Shulou(Shulou.com)06/02 Report--

This article introduces how to use Redis to achieve lightweight delay queue in PHP, the content is very detailed, interested friends can refer to, I hope it can be helpful to you.

I. background

Let's take a look at the business scenario:

1. Send a recall notice 3 days before the expiration of the member

two。 After the payment of the order is successful, check whether all the downstream links are normal after 5 minutes, such as whether all kinds of member status are set successfully after the user buys the member.

3. How to check periodically whether the refund order has been refunded successfully?

4. Failed to realize the notification. Repeat the notification for 7 minutes until the other party replies.

Usually, the simplest and most direct way to solve the above problems is to scan the meter regularly.

The problems with table scanning are:

1. Scan the table to connect with the database for a long time, in the case of a large number of connections, it is easy to break the connection, which requires more exception handling and high requirements for program robustness.

two。 In the case of a large amount of data, the delay is high, which can not be processed within the regulations, which affects the business, although multiple processes can be started to deal with it, which will bring additional maintenance costs and cannot be solved fundamentally.

3. Each business maintains its own table scanning logic. As there are more and more businesses, it is found that the logic of the table scan part will be developed repeatedly, but very similar.

The delay queue can solve the above requirements very well.

Second, investigation and research

We have investigated some open source solutions in the market, as follows:

1. There are likes of technology: only principles, no open source code

2.github 's personal: https://github.com/ouqiang/delay-queue

1. Based on the redis implementation, only one redis can be configured. If the redis fails, the whole service is not available, and the availability is poor.

two。 The consumer side implements the pull mode, the access cost is high, and each project has to implement the access code.

3. The number of people who use star is small, it is risky to put it in the production environment, and because of the lack of knowledge of go language, it is difficult to maintain.

3. Scheduler X- Ali open source: powerful, but complex operation and maintenance, many dependent components, not light enough

4. RabbitMQ-delay task: there is no delay function and needs to be implemented by itself with a feature. Moreover, the company does not deploy this queue, so it is a bit expensive to deploy one alone to do delay queue, and it also requires special operation and maintenance staff to maintain it. Currently, the team does not support it.

For the above reasons, I plan to write one myself. I usually use the zset structure of php and the basic redis of the project as storage, which is implemented in php language. The implementation principle refers to the likes of the team: https://tech.youzan.com/queuing_delay/.

The whole delay queue is mainly composed of four parts.

JobPool is used to store meta-information for all Job.

DelayBucket is a set of ordered queues based on time, which are used to store all Job that need to be delayed (here only Job Id is stored).

Timer is responsible for scanning each Bucket in real time and putting the Job whose delay time is greater than or equal to the current time into the corresponding Ready Queue.

ReadyQueue stores Job in the Ready state (only JobId is stored here) for consumption by consumer programs.

Message structure

Each Job must contain the following attributes:

Topic:Job type. Can be understood as a specific business name.

Unique identification of the id:Job. Used to retrieve and delete specified Job information.

DelayTime:jod delayed execution time, 13-bit timestamp

Ttr (time-to-run): Job execution timeout.

Body:Job content, for consumers to do specific business processing, stored in json format.

For the same type of topic delaytime,ttr, it is generally fixed. Job can simplify the attributes.

1.topic:Job type. Can be understood as a specific business name

Unique identification of the 2.id:Job. Used to retrieve and delete specified Job information.

3.body:Job content, for consumers to do specific business processing, stored in json format.

Delaytime,ttr is configured in the topicadmin background

III. Objectives

Lightweight: it can be run directly with fewer extensions of php, without the introduction of network frameworks such as swoole,workman

Stability: master-work architecture is adopted. Master does not do business processing and is only responsible for managing child processes. When a child process exits abnormally, it will be pulled up automatically.

Availability:

1. Multi-instance deployment is supported. Each instance is stateless, and the service is not affected by the failure of one instance.

two。 Multiple redis can be configured. Only some messages will be affected if a redis fails.

3. It is convenient for the business side to access, and you only need to fill in the relevant message types and callback interfaces in the background.

Scalability: when there is a bottleneck in the consumption process, you can configure to increase the number of consumption processes. When there is a bottleneck in writing, the number of instances can be increased and the writing performance can be linearly improved.

Real-time: a certain time error is allowed.

Message deletion is supported: the business user can delete the specified message at any time.

Message transmission reliability: after the message enters the delay queue, it is guaranteed to be consumed at least once.

Write performance: qps > 1000 +

IV. Architecture design and explanation

Overall architecture

The master-work architecture model is adopted, which mainly includes 6 modules:

1.dq-mster: the main process, responsible for managing the creation, destruction, recycling and signaling notification of child processes

2.dq-server: responsible for message writing, reading, deleting and maintaining redis connection pool

3.dq-timer-N: responsible for scanning expired messages from redis's zset structure and writing to ready queues. The number can be configured, usually 2, because messages are ordered in time in zset structure.

4.dq-consume-N: responsible for reading messages from the ready queue and notifying the corresponding callback interface. The number can be configured.

5.dq-redis-checker: responsible for checking the service status of redis and sending alarm email if redis is down.

6.dq-http-server: provides a web background interface for registering topic

Fifth, module flow chart

Message write:

Timer looks for expiration messages:

Consumer consumption process:

VI. Deployment

Environmental dependency: PHP 5.4 + installs sockets,redis,pcntl,pdo_mysql extension

Ps: students who are familiar with docker can directly use the mirror image: shareclz/php7.2.14 contains the required extensions.

Step1: install the database to store some topic and alarm information

Execute:

Mysql > source dq.sql

Step2: in DqConfg. Configuration database information in file: DqConf::$dbstep3: start the http service

Modified the path of php in the DqConf.php file

Command:

Php DqHttpServer.php-port 8088

Access: configuration interface appears in http://127.0.0.1:8088,

Image

Redis message format: host:port:auth, for example, 127.0.0.1, 6379, 12345.

Stop4: configuration notification information (such as redis downtime)

Stop5: register topic

Retry tag description:

1. The interface returns empty default retry 2. The returned expression will be retried if the specified expression is met. Res indicates the returned json array. For example, the callback API returns the json string: {"code": 200," data ": {" status ": 2," msg ":" return failure "}}. The retry condition can be written as follows: {res.code}! = 200 {res.code}! = 200 & & {res.data.status}! = 2 {res.code} = & & {res.data.status} = = 2 | {res.data.msg} = = 'return failure'

Step6: start the service process:

Php DqInit.php-- port 6789 &

Execute ps-ef | grep dq sees the following message indicating that the startup is successful

Step7: write data. Refer to demo.phpstep8: view logs

The default log directory is under the logs directory of the project directory. Modify $logPath in DqConf.php.

1. Request log: request_ymd.txt

two。 Notification log: notify_ymd.txt

3. Error log: err_ymd.txt

Step9: if the configuration file is changed

1. The system automatically detects that the configuration file is new, and automatically exits if there are any changes (no better hot update solution is found). If you need to restart, you can create a task in crontab and execute it every minute. The program has the judgment of check_self.

two。 Elegant exit command: master detects and listens to the USR2 signal, and notifies all child processes after receiving the signal. The child process automatically exits after completing the current task.

Ps-ef | grep dq-master | grep-v grep | head-n 1 | awk'{print $2}'| xargs kill-USR2

VII. Performance testing

Pthreads extension needs to be installed:

Testing principle: using multi-thread to simulate concurrency, the number of successful requests can be returned successfully within 1 second.

8. Performance optimization points worth mentioning:

1.redis multi command: package multiple operations on redis into one to reduce network overhead

two。 The operation of counting is handled asynchronously, which is saved with the static variable of the function in the asynchronous logic. When the static variable is released after the redis is successfully written, the count can be kept consistent when an exception occurs in the redis, unless the process exits.

3. Memory leak detection is necessary: all memory allocations are called brk or mmap at the bottom. As long as the program has only a large number of brk or mmap system calls, the possibility of memory leak is very high. Detection command: strace-c-p pid | grep-P 'mmap | brk'

4. Check the system call of the program: strace-c-p pid, and find that one system function call is several times higher than the other, and there may be a problem with the high probability program.

IX. Exception handling

1. If the notification API is called within the timeout period and no reply is received, the system will queue the data again and notify again. The system defaults to a maximum of 10 notifications (you can modify $notify_exp_nums in the Dqconf.php file) the notification interval is 2n+1, such as the first one minute, the notification fails, and the second three minutes later, until the reply is received. After exceeding the maximum number of notifications, the system discards automatically. Send an email notification at the same time

two。 Online redis is persisted every 1s, and 1s data is lost. If this happens, you can recover it manually compared with request_ymd.txt and notify_ymd.txt logs.

3.redis downtime Notification:

Ps: network jitter is inevitable. If the notification interface involves core services, it must be idempotent!

10. Online situation

Two instances are deployed online, one for each engine room, and four redis with a total of 16G memory for storage. The service has been running steadily for several months, and all the indicators are in line with expectations.

Main access services:

Order 10-minute recall notice

Make compensation when calling the interface times out or fails

Recall notice 3 days prior to expiration of member

Eleventh, deficiency and Prospect

1. Due to the lack of libevent extension of the images used by the team, dq-server is based on the select model, and there is a bottleneck in the performance of scenarios with high concurrency. Later, it can be based on the libevent event model to improve concurrency performance.

2.timer and consumer are currently done by multi-processes. This granularity feels a bit coarse, so we can consider using multi-thread mode, and support the dynamic creation of the number of threads to improve the performance of consumer, so as to ensure timely consumption.

3.dq-server and redis are synchronous calls, which is also the bottleneck of performance. It is planned to process asynchronously based on swoole_redis.

On how to use Redis in PHP to achieve lightweight delay queue is shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.