Shulou(Shulou.com)06/01 Report--

'connections' = > [.... 'database' = > [' driver' = > 'database',' table' = > 'jobs',' queue' = > 'default',' expire' = > 60,], 'redis' = > [' driver' = > 'redis',' connection' = > 'default',' queue' = > 'default',' expire' = > 180 ],.... ]

Laravel5.2 queue driver config/queue.php profile, "database" and "redis" have an expire parameter, which is interpreted in the manual as "queue task expiration time in seconds". The default is 60 seconds.

(note: 5.2and later configuration files have been changed to 'retry_after' parameters, see manual for details)

Searching for this configuration on the Internet, there is not much explanation, but in the process of actual use, it is found that this parameter and this design pattern are a big pit for queue processes that run longer than the time set by expire, and for using queues for distributed program deployment.

The problem was found to be using distributed application deployment processing queues, two servers deploying Laravel framework artisan scripts, connecting to an MYSQL database, and using a list of jobs teams.

After deployment, start the scripts of the two servers respectively, find the scripts executed after discovery, fetch data from the queue driver, such as the jobs table of MYSQL, and will not skip the queue data when you encounter the first executed script queue data, but regard this data as Failed, and store a new piece of data into the failed_ JobsTable (when the Laravel queue fails, the queue data will be stored in the failed_jobs table), resulting in data duplication.

This error will not occur when you start three processes to execute a script on a server before, and the script executed after that will not get the queue data of the previous process, let alone be judged as Failed. What is the cause of the data error in the queue driver in multi-service processing?

According to the process of queue execution, when the program is executed, the queue driver fetches the task from the queue driver, and the queue driver that obtains the task should do transaction processing, so that the second process fetching the task will skip the queue data that is being executed.

Check some information, understand the principle of Laravel queue, and finally have to look at the source code of Queue.

The source code of Laravel's Queue is in the Illuminate\ Queue directory.

First analyze the jobs table driven by MYSQL:

CREATE TABLE `jobs` (`id` bigint (20) unsigned NOT NULL AUTO_INCREMENT, `queue` varchar (255) COLLATE utf8_unicode_ci NOT NULL, `payload` longtext COLLATE utf8_unicode_ci NOT NULL, `attempts` tinyint (3) unsigned NOT NULL, `reserved`unsigned NOT NULL (3) unsigned NOT NULL, `reserved_ at`int (10) unsigned DEFAULT NULL, `available_ at`int (10) unsigned NOT NULL, `created_ at`int (10) unsigned NOT NULL, PRIMARY KEY (`id`), KEY `jobs_queue_reserved_reserved_at_ index` (`queue`, `reserved`) `reserved_ at`) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci

The manual mainly introduces the saving of queue tasks, the payload field stores the serialized tasks, the Laravel queue can serialize the data model, and the queue system will automatically obtain the entire model instance from the database during execution, as described in the manual.

But several other status and time fields are the key fields to ensure queue transaction processing.

"attempts" execution times, "reserved" execution status, "reserved_at" execution time, 'available_at' subscription execution time,' created_at' is the queue creation time.

The scripts that listen for events include Listener.php and Worker.php. The source code shows that Listener can handle specified queues and connection parameters, but in fact, queues are handled through work. Laravel5.4 has canceled the queue:listen parameter and uses queue:work to execute it. However, what I'm talking about here is the problem of Laravel5.2. I don't know if the following reasons make Laravel optimization get rid of listen.

Continue to analyze the source code of the Worker class processed by the queue, using the pop method when fetching the queue data, which calls the pop method of the driver depending on the type of driver passed, such as database or redis.

$connection = $this- > manager- > connection ($connectionName); $job = $this- > getNextJob ($connection, $queue); / / If we're able to pull a job off of the stack, we will process it and / / then immediately return back out. If there is no job on the queue / / we will "sleep" the worker for the specified number of seconds. If (! Is_null ($job) {return $this- > process ($this- > manager- > getName ($connectionName), $job, $maxTries, $delay);}

Here is the pop method of DatabaseQueue.php.

/ * Pop the next job off of the queue. * * @ param string $queue * @ return\ Illuminate\ Contracts\ Queue\ Job | null * / public function pop ($queue = null) {$queue = $this- > getQueue ($queue); $this- > database- > beginTransaction (); if ($job = $this- > getNextAvailableJob ($queue)) {$job = $this- > markJobAsReserved ($job); $this- > database- > commit () Return new DatabaseJob ($this- > container, $this, $job, $queue);} $this- > database- > commit ();}

The process of fetching data transaction processing has been turned on.

The core of the queue data is still $this- > getNextAvailableJob ($queue).

Open the sql log to see how the queue data is queried.

/ * Get the next available job for the queue. * * @ param string | null $queue * @ return\ StdClass | null * / protected function getNextAvailableJob ($queue) {$this- > database- > enableQueryLog () $job = $this- > database- > table ($this- > table)-> lockForUpdate ()-> where ('queue', $this- > getQueue ($queue))-> where (function ($query) {$this- > isAvailable ($query); $this- > isReservedButExpired ($query) })-> orderBy ('id',' asc')-> first (); var_dump ($this- > database- > getQueryLog ()); return $job? (object) $job: null;} array (1) {[0] = > array (3) {'query' = > string (3) "select * from `jobs`where `queue` =? and ((`reserved` =? and `available_ at` database- > commit ());" the former sleep (10) will obviously see that the second queue does not acquire other queue data, indicating that "for update" is only an update exclusive lock and will not exclude select.

Laravel sometimes has blocking when using database queues. I don't know if this is the reason.

If the execution time is too long and exceeds the setting time of the 'expire' parameter, the second queue will get the first queue data and judge the timeout. At this time, it will determine whether to insert the new queue data to continue the attempt or insert it into the error queue "failed_jobs" table to judge whether the queue execution failed.

This is the logic of Laravel using mysql to execute queues. The two servers mentioned earlier deploy Laravel framework to execute artisan scripts. The problem of jobs table queue Failed is the reason for inconsistent server time. When the latter server executes, the former queue data is judged as timeout and inserted into "failed_jobs" a new data, which has reached the maximum number of failures, otherwise new data will be inserted to continue to try.

Therefore, the execution time parameter of queue:listen, timeout=60, must be set to be less than the queue task expiration time expire parameter!

Also, Laravel5.2 's queue:work does not have the parameter-- timeout=60.

Finally, there is the processing logic for the completion of the queue.

If the queue executes successfully, the jobs's data will be deleted, which is fine. If it fails, including timeouts, exceptions, etc., it will determine whether to insert a new piece of data based on the set maximum number of failures, or insert a piece of Failed data into the "failed_jobs" table.

When an error occurs, handleJobException's exception handling calls DatabaseQueue.php 's release method, $job- > release ($delay), which is ultimately the pushToDatabase implementation.

When inserting new data, attempts is the number of failures, and reserved is 0 for the timestamp plus timeout for the current timestamp.

In this way, the whole queue processing forms a complete data operation.

Laravel5.4 has made a lot of changes to the queue function, as suggested in the manual

Task expiration and timeout

Task execution time

In the configuration file config/queue.php, each connection has a retry_after entry defined. The purpose of this configuration item is to define how many seconds after execution the task is released back to the queue. If the value set by retry_after is 90 and the task is not finished after 90 seconds of running, it will be released back to the queue rather than deleted. There is no doubt that you need to set the value of retry_after to the maximum possible value of task execution time.

Laravel5.4 removes queue's listen command, and work adds a timeout parameter. Laravel5.5 should have to be upgraded directly when it comes out.

Appendix: Laravel5.2 test scripts, previously found on the Internet are relatively early, or write job into a command way, in fact, after 5.2 job is very easy to use.

Job task is defined under jobs. Handle can add some test scenarios, such as mine, which throws an exception and directly Failed

Class MyJob extends Job implements ShouldQueue {use InteractsWithQueue, SerializesModels; private $key; private $value; / * * Create a new job instance. * * @ return void * / public function _ construct ($key, $value) {$this- > key = $key; $this- > value = $value;} / * Execute the job. * * @ return void * / public function handle () {for. "\ t" .date ("Y-m-d H:i:s"). "\ n"; throw new\ Exception ("test\ n"); / / Redis::hset ('queue.test', $this- > key, $this- > value);} public function failed () {dump (' failed');}}

The controller accesses and sets the task queue. Key and value have previously used the test redis to insert, and can set the job parameters according to their own test scheme.

For ($I = 0; $I

Delay (20); $this- > dispatch ($job);}

My example sets up 5 queues, open multiple shell and run artisan tests concurrently.

I originally wanted to read the redis queue code and send it out together, but recently there are too many things, and the redis code doesn't look much.

Redis driver can refer to http://www.cnblogs.com/z1298703836/p/5346728.html this article on the Laravel queue redis driver logic is very detailed, the redis driver uses list and zset structure to store the queue, the execution process will remove the rollover queue, there is no database "for update" operation, so there should not be queue blocking.

The BUT queue task expiration time setting is the same as the database driver, so the same

The execution time parameter of queue:listen-timeout=60, be sure to set the expire parameter which is less than the expiration time of queue tasks!

I finally finished it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.