Shulou(Shulou.com)06/02 Report--

This article mainly explains the "PHP Swoole multi-process reading large file example", the article explains the content is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in-depth, together to study and learn "PHP Swoole multi-process read large file example" it!

PHP read large file source code example, through PHP to read too large, super-large file ideas and solutions.

When reading a file daily, if the file is not very large, you can usually use file_get_contents to load the content into a variable at one time, or you can load a web page or a remote file remotely.

If you load a file process that exceeds the PHP limit or exceeds the native memory size, it will report an error or crash.

To solve this problem, we use the principle of finished use and release to read large files.

Single thread read in

If you do not consider the case of multithreading, single-thread reading of large files can be achieved using while fread.

The code is as follows

$handle = fopen (". / big.txt", "rb"); while (! feof ($handle)) {$contents = fread ($handle, 8192); / / Business processing unset ($contents); / / release the variable} fclose ($handle)

Feof determines whether to reach the end of the file. If it does not reach the end of the file, it will while the loop all the time and perform the read operation. Read 8192 bytes at a time and release them after use.

Often many files are not one-line, there are multiple lines of content. You need to read each line as if it were a piece of data, or you can use fgets.

This is the following code:

$handle = fopen (". / big.txt", "rb"); while (! feof ($handle)) {$contents = fgets ($handle, 1024); / / Business processing unset ($contents); / / release the variable} fclose ($handle)

The second parameter of fgets here defaults to 1024 bytes. That is, a row of data is read by default, and if the row of data is less than 1024 bytes, it is fully read. If it exceeds 1024 bytes, only the first 1024 bytes are taken. Encounters the newline character "\ n" or "\ r\ n" or the Terminator will stop reading.

If you encounter abnormal files, many lines are only 1000 long, a line is 8000 long, if not clear, it is difficult to control, if you want to read completely, you need to specify the maximum bytes to read, 8000 to read each line completely.

Another way is to handle newline characters yourself. 1024 bytes are read by default, then placed in memory, and then released after use. This saves memory, but you need to handle newline characters yourself in logical processing.

For example, if you read 1024 bytes without a newline character, save the data and continue reading. Read another 1024 bytes to determine whether there is a newline character, and if so, process the data from the beginning to the newline character. Then save the rest of the data and wait for the next read until the entire file is read.

$handle = fopen (". / big.txt", "rb"); $contents = ""; while (! feof ($handle)) {$contents. = fread ($handle, 8192); / / determine whether the read contains a newline character, and enter the loop body while (strpos ($contents, "\ n")! = = false) {$eol_pos= strpos ($contents, "\ n"); $line = substr ($contents, 0, $eol_pos) / / $line is a row of data for business processing and releases unset ($line); $contents = substr ($eol_pos, 0); / / places the rest in a variable for next use}} fclose ($handle); multithread reads

There is no multithreading by default in PHP, which can be implemented in a multi-process way, or multi-process in swoole.

For example, read a 8GB file, divided into eight threads, each thread reads the 1GB data content. Or more threads to split the work content.

Get file size

The first step is to get the size of the entire file, and then calculate what each thread should be responsible for processing.

Function length ($filename) {$handle = fopen ($filename, "rb"); $currentPos = ftell ($handle); fseek ($handle, 0, SEEK_END); $length = ftell ($handle); fseek ($handle, $currentPos); / / $length file total length return $length;} echo length (". / big.txt"); processing logic implementation process

Here is the main description of what has been done, the first is to set the total number of threads allocated. Then, according to the set number of threads, calculate the size of the data to be read by each thread, that is, where to start and where to end.

Then there is bound to be a split and read the incomplete line, here to solve the unexpected situation of the first half or the second half of the row.

The solution logic is to assume that the reading position at the beginning of the thread is in the middle of a line, and we move one character forward to the last newline character (or maybe the beginning) to get the entire line of text.

After the remnant data is disposed of, yield is used to pass the data to the business process.

$filename = ". / big.txt"; $maxProcess = 8 maxProcess / allocate 8 threads $length = length ($filename); $singleProcessLength = ceil ($length / $maxProcess); / / content read by the thread function processRead ($filename, $index, $singleProcessLength) {$fh = fopen ($filename,'r'); $beginPos = $index * $singleProcessLength / / end position = thread sequence * thread processing data length + thread processing data-1 (length to pointer, actual end pointer is less than end length) $endPos = $index * $singleProcessLength + $singleProcessLength-1; fseek ($fh, $beginPos); echo 'thread:'. $index. ', starting position:'. $beginPos. ', end position:'. $endPos. PHP_EOL; / / move to the previous\ nto get the entire line while (fseek ($fh,-1, SEEK_CUR) = 0) {if (fread ($fh, 1) = = "\ n" | | ftell ($fh)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.