How to deal with the problem that phpQuery takes up too much memory
The main content of this article is "how to deal with the problem that phpQuery takes up too much memory". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to deal with the problem that phpQuery takes up too much memory.
PhpQuery is an open source project similar to jQuery implemented in php that parses web page elements in the form of jQuery syntax on the server side. PhpQuery is much more convenient to use than regular or other ways to match web pages.
When using phpQuery to collect web pages, there is a problem: after processing a large number of web pages, the amount of memory consumed by phpQuery is staggering (it soon exceeds 1G).
For example, this code:
The copy code is as follows:
While (true) {
PhpQuery::newDocumentFile ($htmlFile)
/ / processing web page elements.
Echo memory_get_usage (). "\ n"
}
Run the above code carefully and it will quickly run out of memory.
After looking at the source code of phpQuery, we finally found the problem. PhpQuery will generate a DOMDocumentWrapper object for each page it processes, and each DOMDocumentWrapper object will be saved in the static member $documents (phpQuery::createDocumentWrapper). This variable is an array, and each page array element is parsed by an additional one.
PhpQuery::$documents [$wrapper- > id] = $wrapper
Once the problem is found, it is easy to solve it. After parsing one web page at a time, just leave the phpQuery::$documents blank.
The copy code is as follows:
While (true) {
PhpQuery::newDocumentFile ($htmlFile)
/ / processing web page elements.
PhpQuery::$documents = array ()
Echo memory_get_usage (). "\ n"
}
Memory footprint has stabilized.
At this point, I believe you have a deeper understanding of "how to deal with the problem of phpQuery taking up too much memory". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!
