Shulou(Shulou.com)06/03 Report--

This article introduces the relevant knowledge of "Why to write a crawler program to choose Python". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1. An unpredictable web crawler

Once I have written about crawlers, I may have such a feeling that the reptiles that ran well yesterday may have a problem today and will not work. The reason for this may be the revision of the web page, the blockade of the website, and so on. In this situation, we must debug and find out the problem in the fastest time, and fix it as quickly as possible, so that it can be online as soon as possible.

two。 Adaptive Python

In view of the complex changes of the above crawlers, writing web crawlers must rely on a rapid development and flexible language, while being supported by a complete and rich library. And the language that has these advantages at the same time is undoubtedly Python. Therefore, Python is born naturally for reptiles, and reptiles naturally choose Python.

3. Concise and rich Python

Seeing the natural connection between Python and web crawlers, little apes can't help but ask, what are the natural properties of Python that are suitable for web crawlers? Don't worry, listen to the old man come slowly.

3.1 succinct grammar

The syntax of Python is very simple, advocating simplicity rather than simplicity. The philosophy of Python developers is "one way, preferably only one way to do one thing". This philosophy makes the code you write without too much personal style, easy for others to understand your code, and easy for you to understand other people's code. The simplicity of Python also allows developers to implement a function in just a few lines of code, while the same function may take dozens of lines in Java or hundreds of lines in C++.

You can try running import this in the Python interpreter to taste the philosophy of Python:

> > import this > The Zen of Python > by Tim Peters > Beautiful is better than ugly. > Explicit is better than implicit. > Simple is better than complex. > Complex is better than complicated. > Sparse is better than dense. > Readability counts. > Special cases aren't special enough to break the rules. > Although practicality beats purity. > Errors should never pass silently. > Unless explicitly silenced. > In the face of ambiguity Refuse the temptation to guess. > There should be one-- and preferably only one-- obvious way to do it. > Although that way may not be obvious at first unless you're Dutch. > Now is better than never. > Although never is often better than * right* now. > If the implementation is hard to explain, it's a bad idea. > If the implementation is easy to explain, it may be a good idea. > Namespaces are one honking great idea-- let's do more of those!

The concise syntax of Python makes it easy for you to implement and modify crawlers. In other words, write fast! Life is too short, why not Python?

3.2Rich Python modules

You should have heard of the richness of Python modules (libraries), but you may not have the time and opportunity to come into contact with so many. "almost all the functions you want Python have libraries to implement." This sentence may seem arrogant, but it is no problem to meet 90% of your needs. So, we have to remember this sentence, in the future development process, need any basic function, you might as well search, ask, to see if someone has achieved this function, and uploaded to pypi, and you may only be pip install. At the same time, also verify that this sentence is not the case.

For example,

If I want to download the web page, I will use it.

Python standard module urllib.request, and a good third-party open source module requests

Asynchronous http requests include aiohttp

If I want to deal with the URL url, I will use:

The module urllib.parse that comes with Python

If I want to parse html, I use:

Based on the C language library of high efficiency module lxml, easy to use beautifulsoap.

I want to manage the URL and record the status of successful downloads, failed url, and undownloaded URLs, using:

Key-value database leveldb encapsulated by Python

I want to use a mature crawler framework, just:

Scrapy with a long history, rising star pyspider.

To support javascript and ajax, I use:

The browser simulation framework Selenium, plus the famous Google Headless Chrome running on the Linux server without the need for a desktop environment.

There is also a Phantomjs, but development has been stopped.

This is the end of the content of "Why you choose Python to write a crawler". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.