Shulou(Shulou.com)06/02 Report--

This article introduces you what are the knowledge points of Python crawler, the content is very detailed, interested friends can refer to, hope to be helpful to you.

To do data analysis, like any technology, you should learn with a goal. The goal is like a beacon that guides you forward. Many people learn to learn and give up, in large part because they do not have a clear goal. So, be sure to be clear about the purpose of learning. Before you prepare to learn crawlers, ask yourself why you want to learn crawlers. Some people are for a job, some are for fun, and some are for a cool techs function. But to be sure, learning to crawl can provide a lot of convenience to your work.

As a rookie with zero foundation, it can be divided into three stages.

The first stage is an introduction, mastering the necessary basic knowledge, such as the basics of Python, the basic principles of network requests, etc.

The second stage is imitation, follow other people's crawler code, understand every line of code, familiar with the mainstream crawler tools

The third stage is to do it yourself. at this stage, you begin to have your own ideas for solving problems, and you can design the crawler system independently.

The techniques involved in crawlers include, but are not limited to, proficiency in a programming language (here Python as an example), basic knowledge of HTTP protocols, regular expressions, database knowledge, the use of commonly used packet crawling tools, crawler frameworks, large-scale crawlers, distributed concepts, message queues, commonly used data structures and algorithms, caching, and even machine learning applications. Large-scale systems are supported by a lot of technology. Data analysis, mining, and even machine learning are inseparable from data, and data often need to be obtained through crawlers, so it is promising to learn crawlers as a major.

So do you have to learn all the above before you can start writing about crawlers? Of course not, learning is a lifetime thing, as long as you can write Python code, just start crawlers, like learning to drive, as long as you can start to hit the road, writing code is much safer than driving.

Write crawlers with Python

First of all, you need to know Python, understand the basic syntax, and know how to use functions, classes, list, and common methods in dict. Then you need to understand that HTML,HTML is a document tree structure.

Knowledge of HTTP

The basic principle of a crawler is the process of downloading data from a remote server through a network request, and the technology behind this network request is based on the HTTP protocol. As a starter crawler, you need to understand the basic principles of the HTTP protocol. Although the HTTP specification can not be written in a book, the in-depth content can be read later, combining theory with practice.

The network request framework is the implementation of the HTTP protocol. For example, the famous network request library Requests is a network library that simulates browsers to send HTTP requests. After understanding the HTTP protocol, you can specifically learn network-related modules. For example, Python comes with urllib, urllib2 (urllib in Python3), httplib,Cookie and so on. Of course, you can skip these directly and learn how to use Requests directly, provided that you are familiar with the basic contents of the HTTP protocol and the data climbs down. Most of the cases are HTML text, and a few are based on XML format or Json format. To deal with these data correctly, you need to be familiar with the solution of each data type, for example, JSON data can directly use Python's own module json, for HTML data, you can use BeautifulSoup, lxml and other libraries to deal with, for xml data, in addition to using untangle, xmltodict and other third-party libraries.

Crawler tool

In crawler tools, learn to use Chrome or FireFox browsers to review elements, track requests, and so on. Most websites now have addresses with APP and mobile browsers, so it's relatively easy to give priority to these interfaces. And the use of proxy tools such as Fiddler.

For starter crawlers, it is not necessary to learn regular expressions. You can learn them when you really need them. For example, after you crawl the data back, you need to clean the data. When you find that it is impossible to deal with it using conventional string manipulation methods, you can try to understand regular expressions, which can often get twice the result with half the effort. Python's re module can be used to process regular expressions.

Data cleaning

Data cleaning eventually requires persistent storage, you can use file storage, such as CSV files, you can also use database storage, simply use SQLite, professional point to use MySQL, or distributed document database MongoDB, these databases are very friendly to Python, have off-the-shelf library support, all you need to do is to be familiar with how to use these API.

The way to advance

The basic process from data crawling to cleaning to storage is over, and then it's time to test your internal skills. Many websites have anti-crawler strategies. They try their best to prevent you from obtaining data by abnormal means. For example, there will be all kinds of strange CAPTCHAs that restrict your request operation, limit the speed of your request, restrict IP, or even encrypt the data. Is to increase the cost of obtaining data. At this point, you need to know more, you need to have an in-depth understanding of the HTTP protocol, you need to understand common encryption and decryption algorithms, you need to understand the cookie,HTTP proxy in HTTP, the various HEADER in HTTP. Reptiles and anti-reptiles are a couple who fall in love and kill each other, vice rises ten.

There is no established and unified solution to how to deal with anti-reptiles, which depends on your experience and your knowledge system. This is not a height that can be achieved with just a 21-day introductory tutorial.

For large-scale crawlers, we usually start with a URL, and then add the parsed URL links in the page to the URL collection to be crawled. We need to use queues or priority queues to distinguish some websites from climbing first and others from climbing later. For each page you climb, whether to use the depth-first or breadth-first algorithm to climb the next link. Each time a network request is initiated, a DNS resolution process (converting the URL into IP) is involved. In order to avoid repeated DNS parsing, we need to cache the parsed IP. There are so many URL, how to judge which URL has been crawled and which have not, the simple point is to use the dictionary structure to store the URL that has been crawled, but if you have encountered a large amount of URL, the dictionary takes up a lot of memory space, so you need to consider using Bloom Filter (Bloom filter) to crawl data one by one, which is pitifully inefficient. If you improve the efficiency of the crawler, use multi-thread. Multi-process, cooperative process or distributed operation need to be practiced over and over again.

On the Python crawler knowledge points are shared here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.