Shulou(Shulou.com)06/02 Report--

This article shows you what the principle of Python crawler is, the content is concise and easy to understand, it can definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

1. The principle of network connection

As shown above, to put it simply, a network connection is a request initiated by a computer, and the server returns the corresponding HTML file. The request header and message body to be crawled are explained in detail.

2. Crawler principle

The principle of crawler is to simulate that the computer initiates a Request request to the server, receives the Response content of the server and parses it, and extracts the required information.

Often, one request can not completely get the information data of all web pages, and then it is necessary to reasonably design the crawling process to achieve multi-page and cross-page crawling.

What is the process of multi-page crawling?

Basic ideas:

1. Since the multi-page structure may be similar, you can manually turn the page to observe the URL.

2. Get all the URL

3. Crawl data according to the URL definition function of each page

4. Circular URL crawling storage

What is the process of cross-page crawling?

Basic ideas:

1. Find all URL

2. Define the function code for crawling the detail page

3. Go to the detail page to get detailed data

4. Storage, cycle completion, end

3. What is the web page like?

Right select "check", open the source code of the web page, you can see that the above is the HTML file, below is the CSS style, in which HTML contains the part is the JavaScript code.

The web page we browse is the result of the browser rendering, which is the page interface obtained by translating the HTML, CSS and JavaScript code. There is a popular analogy: joining the web page is a house, HTML is the frame and pattern of the house, CSS is the soft style of the house, such as floor and paint, and javaScript is electrical appliances.

If you open Baidu search, move the mouse to the "Baidu once" button location, right-click to select "check", you can see the page source location.

Or directly open the right-click source code, by clicking on the upper left corner of the source page mouse icon, and then move to the specific location of the page, you can see.

The above is what the principle of Python crawler is, have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.