What are the three ways of writing python crawler?
This article introduces the relevant knowledge of "what are the three ways of writing python crawler". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!
What is a reptile?
If we compare the Internet to a large spider's web, the data is stored at each node of the spider's web, and the reptile is a small spider.
Crawlers crawling their prey (data) along the web refer to a program that initiates a request to a website, acquires resources, analyzes and extracts useful data.
From the technical level, it is through the program to simulate the behavior of the browser to request the site, crawl the HTML code / JSON data / binary data (pictures, videos) returned by the site locally, and then extract the data you need and store it for use.
Second, the basic process of the crawler:
How users obtain network data:
Method 1: the browser submits the request-> download the web page code-> parse into the page.
Method 2: simulate the browser to send a request (get the web page code)-> extract useful data-> store it in a database or file
What a crawler has to do is way 2.
1. Initiate a request
Use the http library to initiate a request to the target site, that is, send a Request
Request includes: request header, request body, etc.
Request module bug: unable to execute JS and CSS code
2. Get the response content
If the server responds normally, you will get a Response
Response includes: html,json, pictures, videos, etc.
3. Analyze the content
Parsing html data: regular expressions (RE modules), third-party parsing libraries such as Beautifulsoup,pyquery, etc.
Parsing json data: json module
Parsing binary data: writing to a file in wb
4. Save data
Database (MySQL,Mongdb, Redis)
File
This is the end of the content of "what are the three ways to write python crawler". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!
