Shulou(Shulou.com)06/02 Report--

This article mainly shows you "how to achieve python crawler", the content is easy to understand, clear, hope to help you solve your doubts, the following let the editor lead you to study and learn "how to achieve python crawler" this article.

One: reptile preparation

1. The first thing a crawler needs to do is to determine the object you want to crawl. Here I will take the address of the logo image on Baidu's home page as an example.

two。 First of all, open the Baidu home page interface, then move the mouse over the Baidu logo icon on the home page interface, click the right mouse button, and then click the review element to open the developer interface.

3. Then in the interface below, you can see the typesetting mode of the logo icon in HTML.

I replaced Baidu here with the word.

Two: start crawling

1. The crawler is mainly divided into two parts, the first is the acquisition of the web page interface, and the second is the parsing of the web page interface; the principle of the crawler is to use the code to simulate the browser to visit the website. unlike the browser, the crawler gets the source code of the web page without the translation effect of the browser.

two。 First of all, we fetch pages. For python crawler, many module packages are provided to developers to crawl web pages directly, urllib,urllib2,requests (urllib3), and so on. Here we use urllib2 to obtain website pages. First, import the urllib2 module package (which is installed by default): import urllib2

3. After importing the module package, call the urlopen method in urllib2 to link to the Web site, with the code repr = urllib2.urlopen ("XXXXXX"), where XXXXXX represents the website name.

4. After getting the response from the website, then read out the source code of the page and call the read method, html = repr.read ()

5. After getting the source code of the page, the next work is to parse the data you want from the source code of the html interface. There are many module packages for the parsing interface, including original re, useful BeautifulSoup, high-end lxml and so on. Here I will briefly introduce it with re. First, import the re module package: import re.

6. Then use re to search, here I have to use regular expressions, students who do not understand need to supplement the knowledge of regular expressions.

7. Then, I have implemented a simple crawler process here, printing url, and you can see that it happens to be the address of the Baidu home page logo that we saw earlier.

8. Source code:

Import urllib2

Repr = urllib2.urlopen ("URL")

Html = repr.read ()

Import re

Omit a line of code

Print url

The above is all the content of this article "how to achieve python crawler". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.