Shulou(Shulou.com)06/01 Report--

This article is about how Python implements a simple Web crawler. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Brief introduction:

Web crawler (also known as web spider), web robot, is a program or script that automatically grabs information according to certain rules. Assuming that the Internet is a large spider web, and each page is connected to each other through the line of hyperlink, then our reptile Mini Program will be able to constantly search for new pages through these lines.

Python is an interpretive, object-oriented and powerful high-level programming language that represents the idea of simplism. Its syntax is simple and has dynamic data types and high-level abstract data structure, which makes it have good cross-platform characteristics, especially suitable for the implementation of programs such as crawlers. In addition, Python also provides a crawler framework such as Spyder, BeautifulSoup parsing framework, which can easily develop a variety of complex crawler programs.

In this article, a simple web crawler is implemented using Python's native urllib and BeautifulSoup libraries to crawl each URL address and its corresponding title content.

Process:

The crawler algorithm uses a URL read from the input as the initial address and sends a Request request to that address.

The requested address returns an all-content object, which is stored in a String variable, which is used to instantiate a BeautifulSoup object that can parse the content into a DOM tree.

Build regular expressions according to your needs, and finally parse out the needed content and the new URL with the help of HTML tags, and put the new ones in the queue.

For the current URL address and crawled content, an index will be established after certain filtering and sorting, which is a word-page storage structure. When the user enters the search sentence, the corresponding word segmentation function will decompose the sentence to get keywords, and then find the corresponding URL according to each keyword. Through this structure, the address list corresponding to this word can be quickly obtained. Using the storage method of tree structure here, the dictionary and list types of Python can better build a word dictionary tree.

The current URL address is popped out from the queue. Under the condition that the crawling queue is not empty, the algorithm keeps getting new web addresses from the queue, and repeats the above process.

Achieve:

Environment:

Python3.5orAnaconda3

BeautifulSoup4

You can install BeautifulSoup4 using the following directive. If you are a Ubuntu user, remember to add sudo before the command:

The program implements several classes, which are used for URL address management, Html content request, Html content parsing, index building and crawler main process. I explain the whole program separately according to each Class, and finally I just put them together and the code can be executed.

UrlManager class

This class is used to manage URL addresses, new_urls is used to store uncrawled URL addresses, old_urls holds crawled addresses, and both variables use the set type to ensure the uniqueness of the contents. In each loop, add_new_urls () provides a way to add a new urls to the new_urls variable; the add_new_url () method, which repeats each url address, and only those that meet the criteria; get_urls (), which provides a way to get a new url address; and the has_new_url () method, which is used to check whether the crawl queue is empty.

HtmlDownloader class

This class implements the method of sending a Request request to the url address and getting its response, which can be achieved by calling the download () method within the class. What we should pay attention to here is the coding of the page. Here I use UTF-8 to decode decode, and some web pages may use GBK encoding, which should be modified according to the actual situation.

HtmlParser class

This class parses the page by instantiating a BeautifulSoup object. It is an HTML/XML document parser written in Python. It provides users with the data they need to grab by parsing the document into a DOM tree, and provides some simple functions to deal with navigation, search, modify the analysis tree and other functions.

The keys to this class are _ get_new_urls (), _ get_new_content (), and get_url_title (). The first method is used to parse the hyperlinks contained in the page, and the most important thing is to select the tag to be parsed and construct an appropriate regular expression for it. Here I define a matching rule for the a tag, which is used to get all the intra-site links, as follows:

The next two classes get the title method by parsing the Html tag, and finally get the title content by calling _ get_new_content () in parse (). The specific method of tag access will not be discussed in detail, and readers can flip through the official documents of BeautifulSoup themselves.

BuildIndex

This class establishes an index relationship between each URL address and the keywords contained in its title and stores them in a Dict variable. Each title corresponds to multiple keywords, and each title also corresponds to multiple url addresses, so each keyword also corresponds to multiple url addresses, as shown below:

Index= {'keyword': [url1,url2,...,urln],...}

Where the add_page_index () method performs word segmentation on each title, and calls the add_key_index () method to store the keyword-url correspondence in the index, which is also checked repeatedly. The idea is that this method of word segmentation is limited to English sentences. Chinese words need to use specific word segmentation tools.

SpiderMain

This is the theme class of the crawler, which implements the operation of the crawler by calling objects generated by several other classes. When this class is instantiated, the objects of the above classes will be generated permanently. When the url address provided by the user is obtained through the craw () method, the request, download, parsing, and indexing will be done in turn. Finally, the method returns two variables index,graph, which are:

The corresponding address for each keyword set, the keyword-urls index, is as follows

Index= {'keyword': [url1,url2,...,urln],...}

The urls,url-suburls indexes contained in each url and its pages are as follows

Graph= {'url': [url1,url2,...,urln],...}

Finally, we can successfully execute our crawler by adding the following code to the program

Thank you for reading! This is the end of this article on "how to achieve a simple Web crawler in Python". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it out for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.