Shulou(Shulou.com)06/02 Report--

This article mainly introduces "how to create a Python-based crawler". In daily operation, I believe many people have doubts about how to create a Python-based crawler. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "how to create a Python-based crawler"! Next, please follow the editor to study!

What is web crawling?

This is a way to extract information from a website. A HTML page is nothing more than a collection of nested tags. Tags form some kind of tree whose roots are in the tag and divide the page into different logical parts. Each tag can have its own descendants (children) and parents.

For example, the HTML page tree could look like this:

To handle this HTML, you can use text or a tree. Bypassing this tree is a web crawl. We will only find the nodes we need in all this diversity and get information from them! This approach focuses on converting unstructured HTML data into easy-to-use structured information into a database or worksheet. Data crawling requires a robot to collect information and connect to the Internet through a HTTP or Web browser. In this guide, we will use Python to create a scraper.

What we need to do:

Get the URL of the page from which we want to crawl data

Copy or download the HTML content of this page

Process this HTML content and get the required data

This sequence allows us to pop up the URL we need, get the HTML data, and then process it to receive the data we need. But sometimes we need to go to the website and then go to a specific URL to receive the data. Then we have to add one more step-log on to the site.

Matching

We will use the Beautiful Soup library to analyze the HTML content and get all the necessary data. This is an excellent Python package for crawling HTML and XML documents.

The Selenium library will help us get the crawler into the site and go to the desired URL address in a session. Selenium Python can help you perform operations such as clicking buttons and typing content.

Let's delve into the code

First, let's import the library that will be used.

# Import library from selenium import webdriver from bs4 import BeautifulSoup

Then we need to show the browser driver how Selenium launches the web browser (we'll use Google Chrome here). If we do not want the robot to display the graphical interface of the Web browser, we will add the "headless" option to the Selenium.

Web browsers without a graphical interface (headless) can automatically manage web pages in an environment very similar to all popular Web browsers. In this case, however, all activities are carried out through the command line interface or using network communication.

# path to the chrome driver chromedriver ='/ usr/local/bin/chromedriver' options= webdriver.ChromeOptions () options.add_argument ('headless') # open a headless browser browser = webdriver.Chrome (executable_path=chromedriver, chrome_options=options)

After setting up the browser, installing the library, and creating the environment, we started using HTML. Let's go to the input page and find the identifier, category, or field name in which the user must enter an email address and password.

# enter the login page browser.get ('http://playsports365.com/default.aspx') # search by name for the tag email = browser.find_element_by_name (' ctl00 $MainContent$ctlLogin$_UserName') password = browser.find_element_by_name ('ctl00 $MainContent$ctlLogin$_Password') login = browser.find_element_by_name (' ctl00 $MainContent$ctlLogin$BtnSubmit')

We will then send the login data to these HTML tags. To do this, we need to press the action button to send the data to the server.

# add login credentials email.send_keys ('*') password.send_keys ('*') # Click submit button login.click () email.send_keys ('*') password.send_keys ('*') login.click ()

After successfully entering the system, we will go to the required page and collect the HTML content.

# after successfully logging in, go to the "OpenBets" page browser.get ('http://playsports365.com/wager/OpenBets.aspx') # get HTML content requiredHtml = browser.page_source

Now, when we have HTML content, the only thing left is to process the data. We will do this with the help of the Beautiful Soup and html5lib libraries.

Html5lib is a Python software package that implements HTML5 crawling algorithms influenced by modern Web browsers. Once you have a standardized structure of the content, you can search for data in any child element of the HTML tag. The information we are looking for is in the table tag, so we are looking for it.

Soup = BeautifulSoup (requiredHtml, 'html5lib') table = soup.findChildren (' table') my_table = table [0]

We will find the parent tag once, then iterate through the child tag recursively and print out the value.

# receive label and print values rows = my_table.findChildren (['th',' tr']) for row in rows: cells = row.findChildren ('td') for cell in cells: value = cell.text print (value)

To execute this program, you will need to install Selenium,Beautiful Soup and html5lib using pip. After installing the library, the command is as follows:

# python

These values will be printed to the console, which is how you crawl any Web site.

If we crawl sites that update content frequently (for example, sports scores), we should create a cron task to start the program at a specific time interval.

Everything is fine, everything is fine, the content is crawled, the data is populated, everything is fine except that, this is the number of requests we want to get the data.

Sometimes, the server gets tired of making a bunch of requests from the same person, and the server forbids it. Unfortunately, people's patience is limited.

In this case, you must hide yourself. The most common causes of prohibition are 403 errors and frequent requests to the server when the IP is blocked. The server throws a 403 error when the server is available and able to process requests, but refuses to do so for some personal reasons. The first problem has been solved-we can use html5lib to generate fake user agents to pretend to be human and pass a random combination of operating systems, specifications, and browsers to our requests. In most cases, this is a good way to accurately collect the information you are interested in.

But sometimes it's not enough to just put time.sleep () in the right place and fill in the request header. Therefore, you need to find a powerful way to change this IP. To grab a large amount of data, you can:

-develop your own IP address infrastructure

-use Tor-this topic can be devoted to several large articles, but has actually been completed

-use a network of commercial agents

For beginners of network crawling, the best option is to contact a proxy provider, such as Infatica, who can help you set up a proxy and solve all the difficulties in proxy server management. Collecting large amounts of data requires a lot of resources, so there is no need to "reinvent the wheel" by developing your own internal infrastructure to act as proxies. Even many of the largest e-commerce companies use agent network services to outsource agent management, because the first priority of most companies is data, not agent management.

At this point, the study on "how to create a Python-based crawler" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.