How the python crawler bypasses the CloudFlare 5 second shield
This article mainly explains how the python crawler bypasses the CloudFlare 5-second shield. Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how the python crawler bypasses the CloudFlare 5 second shield.
Skillfully break the Cloudflare 5-second shield
I believe that the following interface will be familiar to everyone. [figure 1-1]
Figure 1-1
When we visit a site that uses CloudFlare acceleration for the first time, there is a prompt asking us to wait 5 seconds. When we need to crawl such a site, how should we crawl it?
Analysis request
First of all, we need to analyze what the browser did during this waiting time.
By grabbing the packet, we can see that while waiting, the browser made the following three requests [figure 1-2]-[figure 1-4]:
[figure 1-2] request 1 to write cookie field _ _ cfduid
Figure 1-2
[figure 1-3] request 2 request parameters with suspected encryption and write to cookie field cf_clearance
Figure 1-3
[figure 1-4] request 3 returns the home page of the cookie request website written earlier.
Figure 1-4
Now that the requests needed for the whole process have been analyzed, the next step is to use Python to implement the request process, but in this case it is not worthy of the title.
First, let's talk about how the capture bypass is implemented according to the normal process:
Use browser simulation technology to request target websites, such as Selenium, PhantomJS, etc.
Crack the encryption parameters of request 2 and use the request library to simulate the whole request process.
Of course, these two methods can be crawled, but neither of them accords with the skillful words of the title.
Next, I'd like to introduce you to cloudflare-scrape, a Python library developed specifically to bypass this CloudFlare.
With it, you can crawl websites that use CloudFlare, and it is very easy to use this library.
Installation
Use pip install cfscrape to install cloudflare-scrape and confirm that the node.js development environment is installed locally, and if not, you need to install and configure the nodejs development environment.
Use an example
CloudFlare that processes get requests
Import cfscrape
# instantiate a create_scraper object
Scraper = cfscrape.create_scraper ()
# request error, you can add delay
# scraper = cfscrape.create_scraper (delay = 10)
# get the source code of the web page
Web_data = scraper.get ("https://wallhere.com/").content
Print (web_data)
CloudFlare that processes post requests
# import cfscrape
# instantiate a create_scraper object
Scraper = cfscrape.create_scraper ()
# get the source code of a real web page
Web_data = scraper.post ("http://example.com").content
Print (web_data)
The whole request process is as smooth as silk after using cloudflare-scrape.
At this point, I believe you have a deeper understanding of "how the python crawler bypasses the CloudFlare 5-second shield". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!
