Case analysis of Python crawler
Today, the editor will share with you the relevant knowledge points of Python crawler case analysis. The content is detailed and the logic is clear. I believe most people still know too much about this knowledge, so share this article for your reference. I hope you can get something after reading this article. Let's take a look at it.
Environment building
Since you use python, you can't do without a locale. So I downloaded version 3.5 on the official website. After installation, I randomly selected an editor called PyCharm, so there are quite a lot of python editors.
Send a request
Of course, I don't know how python makes web requests, and what's the difference between 2.0 and 3.0, there are a lot of twists and turns in the middle, and finally write the simplest piece of request code.
Import urllib.parse
Import urllib.request
# params CategoryId=808 CategoryType=SiteHome ItemListActionName=PostList PageIndex=3 ParentCategoryId=0 TotalPostCount=4000
Def getHtml (url,values):
User_agent='Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36'
Headers = {'User-Agent':user_agent}
Data = urllib.parse.urlencode (values)
Response_result = urllib.request.urlopen (url+'?'+data). Read ()
Html = response_result.decode ('utf-8')
Return html
# get data
Def requestCnblogs (index):
Print ('request data')
Url = 'http://www.cnblogs.com/mvc/AggSite/PostList.aspx'
Value= {
'CategoryId':808
'CategoryType': 'SiteHome'
'ItemListActionName': 'PostList'
'PageIndex': index
'ParentCategoryId': 0
'TotalPostCount': 4000
}
Result = getHtml (url,value)
Return result
In fact, the request of the blog Park is quite standard, is just suitable for crawling. Because what he returns is a html. (it would be better to return json. )
Data parsing
As mentioned above, BeautifulSoup is used, and the advantage is that you don't have to write the rules yourself, just write according to his grammar, and finally complete the data parsing after many tests. Let's start with a HTML. Then it may look easier to correspond to the following code.
