Shulou(Shulou.com)06/03 Report--

This article mainly introduces "what are the necessary skills of Python crawler". In daily operation, I believe many people have doubts about the necessary skills of Python crawler. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful for you to answer the doubts about "what are the necessary skills of Python crawler?" Next, please follow the editor to study!

Custom function import requestsfrom bs4 import BeautifulSoupheaders= {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) Rv:93.0) Gecko/20100101 Firefox/93.0'} def baidu (company): url = 'https://www.baidu.com/s?rtt=4&tn=news&word=' + company print (url) html = requests.get (url, headers=headers). Text s = BeautifulSoup (html,' html.parser') title=s.select ('.news-title_1YtI1 a') for i in title: print (i.text) # batch call function companies = [Tencent' 'Alibaba', 'Baidu Group'] for i in companies: baidu (I)

Batch output the titles of multiple search results

The result is saved as a text file import requestsfrom bs4 import BeautifulSoupheaders= {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) Rv:93.0) Gecko/20100101 Firefox/93.0'} def baidu (company): url = 'https://www.baidu.com/s?rtt=4&tn=news&word=' + company print (url) html = requests.get (url, headers=headers). Text s = BeautifulSoup (html,' html.parser') title=s.select ('.news-title_1YtI1 a') fl=open ('test.text','a' Encoding='utf-8') for i in title: fl.write (i.text +'\ n') # batch call function companies = ['Tencent', 'Alibaba', 'Baidu Group'] for i in companies: baidu (I)

Write code

Fl=open ('test.text','a', encoding='utf-8') for i in title: fl.write (i.text +'\ n') exception handling for i in companies: try: baidu (I) print ('run successfully') except: print ('run failed')

Writing in a loop will not cause the program to stop running, but will output the failure to run.

Dormancy time import timefor i in companies: try: baidu (I) print ('run successful') except: print ('run failed') time.sleep (5)

Time.sleep (5)

The units in parentheses are seconds.

Where to put and where to sleep (pause)

Crawl multiple pages of content

Baidu searches for Tencent

Switch to page 2

Get rid of superfluous ones

Https://www.baidu.com/s?wd= Tencent & pn=10

Analyze out

Https://www.baidu.com/s?wd= Tencent & pn=0 is the first page

Https://www.baidu.com/s?wd= Tencent & pn=10 is the second page

Https://www.baidu.com/s?wd= Tencent & pn=20 is the third page

Https://www.baidu.com/s?wd= Tencent & pn=30 is the fourth page

.

Code from bs4 import BeautifulSoupimport timeheaders= {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) Rv:93.0) Gecko/20100101 Firefox/93.0'} def baidu (c): url = 'https://www.baidu.com/s?wd= Tencent & pn=' + str (c) +' 0' print (url) html = requests.get (url, headers=headers). Text s = BeautifulSoup (html) 'html.parser') title=s.select (' .t a') for i in title: print (i.text) for i in range (10): baidu (I) time.sleep (2)

At this point, the study of "what are the necessary skills of Python crawler" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.