Shulou(Shulou.com)06/02 Report--

This article mainly introduces Python how to achieve short video crawler, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, let the editor take you to know about it.

The address of the website is in the code, and you can see it with your heart.

The software used

Python 3.8pycharm 2021.2

Module

Requestsparselreconcurrent.futurestimewarnings

Will not install the module see this article: the installation of the python module and the solution to installation failure

I know you don't want to see those steps, so I'll just go to the code.

Import requestsimport parselimport reimport concurrent.futuresimport timeimport warnings# cancels warning warnings.filterwarnings ("ignore") def get_html (url): "" send a request to get the web page source code "" html_data = requests.get (url=url, verify=False) .text return html_datadef parse_data_1 (html_data): "" parse for the first time Get all the details page links "" selector = parsel.Selector (html_data) url_list = selector.xpath ('/ / a [@ class= "meta-title"] / @ href'). Getall () return url_listdef parse_data_2 (html_data): "" second resolution, get the video link "" video_url = re.findall ('url: "(. *?)",' Html_data) [0] return video_urldef save (video_url): "Save video"title = video_url.split ('/') [- 1] # take the field in the link as the title video_data = requests.get (video_url, verify=False). Content # send network request with open (f'video/ {title}' Mode='wb') as f: f.write (video_data) print (title, "crawled successfully!!") start_time = time.time () url = 'https://www.520mmtv.com/hd/rewu.html'# 1. Send the request html_data = get_html (url=url) # 2 to the target website. First parse data extraction details page link url_list = parse_data_1 (html_data=html_data) for info_url in url_list [: 10]: # 3. Send the request html_data_2 = get_html (url=info_url) # 4 to the details page. The second time parse the data to extract the video playback address video_url = parse_data_2 (html_data=html_data_2) # 5. Save video save (video_url=video_url) print ('take time:', time.time ()-start_time) Thank you for reading this article carefully. I hope the article "how to achieve short Video Crawler" shared by the editor will be helpful to everyone. At the same time, I also hope that you will support and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.