Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "how Python uses Requests to capture package map network videos". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

The target grabs the site-wide video data and saves it locally, and names the video file after the video name of the website.

Analyze the data structure of web pages

After analysis, we can find the data of the main station, we can start with these four options.

Analyze the data format of web pages

Image.png

The web page data is static

Grab the link to the next page

Grab the link to the next page

OK, get the code!

Import requests

From lxml import etree

Import threading

Class Spider (object):

Def _ init__ (self):

Self.headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"}

Self.offset = 1

Def start_work (self, url):

Print ("crawling page d." self.offset)

Self.offset + = 1

Response = requests.get (url=url,headers=self.headers)

Html = response.content.decode ()

Html = etree.HTML (html)

Video_src = html.xpath ('/ / div [@ class= "video-play"] / video/@src')

Video_title = html.xpath ('/ / span [@ class= "video-title"] / text ()')

Next_page = "http:" + html.xpath ('/ / a [@ class= "next"] / @ href') [0]

# crawl completed.

If next_page = "http:":

Return

Self.write_file (video_src, video_title)

Self.start_work (next_page)

Def write_file (self, video_src, video_title):

For src, title in zip (video_src, video_title):

Response = requests.get ("http:" + src, headers=self.headers)

File_name = title + ".mp4"

File_name = ". Join (file_name.split (" / "))

Print (crawling s' file_name)

With open (file_name, "wb") as f:

F.write (response.content)

If _ name__ = = "_ _ main__":

Spider = Spider ()

For i in range (0Pol 3):

# spider.start_work (url= "https://ibaotu.com/shipin/7-0-0-0-"+ str (I) +"-1.html ")

T = threading.Thread (target=spider.start_work, args= ("https://ibaotu.com/shipin/7-0-0-0-"+ str (I) +"-1.html ",))

T.start ()

Running result

Isn't it easy!

This is the end of the content of "how Python uses Requests to capture short videos of packet maps". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.