Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "how python crawls beautiful pictures as wallpaper". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

First of all, you can look at the address bar on the browser, you can see that the URL composition of the search results is quite regular, in front is the home address of sugar pile,? Behind is our request data, the most important thing in the request data is (kw= "search keywords"), which will change according to the content of our search.

Then press f12 on the keyboard to call up developer tools, click Network, and then press f5 to refresh the web page to re-request data, you can see that there are many request files under the drop-down box of Name, we can click on a file to view the request header data, which is a very important request header structure data source for crawlers later.

Do not see the rule can click on the resource file, the right window of the Request URL line also has this url, we click on a few files to view, and copy the url address inside to the text to view.

Click a resource file, and then click on the right Preview, you can see that there is a lot of data, and we want the data is the previous json data, the black triangle is clickable, click will expand the detailed data information.

At this point, we can write our code, open pycharm, and import the necessary libraries first.

From urllib.parse import urlencodeimport requestsimport timeimport os

Then construct our request header and url for each page.

Base_url=' https://www.duitang.com/napi/blog/list/by_search/?'headers = {'Host':'www.duitang.com',' Referer':' https://www.duitang.com/search/?kw=%e7%be%8e%e5%a5%b3&type=feed', 'User-Agent':'Mozilla/5.0 (Windows NT 10.0 WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36', 'Xmuri RequestedMuffle Withwash Vane XMLHttpRequest'} def get_page (start,count): params= {'kw':'', 'type':'feed',' include_fields':'top_comments,is_root,source_link,item,buyable,root_id,status,like_count,like_id,sender,album,reply_count Favorite_blog_id','_ type':'', 'start':start,' _': count} url = base_url + urlencode (params) try: response = requests.get (url Headers=headers) if response.status_code = = 200: return response.json () else: print ("error") except requests.ConnectionError as e: print ('Error',e.args)

When you get the picture url, you can read and write.

Def parse_page (json): if json: items = json.get ('data'). Get (' object_list') for item in items: data = {} data ['favorite_count'] = item.get (' favorite_count'); item = item.get ('photo') data [' path'] = item.get ('path') # # get the url address file_path ='{0} / {1} '.format ('. /'+ download_dir, data ['path'] .split (' /') [- 1]) # # write the picture to the local hard disk with open (file_path) 'wb') as f: img = requests.get (data [' path']) print (img) f.write (img.content) yield data

Write another main function and finish it.

Start_page = 0 # # from which page to climb page_size = 5 # # how many pages to climb count = 1597729530322 # # the value of the underscore in the request data on page 1 download_dir = "downlode_img" >

This is the end of "how python crawls beautiful pictures as wallpaper". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.