Shulou(Shulou.com)06/02 Report--

This article will explain in detail for you how to use Python to climb to benefit the agricultural network Apple data, the content of the article is of high quality, so the editor will share it for you as a reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

This goal

Climb to get the information of Huinong website.

Victim address

Https://www.cnhnb.com/

Environment

Python3.6

Pycharm

Crawler code

Import tool

Import requestsimport parselimport csvimport time

Request header

Headers = {'Cookie':' deviceIdRenew=1; Hm_lvt_91cf34f62b9bedb16460ca36cf192f4c=1604579356,1604659451; deviceId=d1dd5b9-d191-406b Windows NT 971dmur391916a0e; sessionId=S_0KH64T2IHLHSO77N; lmvid=b24dcd0ad2a8f0b783f248c7ff2675a8; lmvid.sig=w1UBnTUKSDq-GfAlx6TyR_K7SjyujGIlF-1kRjTrOAI; hnUserTicket=b80e6b3a-38a3-432cMur816dMuyaeb0376228bd; hnUserId=870048250; Hm_lpvt_91cf34f62b9bedb16460ca36cf192f4c=1604659904', 'Host':' www.cnhnb.com', 'Referer':' https://www.cnhnb.com/supply/pingguo/', 'User-Agent':' Mozilla/5.0 (Windows NT 10.0 WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36',}

Parsing websites and crawling data

Def get_page_url (page_url, page_id): response_2 = requests.get (url=page_url Headers=headers) selector_2 = parsel.Selector (response_2.text) num_id = selector_2.css ('# _ layout > div > div > div > div > ul > li:nth-child (1) > a::attr (href)'). Get (). Split ('/') [2] title = selector_2.css ('. Proinfo-title::text'). Get (). Strip () # title update_time = selector_ 2.css ('. Update-time::text'). Get (). Strip ('update time:') # Update time price = selector_2.css ('.priceTxt .orange .fs30:: text'). Get (). Strip () +' yuan'# Price ads = selector_2.css ('div:nth-child (5) > span.fs14.gray6::text'). Get () # place of shipment Address get_phone_url = 'https://gateway.cnhnb.com/banana/im/operate/wechatcall' data = {' businessType':'1' 'sourceFrom':' 2, 'ticket':', "b80e6b3a-38a3-432c-816d-aeb0376228bd", 'userId':' {} '.format (num_id),} head = {' authority': 'gateway.cnhnb.com',' method': 'POST',' path':'/ banana/im/operate/wechatcall', 'scheme':' https' 'accept':' application/json, text/plain, * / *, 'accept-encoding':' gzip, deflate, br', 'accept-language':' zh-CN,zh QQ 0.9, 'cache-control':' no-cache', 'content-length':' 98, 'content-type':' application/json', 'origin':' https://www.cnhnb.com', 'pragma':' no-cache', 'referer':' https://www.cnhnb.com/gongying/{}/'.format(page_id), # 'Cookie':' deviceIdRenew=1 Hm_lvt_91cf34f62b9bedb16460ca36cf192f4c=1604579356,1604659451; deviceId=d1dd5b9-d191-406b empty', 971dmure 391916a0e; sessionId=S_0KH64T2IHLHSO77N; lmvid=b24dcd0ad2a8f0b783f248c7ff2675a8; lmvid.sig=w1UBnTUKSDq-GfAlx6TyR_K7SjyujGIlF-1kRjTrOAI; hnUserTicket=b80e6b3a-38a3-432c mur816dmuraeb0376228bd; hnUserId=870048250; Hm_lpvt_91cf34f62b9bedb16460ca36cf192f4c=1604659904', 'sec-fetch-dest':' empty', 'sec-fetch-mode':' cors', 'sec-fetch-site':' same-site', 'user-agent':' Mozilla/5.0 (Windows NT 10.0 WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36', 'xKH64WG5WL1GXPNG: '0KH64WG5WL1GXPNG,' xMeiMeiMeiL1GXPNG: '5000WG5WL1GXPNGG:' 5000WG5WL1GXPNG: 'c10e4e9a-5e19-4ba2WL5WL1KH64WL1c9e660a8d5' KH64WG5WL1c9e660a8d5' / gongying/ {} / '.format (page_id),' Xmuri client sidestep: 'Song0KH64W0GT18JX07L copyright,' xMurclientMuttink: 'b80e6b3a-38a3-432cMube 816dmuraeb0376228bdbath,' xmurclientwaytimekeeper: '1604659611092,' xmurhnlyjobless:'If you see these message, I hope you dont hack us, I hope you can join us! Please visit https://www.cnhnkj.com/job.html',} response_3 = requests.post (url=get_phone_url, json=data, headers=head) html_data = response_3.json () if html_data ['errorCode'] = = 0: dit = {' title': title, 'Update time': update_time, 'Price': price, 'Shipping address': ads Contact information: html_data ['data'] [' messageTitle']} csv_writer.writerow (dit) print (dit) else: dit = {'title': title, 'Update time': update_time, 'Price': price, 'Shipping address': ads 'contact information': 'merchant sets do not disturb'} csv_writer.writerow (dit) print (dit)

Save data

F = open ('Huinong information .csv', mode='a', encoding='utf-8-sig', newline='') csv_writer = csv.DictWriter (f, fieldnames= ['title', 'update time', 'price', 'shipping address', 'contact information']) csv_writer.writeheader () effect diagram

On how to use Python to climb to benefit the agricultural network Apple data is shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.