In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-01-18 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Internet Technology >
Share
Shulou(Shulou.com)06/02 Report--
This article introduces the relevant knowledge of "how python crawls JD.com Commodity Review". In the operation of actual cases, many people will encounter such a dilemma. Next, let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!
#! / usr/bin/python#-*-coding: UTF-8-*-import requestsimport reimport jsonimport timeimport xlwtimport random### configuration form # you don't need to know what it is # you can write Chinese to the table with the following four lines of code # style=xlwt.XFStyle () font=xlwt.Font () font.name='SimSun'style.font=font# create a form w=xlwt.Workbook (encoding='utf-8') # add a sheetws=w.add_sheet ('sheet 1') Cell_overwrite_ok=True) # currently writes the form to the row line row=1## write table header # ws.write (0meme 0meme content`) ws.write (0meme 1pr userClientShow') ws.write (0meme 2pr creationTime') ws.write (0meme 3meme user LevelName') ws.write (0min 4Me productColor') ws.write (0mee 5ref userLevelID') ws.write (0min 6m score`) ws.write (0mei 7pm referenceName') ws.write (0m8pr referenceTime') ws.write (09mcmlmsMobile') ws.write (0pl 10) 'nickname') # # accept a json object # write content into the table # def write_json_to_xls (dat): global row for comment in dat [' comments']: ws.write (row,0,comment ['content']) ws.write (row,1,comment [' userClientShow']) ws.write (row,2,comment ['creationTime']) ws.write (row,3 Comment ['userLevelName']) ws.write (row,4,comment [' productColor']) ws.write (row,5,comment ['userLevelId']) ws.write (row,6,comment [' score']) ws.write (row,7,comment ['referenceName']) ws.write (row,8,comment [' referenceTime']) ws.write (row,9,comment ['isMobile']) ws.write (row 10 https://item.jd.hk/2990360.html","User-Agent": comment ['nickname']) row+=1### loop to get data # # hearders = {"Referer": "https://item.jd.hk/2990360.html","User-Agent":" Mozilla/5.0 (Windows NT 6.1) WOW64) AppleWebKit/537.36 (KHTML Like Gecko) Chrome/70.0.3538.110 Safari/537.36' "} auto_jquery = 8809536for i in range (1meme 1010): # url=' https://club.jd.com/comment/productPageComments.action?productId=1475512465&score=0&sortType=5&page=%d&pageSize=100&isShadowSku=0&fold='% I # url=' https://sclub.jd.com/comment/productPageComments.action?callback=jQuery2663266&productId=2990360&score=2&sortType=5&page=%d&pageSize=10&pin=null&_=1563330030798'% I url=' Https://sclub.jd.com/comment/productPageComments.action' # print (url) try: auto_jquery = auto_jquery+1 jquery = 'jQuery%d'% auto_jquery times = random.randint (100999) print times auto_tims = int (time.time ()) true_string ='% d% d'% (auto_tims Times) print true_string params= {'callback':jquery,'productId':'2990360','sortType':'5','page':i,'pageSize':'10','pin':'null','_':true_string,'score':'2'} json_req = requests.get (url,params=params Headers=hearders) print ('11111111111111111111111111') print (json_req.url) print (json_req.text) print ('222222222222222222222222') flag = json_req.text.split ('(') flag_two = flag [1] .split (')') print flag_two [0] Print ('6666666666666666666666666666') json_flag = json.loads ((flag_ two [0]) print json_flag ['comments'] # dat = json_req.json () write_json_to_xls (json_flag) print (u' write% d page data'% I) except Exception as e: print (u' failed to get data' E) time.sleep (0.5) # storing data in table w.save ('result.xls') "this is the end of the introduction to" how python crawled JD.com 's Commodity Review " Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 0
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.