Sample Analysis of Python downloading Commodity data and connecting to Database and saving data

This article mainly introduces the Python download commodity data and connect to the database and save the data example analysis, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let Xiaobian with you to understand.

Foreword:

Development environment:

Python 3.8

Pycharm 2021.2 Professional Edition

Code implementation:

Send a request

Get data

Parsing data (filtering data)

Save data

Connect to the database

Request data # camouflage headers = {'cookie':' miid=4137864361077413341; tracknick=%5Cu5218%5Cu6587%5Cu9F9978083283; thw=cn; hng=CN%7Czh-CN%7CCNY%7C156; cna=MNI4GicXYTQCAa8APqlAWWiS; enc=%2FWC5TlhZCGfEq7Zm4Y7wyNToESfZVxhucOmHkanuKyUkH1YNHBFXacrDRNdCFeeY9y5ztSufV535NI0AkjeX4g%3D%3D; tweead15767ffa6febb4d2a8709edebf63d3; lgc=%5Cu5218%5Cu6587%5Cu9F9978083283; sgcookie=E100EcWpAN49d4Uc3MkldEc205AxRTa81RfV4IC8X8yOM08mjVtdhtulkYwYybKSRnCaLHGsk1mJ6lMa1TO3vTFmr7MTW3mHm92jAsN%2BOA528auARfjf2rnOV%2Bx25dm%2BYC6l; uc3=nk2=ogczBg70hCZ6AbZiWjM%3D&vt3=F8dCvCogB1%2F5Sh2kqHY%3D&lg2=Vq8l%2BKCLz3%2F65A%3D%3D&id2=UNGWOjVj4Vjzwg%3D%3D; uc4=nk4=0%40oAWoex2a2MA2%2F2I%2FjFnivZpTtTp%2F2YKSTg%3D%3D&id4=0%40UgbuMZOge7ar3lxd0xayM%2BsqyxOW; _ cc_=W5iHLLyFfA%3D%3D _m_h6_tk=ac589fc01c86be5353b640607e791528_1647451667088; _m_h6_tk_enc=7d452e4e140345814d5748c3e31fc355; xlly_s=1; x5sec=7b227365617263686170703b32223a223264393234316334363365353038663531353163633366363036346635356431434c61583635454745506163324f2f6b2b2b4b6166686f4d4d7a45774e7a4d794d6a59324e4473784d4b6546677037382f2f2f2f2f77453d227d; JSESSIONID=1F7E942AC30122D1C7DBA22C429521B9; tfstk=cKKGBRTY1F71aDbHPcs6LYjFVa0dZV2F6iSeY3hEAYkCuZxFizaUz1sbK1hS_r1..; l=eBEVp-O4gnqzSzLbBOfwnurza77OIIRAguPzaNbMiOCPO75p5zbNW60wl4L9CnGVhsTMR3lRBzU9BeYBqo44n5U62j-la1Hmn; isg=BDw8SnVxcvXZcEU4ugf-vTadDdruNeBfG0WXdBa9WicK4dxrPkd97hHTxQmZqRi3', 'referer': 'https://s.taobao.com/search?q=%E4%B8%9D%E8%A2%9C&imgfile=&js=1&stats_click=search_radio_all%3A1&initiative_id=staobaoz_20220323&ie=utf8&bcoffset=1&ntoffset=1&p4ppushleft=2%2C48&s=', 'sec-ch-ua': '"Not A;Brand"; V = "99", "Chromium"; v = "99", "Google Chrome" V = "99", 'sec-ch-ua-mobile':'? 050, 'sec-ch-ua-platform':', 'Windows', 'sec-fetch-dest':' document', 'sec-fetch-mode':' navigate', 'sec-fetch-site':' same-origin', 'sec-fetch-user':'? 1century, 'upgrade-insecure-requests':' 1' 'user-agent': 'Mozilla/5.0 (Windows NT 10.0 Win64 X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.82 Safari/537.36',} url= f 'https://s.TB.com/search?q=%E4%B8%9D%E8%A2%9C&imgfile=&js=1&stats_click=search_radio_all%3A1&initiative_id=staobaoz_20220323&ie=utf8&bcoffset=1&ntoffset=1&p4ppushleft=2%2C48&s={44*page}'response = requests.get (url=url) Headers=headers) get data html_data = response.text parse data (filter data) json_str = re.findall ('g_page_config = (. *) 'To Html_data) [0] # string definitely cannot be taken by dictionary key-value pair # json_str: string # type conversion: only when converted to a dictionary type can the value json_dict = json.loads (json_str) auctions = json_dict ['mods'] [' itemlist'] ['data'] [' auctions'] for auction in auctions: raw_title = auction ['raw_title'] Pic_url = auction ['pic_url'] detail_url = auction [' detail_url'] view_price = auction ['view_price'] item_loc = auction [' item_loc'] view_sales = auction ['view_sales'] nick = auction [' nick'] print (raw_title Pic_url, detail_url, view_price, item_loc, view_sales, nick) Save data with open ('TB.csv', mode='a', encoding='utf-8', newline='') as f: csv_writer = csv.writer (f) csv_writer.writerow ([raw_title, pic_url, detail_url, view_price, item_loc, view_sales, nick])

Connect to the database import pymysqldef save_sql (title, pic_url, detail_url, view_price, item_loc, view_sales, nick): count = pymysql.connect (host='xxx.xxx.xxx.xxx', # database address port=3306, # database port user='xxxx', # database account password='xxxx' # Database password db='xxxx' # Database Table name) # create database object db= count.cursor () # write sql sql = f "insert into goods (title, pic_url, detail_url, view_price, item_loc, view_sales, nick) values ('{title}','{pic_url}','{detail_url}', {view_price},'{item_loc}') '{view_sales}','{nick}') "# execute sql db.execute (sql) # Save changes count.commit () db.close () Thank you for reading this article carefully I hope the article "Python downloads commodity data and connects to the database and saves the data" shared by the editor will be helpful to everyone. At the same time, I also hope that you will support and pay attention to the industry information channel. More related knowledge is waiting for you to learn!