How to use Python to collect Tencent recruitment data

This article will explain in detail how to use Python to collect Tencent recruitment data. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

Today we crawled the recruitment information for Python positions on Tencent recruitment website. As shown in the figure:

Then let's take a look at the final result and save it to the csv file.

Crawl 1000 pieces of data, the first line is title:

The old rules, let's follow the process.

01 demand analysis

Grab the job data related to Python on Tencent recruitment website, open the target website, F12 open the developer tool, and find that the job information is not in the page source code, which is obviously the json data in XHR.

It just corresponds to ten pieces of data on the page, and then it's easy to do.

02 send request

Find the page rules of url

First page url: https://careers.tencent.com/tencentcareer/api/post/Query?timestamp=1620561777984&countryId=&cityId=&bgIds=&productId=&categoryId=&parentCategoryId=&attrId=&keyword=python&pageIndex=1&pageSize=10&language=zh-cn&area=cn

The corresponding value of the index parameter is the page number, so the url chain is looped.

Def getPage (self): url_list = [] for i in range: url = self.url.format (I + 1) url_list.append (url) return url_list03 parsing page

After getting the url chain, we loop through the request, get the json data, convert the json data into a dictionary, and parse the data.

For url in urlList: resp = requests.get (url, headers=self.headers) data = resp.content.decode ('utf-8') dic_data = json.loads (data) dic_data_list = dic_data ["Data"] ["Posts"] for i in dic_data_list: work_list.append (I)

In this way, we get the job information data.

04 save data

After getting the job data, we need to save it locally, here we save the data to the local csv file.

Try: with open ('Tx_work.csv',' walled, encoding='utf-8', newline='') as f: writer = csv.DictWriter (f) Self.title) writer.writeheader () writer.writerows (dic) print ('write successful') except Exception ase: print (e) 05 run program if _ _ name__ = ='_ main__': tx = Tx (base_url,hd,titles) tx.run ()

On how to use Python to collect Tencent recruitment data to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.