Shulou(Shulou.com)06/01 Report--

This article mainly shows you "Python how to climb the intern recruitment website", the content is easy to understand, well-organized, hope to help you solve your doubts, the following let the editor lead you to study and learn this article "Python how to climb the intern recruitment website".

The background of this mission:

Https://www.shixiseng.com

Climb the Python internship information of the intern monk IT Internet.

As shown in the figure above, the data in this field is invisible, and maybe it doesn't want you to simply get the data from its website, which is more important to him, so anti-crawling skills are enabled.

If you run it directly, the data cannot be crawled down, as shown below:

As shown in the figure above, the relevant data has been presented in the form of "utf-8" coding.

Create a function hack_number () to decode numbers

After you have written the relevant code, check the running result

The complete code is as follows:

Import requestsfrom bs4 import BeautifulSoupheaders = {"user-agent": "Mozilla/5.0"} def hack_number (text): text = text.encode ('utf-8') text = text.replace (b'\ xef\ x82\ x9dbath, baux0') text = text.replace (b'\ xee\ xa6\ x88mm, baux1') text = text.replace (b'\ xee\ xa8\ xb4', bread2') text = text.replace (b'\ xef\ x91\ xbe'') Bread3') text = text.replace (b'\ xee\ x88\ x9dwells, baux4') text = text.replace (b'\ xef\ x97\ x80,baux5') text = text.replace (b'\ xee\ x85\ x9fags, baux6') text = text.replace (b'\ xee\ x98\ x92cycles, baux7') text = text.replace (b'\ xef\ x80\ x95cycles, baux8') text = text.replace (b'\ xef\ x94\ x9b') Bread9') text = text.decode () return textdef detail_page (url): req = requests.get (url,headers=headers) html = req.text soup = BeautifulSoup (html 'lxml') job_name = soup.select (' .new _ job_name') [0] .text.strip () job_money = hack_number (soup.select ('.job _ money') [0] .text.strip () job_position = soup.select (' .job _ position') [0] .text.strip () job_academic = soup.select ('.job _ academic') [0] .text.strip () job_detail = soup.select ( Job_detail') [0] .text.strip () job_week = hack_number (soup.select ('.job _ week') [0] .text.strip ()) job_time = hack_number (soup.select (' .job _ time') [0] .text.strip () print (job_name) Job_money,job_position,job_academic,job_week Job_time) print (job_detail) # detail_page ('https://www.shixiseng.com/intern/inn_1k3vhcwwguaf?pcm=pc_SearchList')#detail_page('https://www.shixiseng.com/intern/inn_uk1lm380lngh?pcm=pc_SearchList')#detail_page('https://www.shixiseng.com/intern/inn_fr1o1nii5knw?pcm=pc_SearchList')for pages in range (1): url = f' https://www.shixiseng.com/interns?page={pages } & keyword=Python&type=intern&area=&months=&days= °ree = & official=&enterprise=&salary=-0&publishTime=&sortType=&city=%E8%B4%B5%E9%98%B3&internExtend=' req = requests.get (url Headers=headers) html = req.text soup = BeautifulSoup (html,'lxml') for item in soup.select ('a.title ellipsis font'): detail_url = f "https://www.shixiseng.com{item.get('href')}" detail_page (detail_url) these are all the contents of the article" how Python crawls the internship recruitment website " Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.