Shulou(Shulou.com)06/02 Report--

In view of how to use the urllib library in python, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and easy way.

The urllib library is used to crawl network data. Urllib includes the following four modules:

Urllib.request is used to initiate network requests

Urllib.parse is used to operate url

Urllib.errer is used to handle network request errors

Urllib.robotparser is used to parse the website robots.txt file

Crawling the 51job website work case import urllib.requestimport user_agent # is used to generate User-Agentimport re import lxml.etree as le # matching data with xpath import toolimport pandas as pd keywords = input ('keywords >') page = input ('page >') data_best = [] header = {'User-Agent':user_agent.get_user_agent_pc ()} request = urllib.request.Request (# Construction request header Url=' https://search.51job.com/list/170200,000000,0000,00,9, 99, {keywords}, 2, {page} .html? lang=c&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&ord_field=0&dibiaoid=0&line=&welfare='.format (keywords=keywords,page=page), headers=header) response = urllib.request.urlopen (request) pattern = 'window.__SEARCH_RESULT__ = (. *?)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.