Shulou(Shulou.com)06/02 Report--

This article shows you how to use the pycharm-based requests library, which is concise and easy to understand, which will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Requests library installation and import

Step 1: cmd opens the command line and installs the requests library using the following command.

Pip install requests

Since I have installed it, it is as follows:

If you are prompted that the pip version needs to be updated, follow the instructions prompted to update it.

Step 2: cmd uses the following command to verify that the requests library installation is complete.

Pip list

Step 3: in pycharm, click file--settings--project--python interpreter-- and click the + sign-search for requests--install package!

Step 4: in the .py file you write, use the following command to import.

Import requests

One type and six properties of the requests library

Import requestsurl = "https://www.baidu.com"response = requests.get (url=url) # one type six attributes # Type print (type (response)) # set the encoding format of the response response.encoding = 'utf-8'# returns the source code of the web page as a string print (response.text) # returns an url address print (response.url) # returns a binary number according to print (response.content) # return Return the response header print (response.headers) returned by the corresponding status code print (response.status_code) #

The output is as follows:

Baidu, you will know.

News hao123 map video post bar login [xss_clean] ('login'); more products about Baidu About Baidu

©2017 Baidu must read comments before using Baidu feedback Beijing ICP Certificate 030173

Post is generally a form request, if you search something directly in Baidu, it is a get request!

Get request for requests library

First of all, write the code, and then give you the corresponding knowledge points according to the code, which can be regarded as an introduction.

Import requestsurl = "https://www.baidu.com/s?"headers = {'User-Agent':' Mozilla/5.0 (Windows NT 10.0; Win64) X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'} data = {'wd':' Beijing'} # url request path params Parameter kwargs Dictionary response = requests.get (url=url, params=data, headers=headers) # Parameter is passed using params, and the parameter does not need to be encoded by url encode, and the parameter does not need to be customized by object. With or without print (response.text)

Step 1: first, let's take a look at the use of the get method of the requests library and the meaning of the parameters.

Response = requests.get (url=url, params=data, headers=headers)

Url represents the request path, params represents the parameter, and kwargs represents the dictionary.

The parameters are passed using params, and the parameters do not need to be encoded by url encode, and the parameters do not need to be customized by objects. Yes or no.

Step 2: let's demonstrate how to pass these three parameters.

In the following explanation, those who have studied the front end should know how to do it, right?

Right-click to check-- select the following-- and refresh

This place is the url we asked for!

This place is the data we passed params!

Maybe a lot of people will look for From Data, this place should be PayLoad, pay attention!

This place is the dictionary we passed on!

Select the user agent below, which contains information about our browser.

In the above, it should be noted that since the last two of get are actually stored in the form of dictionaries in python, pay attention to the format after you get the data.

Step 3: let's see if there is any data. We can use ctrl + f to search and verify whether the content we want is there or not at the output.

Post request for requests library

First of all, write the code, and then give you the corresponding knowledge points according to the code, which can be regarded as an introduction.

Import requestsurl = "https://fanyi.baidu.com/sug"headers = {'User-Agent':' Mozilla/5.0 (Windows NT 10.0; Win64) X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'} data= {'kw':' eye'} # url request path data request parameters kwargs dictionary response = requests.post (url=url, data=data, headers=headers) # parameters are passed using data, and the parameters do not require url encode encoding, and the parameters do not require object customization print (response.text)

Output result:

{"errno": 0, "data": [{"k": "eye", "v": "n.\ u773c\ u775b;\ u89c6\ u529b;\ u773c\ u72b6\ u7269;\ u98ce\ u7eaa\ u6263\ u6263\ u773c vt. \ u5b9a\ u775b\ u5730\ u770b;\ u6ce8\ u89c6;\ u5ba1\ u89c6;\ u7ec6\ u770b "}, {" k ":" Eye "," v ":" [\ u4eba\ u540d]\ u827e; [\ u5730\ u540d] [\ u82f1\ u56fd]\ u827e\ u4f0a "}, {" k ":" EYE "," v ":" abbr. European Year of the Environment\ u6b27\ u6d32\ u73af\ u5883\ u5e74; Iwas "}, {" k ":" eyed "," v ":" adj. \ u6709\ u773c\ u7684 "}, {" k ":" eyer "," v ":" n.\ u6ce8\ u89c6\ u7684\ u4eba "}]}

Step 1: first, let's take a look at the use of the post method of the requests library and the meaning of the parameters.

Response = requests.post (url=url, data=data, headers=headers)

The parameters here are a little different from the get method, and we can look at it this way if we want to see the details. Select the method in pycharm and you will see the prompt.

Url represents the request path, data represents the request parameters, and kwargs represents the dictionary.

In fact, the difficulty is how to find this url! That is, which one is the urlboat we want! Let's take Baidu translation as an example!

I circle these places, be sure to pay attention to, select Preserve logbooks!

Look for it in the Name on the left. If the kw in the corresponding PayLoad is the same as our search, that's it!

The second step, we may not understand the returned data, so just convert it to json format!

Import requestsimport jsonurl = "https://fanyi.baidu.com/sug"headers = {'User-Agent':' Mozilla/5.0 (Windows NT 10.0; Win64) X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'} data= {'kw':' eye'} # url request path data request parameters kwargs dictionary response = requests.post (url=url, data=data, headers=headers) obj = json.loads (response.text, encoding='utf-8') print (obj)

Output result:

{'errno': 0,' data': [{'Kwon:' eye', 'vested:' n. Eye; eyesight; ophthalmus; wind QQ eye vt. Look intently; gaze; examine; take a closer look at'}, {'knot:' Eye', 'vested:' [name] ai; [place name] [UK] ai'}, {'Kwon:' EYE', 'vested:' abbr. European Year of the Environment year of the Environment in Europe; Iwas'}, {'Kwon:' eyed', 'vested:' adj. With eyes'}, {'knot:' eyer', 'vested:' n. The person watching'}]}

Proxy for the requests library

The main task of the agent is that when we simulate the browser to send a request to the server, we visit a website with high speed and high frequency, so the website will crash, so our ip will be blocked, so what are we going to do? Just change the ip address!

Import requestsurl = "https://www.baidu.com/s?"headers = {'User-Agent':' Mozilla/5.0 (Windows NT 10.0; Win64) X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'} data = {'wd':' ip'} response = requests.get (url=url, params=data, headers=headers) with open ('daili.html', 'walled, encoding='utf-8') as fp: fp.write (response.text)

You will find that you have written this file!

Cookie of the requests library

We are taking the ancient poetry network as an example!

What we want to achieve now is to go directly to the internal page without logging in.

# enter the main page by logging in # by finding the login interface, we find that we need a lot of parameters "_ _ VIEWSTATE: 9Y4yHRQS2k2z739MJJ/8Z0sKfZNltkFId83Z8jCtY3g00xYgg9bsv5oK+KT5DypNl37KWa0IyB+uOwrRPBvTybqGLDdd0chyrWLxhhlHBeAGWL/SLTGYfOh6L1M=__VIEWSTATEGENERATOR: C93BE1AEfrom: http://so.gushiwen.cn/user/collect.aspxemail: 13237153218pwd: wxm20010428code: PDBGdenglu: login" # We have observed that _ _ VIEWSTATE _ _ VIEWSTATEGENERATOR code is a variable quantity # _ _ VIEWSTATE _ _ data that cannot be seen is generally found in the In the source code of the page # we observe that it is in the source code of the page, so we need to get the source code of the page and parse it. # code is the verification code import requests# login url page url = 'https://so.gushiwen.cn/user/login.aspx?from=http://so.gushiwen.cn/user/collect.aspx'headers = {' User-Agent': 'Mozilla/5.0 (Windows NT 10.0) Win64 X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'} response = requests.get (url=url, headers=headers) # print (response.text) # parse the page source code and get _ _ VIEWSTATE _ VIEWSTATEGENERATORfrom bs4 import BeautifulSoupsoup = BeautifulSoup (response.text) 'lxml') # get _ _ VIEWSTATEviewstate = soup.select (' # _ _ VIEWSTATE') [0] .attrs.get ('value') # get _ VIEWSTATEGENERATORviewstategenerator = soup.select (' # _ VIEWSTATEGENERATOR') [0] .attrs.get ('value') # print (viewstate) # print (viewstategenerator) # get verification code image code = soup.select (' # imgCode') [0] .attrs.get ('src') # print (code) code_url =' https : / / so.gushiwen.cn' + code# print (code_url) # get the picture of the CAPTCHA, download it locally, observe the CAPTCHA, and then enter the CAPTCHA on the console and send this value to code#. # import urllib.request# if the request verification code is not the same as the following request # urllib.request.urlretrieve (url=code_url) Filename='code.jpg') # request has a method session () that makes the request an object through the return value of session session = requests.session () response_code = session.get (code_url) # Note that binary data is used here because if we download the image content_code = response_code.contentwith open ('code.jpg') 'wb') as fp: fp.write (content_code) code_name = input (' Please enter CAPTCHA:') # Click to log in url_post = 'https://so.gushiwen.cn/user/login.aspx?from=http%3a%2f%2fso.gushiwen.cn%2fuser%2fcollect.aspx'data_post = {' _ _ VIEWSTATE': viewstate,'_ _ VIEWSTATEGENERATOR': viewstategenerator, 'from':' http://so.gushiwen.cn/user/collect.aspx', 'email':' 13237153218, 'pwd':' wxm20010428', 'code': code_name,' denglu': 'login',} response_post = session.post (url=url_post, headers=headers, data=data_post) with open ('gushiwen.html',' walled, encoding='utf-8') as fp: fp.write (response_post.text)

First of all, let's open the login page of this ancient poetry network (assuming it has already been registered). Now we need to enter the correct account, the wrong password, the correct verification code, click to log in, but do not click OK after the prompt, otherwise the page will jump, and then grab the parameters needed for the login.

After observing the parameters, first find the changed parameters, and then try to get the changed parameters, and generally this kind of invisible parameters, generally in the source code, we click to view the source code, and then ctrl+F searches for invisible parameters to find its location.

Then we simulate the browser to send a request to the server, get the web page source code, use bs4 to parse the source code, and then send the request after changing the corresponding parameters!

Two files will be generated here, and code.jpg, if it cannot be loaded at run time, look in the project folder.

Automatic identification verification code

Super Eagle! Download the python development document and copy .py and an image into the project!

After opening it, take a look at the .py file and change the user name and password!

Follow the prompts to change the user ID

But because I did not recharge, did not return to me ha!

The above content is how to use the pycharm-based requests library. Have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.