Shulou(Shulou.com)06/01 Report--

This article introduces the knowledge of "how to call Web API". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Step 1: open the developer tool and find a JSON response

I browsed https://hangouts.google.com, opened the "Web Network" tab in Firefox's developer tool, and found a JSON response. You can also use Chrome's developer tools.

After opening, the interface is as follows:

Find one of the requests whose "Type Type" column is displayed as json.

I looked for a request I was interested in for a long time, and suddenly I found an endpoint of "people" that seemed to return our contact information. That sounds interesting. Let's take a look.

Step 2: copy to cURL

Next, I right-click on the request I am interested in and click "copy Copy"-> "copy as cURLCopy as cURL".

Then I paste the curl command into the terminal and run it. The following is the running result:

$curl 'https://people-pa.clients6.google.com/v2/people/?key=REDACTED'-X POST. (omitted a large number of request headers) Warning: Binary output can mess up your terminal. Use "- output -" to tell Warning: curl to output it to your terminal anyway, or consider "--output Warning:" to save to a file.

You might think-- strangely, what's wrong with "binary output doesn't display properly on your terminal"? The reason is that by default, the request header sent by the browser to the server has the parameter Accept-Encoding: gzip, deflate, which will compress the output.

We can pipe the output to gunzip to extract it, but we find it easier to request without this parameter. So we remove some irrelevant request headers.

Step 3: remove irrelevant request headers

Here is the complete curl command I got from the browser. There are a lot of lines! I separate the requests with a backslash (\) so that each request header has a line, which looks clearer:

Curl 'https://people-pa.clients6.google.com/v2/people/?key=REDACTED'\-X POST\-H' User-Agent: Mozilla/5.0 (X11; Linux x86'64 Rv:96.0) Gecko/20100101 Firefox/96.0'\-H 'Accept: * / *'\-H 'Accept-Language: en'\-H' Accept-Encoding: gzip Deflate'\-H 'X-HTTP-Method-Override: GET'\-H'Authorization: SAPISIDHASH REDACTED'\-H'Cookie: REDACTED'-H 'Content-Type: application/x-www-form-urlencoded'\-H' X-Goog-AuthUser: 0'\-H'Origin: https://hangouts.google.com'\-H'Connection: keep-alive'\-H'Referer: https://hangouts.google.com/'\-H 'Sec-Fetch -Dest: empty'\-H 'Sec-Fetch-Mode: cors'\-H 'Sec-Fetch-Site: same-site'\-H 'Sec-GPC: 1'\-H 'DNT: 1'\-H' Pragma: no-cache'\-H 'Cache-Control: no-cache'\-H'TE: trailers'\-- data-raw 'personId=101777723309&personId=1175339043204&personId=1115266537043&personId=116731406166&extensionSet.extensionNames=HANGOUTS_ADDITIONAL_DATA&extensionSet.extensionNames=HANGOUTS_OFF_NETWORK_GAIA_ GET&extensionSet.extensionNames=HANGOUTS_PHONE_DATA&includedProfileStates=ADMIN_BLOCKED&includedProfileStates=DELETED&includedProfileStates=PRIVATE_PROFILE&mergedPersonSourceOptions.includeAffinity=CHAT_AUTOCOMPLETE&coreIdParams.useRealtimeNotificationExpandedAcls=true&requestMask.includeField.paths=person.email&requestMask.includeField.paths=person.gender&requestMask.includeField.paths=person.in_app_reachability&requestMask.includeField.paths=person.metadata&requestMask.includeField.paths=person.name&requestMask.includeField.paths=person.phone&requestMask.includeField.paths=person.photo&requestMask.includeField.paths=person.read_only_profile_info&requestMask.includeField.paths=person.organization&requestMask.includeField.paths=person.location& RequestMask.includeField.paths=person.cover_photo&requestMask.includeContainer=PROFILE&requestMask.includeContainer=DOMAIN_PROFILE&requestMask.includeContainer=CONTACT&key=REDACTED'

At first glance, there seems to be a lot of content, but now you don't have to think about what each line means. All you have to do is delete the irrelevant lines.

I usually verify that a line can be deleted by deleting a line to see if there are any errors-- keep deleting headers as long as there are no errors in the request. In general, you can delete Accept*, Referer, Sec-*, DNT, User-Agent and cache-related headers.

In this example, I delete the request as follows:

Curl 'https://people-pa.clients6.google.com/v2/people/?key=REDACTED'\-X POST\-H' Authorization: SAPISIDHASH REDACTED'\-H 'Content-Type: application/x-www-form-urlencoded'\-H' Origin: https://hangouts.google.com'\-H 'Cookie: REDACTED'\-- data-raw' personId=101777723309&personId=1175339043204&personId=1115266537043&personId=116731406166&extensionSet.extensionNames=HANGOUTS_ADDITIONAL_DATA&extensionSet.extensionNames=HANGOUTS_OFF_NETWORK_GAIA_GET&extensionSet.extensionNames=HANGOUTS _ PHONE_DATA&includedProfileStates=ADMIN_BLOCKED&includedProfileStates=DELETED&includedProfileStates=PRIVATE_PROFILE&mergedPersonSourceOptions.includeAffinity=CHAT_AUTOCOMPLETE&coreIdParams.useRealtimeNotificationExpandedAcls=true&requestMask.includeField.paths=person.email&requestMask.includeField.paths=person.gender&requestMask.includeField.paths=person.in_app_reachability&requestMask.includeField.paths=person.metadata&requestMask.includeField.paths=person.name&requestMask.includeField.paths=person.phone&requestMask.includeField.paths=person.photo&requestMask.includeField.paths=person.read_only_profile_info&requestMask.includeField.paths=person.organization&requestMask.includeField.paths=person.location&requestMask.includeField.paths=person .cover _ photo&requestMask.includeContainer=PROFILE&requestMask.includeContainer=DOMAIN_PROFILE&requestMask.includeContainer=CONTACT&key=REDACTED'

So I only need four request headers: Authorization, Content-Type, Origin, and Cookie. It's much easier to manage.

Step 4: send a request in Python

Now that we know which request headers we need, we can translate curl commands into Python programs! This part is a fairly mechanized process, and the goal is simply to send the same data as cUrl using Python.

The following is a code example. We use Python's requests package to achieve the same functionality as the previous curl command. I decomposed the entire long request into an array of tuples to make it look more concise.

Import requestsimport urllibdata = [(personId','101777723'), # I redacted these IDs a bit too ('personId','117533904'), (' personId','111526653'), ('personId','116731406'), (' extensionSet.extensionNames','HANGOUTS_ADDITIONAL_DATA'), ('extensionSet.extensionNames','HANGOUTS_OFF_NETWORK_GAIA_GET'), (' extensionSet.extensionNames','HANGOUTS_PHONE_DATA'), ('includedProfileStates') 'ADMIN_BLOCKED'), (' includedProfileStates','DELETED'), ('includedProfileStates','PRIVATE_PROFILE'), (' mergedPersonSourceOptions.includeAffinity','CHAT_AUTOCOMPLETE'), ('coreIdParams.useRealtimeNotificationExpandedAcls','true'), (' requestMask.includeField.paths','person.email'), ('requestMask.includeField.paths','person.gender'), (' requestMask.includeField.paths','person.in_app_reachability'), ('requestMask.includeField.paths') 'person.metadata'), (' requestMask.includeField.paths','person.name'), ('requestMask.includeField.paths','person.phone'), (' requestMask.includeField.paths','person.photo'), ('requestMask.includeField.paths','person.read_only_profile_info'), (' requestMask.includeField.paths','person.organization'), ('requestMask.includeField.paths','person.location'), (' requestMask.includeField.paths') 'person.cover_photo'), (' requestMask.includeContainer','PROFILE'), ('requestMask.includeContainer','DOMAIN_PROFILE'), (' requestMask.includeContainer','CONTACT'), ('key','REDACTED')] response = requests.post (' https://people-pa.clients6.google.com/v2/people/?key=REDACTED', headers= {'Xmuri HTTPMAT MethodMurray overrides:' GET', 'Authorization':' SAPISIDHASH REDACTED' 'Content-Type':' application/x-www-form-urlencoded', 'Origin':' https://hangouts.google.com', 'Cookie':' REDACTED',}, data=urllib.parse.urlencode (data),) print (response.text)

After I executed this program, I ran it normally-I output a bunch of JSON data! great!

You will notice that in some places I use REDACTED instead, because if I list the raw data, you can use my account to access Google Forum, which is very bad.

End of operation

Now I can modify the Python program at will, such as passing in different parameters, or parsing the results.

I'm not going to use it for anything else interesting, because I'm not interested in this API at all, I just use it to illustrate the process of requesting an API.

But you can do something with the returned pile of JSON.

Curlconverter looks strong.

Some people commented that you can use https://curlconverter.com/ to automatically convert curl to Python (and some other languages! ), it looks amazing-- I turn it all by hand. I used it in this example, and everything looks fine.

Tracking the process of API is not easy

I'm not going to exaggerate the difficulty of tracking the API process-- the API process is not obvious! I don't know what all the parameters passed to this Google Forum API are for!

But there are some parameters that look intuitive, such as

RequestMask.includeField.paths=person.email

It may mean "include everyone's email address". So I only care about the parameters I can understand, not what I don't understand.

(theoretically) applicable to all scenarios

Some people may question-does this method apply to all scenarios?

The answer is yes-browsers are not magic! All the information that the browser sends to your server is a HTTP request. So if I copy all the HTTP request headers sent by the browser, the back end will assume that the request is from my browser, not from the Python program.

Of course, we removed some of the request headers sent by the browser, so theoretically the back end can tell whether the request came from the browser or the Python program, but they usually don't check.

Here are some caveats for readers-the back end of some Google services communicates with the front end in an incomprehensible (for me) way, so even if you can simulate the front end request in theory, it may not work in practice. Large API that may be subject to more attacks will have more protection.

We already know how to call an undocumented API. Now let's talk about the problems we may encounter.

Question 1: session cookie expires

A big problem is that I use my Google session cookie as authentication, so when my browser session expires, this script won't work.

This means that this approach won't last long (I'd rather call a real API), but if I just want to grab a small piece of data quickly at once, I can use it.

Problem 2: abuse

If I am requesting a small website, then my Python script may bring down the service because the number of requests exceeds their processing capacity. So when I ask, I try to be careful not to send a large number of requests too quickly.

This is particularly important because sites without official API tend to be small and do not have enough resources.

Obviously this is not a problem in this case-I think I sent a total of 20 requests to the back end of Google Forum in the course of writing this article, and they can definitely handle it.

If you overaccess the API using your account identity and cause a malfunction, your account may be temporarily blocked (understandably).

I only download my own data or public data-my goal is not to look for weaknesses in the site.

That's all for "how to call Web API". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.