Shulou(Shulou.com)06/01 Report--

This article will explain in detail what to do when writing crawlers encounter APP requests with encryption parameters. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

Here I write a demo APP for you to play with, which, as I said earlier, sends a request with an encryption parameter-sign, and the sign changes each time it is requested.

After installation, open it and prepare your package grab tool, and then click the "Click to send request" button.

If nothing happens, there will be a "sign check passed" prompt, and then let's take a look at the bag we caught.

Bag grabbing result

We can clearly see from the package grabbing tool that there are two parameters that will change, one is ts and the other is sign. (it is recommended to send two more requests to see the change.)

First analyze the meaning of the parameters. Ts can be seen from the name and should be a timestamp. Actually, formatting the value of ts can also determine that this is the timestamp of the request; then sign, which looks like about 32 bits at a glance, and the appearance of such a person is usually hash. Guess that the most likely thing is md5, but you don't know how it is actually generated, so you can only look at it in reverse.

It's time to officially start cracking this encryption parameter, because Android APP is statically compiled, unlike JS, you can see the source code directly, so... We need to decompile APP. The tool I use here is called Jadx. What tools do you need to write APP crawlers? "and" what do you do when you write that the crawler can't catch the APP request packet? [advanced article-confusion leads to failure of general Hook tools] is also mentioned in the article, so I won't repeat it here.

Decompile APK using jadx

After decompilation, you can see such a mess, so how do we find the place to generate the sign parameters?

See that button that looks like a magic wand? Click on it.

Jadx search text

Then it pops up a "search text" window, and then we have two ways to quickly navigate to the generated location:

Search for the path portion of URL

The path part refers to / learning/hash_sign. Of course, some APP may split the path into multiple segments for reuse. If you can't search the full path directly, you can try to use the backslash as the separator to split the path into multiple segments (remember to search from right to left, don't ask why). Here, we can search hash_sign directly, because the name is unique. Generally speaking, there should be no other irrelevant things with the same name.

With a search, you can locate the requested location.

Locate by search path

Search for the parameters you are looking for

For example, here we need to find the parameter sign, you can directly search for "sign" (note the double quotation marks), but if there are a lot of results, but also very similar to the place where sign is generated / set, you can search for some other unique parameters, such as model, brand and other words that do not often appear in the code.

Since this DEMO APP is relatively simple, a direct search for "sign" can locate the location of the set value.

Locate by searching the parameter name

After locating the location of the code, we can start to look at the code. From the search results, we can see that the code for generating sign, setting sign, and setting path are all under this l method, and then we start from the location of setting sign to analyze from bottom to top, so that the logic of the code will be easier to understand.

Navigated to the code

Here I have marked all the key points in the code, and you can follow me to look at the code according to the serial number marked next to it.

First of all, the aVar4.an appears in two places, but the second parameter passed in is stringBuilder2, and there is a null value directly set there below, which is obviously not what we are looking for, so we can ignore the following line with the sign keyword and look directly at the line marked with 1.

Select the variable stringBuilder2, you can see that its value is obtained from the above stringBuilder3.toString (), and then look at the generation of stringBuilder3, the for loop does not understand what is done here, but you can see that there is a very prominent string "MD5". So we can boldly guess that this stringBuilder3 is actually just a MD5 operation, which is the same as our guess when we first grabbed the bag. Go straight up to see what the string in front of the Hash looks like, and then test it. If not, come back.

From the position of the previous MD5 operation, you can see that the parameter used in the digest method is another variable called stringBuilder2. If you continue to look up, you can see that it is actually obtained from stringBuilder.toString (), so how did this stringBuilder come from? We can see from the code that it seems to be for a TreeMap and then write the format of each key and value into the stringBuilder, and then add the & symbol if there is already a value in the stringBuilder, so what can you think of in the end? Yes! That's the queryString part, except that its parameters are sorted (because TreeMap sorts automatically).

Then we look up to verify whether the idea is correct. The place where we can see the source is a HashMap, and what is entered by put is the parameters we saw when we grabbed the package.

Now we have a clear idea of the generation logic of the sign. In fact, it is the product of a MD5 operation of a queryString sorted by the parameter name. Then we just need to implement the generation logic in the code. In Python, you can use its own official library hashlib to do MD5 operations on a string.

So let's write a piece of code to simulate the request, and the sign can indeed pass the verification, indicating that the sign generated by us can be used. At this point, the encryption parameters are cracked.

Tip: it is recommended not to test like this in practice, as it is easy to trigger anti-crawling. You can first take the parameters obtained from the grab package to generate a comparison. If the same is the same, then there is no problem with the generated sign.

On the crawler encountered APP request how to solve the encryption parameters to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.