Shulou(Shulou.com)06/01 Report--

What are WeChat Mini Programs's crawling skills? I believe many inexperienced people can do nothing about it. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

Today, let's talk about the crawling of WeChat Mini Programs. In fact, it is not difficult to crawl Mini Program. It mainly solves the two problems of grabbing bags and how to debug Mini Program. If you are already proficient in debugging with chrome, it will be easy. Let's start with the problem of grabbing bags by Mini Program.

How to catch Mini Program's bag without cracking it? Cracking is a laborious task, and the Wechat account may also be blocked if you are not careful. Mini Program can't catch the bag usually because your phone's Android version is too high and Wechat's APP version is too high. The higher the version, the more secure it is usually. Switching to Android is 4.4% of the phone and Wechat APP version is around 6.7. Use Fiddler or Charles to grab the bag properly.

If you really don't have a low Android version or a low version of Wechat, read the text below and introduce a way to grab the bag later. As long as the bag is done, a lot of Mini Program can be crawled, and the rest is to solve the IP problem. Another part of Mini Program has anti-crawling measures at the front end, encrypting or confusing the request parameters. So we still have to solve the problem of Mini Program debugging.

Let's talk about the debugging of Mini Program.

First of all, we have to have a little understanding of Mini Program. Mini Program is simply a website, but it can only be opened in Wechat, not in the browser.

We all know that the front page of a website is composed of html, css and javascript, and the front page of Mini Program is also made up of something like this. Javascript is also responsible for the data interaction in Mini Program. So the crawler debugging Mini Program is mainly to debug javascript. How to debug the javascript of Mini Program? When we click Mini Program in Wechat, Wechat will download the front-end code of this Mini Program to your phone. As long as we get the Mini Program front-end code, we can debug it on the Mini Program developer tool provided by Wechat.

The path to Mini Program code is as follows:

/ data/data/com.tencent.mm/MicroMsg/ WeChat id folder / appbrand/pkg/ the files ending with .wxapkg in this path are the compiled form of the Mini Program front-end code.

You will see a lot of .wxapkg files, because you have opened a lot of Mini Program, it is best if you delete all the files under this file and reopen the destination Mini Program. Then the .wxapkg files inside are all Mini Program's. You need to copy all the .wxapkg files to your computer. (you need to have root permission to copy this directory, so you have to root your phone first, or use Wechat in the Android simulator, which is root by default). Unpack wxapkg

Our goal is to get the front-end code of Mini Program to debug in the Mini Program developer tool. Wxapkg is the compiled Mini Program, so we have to decompile and let wxapkg unpack the source code.

Thanks to the open source world, some enthusiasts have developed a .wxapkg unpacking program, so we'll just bring it over and use it.

The github address of the unpacking program is: https://github.com/qwerty472123/wxappUnpacker unpackaged program is developed by node.js, so you have to install node.js first, and then install some dependent packages of node.js, the author has made it very clear in the text. I won't repeat it any more, and there are many articles on the use of this program online. Finally, you only need to run the command: node xxxxxx.wxapkgnode is the command to run node.js, and you can restore the Mini Program front-end source code. Something like the following picture.

Debug Mini Program to register a Mini Program developer account registration address: https://mp.weixin.qq.com/cgi-bin/registermidpage?action=index&lang=zh_CN&token= download Mini Program developer tools:

Https://developers.weixin.qq.com/miniprogram/dev/devtools/download.html opens Mini Program developer tools and chooses to import an existing project, which is the folder unpackaged above. Click OK. The following interface appears. This allows you to debug the Mini Program, whether the interface is familiar or similar to the debugging of the chrome browser. You can click on the sources panel and break the js, or you can run a piece of js code directly in console. If you want to know how the requested URL is encrypted, just like chrome, you can debug the breakpoint. I have written several articles on the official account of ape-man learning about how to interrupt debugging. You can go back and take a look at it. In addition, the first half of my article talked about a bag grab question, which has not been answered. If it is a high version of Android, how does the high Wechat version grab the bag Mini Program? One trick is to grab the package with the help of Mini Program developer tools. Take a closer look at the red box above. There is a Network panel with the same function as chrome. This Mini Program web request can be seen in the Network panel. In addition, because this Mini Program is developed by someone else, you need to run this Mini Program so that it does not verify domain names and ssl certificates, as follows.

To sum up, after solving the problem of grabbing and debugging Mini Program, we will be able to crawl the vast majority of Mini Program. There are some Mini Program must be Wechat login to visit, to crawl on a large scale, you still have to solve the problem of a large number of accounts.

After reading the above, have you mastered WeChat Mini Programs's grasping skills? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.