Shulou(Shulou.com)05/31 Report--

The main content of this article is to explain "how to use Python bag grabbing tool to check the carpool around". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Next, let the editor take you to learn "how to use Python bag grabbing tool to check the surrounding carpool situation"!

For APP crawlers, the first thing that comes to mind is to grab packages. Charles and mitmproxy are two tools commonly used under macOS, while mitmproxy is deeply loved by me because of its support for interception and scripting, so I won't introduce the usage here.

Start mitmweb (the mitmproxy brother with a page) to grab the bag, open a carpool APP client on your mobile phone, and check the nearby order (the owner has been verified in advance and switched to the owner identity)

Correspondingly, we can find that the interface URL for querying nearby orders is this getNearbyBookingRideList, so I naively took this interface and grabbed the packet data, and went to request it.

And I was told

{"code": 205, "message": "signature error!" } because the client APP has a signature algorithm, which is calculated according to the parameters given to the server by POST, unless I break the APP and find out the algorithm, think about it or forget it (mainly because I don't know how to do it at all)!

This method won't work, so we have to find another way.

The artifact is in hand. I mentioned earlier that mitmproxy has a brother with web interface called mitmweb. In fact, there is a brother named mitmdump who supports scripts. Write a good script.

Import jsondef response (flow): # Don't ask why, you must write url = 'http://jiaruwoshiurl.com' if url in flow.request.url: text = json.loads (flow.response.content) print (text) save the script as 111.py, and then the terminal executes the command:

Mitmdump-s 111.py-p 8080 all that needs to be done is to keep scratching on a carpool APP. As for the data, just give it to the Dump brothers.

Well, see ~ where you want to see, the fingers can slide, the more slippery!

The next step is to save the data we captured for later use. Here I use Mongodb, although it is also the first time to use it. For convenience, I write a method to insert data and call it in the crawler file.

Import pymongodef insert_nearbyroute (info): client = pymongo.MongoClient ('127.0.0.1, 27017) # Connect database db = client.dida # Select dida this database nearbyroute_collection = db.nearby # Select collection, if it does not exist, automatically create (corresponding to the table concept of mysql) nearbyroute_collection.update ({: info [' id']}, info, True) # insert data. I use the update statement here. It means that if id already exists, the insert action of this piece of data will not be performed. You can effectively go back to our database and have a look at it. It has been fruitful, and if it is not enough, just slide a few more times.

The idea of data capture is probably like this, because there is no time to make trouble, so there is no APP automation, in fact, you can set up a simulator + keystroke wizard, or appium, but you are too lazy to be afraid of trouble, and you can take a stroke when you go to work as a sport.

Data display time problem, there is no data analysis, but the data is very clean, and do not need to deal with any keywords Mongodb, Pandas, Pyecharts took the Hangzhou cross-city ride data on Saturday combined with Aliyun's DataV to make a simple flight chart, the kind that can move, click to read the original text to view the dynamic picture.

At this point, I believe you have a deeper understanding of "how to use the Python bag grabbing tool to check the surrounding carpool". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.