Shulou(Shulou.com)06/01 Report--

How to greatly improve the access speed of requests, I believe that many inexperienced people are helpless about this, for this reason this article summarizes the causes and solutions of the problem, through this article I hope you can solve this problem.

I made a spam-filtering HTTP interface. There are now 10 million messages that need to pass through this interface for garbage detection.

At first my code looked like this:

import requests

messages = ['Article 1',' Article 2','Article 3']

for message in messages:

resp = requests.post(url, json={'msg': message}).json()

if resp['trash']:

print ('is spam message')

Let's write a code to see how fast it runs:

A hundred visits to Baidu took 20 seconds. Then I have 10 million messages. That's too long.

Is there any way to speed it up? In addition to multithreading, aiohttp, or simply Scrapy, as discussed in our previous article, it is possible to keep requests connected and thus reduce the time consumption of frequent TCP three-way handshakes.

So how do you keep requests connected? It's actually quite simple, using the Session object.

Modified code:

import requests

import time

start = time.time()

session = requests.Session()

for _ in range(100):

resp = session.get('https://baidu.com').content.decode()

end = time.time()

print(f'100 web page visits, time: {end - start}')

The operation effect is as shown in the following figure:

Performance has been significantly improved. It takes only 5 seconds to access 100 pages.

In the official documentation [1], requests also state that Session objects can remain connected:



The Session object allows you to persist certain parameters across requests. It also persists cookies across all requests made from the Session instance, and will use urllib3's connection pooling. So if you're making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase (see HTTP persistent connection).

"

Excellent news - thanks to urllib3, keep-alive is 100% automatic within a session! Any requests that you make within a session will automatically reuse the appropriate connection!

"After reading the above, do you know how to greatly improve the speed of requests? If you still want to learn more skills or want to know more related content, welcome to pay attention to the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.