Shulou(Shulou.com)06/01 Report--

This article mainly introduces how http completes the long link of continuous data transmission through StreamingHttpResponse, which has certain reference value, which can be used for reference by interested friends. I hope you can learn a lot after reading this article.

Http accomplishes the problem of continuous data transmission and long link through StreamingHttpResponse

Transfer result flow between http services

An algorithm encapsulated by flask and a background encapsulated by django. I hope to call the algorithm interface of flask through requests in django. Flask can analyze and return a frame of results, pursuing the real-time return of the analysis results, instead of returning the results completely after complete analysis.

In order to return the results completely, there are three modes that come to mind for the time being:

Ask and answer: wait for the complete analysis results, and then return, the worst use of this

I want you to give (long link): flask returns a generator,django to fetch next and get the next result

You have (ideal, long link): set up a long link, and flask will return every time you analyze a frame of result.

One result, until the end of the analysis, close the connection

See that there is a flask_socketio in flask to establish a socket connection. There is no experiment yet.

Temporarily using StreamingHttpResponse,generater can realize the feeling of real-time analysis, which belongs to the third mode (you have what you give)

The StreamingHttpResponse of django can return generater,request when the API that returns generate is called, the stream is processed through the closing of contextlib:

Output # django algorithm side Output stream from django.http import StreamingHttpResponsedef stream_response (request): def generate (): for i in range (10): print (I) yield'hi'+ str (I) print ('sleep 3') time.sleep (1) return StreamingHttpResponse (generate () ) # flask algorithm side Output stream @ app.route ('/ re', methods= ('POST') ) def re (): @ flask.stream_with_context def generate (): for i in range (10): print (I) yield'hi'+ str (I) print ('sleep 3') time.sleep (1) return flask.Response (generate ()) input

Regardless of flask,django, it can be implemented through request,contextlib

# flask algorithm side @ app.route ('/ test', methods= ['POST',' GET']) def test (): url = 'http://172.16.68.151:8000/test2' from contextlib import closing with closing (requests.get (url, stream=True)) as R1: for i in r1.iter_content (): print (I) StreamingHttpResponse and HttpResponse

When modifying the previous file download function, it was found that a file had 5G. When implemented with HttpResponse, the server returned a 502 error. When looking at nginx log, it was found that nginx log recorded: upstream prematurely closed connection while reading response header from upstream. I think the nginx server timed out while fetching data from upstream.

Checked a lot of ways, modified the configuration of nginx, but still timed out.

In despair, I checked the documents of Django and found StreamingHttpResponse. I tried to improve my efficiency a lot.

Later, after a closer inspection, it was found that when HttpResponse used the file iterator:

HttpResponse will consume the iterator immediately, store its content as a string, and discard it.

HttpResponse uses the iterator object directly, stores the contents of the iterator object in the city string, and then returns it to the client, freeing up memory. You can see that this is a very time-consuming and memory-consuming process when the file gets larger.

StreamingHttpResponse streams the contents of the file.

The explanation of StreamingHttpResponse in the official document is:

The StreamingHttpResponse class is used to stream a response from Django to the browser. You might want to do this if generating the response takes too long or uses too much memory.

This is a very time-saving and memory-saving method. However, because the StreamingHttpResponse file transfer process continues throughout the response process, this may degrade the performance of the server.

Thank you for reading this article carefully. I hope the article "how http completes continuous long data transmission through StreamingHttpResponse" shared by the editor will be helpful to everyone. At the same time, I also hope that you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.