Shulou(Shulou.com)06/03 Report--

Python3 under multiprocessing, threading and gevent performance comparison and process pool, thread pool and collaborative pool performance comparison, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain in detail for you, people with this need can come to learn, I hope you can get something.

At present, computer programs generally encounter two types of Imax O: hard disk Imax O and network Imax O. I will analyze the comparison of the efficiency of processes, threads, and co-programs under python3 according to the scenario of network Iamp O. The process adopts the multiprocessing.Pool process pool, the thread is the process pool encapsulated by itself, and the co-program uses the library of gevent. Compare python3's native urlllib.request with open source requests. The code is as follows:

Import urllib.requestimport requestsimport timeimport multiprocessingimport threadingimport queuedef startTimer (): return time.time () def ticT (startTime): useTime = time.time ()-startTime return round (useTime, 3) # def tic (startTime, name): # useTime = time.time ()-startTime# print ('[% s] use time:% 1.3f'% (name, useTime)) def download_urllib (url): req = urllib.request.Request (url Headers= {'user-agent':' Mozilla/5.0'}) res = urllib.request.urlopen (req) data = res.read () try: data = data.decode ('gbk') except UnicodeDecodeError: data = data.decode (' utf8', 'ignore') return res.status, datadef download_requests (url): req = requests.get (url Headers= {'user-agent':' Mozilla/5.0'}) return req.status_code, req.textclass threadPoolManager: def _ _ init__ (self,urls, workNum=10000,threadNum=20): self.workQueue=queue.Queue () self.threadPool= [] self.__initWorkQueue (urls) self.__initThreadPool (threadNum) def _ initWorkQueue (self Urls): for i in urls: self.workQueue.put ((download_requests,i)) def _ initThreadPool (self ThreadNum): for i in range (threadNum): self.threadPool.append (work (self.workQueue)) def waitAllComplete (self): for i in self.threadPool: if i.isAlive (): i.join () class work (threading.Thread): def _ _ init__ (self WorkQueue): threading.Thread.__init__ (self) self.workQueue=workQueue self.start () def run (self): while True: if self.workQueue.qsize (): do Args=self.workQueue.get (block=False) do (args) self.workQueue.task_done () else: breakurls = ['http://www.ustchacker.com'] * 10urllibL = [] requestsL = [] multiPool = [] threadPool = [] N = 20PoolNum = 100for i in range (N): print ('start% d try'% I) urllibT = startTimer () jobs = [download_urllib (url) for url in urls] # for status Data in jobs: # print (status, data [: 10]) # tic (urllibT, 'urllib.request') urllibL.append (ticT (urllibT)) print (' 1') requestsT = startTimer () jobs = [download_requests (url) for url in urls] # for status, data in jobs: # print (status, data [: 10]) # tic (requestsT) 'requests') requestsL.append (ticT (requestsT)) print (' 2') requestsT = startTimer () pool = multiprocessing.Pool (PoolNum) data = pool.map (download_requests, urls) pool.close () pool.join () multiPool.append (ticT (requestsT)) print ('3') requestsT = startTimer () pool = threadPoolManager (urls ThreadNum=PoolNum) pool.waitAllComplete () threadPool.append (ticT (requestsT)) print ('4') import matplotlib.pyplot as pltx = list (range (1, Numb1)) plt.plot (x, urllibL, label='urllib') plt.plot (x, requestsL, label='requests') plt.plot (x, multiPool, label='requests MultiPool') plt.plot (x, threadPool, label='requests threadPool') plt.xlabel ('test number') plt.ylabel (' time (s)') plt.legend () plt.show ()

The running results are as follows:

As can be seen from the above figure, the urllib.request efficiency of python3 is not as efficient as the open source requests,multiprocessing process pool, but it is still lower than the thread pool encapsulated by itself, in part because the overhead of creating and scheduling processes is higher than that of creating threads (I included the cost of creation in the test program).

Here is the test code for gevent:

Import urllib.requestimport requestsimport timeimport gevent.poolimport gevent.monkeygevent.monkey.patch_all () def startTimer (): return time.time () def ticT (startTime): useTime = time.time ()-startTime return round (useTime, 3) # def tic (startTime, name): # useTime = time.time ()-startTime# print ('[% s] use time:% 1.3f'% (name, useTime)) def download_urllib (url): req = urllib.request.Request (url Headers= {'user-agent':' Mozilla/5.0'}) res = urllib.request.urlopen (req) data = res.read () try: data = data.decode ('gbk') except UnicodeDecodeError: data = data.decode (' utf8', 'ignore') return res.status, datadef download_requests (url): req = requests.get (url Headers= {'user-agent':' Mozilla/5.0'}) return req.status_code, req.texturls = ['http://www.ustchacker.com'] * 10urllibL = [] requestsL = [] reqPool = [] reqSpawn = [] N = 20PoolNum = 100for i in range (N): print (' start% d try'% I) urllibT = startTimer () jobs = [download_urllib (url) for url in urls] # for status Data in jobs: # print (status, data [: 10]) # tic (urllibT, 'urllib.request') urllibL.append (ticT (urllibT)) print (' 1') requestsT = startTimer () jobs = [download_requests (url) for url in urls] # for status, data in jobs: # print (status, data [: 10]) # tic (requestsT) 'requests') requestsL.append (ticT (requestsT)) print (' 2') requestsT = startTimer () pool = gevent.pool.Pool (PoolNum) data = pool.map (download_requests, urls) # for status, text in data: # print (status, text [: 10]) # tic (requestsT 'requests with gevent.pool') reqPool.append (ticT (requestsT)) print (' 3') requestsT = startTimer () jobs = [gevent.spawn (download_requests, url) for url in urls] gevent.joinall (jobs) # for i in jobs: # print (i.value [0], i.value [1] [: 10]) # tic (requestsT 'requests with gevent.spawn') reqSpawn.append (ticT (requestsT)) print (' 4') import matplotlib.pyplot as pltx = list (range (1, Numb1)) plt.plot (x, urllibL, label='urllib') plt.plot (x, requestsL, label='requests') plt.plot (x, reqPool, label='requests geventPool') plt.plot (x, reqSpawn, label='requests Spawn') plt.xlabel ('test number') plt.ylabel (' time (s)') plt.legend () plt.show ()

The running results are as follows:

As you can see from the figure above, gevent can greatly improve the performance of I gevent O-intensive tasks. Since the overhead of creating and scheduling protocols is much lower than that of threads, you can see that there is not much difference in performance whether using gevent's Spawn mode or Pool mode.

Because you need to use the monkey patch in gevent, it will improve the performance of gevent, but will affect the operation of multiprocessing. If you want to use it at the same time, you need the following code:

Gevent.monkey.patch_all (thread=False, socket=False, select=False)

But this can not give full play to the advantages of gevent, so it is not possible to compare multiprocessing Pool, threading Pool and gevent Pool in one program. However, comparing the two figures, it can be concluded that thread pool and gevent have the best performance, followed by process pool. Incidentally, it is concluded that the performance of the requests library is better than that of the urllib.request library.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.