Shulou(Shulou.com)06/02 Report--

This article mainly introduces "Gevent meets installation and use". In daily operation, I believe many people have doubts about Gevent meeting installation and use. Xiaobian consulted various materials and sorted out simple and easy operation methods. I hope to help you answer the doubts of "Gevent meets installation and use"! Next, please follow the small series to learn together!

Gevent is a Python concurrency framework based on greenlets, with microthreaded greenlets at its core, made efficient by using epoll event listening mechanisms and many other optimizations.

Compared with greenlet and eventlet, the performance is slightly lower, but the API it encapsulates is very complete. The best thing is to provide a monkey class that can directly convert existing Python-based threads into greenlet, which is equivalent to proxy (patch).

1. Installation

Gevent relies on libevent and greenlet, which need to be installed separately.

Python

#libevent 1.4.x

sudo apt-get install libevent-dev

#python_dev

sudo apt-get install python-dev

#easy_install

wget -q http://peak.telecommunity.com/dist/ez_setup.py

sudo python ./ ez_setup.py

#greenlet

wget http://pypi.python.org/packages/source/g/greenlet/greenlet-0.3.1.tar.gz#md5=8d75d7f3f659e915e286e1b0fa0e1c4d

tar -xzvf greenlet-0.3.1.tar.gz

cd greenlet-0.3.1/

sudo python setup.py install

#gevent

wget http://pypi.python.org/packages/source/g/gevent/gevent-0.13.6.tar.gz#md5=7c836ce2315d44ba0af6134efbcd38c9

tar -xzvf gevent-0.13.6.tar.gz

cd gevent-0.13.6/

sudo python setup.py install

At this point, the installation is complete.

2. Test code: XML-RPC

XML-RPC with thread support must be used here, otherwise the advantages of gevent cannot be exploited!

Traditional version:

It should be noted that this and a lot of data describes not a single thread, but a select version, so sometimes it performs better than the threaded version.

Python

from SocketServer import ThreadingMixIn

from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler

from SocketServer import TCPServer

TCPServer.request_queue_size = 10000

#Logic function

def add(a, b):

return a + b

#Logic function 2

def gen(n):

return '0' * n

#create server

server = SimpleXMLRPCServer(('', 8080), SimpleXMLRPCRequestHandler,False)

server.register_function(add, "add")

server.register_function(gen, "gen")

server.serve_forever()

Thread version: 3. Test client

Python

from xmlrpclib import ServerProxy

#Execute RPC

server = ServerProxy("http://localhost:8080")

#print server.add(3,5)

print server.gen(2048)

4. XML-RPC wrapped by monkey of gevent

monkey is a non-invasive patch, just need to show the call you need patch on the line, although I used three lines, in fact, patch_all()

Python

from SocketServer import ThreadingMixIn

from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler

from gevent import monkey

#Threaded XML-RPC && Monkey Patch

monkey.patch_socket() #Just 2 line!

monkey.patch_thread() #Just 3 line!

monkey.patch_select() #Just 3 line!

class TXMLRPCServer(ThreadingMixIn, SimpleXMLRPCServer): pass

#Logic function

def add(a, b):

return a + b

#Logic function 2

def gen(n):

return "0" * n

#create server

server = TXMLRPCServer(('', 8080), SimpleXMLRPCRequestHandler)

server.register_function(add, "add")

server.register_function(gen, "gen")

server.serve_forever()

5. Test results

Now there is only one machine, go to the laboratory in the afternoon after the two machines run, put the results. I still hope for gevent. I hope it's not too bad.

Special configuration of client:

echo -e '1024t65535' | sudo tee /proc/sys/net/ipv4/ip_local_port_range

echo 1 | sudo tee /proc/sys/net/ipv4/tcp_tw_recycle

echo 1 | sudo tee /proc/sys/net/ipv4/tcp_syncookies

ulimit -n 10240

Special configuration on the server side:

echo "10152 65535″ > /proc/sys/net/ipv4/ip_local_port_range

echo 1 | sudo tee /proc/sys/net/ipv4/tcp_tw_recycle

sysctl -w fs.file-max=128000

sysctl -w net.ipv4.tcp_keepalive_time=300

sysctl -w net.core.somaxconn=250000

sysctl -w net.ipv4.tcp_max_syn_backlog=2500

sysctl -w net.core.netdev_max_backlog=2500

ulimit -n 10240

Then say let everyone disappointed results: test effect is very failed, often abnormal situation, according to my analysis is the default XML-RPC no backlog(or default too low), resulting in a large pressure, will fail accept, resulting in RESET (connection rejected). Xiamen Forklift Rental Company

So don't expect too much from monkey patch, it's closely related to the original code.

Supplement: We have found a way to modify the default backlog as follows:

Python

from SocketServer import TCPServer

#Modify this global variable

TCPServer.request_queue_size = 5000

Of course, the test data shows that don't be too obsessed with monkey, it's just a legend ~

Test data:

c=500 n=50000

Default: 2845/s, 8M

Multithreading: 1966/s, 51M

gevent：1888/s, 11M

c=1000 n=100000

Default: 3096/s, 8M

Multithreading: 1895/s, 52M

gevent：1936/s, 11M

c=5000 n=500000

Default: 3009/s, 8M

Multithreading: Failed, unable to create new thread

gevent：1988/s, 11M

c=10000 n=1000000

Default: 2883/s, 8M

Multithreading: Failed, unable to create new thread

gevent：1992/s, 20M

The advantage of monkey is: save memory, I am compared with thread.

I carefully analyzed it, XML-RPC still uses a large proportion of CPU, compared to direct http calculations, xmlrpc is still CPU intensive.

In this case, where CPU usage is high and microgreenlets need to be fought repeatedly, gevent does not have an advantage.

Or, to put it another way, the tester isn't powerful enough to feed gevent (you can see that gevent performance goes up rather than down as concurrent threads go up, whereas the default is declining).

At this point, the study of "Gevent satisfies installation and use" is over. I hope to solve everyone's doubts. Theory and practice can better match to help everyone learn, go and try it! If you want to continue learning more relevant knowledge, please continue to pay attention to the website, Xiaobian will continue to strive to bring more practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.