Shulou(Shulou.com)06/02 Report--

This article mainly introduces "how to increase the running speed of Python by 100 times". In daily operation, I believe many people have doubts about how to increase the running speed of Python by 100 times. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the questions of "how to increase the running speed of Python by 100 times". Next, please follow the editor to study!

Let's take a look at the simplest example, which adds up from 1 to 100 million.

The original code:

Import time def foo (xpene y): tt = time.time () s = 0 for i in range (xpene y): s + = I print ('Time used: {} sec'.format (time.time ()-tt)) return s print (foo (1pm 100000000)

Results:

Time used: 6.779874801635742 sec 4999999950000000

Let's add a line of code and look at the results:

From numba import jit import time @ jit def foo (xpene y): tt = time.time () s = 0 for i in range (xpene y): s + = I print ('Time used: {} sec'.format (time.time ()-tt)) return s print (foo (1meme 100000000)

Results:

Time used: 0.04680037498474121 sec 4999999950000000

Is it more than 100 times faster?

So let's share "Why is the jit module of the numba library so awesome?"

After leaving Enthought, Travis Oliphant, the founder of NumPy, created CONTINUUM to deal with the application of Python big data. The recently launched Numba project can compile the Python function JIT dealing with NumPy array into machine code for execution, thus increasing the operation speed of the program hundreds of times.

The home page of the Numba project has detailed installation steps under Linux. Compiling LLVM takes some time.

Windows users can download and install several extension libraries such as LLVMPy, meta and numba from Unofficial Windows Binaries for Python Extension Packages.

Let's look at an example:

Import numba as nb from numba import jit @ jit ('f8 (f8 [:])') def sum1d (array): s = 0.0n = array.shape [0] for i in range (n): s + = array [I] return s import numpy as np array = np.random.random (10000)% timeit sum1d (array)% timeit np.sum (array)% timeit sum (array) 10000 loops, best of 3: 38.9 us per loop 10000 loops, best of 3: 32.3 us per loop 100 loops Best of 3: 12.4 ms per loop

Numba provides modifiers that compile its decorated function JIT into a machine code function and return a wrapper object that can call the machine code in Python. In order to compile the Python function into machine code that can be executed at high speed, we need to tell the JIT compiler the parameters of the function and the type of the return value. We can specify type information in a number of ways. In the above example, the type information is specified by a string'f8 (f8 [:])'. Where 'f8' represents an 8-byte double-precision floating point number, the' f8' in front of the parentheses indicates the return value type, the parameter type in the parentheses indicates the parameter type, and the'[:] 'represents an one-dimensional array. So the entire type string indicates that sum1d () is an one-dimensional array of double-precision floating-point numbers, and the return value is a double-precision floating-point number.

It is important to note that the functions generated by JIT can only operate on parameters of the specified type:

Print sum1d (np.ones (10, dtype=np.int32)) print sum1d (np.ones (10, dtype=np.float32)) print sum1d (np.ones (10, dtype=np.float64)) 1.2095376009e-312 1.46201599944e+185 10.0

If you want JIT to operate on all types of parameters, you can use autojit:

From numba import autojit @ autojit def sum1d2 (array): s = 0.0n = array.shape [0] for i in range (n): s + = array [I] return s% timeit sum1d2 (array) print sum1d2 (np.ones (10, dtype=np.int32)) print sum1d2 (np.ones (10, dtype=np.float32)) print sum1d2 (np.ones (10, dtype=np.float64)) 10000 loops, best of 3: 143 us per loop 10.0 10.0 10.0

Although autoit can dynamically generate machine code functions according to parameter types, the calculation speed is also reduced because it needs to check the parameter types every time. The use of numba is simple, basically using two modifiers, jit and autojit, and some type objects. The following program lists all the types supported by numba:

Print [obj for obj in nb.__dict__.values () if isinstance (obj, nb.minivect.minitypes.Type)] [size_t, Py_uintptr_t, uint16, complex128, float, complex256, void, int, long double, unsigned PY_LONG_LONG, uint32, complex256, complex64, object_, npy_intp, const char *, double, unsigned short, float, object_, float, uint64, uint32, uint8, complex128, uint16, int, int, uint8, complex64, int8, uint64, double, double, long double, long double, int32, int32, int32 Unsigned char, PY_LONG_LONG, int64, int16, unsigned long, int8, int16, int32, unsigned int, short, int64, Py_ssize_t]

working principle

Numba parses the ast syntax tree of the Python function through the meta module and adds the corresponding type information to each variable. Then call llvmpy to generate the machine code, and then generate the Python calling API of the machine code.

Meta module

By studying how numba works, we can find many useful tools. For example, the meta module can convert between program source code, ast syntax tree, and Python binary code. Let's look at an example:

Def add2 (a, b): return a + b

Decompile_func can decompile the code object of a function into an ast syntax tree, while str_ast can display the ast syntax tree visually. It is very helpful to use these two tools to learn the ast syntax tree of Python.

From meta.decompiler import decompile_func from meta.asttools import str_ast print str_ast (decompile_func (add2)) FunctionDef (args=arguments (args= [name (ctx=Param (), id='a'), Name (ctx=Param (), id='b')], defaults= [], kwarg=None, vararg=None), body= [Return (left=Name (ctx=Load (), id='a'), op=Add (), right=Name (ctx=Load (), id='b'))], decorator_list= [], name='add2')

Python_source can convert the ast syntax tree into Python source code:

From meta.asttools import python_source python_source (decompile_func (add2)) def add2 (a, b): return (a + b)

Decompile_pyc combines the above two, it can decompile the pyc or pyo files compiled by Python into source code. Let's write a tmp.py file and then compile it to tmp.pyc through py_compile.

With open ("tmp.py", "w") as f: f.write ("def square_sum (n): s = 0 for i in range (n): s + = iTunes 2 return s") import py_compile py_compile.compile (" tmp.py ")

The following calls decompile_pyc to display tmp.pyc as source code:

With open ("tmp.pyc", "rb") as f: decompile_pyc (f) def square_sum (n): s = 0 for i in range (n): s + = (I * * 2) return sllvmpy module

LLVM is a dynamic compiler, while llvmpy can dynamically create machine code by calling LLVM through Python. It is tedious to create machine code directly through llvmpy. For example, the following program creates a function that calculates the sum of two integers and calls it to calculate the result.

From llvm.core import Module, Type, Builder from llvm.ee import ExecutionEngine, GenericValue # Create a new module with a function implementing this: # # int add (int a, int b) {# return a + b #} # my_module = Module.new ('my_module') ty_int = Type.int () ty_func = Type.function (ty_int, [ty_int, ty_int]) f_add = my_module.add_function (ty_func Add) f_add.args [0] .name = "a" f_add.args [1] .name = "b" bb = f_add.append_basic_block ("entry") # IRBuilder for our basic block builder = Builder.new (bb) tmp = builder.add (f_add.args [0], f_add.args [1], "tmp") builder.ret (tmp) # Create an execution engine object. This will create a JIT compiler # on platforms that support it, or an interpreter otherwise ee = ExecutionEngine.new (my_module) # Each argument needs to be passed as a GenericValue object, which is a kind # of variant arg1 = GenericValue.int (ty_int, 100) arg2 = GenericValue.int (ty_int, 42) # Now let's compile and run! Retval = ee.run_function (f_add, [arg1, arg2]) # The return value is also GenericValue. Let's print it. Print "returned", retval.as_int () returned 142

F_add is a dynamically generated machine code function, which we can think of as a compiled function in C language. In the above program, we call this function through ee.run_function, but we can actually get its address and then call it through Python's ctypes module.

First, get the address of the f_add function through ee.get_pointer_to_function:

Addr = ee.get_pointer_to_function (f_add) addr 2975997968L

Then create a function type through ctypes.PYFUNCTYPE:

Import ctypes f_type = ctypes.PYFUNCTYPE (ctypes.c_int, ctypes.c_int, ctypes.c_int)

* convert the address of the function to a callable Python function through f_type, and call it:

F = f_type (addr) f (100,42) 142

What numba has done is:

Parse the ast syntax tree of Python function and modify it to add type information

The ast syntax tree with type information is dynamically converted into machine code functions through llvmpy, and then a wrapper function is created for machine code functions to be called by Python through a technique similar to ctypes.

At this point, the study on "how to increase the running speed of Python by 100 times" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.