Shulou(Shulou.com)06/03 Report--

This article mainly introduces "the optimization skills of python code". In the daily operation, I believe many people have doubts about the optimization skills of python code. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "optimization skills of python code". Next, please follow the editor to study!

Common skills of Python Code Optimization

Code optimization can make the program run faster, it makes the program run more efficiently without changing the running result of the program, according to the principle of 80x20, it usually takes 80% of the work to realize the refactoring, optimization, extension and documentation-related things of the program. Optimization usually includes two aspects: reducing the size of the code and improving the running efficiency of the code.

Improve the algorithm and select the appropriate data structure

A good algorithm can play a key role in performance, so the first point of performance improvement is the improvement of the algorithm. The order of time complexity of the algorithm is as follows:

O (1)-> O (lg n)-> O (n lg n)-> O (n ^ 2)-> O (n ^ 3)-> O (n ^ k)-> O (k ^ n)-> O (n!)

Therefore, if the algorithm can be improved in terms of time complexity, the improvement in performance is self-evident. However, the improvement of the specific algorithm does not fall within the scope of this article, readers can refer to this aspect of information. The following will focus on the choice of data structures.

Dictionary (dictionary) and list (list)

Hash table is used in the Python dictionary, so the complexity of the lookup operation is O (1), while the list is actually an array. In list, the lookup needs to traverse the entire list, and its complexity is O (n), so the dictionary of operations such as lookup and access to members is faster than list.

Listing 1. Code dict.py

From time import time t = time () list = ['dfdf','apple','pddf','ind','basic','none','baecr','var','bana','dd','wrd'] # list = dict.fromkeys (list) True) print list filter = [] for i in range (1000000): for find in ['is','hat','new','list','old','.']: if find not in list: filter.append (find) print "total run time:" print time ()-t

The above code probably requires 16.09seconds to run. If you uncomment the line # list = dict.fromkeys (list,True) and convert list to a dictionary and then run it, it takes about 8.375 seconds, which is about half the efficiency. Therefore, when multiple data members are needed for frequent lookup or access, it is a better choice to use dict instead of list.

Collection (set) and list (list)

Set's union and intersection,difference operations are faster than list's iterations. So if it comes to finding the intersection of list, the problem of union or difference can be converted to set.

Listing 2. Find the intersection of list:

From time import time t = time () lista= [1 for i in range 2: for b in listb: if a = b: intersection.append (a) print "total run time:" print time ()-t

The running time of the above program is approximately:

Total run time: 38.4070000648

Listing 3. Use set to find the intersection

From time import time t = time () lista= [1 set (lista) & set (listb)) print "total run time:" print time ()-t

After changing to set, the running time of the program is reduced to 8.75, which is increased by more than 4 times, and the running time is greatly shortened. Readers can test themselves using the other operations in Table 1.

Table 1. Common usage of set

Syntax operation instructions set (list1) | set (list2) union contains a new collection of all list1 and list2 data set (list1) & set (list2) intersection contains a new collection of elements common to list1 and list2 set (list1)-set (list2) difference appears in list1 but does not appear in list2

Optimization of the cycle

The principle of the optimization of the cycle is to reduce the amount of calculation in the cycle as much as possible, and if there are multiple cycles, the calculation of the inner layer will be raised to the upper level as far as possible. The following examples are used to compare the performance improvement brought about by loop optimization. In listing 4, without loop optimization, the approximate run time is about 132.375.

Listing 4. Before optimizing the cycle

From time import time t = time () lista = [1 len (lista)): for b in range (len (listb)): x=lista [a] + listb [b] print "total run time:" print time ()-t

Now do the following optimization, bringing the length calculation out of the loop, replacing range with xrange, and raising the calculation lista [a] of the third layer to the second layer of the loop.

Listing 5. After cycle optimization

From time import time t = time () lista = [1, 2, 3, 4, 5, 6, 7, 8 Len1=len (lista) len2=len (listb) for i in xrange (1000000): for an in xrange (len1): temp=lista [a] for b in xrange (len2): x=temp+listb [b] print "total run time:" print time ()-t

The running time of the optimized program is reduced to 102.171999931. In listing 4, the number of times lista [a] is calculated is 1000000 / 1010, compared with 1000000 / 1010 in the optimized code, which is significantly reduced, resulting in a performance improvement.

Make full use of the characteristics of Lazy if-evaluation

In python, the conditional expression is lazy evaluation, which means that if there is a conditional expression if x and y, the value of the y expression will no longer be evaluated if x is false. Therefore, this feature can be used to improve the efficiency of the program to some extent.

Listing 6. Take advantage of the characteristics of Lazy if-evaluation

From time import time t = time () abbreviations = ['cf.',' e.g.cake, 'ex.',' etc.', 'fig.',' i.e.vs.', 'Mr.',' vs.'] For i in range (1000000): for w in ('Mr.',' Hat', 'is',' chasing', 'the',' black', 'cat','.): if w in abbreviations: # if w [- 1] ='. And w in abbreviations: pass print "total run time:" print time ()-t

Before optimization, the run time of the program is about 8.84. if you use comment lines instead of the first if, the run time is about 6.17.

Optimization of string

The string object in python is immutable, so any string operations such as concatenation, modification, etc., will result in a new string object, not based on the original string, so this persistent copy will affect the performance of python to some extent. String optimization is also an important aspect of improving performance, especially when dealing with a lot of text. String optimization mainly focuses on the following aspects:

Try to use join () instead of + for string concatenation: using + for string concatenation in listing 7 takes about 0.125 seconds, while using join shortens it to 0.016s. Therefore, join is faster than + in character manipulation, so try to use join instead of +.

Listing 7. Use join instead of + connection string

From time import time t = time () s = "" list = ['aaqunyuzhuo () s = "for i in range (10000): for substr in list: swordsmen = substr print" total run time: "print time ()-t

At the same time avoid:

S = "" for x in list: s + = func (x)

Instead, use:

Slist = [func (elt) for elt in somelist] s = ".join (slist)

Select the built-in function when you can use regular expressions or built-in functions for strings. Such as str.isalpha (), str.isdigit (), str.startswith ('x, 'yz')), str.endswith (' x, 'yz')

Formatting characters is faster than reading in direct concatenation, so use the

Out = "% s%s%s%s"% (head, prologue, query, tail)

And avoid

Out = "" + head + prologue + query + tail + ""

Use list parsing (list comprehension) and generator expressions (generator expression)

List parsing is more efficient than rebuilding a new list in a loop, so we can take advantage of this feature to improve running efficiency.

From time import time t = time () list = ['ahem,' dfdf','apple','pddf','ind','basic','none','baecr','var','bana','dd',' 'wrd'] total= [] for i in range (1000000): for w in list: total.append (w) print "total run time:" print time ()-t

Use a list to resolve:

For i in range (1000000): a = [w for w in list]

It takes about 17s for the above code to run directly, but after using list parsing instead, the running time is reduced to 9.29s. It's nearly halved. Generator expression is a new content introduced in 2.4, and the syntax is similar to list parsing, but when dealing with a large amount of data, the advantage of generator expression is more obvious, it does not create a list, it just returns a generator, so it is more efficient. In the above example, the code a = [w for w in list] is changed to a = (w for w in list), and the running time is further reduced to about 2.98s.

Other optimization techniques

If you need to exchange the values of two variables, use a meme baccalaurea instead of an intermediate variable.

> > from timeit import Timer > > Timer. Timeit () 0.25154118749729365 > > Timer. Timeit () 0.17156677734181258 >

Using xrange instead of range; using xrange during loops saves a lot of system memory because xrange () produces only one integer element per call in the sequence. Range (), on the other hand, will directly return the complete list of elements, which will have unnecessary overhead when used in the loop. Xrange no longer exists in python3, where range provides an iterator that can traverse a range of arbitrary lengths.

Use local variables to avoid the "global" keyword. Python accesses local variables much faster than global variables, so you can take advantage of this feature to improve performance.

If done is not None is faster than the statement if done! = None, and readers can verify it for themselves.

In time-consuming loops, you can change the function call to inline

Use cascading to compare "x

< y < z" 而不是 "x < y and y < z"； while 1 要比 while True 更快（当然后者的可读性更好）； build in 函数通常较快，add(a,b) 要优于 a+b。 回页首 定位程序性能瓶颈 对代码优化的前提是需要了解性能瓶颈在什么地方，程序运行的主要时间是消耗在哪里，对于比较复杂的代码可以借助一些工具来定位，python 内置了丰富的性能分析工具，如 profile,cProfile 与 hotshot 等。其中 Profiler 是 python 自带的一组程序，能够描述程序运行时候的性能，并提供各种统计帮助用户定位程序的性能瓶颈。Python 标准模块提供三种 profilers:cProfile,profile 以及 hotshot。 profile 的使用非常简单，只需要在使用之前进行 import 即可。具体实例如下： 清单 8. 使用 profile 进行性能分析 import profile def profileTest(): Total =1; for i in range(10): Total=Total*(i+1) print Total return Total if __name__ == "__main__": profile.run("profileTest()") 程序的运行结果如下： 图 1. 性能分析结果 其中输出每列的具体解释如下： ncalls：表示函数调用的次数； tottime：表示指定函数的总的运行时间，除掉函数中调用子函数的运行时间； percall：（第一个 percall）等于 tottime/ncalls； cumtime：表示该函数及其所有子函数的调用运行的时间，即函数开始调用到返回的时间； percall：（第二个 percall）即函数运行一次的平均时间，等于 cumtime/ncalls； filename:lineno(function)：每个函数调用的具体信息； 如果需要将输出以日志的形式保存，只需要在调用的时候加入另外一个参数。如 profile.run("profileTest()","testprof")。 对于 profile 的剖析数据，如果以二进制文件的时候保存结果的时候，可以通过 pstats 模块进行文本报表分析，它支持多种形式的报表输出，是文本界面下一个较为实用的工具。使用非常简单： import pstats p = pstats.Stats('testprof') p.sort_stats("name").print_stats() 其中 sort_stats() 方法能够对剖分数据进行排序， 可以接受多个排序字段，如 sort_stats('name', 'file') 将首先按照函数名称进行排序，然后再按照文件名进行排序。常见的排序字段有 calls( 被调用的次数 )，time（函数内部运行时间），cumulative（运行的总时间）等。此外 pstats 也提供了命令行交互工具，执行 python - m pstats 后可以通过 help 了解更多使用方式。 对于大型应用程序，如果能够将性能分析的结果以图形的方式呈现，将会非常实用和直观，常见的可视化工具有 Gprof2Dot，visualpytune，KCacheGrind 等，读者可以自行查阅相关官网，本文不做详细讨论。 回页首 Python 性能优化工具 Python 性能优化除了改进算法，选用合适的数据结构之外，还有几种关键的技术，比如将关键 python 代码部分重写成 C 扩展模块，或者选用在性能上更为优化的解释器等，这些在本文中统称为优化工具。python 有很多自带的优化工具，如 Psyco，Pypy，Cython，Pyrex 等，这些优化工具各有千秋，本节选择几种进行介绍。 Psyco psyco 是一个 just-in-time 的编译器，它能够在不改变源代码的情况下提高一定的性能，Psyco 将操作编译成有点优化的机器码，其操作分成三个不同的级别，有"运行时"、"编译时"和"虚拟时"变量。并根据需要提高和降低变量的级别。运行时变量只是常规 Python 解释器处理的原始字节码和对象结构。一旦 Psyco 将操作编译成机器码，那么编译时变量就会在机器寄存器和可直接访问的内存位置中表示。同时 python 能高速缓存已编译的机器码以备今后重用，这样能节省一点时间。但 Psyco 也有其缺点，其本身运行所占内存较大。目前 psyco 已经不在 python2.7 中支持，而且不再提供维护和更新了，对其感兴趣的可以参考 http://psyco.sourceforge.net/ Pypy PyPy 表示 "用 Python 实现的 Python"，但实际上它是使用一个称为 RPython 的 Python 子集实现的，能够将 Python 代码转成 C， .NET， Java 等语言和平台的代码。PyPy 集成了一种即时 (JIT) 编译器。和许多编译器，解释器不同，它不关心 Python 代码的词法分析和语法树。 因为它是用 Python 语言写的，所以它直接利用 Python 语言的 Code Object.。 Code Object 是 Python 字节码的表示，也就是说， PyPy 直接分析 Python 代码所对应的字节码 ,，这些字节码即不是以字符形式也不是以某种二进制格式保存在文件中， 而在 Python 运行环境中。目前版本是 1.8. 支持不同的平台安装，windows 上安装 Pypy 需要先下载 https://bitbucket.org/pypy/pypy/downloads/pypy-1.8-win32.zip，然后解压到相关的目录，并将解压后的路径添加到环境变量 path 中即可。在命令行运行 pypy，如果出现如下错误："没有找到 MSVCR100.dll, 因此这个应用程序未能启动，重新安装应用程序可能会修复此问题"，则还需要在微软的官网上下载 VS 2010 runtime libraries 解决该问题。具体地址为 http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=5555 安装成功后在命令行里运行 pypy，输出结果如下： C:\Documents and Settings\Administrator>

Pypy Python 2.7.2 (0e28b379d8b3, Feb 09 2012, 18:31:47) [PyPy 1.8.0 with MSC v.1500 32 bit] on win32 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``PyPy is vast, and contains multitudes'' >

Take the loop in listing 5 as an example and run it using python and pypy respectively, and the results are as follows:

C:\ Documents and Settings\ Administrator\ Desktop\ doc\ python > pypy loop.py total run time: 8.42199993134 C:\ Documents and Settings\ Administrator\ Desktop\ doc\ python > python loop.py total run time: 106.391000032

It can be seen that using pypy to compile and run the program, its efficiency is greatly improved.

Cython

Cython is a language implemented in python, can be used to write python extensions, the libraries written with it can be loaded through import, the performance is faster than python. In cython, you can load python extensions (such as import math), load the header file of c's library (such as cdef extern from "math.h"), and use it to write python code. Rewrite the key parts into C extension modules

Installation of Linux Cpython:

Step 1: download

[root@v5254085f259 cpython] # wget-N http://cython.org/release/Cython-0.15.1.zip-- 2012-04-16 22 root@v5254085f259 cpython 08 root@v5254085f259 cpython 35-http://cython.org/release/Cython-0.15.1.zip Resolving cython.org... 128.208.160.197 Connecting to cython.org | 128.208.160.197 |: 80. Connected. HTTP request sent, awaiting response... 200 OK Length: 2200299 (2.1m) [application/zip] Saving to: `100% [= >] 2200299 1.96M/s in 1.1s 2012-04-16 22:08:37 (1.96 MB/s)-`Cython-0.15.1.zip' saved [2200299 1.96M/s in 2200299]

Step 2: decompress

[root@v5254085f259 cpython] # unzip-o Cython-0.15.1.zip

Step 3: install

Python setup.py install

Enter cython directly after the installation is completed. If the following appears, the installation is successful.

[root@v5254085f259 Cython-0.15.1] # cython Cython (http://cython.org) is a compiler for code written in the Cython language. Cython is based on Pyrex by Greg Ewing. Usage: cython [options] sourcefile. {pyx,py}... Options:-V,-- version Display version number of cython compiler-l,-- create-listing Write error messages to a listing file-I,-- include-dir Search for include files in named directory (multiple include directories are allowed). -o,-- output-file Specify name of generated C file-t,-- timestamps Only compile newer source files-f,-- force Compile all source files (overrides implied-t)-Q,-- quiet Don't print module names in recursive mode-v,-- verbose Be verbose, print file names on multiple compil ation-p -- embed-positions If specified, the positions in Cython files of each function definition is embedded in its docstring. -- cleanup Release interned objects on python exit, for memory debugging. Level indicates aggressiveness, default 0 releases nothing. W,-- working Sets the working directory for Cython (the directory modules are searched from)-- gdb Output debug information for cygdb-D,-- no-docstrings Strip docstrings from the compiled module. -a,-annotate Produce a colorized HTML version of the source. -- line-directives Produce # line directives pointing to the .pyx source-- cplus Output a C++ rather than C file. -embed [=] Generate a main () function that embeds the Python interpreter. -2 Compile based on Python-2 syntax and code seman tics. -3 Compile based on Python-3 syntax and code seman tics. -- fast-fail Abort the compilation on the first error-- warning-error,-Werror Make all warnings into errors-- warning-extra,-- Wextra Enable extra warnings-X,-- directive = [, > > import pyximport; pyximport.install () > import sum > sum.sum (1Magne3)

Here's a simple performance comparison:

Listing 9. Cython test code

From time import time def test (int n): cdef int a = 0 cdef int i for i in xrange (n): i return a t = time () test (10000000) print "total run time:" print time ()-t

Test results:

[GCC 4.0.2 20051125 (Red Hat 4.0.2-8)] on linux2 Type "help", "copyright", "credits" or "license" for more information. > import pyximport; pyximport.install () > import ctest total run time: 0.00714015960693

Listing 10. Python test code

From time import time def test (n): a = 0; for i in xrange (n): for i in xrange = i return a t = time () test (10000000) print "total run time:" print time ()-t [root@v5254085f259 test] # python test.py total run time: 0.971596002579

From the above comparison, we can see that the speed of using Cython has increased by nearly 100 times.

At this point, the study of "python code optimization skills" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.