Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "how to use python numexpr". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

The beginning

Python language is widely used in data analysis and machine learning. However, due to the underlying characteristics of python, the low speed of python has been widely criticized. Among them, the widespread use of numpy and pandas has increased the speed of data processing and machine learning by several levels.

However, with more and more data, many people no longer meet the speed of numpy and pandas, so they have withdrawn from a number of accelerated optimization expansion packages. This article focuses on a lightweight but powerful python extension package "NumExpr" to see how it parses mathematical formulas efficiently.

NumExpr

The use of NumExpr is as simple as enclosing the original numpy statement in double quotes and using the evaluate method call in numexpr.

Step 1: need to introduce numexpr and numpy expansion packs

Import numexpr as ne

Import numpy as np

Step 2: create two numpy array-an and b

The number of data contained in an and b is 1 million.

When we need to perform simple addition, subtraction, multiplication and division, the efficiency of numexpr has been well demonstrated. As shown in the figure above, by performing 2 * a + 3 * b, it takes 3.39 milliseconds to operate directly. However, if we are using ne.evaluate acceleration, we can reduce the time to 1.55ms.

How does numexpr perform in more complex mathematical expression operations?

When we use the mathematical expression shown above, it takes 28.3 ms for normal execution. Through the acceleration of numexpr, only 3.03 ms is needed.

Note that numexpr can recognize the sin function, so we don't need to write np.sin in evaluate, we can write sin directly.

Conclusion: the larger the amount of data processed and the more complex the mathematical calculation, the more obvious the acceleration effect of numexpr.

Comparison between numpy and numexpr

We can see that when the number of elements of np.array exceeds that of 10e8, the acceleration effect is more significant.

Numexpr also supports the acceleration of logical and plural expressions, which can be compared by interested readers.

Numexpr multithreading acceleration

Numexpr also has an important acceleration tool, multithreaded operation. The number of threads can be set through ne.set_num_threads (1), and more threads indicate that the program can evaluate mathematical expressions at the same time.

As shown above, if we set up a single thread, it takes 13.4 ms for the program to run. If dual threads are set, the speed can be doubled.

Acceleration of pandas by numexpr

Numexpr is designed for numpy. Similarly, we know that pandas is also based on numpy. Naturally, numexpr can also be used to accelerate pandas.

One of the eval methods in pandas is to use numexpr to optimize and accelerate pandas code.

When we build several dataframe of pandas and then operate on them, pd.eval can accelerate the program from 47.4 ms to 17.6 ms.

That's all for the content of "how to use python numexpr". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.