Shulou(Shulou.com)06/03 Report--

This article mainly introduces how to use the combination of Numba and Cython to improve the running efficiency of python. It is very detailed and has certain reference value. Friends who are interested must finish reading it.

Numba

Numba is an just-in-time (JIT) compiler that converts Python code into native machine instructions for CPU and GPU. Code can be compiled at import time, run time, or in advance.

By using the jit decorator, using Numba is very easy:

As you know, in Python, all blocks of code are compiled into bytecode:

Code optimization

To optimize the Python code, Numba extracts a bytecode from the provided function and runs a set of parsers on it. The Python bytecode contains a series of small and simple instructions, so you can reconstruct the logic of the function from the bytecode without using source code from the Python implementation. The conversion process involves multiple stages, but Numba converts Python bytecodes into LLVM intermediate representations (IR).

Note that LLVM IR is a low-level programming language that is similar to assembly syntax and is independent of Python.

Numba mode

There are two modes in Numba: nopython and object. The former does not use the Python runtime and generates native code without Python dependencies. Native code is statically typed and runs very fast. The object mode, on the other hand, uses Python objects and Python C API, which usually does not result in significant speed improvements. In both cases, the Python code is compiled using LLVM.

What is LLVM?

LLVM is a compiler that takes a special intermediate representation (IR) of code and compiles it into machine code. The compilation process involves many additional passes, in which the compiler optimizes IR. The LLVM toolchain optimizes IR well, not only compiling code for Numba, but also optimizing Numba. The whole system is roughly as follows:

Python numba architecture

Advantages of Numba:

Ease of use

Automatic parallelization

Support for numpy operations and objects

GPU support

Numba's disadvantages:

Multi-level abstraction makes debugging and optimization very difficult.

Cannot interact with Python and its modules in nopython mode

Limited class support

Cython

Instead of parsing bytecode and generating IR,Cython using a superset of Python syntax, it was later converted to C code. When using Cython, C code is basically written in advanced Python syntax.

In Cython, you usually don't have to worry about Python wrappers and low-level API calls, because all interactions are automatically extended to the appropriate C code.

Unlike Numba, all Cython code should be separated from regular Python code in a special file. Cython parses and converts these files into C code, and then compiles it using the provided C compiler (for example, gcc).

The Python code is already valid Cython code.

However, the type version works faster.

Writing fast Cython code requires an understanding of the internal structure of C and Python. If you are familiar with C #, your Cython code can run as fast as C code.

Advantages of Cython:

Control through the use of Python API

A simple interface with the Cpicket + library and the Cpicurus + code.

Parallel execution support

Support Python classes and provide object-oriented features in C #

Cython's disadvantages:

Learning curve

C and Python internal expertise is required.

The organization of the module is not convenient.

Numba versus Cython

Personally, I prefer small projects and ETL experiments with Numba. You can insert it into an existing project. If I need to start a large project or write a wrapper for the C library, I will use Cython because it provides more control and easier debugging.

In addition, Cython is the standard for many libraries, such as pandas, scikit-learn, scipy, Spacy, gensim, and lxml.

The above is all the contents of the article "how to use Numba and Cython to improve the efficiency of python". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.