Shulou(Shulou.com)06/01 Report--

This article is about how to analyze the NumPy broadcasting mechanism and C language extension, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

This paper focuses on the broadcast mechanism and the axis operation for high-dimensional arrays, and finally introduces the C language extension of NumPy.

1 broadcast

NumPy operations are usually performed at the element level of two arrays. In the simplest case, two array operations with exactly the same shape are performed, as shown in the following example

A = np.array ([1.0,2.0,3.0])

B = np.array ([2.0,2.0])

A * b

The broadcast mechanism of numpy refers to the way in which arrays of different shape are processed when performing arithmetic operations. Under certain rules, smaller arrays are broadcast on larger arrays, making the array have a compatible shape.

A = np.array ([1.0,2.0,3.0])

B = 2.0

A * b

It is found that the results of the two calculations are the same, but the second is that there is a broadcasting mechanism at work. Broadcast rules

When performing operations on two arrays, NumPy compares their shapes. It starts from the far right of the shape and compares it one by one to the left. If all the seats are compared to one of the following two situations

Two numbers in the same seat are equal or one of them is 1

Then these two arrays can be operated. If these conditions are not met, a ValueError is thrown, indicating that the shape of the array is incompatible.

It can be seen that the shape of an array is like the eight characters of a person. If two people do not agree with each other, they cannot be together.

In the following examples, the axes of length 1 in the An and B arrays (the missing axes are automatically filled by 1) are expanded to a larger length in the same bit of another array during the broadcast.

A (3D array): 15 x 3 x 5

B (3D array): 15 x 1 x 5

Result (3D array): 15 x 3 x 5

A (3D array): 15 x 3 x 5

B (2d array): 3 x 5

Result (3D array): 15 x 3 x 5

A (3D array): 15 x 3 x 5

B (2d array): 3 x 1

Result (3D array): 15 x 3 x 5

A (4d array): 8 x 1 x 6 x 1

B (3D array): 7 x 1 x 5

Result (4d array): 8 x 7 x 6 x 5

In the following example, the shape of the first array is (3), and the shape of the second array is (3,), which is equivalent to (1), so change the shape of the second array to (3), which is equivalent to 2 copies of the original array along the 0 axis.

A diagram of the broadcasting mechanism. MatA = np.array ([[1, 2, 3], [4, 5, 5, 6], [7, 8, 9])

MatB = np.array ([1,2,3])

MatA + MatB

To better understand this mechanism, here are a few more examples. The following figure consists of three lines, corresponding to three broadcast methods, please compare with the following code.

Each line corresponds to a broadcast. A = np.array ([0re10pr 20pr 30])

B = np.array ([0je 1jue 2])

A = np.stack ((amatheta), axis=1)

B = np.stack ((brecrum brem bjorb))

# corresponding to the first case

A + B

# corresponding to the second case

A + b

A1 = np.array ([[0pr 10pm 20je 30]). T

# corresponding to the third situation

A1 + b

The following example does not satisfy the broadcast rules, so the operation cannot be performed.

A (1D array): 3

B (1D array): 4 # the last axis length is not compatible

A (2d array): 4 x 3

B (1D array): 4 # the last axis length is not compatible

A (2d array): 2 x 1

B (3D array): 8 x 4 x 3 # the penultimate axis length is not compatible

An example that cannot be broadcast. Broadcast mechanism summary broadcast mechanism provides a convenient way for array operation. That said, it does not work in all cases and actually imposes strict rules that must be met in the execution of broadcasts. Arithmetic operations can be performed only if the shape of each dimension in the array is equal or the size of the dimension is 1. 2Dimensions increase or decrease.

Use np.newaxis or None in seats where additional axes are needed.

X = np.arange (6) .reshape (2prime3)

X, x.shape

X1 = x [:, np.newaxis,:]

X1, x1.shape

# or

X2 = x [:, None,:]

X2, x2.shape

Dimension compression sometimes requires the removal of excess axes from the array to reduce the purpose of the array dimension. Numpy.squeeze () removes the axis of a single dimension from the array, that is, removes the dimension of 1 in the shape. X = np.arange (6) .reshape (2Jing 1Jue 3)

Y = x.squeeze ()

Xd = x.interfaceability _ ['data'] [0]

Yd = y.starting arraylike interfaceability _ ['data'] [0]

Check the address of the data in memory to verify that it points to the same block of memory. 3 Array transpose (change axis) x = np.arange (9) .reshape (3,3)

Y = np.transpose (x) # or y = x.transpose () or x.T

Y = np.transpose (x, [1,0])

X = np.array ([3, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] .reshape (2,2,4)

Y1 = np.transpose (x, [1,0,2])

Please compare the following figure to understand what this three-dimensional array looks like in memory and different views of it (view). On this point, the advanced article attached at the end of the article is interpreted in detail.

Note that the axis can be changed, but the data is immobile. Y2 = np.transpose (x, [2,0,1])

# Code together

X = np.array ([3, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] .reshape (2,2,4)

Y0 = np.transpose (x, [1,2,0])

Y1 = np.transpose (x, [1,0,2])

Y2 = np.transpose (x, [2,0,1])

Take a look at what the elements of each array look like after changing the axis, and notice that they all point to the same data.

How does this make it possible to use different axes for the same data in memory? In fact, the data is the same, changing the step size stride on each axis.

X.strides, y1.strides, y2.strides

# the data is still the same

Id (x.data), id (y1.data), id (y2.data)

Looking at another example, a three-dimensional array has three axes. Note the step size of each axis after changing the axis.

X = np.arange (16) .reshape (2,2,4)

Y = x.transpose ((1,0,2))

The corresponding steps of the three axes of the two arrays are different.

After the axis is changed, the subscript is also changed, so the data pointed to by the same subscript before and after the axis change is different.

The axis has been changed, and so has the subscript. In fact, the significance of the axis is mainly reflected in the step size, so changing the axis means changing the step size in a certain sense. Practical example

RGB image data

Each image consists of three red, green and blue channels, each of which corresponds to a 32 × 32 two-dimensional array of 32x32 pixels.

Look at the following figure, from left to right, corresponding to the storage of image data in memory, convert one-dimensional array into three-dimensional array, and change the axis.

So, why change the shaft? Because different packages have different requirements for data, in order to use them, we need to adjust the data according to their requirements for parameters.

Sometimes, you don't need to change the axis, you just need to change the order of the elements on a certain axis, for example

# change the order of elements on an axis

Z = x [., (3,2,1,0)]

4 General function ufunc function ufunc is the abbreviation of universal function, which refers to the function that can operate on each element of the array, not on the narray object. NumPy provides a large number of ufunc functions. These functions operate on narray much faster than using loops or list deductions. Many of the ufunc functions built into NumPy are implemented in C language, so the computation efficiency is very high. X = np.linspace (0, 2*np.pi, 5)

Y, z = np.sin (x), np.cos (x)

# pass the result directly to input x

Np.sin (x, x)

Performance comparison import time

Import math

Import numpy as np

X = [i for i in range (1000000)]

# math.sin

Start = time.process_time ()

For I, t in enumerate (x):

X [I] = math.sin (t)

Math_time = time.process_time ()-start

# numpy.sin

X = np.array (x, dtype=np.float64)

Start = time.process_time ()

Np.sin (x, x)

Numpy_time = time.process_time ()-start

# comparison

Math_time, numpy_time, math_time/numpy_time

Reduce operation this is a general function built-in by NumPy. If you need such a calculation, it is recommended to use it directly and not to implement it yourself. Manipulating arrays along the axis is equivalent to inserting operators into all subarrays or elements along the axis. The format is:. Reduce (array=, axis=0, dtype=None) np.add.reduce ([1, 2, 3])

Np.add.reduce ([[1, 2, 3], [4, 5, 5, 6]], axis=1)

Np.multiply.reduce ([[1, 2, 3], [4, 5, 5, 6]], axis=1)

Accumulate operation

This is also a general function built into NumPy. If you need such a calculation, it is recommended to use it directly and not to implement it yourself.

Similar to reduce, except that the array it returns is the same as the shape of the input array, saving all intermediate calculations. Np.add.accumulate ([1pm 2pm 3])

Np.add.accumulate ([[1, 2, 3], [4, 5, 5, 6]], axis=1)

Custom ufunc function # defines a python function

Def ufunc_diy (x):

C, c0, hc = 0.618, 0.518, 1.0

X = x-int (x)

If x > = c:

R = 0.0

Elif x < c0:

R = x / c0 * hc

Else:

R = (cmurx) / (c-c0) * hc

Return r

X = np.linspace (0, 2, 1000000)

Ufunc_diy (x)

Start = time.process_time ()

Y1 = np.array ([ufunc_diy (t) for t in x])

Time_1 = time.process_time ()-start

Time_1

The np.frompyfunc function converts a function that calculates a single element into the ufunc function ufunc = np.frompyfunc (ufunc_diy, 1,1).

Start = time.process_time ()

Y2 = ufunc (x)

Time_2 = time.process_time ()-start

Time_2

C extension of NumPy

This paper mainly introduces two expansion methods.

CtypesCythonctypesctypes is an external library of Python, which provides data types compatible with C language, and can easily call C interface functions output in dll/so. # ufunc.c

''

Void ufunc_diy (double x, double y, int size) {

Double xx,r,c=0.618,c0=0.518,hc=1.0

For (int iTunes 0 / 1 / 2 / 0 / 0 / 0 / 2 / 0 / 2 / 2 / 2 / 3 / 2 / 3 / 2 / 2 / 3 / 2 / 2 / 3 / 2 / 3 / 2 / 2 / 2 / 2 / 2 / 2 / 2 / 2 / 2 / 2 / 2 / 2 / 3 / 2 / 2 / 2 / 2 / 2 / 3 / 2 / 2 / 3 / 2 / 3 / 2 / 3 / 2 / 2 / 3 / 2 / 3 / 2 / 3 / 2 / 3 / 2 / 3 / 2 / 3 / 2 / 3 / 3 / 2 / 3 / 3 / 3 / 3 / 2 / 3 / 3 / 2 / 3 / 3 / 3 / 2 / 3 / 3 / 2 / 3 / 3 / 3 / 2 / 3 / 2 / 3 / 2 / 3 / 3 / 2 / 3 / 2 / 3 / 3 / 3 / 3 /

Else if (xx

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.