Shulou(Shulou.com)06/01 Report--

This article mainly introduces "what are the reasons why Python big data uses Numpy Array". In daily operation, I believe many people have doubts about the reasons why Python big data used Numpy Array. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the questions of "why Python big data uses Numpy Array". Next, please follow the editor to study!

Numpy is a core module of Python scientific computing. It provides very efficient array objects and tools for working with these array objects. An Numpy array consists of many values, all of the same type.

Python's core library provides a list of List. List is one of the most common Python data types, which can be resized and contain different types of elements, which is very convenient.

So what's the difference between List and Numpy Array? Why do we need to use Numpy Array when big data handles it? The answer is performance.

Numpy data structures perform better in the following areas:

1. Memory size-Numpy data structures take up less memory.

two。 Performance-the underlying Numpy is implemented in C, which is faster than lists.

3. Operation method-built-in optimization of algebraic operation and other methods.

The following explains the advantages of Numpy array over List when dealing with big data.

1. Smaller memory footprint

By using Numpy arrays instead of List properly, you can reduce your memory footprint by 20 times.

For Python's native List list, because each new object requires 8 bytes to refer to the new object, the new object itself accounts for 28 bytes (take integers as an example). So the size of the list list can be calculated using the following formula:

64 + 8 * len (lst) + len (lst) * 28 bytes

With Numpy, you can reduce the footprint by a lot. For example, the Numpy shaping Array with length n requires:

96 + len (a) * 8 bytes

It can be seen that the larger the array, the more memory space you save. Assuming that your array has 1 billion elements, the difference in memory footprint will be GB-level.

two。 Faster, built-in calculation method

Run the following script, which also generates two arrays of a dimension and adds them, and you can see the performance gap between native List and Numpy Array.

Import timeimport numpy as npsize_of_vec = 1000def pure_python_version (): T1 = time.time () X = range (size_of_vec) Y = range (size_of_vec) Z = [X [I] + Y [I] for i in range (len (X))] return time.time ()-t1def numpy_version (): T1 = time.time () X = np.arange (size_of_vec) Y = np.arange (size_of_vec) Z = X + Y return time.time ()-t1t1 = pure_python_version () T2 = numpy_version () print (T1) T2) print ("Numpy is in this example" + str (t1/t2) + "faster!")

The results are as follows:

0.00048732757568359375 0.0002491474151611328

Numpy is in this example 1.955980861244019 faster!

As you can see, Numpy is 1.95 times faster than the native array.

If you are careful, you can also find that Numpy array can perform addition operations directly. Native arrays can't do this, which is the advantage of Numpy.

Let's repeat a few more experiments to prove that this performance advantage is persistent.

Import numpy as npfrom timeit import Timersize_of_vec = 1000X_list = range (size_of_vec) Y_list = range (size_of_vec) X = np.arange (size_of_vec) Y = np.arange (size_of_vec) def pure_python_version (): Z = [XList [I] + YList [I] for i in range (len (X_list))] def numpy_version (): Z = X + Ytimer_obj1 = Timer ("pure_python_version ()" "from _ main__ import pure_python_version") timer_obj2 = Timer ("numpy_version ()", "from _ main__ import numpy_version") print (timer_obj1.timeit (10)) print (timer_obj2.timeit (10)) # Runs Fastener (timer_obj1.repeat (repeat=3, number=10)) print (timer_obj2.repeat (repeat=3, number=10)) # repeat to prove it!

The results are as follows:

0.0029753120616078377

0.00014940369874238968

[0.002683573868125677, 0.002754641231149435, 0.002803879790008068]

[6.536301225423813e-05, 2.9387418180704117e-05, 2.9171351343393326e-05]

As you can see, the time of the second output is always much smaller, which proves that this performance advantage is persistent.

So, if you are doing some big data research, such as financial data, stock data research, using Numpy can save you a lot of memory space and have more powerful performance.

At this point, the study on "what are the reasons why Python big data uses Numpy Array" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.