Shulou(Shulou.com)06/02 Report--

This article focuses on "how to understand Python's data manipulation library Pandas". Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to understand Python's data manipulation library Pandas.

Learn about Pandas

One of the keys to a good understanding of pandas is to understand that pandas is a wrapper for a range of other python libraries. The main ones are Numpy, SQL alchemy, Matplot lib and openpyxl.

The core internal model of data frame is a series of NumPy arrays and pandas functions.

Pandas uses other libraries to get data from data frame. For example, SQL alchemy is used through the read_sql and to_sql functions; openpyxl and xlsx writer are used for the read_excel and to_excel functions. Matplotlib and Seaborn are used to provide a simple interface to draw the information available in data frame using commands such as df.plot ().

Numpy's Pandas-, efficient Pandas

One of the complaints you often hear is that Python is slow or difficult to process large amounts of data. Typically, this is due to the inefficiency of the code written. Native Python code is indeed slower than compiled code. However, libraries like Pandas provide a python interface for compiling code and know how to use it correctly.

Vectorization operation

Like the underlying library Numpy, pandas performs vectorization operations more efficiently than loops. These efficiencies are due to the fact that vectorization operations are performed through C compiled code rather than native python code. Another factor is the ability to vectorize operations, which can operate on the entire dataset, not just one subdataset.

The application interface allows some efficiency to be achieved by looping through the CPython interface:

Df.apply (lambda x: X ['col_a'] * x [' col_b'], axis=1)

However, most of the performance benefits can be achieved by using the vectorization operation itself, which can be used directly in pandas, or you can call its internal Numpy array directly.

Store data efficiently through DTYPES

When a data frame is loaded into memory through read_csv, read_excel, or other data frame reading functions, pandas does type inference, which may be inefficient. These api allow you to explicitly use dtypes to specify the type of each column. Specifies that dtypes allows data to be stored more efficiently in memory.

Df.astype ({'testColumn': str,' testCountCol': float})

Dtypes is a native object from Numpy that allows you to define the exact type and number of bits used to store specific information.

For example, the type np.dtype ('int32') of Numpy represents a 32-bit integer. Pandas defaults to 64-bit integers, so we can save half the space to use 32-bit:

Working with large datasets with blocks

Pandas allows data in data frames to be loaded in blocks (chunk). Therefore, the data frame can be processed as an iterator, and data frames larger than the available memory can be processed.

The combination of defining the block size and the get_chunk method when reading the data source allows panda to process the data as an iterator, as shown in the example above, where the data frame reads two rows at a time. Then we can iterate through these blocks:

I = 0for an in df_iter: # do some processing chunk = df_iter.get_chunk () I + = 1 new_chunk = chunk.apply (lambda x: do_something (x), axis=1) new_chunk.to_csv ("chunk_output_%i.csv"% I)

Its output can be provided to a CSV file, pickle, export to a database, and so on.

At this point, I believe you have a deeper understanding of "how to understand Python's data manipulation library Pandas". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.