Shulou(Shulou.com)06/01 Report--

This article mainly introduces what is the use of the Pandas library in Python, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

Pandas library is the most popular data manipulation library in Python. Inspired by the frames of R, it provides an easy way to manipulate data through its data-frame API. 0 1 understand Pandas

One of the keys to a good understanding of pandas is to understand that pandas is a wrapper for a range of other python libraries. The main ones are Numpy, SQL alchemy, Matplot lib and openpyxl.

The core internal model of data frame is a series of NumPy arrays and pandas functions.

Pandas uses other libraries to get data from data frame. For example, SQL alchemy is used through the read_sql and to_sql functions; openpyxl and xlsx writer are used for the read_excel and to_excel functions. Matplotlib and Seaborn are used to provide a simple interface to draw the information available in data frame using commands such as df.plot ().

0 2Numpy Pandas-, efficient Pandas

One of the complaints you often hear is that Python is slow or difficult to process large amounts of data. Typically, this is due to the inefficiency of the code written. Native Python code is indeed slower than compiled code. However, libraries like Pandas provide a python interface for compiling code and know how to use it correctly.

Vectorization operation

Like the underlying library Numpy, pandas performs vectorization operations more efficiently than loops. These efficiencies are due to the fact that vectorization operations are performed through C compiled code rather than native python code. Another factor is the ability to vectorize operations, which can operate on the entire dataset, not just one subdataset.

The application interface allows some efficiency to be achieved by looping through the CPython interface:

Df.apply (lambda x: X ['col_a'] * x [' col_b'], axis=1)

However, most of the performance benefits can be achieved by using the vectorization operation itself, which can be used directly in pandas, or you can call its internal Numpy array directly.

03 efficiently store data through DTYPES

When a data frame is loaded into memory through read_csv, read_excel, or other data frame reading functions, pandas does type inference, which may be inefficient. These api allow you to explicitly use dtypes to specify the type of each column. Specifies that dtypes allows data to be stored more efficiently in memory.

Df.astype ({'testColumn': str,' testCountCol': float})

Dtypes is a native object from Numpy that allows you to define the exact type and number of bits used to store specific information.

For example, the type np.dtype ('int32') of Numpy represents a 32-bit integer. Pandas defaults to 64-bit integers, so we can save half the space to use 32-bit:

04 dealing with large datasets with blocks

Pandas allows data in data frames to be loaded in blocks (chunk). Therefore, the data frame can be processed as an iterator, and data frames larger than the available memory can be processed.

The combination of defining the block size and the get_chunk method when reading the data source allows panda to process the data as an iterator, as shown in the example above, where the data frame reads two rows at a time. Then we can iterate through these blocks:

I = 0for an in df_iter: # do some processing chunk = df_iter.get_chunk () I + = 1 new_chunk = chunk.apply (lambda x: do_something (x), axis=1) new_chunk.to_csv ("chunk_output_%i.csv"% I)

Its output can be provided to a CSV file, pickle, export to a database, etc.

Thank you for reading this article carefully. I hope the article "what is the use of Pandas Library in Python" shared by the editor will be helpful to you. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.