Shulou(Shulou.com)06/02 Report--

This article mainly introduces "what are the Python packages commonly used in the field of data science". In the daily operation, I believe that many people have doubts about the Python packages commonly used in the field of data science. I have consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the questions of "what are the Python packages commonly used in the field of data science?" Next, please follow the editor to study!

Core library

1 、 Numpy

Numpy (stands for Numerical Python) when trying to solve scientific tasks with Python, numpy is the cornerstone, providing rich features for its ability to manipulate arrays and matrices. The library provides vectorization of mathematical operations of NumPy array types, which can improve performance and speed up execution.

2.SciPy

SciPy is a library of engineering and scientific software. SciPy includes modules for linear algebra, optimization, integration and statistics. The main function of the SciPy library is based on NumPy, so its array uses NumPy heavily.

3.Pandas

Pandas is an easy to process table data (frequent contact, easy to understand excell table). The perfect tool for data cleaning, designed for fast and simple data manipulation, aggregation and visualization. There are two main data structures in this library:

Pandas.Series-1 dimension

Pandas.DataFrames-2 dimension

The following is just a small list of things we can do based on Pandas:

Easily delete or add columns in DataFrame

Convert data structures to DataFrame objects

Deal with missing data, represented by NaNs

GroupBy method

Visualization

Google Trends history

GitHub pull requests history

4.Matplotlib

MatPlotlib is the python visualization library that makes Python a strong contender for scientific tools like MatLab or Mathematica. However, the library is fairly low-level, which means that you need to write more code to achieve advanced visualization, usually with more effort than using more advanced tools, but overall the effort is worth it.

With a little kung fu, you can make any of the following visualization methods:

Line diagram

Scatter plot

Bar chart and histogram

Pie chart

Stem diagram

Contour map

Vector field diagram

Frequency spectrum diagram

There is also the ability to create tags, legends, and many other formatted entities using Matplotlib. Basically, everything is customizable.

The library is supported by different platforms and uses different GUI suites to describe the resulting visualization. Different IDE, such as IPython, support the functionality of Matplotlib.

There are other libraries that can make visualization easier.

5.Seaborn

Seaborn focuses primarily on the visualization of statistical models; these visualizations include heat maps that summarize the data but still depict the overall distribution. Seaborn is based on Matplotlib and is highly dependent on that package.

6.Bokeh

Boken is another powerful visualization library with the goal of creating icons for interactive visualization. Compared to previous libraries, this library is independent of Matplotlib. As we have already mentioned, the main focus of Bokeh is on interactivity, which is presented in a data-driven document (d3.js) style through modern browsers.

7.Plotly

A brief introduction to Plotly. It is a Web-based toolkit that exposes API to some programming languages, including Python, to build visualization. There are some powerful, out-of-the-box graphics on the http://plot.ly website. In order to use Plotly, you will need to set your API key. The graphics will be processed on the server side and will be published on the Internet.

Google Trends history

GitHub pull requests history

Machine learning

8.SciKit-Learn

Scikits is designed for specific functions such as image processing and machine learning assistance. In these areas, one of the most prominent is scikit-learn. The software package is built on the upper layer of SciPy and makes extensive use of its mathematical operations.

Scikit-learn exposes a simple and consistent interface, combined with common machine learning algorithms, which makes it easy to bring machine learning into the production system. With high-quality code, good documentation, easy to use and excellent performance, this library is in fact the industry standard for machine learning using Python.

Deep Learning-Keras/TensorFlow/Theano

In terms of deep learning, one of the most prominent and convenient libraries in Python is Keras, which can run on TensorFlow or Theano. Let's take a look at some of their details.

9.Theano

First, let's talk about Theano.

Theano is a Python package that defines multidimensional arrays similar to NumPy, as well as mathematical operations and expressions. This library is self-compiled, allowing it to run efficiently on all architectures. Originally developed by the Machine Learning Group of the University of Montreal, it is mainly used for machine learning needs.

It is important to note that Theano and NumPy are tightly integrated at low-level operations. The library also optimizes the use of GPU and CPU to make data-intensive computing faster.

Efficiency and stability adjustments allow for more accurate results, even very small values. For example, the calculation of log (1 + x) will give a cognitive result of the minimum value of x. TensorFlow

10TensorFlow

TensorFlow, developed by Google developers, is an open source library for graphical data stream computing that focuses on machine learning. It aims to meet the high requirements of training neural networks in Google environment, and is the successor of DistBelief, a machine learning system based on neural networks. However, TensorFlow is not strictly used for Google-wide scientific purposes-it is just as effective in general practical applications.

The key feature of TensorFlow is its multi-layer node system, which can quickly train artificial neural networks on large data sets. This provides support for speech recognition and image object recognition of Google.

11.Keras

Finally, let's take a look at Keras. It is an open source library written in Python for building neural networks in high-level interfaces. It is easy to understand and highly extensible. It uses Theano or TensorFlow as the backend, but Microsoft now integrates CNTK (Microsoft's cognitive toolkit) into a new backend.

The minimalist method in the design is designed to carry out fast and simple experiments by establishing minimal sets.

Keras is really easy to get started and can be delved into through quick standards. It is written in pure Python and is highly modular and extensible. Despite its ease, simplicity and high orientation, Keras still has a deep and powerful machine learning ability for large models.

The core of Keras is based on layers, and everything else is built around them. The data preprocessing is tensor tensor, the first layer layer is responsible for the input tensor, the last layer is responsible for output, and establish the model.

Google Trends history

GitHub pull requests history

Natural language processing.

12.NLTK

The name of this toolkit stands for the natural language toolkit, which, as its name implies, is used for common tasks of symbolic and statistical natural language processing. NLTK aims to promote the teaching and research of NLP and related fields (linguistics, cognitive science, artificial intelligence, etc.).

The functions of NLTK allow many operations, such as text tagging, classification and tagging, name entity identification, building language trees, displaying inter-language and intra-sentence dependencies, roots, and semantic reasoning. All building blocks can build complex research systems for different tasks, such as emotional analysis and automatic summarization.

13.Gensim

It is an open source library for Python and can be used for vector space modeling and topic modeling. This toolkit can be used not only for memory processing, but also for efficient processing of large text. Efficiency is achieved through the use of NumPy data structures and SciPy operations. It is efficient and easy to use.

Gensim is designed to be used with raw and unstructured numeric text. Gensim implements algorithms such as hierarchical Dirichlet processes (HDP), latent semantic analysis (LSA), and latent Dirichlet allocation (LDA), as well as tf-idf, random projection, word2vec, and document2vec a set of files (often referred to as corpora) that facilitate checking repetitive patterns of text in text. All algorithms are unsupervised-no parameters are required and the only input is the corpus.

Google Trends history

GitHub pull requests history

Data mining, statistics

14.Scrapy

Scrapy is a library of crawlers (also known as spider robots) that retrieve structured data from the web, such as contact information or URL.

It is open source and written in Python. It is designed strictly in the way of crawling, as its name suggests, but it has been developed in a complete framework, capable of collecting data from API and acting as a general crawler.

The library is known in interface design as "Don't repeat yourself"-it prompts users to write common code that will be reused to build and scale large crawlers.

The architecture of Scrapy is built around the Spider class and contains a series of instructions tracked by the crawler.

15.Statsmodels

As you might guess from the name, statsmodels is a library for Python that enables users to mine data and perform statistical assertions and analysis by using various statistical model estimation methods.

Many useful features are descriptive and are counted by using linear regression models, generalized linear models, discrete selection models, robust linear models, time series analysis models, and various estimators.

The library also provides extensible drawing functions, specially designed for statistical analysis and good performance in big data statistics.

At this point, the study of "what are the Python packages commonly used in the field of data science" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.