Shulou(Shulou.com)06/02 Report--

This article focuses on "how to use Python for data science research", interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to use Python for data science research.

1. Why choose Python?

Python as a language, decathlon, easy to learn, easy to install. At the same time, there are many extensions, which are very suitable for data science research. Star websites such as Google, Instagram, Youtube and Reddit are using Python to build core business.

Python is not only used in data science, but also uses Python to do more work-- such as writing scripts, building API, building websites, and so on.

There are a few important things to note about Python.

Currently, there are two commonly used versions of Python. They are versions 2 and 3. Most tutorials and articles will use Python 3, the * * version of Python, by default. But sometimes you come across books or articles that use Python 2. There is not much difference between versions, but sometimes copying and pasting version 2 code while running version 3 will not work properly, so some minor edits are required.

Note that Python cares a lot about white space (that is, spaces and return characters). If you put the space in the wrong place, the program is likely to produce an error.

Compared with other languages, Python does not need to manage memory and has good community support.

two。 Install Python

The * * method to install Python for data science is to use the Anaconda distribution.

Anacoda has the information you need to use Python for data science research, including many of the software packages that will be introduced in this article.

Click Products-> Distribution and scroll down to see the installer for Mac,Windows and Linux. Even if you already have Python on Mac, you should consider installing the Anaconda distribution because it facilitates the installation of other packages.

In addition, you can download the installer from the official Python website.

Package Manager:

Packages are a piece of Python code, not part of the language, and they are very helpful for performing certain tasks. With the package, we can copy and paste the code and put it where the Python interpreter (used to run the code) can find it.

But this is troublesome, and the content must be copied and pasted every time you start a new project or update a package. Therefore, we can use the package manager. The package manager is included in the Anaconda distribution. If not, it is recommended to install pip.

Whichever you choose, you can easily install and update the package using commands on the terminal (or command prompt).

3. Using Python for data Science Research

Python caters to the technical requirements of many different developers (Web developers, data analysts, data scientists), so there are many different programming methods to use the language.

Python is an interpreted language, so you don't have to compile the code into an executable file, you just need to pass the text document containing the code to the interpreter.

Take a quick look at the different ways to interact with the Python interpreter.

(1) at the terminal

If you open the terminal (or command prompt) and type the word 'Python', you will start a shell session. You can enter valid Python commands in the dialogue to achieve the corresponding program operation.

This can be a good way to debug something quickly, but even a small project can be difficult to debug in a terminal.

(2) use a text editor

If you write a series of Python commands in a text file and save it with the .py extension, you can use the terminal to navigate to the file and run the program by typing python YOUR_FILE_NAME.py.

This is basically the same as entering commands one by one in the terminal, except that it is easier to fix errors and change the functionality of the program.

(3) in IDE

IDE is a professional-level software that can manage software projects.

One of the benefits of IDE is that debugging can tell you where to go wrong before you try to run the program.

Some IDE come with project templates (for specific tasks) that you can use to set up projects based on * practices.

(4) Jupyter Notebooks

None of these methods use python for data science, but use Jupyter Notebooks.

Jupyter Notebooks allows you to run one "block" of code at a time, which means you can see the output before deciding what to do next-which is very important in data science projects, and we often need to look at the chart before getting the output.

If you are using Anaconda and have Jupyter lab installed. To start it, just type 'jupyter lab'' in the terminal.

If you are using pip, you must install Jupyter lab using the command 'python pip install jupyter'.

4. Numerical calculation in Python

The NumPy package contains many useful functions for performing mathematical operations needed for data science work.

It is installed as part of the Anaconda distribution and installed using pip, which is as simple as installing Jupyter Notbooks ('pip install numpy').

The most common mathematical operations we need to do in data science are matrix multiplication, calculating the dot product of vectors, changing the data type of arrays, and creating arrays!

Here is how to compile a list into an NumPy array:

Here is how to multiply arrays and calculate dot products in NumPy:

Here is how to do matrix multiplication in NumPy:

5. Statistical analysis in Python

The Scipy package contains modules dedicated to statistics (subsections of the code of the package).

You can use the 'from scipy import stats' command to import it (make its functionality available in the program) into your notebook. The software package contains everything needed to calculate data statistical measurements, perform statistical tests, calculate correlations, aggregate data, and study various probability distributions.

The following is a way to quickly access the summary statistics (minimum, * *, mean, variance, skew, and kurtosis) of an array using Scipy:

6. Data manipulation in Python

Data scientists must spend a lot of time cleaning and collating data. Fortunately, the Pandas package can help us do this in code rather than by hand.

The most common task performed with Pandas is to read data from CSV files and databases.

It also has a powerful syntax to combine different datasets (called DataFrame in Pandas) and perform data operations.

Use the .head method to view the first few lines of DataFrame:

Use square brackets to select a column:

Create a new column by combining other columns:

7. Using databases in Python

In order to use the pandas read_sql method, a connection to the database must be established in advance.

The safest way to connect to a database is to use Python's SQLAlchemy package.

SQL itself is a language, and the way you connect to a database depends on the database you are using.

8. Data engineering in Python

Sometimes we tend to do some calculations on the data before it reaches our project in the form of Pandas DataFrame.

If you are using a database or fetching data from Web (and storing it somewhere), the process of moving data and transforming it is called ETL (extract, transform, load).

You extract data from one place, make some transformations (by adding data to summarize the data, find the mean, change the data type, etc.), and then load it into an accessible location.

There is a very cool tool called Airflow, which is very good at helping manage ETL workflows. Better yet, it is written in Python and developed by Airbnb.

9. Big data Project in Python

Sometimes the ETL process can be very slow. If you have billions of rows of data (or if they are a strange data type, such as text), you can use many different computers to process the conversion separately and put all the data together in * seconds.

This architectural pattern is called MapReduce, and it is very popular with Hadoop.

Nowadays, many people use Spark for this kind of data conversion / retrieval, and there is a Python interface for Spark called PySpark.

The MapReduce architecture and Spark are very complex tools, and I won't go into detail here. As long as you know they exist, PySpark may be helpful if you find yourself dealing with a very slow ETL process.

10. Further statistics in Python

We already know that we can use Scipy's statistics module to run statistical tests, calculate descriptive statistics, p-values, and skew and kurtosis, but what else can Python do?

One special bag you should know is the Lifelines bag.

Using the Lifelines package, you can calculate various functions from a statistical subfield called survival analysis.

Survival analysis has many applications. We can use it to predict customer churn (when customers unsubscribe) and when retail stores may be stolen.

These and the package's creators imagine that it will be used in a completely different field (survival analysis is traditionally a medical statistical tool). But this just shows different ways of building data science problems!

11. Machine Learning in Python

This is an important theme. Machine learning is sweeping the world and is an important part of the work of data scientists.

In short, machine learning is a set of technologies that allow computers to map input data to output data. There are situations where this is not the case, but they are in the minority, and it is often helpful to consider ML in this way.

Python has two very good machine learning packages.

(1) Scikit-Learn

When using Python for machine learning, you spend most of your time using the Scikit-Learn package (sometimes abbreviated to sklearn).

This package implements a host of machine learning algorithms and exposes them through a consistent syntax. This makes it easy for data scientists to make full use of each algorithm.

The general framework for using Scikit-Learn is to split the dataset into training and test datasets:

Instantiate and train a model:

Use the metrics module to test the operation of the model:

(2) XGBoost

The second package commonly used for machine learning in Python is XGBoost.

Scikit-Learn implements a series of algorithms, while XGBoost only implements a gradient lifting decision tree.

Recently, this package (and algorithm) has become very popular because of its success in the Kaggle competition, an online data science competition that anyone can participate in.

The training model works in much the same way as the Scikit-Learn algorithm.

12. Deep learning in Python

The machine learning algorithm provided in Scikit-Learn can satisfy almost any problem. That said, sometimes you need to use a progressive algorithm.

Because the systems that use them are better than almost all other algorithms, the penetration of deep neural networks has increased sharply.

But it's hard to say what the neural network is doing and why it is doing it. Therefore, their use in finance, medicine, law and related professions has not been widely recognized.

The two main categories of neural networks are convolution neural networks (used to classify images and accomplish many other tasks in computer vision) and cyclic neural networks (used to understand and generate text).

Explore the mechanism of neural network work beyond the scope of this article. If you want to do this kind of work, just know that the package you are looking for is TensorFlow (Google contibution!). It's still Keras.

Keras is essentially a wrapper for TensorFlow, making it easier to use.

13. Data Science API in Python

Once the model has been trained, its predictions can be accessed in other software by creating an API.

API allows the model to receive data one at a time from external sources and return predictions. Because Python is a general programming language and can also be used to create Web services, it is easy to use Python to service the model through API.

If you need to build API, you should look at pickle and Flask. Pickle allows trained models to be saved on a hard drive for later use. Flask is the easiest way to create a Web service.

14. Web applications in Python

* if you want to build full-featured Web applications around data science projects, you should use the Django framework.

Django is very popular in the Web development community and is used to build * * versions of Instagram and Pinterest (and many other versions).

At this point, I believe you have a deeper understanding of "how to use Python for data scientific research". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.