Shulou(Shulou.com)06/01 Report--

This article mainly explains "what are the basic knowledge points of python data mining". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what are the basic knowledge points of python data mining".

Basis of data mining

Data mining is to mine implicit, previously unknown and potentially valuable relationships, patterns and trends from a large amount of data (including text), and use these knowledge and rules to build models for decision support. methods, tools and processes that provide predictive decision support.

1.3. Basic tasks of data Mining

The basic tasks of data mining include the use of classification and prediction, clustering analysis, association rules, time series patterns, deviation detection, intelligent recommendation and other methods to help enterprises extract business value from data and improve the competitiveness of enterprises.

1.4. Data mining modeling process 1.4.1, defining mining objectives

Understand mining tasks and define mining goals

1.4.2. Data sampling

The criteria for extracting data, one is relevance, the second is reliability, and the third is effectiveness, rather than using all enterprise data. Through the selection of data samples, we can not only reduce the amount of data processing, save system resources, but also make the regularity we are looking for more prominent.

Quality standards for measuring data sampling:

1) the data is complete and all kinds of indicators are complete.

2) the data are accurate and reflect the level in the normal (rather than abnormal) state.

The obtained data can be sampled from it. There are a variety of sampling methods, and the common ones are as follows:

Random sampling

Isometric sampling

Stratified sampling

Sampling from the starting sequence

Classified sampling

1.4.3, data exploration

Data exploration mainly includes: outlier analysis, missing value analysis, correlation analysis and periodic analysis.

1.4.4. Data preprocessing

Data preprocessing mainly includes: data screening, data variable conversion, data missing value processing, bad data processing, data standardization, principal component analysis, attribute selection, data specification and so on.

1.4.5, mining modeling

After the sample is extracted and preprocessed, the next question to be considered is: what kind of problems (classification, clustering, association rules, time series patterns or intelligent recommendation) in the application of data mining? which algorithm is chosen to build the model? This step is the core link of data mining.

1.4.6. Model evaluation.

One of the purposes of model evaluation is to automatically find the best model from these models, and to explain and apply the model according to the business.

1.5. Commonly used data mining modeling tools

Data mining is a process of repeated exploration. Only by closely combining the technology and implementation experience provided by data mining tools with the business logic and needs of enterprises, and constantly running-in in the process of implementation, can good results be achieved. Here is a brief introduction to several commonly used data mining modeling tools.

SAS Enterprise Miner, integrated data mining system

IBM SPSS Modeler

SQL Server

Python, an object-oriented interpretive high-level programming language

WEKA, well-known machine Xu Wenqi and data mining software

KNIME, you can extend the use of mining algorithms in WEKA

RapidMiner

TipDM, data mining modeling platform

(1) SAS Enterprise Miner

Enterprise Miner (EM) is an integrated data mining system developed by SAS, which allows the use and comparison of different technologies, as well as the integration of complex database management software. Its operation mode is by adding all kinds of nodes that can achieve different functions in a workspace (workspace) in a certain order, then setting different nodes accordingly, and finally running the whole workflow (workflow), then the corresponding results can be obtained.

(2) IBM SPSS Modeler

IBM SPSS Modeler, formerly known as Clementine, has greatly improved the performance and function of the product after it was acquired by IBM in 2009. It encapsulates the most advanced statistics and data mining technology to obtain forecasting knowledge, and deploys the corresponding decision-making scheme to the existing business system and business process, so as to improve the efficiency of the enterprise. IBM SPSS Modeler has an intuitive operating interface, automated data preparation and mature predictive analysis model, combined with commercial technology to quickly build predictive model.

(3) SQL Server

Analysis Servers, a data mining component, is integrated into the SQL Server of Microsoft Company. With the database management function of SQL Server, it can be seamlessly integrated into the SQL Server database. Nine commonly used data mining algorithms are provided in SQL Server 2008, such as decision tree algorithm, clustering analysis algorithm, Naive Bayes algorithm, association rule algorithm, time series algorithm, neural network algorithm, linear regression algorithm and so on. However, the implementation of predictive modeling is based on the SQL Server platform, and the portability of the platform is relatively poor.

(4) Python

Python (Matrix Laboratory, Matrix Lab) is an application software developed by Mathworks Company in the United States, which has powerful scientific and engineering computing capabilities. It not only has powerful mathematical computing capabilities and analytical functions based on matrix computing, but also has rich visual graphic performance functions and convenient programming capabilities. Python does not provide a specialized data mining environment, but it provides a lot of implementation functions of related algorithms, so it is a good choice for learning and developing data mining algorithms.

(5) WEKA

WEKA (Waikato Environment for Knowledge Analysis) is a well-known open source machine learning and data mining software. Advanced users can invoke their analysis components through Java programming and the command line. At the same time, WEKA also provides graphical interfaces for ordinary users, called WEKA Knowledge Flow Environment and WEKA Explorer, which can realize preprocessing, classification, clustering, association rules, text mining, visualization and so on.

(6) KNIME

KNIME (Konstanz InformationMiner, http://www.knime.org)) is developed based on Java and can be extended to use mining algorithms in Weka. KNIME establishes the analysis and mining process in a way similar to data flow (data flow). The mining process consists of a series of functional nodes, each of which has an input / output port to receive data or models and export results.

(7) RapidMiner

RapidMiner, also known as YALE (Yet Another Learning Environment, https://rapidminer.com), provides a graphical interface and organizes analysis components with a tree structure similar to that in Windows explorer. Each node on the tree represents a different operator (operator) o YALE provides a large number of operators, including data processing, transformation, exploration, modeling, evaluation and other aspects. YALE is developed in Java, built on Weka, and can call various analysis components in Weka. RapidMiner has an extended suite of Radoop that can be integrated with Hadoop to run tasks on the Hadoop cluster.

(8) TipDM

TipDM (top data mining platform) is developed in Java language, which can obtain data from various data sources and establish a variety of data mining models. At present, TipDM has integrated dozens of prediction algorithms and analysis techniques, which basically cover the algorithms supported by foreign mainstream mining systems. TipDM supports the main processes of data mining process: data exploration (correlation analysis, principal component analysis, periodic analysis); data preprocessing (attribute selection, feature extraction, bad data processing, null value processing); predictive modeling (parameter setting, cross-validation, model training, model verification, model prediction); clustering analysis, association rule mining and a series of functions.

Thank you for your reading, the above is the content of "what are the basic knowledge points of python data mining". After the study of this article, I believe you have a deeper understanding of what the basic knowledge points of python data mining have, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.