Shulou(Shulou.com)06/01 Report--

This article mainly introduces "what is Spark MLlib machine learning". In daily operation, I believe many people have doubts about what Spark MLlib machine learning is. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "what is Spark MLlib machine learning?" Next, please follow the editor to study!

MLlib is a machine learning library provided by Spark. By calling the algorithm encapsulated by MLlib, machine learning applications can be easily constructed. It provides a wealth of machine learning algorithms, such as classification, regression, clustering and recommendation algorithms. In addition, MLlib standardizes the API for machine learning algorithms, making it easier to combine multiple algorithms into a single Pipeline or workflow.

Machine learning is a branch of artificial intelligence and a cross-discipline in many fields, including probability theory, statistics, approximation theory, convex analysis, computational complexity theory and so on. The main purpose of machine learning theory is to design and analyze algorithms that enable computers to "learn" automatically. Because a large number of statistical theories are involved in learning algorithms, machine learning is particularly closely related to inference statistics, also known as statistical learning theory. In the aspect of algorithm design, machine learning theory focuses on achievable and effective learning algorithms.

What is machine learning?

Machine learning has been widely used in various branches of artificial intelligence, such as expert system, automatic reasoning, natural language understanding, pattern recognition, computer vision, intelligent robot and so on. Machine learning is a branch of artificial intelligence, which mainly studies how to let machines learn from past experience, model the uncertainty of data and predict the future. There are many applications of machine learning, such as search, recommendation system, spam filtering, face recognition, speech recognition and so on.

Big data and Machine Learning

In the era of big data, the speed of data generation is very amazing. Internet, mobile Internet, Internet of things, GPS and so on will generate data all the time. The storage and computing capabilities needed to deal with these data are also growing geometrically, thus giving birth to a series of big data technologies represented by Hadoop, which provide a reliable guarantee for processing and storing these data.

Data, information and knowledge are three levels from big to small. Simple data is difficult to explain some problems, we need to add some people's experience, to convert it into information, the so-called information, that is, in order to eliminate uncertainty, we often say that information asymmetry, that is, when we are unable to get enough information, it is difficult to eliminate some uncertain factors. Knowledge is the highest stage, so data mining is also called knowledge discovery.

The task of machine learning is to use some algorithms to act on big data, and then mine the potential knowledge behind it. The more training data, the more advantages can be reflected in machine learning. The problems that can not be solved by machine learning in the past can be well solved through big data technology, and the performance will be greatly improved, such as speech recognition, image recognition and so on.

Machine learning classification

Machine learning is mainly divided into the following categories:

Supervised learning (supervised learning)

It's basically a synonym for classification. The supervision in learning comes from examples marked in the training data set. For example, in the postal code recognition problem, a group of handwritten postcode images and their corresponding machine-readable transformations are used as training examples to supervise the learning of classification models. Common supervised learning algorithms include: linear regression, logical regression, decision tree, naive Bayes, support vector machine and so on.

Unsupervised learning (unsupervised learning)

It is essentially a synonym for clustering. The learning process is unsupervised because the input instance has no class tags. The task of unsupervised learning is to mine potential structures from a given dataset. For example, give the pictures of cats and dogs to the machine, do not label them, but hope that the machine will be able to classify these photos, eventually the machine will divide these photos into two categories, but do not know which are cat photos and which are dog photos, for the machine, it is equivalent to two categories An and B. Common unsupervised learning algorithms include K-means clustering, principal component analysis (PCA) and so on.

Semi-supervised learning (Semi-supervised learning)

Semi-supervised learning is a kind of machine learning technology, which uses labeled and unlabeled examples when learning models. It is semi-supervised learning that the learner does not rely on external interaction and automatically uses unlabeled samples to improve learning performance.

The practical demand of semi-supervised learning is very strong, because a large number of unlabeled samples can be easily collected in practical applications, but it takes manpower and material resources to obtain markers. For example, when computer-aided medical image analysis is carried out, a large number of medical images can be obtained from the hospital, but it is unrealistic for medical experts to identify all the lesions in the image. the phenomenon of more untagged data is more obvious in Internet applications, such as asking users to mark pages of interest when making web page recommendations, but few users are willing to spend a lot of time providing tags. Therefore, there are few tagged web pages, but there are countless web pages on the Internet that can be used as untagged samples.

Reinforcement learning (reinforcement learning)

Also known as reinforcement learning and evaluation learning, it is an important machine learning method, which has many applications in the fields of intelligent control robot and analysis and prediction. The common model of reinforcement learning is the standard Markov decision process (Markov Decision Process, MDP).

Spark MLLib introduction

MLlib is the machine learning library of Spark, through which the engineering practice of machine learning can be simplified. MLlib contains a wealth of machine learning algorithms: classification, regression, clustering, collaborative filtering, principal component analysis and so on. Currently, MLlib is divided into two code packages: spark.mllib and spark.ml.

Spark.mllib

Spark MLlib is an important part of Spark and a machine learning library originally provided. The library has a disadvantage: if the dataset is very complex and needs to be processed many times, or when the new data needs to be calculated comprehensively with multiple single models that have been trained, the use of Spark MLlib will make the program structure complex, even difficult to understand and implement.

Spark.mllib is the original algorithm API based on RDD and is currently in a state of maintenance. The library contains four kinds of common machine learning algorithms: classification, regression, clustering and collaborative filtering. Note that no new features will be added to RDD-based API.

Spark.ml

Spark1.2 version introduces ML Pipeline. After the development of several versions, Spark ML overcomes some shortcomings of MLlib in dealing with machine learning problems (complexity, unclear flow), and provides users with a machine learning library based on DataFrame API, which makes the whole process of machine learning application simple and efficient.

Spark ML is not an official name and is used to refer to the DataFrame API-based MLlib library. DataFrame provides a more friendly API than RDD. Many of the benefits of DataFrame include Spark data sources, SQL / DataFrame queries, Tungsten and Catalyst optimization, and unified API across languages.

Spark ML API provides many data feature processing functions, such as feature selection, feature transformation, category digitization, regularization, dimensionality reduction and so on. In addition, the ml library based on DataFrame API supports the construction of machine learning Pipeline, which organizes some tasks in the machine learning process in an orderly way, which is easy to run and transfer. The spark.ml library is officially recommended by Spark.

Data transformation

Data transformation is an important work of data preprocessing, such as data standardization, discretization, derived indicators and so on. Spark ML provides a wealth of data conversion algorithms. For more information, please refer to the official website, which is summarized as follows:

In the above conversion algorithm, word Frequency inversion document Frequency (TF-IDF), Word2Vec, PCA are relatively common, if you have done text mining processing, then this should be no stranger.

Data specification

Big data is the foundation of machine learning and provides sufficient data training set for machine learning. When the amount of data is very large, it is necessary to delete or reduce redundant dimensional attributes through data specification technology to achieve the purpose of reducing data sets, which is similar to the idea of sampling, although it reduces the data capacity, but does not change the integrity of the data. The feature selection and dimensionality reduction methods provided by Spark ML are shown in the following table:

Feature selection and dimensionality reduction are commonly used in machine learning. The above methods can be used to reduce feature selection, eliminate noise and maintain the original data structure features. In particular, principal component analysis (PCA), whether in the field of statistics or machine learning, plays a very important role.

Machine learning algorithm

Spark supports classification, regression, clustering, recommendation and other commonly used machine learning algorithms. See the following table:

At this point, the study of "what is Spark MLlib machine learning" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.