Shulou(Shulou.com)05/31 Report--

This article mainly introduces how to use the naive Bayesian algorithm in spark mllib. The article is very detailed and has certain reference value. Interested friends must read it!

advantages

Predicting the sample to be predicted is simple and fast (think of the problem of mail classification, prediction is probability multiplication after word segmentation, and it is faster to add directly in log domain).

It works equally well for multi-class problems, and complexity does not increase significantly.

Under the assumption of independence of distributions, Bayesian classifiers perform exceptionally well, slightly better than logistic regression, and require a smaller sample size.

For input feature variables of category classes, this works very well. For numerical variable characteristics, we assume that they are normally distributed.

disadvantages

For a categorical variable feature in the test set, if it is not seen in the training set, the probability of direct calculation is 0, and the prediction function fails. Of course, we mentioned earlier that we have a technique called smoothing that can alleviate this problem, and the most common smoothing technique is Laplacian estimation.

That…cough cough, naive Bayes calculated the probability result, the size is not bad, the actual physical meaning…mm, don't take it too seriously.

Naive Bayes has the assumption of distribution independence, but in real life these predictors are hardly completely independent.

Most common application scenarios

Text classification/junk text filtering/sentiment discrimination: This is probably where Naive Bayes is used a lot. Even in the current era of such classifiers, Naive Bayes still has a strong place in the text classification scene. The reason is, you know, because multi-classification is very simple, and in text data, the assumption of distribution independence is basically true. And spam text filtering (e.g. spam identification) and sentiment analysis (positive and negative sentiment on tweets) usually work well with Naive Bayes.

Multi-classification real-time prediction: can't this be called a scene? For text-dependent multi-classification real-time prediction, it is widely used because of the advantages mentioned above, simple and efficient.

Recommendation System: Yes, you heard it right, it is used in the recommendation system! Naive Bayes and Collaborative Filtering are a good pair. Collaborative filtering has strong correlation, but its generalization ability is slightly weak. Naive Bayes and Collaborative Filtering together can enhance the coverage and effect of recommendations.

The runtime code is as follows: package spark. logisticRegression import org.apache.spark.mllib.classification.NaiveBayesimport org.apache.spark.mllib.linalg.Vectorsimport org.apache.spark.mllib.regression. LabeledPointport org.apache.spark.mllib.util.MLUtilsimport org. apache.spark. {SparkContext, SparkConf}/** * Naive Bayes zombie fan identification (Naive Bayes requires non-negative eigenvalues) * Normal user marked as 1, false user marked as 0 * V(v1,v2,v3) * v1 = Weibo sent/registered days * v2 = number of friends/registered days * v3 = mobile phone * Weibo sent/registered days

label._ 1 == label._ 2).count() //Compare results println(accuracy) val test = Vectors.dense(0, 0, 10) val result = model.predict(test)//predict a feature println(result)//2 }}

data.txt

0, 1 0 00, 2 0 00, 3 0 00, 4 0 0,1, 0 1 0,1, 0 2 0,1, 0 3 0,1, 0 4 02, 0 0 12, 0 0 22, 0 0 32, 0 0 4 The results are as shown in the figure

The above is "how to use naive Bayesian algorithm in spark mllib" all the contents of this article, thank you for reading! Hope to share the content to help everyone, more relevant knowledge, welcome to pay attention to the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.