Shulou(Shulou.com)06/01 Report--

This article introduces you to Bayesian law in big data. The content is very detailed. Interested friends can refer to it for reference. I hope it can help you.

Bayes 'law is probably the most robust formula in probability theory. It can be used to calculate conditional probabilities or subjective probabilities.

Bayes 'rule is simple: the probability of a random event changing with the occurrence of relevant conditions, and the belief that a proposition is true or false, i.e., the subjective probability, changes with the discovery of relevant evidence. When a positive correlation condition occurs, the conditional probability is raised, and when a negative correlation condition occurs, the conditional probability is lowered. Subjective probabilities are raised when favorable evidence is found and lowered when unfavorable evidence is found.

However, Bayes 'rule has profound philosophical significance and wide application value. Philosophers use it to solve Hume's induction problem; life scientists use it to study how genes are controlled; educators suddenly realize that the student learning process is the application of Bayesian law; fund managers use Bayesian law to find investment strategies;Google uses Bayesian law to improve search function; artificial intelligence, machine translation uses Bayesian law heavily...

Xiao Bian will lead you to understand Bayesian Law and appreciate its unique charm through formula deduction and three simple examples.

Conditional Probability and Bayes Theorem

conditional probability P(A| B) represents the probability of occurrence of random event A if random event B occurs. conditional probability P(A| B) is also called a posteriori probability and P(A) is its corresponding prior probability.

Here is an example of the first application:

HIV carrier test

Assuming that the HIV rate in the population is 0.01%, the current medical technology to detect it is very sophisticated, if a person really carries HIV, then the blood test has a 99.9% probability of being positive, that is, the probability of detection is very high. If a person does not carry HIV, the blood test has a 0.01% probability of being positive, which means that the probability of falsely accusing a normal person is very low. Now from the street to find a random person to do a test for him, found that the test results are not good, HIV positive, then he really carries the probability of HIV virus?

From this example, we find that if the prior probability of an event occurring is low, then the posterior probability of the event occurring is not necessarily high, even if there is very strong evidence.

2. Naive Bayes method

When Bayes 'rule is actually applied, there are usually many conditions rather than a single one. In order to simplify the problem, we sometimes make the naive assumption that these conditional events are independent of each other, and then we get naive Bayesian methods.

Here is the second example:

spam identification

Suppose you now receive an email that reads as follows:

"Southeast Asia 7-day tour, only 6999. "

Is this email spam? To classify spam algorithmically, we tagged 100000 emails, 80000 of which were normal and 20000 spam. We divided the received email into four words: "Southeast Asia,""7-day tour,""as long as," and "6999." The number of times they appear in marked messages is counted as follows.

Now, using naive Bayes, we can calculate the probability that this email is spam.

So there is a 0.96 probability that this email is spam.

Third, Bayesian ranking model

In addition to the naive Bayes hypothesis, we can use another iterative method to expand posterior probabilities under multiple conditions.

When more conditions exist, you can continue to follow this pattern. The above expansion expression is independent of the iteration order of each conditional event. Here is a simple proof.

Using this iterative expansion, we can construct a Bayesian sorting model to process a lot of information and generate subjective probabilities.

Here is the third example:

Bayesian ranking model

There are two products A and B in the same category, A has 1 five-star rating, B has 5 five-star ratings and 1 four-star rating, so which of the two products do you think is better?

Some students will think that commodity A is better because A has an average star rating of 5 and B has an average star rating of 4.83.

Other students will think B is better because B has more five-star reviews and is more reliable.

In fact, when we process a whole evaluation of a commodity from a lot of comment information, we use Bayesian formula.

In the absence of any information, let's assume that the probability that a product is a great product is 0.5.

And we assume that the probability of a great product receiving star ratings is as follows. That is, we assume that great products tend to receive higher ratings.

The probability that a product that is not excellent will receive a star rating is as follows: we assume that products that are not excellent tend to receive lower ratings.

The iteration is calculated as follows.

So we conclude that commodity B is better.

About how Bayesian law in big data is shared here, I hope the above content can be of some help to everyone and learn more knowledge. If you think the article is good, you can share it so that more people can see it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.