Shulou(Shulou.com)06/02 Report--

This article will explain in detail what is the core value of big data. The editor thinks it is very practical, so I share it for you as a reference. I hope you can get something after reading this article.

The core of big data is prediction. The essence of big data is to solve problems, and the core value of big data lies in prediction. Big data applies mathematical algorithms to massive data to predict the possibility of things happening, which is based on big data and the prediction model to predict the probability of something in the future.

The core of big data is prediction. It is usually regarded as part of artificial intelligence, or rather, as a kind of machine learning. But this definition is misleading. Big data is not trying to teach machines to think like people.

Instead, it applies mathematical algorithms to vast amounts of data to predict the possibility of things happening. The possibility of an email being filtered out as spam, the possibility that the input "teh" should be "the". Judging from the trajectory and speed of a person jaywalking, the possibility that he can cross the road in time is within the range that big data can predict. Of course, if a person can cross the road in time, then when he jaywalks, the car only needs to slow down a little. The key to the success of these prediction systems is that they are based on massive data. In addition, as the system receives more and more data, they can be smart enough to automatically search for the best signals and patterns and improve themselves.

Big data Forecast (big data Core Application)

Big data prediction is the core application of big data, which extends the traditional prediction to "current test". The advantage of big data prediction is that it transforms a very difficult prediction problem into a relatively simple description problem, which is beyond the reach of traditional small data sets. From the perspective of prediction, the results of big data's prediction are not only simple and objective conclusions used to deal with real business, but also can be used to help business decisions.

1. Prediction is the core value of big data

The essence of big data is to solve problems, the core value of big data lies in prediction, and the core of enterprise management is to make a correct judgment based on prediction. When talking about big data's application, the most common application cases are "predicting the stock market", "predicting the flu", "predicting consumer behavior" and so on.

Big data's prediction is based on big data and the prediction model to predict the probability of something in the future. Changing the analysis from "facing the past that has happened" to "facing the future that is about to happen" is the biggest difference between big data and traditional data analysis.

The logical basis of big data's prediction is that every unconventional change must have signs in advance, and everything can be followed. If the law between signs and changes is found, it can be predicted. Big data's prediction is unable to determine that something is bound to happen, it is more about the probability that an event will happen.

With the continuous repetition of experiments and the increasing accumulation of big data, human beings continue to discover all kinds of laws, so that they can predict the future. Using big data to predict possible disasters, using big data to analyze the possible causes of cancer and find out treatments are all causes that can benefit mankind in the future.

For example, big data has been used by the Los Angeles Police Department and the University of California to predict crime; Google flu trends use search keywords to predict the spread of bird flu; MIT uses mobile location data and traffic data for urban planning; and the Meteorological Bureau determines future weather conditions more accurately by collating recent weather conditions and satellite cloud images.

two。 The thinking change predicted by big data

In the past, people's decisions mainly relied on 20 per cent of structured data, while big data predicted that another 80 per cent of unstructured data could be used to make decisions. Big data predicts that it has more data dimensions, faster data frequency and wider data width. Compared with the era of small data, big data's thinking of prediction has three major changes: real samples rather than sampling; prediction efficiency rather than accuracy; and correlation rather than causality.

1) actual sample rather than sampling

In the era of small data, due to the lack of means to obtain all samples, people invented the method of "random survey data". In theory, the more random the sample is, the more representative the overall sample is. But the problem is that getting a random sample is expensive and time-consuming. Population survey is a typical example. It is difficult for a country to complete a census every year, because random research is too time-consuming and labor-consuming. However, the emergence of cloud computing and big data technology makes it possible to obtain large enough sample data and even all data.

2) efficiency rather than precision

In the era of small data, due to the use of sampling methods, it is necessary to be very accurate in the specific operation of data samples, otherwise it will be "by a margin of error and by a thousand miles". For example, if 1000 people are randomly selected from a total sample of 100 million people to conduct a census, if there is an error in the calculation on 1000 people, then the deviation will be very large when zoomed in to 100 million. However, in the case of a full sample, the deviation is as much as the deviation and will not be magnified.

In big data's era, getting a general outline and development context quickly was much more important than strict accuracy. Sometimes, when we have a lot of new data, accuracy is less important, because we can still keep abreast of the trend of things. The simple algorithm based on big data is more effective than the complex algorithm based on small data. The purpose of data analysis is not data analysis, but for decision-making, so timeliness is also very important.

3) correlation rather than causality

Big data's research is different from the traditional logical reasoning research, it needs to do statistical search, comparison, clustering, classification and other analysis and induction of a large number of data, and pay attention to the relevance or relevance of the data. Correlation means that there is a certain regularity between the values of two or more variables. There is no absolute correlation, only possibility. However, if the correlation is strong, the probability of success of a correlation is very high.

Correlation can help us capture the present and predict the future. If An and B often happen together, then we only need to notice that B happens to predict that A will also happen.

According to the correlation, we understand that the world no longer needs to be based on the hypothesis, which refers to the hypothesis about the mechanism and internal mechanism of the phenomenon. So we don't need to make assumptions about which search terms can indicate when and where the flu is spreading, how airlines price tickets, and what Wal-Mart customers like to cook. Instead, we can conduct a correlation analysis of big data to find out which search terms best show the spread of influenza, whether air ticket prices will soar, and which foods people who stay at home want to eat most during the hurricane.

The data-driven correlation analysis of big data replaces the error-prone method based on hypothesis. Big data's correlation analysis method is more accurate, faster, and not easily affected by prejudice. The prediction based on correlation analysis is the core of big data.

Correlation analysis itself is of great significance, and it also lays a foundation for the study of causality. By finding out the things that may be relevant, we can conduct further causality analysis on this basis. If there is a causal relationship, then further find out the cause. This convenient mechanism reduces the cost of causal analysis through rigorous experiments. We can also find some important variables from the interrelationships, which can be used in experiments to verify causality.

3. The typical Application Field of big data's Prediction

The Internet has brought convenience to the popularity of big data prediction application, combined with domestic and foreign cases, the following 11 areas are the most opportunity big data prediction application areas.

1) Weather forecast

Weather forecast is a typical application field of big data prediction. The granularity of weather forecast has been shortened from day to hour, with stringent time requirements. If the calculation is carried out in the traditional way based on the massive data, the conclusion will come a long time ago, and the prediction is of no value, while the development of big data technology provides high-speed computing power, which greatly improves the effectiveness and accuracy of the weather forecast.

2) Forecast of sports events

During the 2014 World Cup, companies such as Google, Baidu, Microsoft and Goldman Sachs all launched platforms to predict the results of the competition. Baidu's prediction result is the most eye-catching, with an accuracy of 67% in 64 games and 94% after entering the knockout stage. This means that future sports events will be controlled by big data's prediction.

The Google World Cup prediction is based on the massive match data of Opta Sports to build the final prediction model. Baidu searches 37000 matches of 987 teams (including national teams and clubs) around the world in the past five years.

At the same time, we cooperate with Chinese lottery website Lottery website and SPdex, an European must-send index data provider, to import the forecast data of the gambling market, and establish a prediction model including 199972 players and 112 million data, and forecast the results on this basis.

From the successful experience of Internet companies, as long as we have historical data on sports events and cooperate with index companies, we can predict other events, such as UEFA Champions League, NBA and so on.

3) Stock market forecast

Last year, a study by the Warwick School of Business in the UK and the Department of Physics at Boston University in the US found that users might be able to predict the direction of the financial market through the financial keywords searched by Google, with a corresponding strategic return of 326%. Previously, some experts have tried to predict stock market fluctuations through Twitter blog sentiment.

4) Market price forecast

CPI is used to characterize price fluctuations that have occurred, but the data from the Bureau of Statistics are not authoritative. Big data may help people understand the future trend of prices and predict inflation or economic crisis in advance. The most typical case is Jack Ma's knowledge of the Asian financial crisis in advance through Ali B2B big data.

It is easier to predict the price of individual goods, especially standardized products such as air tickets. The "ticket calendar" provided by "where to go" is the price forecast, which can tell you the approximate price of the ticket in a few months' time.

Because the production, channel cost and approximate gross profit of goods are relatively stable in a fully competitive market, and the variables related to price are relatively fixed, the relationship between supply and demand of goods can be monitored in real time on the e-commerce platform, so the price can be predicted. Based on the forecast results, we can provide purchase time advice, or guide merchants to carry out dynamic price adjustment and marketing activities to maximize profits.

5) user behavior prediction

Based on the data of users' search behavior, browsing behavior, comment history and personal data, Internet business can gain insight into the overall needs of consumers, and then carry out targeted product production, improvement and marketing. "House of Cards" selects actors and plots, Baidu conducts accurate advertising marketing based on user preferences, Ali customizes products under the production line based on Tmall user characteristics, and Amazon predicts user click behavior for early delivery all benefit from Internet user behavior prediction. As shown in figure 1.

Figure 1 user behavior prediction

Thanks to the development of sensor technology and the Internet of things, offline user behavior insight is brewing. Free commercial Wi-Fi,iBeacon technology, camera image monitoring, indoor positioning technology, NFC sensor network, queuing system, can find out users' offline movement, stay, travel rules and other data, so as to carry out accurate marketing or product customization.

6) Human health prediction

Traditional Chinese medicine can discover the hidden chronic diseases in some people by means of seeing, hearing and asking, and even know what symptoms a person may have in the future by looking at his physique. There is a certain regularity in the changes of human signs, and there will be some persistent abnormalities before the occurrence of chronic diseases. In theory, if big data had mastered such an abnormal situation, he could make a prediction of chronic diseases.

Nature News and Viewpoint reported on a study by Zeevi et al on the complex question of how a person's blood sugar concentration is affected by a particular food. Based on microbes and other physiological conditions in the gut, the study suggests a prediction model that can provide personalized food advice that can predict blood sugar responses more accurately than current standards. As shown in figure 2.

Fig. 2 Prediction model of blood glucose concentration

Smart hardware makes big data's prediction of chronic diseases possible. Wearable devices and smart health devices can help the network collect human health data, such as heart rate, body weight, blood lipids, blood sugar, exercise, sleep and so on. If the data are accurate and comprehensive enough, and there are algorithmic chronic disease prediction patterns, perhaps in the future these wearable devices will remind users of the risk of a certain chronic disease.

7) Forecast of disease epidemic situation

Disease prediction refers to the possibility of predicting large-scale outbreaks based on people's search and shopping behavior, and the most classic "influenza prediction" falls into this category. If there is more and more search demand for "influenza" and "Banlangen" from a certain area, it is natural to speculate that there is an influenza trend there.

Baidu has launched a disease prediction product, which can comprehensively monitor the activity and trend maps of each province and most prefecture-level cities and counties in the country with regard to influenza, hepatitis, tuberculosis and venereal diseases. In the future, the types of diseases monitored by Baidu disease forecast will expand from the current 4 to more than 30, covering more common diseases and epidemics. Users can take targeted prevention according to the local forecast results.

8) disaster prediction

Weather forecast is the most typical disaster prediction. Natural disasters such as earthquakes, floods, high temperatures and torrential rains, if we can make use of big data's ability to predict and inform them in advance, will contribute to disaster reduction, disaster prevention, disaster relief and disaster relief. Different from the past, the past data collection methods have some problems such as dead corner and high cost, but in the era of the Internet of things, people can carry out real-time data monitoring and collection with the help of cheap sensor cameras and wireless communication networks. then use big data prediction analysis to achieve more accurate natural disaster prediction.

9) Prediction of environmental change

In addition to short-term micro weather and disaster prediction, more long-term and macro environmental and ecological changes can be predicted. Shrinking forests and farmland, endangered wildlife, rising coastlines and Greenhouse Effect are "chronic problems" facing the planet. The more data humans know about the earth's ecosystems and weather patterns, the easier it will be to model future environmental changes and prevent bad changes from happening. Big data can help humans collect, store and mine more earth data, as well as provide forecasting tools.

10) Traffic behavior prediction

Traffic behavior prediction refers to the prediction of traffic behavior based on the LBS positioning data of users and vehicles to analyze the individual and group characteristics of human and vehicle travel. The transportation department can carry out intelligent vehicle scheduling or apply tidal lanes by predicting the traffic flow at different time points and different roads, and users can choose the roads with lower congestion probability according to the predicted results.

Baidu's LBS forecast based on map applications covers a wider range. During the Spring Festival travel period, it can predict people's migration trend to guide the setting of train lines and routes, and the flow of people in predictable scenic spots during holidays to guide people's choice of scenic spots. Usually, Baidu Heat tries to tell users about the flow of people in places such as the city's business district and zoo, so as to guide users' travel choice and the location of businesses.

11) Energy consumption forecast

The Likou State Grid system Operations Center manages more than 80 per cent of California's power grid, delivering 289 megawatts of power to 35 million users each year, with power lines more than 40,000 kilometers long. The center uses Space-Time Insight software for intelligent management, comprehensively analyzes massive data from weather, sensors, metering equipment and other data sources, predicts changes in energy demand around the country, conducts intelligent power dispatching, balances power supply and demand throughout the network, and responds quickly to potential crises. China's smart grid is already trying a similar big data forecasting application.

In addition to the 11 areas listed above, big data forecast can also be applied to real estate prediction, employment forecast, college entrance examination score prediction, election result prediction, Oscar prediction, insurance policy-holder risk assessment, financial borrower repayment ability assessment and other fields, so that human beings have quantifiable, persuasive and verifiable insight into the future, and the charm of big data's prediction is being released.

This is the end of the article on "what is the core value of big data". I hope the above content can be helpful to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.