Shulou(Shulou.com)06/02 Report--

Three important Technical problems of big data

Today, the arrival of big data has become an unavoidable challenge in real life. Whenever we have to make a decision, big data is everywhere. The widespread emergence of big data terminology also makes people gradually understand its importance. Big data gradually showed people the great opportunities it brought to academia, industry and government. At the same time, big data also posed great challenges to all parties involved, first of all, three important technical issues:

1. How to deal with unstructured and semi-structured data by means of information technology

In big data, structured data accounts for only about 15 per cent, while the remaining 85 per cent are unstructured data, which are abundant in areas such as social networks, the Internet and e-commerce. On the other hand, perhaps 90% of the data comes from open source data, and the rest is stored in the database. Big data's uncertainty is shown in high-dimensional, changeable and strong randomness and so on. Stock trading data flow is a typical example of uncertainty big data.

Big data stimulated a large number of research problems. The individual performance, general characteristics and basic principles of unstructured and semi-structured data are not clear, which need to be studied and discussed through the interdisciplinary study of mathematics, economics, sociology, computer science and management science. Given a semi-structured or unstructured data, such as an image, how can it be transformed into a multidimensional data table, an object-oriented data model, or a direct image-based data model? It is worth noting that each form of representation of big data only presents the side performance of the data itself, not the whole picture.

If the process of extracting "rough knowledge" through data mining is called "primary mining" process, then the process of combining rough knowledge with quantified subjective knowledge, including specific experience, common sense, instinct, situational knowledge and user preference, to produce "intelligent knowledge" is called "secondary mining". The leap from "primary mining" to "secondary mining" similar things "quantity" to "quality".

Because of the semi-structured and unstructured characteristics of big data, the structured "rough knowledge" (potential pattern) generated by big data's data mining is also accompanied by some new features. These structured rough knowledge can be processed and transformed by subjective knowledge to generate semi-structured and unstructured intelligent knowledge. The search for "intelligent knowledge" reflects the core value of big data's research.

Second, how to explore the description method of big data's complexity and uncertainty and big data's system modeling.

The breakthrough of this problem is the premise and key to realize big data's knowledge discovery. From a long-term point of view, the challenges brought by big data's individual complexity and randomness will promote the formation of big data's mathematical structure, which will lead to the completion of big data's unified theory. In the short term, academia encourages the development of a general transformation principle between structured data and semi-structured and unstructured data to support big data's cross-industrial applications. Management science, especially the theory based on optimization, will play an important role in developing the general method and regularity of big data's knowledge discovery.

Big data's complex form leads to many research problems related to the measurement and evaluation of "rough knowledge". The known optimization, data envelopment analysis, expectation theory and utility theory in management science can be applied to the "secondary mining" process of how to integrate subjective knowledge into rough knowledge generated by data mining. Here human-computer interaction will play a vital role.

Third, the influence of the relationship between data heterogeneity and decision heterogeneity on big data's knowledge discovery and management decision.

Due to the complexity of big data itself, this problem is undoubtedly an important scientific research topic, which poses a new challenge to the traditional data mining theory and technology. In the environment of big data, management decision-making is faced with two "heterogeneity" problems: "data heterogeneity" and "decision heterogeneity". The traditional management decision model depends on the learning of business knowledge and the accumulation of practical experience, and management decision is based on data analysis.

Big data has changed the traditional mode of management decision-making structure. The study of big data's influence on management decision-making structure will become an open scientific research issue. In addition, the change of decision structure requires people to explore how to do "secondary mining" in order to support higher-level decision-making. No matter what kind of data heterogeneity big data brings, the "rough knowledge" in big data can still be regarded as the category of "one-time mining". It is necessary to find the "intelligent knowledge" generated by "secondary mining" as a bridge between data heterogeneity and decision heterogeneity. Exploring how the decision-making structure is changed in the environment of big data is equivalent to studying how to participate the subjective knowledge of decision-makers in the decision-making process.

Big data is an artificial nature with hidden rules. Finding big data's scientific model will lead to an exploration of the general method of studying the beauty of big data, although such an exploration is very difficult. however, if we find a way to transform unstructured and semi-structured data into structured data, the known data mining methods will become big data's mining tools.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.