Shulou(Shulou.com)06/03 Report--

Speaking of big data, many people can chat for a while, but if you ask big data what the core technology is, it is estimated that many people will not be able to say one or two.

From machine learning to data visualization, big data has a set of mature technology tree, different technical levels have different technical architecture, and new technical terms emerge every year. In the face of such a complex technical architecture, many rookies who come into contact with big data for the first time are almost daunting.

In fact, want to know what core technology big data has is very simple, nothing more than three processes: take data, calculate data, and use data. In this way, some people may still think that space is universal. To put it simply, from the perspective of big data's life cycle, there are no more than four aspects: big data collection, big data preprocessing, big data storage, and big data analysis. together constitute the core technology in big data's life cycle, let's talk about it separately:

First, big data collected

Big data collection is the collection of structured and unstructured massive data from various sources.

Database collection: Sqoop and ETL are popular, and the traditional relational databases MySQL and Oracle still act as data storage methods for many enterprises. Of course, for the open source Kettle and Talend itself, big data integration content is also integrated to achieve data synchronization and integration between hdfs,hbase and mainstream Nosq databases.

Network data collection: a data collection method that obtains unstructured or semi-structured data from web pages with the help of web crawlers or website public API, and uniformly structures it into local data.

File collection: including real-time file acquisition and processing technology flume, ELK-based log acquisition and incremental acquisition and so on.

Recommend big data Learning Exchange skirt 606859705 every night at 20:10 there will be big data live courses, focusing on big data development courses, data analysis methods, big data programming, big data warehouse, big data case, artificial intelligence, data mining is pure practical information sharing, learn a little technical knowledge every day.

Second, big data pretreatment

Big data preprocessing refers to a series of operations on the original data collected before data analysis, such as "cleaning, filling, smoothing, merging, normalization, consistency check", etc., in order to improve data quality and lay the foundation for later analysis work. Data preprocessing mainly includes four parts:

Data cleaning, data integration, data conversion, data specification.

Data cleaning: refers to the use of cleaning tools such as ETL to deal with missing data (lack of attributes of interest), noise data (data with errors or deviating from the expected value), and inconsistent data.

Data integration: refers to the storage method of merging data from different data sources into a unified database, focusing on solving three problems: pattern matching, data redundancy, data value conflict detection and processing.

Data conversion: refers to the process of dealing with the inconsistencies in the extracted data. It also includes the work of data cleaning, that is, cleaning abnormal data according to business rules to ensure the accuracy of subsequent analysis results.

Data specification: it refers to the operation of minimizing the amount of data to get a smaller data set on the basis of keeping the original appearance of the data to the maximum, including: data side aggregation, dimensional specification, data compression, numerical specification, concept layering and so on.

3. Big data Storage

Big data storage refers to the process of using memory to store collected data in the form of a database, including three typical routes:

1. A new database cluster based on MPP architecture.

The use of Shared Nothing architecture, combined with the efficient distributed computing model of MPP architecture, through column storage, coarse-grained index and other big data processing technologies, focusing on the industry big data launched the data storage mode. With the characteristics of low cost, high performance and high expansibility, it is widely used in the application field of enterprise analysis.

Compared with the traditional database, its PB-level data analysis ability based on MPP products has significant advantages. Naturally, MPP database has also become the best choice for a new generation of data warehouse for enterprises.

2. Technology extension and encapsulation based on Hadoop

The technical expansion and encapsulation based on Hadoop is aimed at the data and scenarios that are difficult to deal with in traditional relational databases (for the storage and calculation of unstructured data, etc.), and make use of the open source advantages of Hadoop and related features (good at dealing with unstructured and semi-structured data, complex ETL processes, complex data mining and computing models, etc.) to derive the related big data technology process.

With the technological progress, its application scenario will be gradually expanded. At present, the most typical application scenario is to support the storage and analysis of Internet big data by extending and encapsulating Hadoop, which involves dozens of NoSQL technologies.

3. Big data all-in-one

This is a combination of software and hardware designed for big data's analysis and processing. It consists of a set of integrated servers, storage devices, operating systems, database management systems, and pre-installed and optimized software for data query, processing and analysis. It has good stability and vertical expansibility.

Fourth, big data analysis and mining

The process of extraction, extraction and analysis of disorganized data from the aspects of visual analysis, data mining algorithm, predictive analysis, semantic engine, data quality management and so on.

1. Visual analysis

Visual analysis refers to the analytical means to convey and communicate information clearly and effectively with the help of graphical means. Mainly used in massive data association analysis, that is, with the help of visual data analysis platform, the association analysis of decentralized heterogeneous data, and make a complete analysis chart process.

FineBI visualization

2. Data mining algorithm

Data mining algorithm, that is, through the creation of data mining model, and data exploration and calculation, data analysis means. It is the theoretical core of big data's analysis.

There are a variety of data mining algorithms, and different algorithms based on different data types and formats will show different data characteristics. But generally speaking, the process of creating a model is similar, that is, it first analyzes the data provided by users, then looks for specific types of patterns and trends, and uses the analysis results to define the best parameters for creating a mining model. These parameters are applied to the entire data set to extract feasible patterns and detailed statistical information.

3. Predictive analysis

Predictive analysis is one of the most important application fields of big data analysis. By combining a variety of advanced analysis functions (special statistical analysis, predictive modeling, data mining, text analysis, entity analysis, optimization, real-time scoring, machine learning, etc.), to achieve the purpose of predicting uncertain events.

Help users analyze trends, patterns and relationships in structured and unstructured data, and use these indicators to predict future events and provide a basis for measures to be taken.

4. Semantic engine

Semantic engine refers to the operation of adding semantics to existing data to improve users' Internet search experience.

5. Data quality management

It refers to the identification, measurement, monitoring and early warning of all kinds of data quality problems that may arise in each stage of the data life cycle (planning, acquisition, storage, sharing, maintenance, application, extinction, etc.). A series of management activities to improve data quality.

-dividing line-

The above is from a large point of view, specifically big data has a lot of framework technology, here are some of them:

File storage: Hadoop HDFS, Tachyon, KFS

Offline calculation: Hadoop MapReduce, Spark

Streaming, real-time computing: Storm, Spark Streaming, S4, Heron

Kmurv, NOSQL database: HBase, Redis, MongoDB

Resource management: YARN, Mesos

Log collection: Flume, Scribe, Logstash, Kibana

Message system: Kafka, StormMQ, ZeroMQ, RabbitMQ

Query analysis: Hive, Impala, Pig, Presto, Phoenix, SparkSQL, Drill, Flink, Kylin, Druid

Distributed Coordination Service: Zookeeper

Cluster management and monitoring: Ambari, Ganglia, Nagios, Cloudera Manager

Data mining, machine learning: Mahout, Spark MLLib

Data synchronization: Sqoop

Task scheduling: Oozie

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.