Shulou(Shulou.com)06/01 Report--

Today, I would like to introduce to you what the enterprise big data technical system is like. The content of the article is good. Now I would like to share it with you. Friends who feel in need can understand it. I hope it will be helpful to you. Let's read it along with the editor's ideas.

Doing what you haven't done is called growth, doing what you don't want to do is called change, doing what you don't dare to do is called a breakthrough.

Enterprise big data technical framework (six-tier big data technical system)

1. Data collection layer: distributed, heterogeneous, diversified, streaming generation

It is mainly composed of relational and non-relational data collection components and distributed message queues.

Sqoop/Canal: a relational data collection and import tool, which is a bridge between relational databases and Hadoop. Sqoop can import all data from relational databases into Hadoop, and vice versa. Canal can be used for incremental import of real-time data.

Flume: non-relational data collection tool, mainly streaming log data, which can be collected in near real time, filtered, aggregated and loaded into storage systems such as HDFS.

Kafka: distributed message queue, generally used as a data bus, which allows multiple data consumers to subscribe to and fetch data of interest.

2. Data storage layer

It is mainly composed of distributed file system (for file storage) and distributed database (row / column-oriented storage).

HDFS:Hadoop distributed file system, the open source implementation of Google GFS, has good scalability and fault tolerance. Directories have supported various types of data storage formats, including SSTable, text files, binary key/value format Sequence File, column storage formats Parquet,ORC and Carbondata and so on.

Hbase: a distributed database built on HDFS, which requires users to store structured and semi-structured data, support unlimited expansion of rows and rows, and random search and deletion of data.

Kudu: distributed column database that allows users to store structured data, supports unlimited row expansion and random search and update of data.

3. Resource management and service coordination layer: sharing cluster resources (advantages: high resource utilization, low operation and maintenance costs, data sharing)

YARN: unified resource management and scheduling system, which can manage all kinds of resources (eg:CPU, memory, etc.) in the cluster, and allocate all kinds of applications in the upper layer according to certain policies. YARN has built-in multi-tenant resource schedulers, which allow users to organize and manage resources according to queues, and the scheduling mechanism of each queue can be customized independently.

ZooKeeper: a service coordination system based on the simplified Paxos protocol. It provides a data model similar to the file system, allowing users to implement complex distributed general modules such as leader election, service naming, distributed queues and distributed locks through a simple API.

4. Computing engine layer

Including batch processing (low time requirements, high throughput), interactive processing (high time requirements, sql query), streaming real-time processing (very high time requirements, advertising, etc.).

MapReduce/Tez:MapReduce is a classic batch computing engine with good scalability and fault tolerance, which allows users to write distributed programs through simple Api. Tez is a general DAG (directed acyclic graph) computing engine developed based on MapReduce, which can implement complex data processing logic more efficiently, and is currently used in hive, pig and other data analysis systems.

Spark: a general DAG computing engine that provides an abstract representation of data based on RDD, allowing users to make full use of memory for fast data mining and analysis.

Impala/Presto: an open source MPP system that allows users to use standard SQL to process data stored in Hadoop. They use parallel database architecture, built-in query optimizer, query push-down, code generation and other optimization mechanisms, which greatly improve the processing efficiency of big data.

Storm/Spark Streaming: distributed streaming real-time computing engine, with good fault tolerance and expansibility, can efficiently deal with streaming data, it allows users to complete real-time application development through simple API.

5. Data analysis layer

A variety of data analysis tools are provided to facilitate users to solve big data problems.

Hive/Pig/SparkSQL: the computing engine is only an analysis system built to support SQL or scripting language, which greatly reduces the threshold for users to conduct big data analysis. Hive is a sql engine based on MapReduce/Tez, pig is a workflow engine based on MapReduce/Tez, and SparkSQL is a sql engine based on spark.

Mahout/MLib: the machine learning library built on the computing engine to implement the commonly used machine learning and data mining algorithm Mahout was originally implemented based on MapReduce, and is currently being migrated to spark,mlib based on spark.

Apache Beam/Cascading: high-level API encapsulated based on various computing frameworks to facilitate the construction of complex pipelines. Apache Beam unifies two kinds of computing frameworks: batch processing and streaming processing, and provides a more advanced API for users to write logic code independent of specific computing engines; Cascading has a built-in query plan optimizer, which can automatically optimize the data flow realized by users. Using the tuple-oriented data model, if your data can be represented in a format similar to database rows, it will be easy to use Cascading processing.

6. Data visualization layer

Application of UI display, such as: strategic Dashboard, eye-catching user analysis platform.

Above is what the enterprise big data technology system is all about, more and enterprise big data technology system is what kind of related content can search the previous article or browse the following article to learn ha! I believe the editor will add more knowledge to you. I hope you can support it!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.