Shulou(Shulou.com)05/31 Report--

This article mainly introduces the Hadoop technology system example analysis, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

The two cores of Hadoop are HDFS and MapReduce, and the whole architecture of Hadoop is mainly supported by the distributed storage of HDFS as the underlying data. And will carry on the calculation and analysis through MapReduce.

The core of Hadoop1.x:

1. Hadoop Common

2. Hadoop Distributed File System (HDFS)

3. Hadoop MapReduce

The core of Hadoop2.x:

1. Hadoop Common

2. Hadoop Distributed File System (HDFS)

3. Hadoop MapReduce

4. Hadoop YARN

Hadoop1.x ecosystem:

Hadoop2.x ecosystem map:

Then from the point of view of understanding, from the bottom up, there are several processes such as data storage, data integration management, data computing, and data mining processing. ETL and log collection tools run through each layer. This is a rough knowledge architecture of Hadoop. Let's take a look at the whole Hadoop technology ecosystem plate one by one.

1. HDFS

A distributed file system divides a file into multiple blocks and stores (copies) them on different nodes. It is the foundation of data storage management in Hadoop system. It is a highly fault-tolerant system that can detect and respond to hardware failures and is used to run on low-cost general-purpose hardware. HDFS simplifies the file consistency model, provides high-throughput application data access through streaming data access, and is suitable for applications with large datasets. It provides the functions of data storage, data backup and data error check in the process of hadoop operation.

2. MapReduce

Distributed computing framework, which is a distributed computing processing model and execution environment, is used for computing with large amounts of data. It includes Map and Reduce parts. Map accepts a key-value pair (key-value) and produces a set of intermediate key-value pairs. The MapReduce framework passes the same value in the middle key value pair generated by the map function to a reduce function. Reduce function: accepts a key and a related set of values, which are combined to produce a smaller set of values (usually only one or zero values).

3. Hive

The data warehouse tool based on Hadoop can map the structured data file to a database table and provide the same query language HiveQL like SQL to manage the data. Hive defines a query language (HQL) similar to SQL, which converts SQL to MapReduce tasks and executes on Hadoop. It is usually used for offline analysis.

4. Pig

Pig is a Hadoop-based big data analysis platform, it provides a high-level language called PigLatin to express big data analysis program, the script into MapReduce tasks to be executed on Hadoop. It is usually used for offline analysis.

5. Mahoutt

Data mining algorithm library, Mahout originated in 2008, was originally a sub-project of Apache Lucent, it has made great progress in a very short time, and is now the top project of Apache. The main goal of Mahout is to create scalable implementations of classic algorithms in the field of machine learning, which are designed to help developers create smart applications more easily and quickly. Mahout now includes widely used data mining methods, such as clustering, classification, recommendation engine (collaborative filtering) and frequent set mining. In addition to algorithms, Mahout also includes data input / output tools, data mining support architectures such as integration with other storage systems such as databases, MongoDB, or Cassandra.

6. ZooKeeper

Distributed collaboration service is a reliable coordination system for large-scale distributed systems, which provides functions including configuration maintenance, name service, distributed synchronization and group service. ZooKeeper is used to manage Hadoop.

7. HBase

HBase is a distributed database of column storage, which provides functions similar to BigTable based on Hadoop. HBase is a scalable, highly reliable, high-performance, distributed and column-oriented dynamic schema database for structured data. Unlike traditional relational databases, HBase uses BigTable's data model: enhanced sparse sort Mapping Table (Key/Value), where keys are made up of row keywords, column keywords, and timestamps. HBase provides random, real-time read and write access to large-scale data. At the same time, the data stored in HBase can be processed by MapReduce, which perfectly combines data storage and parallel computing.

8. Sqoop

Data synchronization tool, abbreviation for SQL-to-Hadoop. Sqoop is a data transfer tool between Hadoop and relational database. The data from the relational database can be imported into the HDFS of Hadoop, or the data from HDFS can be imported into the relational database, which is mainly used to transfer data before the traditional database and Hadoop. The import and export of data is essentially a Mapreduce program, which makes full use of the parallelization and fault tolerance of MR.

9. Flume

Log collection tool, Cloudera open source log collection system, has the characteristics of distributed, high reliability, high fault tolerance, easy to customize and expand. It abstracts the data from the process of generating, transmitting, processing and finally writing to the target path to the data flow. In the specific data flow, the data source supports customizing the data sender in the Flume, thus supporting the collection of different protocol data. At the same time, Flume data stream provides the ability to deal with log data simply, such as filtering, format conversion and so on. In addition, Flume has the ability to write logs to various data targets (customizable). Generally speaking, Flume is a massive log collection system that is scalable and suitable for complex environments.

10. Ambari

It is a Web-based system for monitoring and managing Hadoop clusters. Components such as HDFS,MapReduce,Hive,HCatalog,HBase,ZooKeeper,Oozie,Pig and Sqoop are already supported.

11.Apache Spark

Apache Spark is a computing engine that provides fast data analysis on the big data set. It is built on top of HDFS, but bypasses MapReduce using its own data processing framework. Spark is often used in real-time query, stream processing, iterative algorithm, complex operation and machine learning.

Thank you for reading this article carefully. I hope the article "sample Analysis of Hadoop Technology system" shared by the editor will be helpful to you. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.