Shulou(Shulou.com)06/01 Report--

Hadoop ecosystem

1. Summary

After several years of rapid development, Hadoop has now developed into a software ecosystem containing many related projects. In a narrow sense, the core of Hadoop only includes three sub-projects: Hadoop Common, Hadoop HDFS and Hadoop MapReduce, but closely related to the core of Hadoop include Avro, Zookeeper, Hive, Pig and Hbase and so on. Based on these projects, domain-specific and application-oriented projects such as Mahout, X-Rime, Crossbow and Ivory, as well as peripheral support systems such as Chukwa, Flume, Sqoop, Oozie and Karmasphere, such as data exchange, workflow and development environment. They provide complementary services and together provide a software ecosystem for massive data processing.

Second, detailed explanation

1 、 Hadoop Common

Starting with the Hadoop0.20 version, the Core part of the original Hadoop project was renamed Hadoop Common. Common provides some common tools for other Hadoop projects, including the system configuration tool Configuration, remote procedure call RPC, serialization mechanism, and Hadoop abstract file system FileSystem. They provide basic services for building a cloud environment on general hardware and provide the required API for software development running on the platform.

2 、 HDFS

HDFS,Hadoop distributed file system is the foundation of data storage management in Hadoop system. It is a highly fault-tolerant system that can detect and respond to hardware failures and is used to run on low-cost general-purpose hardware. HDFS simplifies the file consistency model, provides high-throughput application data access through streaming data access, and is suitable for applications with large datasets.

3 、 MapReduce

MapReduce is a computing model, swimsuit for a large amount of data calculation. The MapReduce implementation of Hadoop, together with Common and HDFS, constitutes the three components in the early stage of Hadoop development. MapReduce divides the application into two steps: Map and Reduce, in which Map performs specified operations on the independent elements on the dataset to generate intermediate results in the form of key-value pairs. Reduce regulates all the values of the same key in the intermediate result, and the final result has been obtained. The function partition such as MapReduce is very suitable for data processing in the distributed and parallel environment of a large number of computers.

4 、 Avro

Avro is led by Doug Cuttiing and is a data serialization system. Similar to other serialization mechanisms, Avro can convert data structures or objects into a format that is easy to store and transmit. It is designed to support data-intensive applications and is suitable for large-scale data storage and exchange. Avro provides functions such as rich data structure types, fast and compressible binary data format, file set for storing eat-and-go data, remote calling RPC and simple dynamic language inheritance.

5 、 Zookeeper

How to agree on a certain value (resolution) in a distributed system is a very important fundamental issue. As a distributed service framework, Zookeeper solves the problem of consistency in distributed computing. On this basis, Zookeeper can be used to deal with some data management problems often encountered in distributed applications, such as unified naming service, state synchronization service, cluster management, distributed application configuration item management and so on. Zookeeper, often as a major component of other Hadoop-related projects, is playing an increasingly important role.

6 、 Hive

Hive is an important sub-project in Hadoop, which was first designed by Facebook. It is a data warehouse architecture based on Hadoop. It provides many functions for data warehouse management, including data ETL (extraction, transformation and loading) tools, data storage wall calendar and large data set query and analysis capabilities.

A structured data mechanism provided by Hive defines a SQL-like language similar to the traditional relational database: HiveQL. Through the query language, data analysts can easily run data analysis business.

7 、 HBase

After Google published the BigTable system paper, the open source community began to build the corresponding implementation HBase on HDFS. Hbase is a scalable, highly reliable, high-performance, distributed and column-oriented dynamic schema database for structured data. Unlike traditional relational databases, HBase uses BigTable's data model: enhanced sparse sort Mapping Table (Key/Value), where keys are made up of row keywords, column keywords, and timestamps. HBase provides random, real-time read and write access to large-scale data. At the same time, the data stored in HBase can be processed using MapReduce, which perfectly combines data storage and parallel computing.

8 、 Pig

Pig runs on Hadoop and is a platform for analyzing and evaluating large data sets. It simplifies the requirements of using Hadoop for data analysis and provides a high-level, domain-oriented abstract language: Pig Latin. With PigLatin, data engineers can encode complex and interrelated data analysis tasks into data flow scripts on Pig operations, which can be executed on hadoop by converting the script into a chain of MapReduce tasks. Like Hive, Pig lowers the bar for analyzing and evaluating large data sets.

9 、 Mahout

Mahout originated in 2008 as a sub-project of Apache Lucene. It has made great progress in a very short period of time and is now a top-level project of Apache. The main goal of Mahout is to create scalable implementations of classic algorithms in the field of machine learning, which are designed to help developers create smart applications more easily and quickly. Mahout now includes widely used data mining methods, such as clustering, classification, recommendation engine (collaborative filtering) and frequent set mining. In addition to the algorithm, Mahout also includes data mining supporting architectures such as data input / output tools and inheritance with other storage systems.

10 、 X-RIME

X-RIME is an open source social network analysis tool, which provides a set of large-scale social network / complex network analysis toolkit based on Hadoop. X-RIME parallelizes and distributes more than a dozen social network analysis algorithms on the framework of MapReduce, thus realizing the analysis of Internet-level large-scale social networks / complex networks. It includes a set of data model suitable for large-scale social network analysis on HDFS storage system, a series of distributed computing parallel algorithms for social network analysis based on MapReduce and X-RIME processing bewilderment, that is, X-RIME tool chain.

11 、 Crossbow

CrossBow is an extensible tool based on Bowtie and SOAPsnp, combined with Hadoop, which can make full use of clusters for biological computing. Among them, Bowtie is a fast and efficient tool for splicing short gene sequences into template genomes, while SOAPsnp is a sequence construction program for resequencing consistency. They play an important role in the gene mapping of complex genetic diseases and tumor susceptibility to population and evolutionary genetics. CrossBow uses Hadoop Stream to distribute computing tasks on Bowtie and SOAPsnp to Hadoop clusters, which meets the requirements of massive data storage and computing analysis brought by the new generation of gene sequencing technology.

12 、 Chukwa

Chukwa is an open source data collection system used to monitor large-scale distributed systems (more than 2000 + nodes, the amount of monitoring data generated by the system every day is at T level). It is built on the basis of HDFS and MapReduce of Hadoop and inherits the scalability and robustness of Hadoop. Chukwa contains a powerful and flexible tool set, which provides a series of functions for data generation, collection, sorting, deduplication, analysis and display. It is a necessary tool for Hadoop users, cluster operators and managers.

13 、 Flume

Flume is a distributed, reliable and highly available log collection system developed and maintained by Cloudera. It abstracts the data from the process of generating, transmitting, processing and finally writing to the target path into the data flow. In the specific data flow, the data source supports customizing the data sender in the Flume, thus supporting a variety of mobile phone protocol data. At the same time, Flume data stream provides the ability to deal with log data simply, such as filtering, format conversion and so on. In addition, Flume has the ability to write enough logs to various data targets (customizable). Generally speaking, Flume is a massive log collection system that is scalable and suitable for complex environments.

14 、 Sqoop

Sqoop is the abbreviation of SQL-to-Hadoop and the peripheral tool of Hadoop. Its main function is to exchange data between structured data storage and Hadoop. Sqoop can import data from a relational database into HDFS and Hive of Hadoop, or data from HDFS and Hive into relational database. Sqoop takes full advantage of Hadoop. The whole process of data import and export is parallelized with MapReduce. At the same time, most of the steps in the process are performed automatically, which is very convenient.

15 、 Oozie

To perform data processing in Hadoop, it is sometimes necessary to connect multiple jobs together in order to achieve the ultimate goal. In response to the above requirements, Yahaoo developed the open source work engine flow Oozie, which is used to manage and coordinate multiple jobs running on the Hadoop platform. In Oozie, computing jobs are abstracted as actions, and control flow nodes are used to build dependencies between actions. Together, they form a directed acyclic workflow and describe a complete data processing work. Oozie workflow system can improve the flexibility of data processing process, improve the efficiency of Hadoop cluster, and reduce the workload of developers and operators.

16 、 Karmasphere

Karmasphere includes Karmasphere Analyst and Karmasphere Studio. Among them, Analyst provides the ability to access structured and unstructured data stored in Hadoop, and users can use SQL or other languages for real-time query and further analysis. Studio is a NetBeans-based MapReduce integrated development environment, developers can use it to easily and quickly create Hadoop-based MapReduce applications. At the same time, the tool also provides some visualization tools to monitor the execution of tasks and display the input, output and interaction between tasks. It should be noted that in the above-mentioned projects, the project is not open source.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.