Shulou(Shulou.com)06/01 Report--

This article mainly introduces "the introduction of various components in hadoop". In daily operation, I believe that many people have doubts about the introduction of various components in hadoop. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful for you to answer the doubts of "introduction of various components in hadoop"! Next, please follow the editor to study!

HDFS (Hadoop distribute file system)-the basic component of the Hadoop ecosystem, the Hadoop distributed file system. It is the basis of some other tools HDFS mechanism is to distribute a large amount of data over the computer cluster, the data is written once, but can be read multiple times for analysis. HDFS allows Hadoop to maximize disk utilization. HBase-- is a column-oriented NoSql database built on top of HDFS. HBase is used for fast reading / writing of volume data. HBase uses Zookeeper for its own management to ensure that all its components are running. HBase enables Hadoop to maximize memory utilization. MapReduce--MapReduce is the main execution framework of Hadoop. It is a programming model for distributed parallel data processing. It divides jobs into mapping phase and reduce phase. Developers say that Hadoop writes MapReduce jobs and uses data stored in HDFS, while HDFS ensures fast data access. Due to the nature of MapReduce jobs, Hadoop moves processing to data in a parallel manner. MapReduce enables Hadoop to maximize the use of CPU. Zookeeper--Zookeeper is the distributed coordination service of Hadoop. Zookeeper is designed to run on machine clusters, is a highly available service for managing Hadoop operations, and many Hadoop components rely on it. Oozie-- Oozie is an extensible Workflow system that makes arctic testing difficult to get into the Hadoop software stack. Used to coordinate the execution of multiple MapReduce jobs. It can handle a great deal of complexity and manage execution based on external events. Pig--Pig is an abstraction of the complexity of MapReduce programming, and the Pig platform contains an execution environment and scripting language (Pig Latin) for analyzing Hadoop datasets. Its compiler translates Pig Latin into a sequence of MapReduce programs. Hive-- is similar to the high-level language of SQL and is used to execute queries for data stored in Hadoop. Hive allows developers who are not familiar with MapReduce to write data query statements that will be translated into MapReduce jobs in Hadoop. Similar to Pig. Hive is an abstraction layer suitable for database analysts who are more familiar with SQL than java programming. The Hadoop ecosystem also includes frameworks for integration with other enterprise applications, such as Sqoop and Flume shown in the figure above: Sqoop is a connectivity tool for moving data between relational databases and data warehouse Hadoop. Sqoop uses databases to describe the patterns of importing / exporting data, and uses MapReduce to implement parallel operations and fault tolerance. Fulme is a distributed, reliable and highly available service for efficiently collecting, aggregating, and moving large amounts of data from separate machines to HDFS. It gives a simple and flexible framework for child labourers to stream data. With the help of a simple and extensible data model, it allows data from multiple machines in the enterprise to be moved to Hadoop.

Hbase is a distributed database based on hadoop. Hive is hive sql. Users can write hive sql,hive to generate relevant mapreduce jobs for sql and submit them to the hadoop cluster to run such jobs. Hive can analyze files on hdfs directly, or it can analyze hbase table data. Generally, hive is installed and run on namenode.

To put it simply, hive is used to process data in batches, and HBase is used to quickly index data.

HBase is a distributed non-relational database based on column storage. The query efficiency of HBase is very high, mainly due to query and display results.

Hive is a distributed relational database. It is mainly used for parallel and distributed processing of large amounts of data. All queries in hive except "select * from table;" need to be executed through Map\ Reduce. Because of Map\ Reduce, even a table with only one row and one column may take 8 or 9 seconds if it is not queried through select * from table;. But hive is good at dealing with large amounts of data. When there is a lot of data to deal with, and the Hadoop cluster is large enough, it shows its advantages.

At this point, the study of "introduction to various components in hadoop" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.