Shulou(Shulou.com)06/01 Report--

What this article shares with you is about the Hadoop biosphere and what its components are. The editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

What is 1.Hadoop?

Distributed Storage and Computing platform suitable for big data

HDFS: Hadoop Distributed File System distributed file system

MapReduce: parallel computing framework

2.Hadoop biosphere

① HBase

Open source implementation of Google Bigtable

Column database

Clusterable

It can be accessed by shell, web, api, etc.

Scenarios suitable for high reading and writing (insert)

HQL query language

Typical representative products of NoSQL

② Hive

Data warehouse tools. You can change the original structured data under Hadoop into tables in Hive.

HiveQL, a language almost identical to SQL, is supported. Almost all other features of SQL support except that updates, indexes and transactions are not supported.

Can be thought of as a mapper from SQL to Map-Reduce

Provide shell, JDBC/ODBC, Thrift, Web and other interfaces

③ Zookeeper

Open source implementation of Google Chubby

Used to coordinate various services on distributed systems. Such as confirming the accurate arrival of messages, preventing single point of failure, dealing with load balancing, etc.

Application scenario: Hbase to realize automatic switching of Namenode

How it works: *, followers and the election process

④ Sqoop

Used to exchange data between Hadoop and relational databases

Connect to relational database through JDBC interface

⑤ Chukwa

Data acquisition and Analysis Framework based on Hadoop

Mainly for log collection and analysis.

Collect the most original log data through the "agent" installed on the collection node

The agent sends data to the collector

The collector writes data to the Hadoop cluster regularly.

Specify the timed start-up Map-Reduce job team data for processing and analysis

⑥ Pig

Hadoop client

Using Pig Latin, a data flow-oriented language similar to SQL

Pig Latin can complete sorting, filtering, summation, clustering, association and other operations, can support custom functions

Pig automatically maps Pig Latin to Map-Reduce jobs and uploads them to the cluster to run, reducing the trouble for users to write Java programs.

⑦ Avro

Data serialization tool, developed by Doug Cutting, founder of Hadoop

Used for applications that support mass data exchange. Support binary serialization, can easily and quickly deal with large amounts of data

Dynamic language is friendly, and the mechanism provided by Avro makes it easy for dynamic language to process Avro data.

Thrift interface

⑧ Cassandra

NoSQL, distributed Key- value-based database, contributed by Facebook

Similar to Hbase, it also draws lessons from the ideological system of Google Bigtable.

Only sequential writing, no random writing design, to meet the performance requirements of high load situations.

Flow chart of 3.Hadoop biosphere

The above is the Hadoop biosphere and what are its components. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.