Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "what modules does the hadoop project include". In the operation of actual cases, many people will encounter such a dilemma. Next, let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Apache Hadoop Engineering has developed into a reliable (reliable), lightweight (scalable), distributed computing (distributed computing) open source software.

Apach Hadoop software library is a framework that allows distributed processing of big data sets across computer clusters with a simple program model. Its purpose (designed to) is to

Expand computing power from a single server to thousands of machines, each of which can provide local computing and storage. Instead of relying on a single hardware for high availability

This library enables purposeful detection and handling of application layer failures, thus providing very high availability * * on computer clusters and easy prone to for a single hardware.

Failure!

The project includes the following modules:

Hadoop Common: common utilities, a general tool that supports other Hadoop modules.

Hadoop Distributed File System (HDFS?): a distributed file system (distributed file system) that provides high throughput (high-throughput) when accessing application data

Hadoop YARN: a framework for job scheduling and cluster resource management

Hadoop MapReduce: a YARN-based concurrent processing (parallel processing) system for large datasets

Other items related to Hadoop on Apache:

Ambari?: is a web-based tool for configuration (provisioning), management (managing) and monitoring, supporting Apache Hadoop biosphere, including Hadoop HDFS, Hadoop MapReduce, Hive, HCatalog, HBase, ZooKeeper, Oozie, Pig and Sqoop. Ambari also provides concise diagrams to observe the health of the cluster, such as hot spot maps (heatmaps) and a friendly user interface to monitor and diagnose the visual characteristics of MapReduce and Pig and Hive applications.

Avro?: a data serialization (serialization) system

Cassandra?: a lightweight multi-master database without a single point of failure

Chukwa?: manages the dataset system of large distributed system

HBase?: is a lightweight distributed database bles. It provides structured data storage for large tables.

Hive?: data warehouse tools that provide data summaries and simple queries

Mahout?: a lightweight machine learning (machine learning) and data mining (data mining) library

Pig?: a high-level (high-level) data flow language and supporting framework for parallel parallel computation computing.

A fast and general computing engine (general compute engine) for Spark?: Hadoop data. Spark is a simple and expressive programming model (expressive programming model) that provides a wide range of applications, including ETL, machine learning (machine learning), pipelined processing (stream processing) and graphical computing (graph computation).

Tez?: is a generalized data flow programming framework built on Hadoop YARN that provides a powerful and flexible engine to run an arbitrary DAG task to process batch and interactive use case data. Tez was first adopted by Hive,Pig and other frameworks on the Hadoop ecosystem, as well as by other commercial software (such as ETL tools) as a potential execution engine to replace Hadoop MapReduce.

ZooKeeper?:, a high-performance distributed application coordination service (coordination service)

This is the end of the content of "what modules does the hadoop project include". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.