Shulou(Shulou.com)06/02 Report--

This article is to share with you about what Hadoop is. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

What is 1.Hadoop?

Hadoop was originally a sub-project under ApacheLucene. It was originally a project separated from the Nutch project that specializes in distributed storage and distributed computing. In a nutshell, Hadoop is a software platform that makes it easier to develop and run software that handles large-scale data.

two。 Here are some of the main features of hadoop:

1 capacity expansion (Scalable): it can store and process gigabyte (PB) data reliably (reliably).

2 low cost (Economical): data can be distributed and processed through a server farm composed of ordinary machines. In total, these server farms can reach thousands of nodes.

3High efficiency (Efficient): by distributing data, hadoop can parallel the data on the node where it resides, which makes processing very fast.

4 Reliable: hadoop can automatically maintain multiple copies of data and automatically redeploy (redeploy) computing tasks after task failure.

3.Hadoop implements a distributed file system (HadoopDistributedFileSystem), referred to as HDFS.

HDFS has high fault tolerance (fault-tolerent) and is designed to be deployed on low-cost (low-cost) hardware. And it provides high transfer rate (highthroughput) to access the application's data, which is suitable for applications with very large data sets (largedataset). HDFS relaxes the requirement of (relax) POSIX (requirements) so that data in the file system can be accessed (streamingaccess) by stream.

4.Hadoop also implements the MapReduce distributed computing model.

MapReduce breaks down the work of the application into many small smallblocksofwork. In order to achieve reliability, HDFS creates replicas of multiple data blocks (datablocks) and places them in the compute node (computenodes) of the server farm, and MapReduce can process the data on the node where they are located.

As shown in the following figure:

5.HadoopAPI is divided into (divideinto) the following main packages (package)

Org.apache.hadoop.conf defines the configuration file processing API for system parameters.

Org.apache.hadoop.fs defines the abstract file system API.

The implementation of org.apache.hadoop.dfsHadoop distributed file system (HDFS) module.

Org.apache.hadoop.io defines a general I/OAPI for reading and writing to data objects such as networks, databases, files, and so on.

Org.apache.hadoop.ipc is a tool for network server and client, which encapsulates the basic module of network asynchronous Icano.

The implementation of org.apache.hadoop.mapredHadoop distributed computing system (MapReduce) module, including task distribution and scheduling, etc.

Org.apache.hadoop.metrics defines API for performance statistics, mainly for mapred and dfs modules.

Org.apache.hadoop.record defines an I/OAPI class for records and a record description language translator to simplify serializing records into a language-neutral format (language-neutralmanner).

Org.apache.hadoop.tools defines some common tools.

Org.apache.hadoop.util defines some common API.

Thank you for reading! This is the end of this article on "what is Hadoop?". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.