Shulou(Shulou.com)06/02 Report--

This article mainly explains "what are the main advantages and disadvantages of Hadoop3". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn what are the main advantages and disadvantages of Hadoop3.

Main advantages and disadvantages of Hadoop 3

Hadoop is designed to store and manage large amounts of data. Hadoop has many advantages, for example, it is free and open source, easy to use, its performance and so on. But, on the other hand, it also has some shortcomings. So let's start exploring the main advantages and disadvantages of Hadoop.

Advantages of Hadoop

Hadoop is easy to use, scalable, and cost-effective. Here, we will discuss the 12 advantages of Hadoop

Advantages of Hadoop

1. Various data sources

Hadoop stores all kinds of data. Data can come from a variety of sources and can be structured or unstructured. Hadoop can derive value from a variety of data. Hadoop can accept data from text files, XML files, images, CSV files, etc.

two。 High performance / price ratio

Hadoop is an economical solution because it uses clustering to store data. The hardware is a cheap machine, so it is not very expensive to add nodes to the framework. In Hadoop 3.0, there is only 50% storage overhead, compared with 200% in Hadoop2.x. Due to the significant reduction of redundant data, fewer machines are needed to store the data.

3. Performance

Hadoop and its distributed processing and distributed storage architecture can process large amounts of data at high speed. Hadoop even beat the supercomputer to become the fastest computer in 2008. It divides the input data file into blocks and stores the data in block blocks on multiple nodes. It also divides the tasks submitted by users into multiple subtasks, which are assigned to these work nodes that contain the required data, and these subtasks run in parallel, thus improving performance.

4. Fault tolerance

In Hadoop 3.0, erasure coding provides fault tolerance. For example, six data blocks produce three parity blocks by using erasure coding techniques, so HDFS stores a total of nine blocks. If any node fails, you can use these parity blocks and the rest of the data blocks to recover the affected blocks.

5. Highly available

In Hadoop 2.x, the HDFS architecture has an active NameNode and a Standby NameNode, so if the NameNode fails, we can rely on the standby NameNode. However, Hadoop 3.0 supports multiple standby NameNode, which makes the system more available, so if two or more NameNode crashes, it can continue to run.

6. Low network traffic

In Hadoop, each job submitted by a user is divided into multiple independent subtasks, and these subtasks are assigned to the data node, thus moving a small amount of code to the data, rather than moving a large amount of data to the code, resulting in low network traffic.

7. High flux

Throughput refers to work done in a unit of time. Hadoop stores data in a distributed manner, making it easy to use distributed processing. A given job is divided into a number of small jobs that process data blocks in parallel, providing high throughput.

8. Open source

Hadoop is an open source technology, that is, its source code is freely available. We can modify the source code to suit specific requirements.

9. Scalable

Hadoop works according to the principle of horizontal scalability, that is, we need to add the entire computer to the node cluster, rather than change the configuration of the computer like adding RAM, disk, etc., which is called vertical scalability. Nodes can be dynamically added to the Hadoop cluster to make it an extensible framework.

10. Easy to use

Hadoop framework provides a distributed programming model, MapReduce programmers only need to write distributed computing programs according to a fixed template, without caring about how they implement distributed processing, it is done automatically in the background.

11. Compatibility

Most of big data's emerging technologies are compatible with Hadoop, such as Spark,Flink. They have a processing engine that works as a back-end on Hadoop, that is, we use Hadoop as their data storage platform.

twelve。 Support for multiple languages

Developers can code on Hadoop in a variety of languages, such as Clare C + +, Perl,Python,Ruby, and Groovy.

Shortcomings of Hadoop

Shortcomings of Hadoop

1. Small file problem

Hadoop is suitable for dealing with relatively large files, but when it comes to dealing with a large number of small files (files that are much smaller than the block size of Hadoop, which by default can be 128MB or 256MB), Hadoop is inefficient. These large numbers of small files overload Namenode because Namenode stores the system's namespace and makes it difficult for Hadoop to run.

two。 Born vulnerable.

Hadoop is written in Java. Java is a widely used programming language, so it is easy to be used by cyber criminals, which makes Hadoop vulnerable to security vulnerabilities.

3. Handling cost

In Hadoop, data is read from and written to disk, which makes read / write operations very expensive when dealing with terabytes and PB-level data. Hadoop cannot perform in-memory calculations, which increases processing overhead.

4. Only batch processing is supported

At the heart of Hadoop is a batch engine that is inefficient in stream processing. It cannot generate output in real time with low latency. It applies only to data that we collect in advance and store in the file before processing.

5. Iterative processing

Hadoop itself cannot be iterated. Machine learning or iterative processing has periodic data streams, while Hadoop data flows in multiple phase chains, in which the output of one stage becomes the input of the other.

6. Security.

For security reasons, Hadoop uses Kerberos authentication, which is difficult to manage. It lacks storage and network-level encryption, which is a major problem.

Thank you for your reading, the above is the content of "what are the main advantages and disadvantages of Hadoop3". After the study of this article, I believe you have a deeper understanding of the main advantages and disadvantages of Hadoop3, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.