Shulou(Shulou.com)05/31 Report--

This article mainly introduces the role of HDFS, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

1. What is HDFS?

HDFS (Hadoop Distributed File System,Hadoop distributed File system), which is a highly fault-tolerant system, suitable for deployment on cheap machines. HDFS provides high-throughput data access and is suitable for applications with very large data sets (large data set).

2. What is the characteristic of HDFS?

1. Big data files, you can store big data files, if you don't need them, you can use a stand-alone machine to kill chickens. In theory, if the memory of the management node is enough, there is no upper limit, but after all, the hardware conditions are limited. If a block 128m, each block needs 4kb's management information namenode memory has 8G, then 8G/4kb*128 is the upper limit of her cluster.

2. File block storage. HDFS will store a complete large file in different calculators averagely. The block size is 64m, 128m and 256m (depending on the situation). Its significance is that when reading files, you can get different chunks of files from multiple hosts at the same time. Multi-host reading efficiency is much higher than that of a single host.

3. Streaming data access, read and write multiple times at a time, this mode is different from traditional files, it does not support dynamic changes in the contents of the file, but requires that the file should not be changed once written, and content can only be added at the end of the file to change.

4. Cheap hardware, HDFS can be applied to ordinary PCs, this mechanism allows some companies to use dozens of cheap computers to support a big data cluster.

5, hardware failure, HDFS believes that all computers may have problems. In order to prevent a host from failing to read the block file of the host, it allocates a copy of the same block to several other hosts. If one of the hosts fails, it can quickly find another copy to get the file. So usually there are three block backups and one namenode backup.

3. Summary

HDFS is a file that divides a large amount of data into a block of the same size. The block is generally 62m, 128m, etc., and stored in the cluster. It can be stored once, read many times, is not easy to modify, is not suitable for interaction, and has low timeliness.

Thank you for reading this article carefully. I hope the article "what is the role of HDFS" shared by the editor will be helpful to everyone? at the same time, I also hope that you will support and pay attention to the industry information channel, and more related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.