Shulou(Shulou.com)06/01 Report--

Editor to share with you what is the background of HDFS, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

Background of HDFS production

With the increasing amount of data, all the data cannot be stored in one operating system, so it will be allocated to more disks managed by the operating system, but it is not convenient to manage and maintain, so there is an urgent need for a system to manage files on multiple machines. This is the distributed file management system. HDFS is just one of the distributed file management systems.

HDFS definition

HDFS (Hadoop Distributed File System), which is a file system that stores files and locates files through a directory tree; secondly, it is distributed and is implemented by many servers, and the servers in the cluster have their own roles

Keywords: file system, distributed

Working with scen

It is suitable for scenarios that are written once and read many times, and the modification of the file is not supported. Suitable for data analysis, but not suitable for network disk applications

Advantages

High fault tolerance (1) multiple copies of data are automatically saved. It improves fault tolerance by adding copies. (2) after a copy is lost, it can be restored automatically.

Suitable for dealing with big data (1) data scale: can handle data with data scale up to GB, TB, or even PB level; (2) File size: can handle more than one million files, and the number is quite large; (3) it can be built on cheap machines to improve reliability through multi-copy mechanism.

Shortcoming

It is not suitable for low-latency data access, such as millisecond storage data.

Unable to store a large number of small files efficiently

(1) if you store a large number of small files, it will take up a lot of memory in Namenode to store file directories and block information. This is not desirable because Namenode's memory is always limited:

(2) the addressing time of small file storage will exceed the read time, which violates the design goal of HDFS.

Do not support concurrent writing and random file modification HDFS

(1) A file can only have one write, and multiple threads are not allowed to write at the same time:

(2) only data append (append) is supported, and random modification of files is not supported.

HDFS composition architecture diagram

HDFS file block size

Think about it: why can't you set it too small or too big?

(1) HDFS's block setting is too small, which will increase the addressing time, and the program has been looking for the start position of the block: (2) the block of HDFS is larger than the block of the disk, and its purpose is to minimize the addressing overhead: (3) if the block is set too large, the time to transfer data from the disk will be significantly longer than the time needed to locate the starting position of this block. Causes the program to be very slow in processing this piece of data.

The above is all the content of this article "what is the background of HDFS?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.