Shulou(Shulou.com)06/01 Report--

This article mainly explains "the comparison between HDFS and other parallel file systems". The explanation in this article is simple and clear, and it is easy to learn and understand. Please follow the ideas of Xiaobian to study and learn "the comparison between HDFS and other parallel file systems" together.

HDFS, as a new parallel file system, is similar to the existing distributed file system, they are all distributed file systems running on ordinary hardware, but HDFS also has some differences with other distributed file systems. HDFS, for example, is highly fault-tolerant and can be deployed on low-cost hardware, while relaxing the need for POSIX to allow file data to be accessed in streaming form, making it ideal for applications with large data sets. By analyzing the similarities and differences between HDFS and other parallel file systems, we can deeply understand the application scenarios and design concepts of HDFS system.

Transparency of access, where users can access local files and remote files through the same operation. HDFS can do this, but if HDFS is set up as a local file system rather than distributed, then HDFS applications can read and write local files directly, only modifying the configuration file. The access transparency HDFS provides is incomplete; after all, it is built on Java and cannot modify the Unix kernel as NFS and Lustre do to treat local and remote files in a consistent manner.

(2) Concurrent control: reading and writing of files by clients should not affect reading and writing of the same file by other clients. To achieve this single-file copy semantics, distributed file systems require complex interactions, such as timestamping, mutual exclusion, atomic operations, and so on. HDFS uses a very simple mechanism, allowing only one client to write at any time. When a client needs to write a file, it applies for a lease on the file. Only when the lease is released can other clients apply for write operations on the file. A file is created and written closed without changing, and its model is Write-Once-Read-Many.

(3) Resource mobility policy, HDFS supports "move compute to data." If a Map task is an operation on a data block, then the operation must be completed on the node storing the data block. HDFS supports multiple copies of the data (default is 3 copies). The node storing any copy can execute the operation of the Map task. JobTracker will assign the task to the node closest to the client and with copies.

Through the comparison of the above three aspects, we can deeply understand the advantages and disadvantages of HDFS, as well as the design application scenarios of HDFS. HDFS can undoubtedly exert great power for high-throughput and batch processing tasks that pursue massive data, but it is not suitable for random reading and writing of files. HDFS is also not suitable for low-latency access, storage and processing of large numbers of small files.

Thank you for reading, the above is "HDFS and other parallel file system comparison" content, after the study of this article, I believe you have a deeper understanding of HDFS and other parallel file system comparison, the specific use of the situation also needs to be verified. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.