Shulou(Shulou.com)06/01 Report--

Today, I will talk to you about the main problems solved by HDFS and what is the difference between HDFS and IPFS. Many people may not know much. In order to let you know more, Xiaobian summarizes the following contents for you. I hope you can gain something according to this article.

What problems does HDFS mainly solve and how is it different from IPFS?

In recent years, with the promotion of blockchain, big data and other technologies, the global data volume is expanding and increasing without limit. The rise of distributed storage is inseparable from the development of the Internet. Internet companies usually use large-scale distributed storage systems due to their characteristics of big data and light assets.

Unlike traditional high-end servers, high-end memory, and high-end processors, Internet companies 'distributed storage systems consist of a large number of low-cost, cost-effective, common PC servers connected through a network. Due to the rapid development of Internet services, storage system architecture can not rely on the traditional vertical expansion method, that is, buy a small computer first, and then buy a medium computer, or even a mainframe. Distributed systems on the Internet backend require support for horizontal scaling, that is, by adding ordinary PC servers to improve the overall processing capacity of the storage system.

In addition, with the continuous addition of servers, it is necessary to be able to implement automatic Load Balancer at the software level, so that the processing capacity of the system can be linearly expanded. In this case, distributed storage becomes the inevitable choice for most enterprises.

What are the types of distributed storage?

Distributed storage includes a wide variety of types, in addition to the traditional sense of distributed file system, distributed block storage and distributed object storage, including distributed database and distributed cache, but the architecture is nothing more than three:

A. Intermediate control node architecture-The architecture represented by HDFS is a typical representative

B. Completely no-center architecture-computing model, represented by Ceph architecture is its typical representative

C. Completely centerless architecture-consistent hash, represented by swift architecture is its typical representative

Here we mainly compare HDFS and IPFS

Introduction to HDFS

HDFS (Hadoop Distributed File System) is a core subproject of the hadoop project and is the foundation for data storage management in distributed computing. It was developed based on the need to access and process very large files in streaming data mode and can run on inexpensive commercial servers.

It has the characteristics of high fault tolerance, high reliability, high scalability, high availability, high throughput and so on, which provide fault-free storage for massive data and bring a lot of convenience to the application processing of large data sets.

HDFS is open source and stores the data that Hadoop applications will process, similar to ordinary Unix and Linux file systems, except that it implements the idea of Google's GFS file system and is a scalable distributed file system suitable for large-scale distributed data processing-related applications.

Why HDFS?

A small amount of data, a single disk is able to handle the data well, but when the data volume is huge (PB), the disk begins to struggle with the huge amount of information we need. We couldn't speed up individual disks, because the technology ran out of space, and we had to break big tasks down into smaller tasks, a disk into multiple disks. To manage files on multiple disks is a distributed file management system-HDFS

Features of HDFS

1) Distributed storage and processing of data.

Hadoop provides a command interface to interact with HDFS.

3) Built-in servers for namenode and datanodes help users easily check the state of the cluster.

4) Streaming access to file system data.

HDFS provides file permissions and authentication.

HDFS System Architecture and Main Components

As you may have noticed when starting a Hadoop cluster earlier, there are two types of HDFS-related processes in the cluster: namenode and datanode. HDFS is a master-slave architecture system in which the namenode acts as the master node managing multiple slave datanodes. Its architecture is shown below:

Namenode：

Management maintains the file system tree and all files and directories within the tree, i.e. file system metadata; controls client access to files; and performs file system operations such as renaming, closing, and opening files/directories. DateNode：

Manage stored data; perform read and write operations on the file system as requested by clients; and perform operations such as block creation, deletion, and backup as directed by NameNode.

Block

Typically the user's data is stored in a file on HDFS; the file is split into one or more fragments and stored in individual data nodes; these file fragments are called blocks. In other words, the minimum amount of data HDFS can read and write is called a Block. The default block size is 64MB/128 MB (can be increased depending on configuration).

Rack

A rack for installing cluster computers. A rack can install several computers, and there will be several such racks in the entire Hadoop cluster.

If the client needs to read data from a file, it first gets the location of the file from the NameNode, and then gets the specific data from the NameNode. In this architecture, NameNode is usually a Secondary NameNode, while DataNode is a cluster composed of a large number of nodes. Because the access frequency and access volume of metadata are much smaller than those of data, NameNode usually does not become a performance bottleneck, while DataNode clusters can have replicas of data, which can ensure high availability and disperse client requests. Therefore, through this distributed storage architecture, the bearing capacity can be increased by horizontally expanding the number of datanodes, that is, the ability to dynamically scale horizontally is realized.

Typically, user data is stored in HDFS files. Files in a file system are divided into one or more fragments stored in a single data node. These file segments are called blocks. In other words, the minimum amount of data HDFS can read or write is called a block. The default block size is 64MB, which can be changed depending on HDFS configuration.

Features of HDFS

Fault detection and recovery-Because HDFS contains a large amount of product hardware, component failures occur frequently. Therefore, HDFS should have a mechanism for fast automatic fault detection and recovery.

Management of datasets- HDFS has hundreds of nodes per cluster to manage applications with large datasets.

Data Hardware Processing-When computing is physically near the data, requested tasks can be efficiently completed. Especially when large data sets are involved, it reduces network traffic and improves throughput.

Introduction to IPFS

IPFS (Inter Planetary File System) is also known as the Interplanetary File System. IPFS opened in 2015 and is now five years old. IPFS and Filecoin have always been hot and influential. Here we put aside the Filecoin part of the blockchain, focusing on the application of IPFS in distributed storage.

How IPFS works

The first principle is that in IPFS, every file is hashed and a digital fingerprint is generated.

Second, when we want to find a file, IPFS can quickly find the node that owns the data by using a distributed hash table, and use the hash to verify whether it is the correct data, so as to find the file we want.

Third, IPFS removes duplicate files with the same hash value across the network, that is, it calculates which files are redundant and tracks the version history of each file.

Fourth, each network node stores only what it is interested in, along with some index information that helps us figure out who is storing what.

Fifth, using what is known as IPNS (Decentralized Naming System), each file can be collaboratively named with an easy-to-read name, and by searching, we can easily find the file we want to view.

IPFS and HTTP are both referred to as the underlying Internet protocol. Well, when surfing the Internet, we can often see such a string of characters, http://www.baidu.com, or http://www.taobao.com, or http://www.aiqiyi.com, etc. This is what we commonly call the domain name. However, IPFS has a lot of advantages over HTTP, mainly reflected in the following aspects:

IPFS is more secure. On the one hand, each file in IPFS and all the blocks in it are given a unique fingerprint called cryptographic hash; on the other hand, IPFS is a point-to-point distributed file system that can be used to store files, which we can understand as: including text, pictures, audio, video, etc.; furthermore, because IPFS works by breaking up the entire file and then storing it in different nodes around the world. When data is needed, it can be retrieved from the original storage location through the index of the file to protect the privacy and security of the data.

For example, BAT, the cloud storage method we use now is: we give the data to BAT (Baidu Cloud, Alibaba Cloud, Tencent Cloud), and find BAT to get it back when we need data. This process may seem like nothing wrong, but what if BAT's servers go down, or your privacy is being snooped on?

IPFS makes it possible to upload and download data faster and to store data permanently. Because IPFS is composed of global storage nodes, that is to say, in the future, we can quickly access files stored on the ipfs network from every corner of the world. Simply put, these files are encrypted and stored in computers, mobile phones, etc., which use hard disks.

From the above principles, we can clearly see that IPFS is completely different from traditional distributed storage in terms of storage and is completely decentralized.

HDFS vs IPFS

a. Application object

HDFS is mainly an enterprise-level application, aiming at large file storage of enterprises, because HDFS adopts file management in the form of metadata, and information such as related directories and blocks of metadata is stored in the memory of NameNode, and the increase in the number of files will occupy a large amount of NameNode memory. If there are a large number of small files, it will take up a lot of memory space, causing the performance of the entire distributed storage to decline, so it is more appropriate to use HDFS to store large files as much as possible. IPFS is mainly aimed at the individual user market, according to the individual file storage, the more nodes stored, the more files stored, the higher the stability of the entire file system.

b. Frequency of reading and writing

HDFS is suitable for low-write, multi-read businesses. HDFS has high data transmission throughput, but poor data read latency and is not suitable for frequent data writes. IPFS is very inclusive and scalable for file reading and writing. The more files are read and written, the more prosperous the entire IPFS-based economic ecosystem will be and the more users in the system will benefit.

c. Storage environment

HDFS adopts a multi-copy data protection mechanism, which can guarantee the reliability of data using ordinary X86 servers and is not recommended for use in virtualized environments. IPFS uses personal ordinary servers as nodes to run IPFS systems and provide decentralized storage services

d. Storage system

HDFS is mainly aimed at large enterprises, although it is distributed storage, its main control is still the main body of the enterprise, belonging to a closed storage system. IPFS is a fully decentralized operation, allowing any business or individual to access the storage network.

e. Addressing mode

HDFS If the client needs to read data from a file, it first obtains the location of the file from the NameNode, and then obtains the specific data from the NameNode. IPFS obtains the file directly from the node where the content is located, which is a content-based method.

Applications developed based on IPFS technology are also emerging, IPFS is directly integrated into Brave browser, Hadoop is placed on IPFS for p2p data analysis, PeerPad uses IPFS to build serverless, real-time, offline collaborative applications, etc. After establishing cooperative relations with well-known institutions and enterprises such as Microsoft and NASA, the practical application value of IPFS has been further deepened.

Summarize the advantages of IPFS/IPSE distributed architecture:

a decentralized

Distributed node network, no single point problem

Encryption protects data integrity and security

Storage costs and transmission costs are much lower than centralized systems

After reading the above, do you have any further understanding of the main problems HDFS solves and what is the difference between HDFS and IPFS? If you still want to know more knowledge or related content, please pay attention to the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.