Shulou(Shulou.com)05/31 Report--

This article mainly introduces what is the copy placement strategy in HDFS. It is very detailed and has a certain reference value. Friends who are interested must read it!

Cloud Computing

Cloud computing (cloud computing) is a model for the increase, use, and delivery of related services based on the Internet, which usually involves the provision of dynamic, scalable and often virtualized resources over the Internet. Cloud is a metaphor for the Internet and the Internet. In the past, clouds were often used to represent telecommunications networks, and later they were also used to represent the abstraction of the Internet and the underlying infrastructure. As a result, cloud computing can even allow you to experience 10 trillion computing power per second, with such powerful computing power to simulate nuclear explosions, predict climate change and market trends. Users access the data center through computers, notebooks, mobile phones and other ways to calculate according to their own needs. As defined by the National Institute of Standards and Technology (NIST), cloud computing is a pay-per-use model that provides available, convenient, on-demand network access to a pool of configurable computing resources (including networks, servers, storage, applications, services) that can be provided quickly with little management effort. Or have little interaction with the service provider.

Cloud computing is the product of the development and integration of traditional computer and network technologies, such as distributed computing (Distributed Computing), parallel computing (Parallel Computing), utility computing (Utility Computing), network storage (Network Storage Technologies), virtualization (Virtualization), load balancing (Load Balance), hot backup redundancy (High Available) and so on.

Cloud computing can be considered to include the following levels of services: infrastructure as a service (IaaS), platform as a service (PaaS) and software as a service (SaaS).

Cloud Stora

Cloud storage is a new concept extended and developed from the concept of cloud computing, which refers to a system that integrates a large number of different types of storage devices in the network through application software to provide data storage and business access functions through cluster applications, grid technology, distributed file systems and other functions. When the core of the operation and processing of the cloud computing system is the storage and management of a large amount of data, the cloud computing system needs to be configured with a large number of storage devices, then the cloud computing system will be transformed into a cloud storage system. So cloud storage is a cloud computing system with data storage and management as the core. To put it simply, cloud storage is a new scheme to put storage resources on the cloud for human access. Users can easily access data on the cloud at any time, anywhere, through any networked device.

The structure model of cloud storage system consists of four layers.

Storage layer

The storage layer is the most basic part of cloud storage. Storage devices can be FC fibre Channel storage devices, IP storage devices such as NAS and iSCSI, or DAS storage devices such as SCSI or SAS. The storage devices in cloud storage are often large in number and distributed in many different regions, and are connected to each other through wide area network, Internet or FC fibre Channel network.

On top of the storage device is a unified storage device management system, which can realize logical virtualization management, multi-link redundancy management, as well as state monitoring and fault maintenance of hardware devices.

Basic management

The basic management layer is not only the core part of cloud storage, but also the most difficult part to achieve. The basic management layer realizes the cooperative work among multiple storage devices in cloud storage through cluster, distributed file system and grid computing technology, so that multiple storage devices can provide the same service to the outside world. and provide larger, stronger and better data access performance.

Application interface

The application interface layer is the most flexible part of cloud storage. Different cloud storage operators can develop different application service interfaces and provide different application services according to the actual business types. Such as video surveillance application platform, IPTV and video-on-demand application platform, network hard disk reference platform, remote data backup application platform and so on.

Access layer

Any authorized user can log in to the cloud storage system through the standard public application interface and enjoy the cloud storage service. Cloud storage provides different access types and means according to different cloud storage operators.

HDFS

Hadoop distributed file system (HDFS) is designed to be suitable for distributed file system running on general hardware (commodity hardware). It has a lot in common with existing distributed file systems. But at the same time, the difference between it and other distributed file systems is also obvious. HDFS is a highly fault-tolerant system that is suitable for deployment on cheap machines. HDFS can provide high-throughput data access, which is very suitable for applications on large-scale data sets. HDFS relaxes some of the Portable operating system Interface (POSIX) constraints to achieve the purpose of streaming file system data. HDFS was originally developed as the infrastructure of the Apache Nutch search engine project. HDFS is part of the Apache Hadoop Core project.

The following is a detailed description of HDFS and its copy placement strategy in this article.

Replica technology

Replica technology, that is, distributed data replication technology, is an important part of distributed computing. This technology allows data to be shared on multiple servers, a local server can access data on remote servers in different physical locations, and all servers can hold copies of the data.

Through copy technology, you can have the following advantages:

(1) improve the reliability of the system: the system will inevitably produce failures and errors, and the file system with multiple copies will not cause inaccessibility, thus improving the availability of the system. In addition, the system can repair the error copy through other intact copies, thus improving the fault tolerance of the system.

(2) load balancing: the replica can expand the load of the system. Multiple copies are stored on different servers, which can effectively share the workload, thus the larger workload will be effectively distributed on different sites.

(3) improve the access efficiency: create the copy in the area with high access frequency, that is, the copy is near the access node, accordingly reduce its communication overhead, thus improving the overall access efficiency.

HDFS architecture

Figure 1.HDFS architecture diagram

HDFS is a master-slave structure, a HDFS cluster has a name node, it is a master server that manages file namespaces and regulates client access to files, and of course there are some data nodes, usually a node and a machine, which manages the storage of the corresponding nodes. HDFS opens the file namespace to the outside world and allows user data to be stored as files.

The internal mechanism is to split a file into one or more blocks, which are stored in a set of data nodes. The name node is used to manipulate the file or directory operations of the file namespace, such as opening, closing, renaming, and so on. It also determines the mapping of the block to the data node. The data node is responsible for read and write requests from file system customers. The data node also performs block creation, deletion, and block copy instructions from the name node.

Only one name node in the cluster greatly simplifies the architecture of the system. The name node is the repository of the arbitrator and all HDFS metadata, and the actual data of the user does not pass through the name node.

Data replication

HDFS is designed to reliably store a large number of files between a large number of machines in a cluster, which stores files in the form of block sequences. All blocks in the file have the same size except the last block. Blocks belonging to the file are copied for fault tolerance. The block size and replication number are configured on a file-by-file basis, and the application can modify the replication factor when or after the file is created. Files in HDFS are written at once, and there is only one write operation at any one time.

The name node is responsible for handling decisions related to all block copies. It periodically accepts heartbeats and block reports from data nodes in the cluster. The arrival of a heartbeat indicates that the data node is normal. A block report includes a list of all blocks on the data node.

Copy placement strategy

The choice of storage location of block copies seriously affects the reliability and performance of HDFS. HDFS adopts a rack-sensitive (rack awareness) replica storage strategy to improve data reliability, availability and network bandwidth utilization.

Figure 2. Copy placement strategy map

HDFS runs on clusters that span a large number of racks. Nodes on two different racks communicate through switches, and in most cases, the network bandwidth between machines on the same rack is better than that on different racks.

At the beginning, each data node self-tests the rack id to which it belongs, and then informs its rack id when registering with the name node. HDFS provides an interface to easily mount modules that detect rack markings. A simple but not optimal way is to place copies on different racks, which prevents data loss in the event of rack failure and makes full use of the bandwidth of different racks when reading data. In this way, replication is evenly distributed in the cluster, which simply achieves load balancing in the event of a failure. However, this approach increases the cost of writing because you need to transfer file blocks across multiple racks.

In general, the replication factor (number of copies of files) is 3. The replica placement strategy for the HDFS is to place the first copy on the local node, the second copy on another node on the local rack, and the third copy on a different rack node. This approach reduces write traffic between racks, thereby improving write performance. The probability of rack failure is much less than that of node failure. This approach does not affect the limits of data reliability and availability, and it does reduce the network aggregate bandwidth for read operations, because there are only two different racks for file blocks instead of three. The copies of the files are not evenly distributed in the rack, 1x3 copies are on the same node, 1x3 copies are on the same rack, and the other 1Action3 replicas are evenly distributed on other racks. This method improves the performance of writing without affecting the reliability and read performance of the data.

Figure 3. Copy placement policy source code analysis diagram

HDFS copy placement policy flaw

The default replica placement policy of HDFS is a random selection policy, and the final state of replica placement is difficult to control. The specific manifestations are as follows:

(1) it achieves high reliability and uniform distribution of data blocks by placing replicas on different racks, but if there is only one rack in the data center, then the replica placement strategy is reduced to a random selection strategy. it is difficult to guarantee the uniform distribution and reliability of data blocks.

(2) it does not take into account the load of the node, even if the load of one node is much higher than that of the other nodes, according to the default replica placement policy, this node may still be continuously written to data.

(3) it mainly considers the placement location of the first three replicas, and if more copies need to be placed, the placement of the later copies will adopt the strategy of random selection.

Of course, there is a daemon called equalizer (Balancer) in HDFS system. It will move the data blocks from the nodes with higher load to the nodes with lower load, so as to achieve the purpose of redistribution of data blocks, and finally make the distribution of the whole cluster balanced. Although the equalizer can solve the problem to some extent, it is also flawed:

The main results are as follows: (1) it lags behind the adjustment of cluster data block balance.

(2) the adjustment of equalizer and the movement of data blocks require the consumption of certain resources.

Therefore, the key to solving the problem should be placed on the placement of the original copy rather than remedial afterwards. This requires appropriate optimization of the replica placement strategy for HDFS.

Optimization of HDFS replica placement strategy

According to the defects mentioned above, this paper proposes a replica placement strategy UILBPS (Placement Strategy based on User Information and Load Balance) based on user information and load.

Based on the characteristics of HDFS, the operation after the replica is created is mainly read, which takes the availability and access rate of the load-balanced replica as the optimization goal of the replica placement policy. Among them, the performance of the replica access rate is affected by the load of the replica node, especially the Imax O load will become the bottleneck of access; the state of the replica node will also have a certain impact on the replica access, and affect the later load balancing. In addition, the access to the copy created by the same user is geographically similar. Therefore, UILBPS will solve the problem of determining the number and location of replicas in the initial creation phase through the extraction of replica access features and the calculation of node load.

Replica access feature extraction

The feature extraction of user copy placement is mainly based on the user's historical information. For a user with historical information, the copy he created already has a certain access record. We define the replica access location vector cn_P and the access location range cn_cov to extract the features of replica access. Where cn_P represents the specific location where the copy is accessed, and cn_cov represents the main distribution range of clients accessing the same copy (Client).

For the data block I with k replica access location records, its cn_cov is the minimum location range corresponding to which the Q records are greater than Q records (qvalue 0.8k). If there is no such range, the cn_cov is a full set.

The extraction process is as follows:

(1) Client randomly selects n data blocks created by users to form a sampling sample set.

(2) Client sends the information of the number of copies of the set N and the cn_P request to the name node to obtain the set of the number of copies of the sample and the set of cn_P.

(3) Client calculates the lower bound of the average value of copies of n data blocks in the sample set as the number of copies created rep_num. The default copy lower bound is 2, that is, there are at least 2 copies.

(4) Client calculates the cn_cov according to the cn_P of the sample and forms a set containing the location range of n replicas, F = {F1, f2, and f3,. , fn} .

(5) Client selects the set F' (the union of F child elements) according to the set F to determine the location of the copy.

(6) Client sends rep_num and F' to the name node.

As shown in the following table, according to the records of the access node, it can be concluded that the access location range is a vector (0574).

The number of visits to the international region of Node Chau a08108015b08602121c0860574109d086057481e086057490f086058017g118371h3550899i437213j5145032

Node load

After obtaining the replica creation number and replica location selection set, the name node selects req_num nodes with low load to place copies according to the load of the nodes in the replica location selection set.

The greater the node load, the busier the disk, the more space used; the smaller the disk, the less space used. Therefore, by considering the load of nodes, on the one hand, block write operations caused by busy disks can be avoided, resulting in a decline in system performance; on the other hand, the distribution of replicas can be balanced as much as possible. reduce the pressure of subsequent copy adjustment operations and save system resources.

Therefore, disk Imax O load and disk space load are the main factors affecting the load of the whole node. The formula is as follows:

Listing 1. Formula for calculating node load

LOAD=W1*IO + W2*SPACE

Where W1 and W2 are the weight values of the disk IZP O load and the disk space load.

Placement algorithm

Figure 4. Copy creation flowchart

Comparative Analysis and thinking of copy Placement Strategy

This section compares the HDFS default algorithm with the UILBPS algorithm through simulated data. The initialization simulation system has 3 continents, 5 countries, 10 regions, 10 data centers, 20 racks, 200 nodes, 20 Client,250 data and 2 copies. The nodes and Client are randomly distributed on different racks, and the copies of each data object are randomly distributed on different nodes, and are accessed by Client for 10 to 20 times. However, the access nodes of the same Client are similar to each other, which ensures that the access nodes are distributed in a certain range. The weight value of the system default disk load is 0.5.

The following figure shows the comparison of the simulation system after re-creating M data objects:

Figure 5. Load analysis diagram

As can be seen from the above figure, when the data object M is the same, the load of UILBPS is significantly lower than that of the HDFS default algorithm. This is precisely because, considering the reason of load balancing, the nodes with lower load are selected as far as possible to place replicas.

Figure 6. Access rate analysis chart

As can be seen from the above figure, when the data object M is the same, the average access distance of UILBPS is slightly less than the load under the HDFS default algorithm. This is precisely because considering the user access characteristics, its Client can access the nodes close to it as much as possible, thus saving network bandwidth, reducing the risk of network failure, and speeding up the rate of node access.

HDFS's rack-sensitive placement policy is random to its replica placement location, while UILBPS is based on user history information and load, fully considering the node load and the regional characteristics of user history access, to some extent, optimizing the rack-sensitive placement strategy, improving system performance and balancing system load, reducing user access latency and improving data availability.

Because the HDFS system will be made up of hundreds or thousands of servers storing fragments of file data. In fact, there are very large components in it, and every component is likely to fail, which means that there are always some components in HDFS that fail, that is, hardware failures are normal, not abnormal. So how to recover automatically and quickly is a very important goal of HDFS. On the other hand, the optimization of the replica placement policy reduces the system load and speeds up the access rate of the replica, so that the failed components can be recovered as soon as possible or play the same role as the original failed components.

HDFS has high fault tolerance and is designed to be deployed on low-cost hardware. At the same time, the user history information and load status are managed and calculated through the name node. Therefore, the additional information in the optimization will not occupy the disk space needed by HDFS to a large extent, nor will it cause the system performance degradation caused by the reduction of disk space.

For the adjustment of subsequent replicas caused by the failure of data nodes, the damage of replicas, the damage of disk of data nodes or the increase of replication factor of files, a better replica location is more conducive to the creation, deletion and migration of replicas, that is, it is more conducive to subsequent replica adjustments, ensuring the consistency, availability and fault tolerance of distributed systems.

The above is all the content of the article "what is the copy placement strategy in HDFS". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.