Shulou(Shulou.com)05/31 Report--

This article is about how to implement Ceph erasure codes. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

1. Erasure codes

Erasure code Erasure Code is a kind of coding technology, which can add m pieces of original data to n pieces of original data, and can restore any n pieces of data into original data. That is, if any data less than or equal to m is invalid, it can still be restored through the rest of the data. There are three main applications of erasure codes in distributed storage systems: array erasure codes (Array Code: RAID5, RAID6, etc.), RS (Reed-Solomon) Reed-Solomon erasure codes and LDPC (LowDensity Parity Check Code) low-density parity erasure codes. At present, LDPC code is mainly used in communication, video and audio coding and other fields.

At present, the mainstream distributed storage including HDFS, Ceph, Minio and so on mainly uses RS (Reed-Solomon) code.

2. Ceph erasure code 2.1.Writing of erasure correction code.

The data will be encoded in the main OSD and then distributed to the appropriate OSDs.

2.2, erasure code read 2.2.1, normal erasure code read

The data is obtained from the corresponding OSDs and decoded.

2.2.2, erasure code reading in case of data loss

If any data is lost at this time, Ceph will automatically read the data from the OSD where the check code is stored and decode it.

3. Plug-ins for erasure codes supported by Ceph

Ceph supports different erasure correction code plug-ins: Jerasure,ISA-l and LRC. The default is Jerasure.

# ceph osd erasure-code-profile get defaultk=2m=1plugin=jerasuretechnique=reed_sol_van

The three erasure code plug-ins listed here are actually divided into two types.

Jerasure and ISA-l belong to the erasure correction code base LRC is a kind of erasure correction code coding, similar to RS code

3.1.1Code Base for erasure Correction, Jerasure

Jerasure2.0 uses instructions encapsulated in C language, which makes the code more friendly. In addition, Jerasure2.0 not only supports the calculation of GF (2 ^ 8) finite fields, but also can carry out finite fields between GF (2 ^ 4) and GF (2 ^ 128). Besides RS code, it also provides support for other coding methods such as Cauchy Reed-Solomon code (CRS code). In addition to its industrial application, its academic value is also very high. At present, it is one of the most widely used coding libraries. Currently, Jerasure2.0 does not support AVX acceleration.

3.1.2 、 ISA-l

Intel ISA-L stands for Intel Smart Storage Accelerator Library, using assembly code, and ISA-L is an open source collection of low-level features optimized for storage applications. It includes fast block Reed-Solomon type erasure codes optimized for Intel AVX and AVX2 instruction sets, and its performance is the best in the industry, mainly because the overall matrix operations are placed in assembly.

3.1.3, comparison

3.2. erasure correction code 3.2.1, RS (Reed-Solomon)

Reed-Solomon (RS) code is a kind of erasure correction code commonly used in storage systems. It has two parameters n and m, which are marked as RS (NMagne m). N represents the number of original data blocks. M represents the number of check blocks.

To put it simply, when you specify RS (4p. 1), there are 4 data blocks and 1 check block

D1-4 any data block damaged can be decoded and reconstructed by P1, and the same P1 check block damage can also be calculated and reconstructed by data block coding.

3.2.2, LRC (Locally Repairable Codes)

LRC (Locally Repairable Codes) is a local parity coding method. Its core idea is that the parity block is divided into global parity block (global parity) and local parity block (local reconstruction parity).

Compared to RS, here are four blocks of data divided into two groups

D1+D2=P1

D3+D4=P2

D1+D2+D3+D4=P3

P1 and P2 are local check blocks and P3 is global check blocks.

The advantage is that when the data block in the group is damaged, the data amount and reconstruction time of the recovery operation in the group can be basically reduced by half due to the reduction of the amount of data transmitted by the network Iripple O.

The disadvantage is that the disk utilization and reliability will decrease with the increase of check block.

3.2.2, coding comparison

Thank you for reading! This is the end of this article on "how to implement Ceph erasure codes". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.