Shulou(Shulou.com)06/01 Report--

This article is about how to use Intel's arrogant technology and CPU to improve the performance of Ceph. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Ceph is the most common back-end solution for block and object storage, and now the new version also adds file support. As an open source distributed storage software solution, because of its rich features and active community, it is widely used in public and private cloud environments.

However, due to the large-scale distributed architecture of Ceph, its IO path is too long, so its performance has been criticized by the industry. In reality, Ceph is more used in scenarios with low performance requirements, such as backup and archiving. In this scenario, the general storage medium uses HDD, and the configuration of all-flash memory is rarely used.

With the continuous decline of solid state disk (SSD) prices, cloud providers have begun to build all-flash storage with excellent performance and high reliability for their customers. To this end, they are eager to obtain a Ceph-based all-flash reference architecture and understand specific performance and best practices.

Intel ®arrogant ™technology combines unprecedented advantages of high throughput, low latency, high quality of service and high durability. It is a combination of 3D XPoint ™memory media and Intel ®software. These building modules cooperate with strong scalable processors to achieve specific improvements in reducing latency and accelerating system performance, which can fully meet the workload needs for large capacity and fast storage.

This article will introduce the progress made by Ceph around the reference architecture of Intel all-flash storage system and software optimization based on Intel ®strong ®scalable processors. In this article, we will focus on the Ceph reference architecture and performance results. The configuration of this architecture includes the RADOS Block device (RBD) interface, Intel ®arrogant ™technology, and Intel ®Xeon ®scalable processor product family (Intel ®Xeon ®Platinum 8180 processor and Intel ®Xeon ®Gold 6140 processor).

We first introduce the Ceph all-flash array (AFA) reference architecture using Intel ®arrogant ™technology and Intel ®powerful ®scalable processors, and then introduce the performance results and system characteristics of typical workloads.

Intel recommends that Ceph AFA use Intel ®Xeon ®Platinum 8180 processor, which is the advanced processor in Intel ®Xeon ®scalable processor family. It is recommended to use Intel ®arrogant ™solid state disk (SSD) as BlueStore WAL (Write-Ahead Logging) device, NAND-based solid state disk as data hard drive, and Mellanox 40 GbE network interface card (NIC) as high-speed Ethernet data port for the highest performance (throughput and latency). It is the best choice for Icano-intensive workloads.

The test system consists of five Ceph storage servers and five client nodes. Each storage node is configured with an Intel Xeon Platinum 8180 processor and 384 GB memory, using 1x Intel Optane SSD DC P4800X 375GB as a BlueStore WAL device, 4x Intel ®SSD DC P3520 2TB as a data drive, and 2x Mellanox 40 GbE NIC as a separate cluster and public network for Ceph.

At the same time, each node uses Ceph 12.2.2, and each Intel ®solid state disk DC P3520 series runs an object storage daemon (OSD). The RBD pool configuration for testing has 2 copies.

For clients, each node is configured with an Intel ®Xeon ®platinum 8180 processor, 384 GB memory, and 1 Mellanox 40GbE NIC.

Intel designed four different workloads to simulate typical all-flash Ceph clusters in the cloud (based on fio with librbd), including 4K random reads and 64K sequential reads and writes to simulate random and sequential workloads, respectively. For each test case, IO performance (IOPS or bandwidth) is measured by the number of volume extensions (up to 100), with each volume configured for 30 GB. These volumes have been pre-allocated to eliminate the impact of the Ceph thin configuration mechanism and achieve stable and replicable results. Stop OSD page caching before each test to eliminate the impact of page caching. In each test case, fio is configured with a preparation time of 300 seconds and a data collection time of 300 seconds.

4K random write characteristic

The CPU utilization of user space consumption is 37%, accounting for 75% of the total CPU utilization. The analysis results show that the Ceph OSD process consumes most of the CPU cycles; the suspicious reason why there is still room for CPU is that software threads and locking model implementations limit the scalability of Ceph on a single node, which is still the next step of optimization.

4K random write system index

4K random read characteristic

The CPU utilization is about 60%, of which IOWAIT accounts for about 15%, so the actual CPU consumption is about 45%, which is similar to random writing. The read IOPS of OSD disk is very stable at 80K ~ 40GBbE NIC with a bandwidth of about 2.1GB/s. No obvious hardware bottlenecks were observed; suspected software bottlenecks are similar to 4K random write cases and need further investigation.

4K random read system index

64K sequential write characteristic

CPU utilization and memory consumption for sequential writes are very low. Because the number of OSD replicates is 2, the transmission bandwidth of NIC data is twice the receive bandwidth, which includes the bandwidth of two NIC, one for the public network and one for the cluster network, approximately 1.8 GB per NIC per port. OSD disk AWAIT time is seriously fluctuated, with a maximum disk latency of more than 4 seconds, while disk IOPS is very stable.

System metrics for 64k sequential writes

64K sequential read characteristic

For the sequential read case, we observed that a NIC has a bandwidth of 4.4 GB/s, accounting for about 88% of the total bandwidth. CPU utilization and memory consumption for sequential writes are very low. OSD disk read IOPS and latency stable.

The system index of 64K sequential reading

Overall, Ceph AFA clusters based on Intel Optane technology demonstrate excellent throughput and latency. 64k sequential read and write throughput is 21949 MB/s and 8714 MB/s (maximum 40 GbE NIC), respectively. The 4K random read throughput is 2453K IOPS with an average latency of 5.36ms, while the 4K random write throughput is 500K IOPS with an average latency of 12.79ms.

In fact, Intel has been working closely with communities, ecosystems and partners to optimize the performance of Ceph Giant since its launch. The following figure shows the performance tuning history of the major versions of Ceph and the 4K random write workload on different Intel platforms. With the new major version of Ceph, back-end storage, combined with core platform changes and SSD upgrades, 4K random write performance of a single node has been improved by 27 times (3673 input / output operations per second (IOPS) to 100052 IOPS per node)! This makes it possible to build high-performance storage solutions using Ceph.

In this article, we see the performance results of Intel Optane technology using Ceph AFA reference architecture on Intel Xeon scalable processors. This configuration shows excellent throughput and latency. In addition to the latency gap compared with traditional high-end storage, bandwidth and IOPS have reached the level of high-end storage.

For read-intensive workloads, especially small block reads, there is a high demand for CPU performance. It is recommended to use the top processors of the Intel Xeon scalable processor family, such as the Intel Xeon platinum 8000 series processors. Software tuning and optimization also provides up to 19% performance improvements for reads and writes compared to Intel Optane technology, which uses the default configuration of Ceph AFA clusters on Intel Xeon scalable processors. Since hardware performance can be observed with current hardware configuration, performance is expected to improve continuously in the near future through continuous Ceph optimization (such as RDMA messenger,NVMe-focus object storage, async-osd, etc.).

It is believed that with the support of Intel Xeon scalable processor and arrogant technology, coupled with the continuous optimization of Ceph, the performance of Ceph will be better in the future, and Ceph will be more and more used in the main storage scenario, not just the second storage scenario that is currently focused.

Thank you for reading! This is the end of this article on "how to use Intel's arrogant technology and CPU to improve Ceph performance". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it out for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.