Shulou(Shulou.com)06/02 Report--

What is the performance comparison of several Kubernetes storage methods? I believe many inexperienced people are at a loss about this. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

Abstract

Shows a simple storage comparison that is tested and compared using storage volumes provided by a variety of storage without performance optimization.

Ignoring the native PVC or hostPath of Azure, we can get the following test results:

Portworx is the fastest container store on AKS.

Ceph is the fastest open source storage backend on private cloud clusters. The operation of the public cloud is too complex and these extra complexities do not provide better test performance.

The concept of OpenEBS is great, but its back end needs more optimization.

If you are running Kubernetes, you may be using or preparing to use dynamically supplied block volumes, and the first problem is to choose the appropriate storage technology for the cluster. You can't give a simple answer with a simple test to tell you what is the best technology on the market right now. The type of load running on the cluster is an important input in the selection of storage technology. For bare metal clusters, they need to be selected according to actual use cases and integrated into their own hardware. Managed K8s in the public cloud, such as AKS, EKS, or GKE, have block storage capabilities available out of the box, but this is not necessarily the best choice. There are many factors to consider, such as the long failover time of StorageClass in the public cloud. For example, in a failure test for AWS EBS, it took more than five minutes for the Pod with the volume loaded to start successfully on another node. Cloud native storage products such as Portworx or OpenEBS are trying to solve these problems.

The goal is to make a basic performance comparison using the most common Kubernetes storage scheme. I think the following backends are used on Azure AKS:

AKS native Storageclass:Azure native premium

Using OpenEBS at the back end of cStor

Portworx

Gluster managed by Heketi

Ceph managed by Rook

Now let's introduce each storage backend, explain the installation process, then test it in the AKS test environment, and finally get the results.

Storage

This section describes the storage scheme used in the test, including the installation process and the pros and cons of the scheme.

Azure native StorageClass

My motivation for choosing this option is to use it as the baseline for all tests. This scheme should provide the best performance. Azure dynamically creates managed disks and maps them to the virtual machine of K8s, eventually becoming the storage volume of Pod.

This scheme is so convenient that no extra steps are needed. After you create a new AKS cluster, two predefined StorageClass are automatically provided, default and managed-premium, and premium uses SSD-based high-performance, low-latency disks.

$kubectl get storageclassesNAME PROVISIONER AGEdefault (default) kubernetes.io/azure-disk 8mmanaged-premium kubernetes.io/azure-disk 8m $kubectl get pvcNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGEdbench-pv-claim Bound pvc-e7bd34a4-1dbd-11e9-8726-ae508476e8ad 1000Gi RWO managed-premium 10s$ kubectl Advantages of get poNAME READY STATUS RESTARTS AGEdbench-w7nqf 0/1ContainerCreating029s

AKS is available right out of the box.

Shortcoming

Failover is so slow that it sometimes takes ten minutes for the storage volume to be remounted to the Pod on different nodes.

OpenEBS

OpenEBS is a new thing to me, so I am very interested in doing his test. He proposes a new concept of Container Attached Storage (Container-mounted Storage), which is a microservice-based storage controller and multiple microservice-based storage replicas. Like Portworx, he is a member of the cloud native storage category.

It is a completely open source solution that currently offers two backends-Jiva and cStor. I chose Jiva at first, and then switched to cStor. CStor has many advantages, such as its controller and replica being deployed to a single OpenEBS namespace, the ability to manage raw disks, and so on. Each K8s volume has its own storage controller, which can expand storage within the permitted range of node storage capacity.

Run on AKS

It is very easy to install on AKS.

Connect to all K8s nodes and install iSCSI, because he needs to use iSCSI protocol to connect Pod and controller between K8s nodes.

Apt-get updateapt install-y open-iscsi

Complete the deployment on the K8s cluster using a YAML definition:

Kubectl apply-f https://openebs.github.io/charts/openebs-operator-0.8.0.yaml

Next, the OpenEBS controller discovers all the disks in the node. But I had to manually identify my attached AWS managed disk.

$kubectl get diskNAME AGEdisk-184d99015253054c48c4aa3f17d137b15mdisk-2f6bced7ba9b2be230ca5138fd0b07f15mdisk-806d3e77dd2e38f188fdaf9c46020bdc5m

Then add these disks to StoragePoolClaim, and this object will be referenced in StorageClass:

-apiVersion: storage.k8s.io/v1kind:StorageClassmetadata: name: openebs-custom annotations: openebs.io/cas-type: cstor cas.openebs.io/config????-name:StoragePoolClaim value: "cstor-disk" provisioner: openebs.io/provisioner-iscsi---apiVersion: openebs.io/v1alpha1kind:StoragePoolClaimmetadata: name: cstor-diskspec: name: cstor-disk type: disk maxPools:3 poolSpec: poolType: striped disks: diskList:- Disk-2f6bced7ba9b2be230ca5138fd0b07f1-disk-806d3e77dd2e38f188fdaf9c46020bdc-disk-184d99015253054c48c4aa3f17d137b1

After completing these steps, you can use the PVC of K8s to create storage volumes dynamically.

Advantages

Open source

Maya does a great job of visualizing the use of resources. Multiple services can be deployed in the middle of K8s to conveniently set up monitoring and logging for all aspects of the cluster data. This is very important for troubleshooting work.

CAS concept: I appreciate this concept very much, and I believe it is the trend of the future.

OpenEBS community: any of my problems in the community can be solved in a few minutes. The team on Slack is very helpful.

Shortcoming

Immature: OpenEBS is still very young and has not released a stable version yet. The core team is still working on back-end optimization, and performance will be greatly improved in the coming months.

The iSCSI connection between Kubelet and the storage controller is made through K8s Service, which can cause problems in CNI plug-in environments such as Tungsten Fabric.

Additional software (iSCSI) needs to be installed on the K8s node, which is very inconvenient for hosting clusters.

Note: the OpenEBS team adjusted my case scenario:

Https://github.com/kmova/openebs/tree/fio-perf-tests/k8s/demo/dbench

Portworx

Portworx is another Kubernetes-oriented container native storage scheme that focuses on highly distributed environments. This is host addressable storage, and each volume is mapped directly to the host on which it is mounted. It provides automatic fine-tuning capabilities based on the application of the Ihammer O type. The official website provides more information. Unfortunately, it is also the only non-open source product in this article. However, it offers a free trial of 3 nodes.

Run on AKS

The installation on AKS was just as easy, and I used the generator provided on their website.

Azure0 $kubectl get pods-o wide-n kube-system-l name=portworxNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODEportworx-g9csq 1/1Running014m10.0.1.66 aks-agentpool-20273348-2portworx-nt2lq 1/1Running014m10.0.1.4 aks-agentpool-20273348-0portworx-wcjnx 1/1Running014m10.0.1.35 aks-agentpool-20273348-1

Create a StorageClass for PVC, define a high priority, and three copies:

Root@aks-agentpool-20273348-0 yaml portworx-scapiVersion: storage.k8s.io/v1kind:StorageClassmetadata: creationTimestamp:2019-01-28T21:10:28Z name: portworx-sc resourceVersion: "55332" selfLink:/apis/storage.k8s.io/v1/storageclasses/portworx-sc uid:23455e40-2341-11e9-bfcb-a23b1ec87092parameters: priority_io: high repl: "3" provisioner: advantages of kubernetes.io/portworx-volumereclaimPolicy:DeletevolumeBindingMode:Immediate

Easy to deploy: the generator contains configuration details.

There is no need for additional configuration like Ceph and Glusterfs.

Cloud native storage: both public clouds and bare metal can run.

Storage-level-aware and application-aware Imax O fine-tuning.

Shortcoming

Closed source: commercial solution

GlusterFS Heketi

GlusterFS is a well-known open source storage solution, which is provided by Redhat. Heketi is GlusterFS's RESTful volume management interface. It provides an easy-to-use way to provide dynamic provisioning for GlusterFS volumes. Without the assistance of Heketi, you can only manually create GlusterFS volumes and map to K8s PV. For more information about GlusterFS, please read the official documentation.

Run on AKS

Deploy according to Heketi's Quick start documentation.

Referring to the sample, create a topology file that contains the disk and hostname.

The main development and testing of Heketi is on the RHEL-based operating system. When I used the Ubuntu host on AKS, I had a problem with the wrong kernel module path. I submitted a PR to fix this problem.

~ ~ + + b/deploy/kube-templates/glusterfs-daemonset.yaml @ @-67 b/deploy/kube-templates/glusterfs-daemonset.yaml 7 + 67 b/deploy/kube-templates/glusterfs-daemonset.yaml 7 @ @ spec: mountPath: "/ etc/ssl" readOnly: true-name: kernel-modules

Name: kernel-modules hostPath:

MountPath: "/ usr/lib/modules"

MountPath: "/ lib/modules" readOnly: true securityContext: capabilities: {} @ @-131 spec 4 + 131 spec: path: "/ etc/ssl"

Path: "/ usr/lib/modules"

Path: "/ lib/modules" ~ ~

Another problem I encountered in the AKS environment was a non-empty disk, so I used wipefs to clean up the glusterfs. This disk has not been used.

$wipefs-a / dev/sdc / dev/sdc: 8 bytes were erased at offset 0x00000218 (LVM2_member): 4c 56 4d 32 20 30 30 31

Finally, run gk-deploy-g-t topology.json, which runs GlusterFS Pod under the management of the Heketi controller on each node.

$kubectl get po-o wideNAME READY STATUS RESTARTS IP NODE NOMINATED NODEglusterfs-fgc8f 1/1Running010.0.1.35 aks-agentpool-20273348-1glusterfs-g8ht6 1/1Running010.0.1.4 aks-agentpool-20273348-0glusterfs-wpzzp 1/1Running010.0.1.66 aks-agentpool-20273348-2heketi-86f98754c-n8qfb 1/1Running010.0.1.69 aks-agentpool-20273348-2

And then I had a new problem. The restURL of Heketi cannot be used on the K8s control plane. I tested the kube dns record, and neither pod IP nor svc IP worked. In the end, you can only use Heketi CLI manually to create storage volumes.

$export HEKETI_CLI_SERVER= http://10.0.1.69:8080$ heketi-cli volume create-- size=10--persistent-volume-- persistent-volume-endpoint=heketi-storage-endpoints | kubectl create-f-persistentvolume/glusterfs-efb3b155 created$ kubectl get pvNAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGEglusterfs-efb3b155 10Gi RWX RetainAvailable

Then map the existing PV to PVC and load it to the test tool for testing.

Kind:PersistentVolumeClaimapiVersion: v1metadata: name: glusterfs-efb3b155spec: accessModes:-ReadWriteMany storageClassName: "" resources: requests: storage:10Gi volumeName: glusterfs-efb3b155 $kubectl get pvcNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGEglusterfs-efb3b155 Bound glusterfs-efb3b155 10Gi RWX 36m

More output from Heketi:

$gluster volume info vol_efb3b15529aa9aba889d7900f0ce9849VolumeName: vol_efb3b15529aa9aba889d7900f0ce9849Type:ReplicateVolume ID:96fde36b-e389-4dbe-887b-baae32789436Status:StartedSnapshotCount:0Number of Bricks:1 x 3=3Transport-type: tcpBricks:Brick1:10.0.1.66:/var/lib/heketi/mounts/vg_5413895eade683e1ca035760c1e0ffd0/brick_cd7c419bc4f4ff38bbc100c6d7b93605/brickBrick2:10.0.1.35:/var/lib/heketi/mounts/vg_3277c6764dbce56b5a01426088901f6d/brick_6cbd74e9bed4758110c67cfe4d4edb53/brickBrick3:10.0.1.4:/var/lib/heketi/mounts/vg_29d6152eeafc57a707bef56f091afe44/brick_4856d63b721d794e7a4cbb4a6f048d96/ BrickOptionsReconfigured:transport.address-family: inetnfs.disable: onperformance.client-io-threads: off$ kubectl get svcNAME TYPE CLUSTER-IP EXTERNAL-IP PORT (S) AGEheketi ClusterIP192.168.101.758080/TCP 5hheketi-storage-endpoints ClusterIP192.168.103.661/TCP 5h $kubectl get endpointsNAME ENDPOINTS AGEheketi 10.0.1.69:80805hheketi-storage-endpoints 10.0.1.35 yamlapiVersion 1Magne 10.0.1.4 AGEheketi 10.0.1.69:80805hheketi-storage-endpoints 10.0.1.66 kubectl get endpoints heketi-storage-endpoints-o yamlapiVersion: v1kind:Endpointsmetadata: creationTimestamp:2019-01-29T15:14:28Z name: heketi-storage-endpoints namespace:default resourceVersion: "142212" selfLink:/api/v1/namespaces/default/endpoints/heketi-storage-endpoints uid:91f802eb-23d8-11e9murbbcbMura23b1ec87092 subsets ip:10.0.1.66 ports-addresses:-ip:10.0.1.35-ip:10.0.1.4-ip:10.0.1.66 ports:-port:1 protocol: advantages of TCP

A tried and tested storage solution.

Lighter than Ceph.

Shortcoming

Heketi is not performing well on the public cloud. If you perform well on a private cloud, it will be easier to install.

Not designed for structured data, such as SQL databases. However, you can use GlusterFS to provide backup and recovery support for databases.

Ceph Rook

I have tried to install and run Ceph on the OpenStack private cloud. It needs to customize parameters for specific hardware and design pg groups, SSD partitions and CRUSH diagrams according to data types. So when I first heard that Ceph was running on a 3-node K8s cluster, I didn't really believe it would work. As a result, I was impressed by Rook's orchestration tool, which combines all the steps with the orchestration capabilities of K8s to make installation very easy.

Run on AKS

The default installation of Rook does not require any specific steps, and it will be very simple if there is no advanced configuration.

I am using the Ceph Quick start Guide

Configure FLEXVOLUME_DIR_PATH for AKS because it requires / etc/kubernetes/volumeplugins/, not the default / usr/libexec in Ubuntu, without which Kubelet cannot load PVC.

~ ~ diff-git a/cluster/examples/kubernetes/ceph/operator.yaml b/cluster/examples/kubernetes/ceph/operator.yaml index 73cde2e..33f45c8 100755-a/cluster/examples/kubernetes/ceph/operator.yaml + + b/cluster/examples/kubernetes/ceph/operator.yaml @ @-431 Set the path where the Rook agent can find the flex volumes 8 + 431 spec: #-name: AGENT_MOUNT_SECURITY_MODE # value: "Any" # Set the path where the Rook agent can find the flex volumes

#-name: FLEXVOLUME_DIR_PATH

# value: "

"

-name: FLEXVOLUME_DIR_PATH

Value: "/ etc/kubernetes/volumeplugins" # Set the path where kernel modules can be found #-name: LIB_MODULES_DIR_PATH # value:

"~

Also specify the device to be used in deviceFilter, in this case / dev/sdc.

~ ~ diff-git a/cluster/examples/kubernetes/ceph/cluster.yaml b/cluster/examples/kubernetes/ceph/cluster.yaml index 48cfeeb..0c91c48 100755-a/cluster/examples/kubernetes/ceph/cluster.yaml + + b/cluster/examples/kubernetes/ceph/cluster.yaml @ @-227 storage 7 @ @ spec: storage: # cluster level storage configuration and selection useAllNodes: true useAllDevices: false

DeviceFilter:

DeviceFilter: "^ sdc" location: config: ~ ~

After installation, create a Ceph block pool, as well as StorageClass, using the following configuration.

ApiVersion: ceph.rook.io/v1kind:CephBlockPoolmetadata: name: replicapool namespace: rook-cephspec: failureDomain: host replicated: size:3---apiVersion: storage.k8s.io/v1kind:StorageClassmetadata: name: rook-ceph-blockprovisioner: ceph.rook.io/blockparameters: blockPool: replicapool clusterNamespace: rook-ceph fstype: xfsreclaimPolicy:Retain

Finally, use the deployment tool to check.

Ceph status cluster: id: bee70a10-dce1-4725-9285-b9ec5d0c3a5e health: HEALTH_OK services: mon:3 daemons, quorum c usage:3.0GiB used a mgr: a (active) osd:3 osds:3 up,3in data: pools:0 pools,0 pgs objects:0 objects,0 B usage:3.0GiB used 3.0TiB/3.0TiB avail pgs: [root@aks-agentpool-27654233-0 /] # [root@aks-agentpool-27654233-0 /] # [root@aks-agentpool-27654233-0 /] # ceph osd status+----+--+ -+ | id | host | used | avail | wr ops | wr data | rd ops | rd data | state | + -+ | 0 | aks-agentpool-27654233-0 | 1025m | 1021G | 0 | 0 | 0 | exists Up | | 1 | aks-agentpool-27654233-1 | 1025m | 1021G | 0 | 0 | 0 | exists,up | | 2 | aks-agentpool-27654233-2 | 1025m | 1021G | 0 | 0 | 0 | exists,up | +-+-- advantage

A robust storage system in a large production environment.

Rook simplifies lifecycle management very well.

Shortcoming

Complex: more heavyweight and less suitable for running on public clouds. Running on a private cloud may be more appropriate.

AKS test environment

I created a basic Azure AKS cluster with three virtual machines. In order to connect to the Premium SSD, I can only use virtual machines above type E. So I chose Standard_E2s_v3, which is equipped with 2 vCPU and 16GB memory.

In the resource foot where the AKS cluster is located, you can see all the virtual machines, network interfaces and other resources. Create three Premium SSD stores for 1TB here and manually mount them to each virtual machine.

So I have an empty disk for 1TB on each instance. On the Azure page, depending on the virtual machine and disk size we selected, the performance should be 5000 IOPS and 200MB/s throughput. The last section will show our real results.

Performance result

Note: the results of each storage can not be used as independent evaluation results, but the comparison can be used for reference. There are many ways to compare tests, and this is the simplest one.

To run the tests, I decided to use the off-the-shelf test tool Dbench, which is a k8s YAML file that uses FIO to run eight test cases. You can specify different tests in Dockerfile:

Random read and write bandwidth.

Read and write IOPS randomly.

Read and write delay.

Read and write sequentially.

Mixed read and write IOPS.

The results of all tests can be found on Github.

Random read and write bandwidth

Random read and write tests show that the read performance of GlusterFS, Ceph and Portworx is several times faster than that of AWS's local hostPath. Read caching is the culprit. GlusterFS and Portworx write faster and are as efficient as local disks.

Random read and write IOPS

In the random IOPS test, Portworx and Ceph performed best. Portworx gets an IOPS close to Azure's native PVC in terms of writing.

Read and write delay

The results of the latency test are interesting, and Azure native PVC is worse than most other storage. Portworx and Ceph performed best. GlusterFS is better than Ceph in terms of writing. The latency of OpenEBS is relatively high.

Sequential reading and writing

The result of sequential read and write is similar to that of the previous random test, but Cpeh is more than twice as fast as GlusterFS in terms of reading. The write results are basically the same, except for the poor performance of OpenEBS.

Mixed read and write IOPS

The last test case examines that both IOPS,Portworx and Ceph give better results than Azure native PVC in the case of mixed read and write.

This article shows a simple storage comparison that is tested and compared using storage volumes provided by a variety of storage without performance optimization. It is recommended to pay attention to the methods described in this article, and it is not recommended to use the results directly to judge.

Ignoring the native PVC or hostPath of Azure, we can get the following test results:

Portworx is the fastest container store on AKS.

Ceph is the fastest open source storage backend on private cloud clusters. The operation of the public cloud is too complex and these extra complexities do not provide better test performance.

The concept of OpenEBS is great, but its back end needs more optimization.

It should be interesting to adjust the test size of the performance data. Another noteworthy comparison is CPU and memory consumption. I will continue to follow and share more.

After reading the above, have you mastered the method of comparing the performance of several Kubernetes storage methods? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.