Shulou(Shulou.com)06/02 Report--

Look back

In the previous article in this series, we talked about PV,PVC,Storage Class and Provisioner

A brief review:

PV was originally designed as a storage block that needs to be pre-allocated by the administrator. With the introduction of Storage Class and Provisioner, users can supply PV dynamically.

PVC is a request for PV, and when used with Storage Class, it triggers a dynamic provisioning of the matching PV.

PV and PVC always correspond one to one.

Provisioner is a plug-in that provides PV to users. It frees administrators from the onerous role of creating workloads for persistence.

Storage Class is the classifier of PV. PV in the same Storage Class can share some properties. In most cases, Storage Class can be thought of as a Provisioner with predefined properties when used with Provisioner. Therefore, when a user requests it, it can provide PV dynamically with these predefined properties.

But these are just one of the ways to use persistent storage in Kubernetes

Volume

In the previous article, we mentioned that there is also a concept of Volume in Kubernetes. To distinguish a Volume from a persistent volume (Persistent Volume), people sometimes call it In-line Volume or Ephemeral Volume.

Here we quote the definition of Volume:

Kubernetes Volume... There is an explicit life cycle-- this is the same as the life cycle of the pod that contains it. As a result, the life cycle of Volume is longer than any container running in pod, and data is saved when the container is restarted. Of course, when Pod terminates, Volume also terminates. More importantly, Kubernetes supports multiple types of Volume, and any number of Volume can be used in a pod at the same time.

At its core, Volume is just a directory that may contain data that can be accessed by containers in pod. How these directories are created, the media that supports them, and their contents are all determined by the type of specific volume used.

An important attribute of Volume is that it has the same lifecycle as the pod to which it belongs. If pod disappears, so will it. This is different from Persistent Volume because Persistent Volume will continue to exist in the system until the user deletes it. Volume can also share data between containers in the same pod, but this is not a major use case because users typically use only one container per pod.

Therefore, it is more possible to think of Volume as a property of pod rather than as a separate object. As its definition says, Volume represents a directory in pod, while the type of Volume defines the contents of the directory. For example, the Config Map Volume type will create a configuration file from the API server in the Volume directory; the PVC Volume type will hang on the file system from the corresponding PV in the directory, and so on. In fact, Volume is almost the only way to use storage locally in pod.

It is easy to get confused between Volume, Persistent Volume, and persistent volume declaration (Persistent Volume Claim). Suppose you have a data stream that looks like PV- > PVC- > Volume. PV contains real data, binds to PVC, and eventually becomes Volume in pod.

However, in addition to PVC,Volume, there are various types of repositories that can be directly supported by Kubernetes, and in this sense, the definition of Volume is also confusing.

What we need to know is that we already have Persistent Volume, which supports different types of storage solutions. We also have Provisioner, which supports similar (not exactly the same) solutions. And we have different types of Volume.

So what's the difference between them? How to choose between them?

Multiple ways to persist data

Take AWS EBS as an example. Let's take a closer look at how data is persisted in Kubernetes.

Volume mode

AwsElasticBlockStore is a Volume type.

You can create a Pod, define a volume of type awsElasticBlockStore, set up the volumeID, and then use the EBS volume that exists in the pod.

The EBS volume must already exist before it can be used directly with Volume.

PV mode

AWSElasticBlockStore is also a PV type.

So you can create a PV, use it to represent EBS volume (assuming you have such permission), and then create a PVC volume bound to it. Finally, make PVC the volume, and then you can use it in pod.

Similar to the Volume method, EBS volume must exist before a PV can be created.

Provisioner mode

Kubernetes.io/aws-ebs is a built-in Provisioner for EBS in Kubernetes.

You can use Provisioner kubernetes.io/aws-ebs to create a Storage Class and Storage Class to create a PVC. Kubernetes will automatically create the corresponding PV for you. Next, specify PVC as volume and you can use it in pod.

In this use case, you do not need to create EBS,EBS Provisioner for you before using it.

Third-party mode

All the options listed above are Kubernetes built-in options, and if you're not satisfied, there are actually some third-party EBS implementations in Flexvolume driver format that can help you connect to Kubernetes.

If Flexvolume is not for you, you can also use CSI drivers with the same functionality (why? This will be described in more detail later)

VolumeClaimTemplate mode

If you are using StatefulSet, congratulations! You now have an extra way to use EBS in your workload-VolumeClaimTemple.

VolumeClaimTemple is an StatefulSet specification attribute that provides a way for Pod created by StatefulSet to create a match between PV and PVC. These PVC will be created through Storage Class so that they can be created automatically when StatefulSet is extended. When the StatefulSet shrinks, the excess PV/PVCs remains in the system. Therefore, when the StatefulSet extends again, they will once again act on the new pods created by Kubernetes. We will talk about StatefulSet in more detail later.

For example, suppose you use replica 3 to create a StatefulSet called www and use it to create a VolumeClaimTemplate called data. Kubernetes creates three pods, named www-0, www-1, and www-2. Kubernetes also creates a PVC, where www-data-0 is used for pod www-0,www-data-1 to www-1,www-data-2 to www-2. If you extend StatefulSet to 5M Kubernetes, you will create www-3, www-data-3, www-4, and www-data-4, respectively. If you then reduce StatefulSet to 1, www www-4, all will be deleted, while www-data-1 to www-data-4 will remain in the system. So when you decide to expand to 5 again, pod www-1 to www-4 will be created again, and PVC www-data-1 will still serve the Pod www-1,www-data-2 corresponding www-2, and so on. This is because the identity of pod in StatefulSet is stable. When using StatefulSet, both names and relationships are predictable.

VolumeClaimTemple is important for block storage solutions like EBS and Longhorn. Because these solutions are essentially ReadWriteOnce, you cannot share them between Pod. If you have more than one pod running persistent data, you won't be able to deploy smoothly. Therefore, the advent of VolumeClaimTemplate provides a way for block storage solutions to scale Kubernetes workloads horizontally.

How to choose between Volume, Persistent Volume and Provisioner

As you can see, there are built-in Volume types, PV types, Provisioner types, and external plug-ins that use Flexvolume and / or CSI. The big thing is that the functions they provide are basically the same, but there are slight differences.

I think there should be at least one criterion to determine how to choose between them.

But I didn't find it.

So I went through the code and documentation and drew the following comparison table, as well as the criteria that made the most sense to me, comparing Volume, Persistent Volume and Provisioner.

Here I only touch on what is supported by in-tree in Kubernetes, except for some official out-of-tree Provisioners:

Https://github.com/kubernetes-incubator/external-storage

As you can see, Volume, Persistent Volume, and Provisioner are different in some subtle ways.

1. Volume supports most volume plug-ins.

a. It is the only way to connect PVC and pod

b. It is also the only one that supports Config Map, Secret, Downward API, and Projected. All of these are closely related to the Kubernetes API server.

c. It is also the only one that supports EmptyDir, and EmptyDir can automatically assign and clean up temporary volume to pod. (note: as early as 2015, Clayton Coleman raised a question about supporting EmptyDir. This is useful for workloads that require persistent storage but only local volumes are available. However, this view has not received much attention. Without the support of scheduler, this goal was difficult to achieve at the time. Now, in 2018, Kubernetes v1.11 version of Local Volume has added node affinity support (node affinity support) for scheduler and PV, but there is still no EmptyDir PV. And the Local Volume feature is not what I expected, because it doesn't have the ability to create new volumes with new directories on the node. So I wrote Local Path Provisioner, which leverages scheduler and PV node affinity changes to provide dynamic Host Path type PV for workloads. )

2. The plug-in supported by PV is a superset supported by Provisioner, because Provisioner needs to create a PV before the workload uses it. However, there are some plug-ins that are supported by PV but not by Provisioner, such as Local Volume (which is being modified).

3. There are two other types of Volume that are not supported. They are the two latest features, CSI and Local Volume, and there is still some work in progress that will be used in Volume later.

Criteria for choosing between Volume, Persistent Volume, and Provisioner

So which way should users choose?

In my opinion, users should adhere to one principle:

If conditions permit, choose Provisioner instead of Persistent Volume, followed by Volume.

In detail:

1. For Config Map, Downward API, Secret, or Projected, use Volume because PV does not support them.

two。 For EmptyDir, use Volume directly, or use Host Path instead.

3. For Host Path, Volume is usually used directly because it is bound to a specific node and it is isomorphic between nodes.

a. If you want to use heterogeneous Host Path Volume, it can only be used after Kubernetes v1.11, because there is a lack of knowledge of node affinity with PV. With v1.11 +, you can use my Local Path Provisioner to create a Host Path PV with node affinity:

Https://github.com/rancher/local-path-provisioner .

4. For other cases, unless you need to hook to an existing volume (in which case you should use PV), use Provisioner instead. Some Provisioner are not built-in options, but you should be able to find them in this link (https://github.com/kubernetes-incubator/external-storage) or in the vendor's official repository.

The principle behind this principle is very simple. Objects (PV) are easier to manage than attributes (Volume) when operating within a Kubernetes, and it is much easier to create a PV automatically than to create a PV manually (Provisioner).

However, there is one exception: if you prefer to store outside the Kubernetes, it is best to use Volume, although using this approach requires another set of API to create / delete. In addition, due to the lack of VolumeClaimTemplate, you will lose the ability to use StatefulSet auto-scaling. I don't think this is the way most Kubernetes users will choose.

Why are there so many options for doing the same thing?

When I started working on Kubernetes storage, this was the first thing that came to mind. Due to the lack of consistency and intuition, Kubernetes storage looks like an afterthought. So I tried to study the historical reasons behind these design decisions, but I got nothing until 2016.

Finally, I tend to believe that these are caused by some early designs, which may be due to the urgent need to obtain supplier support, which leads to more responsibilities assigned to Volume than originally. In my opinion, all built-in volume plug-ins that replicate PV should not exist.

In the course of studying history, I found that dynamic provisioning had become an alpha feature in Kubernetes v1.2, which was released in early 2016. It requires two release cycles to become beta and stable in two cycles, which is very reasonable.

SIG Storage (which drives Kubernetes storage development) has also done a lot of work, using Provisioner and CSI to remove Volume plug-ins from tree. I think this is a big step towards a more consistent and streamlined system.

On the other hand, I don't think this bunch of Volume types will disappear. This seems to be at odds with Silicon Valley's unofficial motto: move quickly and break the rules. Sometimes it is too difficult to modify the designs left over from fast iterative projects. We can only live with them, work carefully around them, and don't call them in the wrong way.

Next step

In the next section of this series, we will discuss the mechanisms for extending Kubernetes storage systems, namely Flexvolume and CSI. A little hint: as you may have noticed, I'm not a fan of Flexvolume, and it's not a storage subsystem problem.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.