Shulou(Shulou.com)06/01 Report--

This article introduces how to use distributed storage VSAN to achieve high availability of Harbor Registry, the content is very detailed, interested friends can refer to, hope to be helpful to you.

A High availability Scheme for Harbor Registry with distributed Storage VSAN

Not long ago, VMware released the Docker Volume Driver for vSphere 1.0 beta version of Docker container data volumes, which enables Docker hosts to create volumes directly in vSphere data stores (VSAN,VMFS,NFS, etc.) and mount them directly to Docker containers, which can solve the problem of persistent data storage in Docker containers. In addition to providing storage, these volumes can also take advantage of vSphere's Storage Policy-based Management (SPBM, Storage Policy Based Management), set higher tolerable host failures (FTT), set larger stripes (SW), and so on, as needed, to achieve higher levels of data protection and better performance.

Detailed explanation of the principle

The architecture described in this article, as shown in the following figure, consists of a VSAN distributed storage cluster consisting of three ESXi nodes and a virtual machine running by Harbor Registry. In addition, three Docker external volumes are created in Virtual SAN for persistent storage of data in Harbor. The cluster is pooled by the local disks of each node, which can withstand the failure of one node without affecting availability. The detailed configuration steps are as follows:

1. First, a Virtual SAN cluster with three hosts is built, and a Photon OS virtual machine is installed on one of the hosts as the host for running Docker. Of course, readers can also install other Linux versions of virtual machines such as Ubuntu without Photon OS, as long as they can run services such as Docker Engine and Docker-Compose.

two。 On the release page (https://github.com/vmware/docker-volume-vsphere/releases) of the "Docker Volume Driver for vSphere" project, download the plug-ins installed on the ESXi host and the virtual machine, for example, for the 1.0Beta version, the file names are:

Vmware-esx-vmdkops-1.0.beta.zip

Docker-volume-vsphere-1.0.beta-1.x86_64.rpm

3. On the ESXi host, install the plug-in with the following command, and no reboot is required after the installation is complete.

# esxcli software vib install-d "/ vmware-esx-vmdkops-1.0.beta.zip" >

4. On the Photon OS virtual machine, install the RPM package. For Debian-based systems, install the corresponding deb package.

# rpm-ivh docker-volume-vsphere-1.0.beta-1.x86_64.rpm

5. After the plug-in on the ESXi host is installed, an administrative script, located at / usr/lib/vmware/vmdkops/bin/vmdkops_admin.py, will be installed on the host, which can help the vSphere administrator manage the Docker external volumes created. For example, you can create different storage policies. In Virtual SAN, the default storage policy stripe number is 1 (that is, SW=1). As an example, we create a policy with a stripe number of 2. SSH to any ESXi host in the cluster and run the following command:

# / usr/lib/vmware/vmdkops/bin/vmdkops_admin.py policy create-name SW=2-content'(("stripeWidth" i2))'

Where FTT=0 is the name of this policy, the key point is to set the content of the policy, in this case'(("stripeWidth" i2))'. Other parameters that can be set are the same as the standard parameters of Virtual SAN, and their names and descriptions are as follows:

6. At this point, you can create Docker volumes with the command on the Photon virtual machine. Let's first create two volumes that use the default storage policy, and then create a volume that uses SW=2 as an example.

# docker volume create-- driver=vmdk-- name=vsanvol1-o size=50gb

Vsanvol1

# docker volume create-- driver=vmdk-- name=vsanvol2-o size=20gb

Vsanvol2

# docker volume create-driver=vmdk-name=vsanvol3-o size=20gb-o vsan-policy-name=SW=2

Vsanvol3

A volume can be created in the vSphere data store by specifying the-- driver=vmdk parameter. The created volume storage location is the same as the location of the Photon virtual machine. Because the Photon OS virtual machine is placed on Virtual SAN storage, the volumes created are also placed on Virtual SAN storage. This volume exists in the form of VMDK, and it is worth noting that since this VMDK is not mounted to any virtual machine at this time, no information about this volume can be found through the virtual machine page when browsing the vSphere client.

However, these VMDK can be seen under the dockvols directory of vsanDatastore:

As we can see below, when this volume is mounted to a running container, the corresponding VM can be found in the vSphere client through the associated VMDK.

7. Download the Harbor source code and before installation, you need to modify the harbor/Deploy/docker-compose.yml file in the Harbor configuration in order to use our newly created volume. After that, refer to the Harbor installation documentation for installation.

Open the docker-compose.yml file, locate the 'registry' section, and configure:

Volumes:

-/ data/registry:/storage

-. / config/registry/:/etc/registry/

Modified to:

Volumes:

-vsanvol1:/storage

-. / config/registry/:/etc/registry/

Where 'vsanvol1' is the external volume we just created.

In addition, find the 'mysql' section, and similarly configure the following:

Volumes:

-/ data/database:/var/lib/mysql

Modified to:

Volumes:

-vsanvol2:/var/lib/mysql

Again, 'vsanvol2' is another external volume we just created.

In addition, find the 'jobservice' section, and similarly configure the following:

Volumes:

-/ data/job_logs:/var/log/jobs

-. / config/jobservice/app.conf:/etc/jobservice/app.conf

Modified to:

Volumes:

-vsanvol3:/var/log/jobs

-. / config/jobservice/app.conf:/etc/jobservice/app.conf

Again, 'vsanvol3' is another external volume we just created.

At the end of the file, add the following configuration:

Volumes:

Vsanvol1:

External: true

Vsanvol2:

External: true

Vsanvol3:

External: true

To indicate that the three volumes have been successfully created externally (external) and do not need to be created separately. Other configurations remain the same. Then, follow the installation tutorial for Harbor to install.

8. After Harbor starts, look at the vSphere client and find that these three external volumes have been mounted to the Photon virtual machine, as' Hard Disk 2 'and' Hard Disk 4'. In this version, there seems to be some bug, for example, the storage policy of these three VMDK is shown as' None', but you can see that the components of Hard Disk3 are created in the form of SW=2 in Virtual SAN, and the other two volumes are created using the default storage policy, such as Hard Disk 4:

This should be that Virtual SAN still has some problems in identifying the policy created by "Docker Volume Driver for vSphere", which I hope will be resolved by subsequent versions.

9. Upload two test image to test whether the data will be lost.

10. Test HA: first, enable vSphere HA on the cluster and choose the default configuration for all configurations. Then confirm that the Photon virtual machine is currently on the host 10.162.102.130.

11. Power off this host, wait a while for HA to restart the virtual machine, and check the status of the Photon virtual machine.

As you can see, it has been rebooted on another healthy host.

The original three external volumes have also been mounted to the rebooted virtual machine. Because we powered off a host, there was a missing component (Absent Component) for each VMDK, but VSAN's default storage policy tolerated the failure of a host, so the data could still be accessed properly.

twelve。 After the virtual machine is restarted, check the status of Harbor and show that all containers are running automatically.

Looking at the Harbor management interface, both test Image are normal, indicating that no data loss has occurred.

When vSphere HA restarts the Harbor virtual machine on another host, all containers in Harbor restart, but the attached external volumes remain the same, as shown in the figure:

About how to use distributed storage VSAN to achieve Harbor Registry high availability scheme is shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.