Shulou(Shulou.com)06/01 Report--

Today, I will talk to you about how to build Harbor multi-instance high-availability shared storage. Many people may not know much about it. In order to make you understand better, the editor has summarized the following for you. I hope you can get something according to this article.

Multi-instance shared storage architecture diagram

The LB of this article does not use Nginx, but Ali SLB.

There are three issues to be considered in the architecture of this article.

1. Selection of shared storage. Harbor backend storage currently supports AWS S3, Openstack Swift, Ceph and so on. This article uses Aliyun's fast NAS, and the disk IO performance is better than the read and write performance of a single disk. Mount using NFS V3 version.

2. Session cannot be shared on different instances, so Harbor Redis needs to be deployed separately, and multiple instances are connected to the same Redis.

3. Harbor multi-instance database problem, you must deploy a separate database, and multiple instances connect to the same database.

Note: if Aliyun NAS is used in production environment, extreme NAS is recommended, but universal NAS is not recommended.

Aliyun NAS performance reference document https://help.aliyun.com/document_detail/124577.html?spm=a2c4g.11186623.6.552.2eb05ea0HJUgUB

Deploy resources

Deployment

Harbor chooses to deploy online and uses docker-compose deployment. Docker-compose and Docker deployment environments are not described in this article. Relevant documents can be found online.

1. Mount Aliyun extreme NAS

Both harbor1 and harbor2 machines need to mount NAS

Configure auto mount on boot, open / etc/fstab configuration file, and add mount command.

# create NAS mount directory $mkdir / data # increase the number of simultaneous NFS requests $sudo echo "options sunrpc tcp_slot_table_entries=128" > > / etc/modprobe.d/sunrpc.conf $sudo echo "options sunrpc tcp_max_slot_table_entries=128" > > / etc/modprobe.d/sunrpc.conf

Mount the NFS v4 file system and add the following command:

File-system-id.region.nas.aliyuncs.com:/ / data nfs vers=4,minorversion=0,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,_netdev,noresvport 00

If you want to mount the NFS v3 file system, add the following command:

File-system-id.region.nas.aliyuncs.com:/ / data nfs vers=3,nolock,proto=tcp,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,_netdev,noresvport 00

# add the mount to the / etc/fstab configuration file, and mount $mount-a # to check the mount. If the mount address of the NFS file system exists in the result, the mount is successful $df-h | grep aliyun2, temporary deployment of stand-alone Harbor

Operate on a harbor1 machine

# online deployment of Harbor $cd / opt/ $wget https://github.com/goharbor/harbor/releases/download/v2.2.1/harbor-online-installer-v2.2.1.tgz $tar xf harbor-online-installer-v2.2.1.tgz $cd / opt/harbor $cp harbor.yml.tmpl harbor.yml # create harbor data store $mkdir / data/harbor # add domain name certificate The existing domain name SSL certificate $mkdir / data/harbor/cert # uploads the public key and private key of the SSL certificate to the $scp harbor.example.pem root@192.168.10.10:/data/harbor/cert/ $scp harbor.example.key root@192.168.10.10:/data/harbor/cert/ # configuration harbor.yml file in the / data/harbor/cert directory. The following is the result of comparing the modified file with the original file $diff harbor.yml harbor.yml.tmpl 5c5

< hostname: harbor.example.com --- >

Hostname: reg.mydomain.com 17,18c17,18

< certificate: /data/harbor/cert/harbor.example.pem < private_key: /data/harbor/cert/harbor.example.key --- >

Certificate: / your/certificate/path > private_key: / your/private/key/path 29c29

< external_url: https://harbor.example.com --- >

# external_url: https://reg.mydomain.com:8433

< data_volume: /data/harbor --- >

Data_volume: / data # generate configuration file $cd / opt/harbor # harbor enable helm charts and image vulnerability scanning $. / prepare-- with-notary-- with-trivy-- with-chartmuseum # install $. / install.sh-- with-notary-- with-trivy-- with-chartmuseum # View $docker-compose ps3, deploy Harbor database and Redis separately

# create postgres and redis storage directories

$mkdir-p / data/harbor-redis / data/harbor-postgresql

# modify the group to which it belongs

$chown-R 999.999 / data/harbor-redis / data/harbor-postgresql

# create postgres and redis docker-compose.yml files $vim docker-compose.yml version: '2.3' services: redis: image: goharbor/redis-photon:v2.2.1 container_name: harbor-redis restart: always cap_drop:-ALL cap_add:-CHOWN-SETGID-SETUID volumes:-/ data/harbor-redis:/var/lib/ Redis networks:-harbor-db ports:-6379 postgresql: image: goharbor/harbor-db:v2.2.1 container_name: harbor-postgresql restart: always cap_drop:-ALL cap_add:-CHOWN-DAC_OVERRIDE-SETGID-SETUID environment: POSTGRES_USER: postgres POSTGRES_PASSWORD: test2021 Volumes:-/ data/harbor-postgresql:/var/lib/postgresql/data:z networks:-harbor-db ports:-5432 networks: harbor-db: driver: bridge # deployment postgres and redis $docker-compose up-d4, Import postgres data # enter temporary harbor-db container export related tables and data $docker exec-it-u postgres harbor-db bash # Export data $pg_dump-U postgres registry > / tmp/registry.sql $pg_dump-U postgres notarysigner > / tmp/notarysigner.sql $pg_dump-U postgres notaryserver > / tmp/notaryserver.sql # import the data into a separately deployed PostgreSQL database $psql-h 192.168.10.10-U postgres registry-W

< /tmp/registry.sql $ psql -h 192.168.10.10 -U postgres notarysigner -W < /tmp/notarysigner.sql $ psql -h 192.168.10.10 -U postgres notaryserver -W < /tmp/notaryserver.sql5、清理临时部署单机Harbor数据和相关配置文件 # 清理harbr数据和配置文件 $ cp -a /data/harbor/cert /tmp/ $ rm -rf /data/harbor/* $ rm -rf /opt/harbor $ cp -a /tmp/cert /data/harbor/ # 重新创建配置文件 $ cd /opt/ $ tar xf harbor-online-installer-v2.2.1.tgz $ cd /opt/harbor # 修改配置文件，连接单独部署postgres和redis，注释harbor自带的postgres和redis $ cp harbor.yml.tmpl harbor.yml $ diff harbor.yml harbor.yml.tmpl 5c5 < hostname: harbor.example.com --- >

Hostname: reg.mydomain.com

17,18c17,18

< certificate: /data/harbor/cert/harbor.example.pem < private_key: /data/harbor/cert/harbor.example.key --- >

Certificate: / your/certificate/path

> private_key: / your/private/key/path

29c29

< external_url: https://harbor.example.com --- >

# external_url: https://reg.mydomain.com:8433

37c37

< # database: --- >

Database:

39c39

< # password: root123 --- >

Password: root123

41c41

< # max_idle_conns: 50 --- >

Max_idle_conns: 50

44c44

< # max_open_conns: 1000 --- >

Max_open_conns: 1000

47c47

< data_volume: /data/harbor --- >

Data_volume: / data

135158c135158

< external_database: < harbor: < host: 192.168.10.10 < port: 5432 < db_name: registry < username: postgres < password: test2021 < ssl_mode: disable < max_idle_conns: 50 < max_open_conns: 1000 < notary_signer: < host: 192.168.10.10 < port: 5432 < db_name: notarysigner < username: postgres < password: test2021 < ssl_mode: disable < notary_server: < host: 192.168.10.10 < port: 5432 < db_name: notaryserver < username: postgres < password: test2021 < ssl_mode: disable --- >

# external_database:

> # harbor:

> # host: harbor_db_host

> # port: harbor_db_port

> # db_name: harbor_db_name

> # username: harbor_db_username

> # password: harbor_db_password

> # ssl_mode: disable

> # max_idle_conns: 2

> # max_open_conns: 0

> # notary_signer:

> # host: notary_signer_db_host

> # port: notary_signer_db_port

> # db_name: notary_signer_db_name

> # username: notary_signer_db_username

> # password: notary_signer_db_password

> # ssl_mode: disable

> # notary_server:

> # host: notary_server_db_host

> # port: notary_server_db_port

> # db_name: notary_server_db_name

> # username: notary_server_db_username

> # password: notary_server_db_password

> # ssl_mode: disable

161175c161175

< external_redis: < # support redis, redis+sentinel < # host for redis: : < # host for redis+sentinel: < # :,:,: < host: 192.168.10.10:6379 < password: < # sentinel_master_set must be set to support redis+sentinel < #sentinel_master_set: < # db_index 0 is for core, it's unchangeable < registry_db_index: 1 < jobservice_db_index: 2 < chartmuseum_db_index: 3 < trivy_db_index: 5 < idle_timeout_seconds: 30 --- >

# external_redis:

> # # support redis, redis+sentinel

> # # host for redis::

> # # host for redis+sentinel:

> # #:,:

> # host: redis:6379

> # password:

> # # sentinel_master_set must be set to support redis+sentinel

> # # sentinel_master_set:

> # # db_index 0 is for core, it's unchangeable

> # registry_db_index: 1

> # jobservice_db_index: 2

> # chartmuseum_db_index: 3

> # trivy_db_index: 5

> # idle_timeout_seconds: 30

# deploy the first node harbor $cd / opt/harbor # harbor to enable helm charts and image vulnerability scanning $. / prepare-- with-notary-- with-trivy-- with-chartmuseum # install $. / install.sh-- with-notary-- with-trivy-- with-chartmuseum # View $docker-compose ps # copy configuration to harbor2 machine $scp-r / opt/harbor 192.168.10.11:/opt/

Operate on a harbor2 machine

# deploy the second node harbor $cd / opt/harbor # harbor to enable helm charts and image vulnerability scanning $. / prepare-- with-notary-- with-trivy-- with-chartmuseum # install $. / install.sh-- with-notary-- with-trivy-- with-chartmuseum # View $docker-compose ps

The SLB configuration method is not described here. For more information, please see the Aliyun SLB configuration document below. Port 443is configured, TCP protocol is used, and the backend is mapped to two harbor1 and harbor2 ports.

After reading the above, do you have any further understanding of how to build Harbor multi-instance high-availability shared storage? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.