Shulou(Shulou.com)06/02 Report--

Proxmox+ceph cluster installation notes

1. Create 3 vmware virtual machines

Cpu:2x2 4 Core Open Virtualization

Memory: 2G

Hard disk 1Rom 20g, system disk

Hard disk 2Rom 200Gcceph storage disk

Networks: bridging mode

IP:10.8.20.11/12/13

Hostname: pve1.abc.com/pve2.abc.com/pve3.abc.com

2. Install proxmox on a pve1 virtual machine and install it on a 20g hard disk. You can install it by default.

3. System configuration

1) change to domestic source

Install vim

Apt update

Apt install vim-y

Install the source of Ali Cloud

Cd / etc/apt

Vi sources.list.d/pve-enterprise.list

Modify to

Deb http://download.proxmox.wiki/debian/pve buster pve-no-subscription

Vi sources.list

Modified to:

Deb http://mirrors.aliyun.com/debian buster main contrib

Deb http://mirrors.aliyun.com/debian buster-updates main contrib

# Security updates

Deb http://mirrors.aliyun.com/debian-security/ buster/updates main contrib

Update the system

Apt update

Apt upgrade-y

Init 6

2) remove the subscription prompts on the web management platform

Sed-I "s/data.status! = 'Active'/false/g" / usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js

Last but not least: systemctl restart pveproxy

3) extend local, that is, the root partition, without using lvm-thin

When pve is installed by default, the local and local-lvm sizes are 4.61G (root partition lvm for iso) and 8G (for virtual machines), respectively.

Next, delete 8g and merge it to 4.61g.

Is to delete the logical volume data and expand the logical volume root

Modify the storage configuration of pve

Vi / etc/pve/storage.cfg

Modified to:

Dir: local

Path / var/lib/vz

Content rootdir,iso,vztmpl,backup

(delete lvm-thin paragraph)

Delete the lv where the data is located

Lvremove / dev/pve/data (pve is the volume group name data is the logical volume name)

View volume group information

Vgs

VG # PV # LV # SN Attr VSize VFree

Pve 120 wz--n- system-> host, add the following 2 lines

10.8.20.12 pve2.abc.com pve2

10.8.20.13 pve3.abc.com pve3

Above, install the host with pve1 configured.

Just repeat the above on pve2 and pve3.

4. Set up pve cluster

Execute on pve1:

Pvecm create pve-cluster (pve-cluster is the cluster name)

Pve2, pve3 execution:

Pvecm add 10.8.20.11

View cluster status on pve1

Pvecm status

Pvecm nodes

5. Set up a ceph storage cluster (store virtual machine images, and use nfs for iso storage)

1) install ceph (executed on all nodes)

Pveceph install (install the latest nautilus version without adding the version number)

The following words indicate that the installation was successful

Installed ceph nautilus successfully

2) set up a ceph cluster network, which is shared with the management network for convenience (executed on all nodes)

Pveceph init-network 10.8.20.0 Compact 24

3) create ceph cluster storage Mon monitoring (executed on all nodes)

Pveceph createmon

4) create a ceph cluster storage OSD service (executed on all nodes)

Pveceph createosd / dev/sdb

5) create a cluster storage resource pool

Ceph osd pool create pve-pool 128 128 (executed only on pve1, where pve-pool is the name of the storage pool)

The origin of pg_num:

Set pg_num to 128when there are less than 5 OSD

When the number of OSD is 5 to 10, you can set the pg_num to 512.

When the number of OSD is 10 to 50, you can set the pg_num to 4096.

6) Log in to the web management page of pve1 to add RBD cluster storage

Data Center-> Storage-> add-> Select RBD

ID: enter as pve-ceph

Resource pool: pve-pool

Monitor:10.8.20.11 10.8.20.12 10.8.20.13 (note that multiple Mon are added to separate spaces)

Add Node: pve1,pve2,pve3

After you add it, view the rbd cluster storage configuration information

Cat / etc/pve/storage.cfg

Dir: local

Path / var/lib/vz

Content vztmpl,images,backup,iso,rootdir

Rbd: pve-ceph

Content rootdir,images

Krbd 1

Nodes pve3,pve2,pve1

Pool pve-pool

7) check whether the relevant key exists

Cd / etc/pve/priv

See if there is a ceph directory. If not, create it and copy the key in.

Mkdir ceph

Cp / etc/ceph/ceph.client.admin.keyring / etc/pve/priv/ceph/ceph.keyring

Generally speaking, it does exist:

Root@pve1:/etc/pve/priv# cd ceph

Root@pve1:/etc/pve/priv/ceph# ls

Pvecluster-ceph.keyring

8) Grant relevant permissions to the application

Ceph osd pool application enable pve-pool rgw rbd

9) install ceph Dashboard

Execute on pve1

Apt install ceph-mgr-dashboard

Ceph mgr module enable dashboard

Ceph dashboard create-self-signed-cert

Ceph dashboard ac-user-create admin admin123 administrator (admin is the user name, admin123 is the password, administrator specifies that the user is the administrator)

Systemctl restart ceph-mgr@pve1.service

Visit https://10.8.20.11:8443 and log in with the username admin password admin123

At this point, the ceph installation is complete

6. Install nfs server and provide iso storage

Install a centos server and enable nfs services

Suppose the output directory of nfs is the / data/nfs directory on this centos

Yum install rpcbind nfs-utils-y

Vi / etc/exports

Join:

/ data/nfs 10.8.20.0 Compact 24 (rw,sync,no_root_squash)

Start the nfs service

Systemctl enable rpcbind nfs-server

Systemctl start rpcbind nfs-server (centos7)

View shared directory

Exportfs

Activate configuration

Exportfs-rav

Finally, add nfs storage to the pve cluster

Pve1web management page: data Center-> Storage-> add-> Select nfs

After adding, you can upload the iso file, which will be stored in the / data/nfs/template/iso directory of the centos server.

7. Install the virtual machine test cluster, and choose to store the virtual machine hard disk under pve-ceph storage before you can use the migration and HA functions.

My technical blog: memories of autumn

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.