Shulou(Shulou.com)06/02 Report--

RBD introduction

RBD block storage is the most widely used and stable storage type among the three storage types provided by ceph. RBD blocks are similar to disks and can be mounted to physical machines or virtual machines. There are usually two ways to mount them:

Kernel module (KRBD) uses QEMU simulator through LIBRBD mode

A block is an ordered byte, and a normal block size is 512 bytes. Block-based storage is the most common way, and common hard disks, floppy disks and CD optical drives are the easiest and quickest devices to store data.

When providing block devices on physical machines, you use Kernel's RBD module, and when driving based on kernel modules, you can use Linux's own page cache (Page Caching) to improve performance.

When a block device is provided in a virtual machine (such as QUEM/KVM), the block device is usually provided by calling the librbd library using LIBVIRT.

Deploy RBD

Before deployment, you need to check the kernel version to see if it supports RBD. It is recommended to upgrade to version 4.5 or above.

[root@local-node-1] # uname-r 4.4.174-1.el7.elrepo.x86_64 [root@local-node-1] # modprobe rbd one. Initialize a cluster before deploying RBD, you need to deploy a Ceph cluster The cluster status is as follows: [root@local-node-1 ~] # ceph- s cluster: id: 7bd25f8d-b76f-4ff9-89ec-186287bbeaa5 health: HEALTH_OK services: mon: 2 daemons, quorum local-node-2,local-node-3 mgr: ceph-mgr (active) osd: 9 osds: 9 up, 9 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0B usage: 9.2 GiB used 81 GiB / 90 GiB avail pgs: [root@local-node-1 ~] # ceph osd treeID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF-1 0.08817 root default-3 0.02939 host local-node-1 0 hdd 0.00980 osd.0 up 1.00000 1.00000 1 hdd 0.00980 osd.1 up 1.00000 1.00000 2 hdd 0.00980 osd.2 up 1.00000 1.00000-5 0.02939 host local-node-2 3 hdd 0.00980 osd.3 up 1.00000 1.00000 4 hdd 0.00980 osd.4 Up 1.00000 1.00000 5 hdd 0.00980 osd.5 up 1.00000 1.00000-7 0.02939 host local-node-3 6 hdd 0.00980 osd.6 up 1.00000 1.00000 7 hdd 0.00980 osd.7 up 1.00000 1.00000 8 Hdd 0.00980 osd.8 up 1.00000 1.00000 initial poolceph osd pool create ceph_rbd 128rbd pool init ceph_rbd creates a block storage user: ceph auth get-or-create client. {ID} mon 'profile rbd' osd' profile {profile name} [pool= {pool-name}] [ Profile.] 'EG:# ceph auth get-or-create client.docker mon' profile rbd' osd 'profile rbd pool=ceph_rbd, profile rbd-read-only pool=images' [client.docker] key = AQDQkK1cpNAKJRAAnaw2ZYeFHsXrsTWX3QonkQ== add this user file to the configuration directory [root@local-node-1 ~] # cat / etc/ceph/ceph.client.docker.keyring [client.docker] key = AQDQkK1cpNAKJRAAnaw2ZYeFHsXrsTWX3QonkQ== II. Create a block device image create a block device image of a specified size, where the size unit is M. In all of the following commands If you do not specify the pool==rbd create named rbd for the pool default operation: size {megabytes} {pool-name} / {image-name} eg: [root@local-node-1 ~] # rbd create-- size 1024 ceph_rbd/docker_image View the created image [root@local-node-1 ~] # rbd ls ceph_rbddocker_image lists the block devices [root@local-node-1 ~] # rbd trash ls ceph_rbd that will be delayed to delete the image in the rbd pool Information: [root@local-node-1 ~] # rbd info ceph_rbd/docker_imagerbd image' docker_image': size 1 GiB in 256 objects order 22 (4 MiB objects) id: 1bedc6b8b4567 block_name_prefix: rbd_data.1bedc6b8b4567 format: 2 features: layering Exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Wed Apr 10 14:52:48 2019 if necessary You can expand or reduce the capacity of an image. [root@local-node-1] # rbd resize-- size 2048 ceph_rbd/docker_imageResizing image: 100% complete...done.# reduce capacity [root@local-node-1] # rbd resize-- size 1024 ceph_rbd/docker_image-- allow-shrinkResizing image: 100% complete...done.

Ceph's block mirroring devices are thinly configured and do not use physical storage space when they are created and specified until data is stored. But they can specify the maximum capacity limit through-- size.

Delete block device image rbd rm ceph_rbd/docker_image can move the temporarily unused image to trash# rbd trash mv ceph_rbd/docker_image# rbd trash ls ceph_rbd1beb76b8b4567 docker_image if you want to restore the image in trash, you can execute the following command: rbd trash restore ceph_rbd/1beb76b8b4567# at this time there will be no data in trash, the original image has been restored # rbd ls ceph_rbddocker_image if you want to rename the image during restore Execute the following command: rbd trash restore ceph_rbd/1beb76b8b4567-- image docker # rename the image to docker to delete the image in trash, using the IDrbd trash rm ceph_rbd/1beb76b8b4567 of the image

Tip:

Even if the mirror has a snapshot, or one of its own clones is in use, the mirror can be moved to the trash, but cannot be deleted from the trash. If you want to delete, you can use-expires-at to set the delay time (default is now), and if the delay time has not expired, you cannot delete three unless you use-force. Use the kernel module mapping RBD to check the cluster version and adjustable parameter version: # if the kernel version is below 4.5, it is recommended to use hammer Otherwise, rbd cannot be mapped [root@local-node-1 ~] # ceph osd crush show-tunables {"choose_local_tries": 0, "choose_local_fallback_tries": 0, "choose_total_tries": 50, "chooseleaf_descend_once": 1, "chooseleaf_vary_r": 1, "chooseleaf_stable": 0, "straw_calc_version": 1, "allowed_bucket_algs": 54 "profile": "hammer", "optimal_tunables": 0, "legacy_tunables": 0, "minimum_required_version": "hammer", "require_feature_tunables": 1, "require_feature_tunables2": 1, "has_v2_rules": 0, "require_feature_tunables3": 1, "has_v3_rules": 0, "has_v4_buckets": 1, "require_feature_tunables5": 0 "has_v5_rules": 0} # set version [root@local-node-1 ~] # ceph osd crush tunables hammer to view the existing image [root@local-node-1 ~] # rbd list ceph_rbddocker_image

3. Map block devices on the client side (need to install ceph)

[root@local-node-1] # rbd device map ceph_rbd/docker_image-- id admin/dev/rbd0

= = prompt: =

If the following error occurs when performing the mapping step, some features of the current kernel rbd are not supported, and some features need to be disabled:

# rbd device map ceph_rbd/docker_image-id adminrbd: sysfs write failedRBD image feature set mismatch. Try disabling features unsupported by the kernel with "rbd feature disable" .in some cases useful info is found in syslog-try "dmesg | tail" .rbd: map failed: (6) No such device or address

Disable properties:

# rbd feature disable ceph_rbd/docker_image exclusive-lock, object-map, fast-diff, deep-flatten if the key authentication policy is set, you need to specify the key when mapping, which can be the specified keyring file. It can also be a file to save the key separately: sudo rbd device map rbd/myimage-- id admin-- keyring / path/to/keyringsudo rbd device map rbd/myimage-- id admin-- keyfile / path/to/fileeg:#rbd device map ceph_rbd/docker_image-- id docker-- keyring / etc/ceph/ceph.client.docker.keyring / dev/rbd0 to view the mapped block storage [root@local-node-1 ~] # rbd device listid pool image snap device 0 ceph_rbd docker_image-/ dev/rbd0 [root@local-node-1 ~] # lsblk | grep rbdrbd0 252root@local-node-1 0 01G 0 disk uses rbd block storage [root@local-node-1 ~] # mkfs.xfs / dev/rbd [root@local-node-1 ~] # mount / dev/rbd0 / mnt

7. If you want to uninstall the device, use the following command:

[root@local-node-1 ~] # umount / mnt/ [root@local-node-1 ~] # rbd device unmap / dev/rbd0RBD

When the mounted rbd is unmounted, the data in the block device is generally not lost (it may damage the data and become unavailable after a forced restart) and can be remounted to another host.

RBD supports replication, snapshot and online expansion.

RBD Snapshot

A snapshot is a read-only copy of an image at a specific point in time. An advanced feature of Ceph block devices is that you can create snapshots of images to preserve their history. Ceph also supports hierarchical snapshots, allowing you to clone images (such as VM images) quickly and easily. Ceph's snapshot feature supports rbd commands and a variety of advanced interfaces, including QEMU, libvirt, OpenStack, and CloudStack.

If the image is still in the process of performing an Icano operation when the snapshot is taken, the snapshot may not get the accurate or up-to-date data of the image, and the snapshot may have to be cloned to a new mountable image. Therefore, we recommend that you stop the Iripple O operation before taking a snapshot. If the image contains a file system, make sure the file system is in a consistent state before taking a snapshot or use the fsck command to check the mounted block devices. To stop the fsfreeze O operation, you can use the command. For virtual machines, qemu-guest-agent is used to automatically freeze the file system when taking snapshots.

Check the file system Freeze IO operations before creating snapshots (for more operations on fsfreeze reference: https://blog.pythian.com/fsfreeze-in-linux/) = = all operations on this directory file will hang after the execution of this command = = fsfreeze-f / mnt create snapshot: unfreeze IO after rbd snap create rbd/foo@snapnameEG:rbd snap create rbd/test@test-snap snapshot is created successfully fsfreeze-u / mnt/ view snapshot [root@local-node-1 ~] # rbd snap ls rbd/testSNAPID NAME SIZE TIMESTAMP 4 test-snap 1 GiB Tue Apr 16 14: 49:27 2019 5 test-snap-2 1 GiB Tue Apr 16 15:56:18 2019 rollback The rolled-back mirror retains the previous snapshot data, but it is read-only and cannot be written or deleted. Rbd snap rollback rbd/test@test-snap: for mounted disks, you need to unmount them and then mount them again. If the following error occurs when mounting: # mount / dev/rbd0 / mnt/mount: wrong fs type, bad option, bad superblock on / dev/rbd0, missing codepage or helper program Or other error In some cases useful info is found in syslog-try dmesg | tail or so.# encountered the following error when using xfs_repair repair: xfs_repair / dev/rbd0Phase 1-find and verify superblock...Phase 2-using internal log-zero log...ERROR: The filesystem has valuable metadata changes in a log which needs tobe replayed. Mount the filesystem to replay the log, and unmount it beforere-running xfs_repair. If you are unable to mount the filesystem, then usethe-L option to destroy the log and attempt a repair.Note that destroying the log may cause corruption-- please attempt a mountof the filesystem before doing this.# uses the-L parameter to fix the file mounted by xfs_repair-L / dev/rbd0, the system is still read-only. Mount / dev/rbd0 / mnt/ if you want to delete a snapshot, execute the following command: rbd snap rm rbd/test@test-snap # if you want to delete all snapshots of an image, perform: rbd snap purge rbd/fooRBD copy

Ceph can clone a write-time replica from a snapshot, and because the snapshot is read-only, we can use the snapshot to create a write-time replica when we need to modify the file. (in Openstack, use this mechanism to create a new virtual machine, usually use a snapshot to save the image, copy this snapshot to create a new virtual machine)

Ceph only supports cloning images of format 2 (that is, created with rbd create-- image-format 2, which is the default in the new version). The kernel client supports cloned images from version 3.10.

The specific process is as follows:

Create a block device image-> create a snapshot-- > protect a snapshot-- > Clone a snapshot to protect a snapshot. The clone image wants to access the parent snapshot. If the user accidentally deletes the parent snapshot, all cloned images are corrupted. To prevent data loss, snapshots must be protected before cloning: # create snapshot rbd snap create rbd/test@test-snap# list snapshot # rbd snap list rbd/testSNAPID NAME SIZE TIMESTAMP 10 test-snap 1 GiB Tue Apr 16 17:46:48 20 protect snapshot so that rbd snap protect rbd/test@test-snap clone snapshot cannot be deleted, parent storage pool, parent image name and snapshot need to be specified Child storage pool and child image name, you can specify a different storage pool: rbd clone {pool-name} / {parent-image} @ {snap-name} {pool-name} / {child-image-name} EG:rbd clone rbd/test@test-snap rbd/test-new

View the newly created image:

# rbd lstesttest-new cancel snapshot protection rbd snap unprotect rbd/test@test-snap view the generated sub-image of the snapshot # rbd children rbd/test@test-snaprbd/test-new [root@local-node-1 ~] # rbd-- pool rbd-- image test-new inforbd image 'test-new': size 1 GiB in 256 objects order 22 (4 MiB objects) id: ba9096b8b4567 block_name_prefix: rbd_data.ba9096b8b4567 format: 2 features: layering op_ Features: flags: create_timestamp: Tue Apr 16 17:53:51 2019 parent: rbd/test@test-snap # correlation information such as parent snapshot is shown here overlap: 1 GiB flattened image Disassociate it from the parent image to prevent changes in the parent image from affecting the new image: rbd flatten rbd/test-new, the newly generated image can be read and written at will. RBD online expansion is used to expand the mounted rbd: # rbd resize ceph_rbd/docker_image-- size 4096Resizing image: 100% complete...done. Check the expanded status: # rbd info ceph_rbd/docker_imagerbd image' docker_image': size 4 GiB in 1024 objects order 22 (4 MiB objects) id: 1bef96b8b4567 block_name_prefix: rbd_data.1bef96b8b4567 format: 2 features: layering op_features: flags: create_timestamp: Wed Apr 10 15:50:21 2019 check whether the capacity on the client is accepted by the kernel: # xfs_growfs-d / mntmeta-data=/ Dev/rbd0 isize=512 agcount=9 Agsize=31744 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0 spinodes=0data = bsize=4096 blocks=262144, imaxpct=25 = sunit=1024 swidth=1024 blksnaming = version 2 bsize=4096 ascii-ci=0 ftype=1log = internal bsize=4096 blocks=2560 Version=2 = sectsz=512 sunit=8 blks, lazy-count=1realtime = none extsz=4096 blocks=0, rtextents=0data blocks changed from 262144 to 104857 upload lsblk | grep mntrbd0 252G0 04G 0 disk / mnt directly upload files exceeding 1G It is found that the xfs file system has been automatically expanded: # df-h | grep mnt/dev/rbd0 4.0g 3.1g 998M 76% / mntRBD online reduction (dead operation, only for testing) when performing the reduction operation, make sure there is enough free space, otherwise data may be lost: # rbd resize-size 2048 ceph_rbd/docker_image-allow-shrinkResizing image: 100% complete...done. View the reduced mount disk: # lsblk | grep mntrbd0 252disk 002G 0 / mnt# xfs_growfs-d / mnt/meta-data=/dev/rbd0 isize=512 agcount=34, agsize=31744 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0 spinodes=0data = bsize=4096 blocks=1048576 Imaxpct=25 = sunit=1024 swidth=1024 blksnaming = version 2 bsize=4096 ascii-ci=0 ftype=1log = internal bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=8 blks, lazy-count=1realtime = none extsz=4096 blocks=0, rtextents=0data size 524288 too small, old size is 1048576 is not recommended. RBD uninstall

= = if you uninstall RBD, the stored data will be lost. = =

Unmount disk Unmap [root@local-node-3 mnt] # df-h | grep mnt.../dev/rbd0 2.0G 33M 2.0G 2% / mnt [root@local-node-3 ~] # rbd device listid pool image snap device 0 ceph_rbd docker_image-/ dev/rbd0 [root@local-node-3 ~] # rbd device unmap / dev/rbd0 Delete rbd Image [root@local-node-3 ~] # rbd ls ceph _ rbddocker_ image [root @ local-node-3 ~] # rbd info ceph_rbd/docker_imagerbd image' docker_image': size 2 GiB in 512 objects order 22 (4 MiB objects) id: 1bef96b8b4567 block_name_prefix: rbd_data.1bef96b8b4567 format: 2 features: layering op_features: flags: create_timestamp: Wed Apr 10 15:50:21 2019 [root@local-node-3 ~] # rbd trash ls ceph_rbd# remove rbd to trash You can also delete [root@local-node-3 ~] # rbd trash mv ceph_rbd/docker_ image [root @ local-node-3 ~] # rbd trash ls ceph_rbd1bef96b8b4567 docker_image# delete the image [root@local-node-3 ~] # rbd trash rm ceph_rbd/1bef96b8b4567Removing image: 100% complete...done. [root@local-node-3 ~] # rbd trash ls ceph_ rbd [root @ local-node-3 ~] # rbd ls ceph_ Rbd delete pool [root@local-node-3 ~] # ceph osd lspools7 ceph_ rbd [root @ local-node-3 ~] # ceph osd pool rm ceph_rbd ceph_rbd-- yes-i-really-really-mean-itpool 'ceph_rbd' removed View cluster status: [root@local-node-3 ~] # ceph-s cluster: id: 7bd25f8d-b76f-4ff9-89ec-186287bbeaa5 health: HEALTH_OK services: mon: 3 daemons Quorum local-node-1,local-node-2,local-node-3 mgr: ceph-mgr (active) osd: 9 osds: 9 up, 9 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 9.3 GiB used 81 GiB / 90 GiB avail pgs: [root@local-node-3 ~] # ceph osd treeID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF-1 0.08817 root default-3 0.02939 host local-node-1 0 hdd 0.00980 osd.0 up 1.00000 1.00000 1 hdd 0.00980 osd.1 up 1.00000 1.00000 2 hdd 0.00980 osd.2 up 1.00000 1.00000-5 0.02939 host local-node-2 3 hdd 0.00980 osd.3 up 1.00000 1.00000 4 hdd 0.00980 osd.4 Up 1.00000 1.00000 5 hdd 0.00980 osd.5 up 1.00000 1.00000-7 0.02939 host local-node-3 6 hdd 0.00980 osd.6 up 1.00000 1.00000 7 hdd 0.00980 osd.7 up 1.00000 1.00000 8 Hdd 0.00980 osd.8 up 1.00000 1.00000

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.