Shulou(Shulou.com)05/31 Report--

Editor to share with you what skills to use ceph, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

1. Set up cephx keys

If ceph sets cephx, you can set permissions for different users.

# create key$ ceph auth get-or-create client.dummy mon 'allow r' osd 'allow rwx pool=dummy' of dummy [client.dummy] key = AQAPiu1RCMb4CxAAmP7rrufwZPRqy8bpQa2OeQ==$ ceph auth listinstalled auth entries:...client.dummy key: AQAPiu1RCMb4CxAAmP7rrufwZPRqy8bpQa2OeQ== caps: [mon] allow r caps: [osd] allow rwx pool=dummy...# to key reassign permission of dummy $ceph auth caps client.dummy mon' allow rwx' osd 'allow rwx pool=dummy'updated caps for client.dummy$ ceph auth listinstalled auth entries:client.dummy Key: AQAPiu1RCMb4CxAAmP7rrufwZPRqy8bpQa2OeQ== caps: [mon] allow rwx caps: [osd] allow allow rwx pool=dummy/dev/sda2 / srv/ceph/osdX1 xfs rw Noexec,nodev,noatime,nodiratime,barrier=0 0 02. See where rbd is mounted

Since rbd showmapped can only display locally mounted rbd devices, if there are more machines and you happen to forget where the map is, you can only look for them one by one. This problem can be solved by using listwatchers.

For blocks with an image format of 1:

$rbd info bootrbd image 'boot': size 10240 MB in 2560 objects order 22 (4096 kB objects) block_name_prefix: rb.0.89ee.2ae8944a format: 1$ rados-p rbd listwatchers boot.rbdwatcher=192.168.251.102:0/2550823152 client.35321 cookie=1

For blocks with an image format of 2, there are some differences:

[root@osd2 ceph] # rbd info myrbd/rbd1rbd image 'rbd1': size 8192 kB in 2 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.13436b8b4567 format: 2 features: layering [root@osd2 ceph] # rados-p myrbd listwatchers rbd_header.13436b8b4567watcher=192.168.108.3:0/2292307264 client.5130 cookie=1

You need to append the sequence number obtained by rbd info to rbd_header.

3. How to delete a giant rbd image

I've seen in some blogs that deleting a giant rbd image can be time-consuming (a long night) if you go directly through rbd rm. However, after a try on ceph 0.87, this problem no longer exists, and the specific process is as follows:

# create a 1PB-sized block [root@osd2 ceph] # time rbd create myrbd/huge-image-s 1024000000real 0m0.353suser 0m0.016ssys 0m0.009s [root@osd2 ceph] # rbd info myrbd/huge-imagerbd image' huge-image': size 976 TB in 256000000 objects order 22 (4096 kB objects) block_name_prefix: rb.0.1489.6b8b4567 format: 1 [root@osd2 ceph] # time rbd rm myrbd/huge-imageRemoving Image: 2% complete. ^\ Quit (core dumped) real 10m24.406suser 18m58.335ssys 11m39.507s

A 1PB-sized block is created above, which may be too large, but it is still very slow to delete rbd rm directly, so the following method is used:

[root@osd2 ceph] # rados-p myrbd rm huge-image.rbd [root@osd2 ceph] # time rbd rm myrbd/huge-image2014-11-06 09 root@osd2 ceph 42.916826 7fdb4fd5a7e0-1 librbd::ImageCtx: (2) No such file or directoryRemoving image: 100% complete...done.real 0m0.192suser 0m0.012ssys 0m0.013s

Try a block the size of 1TB:

[root@osd2 ceph] # rbd create myrbd/huge-image-s 1024000 [root@osd2 ceph] # rbd info myrbd/huge-imagerbd image' huge-image': size 1000 GB in 256000 objects order 22 (4096 kB objects) block_name_prefix: rb.0.149c.6b8b4567 format: 1 [root@osd2 ceph] # time rbd rm myrbd/huge-imageRemoving image: 100 complete...done.real 0m29.418suser 0m52.467ssys 0m32.372s

So giant blocks should be deleted in the following ways:

Format 1:

[root@osd2 ceph] # rbd create myrbd/huge-image-s 1024000000 [root@osd2 ceph] # rbd info myrbd/huge-imagerbd image' huge-image': size 976 TB in 256000000 objects order 22 (4096 kB objects) block_name_prefix: rb.0.14a5.6b8b4567 format: 1 [root@osd2 ceph] # rados-p myrbd rm huge-image.rbd [root@osd2 ceph] # time rados-p myrbd ls | grep'^ rb.0.14a5.6b8b4567' | | xargs-n 200 rados-p myrbd rm [root@osd2 ceph] # time rbd rm myrbd/huge-image2014-11-06 09Partition 54 time rbd rm myrbd/huge-image2014 12.718211 7ffae55747e0-1 librbd::ImageCtx: error finding header: (2) No such file or directoryRemoving image: 100% complete...done.real 0m0.191suser 0m0.015ssys 0m0.010s |

Format 2:

[root@osd2 ceph] # rbd create myrbd/huge-image-s 1024000000-- image-format=2 [root@osd2 ceph] # rbd info myrbd/huge-imagerbd image' huge-image': size 976 TB in 256000000 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.14986b8b4567 format: 2 features: layering [root@osd2 ceph] # rados-p myrbd rm rbd_id.huge-image [root@osd2 ceph] # rados-p myrbd Rm rbd_ header.14986b8b4567 [root @ osd2 ceph] # rados-p myrbd ls | grep'^ rbd_data.14986b8b4567' | xargs-n 200 rados-p myrbd rm [root@osd2 ceph] # time rbd rm myrbd/huge-image2014-11-06 09 7f6b6923c7e0 59 7f6b6923c7e0-1 librbd::ImageCtx: error finding header: (2) No such file or directoryRemoving image: 100% complete...done.real 0m0.192suser 0m0.016ssys 0m0.010s

Note that if the block is empty, do not want the xargs statement; if it is non-empty, you need it.

Therefore, if it is a block above 100TB, it is best to delete id first, and then delete it by rbd rm.

4. Check whether kvm or qemu supports ceph$ sudo qemu-system-x86_64-drive format=?Supported formats: vvfat vpc vmdk vhdx vdi sheepdog sheepdog sheepdog rbd raw host_cdrom host_floppy host_device file qed qcow2 qcow parallels nbd nbd nbd dmg tftp ftps ftp https http cow cloop bochs blkverify blkdebug$ qemu-img-h.Supported formats: vvfat vpc vmdk vhdx vdi sheepdog sheepdog sheepdog rbd raw host_cdrom host_floppy host_device file qed qcow2 qcow parallels nbd nbd nbd dmg tftp ftps ftp https http cow cloop bochs blkverify blkdebug

You can download the latest rpm or deb packages at http://ceph.com/packages/.

5. Configure nfs with ceph rbd

A simple and practical storage method, which is as follows:

# install nfs rpm [root@osd1 current] # yum install nfs-utils rpcbindLoaded plugins: fastestmirror, priorities, refresh-packagekit SecurityLoading mirror speeds from cached hostfileepel/metalink | 5.5kB 00:00 * base: mirrors.cug.edu.cn * epel: mirrors.yun-idc.com * extras: mirrors.btte.net * rpmforge: ftp.riken.jp * updates: mirrors.btte.netCeph | | 951B 00:00 Ceph-noarch | 951B 00:00 base | | | 3.7 kB 00:00 ceph-source | 951B | 00:00 epel | 4.4 kB 00:00 epel/primary_db | | 6.3 MB 00:01 extras | 3.4 kB 00:00 rpmforge | | | 1.9 kB 00:00 updates | 3.4 kB 00:00 updates/primary_db | | | 188 kB 00:00 69 packages excluded due to repository priority protectionsSetting up Install ProcessPackage rpcbind-0.2.0-11.el6.x86_64 already installed and latest versionResolving Dependencies-- > Running transaction check--- > Package nfs-utils.x86_64 1buret 1.2.3-39.el6 will be updated--- > Package nfs-utils.x86_64 1purl 1.2.3-54.el6 will be an | Update-- > Finished Dependency ResolutionDependencies Resolved= Package Arch Version Repository Size=Updating: nfs-utils x86'64 1purl 1.2.3-54.el6 base 326 kTransaction Summary=Upgrade 1 Package (s) Total download size: 326 kIs this ok [y yDownloading Packages:nfs-utils-1.2.3 N]: yDownloading Packages:nfs-utils-1.2.3-54.el6.x86_64.rpm | 326 kB 00:00 Running rpm_check_debugRunning Transaction TestTransaction Test SucceededRunning Transaction Updating: 1:nfs-utils-1 .2.3-54.el6.x86_64 1 Cleanup 2 Cleanup: 1:nfs-utils-1.2.3-39.el6.x86_64 2 Verifying: 1:nfs-utils-1.2.3-54.el6.x86_64 1 Verifying: 1:nfs-utils-1.2.3-39.el6.x86_64 2 Updated: nfs-utils.x86_64 1 Updated 1.2.3-54.el6 # create a block and format, Mount [root@osd1 current] # rbd create myrbd/nfs_image-s 1024000-- image-format=2 [root@osd1 current] # rbd map myrbd/nfs_image/dev/ rbd0 [root @ osd1 current] # mkdir / mnt/nfs [root@osd1 current] # mkfs.xfs / dev/rbd0log stripe unit (4194304 bytes) is too large (maximum is 256KiB) log stripe unit adjusted to 32KiBmeta-data=/dev/rbd0 isize=256 agcount=33 Agsize=8190976 blks = sectsz=512 attr=2, projid32bit=0data = bsize=4096 blocks=262144000, imaxpct=25 = sunit=1024 swidth=1024 blksnaming = version 2 bsize=4096 ascii-ci=0log = internal log bsize=4096 blocks=128000, version=2 = sectsz=512 sunit=8 blks, lazy-count=1realtime = none extsz=4096 blocks=0 Rtextents=0 [root@osd1 current] # mount / dev/rbd0-o rw,noexec,nodev,noatime,nobarrier / mnt/nfs# overwrites the exports file Add a line [root@osd1 current] # vim / etc/exports/mnt/nfs 192.168.108.0 to 24 (rw,no_root_squash,no_subtree_check Async) [root@osd1 current] # exportfs-r also need to execute the instruction service rpcbind start [root@osd1 current] # service nfs startStarting NFS services: [OK] Starting NFS quotas: [OK] Starting NFS mountd: [OK] Starting NFS daemon: [OK] Starting RPC idmapd: [OK] at this point, the client can mount it. Client runs: showmount-e 192.168.108.2 and then mounts: mount-t nfs 192.168.108.2:/mnt/nfs / mnt/nfs

If it cannot be mounted, run the service rpcbind start or service portmap start command to try it.

The above is all the content of the article "what are the skills in ceph?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.