Shulou(Shulou.com)06/01 Report--

This article mainly introduces the rexray in CentOS can not create ceph rbd on the docker volume problem how to do, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let Xiaobian with you to understand.

Background

We create the docker volume of the ceph rbd device through the rexray plug-in of docker, but always prompt that the creation fails.

# docker volume create-driver=rexray-opt=size=5-name=cephrbd-bookError response from daemon: create test_cephrbd_volume: VolumeDriver.Create: {"Error": "Failed to create new volume"}

OS version:

# lsb_release-aLSB Version:: core-4.1-amd64:core-4.1-noarchDistributor ID: CentOSDescription: CentOS Linux release 7.4.1708 (Core) Release: 7.4.1708Codename: Core

Ceph version:

# ceph versionceph version 0.94.5

Rexray version:

# rexray versionREX-Ray-Binary: / usr/bin/rexrayFlavor: client+agent+controllerSemVer: 0.9.0OsArch: Linux-x86_64Branch: (detached from 2a7458dCommit: 2a7458dd90a79c673463e14094377baf9fc8695eFormed: Wed, 26 Jul 2017 14:35:37 CSTlibStorage-SemVer: 0.6.0OsArch: Linux-x86_64Branch: (detached from fa055d6Commit: fa055d6da595602715bdfd5541b4aa6d4dcbcbd9Formed: Wed, 26 Jul 14:35:11 CST Analysis

We have also built a docker+ceph+rexray environment in the environment of ubuntu 16.04. using the docker command to create a volume did not report an error. The normal process is:

1. Creating a docker volume:rexray creates a rbd device using the rbd create command; 2. Use this volume to create a docker container: rexray will map the rbd device to the docker host to become a / dev/rbd device, and then use the mount command to mount the / dev/rbd device to the / var/lib/libstorage/volumes/ directory for container use. When deleting the container: first umount the mount of / dev/rbd device in / var/lib/libstorage/volumes/ directory, and then rbd unmap the mapping of rbd device on docker host; 4. Deleting a docker volume:rexray deletes the rbd device using the rbd rm command

In our actual use, we found that only the first step would go wrong, and there were no exceptions in the next three steps. Moreover, if we do not use the docker volume create command to create a rbd device, but directly use the "rbd create" command to create a rbd device, there is no error, and the rbd device can be used properly. This is a very strange place.

Checking the docker log only prompts "Failed to create new volume", and there is no other useful information.

# journalctl-xu docker...Jan 22 10:45:13 dcos-agent2 dockerd [66886]: time= "2018-01-22T10:45:13.937645601+08:00" level=error msg= "Handler for POST / v1.29/volumes/create returned error: create cephrbd-book: VolumeDriver.Create: {\" Error\ ":\" Failed to create new volume\ "}\ n"

Since there are no more valuable logs in docker, try taking a look at rexray's logs, where you need to turn on the debug option of rexray (Open method reference) so that you can see more logs.

# vim / var/log/rexray/rexray.log...time= "2018-01-19T19:12:32+08:00" level=info msg= "--HTTP REQUEST (CLIENT) -" time= "2018-01-19T19:12:32+08:00" level=info msg= "GET / volumes/rbd?attachments=0 HTTP/ 2018-01-19T19:12:32+08:00 "level=info msg=" Host: libstorage-server "time=" 2018-01-19T19:12:32+08:00 "level=info msg=" Libstorage-Instanceid: rbd=109.105.115.73 "time=" 2018-01-19T19:12:32+08:00 "level=info msg=" Libstorage-Localdevices: rbd= "time=" 2018-01-19T19:12:32+08:00 "level=info msg=" Libstorage-Tx: txID=b9fb044b-d47a-4ac5-7533-5824b8dc737a TxCR=1516360352 "time=" 2018-01-19T19:12:32+08:00 "level=info msg="time=" 2018-01-19T19:12:33+08:00 "level=infotime=" 2018-01-19T19:12:33+08:00 "level=info msg="-HTTP RESPONSE (CLIENT)-"time=" 2018-01-19T19 Time= "2018-01-19T19:12:33+08:00" level=info msg= "Content-Length: 228" time= "2018-01-19T19:12:33+08:00" level=info msg= "Content-Type: application/json" time= "2018-01-19T19:12:33+08:00" level=info msg= "Date: Fri 19 Jan 2018 11:12:33 GMT "time=" 2018-01-19T19:12:33+08:00 "level=info msg=" Libstorage-Servername: jade-chopper-ky "time=" 2018-01-19T19:12:33+08:00 "level=info msg="time=" 2018-01-19T19:12:33+08:00 "level=info msg=" {"time=" 2018-01-19T19:12:33+08:00 "level=info msg="\ "rbd.test_fs_device2\": {"time=" 2018-01 -19T19:12:33+08:00 "level=info msg="\ "name\":\ "test_fs_device2\" "time=" 2018-01-19T19:12:33+08:00 "level=info msg="\ "size\": 16, "time=" 2018-01-19T19:12:33+08:00 "level=info msg="\ "id\":\ "rbd.test_fs_device2\", "time=" 2018-01-19T19:12:33+08:00 "level=info msg="\ "type\":\ "rbd\"time=" 2018-01-19T19:12:33+08:00 "level=info msg="} Time= 2018-01-19T19:12:33+08:00 "level=info msg="\ "rbd.test_majk\": {"time=" 2018-01-19T19:12:33+08:00 "level=info msg="\ "name\":\ "test_majk\", "time=" 2018-01-19T19:12:33+08:00 "level=info msg="\ "id\":\ "rbd.test_majk\" "time=" 2018-01-19T19:12:33+08:00 "level=info msg="\ "type\":\ "rbd\"time=" 2018-01-19T19:12:33+08:00 "level=info msg="} "time=" 2018-01-19T19:12:33+08:00 "level=info msg="} "time=" 2018-01-19T19:12:33+08:00 "level=infotime=" 2018-01-19T19:12:33+08:00 "level=info msg="- -HTTP REQUEST (CLIENT)-"time=" 2018-01-19T19:12:33+08:00 "level=info msg=" POST / volumes/rbd HTTP/1.1 "time=" 2018-01-19T19:12:33+08:00 "level=info msg=" Host: libstorage-server "time=" 2018-01-19T19:12:33+08:00 "level=info msg= "Libstorage-Instanceid: rbd=109.105.115.73" time= "2018-01-19T19:12:33+08:00" level=info msg= "Libstorage-Localdevices: rbd=" time= "2018-01-19T19:12:33+08:00" level=info msg= "Libstorage-Tx: txID=b1565614-1b5f-4de2-5756-b74fb99887aa TxCR=1516360353 "time=" 2018-01-19T19:12:33+08:00 "level=info msg="time=" 2018-01-19T19:12:33+08:00 "level=info msg=" {\ "name\":\ "cephrbd-book\",\ "availabilityZone\":\ "iops\": 0,\ "size\": 5,\ "type\":\ " \ "opts\": {\ "size\":\ "5\"} "time=" 2018-01-19T19:12:33+08:00 "level=infotime=" 2018-01-19T19:12:33+08:00 "level=info msg="-HTTP RESPONSE (CLIENT)-"time=" 2018-01-19T19: 12 level=info msg= "HTTP/1.1 500 Internal Server Error" time= "2018-01-19T19:12:33+08:00" level=info msg= "Content-Length: 319" time= "2018-01-19T19:12:33+08:00" level=info msg= "Content-Type: application/json" time= "2018-01-19T19:12:33+08:00" level=info msg= "Date: Fri 19 Jan 2018 11:12:33 GMT "time=" 2018-01-19T19:12:33+08:00 "level=info msg=" Libstorage-Servername: jade-chopper-ky "time=" 2018-01-19T19:12:33+08:00 "level=info msg="time=" 2018-01-19T19:12:33+08:00 "level=info msg=" {"time=" 2018-01-19T19:12:33+08:00 "level=info msg="\ "message\":\ "Failed to create new volume\" "time=" 2018-01-19T19:12:33+08:00 "level=info msg="\ "status\": 500,500, "time=" 2018-01-19T19:12:33+08:00 "level=info msg="\ "error\": {"time=" 2018-01-19T19:12:33+08:00 "level=info msg="\ "driverName\":\ "rbd\" Time= 2018-01-19T19:12:33+08:00 "level=info msg="\ "inner\": {"time=" 2018-01-19T19:12:33+08:00 "level=info msg="\ "inner\":\ "Error running command: [rbd: strict_strtoll: garbage at end of string. Got:'5G'\ n]\ "," time= "2018-01-19T19:12:33+08:00" level=info msg= "\" msg\ ":\" unable to create rbd\ "" time= "2018-01-19T19:12:33+08:00" level=info msg= "}," time= "2018-01-19T19:12:33+08:00" level=info msg= "\" opts.Size\ ": 5 "time=" 2018-01-19T19:12:33+08:00 "level=info msg="\ "volumeName\":\ "cephrbd-book\"time=" 2018-01-19T19:12:33+08:00 "level=info msg="} "time=" 2018-01-19T19:12:33+08:00 "level=info msg="} "

As you can see from the log, each time the docker volume create command is sent, two http requests are actually sent, the first is to list the current rbd device, and the second is to issue a post request to create a rbd device. The log here gives more information about "rbd: strict_strtoll: garbage at end of string. Got:'5G'". In other words, it is likely that an error occurred when rexray called the rbd create command. Here, we open the rexray source code to continue tracking, the specific source file is rexray/blob/master/libstorage/drivers/storage/rbd/utils/utils.go.

/ RBDCreate creates a new RBD volume on the clusterfunc RBDCreate (ctx types.Context, pool * string, image * string, sizeGB * int64, objectSize * string, features [] * string) error {cmd: = exec.Command (rbdCmd, "create", poolOpt, * pool, "--object-size", * objectSize, "--size" Strconv.FormatInt (* sizeGB, 10) + "G",) for _, feature: = range features {cmd.Args = append (cmd.Args, "--image-feature") cmd.Args = append (cmd.Args, * feature)} cmd.Args = append (cmd.Args, * image) _, err: = RunCommand (ctx) Cmd) if err! = nil {return goof.WithError ("unable to create rbd", err)} return nil}

From the above code, we can roughly see that rexray is directly calling the rbd create command to create a rbd device, which is different from our direct call command in that it converts the size of the rbd into GB, and then concatenates a "G" string. Here we assemble a new rbd command as it does here, and then test it on CentOS:

# rbd create cephrbd-test-size 5Grbd: strict_strtoll: garbage at end of string. Got:'5G'

Sure enough, an error was reported, and we used this command to test it on ubuntu:

# rbd create cephrbd-test-- size 5G

Running on ubuntu, no error was reported. The same command has different results in different circumstances. Then the biggest possibility is that the version of rbd (ceph) is inconsistent. Get the ceph version under ubuntu.

# ceph versionceph version 10.2.7

The default version of ceph installed on CentOS is 0.94.5 (Hammer version), while the higher 10.2.7 (jewel version) is installed by default on Ubuntu. The 0.94.5 ceph version of rbd cannot parse parameters such as "G" and "M" that describe the size of the rbd. This is the cause of the problem.

Solution method

Install a newer version of Ceph (jewel or newer version) on CentOS.

Thank you for reading this article carefully. I hope the article "what to do if rexray cannot create ceph rbd on CentOS" shared by the editor will be helpful to everyone. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.