Shulou(Shulou.com)06/01 Report--

This article introduces the knowledge of "how to run Ceph in Docker". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Ceph is a completely open source distributed storage scheme, network block device and file system, which has the characteristics of high stability, high performance and high scalability, and can cope with the amount of data from terabyte to exabyte.

By using innovative scheduling algorithms (CRUSH), active storage nodes, and peer-to-peer 's gossip protocol, Ceph avoids scalability and reliability issues in traditional centralized control and lookup table.

Ceph is currently highly respected in the open source community and has been widely used in virtualization platforms (Proxmox), cloud computing platforms (OpenStack, CloudStack, OpenNebula), container technology (Docker), and big data analysis system (Hadoop, meted server as HDFS).

I've been trying to run Ceph in Docker for almost two years. To this day, I am still doing this work. Recently, I have put a lot of effort into deploying Ceph in Docker.

Before going into the technical details, I would like to thank Sean C McCord for his strong support for this work, and the open source ceph-docker project of that year was indeed based on the early work of Sean.

Now let's take a look at how to run Ceph in Docker!

Principle

Running Ceph in Docker is a controversial topic, and many people question the significance of this operation. While there is no big problem with instrumentation modules, metadata servers, and RADOS gateway containerization, things can get tricky for OSD (object storage daemon). Ceph's OSD is optimized for physical machines and has a lot to do with the underlying hardware. If the physical hard disk fails, the OSD will not work, which brings problems to the containerized scenario.

Frankly, at some point in the past, I was thinking:

"I don't know why I'm doing this. I just know that someone needs it (of course, they probably don't know why they want it). I just think if I want to try the technology, give it a try!"

Of course, the above idea doesn't sound optimistic, but it was really what I thought at the time. There has been some change in my point of view, and let me explain why it is worth it. I hope this explanation will also change your mind (my explanation is not just "Docker is cool, so we have to run everything in Docker!").

Many developers have spent a lot of time containerizing their software. In the process, they have also used a variety of different tools to build and manage their environment. I wouldn't be surprised to see someone using Kubernetes as a management tool.

Some people like to apply the latest technology to production, otherwise they will find their work boring. So when they see that their favorite open source storage solution is also being containerized, they will be happy that it conforms to the "all containerization" approach.

Unlike traditional yum or apt-get, containers make it easy to upgrade and roll back software: we can release new daemons versions through docker stop or docker run. We can even run multiple isolated clusters on a single physical machine. All these provide great convenience for the development process.

Project

As mentioned above, all the work is based on Sean C McCord's early contributions, and we have since perfected his work. Now if you use ceph-docker, you can run each single Ceph daemon on Ubuntu or CentOS.

We have a lot of images in Docker Hub, and we use the namespace of Ceph, so our image prefix is ceph/. We use automatic builds, so each time we integrate a new patch, a new build is triggered, resulting in a new container image.

Since we are now working on code refactoring, you will see a lot of mirrored versions. We have always built a separate image of each daemon (as we do when we integrate these patches). So monitoring, OSD, mds, and radosgw each have separate images. This is not the ideal solution, so we are trying to integrate all the components into a single image called daemon.

This image contains all the modules, and you can selectively activate different modules from the command line while running docker run. If you want to try our image, we recommend using ceph/daemon image. Let me give you an example of how to run it.

Containerized Ceph

Monitor and control

Since the monitoring module cannot communicate over the NAT network, we must use-- net=host to open the host's network layer to the container:

$sudo docker run-d-net=host\

-v / etc/ceph:/etc/ceph\

-v / var/lib/ceph/:/var/lib/ceph\

-e MON_IP=192.168.0.20\

-e CEPH_PUBLIC_NETWORK=192.168.0.0/24\

Ceph/daemon mon

You can configure the following options:

MON_IP is the host IP running Docker

MON_NAME is the name of your monitoring module (default is $(hostname))

CEPH_PUBLIC_NETWORK is the CIDR of the host running Docker. It must be the same network as MON_IP.

CEPH_CLUSTER_NETWORK is the CIDR of the alternate network port of the host running Docker and is used for backup traffic of OSD.

Object Storage Daemon

We can now implement allowing each OSD process to run in a separate container. According to the concept of microservices, no more than one service should run in a container. In our case, running multiple OSD processes in the same container breaks this idea, which also brings additional complexity to the configuration and maintenance of the system.

In this configuration, we must use-- privileged=true to enable processes in the container to access other kernel functions such as / dev. Then, we also support other configurations on top of opening the OSD directory, and opening the OSD directory allows operators to prepare the device appropriately.

In this way, we can simply open the OSD directory and configure OSD (ceph-osd mkfs) through Entry Point. The configuration method I describe below is the simplest, because it only requires you to specify a block device, and the rest will be done by Entry Point.

If you don't want to use it-- privileged=true can use my second example.

$sudo docker run-d-net=host\

-- privileged=true\

-v / etc/ceph:/etc/ceph\

-v / var/lib/ceph/:/var/lib/ceph\

-v / dev/:/dev/\

-e OSD_DEVICE=/dev/vdd\

Ceph-daemon osd_ceph_disk

If you don't want to use-- privileged=true, you can also use your favorite configuration management tool to configure OSD manually.

In the following example, I assume that you have implemented partitioning and configured the file system. Run the following command to generate your OSD:

$sudo docker exec ceph osd create.

Then run your container:

Docker run-v / osds/1:/var/lib/ceph/osd/ceph-1-v / osds/2:/var/lib/ceph/osd/ceph-2

$sudo docker run-d-net=host\

-v / etc/ceph:/etc/ceph\

-v / var/lib/ceph/:/var/lib/ceph\

-v / osds/1:/var/lib/ceph/osd/ceph-1\

Ceph-daemon osd_disk_directory

The configurable options are as follows:

OSD_DEVICE I is an OSD device, for example: / dev/sdb

A device used by OSD_JOURNAL to store OSD journal, for example: / dev/sdz

HOSTNAME is the host running OSD (default is $(hostname))

OSD_FORCE_ZAP will force the established device content zapping (default is 0, set to 1 to enable)

OSD_JOURNAL_SIZE is the size of the OSD journal (default is 100)

Metadata server

The setup of this component is more intuitive. The only thing to note is that we can access the Ceph administrator key in Docker. This key will be used to generate CephFS pools and file systems.

If you run a version of Ceph prior to 0.87, you don't need to do this configuration, but we'd better run the latest version!

$sudo docker run-d-net=host\

-v / var/lib/ceph/:/var/lib/ceph\

-v / etc/ceph:/etc/ceph\

-e CEPHFS_CREATE=1\

Ceph-daemon mds

The configurable options are as follows:

MDS_NAME is the name of the Metadata server (default is mds-$ (hostname)).

CEPHFS_CREATE generates a file system for the Metadata server (default is 0, set to 1 to turn it on).

CEPHFS_NAME is the name of the Metadata file system (default is cephfs).

CEPHFS_DATA_POOL is the name of the Metadata server data pool (default is cephfs_data).

CEPHFS_DATA_POOL_PG is the number of placement groups for data pool (default is 8).

CEPHFS_DATA_POOL is the name of the Metadata server metadata pool (default is cephfs_metadata).

CEPHFS_METADATA_POOL_PG is the number of placement groups for metadata pool (default is 8).

RADOS gateway

When we deploy RADOS gateway, civetweb is enabled by default. Of course, we can also use different CGI front ends by specifying addresses and ports:

$sudo docker run-d-net=host\

-v / var/lib/ceph/:/var/lib/ceph\

-v / etc/ceph:/etc/ceph\

Ceph-daemon rgw

The configurable options are as follows:

RGW_REMOTE_CGI specifies whether to use an embedded web server (default is 0, set to 1 to turn it off).

RGW_REMOTE_CGI_HOST specifies the remote host on which the CGI process is running.

RGW_REMOTE_CGI_PORT is the remote host port that runs CGI.

RGW_CIVETWEB_PORT is the listening port for civetweb (default is 80).

RGW_NAME is the name of the RADOS gateway instance (default is $(hostname)).

Follow-up work

Backend configuration storage

In the default configuration, ceph.conf and all Ceph keys are generated during the startup phase of the monitoring module. This process assumes that you have to transfer these configurations to all nodes when you extend the cluster to multiple nodes. This operation is not flexible and we hope to improve it. One of the solutions I'm going to propose soon is to use Ansible to generate these configuration files and passwords and install them on all machines.

Another way is to store all configuration information on different back-end servers, such as etcd or consul.

Deployment management

The most intuitive solution is to use off-the-shelf ceph-ansible, although I still need to make some changes, but the main work is done. Another option is to use Kubernetes, whose preview version has been released.

Support for other container technologies such as Rocket

You don't need to do anything, because you can ship your Docker image directly to Rocket and run it.

This is the end of "how to run Ceph in Docker". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.