Shulou(Shulou.com)06/01 Report--

This article introduces the knowledge of "how to use Mesos and Marathon to manage Docker clusters". Many people will encounter this dilemma in the operation of actual cases, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Distributed systems are difficult to understand, design, build and manage. They introduce more variables into the design than a single machine, making it more difficult to find the root causes of the application. SLA (service level agreement) is a measure of downtime and / or performance degradation, and most modern applications have a desired level of resilient SLA, usually increasing by the number of "9" (for example, 99.9 or 99.99% availability per month). Each extra 9 is becoming more and more difficult to achieve.

Distributed systems are usually divided into static partitions, such as Akka/Play, Spark/Hadoop, Storm, and Redis. The disadvantage of static partitioning is the increase of complexity. With the increase of the number of machines, software management becomes more and more complex, and failure management and maintenance become more and more difficult. And resource consumption is very uneconomical. The following figure shows the resource utilization under static partitions:

Apache Mesos can run multiple distributed system types on the same cluster machine, which makes it more dynamic and efficient to share resources. Provide failure detection, task release, task tracking, task monitoring, low-level resource management and fine-grained resource sharing, which can be scaled to thousands of nodes. Mesos has been used by Twitter to manage their data centers.

The Mesos architecture diagram is as follows:

The Mesos framework is an application that runs distributed applications on Mesos. It has two components:

Scheduler: interact with Mesos, subscribe to resources, and then load tasks from the server in mesos.

Executor: get the information from the framework's environment variable configuration and run the task in the mesos slave server.

Let's take a look at how it implements resource calls. Mesos allocates resources through "resources offers", which is actually a snapshot of the currently available resources that the scheduler will use to run tasks on the mesos slave server.

The sequence diagram of Mesos master-slave server scheduling resources is as follows:

First of all, the Mesos master server queries the available resources to the scheduler, the second step dispatches the loading task to the master server, and the master server transmits the load task to the slave server, and the slave server commands the load task to execute. After the executor executes the task, the status feedback is reported to the slave server, and finally the scheduler is informed.

Multiple executors are managed from the server, each of which is a container. Previously, you could use the Linux container LXC, but now you can use the Docker container.

Failed recovery and high availability

The Mesos master server uses Zookeeper for service election and discovery. It has a registry that records all running any and slave server information, using MultiPaxos for log replication to achieve consistency.

Mesos has a recovery mechanism from the server. Whenever a slave server crashes, the user's task can continue to run. The server persists some key information such as task information status updates to the local disk, and these tasks can be resumed from the disk when restarted (similar to passivation and wake-up in Java).

What is Marathon?

It is a mesos framework that supports running long services, such as web applications. Is the distributed Init.d of the cluster, can run any Linux binary release version as is, such as Tomcat Play, etc., and can be managed by multiple processes of the cluster. It is also a private Pass that implements service discovery, provides REST API services for deployment, has authorization and SSL, configuration constraints, and implements service discovery and load balancing through HAProxy.

In this way, we can manage thousands of servers like a Linux host. Their corresponding principles are shown below. They are managed using shells such as Marathon, which is similar to init Systemd in a Linux host, while Mesos contains not only one Linux core, but also a Linux core that can schedule thousands of servers, which is actually the core of a data center:

Installation configuration

First, we need to set up the Mesos cluster environment, so let's set up Mesos master/slave and Zookeeper on the Ubuntu 14.04 vagrant node. Installation dependencies:

$apt-get install curl python-setuptools python-pip python-dev python-protobuf

Install zookeeper:

$apt-get install zookeeperd

After installation, Zookeeper has a configuration, and each Zookeeper needs to know its location in quorum.

$echo 1 | sudo dd of=/var/lib/zookeeper/myid

Set the Docker as follows:

$echo "deb http://get.docker.io/ubuntu docker main" > / etc/apt/sources.list.d/docker.list

$apt-get update & & apt-get install lxc-docker

$docker version

Client version: 1.0.0 Client API version: 1.12 Go version (client): go1.2.1 Git commit (client): 63fe64c Server version: 1.0.0 Server API version: 1.12 Go version (server): go1.2.1 Git commit (server): 63fe64c

Pull a ubuntu image from Docker Hub

$docker pull libmesos/ubuntu

Configure Mesos:

$curl-fL http://downloads.mesosphere.io/master/ubuntu/14.04/mesos_0.19.0~ubuntu14.04%2B1_amd64.deb-o / tmp/mesos.deb$ dpkg-I / tmp/mesos.deb$ mkdir-p / etc/mesos-master$ echo in_memory | sudo dd of=/etc/mesos-master/registry## Mesos Python egg for use in authoring frameworks$ curl-fL http://downloads.mesosphere.io/master/ubuntu/14.04/mesos-0.19.0_rc2- Py2.7-linux-x86_64.egg-o / tmp/mesos.egg$ easy_install / tmp/mesos.egg

Download Marathon:

$tar xvzf marathon-0.6.0.tgz

Mesos manages Docker through Deimos and installs Deimos through pip:

$pip install deimos

Configure mesos to use Deimos

$mkdir-p / etc/mesos-slave$ echo / usr/local/bin/deimos | sudo dd of=/etc/mesos-slave/containerizer_path$ echo external | sudo dd of=/etc/mesos-slave/isolation

Start all services:

$initctl reload-configuration

$service docker start

$service zookeeper start

$service mesos-master start

$service mesos-slave start

# Starting Marathon #

$cd marathon-0.6.0

$. / bin/start-master zk://localhost:2181/mesos-zk_hosts localhost:2181

Marathon has started listening on port 8080, and we can access it through the browser:

Curl localhost:8080/help # gives us some details about the API's

Start the container

We start the REST API submission of a container through Marathon as follows:

Curl-X POST-H "Accept: application/json"-H "Content-Type: application/json"\

Localhost:8080/v2/apps-d'{

"container": {"image": "docker:///libmesos/ubuntu", "options": ["--privileged"]}

"cpus": 0.5

"cmd": "sleep 500"

"id": "docker-tester"

"instances": 1

"mem": 300

}'

By submitting the custom Docker by selecting "options" after the curl command, we can check the syslog because the mesos default log is placed in the syslog.

Jun 27 07:24:58 vagrant-ubuntu-trusty-64 deimos [19227]: deimos.containerizer.docker.launch () exit 0 / / docker run-- sig-proxy-- rm-- cidfile / tmp/deimos/mesos/00d459fb-22ca-4af7-9a97-ef8a510905f2/cid-w / tmp/mesos-sandbox-v / tmp/deimos/mesos/00d459fb-22ca-4af7-9a97-ef8a510905f2/fs:/tmp/mesos-sandbox-- privileged-p 31498 Race 31498-c 512-m 300m-e PORT=31498-e PORT0=31498-e PORTS=31498 libmesos/ubuntu sh-c 'sleep 500'

We can also check the status of our startup task through Marathon Rest API:

Curl-X GET-H "Content-Type: application/json" localhost:8080/v2/apps

The following is a screenshot of the information obtained by Marathon UI:

For more information about Docker tasks, you can use Mesos GUI on port 5050 of the Mesos master server. Now let's test the scalability of the task. For example, we need to join more node servers. There are two ways to use GUI or by issuing a put request.

Curl-X PUT-H "Content-Type: application/json" localhost:8080/v2/apps/docker-tester\

"container": {"image": "docker:///libmesos/ubuntu", "options": ["--privileged"]}

"cpus": 0.5

"cmd": "sleep 500"

"id": "docker-tester"

"instances": 2, # increasing the instance count to 2

"mem": 300

}'

Marathon ensures that all docker processes are up and running, and if a process crashes, Marathon restarts the same process to ensure that each configuration runs an instance, as well as other open source Mesos schedulers such as Apache Aurora, Airbnb's Chronos. But marathon is relatively straightforward to provide good REST API for managing containers. Although Mesos, Marathon and Docker are still young, they provide a killer cluster management combination based on Docker.

That's all for "how to use Mesos and Marathon to manage Docker clusters". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.