Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "how to use Mesos to build a multi-task scheduling system". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Background

The company's internal cloud platform provides a large number of physical machines and virtual machines for various business lines to run business services. according to statistics, these machines allocated to the business, such as cpu, memory and other resources are not fully utilized.

If we can make full use of the idle resources on these machines while ensuring the normal operation of business services, a lot of machine resources will be saved.

Type selection

When it comes to multitasking and scheduling, most people may first think of Kubernetes (K8s) + Docker, which runs like a breeze. However, most of our business machines are centos 6.2and linux kernel 2.6environment, and docker requires Linux kernel version 3.10 + to run.

(please refer to https://docs.docker.com/engine/faq/#how-do-i-connect-docker-containers)

Because you want to take advantage of the machine that the business is using without affecting the existing services that are already running, you can't upgrade the kernel or restart the machine. Obviously, K8s can't get through.

Fortunately, the world always offers us a variety of choices, in addition to Kubernetes (K8s) + Docker, we also have mesos

Introduction to Mesos

Put it on the official website first, which has very detailed instructions.

Http://mesos.apache.org/

To put it simply, Mesos is the operating system used in the entire computing center. It uniformly manages cpu, memory, disk, network and other computing resources of all machines in the computing center, allocates resources according to task requirements, schedules tasks, supports failover, and so on.

Mesos characteristics

The most important feature of Mesos is two-level resource scheduling, as shown below:

The architecture diagram above is briefly described as follows:

Each Agent reports its own computing resources to Master

Master sends resource offer to each second-level scheduling framework Framework

Framework matches the task waiting for scheduling with the received resource offer and feeds it back to Master

Master sends the task and resource offer of the corresponding Framework feedback to the corresponding Agent

Agent uses Executor to run task and limit the use of resources

You can run various Framework on Mesos, such as Spark, Storm, Hadoop, Marathon, etc.

Mesos system architecture

Official documents:

Http://mesos.apache.org/documentation/latest/architecture/;

For task isolation, Mesos not only supports docker container technology, but also provides its own Mesos Containerizer, which is exactly what we need. In fact, Mesos Containerizer also uses Linux Cgroup for resource restriction and Linux namespace for resource isolation.

Mesos Containerizer:

Http://mesos.apache.org/documentation/latest/mesos-containerizer/

Face the challenge

Several problems that need to be solved in our multitasking scheduling system

Mesos agent needs to be deployed non-intrusively on business machines and cannot contaminate the environment of the deployed machines.

Real-time monitoring and adjustment of computing resources available to Mesos Agent

Rapid deployment and resource isolation of Task

Monitoring of the overall operation of the cluster

Overall architecture of multi-task scheduling system

Architecture design diagram

Brief introduction of each component:

1.1Principal or Mesos master + Mesos agent

1.2 the second-level scheduling framework uses Marathon

1.3 simultaneously deploy monitor on machines where Mesos agent is deployed to monitor and adjust the available computing resources of Agent in real time

The running process of the system, in the order marked in the figure above.

2.1 the Monitor real-time monitoring component collects available resources on the host machine

2.2 Monitor dynamically adjusts the reserved resources of agent according to the available resources on the machine

2.3 Agent dynamically and real-time reports its own reserved resources to Mesos master

2.4 Mesos Master is sent to Marathon in resource offer

2.5 Marathon allocates resources according to the received resource offer and the task that needs to be run, and feeds back the allocation result to Mesos Master

2.6Mesos Master assigns task to a specific Agent for execution

Non-intrusive deployment of Mesos agent on business machines

We use version 1.4.1 of Mesos, which is written in Category 11. The Mesos project itself is very large, and there must be a lot of dependent libraries, so we should bear the brunt of solving these operational dependency problems.

The deployment principle is not to change, not to pollute the deployed machine environment. For libstdc++ and some other so, we will not install it to a system directory such as / usr/local/lib. Instead, we will use the method of dynamically changing the rpath of the executable program during packaging to load the corresponding so library from our installation directory at runtime.

We centralize all the lib files needed for mesos to run in the libs directory

The compiled mesos executable file, use patchelf to update the rpath path, and point to our own libs directory

Patchelf is a tool that can modify some attribute values in its elf file structure without affecting the execution of executable files. For more information, please see: https://nixos.org/patchelf.html

After this deployment, Mesos agent is just a separate directory. To uninstall, just stop the process and delete the directory.

When it comes to the compilation process, although it has been described in more detail on the official website, there are still some problems encountered in the compilation process:

Official website compilation steps: please refer to

Http://mesos.apache.org/documentation/latest/building/;

The official requirement for gcc is 4.8.1 cycles. I used gcc 5.4, which I recompiled on centos 6.2, which is Linux 2.6.32.

By default, the binding of java language is compiled, but the jar package of protobuffer is missing when compiling "Build and attach javadoc". The solution: modify src/java/mesos.pom.in, find ``first, and add true under it. This will not compile the javdoc, but does not affect the use of java binding.

Real-time monitoring and adjustment of computing resources available to Agent

The Monitor program is developed and deployed on the business machine together with Agent to periodically monitor the resources available on the machine and the resources used by the Agent itself.

Mesos provides us with the super-practical feature of dynamic resource reservation, which can limit the computing resources currently available to Agent for a certain Role:

The monitoring and evaluation results of Monitor are the computing resources currently available to Agent.

I thought this would be the end of the problem. During the test, I found that Agent could not adjust the dynamic resource reservation online in real time. We need to update the dynamic resources that can be used in the configuration file, and then restart Agent.

Restarting Agent is unbearable, so we modified the source code and adjusted the dynamic resource reservation online through the http API. In fact, this part is not difficult. The definition of the mesos http API is very clear.

Rapid deployment and resource isolation of Task

At present, we use Marathon to deploy task, which is easy to use and functional. If we need a more flexible adjustment strategy, we may need to exploit our own framework or re-develop based on a framework.

As a matter of fact, task is important, and it is urgent, and the amount of resources taken up is also different. For important urgent tasks, in order to ensure that the tasks run better, we will use Mesos attribute to schedule tasks so that specific tasks only run on the agent with a specific attributes, which requires setting the corresponding attributes for each mesos agent.

Encountered the same problem, mesos can not dynamically adjust attributes online:-(, in fact, it is relatively simple, slightly comb the structure of mesos source code, it is not difficult to change

There is another problem. Attributes is dynamically adjusted. What if agent is restarted? For this purpose, we deploy an etcd cluster to manage. Each agent is a node on the etcd. Update its attribrtes through the http interface provided by etcd, and the agent will periodically synchronize from the etcd; at the same time, the attributes information on each agent can be easily obtained from the etcd

If you directly manipulate etcd and delete the attribute on an agent, the tasks that depend on this attribute deployment will be automatically dispatched away and will no longer run on this agent.

For the problem of task isolation, both cpu and memory,mesos are accomplished through cgroup. For the restriction of cpu, we use cfs method, as long as we need to judge whether the current kernel supports it or not. For the limitation of disk, mesos currently uses the periodic detection method of the du command.

There are two ways for mesos to restrict cpu:

Cpu shared mode: there is no strict restriction on cpu in this way. Any task on the machine can access all cpu resources on the machine. For example, your limited cpu usage resource is 2. In this way, you may use 4, 6 or more.

Cpu CFS method: it is equivalent to configuring exclusive cpu, for example, if cpu is configured as 1, then the cpu utilization of this task will not exceed 100%, which is equivalent to setting a hard limit.

In most of our environments, due to the limited kernel version, mount namespace does not support it, so we use rootfs + chroot for simple resource isolation.

We have customized several versions of rootfs. Task packaging is a process of combining the dependencies of the task itself with the corresponding rootfs, which can be packaged and stored on storage such as S3 for marathon to call when deploying the task.

Here, let's talk about the json configuration of a task deployment in marathon. Let's take a look at this configuration first. I'll write the instructions directly in it.

This is the end of the content of "how to use Mesos to build a multi-task scheduling system". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.