Shulou(Shulou.com)06/02 Report--

Original articles, welcome to reprint. Reprint please indicate: Reprint from IT people story meeting, thank you!

Link to the original address: "Advanced" docker's service arrangement to understand Mesos (22)

Mesos is an open source distributed resource management framework under Apache, which is called the kernel of distributed systems. Mesos was originally developed by AMPLab at the University of California, Berkeley, and has since become widely used on Twitter.

achieved initial

http://mesos.apache.org/

Run data (collections of data) in your data center as if it were running in a single resource pool

Mesos abstracts CPU, memory, disk, and other computer resources from physical or virtual machines, making fault-tolerant and scalable systems easier to build and simpler to run. If it is an old iron without foundation, it may be thought that it is to directly plug out the hardware of the server and rebuild a new machine. In fact, it is not like this. It extracts the required hardware equipment through software, unified scheduling, and rational use of these resources.

History Origin

When twitter first started, the site experienced explosive growth and was unstable. It is often destroyed by traffic. Every time the website hangs, the following picture will appear. If you look at it more, everyone will be more pleasing to the eye, resulting in this picture also becoming popular,"failure whale"! To address this failure whale education, twitter took inspiration from google's Borg system and developed a similar resource management system to help them get rid of the dreaded "failure whale"! Then they noticed a project called Mesos being developed by AMPLab at UC Berkeley, led by Ben Hindman, a PhD student at UC Berkeley. Ben Hindman later joined Twitter and was responsible for developing and deploying Mesos. Mesos now manages Twitter's app deployment on more than 30,000 servers. Finally Twitter open-sourced Mesos to apache.

version iteration

Keep updating a version of the frequency of a month, until today still maintain this frequency of the latest version: 1.7.0, which can also see confidence in the market!

In January 2013, version 0.12.1 was released, and more than 30 versions were experienced in the middle. In July 2016, version 1.0.0 was released. Two-level scheduling

Mesos is how to get twitter out of the whale of failure, see below

Mesos master is a daemon that supports high availability clusters. Mesos master is elected through zookeeper. Mesos slave manages all nodes. Mesos slave runs on physical machines or virtual machines. Each specific task or service is run. Each salve will register as mastermaster when it is started. Coordinate the slaves of each node and determine the available resources of each slave. The first level of scheduling Hadoop and Mpi, master scheduling slave. The components of the second level scheduling Framework, we see the dotted line in the figure below. The Framework includes two parts: scheduler and executor. Master can communicate with various frameworks. The scheduler shown above is also Hadoop and MPI. The actuator shown below runs in a slave.

Dispatching process

see the picture below

Slave1, that is, Agent1 tells the master that there are 4cpu, 4gb free resources. Mesos triggers the scheduling resource allocation module to request the allocation of 4cpu, 4gb, Framework1 requires the use of available resources, and then the master issues an offer describing all available resources in slave1. 4cpu, 4gb memory Framework scheduler tells master to run 2 tasks on slave, task1 (2cpu, 1gb)

task2 (1cpu, 2gb) Mesos issues tasks to slaves, and allocates appropriate resources to Franmework's executors, and then there are executors to start these 2 tasks, fw1, task1; fw2, task2;

At this time, slave1 still has 1 CPU left, 1 GB is not occupied, and can continue to be assigned to other tasks to run. In fact, the scheduler is to negotiate resources for Mesos. See how many resources you have. You need to run a program to see if there are enough resources. If there are enough, I will tell you that I want to run on that machine. Then tell the actuator to the master. The master tells the actuator to slave and execute it on the slave. The actuator can actually be understood as a piece of code. Code that can interface master and slave. In order to run multiple tasks in a slave, Mesos uses isolation modules, which use process isolation mechanisms to run these tasks. Mesos started using Linux isolation technology as early as 2009, and later Mesos added support for docker, you can use docker's own isolation mechanism, no matter what isolation mechanism you use, you need to implement all the packaging, and send it to the response slave, start on the slave.

Marathon

Mesos does not exist alone, there must be a Framework to cooperate with the existence, also know that mesos has a variety of frameworks responsible for running a variety of programs, Marathon suitable for long-term running projects (databases, application services, etc.), the following picture is mesos+Marathon and linux kernel comparison.

Process for integrating mesos marathon Load Balancer service discovery

Mesos characteristics

Powerful resource management

The core of Mesos ensures that all users in the cluster have equal access to resources, including memory, CPU, disk, and so on.

Separation of Kernel and Framework

Mesos is only responsible for the scheduling management of resources, various programs use Mesos resources, you can also develop your own Framework.

Low threshold, easy to use

The threshold is low compared to other service orchestration tools, the environment is relatively easy to build according to the basic documents will not encounter major problems, if you use long-term running services can use Marathon this service can be. Marathon environment is relatively easy to set up to start quickly completed. If you have a special situation you need to develop your own Framework

Congratulations, Old Tie. You won the prize. The threshold is too high!

Big factory use

Twitter founder 300,000 + services in use, apple cluster, youtube also used. Domestic iQiyi, digital technology are also used.

Marathon features

highly available

support cluster

Constraints

Label the machine, CPU high, memory high, hard disk good, and then give the specified label to the machine when the resource is offered.

Service Discovery and Load Balancer

Equivalent to a registry of services.

health check

When executing the scheduler, there are health check functions for machines, including three ways: http, tcp, shell commands, such as: web services to join http-based health check, access fixed pages, if the access is 200, the service is no problem. Stop restarting a service if it is found unhealthy several times in a row, which is not the case with 200.

event subscription

Start a service yourself, register an event subscription, and it will automatically push the event information of the subscription, including the service stopping, being killed, etc.

Complete REST API

A nice UI page, the API interface provides callers to view the status of the service. The running instance of each service, the state of each instance, can be integrated via scripts API.

PS: That's the general understanding of Mesos and Marathon.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.