Shulou(Shulou.com)05/31 Report--

In this issue, the editor will bring you about how to understand Mesos. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

I have heard many people talking about Mesos, but they do not quite understand what problems Mesos can solve and what the use scenario is. Zhou Weitao (one of the earlier domestic developers who contacted the use of Docker,Mesos and other technologies) described it in one sentence. Mesos can manage the CPU, memory and other resources of each machine, allowing you to manipulate the entire data center like a single resource pool.

Apache Mesos is a cluster manager that provides effective resource isolation and sharing across distributed applications or frameworks, and can run Hadoop, MPI, Hypertable, Spark.

13 questions will take you to learn more about Mesos

(the question comes from the 100th master Q & An of the OSChina open source Chinese community-Apache Mesos)

Q1: most people still don't know what Mesos is. Please tell me what it does, what's its use, and how to use it?

A1: Hello, although there is not much information about Mesos in China at present, there are still some information about Baidu and Google. Here, I would like to take an example to explain Mesos. Suppose a company needs to do big data calculation frequently, and the task requires more than N CPU and memory to run. In order to meet this requirement, we have two ways of thinking:

Idea 1) using a minicomputer, a single computer can provide sufficient resources for the task

Idea 2) distributed computing, that is, to provide a group of commonly configured machines (computing nodes), that is, clusters, split the computing tasks into each machine, and then summarize the results.

The second idea is the current popular practice, the advantages of this approach are no longer said. In order to meet the requirements of idea 2, we need to establish a data center (cluster). Furthermore, in order to make full use of the resources of the data center (cluster) (such as assigning different resources to different tasks, allocating resources according to task priority, etc.), we need a tool to manage, allocate, etc., the entire data center resources, this tool is Mesos. Other tools similar to Mesos are YARN.

In addition, Mesos is not only an Offer resource for computing tasks, it also supports running long-term tasks (such as Web applications). At present, many foreign Internet companies are using Mesos as their cluster management tool. Here is a Powered by Mesos list: https://mesos.apache.org/documentation/latest/powered-by-mesos/

Q2: let's use Cloudera now. Can you briefly introduce the difference between Mesos and Cloudera?

The main goal of A2:Mesos is to help manage cluster resources between different frameworks (or application stacks). For example, there is a business that needs to run both Hadoop,Storm and Spark on the same physical cluster. In this case, the existing scheduler cannot achieve such fine-grained resource sharing across frameworks. Hadoop's YARN scheduler is a central scheduler that allows multiple frameworks to run in a cluster.

However, it becomes difficult to use a framework-specific algorithm or scheduling strategy, because there is only one scheduling algorithm among multiple frameworks. For example, MPI uses a group scheduling algorithm, while Spark uses delayed scheduling. The two of them running on the same cluster at the same time will lead to the conflict between supply and demand. Another way is to physically split the cluster into multiple small clusters, and then run different frameworks on these small clusters independently. Another way is to assign a set of virtual machines to each framework. As Regola and Ducom say, virtualization is considered a performance bottleneck, especially in high-performance computing (HPC) systems. This is the scenario where Mesos fits-- it allows users to manage cluster resources across frameworks.

Mesos is a double-layer scheduler. In the first layer, Mesos provides certain resources (in the form of containers) to the corresponding framework. After receiving the resources at the second layer, the framework runs its own scheduling algorithm to assign tasks to the resources provided by Mesos. Compared with Hadoop YARN's central scheduler, it may not be so efficient in terms of cluster resource usage. But it brings flexibility-for example, multiple framework instances can run in a cluster. This is what none of these existing schedulers can achieve. Even Hadoop YARN is just trying to support third-party frameworks like MPI on the same cluster. More importantly, with the birth of new frameworks, such as Samza, which has recently been opened up by LinkedIn-with Mesos, these new frameworks can be deployed to existing clusters and coexist peacefully with other frameworks.

Q3: Hello, what are the typical application scenarios of Mesos? Read some introductions, said that can do Docker choreography service. What is the difference between cloud platform such as OpenStack managing physical machine CPU and memory, and Cloudera Manager managing Hadoop cluster service?

A3: now there are many Mesos application scenarios, such as 1) Spark on Mesos (this is standard) 2) Jenkins on Mesos

3) Mesos does docker choreography service and so on.

Compared with OpenStack, first of all, physical machines and virtual machines can be used as cluster nodes of Mesos; secondly, the granularity is different: the basic computing unit of Mesos is LXC, while VM of OpenStack (I heard that Docker container technology is also supported now), the former has higher resource utilization; finally, lightweight, Mesos is only responsible for Offer resources to Framework, not for scheduling resources. OpenStack is closer to the IaaS layer, while Mesos is on top of IaaS. So some people call it DCOS, or distributed operating system.

Q4: where are the boundaries in all aspects and what are the advantages and disadvantages? thank you.

A4: benefits

Resource management strategy Dominant Resource Fairness (DRF), which is the core of Mesos and the fundamental reason why we compare Mesos to distributed system Kernel. Generally speaking, Mesos can ensure that all users in the cluster have equal opportunities to use the resources in the cluster, including CPU, memory, disks and so on. Many people compare Mesos with K8s. I don't know much about K8s, but I don't think we can compare them with different priorities. K8s is only responsible for container orchestration rather than cluster resource management. Just because both can manage Docker, we can't confuse them.

Lightweight. As opposed to YARN,Mesos, it is only responsible for Offer resources to Framework, not for scheduling resources. In this way, in theory, we can make all kinds of things use Mesos cluster resources, instead of YARN sticking to Hadoop only, what we need to do is to develop a Mesos Framework.

Improve the resource utilization of distributed clusters: this is an advantage of Generic. In some ways, all cluster management tools are designed to improve resource utilization. The emergence of VM gave birth to the emergence of IaaS; containers, gave birth to K8s, Mesos and so on. To put it simply, with the same amount of resources, we use IaaS to split them into VM and K8s/Mesos to split them into containers, obviously the latter has higher resource utilization. (I'm not talking about security here, and let's assume that the internal subnet environment doesn't need to consider this.)

Shortcoming

The threshold is too high. You can't do anything with just one set of Mesos. In order to use it, you need different Mesos Framework, like Marathon,Chronos,Spark, and so on. Or write their own Framework to dispatch the resources given by Mesos, which makes everyone prohibitive.

There is not enough support for Stateful Service at present. The Mesos cluster is currently unable to persist data. With the addition of Persistent resource and Dynamic reserver in version 0.23, the data persistence problem will be improved.

There will be no less dirty work. Team was optimistic in the early days of using Mesos, thinking that once Mesos was done, our operation and maintenance classmates would be much more relaxed. However, this is not the case at all. Mesos will not help you with the optimization of cluster nodes, disk, network settings, and so on. At the initial stage of use, the workload of operation and maintenance has not been reduced, but has become heavier. The Mesos project is still in full swing, and many functions are not perfect.

Q5: I would like to ask how to select Mesos and Kubernates if you want to build a cloud service platform.

A5: the current situation is that the biosphere of Mesos and K8s are both developing well, and it is very difficult to discard either one. You might as well choose one and use it first according to your personal preference. For example, it is too convenient for several people to deploy directly with one click. You can quickly experience the benefits of Mesos.

It depends on your specific needs. As far as I know, K8s currently only supports Docker and has few use cases in a production environment; Mesos does not need your application package in Docker and it has been tested in a production environment. However, in turn, the community of K8s is more active, it is developing at a high speed, and the prospect is very good. Of course, none of the above is the key, a good cloud platform is more about having a good product concept. Please refer to the number of people.

Q6: is there a good scheduler algorithm or strategy for long-term tasks

A6: long tasks rely on the marathon Marathon framework, which is basically the most mature distributed running framework for Docker,Mesos + Marathon. Long tasks rely on the marathon Marathon framework, which is basically the most mature distributed running framework for Docker,Mesos + Marathon.

Q7: with the combination of Mesos and Docker, Mesos can only solve the problem of resource allocation, right?

A7: yes, Mesos is responsible for resource allocation, and you need something to schedule tasks for Docker, so you can automatically send Docker instances to the cluster to run. This component is called Marathon Marathon. Mesos + Marathon is basically the most stable Docker cluster scheduling framework now.

Can Q8:Mesos be gradually applied to production environments now?

A8:Mesos can be applied to the production environment for a long time. Foreign Airbnb, Apple, Uber, Twitter, domestic Ctrip, iqiyi, and several technologies of our company all use Mesos in the production environment. You can see the list of https://mesos.apache.org/documentation/latest/powered-by-mesos/ using Mesos here.

Is there any connection between Q9:Mesos and Zookeeper?

A9:Zookeeper is a software that provides consistent services for distributed applications, while Mesos is a distributed application. So in a production environment, we need to use Zookeeper to provide consistent services for Mesos.

Q10 is there any competition between Q10 Ranges Mesostemps and Kubernetes? Although the three families all say they support each other, is it too verbose to do so?

There are many crossover between A10:Swarm and K8s. Mesos is more about Focus in resource management, but just happens to be able to use Container for resource isolation. Whether it is competitive or not depends on the trend of the community.

Q11: Hello, after looking at this framework, I'd like to ask you a few questions:

1. Does this framework have its own log collection module? two。 Can this framework perform performance statistics? 3. Can this framework switch automatically when a node runs out of resources? Is it easy to crash if all nodes run out of resources, and what is the self-recovery ability? 4. Can this framework be configured with load balancing? Thank you:)

A11:

With log module, but the function is relatively simple, without a global display

Performance statistics can be performed.

Mesos determines how many resources are allocated to the scheduler according to the current cluster resource statistics. Resource exhaustion will only prevent new applications from being deployed and will not affect what is running.

Load balancing can be configured. And Mesos itself has a multi-Master mechanism.

Q12: how does Mesos decide how many resources to allocate? When will the allocated resources be recycled?

A12:Mesos is different from other cluster management tools, Mesos itself is not responsible for allocating resources, it only provides the remaining resources of the current cluster to the scheduler that registers with it, and it is up to the scheduler itself to decide how many resources to use and to release resources appropriately.

Q13: suppose there are three servers in the cluster, and each server has 16 GB of available memory. Now the scheduler needs 24 GB of memory to run a task. Will Mesos provide 48 GB of memory for the entire cluster as a whole, or will it provide the scheduler with the remaining memory of each server? that is, which of the following two situations is correct:

1. The scheduler first applies for 16 gigabytes of memory of node 1, and then applies for 8 gigabytes of memory of node 2. Which node's memory is completely controlled by the scheduler.

two。 The scheduler requests 24 gigabytes of memory at a time, and the Mesos controls which node's memory is used. It is possible that each node is assigned 8G, or it may be 16G for one node and 8G for the other node.

A13: I've seen the DPark scheduler that implements Mesos. You need 24 gigabytes of memory for a task, which needs to be split before it can be scheduled. Each small task requires less than 16GB of memory. Can be dispatched to a specific server through the scheduler. Schedulers generally schedule tasks to the machine where the file is located. The scheduler controls where resources are used, and Mesos tells the scheduler which resources are available.

This is how the Mesos shared by the editor is understood. If you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.