Shulou(Shulou.com)06/03 Report--

Original text link

Author: wish William

Editor's note

Both Mesos and Yarn are excellent resource scheduling frameworks, and there are many people in the community who analyze the differences and usage scenarios between them. InfoQ has also asked questions about the relationship between the two before. At present, Mesos is widely used in the industry. The purpose of this article is to explain why the author chooses to use Yarn instead of Mesos, and introduces how to develop distributed programs based on Yarn. This article was first posted on William Zhu's blog and was reproduced and published by InfoQ under authorization.

Preface

In fact, I am not very familiar with Mesos, so some of the content may be biased, with personal preferences. People still need to have their own ability to distinguish.

Mesos and Yarn are both great, programmable frameworks. A hardware, can not be programmed, is dead, once it can be programmed to live, it can be a variety of twists and turns, there are a variety of ingenious ideas can be realized, similarly, a software, as long as it is programmable, basically alive, easy to form ecology.

Yarn VS Mesos

Let me first talk about why you chose Yarn over Mesos when working as a container scheduling engine.

Deployability

First of all, what we are discussing here is the deployment of Yarn or Mesos clusters, regardless of the applications on them. Apart from relying on JDK, Yarn has no dependence on the operating system and can basically run on it. Mesos may have library dependencies for installation and deployment because it was developed by CumberCraft +. I don't know if you take this seriously. Anyway, I take it very seriously. The software should be able to Run as soon as possible. So I developed my own Java services framework in 12 years, and then I just ran a main method. It is too complex and counterintuitive to let applications include containers rather than throwing them into Tomcat containers.

Secondary development

To Java/Scala engineers, Yarn is just a Jar package, similar to the index development kit Lucene, which you can bring into the project and do any wrapper you want. This is one of them.

Second, Yarn provides a lot of extension interfaces, and many implementations are pluggable. Alternatively, in the XML configuration, you can easily replace the original implementation with your implementation, without much intrusiveness, so even if you upgrade Yarn in the future, it won't be a big problem.

In comparison, Mesos is more like a ready-made product, which can be used directly when deployed, but is not friendly to secondary development.

Ecological advantage

Yarn was born in Hadoop, the "initiator" project of big data, so it has an inherent advantage in the field of big data.

The underlying layer is naturally distributed storage system HDFS, which is stable and efficient.

It has supported the top seat in the field of big data, such as Spark and MR, and has been tested for a long time.

The community is strong, recent releases are significantly faster, and support for long tasks is getting better and better.

Long task support

When it comes to support for long tasks (long running services), some people think that earlier Yarn was designed to support offline short-term tasks, so support for long tasks may be limited. In fact, there is no need to worry, for example, based on the fact that the Spark Streaming on it runs on 7x24 hours, there is no problem with running. In general, to support long tasks, you need to consider the following:

Fault tolerance, mainly AM fault tolerance.

Yarn Security, if the security mechanism is enabled, the expiration time of tokens and so on should also be paid attention to.

Logs are collected to the cluster.

Then there is resource isolation and priority. If the resource isolation is too poor, it will have an impact on long-term tasks.

If you are interested, you can refer to Jira first. I think this Jira began 13 years ago, which shows that this matter has been taken seriously a long time ago. The following points mentioned by our team will be explained.

Fault tolerance

Yarn itself is highly available. Currently, HA has been implemented in Yarn's Master.

AM fault tolerance, I think the keep containers across attempt option has been supported since version 2.4 (look at the source code, it may have been supported by earlier versions). What do you mean? That is, if AM dies, all containers managed by AM will be maintained without being killed during Yarn's restart of AM. Unless Yarn tries many times, there is no way to restart AM (the default is twice). This shows that from the perspective of underlying scheduling, it has been done very well.

Log collection to the cluster

Log collection has been collected while running in version 2.6.

Resource isolation

Resource isolation, Yarn does not do well, currently effective is memory, has always wanted to do support for other aspects, but has been limited. This is probably the reason why many people choose Mesos finally. But now the advantage of Mesos is gone, because the Docker container has done a good job of isolating resources. As soon as Yarn and Docker are integrated, they complement each other.

Summary

Mesos and Yarn are excellent scheduling frameworks, each has its own advantages and disadvantages. Flexible scheduling and unified resource management is a trend of the future platform. Similar resource management and scheduling framework will be popular.

Some common misunderstandings

Born in Hadoop, inherited his halo and ecology, but this will also bring some confusion, first of all, the halo has been covered by Hadoop, and because of the inherent inertia, people will take it for granted that Yarn is just a component in Hadoop, will anyone ever think of taking Yarn out and using it alone?

However, as I repeatedly emphasized in a previous course, Hadoop is a collection of software that consists of three parts: distributed storage, resource management and scheduling, and computing framework. There is no inevitable relationship between them, and they can come independently. Yarn is a resource management and scheduling engine, and its initial design goal is for general purpose, not just to run MR. Now there are a lot of services based on Yarn, such as Spark.

There is another misunderstanding. MR is basically synonymous with offline batch at present, which makes people mistakenly think that Yarn is only suitable for batch offline task scheduling. In fact, I have given the analysis above, Yarn is completely able to ensure the stable and reliable operation of long tasks.

How to develop distributed programs based on Yarn

This article won't teach you exactly how to use Yarn's API, but if you want to know Yarn's API, but think the official documentation is too brief, I can give you two suggestions:

Code usage examples can be found in Spark Yarn-related code. It can be regarded as a very compact Yarn adaptor.

Buying a book about Yarn and understanding its architecture can also help you understand the design of its API.

The following two topics will be discussed:

Some preparatory work to be done in developing distributed programs based on Yarn

Some basic ideas of developing Container scheduling system based on Yarn

Some preparatory work to be done in developing distributed programs based on Yarn

You can't roll up your sleeves and start working. What do we need to know before we start?

The native API of Yarn is too low-level and complex.

If you want to happily develop Yarn applications, it is necessary to encapsulate Yarn's API. Yarn in order to be flexible, or to meet most of the needs of developers, the underlying interaction API appears to be more primitive. Naturally, it is very difficult to develop. I am not the only one who thinks that the current Twill of Apache and the Adaptor layer of Hulu when they were developed are actually designed to solve this problem. Then why didn't I use Twill? the first is that there are too few documents, and the second is that it is a little complicated. I don't need something so complicated. In my opinion, instead of developing so many functions, Twill is really better to write documents.

It is best to develop a Framework that can solve a kind of problem.

Yarn is just an underlying resource management and scheduling engine. Generally, you need to develop a set of Framework based on the above to solve specific problems. Take Spark as an example, he solves some problems related to distributed computing. The container scheduler I developed is actually to solve the problem of dynamic deployment of Web applications. On top of them, it is your application. For example, if you want to count logs, all you have to do is develop an Application on Spark. For example, if you want to provide a recommendation system, you can be scheduled and deployed by the container scheduler as long as you are packaged in a container.

So generally speaking, Yarn-based distributed applications should conform to this level:

Yarn-> Adapter-> Framework-> Application

Adapter is the first thing I said, you encapsulated the API of Yarn yourself. Framework is the programming framework for solving a class of problems, and Application is the system you really want to solve the business. Through this decoupling, each level only needs to focus on its own core function points.

Make sure that your upper Framework/Application can be transplanted.

Spark is a typical example, he can run on Mesos, can also run on Yarn, can also run on himself (Standalone), real-time, soak in Yarn, and run Standalone mode, there are a lot of people. This benefits from the fact that Spark itself does not rely on the underlying resource management and scheduling engine.

This is actually the benefit of the two items I mentioned above, because with Adaptor, the upper Framework does not have to be tied to a resource scheduling engine. On the other hand, Framework allows Applicaiton to pay no attention to the underlying scheduling, but only to the business.

In addition, on the Framework you have worked so hard to develop, you naturally hope that it can run on more platforms and meet the needs of more people, right?

Some basic ideas of developing Container scheduling system based on Yarn

First of all, we need to understand two concepts:

Dumb application. The so-called dumb application means that it is unable to interact directly with the distributed system, and the distributed system can only control the life cycle through the container, such as closed or open applications. Typical basic applications such as MySQL, Nginx and so on. They generally have their own unique ways of interaction, such as the command line or the socket protocol or the HTTP protocol.

Accompanying components. Because of the existence of dumb applications, distributed systems need an agent in order to interact with these applications. The dumb application of this agent and the agent has the same life cycle. For example, after a service is shut down, the event will be known by the distributed system, and the distributed system will send the event to the accompanying components of Nginx, which will be converted into instructions that Nginx can recognize and remove the stopped services from the ProxyBackend list of Nginx.

In the container scheduling system, if the NodeManager of Yarn manages the Docker directly, it needs to be supported by Yarn itself, which I don't think is appropriate. The responsibility of Yarn is to do a good job of resource management, allocation, scheduling, and does not need to be coupled with a specific technology. After all, Yarn is a general resource scheduling management framework.

Based on the above theory, we develop a framework based on Yarn, which will be a typical master-slave structure (this is determined by Yarn). The slaves of this framework is actually the companion object of Docker. Master can manage containers indirectly through these Slave.

Let's briefly describe their process:

Users submit Application and apply for resources

Yarn starts master of Framework

Yarn starts slave of Framework

The slave connects to the master and sends a heartbeat so that the master knows the condition of the slave. The slave starts the Docker,slave and corresponds to the docker container that is started one by one.

Slave regularly monitors Container

Slave found that container crash,slave automatically exited, and Yarn was notified to withdraw resources.

Master finds that a node has failed, issues a new node request, and restarts slave on another server, repeating the steps starting at 2.

There is another problem. If slave is killed normally, you can turn off Container by the way through JVM ShudownHook. But if the slave is dropped by kill-9 or an exception crash, it may lead to resource leakage. At present, this information is reported by master to the cluster management platform, which will be cleaned regularly. You can also store this information, such as putting it in Redis or MySQL, and then start the background cleanup task.

After understanding this idea, the specific implementation becomes simple, that is, to develop a master-slave program based on Yarn, and then slave to manage the corresponding Docker container, including accepting new instructions. Master provides a management interface to display container information, and the running status is fine.

Of course, you can also develop a set of Framework B specifically to interact with Nginx, so that, for example, the above system makes node changes and notifies B's master, and then B's master completes the Nginx update through its own companion component Slave, thus realizing the automatic change and notification of back-end services.

Now, does this concept perfectly cover the interaction between applications?

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.