Shulou(Shulou.com)06/01 Report--

This article shows you how to simply write yarn app, the content is concise and easy to understand, can definitely make your eyes bright, through the detailed introduction of this article, I hope you can get something.

Although YARN's native programming API has been greatly simplified, it is still very difficult to develop a YARN application from scratch. To write an application on YARN, you need to develop Client and ApplicationMaster modules, and understand a number of API and parameter lists of several protocols involved, in which ApplicationMaster is also responsible for resource application, task scheduling, fault tolerance, etc., in short, the whole process is very complex.

Apache Twill (http://twill.apache.org/index.html)) is a project set up to simplify application development on YARN, which encapsulates repetitive work related to YARN into libraries that allow users to focus on their own application logic.

The following code example uses Apache Twill to develop a helloworld program that runs on YARN:

Public class HelloWorld {

Static Logger LOG = LoggerFactory.getLogger (HelloWorld.class)

Static class HelloWorldRunnable extends AbstractTwillRunnable {

@ Override

Public void run () {

LOG.info ("Hello World")

}

}

Public static void main (String [] args) throws Exception {

YarnConfiguration conf = new YarnConfiguration ()

TwillRunnerService runner = new YarnTwillRunnerService (conf, "localhost:2181")

Runner.startAndWait ()

TwillController controller = runner.prepare (new HelloWorldRunnable ()) 
 .start ()

Services.getCompletionFuture (controller) .get ()

}

What Twill does is develop a general implementation of Client and ApplicationMaster, and provide a set of programming API, so that users can directly implement this set of API and write distributed programs running on YARN. The flow of Twill applications running on YARN is the same as that of ordinary programs, under the flowchart:

To make it very easy for users to write distributed applications running on YARN, Twill has made a large number of common modules and features, including:

(1) Real-time log collection

The log collection function is realized based on kafka, which can collect the logs generated by the application in real time, which is convenient for centralized analysis and display.

(2) Resource reporting

It can collect all kinds of resources occupied by the task in real time, including cpu utilization, physical memory and so on.

(3) flexible expansion of resources

The resources consumed by the application can be changed dynamically, such as increasing the number of container online from 5 to 10.

(4) status recovery

When the client loses its link to the application, it can reconstruct a new client through API to retrieve the state.

(5) Service discovery

An application can register some information with zookeeper so that other services or clients can discover the location of the application in order to communicate with it.

As YARN becomes more and more mature, it will become a basic core system, and projects similar to Apache Twill will be particularly important.

The above is how to simply write yarn app. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.