Shulou(Shulou.com)05/31 Report--

This article mainly introduces the example analysis of job configuration in azkaban, which has a certain reference value, and interested friends can refer to it. I hope you can learn a lot after reading this article.

A) Job type

All job need a type that knows how they execute it. In general, there are four types of job: java, command, javaprocess, and pig.

Property

Require?

Meaning

Type

Required

Job type: java, command, javaprocess, or pig

Each type has multiple options, as described below.

I. Command jobs

The command line jobs is essentially a Unix command executed as a separate process. Any output is written to the log as standard output or standard error. If a job completes with a zero exit code, the execution succeeds, and if it is not zero, it fails.

Azkaban web dynamically passes the parameter, which command job can get through ${variable name}.

ii. Javaprocess jobs

Java process jobs is a container that makes it easy to start java programs. It is equivalent to executing a main method of class from the command line. The following describes the properties available in javaprocess jobs:

Env.property can be obtained through the program to obtain environment variables.

iii. Pig jobs

This job type runs the pig script through Grunt. The following describes the available properties of pig jobs:

iv. Java jobs

Java jobs is a java class with a run () method, for example. Java class does not need to implement any interfaces. Azkaban takes full advantage of the methods defined in the following class:

B) Job lock

There are three types of locks in Azkaban: permit, read.lock, and write.lock.

I. Permits

Permit locks are used to restrict concurrent access to a resource. For example, if you want to ensure that there are no more than four job accesses to a particular database at a time, you can create a pool with four permit and each job needs one permit to run. The number of Perimits can be set in the properties in the .property file under the job directory.

The number of Permit job must be provided through the job parameter setting job.permits to run. All permits will be released immediately when the job ends or fails.

ii. Read and Write lock

Azkaban supports bit resource setting Read/Write locks. A common use case is to lock the permission to modify a file in HDFS. For example, when you have a lot of job reading a file and a job wants to rebuild it, you need to make sure that you don't rebuild the file while other job reads it. Readers do not interfere with other readers and allow any number of readers. In any case, only one author is allowed, and at the beginning of writing, all readers must finish.

These must be set by defining parameters below read.lock and write.lock:

E) Azkaban system properties

Thank you for reading this article carefully. I hope the article "sample Analysis of job configuration in azkaban" shared by the editor will be helpful to you. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.