Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to configure the coordinator of Oozie". In the daily operation, I believe that many people have doubts about how to configure the coordinator of Oozie. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the doubts of "how to configure coordinator of Oozie". Next, please follow the editor to study!

Coordinator is a timing submitter of workflow, which generates triggers (based on time and data triggers) based on time conditions and data. To put it simply, coordinator polls according to the defined time period and triggers the workflow task if the data generation condition is met; otherwise, it waits for the data to be generated or skipped (the scheduling policy is determined by the set timeout).

Coordinator-app

Coordinator-app name= "coord-demo" frequency= "${coord:days (1)}" start= "${start_time}" end= "${end_time}" timezone= "Asia/Shanghai" xmlns= "uri:oozie:coordinator:0.1"

The header defines the name of the coordinator task, the frequency of execution (frequency), and the start and end times (start, end). ${coord:days (1)} is coordinator's built-in EL function (Expression Language), which means that the execution cycle is one day. If the time unit of frequency is minutes, then ${coord:days (1)} = 1440 coord:hours (3)} = 180. Generally speaking, the definition of job.properties, such as start_time=2016-06-07T00:00Z, should be based on the UTC time zone.

Control information

-1 1 ${execution_order}

Datasets

${nameNode} / path/to/$ {YEAR}-${MONTH}-${DAY} / ${HOUR} /

Related parameters:

Initial-instance, the data directory corresponding to the initial time, as the base directory for the time period offset

Uri-template, file directory HDFS directory, ${YEAR}, ${MONTH} and so on are EL Time Constants. Specific values are passed, or custom parameters can be passed through job.properties.

Done-flag, the file ID generated by the data directory. If it is not specified, it defaults to the _ SUCCESS file; if it is empty, it represents the folder itself.

Input-events

${coord:current (- 1)}

The data trigger condition is defined, and ${coord:current (- 1)} represents the offset for the current time, where offset = numeric * dataset frequency. Sometimes, when we want to determine whether multiple directories are generated, for example, to determine the 24-hour directory of the day, we should write:

${coord:current (- 24)} ${coord:current (- 1)}

Action

${wf_app_path} dayTime ${coord:formatTime (coord:dateOffset (coord:nominalTime (),-1, 'DAY'),' yyyy-MM-dd')}

Define the workflow to be triggered by coordinator

The HDFS directory where app-path,workflow is located

Property, which defines the parameters passed to workflow

Coordinator provides some functions for time calculation, such as:

Coord:nominalTime (), which returns the current time of each execution cycle of the coordinator

Coord:dateOffset (String baseDate, int instance, String timeUnit), calculate the time after migration, newDate = baseDate + instance * timeUnit

Coord:formatTime (String ts, String format), which defines the time of formatting, where format should follow Java's SimpleDateFormat.

Return multiple times:

${coord:formatTime (coord:dateOffset (coord:dateOffset (coord:nominalTime (),-12, 'HOUR'),-6,' DAY'), 'yyyy-MM-dd')}, ${coord:formatTime (coord:dateOffset (coord:dateOffset (coord:nominalTime (),-12,' HOUR'),-5, 'DAY') so far, the study on "how to configure coordinator for Oozie" is over. I hope I can solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.