Shulou(Shulou.com)05/31 Report--

This article mainly introduces the concept of job and task is what the relevant knowledge, the content is detailed and easy to understand, the operation is simple and fast, has a certain reference value, I believe that everyone will have a harvest after reading the concept of job and task, let's take a look at it together.

1. profile

The cluster management system, internally called Borg, manages, schedules, starts, restarts, and monitors the lifecycle of applications running on Google. This article describes how it does this.

Borg provides three main benefits: it (1) hides the details of resource management and fault handling so that its users can focus on application development;(2) operates with high reliability and availability and enables applications to be highly reliable and highly available; and (3) allows us to run efficiently across tens of thousands of machines. Borg isn't the first system to solve these problems, but it's one of the few systems that operates at this scale, with this degree of resilience and integrity.

This article is written around these themes and includes some of our skills from ten years of operating in a production environment.

2. user perspective

Borg users are Google developers and system administrators (SRE) who run Google apps and services. Users submit their work to Borg as a job, which consists of one or more tasks, each task containing the same binary program. A job runs in a Borg Cell, which is a unit that includes multiple machines. This section focuses on the Borg system from the user's perspective.

2.1 workloads

Borg Cell primarily runs two heterogeneous workloads. The first is a long-term service that should run "forever" and handle sensitive requests of short duration (microseconds to hundreds of milliseconds). Such services are end-user oriented products such as Gmail, Google Docs, web search, internal Infrastructure services (e.g. Bigtable). The second is batch processing, which takes seconds to days to complete and is insensitive to short-term performance fluctuations. These two workloads are mixed on a single Cell, depending on their primary tenant (e.g., some Cells are specifically designed to run intensive batch tasks). Workload also varies over time: batch tasks are done, end-user service loads are on a daily basis. Borg needs to handle both situations.

Borg has a load data for May 2011 [80] that has been extensively analyzed [68, 26, 27, 57, 1].

In recent years, many application frameworks have been built on Borg, including our internal MapReduce[23], flumejava[18], Millwheel[3], Pregel[59]. Most of them have a controller that can submit jobs. The first two frameworks are similar to YARN's application manager.[76] Our distributed storage systems, such as GFS[34] and its successors CFS, Bigtable[19] and Megastore[8], all run on Borg.

In this article, we define high-priority Borg jobs as productive (prod) and the rest as non-productive (non-prod). Most long-term services are prod, and most batch tasks are non-prod. In a typical Cell, the prod job allocates 70% of CPU resources and then uses 60%; it allocates 55% of memory resources and then uses 85%. At $5.5 it will be shown that the difference between allocation and actual value is important.

2.2 Cluster and Cell

All the machines in a Cell belong to a single cluster, which is connected by a high-performance data center-level fiber optic network. A cluster is installed in one building of the data center, and n buildings are combined to form a site. A cluster usually consists of one large Cell and several smaller or test Cells. We try to avoid any single point of failure.

In addition to the Cells we tested, our medium-sized Cell probably contained 10000 machines; some Cells were much larger. The machines in a Cell are heterogeneous in many ways: size (CPU,RAM,disk,network), processor type, performance, and external IP address or flash storage. Borg isolates these differences, allowing users to simply choose which Cell to run tasks with, allocate resources, install programs, and other dependencies, monitor system health, and restart in case of failure.

A Cell is actually a logical cluster.

2.3 job and task

The attributes of a Borg job are: name, owner, and number of tasks. A job can have constraints that specify what architecture processor, operating system version, and external IP the job runs on. Constraints can be hard or soft. A job can be specified to start after another job has finished. A job only runs in one Cell.

Each task consists of a set of Linux processes running in a container on a machine.[62] Most of Borg's workloads don't run inside virtual machines (VMs) because we don't want to pay the price of virtualization. And Borg wasn't designed for hardware virtualization.

Task also has a number of attributes, including resource usage, sort in job. Most task attributes are the same as job generic task attributes and can be overridden-for example, providing task-specific command-line parameters, including CPU core, memory, disk space, disk access speed, TCP port, etc., that can be set separately and provided at a good granularity. We do not provide fixed units of resources. Borg programs are statically compiled so that there are no dependencies in the running environment. These programs are packaged into a package, including binary and data files, that can be installed by Borg.

Users operate Borg jobs via RPC, mostly from command-line tools, or from our monitoring system ($2.6). Most job description files are a variant of the declarative configuration file BCL -- GCL[12], which produces a protobuf file [67]. BCL has some keywords of its own. GCL provides lambda expressions to allow computation, which allows applications to adjust their configuration within the environment. Tens of thousands of BCL configuration files have more than 1,000 lines, and the system has accumulated tens of millions of BCL lines. Borg's job configuration is very similar to the Aurora profile [6].

Figure 2 shows the state machine and life cycle of job and task.

Users can change task properties in a job at runtime by pushing a new job configuration to Borg. This new configuration command Borg updates the task specification. It's like running a lightweight, non-atomic transaction that can be easily changed back after it's committed. The update is rolling, limiting the number of task restarts in the update, and terminating the operation if too many tasks are stopped.

Some task updates, such as updating binary programs, require task restart; others, such as modifying resource requirements and constraints, cause the machine to be unsuitable for running existing tasks and require task to be stopped and rescheduled to other machines; and others, such as modifying priorities, do not need to restart or move tasks.

Tasks need to be able to accept Unix SIGTERM signals before they are forced to send SIGKILL, so that there is time to clean up, save state, end execution of existing requests, and reject new requests. The actual delay bound of notice. In practice, 80% of tasks can handle termination signals properly.

2.4 Allocs

Borg alloc(short for allocation) is a set of reserved resource quotas on a single machine that allow one or more tasks to run; these resources are always allocated there, whether or not they are used. Allocs can be allocated to future tasks, used to keep resources between stopping and restarting a task, and used to aggregate tasks from different jobs onto the same machine--for example, a web server instance and additional log collection instances that send serverURL logs to a distributed file system. An alloc manages resources in a similar way to a machine; multiple tasks run on an alloc and share resources. If an alloc must be relocated to another machine, its task must be rescheduled accordingly.

An alloc set is like a job: it is a set of allocs that reserve resources on multiple machines. Once an alloc set is created, one or more jobs can be submitted to run. In short, we will use task to represent an alloc or a top-level task(outside of an alloc) and job to represent a job or alloc set.

2.5 Priorities, quotas and administrative controls

What happens when there is an excess workload running? Our solution is priorities and quotas.

All jobs have priority, a small positive integer. High-priority tasks get priority access to resources, even if they are later killed. Borg defines non-overlapping priority segments for different tasks, including (in descending order of priority): monitoring, production, batch tasks, high performance (test or free). In this article, prod jobs are in the monitoring and production segments.

Although a demoted task will always find a place elsewhere in the cell. Downgrade waterfall may also occur, that is, after a task falls, the task running below is squeezed onto another machine, and so on. To avoid this, we prohibit prod-level tasks from crowding out each other. Reasonably granular priorities are useful in other scenarios as well-MapReduce's master runs have a higher priority than workers to ensure their availability.

Priority is the relative importance of jobs, which determines whether jobs are running or pending in a cell. Quotas are used to determine whether jobs are scheduled. A quota is the amount of a resource (CPU, RAM, disk) at a specified priority for a specified period of time (on the order of months). The quantity determines the maximum resources available for this user's job (example: 20TB of memory and prod priority from now until July in xx cell). Quota checking is part of administrative control, not at the scheduling level: tasks with insufficient quotas are rejected when submitted.

A high-priority quota always costs more than a low-priority quota. Prod-level quotas are limited to the actual amount of resources in a cell, so when users submit prod-level job quotas, they can expect the job to run, removing some fragments. Even so, we encourage users to buy a little more quota than they need, and many users overbuy because their applications need more quota as they grow in user count. For overbought, our solution is to oversell low-priority resources: all users at priority 0 can use unlimited quotas, although this situation is difficult to run in practice. A low-priority job will remain pending when resources are scarce.

Quota allocation is outside the Borg system and related to our physical resource planning. These resource plans generate different prices and quotas in different data centers. User jobs can only be started if there is enough quota and enough priority. The use of quotas makes Dominant Resource Fairness(DRF)[29, 35, 36, 66] less necessary.

Borg has a capacity system that gives certain users special privileges, such as allowing administrators to delete or modify jobs in a cell, or allowing user zones to access certain kernel features, or allowing Borg not to make resource estimates for its own jobs ($5.50).

2.6 Naming and monitoring

It's not enough to create and deploy tasks: clients of a service and other systems need to be able to find them, even if it's somewhere else. To get around this, Borg created a stable "Borg name Service"(BNS) name for each task, which includes the cell name, job name, and task number. Borg writes the task hostname and port to a persistent high availability file named BNS on Chubby[14]. This file is used by our RPC system to find the terminal address of a task. The BNS name is also the basic component of the DNS name of the task, so the DNS name of the 50th task of the jfoo job of the ubar user of cc cell will be 50.jfoo.ubar.cc.borg.google.com. Borg also writes job size and task health information to Chubby whenever circumstances change so that Load Balancer knows how to route requests.

Almost all Borg tasks include a built-in HTTP service for publishing health information and thousands of performance metrics (such as RPC latency). Borg monitors these health check URLs and restarts tasks that respond to timeouts and errors. Other data is also tracked by monitoring tools and displayed on Dashboards, which alert when service level objects (SLO) go wrong.

Users can use a web UI called Sigma to check all of their job status, a particular cell, or drill down into resource usage, detailed logs, operation history, and final fate for a task of a job. Our application generates a large number of logs, which are automatically scrolled to avoid clogging the hard drive, and are kept for a short period of time after a task is completed for debugging. If a job is not running, Borg provides an explanation of why it is pending, instructing the user how to modify the job's resource requirements to match the current cell situation. We publish guidelines for the use of resources, and it is easy to dispatch them according to this policy.

Borg records all job commit and task times, as well as resource usage details for each task within the underlying storage service. This storage service has a distributed, read-only SQL-like interactive interface provided through Dremel[61]. This data is useful for real-time usage, debugging, system troubleshooting, and long-term capacity planning. This data is also one of the sources of data for Google Cluster Load Tracking.[80]

All of these features help users understand and debug Borg behavior and manage their jobs, and help each of our SRE manage over tens of thousands of machines.

About "what is the concept of job and task" the content of this article is introduced here, thank you for reading! I believe everyone has a certain understanding of the knowledge of "what is the concept of job and task". If you still want to learn more knowledge, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.