Shulou(Shulou.com)06/02 Report--

This article mainly explains "what is elastic-job". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what is elastic-job".

ElasticJob was born in 2015, when the industry had outstanding timed task frameworks such as QuartZ, but there was a lack of distributed exploration.

Due to the lack of products of distributed scheduling cloud platform, ElasticJob has attracted much attention from the very beginning. It effectively makes up for the shortcomings of jobs in the distributed field, and provides an one-stop automatic operation and maintenance control terminal.

In technology selection, ElasticJob chose to stand on the shoulders of giants rather than repeat the idea of manufacturing wheels, and perfectly combined the standard QuartZ of timed tasks with the sharp weapon ZooKeeper of distributed coordination to build a new concept of distributed scheduling framework quickly and stably.

What is ElasticJob?

ElasticJob is a distributed scheduling solution, which consists of two independent subprojects, ElasticJob Lite and ElasticJob Cloud.

ElasticJob Lite is positioned as a lightweight decentralized solution that provides coordination services for distributed tasks in the form of jar.

ElasticJob Cloud adopts the solution of self-developed Mesos Framework, which provides additional functions such as resource governance, application distribution and process isolation.

Through the functions of flexible scheduling, resource control and job governance, it creates a distributed scheduling solution suitable for Internet scenarios, and provides a diversified job ecology through open architecture design.

Using ElasticJob can make development engineers no longer worry about non-functional requirements such as linear throughput improvement of tasks, and enable development engineers to focus more on business-oriented coding design.

At the same time, it can liberate the operation and maintenance engineers so that they no longer have to worry about the availability of tasks and related management requirements, and can achieve the goal of automated operation and maintenance only by easily adding service nodes.

ElasticJob scheduling model

Unlike most job platforms, ElasticJob's scheduling model is divided into intra-process scheduling ElasticJob Lite, which supports thread-level scheduling, and process-level scheduling ElasticJob Cloud.

In-process scheduling

ElasticJob Lite is an intra-process thread-level scheduling framework. Through ElasticJob, jobs can be transparently combined with business applications.

It can be easily used with Spring, Dubbo and other Java frameworks, and the Bean injected by Spring can be used freely in the job, such as data source connection pool, Dubbo remote service and so on.

ElasticJob Lite is deployed with business applications, and its life cycle is consistent with business applications, so it is a typical embedded lightweight architecture.

ElasticJob Lite is very suitable for ordinary Java applications with stable use of resources and simple deployment architecture, which can be understood as Java development framework.

ElasticJob Lite itself is a non-centralized architecture, there is no need for independent centralized scheduling nodes, and each task node in the distributed environment schedules jobs timely in a self-scheduling way.

Only one registry is needed between tasks to coordinate the status of tasks in distributed scenarios. Currently, ZooKeeper and ETCD are supported as registries.

The architecture diagram is as follows:

Product orientation and New version Design concept of ElasticJob

As can be seen from the figure, the distributed job node of ElasticJob Lite obtains the master node through election and splits through the master node.

After slicing, the master node is no different from the slave node, and both perform tasks in a self-scheduling way.

Process-level scheduling

ElasticJob Cloud has two ways: in-process scheduling and process-level scheduling. Because ElasticJob Cloud can control the resources of the job server, its job types can be divided into resident tasks and instantaneous tasks.

Resident task is similar to ElasticJob Lite, which is an in-process scheduling; instantaneous task is completely different, it makes full use of the peak-cutting and valley-filling ability of resource allocation, it is a process-level scheduling, and each task will start a new process processing.

ElasticJob Cloud needs to control resources through Mesos and allocate tasks and resources through a scheduler deployed in Mesos Master. Cloud adopts a centralized architecture, and the high availability of the dispatch center is managed by Mesos.

Its architecture is as follows:

Product orientation and New version Design concept of ElasticJob

As can be seen from the figure, ElasticJob Cloud not only has all the capabilities of Lite, but also has the ability to allocate resources and distribute tasks.

It completely hosts the life cycle of job development, packaging, distribution, scheduling, governance, slicing and so on. It is a real job cloud scheduling system.

Compared with the ease of use of ElasticJob Lite, ElasticJob Cloud's strong dependence on Mesos increases the complexity of system deployment, so it is more suitable for large-scale operating systems.

ElasticJob feature list

The main functions of ElasticJob are flexible scheduling, resource allocation, job management and visual control.

Flexible scheduling

Flexible scheduling is the most important function of ElasticJob and the origin of the product name. It is a task processing system that enables tasks to scale horizontally through fragmentation.

The concept of task fragmentation item in ElasticJob enables tasks to run in a distributed environment, and each task server only runs the shards assigned to that server.

With the increase or downtime of the server, ElasticJob will perceive the change of the number of servers in near real time, thus reassigning more reasonable task fragmentation items to the distributed task server, so that the task can improve the efficiency with the increase of resources.

For example, if the job is divided into 4 slices and executed with two servers, each server is allocated 2 slices, as shown in the following figure:

Product orientation and New version Design concept of ElasticJob

When a new job server is added, ElasticJob will perceive the existence of the new server through the change of the temporary node in the registry, and refragment it the next time the task is scheduled, and the new server will host part of the job shard.

The fragmentation is shown in the following figure:

Product orientation and New version Design concept of ElasticJob

When the job server is down while running, the registry will also perceive it through the temporary node, and will transfer the shards to the surviving server the next time it runs, so as to achieve the effect of high job availability.

Jobs that have not been completed due to server downtime can continue to be executed by means of failure transfer.

The high availability of jobs is shown in the following figure:

Product orientation and New version Design concept of ElasticJob

Allocation of resources

As mentioned in the introduction, scheduling refers to assigning the right resources to the task at the right time and making it effective.

ElasticJob has the ability to allocate resources, and it can schedule tasks like a distributed operating system.

Resource allocation is implemented by Mesos, and Mesos is responsible for allocating the required resources (CPU and memory) declared by the task and isolating the allocated resources. ElasticJob does not perform the task until it has obtained the resource.

Considering the relatively complex deployment of the Mesos system, ElasticJob split this part into the ElasticJob Cloud section for advanced users.

With the strong development of Kubernetes, ElasticJob will also complete the docking of Cloud with it in the future.

Operation management

Governance and coordination of high availability of jobs, failure transfer, missed job re-execution and other behaviors in distributed scenarios.

Visual control terminal

It mainly includes the control side of adding, deleting, changing and checking the job, the execution of the history query, the management of the configuration center and so on.

Typical application scenarios of ElasticJob

ElasticJob focuses on solving the problems of complex tasks, resource-oriented tasks and business application tasks.

Complex task

Data migration: if you migrate tens of billions of data from one set of database clusters to another, single-threaded jobs may take from days to weeks.

Through the elastic slicing capability of ElasticJob, the time required for massive data migration can be greatly reduced.

Resource-oriented task

Report jobs that take up a lot of computing resources. If it takes several hours every morning to calculate the business report of Tread1, without the control of resources, sufficient resources should be allocated to the report job regardless of whether it is started or not.

ElasticJob divides jobs into resident jobs and instantaneous jobs. For report jobs, instantaneous jobs are very suitable.

Can it obtain resources at the start of the operation, return resources after the end of the operation, truly cut peaks and fill valleys, and make more rational use of resources.

Business application

Order pull operation: order systems mostly use message middleware or jobs to realize order pull, which is used to decouple the order generation system from the back-end fulfillment system to facilitate the separation of front and back end traffic.

The order system realized by job can realize the order-related business logic through ElasticJob, and can easily use the dependency injection service provided by the peripheral system to seamlessly integrate into the business-side research and development.

Design concept of new version of ElasticJob

After more than a month of development, the ElasticJob community plans to release 3.0.0-alpha in the near future as its first release into the Apache Software Foundation.

Its main functions include:

Operational biosphere

Flexible customization is the most important design change in version 3.x. The new version is based on the design concept of Apache ShardingSphere pluggable architecture and creates a new job API.

It is intended to enable developers to expand job types and create an ecological circle of ElasticJob jobs in a more convenient and isolated way.

ElasticJob provides flexible job API, which decouples jobs into job interfaces and actuator interfaces. Users can customize new job types, such as script execution, HTTP service execution, big data jobs, file jobs, and so on.

At present, ElasticJob has built-in script execution jobs, and fully opened the extension interface, developers can introduce new job types through SPI, and can easily give back to the community.

Multiple scheduler

On the basis of keeping the original time-triggered scheduler based on Cron, an one-time scheduling API is added to ElasticJob, which adds a new scheduling dimension beyond the time dimension.

Micro-kernel & ecological separation

Abstract the job kernel module, and completely extract the extensible modules such as job execution trajectory tracking and job ecology from the kernel module.

As a second-level ecology, the job execution trajectory tracking module modifies the previous limitation that only MySQL is supported as a storage medium, and the adaptation is completely open and persistent.

Future planning

Version 3.0.0, as a quick feedback to the community, does not make a disruptive innovation, but attempts to decouple the project kernel bit by bit.

In the future planning, ElasticJob will take drastic steps forward, the main plans are as follows:

Job dependence

Job dependency based on directed acyclic graph (DAG) is supported. Dependency includes dependencies based on the overall dimension of the job, as well as dependencies based on job fragmentation items, to create a more flexible job governance solution.

Dispatch execution separation

Completely separate the scheduler from the actuator. The scheduler can be deployed with the executor, that is, it is a non-centralized lightweight version of ElasticJob Lite; the scheduler can be deployed separately from the executor, that is, an one-stop distributed scheduling system controlled by ElasticJob Cloud resources.

Easier to use cloud tube products

Build ElasticJob cloud, which only supports Mesos, into a job cloud management platform that supports Mesos and Kubernetes, and provides a pure job management and control platform that can be used independently without Mesos and Kubernetes.

Pluggable ecology

In line with Apache ShardingSphere, ElasticJob will also provide a more pluggable and modular architecture to provide infrastructure for developers.

Developers can easily redevelop based on ElasticJob and add a variety of customized functions, including but not limited to job types (e.g. big data job, HTTP job, etc.), registry type (e.g. Eureka, etc.), execution track storage media (such as other database types), etc.

The positioning of ElasticJob is shown in the following figure:

Product orientation and New version Design concept of ElasticJob

About the ElasticJob Community

The goal of the ElasticJob community is to become a top-level project of the Apache Software Foundation like Apache ShardingSphere and to achieve a wider range of applications.

During the restart of the project, ElasticJob continued to be famous on GitHub's weekly and monthly rankings:

In the last month, the ElasticJob community merged 152 Pull Requests, closed 105 Issues;25 submitters, and completed 4w + lines of code changes.

Thank you for your reading, the above is the content of "what is elastic-job", after the study of this article, I believe you have a deeper understanding of what is elastic-job, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.