Shulou(Shulou.com)06/02 Report--

This article mainly explains the "web distributed timing task scheduling framework how to use", the content of the article is simple and clear, easy to learn and understand, now please follow the editor's ideas slowly in depth, together to study and learn "web distributed timing task scheduling framework how to use" it!

Business background 1.1 Why do you need to use scheduled task scheduling

(1) time-driven processing scenarios: send coupons on the hour, update earnings every day, and refresh tag data and crowd data every day.

(2) batch processing data: batch statistics of report data on a monthly basis, batch update of SMS status, and low real-time requirements.

(3) decoupling asynchronous execution: activity state refresh, asynchronous execution of offline query, decoupling from internal logic.

1.2 use requirements and pain points

(1) Task execution, monitoring and alarm capabilities.

(2) the task can be configured flexibly and dynamically without restarting.

(3) transparent business, low coupling, simplified configuration and convenient development.

(4) easy to test.

(5) High availability and no single point of failure.

(6) the task can not be repeated to prevent logical anomalies.

(7) the parallel processing ability of the distribution of large tasks.

II. Practice and Exploration of Open Source Framework 2.1 Java Native Timer and ScheduledExecutorService2.1.1 Timer usage

Timer defect:

At the bottom of Timer, a single thread is used to handle multiple Timer tasks, which means that all tasks are actually executed serially, and the delay of the previous task will affect the execution of subsequent tasks.

Because of a single thread, once an unhandled exception is generated while a scheduled task is running, not only the current thread will stop, but all scheduled tasks will stop.

The execution of Timer tasks depends on the absolute time of the system, and the change of system time will lead to changes in the execution plan.

Because of the above defects, try not to use Timer, idea will also explicitly prompt to use ScheduledThreadPoolExecutor instead of Timer.

2.1.2 ScheduledExecutorService usage

ScheduledExecutorService fixes the defects of Timer. First of all, the internal implementation of ScheduledExecutorService is the Scheduled ThreadPool thread pool, which can support multiple tasks to execute concurrently.

If an exception occurs in a task executed by one thread, it will also be handled, which will not affect the execution of other thread tasks. In addition, ScheduledExecutorService is based on the delay of time interval, and the execution will not change due to the change of system time.

Of course, ScheduledExecutorService also has its own limitations: it can only be scheduled according to the delay of tasks, and can not meet the needs of scheduling based on absolute time and calendar.

2.2 Spring Task2.2.1 Spring Task usage

Spring task is a lightweight scheduled task framework developed by spring, which does not need to rely on other additional packages and is relatively simple to configure.

Use annotations to configure here

2.2.2 Spring Task defect

Spring Task itself does not support persistence, nor does it launch an official distributed cluster model. It can only be implemented by developers manually expanding their business applications, which can not meet the needs of visualization and easy configuration.

2.3 basic introduction of timeless classic Quartz2.3.1

Quartz framework is the most famous open source task scheduling tool in the field of Java, and it is also the current de facto timed task standard. Almost all open source timed task frameworks are based on Quartz core scheduling.

2.3.2 principle Analysis

Core components and Architectur

Key concepts

(1) Scheduler: task scheduler, which is the controller that performs task scheduling. In essence, it is a planning and scheduling container, which registers all Trigger and corresponding JobDetail, and uses thread pool as the basic component of task execution to improve task execution efficiency.

(2) Trigger: trigger, which defines the time rules for task scheduling and tells the task scheduler when to trigger the task, where CronTrigger is a powerful trigger based on cron expressions.

(3) Calendar: a collection of calendar points in time. A trigger can contain multiple Calendar, which can be used to exclude or include certain time steps.

(4) JobDetail: an executable job to describe the Job implementation class and other related static information, such as the name of the Job, listeners, and other related information.

(5) Job: task execution interface with only one execute method for executing real business logic.

(6) JobStore: task storage method. RAMJobStore and JDBCJobStore,RAMJobStore are mainly stored in the memory of JVM, and there is a risk of loss and limited quantity. JDBCJobStore persists task information to the database and supports clusters.

2.3.3 practical explanation

(1) on the basic use of Quartz

You can refer to Quartz official documentation and online blog practice tutorials.

(2) in order to meet the requirements of dynamic modification and restart without loss, it is generally necessary to use a database to save the business.

Quartz supports JDBCJobStore, but its configuration has a large number of data sheets. The official recommended configuration can refer to the official documents, more than 10 tables, and the business is heavily used.

When using it, you only need to have the basic trigger configuration and corresponding tasks, as well as the tables of related execution logs to meet most of the requirements.

(3) componentization

The quartz dynamic task configuration information is persisted to the database, and the data operations are packaged into basic jar packages for use between projects. Referencing projects only need to introduce jar package dependencies and configure corresponding data tables, which can be transparent to Quartz configuration.

(4) expansion

Cluster mode

High availability of tasks is achieved through failover and load balancing, and the uniqueness of task execution is ensured through the locking mechanism of the database, but the cluster feature is only used for HA. The increase in the number of nodes will not improve the execution efficiency of a single task and cannot achieve horizontal expansion.

Quartz plug-in

You can extend specific needs, such as adding triggers and task execution logs. Tasks rely on serial processing scenarios. For more information, please see: quartz plug-in, which implements serial scheduling between tasks.

2.3.4 defects and deficiencies

(1) the task information needs to be persisted to the business data table, which is coupled with the business.

(2) scheduling logic and execution logic coexist in the same project, and when the machine performance is fixed, the business and scheduling will inevitably influence each other.

(3) in quartz cluster mode, the task is obtained only through database exclusive lock, and there is no perfect load balancing mechanism for task execution.

2.4 basic introduction to lightweight artifact XXL-JOB2.4.1

XXL-JOB is a lightweight distributed task scheduling platform, which is characterized by platform, easy deployment, rapid development, simple learning, lightweight and easy to expand, and the code is still being updated continuously.

The "scheduling center" is the task scheduling console, and the platform itself does not undertake the business logic, but is only responsible for the unified management and scheduling of tasks, and provides a task management platform. The "executor" is responsible for receiving the scheduling and execution of the "scheduling center". The executor can be deployed directly or integrated into the existing business project. By decoupling the scheduling control of tasks from the execution of tasks, business usage only needs to focus on the development of business logic.

It mainly provides several functional modules such as task dynamic configuration management, task monitoring and statistical report, and scheduling log, which supports a variety of operation modes and routing strategies, and can carry out simple slicing data processing based on the number of corresponding executor machine clusters.

2.4.2 principle Analysis

Before version 2.1.0, the core scheduling modules are all based on the quartz framework. Since version 2.1.0, self-developed scheduling components have been developed to remove quartz dependencies and use time round scheduling.

2.4.3 practical explanation

Refer to the official documentation for detailed configuration and introduction.

2.4.3.1 demo uses:

Example 1: to implement a simple task configuration, you only need to inherit the IJobHandler abstract class and declare comments

@ JobHandler (value= "offlineTaskJobHandler") to implement the business logic. (note: dubbo is introduced this time, which will be introduced later).

JobHandler (value= "offlineTaskJobHandler") @ Componentpublic class OfflineTaskJobHandler extends IJobHandler {@ Reference (check = false,version = "cms-dev", group= "cms-service") private OfflineTaskExecutorFacade offlineTaskExecutorFacade; @ Override public ReturnT execute (String param) throws Exception {XxlJobLogger.log ("offlineTaskJobHandler start."); try {offlineTaskExecutorFacade.executeOfflineTask ();} catch (Exception e) {XxlJobLogger.log ("offlineTaskJobHandler-- > exception.", e); return FAIL } XxlJobLogger.log ("XXL-JOB, offlineTaskJobHandler end."); return SUCCESS;}}

Example 2: multipart broadcast task.

@ JobHandler (value= "shardingJobHandler") @ Servicepublic class ShardingJobHandler extends IJobHandler {@ Override public ReturnT execute (String param) throws Exception {/ / sharding parameter ShardingUtil.ShardingVO shardingVO = ShardingUtil.getShardingVo (); XxlJobLogger.log ("shard parameters: current shard serial number = {}, total number of shards = {}", shardingVO.getIndex (), shardingVO.getTotal ()); / / business logic for (int I = 0; I

< shardingVO.getTotal(); i++) { if (i == shardingVO.getIndex()) { XxlJobLogger.log("第 {} 片, 命中分片开始处理", i); } else { XxlJobLogger.log("第 {} 片, 忽略", i); } } return SUCCESS; }}2.4.3.2 整合dubbo （1）引入dubbo-spring-boot-starter和业务facade jar包依赖。 com.alibaba.spring.boot dubbo-spring-boot-starter 2.0.0 com.demo.service xxx-facade 1.9-SNAPSHOT （2）配置文件加入dubbo消费端配置（可根据环境定义多个配置文件，通过profile切换）。 ## Dubbo 服务消费者配置spring.dubbo.application.name=xxl-job spring.dubbo.registry.address=zookeeper://zookeeper.xyz:2183spring.dubbo.port=20880 spring.dubbo.version=demospring.dubbo.group=demo-service （3）代码中通过@Reference注入facade接口即可。 @Reference(check = false,version = "demo",group="demo-service")private OfflineTaskExecutorFacade offlineTaskExecutorFacade; （4）启动程序加入@EnableDubboConfiguration注解。 @SpringBootApplication@EnableDubboConfigurationpublic class XxlJobExecutorApplication { public static void main(String[] args) { SpringApplication.run(XxlJobExecutorApplication.class, args); }}2.4.4 任务可视化配置 内置了平台项目，方便了开发者对任务的管理和执行日志的监控，并提供了一些便于测试的功能。 2.4.5 扩展 （1）任务监控和报表的优化。 （2）任务报警方式的扩展，比如加入告警中心，提供内部消息，短信告警。 （3）对实际业务内部执行出现异常情况下的不同监控告警和重试策略。 2.5 高可用 Elastic-Job2.5.1 基本介绍 Elastic-Job是一个分布式调度解决方案，由两个相互独立的子项目Elastic-Job-Lite和Elastic-Job-Cloud组成。 Elastic-Job-Lite定位为轻量级无中心化解决方案，使用jar包的形式提供分布式任务的协调服务。 Elastic-Job-Cloud使用Mesos + Docker的解决方案，额外提供资源治理、应用分发以及进程隔离等服务。 可惜的是已经两年没有迭代更新记录。 2.5.2 原理解析 2.5.3 实践说明 2.5.3.1 demo使用 （1）安装zookeeper，配置注册中心config，配置文件加入注册中心zk的配置。 @Configuration@ConditionalOn_Expression("'${regCenter.serverList}'.length() >

0 ") public class JobRegistryCenterConfig {@ Bean (initMethod =" init ") public ZookeeperRegistryCenter regCenter (@ Value (" ${regCenter.serverList} ") final String serverList, @ Value (" ${regCenter.namespace} ") final String namespace) {return new ZookeeperRegistryCenter (new ZookeeperConfiguration (serverList, namespace)) }} spring.application.name=demo_elasticjob regCenter.serverList=localhost:2181regCenter.namespace=demo_elasticjob spring.datasource.url=jdbc:mysql://127.0.0.1:3306/xxl-job?Unicode=true&characterEncoding=UTF-8spring.datasource.username=userspring.datasource.password=pwd

(2) configure the data source config and add the data source configuration to the configuration file.

@ Getter@Setter@NoArgsConstructor@AllArgsConstructor@ToString@Configuration@ConfigurationProperties (prefix = "spring.datasource") public class DataSourceProperties {private String url; private String username; private String password; @ Bean @ Primary public DataSource getDataSource () {DruidDataSource dataSource = new DruidDataSource (); dataSource.setUrl (url); dataSource.setUsername (username); dataSource.setPassword (password); return dataSource }} spring.datasource.url=jdbc:mysql://127.0.0.1:3306/xxl-job?Unicode=true&characterEncoding=UTF-8spring.datasource.username=userspring.datasource.password=pwd

(3) configure event config.

@ Configurationpublic class JobEventConfig {@ Autowired private DataSource dataSource; @ Bean public JobEventConfiguration jobEventConfiguration () {return new JobEventRdbConfiguration (dataSource);}}

(4) in order to flexibly configure different task trigger events, add ElasticSimpleJob comments.

@ Target ({ElementType.TYPE}) @ Retention (RetentionPolicy.RUNTIME) public @ interface ElasticSimpleJob {@ AliasFor ("cron") String value () default "; @ AliasFor (" value ") String cron () default"; String jobName () default"; int shardingTotalCount () default 1; String shardingItemParameters () default"; String jobParameter () default";}

(5) initialize the configuration.

@ Configuration@ConditionalOn_Expression ("'${elaticjob.zookeeper.server-lists} '.length () > 0") public class ElasticJobAutoConfiguration {@ Value ("${regCenter.serverList}") private String serverList; @ Value ("${regCenter.namespace}") private String namespace; @ Autowired private ApplicationContext applicationContext; @ PostConstruct public void initElasticJob () {ZookeeperRegistryCenter regCenter = new ZookeeperRegistryCenter (new ZookeeperConfiguration (serverList, namespace)) RegCenter.init (); Map map = applicationContext.getBeansOfType (SimpleJob.class); for (Map.Entry entry: map.entrySet ()) {SimpleJob simpleJob = entry.getValue (); ElasticSimpleJob elasticSimpleJobAnnotation = simpleJob.getClass (). GetAnnotation (ElasticSimpleJob.class); String cron = StringUtils.defaultIfBlank (elasticSimpleJobAnnotation.cron (), elasticSimpleJobAnnotation.value ()) SimpleJobConfiguration simpleJobConfiguration = new SimpleJobConfiguration (JobCoreConfiguration.newBuilder (simpleJob.getClass (). GetName (), cron, elasticSimpleJobAnnotation.shardingTotalCount ()) .shardingItemParameters (elasticSimpleJobAnnotation.shardingItemParameters ()). Build (), simpleJob.getClass (). GetCanonicalName (); LiteJobConfiguration liteJobConfiguration = LiteJobConfiguration.newBuilder (simpleJobConfiguration) .overwrite (true). Build (); JobEventRdbConfiguration jobEventRdbConfiguration = new JobEventRdbConfiguration (dataSource); SpringJobScheduler jobScheduler = new SpringJobScheduler (simpleJob, regCenter, liteJobConfiguration, jobEventRdbConfiguration) JobScheduler.init ();}

(6) implement the SimpleJob interface, integrate dubbo according to the above method, and complete the business logic.

@ ElasticSimpleJob (cron = "* / 10 *?", jobName = "OfflineTaskJob", shardingTotalCount = 2, jobParameter = "Test parameters", shardingItemParameters = "0,0,0,0,5) @ Logger logger = LoggerFactory.getLogger (OfflineTaskJob.class); @ Reference (check = false, version =" cms-dev ", group =" cms-service ") private OfflineTaskExecutorFacade offlineTaskExecutorFacade @ Override public void execute (ShardingContext shardingContext) {offlineTaskExecutorFacade.executeOfflineTask (); logger.info (String.format ("Thread ID:% s, total number of job shards:% s," + "current shard item:% s. Current parameter:% s, "+" job name:% s. Job customization parameters:% s ", Thread.currentThread () .getId (), shardingContext.getShardingTotalCount (), shardingContext.getShardingItem (), shardingContext.getShardingParameter (), shardingContext.getJobName (), shardingContext.getJobParameter ());}}

2.6 remaining open source frameworks

(1) Saturn:Saturn is a distributed task scheduling platform opened by VIPSHOP, which is modified on the basis of Elastic Job.

(2) SIA-TASK: it is a reliable open source distributed task scheduling platform.

Third, the comparison of advantages and disadvantages and the adaptation of business scenarios.

Business thinking:

Enrich task monitoring data and alarm policies.

Access to unified login and access control.

Further simplify the service access steps.

Thank you for reading, the above is the content of "how to use web distributed timing task scheduling framework". After the study of this article, I believe you have a deeper understanding of how to use web distributed timing task scheduling framework, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.