Shulou(Shulou.com)06/01 Report--

This article mainly introduces the example analysis of Spring Batch lightweight batch framework, which is very detailed and has certain reference value. Friends who are interested must finish it!

1 what is the theoretical basis before actual combat 1.1 Spring Batch

Spring Batch is a lightweight, comprehensive batch framework designed to support the development of powerful batch applications that are critical to the day-to-day operation of enterprise systems. It also makes it easy for developers to access and take advantage of more advanced enterprise services when necessary. Spring Batch is not a scheduling framework, it is designed to work with the scheduler, not replace the scheduler.

1.2 what can Spring Batch do?

Automatic and complex processing of a large amount of information, which can be processed most effectively without user interaction. These actions typically include time-based events (such as month-end calculations, notifications, or communications).

Complex business rules that are periodically applied to very large datasets that are processed repeatedly (for example, insurance benefit determination or rate adjustment).

Integrate information received from internal and external systems into the recording system, which usually needs to be formatted, verified, and processed in a transactional manner. Batch processing is used to process billions of transactions for the enterprise every day.

Business scenario:

Submit batch processing periodically

Concurrent batch processing: parallel processing of jobs

Phased, enterprise message-driven processing

Massively parallel batch processing

Manual or planned restart after failure

Dependent sequential processing of steps (extended to workflow-driven batching)

Partial processing: skip records (for example, on rollback)

Whole batch transactions, suitable for small batches or existing stored procedures / scripts

All in all, what Spring batch can do:

Read a large number of records from a database, file, or queue.

Process data in some way.

Write back the data in a modified form.

1.3 Infrastructure

1.4 Core concepts and abstractions

Core concept: a Job has an one-to-many Step, and each step happens to have an ItemReader, an ItemProcessor, and an ItemWriter. The job needs to be started (using JobLauncher) and metadata about the currently running process needs to be stored (JobRepository in).

2 introduction of each component 2.1 Job

Job is an entity that encapsulates the entire batch process. Like other Spring projects, a Job is connected to an XML configuration file or a Java-based configuration. This configuration can be called "job configuration".

Configurable items:

The simple name of the job.

Definition and sorting of Step instances.

Whether the job can be restarted.

2.2 Step

A Step is a domain object that encapsulates a separate, continuous phase of a batch job. Therefore, each Job consists entirely of one or more steps. A Step contains all the information needed to define and control the actual batch.

One StepExecution represents an attempt to execute one Step at a time. StepExecution creates a new one each time Step runs, similar to JobExecution.

2.3 ExecutionContext

An ExecutionContext represents a collection of key / value pairs persisted and controlled by the framework to allow developers a place to store persistent state scoped to StepExecution objects or JobExecution objects.

2.4 JobRepository

JobRepository is the persistence mechanism for all of the above Stereotypes. It provides CRUD operation JobLauncher,Job as well as Step implementation. When Job is first started, a JobExecution is obtained from the repository, and, during execution, the StepExecution and JobExecution implementations persist by passing them to the repository.

When using Java configuration, the @ EnableBatchProcessing annotation provides a JobRepository as one of the components that are automatically configured out of the box.

2.5 JobLauncher

JobLauncher represents a simple interface for Job to start JobParameters with a given collection, as shown in the following example:

Public interface JobLauncher {public JobExecution run (Job job, JobParameters jobParameters) throws JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException, JobParametersInvalidException;}

Expect the implementation JobExecution to get a valid JobRepository from and execute the Job.

2.6 Item Reader

ItemReader is an abstraction that represents the input that retrieves Step one item at a time. When ItemReader uses up the items it can provide, it indicates this by returning null.

2.7 Item Writer

ItemWriter is an abstraction that represents the output of a Step, batch, or chunk of a project at a time. Typically, the anItemWriter does not know the input it should receive next, and only knows the items passed in its current call.

2.8 Item Processor

An ItemProcessor is an abstraction that represents the business processing of a project. When ItemReader reads an item and ItemWriter writes to it, it ItemProcessor provides an access point to transform or apply other business processes. If it is determined that the project is invalid when processing the project, returning null indicates that the project should not be written out.

3 Spring Batch actual combat

Let's use the theory we have learned to implement the simplest Spring Batch batch project.

3.1 dependency and project structure and configuration files

Dependence

Org.springframework.boot spring-boot-starter-batch org.springframework.boot spring-boot-starter-web org.projectlombok lombok 1.18.20 mysql mysql-connector-java 5.1.47 com.baomidou mybatis-plus-boot-starter 3.2.0

Project structure

Configuration file

Server.port=9000spring.datasource.url=jdbc:mysql://localhost:3306/testspring.datasource.username=rootspring.datasource.password=12345spring.datasource.driver-class-name=com.mysql.jdbc.Driver3.2 code and data sheet

Data sheet

CREATE TABLE `student` (`id` int (100) NOT NULL AUTO_INCREMENT, `name` varchar (45) DEFAULT NULL, `age` int (2) DEFAULT NULL, `address` varchar (45) DEFAULT NULL, PRIMARY KEY (`id`), UNIQUE KEY `id_ UNIQUE` (`id`) ENGINE=InnoDB AUTO_INCREMENT=203579 DEFAULT CHARSET=utf8 ROW_FORMAT=REDUNDANT

Student entity class

/ * @ desc: Student entity class * @ author: YanMingXin * @ create: 2021-10-15-12:17 * * / @ Data@Accessors (chain = true) @ NoArgsConstructor@AllArgsConstructor@ToString@TableName ("student") public class Student {@ TableId (value = "id", type = IdType.AUTO) private Long sId; @ TableField ("name") private String sName; @ TableField ("age") private Integer sAge; @ TableField ("address") private String sAddress;}

Mapper layer

/ * * @ desc: Mapper layer * @ author: YanMingXin * @ create: 2021-10-15-12:17 * * / @ Mapper@Repositorypublic interface StudentDao extends BaseMapper {}

Reading classes in a simulated database (file)

/ * * @ desc: read from simulated database * @ author: YanMingXin * @ create: 2021-10-16-10:13 * * / public class StudentVirtualDao {/ * simulated read from database * * @ return * / public List getStudents () {ArrayList students = new ArrayList (); students.add (new Student (1L, "zs", 23, "Beijing")) Students.add (new Student (2L, "ls", 23, "Beijing")); students.add (new Student (3L, "ww", 23, "Beijing")); students.add (new Student (4L, "zl", 23, "Beijing")); students.add (new Student (5L, "mq", 23, "Beijing")) Students.add (new Student (6L, "gb", 23, "Beijing")); students.add (new Student (7L, "lj", 23, "Beijing")); students.add (new Student (8L, "ss", 23, "Beijing")); students.add (new Student (9L, "zsdd", 23, "Beijing")) Students.add (new Student (10L, "zss", 23, "Beijing"); return students;}}

Service layer interface

/ * * @ desc: * @ author: YanMingXin * @ create: 2021-10-15-12:16 * / public interface StudentService {List selectStudentsFromDB (); void insertStudent (Student student);}

Service layer implementation class

/ * * @ desc: Service layer implementation class * @ author: YanMingXin * @ create: 2021-10-15-12:16 * * / @ Servicepublic class StudentServiceImpl implements StudentService {@ Autowired private StudentDao studentDao; @ Override public List selectStudentsFromDB () {return studentDao.selectList (null);} @ Override public void insertStudent (Student student) {studentDao.insert (student);}}

The core configuration class BatchConfiguration

/ * @ desc: BatchConfiguration * @ author: YanMingXin * @ create: 2021-10-15-12:25 * * / @ Configuration@EnableBatchProcessing@SuppressWarnings ("all") public class BatchConfiguration {/ * * injection JobBuilderFactory * / @ Autowired public JobBuilderFactory jobBuilderFactory; / * * injection StepBuilderFactory * / @ Autowired public StepBuilderFactory stepBuilderFactory; / * * injection JobRepository * / @ Autowired public JobRepository jobRepository / * inject JobLauncher * / @ Autowired private JobLauncher jobLauncher; / * inject custom StudentService * / @ Autowired private StudentService studentService; / * inject custom job * / @ Autowired private Job studentJob / * * Encapsulation writer bean * * @ return * / @ Bean public ItemWriter writer () {ItemWriter writer = new ItemWriter () {@ Override public void write (List list) throws Exception {/ / debug found that the thread List of the nested List reader nests the real List list.forEach ((stu)) -> {for (Student student: (ArrayList) stu) {studentService.insertStudent (student) });}}; return writer } / * * encapsulate reader bean * * @ return * / @ Bean public ItemReader reader () {ItemReader reader = new ItemReader () {@ Override public Object read () throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {/ / Analog data acquisition StudentVirtualDao virtualDao = new StudentVirtualDao (); return virtualDao.getStudents () }}; return reader } / * * Encapsulation processor bean * * @ return * / @ Bean public ItemProcessor processor () {ItemProcessor processor = new ItemProcessor () {@ Override public Object process (Object o) throws Exception {/ / debug discovers that o is the data return o read by reader in a single single thread;}} Return processor } / * encapsulate custom step * * @ return * / @ Bean public Step studentStepOne () {return stepBuilderFactory.get ("studentStepOne") .chunk (1) .reader (reader ()) / / add reader .processor (processor ()) / / add processor .writer (writer ()) / / add writer .build () } / * encapsulate custom job * * @ return * / @ Bean public Job studentJob () {return jobBuilderFactory.get ("studentJob") .flow (studentStepOne ()) / add step .end () .build () } / * use spring scheduled task execution * / @ Scheduled (fixedRate = 5000) public void printMessage () {try {JobParameters jobParameters = new JobParametersBuilder () .addLong ("time", System.currentTimeMillis ()) .toJobParameters (); jobLauncher.run (studentJob, jobParameters) } catch (Exception e) {e.printStackTrace ();} 3.3 Test

1s after the start of the project

If you look at the database, there are so many tables besides the tables defined by our entity class. These tables are logs and error tables that come with spring batch. The specific meaning of the fields needs to be studied.

4 summary after actual combat

Spring Batch has a very fast write and read speed, but the impact is that it is very memory-consuming and database connection pool resources will be abnormal if not used properly, so we need to make the correct configuration. Next, let's do a simple source code exploration:

4.1 JobBuilderFactory

The acquisition of job uses simple factory pattern and builder mode JobBuilderFactory to get JobBuilder to return an instance of a job object after configuration, which is the top-level component in Spring Batch, including n and step

Public class JobBuilderFactory {private JobRepository jobRepository; public JobBuilderFactory (JobRepository jobRepository) {this.jobRepository = jobRepository;} / / returns JobBuilder public JobBuilder get (String name) {JobBuilder builder = new JobBuilder (name) .repository (jobRepository); return builder;}}

JobBuilder class

Public class JobBuilder extends JobBuilderHelper {/ * creates a new builder * / public JobBuilder (String name) {super (name);} / * creates a new job builder that will execute a step or sequence of steps. * / public SimpleJobBuilder start (Step step) {return new SimpleJobBuilder (this) .start (step);} / * create a new job builder that will execute the flow. * / public JobFlowBuilder start (Flow flow) {return new FlowJobBuilder (this) .start (flow);} / * create a new job builder that will execute steps or step sequences * / public JobFlowBuilder flow (Step step) {return new FlowJobBuilder (this) .start (step);} 4.2 StepBuilderFactory

Look directly at the StepBuilder class

Public class StepBuilder extends StepBuilderHelper {public StepBuilder (String name) {super (name);} / * * build steps with custom tasklets, not necessarily an item processing. * / public TaskletStepBuilder tasklet (Tasklet tasklet) {return new TaskletStepBuilder (this) .tasklet (tasklet);} / * * build a step to process items as blocks according to the size provided. To extend this step to fault tolerance, * call the faultolerant () method of SimpleStepBuilder on the builder. * @ param input type * @ param output type * / public SimpleStepBuilder chunk (int chunkSize) {return new SimpleStepBuilder (this) .chunk (chunkSize);} public SimpleStepBuilder chunk (CompletionPolicy completionPolicy) {return new SimpleStepBuilder (this) .chunk (completionPolicy);} public PartitionStepBuilder partitioner (String stepName, Partitioner partitioner) {return new PartitionStepBuilder (this) .partitioning (stepName, partitioner) } public PartitionStepBuilder partitioner (Step step) {return new PartitionStepBuilder (this) .step (step);} public JobStepBuilder job (Job job) {return new JobStepBuilder (this) .job (job);} / * create a new step builder that will execute the flow. * / public FlowStepBuilder flow (Flow flow) {return new FlowStepBuilder (this) .flow (flow);}} above is all the content of the article "sample Analysis of the Spring Batch lightweight batch Framework". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.