Shulou(Shulou.com)06/03 Report--

This article mainly explains "how to use the Spring Batch batch framework", the content of the article is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "how to use the Spring Batch batch framework" bar!

1 preface

Spring Batch is a lightweight and perfect batch processing framework. As a member of Spring system, it has the characteristics of flexibility, convenience and production availability. It is very simple to deal with scenarios such as efficient processing of a large amount of information and regular processing of a large amount of data.

Combined with the scheduling framework, the role of Spring Batch can be brought into full play.

2 Conceptual knowledge of Spring Batch

2.1 hierarchical architecture

The hierarchical architecture of Spring Batch is as follows:

Through examples to explain the introduction to Spring Batch, excellent batch processing framework

You can see that it is divided into three layers, namely:

Application application layer: contains all tasks batch jobs and developer-defined code, mainly business processes developed according to the needs of the project, and so on.

Batch Core core layer: contains runtime environment classes for launching and managing tasks, such as JobLauncher, etc.

Batch Infrastructure basic layer: the upper two layers are built on top of the basic layer, including basic read-in reader and write-out writer, retry framework, etc.

2.2 key concepts

It is important to understand the concepts involved in the following figure, otherwise it is difficult to carry out follow-up development and problem analysis.

Through examples to explain the introduction to Spring Batch, excellent batch processing framework

2.2.1 JobRepository

Specially responsible for dealing with the database, recording the addition, update and execution of the entire batch. So Spring Batch needs to rely on the database to manage.

2.2.2 Task initiator JobLauncher

Responsible for starting the task Job.

2.2.3 Task Job

Job is the unit that encapsulates the entire batch process. To run a batch task is to run the content defined by a Job.

Through examples to explain the introduction to Spring Batch, excellent batch processing framework

The figure above introduces some related concepts of Job:

Job: encapsulates the processing entity and defines the process logic.

The running instance of JobInstance:Job varies with different parameters, so after defining a Job, you can run it multiple times with different parameters.

JobParameters: the parameter associated with the JobInstance.

JobExecution: represents an actual execution of Job that may succeed or fail.

So, what developers need to do is define Job.

2.2.4 step Step

Step is the encapsulation of a process of Job. A Job can contain one or more Step. Step-by-step Step execution according to specific logic represents the completion of Job execution.

Through examples to explain the introduction to Spring Batch, excellent batch processing framework

By defining Step to assemble Job, you can implement complex business logic more flexibly.

2.2.5 input-processing-output

Therefore, the key to defining a Job is to define one or more Step and then assemble them. There are many ways to define Step, but one commonly used model is input-process-output, namely Item Reader, Item Processor, and Item Writer. For example, input data from a file through Item Reader, then carry out business processing and data conversion through Item Processor, and finally write to the database through Item Writer.

Spring Batch provides us with many out-of-the-box Reader and Writer, which is very convenient.

3 code examples

Once you understand the basic concepts, feel it directly through the code. The function of the whole project is to read data from multiple csv files and output them to a csv file after processing.

3.1 basic framework

Add dependencies:

Org.springframework.boot spring-boot-starter-batch com.h3database h3 runtime

Need to add Spring Batch dependency, while using H2 as an in-memory database is more convenient, the actual production is definitely to use external databases, such as Oracle, PostgreSQL.

Entry main class:

@ SpringBootApplication @ EnableBatchProcessing public class PkslowBatchJobMain {public static void main (String [] args) {SpringApplication.run (PkslowBatchJobMain.class, args);}}

It's also simple, just add the annotation @ EnableBatchProcessing to Springboot.

Domain entity class Employee:

Package com.pkslow.batch.entity; public class Employee {String id; String firstName; String lastName;}

The contents of the corresponding csv file are as follows:

Id,firstName,lastName 1,Lokesh,Gupta 2,Amit,Mishra 3,Pankaj,Kumar 4,David,Miller

3.2 input-processing-output

3.2.1 read ItemReader

Because there are multiple input files, the definition is as follows:

@ Value ("input/inputData*.csv") private Resource [] inputResources; @ Bean public MultiResourceItemReader multiResourceItemReader () {MultiResourceItemReader resourceItemReader = new MultiResourceItemReader (); resourceItemReader.setResources (inputResources); resourceItemReader.setDelegate (reader ()); return resourceItemReader;} @ Bean public FlatFileItemReader reader () {FlatFileItemReader reader = new FlatFileItemReader (); / / Skip the first line of the csv file with the header reader.setLinesToSkip (1) Reader.setLineMapper (new DefaultLineMapper () {{setLineTokenizer (new DelimitedLineTokenizer () {{/ / Field name setNames (new String [] {"id", "firstName", "lastName"});}}) SetFieldSetMapper (new BeanWrapperFieldSetMapper () {{/ / converted target class setTargetType (Employee.class);}});}}); return reader;}

FlatFileItemReader is used here to make it easy for us to read data from files.

3.2.2 processing ItemProcessor

For a simple demonstration, the process is simple: convert the last column to uppercase:

Public ItemProcessor itemProcessor () {return employee-> {employee.setLastName (employee.getLastName () .toUpperCase ()); return employee;};}

3.2.3 output ItremWriter

Relatively simple, the code and comments are as follows:

Private Resource outputResource = new FileSystemResource ("output/outputData.csv"); @ Bean public FlatFileItemWriter writer () {FlatFileItemWriter writer = new FlatFileItemWriter (); writer.setResource (outputResource); / / whether it is append mode writer.setAppendAllowed (true); writer.setLineAggregator (new DelimitedLineAggregator () {{/ / set delimiter setDelimiter (",") SetFieldExtractor (new BeanWrapperFieldExtractor () {{/ / set fields setNames (new String [] {"id", "firstName", "lastName"});}});}}); return writer;}

3.3 Step

With Reader-Processor-Writer, you can define Step:

@ Bean public Step csvStep () {return stepBuilderFactory.get ("csvStep") .chunk (5) .reader (multiResourceItemReader ()) .processor (itemProcessor ()) .writer (writer ()) .build ();}

There is a chunk setting with a value of 5, which means 5 records before submitting the output, which can be defined according to your own needs.

3.4 Job

With the coding of Step completed, it is easy to define Job:

@ Bean public Job pkslowCsvJob () {return jobBuilderFactory .get ("pkslowCsvJob") .incrementer (new RunIdIncrementer ()) .start (csvStep ()) .build ();}

3.5 run

After completing the above coding, execute the program, and the result is as follows:

Through examples to explain the introduction to Spring Batch, excellent batch processing framework

The data is read successfully, and the last field is capitalized and output to the outputData.csv file.

4 listening to Listener

You can listen for specific events through the Listener interface to achieve more business functions. For example, if the processing fails, a failure log is recorded; when the processing is completed, the downstream is notified to take the data, and so on.

We listen for Read, Process and Write events respectively, and implement ItemReadListener interface, ItemProcessListener interface and ItemWriteListener interface respectively. Because the code is relatively simple, just print the log. Only the implementation code of ItemWriteListener is posted here:

Public class PkslowWriteListener implements ItemWriteListener {private static final Log logger = LogFactory.getLog (PkslowWriteListener.class); @ Override public void beforeWrite (List

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.