Shulou(Shulou.com)06/02 Report--

This article mainly introduces "Java multithreading how to import data in batches". In daily operation, I believe many people have doubts about how Java multithreading imports data in batches. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the doubts of "Java multithreading how to import data in batches". Next, please follow the editor to study!

Design ideas

As the scene is characterized by fast reading and slow writing, if multithreading is used, it is recommended that the data writing part be transformed into multithreading. And data reading can be transformed into batch reading data. To put it simply, there are two main points:

Read data in batches and write data in multiple threads

Example

The simplest solution for multithreaded batch processing is to use a thread pool for processing. The following shows how to import multithreaded batch data through a service that simulates batch reads and writes, as well as multithreaded write calls to this service.

Simulation service

Simulation service for bulk writing of import java.util.concurrent.atomic.AtomicLong;/*** data * * @ author RJH* create at 2019-04-01*/public class MockService {/ * * Total readable * / Total private long canReadTotal;/*** writes * / private AtomicLong writeTotal=new AtomicLong (0); / * write hibernation time (in milliseconds) * / private final long sleepTime / * Construction method * * @ param canReadTotal* @ param sleepTime*/public MockService (long canReadTotal, long sleepTime) {this.canReadTotal = canReadTotal;this.sleepTime = sleepTime;} / * bulk read data Interface * * @ param num* @ return*/public synchronized long readData (int num) {long readNum;if (canReadTotal > = num) {canReadTotal-= num;readNum = num;} else {readNum = canReadTotal;canReadTotal = 0;} / / System.out.println ("read data size:" + readNum) Return readNum;} / * write data interface * / public void writeData () {try {/ / hibernate for a certain period of time to simulate slow write speed Thread.sleep (sleepTime);} catch (InterruptedException e) {e.printStackTrace ();} / / the total number of writes is self-increasing System.out.println ("thread:" + Thread.currentThread () + "write data:" + writeTotal.incrementAndGet ()) } / * get the total number of writes * * @ return*/public long getWriteTotal () {return writeTotal.get ();}}

Batch data processor

Import java.util.concurrent.ExecutorService;import java.util.concurrent.Executors;/*** multithreaded bulk write processor based on thread pool * @ author RJH* create at 2019-04-01*/public class SimpleBatchHandler {data read per batch by private ExecutorService executorService;private MockService service;/*** * / number of private int batch;/*** threads * / private int threadNum;public SimpleBatchHandler (MockService service, int batch,int threadNum) {this.service = service;this.batch = batch / / use a fixed number of thread pools this.executorService = Executors.newFixedThreadPool (threadNum);} / * start processing * / public void startHandle () {/ / start processing time long startTime = System.currentTimeMillis (); System.out.println ("start handle time:" + startTime); long readData;while ((readData = service.readData (batch))! = 0) {/ / batch read data and stop for until the data cannot be read (long I = 0; I

Service.writeData ();}} / / close thread pool executorService.shutdown (); while (! executorService.isTerminated ()) {/ / wait for threads in the thread pool to finish execution} / / end time long endTime = System.currentTimeMillis (); System.out.println ("end handle time:" + endTime); / / Total time-consuming System.out.println ("total handle time:" + (endTime-startTime) + "ms") / / Total number of writes System.out.println ("total write num:" + service.getWriteTotal ());}}

Test class

/ * * SimpleBatchHandler test class * @ author RJH* create at 2019-04-01*/public class SimpleBatchHandlerTest {public static void main (String [] args) {/ / Total long total=100000;// hibernation time long sleepTime=100;// number of int batch=100;// threads pulled each time int threadNum=16;MockService mockService=new MockService (total,sleepTime); SimpleBatchHandler handler=new SimpleBatchHandler (mockService,batch,threadNum); handler.startHandle ();}}

Running result

Start handle time:1554298681755thread: Threadpool-1While threadly2jin5main] write data:1thread: threadpool-1Muthreadmer1jing5mai main] write data:2... Omit part of the output thread: Thread [pool-1 color threadmuri 4jin5recom main] write data:100000end handle time:1554299330202total handle time:648447mstotal write num:100000

Analysis.

In the case of a single thread, the execution time should be total*sleepTime, that is, 10000000ms, while the execution time should be 648447ms after the transformation to multithreading.

At this point, the study on "how to import data in batches by Java multithreading" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.