Shulou(Shulou.com)06/01 Report--

This article shows you what is the implementation of big data text parallel computing, the content is concise and easy to understand, it can definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

For the processing of big data files, we can make full use of the multi-core CPU of modern computer to implement multi-thread parallel computing, so as to achieve the purpose of increasing speed. However, it is not an easy task to write multithreaded parallel programs in a programming language.

To process in parallel, you need to segment the source file, and each thread processes one of the segments. Text files generally correspond to one record per line, and each line is not necessarily the same in length. Therefore, it cannot be segmented by the number of rows, because traversing from scratch each time does not achieve the goal of improving performance; if it is segmented by bytes, there is no need to traverse, but it is possible that the segmentation point falls right in the middle of the row, resulting in a row being split into two segments, resulting in data errors. The effective solution is to adopt the byte segmentation mechanism of automatic head and tail, that is, the line where the segment start point is discarded and the row where the segment end point is made up, which will ensure that each segment is made up of complete lines and there will be no data errors. In addition, there are problems such as thread management and control, which will be out of bounds if not managed properly.

It would be much easier if there was an aggregator SPL, which encapsulates multithreaded parallel computing so that the code is not only shorter but also easier to understand, so that while achieving high performance, programmers focus more on the overall logic of computing rather than on the parallel details used to improve performance. Write the parallel computing code with the aggregator SPL:

A

B

C

one

= file ("data.txt")

/ source file

two

Fork 4

= A1.cursor@t (amount;A2:4)

/ divided into 4 segments in parallel to set up cursors respectively

three

= B2.groups (; sum (amount): am)

/ calculate the sum of amount by traversing cursors

four

A2.conj () .sum (am)

/ summarize the results of each thread

Text parsing often takes much longer than computing, and sometimes as long as parsing can be parallel, it doesn't matter whether the computation itself is parallel or not. So SPL provides a simple built-in parallel option for reading data. If you don't care about the data reading order, such as grouping and summation, you can write code more easily:

A

B

one

= file ("orders.txt") .cursor@mt ()

The / @ m option automatically determines the number of parallel threads based on the system configuration

two

= A1.select (month (Date) = = 10)

/ filter

three

= A2.groups (ID;sum (COST*WEIGHT): VALUE)

/ grouping, summarizing (serial)

The above content is what is the implementation of big data text parallel computing? have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.