Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to execute distributed MapReduce on Java on Redis". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to implement distributed MapReduce on Java on Redis".

What is MapReduce?

MapReduce is a distributed computing program model that can be implemented with Java. The algorithm consists of two key tasks, called Map and Reduce.

The purpose of this Map task is to transform the dataset into another dataset, where elements are decomposed into key / value pairs called tuples. The Reduce task combines these data tuples into a small tuple, using the output of the map as input.

Distributed computing means dividing tasks into separate processes that can then be executed in parallel on a large commercial hardware cluster. Once MapReduce breaks down the elements of a large dataset into tuples and then further reduces them to smaller sets the remaining data can be processed in parallel which can significantly speed up the processing that needs to be performed on the data.

When do I need to use MapReduce to process Redis data?

In many cases, it is helpful to use it MapReduce to process Redis data. Usually, what they have in common is that you need to deal with a very large amount of data.

As a simple example, you can consider a situation where you have a large amount of monthly energy consumption data for your organization. Now suppose you need to process this data to generate results such as the maximum use year, the minimum use year, and so on, for each organization. While it is not difficult for experienced programmers to write algorithms to perform this processing, many of these algorithms will take a long time to execute if you have to run large amounts of data.

As a solution to long processing times, you can use MapReduce to reduce the overall size of the dataset, thus making processing faster. For many organizations, the reduction in processing time may be important because it frees up hardware so that it can be used for other computing tasks.

There are many more situations where MapReduce can be very useful using distributed data stored in Redis by Redisson. For example, MapReduce is especially useful if you need to calculate the word count of a very large file or collection of files quickly, reliably, and accurately.

An example of performing a distributed MapReduce on data stored in Redis

The following is an example of how to use MapReduce to create an effective algorithm that generates the exact number of words. This seems like a very simple task, but using MapReduce is very important to reduce the processing time of very large blocks of text or a large set of files.

Take a look at the following code to see how this algorithm uses and processes text data provided by RedissonMapReduce to reliably generate an accurate word count.

Step 1

Create a Redisson configuration:

/ / from JSON configuration config = configuration. FromJSON (...) / / from YAML configuration config = configuration. From YAML (...) / / or dynamic Config config = new Config ();... Step 2

Create a Redisson instance:

RedissonClient redisson = Redisson. Create (config); step 3

Define the Mapper object. This applies to each Map entry and divides the values by spaces to separate words:

The public class WordMapper implements RMapper

< String，String，String，Integer >

{

@ overwrite public void map (String key,String value,RCollector

< String，Integer >

Collector) {String [] words = value. Split ("[^ a-zA-Z]"); for (String word:words) {collector. Launch (word, 1);}

} step 4

Define the Reducer object. This calculates the sum of each word.

The public class WordReducer implements RReducer

< String，Integer >

{

@ overwrite public Integer reduce (String reducedKey,Iterator

< Integer >

Iter) {int sum = 0; and (ITER. HasNext () {integer I = (integer) iter. Next (); sum + = I;} return amount;}} step 5

Define the Collator object (optional). This calculates the total number of words.

The public class WordCollator implements RCollator

< String，Integer，Integer >

{

@ overwrite public Integer collate (Map

< String，Integer >

ResultMap) {int result = 0; is (integer count: result mapping. Value () {result + = count;}

Return the result; step 6

Here's how to run it together:

RMap

< String，String >

Map = redisson. GetMap ("wordsMap"); map. Put ("line1", "Alice began to get very tired"); map. Put ("line2", "sitting in her sister's bank and"); map. Play ("line3", "she has nothing to do once"); map. Put ("line4", "peeked at the book her sister read"); map. Put ("line5", "but it has no pictures or conversations"); map. Put ("line6", "and the use of books"); maps. Put ("line7", "thought Alice has no picture or conversation")

RMapReduce

< String，String，String，Integer >

MapReduce = map.

< String，Integer >

MapReduce (). Mapper (the new WordMapper ()). Reducer (new WordReducer ())

/ / calculate the number of occurrences of words Map

< String，Integer >

MapToNumber = mapReduce. Execute (); / / calculates the total number of words, an integer totalWordsAmount = mapReduce. Execute (new WordCollator ())

MapReduce can also be used to collect objects of type, including Set, SetCache, List, SortedSet, ScoredSortedSet, Queue, BlockingQueue, Deque, BlockingDeque, PriorityQueue, and PriorityDeque.

How to use Redisson to MapReduce data stored in Redis

Redisson is a state-of-the-art Redis client that offers unlimited possibilities for programming and data processing with Java. From the largest to the smallest startups, a wide variety of companies use Redisson to support their Java applications through Redis.

As a highly complex Redis client, Redisson provides distributed implementation of services, objects, collections, locks and synchronizers. It supports a range of Redis configurations, including single, clustered, tagged or master-slave configurations.

MapReduce if you have already used Redisson to store large amounts of data in Redis, using it is a good choice. Redisson provides a Java-based MapReduce programming model that can easily handle large amounts of data stored in Redis.

Thank you for reading, the above is the content of "how to execute distributed MapReduce on Java on Redis". After the study of this article, I believe you have a deeper understanding of how to implement distributed MapReduce on Java on Redis, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.