Shulou(Shulou.com)05/31 Report--

This article mainly introduces "what is the Grouping of Storm". In the daily operation, I believe that many people have doubts about the Grouping of Storm. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful for you to answer the doubts about "what is the Grouping of Storm?" Next, please follow the editor to study!

# # Storm Grouping

ShuffleGrouping

Flow grouping is defined as mixed arrangement. This mixed grouping means that input from Spout will be mixed or randomly distributed to tasks in this Bolt. Shuffle grouping distributes evenly the tuple of each task.

FieldsGrouping

This grouping mechanism ensures that tuple with the same field value will go to the same task, which is very critical for WordCount. If the same word does not go to the same task, then the counted number of words is not correct.

All grouping

Broadcast transmission, for each tuple will be copied to each bolt for processing.

Global grouping

All tuple in Stream will be sent to the same bolt task processing, and all tuple will be sent to bolt task processing with the smallest task_id.

None grouping

This method is not concerned with parallel processing load balancing strategy, which is currently equivalent to shuffle grouping, and storm will arrange the bolt task under the same thread as its upstream data-providing task.

Direct grouping

The transmitting unit of the tuple directly determines which bolt the tuple will be transmitted to. In general, it is up to the bolt that receives the tuple to decide which bolt-transmitted Tuple to receive. This is a special grouping method, which means that the sender of the message specifies which task of the message receiver will process the message. Only message flows declared as Direct Stream can declare this grouping method. And the message tuple must be emitted using the emitDirect method. The message handler can use TopologyContext to get the taskid of the message that processes it (the OutputCollector.emit method also returns taskid)

# # fieldsGrouping

If you know Storm, I think you can understand most of the Grouping. The Grouping strategy here I would like to focus on fieldsGrouping, but also the most difficult to understand.

FieldsGrouping is grouped by the value of the field Field in the data. Here is my test code:

TopologyBuilder builder = new TopologyBuilder (); builder.setSpout ("words", new TestWordSpout (), 2); builder.setBolt ("exclaim2", new DefaultStringBolt (), 5) .fieldsGrouping ("words", new Fields ("word"))

The test example, Spout, is an example that comes with Storm. The Blot code is as follows:

Public void execute (Tuple tuple) {log.info ("rev a message:" + tuple.getString (0)); collector.emit (tuple, new Values (tuple.getString (0) + "!!")); collector.ack (tuple);} public void declareOutputFields (OutputFieldsDeclarer declarer) {declarer.declare (new Fields ("word");}

The example Spout that comes with Storm can randomly return several strings in the new String [] {"nathan", "mike", "jackson", "golda", "bertels"}; list. This is also a good example of testing FieldGroup.

According to my understanding before I first started to do Storm, since it is grouped by Field, then all the same Field worth data will reach a Blot. I have tested it many times, and the result is not like this. A Blot will receive several different values. I did not carefully explore what is so special about Storm grouping that my study of Storm has been stagnant for a long time.

Storm guarantees that all data with the same field value will arrive at the same Blot, but there is no guarantee that a Blot will only handle one range.

That is to say, all values nathan can reach a Blot, but there may be multiple values that reach the same Blot, such as "nathan" and "mike" data.

With this in mind, fieldsGrouping can be regarded as understanding.

The following is the test log:

9144 [Thread-35-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: bertels9234 [Thread-35-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: mike9245 [Thread-33-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: nathan9335 [Thread-26-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: golda9346 [Thread-26-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: golda9437 [Thread-35 -exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: jackson9447 [Thread-35-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: mike9537 [Thread-26-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: golda9548 [Thread-35-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: jackson9639 [Thread-33-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: nathan9649 [Thread-35-exclaim2] INFO cn .pointways.dstorm.bolt.DefaultStringBolt-rev a message: jackson9740 [Thread-33-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: nathan9749 [Thread-35-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: jackson9841 [Thread-35-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: bertels9850 [Thread-26-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: golda

As you can see from the log above, the data of the value of golda is indeed merged into a Blot. Thread number: Thread-26-exclaim2. All other values are the same. All values are processed in one thread.

At this point, the study of "what are the Grouping of Storm" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.