Shulou(Shulou.com)06/03 Report--

Storm applicable scenario

Stream aggregation:

Flow aggregation is the aggregation of two or more data streams into a single data stream-based on some common tuple fields.

Builder.setBolt (5 ~ new MyJoiner (), parallelism)

.fieldsGrouping (1JI new Fields ("joinfield1", "joinfield2"))

.fieldsGrouping (2je new Fields ("joinfield1", "joinfield2"))

.fieldsGrouping (3je new Fields ("joinfield1", "joinfield2"))

Batch processing:

Sometimes for performance or other reasons, you may want to deal with a set of tuple together, rather than individually.

BasicBolt:

A, read an input tuple

B. Emit one or more tuple according to this input tuple

C. Enter tuple in the last ack of the execute method

Bolt that follows this type of pattern is usually a function or filter, which is so common that storm encapsulates a separate interface for this type of pattern: IbasicBolt.

In-memory cache + Fields grouping combination

It is very common to cache something in bolt memory. Caching is even more useful when combined with fields grouping. For example, you have a bolt that turns short links into long links (bit.ly,t.co and the like). You can cache short links to long links in memory using LRU scores to avoid double counting. For example, component one launches short links, and component two converts short links into growing links and caches them in memory. Take a look at the difference between the following two codes:

Builder.setBolt (2 new ExpandUrl (), parallelism) .shuffleGrouping (1)

Builder.setBolt (2) new ExpandUrl (), parallelism) .fieldsGrouping (1) new Fields ("url")

Calculate top N

For example, you have a tuple like bolt emission: "value", "count" and you want a bolt to calculate the tuple of top N based on this information. The easiest way is to have a bolt that can do a global grouping action and keep the value of top N in memory.

This approach is obviously not scalable for streams with large amounts of data, because all data will be sent to the same machine. A better approach is to calculate the top N of each part of the stream in parallel on multiple machines, and then a bolt merges the top N calculated on these machines to calculate the final top N, the code looks something like this:

Builder.setBolt (2) new RankObjects (), parallellism) .fieldsGrouping (1) new Fields ("value")

Builder.setBolt (3) new MergeObjects (). GlobalGrouping (2)

This pattern is successful because the first bolt's fieldsgrouping makes this parallel algorithm semantically correct.

Use TimeCacheMap to efficiently save the cache of a recently updated object:

Sometimes you want to keep some recently active objects in memory, as well as those that are no longer active. TimeCacheMap is a very efficient data structure that provides callback functions that allow us to do something when the object is no longer active.

Distributed RPC:CoordinatedBolt and KeyedFairBolt:

There are two common modes when using storm to do distributed RPC applications: they are encapsulated in CoordinatedBolt and KeyedFairBolt.

CoordinatedBolt wraps your bolt and determines when your bolt has received all the tuple. It mainly uses Direct Stream to do this.

KeyedFairBolt also wraps your bolt and ensures that your topology handles multiple DRPC calls at the same time, rather than executing only one serial at a time.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.