Shulou(Shulou.com)06/03 Report--

First, let's understand the basic concepts of storm by comparing a storm with a hadoop.

HadoopStorm System Role JobTrackerNimbusTaskTrackerSupervisorChildWorker Application Name JobTopology Component Interface Mapper/ReducerSpout/Bolt

Now let's look at these concepts in detail.

Nimbus: Responsible for resource allocation and task scheduling.

Supervisor: Responsible for accepting tasks assigned by nimbus, starting and stopping worker processes that belong to their own management.

c. Worker: A process that runs the logic of a specific processing component.

Task: Each spurt/bolt thread in a worker is called a task. After storm0.8, tasks no longer correspond to physical threads, and tasks of the same spout/bolt may share a physical thread, which is called executor.

The following diagram illustrates the relationship between these roles.

Topology: A real-time application running in a storm because the flow of messages between components forms a logical topology.

Spout: A component that produces a source data stream within a topology. Typically, spouts read data from external data sources and convert it to source data internal to a topology. Spout is an active actor with a nextTuple() function in its interface that the storm framework calls over and over again, where the user simply generates source data.

Bolt: A component that accepts data in a topology and performs processing. Bolt can perform filtering, function manipulation, merging, writing to databases, and anything else. Bolt is a passive role, its interface has an execute(Tuple input) function, after receiving the message will call this function, the user can perform their desired operation in it.

Tuple: The basic unit of a message delivery. It should be a key-value map, but since the field names of the tuple passed between the components have been defined in advance, it is only necessary to fill in the values in the tuple in order, so it is a value list.

Stream: Tuples that are passed continuously form a stream.

Storm grouping mechanism:

Stream Grouping defines how a stream should be split between Bolt tasks. Here are 6 Stream Grouping types provided by Storm:

Shuffle grouping: Randomly distribute tuples to Bolt tasks, ensuring that each task gets an equal number of tuples.

Fields grouping: Divide the data stream according to the specified fields and group them; for example, according to the "user-id" field, tuples of the same "user-id" are always distributed to the same task, and tuples of different "user-ids" may be distributed to different tasks.

All grouping: tuples are copied to all tasks of bolt, and this type needs to be used with caution.

Global grouping: All flows are assigned to the same task of the bolt, specifically, the task assigned to the ID with the latest.

None grouping: You don't need to care how churn is grouped. Currently, no grouping is equivalent to randomization. But eventually Storm will put ungrouped Bolts into the same thread to execute (if possible) as Bolts or Spouts subscribe to them.

Direct grouping: This is a special grouping type where the tuple producer decides which tuple handler task receives the tuple.

Of course, you can also implement the CustomGroup Streaming interface to customize the grouping you need.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.