Shulou(Shulou.com)06/03 Report--

Storm is an open source distributed real-time computing system, which can handle a large number of data streams simply and reliably. The deployment and operation and maintenance of Storm are very convenient, and more importantly, you can use any programming language to develop applications.

Storm: real-time computing system

Low latency, high performance, distributed, scalable, fault tolerant

Features: simple programming model, hot deployment, various programming languages, extensibility, fault tolerance, reliable message processing, fast, local mode

Basic concepts of storm:

Nimbus: responsible for resource allocation and task scheduling

Supervisor: responsible for accepting tasks assigned by nimbus, starting and stopping worker processes managed by yourself

Worker: runs a process that specifically deals with component logic

The thread of each spout/bolt in Task:worker is called a task. After storm0.8, the task no longer corresponds to the physical thread, and the task of the same spout/bolt may share a physical thread, which is called executor.

A real-time application running in Topology:storm because the flow of messages between components forms a logical topology

Spout: a component that generates source data in a topology. Typically, spout reads the data from an external data source and converts it to the internal source data of the topology. Spout is an active role with a function of nextTuple () in its interface. The storm framework will keep calling this function, and users only need to generate source data in it.

Bolt: a component that accepts data in a topology and then performs processing. Bolt can perform filtering, function operations, merging, writing to the database, and so on. Bolt is a passive role, and there is an execute (Tupleinput) function in its interface, which is called after receiving a message, where users can perform the operation they want.

Tuple: the basic unit of a message delivery. It should be a key-value map, but since the field names of the tuple passed between components have been defined in advance, all you have to do in the Tuple is to fill in each value in order, which is a valueList.

Stream: the continuous flow of Tuple makes up stream.

Strom usage scenarios:

1. Stream aggregation: aggregates two or more data streams into a single data stream-based on some common tuple fields.

two。 Batch processing: for performance or other reasons, a set of tuple is processed together, rather than individually.

3.BasicBolt

a. Read an input tuple

b. Emit one or more tuple based on an input tuple

c. At the end of the execute method, the ack that the input tuple follows is usually a function or filter, which is so common that storm encapsulates a separate interface for this type of pattern: IbasicBolt

d. In-memory cache + Fields grouping combination

e. It is calculated that topN

f. Use TimeCacheMap to efficiently save the cache of a recently updated object

g. Distributed RPC

Storm grouping mechanism

1. Random grouping (Shuffle grouping)

two。 Field grouping (Fields grouping)

3. Grouping all (All grouping)

4. Global grouping (Global grouping)

5. No grouping (None grouping)

6. Direct grouping (Direct grouping)

7. Implement the CustomStreamGrouping interface to define your own grouping

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.