Shulou(Shulou.com)06/03 Report--

Brief introduction

Strom is an open source distributed streaming computing system, which is used to deal with streaming data, which is called streaming hadoop. In the telecommunications industry, it can be used to do large traffic early warning, terminal marketing, access to competitors' products to retain and other services. This article will introduce storm in detail from the position of storm in the hadoop ecosystem, terms in storm, storm platform construction, storm application construction and so on.

The position of Strom in big data's biosphere

As can be seen in the figure above, Storm is above HDFS, but it does not mean that Storm can only deal with data in HDFS. Instead, the data source of Storm is usually Log log or data in Kafka. When the data is processed through Strom, its flow can be HDFS, HBase, relational database, etc.

Strom is a computing system, in big data processing, we are familiar with the computing system is mapreduce, this architecture diagram shows that storm and mapreduce are peer relationship, and storm is called streaming hadoop. So next we will introduce storm by comparing it with mapreduce.

Introduction to common terms of 3.Strom

StromMapReduce (based on hadoop2.X) describes that Resourcemanager in NimbusApplicationMasterMapReduce is responsible for task allocation and resource request, while Nimbus is also responsible for code distribution, task allocation and scheduling in Strom, NodeManager is responsible for resource application, work process startup and monitoring in SupervisorNodeManagerMapReducer, and Supervisor in Strom is also responsible for starting and stopping Worker of task process.

YarnChild the process TopologyMapReduce driver that is really responsible for task processing

4.

Strom architecture Topology: applications built through Storm describe the source of data, the logic of data processing, and the flow of data.

The component in Spout:Topology describes the source of the data through Spout. There is a nextTuple () function in Spout, which is called constantly. The source data is generated in this function, and then the data flows to the next node. Only one Spout is allowed in each Topology.

A component in Bolt:Topology that receives data from the previous node (Spout or Bolt). There is an execute (Tuple tuple) method in the component. When the data is received, the function is passively executed to merge, filter, persist, and so on. Bolt can be the end point of a complete data processing process in Topology, or a transfer point.

Tuple:Tuple is the basic unit of message delivery in Storm, and the data structure of Tuple is a List.

Stream: continuous Tuple makes up Stream.

Stream grouping: describes the rules for partition when data flows between different components (Spout/Bolt). The types are as follows:

1.shuffle Grouping: randomly group the tuple in the stream to ensure that the number of tuple received by each bolt is the same 2.Fields Grouping: grouped by field, the same field will be assigned to the same bolt, and different fields will be assigned to different Bolt3.ALL Grouping: broadcast, for each tuple, all bolt will receive 4.Global Grouping: global packet, and this tuple will be assigned to a task in a bolt of storm That is, the 5.Non Grouping with the lowest id: no grouping, the current effect is the same as Global Groouping 6.Direct Grouping: direct grouping, specifying the recipient of the message.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.