Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "what are the Storm interview questions". In the actual case operation process, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to deal with these situations! I hope you can read carefully and learn something!

I. Structure

1、Nimbus

Responsible for resource allocation and task scheduling. The nimbus node in the new version can have multiple masters.

2、Zookeeper

Coordinates cluster, stores common data (such as heartbeat data, cluster status and configuration information), nimbus writes tasks assigned to Supervisor to Zookeeper

3、supervisor

Responsible for accepting tasks assigned by nimbus, starting and stopping worker processes that belong to their own management.

4、worker

A process that runs the logic of a specific process component. Each spout/bolt thread in a worker is called a task. After storm0.8, tasks no longer correspond to physical threads, and tasks of the same spout/bolt may share a physical thread, which is called executor. The latest version of Jstorm has abolished the concept of task

II. Programming model

1、Spout

Spout is a component that accepts external data sources, converts external data sources into internal Storm data, and sends it to Bolt using Tuple as the basic transmission unit. (Tuple is the basic unit of data transfer in Storm, which encapsulates a List object to store data.)

2、Bolt

Bolt is the data received from Spout or upstream bolt. Process according to business logic. Send it to the next Bolt or store it on some medium. The medium can be mongodb or mysql, or something else.

3. Degree of parallelism

Worker: represents a process

Executor: Indicates the thread started by worker

Task: The smallest unit of work that actually performs data processing (note that a task is not a thread)

Parallel degree setting: estimate the amount of data produced per second by upstream kafka, analyze each partition of topic

Data per second, partition =SpoutTask accepted data SpoutTask =partition

Worker setting: If the data volume is large, the number of workers is equal to the number of spouttasks

4, the message is not lost

ack mechanism, that is, every message sent by Spout

l Within the specified time, spout receives an ack response from the Acker, that is, the tuple is considered to be rejected.

l

continued bolt successfully processed

l If no ack response tuple from Acker is received within the specified time, the failure action is triggered, that is, the tuple is considered to have failed.

l or receive the fail response tuple sent by the Acker, which is also considered as a failure and triggers the fail action. With the Ack mechanism, every message sent by spout can be determined whether it was successfully processed or failed, so that developers can take action. For example, in Meta, if it is successfully processed, the offset can be updated, and when it fails, the data is repeatedly sent. Therefore, with the Ack mechanism, it is easy to ensure that all data is processed and no one is missed.

"Storm interview questions what" content is introduced here, thank you for reading. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.