Shulou(Shulou.com)05/31 Report--

This article mainly explains "what is the relationship between Worker, Executor and Task". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what is the relationship between Worker, Executor and Task".

1 simple introduction

When Storm runs a Topology on a cluster, the Topology is executed mainly through the following three entities

1 Worker

2 Executor

3 Task

A Worker process executes a subset of topology. Here we must emphasize that there will be no one worker serving multiple topology.

A worker process starts one or more executor threads to execute a topology's compotent-, that is, Spout or bolt

A topology is made up of Worker on multiple physical machines in the middle of the cluster.

An executor is a separate thread started by the Worker process, and each Executor runs only one component of a topology.

By default, a spout, or a bolt, will only generate a real example of a task,Executor thread that calls all task sequentially during each loop.

The task is the unit that ends up running the code in spout or bolt (Note: 1 task is an instance of spout or bolt, and the executor thread calls the nextTuple or execute method of that task during execution). After topology starts, the number of task of a component (spout or bolt) is fixed, but the number of executor threads used by the component can be dynamically adjusted (for example, 1 executor thread can execute one or more task instances of the component). This means that for a component there is such a condition: # threadsnode+port, which is actually a mapping from task-id to supervisor-id+port, that is, assigning the task to a port on a machine to do.

The spout/bolt in the topology is divided into multiple task according to the parallelism, and these task are assigned to multiple worker of the supervisor to execute.

Task is associated with a componment-id. Componment is a general term for spout and bolt.

For each component that is specified at deployment time, there is a discussion in storm-user that illustrates this:

The main idea is to specify the number of threads executing spout/bolt by setting parallelism. There is another place in the configuration (backtype.storm.Config.setNumWorkers (int)) to specify the number of processes executing topolgy in a storm cluster, and all threads will run in these specified worker processes. For example, if a topology needs to start 300 threads to run spout/bolt, and the specified number of worker processes is 60, then storm will assign 5 threads to each worker to run spout/bolt. If you want to tune a topology, you can adjust the number of worker and the number of parallelism of spout/bolt (remember to redeploy topology after adjusting the parameters. A swapping function will be provided later for this operation to reduce the time to redeploy.

For the ratio between worker and task, nathan also gives a reference, that is, a worker contains about 10 to 15. Of course, this reference should be based on configuration and testing.

3: work process internal message passing processing and data structure analysis

This paper starts with the transformation, transmission and processing of external messages within the worker process, and analyzes the reason and significance of the existence of data items in worker-data step by step. From the point of view of code implementation, this paper tries to answer how to define message interfaces and how to implement message handling on their respective interfaces if you implement worker from scratch.

3.1Mapping from Topology to worker

Topology is made up of Spout,Bolt, and the logical relationship is roughly as follows

Please note that the behavior of Acker will not occur until tuple and other tuple generated by tuple are confirmed to be consumed.

With your ACK behavior

The processing logic of both Spout and Bolt needs to be executed within a process or thread, so how do they map to processes and threads? With regard to this issue, Understanding the Parallelism of a Storm Topology has made a good summary, and now repeat its main points.

1 worker is a process, executor corresponds to a thread, and spout or bolt are task one by one

2 the same worker will only execute the task related to the same topology

3 multiple task of the same type can be executed in the same executor, that is, in the same executor, either all the task of the bolt class or all the task of the spout class

4 when running, spout and bolt need to be packaged as one task after another

The relationship between the three components during the period is:

A brief summary, Worker=Process, Executor=Thread, Task=Spout or Bolt.

Each executor uses the processing logic of actor pattern,high level, as shown in the following figure

Thank you for reading, the above is the content of "what is the relationship between Worker, Executor and Task". After the study of this article, I believe you have a deeper understanding of what the relationship between Worker, Executor and Task is, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.