Shulou(Shulou.com)06/03 Report--

Editor to share with you the example analysis of multi-processes and clusters in NodeJS, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

Processes and threads

"process" is the basic unit of resource allocation and scheduling in a computer system. We can understand that every time a computer starts a task, at least one process is created to handle it, and sometimes multiple processes are created, such as Chrome browser tabs. The purpose is to prevent a process from dying and the application stops working. "Thread" is the smallest unit of program execution flow. NodeJS is single-process and single-threaded by default. We call this process the main process, or we can create sub-processes through the child_process module to achieve multiple processes. We call these sub-processes "working processes" and are managed by the main process. By default, processes cannot communicate with each other, and all child processes execute tasks asynchronously.

Implementing multiple processes with spawn

1. Spawn creation child process

Execute a JS file in NodeJS, and if you want to execute another JS file at the same time (asynchronously) in this file, you can use spawn in the child_process module. Spawn can help us create a child process as follows.

/ / File: process.jsconst {spawn} = require ("child_process"); const path = require ("path"); / / create child process let child = spawn ("node", ["sub_process.js", "--port", "3000"], {cwd: path.join (_ _ dirname, "test") / / specify the current working directory of the child process}) / / trigger child.on ("error", err = > console.log (err)) when an error occurs; / / trigger child.on for child process exit ("exit", () = > console.log ("exit")); / / trigger child.on for child process shutdown ("close", () = > console.log ("close")); / / exit// close

The spawn method can help us create a child process, which is the return value of the method, and spawn receives the following parameters:

Command: command to run

Args: the type is an array. The first item in the array is the file name, followed by the command parameters and values of the execution file.

Options: option, object type, which is used to specify the current working directory of the child process and the communication rules of the main process, the child process, and so on. For more information, please see the official documentation.

The error event is triggered when the child process fails, the exit event is triggered when the child process exits, and the close event is triggered after the child process shuts down. Exit will be triggered after the child process task ends, but close may not be triggered.

/ / file: ~ the parameter console.log (process.argv) of the sub_process.js file executed by the test/sub_process.js// print subprocess

The parameters of the child process are printed through the above code, but we find that the main process window is not printed. We hope that the information of the child process can be fed back to the main process. To achieve communication, you need to configure the stdio attribute definition in the third parameter options when creating the child process.

2. Spawn defines input and output

/ / File: process.jsconst {spawn} = require ("child_process"); const path = require ("path") / / create child process let child = spawn ("node", ["sub_process.js", "--port", "3000"], {cwd: path.join (_ _ dirname, "test") / / specify the current working directory of the child process / / stdin: [process.stdin, process.stdout, process.stderr] stdio: [0,1,2] / / configure standard input, standard output, error output}) / / C:\ Program Files\ nodejs\ node.exe,g:\ process\ test\ sub_process.js,--port,3000// file: ~ test/sub_process.js// uses the standard output of the main process to output the parameter process.stdout.write (process.argv.toString ()) executed by the sub_process.js file

By configuring the stdio value of options as an array, the above two words have the same effect, indicating that the child process and the main process share the standard input, standard output, and error output of the main process. In fact, the communication between the main process and the child process is not realized, where 0 and stdin represent standard input, 1 and stdout represent standard output, and 2 and stderr represent error output.

In the above way, as long as the child process executes sub_process.js, it will output in the window. If we want to control whether the output is in the main process, that is, to implement the communication between the child process and the main process, see the usage below.

/ File: process.jsconst {spawn} = require ("child_process"); const path = require ("path"); / / create child process let child = spawn ("node", ["sub_process.js"], {cwd: path.join (_ _ dirname, "test"), stdio: ["pipe"]}); child.stdout.on ("data", data = > console.log (data.toString () / / hello world// file: ~ the test/sub_process.js// child process executes sub_process.jsprocess.stdout.write ("hello world")

If the value of the array in stdio is configured as pipe (pipe is not written by default), the communication between the main process and the child process is realized through the stream, which is written through the standard output of the child process (writable stream). The stream read by the standard output of the child process through the data event in the main process is output to the window (which is rarely used). Only one child process is opened in the main process. Here is an example of starting multiple processes.

The example scenario is that the main process starts two sub-processes, the first runs the child process 1 to pass some parameters, the child process 1 returns the parameters to the main process, and the main process passes the parameters to the child process 2, and writes the parameters into the file param.txt through the child process 2. This process does not represent the real application scenario, but the main purpose is to understand the communication process between the main process and the child process.

/ File: process.jsconst {spawn} = require ("child_process"); const path = require ("path"); / / create child process let child1 = spawn ("node", ["sub_process_1.js", "--port", "3000"], {cwd: path.join (_ _ dirname, "test"),}) Let child2 = spawn ("node", ["sub_process_2.js"], {cwd: path.join (_ _ dirname, "test"),}); / / read the contents written by child process 1 and write to child process 2child1.stdout.on ("data", data = > child2.stdout.write (data.toString)) / / file: ~ test/sub_process_1.js// get-- port and 3000process.argv.slice (2) .forEach (item = > process.stdout.write (item)); / / file: ~ test/sub_process_2.jsconst fs = require ("fs"); / / read the parameters passed by the main process and write to the file process.stdout.on ("data", data = > {fs.writeFile ("param.txt", data, () = > {process.exit ()) );})

It should be noted that when the child process 2 writes to the file, the main process will be stuck because the main process does not know when the child process 2 will finish writing, so the child process needs to call the process.exit method to exit the child process after the writing is completed, and the main process will shut down after the child process exits and shuts down.

When we configure stdio for options, standard input, standard output and error output can be configured separately in the array. If the default array is pipe, all three are pipe. See the following example for configuration.

/ File: process.jsconst {spawn} = require ("spawn"); const path = require ("path"); / / create child process let child = spawn ("node", ["sub_process.js"], {cwd: path.join (_ _ dirname, "test"), stdio: [0, "pipe", 2]}); / / world// file: ~ test/sub_process.jsconsole.log ("hello"); console.error ("world")

The above code for stderr to achieve the default print and no communication, the standard input to achieve a communication, there is another case, if the hope child process is only silent implementation of the task, and in the main process command window what type of output are prohibited, you can give the value ignore in the corresponding position in the array, modify the above case as follows.

/ File: process.jsconst {spawn} = require ("spawn"); const path = require ("path"); / / create child process let child = spawn ("node", ["sub_process.js"], {cwd: path.join (_ _ dirname, "test"), stdio: [0, "pipe", "ignore"]}); / / File: ~ test/sub_process.jsconsole.log ("hello"); console.error ("world")

This time we found that both standard output and error output are not effective, the above methods are actually not very convenient, because the output has stdout and stderr, there is no way to unify the writing, can be unified through the following ways.

3. Standard process communication

/ File: process.jsconst {spawn} = require ("spawn"); const path = require ("path"); / / create child process let child = spawn ("node", ["sub_process.js"], {cwd: path.join (_ _ dirname, "test"), stdio: [0, "pipe", "ignore", "ipc"]}); child.on ("message", data = > {console.log (data)) / reply message to child process child.send ("world"); / / kill child process / / process.kill (child.pid);}); / / hello// file: ~ test/sub_process.js// sends message process.send ("hello") to main process; / / receives message process.on ("message", data = > {console.log (data)) from main process; / / exits child process process.exit () }) / / world

This method is called standard process communication. By configuring ipc to the stdio array of options, as long as there is ipc in the array, it is usually placed at the beginning or end of the array. After configuring ipc, the child process sends a message to the main process by calling its own send method. The main process receives it with the message event of the child process, and it can also reply through the send message of the child process during the callback of the message event that receives the message in the main process. And in the sub-process to receive with message events, this programming method is more unified, closer to the wishes of developers.

4. Exit and kill child processes

In the above code, the sub-process exits directly when it receives the message from the main process, or when the child process sends a message to the main process, the main process receives the message to kill the child process directly, the code is as follows.

/ File: process.jsconst {spawn} = require ("spawn"); const path = require ("path"); / / create child process let child = spawn ("node", ["sub_process.js"], {cwd: path.join (_ _ dirname, "test"), stdio: [0, "pipe", "ignore", "ipc"]}); child.on ("message", data = > {console.log (data)) / / kill child process process.kill (child.pid);}); / / hello world// file: ~ test/sub_process.js// sends message process.send ("hello") to the main process

From the above code, we can see that the method to kill the child process is process.kill, because a main process may have multiple child processes, so specify that the child process to be killed needs to pass the pid property of the child process as the parameter of process.kill.

{% note warning%}

Note: the exit child process process.exit method operates in the child process, where the process represents the child process, the kill child process process.kill is operated in the main process, and the process represents the main process.

{% endnote%}

5. Independent child process

As we said earlier, the child processes created by the child_process module are uniformly managed by the main process. if the main process dies, all the child processes will be affected to die together, but in fact, using multiple processes on the one hand is to improve the efficiency of processing tasks, on the other hand, there are other processes that can continue to work when one process dies, so that the whole application will not fail. There are many examples like this. For example, the tabs of Chrome browsers, such as the VSCode editor, will open multiple processes to process tasks at the same time. In fact, when spawn creates a child process, it can also achieve the independence of the child process, that is, the child process is no longer controlled and affected by the main process.

/ File: process.jsconst {spawn} = require ("spawn"); const path = require ("path"); / / create child process let child = spawn ("node", ["sub_process.js"], {cwd: path.join (_ _ dirname, "test"), stdio: "ignore", detached: true}); / / sever the relationship with the main process child.unref (); / / File: ~ test/sub_process.jsconst fs = require ("fs") SetInterval () = > {fs.appendFileSync ("test.txt", "hello");})

If you want to create a child process independently, you need to configure the detached parameter true when you create the child process, indicating that the child process is out of control, and you need to call the unref method of the child process to sever the relationship with the main process. But only this process may still be affected by the main process. If the child process is completely independent, it is necessary to ensure that the child process cannot share standard input, standard output and error output with the main process. That is, stdio must be set to ignore, which means that independent child processes cannot communicate with the standard process of the main process, that is, ipc cannot be set.

Implementing multiple processes with fork

1. The use of fork

Fork is also a method of the child_process module, similar to spawn, is to do a layer of encapsulation on the basis of spawn, we see an example of the use of fork.

/ File: process.jsconst fork = require ("child_process"); const path = require ("path"); / / create child process let child = fork ("sub_process.js", ["- port", "3000"], {cwd: path.join (_ _ dirname, "test"), silent: true}); child.send ("hello world") / / file: ~ test/sub_process.js// receives the message process.on ("message", data = > console.log (data)) from the main process

The usage of fork has changed compared with spawn. The first parameter is the name of the execution file of the child process, the second parameter is the array, which stores the parameters and values at the time of execution, and the third parameter is options, in which the stdio of spawn is replaced by the slilent attribute. When silent is true, all non-standard communication operations between the main process and the child process will not take effect, including standard input, standard output and error output. When set to false, the output can be normal. The return value is still a child process.

The child process created by fork can communicate with the main process directly through the send method and listening for message events.

2. The principle of fork

In fact, the principle of fork is very simple, just hang a fork method on the child process module child_process, and call spawn in this method and return the child process returned by spawn as the return value.

/ / File: fork.jsconst childProcess = require ("child_process"); const path = require ("path"); / / Encapsulation principle childProcess.fork = function (modulePath, args, options) {let stdio = options.silent? ["ignore", "ipc"]: [0,1,2, "ipc"]; return childProcess.spawn ("node", [modulePath,... args], {... options, stdio}) } / / create a child process let child = fork ("sub_process.js", ["- port", "3000"], {cwd: path.join (_ _ dirname, "test"), silent: false}); / / send a message child.send ("hello world") to the child process; / / File: ~ test/sub_process.js// receives a message process.on ("message", data = > console.log (data)) from the main process; / / hello world

There are some parameters not passed by fork in spawn (such as using node execution file), which pass default values when calling spawn internally or integrate the default parameters with the parameters passed in fork, focusing on the parameter silent that spawn does not have. In fact, it is handled into two extreme cases of stdio parameters of spawn (default uses ipc communication). Encapsulating fork is to make it easier for us to create child processes and pass fewer parameters.

ExecFile and exec implement multi-process

ExecFile and exec are two methods of child_process module. ExecFile is based on spawn encapsulation, while exec is based on execFile encapsulation. These two methods are more or less the same. ExecFile can directly create child processes for file operations, while exec can directly open child processes to execute commands. Common application scenarios such as http-server and weboack-dev-server command line tools automatically open browsers when starting local services.

/ execFile and execconst {execFile, exec} = require ("child_process"); let execFileChild = execFile ("node", ["- version"], (err, stdout, stderr) = > {if (error) throw error; console.log (stdout); console.log (stderr);}); let execChild = exec ("node-version", (err, stdout, stderr) = > {if (err) throw err; console.log (stdout); console.log (stderr);})

Exec differs from execFile in passing parameters. The first parameter of execFile is the executable path or command of the file, the second parameter is the set of parameters of the command (array), the third parameter is options, the last parameter is the callback function, and the formal parameters of the callback function are error, standard output and error output.

Exec integrates the first two parameters of execFile in passing parameters, that is, the command and command parameters are concatenated into a string as the first parameter, and the subsequent parameters are the same as execFile.

Cluster cluster

Starting processes requires memory consumption, so the number of started processes should be appropriate. Rational use of multi-processes can greatly improve efficiency. For example, when Webpack packages resources, it starts multiple processes at the same time, which greatly improves the packaging speed. Cluster is also one of the important applications of multi-processes. It is better to use multiple processes to monitor the same service at the same time. Generally, the number of opened processes is the same as the number of cores in CPU. At this point, services monitored by multiple processes will be diverted according to the request pressure, and "load balancing" can also be achieved by setting the number of requests processed by each child process.

1. Use ipc to implement cluster

When ipc standard process communication uses the send method to send messages, the second parameter supports passing in a service, which must be a http service or a tcp service. The child process receives it through the message event, and the callback parameters correspond to the sent parameters, that is, the first parameter is the message and the second parameter is the service. We can create a service in the child process and listen and operate on the service of the main process (listen can listen to the port number as well as the service), and then implement the cluster, as shown below.

/ File: server.jsconst os = require ("os"); / / os module is used to obtain system information const http = require ("http"); const path = require ("path"); const {fork} = rquire ("child_process"); / / create service const server = createServer (res, req) = > {res.end ("hello");}) .requests (3000) / / create child processes os.cpus (). ForEach () = > {fork ("child_server.js", {cwd: path.join (_ _ dirname);}) .send ("server", server);}); / / File: child_server.jsconst http = require ("http") / / receive the service process.on sent by the autonomous process ("message", (data, server) = > {http.createServer ((req, res) = > {res.end (`child$ {process.pid} `);}) .services (server); / / the child process shares the services of the main process})

The request processed by the main process in the above code returns hello, and the request processed by the child process returns a string consisting of child plus the process's pid.

2. Use cluster to realize cluster

The cluster module is provided by NodeJS to implement the cluster, and it integrates child_process 's method of creating child processes in a simpler way than using ipc.

/ / file: cluster.jsconst cluster = require ("cluster"); const http = require ("http"); const os = require ("os"); / / determine whether the currently executing process is the main process, and create a child process, otherwise use the child process listening service if (cluster.isMaster) {/ / create the child process os.cpus (). ForEach () = > cluster.fork ()) } else {/ / create and listen to the service http.createServer ((req, res) = > {res.end (`child$ {process.pid} `);}) .listen (3000);}

The above code executes both if and else, which seems strange, but it is not executed at the same time. When the main process executes, a child process is created through cluster.fork. When the child process is created, the file will be executed again, and then the service listening in else will be executed. There is another way to separate the code executed by the main process and the child process with clearer logic. The usage is as follows.

/ File: cluster.jsconst cluster = require ("cluster"); const path = require ("path"); const os = require ("os"); / / set the path for the child process to read the file cluster.setupMaster ({exec: path.join (_ _ dirname, "cluster-server.js")}); / / create the child process os.cpus (). ForEach (() = > cluster.fork ()); / / File: cluster-server.jsconst http = require ("http") / / create and listen to the service http.createServer ((req, res) = > {res.end (`child$ {process.pid} `);}) .listen (3000)

After setting the child process execution file through cluster.setupMaster, the logic of the main process and the child process can be separated. In actual development, this way is also the most commonly used, with low coupling, good readability, and more in line with the principles of development.

The above is all the content of the article "sample Analysis of Multi-processes and clusters in NodeJS". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.