Shulou(Shulou.com)06/02 Report--

This article mainly explains "analyzing the Stream module of Nodejs". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "analyze the Stream module of Nodejs".

First, the opening analysis

A stream is an abstract interface implemented by many objects in Node. For example, a request to a HTTP server is a stream, and stdout is also a stream. Streams are readable, writable, or both.

The first contact with Stream is from the early unix, decades of practice has proved that the idea of Stream can easily develop some huge systems.

In unix, Stream is implemented through "|". In node, as a built-in stream module, many core modules and tripartite modules are used.

Like unix, the main operation of node stream is .pipe (), and users can use the reverse pressure mechanism to control the balance between read and write.

Stream can provide developers with a unified interface that can be reused, and an abstract Stream interface is used to control the read-write balance between Stream.

A TCP connection is both a readable stream and a writable stream, whereas a Http connection is different. A http request object is a readable stream, while a http response object is a writable stream.

The transmission process of the stream is transmitted in the form of buffer by default, unless you set another encoding form for it. Here is an example:

1.

two。

3.

Var http = require ('http')

4.

Var server = http.createServer (function (req,res) {

5.

Res.writeHeader (200,{ 'Content-Type':' text/plain'})

6.

Res.end ("Hello, big bear!")

7.

})

8.

Server.listen (8888)

9.

Console.log ("http server running on port 8888...")

After running, there will be garbled code because the specified character set is not set, such as "utf-8".

Just modify it:

Var http = require ('http')

1.

two。

Var server = http.createServer (function (req,res) {

3.

Res.writeHeader (20000, {

4.

'Content-Type':' text/plain;charset=utf-8' / / add charset=utf-8

5.

})

6.

Res.end ("Hello, big bear!")

7.

})

8.

Server.listen (8888)

9.

Console.log ("http server running on port 8888...")

Running result:

Why use Stream

The iCandle O in node is asynchronous, so reading and writing to disk and network requires a callback function to read data. Here is an example of a file download.

The above code:

1.

two。

Var http = require ('http')

3.

Var fs = require ('fs')

4.

Var server = http.createServer (function (req, res) {

5.

Fs.readFile (_ _ dirname +'/ data.txt', function (err, data) {

6.

Res.end (data)

7.

})

8.

})

9.

Server.listen (8888)

The code can achieve the desired function, but the service needs to cache the entire file data to memory before sending the file data, if the "data.txt" file is very

If the concurrency is large, a lot of memory will be wasted. Because the user needs to wait until the entire file is cached in memory to accept the file data, which results in

The user experience is quite bad. Fortunately, (req,res) both arguments are Stream, so we can use fs.createReadStream () instead of fs.readFile (). As follows:

Var http = require ('http')

1.

two。

Var fs = require ('fs')

3.

Var server = http.createServer (function (req, res) {

4.

Var stream = fs.createReadStream (_ _ dirname +'/ data.txt')

5.

Stream.pipe (res)

6.

})

7.

Server.listen (8888)

The .pipe () method listens for the 'data' and' end' events of fs.createReadStream (), so that the "data.txt" file does not need to be cached

You can send a data block to the client as soon as the client connection is completed. Another benefit of using .pipe () is that it can solve when the customer

Read-write imbalance caused by very large end delay.

There are five basic Stream:readable,writable,transform,duplex,and "classic". (for specific use, please refer to api)

Second, the introduction of examples

We need to use data streams when we cannot hold data that needs to be processed at once in memory, or when it is more efficient to read and process at the same time. The operation of data stream is provided by various Stream in NodeJS.

Taking the large file copy program as an example, we can create a read-only data stream for the data source, as shown below:

Var rs = fs.createReadStream (pathname)

1.

two。

Rs.on ('data', function (chunk) {

3.

DoSomething (chunk); / / play with the details at will

4.

})

5.

Rs.on ('end', function () {

6.

CleanUp ()

7.

})

Data events continue to be triggered in the code, regardless of whether the doSomething function can handle it or not. The code can continue to make the following modifications to solve this problem.

1.

two。

3.

Var rs = fs.createReadStream (src)

4.

Rs.on ('data', function (chunk) {

5.

Rs.pause ()

6.

DoSomething (chunk, function () {

7.

Rs.resume ()

8.

})

9.

})

10.

Rs.on ('end', function () {

11.

CleanUp ()

twelve。

})

A callback is added to the doSomething function, so we can pause the data reading before processing the data and continue to read the data after processing the data.

In addition, we can also create a write-only data flow for the data target, as follows:

Var rs = fs.createReadStream (src)

1.

two。

Var ws = fs.createWriteStream (dst)

3.

Rs.on ('data', function (chunk) {

4.

Ws.write (chunk)

5.

})

6.

Rs.on ('end', function () {

7.

Ws.end ()

8.

})

When doSomething writes data to a write-only data stream instead, the above code looks like a file copy program. However, the above code has the problem mentioned above, if the write speed can not keep up with the read speed, the cache within the write-only data stream will burst. We can determine whether the incoming data is written to the target or temporarily placed in the cache according to the return value of the .write method, and determine when the write-only data stream has written the data in the cache to the target and can pass in the next data to be written according to the drain event. So the code is as follows:

Var rs = fs.createReadStream (src)

1.

two。

Var ws = fs.createWriteStream (dst)

3.

Rs.on ('data', function (chunk) {

4.

If (ws.write (chunk) = false) {

5.

Rs.pause ()

6.

}

7.

})

8.

Rs.on ('end', function () {

9.

Ws.end ()

10.

})

11.

Ws.on ('drain', function () {

twelve。

Rs.resume ()

13.

})

Finally, it realizes the transportation of data from read-only data flow to write-only data flow, and includes explosion-proof warehouse control. Because there are many such usage scenarios, such as the large file copy program above, NodeJS directly provides a .pipe method to do this, and its internal implementation is similar to the code above.

Here is a more complete process of copying files:

Var fs = require ('fs')

1.

two。

Path = require ('path')

3.

Out = process.stdout

4.

Var filePath ='/ bb/bigbear.mkv'

5.

Var readStream = fs.createReadStream (filePath)

6.

Var writeStream = fs.createWriteStream ('file.mkv')

7.

Var stat = fs.statSync (filePath)

8.

Var totalSize = stat.size

9.

Var passedLength = 0

10.

Var lastSize = 0

11.

Var startTime = Date.now ()

twelve。

ReadStream.on ('data', function (chunk) {

13.

PassedLength + = chunk.length

14.

If (writeStream.write (chunk) = false) {

15.

ReadStream.pause ()

16.

}

17.

})

18.

ReadStream.on ('end', function () {

19.

WriteStream.end ()

20.

})

21.

WriteStream.on ('drain', function () {

twenty-two。

ReadStream.resume ()

23.

})

24.

SetTimeout (function show () {

25.

Var percent = Math.ceil ((passedLength / totalSize) * 100)

twenty-six。

Var size = Math.ceil (passedLength / 1000000)

twenty-seven。

Var diff = size-lastSize

twenty-eight。

LastSize = size

twenty-nine。

Out.clearLine ()

thirty。

Out.cursorTo (0)

thirty-one。

Out.write ('completed' + size +'MB,'+ percent +'%, Speed:'+ diff * 2 +

thirty-two。 'MB/s')

thirty-three。

If (passedLength < totalSize) {

thirty-four。

SetTimeout (show, 500)

thirty-five。

} else {

thirty-six。

Var endTime = Date.now ()

thirty-seven。

Console.log ()

thirty-eight。

Console.log ('shared time:' + (endTime-startTime) / 1000 + 'seconds.')

thirty-nine。

}

forty。

}, 500)

You can save the above code as "copy.js" and try to add a recursive setTimeout (or just use setInterval) to be a bystander.

Each 500ms observes the progress of completion, writes the size, percentage, and replication speed of the completed to the console, and calculates the total elapsed time when the replication is complete.

Thank you for your reading, the above is the content of "analyzing the Stream module of Nodejs". After the study of this article, I believe you have a deeper understanding of the problem of analyzing the Stream module of Nodejs, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.