Shulou(Shulou.com)06/03 Report--

Node.js version of how to use HTTP to upload G-level files, I believe that many inexperienced people do not know what to do, so this article summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

Like it or not, javascript is everywhere. We can find it in the foreground application of the client, in a large number of frameworks and class libraries, and in the background application on the server side.

In recent years, Javascript has become more and more popular, which seems to be because the Javascript ecosystem is helping to increase productivity and reduce the time required to get started. In my article, I introduced the use of ASP.NET Web back-end API for G-level file uploading. after publishing this article, I decided to try to use Node.js to achieve the same effect. This means that I need to implement the UploadChunk and MergeAll methods.

Development environment

We will use Visual Studio Express 2013 for Web as the development environment, but it cannot be used for Node.js development yet. To do this, we need to install Node.js Tools for Visual Studio. Once installed, the Visual Studio Express 2013 for Web will be transformed into a Node.js IDE environment that provides everything you need to create the application. And based on the guidance provided here, we need:

Download and install the Node.js Windows version and choose the version suitable for your system platform, Node.js (x86) or Node.js (x64).

Download and install Node.js 's Visual Studio tool.

After the installation is complete, we will run Visual Studio Express 2013 for Web and use Node.js 's interactive window to verify the installation. The interaction window for Node.js can be found under View- > Other Windows- > Node.js Interactive Window. After the Node.js interaction window runs, we need to enter some commands to check whether everything is OK.

Figure 1 Node.js Interactive Window

Now that we have verified the installation, we are ready to create a Node.js daemon that supports GB-level file uploads. Let's start by creating a new project and selecting an empty Node.js Web application template.

Figure 2 New project using the Blank Node.js Web Application template

After the project is created, we should see a file called server.js and the Node package manager (npm) in the solution browser.

Figure 3 Node.js application in solution Manager

The server.js file contains code that needs to use Node.js to create a basic hello world application.

Figure 4 The Hello World application

I will now continue to remove this code from server.js and then wear the back-end code for uploading a G-level file in Node.js. Below I need to install some of the dependencies needed for this project with npm:

Express-Node.js web application framework for building single-page, multi-page and hybrid web applications

Formidable-Node.js module for parsing form data, especially file upload

Fs-extra-File system interaction Module

Figure 5 using npm to install the required modules

After the modules are installed, we can see them in solution Explorer.

Figure 6 solution Explorer shows installed modules

Next we need to create a new "Scripts" folder in solution Explorer and add "workeruploadchunk.js" and "workerprocessfile.js" to this folder. We also need to download the jQuery 2.x and SparkMD5 libraries and add them to the "Scripts" folder. * also need to add a "Default.html" page. All of this was introduced in my previous post.

Create Node.js backend

First of all, we need to use Node.js 's "require ()" function to import the module that uploads G-level files in the background. Note that I have also imported the "path" and "crypto" modules. The "path" module provides a method to generate the file name of the uploaded file block. The "crypto" module provides a method to generate the MD5 checksum of the uploaded file.

/ / The required modules var express = require ('express'); var formidable = require (' formidable'); var fs = require ('fs-extra'); var path = require (' path'); var crypto = require ('crypto')

The next line of code is the moment to witness miracles.

Var app = express ()

This line of code is used to create an express application. Express application is a middleware that encapsulates the underlying functions of Node.js. If you remember the "Hello World" program created by the Blank Node.js Web application template, you will find that I imported the "http" module and then called the "http.CreateServer ()" method to create the "Hello World" web application. The express application we just created has all the features built into it.

Now that we have created an express application, we let it render the "Default.html" we created earlier, and then let the application wait for a connection.

/ / Serve up the Default.html page app.use (express.static (_ _ dirname, {index: 'Default.html'})); / / Startup the express.js application app.listen (process.env.PORT | | 1337); / / Path to save the files var uploadpath =' CelerFTCue'

Express applications have the app.VERB () method, which provides routing capabilities. We will use the app.post () method to process the "UploadChunk" request. The * thing we do in the app.post () method is to check whether we are processing the POST request. Next check whether Content-Type is mutipart/form-data, and then check that the size of the uploaded file block cannot be greater than 51MB.

/ / Use the post method for express.js to respond to posts to the uploadchunk urls and / / save each file chunk as a separate file app.post ('* / api/CelerFTFileUpload/UploadChunk*', function (request) Response) {if (request.method = = 'POST') {/ / Check Content-Type if (! (request.is (' multipart/form-data') {response.status (415) .send ('Unsupported media type') Return;} / / Check that we have not exceeded the maximum chunk uploadsize var maxuploadsize = 51 * 1024 * 1024; if (request.headers ['content-length'] > maxuploadsize) {response.status (413). Send (' Maximum upload chunk size exceeded'); return;}

Once we have successfully passed all the checks, we will take the uploaded file block as a separate file and name it sequentially. The most important code below is to call the fs.ensureDirSync () method, which is used to check for the existence of a temporary directory. Create one if the directory does not exist. Note that we are using a synchronized version of this method.

/ / Get the extension from the filename var extension = path.extname (request.param ('filename')); / / Get the base filename var baseFilename = path.basename (request.param (' filename'), extension); / / Create the temporary filename for the chunk var tempfilename = baseFilename +'. + request.param ('chunkNumber'). ToString (). PadLeft (' 0,16) + extension + ".tmp" / / Create the temporary directory to store the file chunk / / The temporary directory will be based on the filename var tempdir = uploadpath + request.param ('directoryname') +' /'+ baseFilename; / / The path to save the file chunk var localfilepath = tempdir +'/'+ tempfilename; if (fs.ensureDirSync (tempdir)) {console.log ('Created directory' + tempdir);}

As I mentioned earlier, we can upload files to the back-end server in two ways. * one way is to use FormData in the web browser, and then send the file block as binary data. Another way is to convert the file block into a base64-encoded string, then create a manual multipart/form-data encoded request, and then send it to the back-end server.

So we need to check to see if we are uploading a manual multipart/form-data encoded request, and by checking the "CelerFT-Encoded" header information, if this header exists, we create a buffer and use request's ondata time to copy the data into the buffer.

In the onend event of request, the base64 string is extracted from the multipart/form-data encoded request by rendering the buffer as a string and separating it by CRLF. Base64-encoded file blocks can be found in the fourth index of the array.

Convert base64-encoded data reproduction to binary by creating a new buffer. The fs.outputFileSync () method is then called to write the buffer to the file.

/ / Check if we have uploaded a hand crafted multipart/form-data request / / If we have done so then the data is sent as a base64 string / / and we need to extract the base64 string and save it if (request.headers ['celerft-encoded'] = =' base64') {var fileSlice = newBuffer (+ request.headers ['content-length']); var bufferOffset = 0 / / Get the data from the request request.on ('data', function (chunk) {chunk.copy (fileSlice, bufferOffset); bufferOffset + = chunk.length }). On ('end', function () {/ / Convert the data from base64 string to binary / / base64data in 4th index of the array var base64data = fileSlice.toString (). Split ('\ r\ n'); var fileData = newBuffer (base64data [4] .toString (), 'base64'); fs.outputFileSync (localfilepath,fileData) Console.log ('Saved file to' + localfilepath); / / Send back a sucessful response with the file name response.status (200.send (localfilepath); response.end ();});}

The upload of binary file blocks is handled through the formidable module. We use the formidable.IncomingForm () method to get the multipart/form-data encoded request. The formidable module will save the uploaded file block as a separate file and save it to a temporary directory. What we need to do is to save the uploaded file block to the name in the onend event of formidable.

Else {/ / The data is uploaded as binary data. / / We will use formidable to extract the data and save it var form = new formidable.IncomingForm (); form.keepExtensions = true; form.uploadDir = tempdir; / / Parse the form and save the file chunks to the / / default location form.parse (request, function (err, fields, files) {if (err) {response.status (500) .send (err) Return;} / / console.log ({fields: fields, files: files});}); / / Use the filebegin event to save the file with the naming convention / * form.on ('fileBegin', function (name, file) {file.path = localfilepath;}) * / form.on ('error', function (err) {if (err) {response.status (500) .send (err); return;}}) / / After the files have been saved to the temporary name / / move them to the to teh correct file name form.on ('end', function (fields,files) {/ / Temporary location of our uploaded file var temp_path = this.openedFiles [0] .path Fs.move (temp_path, localfilepath,function (err) {if (err) {response.status (500) .send (err); return } else {/ / Send back a sucessful response with the file name response.status (200.send (localfilepath); response.end ();}});}) / / Send back a sucessful response with the file name / / response.status (2000.send (localfilepath); / / response.end ();}}

Used by the app.get () method to process "MergeAll" requests. This method implements the functionality described earlier.

/ / Request to merge all of the file chunks into one file app.get ('* / api/CelerFTFileUpload/MergeAll*', function (request,response) {if (request.method = = 'GET') {/ / Get the extension from the filename var extension = path.extname (request.param (' filename')) / / Get the base filename var baseFilename = path.basename (request.param ('filename'), extension); var localFilePath = uploadpath + request.param (' directoryname') +'/'+ baseFilename / / Check if all of the file chunks have be uploaded / / Note we only wnat the files with a * .tmp extension var files = getfilesWithExtensionName (localFilePath, 'tmp') / * if (err) {response.status (500) .send (err); return } * / if (files.length! = request.param ('numberOfChunks')) {response.status (400). Send (' Number of file chunks less than total count'); return;} var filename = localFilePath +'/'+ baseFilename + extension; var outputFile = fs.createWriteStream (filename) / / Done writing the file / / Move it to top level directory / / and create MD5 hash outputFile.on ('finish', function () {console.log (' file has been written'); / / New name for the file var newfilename = uploadpath + request.param ('directoryname') +' /'+ baseFilename + extension / / Check if file exists at top level if it does delete it / / if (fs.ensureFileSync (newfilename)) {fs.removeSync (newfilename) / / Move the file fs.move (filename, newfilename, function (err) {if (err) {response.status (500) .send (err); return } else {/ / Delete the temporary directory fs.removeSync (localFilePath); varhash = crypto.createHash ('md5'), hashstream = fs.createReadStream (newfilename) Hashstream.on ('data', function (data) {hash.update (data)}); hashstream.on (' end', function () {var md5results = hash.digest ('hex')) / / Send back a sucessful response with the filename response.status 200.send ('Sucessfully merged file' + filename + "," + md5results.toUpperCase (); response.end ();}) }});}); / / Loop through the file chunks and write them to the file / / files [index] retunrs the name of the file. / / we need to add put in the full path to the file for (var index infiles) {console.log; var data = fs.readFileSync (localFilePath +'/'+ files [index]); outputFile.write (data); fs.removeSync (localFilePath +'/'+ files [index]) } outputFile.end ()

Note that Node.js does not provide a String.padLeft () method, which is achieved by extending String.

/ / String padding left code taken from / / http://www.lm-tech.it/Blog/post/2012/12/01/String-Padding-in-Javascript.aspx String.prototype.padLeft = function (paddingChar, length) {var s = new String (this); if ((this.length

< length)&& (paddingChar.toString().length >

0)) {for (var I = 0; I

< (length - this.length) ; i++) { s = paddingChar.toString().charAt(0).concat(s); } } return s; } ; 一些其它事情 其中一件事是，发表上篇文章后我继续研究是为了通过域名碎片实现并行上传到CeleFT功能。域名碎片的原理是访问一个web站点时，让web浏览器建立更多的超过正常允许范围的并发连接。 域名碎片可以通过使用不同的域名（如web1.example.com，web2.example.com）或者不同的端口号（如8000, 8001）托管web站点的方式实现。 示例中，我们使用不同端口号托管web站点的方式。 我们使用 iisnode 把 Node.js集成到 IIS（ Microsoft Internet Information Services）实现这一点。 下载兼容你操作系统的版本 iisnode (x86) 或者 iisnode (x64)。 下载 IIS URL重写包。 一旦安装完成（假定windows版Node.js已安装），到IIS管理器中创建6个新网站。将***个网站命名为CelerFTJS并且将侦听端口配置为8000。 图片7在IIS管理器中创建一个新网站 然后创建其他的网站。我为每一个网站都创建了一个应用池，并且给应用池"LocalSystem"级别的权限。所有网站的本地路径是C:\inetpub\wwwroot\CelerFTNodeJS。 图片8 文件夹层级 我在Release模式下编译了Node.js应用，然后我拷贝了server.js文件、Script文件夹以及node_modules文件夹到那个目录下。 要让包含 iisnode 的Node.js的应用工作，我们需要创建一个web.config文件，并在其中添加如下得内容。 web.config中各项的意思是让iisnode处理所有得*.js文件，由server.js 处理任何匹配"/*"的URL。 图片9 URL重写规则 如果你正确的做完了所有的工作，你就可以通过http://localhost:8000浏览网站，并进入CelerFT "Default.html"页面。 web.config文件被修改以支持如前面post中所解释的大文件的上传，这里我不会解释所有的项。不过下面的web.config项可以改善 iisnode中Node.js的性能。 并行上传 为了使用域名碎片来实现并行上传，我不得不给Node.js应用做些修改。我***个要修改的是让Node.js应用支持跨域资源共享。我不得不这样做是因为使用域碎片实际上是让一个请求分到不同的域并且同源策略会限制我的这个请求。 好消息是XMLttPRequest 标准2规范允许我这么做，如果网站已经把跨域资源共享打开，更好的是我不用为了实现这个而变更在"workeruploadchunk.js"里的上传方法。 // 使用跨域资源共享 // Taken from http://bannockburn.io/2013/09/cross-origin-resource-sharing-cors-with-a-node-js-express-js-and-sencha-touch-app/ var enableCORS = function(request,response, next){ response.header('Access-Control-Allow-Origin', '*'); response.header('Access-Control-Allow-Methods', 'GET,POST,OPTIONS'); response.header('Access-Control-Allow-Headers', 'Content-Type, Authorization, Content- Length, X-Requested-With' ) ; // 拦截OPTIONS方法 if ('OPTIONS' ==request.method){ response.send(204); } else { next(); } } ; // 在表达式中使用跨域资源共享 app. use ( enableCORS ) ; 为了使server.js文件中得CORS可用，我创建了一个函数，该函数会创建必要的头以表明Node.js应用支持CORS。另一件事是我还需要表明CORS支持两种请求，他们是： 简单请求： 1、只用GET，HEAD或POST。如果使用POST向服务器发送数据，那么发送给服务器的HTTP POST请求的Content-Type应是application/x-www-form-urlencoded, multipart/form-data, 或 text/plain其中的一个。 2、HTTP请求中不要设置自定义的头（例如X-Modified等） 预检请求： 1、使用GET，HEAD或POST以外的方法。假设使用POST发送请求，那么Content-Type不能是application/x-www-form-urlencoded, multipart/form-data, or text/plain，例如假设POST请求向服务器发送了XML有效载荷使用了application/xml or text/xml，那么这个请求就是预检的。 2、在请求中设置自定义头（比如请求使用X-PINGOTHER头）。 在我们的例子中，我们用的是简单请求，所以我们不需要做其他得工作以使例子能够工作。 // We are going to upload to a backend that supports parallel uploads. // Parallel uploads is supported by publishng the web site on different ports // The backen must implement CORS for this to work else if(workerdata.chunk!= null&& workerdata.paralleluploads ==true){ if (urlnumber >

= 6) {urlnumber = 0;} if (urlcount > = 6) {urlcount = 0;} if (urlcount = = 0) {uploadurl = workerdata.currentlocation + webapiUrl + urlnumber } else {/ / Increment the port numbers, e.g 8000, 8001, 8002, 8003, 8004, 8005 uploadurl = workerdata.currentlocation.slice (0,-1) + urlcount+ webapiUrl + urlnumber;} upload (workerdata.chunk,workerdata.filename,workerdata.chunkCount, uploadurl, workerdata.asyncstate); urlcount++; urlnumber++;}

I saved the current URL on the Default.html page because I am going to send this information to the working program for file upload. This is only done because:

I want to use this information to increase the number of ports.

After making a CORS request, I need to send the complete URL to the XMLHttpRequest object.

/ / Save current protocol and host for parallel uploads

"font-family: & apos;Lucida Console'; font-size: 8pt;" > var currentProtocol = window.location.protocol

"font-family: & apos;Lucida Console'; font-size: 8pt;" > var currentHostandPort = window.location.host

"font-family: & apos;Lucida Console'; font-size: 8pt;" > var currentLocation = currentProtocol + "/ /" + currentHostandPort

The code below shows the modification made to the upload message.

/ / Send and upload message to the webworker

"background-color: # ffff99; font-family: & apos;Lucida Console'; font-size: 8pt;" > case & apos;upload':

/ / Check to see if backend supports parallel uploads

Var paralleluploads = false; if ($('# select_parallelupload'). Prop ('checked')) {paralleluploads = true;} administrators [data.id] .postMessage ({' chunk': data.blob, 'filename':data.filename,' directory': $("# select_directory"). Val (), 'chunkCount':data.chunkCount,' asyncstate':data.asyncstate,'paralleluploads':paralleluploads, 'currentlocation': currentLocation,' id': data.id}) Break

* API CelerFT has been modified to support parallel upload.

CelerFT with parallel upload

The code for this project can be found on my github repository.

After reading the above, have you mastered how to use HTTP to upload G-level files in the Node.js version? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.