Shulou(Shulou.com)06/02 Report--

Introduction

The project needs to do a file upload and download service, use nginx+lua as a proxy service, upload entry is unified, distribute to different machine storage, download links and physical storage isolation, support adding agent to expand capacity, here mainly talk about ideas and build configuration process, do not spray.

Main logic

Upload

The frontend requests the nginx service, and the nginx calls the upload script. The script finds the corresponding logical storage path and the ip and port of the agent of the physical storage machine by looking up the configuration, and sends the packet to the corresponding agent through tcp. The agent deployed on the corresponding machine receives the data and writes it to the local file.

download

Http download request nginx, nginx calls the download script, the script parses the link parameters, finds the corresponding agent address according to the parameters, and requests to return the binary content of the file. The script accepts the data returned by agent and returns it to the client.

Configure Nginx+lua

Next, let's talk about the installation and configuration of nginx (including lua binary stream processing lpack, md5 computing, mysql operation, json operation)

1. Install nginx

Download http://nginx.org/en/download.html

Decompress tar-xvf nginx-1.10.3.tar.gz

2. Install luajit (lightweight lua)

Http://luajit.org/download.html

Modify the installation path export PREFIX= / usr/local/luajit in makefile

Then install make & make install

3. Install nginx_lua_module

Download https://github.com/openresty/lua-nginx-module

Decompression

4. Install ngx_devel_kit (NDK provides functions and macros to handle some basic tasks to reduce the amount of code developed by third-party modules)

Download https://github.com/simpl/ngx_devel_kit/

5. Install and compile, import

Export LUAJIT_LIB=/usr/local/luajit/lib export LUAJIT_INC=/usr/local/luajit/include/luajit-2.0. / configure-- prefix=/usr/local/nginx-- with-http_stub_status_module-- with-http_ssl_module-- add-module=/home/oicq/jeffzhuang/ngx_devel_kit-0.3.0-- add-module=/home/oicq/jeffzhuang/lua-nginx-module-0.10.make-j2 make install

Start / usr/local/nginx/sbin/nginx restart command `usr/local/nginx/sbin/nginx-s reload v

If you report an error and cannot find the luajit library ln-s / usr/local/lib/libluajit-5.1.so.2 / lib64/libluajit-5.1.so.2

Test nginx just open the browser and http:10.x.x.x:8080 will see the welcome interface.

6. Configure conf/nginx.conf to run lua script

Add the search path lua_package_path,lua_package_cpath of Lua library

7. Just add mysql.lua download https://github.com/openresty/lua-resty-mysql and copy it to the lua_package_path directory.

8. Add csjon http://www.kyne.com.au/~mark/software/download/lua-cjson-2.1.0.tar.gz

Modify the PREFIX=/usr/local/luajit in Makefile is the installation path of luajit. After make, copy the generated cjson.so to

Under the lua_package_cpath directory

9. To install lpack, you can use ready-made lpack.lua to copy to lua_package_path or use https://github.com/LuaDist/lpack to compile and generate lpack.so copy to lua_package_cpath 64-bit requires additional compilation command-fPIC

10. Upload.lua download https://github.com/openresty/lua-resty-upload

11. Md5 download https://github.com/openresty/lua-resty-string

Main code

1. Upload the page code at the front end

File upload example

2. Upload upload code, this module mainly adopts a simple algorithm similar to finite state machine in the process of parsing file upload request, and is processed by the corresponding handler in different states.

The file download service writes to saveRootPath.. "/". Under filename function DownLoad () local chunk_size = 4096 local form,err=upload:new (chunk_size) if not form then ngx.log (ngx.ERR, "failed to new upload:", err) ngx.exit (ngx.HTTP_INTERNAL_SERVER_ERROR) end form:set_timeout (100000) while true do local typ,res Err=form:read () if not typ then ErrorMsg= "failed to read:".. err return 1 end if typ = = "header" then local key=res [1] local value=res [2] if key= = "Content-Disposition" then local kvlist=string.split (value,' ') for _, kv in ipairs (kvlist) do local seg = string.trim (kv) if seg:find ("filename") then local kvfile = string.split (seg, "=") filename = string.sub (kvfile [2], 2,-2) if filename then-- get the file suffix fileExtension=getExtension (filename) local linuxTime=tostring (os.time () filePath=saveRootPath. "/".. linuxTime..filename fileToSave,errmsg = io.open (filePath) "w +")-the path to the stored file-- ngx.say ("failed to open file", filePath) if not fileToSave then-- ngx.say ("failed to open file", filePath. Errmsg) ErrorMsg= "failed to open file".. filePath.. Errmsg return 1 end else ErrorMsg= "request parameter cannot find file name" return 1 end-- jump out of loop break end elseif typ = = "body" then if fileToSave then fileToSave:write (res) fileMd5:update (res) end elseif typ = = "part_end" then if fileToSave then local md5_sum=fileMd5:final ()-- ngx.say ("md5:" Str.to_hex (md5_sum)) fileMD532=str.to_hex (md5_sum) fileToSave:close () fileToSave = nil end elseif typ = = "eof" then break else ngx.log (ngx.INFO, "do other things") end end return 0end

3. Tcp receives binary data

-read bytefunction readInt8 (tcp) local next, val = string.unpack (tcp:receive (1), "b") return tonumber (val); end-- read int16function readInt16 (tcp) local next, val = string.unpack (tcp:receive (2), "h"); return tonumber (val); end-- read int32function readInt32 (tcp) local next, val = string.unpack (tcp:receive (4), "> I"); return tonumber (val) End-- reads the string function readString (tcp,len) return tostring (tcp:receive (len)); end

4. Tcp writes binary data. Here, the communication protocol with agent is: start flag bit + packet length + json string + end flag bit, so the parameter used for the corresponding pack is bIAb. > is converted to the big end.

JsonData ["filename"] = fileMD532.. ".". FileExtensionjsonData ["cmd"] = "write" jsonData ["fileSize"] = tostring (filelen) jsonData ["path"] = System.. "/".. StorageDatelocal Jsonstr=cjson.encode (jsonData) local uiLen=string.len (Jsonstr) senddata=bpack ("> b1IAb", startIndex,uiLen,Jsonstr,endIndex) socket:send (senddata)

5. When downloading an error, use redirect to jump directly to the error page to facilitate the output of error messages. In fact, you can also do user token verification here.

There is a problem with local ErrorUrl= "/ downloadError.html" ErrorMsg= "url parameter parsing".. indexreturn ngx.redirect (ErrorUrl.. "? msg=".. ErrorMsg, ```return)

Summary

The above is introduced by the editor to solve the problem of uploading and downloading nginx+lua building files. I hope it will be helpful to you. If you have any questions, please leave me a message and the editor will reply to you in time. Thank you very much for your support to the website!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.