Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to build a short address service based on s3cmd". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to create a short address service based on s3cmd".

Requirement description

The wired project uses S3 to publish the mobile APP. Due to the need for permission authentication, the corresponding Object cannot open the "public-read" permission. The signature URL generated by Presign has the following problems.

The generated signed URL address contains sensitive information, which will expose the bucket name and accesskey, which brings security risks.

The generated signed URL address is too long, affecting the user experience.

The generated address can be accessed arbitrarily within the valid time, so it is impossible to achieve strict anti-hotlink.

Solution

Use openresty to encapsulate the S3 data access process. All data downloads on the client go to the service built on openresty, which provides an one-time short address generation service (the corresponding short address expires after several visits). The specific process is as follows:

1. Upload user data to S3, and set the corresponding Object permission to "public-read" to obtain the corresponding external access URL.

two。 Based on the previously generated external access URL, the corresponding short address service is generated.

3. The client uses a short address for access, and cannot be accessed if the number of visits is exceeded.

If you want to further improve security, you can interconnect RGW and Openresty through the internal network, upload all the data through the internal network, and external access to the external network.

Specific implementation 1. Openresty installation 2. Install dependent lua library files

The lib path of the default openresty is / usr/local/openresty/lualib

You need to rely on two lua files, of which url.lua is mainly used for url address resolution, and redis_iresty.lua is used for redis connection

Save the following source file as / usr/local/openresty/lualib/resty/url.lua https://github.com/golgote/neturl/blob/master/lib/net/url.lua and the following source file as / usr/local/openresty/lualib/resty/redis_iresty.lua https://gist.github.com/moonbingbing/9915c66346e8fddcefb53. RGW service configuration root@demohost:/etc/openresty# cat / etc/ceph/ceph.conf... [client.radosgw.demo] rgw dns name = s3.cephbook.com # S3 endpoint corresponding domain name rgw frontends = "civetweb port=7480" host = demohost keyring = / etc/ceph/ceph.client.radosgw.keyring log file = / home/ceph/log/radosgw.log4. Openresty configuration root@demohost:/etc/openresty# cat / etc/openresty/nginx.conf...server {listen 80; server_name ceph.vip; # short address host header setting location = / url {# generate short address entry content_by_lua_file / etc/openresty/url.lua } location / {# provides data access proxy_http_version 1.1 for short addresses; proxy_set_header Host $host; access_by_lua_file / etc/openresty/access.lua; proxy_pass http://127.0.0.1:7480; # corresponds to the radosgw service entry of the backend}}

The source code for generating the short address service is as follows

Root@demohost:/etc/openresty# cat url.lualocal request_method = ngx.var.request_methodlocal url_ = require "resty.url" # corresponds to the previous url library local redis = require "resty.redis_iresty" # corresponds to the previous redis_ irestyle library local s3_endpoint = "s3.cephbook.com" # here the setting allows only the generation of short addresses related to S3 endpoint local endpoint = s3_endpoint.. "$" local charset = {} for I = 48,57 do table.insert (charset, string.char (I)) endfor I = 65,90 do table.insert (charset, string.char (I)) endfor I = 97,122 do table.insert (charset, string.char (I)) endfunction string.random (length) local urandom = assert (io.open ('/ dev/urandom','rb')) local a, b, c D = urandom:read (4): byte (1Magne 4) urandom:close () local seed = a*0x1000000 + b*0x10000 + c * 0x100 + d math.randomseed (seed) if length > 0 then return string.random (length-1). Charset [math.random (1, # charset)] else return "" endendfunction genera_url (red,url_id,host,path)-counts indicates the maximum number of visits to a short address Current represents the current number of times ok, err = red:hmset (url_id,'host',host,'uri',path,'counts',3,'current',1) if not ok then ngx.status = ngx.HTTP_METHOD_NOT_IMPLEMENTED ngx.say ("failed to hmset:", err) return ngx.exit (ngx.HTTP_METHOD_NOT_IMPLEMENTED) end-sets the maximum length of time that a short address record can be stored in redis Avoid resource exhaustion ok, err = red:expire (url_id,3600) if not ok then ngx.status = ngx.HTTP_METHOD_NOT_IMPLEMENTED ngx.say ("failed to set expire:", err) return ngx.exit (ngx.HTTP_METHOD_NOT_IMPLEMENTED) endendfunction get_info_by_id (red,url_id) ok Err = red:hgetall (url_id) if not ok then ngx.status = ngx.HTTP_SERVICE_UNAVAILABLE ngx.say ("failed to getall:" Err) return ngx.exit (ngx.HTTP_SERVICE_UNAVAILABLE) end local h = red:array_to_hash (ok) return hendif request_method = = "POST" then ngx.req.read_body () local data = ngx.req.get_body_data () local u = url_.parse (data) local hostname_check, hostname_err = ngx.re.match (u.host Endpoint) if not hostname_check then ngx.status = ngx.HTTP_BAD_REQUEST ngx.say ("not an available s3_endpoint") ngx.exit (ngx.HTTP_BAD_REQUEST) end if u.host then if string.len (u.path) > 1 then local url_id = string.random (7) local red = redis:new () genera_url (red,url_id,u.host) U.path) local h = get_info_by_id (red,url_id) ngx.status = ngx.HTTP_CREATED ngx.say ("ShortURL:", ngx.var.scheme, "/ /", ngx.var.host, "/", url_id) ngx.say ("host:", h.host) ngx.say ("uri:" H.uri) ngx.say ("counts:", h.counts) ngx.say ("current:" H.current) ngx.exit (ngx.HTTP_CREATED) else ngx.status = ngx.HTTP_BAD_REQUEST ngx.say ("path err") ngx.exit (ngx.HTTP_BAD_REQUEST) end else ngx.status = ngx.HTTP_BAD_REQUEST ngx.say ("not an available url") ngx.exit (ngx.HTTP_BAD_REQUEST) endend

The source code for accessing the short address data service is as follows

Root@demohost:/etc/openresty# cat / etc/openresty/access.lualocal redis = require "resty.redis_iresty" local red = redis:new () local uri = ngx.var.urilocal url_id = string.sub (uri,2,string.len (uri)) res, err = red:hgetall (url_id) if not res then ngx.exit (ngx.HTTP_NOT_FOUND) else local h = red:array_to_hash (res)-access is denied if the number of visits exceeds If you consider the resource consumption, you can add the operation if h.counts to delete the corresponding redis record.

< h.current then ngx.status = ngx.HTTP_GONE ngx.say("current=",h.current," >

Counts=, h.counts) ngx.exit (ngx.HTTP_GONE) end ngx.req.set_header ("host", h.host) ngx.req.set_uri (h.uri, false) ok, err = red:hincrby (url_id,'current',1) if not ok then ngx.status = ngx.HTTP_METHOD_NOT_IMPLEMENTED ngx.say ("incrby key failed:" Err) ngx.exit (ngx.HTTP_METHOD_NOT_IMPLEMENTED) return endend

Https://github.com/openresty/lua-nginx-module#http-status-constants

5. Use the process to upload data

Use s3cmd or other means to upload files, and set the corresponding Object access to "public-read".

Root@demohost:/tmp# s3cmd put myfile s3://demo-- acl-public'myfile'->'s 3 of kB/s donePublic URL of the object is of'[1 of 1] 21577 of 21577 100% in 0s 227.71 kB/s done'myfile'->'s 3 Velcro kB/s donePublic URL of the object is'[1 kB/s donePublic URL of the object is 1] 21577 of 21577 100% in 0s 203.11 kB/s donePublic URL of the object is: generate a short address Root@demohost:/tmp# curl ceph.vip/url-d "http://demo.s3.cephbook.com/myfile"-v * Hostname was NOT found in DNS cache* Connected to ceph.vip port 80 (# 0) > POST / url HTTP/1.1 > User-Agent: curl/7.38.0 > Host: ceph.vip > Accept: * / * > Content-Length: 29 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 29 out of 29 bytes

< HTTP/1.1 201 Created* Server openresty/1.11.2.5 is not blacklisted< Server: openresty/1.11.2.5< Date: Wed, 15 Nov 2017 09:56:14 GMT< Content-Type: application/octet-stream< Transfer-Encoding: chunked< Connection: keep-alive GET /Vo3t5Ex HTTP/1.1>

User-Agent: curl/7.38.0 > Host: ceph.vip > Accept: * / * > < HTTP/1.1 410 Gone* Server openresty/1.11.2.5 is not blacklisted < Server: openresty/1.11.2.5 < Date: Wed, 15 Nov 2017 10:00:46 GMT < Content-Type: application/octet-stream < Transfer-Encoding: chunked < Connection: keep-alive counts=3* Connection # 0 to host ceph.vip left intact Thank you for reading. This is the content of "how to build a s3cmd-based short address Service". After the study of this article, I believe you have a deeper understanding of how to create a short address service based on s3cmd, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.