Shulou(Shulou.com)06/02 Report--

Background

Most systems involve the processing of thumbnails, such as news systems and e-commerce systems, especially e-commerce systems, where large images of each product correspond to a series of thumbnails for different business scenarios. Some systems also generate thumbnails of different sizes for use by PC, mobile phone and iPad.

Solution exploration:

Load the original image directly and use the css style sheet to control the width and height of the picture. It is obviously not appropriate, and we should try not to do so. After the web program is uploaded successfully, the corresponding thumbnail is generated at the same time. This practice is inefficient and has a serious impact on performance if you encounter bulk imported business. Thumbnails of some pictures are rarely used, wouldn't it be better if they could be generated on demand? Use the cloud storage and data processing services provided by third parties to solve the problems of image processing, storage and multi-node access speed. The advantage of this method is that the solution is mature, and there are certain costs and development work accordingly. In addition, there are some low-probability risks, such as the failure of cloud services to affect the access of this site.

In this paper, Nginx+Lua+GraphicsMagick is used to realize the thumbnail function, the upload and deletion of pictures are handled by web service, and the thumbnails are completed by a separate module. The final effect is similar to Taobao images. You can customize the output image size according to the suffix 100x100.jpg (fixed height and width),-100.jpg (fixed height) and _ 100-.jpg (fixed width).

Github source code address: https://github.com/botaozhao/nginx-lua-GraphicsMagick

Update description

2018-2-9: to add the thumbnail size limit, you need to configure the switch and the allowed size in demo.conf. The code snippet is as follows:

Init_by_lua'--switch needs to limit thumbnail size: true, no need to limit thumbnail size: false image_sizes_check = true-allowable size image_sizes = {"800x800", "400x400", "100x100", "- 800,400", "- 100,800 -", "400 -", "100 -"}'; description

Folder planning

Img.xxx.com (e.g. / usr/local/filebase) ├─ upload │ └─ img │ ├─ 001.jpg │ └─ 002.jpg

Path after Custom Siz

Thumb (/ tmp/thumb, which can be changed in the conf file) ├─ upload │ └─ img │ ├─ 001.jpg_100x100.jpg # fixed height and width │ ├─ 001.jpg_-100.jpg # fixed height │ ├─ 001.jpg_200-.jpg # fixed width │ └─ 002.jpg_300x300.jpg # where img.xxx.com is the root directory of the picture site, and img directory is the original image directory thumbnail directory according to the original structure And set up the directory separately, which can be cleaned regularly.

Link address correspondence

Original image access address: http://img.xxx.com/upload/img/001.jpg

Thumbnail access address: http://img.xxx.com/upload/img/001.jpg_100x100.jpg is 100 wide and 100 high

Automatic wide address: http://img.xxx.com/upload/img/001.jpg_-100.jpg indicates automatic with "-", that is, high 100th and wide automatic.

Automatic high address: http://img.xxx.com/upload/img/001.jpg_200-.jpg indicates automatic with "-", that is, 200 wide and high automatic.

Access proc

First of all, determine whether the thumbnail exists, if it does, directly display the thumbnail; if the thumbnail does not exist, determine whether the original image exists, then splice the graphicsmagick (gm) command if the original image exists, generate and display the thumbnail, otherwise return to 404 installation

System environment

Minimal installation within the centOS7 X64 virtual machine

The following operations are operated in this system, for reference only

1. Environmental preparation

Yum install-y wget git yum install-y gcc gcc-c++ zlib zlib-devel openssl openssl-devel pcre pcre-develyum install-y libpng libjpeg libpng-devel libjpeg-devel ghostscript libtiff libtiff-devel freetype freetype-develyum install-y GraphicsMagick GraphicsMagick-devel

If you are prompted that there is no installation package available for GraphicsMagick, please install GraphicsMagick yourself. For more information, please refer to another article I wrote: installing GraphicsMagick1.3.21 under CentOS7.

2. Download related applications

Cd / usr/local/srcwget http://nginx.org/download/nginx-1.8.0.tar.gzwget http://luajit.org/download/LuaJIT-2.0.4.tar.gzwget http://zlib.net/fossils/zlib-1.2.8.tar.gz

3. Download nginx components

Git clone https://github.com/alibaba/nginx-http-concat.gitgit clone https://github.com/simpl/ngx_devel_kit.gitgit clone https://github.com/openresty/echo-nginx-module.gitgit clone https://github.com/openresty/lua-nginx-module.git

Decompression and installation

Tar-zxf nginx-1.8.0.tar.gztar-zxf LuaJIT-2.0.4.tar.gztar-zxf zlib-1.2.8.tar.gz

1. Install LuaJIT

Cd LuaJIT-2.0.4make-j8make install export LUAJIT_LIB=/usr/local/libexport LUAJIT_INC=/usr/local/include/luajit-2.0ln-s / usr/local/lib/libluajit-5.1.so.2 / lib64/libluajit-5.1.so.2cd..

2. Install nginx

Cd nginx-1.8.0./configure-- prefix=/usr/local/nginx\-- sbin-path=/usr/local/nginx/sbin/nginx\-- conf-path=/usr/local/nginx/conf/nginx.conf\-pid-path=/usr/local/nginx/pid/nginx.pid\-- lock-path=/usr/local/nginx/pid/nginx.lock\-- error-log-path=/usr/local/nginx/logs/error.log\-- http-log-path=/ Usr/local/nginx/logs/access.log\-- with-http_ssl_module\-- with-http_realip_module\-- with-http_sub_module\-- with-http_flv_module\-- with-http_dav_module\-- with-http_gzip_static_module\-- with-http_stub_status_module\-- with-http_addition_module\-with-http_spdy_module\-- with-pcre\-- with-zlib=../ Zlib-1.2.8\-add-module=../nginx-http-concat\-add-module=../lua-nginx-module\-add-module=../ngx_devel_kit make-j8make install

Frequently asked questions about compiling nginx

. / configure: error: invalid option "- with-http_spdy_module"

Nginx 1.9.5 is no longer available-- with-http_spdy_module, replaced by-- with-http_v2_module. If you use this version or later, please replace it yourself.

Official documentation: http://nginx.org/en/docs/http/ngx_http_v2_module.html configuration

Related profile structure location

/ usr/local/nginx │ ├─ conf │ ├─. │ └─ nginx.conf │ ├─ html │ ├─ logs │ ├─ lua │ ├─ autoSize.lua │ └─ cropSize.lua │ ├─ pid │ sbin vhost demo.conf

The relevant configuration files can be downloaded from my github at nginx-lua-GraphicsMagick.

The details of the configuration file are posted below, which are consistent with github, and can be skipped directly.

Modify nginx configuration file

Cd / usr/local/nginx/vi conf/nginx.confuser root;worker_processes 4 binary_remote_addr zone=one:10m; limit_conn_zone workerboards 1000 0100 0010 0001 binary_remote_addr zone=one:10m; limit_conn_zone server_name zone=perserver:10m; include mime.types; include fastcgi.conf log / usr/local/nginx/logs/error.log error;pid / usr/local/nginx/pid/nginx.pid;worker_rlimit_nofile 65535 binary_remote_addr zone=one:10m; limit_conn_zone events {limit_conn_zone 65535;} Default_type application/octet-stream; charset utf-8; server_names_hash_bucket_size 128; client_header_buffer_size 32k; large_client_header_buffers 4 64k; sendfile on; autoindex off; tcp_nopush on; tcp_nodelay on; keepalive_timeout 120; fastcgi_connect_timeout 60; fastcgi_send_timeout 60; fastcgi_read_timeout 60; fastcgi_buffer_size 128k; fastcgi_buffers 8 128k Fastcgi_busy_buffers_size 128k; fastcgi_temp_file_write_size 128k; gzip on; gzip_min_length 1k; gzip_buffers 4 16k; gzip_http_version 1.0; gzip_comp_level 2; gzip_types text/plain application/x-javascript text/css application/xml; gzip_vary on Log_format main'$remote_addr-$remote_user [$time_local] "$request"'$status $body_bytes_sent "$http_referer"'"$http_user_agent" $http_x_forwarded_for'; client_max_body_size 200m; # lua_package_path "/ etc/nginx/lua/?.lua"; include / usr/local/nginx/vhost/*.conf;}

Modify site configuration

The configuration file of an ordinary site, including fixed height width and fixed height, fixed width mode configuration

Cd / usr/local/nginx/mkdir vhost vi vhost/demo.conf# defines the picture size supported by lua thumbnails and switch init_by_lua'--switch needs to limit thumbnail size: true, no need to limit thumbnail size: false image_sizes_check = true-allowable size image_sizes = {"800x800", "400x400", "100x100", "- 800,400", "- 100" "800 -", "400 -", "100 -"}' Server {listen 80; index index.php index.html index.htm; set $root_path'/ var/www'; root $root_path; location / lua {default_type 'text/plain'; content_by_lua' ngx.say ("hello, ttlsa lua")';} location / {try_files $uri $uri/ / index.php?$args # add support for img which has query params, # like: xxx.jpg?a=b&c=d_750x750.jpg if ($args ~ * "^ ([^ _] +) _ (\ d +) + x (\ d +) +\. (jpg | jpeg | gif | png) $") {set $w $2; set $h $3; set $img_ext $4 # rewrite ^\? (. *) $_ ${w} x ${h}. $img_ext? Last; rewrite ([^.] *). (jpg | jpeg | png | gif) $1. 2 img_ext ${w} x ${h}? Permanent;} # set var for thumb pic set $upload_path/ usr/local/filebase; set $img_original_root $upload_path;# original root; set $img_thumbnail_root$ upload_path/cache/thumb; set $img_file $img_thumbnail_root$uri # like:/xx/xx/xx.jpg_100-.jpg or / xx/xx/xx.jpg_-100.jpg location ~ * ^ (. +\. (jpg | jpeg | gif | png)) _ (\ d +\) | (\ -\ d +))\. (jpg | jpeg | gif | png) ${root $img_thumbnail_root; # root path for croped img set $img_size $3 If (!-f $img_file) {# if file not exists add_header X-Powered-By 'Nginx+Lua+GraphicsMagick By Botao'; # header for test add_header file-path $request_filename; # header for test set $request_filepath $img_original_root$1; # origin_img full path:/document_root/1.gif set $img_size $3 # img width or height size depends on uri set $img_ext $2; # file ext content_by_lua_file / usr/local/nginx/lua/autoSize.lua # load lua} # like: / xx/xx/xx.jpg_100x100.jpg location ~ * ^ (. +\. (jpg | jpeg | gif | png)) _ (\ d +) + x (\ d +) +\. (jpg | jpeg | gif | png) ${root $img_thumbnail_root; # root path for croped img if (!-f $img_file) {# if file not exists add_header X-Powered-By 'Nginx+Lua+GraphicsMagick By Botao' # header for test add_header file-path $request_filename; # header for test set $request_filepath $img_original_root$1; # origin_img filepath set $img_width $3; # img width set $img_height $4; # height set $img_ext $5 # file ext content_by_lua_file / usr/local/nginx/lua/cropSize.lua; # load lua}} # if need (all go there) location * / upload {root $img_original_root;} location ~ /\ .ht {deny all;}}

Lua tool for trimming pictures

Cd / usr/local/nginx/mkdir lua

Two files are required under the lua folder

AutoSize.lua height or width mode cropping picture processing lua script

CropSize.lua fixed height and width mode cropping picture processing lua script

The content of the autoSize.lua file is:

-- automatically cut the image size according to the size of the input length or width-- check whether the path is in the directory local function is_dir (sPath) if type (sPath) ~ = "string" then return false endlocal response = os.execute ("cd".. SPath) if response = = 0 whether a then return trueendreturn falseend-- file exists function file_exists (name) local f = io.open (name, "r") if f = nil then io.close (f) return true else return false endend-- get the file path function getFileDir (filename) return string.match (filename, "(. +) / [^ /] *% wicked $")-- * nix systemend-- gets the file name function strippath (filename) return string.match (filename) ". + / ([^ /] *%.% w+) $")-- * nix systemend-- removes the extension function stripextension (filename) local idx = filename:match (". + ()% wicked $") if (idx) then return filename:sub (1) Idx-1) else return filenameendend-- get the extension function getExtension (filename) return filename:match (". +%. (% w+) $") endfunction getImgSize (img) end-- to determine whether the size is legal-- check image sizefunction table.contains (table, element) for _, value in pairs (table) do if value = = element then return true endendreturn falseendif image_sizes_checkthenif not table.contains (image_sizes, ngx.var.img_size) then ngx.exit Endend-- check image end-- starts execution-- ngx.log (ngx.ERR, getFileDir (ngx.var.img_file)); local gm_path = 'gm'-- check image dirif not is_dir (getFileDir (ngx.var.img_file)) thenos.execute ("mkdir-p".. GetFileDir (ngx.var.img_file)) end-- to get height and width 100! Or! 100 mode local uri = ngx.var.img_sizelocal width = string.sub (uri,1,1) local height = 0if width = "-" thenwidth = 0height = string.sub (uri,2,string.len (uri)) elsewidth = string.sub (uri,1,string.len (uri)-1) height = 0endlym-ngx.log (ngx.ERR,uri)-ngx.log (ngx.ERR,width)-ngx.log (ngx.ERR,height)-ngx.log (ngx.ERR,ngx.var.img_file) -- ngx.log (ngx.ERR,ngx.var.request_filepath);-- guarantee proportional thumbnails after cropping (disadvantages: cropping part of the picture)-- such as: gm convert autoSize.jpg-resize x200-quality 100 + profile "*" autoSize.jpg_-200.jpgif (file_exists (ngx.var.request_filepath)) thenlocal cmd = gm_path. ' Convert'.. Ngx.var.request_filepathif height = = 0 then cmd = cmd.. "- resize".. Width.. "x".. "else cmd = cmd.. "- resize".. "x".. Height.. "" because end-- is blurry after compression, the default picture quality is 100. please modify qualitycmd = cmd according to your own situation. "- quality 100" cmd = cmd.. "+ profile\" *\ ".. Ngx.var.img_file;ngx.log (ngx.ERR, cmd); os.execute (cmd); ngx.exec (ngx.var.uri); elsengx.exit (ngx.HTTP_NOT_FOUND); end

The content of the cropSize.lua file is:

-- cut the picture according to the size of the input length and width-- check whether the path is in the directory local function is_dir (sPath) if type (sPath) ~ = "string" then return false endlocal response = os.execute ("cd".. SPath) if response = = 0 whether a then return trueendreturn falseend-- file exists function file_exists (name) local f = io.open (name, "r") if f = nil then io.close (f) return true else return false endend-- get the file path function getFileDir (filename) return string.match (filename, "(. +) / [^ /] *% wicked $")-- * nix systemend-- gets the file name function strippath (filename) return string.match (filename) ". + / ([^ /] *%.% w+) $")-- * nix systemend-- removes the extension function stripextension (filename) local idx = filename:match (". + ()% wicked $") if (idx) then return filename:sub (1) Idx-1) else return filenameendend-- gets the extension function getExtension (filename) return filename:match (". +%. (% w +) $") end-- to determine whether the size is legal-the size of the picture to be cut local img_width_height = ngx.var.img_width.. "x".. Ngx.var.img_height;-- check image sizefunction table.contains (table, element) for _, value in pairs (table) do if value = = element then return true endendreturn falseendif image_sizes_checkthenif not table.contains (image_sizes, img_width_height) then ngx.exit; endend-- check image end-- starts execution-- ngx.log (ngx.ERR, getFileDir (ngx.var.img_file)) Local gm_path = 'gm'-- check image dirif not is_dir (getFileDir (ngx.var.img_file)) thenos.execute ("mkdir-p".. GetFileDir (ngx.var.img_file)) end-- ngx.log (ngx.ERR,ngx.var.img_file);-- ngx.log (ngx.ERR,ngx.var.request_filepath);-- guarantee proportional thumbnails after cropping (disadvantage: clipping part of the picture)-- gm convert cropSize.jpg-thumbnail 300x300 ^-gravity center-extent 300x300-quality 100 + profile "*" cropSize.jpg_300x300.jpgif (file_exists (ngx.var.request_filepath)) thenlocal cmd = gm_path.. ' Convert'.. Ngx.var.request_filepathcmd = cmd.. "- thumbnail".. Ngx.var.img_width.. "x".. Ngx.var.img_height.. "^" cmd = cmd.. "- gravity center-extent".. Ngx.var.img_width.. "x".. Because ngx.var.img_height-- is blurred after compression, the default picture quality is 100. please modify qualitycmd = cmd according to your own situation. "- quality 100" cmd = cmd.. "+ profile\" *\ ".. Ngx.var.img_file;-- ngx.log (ngx.ERR, cmd); os.execute (cmd); ngx.exec (ngx.var.uri); elsengx.exit (ngx.HTTP_NOT_FOUND); end access

Open port 80

Firewall-cmd-permanent-zone=public-add-port=80/tcpfirewall-cmd-reload

Start nginx

Cd / usr/local/nginx/./sbin/nginx

Access to view pictures

Visit the following addresses to test whether you can view and generate thumbnails

Http://XXX/upload/img/001.jpg

Http://XXX/upload/img/001.jpg_-200.jpg

Http://XXX/upload/img/001.jpg_200X200.jpg

The effect is as follows:

At this time, the server has also generated a thumbnail file under the corresponding path:

At this point, nginx+lua+GraphicsMagick generates real-time thumbnails.

Optimize

Now it has been realized that the server-side real-time generation of thumbnails, in order to prevent the server from being maliciously changed width and height parameters to generate a large number of files at will, wasting resources and space, so we added the size limit, the specific configuration has been marked in the demo.conf configuration file.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.