Shulou(Shulou.com)06/02 Report--

Blog structure

The overview and working principle of Varnish, etc.

Install the Varnish cache proxy server

Varnish Overview introduction to 1.Varnish

Varnish is a high-performance and open source reverse proxy server and HTTP accelerator, which adopts a new software architecture and works closely with the current hardware system. Compared with traditional squid, Varnish has the advantages of high performance, high speed and more convenient management. at present, many large websites have begun to try to use Varnish instead of squid, which is the most fundamental reason for the rapid development of Varnish.

The main features of Varnish:

(1) Cache proxy location: either memory or disk can be used; (2) log storage: logs are stored in memory; (3) support the use of virtual memory; (4) there is a precise time management mechanism, that is, cache time attribute control; (5) State engine architecture: different cache and proxy data are processed on different engines. (6) Cache management: manage cached data with binary heap to clean up the data in time; the similarities between 2.Varnish and Squid are both open source software; both are reverse proxy servers; the advantages of Varnish (1) stability: when Varnish and Squid complete the same load of work, the probability of Squid server failure is higher than Varnish, because the use of Squid requires frequent restarts (2) faster access: all cached data in Varnish is read directly from memory, while Squid is read from hard disk. (3) support more concurrent connections: because the TCP connection and release speed of Varnish is much faster than that of Squid. (1) once the Varnish process is restarted, the cached data will be completely released from memory, and all requests will be sent to the back-end server. In the case of high concurrency, it will put a lot of pressure on the back-end server. (2) in the use of Varnish, if a request using a single URL passes through load balancing, each request will fall on a different Varnish server, causing the request to go to the back-end server; and the same Qin Gui caching on multiple servers will also cause a waste of Varnish cache resources and cause performance degradation. The Varnish disadvantage solution aims at disadvantage one: it is recommended to start with varnish's in-memory cache in the case of high traffic, and multiple squid/nginx servers are required. Mainly to prevent a large number of requests from penetrating varnish when the previous varnish service and server are restarted, so that squid/nginx can act as the second layer CACHE, and also make up for the problem that varnish cache will be released when it is restarted in memory; for disadvantage 2: you can do url hashing on load balance, so that a single url request can be fixed to a varnish server; how 3.Varnish works

When the Varnish server receives a request from the client, it is the first choice to check whether there is data in the cache, and if so, respond directly to the client; if not, request the corresponding resources from the back-end server, cache them locally to the Varnish server, and then respond to the client

Select whether the data needs to be cached according to the rules and the type of the request page. You can determine whether the data is cached or not according to the Cache-Contorl in the request header, and whether the cookis is marked. These functions can be achieved by writing configuration files.

4.Varnish simple architecture Varnish is divided into management process and child process management process: manage the child process, colleagues compile the VCL configuration and apply it to different state engines; child process: generate thread pool, which is responsible for processing user requests and return user results through hash lookup; common threads generated by the child process are invite threads: receive new connection requests and respond; worker threads: sessions, processing request resources Expiry thread: clear the expired content in the cache; 5.varnish main configuration part backend configuration: add a reverse proxy server node to Varnish, configure at least one; ACL configuration: add access control lists to Varnish, you can specify or prohibit access to these lists; probes configuration: add rules to Varnish to detect whether the back-end server is normal, to facilitate switching or disable the corresponding back-end server Directors configuration: add load balance mode to Varnish to manage multiple back-end servers; core subroutine configuration: add back-end server switching, request caching, access control, error handling and other rules to Varnish; default variables are built into 6.VCL: variables (also known as object):

As shown in the picture

(1) req: the variable available when the client requests the Varnish server; (2) the variable that the bereq:Varnish server requests the back-end server to use; (3) beresp: the variable used when the back-end server responds to the Varnish server request and returns the result; (4) the variable used by the resp:Varnish server to respond to the client request; (5) obj: cache the object and cache the back-end response request content (6) now: the function is to return the current timest client Clienet.ip: return the IP address of the client Client.port: get the port number requested by the client (you need to call the std module after vatnish5.0 to use it) Syntax is import std Std.port (client.ip) Client.identiy acquires the client identification code. During installation, the software generates a serial number to identify the identity server Server.hostname: server hostname Server.identiy: get server identification code Server.ip: get server IP address Server.prot: get server IP port number Need to call std module client request req (object sent by the client request) Req: data structure of the entire request req.bachend_hint: specify the request backend node such as gif to the picture server Req.can_gzip: whether the client accepts gzip transmission encoding (usually the browser supports all compression formats) req.hash_always_miss: whether to read data from the cache req.hash_ignore_busy A data deadlock that is busy in a slight cache (for example, two varnish servers scramble for a resource during processing, resulting in congestion Req.http: headerreq.method of the request http: request type or method (such as gt,post) req.proto: version of the http protocol used by the client request req.restarts: number of restarts, default maximum value is 4 (usually used to determine whether the server has been accessed) req.url: requested urlreq.xid: unique id X-varnish is added to the header when accessing the varnish server, followed by the id of varnish, and the first data is the identity id of the request. The second data is the cached identity id varnish request backend server (bereq) bereq: the data structure of the entire backend request bereq.backend: the configuration of the requested backend node bereq.between_bytes_timeout: the waiting time or timeout time between each byte received from the backend bereq.http: corresponding to the header information of the http sent to the backend bereq.method: the type or method of the request sent to the backend Bereq.proto: http protocol version of the request sent to the backend bereq.retires: same request retry count bereq.uncacheable: request data is not cached Or request does not cache bereq.url: urlbereq.xid sent to the backend request: request unique id backend server to return data to varnish beresqBeresp: backend server response data Beresp.backend.ip: backend response server IP address Beresp.backend.name: backend response server node name Beresp.do_gunzip: default is false Decompress modified object Beresp.grace before caching: set extra grace time for cache expiration Beresp.http: response http header Beresp.keep: object cache retention time Beresp.proto: response http version Beresp.reason: http status information returned by back-end server Beresp.status: status code returned by back-end server Beresp.storage_hint: specified saved specific memory (memory) Beresp.ttl: change the remaining time of object cache Specify unified cache remaining time Beresp,uncacheable: do not cache data storage Storage..free_space: store free space (bytes) Storage..used_space: store remaining time (bytes) Storage..happy: storage node status deliver sends data to the client, the returned data fetch gets the data from the backend and caches the data locally 7. Specific function statement Ban (expression): clears the specified object cache; Call (subroutine): invokes subroutines; Hash_data (input): generates hash keys based on the values of input subroutines; New (): creates new vcl objects that can only be in the vcl_init subprocess; Return (): ends the current subroutine and specifies to proceed to the next step; Rollback (): restores the HTTP header to its original state and has been deprecated, using std.rollback () instead of std.rollback () Synthetic (STRING): synthesizer that defines the page and status code returned to the client; Regsub (str,regex,sub) replaces the first occurrence of strings with regular expressions; Regsuball (str,regex,sub) replaces all occurrences of strings; steps for 8.varnish request processing

As shown in the picture

The step of varnish request processing is Receive status (vcl_recv). That is, the entry status of the request processing, according to the VCL rules to determine whether the request should be pass (vcl_pass) or pipe (vcl_pipe), or enter lookup (local query). Lookup status. After entering this state, the data is looked up in the hash table, and if found, it enters the hit (vcl_hit) state, otherwise it enters the miss (vcl_miss) state. Pass (vcl_pass) status. In this state, the backend request is directly entered, that is, the fetch (vcl_fetch) status Fetch (vcl_fetch) status is entered. In the fetch state, the request is obtained at the back end, the request is sent, the data is obtained, and the local storage is carried out according to the settings. Deliver (vcl_deliver) status. Send the obtained data to the client, and then complete this request. Pipe status. Establish a direct connection between the client and the server to retrieve data from the back-end server vcl_recv subroutine: start to process the request through return (action); vcl_pipe subroutine: pipe mode processing, which is mainly used to directly take the back-end response content back to the client, you can define the response content to be returned to the client. Vcl_pass subroutine: pass mode processing, this mode is similar to hash cache mode, only do not cache processing. Vcl_hit subroutine: hash cache mode, called when there is a hash cache, for cache processing, you can discard or modify the cache. Vcl_miss subroutine: when there is no hash cache mode, it is called when there is no hash cache. It is used for judging the choice to enter the backend to get the response content, which can be modified to pass mode. Vcl_hash subroutine: hash cache mode, generate hash value as cache lookup key name to extract cache content, mainly used for cache hash key value processing, you can use hash_data (string) to specify key value composition structure, you can generate different cache key values through IP or cookie on the same page. Vcl_purge subroutine: cleanup mode, which is cleared and called when the corresponding cache is found, used to request the method to clear the cache, and reports the vcl_deliver subroutine: client delivery subroutine, called after vcl_backend_response subroutine (non-pipe mode), or vcl_hit subroutine called after vcl_hit subroutine, can be used to append response header information, cookie and so on. Vcl_backend_fetch subroutine: called before sending a back-end request, can be used to change the request address or other information, or to abandon the request. Vcl_backend_response subroutine: called after the response from the backend, which can be used to modify the cache time and cache related information. Vcl_backend_error subroutine: back-end processing failed to call, exception page display effect handling, you can customize the error response content, or modify beresp.status and beresp.http.Location redirection and so on. Vcl_synth subroutine: customize the response content. You can call through synthetic () and the return value synth, where you can customize the exception display or modify the resp.status and resp.http.Location redirection. Vcl_init subroutine: first called when loading vcl to initialize VMODs. This subroutine does not participate in request processing and is only called once when vcl loads. Vcl_fini subroutine: called when uninstalling the current vcl configuration to clean up the VMODs. This subroutine does not participate in request processing and is only called after the vcl is discarded normally. two。 Install Varnish

Download the varnish package

In addition to one varnish server, you can open two more web servers and provide web pages.

[root@localhost ~] # yum-y install autoconf automake libedit-devel libtool ncurses-devel pcre-devel pkgconfig python-docutils python-sphinx [root@localhost ~] # tar zxf varnish-4.0.3.tar.gz [root@localhost ~] # cd varnish-4.0.3/ [root@localhost varnish-4.0.3] #. / configure & & make & & make install [root@localhost varnish-4.0.3] # cp etc/example.vcl / usr/local/var/varnish/// Copy the Varnish main configuration file [root@localhost /] # vim / usr/local/var/varnish/example.vcl / / Edit the Varnish master configuration to see the original modified vcl 4.0 Import directors;import std;backend default {.host = "127.0.0.1"; .port = "80";} probe backend_healthcheck {.url = "/"; .interval = 5s; .timeout = 1s; .window = 5; .threshold = 3;} backend web1 {.host = "192.168.148.132"; .port = "80"; .probe = backend_healthcheck } backend web2 {.host = "192.168.148.133"; .port = "80"; .probe = backend_healthcheck;} acl purgers {"127.0.0.1"; "localhost"; "192.168.148.0 new web_cluster=directors.round_robin 24";! "192.168.148.133";} sub vcl_init {new web_cluster=directors.round_robin () Web_cluster.add_backend (web1); web_cluster.add_backend (web2);} / / add the following sub vcl_recv {set req.backend_hint = web_cluster.backend (); if (req.method = = "PURGE") {if (! client.ip ~ purgers) {return (synth (405, "Not Allowed.")) } return (purge) } if (req.method! = "GET" & & req.method! = "HEAD" & & req.method! = "PUT" & & req.method! = "POST" & & req.method! = "TRACE" & & req.method! = "OPTIONS" & & req.method! = "PATCH" & & req.method! = "DELETE") {return (pipe) } if (req.method! = "GET" & & req.method! = "HEAD") {return (pass);} if (req.url ~ "\. (php | asp | aspx | do | ashx | shtml) ($|\?) {return (pass);} if (req.http.Accept-Encoding) {if (req.url ~"\. (bmp | png | gif | jpeg | ico | gz | tgz | bz2 | tbz | zip | rar | mp3 | mp4 | ogg | ogg) $") } elseif (req.http.Accept-Encoding ~ "gzip") {set req.http.Accept-Encoding = "gzip";} elseif (req.http.Accept-Encoding ~ "deflate") {set req.http.Accept-Encoding = "deflate";} else {unset req.http.Accept-Encoding } if (req.url ~ "\. (css | js | html | htm | bmp | png | gif | jpg | jpeg | ico | gz | tgz | bz2 | tbz | zip | rar | mp3 | mp4 | ogg | swf | flv) ($|\") {unset req.http.cookie; return (hash);} if (req.restarts = 0) {if (req.http.X-Forwarded-For) {set req.http.X-Forwarded-For = req.http.X-Forwarded-For + "," + client.ip " } else {set req.http.X-Forwarded-For = client.ip;}} return (hash);} sub vcl_hash {hash_data (req.url); if (req.http.host) {hash_data (req.http.host);} else {hash_data (server.ip);} return (lookup) } sub vcl_hit {if (req.method = = "PURGE") {return (synth (200, "Purged."));} return (deliver);} sub vcl_miss {if (req.method = = "PURGE") {return (synth (404, "Purged."));} return (fetch) } sub vcl_deliver {if (obj.hits > 0) {set resp.http.CXK = "HIT-from-varnish"; set resp.http.X-Cache-Hits = obj.hits;} else {set resp.http.X-Cache = "MISS";} unset resp.http.X-Powered-By; unset resp.http.Server Unset resp.http.X-Drupal-Cache; unset resp.http.Via; unset resp.http.Link; unset resp.http.X-Varnish; set resp.http.xx_restarts_count = req.restarts; set resp.http.xx_Age = resp.http.Age; set resp.http.hit_count = obj.hits; unset resp.http.Age Return (deliver);} sub vcl_purge {return (synth, "success") } sub vcl_backend_error {if (beresp.status = = 500 | | beresp.status = = 501 | | beresp.status = = 502 | | beresp.status = = 503 | | beresp.status = = 504) {return (retry);}} sub vcl_fini {return (ok) } [root@localhost /] # varnishd-f / usr/local/var/varnish/example.vcl-s malloc 200m-a 0.0.0.0 yum 80max / startup service first web provides page [root@localhost ~] # yum-y install httpd [root@localhost ~] # echo aaa > / var/www/html/index.html [root@localhost ~] # systemctl stop firewalld [root@localhost ~] # systemctl start httpd second [root@localhost ~] # yum-y install httpd [root@localhost ~] # echo bbb > / var/www/html/index.html [root@localhost ~] # Systemctl stop firewalld [root@localhost ~] # systemctl start httpd if you restart Varnishd as follows: [root@localhost /] # netstat-anpt | grep 80 [root@localhost /] # killall-9 varnishd [root@localhost /] # varnishd-f / usr/local/var/varnish/example.vcl-s malloc 200m-a 0.0.0.0 80 client access is as follows:

Refresh it.

[root@localhost /] # curl-X "PURGE" 192.168.148.130max / clear cache

The Varnish configuration file interprets vcl 4.0 per import directors;import std;# Default backend definition. Set this to point to your content server.probe backend_healthcheck {.url = "/"; # access the root path of the backend server .interval = 5s; # request interval .timeout = 1s; # request timeout .window = 5; # specify the number of polls 5 times. Threshold = 3 # if there are 3 failures, it means that the backend server is abnormal} backend web1 {# defines the backend server .host = "192.168.1.7"; # to go to the IP or domain name .port of the host (that is, the backend host); # specify the port number of the backend server .probe = backend_healthcheck # Health check calls the contents defined by backend_healthcheck} backend web2 {.host = "192.168.1.8"; .port = "80"; .probe = backend_healthcheck;} acl purgers {# define access control list "127.0.0.1"; "localhost"; "192.168.1.0 amp 24";! "192.168.1.8" } sub vcl_init {# call the vcl_init initialization subroutine to create a back-end host group, namely directors new web_cluster=directors.round_robin (); # create a drector object using the new keyword and use the round_robin (polling) algorithm web_cluster.add_backend (web1); # add a back-end server node web_cluster.add_backend (web2) } sub vcl_recv {set req.backend_hint = web_cluster.backend (); # specify the backend node if defined by the request backend node web_cluster (req.method = = "PURGE") {# determine whether the client's request header is PURGE if (! client.ip ~ purgers) {# if so, determine whether the client's IP address is in the ACL access control list. Return (synth (405, "Not Allowed.")); # if not, return the client 405 status code and return the defined page. } return (purge) # if it is defined by ACL Leave it to purge to handle.} if (req.method! = "GET" & & req.method! = "HEAD" & & req.method! = "PUT" & & req.method! = "POST" & & req.method! = "TRACE" & & req.method! = "OPTIONS" & & req.method! = "PATCH" & & req.method! = "DELETE") {# determine the request type of the client return (pipe) } if (req.method! = "GET" & & req.method! = "HEAD") {return (pass); # if it is not GET and HEAD, give it to pass.} if (req.url ~ "\. (php | asp | aspx | do | ashx | shtml) ($|\?") {return (pass) # if (req.http.Accept-Encoding) {if (req.url ~ "\. (bmp | png | gif | jpg | ico | gz | bz2 | tbz | zip | mp3 | mp4 | ogg | swf | flv) $") {unset req.http.Accept-Encoding; # cancels the compression type received by the client} elseif (req.http.Accept-Encoding ~ "gzip") {set req.http.Accept-Encoding = "gzip" # if there is a gzip type, mark the gzip type. } elseif (req.http.Accept-Encoding ~ "deflate") {set req.http.Accept-Encoding = "deflate";} else {unset req.http.Accept-Encoding; # other undefined pages also cancel customer-acceptable compression types. } if (req.url ~ "\. (css | js | html | htm | bmp | png | gif | jpg | jpeg | ico | gz | tgz | bz2 | tbz | zip | rar | mp3 | mp4 | swf | flv) ($|\?) {unset req.http.cookie; # cancel the cookie value of the client. Return (hash); # forward the request to the hash subroutine, that is, check the local cache.} if (req.restarts = = 0) {# determine whether the client requests if (req.http.X-Forwarded-For) for the first time {# if it is the first time, set the IP address of the client. Set req.http.X-Forwarded-For = req.http.X-Forwarded-For + "," + client.ip;} else {set req.http.X-Forwarded-For = client.ip;}} return (hash);} sub vcl_hash {hash_data (req.url) # View the page requested by the client and hash if (req.http.host) {hash_data (req.http.host); # set the host of the client} else {hash_data (server.ip); # set the IP} return (lookup) of the server } sub vcl_hit {if (req.method = = "PURGE") {# if it is HIT and when the type of client request is the 200 status code returned by PURGE, and return the corresponding page. Return (synth (200, "Purged."));} return (deliver);} sub vcl_miss {if (req.method = = "PURGE") {return (synth (404, "Purged.")); # return 404} return (fetch) if miss } sub vcl_deliver {if (obj.hits > 0) {set resp.http.CXK = "HIT-from-varnish"; # set http header X-Cache = hit set resp.http.X-Cache-Hits = obj.hits; # the number of times a command is returned} else {set resp.http.X-Cache = "MISS";} unset resp.http.X-Powered-By # cancel display of web version unset resp.http.Server; # cancel display of varnish Service unset resp.http.X-Drupal-Cache; # cancel display of cached frame unset resp.http.Via; # cancel display of file content source unset resp.http.Link; # cancel display of HTML's hyperlink address unset resp.http.X-Varnish # cancel the display of varnish id set resp.http.xx_restarts_count = req.restarts; # set the number of client requests set resp.http.xx_Age = resp.http.Age; # display the length of cache files # set resp.http.hit_count = obj.hits; # display the number of cache hits # unset resp.http.Age; return (deliver) } sub vcl_pass {return (fetch); # caches the data returned by the backend server locally} sub vcl_backend_response {set beresp.grace = 5m; # additional cache grace time if (beresp.status = = 499 | | beresp.status = = 404 | | beresp.status = = 502) {set beresp.uncacheable = true # do not cache} if (bereq.url ~ "\. (php | jsp) (\? | $)") {set beresp.uncacheable = true; # when the corresponding status code of the backend server is 449, etc. (css | js | html | htm | bmp | png | gif | jpg | jpeg | ico) ($|\)) {set beresp.ttl = 15m # when it ends above, cache 15 minutes unset beresp.http.Set-Cookie;} elseif (bereq.url ~ "\. (gz | tgz | bz2 | tbz | zip | rar | mp3 | mp4 | ogg | swf | flv) ($|\)) {set beresp.ttl = 30m; # cache 30 minutes unset beresp.http.Set-Cookie } else {set beresp.ttl = 10m; # Live time 10 minutes unset beresp.http.Set-Cookie;}} return (deliver);} sub vcl_purge {return (synth (200m, "success")) } sub vcl_backend_error {if (beresp.status = = 500 | | beresp.status = = 501 | | beresp.status = = 502 | | beresp.status = = 503 | | beresp.status = = 504) {return (retry); # if the status code is one of the above, rerequest}} sub vcl_fini {return (ok);}

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.