Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to deploy Varnish cache proxy server based on Centos7". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to deploy Varnish cache proxy server based on Centos7".

I. Overview of varnish

Introduction to 1.varnish

Varnish is a high-performance and open source reverse proxy server and http accelerator, which adopts a new software architecture and works closely with the current hardware system. Compared with traditional squid, varnish has the advantages of high performance, high speed and more convenient management. at present, many large websites have begun to try to use varnish instead of squid, which is the most fundamental reason for the rapid development of varnish.

The main features of varnish:

(1) Cache proxy location: you can use memory or disk

(2) Log storage: logs are stored in memory

(3) support the use of virtual memory

(4) there is an accurate time management mechanism, that is, the time attribute control of cache.

(5) State engine architecture: different caches and proxy data are processed on different engines.

(6) Cache management: cache data is managed by binary heap to clean up the data in time.

The comparison between 2.varnish and squid has something in common

It's all open source software.

It's a reverse proxy server.

Advantages of varnish

(1) Stability: when varnish and squid complete the same load of work, the probability of squid server failure is higher than that of varnish, because using squid requires frequent restarts.

(2) faster access: all cached data in varnish is read directly from memory, while squid is read from hard disk.

(3) support more concurrent connections: because the tcp connection and release speed of varnish is much faster than that of squid

Disadvantages of varnish

(1) once the varnish process is restarted, the cached data will be completely released from memory, and all requests will be sent to the back-end server. In the case of high concurrency, it will cause great pressure on the back-end server.

(2) in the use of varnish, if a request using a single url passes through load balancing, each request will fall on a different varnish server, causing the request to go to the back-end server; and the same Qin Gui caching on multiple servers will also cause a waste of varnish cache resources and cause performance degradation.

Solutions to the disadvantages of varnish

For disadvantage 1: in the case of heavy traffic, it is recommended to use varnish's in-memory cache to start, and multiple squid/nginx servers are needed. Mainly in order to prevent the previous varnish service and server from being restarted, a large number of requests penetrate varnish, so that squid/nginx can act as the second layer cache, and it also makes up for the problem that varnish cache will be released when restart in memory.

For the second disadvantage: url hash can be done on load balancer, so that a single url request can be routed to a varnish server.

How 3.varnish works

When the varnish server receives a request from the client, it is the first choice to check whether there is data in the cache, and if so, respond directly to the client; if not, request the corresponding resources from the back-end server, cache them locally to the varnish server, and then respond to the client

Select whether the data needs to be cached according to the rules and the type of the request page. You can determine whether the data is cached or not according to the cache-contorl in the request header, and whether the cookis is marked. These functions can be achieved by writing configuration files.

4.varnish simple Architecture

Varnish is divided into management process and child process.

Management process: manage the child process, and colleagues compile the vcl configuration and apply it to different state engines

Child process: generates a thread pool that processes user requests and returns user results through hash lookups

Common threads generated by the child process are

Accept thread: receives a new connection request and responds

Worker threads: sessions, processing requested resources

Expiry threads: clearing expired content in the cache

The main configuration part of 5.varnish

Backend configuration: add a reverse proxy server node to varnish and configure at least one

Acl configuration: add access control lists to varnish, which can be specified or prohibited

Probes configuration: add a rule to varnish to detect whether the backend server is normal, making it easy to switch or disable the corresponding backend server.

Directors configuration: add load balancing mode to varnish to manage multiple back-end servers

Core subroutine configuration: add back-end server switching, request caching, access control, error handling and other rules to varnish

Default variables are built into 6.vcl: variables (also known as object):

As shown in the picture

(1) req: the variables available when the client requests the varnish server

(2) bereq:varnish server requests variables available to backend server

(3) beresp: the variable used by the back-end server when responding to the varnish server request and returning the result

(4) the variable used by the resp:varnish server to respond to client requests

(5) obj: cache the object and the response request content of the backend.

(6) now: function is to return the current timestamp

Client

Clienet.ip: returns the ip address of the client

Client.port: get the port number requested by the client (after vatnish4.0, you need to call the std module to use it) the syntax is import std; std.port (client.ip)

Client.identiy acquires the client identification code, and the software generates a serial number during installation to identify the identity.

Server

Server.hostname: server hostname

Server.identiy: get the server identification code

Server.ip: get the server ip address

Server.prot: to get the ip port number of the server, you need to call the std module

Client request req (the object that the client requests to send)

Req: the data structure of the entire request

Req.bachend_hint: specify the request backend node such as gif to send to the image server

Req.can_gzip: whether the client accepts gzip transport encoding (usually browsers support all compressed formats)

Req.hash_always_miss: whether to read data from the cache

Req.hash_ignore_busy: ignore the busy data deadlock in the cache (for example, two varnish servers scramble for a resource together during processing, resulting in blockage, and if it is not stopped manually, the deadlock will continue)

Req.http: the header corresponding to the request http

Req.method: the type of request or the way it is requested (e.g. gt,post)

Req.proto: the version of the http protocol used by the client request

Req.restarts: the number of restarts. The default maximum is 4 (usually used to determine whether the server has been accessed)

Req.url: the requested url

Req.xid: x-varnish is added to the header when the unique id,varnish server is accessed, followed by the id of varnish, the first data is the requested identity id, and the second data is the cached identity id

Varnish request backend server (bereq)

Bereq: the data structure of the entire backend request

Bereq.backend: configuration of the requested backend node

Bereq.between_bytes_timeout: the wait time or timeout time between each byte received from the backend

Bereq.http: corresponds to the header information of the http sent to the backend

Bereq.method: the type or method of request sent to the backend

Bereq.proto: the http protocol version of the request sent to the backend

Bereq.retires: same request retry count

Bereq.uncacheable: request data is not cached, or request is not cached

Bereq.url: the url sent to the backend request

Bereq.xid: request a unique id

The back-end server returns data beresq to varnish

Beresp: back-end server response data

Beresp.backend.ip: the ip address of the backend response (processing request data) server

Beresp.backend.name: node name of the backend response server

Beresp.do_gunzip: defaults to false. Decompress the modified object before caching.

Beresp.grace: sets the extra grace time for cache expiration

Beresp.http: the header of the http in response

Beresp.keep: object cache with hold time

Beresp.proto: the http version of the response

Beresp.reason: the http status information returned by the backend server

Beresp.status: the status code returned by the backend server

Beresp.storage_hint: specifies the specific memory (memory) to be saved

Beresp.ttl: change the remaining time of object cache and specify the remaining time of unified cache

Beresp,uncacheable: data is not cached

Storage

Storage..free_space: storage free space (bytes)

Storage..used_space: time remaining for storage (bytes)

Storage..happy: storage node statu

Deliver sends the data to the client, and the returned data

Fetch fetches data from the backend and caches it locally

7. Specific function statement

Ban (expression): clears the specified object cache

Call (subroutine): call subroutine

Hash_data (input): generates a hash key based on the value of a subroutine of input

New (): create a new vcl object, only in the vcl_init child process

Return (): ends the current subroutine and specifies to proceed to the next step

Rollback (): restore the http header to its original state, which has been deprecated, instead of using std.rollback ()

Synthetic (string): a synthesizer that defines the page and status code returned to the client

Regsub (str,regex,sub) replaces the first occurrence of a string with a regular expression

Regsuball (str,regex,sub) replaces all strings that appear

Steps for 8.varnish request processing

As shown in the picture

Steps for varnish request processing

Receive status (vcl_recv). That is, the entry status of the request processing, according to the vcl rules to determine whether the request should be pass (vcl_pass) or pipe (vcl_pipe), or enter lookup (local query).

Lookup status. After entering this state, the data is looked up in the hash table, and if found, it enters the hit (vcl_hit) state, otherwise it enters the miss (vcl_miss) state.

Pass (vcl_pass) status. In this state, the backend request is directly entered, that is, the fetch (vcl_fetch) state is entered.

Fetch (vcl_fetch) status. In the fetch state, the request is obtained at the back end, the request is sent, the data is obtained, and the local storage is carried out according to the settings.

Deliver (vcl_deliver) status. Send the obtained data to the client, and then complete this request.

Pipe status. Establish a direct connection between the client and the server to retrieve data from the back-end server

Vcl_recv subroutine: start processing the request through return (action)

Vcl_pipe subroutine: pipe mode processing, this mode is mainly used to directly take the back-end response content back to the client, you can define the response content to return to the client.

Vcl_pass subroutine: pass mode processing, this mode is similar to hash cache mode, only do not cache processing.

Vcl_hit subroutine: hash cache mode, called when there is a hash cache, for cache processing, you can discard or modify the cache.

Vcl_miss subroutine: when there is no hash cache mode, it is called when there is no hash cache. It is used for judging the choice to enter the backend to get the response content, which can be modified to pass mode.

Vcl_hash subroutine: hash cache mode, generate hash value as cache lookup key name to extract cache content, mainly used for cache hash key value processing, you can use hash_data (string) to specify key value composition structure, you can generate different cache key values through ip or cookie on the same page.

Vcl_purge subroutine: cleanup mode, cleared and called when the corresponding cache is found, used to request the method to clear the cache and report

Vcl_deliver subroutine: client delivery subroutine, called after vcl_backend_response subroutine (non-pipe mode), or vcl_hit subroutine, can be used to append response header information, cookie and so on.

Vcl_backend_fetch subroutine: called before sending a back-end request, can be used to change the request address or other information, or to abandon the request.

Vcl_backend_response subroutine: called after the response from the backend, which can be used to modify the cache time and cache related information.

Vcl_backend_error subroutine: back-end processing failed to call, exception page display effect handling, you can customize the error response content, or modify beresp.status and beresp.http.location redirection and so on.

Vcl_synth subroutine: customize the response content. You can call through synthetic () and the return value synth, where you can customize the exception display or modify the resp.status and resp.http.location redirection.

Vcl_init subroutine: first called when loading vcl to initialize vmods. This subroutine does not participate in request processing and is only called once when vcl loads.

Vcl_fini subroutine: called when uninstalling the current vcl configuration to clean up the vmods. This subroutine does not participate in request processing and is only called after the vcl is discarded normally.

two。 Install varnish

Download varnish package link: https://pan.baidu.com/s/1ofnyr-5xfuxmuyjtnhqesa extraction code: m9q4

In addition to one varnish server, you can open two more web servers and provide web pages.

[root@localhost ~] # yum-y install autoconf automake libedit-devel libtool ncurses-devel pcre-devel pkgconfig python-docutils python-sphinx [root@localhost ~] # tar zxf varnish-4.0.3.tar.gz [root@localhost ~] # cd varnish-4.0.3/ [root@localhost varnish-4.0.3] #. / configure & & make & & make install [root@localhost varnish-4.0.3] # cp etc/example.vcl / usr/local/var/varnish/// Copy the varnish main configuration file [root@localhost /] # vim / usr/local/var/varnish/example.vcl / / Edit the varnish master configuration to see the original modified vcl 4.0 Import directors;import std;backend default {.host = "127.0.0.1"; .port = "80";} probe backend_healthcheck {.url = "/"; .interval = 5s; .timeout = 1s; .window = 5; .threshold = 3;} backend web1 {.host = "192.168.148.132"; .port = "80"; .probe = backend_healthcheck;} backend web2 {.host = "192.168.148.133" .port = "80"; .probe = backend_healthcheck;} acl purgers {"127.0.0.1"; "localhost"; "192.168.148.0 amp 24";! "192.168.148.133";} sub vcl_init {new web_cluster=directors.round_robin (); web_cluster.add_backend (web1); web_cluster.add_backend (web2) } / / add the following sub vcl_recv {set req.backend_hint = web_cluster.backend (); if (req.method = = "purge") {if (! client.ip ~ purgers) {return (synth (405, "not allowed."));} return (purge) } if (req.method! = "get" & & req.method! = "head" & & req.method! = "put" & & req.method! = "post" & & req.method! = "trace" & & req.method! = "options" & & req.method! = "patch" & & req.method! = "delete") {return (pipe) } if (req.method! = "get" & & req.method! = "head") {return (pass);} if (req.url ~ "\. (php | asp | aspx | do | ashx | shtml) ($|\?) {return (pass);} if (req.http.accept-encoding) {if (req.url ~"\. (bmp | png | gif | jpeg | ico | gz | tgz | bz2 | tbz | zip | rar | mp3 | mp4 | ogg | ogg) $") } elseif (req.http.accept-encoding ~ "gzip") {set req.http.accept-encoding = "gzip";} elseif (req.http.accept-encoding ~ "deflate") {set req.http.accept-encoding = "deflate";} else {unset req.http.accept-encoding;}} if (req.url ~ "\. (css | js | htm | bmp | gif | jpg | jpeg | ico | gz | tgz | bz2 | tbz | zip | rar | mp3 | mp4 | ogg | swf) ($|\?) {unset req.http.cookie; return (unset req.http.cookie; return) } if (req.restarts = = 0) {if (req.http.x-forwarded-for) {set req.http.x-forwarded-for = req.http.x-forwarded-for + "," + client.ip;} else {set req.http.x-forwarded-for = client.ip;}} return (hash);} sub vcl_hash {hash_data (req.url) If (req.http.host) {hash_data (req.http.host);} else {hash_data (server.ip);} return (lookup);} sub vcl_hit {if (req.method = = "purge") {return (synth (200, "purged."));} return (deliver) } sub vcl_miss {if (req.method = = "purge") {return (synth (404, "purged."));} return (fetch);} sub vcl_deliver {if (obj.hits > 0) {set resp.http.cxk = "hit-from-varnish"; set resp.http.x-cache-hits = obj.hits;} else {set resp.http.x-cache = "miss" } unset resp.http.x-powered-by; unset resp.http.server; unset resp.http.x-drupal-cache; unset resp.http.via; unset resp.http.link; unset resp.http.x-varnish; set resp.http.xx_restarts_count = req.restarts; set resp.http.xx_age = resp.http.age; set resp.http.hit_count = obj.hits; unset resp.http.age Return (deliver);} sub vcl_purge {return (synth, "success");} sub vcl_backend_error {if (beresp.status = = 500 | | beresp.status = = 501 | | beresp.status = = 502 | | beresp.status = = 503 | | beresp.status = = 504) {return (retry);}} sub vcl_fini {return (ok) } [root@localhost /] # varnishd-f / usr/local/var/varnish/example.vcl-s malloc,200m-a 0.0.0.0

The first web provides a page

[root@localhost ~] # yum-y install httpd [root@localhost ~] # echo aaa > / var/www/html/index.html [root@localhost ~] # systemctl stop firewalld [root@localhost ~] # systemctl start httpd

The second station

[root@localhost ~] # yum-y install httpd [root@localhost ~] # echo bbb > / var/www/html/index.html [root@localhost ~] # systemctl stop firewalld [root@localhost ~] # systemctl start httpd

If you restart varnishd as follows:

[root@localhost /] # netstat-anpt | grep 80 [root@localhost /] # killall-9 varnishd [root@localhost /] # varnishd-f / usr/local/var/varnish/example.vcl-s malloc,200m-a 0.0.0.0 anpt 80

The client access is as follows:

Refresh it.

[root@localhost /] # curl-x "purge" 192.168.148.130max / clear cache

Varnish profile interpretation

Vcl 4.0 China import directors;import std;# default backend definition. Set this to point to your content server.probe backend_healthcheck {.url = "/"; # access the root path of the backend server .interval = 5s; # request interval .timeout = 1s; # request timeout .window = 5; # specify the number of polls 5 times. Threshold = 3 # if there are 3 failures, it means that the backend server is abnormal} backend web1 {# defines the backend server .host = "192.168.1.7"; # to go to the ip or domain name .port of the host (that is, the backend host); # specify the port number of the backend server .probe = backend_healthcheck # Health check calls the contents defined by backend_healthcheck} backend web2 {.host = "192.168.1.8"; .port = "80"; .probe = backend_healthcheck;} acl purgers {# define access control list "127.0.0.1"; "localhost"; "192.168.1.0 amp 24";! "192.168.1.8" } sub vcl_init {# calls the vcl_init initialization subroutine to create the back-end host group, namely directors new web_cluster=directors.round_robin (); # creates the drector object using the new keyword and uses the round_robin (polling) algorithm web_cluster.add_backend (web1); # adds the back-end server node web_cluster.add_backend (web2);} sub vcl_recv {set req.backend_hint = web_cluster.backend () # specify the backend node if defined by the request backend node web_cluster (req.method = = "purge") {# determine whether the client's request header is purge if (! client.ip ~ purgers) {# if so, determine whether the client's ip address is in the acl access control list. Return (synth (405, "not allowed.")); # if not, return the client 405 status code and return the defined page. } return (purge) # if it is defined by acl Leave it to purge to handle.} if (req.method! = "get" & & req.method! = "head" & & req.method! = "put" & & req.method! = "post" & & req.method! = "trace" & & req.method! = "options" & & req.method! = "patch" & & req.method! = "delete") {# judge the client Request type return (pipe) of } if (req.method! = "get" & & req.method! = "head") {return (pass); # if it is not get and head, give it to pass.} if (req.url ~ "\. (php | asp | aspx | do | ashx | shtml) ($|\?") {return (pass) # if (req.http.accept-encoding) {if (req.url ~ "\. (bmp | png | gif | jpg | jpeg | ico | tgz | tbz | zip | rar | mp3 | mp4 | swf | flv) $") {unset req.http.accept-encoding; # cancels the compression type received by the client} elseif (req.http.accept-encoding ~ "gzip") {set req.http.accept-encoding = "gzip"; # if there is a gzip type, mark it. } elseif (req.http.accept-encoding ~ "deflate") {set req.http.accept-encoding = "deflate";} else {unset req.http.accept-encoding; # other undefined pages also cancel customer-acceptable compression types. } if (req.url ~ "\. (css | js | html | htm | bmp | png | gif | jpg | jpeg | ico | gz | tgz | bz2 | tbz | zip | rar | mp3 | mp4 | swf | flv) ($|\?) {unset req.http.cookie; # cancel the cookie value of the client. Return (hash); # forward the request to the hash subroutine, that is, check the local cache.} if (req.restarts = = 0) {# determine whether the client requests if (req.http.x-forwarded-for) for the first time {# if it is the first time, set the ip address of the client. Set req.http.x-forwarded-for = req.http.x-forwarded-for + "," + client.ip;} else {set req.http.x-forwarded-for = client.ip;}} return (hash);} sub vcl_hash {hash_data (req.url); # View the page requested by the client and hash if (req.http.host) {hash_data (req.http.host) # set the host of the client} else {hash_data (server.ip); # set the ip of the server} return (lookup);} sub vcl_hit {if (req.method = = "purge") {# if it is hit and when the type of client request is the 200 status code returned by purge, and return the corresponding page. Return (synth, "purged.");} return (deliver);} sub vcl_miss {if (req.method = = "purge") {return (synth (404, "purged.")); # if miss returns 404} return (fetch);} sub vcl_deliver {if (obj.hits > 0) {set resp.http.cxk = "hit-from-varnish" # set http header x-cache = hit set resp.http.x-cache-hits = obj.hits; # the number of times the command is returned} else {set resp.http.x-cache = "miss";} unset resp.http.x-powered-by; # cancel the display of web version unset resp.http.server; # cancel the display of varnish service unset resp.http.x-drupal-cache # cancel display of cached frame unset resp.http.via; # cancel display of file content source unset resp.http.link; # cancel display of hyperlink address of html unset resp.http.x-varnish; # cancel display of id set resp.http.xx_restarts_count of varnish = req.restarts; # set number of client requests set resp.http.xx_age = resp.http.age # length of time to display cache files # set resp.http.hit_count = obj.hits; # shows the number of cache hits # unset resp.http.age; return (deliver);} sub vcl_pass {return (fetch); # caches the data returned by the back-end server locally} sub vcl_backend_response {set beresp.grace = 5m # additional cache grace time if (beresp.status = = 499 | | beresp.status = = 404 | | beresp.status = = 502) {set beresp.uncacheable = true; # when the corresponding status code of the backend server is 449, etc., no cache} if (bereq.url ~ "\. (php | jsp) (\? | $)") {set beresp.uncacheable = true # when the page of php does not cache} else {if (bereq.url ~ "\. (css | js | html | htm | bmp | gif | jpg | jpeg | ico) ($|\?) {set beresp.ttl = 15m; # when it ends above, cache unset beresp.http.set-cookie;} elseif for 15 minutes (bereq.url ~"\. (gz | tgz | bz2 | tbz | zip | rar | mp3 | mp4 | ogg | swf | flv) ($|\) ") {set beresp.ttl = 30m # cache 30 minutes unset beresp.http.set-cookie;} else {set beresp.ttl = 10m; # Live time 10 minutes unset beresp.http.set-cookie;}} return (deliver);} sub vcl_purge {return (synth (200m, "success")) } sub vcl_backend_error {if (beresp.status = = 500 | | beresp.status = = 501 | | beresp.status = = 502 | | beresp.status = = 503 | | beresp.status = = 504) {return (retry); # if the status code is one of the above, re-request}} sub vcl_fini {return (ok) } Thank you for reading, the above is the content of "how to deploy Varnish cache proxy server based on Centos7". After the study of this article, I believe you have a deeper understanding of how to deploy Varnish cache proxy server based on Centos7, and the specific usage needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.