Shulou(Shulou.com)06/02 Report--

Blog outline:

A brief introduction to Memcache

II. Memcache work flow

Third, Memcache scheduling algorithm

Fourth, the principle of Memcache implementation.

5. Install Memcache

(1) install nginx server

(2) install PHP server

(3) install MySQL database

(4) Test the connectivity between PHP, Nginx and MySQL.

(5) install Memcache server

(6) PHP server installs Memcache client

(7) using memcache to realize session sharing

(8) Test Memcache cache database

A brief introduction to Memcache

Memcache is a free, open source, high-performance, distributed cache system. Because Memcache reduces the number of times the database is read by caching data and objects in memory. At present, it is used by many websites to improve the access speed of the website, especially for some large websites that need to visit the database frequently.

Memcache is a HashMap that stores key-value pairs. Any data can be stored in memory using key-value, and the database can come from database calls or API calls. Memcache design philosophy is small and powerful, her simple design promotes rapid deployment, easy to develop and solve many problems of large-scale data caching, and its open API enables Memcache to be used in most popular programming languages such as Java, C/C++/C#, Perl, Python and so on.

II. Memcache work flow

Note that although Memcache is called "distributed cache", Memcache itself does not have distributed features at all, and Memcache clusters do not communicate with each other. The so-called "distributed" depends entirely on client programs, as shown in the figure:

Memcahe workflow:

(1) the application enters the data that needs to be written to the cache

(2) API inputs Key into the routing algorithm module, and the reason algorithm gets the server number according to the list of Key and Memcache cluster servers.

(3) get the Memcache and its IP address and port number from the server number

(4) API calls the communication module to communicate with the server with the specified number, writes data to the server, and completes a write operation of distributed cache.

Whether it is read cache or write cache, as long as the same routing algorithm and server list are used, the same key,Memcache client always accesses the same client to read data, and as long as there is still cache of the data in the server cache, the cache hit can be guaranteed.

This method of Memcache clustering is also considered in terms of partition fault tolerance. If Node2 is down, the data stored on Node2 is unavailable. Since Node0 and Node2 still exist in the cluster, the next request to get Node2 cached data will definitely miss. At this time, the cached data will be obtained from the database first, and the cached data will be stored on Node0 or Node1 according to the routing algorithm. This kind of clustering approach is very good, but the disadvantage is that the cost is too high.

Third, Memcache scheduling algorithm

There are two routing scheduling algorithms in Memcache: remainder Hash and consistent Hash.

(1) remainder Hash

A simple routing algorithm can use the remainder Hash: divided by the number of servers and the hash value of the cached data Key, and the remainder is the server number in the server list. Because of the strong randomness of HashCode, the use of remainder Hash routing algorithm can ensure a more balanced distribution of cached data in the whole Memcache server cluster.

If you do not consider the scalability of the server cluster, then the remainder Hash algorithm can almost meet the needs of the vast majority of cache routers, but when the distributed cache cluster needs to be expanded, it will be a big problem.

For example, if the cluster of Memcache servers has changed from 3 to 4, suppose there are 20 pieces of data with a HashCode of 0,19, as shown in the figure:

3 Memcache server environments:

Now expand the capacity to 4 Memcache servers:

Just expand to 4 sets, the original cached data can only be hit 6 times, then if the capacity is expanded to 20 sets per row, only the corresponding data in the first three caches can be hit. This can be illustrated by the above examples:

The routing algorithm using the remainder Hash will cause a large amount of data to fail to hit the cache correctly when the capacity is expanded.

In the website business, most of the business data operation requests will be obtained through the cache, and only a small number of data operations will access the database, so the load capacity of the database is designed under the premise of cache. When most of the cached data cannot be read correctly because of the expansion of the Memcache server, the pressure of these data access will fall on the database, which will greatly exceed the load capacity of the database, and in serious cases will cause database downtime.

For the Memcache cluster environment that uses the remainder Hash algorithm, when you need to expand the capacity, the solution:

When the number of visits to the website is low, usually late at night, the technical team works overtime, expands capacity, and restarts the server; gradually warm up the cache by simulating requests to redistribute the data in the cache server; (2) consistent Hash

The consistent Hash algorithm implements the Hash mapping from key to cache server through a data structure called consistent Hash ring. To put it simply, consistent Hash organizes the entire Hash space into a virtual ring. Assuming that the value space of the hash function H in a space is 0 ~ 2 ^ 32-1, then the entire Hash space is shown in the figure:

Use H for a Hash calculation for each server, specifically using the server's IP address or hostname as a keyword, so that each server can determine its location on the above Hash ring and sort it clockwise.

Suppose the calculated locations of the three Memcache are as follows:

Next, the hash value of the data is calculated using the same algorithm, and the position of the data on the hash ring is determined. For example, there are four data, and the position after hash calculation is as follows:

According to the consistent Hash algorithm, according to the clockwise method of finding the nearest service node, the Hash ring scheduling method has high fault tolerance and scalability.

Assuming that server03 is down, the following occurs:

You can see that C and B will be affected at this time, and redirect the B and C nodes to server01.

In the consistent Hash algorithm, if a server is unavailable, only the first server of this server in a clockwise direction is affected, and the other servers are not affected.

Suppose you add another server, server04, in the original environment, and the following occurs:

At this point, A, D, and C are not affected, only B needs to be redirected to the new Server04.

In the consistent Hash algorithm, if you add a server, the affected data is only the first server of the new server in a clockwise direction, and the rest will not be affected.

To sum up, the advantages and disadvantages of consistent Hash algorithm are as follows:

Advantages: for the increase or decrease of nodes, only part of the data in the ring space needs to be relocated, which has good fault-tolerant rows and scalability; disadvantages: when there are too few service nodes, the container has the problem of data skew because of the uniform distribution of nodes.

Of course, we can solve its shortcomings by adding virtual nodes.

In short, the more cache server nodes in the cluster, the smaller the impact of increasing or decreasing nodes; with the increase of the size of the cluster, the probability of continuing to hit the original cached data will become greater and greater, although there are still a small number of data cache servers that cannot be read, but the proportion is relatively small. Even if you access the database, it will not cause fatal load pressure on the database.

Fourth, the principle of Memcache implementation.

Characteristics of Memcache:

Access to the database is faster than traditional relational databases (because Memcache is stored in memory, while traditional relational databases are stored on disk); Memcache data is stored in memory, which means that as soon as Memcache restarts, the data will be lost. Since most of them are 64-bit operating systems, the impact of memory on 32-bit systems will not be discussed here.

The principle of Memcache: the most important thing is how to allocate memory. The memory allocation method adopted by Memcache is fixed space allocation, as shown in the figure:

The four concepts of stab_class, slab, page and chunk are mainly designed in the figure, and the relationship between them:

(1) Memcache divides the memory space into a set of slab

(2) there are several page under each slab, and each page defaults to 1m. If a slab occupies 100m of memory, then there should be 100m page under this slab.

(3) each page contains a set of chunk,chunk, where the real data is stored, and the size of the chunk in the same slab is fixed.

(4) slab with the same size chunk is organized together, which is called slab_class

The method of Memcache memory allocation is called allocator (allocation operation). The number of slab is limited, several, a dozen or dozens, which is related to the configuration of startup parameters.

Where the value in the Memcache is stored is determined by the size of the value, and the value is always stored in a slab that is closest to the size of the chunk.

What if there is no chunk to allocate in this slab? If the Memcache startup does not append "- M", then Memcache will clean up the data from the least recently used chunk in the slab and put the latest data.

The workflow of 1.Memcache

As shown in the figure:

1. Check whether the request data of the client is in the memcached. If so, return the request data directly without any operation on the database. The path is treated as ①②③⑦.

2. If the requested data is not in the memcached, check the database, return the data obtained from the database to the client, and cache a copy of the data to the memcached (the memcached client is not responsible and needs to be explicitly implemented by the program), and the path is operated as ①②④⑤⑦⑥.

3. Update the data in memcached every time you update the database to ensure consistency

4. When the memory space allocated to memcached is used up, the LRU (Least Recently Used, least recently used) policy plus expiration policy is used to replace the invalidated data first, and then replace the recently unused data.

Characteristics of 2.Memcached

The protocol is simple:

Based on the text line protocol, the data can be accessed on the memcached server directly through telnet; based on libevent event processing; all data is stored in memory, access to data is faster than hard disk, when the memory is full, the unused cache is automatically deleted by LRU algorithm, but the problem of data disaster recovery is not considered, restart the service, all data will be lost.

Distributed: each memcached server does not communicate with each other, accesses data independently and does not share any information. The server does not have distributed capabilities, and distributed deployment depends on the memcache client. 5. Install Memcache

The installation of Memcache is divided into two processes: the installation of memcache server and the installation of memcached client. The so-called server-side installation is to install Memcache on the server (usually the linux system) to realize the storage of data.

The so-called client installation means that php (or other programs, Memcache and other good api interfaces provide) to use the functions provided by the server-side Memcache, which requires php to add extensions.

From this we can see that the construction of Memcache needs the help of LAMP or LNMP, and this blog uses LNMP structure.

Installation environment:

(1) install nginx server

Download the Nginx package

For a detailed description of the installation of Nginx, please refer to Nginx depth Optimization (2), so there is no more explanation here!

[root@Nginx ~] # useradd-M-s / sbin/nologin nginx [root@Nginx ~] # yum-y install openssl-devel [root@Nginx ~] # tar zxf pcre-8.39.tar.gz-C / usr/src [root@Nginx ~] # tar zxf zlib-1.2.8.tar.gz-C / usr/src [root@Nginx ~] # tar zxf nginx-1.14.0.tar.gz-C / usr/src [root@Nginx ~] # cd / usr/src/ Nginx-1.14.0/ [root@Nginx nginx-1.14.0] #. / configure-- prefix=/usr/local/nginx\-- user=nginx-- group=nginx-- with-http_dav_module\-- with-http_stub_status_module-- with-http_addition_module\-- with-http_sub_module-- with-http_flv_module-- with-http_mp4_module\-- with-pcre=/usr/src/pcre-8.39-- with-zlib= / usr/src/zlib-1.2.8\-- with-http_ssl_module-- with-http_gzip_static_module & & make & & make install [root@Nginx ~] # ln-s / usr/local/nginx/sbin/nginx / usr/local/sbin [root@Nginx ~] # nginx [root@Nginx ~] # netstat-anpt | grep 80tcp 00 0.0.0.0 with-http_gzip_static_module 80 0.0.0.0 make * LISTEN 8460/nginx: master (2) install PHP server

Download the PHP package

For a detailed description of the installation of PHP, you can refer to the deployment LAMP static and dynamic separation and deployment Discuz forum without more explanation here!

[root@PHP] # yum-y install openssl-devel libxml2-devel bzip2-devel libcurl-devel [root@PHP] # tar zxf libmcrypt-2.5.7.tar.gz-C / usr/src [root@PHP] # cd / usr/src/libmcrypt-2.5.7/ [root@PHP libmcrypt-2.5.7] #. / configure-- prefix=/usr/local/libmcrypt & & make & & make install [root@PHP] # tar zxf php-5.6.27. Tar.gz-C / usr/src [root@PHP ~] # cd / usr/src/php-5.6.27/ [root@PHP php-5.6.27]. / configure-- prefix=/usr/local/php\-- with-mysql=mysqlnd-- with-pdo-mysql=mysqlnd-- with-mysqli=mysqlnd\-- with-openssl-- enable-fpm-- enable-sockets-- enable-sysvshm\-- enable-mbstring-- with-freetype-dir-- with-jpeg-dir-- with-png -dir\-with-libxml-dir=/usr-- enable-xml-- with-mhash-- with-zlib\-- with-mcrypt=/usr/local/libmcrypt-- with-config-file-path=/etc\-- with-config-file-scan-dir=/etc/php.d-- with-bz2\-- enable-maintainer-zts & & make & & make install [root@PHP ~] # cp / usr/src/php-5.6.27/php.ini-production / etc/php.ini [root@PHP ~] # cp / usr/src/php-5.6.27/sapi/fpm/init.d.php-fpm / etc/init.d/php-fpm [root@PHP ~] # chmod + x / etc/init.d/php-fpm [root@PHP ~] # chkconfig-add php-fpm [root@PHP ~] # cd / usr/local/php/etc/ [root@PHP etc] # cp php-fpm.conf.default php-fpm.conf [root@PHP etc] # sed-I's # Pid = run/php-fpm.pid#pid = run/php-fpm.pid#g' php-fpm.conf [root@PHP etc] # sed-I 's/listen = 127.0.0.1:9000/listen = 0.0.0.0V 9000 php-fpm.conf [root@PHP etc] # sed-I' s/pm.max_children = 5/pm.max_children = 50 GB php-fpm.conf [root@PHP etc] # sed-I's / Pm.start_servers = 2/pm.start_servers = 5gamma g 'php-fpm.conf [root@PHP etc] # sed-I' s/pm.min_spare_servers = 1/pm.min_spare_servers = 5gamma g 'php-fpm.conf [root@PHP etc] # sed-I' s/pm.max_spare_servers = 3/pm.max_spare_servers = 35gamma g 'php-fpm.conf [root@PHP ~] # systemctl start php-fpm [root@PHP ~] # netstat-anpt | grep 9000 tcp 00 0.0.0.0 tcp 9000 0.0.0.0 LISTEN 118146/php-fpm: mas (3) install MySQL database

Provide one-click installation of MySQL software and scripts

[root@Mysql ~] # lsanaconda-ks.cfg initial-setup-ks.cfg mysql-5.7.22-linux-glibc2.12-x86_64.tar.gz mysql.sh [root@Mysql ~] # sh mysql.sh Starting MySQL.. SUCCESS! [root@Mysql ~] # mysql-u root-p123mysql > create database testdb1;mysql > use testdb1;mysql > grant all on *. * to lzj@'192.168.1.%' identified by '123456 create table test1 create table test1 (id int not null auto_increment,name varchar (20) default null,primary key (id)) engine=innodb auto_increment=1 default charset=utf8;mysql > insert into test1 (name) values (' tom1'), ('tom2'), (' tom3'), ('tom4'), (' tom5'); mysql > select * from test1 +-to test the connectivity of PHP to Nginx and MySQL.

Operations of the Nginx server:

[root@Nginx ~] # vim / usr/local/nginx/conf/nginx.conf location / {root html; index index.php index.html index.htm;} location ~\ .php$ {root / var/www/html; fastcgi_pass 192.168.1.6 virtual 9000; fastcgi_index index.php Fastcgi_param SCRIPT_FILENAME / scripts$fastcgi_script_name; include fastcgi.conf;} [root@Nginx ~] # nginx-tnginx: the configuration file / usr/local/nginx/conf/nginx.conf syntax is oknginx: configuration file / usr/local/nginx/conf/nginx.conf test is successful [root@Nginx ~] # nginx-s reload

Operations of the PHP server:

[root@PHP ~] # mkdir-p / var/www/html [root@PHP ~] # vim / var/www/html/test.php [root@PHP ~] # vim / var/www/html/test1.php

Client access test:

Client access test to make sure there are no problems!

(5) install Memcache server

Download the Memcache package

[root@Memcache ~] # tar zxf libevent-2.0.22-stable.tar.gz-C / usr/src [root@Memcache ~] # cd / usr/src/libevent-2.0.22-stable/ [root@Memcache libevent-2.0.22-stable] #. / configure & & make & & make install// install Memcache dependency package [root@Memcache ~] # tar zxf memcached-1.4.33.tar.gz-C / usr/src [root@Memcache ~] # Cd / usr/src/memcached-1.4.33/ [root@Memcache memcached-1.4.33] #. / configure-- prefix=/usr/local/memcached\-- with-libevent=/usr/local/ & & make & & make install [root@Memcache ~] # ln-s / usr/local/memcached/bin/memcached / usr/local/bin [root@Memcache ~] # memcached- d-m 2048-l 192.168.1.7-p 11211-c 10240-P / usr/local/ Memcached/memcached.pid-u root [root@Memcache ~] # netstat-anpt | grep 11211tcp 0 0 192.168.1.7 Vera 11211 0.0.0.0 netstat * LISTEN 10886/memcached

The common parameters for starting Memcache are described as follows:

-d: start a daemon process;-m: the amount of memory allocated to Memcache (in MB). Default is 64MB: IP address for listening; (default: INADDR_ANY, all addresses)-p: set the port on which Memcache's TCP listens, preferably above 1024;-u: the user running Memcache, if it is currently root, you need to use this parameter to specify the user. -c: maximum number of concurrent connections running. Default is 1024: pid file path to save Memcache;-M: error is returned when memory is exhausted, rather than delete items;-f: block size growth factor, default is 1.25 Taci n: minimum allocation space, key+value+flags defaults to 48%: show help. (6) PHP server installs Memcache client [root@PHP ~] # scp 192.168.1.7:/root/memcache-3.0.8.tgz. [root@PHP ~] # tar zxf memcache-3.0.8.tgz-C / usr/src [root@PHP ~] # cd / usr/src/memcache-3.0.8/ [root@PHP memcache-3.0.8] # / usr/local/php/bin/phpize// generates configure file / / if you make an error in executing the above command Then the autoconf package [root@PHP memcache-3.0.8] #. / configure-- enable-memcache\-- with-php-config=/usr/local/php/bin/php-config & & make & & make install// that needs to execute the "yun-y install autoconf" installation prompt will display the path where the memcache.so is stored [root@PHP ~] # echo "extension = / usr/local/php/lib/php/extensions/no-debug-zts-20131226/memcache.so" > > / etc/php.ini// fill in the path where the memcache.so module is stored in the PHP main configuration file [root@PHP ~] # systemctl restart php-fpm [root@PHP ~] # vim / var/www/html/test2.php// this test script shows the version of memcached / / and inserts a key value pair "test=123" with a cache time of 600 seconds. Its ID is "key"

The customer segment visits are as follows:

Install the telnet tool test on the PHP server

[root@PHP ~] # yum-y install telnet [root@PHP ~] # telnet 192.168.1.7 11211 / / Log in to port 11211 of memcached Trying 192.168.1.7...Connected to 192.168.1.7.Escape character is'^] .get key / / query the key-value pair whose ID is "key" You can see the "test=123" VALUE key 1 66O:8: "stdClass": 2: {Sv8: "str_attr" written by our test script Test 4: "test"; 8: "int_attr"; iEND// 123;} END// needs to increase / / or revisit the save time value of the key-value pair inserted in the test2.php file to avoid cache invalidation. [root@PHP ~] # (7) use memcache to implement session sharing

Do the following on the PHP server:

[root@PHP ~] # vim / etc/php.ini / / write the PHP main configuration file, and add the following at the end: session.save_handler = memcachesession.save_path = "tcp://192.168.1.7:11211?persistent=1&weight=1&timeout=1&retry_interval=15" [root@PHP ~] # systemctl restart php-fpm [root@PHP ~] # vim / var/www/html/test3.php

What is written in the PHP configuration file is explained as follows:

Session.save_handler: set the storage mode of session to memcache; access session data by file by default; session.save_path: set the location where session is stored; use commas when using multiple memcached server "," separated, you can take additional parameters "persistent", "weight", "timeout", "retry_interval", etc.; something like this: "tcp://host:port?persistent=1&weight=2,tcp://host2:port2"

The client access is as follows:

[root@PHP ~] # telnet 192.168.1.7 11211Trying 192.168.1.7...Connected to 192.168.1.7.Escape character is'^] '.get a8aujbnie16p29rj4pf9ltfjp3 / / use the server_id number that appears on the web page to get its corresponding value VALUE a8aujbnie16p29rj4pf9ltfjp3 0 26session_time | iRover 1576329213 quitConnection closed by foreign host. [root@PHP ~] # (8) Test the Memcache cache database

Because when you create the database in step 3, the values used for the test are already written to the database, so the next step is to create the test script!

Operations of the PHP server:

[root@PHP ~] # vim / var/www/html/test4.php / / areas that often need to be modified have been marked! And this test script is also available in Memcache software.

The client tests:

Visit for the first time

Second access (after refresh)

Before the cache expires, you can also use get to get the corresponding value of the cache:

[root@PHP ~] # telnet 192.168.1.7 11211Trying 192.168.1.7...Connected to 192.168.1.7.Escape character is'^]. Get d8c961e9895ba4b463841924dbcefc2bVALUE d8c961e9895ba4b463841924dbcefc2b 0 251a:5: {i11211Trying 192.168.1.7...Connected to 192.168.1.7.Escape character is 0 251a:5: {SDV 2: "id"; SSV 1: "1"; SRAV 4: "name"; SRAV 4: "tom1";} i1MAV: 2: "id"; SDR 1: "2"; SSR 4: "name" SRV 4: "tom2";} iRAPR 2: "id"; SRAV 1: "3"; SARO 4: "name"; SRAR 4: "tom3";} iRanger 3: 2: "id"; SRAV 1: "4"; SRAV 4: "name"; SRAV 4: "tom4";} iRanger 4: "id"; SSR 1: "5"; Srig 4: "name" SRV 4: "tom5";}} END

-this is the end of this article. Thank you for watching-

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.