Shulou(Shulou.com)06/03 Report--

A brief introduction to MemCache

MemCache is a free, open source, high-performance, distributed distributed memory object caching system, which is used for dynamic Web applications to reduce the load of the database. It reduces the number of times to read the database by caching data and objects in memory, thus improving the speed of website access. MemCaChe is a HashMap that stores key-value pairs. Key-value storage is used in memory for any data (such as strings, objects, etc.). The data can come from database calls, API calls, or the results of page rendering. The design concept of MemCache is small and powerful, and its simple design promotes rapid deployment, easy development and solving many problems facing large-scale data cache. the open API makes MemCache used in most popular programming languages such as Java, C/C++/C#, Perl, Python, PHP, Ruby and so on.

In addition, why are there two names, Memcache and memcached? In fact, Memcache is the name of the project (also the name of its client), and memcached is its server-side main program file name.

Memcached is a key / value system, the system is much simpler than MySQL, although MySQL also has cache, but database SQL parsing will consume performance, query is slower than memcached, in addition, MySQL cache design is more complex, because to consider transactions, logs, storage engine and other modules, its performance is not as good as memcached.

Memcached only does one thing, simple and efficient, and is better than MySQL on cache, which should be easy to understand.

As a high-speed distributed cache server, memcached has the following characteristics: simple protocol; event processing based on libevent; built-in memory storage mode; distributed memcached that does not communicate with each other

1. Agreement

Memcached's server-client communication does not use complex formats such as XML, but uses a simple text-line-based protocol.

Therefore, data can also be saved and obtained on memcached through telnet.

2. Event handling

Libevent is a library that encapsulates event handling functions such as epoll of Linux and kqueue of BSD operating systems into a unified interface. O (1) performance can be achieved even if the number of connections to the server increases. Memcached uses this libevent library, so it can perform its high performance on Linux, BSD, Solaris, and other operating systems.

3. Storage mode

To improve performance, the data saved in memcached is stored in memcached's built-in memory storage space. Because the data exists only in memory, restarting memcached and restarting the operating system will cause all data to disappear. In addition, when the content capacity reaches the specified value, the unused cache is automatically deleted based on the LRU (Least Recently Used) algorithm. Memcached itself is a server designed for caching, so it doesn't think too much about the persistence of the data.

4. Distributed communication

Although memcached is a "distributed" cache server, there is no distributed function on the server side. Memcached does not communicate with each other to share information. So, how to distribute it? It all depends on the implementation of the client.

5. Application scenarios of memcached

1) the front-end cache application of the database: let it share the concurrent pressure of the data. When the data is updated, the program can tell the cache to update.

2) shared storage of session session sharing

6. Workflow in memcached application

It is an in-memory cache, which can read the data cached in memory through API. When users need to read data, they will first access the memcached cache. If there is any data in the cache, it will be directly returned to the front-end application. If not, it will be forwarded to the server at the backend. At this time, in addition to returning the data to the user, the server will update the data to the memcached cache.

If, in the actual production environment, the cache server needs to be restarted (or power is cut off), the data in the cache will be lost, and the concurrent pressure on the replacement server will increase, which may cause the introduced server to shut down as well. Unable to provide services, then our processing flow is as follows:

First stop the WEB application from the load balancer-> make the load balancer stop forwarding data to WEB-- > then start the cache server-> initialize the contents of the database to the cache server through the program-> then enable the web application-> restart the database server

7. Consistent Hash algorithm of memcached

The consistent Hash algorithm implements the Hash mapping from the Key to the cache server through a data structure called the consistent Hash ring. To put it simply, a consistent hash organizes the entire hash space into a virtual ring (this ring is called a consistent Hash ring). For example, assuming that the value space of a space hash function H is 0 ~ 2 ^ 32-1 (that is, the hash value is a 32-bit unsigned integer), the whole hash space is as follows:

Use H for a hash calculation for each server. Specifically, you can use the server's IP address or hostname as a keyword, so that each machine can determine its position on the hash ring above and arrange it clockwise. Here, we assume that the positions of the three nodes memcache are as follows:

Next, the hash value h of the data is calculated using the same algorithm, and the position of the data on the hash ring is determined. If we have data A, B, C, D, 4 objects, the position after hashing is as follows:

According to the consistent hash algorithm, data An is bound to server01, D is bound to server02, and B and C are clockwise to find the nearest service node on server03.

The hash ring scheduling method thus obtained has high fault tolerance and scalability:

Suppose server03 is down:

You can see that C and B are affected at this time, and the B and C nodes are relocated to Server01. In general, in a consistent hash algorithm, if a server is not available, the data affected is only the data between this server and the previous server in its ring space (that is, the first server encountered walking counterclockwise). The rest will not be affected.

Consider another situation, if we add a server Memcached Server 04 to the system:

At this point, A, D, and C are not affected, only B needs to be relocated to the new Server04. In general, in the consistent hashing algorithm, if you add a server, the data affected is only the data between the new server and the previous server in its ring space (that is, the first server encountered when walking counterclockwise). Nothing else will be affected.

To sum up, the consistent hashing algorithm only needs to relocate a small part of the data in the ring space for the increase or decrease of nodes, so it has good fault tolerance and scalability.

The disadvantage of consistent hash: when there are too few service nodes, it is easy to cause the problem of data skew due to the uneven division of nodes. We can solve the problem by adding virtual nodes.

More importantly, the more cache server nodes in the cluster, the smaller the impact of adding / decreasing nodes, which is easy to understand. In other words, with the increase of the size of the cluster, the probability of continuing to hit the original cached data will increase. Although there is still a small amount of data cached in the server that cannot be read, the proportion is small enough that even if the database is accessed, it will not cause fatal load pressure on the database.

Deploy LNMP dynamic and static separation & & memcache cache server

The environment is as follows:

The required source packages can be downloaded here and uploaded to each server: https://pan.baidu.com/s/1-2pS702mz41e94nBXSgUnA

Extraction code: rldk

1. Deploy Nginx service

[root@nginx /] # yum-y install openssl-devel pcre-devel # install the required dependency package [root@nginx /] # mkdir nginx # personal habit is only [root@nginx /] # cd nginx/ [root@nginx nginx] # rz # xshell-connected server Use [root@nginx nginx] # tar zxf nginx-1.14.0.tar.gz # to extract the source code package required for rz upload to the current directory [root@nginx nginx] # cd nginx-1.14.0/ [root@nginx nginx-1.14.0] # useradd-M-s / sbin/nologin www # to create a Nginx running user [root@nginx nginx-1.14.0] #. / configure-- prefix=/usr/local / nginx--user=www-- group=www & & make & & make install # compile and install [root@nginx nginx-1.14.0] # ln-s / usr/local/nginx/sbin/nginx / usr/local/sbin/ # create Command soft connection [root@nginx nginx-1.14.0] # nginx # start Service [root@nginx nginx-1.14.0] # netstat-anput | grep 80 # View Port Confirm that the service has started tcp 0 0 0. 0 master 80 0 0 0 cd * LISTEN 6827/nginx: master [root@nginx nginx-1.14.0] # cd / [root@nginx /] # vim / usr/local/nginx/conf/nginx.conf.. / / omit some contents and add the following content to the server {} field: location ~\ .php$ {root / var/www/html; # specify the web page storage path of PHP fastcgi_pass 192.168.171.133php$ 9000; # specify PHP service listening port and address fastcgi_index index.php Fastcgi_param SCRIPT_FILENAME / scripts$fastcgi_script_name; include fastcgi.conf;} [root@nginx /] # nginx-s reload # restart the service to make the configuration effective

2. Deploy PHP service

# first, you need to install the dependency package for PHP [root@php php] # yum-y install libxml2-devel openssl-devel bzip2-devel [root@php php] # tar zxf libmcrypt-2.5.7 [root@php php] # cd libmcrypt-2.5.7/ [root@php libmcrypt-2.5.7] #. / configure-- prefix=/usr/local/libmcrypt & & make & & make install [root@php libmcrypt-2.5.7] # cd. [root@php php] # Tar zxf php-5.6.27.tar.gz [root@php php] # cd php-5.6.27/ [root@php php-5.6.27] # / configure-- prefix=/usr/local/php5.6-- with-mysql=mysqlnd-- with-pdo-mysql=mysqlnd-- with-mysqli=mysqlnd-- with-openssl-- enable-fpm-- enable-sockets-- enable-sysvshm-- enable-mbstring-- with-freetype-dir-- with-jpeg-dir-- with-png-dir -- with-zlib-- with-libxml-dir=/usr-- enable-xml-- with-mhash-- with-mcrypt=/usr/local/libmcrypt-- with-config-file-path=/etc-- with-config-file-scan-dir=/etc/php.d-- with-bz2-- enable-maintainer-zts & & make & & make install# the following is to adjust the configuration file of PHP and control the start and stop of the service [root@php php-5.6.27] # cp php.ini- Production / etc/php.ini # copy the PHP configuration file provided in the source code [root@php php-5.6.27] # cp sapi/fpm/init.d.php-fpm / etc/init.d/php-fpm# copy its service control script file [root@php php-5.6.27] # chmod + x / etc/init.d/php-fpm# to grant execution permission [root@php php-5.6.27] # chkconfig- -add php-fpm # added as a system service To support systemctl management [root@php php-5.6.27] # chkconfig php-fpm on # enable # copy the default configuration file provided by php-fpm and edit it [root@php php-5.6.27] # cp / usr/local/php5.6/etc/php-fpm.conf.default / usr/local/php5.6/etc/php-fpm.conf [root@php php-5.6.27] # vim / usr/local/php5.6/etc / php-fpm.conflisten = 192.168.171.133 pm.max_children 9000 # listener address is the local IP9000 port pm.max_children = 50 # maximum number of started processes pm.start_servers = 5 # initial start processes pm.min_spare_servers = 5 # minimum idle process pm.max_spare_servers = 35 # maximum idle process # after modification Save and exit to [root@php /] # service php-fpm restart # restart PHP to make the configuration effective Gracefully shutting down php-fpm. DoneStarting php-fpm done [root@php /] # netstat-anput | grep 9000 # check whether to run tcp 00 192.168.171.133anput 9000 0.0.0.0 * LISTEN 3054/php-fpm: maste # prepare the web page test file [root@php /] # mkdir-p / var/www/html [root@php /] # cd / var/www/html/ [root@php html] # cat index.php [root@php html] # cat index1.php

At this point, you can access port 80 of the Nginx server to view the two web page files defined on the php server (when accessing the script file that connects to the database, you need to deploy the database and create users to connect to):

3. Deploy MySQL service

# A simple database can be deployed here [root@mysql /] # mkdir-p mysql [root@mysql /] # cd mysql/ [root@mysql mysql] # rz # upload required [root@mysql mysql] # sh mysql.sh # Direct sh script installation. After installation, the default password is 123Starting MySQL... SUCCESS! Mysql: [Warning] Using a password on the command line interface can be insecure. [root@mysql mysql] # netstat-anput | grep 3306 # make sure the service has started tcp6 0 0:: 3306: * LISTEN 3290/mysqld [root@mysql mysql] # mysql-u root-p # Log in to the database Enter password: mysql > create database bbs Query OK, 1 row affected (0.00 sec) mysql > grant all on bbs.* to zyz@ "192.168.171.%" identified by 'pwd@123';Query OK, 0 rows affected, 1 warning (0.00 sec)

4. Deploy Memcached service

[root@memcached /] # mkdir memcached [root@memcached /] # cd memcached/ [root@memcached memcached] # rz # upload the required source code package [root@memcached memcached] # tar zxf libevent-2.0.22-stable.tar.gz # unpack [root@memcached memcached] # cd libevent-2.0.22-stable/ [root@memcached libevent-2.0.22-stable] #. / configure & & make & & make install # compile and install [root@memcached libevent-2.0.22-stable] # cd. [root@memcached memcached] # tar zxf memcached-1.4.33.tar.gz [root@memcached memcached] # cd memcached-1.4.33/ [root@memcached memcached-1.4.33] #. / configure-- prefix=/usr/local/memcached-- with-libevent=/usr/local/ & & make & & make install [root@memcached memcached-1.4.33] # ln-s / usr/local/memcached/bin/memcached / usr/local/bin/ # command to make a soft connection [root@memcached memcached-1.4.33] # memcached- d-m 1024-l 192.168.171.132-p 11211-c 10240-P / usr/local/memcached/memcached.pid-u root# start the memcached service The above startup parameters are described as follows: the #-d option starts a daemon. #-m the amount of memory allocated to Memcache for use, in MB (default 64MB). #-l the IP address to listen to. (default: INADDR_ANY, all addresses) #-p sets the port on which Memcache's TCP listens, preferably above 1024. #-u the user running Memcache needs to use this parameter to specify the user if it is currently root. The #-c option is the maximum number of concurrent connections running, and the default is 1024. #-P sets the path of the pid file where Memcache is saved. #-M returns an error when memory is exhausted, instead of deleting the item #-f block size growth factor. The default is 1.2 blocks-n minimum allocated space. Key+value+flags defaults to 4 minutes-h display help [root@memcached memcached-1.4.33] # netstat-anput | grep 11211 # make sure that both TCP and udp are listening on tcp 0 0192.168.171.132netstat 11211 0.0.0.0netstat * LISTEN 11666/memcached udp 0192.168.171.132netstat 11211 0.0.0.0netstat * 11666/memcached

5. Deploy Memcache client (return PHP server operation)

[root@php /] # mkdir memcache [root@php /] # cd memcache/ [root@php memcache] # rz # upload the following source code package [root@php memcache] # lsmemcache-3.0.8.tgz [root@php memcache] # tar zxf memcache-3.0.8.tgz [root@php memcache] # cd memcache-3.0.8/ [root@php memcache-3.0.8] # / usr/local/php5.6/bin/phpize # # execute the command In order to generate the configure file # if you make an error in executing the above command, you need to execute the autoconf package of the "yun-y install autoconf" installation prompt. Configuring for: # successful execution will display the following lines: PHP Api Version: 20131106Zend Module Api No: 20131226Zend Extension Api No: 220131226 [root@php memcache-3.0.8] #. / configure-- enable-memcache-- with-php-config=/usr/local/php5.6/bin/php-config & & make & & make install # compile and install # after executing the above command The path where the memcache.so is stored will be displayed [root@php memcache-3.0.8] # vim / etc/php.ini # Edit this file # add the following on the last line Be careful not to directly copy your own path extension = / usr/local/php5.6/lib/php/extensions/no-debug-non-zts-20131226/memcache.so [root@php memcache-3.0.8] # service php-fpm restart # restart php to make the configuration effective Gracefully shutting down php-fpm. DoneStarting php-fpm done

Write a test file:

[root@php memcache-3.0.8] # cd / var/www/html/ [root@php html] # vim test1.php#, save and exit. This test script displays the version of memcached # and inserts a key value pair "test=123" with a cache time of 600s, and its ID is "key".

Install the Telnet command on the memcached server and log in to the cache to see if you can get its key-value pair

[root@memcached /] # yum-y install telnet # install the Telnet command [root@memcached /] # telnet 192.168.171.132 11211Trying 192.168.171.132...Connected to 192.168.171.132.Escape character is'^] ..get key # query the key-value pair whose ID is "key". You can see the "test=123" VALUE key 1 66O:8: "stdClass": 2: {sV8: "str_attr" written in our test script. SRV 4: "test"; SRV 8: "int_attr"; iRV 123;} END

# when performing the above get verification, you need to increase the save time value of the key-value pair inserted in the test1.php file.

# or revisit it to avoid cache invalidation and failure to query

At this point, LNMP separation & & memcache cache server has been basically deployed. Next, configure PHP to communicate with memcached server to save session session.

6. Use memcache to achieve session sharing (do the following on the PHP server)

[root@php /] # vim / etc/php.ini # add the following at the end: session.save_handler = memcachesession.save_path = "tcp://192.168.171.132:11211?persistent=1&weight=1&timeout=1&retry_interval=15" # the content is explained as follows: # session.save_handler: set the storage mode of session to memcache. # session data is accessed by file by default. # session.save_path: set the location of session storage # comma when using multiple memcached server, "separated, # can take additional parameters" persistent "," weight "," timeout "," retry_interval ", and so on, # such as:" tcp://host:port?persistent=1&weight=2,tcp://host2:port2 "[root@php /] # service php-fpm restart # restart the service to make the configuration effective Gracefully shutting down php-fpm. DoneStarting php-fpm done [root@php /] # vim / var/www/html/test2.php # write configuration file

The client accesses the written test2.php test file as follows:

Similarly, use the Telnet command to query the value of its session_id on the memcached server, as follows:

[root@memcached /] # telnet 192.168.171.132 11211Trying 192.168.171.132...Connected to 192.168.171.132.Escape character is'^]. Get d1n4umig3aq8okqrg7ep95t321VALUE d1n4umig3aq8okqrg7ep95t321 0 26session_time | i11211Trying 192.168.171.132...Connected to 192.168.171.132.Escape character is 1581082210 | END# you can see that the value queried is the same as that accessed by our web page, indicating that it is cached

7. Test the Memcached cache database

Create a table for testing on the MySQL database (all operations are on the MySQL database) as follows:

[root@mysql mysql] # mysql-u root-pEnter password: mysql > create database testdb; # create database mysql > use testdb; # enter the library Database changedmysql > create table test1 (id int not null auto_increment,name varchar (20) default null,primary key (id)) engine=innodb auto_increment=1 default charset=utf8; # create table mysql > insert into test1 (name) values ('aaa1'), (' aaa2'), ('aaa3'), (' aaa4'), ('aaa5') # add content to the table mysql > select * from test1 # query the contents of the table->; +-+ | id | name | +-+-+ | 1 | aaa1 | | 2 | aaa2 | 3 | aaa3 | | 4 | aaa4 | 5 | aaa5 | +-+ + 5 rows in set (0.00 sec) mysql > desc test1 # query table structure +-+ | Field | Type | Null | Key | Default | Extra | + -- + | id | int (11) | NO | PRI | NULL | auto_increment | | name | varchar (20) | YES | | NULL | | + -+-+ 2 rows in set (0.00 sec) mysql > grant select on testdb.* to test@'%' identified by 'pwd@123' # create users for testing

Write the following test file on the PHP server to test whether memcache cached data successfully:

[root@php html] # vim test3.php

The client accesses the script file for the test, and the page visited for the first time is as follows:

After the client refreshes, you will see the following page:

Before the cache query expires, you can obtain the corresponding cache data on the memcache through get, as follows (operate on the memcache server):

[root@memcached memcached] # telnet 192.168.171.132 11211Trying 192.168.171.132...Connected to 192.168.171.132.Escape character is'^]. Get d8c961e9895ba4b463841924dbcefc2bVALUE d8c961e9895ba4b463841924dbcefc2b 0 251a:5: {iRAPR 0: 2: "id"; SRAV 1: "1"; SRAV 4: "name"; SRAV 4: "aaa1";} i1MAV: 2: "id"; SSR 1: "2"; SSR 4: "name" SRV 4: "aaa2";} iRAPR 2: "id"; SRAV 1: "3"; SARO 4: "name"; SRAR 4: "aaa3";} iRanger 3: 2: "id"; SRAV 1: "4"; SRAV 4: "name"; SRAV 4: "aaa4";} iRanger 4: "id"; SSR 1: "5"; Srig 4: "name" SRV 4: "aaa5";}} END

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.