Shulou(Shulou.com)05/31 Report--

How to analyze the application of memcached and compatible programs? in view of this problem, this article introduces the corresponding analysis and solutions in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Here are some cases of mixi and topics on practical applications, as well as some programs compatible with memcached.

Mixi case study

Mixi uses memcached in the early stages of providing services. With the rapid increase of website visits, simply adding slave to the database can no longer meet the needs, so memcached is introduced. In addition, we also verify from the aspect of increasing scalability and prove that the speed and stability of memcached can meet the needs. Now, memcached has become a very important part of mixi services.

Figure 1 current system components

Server configuration and quantity

Mixi uses many servers, such as database server, application server, picture server, reverse proxy server and so on. Memcached alone has nearly 200 servers running. A typical configuration of a memcached server is as follows:

CPU:Intel Pentium 4 2.8GHz

Memory: 4GB

Hard disk: 146GB SCSI

Operating system: Linux (x86x64)

These servers have previously been used for database servers, and so on. With the improvement of CPU performance and the decline of memory price, we actively replace database servers, application servers and so on with servers with stronger performance and more memory. In this way, the sharp increase in the number of servers used by mixi as a whole can be suppressed and the management cost can be reduced. Since the memcached server takes up almost no CPU, the replaced server is used as the memcached server.

Memcached process

Each memcached server starts only one memcached process. The memory allocated to memcached is 3GB, and the startup parameters are as follows:

/ usr/bin/memcached-p 11211-u nobody-m 3000-c 30720

Due to the use of x86 / 64 operating system, memory above 2GB can be allocated. In a 32-bit operating system, each process can only use 2GB memory at most. It has also been considered to start multiple processes that allocate memory below 2GB, but the number of TCP connections on one server will multiply and management will become complex, so mixi uses a unified 64-bit operating system.

In addition, although the server's memory is 4GB, only 3GB is allocated because the memory allocation exceeds this value, which may lead to memory swapping (swap). In the second series, Nakasaka explained memcached's memory storage "slab allocator". At that time, he said that the specified amount of memory allocation when memcached was started was the amount of memory used by memcached to save data, excluding the memory occupied by "slab allocator" itself and the management space set to save data. Therefore, it should be noted that the actual amount of memory allocated by the memcached process is larger than the specified capacity.

Most of the data saved by mixi in memcached is relatively small. In this way, the size of the process is much larger than the specified capacity. Therefore, we repeatedly change the amount of memory allocation to verify that the size of 3GB does not cause swap, which is the value applied now.

Memcached usage and client

Today, mixi's service uses about 200 memcached servers as one pool. The capacity of each server is 3GB, so the whole body has a huge in-memory database of nearly 600GB. The client library uses the Cache::Memcached::Fast of the car mentioned many times in this series to interact with the server. Of course, the cached distributed algorithm uses the Consistent Hashing algorithm introduced for the fourth time.

Cache::Memcached::Fast-search.cpan.org

The use of memcached on the application layer is decided and implemented by the engineer who develops the application. However, in order to prevent wheel reengineering and prevent the lessons learned from Cache::Memcached::Fast from happening again, we provide Cache::Memcached::Fast 's wrap module and use it.

Maintain connectivity through Cache::Memcached::Fast

In the case of Cache::Memcached, the connection to memcached (file handle) is saved in the class variable in the Cache::Memcached package. In environments such as mod_perl and FastCGI, variables in the package are not restarted at any time as CGI does, but are persisted throughout the process. The result is that the connection with the memcached will not be disconnected, which reduces the overhead when the TCP connection is established, and at the same time, it can prevent the exhaustion of TCP port resources caused by repeated TCP connection and disconnection in a short time.

However, Cache::Memcached::Fast does not have this capability, so you need to keep the Cache::Memcached::Fast object in the class variable outside the module to ensure a persistent connection.

Package Gihyo::Memcached;use strict;use warnings;use Cache::Memcached::Fast;my @ server_list = qw/192.168.1.1:11211 192.168.1.1 qw/192.168.1.1:11211 11211fast fast my $fast; # # is used to hold objects sub new {my $self = bless {}, shift; if (! $fast) {$fast = Cache::Memcached::Fast- > new ({servers = >\ @ server_list});} $self- > {_ fast} = $fast Return $self;} sub get {my $self = shift; $self- > {_ fast}-> get (@ _);}

In the above example, the Cache::Memcached::Fast object is saved to the class variable $fast.

Public data processing and rehash

Cached data, setting information, and other data shared by all users, such as news on mixi's home page, will take up many pages and be visited a lot. Under such conditions, access can easily be concentrated on a memcached server. The access set itself is not a problem, but once the failure of the server in the access set causes memcached to fail to connect, it will cause a huge problem.

As mentioned in the fourth series, Cache::Memcached has the rehash function, that is, if it cannot connect to the server that holds the data, it will calculate the hash value again and connect to other servers.

However, Cache::Memcached::Fast does not have this feature. However, it can no longer connect to the server for a short period of time when it fails to connect to the server.

My $fast = Cache::Memcached::Fast- > new ({max_failures = > 3, failure_timeout = > 1})

If the max_failures connection fails within failure_timeout seconds, the memcached server will no longer be connected. Our setting is more than 3 times a second.

In addition, mixi also sets naming rules for the key names of cached data shared by all users, and the data that meets the naming rules is automatically saved to multiple memcached servers, from which only one server is selected. After you create this function library, you can make the memcached server failure have no other impact.

Experience in memcached application

So far, the internal structure and function library of memcached are introduced, and then some other application experiences are introduced.

Start via daemontools

Normally, memcached runs fairly steadily, but the latest version 1.2.5 used by mixi today has experienced several memcached process deaths. The architecture ensures that even if there are several memcached failures, the service will not be affected, but for the server where the memcached process is dead, as long as you restart memcached, it can run normally, so the method of monitoring the memcached process and starting it automatically is adopted. So daemontools is used.

Daemontools is a set of UNIX service management tools developed by DJB, the author of qmail, in which a program called supervise can be used for service startup, service restart, etc.

Daemontools

The installation of daemontools is not covered here. Mixi uses the following run script to start memcached.

#! / bin/shif [- f / etc/sysconfig/memcached]; then. / etc/sysconfig/memcachedfiexec 2 > & 1exec / usr/bin/memcached-p $PORT-u $USER-m $CACHESIZE-c $MAXCONN $OPTIONS Monitoring

Mixi uses open source monitoring software called "nagios" to monitor memcached.

Nagios: Home

Plug-ins can be simply developed in nagios to monitor memcached's get, add, and other actions in detail. However, mixi confirms the running status of memcached only through the stats command.

Define command {command_name check_memcachedcommand_line $USER1 $/ check_tcp-H $HOSTADDRESS$-p 11211-t 5-E-s' stats\ r\ nquit\ r\ n'- e 'uptime'-M crit}

In addition, mixi converts the results of the stats directory into graphics through rrdtool, monitors performance, and reports daily memory usage, which is shared with developers via email.

Performance of memcached

As described in the serialization, the performance of memcached is excellent. Let's take a look at the actual case of mixi. The diagram presented here is the most visited memcached server used by the service.

Figure 2 number of requests

Figure 3 Traffic

Figure 4 number of TCP connections

From top to bottom are the number of requests, traffic and TCP connections. The maximum number of requests is 15000qps, and the traffic reaches 400Mbps. At this time, the number of connections has exceeded 10000. The server has no special hardware, just the ordinary memcached server introduced at the beginning. The CPU utilization at this time is:

Figure 5 CPU utilization

It can be seen that there is still a part of idle. As a result, memcached has very high performance and can be used as a place where Web application developers can safely save temporary data or cache data.

Compatible application

The implementation and protocol of memcached are very simple, so there are many implementations compatible with memcached. Some powerful extensions can write memcached's in-memory data to disk to achieve data persistence and redundancy. As described in the series for the third time, the storage layer of the future memcached will become pluggable, gradually supporting these functions.

Here are several memcached-compatible applications.

Repcached

Patch that provides replication (replication) functionality for memcached.

Flared

Save to QDBM. At the same time, it realizes the functions of asynchronous replication and fail over.

Memcachedb

Save to BerkleyDB. Message queue is also implemented.

Tokyo Tyrant

Store the data in Tokyo Cabinet. It is not only compatible with the memcached protocol, but also can be accessed through HTTP.

Tokyo Tyrant case

Mixi uses Tokyo Tyrant from the above compatible applications. Tokyo Tyrant is the network interface of Tokyo Cabinet DBM developed by Pinglin. It has its own protocol, but it also has a memcached compatible protocol, and it can also exchange data through HTTP. Tokyo Cabinet is an implementation of writing data to disk, but it is quite fast.

Instead of using Tokyo Tyrant as a cache server, mixi uses it as a DBMS that holds combinations of key-value pairs. It is mainly used as a database to store the last time the user accessed it. It is related to almost all mixi services, and the data is updated every time the user visits the page, so the load is quite high. The processing of MySQL is very cumbersome, and it is possible to lose data by using memcached alone to save data, so Tokyo Tyrant is introduced. But there is no need to redevelop the client, just use Cache::Memcached::Fast intact, which is one of the advantages.

Mixi Engineers' Blog-Tokyo Tyrant temperature tolerance high load DB temperature tolerance

Mixi Engineers' Blog-Tokyo (Cabinet | Tyrant) new machine

The answers to the questions on how to analyze memcached applications and compatible programs are shared here. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.