Shulou(Shulou.com)06/01 Report--

Codis is written in go language. In order to deal with the redis cluster software in high and parallel environment, the principle is that before operating on a redis key, the key is assigned to a slot of different redis through the crc32 algorithm to realize the function of concurrent reading and writing. And it can call redis-sentinel through zookeeper to realize the failover function. Now the latest version is 3.2.1, which is developed by redis3.2.9.

Advantages: high concurrent reading and writing, high data consistency.

Disadvantages: there is a large loss of performance, failover can not guarantee the loss of key, unable to separate read and write.

Architecture introduction

1. The software you need to use are:

Codis3.2.1

Description: codis cluster suite, which contains redis-related programs, and cluster-specific programs, main function program parsing:

Codis-server: belongs to the optimized version of redis-server and is developed based on the branch of redis-3.2.9. Additional data structures have been added to support slot-related operations and data migration instructions.

Codis-proxy: a client-connected Redis proxy service that implements the Redis protocol. Except for some commands that are not supported (for example: keys *, flush), there is no difference between the presentation and the native Redis (like Twemproxy).

Redis-sentinel: can realize the monitoring, notification and automatic failover of Redis. If Master does not work, the failover process is automatically started, one of the Slave is promoted to Master, and the other Slave resets the new Master service. The configuration of Sentinel is controlled by codis-dashboard and zookeeper together and does not need to be filled in manually.

Codis-dashboard: a cluster management tool that supports the addition, deletion and migration of codis-proxy and codis-server. When the state of the cluster changes, codis-dashboard maintains the consistency of the state of all codis-proxy under the cluster.

Codis-fe: cluster web management interface.

Go1.9.1

Description: codis dependent language pack

Jdk1.8

Description: zookeeper dependent language pack

Zookeeper-3.4.11

Description: used to store data configuration routing table. Zookeeper is abbreviated to zk. In a production environment, the more zk deployments, the higher its reliability. Because the zk cluster takes more than half of the number of outages to bring down the entire cluster, odd zk is better.

two。 The logical architecture is as follows:

Access layer: the access mode can be similar to the vip mode of keepalived cluster, or call the jodis control through java code and then connect to the zookeeper cluster, then find the available proxy side, and then connect and call different codis-proxy addresses to achieve highly available LVS and HA functions.

Proxy layer: in the middle layer, codis-proxy and zookeeper handle the data direction and distribution, and through the crc32 algorithm, the key is evenly distributed in a certain slot of different redis. To achieve striping similar to raid0, in the old version of codis, slot needs to be assigned manually, after codis3.2, as long as a button slot will be automatically assigned, quite convenient, but it can also be manually assigned, need to call another codis-admin command.

Data layer: finally, codis-proxy stores the data on the real redis-server master server. Because the author of codis, Huang Dongxu, pays great attention to data consistency and does not allow data inconsistency caused by data delay, the architecture does not consider the separation of master and slave read and write from the very beginning. The slave server is only used as a redundant architecture for failover, and the status of each service is monitored by codis-dashboard, and then the failover function is realized by rewriting zookeeper data and calling redis-sentinel.

3. Because of the limited machines, the architecture of the deployment is as follows:

Zookeeper Cluster:

10.0.2.5:2181

10.0.2.6:2181

10.0.2.7:2181

Codis-config and codis-dashboard:

10.0.2.6:18087

10.0.2.6:8090

Codis-proxy:

10.0.2.5:19000

10.0.2.7:19000

Codis-server:

10.0.2.5 VR 6379 (master), 10.0.2.5 VR 6380 (slave)

10.0.2.6Plus 6379 (master), 10.0.2.6purl 6380 (slave)

10.0.2.7Plus 6379 (master), 10.0.2.7purl 6380 (slave)

Installation and deployment

1. Download program code

1) download the golang language package

According to the normal way is to × ×, but the domestic address has also been released, because the codis3.2 requires at least 1.7 or more version, then simply under the latest version.

Https://studygolang.com/dl/golang/go1.9.1.linux-amd64.tar.gz

2) download the java language package

The download address of Java is changing all the time, so you'd better go up and watch it and download it.

Http://download.oracle.com/otn-pub/java/jdk/8u151-b12/e758a0de34e24606bca991d704f6dcbf/jdk-8u151-linux-x64.tar.gz?AuthParam=1513326216_bcf60226458d67751e1d8d1bbe6689b4

3) download the zookeeper program

It's just a package. It doesn't need to be compiled. It's convenient.

Http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.4.11/zookeeper-3.4.11.tar.gz

4) download codis3.2.1

It's just a package. It doesn't need to be compiled. It's convenient.

Https://github.com/CodisLabs/codis/releases/download/3.2.1/codis3.2.1-go1.7.6-linux.tar.gz

two。 Installation program

1) install java

# Unzip the package tar xf jdk-8u144-linux-x64.tar.gz# and move to the specified directory mv jdk1.8.0_144/ / usr/local/# to enter the specified directory And create program soft link cd / usr/local/ln-sf jdk1.8.0_144/ jdk# create environment variable file echo "export JAVA_HOME=/usr/local/jdkexport JRE_HOME=/usr/local/jdk/jreexport CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/libexport PATH=$PATH:$JAVA_HOME/bin" > / etc/profile.d/java.sh# overload environment variable source / etc/profile# test Check whether the installation of java-versionjava version "1.8.0mm 144" Java (TM) SE Runtime Environment (build 1.8.0_144-b01) Java HotSpot (TM) 64-Bit Server VM (build 25.144-b01) is complete. Mixed mode)

Installation completed

2) install golang

# Unzip the package tar xf go1.9.1.linux-amd64.tar.gz# to the specified directory mv go/ usr/local/# and soft connect the commands in the package to the system default command directory ln-sf / usr/local/go/bin/* / usr/bin/# test to check whether the installation of go versiongo version go1.9.1 linux/amd64 is complete

Installation completed

3) install zookeeper

# unzip the package tar xf zookeeper-3.4.11.tar.gz# and move to the specified directory mv zookeeper-3.4.11/ usr/local/# to enter the specified directory, and create a program soft connection cd / usr/local/ln-sf zookeeper-3.4.11/ zookeeper

Installation is complete, waiting for configuration.

4) install codis

# unzip the package tar xf codis3.2.1-go1.7.6-linux.tar.gz# and move to the specified directory mv codis3.2.1-go1.7.6-linux/ usr/local/# to enter the specified directory, and create a program soft connection cd / usr/local/ln-sf codis3.2.1-go1.7.6-linux/ codis

Installation is complete, waiting for configuration, because we all use binary packages, as long as the dependent package has a normal installation, it will not report an error and can be used directly, so the installation is very simple.

3. Configuration program

1) configure the zookeeper,3 station in this way.

# set hosts jump rules. If you don't, you can't start echo "10.0.2.5 zookeeper-node110.0.2.6 zookeeper-node210.0.2.7 zookeeper-node3" > > / etc/hosts# creator directory mkdir-p / data/zookeeper# to create a configuration file. There is a template in the folder. If you are interested, you can take a look at the vim / usr/local/zookeeper/conf/zoo.cfg# maximum number of connections setting (single ip limit). Note: the default is 60 and there is no limit if set to 0. The length of a cycle (tick) of maxClientCnxns=500# (in milliseconds). The maximum number of tick consumed in tickTime=28800# initialization synchronization phase. Note: the default value initLimit=10# can be used to wait for the maximum number of tick intervals. Note: you can use the default value syncLimit=5# data storage directory, just create that. Note: do not put the dataDir=/data/zookeeper/# communication port in the / tmp directory. Note: the default value clientPort=2181server.1=zookeeper-node1:2888:3888server.2=zookeeper-node2:2888:3888server.3=zookeeper-node3:2888:3888 is available

To generate ID, it should be noted here that myid corresponds to the server.ID of zoo.cfg. For example, the myid corresponding to zookeeper-node2 should be 2. If it is not set up according to the regulations, the zookeeper cluster will not start.

Echo "1" > / data/zookeeper/myid

=

For example, the 10.0.2.7 server in zookeeper-node3 should be

Echo "3" > / data/zookeeper/myid

=

# Special instructions on the last three lines of zoo.cfg

Description: server.A=B:C:D: where An is a number indicating which server this is, B is the ip address of this server, and C represents the port where this server exchanges information with the Leader servers in the cluster D means that in case the Leader server in the cluster dies, a port is needed for re-election to select a new Leader, and this port is the port through which the servers communicate with each other during the election.

# finally start, because the server of zookeeper is ordered, it is best to start it sequentially, first start server.1 and then start server2, and finally start server.3 so / usr/local/zookeeper/bin/zkServer.sh start# to check the status, there will be the difference between follower and leader. They will choose who is leader/usr/local/zookeeper/bin/zkServer.sh statusZooKeeper JMX enabled by defaultUsing config: / usr/local/zookeeper/bin/../conf/zoo.cfgMode: follower.

Configuration and startup completed.

2) configure the codis-server,3 station in this way.

Note: codis-server is the redis-server program, which belongs to the optimized version of codis and is used with codis clusters.

So it is to configure a master-slave structure of redis, and the actual production environment should not be put together.

# create redis data directory, configuration file directory, and log directory mkdir-p / data/redis/data/config/mkdir-p / data/redis/data/logs/# to create configuration files for the main library, and only configure these files for the time being. Other default vim / data/redis/data/config/redis_6379.conf# allows the background to run daemonize yes# to set the port, preferably a non-default port port 637 binds to log in to IP, security considerations It is better to name the private network bind * # and specify the PID path of the current redis to distinguish multiple redispidfile "/ data/redis/data/config/redis_6379.pid" # names and specify the current redis log file path logfile "/ data/redis/data/logs/redis_6379.log" # specify the RDB file name to back up the data to the hard disk and distinguish between different redis Trigger snapshot function dbfilename "dump_6379.rdb" # when memory exceeds 45% of available memory # specify the root directory of the current redis, which is used to store the authentication key of the RDB/AOF file dir "/ data/redis/data" # the current redis. The redis runs very fast, and this password must be strong enough. # all redis-server authentication passwords related to the codis-proxy cluster must be consistent with requirepass "123" # current redis maximum capacity limit It is recommended to set it within 45% of the available memory, and the maximum can be set to 95% of the available memory of the system. # config set maxmemory can be used to modify online, but restart fails, so you need to use config rewrite command to refresh the configuration file. # Note, if you use codis cluster, you must configure the capacity limit, otherwise you can not start maxmemory 100000kb#LRU. There are four strategies, depending on the situation, choose maxmemory-policy allkeys-lru# if you fail over. No matter the master and slave nodes need to fill in the password and keep consistent masterauth "123" # to create the slave library configuration file, only configure these for the time being. The other default vim / data/redis/data/config/redis_6380.conf# allows background daemonize yes# to set the port, preferably a non-default port port 638 binds to login IP. For security reasons, it is best to name the private network bind * # and specify the PID path of the current redis. Used to distinguish multiple redispidfile "/ data/redis/data/config/redis_6380.pid" # names and specify the current redis log file path logfile "/ data/redis/data/logs/redis_6380.log" # specify the RDB file name to back up data to the hard disk and distinguish between different redis, trigger the snapshot function dbfilename "dump_6380.rdb" # to specify the root directory of the current redis when the memory is more than 45% of the available memory The authentication key used to store the RDB/AOF file dir "/ data/redis/data" # current redis, redis runs very fast, and this password must be strong enough # all redis-server authentication passwords related to codis-proxy clusters must be consistent with requirepass "123" # the maximum capacity limit of the current redis is recommended to be set to within 45% of the available memory, and the maximum can be set to 95% of the system available memory # can be modified online with config set maxmemory But the restart fails, you need to use the config rewrite command to refresh the configuration file # Note, if you use codis cluster, you must configure the capacity limit, otherwise you can't start maxmemory 100000kb#LRU. There are four strategies, choose maxmemory-policy allkeys-lru# if you fail over depending on the situation. Both master and slave nodes need to fill in the password and keep consistent masterauth "123" # configure master node information slaveof 10.0.2.5 6379

Except for the different port number, the file name is different. In fact, the slave library configuration only adds the last line and specifies the address of the master library.

# then you can start. I said at the beginning that codis-server is redis-server/usr/local/codis/codis-server / data/redis/data/config/redis_6379.conf/usr/local/codis/codis-server / data/redis/data/config/redis_6380.conf# verify ss-ntplu | grep codis-servertcp LISTEN 0128 *: 6379 *: * users: ("codis-server", pid=2192 Fd=4)) tcp LISTEN 0 128 *: 6380 *: * users: (("codis-server", pid=2197,fd=4))

Start in the same way as redis-server, you can start it by specifying a configuration file. This is configured and started successfully.

3) configure the redis-sentinel,3 station in this way.

Correctly, redis-sentinel needs to configure the master-slave architecture to take effect, but it is not the same in the codis cluster, because its configuration is maintained by zookeeper, so the redis-sentinel used by codis here only needs to configure some basic configurations.

# We put the configuration in the configuration file directory vim / data/redis/data/config/sentinel.confbind 0.0.0.0protected-mode noport 26379dir "/ data/redis/data" pidfile "/ data/redis/data/config/sentinel_26379.pid" logfile "/ data/redis/data/logs/sentinel_26379.log" daemonize yes# of the redis data directory, and then we can start / usr/local/codis/redis-sentinel / data/redis/data/config/sentinel.conf# Verify / usr/local/codis/redis-cli-p 26379-c info Sentinel# Sentinelsentinel_masters:3sentinel_tilt:0sentinel_running_scripts:0sentinel_scripts_queue_length:0sentinel_simulate_failure_flags:0master0:name=codis-test1-3 Status=ok,address=10.0.2.7:6380,slaves=1,sentinels=3master1:name=codis-test1-1 statusstatuscriokgramme address 10.0.2.5 6379 people master1 sentinelswatch 3master2 purveyornamestatuscodisriptytest1-2 statusstatuscriokle addressperson 10.0.2.6 Fluves6379 slaves1sentinelsquarter3

Configured and started successfully.

Note: if the codis-dashboard is not configured, it will miss the last few lines, because it is not controlled by zookeeper, so it is normal and will be loaded automatically after it is configured.

4) configure codis-proxy. There are only two to be configured this time. Of course, you can also configure three.

This is the core of the codis cluster. In fact, it does not have a master-slave architecture, and the configuration is also used from zookeeper, so let's look at the configuration directly.

# there are many configurations. Our husband becomes the default configuration file / usr/local/codis/codis-proxy-- default-config | tee. / proxy.conf#. Then we put the configuration in the configuration file directory of the redis data directory, and then change the key location. Other default vim / data/redis/data/config/proxy.conf# project names will be registered in zookeeper. If you want a set of zookeeper to manage multiple codis, You must distinguish between product_name = "codis-test1" # set the password for login to dashboard (consistent with the requirepass in the real redis) product_auth = "123" # the login password for the client (redis-cli) (inconsistent with the requirepass in the real redis), is the password for login codis session_auth = "123456" # managed port, 0.0.0.0 is open to all ip, based on security considerations You can restrict the intranet admin_addr = "0.0.0.0 tcp4 11080" # to communicate in that way. If your network supports tcp6, you can set another proto_type = "tcp4" # client (redis-cli) to access the agent's port. 0.0.0.0 is open to all ip. Proxy_addr = "0.0.0.0proto_type 19000" # external configuration storage type, we use zookeeper, of course, there are others that can support Do not expand here to say that jodis_name = "zookeeper" # configure the connection address of zookeeper. Here are three sets of jodis_addr = "10.0.2.5 jodis_addr 2181" # zookeeper password of three sets of proxy_max_clients = "10.0.2.5 jodis_addr 2181" # zookeeper. If there is any, the maximum number of connections of the agent is 1000 by default. If the concurrency is too large, you may need to adjust this pipeline parameter. In most cases, the default value of 10000 is enough for session_max_pipeline = 10000 cycles, or you can change the following parameter backend_max_pipeline = 204800session_recv_bufsize = "256kb" session_recv_timeout = "0s" # and then you can start it. / usr/local/codis/codis-proxy-- ncpu=1-- config=/data/redis/data/config/proxy.conf-- log=/data/redis/data/logs/proxy.log & # verify ss-ntplu | grep codis-proxytcp LISTEN 0 128 *: 19000 *: * users: ("codis-proxy", pid=2075 Fd=4)) tcp LISTEN 0 128: 11080: * users: (("codis-proxy", pid=2075,fd=6))

-- ncpu specifies how many cpu to use. Mine is a virtual machine, so it's 1. If you are multicore, fill in more than one.

-- config specifies the configuration file, which is the configuration file just now

-log specifies the output log file

Configuration and startup succeeded, but not yet, as codis-dashboard still needs to be configured to finalize.

5) to configure codis-dashboard, you only need to configure one machine.

This is a tool for managing configuration codis cluster information. After configuration, the configuration information will be automatically loaded into the zookeeper cluster. Even if the service is down, the configuration is still on zookeeper, so there is no need to consider high availability. A single point is enough. It is not particularly troublesome to restart at the worst. The configuration interface is implemented by codis-fe. Because it is limited to the namespace of zookeeper, it is usually configured with proxy.

# We can also use the program to generate the default configuration file / usr/local/codis/codis-dashboard-default-config | tee. / dashboard.conf#. Then we put the configuration in the configuration file directory of the redis data directory, and then change the key location. Other external configuration storage types can be vim / data/redis/data/config/dashboard.conf# by default. We use zookeeper, of course, there are other supports. Do not expand here to say coordinator_name = "zookeeper" # configure zookeeper connection address, here are three coordinator_addr = "10.0.2.5 coordinator_addr 2181" # Project name, which is registered in zookeeper, if you want a set of zookeeper to manage multiple sets of codis, you must distinguish between product_name = "codis-test1" # login passwords of all redis (same as requirepass in real redis) Because to log in to the communication port that modifies the data product_auth = "123" # codis-dashboard, 0.0.0.0 means that it is open to all, it is best to use the private network address admin_addr = "0.0.0.0 admin_addr 18080" # if you want to execute some scripts on the failover function on the codis cluster, you can configure the following two configurations sentinel_notification_script = "" sentinel_client_reconfig_script = "" start / usr/local/codis/codis-dashboard-- ncpu=1-- config=/data/redis/data/config/dashboard.conf-- log=/data/redis/data/logs/codis_dashboard.log-- log-level=WARN & # verify ss-ntplu | grep codis-dashboardtcp LISTEN 0 128:: 18080: * users: (("codis-dashboard", pid=2021,fd=5))

-- ncpu specifies how many cpu to use

-- config specifies the profile

-log specifies the output log file

-- log-level specifies the log level, with INFO,WARN,DEBUG,ERROR

Installation is complete, only the last step to start the configuration.

Since codis-dashboard itself does not need a password to log in, it will be very dangerous. It is strongly recommended to use an intranet address, and the author said that he will consider adding the authentication password of codis-dashboard in the next version.

6) to configure codis-fe, you only need to configure one machine.

This is a tool that belongs to the web interface to manipulate codis-dashboard configuration. The web code file is located in the directory of the codis installation folder, specifically: / usr/local/codis/assets/. Can be shared by multiple dashboard.

The tool itself does not need a configuration file to start, just specify the ip and port of codis-dashboard, but for ease of management, it is better to generate a configuration file.

# generate the configuration file, that is, the ip and port / usr/local/codis/codis-admin of codis-dashboard-- dashboard-list-- zookeeper=10.0.2.6:2181 > codis.json#. Then let's put the configuration in the configuration file directory of the redis data directory and take a look at cat / data/redis/data/config/codis.json [{"name": "codis-test1" "dashboard": "10.0.2.6 log=/data/redis/data/logs/fe.log 18080"}] # and then you can start, / usr/local/codis/codis-fe-- ncpu=1-- log=/data/redis/data/logs/fe.log-- log-level=WARN-- dashboard-list=/data/redis/data/config/codis.json-- listen=0.0.0.0:8090 &

-- ncpu specifies how many cpu to use

-log specifies the output log file

-- log-level specifies the log level, with INFO,WARN,DEBUG,ERROR

-- dashboard-list specifies the address and project name of dashboard. Here, because the file is generated, it is specified as a file.

-- listen specifies the web login port of codis-fe, that is, we access the management terminal through 8090. 0.0.0.0 means that there is no restriction on visiting IP. In fact, it is best to restrict the private network.

Verify it.

The full set of installation is complete and starts successfully. Start the next step.

Use examples

Because of the web interface, basically is the interface operation, very convenient, configuration will be directly loaded into the zookeeper inside.

First, let's add the codis-proxy address and port

In order:

The first step is to add the address and management port of codis-proxy, which is set to 11080.

Step 2, click the orange button on the left, and then add it.

The third step is to see the codis-proxy address below, and then to see that the color of the SYNC on the right is green, which means the configuration is normal.

If you want to delete the record, just click the red button on the right.

Then, we add the real redis-server (also codis-server) address and port

In order:

The first step is to create a group and prepare to put a related group of masters and subordinates into it.

Step two, click the button to generate the grouping

The third step is to add the real redis-server address and select a packet, such as the one you just created

Step 4, click the button to generate the configuration

Step 5, you can see that the configuration has been registered, pay attention to the sync status.

Step 6, click to rebalance all slots data blocks (this is required for any addition or deletion of old and new nodes)

In the old version, slots needs to be manually configured, but after version 3.2, it has been changed to automatic allocation, so there is no need to configure, just click on it. Of course, you can also assign it manually.

Finally configure the address and port of sentinel

In order:

The first step is to add real sentinel addresses and ports

Step 2, click the button to add

The third step is to check the status. It is a little different here. It will automatically add the number of current master-slave group structures and control the switch.

As I said before, they automatically change the configuration file. You can take a look at the sentinel configuration file to confirm that we will not expand here.

Once all is configured, you can use it. Connect one of the codis-proxy to test, and pay attention to distinguish the login address and port, as well as the password.

/ usr/local/codis/redis-cli-h 10.0.2.5-p 19000-a 12345610.0.2.5 19-generic x86_64arch_bits:64multiplexing_api:epollgcc_version:4.8.4process_id:2032run_id:98e2364d837990dfb47be050901ef9e36ea113fatcp_port:6379uptime_in_seconds:16312uptime_in_days:0hz 19000 > info# Serverredis_version:3.2.9redis_git_sha1:f8bc4e32redis_git_dirty:0redis_build_id:2bdb8aa56be3fbc2redis_mode:standaloneos:Linux 4.10.0-19-generic x86_64arch_bits:64multiplexing_api:epollgcc_version:4.8.4process_id:2032run_id:98e2364d837990dfb47be050901ef9e36ea113fatcp_port:6379uptime_in_seconds:16312uptime_in_days:0hz : 10lru_clock:3634855executable:/usr/local/codis/codis-serverconfig_file:/data/redis/data/config/redis_6379.conf # Clientsconnected_clients:71client_longest_output_list:0client_biggest_input_buf:0blocked_clients:0 # Memoryused_memory:61878808used_memory_human:59.01Mused_memory_rss:76623872used_memory_rss_human:73.07Mused_memory_peak:63148384used_memory_peak_human:60.22Mtotal_system_memory:1529741312total_system_memory_human:1 .42Gused _ memory_lua:37888used_memory_lua_human:37.00Kmaxmemory:102400000maxmemory_human:97.66Mmaxmemory_policy:allkeys-lrumem_fragmentation_ratio:1.24mem_allocator:jemalloc-4.0.3...

It's ready to use.

Pressure testing

1. Performance testing

First, use the built-in redis-benchmark to test the performance, simulating 500 concurrency and 1 million requests. Pay attention to distinguish between login address and port, as well as password

Pressure test the performance of codis-proxy first.

/ usr/local/codis/redis-benchmark-h 10.0.2.5-p 19000-a 123456-c 500-n 1000000-Q

Re-pressure testing the performance of a single node

/ usr/local/codis/redis-benchmark-h 10.0.2.5-p 6379-a 123-c 500-n 1000000-Q

Redis-benchmark parameter resolution:

-h ip address

-p redis port

-an authentication password

-c set how many concurrent connections

-n how many requests in total

-Q display mode: brief mode

And then look at the picture.

As you can see, some operations are not much different, some are very different, and the performance loss is obvious, but as a cluster application, it mainly deals with a high concurrency environment, and the performance loss is acceptable, not to mention that for high-speed memory applications like redis, the performance loss is basically not perceived.

two。 Data distribution test

Then there is the read-write distribution test:

I wrote a script to test:

Cat tmurredis.shemaking binbinapachehaos = "10.0.2.7" pot= "19000" pawd= "123456" cli= "/ usr/local/codis/redis-cli" keyset= "keytest2" jlasdnfnsdfsdf;sdfhlkjahsdjlkfadfjkasdbbcjhdgasfyuefkbadjkhflk "dbname=2a=0for I in `do $cli-h $hos-p $pot-a $pawd-n $dbname 'set' ${keyset} ${a}" ${valueset} ${a} "> / dev/null # echo $a let a++done

The script is simple, just keep writing junk data to the codis cluster and execute the script.

Bash t-redis.sh

Then the result can look at the web interface, because you connect to codis-proxy with info, it is actually not accurate, that only shows the data of a single station.

As you can see, each group is evenly distributed, dividing the pressure among the three redis-server master servers.

3. Failover test

Then let's look at the failover and continue to execute the script.

Bash t-redis.sh

Enter one of the codis-server, such as 10.0.2.5, and the state is normal.

Start the simulation operation

# find the main library process ss-ntplu | grep codis-server tcp LISTEN 0 128 *: 6379 *: * users: (("codis-server", pid=2032,fd=4)) tcp LISTEN 0 128 *: 6380 *: * users: (("codis-server", pid=2037,fd=4)) # kill the main library process kill 2032

At this point, the slave library takes over the process of the master library, and sentinels has a prompt.

Someone may find that the sync button in group 1 turns red, that is, the master and slave fails and returns to normal at one click.

Now, waiting for the data to be written, I will first restart port 6379 of the old main process.

/ usr/local/codis/codis-server / data/redis/data/config/redis_6379.conf

Then take a look:

Look, the state is back to normal, but students with computers can calculate that my script executes a total of 300000 key, but now there are thousands less.

Pit-1:

The problem with missing key is that during the redis-sentinel failover, the entire codis cluster will not close the connection to the failed redis-server, so the codis-proxy will still send data to the current failed redis-server. Obviously, the redis-server cannot store the data, which leads to the phenomenon of missing key. If the whole master and slave dies, all key sent to this redis-server will be discarded unless the failed node is manually removed.

Although codis also comes with a failover program codis-ha, it belongs to a daemon and connects to codis-dashboard to view the status of each node.

# execute the command to start codis-ha, and the port is codis-dashboard port / usr/local/codis/codis-ha-- dashboard=10.0.2.6:18080-- log=/data/redis/data/logs/ha.log-- log-level=WARN &

-- dashboard specifies the address and port of the dashboard

-- log specifies the log file

-- log-level specifies the log level, with INFO,WARN,DEBUG,ERROR

But this software is also defective, it will automatically connect to dashboard to detect the health information of each master-slave structure, the detection interval is very fast (default 3 seconds, can modify the parameter-interval), after the fault is detected, the fault master database or slave library will be forced offline and the information registered in dashboard will be deleted.

Although the switching speed is very fast, only a few key will be lost (some will be lost in 3 seconds), but the old main library will be forced offline later. You need to manually modify the configuration and restart redis-server (codis-server), and then add the configuration to the codis-fe interface.

Obviously this is not fully automatic management, a bit troublesome, and will also make redis-sentinel meaningless, so there are only two ways to choose one.

Although there seems to be less loss of key, there are still cases of loss of key, which can only be said to be 50 steps and 100 steps, and other slave instances in the group will not change their state automatically. These slave will still try to synchronize data from the old master, which will lead to data inconsistency between the new master and other slave in the group. Therefore, when the master-slave switch occurs, the administrator is required to manually create a new sync action to complete the data synchronization between the new master and slave, which increases the workload of manual operation. You still need to consider the actual situation when you choose the cluster of codis-ha and redis-sentinel.

So, in the final analysis, the failover of codis is not done well. If losing key can be tolerated, it is enough to open redis-sentinel. For those with high data consistency, it is better to open codis-ha plus script to achieve it.

Add and delete nodes in real time

Another selling point of codis is that you can add / delete redis-server (codis-server) nodes online to expand capacity and replace problem nodes in real time. For a single point of redis, you can have more space without restarting the service, and you can also switch problematic nodes online. However, it should be noted that real-time capacity expansion and failover does not mean that there is no performance loss. It is really necessary to pay attention to online pressure to avoid service unavailability caused by performance pressure.

The principle of implementation is that the codis cluster divides each redis-server node into multiple slots blocks (a total of 1024) by rules. Need to expand as long as the new nodes are added, the expansion effect can be achieved by re-allocating the slots information, and if you need to fail over, the slots of the problem redis-server node can be removed, and then the node can be removed. It can be said that there is almost no perception of the online environment, and no lost key phenomenon is found during the test, that is, the performance has degraded, but the performance degradation in the environment I tested is acceptable. The performance loss is only about 10% per cent and 30 per cent.

To start the experiment, first of all, let's assume that we add two new redis-server nodes, both of which are directly the master library and have no slave library environment (because there is no need to test for failover this time):

10.0.2.5:16379

10.0.2.6:16379

1. Adding a node is tantamount to capacity expansion, which is relatively simple.

The first step is about the same as before, first create a new group, click the button to confirm the addition

Step 2, add the new address to the new group and click the button to confirm the addition.

Step 3, confirm that the new address has been successfully added

Step 4, click the button to reassign all slots blocks

The only problem here is the last step. Redistribution will consume some resources. Codis will automatically balance the distribution of data blocks, so there will be a process of data migration, but according to the results of my test, it is not very serious, about 20%.

According to the surveillance it comes with,

You can see that before the normal qps is close to 1500, the rebalance drop is more serious, then there is about 20% performance loss, and finally it returns to normal after the migration. Of course, this is the case of a small amount of data, if the amount of data is large, the migration time may not be so simple.

two。 Switch a node and get off the shelf

Just click this button to get off the shelves normally.

But because there is still data in it, it is not allowed to take it off the shelves directly.

So we need to migrate the data first, as follows:

The first step is to confirm the number of the data block of a group that needs to be migrated. For example, the block of 499-512 here belongs to data group 4. I want to migrate group 4 now. I will select this one.

The second step is to fill in the information just obtained, that is, migrate the data blocks numbered 500 from group 4 to group 5, and click the button to execute.

And then you'll see

Obviously, the information of group 4 disappeared, and codis migrated all the data blocks of group 4 to group 5.

At this point, the redis-server node can be deleted.

As for whether it needs to be re-filled, this question needs to be considered by itself. If it doesn't need to be filled, it's better to rebalance slots by clicking again.

As you can see, the balance has been rebalanced again.

Fault handling

1.codis-dashboard fails to start and prompts:

[ERROR] store: acquire lock of codis-test1 failed

[error]: zk: node already exists

As a result of the test environment, I often forced shutdown, resulting in codis-dashboard did not shut down normally. Directly cause the status in the zookeeper not to be updated, and eventually the newly started codis-dashboard cannot register with the zookeeper. It keeps prompting that it already exists and is forcibly closed.

It is not difficult to fix it, just delete the status key of the lock.

# enter project name and zookeeper address / usr/local/codis/codis-admin-remove-lock-product=codis-test1-zookeeper=10.0.2.6:2181

Then, codis-dashboard can start normally again.

Codis-admin is a tool that can fully control the codis cluster, and all the work of adding / deleting / modifying can be realized by it. There are many parameters. Here is only an example. For more information, please see codis-admin-help.

2.codis-proxy exited abnormally so that it cannot be deleted

Usually, codis-proxy is removed through codis-dashboard. During the removal process, codis-dashboard sends an offline instruction to codis-proxy for security, and only when it is successful will the proxy information be removed from external storage (zookeeper). If codis-proxy exits abnormally, the operation will fail. At this point, you need to delete the zookeeper information manually.

# Log in to zookeeper to operate / usr/local/zookeeper/bin/zkCli.sh-server# to confirm that the problematic codis-proxy address corresponds to the message on zookeeper [zk: localhost:2181 (CONNECTED) 6] get / codis3/codis-zyyhj1/proxy/proxy-80722e128c6e8fc3d0da44983343a843 {"id": 1, "token": "80722e128c6e8fc3d0da44983343a843", "start_time": "2018-06-16 2323 id 3057.44436209 + 0800 CST" "admin_addr": "10.21.1.140 tcp4", "proto_type": "tcp4", "proxy_addr": "10.21.1.140 proxy_addr", "jodis_path": "/ jodis/codis-zyyhj1/proxy-80722e128c6e8fc3d0da44983343a843", "product_name": "codis-zyyhj1", "pid": 26493, "pwd": "/ root" "sys": "Linux localhost.localdomain 3.10.0-514.el7.x86_64 # 1 SMP Tue Nov 22 16:42:41 UTC 2016 x86" 64 x86 "64 x86 GNU/Linux", "hostname": "localhost.localdomain", "datacenter": ""} cZxid = 0x100001c7bctime = Fri Jun 15 21:02:22 CST 2018mZxid = 0x1000038aemtime = Sat Jun 16 23:30:57 CST 2018pZxid = 0x100001c7bcversion = 0dataVersion = 2aclVersion = 0ephemeralOwner = 0x0dataLength = 576numChildren = confirmation complete Delete [zk: localhost:2181 (CONNECTED) 7] rmr / codis3/codis-zyyhj1/proxy/proxy-80722e128c6e8fc3d0da44983343a843# is deleted, you still need to reload dashboard, otherwise the information will not be updated / usr/local/codis/codis-admin-- dashboard=10.21.1.124:18081-- reload

Then, the proxy end that can not be deleted is gone, finished.

Remarks:

Codis does not support command lists:

Command TypeCommand NameKeysKEYS

MIGRATE

MOVE

OBJECT

RANDOMKEY

RENAME

RENAMENX

SCAN

StringsBITOP

MSETNX

ListsBLPOP

BRPOP

BRPOPLPUSH

Pub/SubPSUBSCRIBE

PUBLISH

PUNSUBSCRIBE

SUBSCRIBE

UNSUBSCRIBE

TransactionsDISCARD

EXEC

MULTI

UNWATCH

WATCH

ScriptingSCRIPT

ServerBGREWRITEAOF

BGSAVE

CLIENT

CONFIG

DBSIZE

DEBUG

FLUSHALL

FLUSHDB

LASTSAVE

MONITOR

RESTORE

SAVE

SHUTDOWN

SLAVEOF

SLOWLOG

SYNC

TIME

Codis SlotSLOTSCHECK

SLOTSDEL

SLOTSINFO

SLOTSMGRTONE

SLOTSMGRTSLOT

SLOTSMGRTTAGONE

SLOTSMGRTTAGSLOT

The following are semi-supported commands, and Codis does not support cross-node operations, so you must use hash tags to put all keys that may appear in a request in the same slot, and then you can use these commands. Codis does not check whether the key has the same tag, so if you do not use the tag, your program will get the wrong response.

Command TypeCommand NameListsRPOPLPUSHSetsSDIFF

SINTER

SINTERSTORE

SMOVE

SUNION

SUNIONSTORESorted SetsZINTERSTORE

ZUNIONSTOREHyperLogLogPFMERGEScriptingEVAL

EVALSHA

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.