Shulou(Shulou.com)06/01 Report--

In this article, the editor introduces in detail "how to deploy Codis cluster service in Centos system". The content is detailed, the steps are clear, and the details are handled properly. I hope this article "how to deploy Codis cluster service in Centos system" can help you solve your doubts.

A brief introduction to Codis

Codis is a distributed Redis service developed by Wandoujia Infrastructure Team. Users can be regarded as a Redis service with unlimited memory, which has the ability of dynamic expansion / reduction. It is more practical for partial storage business. If you need instructions such as SUBPUB, Codis does not support it. Always remember that Codis is a distributed storage project. For a large number of key, the value is not too large (

II. Codis architecture diagram

1. Component description

Codis consists of four parts:

Codis Proxy (codis-proxy)

Codis Dashboard (codis-config)

Codis Redis (codis-server)

ZooKeeper/Etcd

Codis-proxy: a Redis proxy service connected to a client. Codis-proxy itself implements the Redis protocol and behaves like a native Redis (like Twemproxy). For a business, it is stateless to deploy multiple codis-proxy,codis-proxy.

Codis-config: is a management tool for Codis, which supports operations such as adding / deleting Redis nodes, adding / deleting Proxy nodes, initiating data migration and so on. Codis-config also comes with a http server and launches a dashboard. Users can directly observe the status of the Codis cluster in the browser.

Codis-server: is a Redis branch of Codis project maintenance, based on 2.8.13 development, added slot support and atomic data migration instructions, the upper layer of Codis codis-proxy and codis-config can only interact with this version of Redis to work properly.

ZooKeeper: used to store the data routing table and meta-information of codis-proxy nodes. Commands initiated by codis-config are synchronized to each surviving codis-proxy through ZooKeeper.

Note: Codis supports distinguishing different products according to Namespace, and products with different product name will not conflict with each other.

2. Experimental environment

Hostname IP address roles linux-node110.1.1.148Zookeeper, redis_master,redis_slavelinux-node210.1.1.149Zookeeper, redis_master,redis_slavelinux-node310.1.1.150zookeeper,codis-proxy, redis_master,redis_slave

Note: production environment recommends that redis be placed on a separate server

3. Installation and configuration

Note: for the environment where a single codis-proxy node is deployed, it is recommended that the relevant installation packages be downloaded locally and uploaded to the server.

# operate on codis-proxy (linux-node3) server

Yum install-y git gcc make gathers + gcc-c++ automake openssl-devel zlib-devel

3.1. install GO

Cd / usr/local/src/tar zxvf go1.7.linux-amd64.tar.gz-C / opt

Note: the following steps for installing JDK and zookeeper after Zookeeper has been built can be skipped.

3.2. install JDK

Rpm-ivh jdk-8u65-linux-x64.rpm

3. Install zookeeper

Wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.8/zookeeper-3.4.8.tar.gztar zxvf zookeeper-3.4.8.tar.gz-C / opt/

3.4. Environment variable setting

Modify the environment variable: vim / etc/profile add the following at the end

# codis pathexport GOROOT=/opt/goexport GOPATH=/opt/codisJAVA_HOME=/usr/java/jdk1.8.0_101CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/libexport PATH=$PATH:$GOROOT/bin:$JAVA_HOME/bin:

# Test environment variable settings

# execute source / etc/profile when the environment variable takes effect, and test whether go is installed normally:

Cat hello.gopackage mainimport "fmt" func main () {fmt.Printf ("hello,world\ n")} [root@linux-node-2 ~] # go run hello.gohello,world! 3. Install CODIS [linux-node-3 @ root] # yum-y install git [root@linux-node-3 opt] # go get-u-d github.com/CodisLabs/codis

You can see no buildable Go source files in / opt/codis/src/github.com/wandoulabs/codis. We set GOAPTH=/opt/codis in the above environment, so as long as we execute the acquisition command above, it will be downloaded under / opt/codis:

We go to the prompted path for installation, which takes a long time and waits patiently:

[root@linux-node-3 ~] # cd / opt/codis/src/github.com/CodisLabs/codis

Directory executes the make command to compile the code, and executes make gotest to run the test

Makemake gotest

It is recommended to download codis only through the go get command, unless you are so familiar with the directory reference form of the go language that it will not cause the code to be misplaced. This command downloads the latest version of the master branch, and we will ensure the stability of the master branch.

After all the instructions are executed, three executable files, codis-config, codis-proxy, and codis-server, are generated in the bin folder. In addition, the bin/assets folder is the front-end resource required by codis-config 's dashboard http service, which needs to be placed in the same folder as codis-config)

Install codis through git

Git clone https://github.com/CodisLabs/codis.git

After downloading the git repository, we then do the following. As follows:

Mkdir-p / opt/codis/src/github.com/CodisLabs/ cp-r codis/ opt/codis/src/github.com/CodisLabs/cd / opt/codis/src/github.com/CodisLabs/codis/

# after the above operations are completed, execute the make command to compile, and then execute the make gotest command to test in the same way as downloading and installing through go

Installed through the source package

Wget https://github.com/CodisLabs/codis/archive/3.0.3.tar.gz-P / usr/local/srctar zxvf 3.0.3.tar.gz & & cd codis-3.0.3 & & make & & make gotest

Detailed explanation of codis-related configuration files

# main usage of codis-config after codis installation is completed

#. / bin/codis-config-h (master) usage: codis-config [- c] [- L] [--log-level=] [...] Options:-c configuration file address-L log output file address-log-level= output log level (debug #. / bin/codis-proxy-husage: codis-proxy [- c] [- L] [--log-level=] [--cpu=] [--http-addr=] options:-c configuration file address-- L log output file address-- log-level= output log level (debug IV, configure codis cluster

After the codis installation, let's now configure the codis cluster. Before formally configuring the cluster, create the relevant directory, and then copy the relevant files to the new directory.

1. Codis cluster configuration

Use the following command:

Cd / opt/codis/src/github.com/CodisLabs/codis/mkdir-p / opt/codis/ {log,redis_conf} cp-rf bin/ / opt/codis/cp config.ini / opt/codis/cp extern/redis-test/conf/6379.conf / opt/codis/redis_conf/cp extern/redis-test/conf/6380.conf / opt/codis/redis_conf/

2. Edit the codis configuration file

[root@linux-node3 codis] # egrep-v "^ # | ^ $" config.inicoordinator=zookeeperzk=192.168.1.148:2181192.168.1.149:2181192.168.1.150:2181product=testdashboard_addr=192.168.1.150:18087password=backend_ping_period=5session_max_timeout=1800session_max_bufsize=131072session_max_pipeline=1024zk_session_timeout=30000proxy_id=proxy_3

In this configuration file, we need to pay attention to three parameters: zk, dashboard_addr, and proxy_id.

Where zk is the server IP address of the zookeeper cluster, dashboard_addr is the IP address and port managed by codis web, and proxy_id is the id of codis. Note that this value should be unique for each codis server.

# linux-node1

# linux-node2

# linux-node3

3. Edit the redis configuration file

On each codis server, start two redis instances (you can also start multiple redis instances) and configure two redis, as follows:

Daemonize nopidfile / var/run/redis.pidport 6379tcp-backlog 511timeout 0tcp-keepalive 0loglevel noticelogfile / var/log/redis/6379.logdatabases 16stop-writes-on-bgsave-error yesrdbcompression yesrdbchecksum yesdbfilename dump6379.rdbdir / opt/codis/slave-serve-stale-data yesslave-read-only yesrepl-disable-tcp-nodelay noslave-priority 100appendonly noappendfilename "appendonly.aof" appendfsync everysecno-appendfsync-on-rewrite noauto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64mblua-time-limit 5000slowlog-log-slower-than 10000slowlog-max-len 128latency -monitor-threshold 0notify-keyspace-events "" hash-max-ziplist-entries 512hash-max-ziplist-value 64list-max-ziplist-entries 512list-max-ziplist-value 64set-max-intset-entries 512zset-max-ziplist-entries 128zset-max-ziplist-value 64hll-sparse-max-bytes 3000activerehashing yesclient-output-buffer-limit normal 00 0client-output-buffer-limit slave 256mb 64mb 60client-output-buffer-limit pubsub 32mb 8mb 60hz 10aof-rewrite-incremental-fsync yes

# pay attention to one parameter of the red line, and configure the second redis instance

\ cp / opt/codis/redis_conf/6379.conf 6380.confsed-I's var/log/redis/ 6379Universe 6380Universe / opt/codis/redis_conf/6380.confmkdir-p / var/log/redis/

# the above operations need to be performed on each codis service.

Echo 512 > / proc/sys/net/core/somaxconnsysctl vm.overcommit_memory=1

Note: optimize parameters of the system

4. Initialize slots

Execute bin/codis-config slot init, which creates slot-related information on zookeeper

/ opt/codis/bin/codis-config-c / opt/codis/config.ini slot init

# linux-node3 、 linux-node1 、 linux-node2

/ opt/codis/bin/codis-server / opt/codis/redis_conf/6379.conf & / opt/codis/bin/codis-server / opt/codis/redis_conf/6380.conf & ps-ef | grep codis-server

# start redis instances on the three codis-proxy

5. Start codis dashboard

# try to start dashboard in the redis installation directory or the log directory of redi, which makes it easy to view the log generated by startup, nohup.out

Nohup / opt/codis/bin/codis-config-c / opt/codis/config.ini dashboard &

# you can see from the figure that dashboard has been started, and then visit the web page. The codis dashboard access port is 18087.

# View codis dashboard, and the browser accesses http://192.168.1.150:18087/, as follows:

The default page after the launch of # codis dashboard is shown in the figure above. We can create a new redis_group through the web page, add the newly launched redis instance to different groups, or perform the creation of redis_group through the command line.

6. Add Redis Server Group

# each Server Group exists as a Redis server group. Only one master and multiple slave are allowed. Group id only supports integers greater than or equal to 1.

# use the command to create a new server group on codis dashboard

[root@linux-node3 ~] # cd / opt/codis/ [root@linux-node3 codis] #. / bin/codis-config server-husage:codis-config server listcodis-config server addcodis-config server removecodis-config server promotecodis-config server add-groupcodis-config server remove-group

# for example: add three server group, each group has two redis instances, the id of group is 1 and 2 respectively, and the redis instance is one master and one slave.

# Note: here I add two redis instances on linux-node3,linux-node2,linux-node1 to different server group. Details are as follows:

Group_1: linux-node3 redis_6379 linux-node2 redis_6380group_2: linux-node2 redis_6379 linux-node1 redis_6380group_3: linux-node1 redis_5379 linux-node3 redis_6380

Port 6379 of redis is used as redis master,6380 as slave

# add a group,group with an id of 1, and add a redis master (192.168.1.150 redis master 6379) to the group

. / bin/codis-config server add 1 192.168.1.150:6379 master

# add a redis slave (192.168.1.149 6380) to the group

. / bin/codis-config server add 1 192.168.1.149:6380 slave

# add a group,group with an id of 2, and add a redis master (192.168.1.149 redis master 6379) to the group

. / bin/codis-config server add 2 192.168.1.149:6379 master

# add a redis slave (192.168.1.148 6380) to the group

. / bin/codis-config server add 2 192.168.1.148:6380 slave

# add a group,group with an id of 3, and add a redis master (192.168.1.148 redis master 6379) to the group

. / bin/codis-config server add 3 192.168.1.148:6379 master

# add a redis slave (192.168.1.150 redis slave 6380) to the group

. / bin/codis-config server add 3 192.168.1.150:6380 slave

Note: the first redis instance added by each group cannot be deleted because codis sets the redis instance to master by default.

# We can see that server group has been created successfully from the figure. Check codis dashboard:

7. Set the slot range of server group service

Https://github.com/CodisLabs/codis/blob/master/doc/FAQ_zh.md

Codis uses Pre-sharding technology to implement data slicing. By default, it is divided into 1024 slots (0-1023). For each key, the SlotId is determined by the following formula: SlotId = crc32 (key)% 1024 each slot will have a specific server group id to indicate which server group provides the data of this slot.

[root@linux-node3 codis] #. / bin/codis-config slot-husage:codis-config slot init [- f] codis-config slot infocodis-config slot setcodis-config slot range-setcodis-config slot migrate [--delay=] codis-config slot rebalance [--delay=]

# the slot numbered [0340] is served by server group 1, the slot numbered [341,682] is serviced by server group 2, and the slot numbered [683,1023] is serviced by server group 3

. / bin/codis-config slot range-set 0 340 1 online./bin/codis-config slot range-set 341 682 2 online./bin/codis-config slot range-set 683 1023 3 online

# View the information of slot

/ opt/codis/bin/codis-config-c / opt/codis/config.ini slot info 1/opt/codis/bin/codis-config-c / opt/codis/config.ini slot info 2/opt/codis/bin/codis-config-c / opt/codis/config.ini slot info 3

# after the slot operation is completed, you can check the status in codis dashboard. As follows:

8. Start codis-proxy

. / bin/codis-proxy-c config.ini-L. / log/proxy.log-- cpu=1-- addr=0.0.0.0:19000-- http-addr=0.0.0.0:11000

Nohup / opt/codis/bin/codis-proxy-c / opt/codis/config.ini-log-level=error-L / opt/codis/log/proxy.log-cpu=1-addr=0.0.0.0:19000-http-addr=0.0.0.0:11000 &

# the parameters in the above command are explained below:

-c profile address. -L log output file address. -log-level= output log level (the number of cpu cores occupied by debug-cpu= proxy. The default is 1, which is best set to about half of the physical cpu of the machine to 2 to 3. -redis server listener address of addr= proxy, format:, such as: localhost:9000,: 9001. -the debug information of http-addr= proxy starts the http server, and you can access http://debug_http_server_addr/debug/vars.

After # codis-proxy starts, we can view it on dashboard, as follows:

# after the codis cluster has been built, it is recommended that redis instances and codis-proxy be placed on different servers for performance requirements.

9. Connect to the codis cluster

After the codis cluster is built, test the connection to the codis cluster. To connect to the codis cluster, we just need to connect to the codis-proxy. Codis-proxy server address: 192.168.1.150, then add port 19000. Use the redis-cli command to connect as follows:

# Test on a server with a redis instance installed:

From the figure above, we can clearly see that the connection to the codis cluster is ok.

10. Codis cluster stress test

We will now do some stress tests on the codis cluster and observe the key-value pairs on dashboard. As follows:

. / redis-benchmark-h 192.168.1.150-p 19000-c 1000-d 1000-t set-n 100000-r 100000

The above command means that using the redis-benchmark stress test command to connect to the codis cluster, concurrently 10000 (- c), test set operations (- t), each test data set is 100byte (- d), the number of requests is 100000 (- n), use random numbers to insert values (- r)

11. Data migration

Secure and transparent data migration is an important feature provided by Codis, which distinguishes Codis from static distributed Redis solutions such as Twemproxy.

The smallest unit of data migration is key. We have added some instructions to codis redis to implement key-based migration, such as SLOTSMGRT (command list). Each time, a random key of a specific slot will be sent to another codis redis instance. This command will confirm that the other party has received it, delete the local key v value, and return the number of remaining key of this slot. The whole operation is atomic.

In the codis-config management tool, the smallest unit for each migration task is slot

For example, migrate the slot data whose slot id is [0-340] to server group 2. The-delay parameter indicates the number of milliseconds of sleep after each key is migrated. The default is 0, which is used for speed limit.

# codis dashboard to view the data status in each group

# you can see that after we have done the stress test, the key processed by each redis group is basically the same, and then we migrate the data on the slot in the group_1 to the group_2. The-delay parameter indicates the number of milliseconds of the sleep after each key is migrated. The default is 0, which is used for speed limit.

$bin/codis-config slot migrate 0 511 2-delay=10/opt/codis/bin/codis-config-c / opt/codis/config.ini slot migrate 0 340 2-delay=10

# check the group status on codis dashboard:

# refresh again, 3 seconds later

# comparing the two key changes in group, we can see that the slot data in group_1 is being migrated to group_2.

The migration process is secure and transparent for the upper layer business, the data will not be lost, and the upper layer will not abort the service.

Note that it is OK to interrupt during migration, but if you interrupt a task that is migrating a slot, you need to migrate the slot that is in the migration state next time, otherwise you cannot continue (that is, the migration program will check that only one slot can be migrated at a time).

Auto Rebalance

Codis supports automatic migration of slot according to instance memory to balance data distribution.

$bin/codis-config slot rebalance

Request:

All codis-server must have maxmemory parameter set. All slots should be in online state, that is, no migration task is in progress. All server group must have Master 5. Codis cluster architecture simulation failure test

Server group implements master-slave switching. When a redis instance in the same group stops, other redis instances in the group automatically switch to the master role and provide services.

Need to install the codis-ha plug-in, codis-ha can be installed on any node, only need to install on one node. (this lab is installed on linux-node2)

Go get github.com/ngaut/codis-ha [root@linux-node2 ~] # cd / opt/codis/src/github.com/ngaut/ [root@linux-node2 ngaut] # cp-r codis-ha/ opt/ [root@linux-node2 ngaut] # cd / opt/codis-ha/ [root@linux-node2 codis-ha] # go build

1. Start the codis-ha service

Nohup $codis_ha_home/codis-ha-codis-config=192.168.1.150:18087-log-level=info-productName=test & > / opt/codis-ha/codis-ha.log

Note:-codis-config followed by ip, the server where dashboard resides

Restart the codis-proxy service

Note: (take the 150 machine configuration as an example, the rest of the codis-proxy only need to be changed under the name)

Stop codis-proxy first.

[root@linux-node3] # / opt/codis/bin/codis-config-c / opt/codis/config.ini proxy offline codis_proxy_3

# different machines only need to change the codis_proxy (in this experiment, a codis-proxy has been restarted and the codis-ha service has taken effect)

Restart codis-proxy

Nohup / opt/codis/bin/codis-proxy-c / opt/codis/config.ini-- log-level=error-L / opt/codis/log/proxy.log-- cpu=1-- addr=0.0.0.0:19000-- http-addr=0.0.0.0:11000 &

Set codis-proxy status to online

[root@linux-node3] # / opt/codis/bin/codis-config-c / opt/codis/config.ini proxy online codis_proxy_3

Note: if the codis-ha service does not take effect after restarting one codis-proxy, you need to restart three codis-proxy

Stop the zookeeper of any node and check whether the codis-proxy,dashboard is normal >

Test the server with zookeeper status as leader (linux-node2)

Check the zookeeper status to confirm whether it is leader.

[root@linux-node2 ~] # / opt/zookeeper/bin/zkServer.sh status

Stop the zookeeper node on the linux-node2 and let other zookeeper nodes automatically select the leader node and provide services

[root@linux-node2 ~] # / opt/zookeeper/bin/zkServer.sh stop

# randomly detect the zookeeper status of other nodes, detect that the zookeeper node of linux-node2 has been selected as leader, and start to provide services

# restart refresh codis dashboard and you can see that dashboard is normal.

Check whether the redis client can access codis-proxy properly

Take linux-node3 as an example:

[root@linux-node3 ~] # cd / opt/codis/src/github.com/CodisLabs/codis/extern/redis-2.8.21/src/

# create a new scripts directory to store common redis commands

[root@linux-node3 src] # mkdir-p / opt/codis/scripts/ [root@linux-node3 src] # cp redis-cli redis-server redis-benchmark redis-check-dump / opt/codis/scripts/

From the above, we can see that both of my previous key of set can get to

Stop the redis master instance in any group and check if redis slave automatically switches to the redis-master 6379 port above the master analog linux-node1 (192.168.1.148) hangs up

Log in to the node where codis-ha is located (linux-node2) to view the codis-ha log and print as follows

Open the dashboard interface and you can see the following:

From the figure, you can see that the salve node of group_3 has automatically switched to the master state.

The client writes data to check whether new key is added to the switched redis master instance.

# add key to linux-nod3

Refresh the dashboard interface to see the key changes

# restore port 6379 of redis instance on linux-node1

[root@linux-node1 ~] # / opt/codis/bin/codis-server / opt/codis/redis_conf/6379.conf & [root@linux-node1 ~] # ps-ef | grep-v grep | grep codis-serverroot 16620 13759 0 Aug27 pts/0 00:04:52 / opt/codis/bin/codis-server *: 6380root 27574 13759 2 19:40 pts/0 00:00:00 / opt/codis/bin/codis-server *: 6379

Refresh the dashboard interface to view the status changes of redis instances in group

You can see that the redis instance with port 6379 on linux-node1 has been started and automatically becomes the slave in group_3

Note: if maste is hung up, redis-ha detection will automatically switch slave to master, but after master is restored, if the status is still offline, you can remove the redis node from the group and add it again.

2. Error reporting and resolution:

/ opt / bin/zkCli.shdelete / zk/codis/db_test/dashboard

# restart codis dashboard

Nohup / usr/local/codis/bin/codis-config-c / usr/local/codis/config.ini dashboard & read here, this article "how to deploy Codis cluster service in Centos system" has been introduced. If you want to master the knowledge points of this article, you still need to practice and use it yourself. If you want to know more about related articles, welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.