Shulou(Shulou.com)06/01 Report--

Preface

Mycat is a well-known mysql database middleware in the industry, and its functions are rich in all kinds of mysql middleware. The related components of mycat are written in Java language and are easy to deploy. Its functions are: opaque proxy, read-write separation, load balancing, data slicing and so on. And because the underlying connection mode uses the jdbc component of java to connect to the database, it supports sql server,oracle,pgsql in theory, but the function is mainly focused on mysql.

Advantages: simple deployment, powerful function, high flexibility, make up for some of the lack of functions of mysql

Disadvantages: functional development is not very perfect, we need to pay attention to stepping on the pit.

According to the principle, if mycat does not do shredding and is purely an agent, what he does is actually more data forwarding, and this forwarding capability, of course, depends on how strong he can be.

Since the forwarding ability is very strong in theory, it can be ignored, of course not, so we need to use the test data of directly connected mysql to make a comparison.

Preparation before test

The server is configured with 32-core cpu (virtualized value, including hyper-threading), 120g memory, 16000iops storage device, specific allocation:

10.21.128.156:mycat1.6.5,sysbench0.4

10.21.128.157:mysql5.7.20 main library

10.21.128.158:mysql5.7.20 from Library 1

10.21.128.159:mysql5.7.20 from Library 2

Briefly explain the topological relationship:

The first step, of course, is to install mysql. We will not describe how to install it in detail here. However, the parameters of my.cnf are somewhat different. The main reason is to adapt to the high concurrent pressure testing environment, otherwise it will be limited by the parameters, and then the program will exit. Of course, if you want to simulate the online environment as much as possible, then you have to think about these restrictions, and you need to be careful to change. I just give an example here for reference.

# first of all, it is necessary to match the number of connections to the system and the number of open files. If ulimit-SHn 6553please wants to take effect permanently, it will be like this: echo "* soft nofile 65535 * hard nofile 65535root soft nofile 65535root hard nofile 65535" > > / etc/security/limits.conf.

Then, change the mysql configuration file parameters, other buffer_pool and so on will not be listed, please depend on the situation settings, here only describes the parameters related to stress testing.

# Open the configuration file vim / usr/local/mysql/my.cnf# other parameters we temporarily ignore, only look at these parameters related to stress testing [mysqld] # each machine needs a different ID, server-id = 12815 global maximum number of open files in the master-slave environment, which cannot exceed the maximum number of files set by the system Otherwise, invalid open_files_limit = 65530#innodb engine limits the number of file descriptors for opening tablespace files .ibd at one time innodb_open_files = 6553tables allowed to store in the cache. Naturally, the stress test needs to open a lot of tables, just set it a little larger. The maximum number of connections available for table_open_cache = 6500 instances must be set large enough due to the system limit and the above two parameters, which must be set large enough. It is troublesome to test that the error does not pass. At least it is more than one-third of the maximum number of connections per user in max_connections=30000#. Limit the maximum number of connections per user. It must be a little less than the size of the max_user_connections=20000# most big data package above. It is still necessary to make the pressure test larger. Max_allowed_packet = 128M# limits the upper limit of prepared statements in all session on mysqld at the same time. Usually, you don't have to worry about this parameter. But the pressure test needs to increase the max_prepared_stmt_count=100000 a little bit.

I will not enumerate all the other parameters, just set them according to the situation, and then restart and stand by.

Software installation

First of all, let's talk about the choice of stress testing tools. Mycat is not compatible with tpcc on the MySQL protocol, so give up tpcc. Then sysbench2.0 is not compatible with mycat, and there are many unexplained stress test failures, so it can only be abandoned.

Finally, sysbench0.4 and 0.5 are selected to use, the results can be measured smoothly, and the principle of pressure test is more objective.

Therefore, the software we need to install are: mysql (installed by default), mycat,sysbench0.4 or 0.5

How to install and license mysql, I won't go into details here. Please build it yourself. Add the configuration file mentioned above.

Install mycat:

1) build jdk environment

Since mycat is a java program, you need to install JDK, and the version must be at least jdk1.6 or above

Download the java language package

The download address of Java is changing all the time, so you'd better go up and watch it and download it.

Http://download.oracle.com/otn-pub/java/jdk/8u151-b12/e758a0de34e24606bca991d704f6dcbf/jdk-8u151-linux-x64.tar.gz?AuthParam=1513326216_bcf60226458d67751e1d8d1bbe6689b4

# after downloading the installation package Decompress create soft connection tar xf jdk-8u144-linux-x64.tar.gzmv jdk1.8.0_144/ / usr/local/ln-sf jdk1.8.0_144/ jdk# create environment variable vim / etc/profile.d/java.shexport JAVA_HOME=/usr/local/jdkexport JRE_HOME=/usr/local/jdk/jreexport CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/libexport PATH=$PATH:$JAVA_HOME/bin# Overload the environment variable source / etc/profile# to verify the installation of java-versionjava version "1.8.0,144" Java (TM) SE Runtime Environment (build 1.8.0_144-b01) Java HotSpot (TM) 64-Bit Server VM (build 25.144-b01) Mixed mode)

The installation is complete and can be used.

2) install and configure mycat

Students who know the java program should know that as long as the java program is configured, it can be used directly, so there is no concept of installation, just configure the mycat and use it.

We download the version of mycat1.6.5, which is the latest official version.

Http://dl.mycat.io/1.6.5/Mycat-server-1.6.5-release-20180122220033-linux.tar.gz

# unzip the installation package tar xf Mycat-server-1.6.5-release-20180122220033-linux.tar.gz# and move it to the destination location mv mycat1.6.5/ usr/local/# to create a soft connection ln-sf mycat1.6.5/ mycat# to also do the command soft connection ln-sf / usr/local/mycat/bin/* / usr/bin/

Then you can start to change the configuration file.

Here, the test of sysbench does not involve sub-database and sub-table, so mycat only needs to set server.xml and schema.xml. The specific storage path of mycat is not specified, so I put it in / usr/local.

= = start of split line = =

Some people may wonder that mycat supports sub-database and sub-table, so can the stress test be carried out for sub-database sub-table? In fact, it is feasible in theory, those who intend to do the pressure test of sub-database and sub-meter should do some processing of the data, according to the principle of sysbench, it is feasible, the number of tables he tests is controllable, you can assemble the numbered tables into a new logical table through logical library processing.

= end of split line =

First, take a look at the basic environment settings server.xml. Most of the settings can be left unchanged. Note that you can add and modify the configuration under the tag. For each user you create, you need a separate tag:

Vim / usr/local/mycat/conf/server.xml

2880000 134217728 utf8 2 600 mycat123 sbtest,testppp sb123 sbtest test sbtest true

Set some project parameters related to the pressure test, and create three user root,sysbench,test. These three users are not associated with the users of the database and are independent. Even if the user's password is cracked, the database password is still secure. Among them, root has full control rights, sysbench can only control sbtest libraries, test can only control sbtest libraries, and limit read and write permissions.

Then set the logic library configuration file schema.xml. There are many changes here, so paste the whole file directly:

Select user ()

Set up two logical libraries, sbtest and testppp (with the same name as the real database, but can be different), point to the master-slave environment dn1 and dn2 of the real database, with one master, two slaves and three real server addresses, and enable read-write separation. However, it should be noted that transactions only go to the main library, so the best solution for read-write separation is better by the program, and it is inevitable to do it with middleware. After all, many development frameworks are transactional now.

3) start mycat

# start mycatMycat start# and restart mycatMycat restart# to disable mycatMycat stop# while the default connection port of Mycat is 8066 and the management port is 9066. You can modify .ss-ntplu in server.xml | grep javatcp LISTEN 0 100: 9066: * users: ("java", pid=115779 Fd=168)) tcp LISTEN 0100:: 8066:: * users: (("java", pid=115779,fd=172)) # if you see the port up, you can use it. Because mycat supports mysql native protocol, connecting to port 8066 will not be any different from normal mysql, and you can use it directly. # if you change any configuration file You can log in to the management port and execute the following command to refresh the configuration in real time, which is quite convenient. MySQL-uroot-pmycat123-h20.21.128.156-P9066 # change the configuration of server can Mysql > reload @ @ config# if you change the configuration of schema, you need this command Mysql > reload @ @ config_all

After loading, you can use the new configuration, even if you change the login user name.

Set up sysbench

After downloading, you can compile and install as long as you have the c runtime, but you need to create a soft connection to the mysql library file, otherwise the library file will not be found by error

# install the dependency package yum install-y automake libtool# first create a soft connection to the mysql library file Otherwise, ln-sf / usr/local/mysql/lib/libmysql* / usr/lib64ln-sf / usr/local/mysql/lib/libmysqlclient.so / usr/lib64/libmysqlclient_r.so# will be compiled, and then execute: tar xf sysbench-0.4.12-1.1.tgzcd sysbench-0.4.12-1.1./autogen.sh./configure-- with-mysql-includes=/usr/local/mysql/include-- with-mysql-libs=/usr/local/mysql/libMake# if make does not report an error. The binary command line tool sysbenchcd / tmp/sysbench-0.4.12-1.1/sysbenchls-l sysbench-rwxr-xr-x 1 root root 3293186 Sep 21 16:24 sysbench will be generated under the sysbench directory

Here, the environment is built.

Start testing.

When the environment is ready, the test begins. Before the test, you need to prepare the test data, use the command to create it, and take a long time (maybe more than half a day). The key point is to pay attention to whether the hard disk space is enough:

# to avoid unnecessary errors, directly connect to the database operation mysql-uroot-p128157-h20.21.128.157-P330database to create a test database. The default sysbench test database name is sbtest, or you can specify the library name mysql > create database sbtest # enter the sysbench program directory cd / tmp/sysbench-0.4.12-1.1/sysbench# to start creating data.. / sysbench--mysql-host=10.21.128.157-- mysql-port=3306-- mysql-user=root-- mysql-password=128157-- test=tests/db/oltp.lua-- oltp_tables_count=15-- oltp-table-size=40000000-- mysql-table-engine=innodb-- rand-init=on prepare# emphasizes that it may take a long time and requires patience.

Parameter resolution:

-- test=tests/db/oltp.lua means calling the tests/db/oltp.lua script to test the oltp pattern, but not parsing what oltp is.

-- oltp_tables_count=15 indicates that 15 test tables will be generated. The more the number, the longer the natural time.

-- oltp-table-size=40000000 indicates that each test table is populated with 40000000 rows of data, and the more the number, the longer the time.

-- rand-init=on indicates that each test table is populated with random data, which is objective.

Mysql-table-engine=innodb indicates that the engine of the table is innodb

Prepare is used to prepare the data needed for the test. After the preparation, run is executed to test. If you need to clean up the data after the test, use cleanup to clear the test data.

= start of split line = =

So there are 15 tables created here, each table has 40 million rows of data, the data is randomly input, this amount of data needs about 150G hard disk space, it is estimated to cover most of the situation, of course, you can also create more data.

The data structure is roughly like this, for reference only:

Mysql > show tables;+-+ | Tables_in_sbtest | +-+ | sbtest1 | | sbtest2 | | sbtest3 |. . | | sbtest15 | +-+ 5 rows in set (0.00 sec) mysql > select * from sbtest1 limit 1 | +-+- -+-+ | id | k | c | | pad | +-+ -- +- -+ | 1 | 2507308 | 68487932199-96439406143-93774651418-41631865787-96406072701-20604855487-25459966574-28203206787-41238978918-19503783441 | 22195207048-70116052123-74140395089-76317954521-98694025897 | -+- -+ 1 row in set (0.01 sec)

= end of split line =

Then we are ready for formal testing, let's first test the common master-slave architecture:

# enter the sysbench program directory cd / tmp/sysbench-0.4.12-1.1/sysbench# to execute the test command. / sysbench--mysql-host=10.21.128.157-- mysql-port=3306-- mysql-user=root-- mysql-password=128157-- test=tests/db/oltp.lua-- oltp_tables_count=15-- oltp-table-size=40000000-- mysql-table-engine=innodb-- num-threads=1024-- oltp-read-only=off-- report-interval=10-- rand-type=uniform-- max-time=3600-- Max-requests=0-- percentile=99 run > > / tmp/sysbench_oltpX_32_20171113-1.log# We set it for 3600 seconds Before the termination, that is, the test fails, the error report needs to be analyzed.

Parameter resolution:

-- num-threads=1024 indicates that 1024 concurrent connections are initiated

-- oltp-read-only=off means no read-only test, that is, a mixed read-write test.

-- report-interval=10 indicates that a test progress report is output every 10 seconds

-- rand-type=uniform indicates that the random type is fixed, and there are several other optional random modes: uniform (fixed), gaussian (Gaussian), special (specific), pareto (Pareto)

-- max-time=3600 indicates that the maximum execution time is 3600 seconds, after which the test will end.

-- max-requests=0 indicates that the total number of requests is 0, because the total execution time has been defined above, so the total number of requests can be set to 0, or only the total number of requests can be set without setting the maximum execution time.

-- percentile=99 means to set the sampling ratio. Default is 95%, that is, discard 1% of long requests and take the maximum value in the remaining 99%.

According to the above analysis, the final output to a file, after all, need to be recorded.

Then go to the mycat proxy environment:

. / sysbench-- mysql-host=10.21.128.156-- mysql-port=8066-- mysql-user=sysbench-- mysql-password=sb123-- test=tests/db/oltp.lua-- oltp-table-size=40000000-- mysql-table-engine=innodb-- num-threads=1024-- oltp-read-only=off-- report-interval=10-- rand-type=uniform-- max-time=3600-- max-requests=40000000-- percentile=99 run > > / home/logs/sysbench_oltpX_32_20171113-2.log

As above, the concurrency is set to 1024 threads, the test time is 3600 seconds, that is, 1 hour, and the 15 tables with 40 million rows just created are also used to output the data with an average sampling value of 99% to another log file.

Read the test report

When the test is over, let's take a look at the results. Let's analyze one of the results and say:

# ignore some statistics per 10 seconds and just look at the final statistics vim / tmp/sysbench_oltpX_32_20171113-1.logsysbench 0.5: multi-threaded system evaluation benchmarkRunning the test with following options:Number of threads: 1024Report intermediate results every 10 second (s) Random number generator seed is 0 and will be ignoredThreads startedmusic-report the test results every 10 seconds Tps, read per second, write per second, response time statistics over 99% [10s] threads: 1024, tps: 364.69, reads/s: 6324.66, writes/s: 1765.66, response time: 4292.46ms (99%) [20s] threads: 1024, tps: 475.80, reads/s: 6037.91, writes/s: 1640.10, response time: 4088.05ms (99%) [30s] threads: 1024, tps: 439.40, reads/s: 6349.45, writes/s: 1808.89 Response time: 3248.44ms (99%) [40s] threads: 1024, tps: 487.70, reads/s: 6438.46, writes/s: 1879.72, response time: 4385.98ms (99%) [50s] threads: 1024, tps: 395.70, reads/s: 6498.99, writes/s: 1849.00, response time: 3845.88ms (99%) . . [3560s] threads: 1024, tps: 385.80, reads/s: 4949.60, writes/s: 1503.80, response time: 19827.73ms [3570s] threads: 1024, tps: 249.70, reads/s: 3679.90, writes/s: 1009.40, response time: 12016.58ms (99%) [3580s] threads: 1024, tps: 328.90, reads/s: 4511.40, writes/s: 1301.40, response time: 7419.06ms (99%) [3590s] threads: 1024, tps: 196.40 Reads/s: 3058.90, writes/s: 815.30, response time: 12092.35ms [3600s] threads: 1024, tps: 386.60, reads/s: 5282.74, writes/s: 1537.78 Response time: 13614.18ms OLTP test statistics: queries performed: read: 16913554-- Total number of reads write: 4832444-Total number of writes other: 2416222-- Total number of other operations (SELECT, INSERT, UPDATE, DELETE) For example, COMMIT, etc.) total: 24162220-- Total total transactions: 1208111 (335.29 per sec.)-- Total transactions (transactions per second) deadlocks: 0 (335.29 transactions.)-- Total deadlocks occurred Number of read/write requests: 21745998 (6035.29 per sec.)-- Total number of reads and writes (reads and writes per second) other operations: 2416222 (670.59 per sec.)-Total number of other operations (other operations per second) General statistics: total time: 3603.1388s- -Total time-consuming total number of events: 1208111-- total number of transactions total time taken by event execution: 3688348.3797s-- Total time spent by all transactions (regardless of parallelism) response time: min: 28.41ms-- minimum time-consuming avg: 3052.99ms-- average time-consuming max: 48667.93ms-- longest time-consuming approx. 99 percentile: 12708.40ms-average time spent over 99% Threads fairness: events (avg/stddev): 1179.7959 execution time 29.07 execution time (avg/stddev): 3601.9027max 1.01

Therefore, the number of transactions per second Tps reaches 335.29, the number of queries per second Qps reaches 6035.29, and the average delay 3052.99ms.

After reading the analysis, let's take a look at the results. Here are the results of directly measuring the master-slave environment:

OLTP test statistics: queries performed: read: 19215238 write: 5490068 other: 2745034 total: 27450340 transactions: 1372517 (381.03 per sec.) Deadlocks: 0 (0.00 per sec.) Read/write requests: 24705306 (6858.48 per sec.) Other operations: 2745034 (762.05 per sec.) General statistics: total time: 3602.1538s total number of events: 1372517 total time taken by event execution: 3688254.2686s response time: min: 18.33ms avg: 2687.22ms max: 55386.15ms approx. 99 percentile: 12444.89msThreads fairness: events (avg/stddev): 1340.3486 execution time 33.85 (avg/stddev): 3601.8108Comp0.44

The number of transactions per second Tps reaches 381.03, the number of queries per second Qps reaches 6858.48, the average time-consuming 2687.22ms, after all, the total data volume is 600 million, or should be accepted.

Here are the results through the mycat proxy

OLTP test statistics: queries performed: read: 18078326 write: 5165236 other: 2582618 total: 25826180 transactions: 1291309 (358.40 per sec.) Deadlocks: 0 (0.00 per sec.) Read/write requests: 23243562 (6451.19 per sec.) Other operations: 2582618 (716.80 per sec.) General statistics: total time: 3602.9883s total number of events: 1291309 total time taken by event execution: 3687715.5739s response time: min: 22.45ms avg: 2855.80ms max: 50326.08ms approx. 99 percentile: 13264.21msThreads fairness: events (avg/stddev): 1261.0439 execution time 33.56 (avg/stddev): 3601.2847

The number of transactions per second Tps reaches 358.40, the number of queries per second Qps reaches 6451.19, the average time-consuming 2855.80ms, also is the total data volume of 600 million.

From the comparison of the results, we can see that after using mycat, tps,qps and time consumption are both reduced by 6% and 7%, which is acceptable to me personally, because using mycat as the proxy layer, you can easily manage the backend database, and any switch can be done manually. After using the trigger script, it is a HA framework.

Extraneous explanation

The example shown in the test results shows that the test results with highly available keepalived are added. The number of transactions per second Tps reaches 335.29, the number of queries per second Qps reaches 6035.29, and the average delay 3052.99ms is good, which is generally good, but it is only within the range of 10% master-slave loss of pure mysql, which is acceptable. After all, the availability is much higher, and the number of mycat can be increased at any time when the subsequent pressure increases.

With more functions and more machines, it should be noted that because of the network packet forwarding relationship, keepalived and back-end mycat cannot be on the same server, so they must be independent.

10.21.128.208:keepalived master

10.21.128.209:keepalived standby

10.21.128.200:mycat1

10.21.128.201:mycat2

10.21.128.199:vip

Briefly explain the topological relationship:

Need to build an extra mycat, but I won't go into details here, just copy a copy of the configuration to another machine and then boot it to ok.

Then build a set of keepalived clusters:

In most cases, people only use keepalived to do the HA function, while the LVS function is implemented by other software. But in fact, keepalived+ipvsadm can realize both HA function and LVS function, which is very convenient.

We installed keepalived and modified the configuration on 10.21.128.208 and 10.21.128.209.

Individuals do not want to struggle with functionality and version problems, and these are relatively normal functional software, most of the yum sources are equipped, so it is very convenient to install directly with yum, and those who have extra interest can slowly compile

# install the required software yum install-y keepalived ipvsadm nc rsync telnet tcpdump wget

Installation is quick, let's look directly at the configuration, the default configuration file for yum installation is / etc/keepalived/

# modify the configuration file and pay attention to the comment location vim / etc/keepalived/keepalived.conf! Configuration File for keepalived# Global configuration global_defs {notification_email {root@localhost # define recipient email address} notification_email_from root@localhost # define Sender smtp_server 127.0.0.1 # if you want to use a third-party smtp server It is almost meaningless in reality (the reason that needs to be verified). Set it to smtp_connect_timeout 30 # smtp timeout router_id LVS_HA_MYCAT1 # the ID of this server keepalived, change it at will, note that different servers are different.} # vrrp configuration (HA configuration) vrrp_instance VI_1 {# defines a virtual route, and VI_1 is the identifier of the virtual route. Define the name state MASTER # specify the network interface of interface eth0 # binding virtual IP on the primary node standby node of the current node. Note the virtual_router_id 19 # VRRP group name of the internal and external network. The settings of the two nodes must be the same. To indicate that each node belongs to the same VRRP group, and you can use 0-255. this ID is also the source of the last segment of the virtual MAC, priority 100. the initial priority is between 1 and 254. the master node must be the largest, and other slave nodes must reduce the interval between advert_int 1 # multicast messages, and the settings of the two nodes must be the same authentication {# set authentication information. The two nodes must be consistent auth_type PASS auth_pass 199200} virtual_ipaddress {10.21.128.199 # specify the VIP, the settings of the two nodes must be the same, and the virtual ip is preferably on the same network segment as the real ip. }} # load balancer configuration (LVS configuration) virtual_server 10.21.128.199 8066 {# specify VIP and port. Vip is the above setting that delay_loop 6 # delays the number of cycles before starting the service. For service detection lb_algo rr # load balancer scheduling algorithm lb_kind DR # load balancer type selection, you can choose DR | NAT | the mask of nat_mask 255.255.255.0 # vip with high performance persistence_timeout 0 # session persistence time. If the user does not respond within a certain period of time, the next user request needs to be rerouted, which is generally set to 0, which is not needed. The protocol used by protocol TCP # is generally TCP real_server 10.21.128.200 8066 {# define the real server attributes of the backend realserver, ip and port weight 1 # load balance weight, the higher the value, the more burden the connection MISC_CHECK {# defines the heartbeat detection method, because it is not web, and the mycat behind the tcp_check health test will report an error. So need misc_check way to do heartbeat detection misc_path "/ etc/keepalived/shell/check_mycat_status.sh 10.21.128.200 8066" # Custom heartbeat detection shell script path, detected server ip, detected port. (required) misc_timeout 3 # script execution timeout}} real_server 10.21.128.201 8066 {# ditto weight 2 MISC_CHECK {misc_path "/ etc/keepalived/shell/check_mycat_status.sh 10.21.128.201 8066" misc_timeout 3}} real_server 10. 21.128.156 8066 {# Ibid. Weight 2 MISC_CHECK {misc_path "/ etc/keepalived/shell/check_mycat_status.sh 10.21.128.156 8066" misc_timeout 3}

Special description of the load balancing scheduling algorithm parameter lb_algo, because different algorithms have different effects, if you want the architecture to be stable, it is necessary to find the most suitable algorithm.

Rr: round robin scheduling (Round Robin) algorithm that sends connections to each real_server in turn, regardless of the actual number of connections on the server and the system load. The advantage is that every real_server must have an operation, but the disadvantage is that long and short connections are easily unevenly distributed.

Wrr: weighted round robin scheduling (Weighted Round Robin) algorithm is roughly the same as rr, but there is a weight judgment in the rule. The real_server distributes more tasks by the parameter weight, which has the same advantages and disadvantages as rr.

Lc: minimum connection scheduling algorithm that assigns new connection requests to the real_server with the smallest number of connections. The advantage is that the connection requests can be distributed relatively evenly, and the disadvantage is that a large number of local short connections may hit a node.

Wlc: weighted minimum connection scheduling algorithm, which is roughly the same as lc, and has the same advantages and disadvantages. There is an extra weight judgment in the rule, and which real_server distributes more tasks by the parameter weight.

Lblc: a locality-based least link scheduling algorithm that dispatches requests from the same destination IP address to the same server when the load is basically relatively balanced. The advantage is that it can improve the cache hit rate, but the disadvantage is that it doesn't make much sense, because imbalance is failure.

Lblcr: locality-based minimum links with replication, which is basically the same as the LBLC algorithm, but the algorithm is more complex to achieve higher cache availability. The disadvantage is that it is of little significance for hot data, and this algorithm is rarely used.

Dh: target address hash scheduling. Through the static mapping algorithm, the Hash function maps a target IP address to real_server. The advantage is that an IP address will be permanently connected to a server for a long time, but the disadvantage is that it is easy to make requests unbalanced.

Note that in 1:LVS+DR mode, only IP forwarding is supported, but port forwarding is not supported, that is, the port must be the same in the virtual_server and real_server configuration nodes of keepalived.conf.

Note 2: configuring the load balancer mode strongly recommends that VIP and the real backend server IP share the same network segment, otherwise some strange situations will occur and the loss will outweigh the gain.

It involves a script to detect the heartbeat, as follows:

Cat / etcswap keepalivedamp then exit exit 1fi#; exit 1fi# remember to grant execution permission to the script: chmod 755 check_mycat_status.sh

The meaning is very simple, that is, check the mycat port of the real server through the nc command, and if not, keepalived removes the mycat from the cluster.

Then, you need to do an operation on the real backend server and bind vip to create ipvs rules:

Create and execute the following script on 10.21.128.200, 10.21.128.201, 10.21.128.156

# create rule script vim / shell/realserver #! / bin/bashSNS_VIP=10.21.128.199#/etc/rc.d/init.d/functionscase "$1" instart) ifconfig lo:0 $SNS_VIP broadcast $SNS_VIP netmask 255.255.255.255 up sleep 5 / sbin/route add ${SNS_VIP} / 32 dev lo:0 echo "1" > / proc/sys/net/ipv4/conf/lo/arp_ignore Echo "2" > / proc/sys/net/ipv4/conf/lo/arp_announce echo "1" > / proc/sys/net/ipv4/conf/all/arp_ignore echo "2" > / proc/sys/net/ipv4/conf/all/arp_announce sysctl-p > / dev/null 2 > & 1 echo "RealServer Start OK" Stop) ifconfig lo:0 down / sbin/route del-net $SNS_VIP netmask 255.255.255.255 dev lo:0 echo "0" > / proc/sys/net/ipv4/conf/lo/arp_ignore echo "0" > / proc/sys/net/ipv4/conf/lo/arp_announce echo "0" > / proc/sys/net/ipv4/conf/all/arp_ignore echo "0" > / proc/ Sys/net/ipv4/conf/all/arp_announce echo "RealServer Stoped" ; restart) / sbin/route del-net $SNS_VIP netmask 255.255.255.255 dev lo:0/sbin/route add ${SNS_VIP} / 32 dev lo:0 *) echo "Usage: $0 {start | stop | restart}" exit 1esacexit permission chmod 755 / shell/realserver# execute / shell/realserver start# and then check the status of the network card ifconfig lo:0lo:0 Link encap:Local Loopback inet addr:10.21.128.199 Mask:255.255.255.255 UP LOOPBACK RUNNING MTU:16436 Metric:1

It should be noted that this binding must be started every time the server is restarted, otherwise the lvs will not take effect, for example, put it in / etc/rc.d/rc.local

Echo "/ shell/realserver start" > > / etc/rc.d/rc.local

Everything is ready, only Dongfeng, then start keepalived.

Execute on 10.21.128.208 and 10.21.128.209

# because it is a yum installation So there is a linux service / etc/init.d/keepalived startStarting keepalived: [OK]-# if it is a centos7 or a system above ubuntu14 You need the following command systemctl start keepalivedsystemctl status keepalivedsystemctl stop keepalivedsystemctl reload keepalived----# to look at vip. Because the master is 208, vip will only appear in 208s. Unless the ip addr 1: lo: mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00 link/ether 22:87:35:77:ef:38 brd ff:ff:ff:ff:ff 00 brd 00 00 brd 00 00 inet 127.0.0.1 scope host valid_lft forever preferred_lft forever2:: 1 mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 22:87:35:77:ef:38 brd ff:ff:ff:ff:ff: Ff inet 10.21.128.208 ff inet 24 brd 10.21.128.255 scope global eth0 inet 10.21.128.199 inet6 fe80::2087:35ff:fe77:ef38/64 scope link valid_lft forever preferred_lft forever# 32 scope global eth0 #-VIP in this inet6 fe80::2087:35ff:fe77:ef38/64 scope link valid_lft forever preferred_lft forever# and then look at the current LVS status ipvsadm-LIP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 10.21.128.199Route 8066 rr-> 10.21.128.156 Route 8066 Route 100-> 10.21.128.200 Route 100-> 10.21.128.200

Note: if abnormal IP and port are detected, the IP address cannot be seen under the ipvsadm-L command, because it was actively logged off by keepalived. At this time, you should check the reason why the relevant IP and port failed.

To see the IP address and status of the connection, you can use the following command.

Ipvsadm-LncIPVS connection entriespro expire state source virtual destinationTCP 14:54 ESTABLISHED 10.237.140.90 ESTABLISHED 31472 10.21.128.1998066 10.21.128.200:8066TCP 01:27 FIN_WAIT 10.237.140.90 FIN_WAIT 10.21.128.199 ESTABLISHED 8066 10.21.128.2018066

You can see the status of the connection by looking at the value of state.

Everything is fine, so we can stress test.

# similar to the previous command, except that ip is different. / sysbench-- mysql-host=10.21.128.199-- mysql-port=8066-- mysql-user=sysbench-- mysql-password=sb123-- test=tests/db/oltp.lua-- oltp_tables_count=15-- oltp-table-size=40000000-- mysql-table-engine=innodb-- num-threads=1024-- oltp-read-only=off-- report-interval=10-- rand-type=uniform-- max-time=3600-- max-requests=40000000-- percentile=99 run > > / home/logs/sysbench_oltpX_32_20171128-3.log

Let's take a look at lvs status

# check the link status ipvsadm-LIP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 10.21.128.199 RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 8066 rr-> 10.21.128.156 size=4096 8066 Route 2341 340-> 10.21.128.200 8066 Route 1341 340-> 10.21.128 .201pur8066 Route 2 342 340 # add one more-- stats parameter You can see the traffic data ipvsadm-L-- statsIP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes-> RemoteAddress:PortTCP 10.21.128.1998066 2045 3658545 0326481K 0-> 10.21.128.156 Vista 8066 682 1131691 0114278K 0-> 10.21.128.2008066 1266405 0106344K 0-> 10.21.128.201pur8066682 1260449 0105858K 0

Why is there no out? Isn't that weird? Because our load balancer chooses DR mode, the characteristic of this mode is that when the client establishes a link with the real server, the real server will send the data directly to the client, and there is no need for keepalived to transfer, so there is no out traffic, that is, the reason why it is more efficient.

Then let's take a look at the results of the stress test.

OLTP test statistics: queries performed: read: 17043978 write: 4869708 other: 2434854 total: 24348540 transactions: 1217427 (338.00 per sec.) Deadlocks: 0 (0.00 per sec.) Read/write requests: 21913686 (6084.05 per sec.) Other operations: 2434854 (676.01 per sec.) General statistics: total time: 3601.8265s total number of events: 1217427 total time taken by event execution: 3687506.3802s response time: min: 30.17ms avg: 3028.93ms max: 75102.41ms approx. 99 percentile: 13757.56msThreads fairness: events (avg/stddev): 1188.8936 execution time 33.40 (avg/stddev): 3601.0804

The number of transactions per second Tps reaches 338.00, the number of queries per second Qps reaches 6084.05, and the average delay 3028.93ms is almost the same as the beginning, which is basically consistent with the reality.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.