Shulou(Shulou.com)05/31 Report--

When master down is dropped, pt-heartbeat keeps retrying the reasons and solutions that will lead to slow memory growth. I believe many inexperienced people are at a loss about this. Therefore, this article summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

Recently, my colleague reported that in the process of using pt-heartbeat to monitor master-slave replication latency, if master down is dropped, pt-heartbeat will fail to connect, but will keep retrying.

There is nothing wrong with retrying this. After all, from the user's point of view, I hope pt-heartbeat will try again and again until you reconnect to the database. However, they found that constant retries lead to slow memory growth.

Reproduce

Environment:

Pt-heartbeat v2.2.19 MySQL Community Edition v5.6.31 Perl v5.10.1 Magi RHEL 6.7, 500m memory

In order to avoid the impact of database startup and shutdown on pt-heartbeat memory utilization, MySQL and pt-heartbeat are running on different hosts.

Run pt-heartbeat

# pt-heartbeat-- update-h 192.168.244.10-u monitor-p monitor123-D test-- create-table

Monitor the memory usage of pt-heartbeat

Get pid

# ps-ef | grep pt-heartbeatroot 1505 1471 0 19:13 pts/0 00:00:08 perl / usr/local/bin/pt-heartbeat-- update-h 192.168.244.10-u monitor-p monitor123-D test-- create-tableroot 1563 1545 2 19:50 pts/3 00:00:00 grep pt-heartbeat

View the memory usage of the process

# top-p 1505

Running 0 15.00 (TIME+ column), MEM has been stable at 3.3%.

Now close the database.

# service mysqld stop

The pt-heartbeat command just now keeps outputting the following information

After the same CPU time, MEM increased to 4.4%, an increase of 1%. Considering the memory of 500m, the memory footprint of the process increased by 5m, which is not much, but considering that the memory increase of the process does not mean to stop, this phenomenon should be noted.

At the same time, through the pmap command, it is found that the RSS and Dirry of 0000000001331000 address will also grow, and the growth rate is 4k/s.

Later, when I studied the source code of pt-heartbeat, I found that the code was a little bug.

My $tries = 2bot while (! $dbh & & $tries--) {PTDEBUG & & _ d ($cxn_string,'', $user,'', $pass,join (',', map {"$_ = > $defaults- > {$_}"} keys% $defaults)); $dbh = eval {DBI- > connect ($cxn_string, $user, $pass, $defaults)} If (! $dbh & & $EVAL_ERROR) {if ($EVAL_ERROR = ~ m/locate DBD\ / mysql/i) {die "Cannot connect to MySQL because the Perl DBD::mysql module is". "not installed or not found. Run 'perl-MDBD::mysql' to see". "the directories that Perl searches for DBD::mysql. If" "DBD::mysql is not installed, try:\ n". Debian/Ubuntu apt-get install libdbd-mysql-perl\ n. RHEL/CentOS yum install perl-DBD-MySQL\ n. "OpenSolaris pgk install pkg:/SUNWapu13dbd-mysql\ n";} elsif ($EVAL_ERROR = ~ m/not a compiled character set | character set utf8/) {PTDEBUG & & _ d ('Going to try again without utf8 support'); delete $defaults- > {mysql_enable_utf8};} if (! $tries) {die $EVAL_ERROR;}

The above code is extracted from the get_dbh function and is used to obtain the connection to the database. If the acquisition fails, retry once, and then exit with an exception thrown by the die function.

However, by setting the following breakpoint, it is found that the PTDEBUG & & _ d ("$EVAL_ERROR") statement in the if function can be executed when $tries is 0, but the die function simply does not throw an exception and exits the script

PTDEBUG & & _ d ($tries); if (! $tries) {PTDEBUG & & _ d ("$EVAL_ERROR"); die $EVAL_ERROR;}

Later, the last if function of the above code is modified as follows:

If (! $tries) {die "test:$EVAL_ERROR";}

Test again

Start the database

# service mysqld start

Execute the pt-heartbeat command

# pt-heartbeat-- update-h 192.168.244.10-u monitor-p monitor123-D test-- create-table

Stop the database

# service mysqld stop

The pt-heartbeat command just executed exited abnormally

"test:" is the added test character.

Oddly enough, a simple die $EVAL_ERROR will not throw an exception and exit the script, but the modified die "test:$EVAL_ERROR" will exit the script.

Obviously, this is indeed a bug, and I don't know if it has anything to do with the version of perl.

Curiously, how does a failed connection lead to the continuous growth of memory?

After reading the above, do you understand the reasons and solutions for the slow memory growth that pt-heartbeat keeps retrying when master down is dropped? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.