MySQL, a phenomenon of hang death that makes you doubt your life. 03/29 Update SLTechnology News&Howtos

MySQL, a phenomenon of hang death that makes you doubt your life.

2025-03-29 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Database >

Shulou(Shulou.com)06/01 Report--

1. Failure phenomenon A customer reported that when using the parsed contents of the binlog log to do a point-in-time recovery, hang died (more than 12 hours), the recovery process could not continue, and 100% reappeared.

The implementation of point-in-time recovery is to specify an end time point with the mysqlbinlog command, parse the binlog and import it directly into the database using pipes (for example: mysqlbinlog binlog_file | mysql-uroot-pxx). For this, we have collected some of the most basic environment information (Note: this is the customer environment), as follows

Database version: MySQL 5.7.26 operating system: CentOS 7.2x64 server configuration:

* CPU:8 vcpus

* disks: disk array

* memory: key parameters of 16GB database: buffer pool is half of physical memory, row format replicates database replication topology: one master, one slave, the process of data recovery is realized by using full data snapshot of slave database + binlog of slave database. The recovery target server is another separate database server, which is only used to temporarily recover data and there is no replication topology. No monitoring or heartbeat detection is configured through a simple inquiry to confirm that the system load is not high and the database is almost load-free. Synthesizing this information, before seeing the scene, we made a simple reasoning that the hang death phenomenon was probably caused by a big affair, so as soon as we went to the scene, we conducted a direct investigation around the database, and after several twists and turns, we finally found the cause of the problem, and also found a solution to the problem. However, the cause of hang's death is far from as simple as expected, and it is confusing to some extent. Let's make a simple summary of this and share it with you PS1: the following data is reproduced, and the server configuration is as follows (database uses Walker Best practices configuration template)

CPU:32 vcpus memory: 256g disk: 1.6T LSI flash card network: 10 Gigabit network PS2: pay attention to the Chinese notes in the following code segment 2, analysis and troubleshooting first, check the mysqlbinlog process You can see that it is running [root@localhost ~] # ps aux | grep mysqlbinlogroot 27822 1.30 24568 3204 pts/2 R + 15:11 0:04 mysqlbinlog-- stop-datetime='2019-07-14 16 mysqlbinlog 30 pts/2 00' mysql-bin.000014 mysql-bin.000015 mysql-bin.000016 mysql-bin.000017 mysql-bin.000018 mysql-bin.000019.

Then, I looked at the status information of the currently running thread and found a thread with a sleep length of 157269S. no, no, no. Is the big deal uncommitted? Rejoice!

Then we looked at the transaction and lock information

# check transaction information admin@localhost: test 03:02:36 > select * from information_schema.innodb_trx\ GEmpty set (0.00 sec) # Nani! , there is no transaction running # then check the lock information admin@localhost: test 03:30:25 > select * from information_schema.innodb_locks;Empty set, 1 warning (0.00 sec) # WTF! Unexpectedly, there is no locked information so far, and the investigation seems to have reached an impasse. Since there is no large transaction, no lock information, and the client connection used to recover the data is also in the Sleep state, it means that the database is not doing anything at this time (the database is a temporary library for data recovery, there is no other access business, and there is no monitoring, heartbeat, etc., there are only two connections, one is the connection on which we log in to troubleshoot the problem The other is to use the mysqlbinlog command to restore the client connection to the data), so why hang? I don't understand!

Then we looked at the system load, memory, network, disk almost no load, CPU no load, but there is a strange phenomenon, there is a CPU core utilization of 100%, as follows

Top-15:40:50 up 117 days, 8:09, 5 users, load average: 1.97,1.36, 1.15Tasks: 496total, 2 running, 494 sleeping, 0 stopped, 0 zombie%Cpu0: 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st%Cpu1: 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si % Cpu2: 34.4 us, 65.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st%Cpu3: 0.3 us, 0.3 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st%Cpu4: 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa 0.0 hi, 0.0 si, 0.0 st%Cpu5: 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st%Cpu6: 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st%Cpu7: 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi 0.0 si, 0.0 st.

Finally, in the recovery library, we looked at the GTID information and found that the GTID number was discontiguous and a transaction GTID (3759) was missing.

By looking at the GTID in the slave library (that is, the database corresponding to the data source of the recovery library), there is no missing transaction with GTID number 3759, so why is it inexplicably less in the recovery library instead of continuous? With curiosity, we use the GTID number to parse the list of all the binlog file designated for recovery from the library (in this case, mysql-bin.000014 mysql-bin.000015 mysql-bin.000016 mysql-bin.000017 mysql-bin.000018 mysql-bin.000019), and extract the binlog log contents of the transaction with GTID number 3759 from these binlog

[root@localhost binlog] # mysqlbinlog mysql-bin.000014 mysql-bin.000015 mysql-bin.000016 mysql-bin.000017 mysql-bin.000018 mysql-bin.000019-- include-gtids='5de97c46-a496-11e9-824bMuo0025905b06dadab.sql 3759' > b.sql [root@localhost binlog] # ls-lh b.sqlmurr RWMULI RWMULI-1 root root 996m Jul 14 15:59 b.sql # My God The number of binlog logs of a transaction with GTID number 3759 is close to a G # use the mysqlbinlog command plus the-vv option to reparse (parsing using the-vv option will cause the size of the parsing result to increase by about 3 times) Check what the original statement of the transaction looks like (only binlog_rows_query_log_events=ON will record the original statement text of the transaction) [root@localhost binlog] # mysqlbinlog-vv mysql-bin.000014 mysql-bin.000015 mysql-bin.000016 mysql-bin.000017 mysql-bin.000018 mysql-bin.000019-- include-gtids='5de97c46-a496-11e9-824bWhitt0025905b06dacon3759'> c.sql# vim opens the c.sql file to view [root@localhost binlog] # ls-lhtotal 12G root root Jul-1 end_log_pos 994M Jul 14 16:38 b.sql RW Jul 14 16:40 c.sql. [root@localhost binlog] # vim c.sql.# at 336lines 190714 16:29:10 server id 3306102 end_log_pos 422 CRC32 0x3b2c9b07 Rows_query# found a large transaction with 400W rows (by looking at the amount of data in the sbtest.sbtest1 table Confirm that the statement inserts 400W rows into the test.sbtest1 table) # insert into sbtest1 select * from sbtest.sbtest1 limit 4000000.BINLOG '1ucqXRN2cjIAOQAAAN8BAAAAAHUAAAAAAAEABHRlc3QAB3NidGVzdDEABAMD/v4E7mj+tABkzbY11ucqXR52cjIADSAAAOwhAAAAAHUAAAAAAAAAAgAE//ABAAAAlyomAHcAMzMzNjgzMDQzMTQtMjYxMDIwNDA4ODktMTU4NzIxODA4NzYtMTU0NDIxOTgxNjQtNDYzOTM1NDIxMzEtMTQwODg3NzUzNTQtNzY4MDU0ODgyMTEtNzg0ODM1NTU5NjEtNjMyMDM5NjA0ODEtNDcxNjQ5MDg4MjY7NTA3NzUyOTM0MzctMjE4MzMzNTAzNzYtODc2MTE2NjU0NTYtNjI4NTU3NjAyOTItNDQ3Mjc0MzA4MTjwAwAAAA0zHQB3ADM4NjUyNzU5NDQwLTg4NjY4MjU5MDE3LTk0MDk4ODI4Nzc1LTYxMzMzNjEwMjg0LTYyODc3NDgxMTY3LTY1NzM3Mjg3NTExLTA4MDYwNzA5NTU1LTIzMTUyNjI5NTcxLTE2MDMzMDM2NDE5LTYyMjA0MDgxMzc0OzczOTE3OTQ0NjMzLTc0ODMwMjgwMjE3LTAxMTYzODkwMzkzLTU3NTEzNDA4MDY1LTMwNjgzOTA1MTQ08AUAAAAlNCYAdwAwMjU2MTIyODQ3MC05OTAwMzk1ODMyOC0zMDQ1OTgyMzQwNC0zMTY1MTgyNzE4OC02MDMxODU1MDA5OS03Njk5Njk5MTY3Mi02MTI1Nzg5NTU5MC03NDA3OTQzMDg4MC01NzMyMDA4MzY4NC0zNjAzMDY2NDE4NjszOTA4Nzk4NjM5NC02MjA0NjQ4MDk0Ny01NjQ0NTE4NzA3My0yODQxNDg5MzQyNC03OTYxOTMzMTg1N/AHAAAAn1MmAHcAODgxMzg4MjkxMjEtMDkxNTk1NDI1OTctNzc4ODUwODczMzMtMjA1MzE3NDM2MjktODE3NTQ0NDc2MjgtMjczNDMyMzQ2ODEt. # this is the BINLOG code corresponding to this large transaction. The 994MB content of the whole b.sql file is mainly these BINLOG codes. After analyzing these BINLOG codes, we finally found the direct cause of the MySQL client hang death (note that the client hang died, not the server side), that is, when the MySQL client applied a binlog with a 400W line transaction, it caused the MySQL client to die hang. However, the question comes again: why does a large transaction with 400W lines cause the client hang to die for as long as 12 hours? And during this period, the load of the database and the operating system is not high, the parameters of the database have not reached the upper limit, and the error log and operating system log (/ var/log/messages) of the database have been checked, and no useful information has been found. So.., next we need to further analyze why a large transaction led to the death of the client hang 3, to find the culprit, as we mentioned above, the failure phenomenon can be reproduced, which facilitates our further analysis of the problem. Before starting any further analysis, let's list the time reference data for inserting 400W rows of large transactions in server environments of different disk devices (empirical value, imprecise)

Assuming that memory and CPU are not bottlenecks

* when using 15000 rpm SAS disk, insert into x select * from y; the statement inserts 400W rows of data in about 15 minutes

* when using ordinary SSD, insert into x select * from y; the statement inserts 400W rows of data in about 5 minutes

* insert into x select * from y when using flash card Although the reference time data provided above is not accurate, and the BINLOG encoding of a large transaction may have some additional overhead, it is impossible to import a 400W row binlog of a large transaction for more than 12 hours so..,. Next, we will use strace, pstack and other tools, combined with looking through the source code to troubleshoot the problem 3.1. Strace tool troubleshooting

Use the strace tool to view some of the output from logging in to MySQL and loading binlog parsing text (here use the b.sql mentioned above directly)

[root@localhost] # strace mysql 2 > strace.txtWelcome to the MySQL monitor. Commands end with; or\ g.Your MySQL connection id is 22Server version: 5.7.26-log MySQL Community Server (GPL) Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or'\ h' for help. Type'\ c'to clear the current input statement.admin@localhost: (none) 10:49:36 > use testDatabase changedadmin@localhost: test 10:49:38 > source / data/mysqldata1/binlog/b.sql;.Charset changed.Warning (Code 1681): Updating 'collation_database' is deprecated. It will be made read-only in a future release.Query OK, 0 rows affected (0.00 sec) # is stuck here. You can do something else at this time and come back in 20 minutes to see if the import is successful. However, no matter you wait, this client should be reserved for follow-up troubleshooting and cannot be disconnected.

Now, start another terminal session and view the contents of the strace.txt file

[root@localhost ~] # tailf strace.txt.munmap (0x7f8d6607d000, 58281984) = 0read (4, "wMjE4ODQzLTE5MTYzMTE4NDk4LTQwNTA"..., 4096) = 4096mmap (NULL, 58290176, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS,-1,0) = 0x7f8d5f150000. # it seems to be stuck in the memory allocation where munmap (0x7f8d628e1000, 58298368) = 0read (4, "UxNzUtODUyMjQ1MTkxMDEtMTM4MTk3OD"..., 4096) = 4096mmap (NULL, 58306560, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS,-1,0) = 0x7f8d66077000. Unfortunately, we use the strace tool to find no substantial and useful information here, and we need to use other tools for troubleshooting. Now, you can close the client connection for derivative testing (note that after using the strace command, the client connection can be terminated immediately. If you do not use the strace command, the terminal session hang of the database is dead, unable to terminate the derivative operation of the database, don't say I didn't wait for it to roll back. I waited for hours.

Note: if you are following our playback process, it is recommended that you check the show processlist information. You may find that the client connection that imported the b.sql file has not been disconnected in the database. If you find this, you need to manually perform the kill operation in the database.

[root@localhost] # mysql-uadmin-ppasswordmysql: [Warning] Using a password on the command line interface can be insecure.Welcome to the MySQL monitor. Commands end with; or\ g.Your MySQL connection id is 31Server version: 5.7.26-log MySQL Community Server (GPL) Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or'\ h' for help. Type'\ c'to clear the current input statement.admin@localhost: (none) 11:08:58 > use testDatabase changedadmin@localhost: test 11:09:00 > source / data/mysqldata1/binlog/b.sql;.

Use the pstack tool to view process stack information for MySQL client connections

# check the MySQL client connection process number [root@localhost ~] # ps aux | grep 'mysql-uadmin' | grep-v greproot 4614 81.2 0.0 175080 41284 pts/21 R+ 11:18 mysql-uadmin-px xxxx# use the pstack tool to view [root@localhost ~] # pstack 4614 | tee-a pstack.txt# finds it stuck in the stage of _ memmove_ssse3_back () memory copy. At this point, you can see that the idle of a certain CPU in the system is 0% (consistent with the weird phenomenon of CPU load found above). What this CPU-exhausted CPU does is continuously request memory, copy data, and free memory. Naturally used up a CPU # 0 0x00007f1009f5c315 in _ memmove_ssse3_back () from / lib64/libc.so.6#1 0x0000000000422adf in my_realloc (key=, ptr=0x7f0ffde45030, size=30197464, flags=) at / export/home/pb2/build/sb_0-33648028-1555164244.06/mysql-5.7.26/mysys/my_malloc.c:100#2 0x000000000041a2ba in String::mem_realloc (this=0xa45460, alloc_length=30197460) Force_on_heap=) at / export/home/pb2/build/sb_0-33648028-1555164244.06/mysql-5.7.26/sql-common/sql_string.cc:121#3 0x0000000000417922 in add_line (truncated=false, ml_comment=, in_string=, line_length=76, line=0x16b4309 "NDQ4NDA0LTE5MjQ2NjgzMDgxLTY2MTA0Mjk0ODQ2LTYzNzk3MjcwMjU0LTQ3NjA2Nzk0MTY0LTEx\ nODQwNjExOTY5OzY5NTQ1Mjc5MDA2LTM5NTgwMjUzMDEzLTgzMjQxNjU0MzQ1LTA4MDkxMDEzODk1\ nLTk5NzMxMDYyMzU58B2TAwABKiYAdwAwNTEzMjEzNDUzNC"..., buffer=...) At / export/home/pb2/build/sb_0-33648028-1555164244.06/mysql-5.7.26/client/mysql.cc:2795#4 read_and_execute (interactive=false) at / export/home/pb2/build/sb_0-33648028-1555164244.06/mysql-5.7.26/client/mysql.cc:2364#5 0x00000000004181ef in com_source (buffer= Line=) at / export/home/pb2/build/sb_0-33648028-1555164244.06/mysql-5.7.26/client/mysql.cc:4709#6 0x0000000000417099 in add_line (truncated=false, ml_comment=, in_string=, line_length=37, line=0x16b2a10 "source / data/mysqldata1/binlog/b.sql ", buffer=...) At / export/home/pb2/build/sb_0-33648028-1555164244.06/mysql-5.7.26/client/mysql.cc:2664#7 read_and_execute (interactive=true) at / export/home/pb2/build/sb_0-33648028-1555164244.06/mysql-5.7.26/client/mysql.cc:2364#8 0x0000000000418c98 in main (argc=15, argv=0x160c348) at / export/home/pb2/build/sb_0-33648028-1555164244.06/mysql-5.7.26/client/mysql.cc:1447 happy Using the pstack tool to find a more useful information, from this we know that when the MySQL client imports b.sql, it is stuck in the stage of using the _ _ memmove_ssse3_back () function to make a memory copy, so the question comes again, why is it stuck here? Combined with the information obtained above, we know that the MySQL client only established a connection with the server, and the subsequent server did not receive any information (that is, the server did not do anything during the death of the client hang). Next, we have to look through the source code to find the answer: why does the MySQL client get stuck in the memory copy phase of the _ _ memmove_ssse3_back () function? 3.3. Browse the source code of the client command mysql

Since the failure occurred in MySQL 5.7.26, we first looked at the client source code of MySQL 5.7.26. When the MySQL client reads a file (or determines when an operation can be sent to the server), it encounters' The semicolon will directly throw the previously read content to the server as a whole data packet. For the BINLOG code in the b.sql file mentioned above, the entire BINLOG code has only one semicolon, so it will not be sent to the server until the entire BINLOG code is read. In the process of reading this super-large BINLOG code, the MySQL client will read it one line and record it in the global_buffer. When the space requested by global_buffer is insufficient, you need to expand the memory space. The code to expand the memory space is as follows:

/ / client/mysql.cc file if (buffer.length () + length > = buffer.alloced_length ()) buffer.mem_realloc (buffer.length () + length+IO_SIZE) # each time, in addition to insufficient extended memory (original memory length + newly read content length), only 4k additional space is expanded (that is, the definition size of IO_SIZE,IO_SIZE variable is detailed in the "include/my_global.h file" following the code snippet) / / mysys/my_malloc.c file new_ptr= my_malloc (key, size, flags) # apply for a new and larger memory when expanding memory (the size of the newly requested memory is the memory size described in the client/mysql.cc file) if (likely (new_ptr! = NULL)) {# ifndef DBUG_OFF my_memory_header * new_mh= USER_TO_HEADER (new_ptr) # endif DBUG_ASSERT ((new_mh- > m_key = = key) | (new_mh- > m_key = = PSI_NOT_INSTRUMENTED)); DBUG_ASSERT (new_mh- > m_magic = = MAGIC); DBUG_ASSERT (new_mh- > m_size = = size); min_size= (old_size

< size) ? old_size : size; memcpy(new_ptr, ptr, min_size); # 然后把数据拷贝过去 my_free(ptr); return new_ptr; }// include/my_global.h 文件#define OS_FILE_LIMIT UINT_MAX/* Io buffer size; Must be a power of 2 and a multiple of 512. May be smaller what the disk page size. This influences the speed of the isam btree library. eg to big to slow.*/#define IO_SIZE 4096 # IO_SIZE 变量的大小定义为4K通过翻阅源码中的相关代码，再结合b.sql中400W行大事务产生的BINLOG编码，我们可以得出如下结论：MySQL客户端读取b.sql文件中的BINLOG编码时，会把整个BINLOG编码当做一整个数据包对待，也就是说客户端先要将其完整读取并存放在内存中，然后再将其作为一整个数据包发送给服务端（包的大小由客户端和服务端的max_allowed_packet参数共同控制，以最小的为准），客户端在读取BINLOG编码的过程中，每读一行都会先判断原先分配的内存是否足够，如果内存不足就需要扩展一下，申请新的一段内存，并且把原来的内存中的数据拷贝过去（新申请的内存），自然就非常慢，看上去就好像客户端hang死一样，而客户端读取BINLOG编码的整个过程中，服务端都没有收到任何的操作，所以，从服务端的show processlist看到线程状态一直是Sleep状态，也没有事务，也没有锁信息。也就是说，这里可以排除掉mysqlbinlog命令解析binlog的嫌疑4、干掉罪魁祸首虽然，我们发现了问题的根本原因，是因为MySQL客户端读取大事务的BINLOG编码时分配内存的效率太低导致的，但是，怎么解决它呢？改代码？怎么改？改了出其他问题咋办？IO_SIZE变量是一个比较底层的变量，不同用途的buffer分配都会使用该变量作为一个基准参考，所以不能直接修改此变量的大小。因此，我们并没有急于去修改客户端代码，而是先尝试使用MySQL 8.0.16、MariaDB 10.1.40/10.2.25/10.3.16/10.4.6、Percona Server 5.7.26进行测试，发现如下现象 MySQL 5.7.x和Percona Server 5.7.x：表现一致，客户端hang死MySQL 8.0.x：在两三分钟之后，客户端连接被服务端断开MariaDB：表现与MySQL 8.0.x一致那么，问题又双叒叕来了 1）、为什么MariaDB和MySQL 8.0.x会将客户端连接断开，而不像MySQL 5.7.x和Percona Server 5.7.x那样导致MySQL客户端hang死2）、MySQL 8.0.x在MySQL 5.7.x的基础上做了什么改动使得他们在导入相同数据时的表现不同呢？我们先解决第一个问题（我想大多数人也是这么想的，对吧），过程很辛酸，最后发现一个很low的问题，就是。。服务端的max_allowed_packet系统变量值设置太小了（64M），将其改为1G之后在MariaDB和MySQL 8.0.x中，这个400W行的大事务产生的binlog可以在3分钟以内应用完成（注意，该参数最大值为1G，如果你的事务产生的BINLOG编码长度大于该参数，则仍然会被服务端断开连接）在MySQL 5.7.x和Percona Server 5.7.x版本中修改该参数为1G之后，也可以应用完成，只是要差不多2天左右的时间，这跟hang死没什么两样，所以问题并没有解决（因为客户使用的是MySQL 5.7.26版本嘛...）接下来，我们解决第二个问题，比对MySQL 5.7.26和MySQL 8.0.16版本中，关于客户端缓存读取文件数据这块的内存分配代码有什么不同？ # 以下只列出MySQL 8.0.16版本中的代码// client/mysql.cc 文件 if (buffer.length() + length >

= buffer.alloced_length () buffer.mem_realloc (buffer.length () + length + batch_io_size); each time memory is expanded in # # 8.0, the extra allocated memory size is changed to batch_io_size. In 5.7, the batch_io_size location is the IO_SIZE// client/my_readline.h file static const unsigned long int batch_io_size = 16 * 1024 * 1024 # # batch_io_size is defined as 16m, from 4K to 16m, which increases the memory allocation efficiency by several orders of magnitude when the client caches the read data and finds that there is not enough memory. So... It seems clear how to modify the client code of MySQL. All you need to do is to define a new batch_io_size variable, and then change 'buffer.mem_realloc (buffer.length () + length+IO_SIZE)' to 'buffer.mem_realloc (buffer.length () + length+ batch_io_size)'. According to this conclusion, we modify, recompile, and then recompile the source code of MySQL's client side mysql. Set the max_allowed_packet parameters of both the server and the client to 1G, and re-import the row format binlog generated by 400W rows of large transactions (that is, use the row format binlog parsed text generated by the insert into sbtest1 select * from sbtest.sbtest1 limit 4000000 statement to test). The import is completed in 3 minutes. Problem solved!

Finally, I would like to say that although MariaDB also solves this problem, the solution is completely different (a cursory glance at the code, but not a closer look). By comparing the parsed BINLOG code, it is found that there are gaps in the BINLOG code of MariaDB, so it is speculated that the problem may be solved by using "parsing format change of mysqlbinlog" combined with "parsing logic change of mysql client" (when MariaDB imports binlog parsing content. When show processlist looks at the thread status, it can also see that the thread importing the data has been working and is not in the Sleep state all the time, as shown below

MariaDB

* BINLOG encoding format

* show processlist status

MySQL

* BINLOG encoding format

* show processlist status

| | author profile |

Luo Xiaobo Walk senior database technology expert

IT has worked for many years and is mainly responsible for the database support and after-sale second-line support of MySQL products. Participated in the design and preparation of version release system, lightweight monitoring system, operation and maintenance management platform, database management platform, familiar with MySQL architecture, Innodb storage engine, like to specialize in open source technology, has done offline database topic sharing in public many times, and published many database-related research articles.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.