Shulou(Shulou.com)06/01 Report--

Editor to share with you Pciessd abnormal Readonly to Mysql repeated crash how to do, I believe that most people do not understand, so share this article for your reference, I hope you will learn a lot after reading this article, let's go to know it!

In October last year, a fio card changed to readonly (which has nothing to do with Singles Day), and a Fio card changed to readonly, resulting in the failure of mysql crash, which is sorted out as follows.

[machine configuration]

System | Dell Inc.; PowerEdge R710 Processors | physical = 2, cores = 12, virtual = 24 Hyperthreading = yes # Memory # Total | 94.40G Free | 555.50m Swappiness | vm.swappiness = 0 # Disk # # # 2 ioMemory devices in this system Fusion-io driver version: 3.1.5 build 126 Fusion-io ioDrive 640GB * 2-> mdadm-> / dev/md0 ibdata Ib_logfile,bin_log,relay_log on SAS 600GB raid1

[problem performance]

13:28, the monitor issued * db_ping alarms.

The alert log of mysql is as follows:

/ u01/mysql/libexec/mysqld: Can't create/write to file'/ u01max mysql InnoDB' init function returned error TMP Universe ibU5kXB4' (Errcode: 30) 121104 13:28:10 InnoDB: Error: unable to create temporary file; errno: 30 121104 13:28:10 [ERROR] Plugin 'InnoDB' init function returned error. 121104 13:28:10 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 121104 13:28:10 [ERROR] Aborting InnoDB: Error: tried to read 16384 bytes at offset 0 41517056. InnoDB: Was only able to read-1. 121104 13:14:59 InnoDB: Operating system error number 5 in a file operation. InnoDB: Error number 5 means' Input/output error'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.1/en/operating-system-error-codes.html InnoDB: File operation call: 'read'. InnoDB: Cannot continue operation. Mysqld: my_new.cc:51: int _ _ cxa_pure_virtual (): Assertion `! "Aborted: pure virtual method called." 'failed. 121104 13:14:59-mysqld got signal 6

Judging from the above that there is a problem with the IO device, at this point touch / u01/mysql/tmp/ibd:

Touch: cannot touch `/ u01amp mysqlUniple TMP Universe ibdink: Read-only file system

Because it is a core cluster, there is a need for strong consistency of data, so you can manually force the master / slave switch through DBA to troubleshoot.

[cause of the problem]

The fusionIO card appears readonly / var/log/message Nov 4 13:14:59 my160130.cm6 kernel:: fioerr Fusion-io ioDrive 640GB 000015 07 fioerr Fusion-io ioDrive 640GB 00.0: Single Bit Event Upset Error Dete4ted-interrupt: val [0]: 000ff16 fio-status-a fct1 Failed: DEVICE IS OFFLINE. ALL READS AND WRITES WILL FAIL! IoDrive 640GB MLC, Product Number:2TTK9, SN:436946!-> There are active errors or warnings on this device! Read below for details. IoDrive 640GB MLC, PN:00214401201 Located in slot 0 Center of Pseudo Low-Profile ioDIMM Adapter SN:436946 WARNING: READ-ONLY MODE. ALL WRITES WILL FAIL! ACTIVE ERRORS: The ioMemory has encountered an internal error and has been temporarily disabled. All reads and writes will fail. The ioMemory is not allowing write operations.

[problem Analysis]

SEUs are transient soft errors, and are non-destructive. A reset or rewriting of the device results in normal device behavior thereafter

The control module of fio runs on fpga, the metadata is stored on DRAM and SSD, and the power outage can be restored. After the error occurs in the driver of 2.x, it will be repaired by rewriting. 3.x driver improves security. When this error occurs, it will directly reset, and card read _ only will wait for power recycle.

SEU class errors are caused by cosmic ray particles making it's way into the NAND controller or by a failing NAND controller

The media damage of FPGA itself or cosmic rays are the causes of this error. In May, there is a similar Write Path error in message. 2.x driver automatic rewrite has been fixed. 3.x driver has a higher security level, and reset is set to readonly.

Write Path Parity Error

This error is the precursor of SEU errors, most of which can be fixed. In the same cluster, 3 happened and repaired automatically.

The cost of FPGA is much lower than that of opening the chip, and the programming iteration is fast, but the robustness is not open to the chip.

[data loss]

Because the logs of undo,redo,binlog are all complete on the SAS disk of U02, and there is basically no delay in preparing the database, there is no data loss.

However, because SEU may lead to block errors written at that time, resulting in data inconsistency, it is safe to redo the standby database and use binlog to synchronize all data.

SEU class of error my result in data on the device being corrupted.The database should be verified or restored from backup

[improvement measures]

After FPGA aging, there is a certain chance of Single Event Upset errors, and the core library should be replaced in time.

FPGA is sensitive to cosmic rays and needs to control the environment of the computer room and distribute the cabinets on the shelves.

Improve the more sensitive message,dmesg alarm.

The above is all the content of this article "what to do with Pciessd abnormal Readonly causing Mysql repeated crash". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.