Shulou(Shulou.com)06/02 Report--

Author: formyz

Problem description

A project consists of two sets of proxmox, one running all applications and the other running mysql database. To be on the safe side, proxmox plug-in shared storage and automatic backup of all virtual machines at night.

Backup is the use of a 4U server, considering the capacity and cost, using an old 4U server, plugged in a lot of slow Sata disk, the effective capacity is more than 35TB. After the project was launched, the operation in the first half of the year was normal. With the increase of business, the amount of data increased, especially the number and size of databases. As a result, the alarm of the monitoring system is frequent and the user experience becomes worse. And the impact is quite large. Through investigation, it is found that it is caused by the backup of the database virtual machine.

The set backup starts at 0: 30 in the morning and can hardly be completed before going to work during the day, or worse, it will be delayed into the evening. The performance of the database IO, causing access congestion, resulting in a series of chain reactions, operation and maintenance work under great pressure.

Temporary measures

In order to ensure the normal business, but also consider the data security, requisitioned a smaller capacity idle server (originally used for other purposes), its hard disk is all 600G 15000 rotation sas mechanical hard disk. After it is configured as a nfs service, it is hooked into the proxmox data center.

After the setting is set, people are arranged to follow each other in turn at night, and the alarm is notified to each other immediately. fortunately, there is no congestion. This shows that it is really due to the poor performance of sata and the slow backup speed. Observe for a week, and if the problem does not reappear, come up with a formal solution. If you talk about data in this way, you can also get the support of decision makers.

Scheme design

Because it is not the kind of organization that is not short of money, it is impossible to buy a set of storage of SAS disk separately and abandon the existing low-performance storage. It can only be optimized on the existing storage to improve its performance. In another unrelated project, several Aliyun's "efficient cloud disks" were purchased to store computing-intensive applications (java, php, database, etc.). When users visit heavily (when there are tens of thousands of users online), they often have problems, so they are impressed by this. The so-called efficient cloud disk uses ssd to cache the data of the backend Sata disk, and its performance is much better than that of naked sata. Data backup does not meet the requirement of high disk performance, so is it helpful to use this method for reference to the overall write performance of backup?

The original system has a ssd, which is used to install the operating system, and other sata is used for sharing, which is made into raid 5 at the bottom. Buy a 512G ssd and unplug a Sata disk.

Consult the hardware supplier and inform the type and model of the raid card currently used, the answer is that the solution is feasible, and the existing raid card can support ssd cache, only need to purchase a hardware cache acceleration module and pay a small license fee. There is no practice in this area in the past, there is not much confidence in mind, but even if it does not meet the requirements, the loss of funds caused by it is not large (ssd can be used for it).

To sum up, on the existing basis, purchase a 512G ssd hard disk and a raid card cache acceleration module, do the configuration, and then put into use.

Program implementation

The moon is dark and the wind is high at night, send a younger brother to sneak into the computer room quietly. Shut down, remove the shelf, insert the SSD disk, in order to facilitate the insertion of the raid cache acceleration module, cut off the raid card, insert the cache acceleration module and then insert it back into the motherboard.

When the hardware is ready, put it on the shelf and power on.

Go to the raid card setup interface (before the system boots), make raid 5 for the Sata disk, and then use the menu to set the 512g SSD disk as the cache device of the raid group. For specific operation, please refer to the operation manual of each manufacturer.

After setting up, continue to boot and enter the system, you should not see the 512G hard drive that is cached.

Configure the nfs shared directory and start the nfs service, and then hook up the nfs shared directory in the proxmox data center.

Implementation effect

It's a mule or a horse. It's only clear if you pull it out and walk.

First use disk performance tools hdparm and dd and other tools to test, the speed is indeed several times faster than the bare sata disk. Look at the time is almost up, advance the backup time half an hour, from 0:00 let the system automatically start backup. The people concerned pay attention to the mobile phone and notify each other when there is an alarm.

At 7: 00 in the morning, I got up to check the backup status (the proxmox management interface can track the specific backup to that virtual machine, how many backups are), and nearly 90% of the backup was completed. Heaved a sigh of relief, wait until 9 o'clock to see, the backup is complete.

Contact other operators, understand the user access, feedback everything is normal, did not appear before the phenomenon of all stuck.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.