Shulou(Shulou.com)06/01 Report--

In the past two years, with the maturity of the cloud environment, the business of many enterprises are moving to the cloud. As a result, "cloud database" has become one of the most fashionable words, respected by AWS, Microsoft, Ali, Huawei and other big companies. So, for users, how do they view the development of cloud database, and what factors are the key points for their selection? Lao Wang's experience may explain the current situation of cloud database development in China to a certain extent.

Lao Wang's mental journey

Website An is one of the largest social media in China, and Lao Wang is the DBA of this site. As many people say, a good architecture is never designed, but evolved, and so is the database. From commercial databases to open source databases to cloud databases, Lao Wang is the most direct witness and promoter of database change.

At first, the number of daily active users of website A reached 150 million. When users register, they need to assign an independent ID to each user, and save the user name, password, date of birth, unit and other information. In addition, after logging in, you should manage the message, follow, find, comment, forward and so on. The database of this website adopts a standard structure, which is designed according to the separation of read and writing. the main database is responsible for writing and access from the library. if the access pressure is too great, the expansion ability can be obtained by expanding the number of slave libraries. Moreover, all businesses are vertically divided according to users, content, and relationships, using separate databases.

After that, with the rapid development of the mobile Internet, the number of registered users of website An also skyrocketed, and the original database showed unprecedented pressure. In order to support the rapid development of business, we can only purchase hardware equipment with better performance, store each functional module independently, and split the massive business data for the second time. As there is no time for architectural transformation, we can only support the core business by purchasing hardware equipment. At that time, the feed system relied heavily on MySQL, and at its highest concurrency, MySQL wrote to QPS up to 4W. Due to database performance problems, directly led to website paralysis, the loss of users, the scene at that time, Lao Wang still remembers vividly. In order to ensure the performance of the query, Lao Wang and his team members split the index and content, each divided into a lot of ports, each port out a lot of DB.

After the transformation of the database, has gradually become stable, but when the peak occurs, MySQL still has the problem of slow query. Moreover, for the operation and maintenance personnel, the operation and maintenance is not automatic enough. In order to solve these problems, Lao Wang began to study the problem of cloud migration.

Build your own cloud MySQL or choose a mature cloud database?

In order to maintain the consistency of the data, Lao Wang first considered building his own MySQL database on the cloud server.

After all, MySQL is an open source database with a history of more than 20 years, and it is also the most popular open source database on the Internet. Especially after MySQL upgraded to MySQL 8.0, it has done a lot of optimization and has a lot of new features. For example: support the data type of json, realize the reading and access of json data type, support NoSQL interface and so on.

Self-built cloud MySQL has cloud features, including high availability, flexible scalability, simplified operation and maintenance, and so on. However, the disadvantages of self-built cloud MySQL are also obvious: first of all, the cost of CVM is too high, so you need to configure software and hardware. Secondly, the self-built database needs to be maintained by DBA itself, so it is difficult to guarantee the security. In particular, the security risks caused by hardware failures are difficult to control. In addition, the level of DBA also determines the stability of the database. Third, the deployment cycle is too long, need to purchase hardware, find computer room trusteeship, redeploy. For Lao Wang, they do not have more manpower and material resources to solve these problems. Building your own cloud MySQL is tantamount to digging a hole for yourself.

Therefore, Lao Wang decided to choose a mature solution directly. As a result, "cloud database" jumped into his eyes.

Looking ahead, there are many "cloud databases" available on the market. Including: AWS's Amazon Aurora, Ali's PolarDB, Huawei's cloud database MySQL and so on. How to select the type?

Lao Wang made a comprehensive assessment and thought that there were several important points:

First, we must choose a big brand. A website is a large social networking site, the database is the full value of the enterprise, Lao Wang can not bet on the fate of the site and his career. Therefore, cloud database products must be in the top three. AWS is the largest cloud computing provider in the world, occupying most of the market share.

Second, you must choose a mature solution. The reason for choosing to migrate the database to the cloud is to get better performance, but if the higher performance comes at the expense of reliability, Lao Wang would rather maintain the status quo. Moreover, cloud database is a new concept in China after all, if there is no mature case, Lao Wang can not be this "guinea pig". From this point of view, Amazon Aurora is undoubtedly the most dominant. Amazon Aurora was released in 2015, and after three years of verification, it has been recognized by many customers, such as Autodesk, TIBCO, Arizona State University, General Electric, BMC, Daily News and so on.

Third, it should be compatible with MySQL on the basis of higher performance. There is no doubt about the performance of Aurora, as a top masterpiece in the field of cloud database, its throughput capacity is unparalleled. Many database products take Aurora as a benchmark and refer to the architecture of Aurora. In addition, the biggest challenge in making a database is compatibility. If the compatibility is poor, the migration cost will be many times higher, which is the fundamental reason why middleware and sub-library sub-table are anti-human solutions. Aurora happens to have compatibility features, which can completely cover MySQL and PostgreSQL.

Fourth, it can be managed. Lao Wang hopes that after using the cloud database, it can reduce the pressure of database maintenance. There is no need to worry about database management tasks such as hardware, software patches, setup, configuration, or backup. In addition, the cloud database solution can provide automatic monitoring, automatic continuous monitoring and backup to storage services, which can achieve more granular point-in-time recovery. At this point, Amazon Aurora also beats similar solutions.

Fifth, in the process of database migration, the business operation can not be affected. Amazon Aurora can be started quickly, can be directly connected to the source database and target database, and can minimize downtime. Through AWS DMS, users can create tables, load data, and keep them synchronized, switching applications to the target at any time.

How to migrate?

How did Lao Wang migrate from local MySQL to cloud Amazon Aurora?

According to the migration manual on Amazon's official website, Amazon Aurora has a variety of different migration methods. Website A chooses Percona XtraBackup backup tool, which can support full backup and incremental backup. Compared to MySQLdump,XtraBackup, the binary data and logs of the database are backed up, and the files can be compressed smaller. In this way, both backup and restore are faster.

After installing the complete tool, back up the MySQL database, and then upload the backup MySQL file to Amazon S3. Finally, restore the original backup file to Amazon Aurora in Aurora Console.

Database instance and instance monitoring

To further optimize costs and meet the additional requirements of multiple applications, site A performs the following steps:

1. Automatically adjust the number of CPU copies based on Aurora utilization. No longer run multiple copies continuously, only add copies when needed.

2. Deploy the Aurora Snapshot tool for all clusters to automatically replicate snapshots and implement snapshot retention rules for 30 days. Use the console to operate snapshot injection, a read-only copy of Binlog replication, and DMS can also access the database and set it as a source or destination.

3. Full MySQLdump plus incremental Replication is adopted. However, during the period when MySQLdump exports the data and replicates to the target database, the primary library should be set to read-only to avoid new data writes.

4. Enable CloudWatch Logs to create CloudWatch metrics and alerts to continuously monitor activity in the Aurora database cluster.

5. Compatibility testing should be done before deciding on migration. We can define the Master node capacity of Aurora, t-series models can be used in temporary test environment, and r-series models can be used in production environment. The machine of the right size can be selected according to the pressure of production.

In the end, website A was very successful in migrating data from MySQL to Amazon Aurora. Taken together, it is not just a legend that the performance is five times higher than the original. Amazon Aurora greatly improves the performance and scalability of the original database, and optimizes the cost. During the peak period, the system can receive at least 10 times more requests than before.

Performance performance

From then on, Lao Wang can finally breathe a sigh of relief. In the future, he wants to focus more on more valuable business, rather than being overwhelmed by complicated chores every day.

Write at the end:

Amazon Aurora, one of the most charismatic aurora goddesses, created the most global cloud database. For Chinese users, many people may not doubt his technical ability, but have doubts about the effectiveness of local services. In fact, AWS is already laying out the Chinese market. The AWS China (Beijing) region operated by Sinnet and the China (Ningxia) region operated by NWCD provide similar technical service platforms to other AWS regions around the world. Developers can easily and efficiently deploy cloud-based applications in China, using the same API, protocols, and operating standards that are no different from AWS's global customers.

Today, AWS is vigorously promoting Amazon Aurora to Chinese users. If you want to experience the ultimate experience brought by cloud database, click here to apply for AWS China regional account > >

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.