Shulou(Shulou.com)06/01 Report--

AWS RDS is forced to upgrade is a helpless thing, the version is not supported, and being forced to upgrade will affect business availability. Instead of passive forced upgrading, it is better to formulate an active upgrading strategy. This article introduces you to the best practices of AWS RDS version upgrade, which is the author's work of vomiting blood. I hope it can help you.

1. AWS RDS Upgrade Cycle Description

According to Amazon RDS FAQs, the larger version of AWS RDS will last at least 3 years, and the smaller version will last at least 1 year.

According to the communication with AWS, AWS will release RDS based on AWS about 5 months after the release of the basic version of the general community.

Therefore, the RDS upgrade cycle of AWS is about 5 months after the release of the community version, and AWS releases the corresponding version, each major version supports at least 3 years, and each minor version supports at least 1 year.

2. Consequences of AWS RDS version expiration

According to Amazon RDS FAQs, Amazon will alert customers in advance when a major or minor version expires (6 months in advance for major versions and 3 months in advance for minor versions), and after the reminder period, AWS will force automatic upgrading of the database to the latest version (even if the customer chooses to turn off automatic minor version upgrading). The upgrade process, the application can not connect to the database, resulting in business impact.

Note 1: Both large and small versions will face a forced upgrade process once Amazon's service support period expires.

Note 2: The upgrade process for minor versions includes backup, upgrade, and backup again. The experience value refers to the first backup and the last backup, and does not affect normal business access. The process of upgrading the database affects normal business access. The whole upgrade process takes about 30 minutes, of which the time affecting business access is 3 and a half minutes. However, the specific business impact time is subject to actual testing.

Note 3: A week before the deadline of the reminder period, the minor version cannot do any modification operations on the database, including building replicas or changing maintenance windows. However, a database can be restored from the snapshot backup to test the duration of the upgrade.

Note 4: The minor version upgrade procedure is to upgrade the slave library first, and then upgrade the master library.

3. Internal upgrade step analysis

Namely:

a). Before upgrading, take a snapshot, noting the timing of the snapshot and the size of the database.

b). Slow shutdown, i.e. set global innodb_fast_shutdown=0 and then shutdown. Because slow shutdown is set, dirty buffer will be flushed to disk +insert buffer will also be flushed to disk (i.e. system tablespace, ibdata1)+full purge (i.e. clean up useless undo pages)

c). Mount mysql to the new storage engine and disable remote network access;

d). Run mysql_upgrade to upgrade the data dictionary.

e). Run RDS special scripts to upgrade RDS encapsulated tables and stored procedures.

f). Restart instance, open network remote connection.

Note 1, in some cases, mysql_upgrade this step will physically rebuild the table, the size of the table will affect the upgrade time, so the actual upgrade time, need to be subject to testing. For example, MySQL 5.6.4 is upgraded to version 5.7, because the storage types of TIME, DATETIME, and TIMESTAMP in version 5.6.4 have changed. When upgrading, you need to rebuild the table.

Note 2: Since major versions cannot be upgraded across major versions, such as upgrading MySQL 5.5.46 to 5.7.19, they cannot be upgraded directly. You need to upgrade 5.5.46 to 5.6 first, such as 5.6.37, and then upgrade to 5.7.19. Therefore, the time for business to be affected is the time for two upgrades. Not once. So don't do the alternate upgrade of the big version. 5.5 upgrade 5.7, 5.4 upgrade 5.6.

4. Release Roadmap

Depending on when the community released the version, and when AWS has released the version, we can make the following release roadmap.

MySQL：

PostgreSQL：

Note 1: The light green color at the beginning indicates the release time of the first version of the Community Edition, and the gray color behind it indicates the release time of the Community Edition based on the minor version GA after the first version, while the corresponding version released by AWS is colored.

Note 2: Usually, AWS minor versions support at least one year (i.e. 12 months), but some minor versions, AWS has been supported for more than 12 months, and support may be terminated at any time, so I draw the current time (October 2018), and the later time does not continue to draw. (That is, no drawing does not mean that it is not supported, but it only means that the AWS version has been released for more than 12 months, after which it may be terminated and forced to go offline)

5. Upgrade Best Practices

5.1. Major version upgrade:

a). Create two replica instances.

b). Upgrade one of the instances to a higher version. At this time, the master-slave synchronization relationship is maintained.

c). Create a dms instance, configure the endpoint of the source and target, and create a task. Note that when creating a task, select changes only, and cancel the check mark of Start task on create.

d). At the beginning of service interruption, the newly created replica instance is promoted to the master database;

d). Click start in dms task, wait for it to complete the comparison of the full database, and start preparing synchronous incremental data;

e). Switch applications to connect to a higher version of the database;

Note 1: After upgrading from versions below 5.6.4 to 5.6.4, alter table_name force is required to rebuild the table before creating index using online ddl.

Note 2, large version upgrade, need to verify the performance of the application, need to grab at least a week of SQL, sql replay to see the performance changes.

Note 3, after upgrading, in order to reduce physical reading, load more data into memory as soon as possible, you can use mysqldump to do warm

Note 4: To reduce downtime, one of the steps is to click the start of task to calibrate the full amount of data. If the IOPS of the main database is increased, it will help to improve the speed of this step. (This step is a business downtime operation, so reducing the time of this step is equivalent to reducing the downtime.)

aws mysql major version upgrade best practise.pdf

Download link: oracleblog.org/working-case/aws-rds-upgrade-best-practise/

5.2. Minor version upgrade:

Method 1:

a). Create a replica instance first, or use an existing replica instance directly;

b). Upgrade the replica instance to a higher version. At this time, the master-slave synchronization relationship is still maintained.

c). At the beginning of the service interruption, the replica instance of the higher version is promoted to the master database;

d). Switch apps to connect to a higher version of the database. Application connection string configuration, can be configured in advance, restart the application can be;

aws mysql minor version upgrade best practise.pdf

Download link: oracleblog.org/working-case/aws-rds-upgrade-best-practise/

Method 2:

a). Upgrade replica instance to a higher version first, which is the prerequisite for all AWS upgrades, i.e. you must upgrade slave library first;

b). Disconnect the service from the database and start upgrading the master database.

c). Upgrade the master library to a higher version;

d). Restore application connectivity;

aws mysql minor version upgrade best practise_2.pdf

Download link: oracleblog.org/working-case/aws-rds-upgrade-best-practise/

Note 1: Method 1 is recommended by AWS, but Scheme 2 is also very suitable for small systems.

Note 2: The application impact time of method 1 is the time to promote the slave library to the master library + the time to restart the application. According to one of our database tests, the time to upgrade is approximately 3 minutes and 02 seconds. Plus the application restart time, it is also about three and a half minutes.

Note 3. Method 2: The test data of one of our databases is that the overall upgrade time is about 34 minutes (because it includes the backup of the database before the upgrade and the backup after the upgrade is completed, which is done by AWS itself during the upgrade process). However, these 34 minutes are not all unavailable. When doing database backup, the database can still be used. The time when the real business cannot connect to the database is 3 minutes and 32 seconds.

Note 4: The service is unavailable for about three and a half minutes for both methods. However, there is a risk in method 1. If it is because we need to force upgrade the minor version, it is almost the maintenance time of the upgrade, and it is already the maintenance time of the deadline, then although we do not move the main library, we need to switch back to the main library in case of failure, and the time for forced upgrade is up again, triggering forced upgrade, then this is an uncontrollable state. So we chose option two.

Note 5: Which method should be chosen ultimately depends on the actual practice of doing the upgrade test.

6. summary

Therefore, we can develop the following proactive upgrade strategy:

(1). Disable automatic upgrade of all minor versions;

(2). Based on the above, new MySQL installations are specified as 5.7.23;

(3). Within a year, for the previous MySQL 5.5 version, minor versions were unified to 5.5.61, MySQL 5.6 version, minor versions were unified to 5.6.41. This prevents minor MySQL releases from being forced to upgrade because they are not supported, and the next forced upgrade of these two versions is at least after September 2019. (pg similar guidelines);

(4). Upgrade to MySQL version 5.6 for the previous MySQL version within one year; upgrade to MySQL version 5.7 for MySQL version 5.6 within two years; and unify to MySQL version 8.0 within two to three years. Resolves the problem of increased O & M difficulty due to coexistence of multiple versions. (pg similar guidelines);

(5). Subsequent versions will be upgraded at a 1-year, 1-year, 3-year, 1-year pace to comply with AWS RDS version support rules.

Author: He Jianmin, Operation and Maintenance Department of a UAV company, database architect, responsible for the architecture, planning and operation and maintenance of the company's cloud and cloud databases, focusing on the overall operation and maintenance ideas, giving full play to the platform effect, and realizing unified operation and maintenance of cloud and cloud databases.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.