Shulou(Shulou.com)06/01 Report--

This set of maintenance optimization scripts, which I wrote in my spare time, are designed specifically for goldengate data synchronization tools.

The main reason is that I am a little lazy and do not like tedious manual operation. So reduce your workload with the help of shell script

Some references are made to the shell script design of some network Daniel.

Because of leaving, it is estimated that it may not be used in the future career, so share the design and specific content of the script.

For netizens to learn and reference.

Because some of the functions of the ability problem may be written in a rough way, I hope you will not mind. You are welcome to put forward your own opinions. Thank you

I will slowly update the blog post to improve the content.

First, the purpose of script design

I am working on the operation and maintenance of OGG, and I will encounter the following problems

1 > the steps of adding tables are tedious

Some of the required tables have non-standard table format and have been repeatedly configured, so it takes time to confirm the repeated configuration when adding the table.

2 > frequent abnormal interruptions on the target side due to frequent modification of the table structure by the application, and affect the normal synchronization of other tables in the process.

3 > due to maintenance needs to stop the process operation, too many long transactions are encountered, resulting in more tedious operation of skipping long transactions.

4 > the abnormal delay of the process caused by database exception leads to the lag of data synchronization and the failure of monitoring to notify in time, which leads to the interruption of the process caused by backup cleaning of archived files.

5 > due to the increase of lines, the operation of late operation and maintenance (annotation and table) becomes more and more tedious. At the same time, the configuration information of OGG is too confusing to query uniformly.

Therefore, it is necessary to optimize or improve the problems encountered in the current operation and maintenance through effective auxiliary means.

Second, the idea of script design

Use scripts to directly integrate duplicate table filtering and additional log add operations.

1. Optimize the operation of adding tables for large quantities of newly added tables

Direct manipulation of prm files

Backup automatically when editing the original file to ensure that it can be restored normally after an exception caused by misoperation

Each step is logged to ensure traceability of the operator

Tables with parameter conditions need to be adjusted manually

Note: this script needs to be formatted and annotated for all processes to take effect.

Approximate script function ER diagram (designed for requirements only, some of the functions have been completed and some have not yet been completed)

two。 One-click insertion of single table

This is designed separately according to the multi-table batch insert script test, aiming at a small number of additional table requirements.

The user name can be operated according to the domain name of the command table name area.

Quickly insert the format of the process, and complete the addition of additional logs

This script is only for EXTRACT and PUMP processes and cannot operate on REPLICAT processes.

Approximate script function ER diagram

3. Skip long transactions

In normal operation and maintenance, every time you add a table or modify the EXTRACT configuration, you must start and stop each process.

To take effect on the configuration. However, because of the long transaction in the database, the process must perform a long transaction skip operation in order to stop normally.

If there are too many long transactions, the operation is more tedious. You need to obtain the long transaction information through showtrans, then compare it, modify it into instructions, and execute it in ggsci.

Therefore, the script is specially designed to automate long transaction skipping operations on EXTRACT processes.

4. Batch comment

Used to respond to the need for business to cancel the synchronization configuration of related tables

Because some business parties need to clean up the historical data on the source side, because a large number of delete or update operations during the cleaning operation will lead to the generation of a large number of archive files. May result in high latency of the data link.

Therefore, the configured table information needs to be annotated.

Because when the amount of data cleaning is large, the number of tables may reach 40-50.

Adding notes one by one is sure to drive yourself crazy, so the batch comment script is designed when needed.

5. Batch uncomment

The function of this script is opposite to that of batch comment script.

It is mainly used to deal with the operation that requires uncomment after application data cleaning.

6. Information upload

This script is designed to send definition files, logs, or other related information generated by OGG to the specified target machine.

It can be used as an auxiliary function script for synchronizing definition files, sorting logs, or collating configuration information. Can be called in the specified format in other optimization scripts

Because it is sent through the SSH protocol, it needs to be configured on the firewall security policy in order to send it normally.

At the same time, it can be executed regularly by scheduling tasks.

7. Process configuration information query

The script acquires and collates the detailed configuration information table for the process through the process INFO information artificially filled in within the process.

Including source side, source side IP, sending destination side and IP, definition file and other information (can be adjusted according to demand)

It is divided into local query and upload HTML2 versions

8. Define automatic file synchronization

Because there are table structure exceptions between the source side and the destination side, OGG uses definition files to ensure that the data can be synchronized properly

However, because of the current operation and maintenance, some developers on the source side often modify the table structure frequently without warning.

Or because the definition file is corrupted and the configuration is wrong, the target side replicate process is abnormal.

As a result, there is a backlog of trail files on the target side, which leads to interlocking system abnormal problems such as disk space alarm.

Therefore, through the automatic synchronization script of the definition file, the definition file can be updated in real time to ensure the normal operation of the target-side replicate process.

Note: the script needs to configure the format annotation of the definition file to generate the prm file before it can take effect (see the description below). At the same time, it needs to configure the planned task on the source side, and the automatic restart parameters on the target side ensure that the writing process can restart and read the new definition file after the write process is interrupted due to an abnormal data format.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.