Shulou(Shulou.com)05/31 Report--

This article mainly introduces "the course of overall design method of MySQL data transmission service based on Maxwell". In the daily operation, I believe that many people have doubts on the tutorial of overall design method of MySQL data transmission service based on Maxwell. Xiaobian consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts of "overall design method tutorial of MySQL data transmission service based on Maxwell". Next, please follow the editor to study!

1. Overall system design

The overall design of the system is as follows:

DTS is the front end of the platform, which can be supported by a professional front-end team.

Data transmission system DTS is an independent business system, which is currently rebuilt. In order to achieve the initial version, the baseline version is the current version of the database operation and maintenance system, and only the relevant logic on the DTS side is maintained.

In order to consider the subsequent expansibility and maintainability, DTS will be built based on three general modules of reader,write,service, and reader,writer can be subdivided according to the specific technical direction.

The related systems are database operation and maintenance system, task system and big data management system. There will be related calls to the metadata and some logic functions of these related systems.

In the whole data transfer service process, a basic attribute is task_code, which is a common attribute in DTS task creation, mid-range data transfer, and back-end service integration. Task_code means client_id in the format: dts_ [idc] _ [maxwell_ip] _ [maxwell_code]. The topic of the back-end service is composed based on client_id additional database and table information.

When the basic information (such as database IP, port, etc.) is entered in the front-end DTS page, the middle-end service API will be called to generate the corresponding client_id. The back-end service will manage the tasks based on the task_code in the DTS task list, while the middle-end service will perform corresponding operations such as synchronous object configuration management / service start and stop according to the client_id.

The related data transfer streams are as follows:

two。 Mid-range management design

The middle management is mainly based on Maxwell deployment management, configuration management, synchronous object list change, service management (start, stop), service self-management and monitoring alarm. The current implementation is mainly based on API, the preliminary implementation of the local front end.

2.1. Deployment management

Deployment management is mainly to manage the deployment of Maxwell services on a platform. At present, there are two Maxwell application servers, 130.200 and 130.201 respectively. The archive database is the basic configuration data of Maxwell services and the basic database metadata initialized by Maxwell startup.

Through platform deployment, you can achieve the following functions:

1) it can realize automatic addressing, add synchronization tasks for Maxwell to match corresponding application servers, and quickly adjust and adapt in the case of subsequent new application servers.

2) if the maxwell service of instance B has been deployed on server A, the existing maxwell service configuration needs to be reused when adding data synchronization tasks, and there is no need to reconfigure it in the new server.

3) basic preparations for service configuration, such as Maxwell-related accounts and firewall opening, need to be supported by scripted management and provide access in the form of API. Currently, the unified maxwell access account is: dba_maxwell_repl.

The Master end of the main database of ① opens firewall permissions to create corresponding database accounts.

② database enables firewall permissions from the Slave side of the library

4) the topology management of MySQL service is based on the database operation and maintenance management system, which needs to encapsulate the corresponding API to get Slave information, and needs to be maintained and managed in the Maxwell configuration list.

Add:

Predecessor tasks:

Deploy the revised Maxwell service on the 130.201 server

Complete the access test of Maxwell based on the test environment, and producer is stdout

For subsequent development, you can refer to the following implementation points:

① completes Maxwell templated configuration

② can configure templates to generate customized configurations and encapsulate unified start and stop scripts

③ Maxwell automatic addressing logic. Only one Maxwell process service can be reused in an instance.

④ Client_id generation logic

⑤ Maxwell code generation logic

⑥ encapsulates the interface on the DTS side based on the operation and maintenance system, and realizes the functions of firewall activation management and new replication users. The newly created users are in Master, and the permissions are enabled in Master and Slave.

⑦ implements mutex logic for stdout and Kafka configurations

2.2. Configuration management

Configuration management contains maxwell-based configuration files, such as config configuration, log configuration, and monitoring configuration. The catalogue planning and design are as follows:

On this basis, the service can be extended accordingly.

The basic configuration of Maxwell depends on client_id

ü client_id metadata naming

Dts_ [idc] _ [maxwell_ip] _ [maxwell_code]

For example, Maxwell is deployed on server 121.240

Maxwell001 is the business code, which can be mapped to the corresponding database service (such as Slave). In order to facilitate identification and parsing, it is not allowed to include special symbols, such as underlining, etc.

The naming example is: dts_xxxx_121_240_maxwell001

The implementation details can be sorted into the following sections:

1) the corresponding maxwell configuration file can be generated according to the configuration of the template. The configuration file is named after client_id metadata in the format of dts_ [idc] _ [maxwell_ip] _ [maxwell_code] .properties. The configuration file is deployed in the app_config directory.

For example, Maxwell is deployed on server 130.200

Maxwell001 is the business code, which can be mapped to the corresponding database service (such as Slave). In order to facilitate identification and parsing, it is not allowed to include special symbols, such as underlining, etc.

The naming example is: dts_xxxx_130_200_maxwell001.properties

2) for the configuration of Maxwell basic information, because it is based on client_id, it is not easy to display the association relationship at the source end, and the management of this configuration information needs to be maintained in the maxwell archive.

3) basic configuration information includes source service information, source service port, replication account information, client_id, home maxwell application server, monitoring port, filter list, bootstrap_type (sync,async), kafka configuration information, etc.

4) basic configuration management needs to implement local front end

Add:

Complete the platform management of Maxwell service list and service configuration, and the backend is all implemented by API

If the task configuration changes, there is an one-to-many relationship between the service list and the details list, that is, you need to keep the existing configuration file record, and then set the original record status to invalid, so that in the event of an abnormal fallback, this task still ensures that the service is available.

2.3. Synchronize object list changes

The list of synchronization objects is the key management object in data transmission, and the following functions need to be implemented:

1) when adding a table to an existing maxwell service, you need to expand it under the existing maxwell service to modify the synchronization object list. The modification mode of the list is appended, and other modes such as modification and deletion are not supported.

2) after modifying the synchronization list, the maxwell service needs to be restarted, and the startup process needs to be relatively smooth and controllable.

3) if the synchronization list fails to refresh, you need to be able to roll back quickly and restore the data transfer service quickly.

4) version management and archiving of configuration files. Before object changes, you need to back up the configuration files and standardize the naming of backup files.

Add:

Mainly realize the modification and management of synchronous objects, add the corresponding regular configuration, and call the method / interface of detail management.

2.4. Service management

Can realize the start and stop management function of maxwell, including three sub-options; start,stop,restart

Service status check is carried out through API mode, and existing open interfaces can be reused.

2.5. Service self-starting

In the transmission link, if the source end service exception, maxwell side exception or back-end transport service exception will lead to the abnormal termination of the Maxwell service, you need to check and check through the probing mode, and restart the service after confirmation to ensure the persistence of the maxwell service.

Add:

The corresponding script is developed to detect the service status, and the existing monitored API is reused to detect the data transmission status.

3.6. Monitoring and alarm

Monitoring and alarm is the basic guarantee function of maxwell, which needs to monitor the data transmission of Maxwell service, analyze the throughput of binlog, push down the throughput of Kafka, and give an alarm to abnormal situations such as abnormal data transmission and service.

Add:

Configure the corresponding alarm details, set the threshold, and configure the alarm

At this point, the study on the "overall Design method course of Maxwell-based MySQL data transfer Service" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.