Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you an example analysis of the implementation of canal synchronous es based on canal client. The article is rich in content and analyzed and described from a professional point of view. I hope you can get something after reading this article.

Canal is a pure java language mysql incremental subscription tool provided by Ali. It is mainly disguised as a slave node to send the official principle of the dump protocol to mysql as shown below:

Official explanation:

Server represents a canal running instance, corresponding to a jvm

Instance corresponds to a data queue (1 server corresponds to 1.. n instance)

Individuals understand instance as a topic queue in mq (broker in kafka), and canal client is equivalent to a consumer in mq. Instance receives data parsed by parse, then processed by sink, and finally stored in sore, which is consumed by canal client.

Due to the large amount of business data, with a single table exceeding 100 million, the database is divided into databases and tables according to the actual business. The corresponding relationship between the primary key and the account number is stored in es. When accessing the database, the primary key is first sent to es to obtain the primary key, and then the db is checked. In this way, you can directly call back to the corresponding slice in a second.

At first, we planned to use the official tools, but later found that the official ones didn't quite meet our needs (maybe we didn't understand the official tools), and we were divided into databases and tables, which was too cumbersome according to the official configuration. There are hundreds of tables from one table to the bottom, so I decided to implement synchronous es, full quantity and increment based on canal client. But I stepped on a lot of words.

Batch updates using bulk api from es. Set up a custom template, and then read the official source code, in fact, roughly the same way, but the official one is universal.

Startup method:

Connect the canal server code:

The code for parsing binlog is also very simple, but your own implementation is not universal, so don't spray:

Assemble the data and submit it in batch:

The difference between es and msyql varies from 10 to 100. the reason found later is that the setting batch is 1000 and the database update operation tps is low, resulting in data delay between the two. The batch number is reduced one by one and the test is finally adjusted to 50 batches, and the es and mysql data delay is reduced to millimeter.

Full synchronization is relatively simple. The individual is realized in this way. First of all, export the amount of data in mysql to a text file, we only need two fields: primary key and account number, and it is divided into libraries and tables, and the export is very fast. 2 billion of the data is exported within 30 seconds. Then read the file through the buffer stream line by line, call bulk api to batch into es,es 5 data sections, 5 shards and one copy, single node 40 core physical machine memory 512g jvm set 31g raid0 SAS disk, 100 million data can be imported into es within 5 minutes. Currently, no exception has been found in online operation for two months.

The above is the analysis of the example of realizing canal synchronous es based on canal client shared by Xiaobian. If you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.