Shulou(Shulou.com)05/31 Report--

This article mainly introduces the way to synchronize MySQL data to Elasticsearch what the relevant knowledge, the content is detailed and easy to understand, the operation is simple and fast, with a certain reference value, I believe that you will have some gains after reading this article on the way of synchronizing MySQL data to Elasticsearch, let's take a look at it.

1. Business layer synchronization

Because the operation of MySQL data is also done in the business layer, it is natural to synchronize other data sources in the business layer. It is more common to write synchronization code in the hooks hook of ORM.

The disadvantage of this approach is that as there are more and more services, the synchronized parts may be too fragmented, making it difficult to update iterations, such as incompatible migrations of ES indexes.

two。 Middleware synchronization

When the application architecture evolves into micro-services, MySQL may no longer be called directly in each service, but through a layer of middleware middleware, which can synchronize other data sources while the middleware operates MySQL.

This method requires middleware to adapt, which has a certain complexity.

3. Timing tasks are synchronized according to the updated_at field

Set special fields in the MySQL table structure, such as updated_at (data update time), according to this field, the scheduled task to query the actual changed data, so as to achieve incremental data update.

This way you can use the open source Logstash to do it.

Of course, the disadvantage is also obvious, that is, the deletion of data cannot be synchronized.

4. Parsing binlog synchronization

Such as the famous canal.

By pretending to be slave to parse the binary log of MySQL to know the changes in the data.

This is a relatively mature scheme in the industry.

This approach requires you to set the binlog-format of MySQL to ROW mode.

5. Parsing binlog-mixed / statement format

MySQL's binlog is available in three formats:

In ROW mode, binlog records data changes in a row-by-row manner

Statement mode, binlog records SQL statements

When mixed mode is mixed, each line change of SQL statement or ROW mode may be recorded.

In some cases, your MySQL binlog may not be set to ROW mode, in this case, we can still parse binlog uniformly to complete synchronization, but of course, what is parsed here is still the original SQL statement or every line change of ROW schema, which requires us to parse these SQL or every line change according to the business, such as using regular matching or AST abstract grammar tree, etc. Then the data is synchronized according to the results of the parsing.

The limitations of this method are also obvious. First, you need to adapt to the business resolution SQL, and second, it may be difficult to deal with the scenario of batch update. Of course, it can be better if your data is simply modified or deleted according to the primary key.

This is the end of the article on "what are the ways to synchronize MySQL data to Elasticsearch". Thank you for reading! I believe you all have a certain understanding of "how to synchronize MySQL data to Elasticsearch". If you want to learn more, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.