Shulou(Shulou.com)06/03 Report--

ES as a NoSQL, a typical application scenario is to store data. That is, users can add data to es through api. Due to the internal implementation of Lucene, the data added each time is not available in real time. Instead, the index information is maintained in memory, and the data will not fall to the disk until the buffer is full or there is an explicit commit, forming a segement that is saved in the file.

So if, for some reason, the ES process suddenly dies, the data in memory will be lost. In fact, the user calls api and returns the result to confirm that the user data has been added to the index. This kind of data loss is unacceptable. How to solve this problem?

ES implements Translog, which is written to the log file before the data is indexed. If the node dies, the log will be replayed when the node is restarted, which is tantamount to simulating the user's operation. It ensures that the data is not lost.

Learn about the details of the implementation through the source code of ES. Let's start with the Translog class.

The Translog class is a component at the index fragment level, that is, one Translog class per index shard. Its purpose is to record uncommitted index operations in a persistent manner (that is, to write to a file).

InternalEngine records the latest translog generation in commit metadata. Through this generation, you can associate to all operations that do not have commit.

Each Translog instance will have only one translog file in the open state at any time. This translog file and translog generation ID are mapped one by one.

For performance reasons, post-disaster reconstruction is not about playing back all the translog, but the latest part of the uncommitted index. So there must be a checkpoint, the translog.ckp file.

To sum up, looking at the translog mechanism from a file perspective is actually two files:

$tree translogtranslog ├── translog-11.tlog └── translog.ckp

Translog records logs in the following format: | record size | unique id of the operation | content of the operation | checksum | the location returned by each add operation will be recorded in versionMap, so that the function of realtime get can be realized.

With this in mind, there are two ways to improve the performance of ES indexes when configuring es.

a. Configure translog days and indexes to different disks. b. Set the flush interval for translog to be longer. For example, the following parameters: index.translog.sync_interval: 30s index.translog.durability: "async" index.translog.flush_threshold_size: 4G index.translog.flush_threshold_ops: 50000

When you understand the mechanism of translog, you will find that even the translog mechanism can not completely avoid data loss. Some balance still needs to be made in terms of performance and data loss tolerance.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.