Shulou(Shulou.com)06/01 Report--

This article shows you whether Delete index in Elasticsearch will cause nodes to go offline, the content is concise and easy to understand, and it will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

My friend told me that when I Delete an index in his cluster, the node holding the index is offline from the cluster, and this situation can be stably repeated under certain conditions, the two seem to be not directly related, in fact, it is because of a new mechanism in Elasticsearch 7.x.

When master publishes the cluster status, it waits for other nodes to apply the cluster status. The timeout period is cluster.publish.timeout. By default, 30 seconds. If a node does not return a response after this time, the node is considered to be a lagging node. If the cluster.follower_lag.timeout fails to catch up with the master cluster status within 90 seconds by default, the node will be removed from the cluster.

When you remove a node, there will be logs at the following INFO level:

Node [{node-1}] is lagging at cluster state version [0], although publication of cluster state version [87] completed [1.5m] ago

Node-left [{node-1} lagging], term: 8, version: 89, reason: removed {{node-1}}

The state of the node application cluster is slow. It may be that the single execution is slow, exceeding the cluster.follower_lag.timeout time, or the process of multiple application cluster states is relatively slow. The reason for the slowness is not only the node GC, but also the lock. Corresponding to the topic of this article, the close process of Engine will be triggered when the index is deleted, which requires a read-write lock. This lock is used in refresh, flush, and some stages of recovery, etc. Therefore, if these processes are unusually slow, it may cause the application cluster state to block for a long time due to waiting for a lock, and be kicked out of the cluster by master after reaching the timeout.

Take version 7.4 as an example, when soft-delete is enabled, if many update operations on the document cause refresh to be very slow, Delete index or Close index will cause master to remove the nodes holding the index from the cluster.

The offline cost of a node is relatively high. When he rejoins the cluster, it takes a long recovery time. In theory, an index operation should not cause an impact on the cluster topology. However, from the cluster coordination level, if the node does not respond for a long time, it is understandable to kick it out of the cluster. This is the result of the coupling of the cluster coordination layer and control instructions. When HBase implements similar functions, master sends a RPC directly to ask the node to disable region (similar to shard in es), and then update the cluster metadata after it closes. If the node does not return a response for a long time, although it will not cause the node to go offline, it is easy to cause inconsistent state and other problems. There will be bug when the project is implemented carelessly. The implementation mechanism of es is simple, there are fewer errors, but the efficiency is lower, and it is easy to have a chain reaction when there is a problem.

The above content is whether Delete index in Elasticsearch will cause nodes to go offline. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.