How to restart an es cluster when an es cluster has a copy

When you need to perform a rolling restart of the cluster, the cluster is kept online and running, but one node at a time is taken offline.

Common reasons are Elasticsearch version upgrades or some maintenance of the server itself (such as OS updates or hardware). In any case, there is a special way to perform a rolling restart.

In essence, Elasticsearch wants your data to be fully replicated and balanced. If a single node is shut down for maintenance, the cluster immediately identifies the missing node and begins to rebalance. This can be annoying if you know that node maintenance is short-term, because it may take some time for very large shards to be rebalanced.

What we want to do is tell Elasticsearch to postpone rebalancing because we know more about the state of the cluster caused by external factors. The procedure is as follows:

If possible, stop indexing new data and perform a synchronous refresh. This is not always possible, but it helps speed up recovery time. The synchronous refresh request is a "best effort" operation. If there are any pending index operations, it will fail, but you can safely reissue the request multiple times if necessary.

POST / _ flush / synchronously disables sharding allocation. This prevents Elasticsearch from rebalancing lost shards unless you specify otherwise. If you know that the maintenance window is short, this is a good idea. You can disable allocation as follows:

Curl xput-d' {"transient": {"cluster.routing.allocation.enable": "none"} 'stop the node that needs to be restarted: curl xput http://ip:port/_cluster/node/_local/_shutdown

Restart the node and confirm that it joins the cluster.

Repeat steps 3 and 4 for the remaining nodes.

Re-enable sharding allocation as follows:

Curl xput http://ip:9092/_cluster/shard/setting-d' {"transient": {"cluster.routing.allocation.enable": "all"} 'fragments may take some time to rebalance. Wait until the cluster returns to the status green before continuing.

At this point, you can safely restore the index (if you stopped it before), but waiting for the cluster to be fully balanced before restoring the index will help speed up the process.