Shulou(Shulou.com)06/01 Report--

This article mainly introduces the relevant knowledge of "how to tune ElasticSearch cluster". The editor shows you the operation process through an actual case, and the operation method is simple, fast and practical. I hope this article "how to tune ElasticSearch cluster" can help you solve the problem.

Tuning at the system level

The tuning at the system level is mainly to set the memory and avoid swapping memory. After ES is installed, the default heap memory is 1GB, which is obviously not enough, so there is a question: how much memory should we set for ES?

In fact, it depends on the memory size of our cluster nodes, and also depends on whether we have to deploy other services on the server nodes.

If the memory is relatively large, such as 64g or more, and we do not deploy other services on the ES cluster, then I recommend that the ES memory be set to 31G-32G, because there is a 32G performance bottleneck.

To put it bluntly, even if you give the ES cluster more than 32 gigabytes of memory, its performance will not necessarily be better, or even worse than when it is set to 31G-32G.

Take the cluster I tuned as an example, the memory of the server node I tuned is 64G, and there are basically no other services running on the server node, so I set the memory size of the ES cluster to 31g to give full play to the performance of the cluster.

When setting up ES cluster memory, another thing is to make sure that the heap memory minimum value (Xms) is the same as the * * value (Xmx), so as to prevent programs from changing the heap memory size at runtime, which is a very resource-consuming process.

Another thing is to avoid swapping memory, which can be locked in the configuration file to avoid swapping memory (you can also turn off memory swapping at the operating system level).

Corresponding parameters:

Bootstrap.mlockall: true

Fragmentation and copy

Fragmentation (shard)

ES is a distributed search engine, the index is usually divided into different parts, and the part of data distributed in different nodes is fragmented.

ES automatically manages and organizes shards and rebalances the shard data when necessary, so users basically don't have to worry about the details of shard processing. The default number of shards when creating an index is 5, and cannot be changed once created.

Copy (replica)

ES creates a copy by default, that is, on the basis of five master shards, each master shard has a corresponding replica shard.

Additional copies have both advantages and disadvantages. Copies can be more resilient, but they also take up the disk space of the corresponding multiple of copies.

How many shards and replicas should we create when we create the index? For the number of copies, it is easier to determine, which can be determined according to the number of nodes in our cluster and our storage space.

We have a large number of cluster servers and enough storage space, so we can set more copies (usually 1-3 copies). If the cluster servers are relatively small and the storage space is not so loose, you can set only one copy to ensure disaster recovery (the number of copies can be adjusted dynamically).

For the number of fragments, it is difficult to determine, because once the number of index fragments is determined, it cannot be changed.

So before we create the index, we should fully consider the amount of data stored in the index we create in the future, otherwise we create an inappropriate number of fragments, which will have a great impact on our performance.

As for the number of shards, it is agreed that the number of shards is linked to memory. It is believed that the 1GB heap memory corresponds to 20-25 shards, while the size of one shard should not exceed 50 GB. This configuration is conducive to the health of the cluster.

However, I personally think that this configuration method is too rigid. In the process of tuning the ES cluster, I set the corresponding shard according to the total amount of data, to ensure that the size of each shard is no more than 50G (about 40G). However, compared with the previous number of shards, the effect is not obvious.

After that, we tried to increase the number of shards, and found that after the number of shards increased, the query speed was significantly improved, and the amount of data for each shard was controlled at about 10G.

Querying a large number of small shards makes each shard process data faster. Is it true that the more shards, the faster our query and the better ES performance?

In fact, it is not, because in the query process, there is a process of fragment merging, if the number of fragments continues to increase, the merging time will increase.

And as more tasks need to be queued and processed sequentially, more small shards are not necessarily faster than querying a smaller number of larger shards. If there are multiple concurrent queries, many small fragments can also reduce query throughput.

If your scene is now that the number of fragments is not appropriate, but do not know how to adjust, then a good solution is to create an index by time, and then do a wildcard query.

If there is a large amount of data every day, you can create an index on a daily basis, and if it is accumulated in a month resulting in a large amount of data, you can create an index every month.

If you want to refragment an existing index, you need to rebuild the index. I will summarize the process of rebuilding the index in the article.

Parameter tuning

Below I will introduce some tuning of key parameters of ES. There are many scenarios where how much CPU utilization is occupied by our ES cluster and how to adjust it?

CPU usage is high, which may be caused by writing or query, so how to check it?

You can first check the thread stack through GET _ nodes/ {node} / hot_threads to see which thread occupies high CPU:

If it is elasticsearch [{node}] [search] [tally 10], it is caused by the query.

If it is elasticsearch [{node}] [bulk] [data 1], it is caused by data writing.

In my actual tuning, the utilization rate of CPU is very high. If it is not SSD, it is recommended to set the number of index.merge.scheduler.max_thread_count: 1 index merge * * threads to 1. This parameter can effectively adjust the performance of writes.

Because writing simultaneously on the storage media, write performance will not improve because of addressing, but will only decrease.

There are also several important parameters that can be set, and students can depend on their own cluster and data conditions.

Index.refresh_interval: this parameter means that a few seconds after the data is written, it can be searched. The default is 1s.

The refresh of each index will generate a new lucene segment, which will lead to frequent merging behavior. If the business demand is not so demanding for real-time performance, you can increase this parameter. The actual tuning tells me that this parameter is really powerful, and the CPU utilization is plummeting.

Indices.memory.index_buffer_size: if we are going to do very heavy high concurrent writes, then * make the indices.memory.index_buffer_size a little larger.

The size of index buffer is common to all shard. It is generally recommended that for each shard, give 512mb at most, because no matter how big the performance is, there will be no improvement.

ES will use this setting as a shared index buffer for each shard, and those shard that are particularly active will use this buffer more. The default value of this parameter is 10%, which is 10% of jvm heap.

Translog:ES in order to ensure that data is not lost, every time index, bulk, delete, and update are completed, the refresh translog will be triggered to disk.

While improving data security, it also reduces a little bit of performance. If you don't care about this possibility, or you want performance first, you can set the following parameters:

"index.translog": {"sync_interval": "120s",-- sync interval increase "durability": "async",-- Asynchronous update "flush_threshold_size": "1g"-- log file size}

This setting means that asynchronously writing to disk is turned on, and the interval and size of writes are set, which helps to improve write performance.

There are also some settings for timeout parameters:

Discovery.zen.ping_timeout found other timeout settings for node survival during the master election process.

The frequency at which the discovery.zen.fd.ping_interval node is ping is used to detect whether the node is alive.

The response time for the survival of discovery.zen.fd.ping_timeout nodes is 30 seconds by default. If there may be hidden dangers in the network, you can increase it appropriately.

How much discovery.zen.fd.ping_retries ping failure / timeout causes the node to be considered a failure, which defaults to 3.

Other suggestions

There are also some piecemeal optimization suggestions as follows:

Insert index automatically generates id: when the writer writes data to ES using a specific id, ES checks whether the same id exists under the corresponding index.

This operation will consume more and more with the increase of the number of documents, so if there is no rigid requirement in the business, it is recommended to use id automatically generated by ES to speed up the write rate.

Avoid sparse indexes: sparse indexes can cause the index file to grow. The keyword of ES, and the array type adopts doc_values structure.

Even if the field is null, each document takes up a certain amount of space, so sparse indexes can cause the disk to grow, resulting in inefficient queries and writes.

My tuning

Let's talk about my tuning: mainly to rebuild the index, change the number of shards of the existing index, after continuous testing, found a * number of shards.

It takes a long time to rebuild the index, and during this period, ES writes are tuned accordingly to reduce CPU usage.

Attach my tuning parameters:

Index.merge.scheduler.max_thread_count:1 # Index merge * number of threads indices.memory.index_buffer_size:30% # memory index.translog.durability:async # this can write to the hard disk asynchronously Increase the write speed index.translog.sync_interval:120s # translog interval discovery.zen.ping_timeout:120s # heartbeat timeout discovery.zen.fd.ping_interval:120s # Node detection time discovery.zen.fd.ping_timeout:120s # ping timeout discovery.zen.fd.ping_retries:6 # heartbeat retries thread_pool.bulk.size:20 # number of writing threads due to me Our query threads are all set in the code. I only adjusted the number of threads written here thread_pool.bulk.queue_size:1000 # write thread queue size index.refresh_interval:300s # index refresh interval copy code

About re-indexing

Before rebuilding the index, first consider the need to rebuild the index, because rebuilding the index is very time-consuming.

The reindex api of ES will not attempt to set the target index and will not copy the settings of the source index, so we should set the target index before running the _ reindex operation, including mapping, shards, replicas, and so on.

* step to create a new index just like creating a normal index

When the amount of data is large, you need to set the refresh interval and set refresh_intervals to-1, that is, no refresh.

The number of number_of_replicas copies is set to 0 (because the number of copies can be adjusted dynamically, which helps speed up).

{"settings": {"number_of_shards": "50", "number_of_replicas": "0", "index": {"refresh_interval": "- 1"}} "mappings": {}}

The second step is to call reindex interface

It is recommended to add the parameter condition of wait_for_completion=false, so that reindex will return taskId directly.

POST _ reindex?wait_for_completion=false {"source": {"index": "old_index", / / original index "size": 5000 / / amount of data processed in a batch}, "dest": {"index": "new_index", / / target index}}

Step 3: wait

You can query the progress of the rebuild through GET _ tasks?detailed=true&actions=*reindex. Call _ tasks/node_id:task_id/_cancel if you want to cancel task.

Step 4: delete the old index and free up disk space

For more details, see the reindex api on ES's official website. So some students may ask, what if I write ES in real time at the moment?

At this point, when we are about to rebuild the index, add the timestamp of the last re-index to the parameter.

To put it bluntly, for example, our data is 100G, when we rebuild the index, but this 100G is increasing, then when we rebuild the index, we need to record the time stamp of the re-index.

The purpose of recording the timestamp is that the next time the index is rebuilt, it only needs to be rebuilt after the timestamp, so iterate until the amount of data of the new and old indexes is basically the same, and the data flow is switched to the name of the new index.

POST / _ reindex {"conflicts": "proceed", / / means that the conflict is subject to the old index and skips the conflict directly, otherwise an exception will be thrown Stop task "source": {"index": "old_index" / / Old index "query": {"constant_score": {"filter": {"range": {"data_update_time": {"gte": 123456789 / / millisecond timestamp before the start time of reindex}} "dest": {"index": "new_index", / / New index "version_type": "external" / / based on the data of the old index} this is the end of the introduction to "how to tune the ElasticSearch cluster". Thank you for your reading. If you want to know more about the industry, you can follow the industry information channel. The editor will update different knowledge points for you every day.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.