Shulou(Shulou.com)06/02 Report--

Today I will show you how to practice Elasticsearch tuning. The content of the article is good. Now I would like to share it with you. Friends who feel in need can understand it. I hope it will be helpful to you. Let's read it along with the editor's ideas.

Background

As an organic combination of NOSQL+ search engine, Elasticsearch (ES) not only has near real-time query ability, but also has strong aggregation analysis ability. Therefore, ES is widely used in the fields of full-text retrieval, log analysis, monitoring system, data analysis and so on. The complete Elastic Stack system (Elasticsearch, Logstash, Kibana, Beats) provides a complete set of solutions for data acquisition, cleaning, storage and visualization.

Based on ES 5.6.4, this paper introduces the basic scheme of ES tuning from two aspects of performance and stability, from the perspectives of linux parameter tuning, ES node configuration and ES usage. Of course, ES tuning can not be generalized, it needs to make appropriate trade-offs and adjustments according to the actual business scenarios, and the omissions in the article are also welcome to be criticized and corrected at any time.

Performance tuning

One Linux parameter tuning

1. Turn off the swap partition to prevent memory swapping from degrading performance. Release the line in the / etc/fstab file that contains swap

Sed-I'/ swap/s/ ^ / # /'/ etc/fstabswapoff-a

two。 Disk mount option

Noatime: disable recording access timestamps to improve file system read and write performance

Data=writeback: do not record data journal to improve file system write performance

Barrier=0:barrier ensures that journal is brushed to the disk before data. If journal is turned off, there is no need to open barrier here.

Nobh: turn off buffer_head to prevent the kernel from interrupting the IO operation of large chunks of data

Mount-o noatime,data=writeback,barrier=0,nobh / dev/sda / es_data

3. For SSD disks, the elevator scheduling algorithm is used, because SSD provides a more intelligent request scheduling algorithm, which does not require the kernel to make redundant adjustments (for reference only)

Echo noop > / sys/block/sda/queue/scheduler

Two ES node configuration

Conf/elasticsearch.yml file:

1. Appropriately increase the write buffer and bulk queue length to improve write performance and stability indices.memory.index_buffer_size: 15%thread_pool.bulk.queue_size: 1024

two。 When calculating disk usage, the shard being relocated is not taken into account

In large clusters, you can prevent the metadata of all shard from being scanned when creating a new shard, thus increasing the speed of shard allocation.

Cluster.routing.allocation.disk.include_relocations: false

3 ways of using ES

1. Control the storage options for the field

The underlying layer of ES uses Lucene to store data, which mainly includes three parts: row memory (StoreFiled), column memory (DocValues) and inverted index (InvertIndex). In most usage scenarios, it is not necessary to store these three parts at the same time. You can make appropriate adjustments through the following parameters:

StoreFiled: row storage, of which the largest proportion is the source field, which controls the storage of doc raw data. When writing data, ES treats the entire json structure of doc raw data as a string and stores it as a source field. When querying, you can get the entire json structure when it was written through the source field. Therefore, if there is no need to pull out the entire original json structure, you can use the following command to close the source field in mapping or store only some of the fields in source, and you can still get the values of all fields through ES's docvaluefields when querying data.

Note: after shutting down source, update, updatebyquery, reindex and other APIs will not work properly, so index with update and other requirements cannot shut down source.

# close _ sourcePUT my_index {"mappings": {"my_type": {"_ source": {"enabled": false} # _ source stores only some fields Specify fields to store through includes or filter unwanted fields through excludes PUT my_index {"mappings": {"_ doc": {"_ source": {"includes": ["* .count", "meta.*"], "excludes": ["meta.description" "meta.other.*"]}

Docvalues: controls column storage.

ES mainly uses column storage to support sorting, aggregations and scripts functions. For fields that do not have the above requirements, you can use the following command to turn off docvalues to reduce storage costs.

PUT my_index {"mappings": {"my_type": {"properties": {"session_id": {"type": "keyword", "doc_values": false}

Index: controls the inverted index.

ES enables inverted indexing for all fields by default for query purposes. For fields that have no query requirements, you can turn off the inverted index with the following command.

PUT my_index {"mappings": {"my_type": {"properties": {"session_id": {"type": "keyword", "index": false}

A special field of all:ES, ES splices all the field values written by users into json into a string, makes a participle, and then saves the inverted index to support full-text search of the entire json.

This requirement applies to fewer scenarios. You can use the following command to close the all field to save storage costs and cpu overhead. (the _ all field is no longer supported in versions above ES 6.0 + and does not need to be set)

PUT / my_index {"mapping": {"my_type": {"_ all": {"enabled": false}

Fieldnames: this field is used for exists queries to confirm whether a field exists in a doc. If there is no such requirement, you can turn it off.

PUT / my_index {"mapping": {"my_type": {"_ field_names": {"enabled": false}

two。 Turn on optimal compression

For index with the above _ source field turned on, you can use the following command to replace the compression algorithm applicable to lucene with DEFLATE to improve the data compression ratio.

PUT / my_index/_settings {"index.codec": "best_compression"}

3. Bulk batch write

When writing data, try to use the following bulk interface to write in batches to improve writing efficiency. The set range of doc quantity for each bulk request is recommended as 1k~1w. You can select an appropriate number according to the business scenario.

POST _ bulk {"index": {"_ index": "test", "_ type": "type1"} {"field1": "value1"} {"index": {"_ index": "test", "_ type": "type1"} {"field1": "value2"}

4. Adjust translog synchronization policy

By default, translog's persistence policy is to flush once for each write request to refresh the translog data to disk. This frequent disk IO operation seriously affects write performance. If you can accept a certain probability of data loss (the probability of hardware failure is very small), you can use the following command to adjust the translog persistence policy to be executed asynchronously and periodically, and adjust the flush cycle of translog appropriately.

PUT my_index {"settings": {"index": {"translog": {"sync_interval": "5s", "durability": "async"}

5. Adjust refresh_interval

The data written into Lucene is not searchable in real time. ES must convert the data in memory into the complete segment of Lucene through the process of refresh before it can be searched. By default, ES refresh every second, resulting in a new segment, which results in more segment, resulting in more frequent segment merge and higher system overhead. If you have low requirements for real-time visibility of data, you can increase the refresh interval and reduce system overhead by using the following command.

PUT my_index {"settings": {"index": {"refresh_interval": "30s"}

6. Merge concurrency control

An index of ES is composed of multiple shard, and a shard is actually an index of Lucene, which is composed of multiple segment, and Lucene will constantly merge some small segment into a large segment, a process called merge. The default value is Math.max (1, Math.min (4, Runtime.getRuntime (). AvailableProcessors () / 2)). When the number of cpu cores configured by a node is high, the resources consumed by merge may be high, affecting the performance of the cluster. You can adjust the concurrency of the merge process of an index by using the following command:

PUT / my_index/_settings {"index.merge.scheduler.max_thread_count": 2}

7. Write data without specifying _ id and let ES generate it automatically

When the user displays the data written by the specified id, the ES will first launch a query to determine whether the doc of the same id already exists in the index, and if so, delete the old doc before writing the new doc. In this way, ES will spend a certain amount of resources on querying each time it is written. If the user does not specify doc,ES when writing data, a random id is generated through the internal algorithm, and the uniqueness of the id is guaranteed, so that the previous step of querying id can be skipped and the writing efficiency can be improved. Therefore, in scenarios where there is no need to remove duplicates through the id field and update is used, writing without specifying id can increase the write rate. The test results of the database team of the Infrastructure Department show that the data write performance without id may be nearly twice as high as that with _ id, and the actual loss is related to the specific test scenario.

# specify _ idPOST _ bulk {"index": {"_ index": "test", "_ type": "type1", "_ id": "1"} {"field1": "value1"} # write without specifying _ idPOST _ bulk {"index": {"_ index": "test" "_ type": "type1"} {"field1": "value1"}

8. Use routing

For index with a large amount of data, it is common to configure multiple shard to share the pressure. In this scenario, a query searches all shard at the same time, and then merges the results of each shard and returns them to the user. For small query scenarios with high concurrency, each shard usually grabs only a very small amount of data, so the scheduling overhead in the query process is much greater than the actual data reading overhead, and the query speed depends on the slowest shard. When the routing feature is enabled, ES will write the same data of routing to the same shard (or multiple ones, controlled by the index.routingpartitionsize parameter). If routing is specified in the query, then ES will only query the shard pointed to by routing, which can significantly reduce the scheduling overhead and improve the query efficiency. Routing is used as follows:

# write PUT my_index/my_type/1?routing=user1 {"title": "This is a document"} # query GET my_index/_search?routing=user1,user2 {"query": {"match": {"title": "document"}

9. Select the appropriate storage method for fields of string type

Fields saved as text (default type of string field is text): inverted index is stored after word segmentation, and full-text retrieval is supported. The storage method can be optimized by the following parameters:

Norms: the _ score used to calculate the doc during the search (representing the relevance of this data to the search criteria). If you don't need a score, you can turn it off.

Indexoptions: controls what information is included in the inverted index (docs, freqs, positions, offsets). For usage scenarios that do not pay much attention to score/highlighting, you can set it to docs to reduce memory / disk resource consumption.

Fields: used to add subfields. For scenarios with sort and aggregate query requirements, you can add a keyword subfield to support both functions.

PUT my_index {"mappings": {"my_type": {"properties": {"title": {"type": "text", "norms": false, "index_options": "docs" "fields": {"raw": {"type": "keyword"}

Fields saved as keyword type: no word segmentation, full-text search is not supported. Text participle consumes CPU resources, and the redundant storage keyword subfield takes up storage space. If there is no need for full-text indexing, but only to search through the entire field, you can set the type of the field to keyword to increase the write rate and reduce the storage cost. There are two ways to set the field type: one is to specify the field type when creating a specific index; the other is to control the field type of a certain type of index by creating a template.

# 1. Specify the tags field through mapping as the keyword type PUT my_index {"mappings": {"my_type": {"properties": {"tags": {"type": "keyword"} # 2. Specify the index of the my_index* class through template All its string fields default to the keyword type PUT _ template/my_template {"order": 0, "template": "my_index*", "mappings": {"_ default_": {"dynamic_templates": [{"strings": {"match_mapping_type": "string" "mapping": {"type": "keyword", "ignore_above": 256}]}}, "aliases": {}}

10. When querying, use query-bool-filter combination instead of normal query

By default, ES calculates the correlation between each piece of data returned and the query statement through a certain algorithm, and represents it by the score field. However, for non-full-text indexing scenarios, users do not care the correlation between query results and query conditions, but just want to find the target data accurately. At this point, we can use query-bool-filter combination to make ES not calculate score, and cache the result set of filter as much as possible for subsequent queries containing the same filter to improve query efficiency.

# ordinary query POST my_index/_search {"query": {"user": "Kimchy"}} # query-bool-filter accelerated query POST my_index/_search {"query": {"bool": {"filter": {"term": {"user": "Kimchy"}

11. Index scrolls by date for easy management

Data written to ES is best segmented in some way and stored in different index. A common practice is to sort the data by module / function, write it to different index, and then scroll to generate index over time. The advantage of this is that the separate management of all kinds of data will not be confused, and it is easy to improve the query efficiency. At the same time, index scrolls over time, and deleting the entire index when the data expires is much more efficient than deleting the data or deletebyquery one by one, because deleting the entire index directly deletes the underlying files, while deletebyquery is query-tag-delete.

For example, if you have data generated by [modulea,moduleb] two modules, the index plan can look like this: one type of index name is modulea + {date}, and the other type of index name is module_b+ {date}. For the date in the name, you can specify the exact date when you write the data, or you can do it through index-name-processor in ES's ingest pipeline (with write performance loss).

# module_a class index- creates index:PUT module_a@2018_01_01 {"settings": {"index": {"number_of_shards": 3 "number_of_replicas": 2}} PUT module_a@2018_01_02 {"settings": {"index": {"number_of_shards": 3 "number_of_replicas": 2}}.-query data: GET module_a@*/_search# module_b class index- creates index:PUT module_b@2018_01_01 {"settings": {"index": {"number_of_shards": 3 "number_of_replicas": 2}} PUT module_b@2018_01_02 {"settings": {"index": {"number_of_shards": 3, "number_of_replicas": 2}}.-query data: GET module_b@*/_search

twelve。 Control the number of fragments and copies of index as needed

Shard: the index of an ES consists of multiple shard, and each shard carries part of the data of the index.

Replica: index can also set the number of copies (numberofreplicas), that is, how many backups there are in the same shard. For index with high query pressure, you can consider increasing the number of copies (numberofreplicas) and sharing the query pressure through multiple copies.

Setting too much or too low number of shard (numberofshards) will cause some problems: if the number of shard is too large, the batch write / query request will be divided into too many sub-writes / queries, resulting in an increase in the write and query rejection rate of the index; for an inex with a large amount of data, when its shard number is too small, it can not make full use of node resources, resulting in low or uneven machine resource utilization, affecting the efficiency of write / query.

The number of shard for each index can be set according to the total amount of data, writing pressure, number of nodes and other comprehensive considerations, and then regularly check whether the number of shard is reasonable according to the data growth status. The recommended solution for the Infrastructure Database team is:

For index with a small amount of data (less than 100GB), the query pressure is often relatively low. Generally, you can set 3-5 shard,numberofreplicas to 1 (that is, one master, one slave, two copies in total).

For index with a large amount of data (above 100GB):

Generally, the amount of data in a single shard is limited to (20GB~50GB).

Distribute the index pressure to multiple nodes: you can use the index.routing.allocation.totalshardsper_node parameter to force the number of shard of the index on a node to be allocated to different nodes as far as possible.

Considering the number of shard of the entire index, if the number of shard (excluding copies) exceeds 50, it is likely to cause an increase in the rejection rate. At this time, you can consider splitting the index into multiple independent index, sharing the amount of data, and using it with routing to reduce the number of shard that needs to be accessed for each query.

Stability optimization

One Linux parameter tuning

# modify system resource limits

# the maximum number of files that a single user can open can be set to 65536 or more of the official recommendation

Echo "*-nofile 655360" > > / etc/security/limits.conf

# single user memory address space

Echo "*-as unlimited" > > / etc/security/limits.conf

# number of threads per user

Echo "*-nproc 2056474" > > / etc/security/limits.conf

# single user file size

Echo "*-fsize unlimited" > > / etc/security/limits.conf

# Lock memory by single user

Echo "*-memlock unlimited" > > / etc/security/limits.conf

# maximum number of map memory areas that can be used by a single process

Echo "vm.max_map_count = 655300" > > / etc/sysctl.conf

# TCP full connection queue parameter setting. The purpose of this setting is to prevent the full connection queue in the ES cluster with a large number of nodes (for example, more than 100 nodes) from filling up at the moment of startup when the node is abnormally restarted, resulting in the delay of the response of the whole cluster when the node hang resides.

Echo "net.ipv4.tcp_abort_on_overflow = 1" > > / etc/sysctl.conf

Echo "net.core.somaxconn = 2048" > > / etc/sysctl.conf

# reduce tcp alive time to prevent invalid links from taking up the number of links

Echo 300 > / proc/sys/net/ipv4/tcp_keepalive_time

Two ES node configuration

1. Jvm.options

-Xms and-Xmx are set to the same value, which is recommended to set to about half of the machine memory, and the remaining half is left to the system cache.

Jvm memory is recommended not to be less than 2G, otherwise ES may not start normally or OOM may be caused by insufficient memory.

Jvm recommends no more than 32G, otherwise jvm will disable memory object pointer compression technology, resulting in memory waste

2. Elasticsearch.yml

Set memory fuse parameters to prevent OOM caused by high write or query pressure. The specific value can be adjusted according to the usage scenario. Indices.breaker.total.limit: 30% indices.breaker.request.limit: 6% indices.breaker.fielddata.limit: 3%

Reduce the cache used by the query to prevent cache from taking up too much jvm memory. The specific value can be adjusted according to the usage scenario. Indices.queries.cache.count: 500 indices.queries.cache.size: 5%

When there are multiple nodes on a single machine, the master-slave shard allocation is based on ip and is assigned to different machines to avoid data loss caused by single machine hanging. Cluster.routing.allocation.awareness.attributes: ip node.attr.ip: 1.1.1.1

3 ways of using ES

1. For clusters with more nodes, add proprietary master to improve the stability of the cluster

The tasks of cluster management such as meta-information management of ES cluster, addition and deletion of index, addition and deletion of nodes are all responsible for by master node, and master node broadcasts the latest cluster status to each node regularly. Therefore, the stability of master is very important to the overall stability of the cluster. When the number of nodes in the cluster is large (for example, more than 30 nodes), the management of the cluster becomes much more complex. At this point, you should create proprietary master nodes, which are only responsible for cluster management, do not store data, and do not bear the pressure of data reading and writing; other nodes are only responsible for data reading and writing, but not for cluster management.

In this way, the cluster management and data writing / query are separated, and do not affect each other, so as to prevent the overall instability of the cluster caused by the excessive pressure of reading and writing. To separate the proprietary master node from the data node, you need to modify the ES configuration file, and then scroll to restart each node.

# the configuration file (conf/elasticsearch.yml) of the proprietary master node adds the following attributes: node.master: true node.data: false node.ingest: false # the configuration file of the data node adds the following attributes (as opposed to the above attributes): node.master: false node.data: true node.ingest: true

two。 Control the total quantity of index and shard

As mentioned above, the meta-information of ES is managed by the master node and is periodically synchronized to each node, that is, a copy of the meta-information is stored by each node. This meta-information is mainly stored in clusterstate, such as all node meta-information (indices, various statistical parameters of nodes), all index/shard meta-information (mapping, location, size), metadata ingest, and so on.

When creating a new shard, ES needs to specify a shard allocation policy according to the existing shard distribution, so that the number of shards on each node is basically the same. In this process, you need to traverse the clusterstate deeply. When there are too many index/shard in the cluster, the clusterstate structure will become too complex, resulting in inefficient traversal of clusterstate and slow response of the cluster. The Infrastructure database team once created 4w + shard in a 20-node cluster, resulting in a new index taking 60s + to complete. When the number of index/shard is too large, you can consider improvements from the following aspects:

Reduce the number of shard for index with small amount of data

Merge some related index into a single index

The data is split according to a dimension and written to multiple clusters

3. Segment Memory optimization

As mentioned earlier, the underlying ES uses Lucene for storage, while an index of Lucene consists of several segment, and each segment sets up its own inverted index for data query. In order to speed up the query, Lucene makes a prefix index for the inversion of each segment. The data structure of this index after Lucene4.0 is FST (Finite State Transducer). When Lucene loads segment, it loads all of it into memory to speed up the query. This part of the memory is called SegmentMemory, resident memory, occupies heap, can not be GC.

As mentioned earlier, in order to take advantage of JVM's object pointer compression technology to save memory, it is generally recommended that the JVM memory allocation should not exceed 32G. When the amount of data in the cluster is too large, SegmentMemory will eat a large amount of heap memory, and the memory space of JVM is limited. You need to find ways to reduce the usage of SegmentMemory. The common methods are as follows:

Delete unused index on a regular basis

For index that is not accessed frequently, you can turn it off through the close interface and then turn it on when you need it.

Forced merging of segment through force_merge interface to reduce the number of segment

On this basis, the database team of the Infrastructure Department optimized the FST section to free up to 40% of the Segment Memory memory space.

The above is the whole content of how to carry out Elasticsearch tuning practice, more content related to how to do Elasticsearch tuning practice can search the previous articles or browse the following articles to learn ha! I believe the editor will add more knowledge to you. I hope you can support it!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.