Shulou(Shulou.com)06/02 Report--

In this issue, Xiaobian will bring you about how to optimize Elasticsearch performance. The article is rich in content and analyzed and described from a professional perspective. After reading this article, I hope you can gain something.

Elasticsearch is currently one of the most popular technology stacks in the field of big data. After nearly 8 years of development, it has been upgraded from version 0.0.X to version 6.X. Although many features and functions have been added, there are still not many changes in the main architecture. Here is a summary of some of my experiences in ES use practice for your reference; please also clap bricks.

1. Hardware environment selection:

If there are conditions, try to use SSD hard disk, good CPU. ES's strength lies in the distributed architecture of ES itself and the characteristics of lucene. The increase in IO will greatly improve the speed and performance of ES.

II. System topology design:

ES clusters generally adopt Hot-Warm architecture mode when they are in architecture topology, that is, three different types of nodes are set: Master node, Hot node and Warm node.

Master node setting: 3 dedicated maste nodes are generally set to provide the best elastic scalability. Of course, you must pay attention to the setting of the discovery.zen.minimum_master_nodes attribute, in case of split-brain problems, using the formula: N/2+1(N is the number of candidate master nodes). This node maintains: node.data: false ; because the master node does not participate in query and index operations, it is only responsible for cluster management, so in terms of CPU, memory, and disk configuration, it can be much lower than the data node.

Hot node settings: index nodes (write nodes), while maintaining recently used indexes. For IO and CPU intensive operations, SSD disk type is recommended to maintain good write performance; the number of nodes is generally greater than or equal to 3. Set the node to hot type:

node.attr.box_type: hot

For index, you can set the index to be written to the hot node by setting index.routing.allocation.require.box_type: hot.

Warm node settings: For read-only indexes that are accessed infrequently. Since it is not frequently accessed, ordinary disks are generally used. Memory, CPU configuration and Hot node can be consistent; the number of nodes is generally greater than or equal to 3.

When the index is no longer frequently queried, you can mark the index as warm by index.routing.allocation.require.box_type: warm, so as to ensure that the index is not written to the hot node, so that SSD disk resources can be used at the cutting edge. Once this property is set, ES automatically merges the index into the warm node. At the same time, you can also set index.codec: best_compression in elasticsearch.yml to ensure the compression configuration of warm nodes.

Coordinating node: The coordinating node is used to coordinate in the distributed system, integrating the data returned by each fragment or node and returning it. In an ES cluster, all nodes may be coordinator nodes, but you can set a dedicated coordinator node by setting node.master, node.data, and node.ingest to false. A better CPU and higher memory are required.

III. ES memory settings:

Because ES is built on lucene, the strength of lucene design is that lucene makes good use of operating system memory to cache index data to provide fast query performance. Lucene index file segments are stored in a single file, and immutable, for OS, can be very friendly to keep the index file in cache, for fast access; therefore, it is necessary to leave half of the physical memory for lucene ; the other half of the physical memory for ES (JVM heap ). So, in terms of ES memory settings, you can follow the following principles:

1. When the machine memory is less than 64 gigabytes, follow the general principle of 50% for ES and 50% for lucene.

2. When the machine memory is greater than 64G, follow the following principles:

a. If the main usage scenario is full-text search, it is recommended to allocate 4~32G of memory to ES Heap; the rest of the memory is reserved for the operating system for lucene (segments cache) to provide faster query performance.

b. If the main usage scenario is aggregation or sorting, and most of the character types are numerics, dates, geo_points and not_analyzed, it is recommended to allocate 4~32G of memory to ES Heap, and the rest of the memory is reserved for the operating system for lucene (doc values cache) to provide fast document-based clustering and sorting performance.

c. If the usage scenario is aggregation or sorting, and both are based on analyzed character data, then more heap size is required. It is recommended to run multiple ES instances on the machine, and each instance should keep no more than 50% ES heap setting (but no more than 32G. When the heap memory setting is less than 32 G, JVM uses object metric compression techniques to save space), and more than 50% is reserved for lucene.

3. Disallow swap, which causes fatal performance problems once memory and disk swaps are allowed. By: bootstrap.memory_lock: true in elasticsearch.yml, keep JVM locked memory and guarantee ES performance.

4. GC setting principles:

a. Keep GC's current settings, default to: Concurrent-Mark and Sweep (CMS), don't change to G1GC, because there are still many bugs in G1.

b. Keep the existing settings of thread pool. At present, ES thread pool has more optimized settings than 1.X. Just keep the status quo; the default thread pool size is equal to the number of CPU cores. If you must change it, set it according to the formula ((CPU core number * 3)/ 2)+ 1; it cannot exceed 2 times the CPU core number; however, it is not recommended to modify the default configuration, otherwise it will cause hard damage to the CPU.

IV. Cluster fragmentation settings:

Once an ES index is created, it is impossible to adjust the shard settings. In ES, a shard actually corresponds to a lucene index, and the reading and writing of lucene indexes will take up a lot of system resources. Therefore, the number of shards cannot be set too large. Therefore, it is very important to configure the number of shards reasonably when creating an index. In general, we follow some principles:

1. Control the disk capacity occupied by each shard not to exceed the maximum JVM heap space setting of ES (generally set not to exceed 32G, see the JVM setting principle above), so if the total capacity of the index is about 500G, the shard size is about 16; of course, it is best to consider principle 2 at the same time.

2. Consider the number of nodes. Generally, a node is sometimes a physical machine. If the number of shards greatly exceeds the number of nodes, it is likely that there will be multiple shards on a node. Once the node fails, even if more than one replica is maintained, it may also lead to data loss and the cluster cannot be recovered. Therefore, it is generally set that the number of fragments does not exceed 3 times the number of nodes.

V. Mapping modeling:

1. Try to avoid using nested or parent/child, do not use it if you can;nested query is slow, parent/child query is slower, hundreds of times slower than nested query; therefore, if you can get it in the mapping design stage (large wide table design or smart data structure), do not use parent-child mapping.

2. If you must use nested fields, make sure that there are not too many nested fields. Currently, the default limit of ES is 50. Reference:

index.mapping.nested_fields.limit ：50

Because for a document, each nested field will generate a separate document, which will make the number of Doc increase dramatically, affecting query efficiency, especially JOIN efficiency.

3. Avoid using dynamic values as fields (keys). Dynamically increasing mapping will lead to cluster collapse. Similarly, you need to control the number of fields. Fields that are not used in the business do not need to be indexed. Controlling the number of index fields, mapping depth, and index field types is the most important thing for ES performance optimization. Here are some default ES settings for field count and mapping depth:

index.mapping.nested_objects.limit :10000

index.mapping.total_fields.limit:1000

index.mapping.depth.limit: 20

VI. Index optimization settings:

1. Set refresh_interval to-1 and number_of_replicas to 0 to improve write performance by turning off the refresh interval period without setting replicas.

2. Modify the index_buffer_size setting. It can be set as a percentage or a specific size. The size can be tested according to the size of the cluster.

indices.memory.index_buffer_size: 10%(default)

indices.memory.min_index_buffer_size: 48mb (default)

indices.memory.max_index_buffer_size

3. Modify translog-related settings:

a. Control how often data is transferred from memory to hard disk to reduce disk IO. The sync_interval can be set to a longer time.

index.translog.sync_interval: 5s(default).

b. Control the size of tranlog data block, only flush to lucene index file when threshold size is reached.

index.translog.flush_threshold_size: 512mb(default)

4. The use of the_id field should avoid custom_id as much as possible to avoid version management for IDs; it is recommended to use the default ID generation policy of ES or use the numeric type ID as the primary key.

5. The use of_all field and_source field should pay attention to the scene and needs. The_all field contains all index fields, which is convenient for full-text retrieval. If there is no such requirement, it can be disabled._source stores the original document content. If there is no requirement to obtain the original document data, you can define the fields placed in_source by setting the attributes of include and exclude.

6. Reasonable configuration uses index attributes, analyzed and not_analyzed to control whether fields are segmented or not segmented according to business requirements. Only fields required by groupby are configured to not_analyzed to improve query or clustering efficiency.

VII. Query optimization:

1. The more query fields for query_string or multi_match, the slower the query. You can use the copy_to attribute during the mapping phase to index multi-field values to a new field, multi_match, and query with the new field.

2. Date field queries, especially queries using now, actually do not have cache, so you can consider whether to use now from a business perspective. After all, query cache can greatly improve query efficiency.

3. The size of the query result set cannot be arbitrarily set to an excessively large value. For example, query.setSize cannot be set to Integer.MAX_VALUE, because ES needs to establish a data structure internally to store the result set data of the specified size.

4. Avoid script if you have to, and opt for a painless & experts engine. Once you use script queries, be sure to control returns and never have an endless loop (see the following error example), because ES has no timeout control for script execution, and the query will block as long as the current script is not executed.

For example: {

"script_fields"：{

"test1"：{

"lang"："groovy"，

"script"："while（true）{print 'don't use script'}"

}

}

}

5. Avoid aggregate queries with too deep a hierarchy. Group by with too deep a hierarchy will lead to memory and CPU consumption. It is recommended to assemble services through programs at the service layer, or optimize them through pipelines.

6. Reuse pre-indexed data to improve AGG performance:

For example, range aggregates are replaced by terms aggregates. If you want to group by age, the grouping target is: teenagers (under 14 years old) youth (14-28 years old) middle-aged (29-50 years old) elderly (51 years old). You can set an age_group field when indexing to classify the data in advance. So instead of doing range aggregates by age, you can just do it with the age_group field.

7. Cache settings and usage:

a) QueryCache: query cache will be used when querying ES using filter query. If there are many filtered queries in the business scenario, it is recommended to set querycache larger to improve query speed.

indexes. querys.cache.size: 10%(default), can be set as a percentage, or can be set to a specific value, such as 256mb.

Of course, you can disable query caching (which is enabled by default) by setting index.queries.cache.enabled: false.

b) FieldDataCache: field data cache is frequently used during clustering or sorting. Therefore, it is necessary to set the size of field data cache. It can be set by indexes.fielddata.cache.size: 30% or 10GB. However, if the scene or data changes frequently, it is not a good idea to set up cache, because the cache load overhead is also very large.

c) ShardRequestCache: After the query request is initiated, each shard returns the result to the Coordinating Node, which consolidates the result.

If required, you can set it on; turn it on by setting index.requests.cache.enable: true.

However, shard request cache only caches hits.total, aggregates, suggestions and does not cache the contents of hits. You can also control the cache size by setting indexes.requests.cache.size: 1%(default).

The above is how to optimize Elasticsearch performance shared by Xiaobian for everyone. If you happen to have similar doubts, you may wish to refer to the above analysis for understanding. If you want to know more about it, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.