Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about the five common mistakes and solving strategies of Elasticsearch. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

1. Adopt dynamic Mapping

If you don't define Mapping,Elasticsearch, the corresponding Mapping will be created based on the input data, which looks perfect, but the dynamic Mapping of Elasticsearch is not always accurate.

Dynamic Mapping is useful for getting started, but at some point you need to specify Mapping in conjunction with business data.

For example, after version 5.x of keyword, the fields that need word segmentation need to set the text type and the corresponding analyzer; only the fields that need to match exactly can be set to the keyword type.

Example 2: long text highlighting needs to be based on the text type, set fast-vector-highlighter highlighting mode, highlighting efficiency can be increased by more than 20 times.

2. Improper aggregation settings lead to OOM

In some aggregations, there is not enough memory to support complex nested aggregations, causing the aggregate result to time out or even OOM.

Aggregation explosion is a computational problem that can lead to exponential growth in bucket generation of some aggregates and may lead to uncontrolled memory usage.

The Elasticsearch "terms" field builds buckets based on your data, but it is impossible to predict how many buckets will be created in advance. This can be problematic for parent aggregates that consist of multiple child aggregates. Combining unique values in each subaggregate may result in a significant increase in the number of buckets created.

Let's look at an example.

Suppose you have a dataset that represents a sports team. If you want to pay special attention to the top 10 players of that team and their supporting players, the aggregation will be as follows

1 {

2 "aggs": {

3 "play_aggs": {

4 "terms": {

5 "field": "players"

6 "size": 10

7}

8 "aggs": {

9 "other_aggs": {

10 "terms": {

11 "field": "players"

12 "size": 5

13}

14}

15}

16}

17}

18}

The aggregation returns a list of the top 10 players and a list of the top five supporting players for each top player-which returns a total of 50 values. This seemingly simple query can easily consume a lot of memory.

Terms aggregations can be displayed as trees that use buckets at each level. Therefore, the bucket of each top player in the above aggregation will constitute the first level, while the bucket of each supporting player in the other aggregation will constitute the second level. Therefore, a team will produce n ²barrels. Imagine what would happen if you had a dataset of 500 million documents.

Collection Mode is used to help control how subaggregations are performed. The default Collection Mode for aggregation is called depth first, and you first need to build the entire tree and then trim the edges. Although depth first is the appropriate collection mode for most aggregations, it does not apply to the athlete aggregation example above. Therefore, Elasticsearch allows you to change the collection mode in a particular aggregation to a more appropriate way.

Specifications such as the above example should use the breadth-first collection pattern, which builds and prunes trees one level at a time to control aggregation explosions. This collection mode greatly helps to reduce the amount of memory consumed and keep the node stable.

1 {

2 "aggs": {

3 "play_aggs": {

4 "terms": {

5 "field": "players"

6 "size": 10

7 "collect_mode": "breadth_first"

8}

9 "aggs": {

10 "other_aggs": {

11 "terms": {

12 "field": "players"

13 "size": 5

14}

15}

16}

17}

18}

19}

Recommended reading: http://t.cn/RHndSgY

3. Improper ES index configuration 3.1 Cluster name configuration

The default cluster name for ES startup is elasticsearch. If there are many nodes in the cluster, it is best to keep the naming flags as consistent as possible, for example:

1cluster.name:app_es_production

2node.name:app_es_node_001

3.2 Cluster recovery Settings

The recovery settings of the node are also important. Suppose some nodes in the cluster restart because of a failure, and some nodes restart after others. In order to keep the data consistent between all these nodes, we must run the consistency program to keep all the clusters in a consistent state.

Example 1: only 10 data or primary nodes have joined the cluster to recover.

1gateway.recover_after_nodes:10

Example 2: the cluster is expected to restart or resume after 20 startup nodes and more than 7 minutes.

1gateway.expected_nodes:20

2gateway.recover_after_time:7m

With the correct configuration, it may take several hours of recovery to be reduced to minutes, greatly improving productivity.

3.3 Anti-cleft configuration

Minimum_master_nodes is very important for cluster stability. They help prevent brain fissure.

The recommended value for this setting is (N / 2) + 1, where N is the number of nodes of the candidate primary node.

With this, if you have 10 candidate master nodes that can save data and become master data, then the value will be 6.

If you have three dedicated primary nodes and 1000 data nodes, the value is two (only candidate primary nodes are calculated):

Discovery.zen.minimum_master_nodes:2

4. If there is no planning for the cluster, we will talk about it when we encounter problems.

1 "how much storage space and how much memory do I need?" It's a question that users often ask themselves.

Unfortunately, there is no fixed formula, but some steps can be taken to assist in planning resources.

Recommended method: simulate the actual use case.

Step 1: create an ES cluster.

Step 2: use data at almost the same data rate as the production settings.

Step 3: start the nodes, populate them with real documents, and then push the fill data to the index shard.

It is important to understand resource utilization during simulation of actual use cases because it allows you to reserve the appropriate amount of RAM for nodes, configure JVM heap space, and optimize the entire testing process.

According to the simulation results, the memory, CPU and disk capacity of the actual cluster are determined.

5. Unreasonable thread pool setting

The ES node has many thread pools to improve the way threads are managed within the node. But there is a limit to how much data each thread can process. To track this value, we can use the ES property:

1threadpool.bulk.queue_size:2000

This notifies ES of the number of requests in the shard, and when there are no threads available to process the request, new requests can be queued in the node for execution. If the number of tasks is higher than this value, you will get RemoteTransportException. The higher the value, the greater the amount of heap space required on the node machine, and the JVM heap will be consumed.

In addition, you should do exception handling during the development phase of your code.

Note: it is not recommended to modify this value on the ES official website.

In the process of using Elasticsearch, we will always encounter problems of one kind or another. We should sum up and think more to form an effective solution for business scenarios.

At the same time, we should also absorb the quintessence of communities, forums and blogs at home and abroad to learn from each other's strengths to offset our weaknesses.

Note: the network literature generally does not cover the version, some configurations of the old version of ES may not be suitable for the latest version of 6.x, but the underlying technology is never out of date.

These are the five common Elasticsearch mistakes and solutions shared by the editor. If you happen to have similar doubts, please refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.