Shulou(Shulou.com)05/31 Report--

This article introduces the knowledge of "how to manage the life cycle of Elasticsearch index". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Introduce the role of index life cycle

In the daily management of Elasticsearch, there are many application scenarios such as system logs and behavioral data, which are characterized by a large amount of data, and the number of indexes will continue to grow over time. However, in these scenarios, only the recent data is useful or often used (hot data), while historical data is almost useless or rarely used (cold data). At this time, it is necessary to maintain and manage the index or even delete and clean up, otherwise, with the increasing amount of data, in addition to wasting disk and memory space, it will seriously affect the performance of Elasticsearch.

After Elastic Stack version 6.6, a new feature Index Lifecycle Management (Index Lifecycle Management) is introduced, which supports full lifecycle managed management of indexes, and a set of UI interface is also provided on Kibana to configure policies.

Common stages of the index life cycle

Hot: there are also a lot of read and write operations in the index.

Warm: there is no write operation in the index, and there is a need to be queried.

Cold: there are no writes or reads to the data.

Delete: the index is no longer needed and can be safely deleted.

Note: the above are only common definitions of the index lifecycle phase, and the specific strategy can be defined according to the actual business situation.

Deploy Elasticsearch clusters

Deploy an Elasticsearch cluster composed of 2 hot nodes, 2 warm nodes and 2 cold nodes, and deploy Kibana and Cerebro to facilitate debugging and observation. The docker-compose.yaml file is as follows:

Version: '2.2'services: cerebro: image: lmenezes/cerebro:0.8.3 container_name: hwc_cerebro ports:-"9000 cerebro 9000" command:-- Dhosts.0.host= http://elasticsearch:9200 networks:-hwc_es7net kibana: image: docker.elastic.co/kibana/kibana:7.1.0 container_name: hwc_kibana7 environment: #-I18N_LOCALE=zh-CN -XPACK_GRAPH_ENABLED=true-TIMELION_ENABLED=true-XPACK_MONITORING_COLLECTION_ENABLED= "true" ports:-"5601 true 5601" networks:-hwc_es7net elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0 container_name: es7_hot1 environment:-cluster.name=cr7-hwc-node.name=es7_hot1-node.attr.box_type=hot- Bootstrap.memory_lock=true-"ES_JAVA_OPTS=-Xms512m-Xmx512m"-discovery.seed_hosts=es7_hot1 Es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2-cluster.initial_master_nodes=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2 Es7_cold2 ulimits: memlock: soft:-1 hard:-1 volumes:-hwc_es7data_hot1:/usr/share/elasticsearch/data ports:-9200 networks:-hwc_es7net elasticsearch3: image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0 container_name: es7_warm1 environment:-cluster.name=cr7-hwc-node.name=es7_warm1 -node.attr.box_type=warm-bootstrap.memory_lock=true-"ES_JAVA_OPTS=-Xms512m-Xmx512m"-discovery.seed_hosts=es7_hot1 Es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2-cluster.initial_master_nodes=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2 Es7_cold2 ulimits: memlock: soft:-1 hard:-1 volumes:-hwc_es7data_warm1:/usr/share/elasticsearch/data networks:-hwc_es7net elasticsearch4: image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0 container_name: es7_cold1 environment:-cluster.name=cr7-hwc-node.name=es7_cold1-node.attr.box_type=cold -bootstrap.memory_lock=true-"ES_JAVA_OPTS=-Xms512m-Xmx512m"-discovery.seed_hosts=es7_hot1 Es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2-cluster.initial_master_nodes=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2 Es7_cold2 ulimits: memlock: soft:-1 hard:-1 volumes:-hwc_es7data_cold1:/usr/share/elasticsearch/data networks:-hwc_es7net elasticsearch5: image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0 container_name: es7_hot2 environment:-cluster.name=cr7-hwc-node.name=es7_hot2-node.attr.box_type=hot -bootstrap.memory_lock=true-"ES_JAVA_OPTS=-Xms512m-Xmx512m"-discovery.seed_hosts=es7_hot1 Es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2-cluster.initial_master_nodes=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2 Es7_cold2 ulimits: memlock: soft:-1 hard:-1 volumes:-hwc_es7data_hot2:/usr/share/elasticsearch/data networks:-hwc_es7net elasticsearch6: image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0 container_name: es7_warm2 environment:-cluster.name=cr7-hwc-node.name=es7_warm2-node.attr.box_type=warm -bootstrap.memory_lock=true-"ES_JAVA_OPTS=-Xms512m-Xmx512m"-discovery.seed_hosts=es7_hot1 Es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2-cluster.initial_master_nodes=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2 Es7_cold2 ulimits: memlock: soft:-1 hard:-1 volumes:-hwc_es7data_warm2:/usr/share/elasticsearch/data networks:-hwc_es7net elasticsearch7: image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0 container_name: es7_cold2 environment:-cluster.name=cr7-hwc-node.name=es7_cold2-node.attr.box_type=cold -bootstrap.memory_lock=true-"ES_JAVA_OPTS=-Xms512m-Xmx512m"-discovery.seed_hosts=es7_hot1 Es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2-cluster.initial_master_nodes=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2 Es7_cold2 ulimits: memlock: soft:-1 hard:-1 volumes:-hwc_es7data_cold2:/usr/share/elasticsearch/data networks:-hwc_es7netvolumes: hwc_es7data_hot1: driver: local hwc_es7data_warm1: driver: local hwc_es7data_cold1: driver: local hwc_es7data_hot2: driver: local hwc_es7data_warm2: driver: local hwc_es7data_ Cold2: driver: localnetworks: hwc_es7net: driver: bridge

Execute the docker-compose up-d command to start the Elasticsearch cluster. (provided that docker and docker-compose are installed)

Modify IML refresh time

Set the ILM refresh time to 1 second for experimental demonstration (default is 10 minutes):

PUT _ cluster/settings {"persistent": {"indices.lifecycle.poll_interval": "1s"}} set Index Lifecycle Policies

Set up the four phases of hot/warm/cold and delete:

Hot: rollover after more than 5 documents.

After warm:20s, enter the warm phase and set the index to read-only.

After cold:40s, enter the warm phase, reducing the copies from 1 to 0, respectively.

After delete:60s, enter the delete phase and delete the index.

PUT / _ ilm/policy/log_ilm_policy {"policy": {"phases": {"hot": {"actions": {"rollover": {"max_docs": 5}, "warm": {"min_age": "20s" "actions": {"allocate": {"include": {"box_type": "warm"}}, "readonly": {}, "cold": {"min_age": "40s" "actions": {"allocate": {"include": {"box_type": "cold"}, "number_of_replicas": 0}, "delete": {"min_age": "60s" "actions": {"delete": {} set the index template

The index template is set so that the indexes from the new rollover can apply the configuration of iml:

PUT / _ template/log_ilm_template {"index_patterns": ["ilm_index-*"], "settings": {"index": {"lifecycle": {"name": "log_ilm_policy", "rollover_alias": "ilm_alias"} "routing": {"allocation": {"include": {"box_type": "hot"}, "number_of_shards": "1", "number_of_replicas": "1"}} create the first index

Assign the index to the hot node, and call the previously set IML policy log_ilm_policy, set the alias of rollover to ilm_alias, set the primary shard to 1, and the replica shard to 1. The index must be followed by a number, or the rollover policy will not take effect.

PUT ilm_index-000001 {"settings": {"number_of_shards": 1, "number_of_replicas": 1, "index.lifecycle.name": "log_ilm_policy", "index.lifecycle.rollover_alias": "ilm_alias", "index.routing.allocation.include.box_type": "hot"}, "aliases": {"ilm_alias": {# in rollover Alias will contain all rollover indexes and automatically set is_write_index ": # true to the latest rollover index, that is, the write operation will go to the latest index. "is_write_index": true} create document

Execute POST 5 times in a row and create 5 documents.

POST ilm_alias/_doc {"name": "cr7", "age": 15} observation

At first, you can see that there are a total of five doc.

When the index iml_index-000001 reaches 5 doc, the rollover operation is done and a new index iml_index-000002 is created.

After waiting 20 seconds, move the index iml_index-000001 to the warm node.

After waiting for 40 seconds, move the index iml_index-000001 to the cold node and reduce the number of copies from 1 to 0.

After waiting for 60 seconds, the index iml_index-000001 is deleted.

Because readonly is set up in the warm phase, if you try to write or modify data to iml_index-000001 during the warm and cold phases, you will receive the following error:

{"error": {"root_cause": [{"type": "cluster_block_exception", "reason": "blocked by: [FORBIDDEN/8/index write (api)];"}], "type": "cluster_block_exception", "reason": "blocked by: [FORBIDDEN/8/index write (api)] "}," status ": 403}" how to manage the life cycle of Elasticsearch indexes "is introduced here. Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.