Shulou(Shulou.com)06/02 Report--

This article introduces how to achieve automatic hot and cold separation of data in ElasticSearch, the content is very detailed, interested friends can refer to, hope to be helpful to you.

1. Node tag

Start three nodes in turn, and add box_type and resource_level tags at the same time. Box_type marks node1 and node2 as warm nodes, node3 as hot nodes, and resource_level marks the performance of machine resources as high, medium and low.

Bin/elasticsearch-d-p pid-E node.name=node1-E node.max_local_storage_nodes=3-E path.data=node1_data-E path.logs=node1_logs-E node.attr.box_type=warm-E node.attr.resource_level=highbin/elasticsearch-d-p pid-E node.name=node2-E node.max_local_storage_nodes=3-E path.data=node2_data-E path.logs=node2_logs-E node.attr.box_type=warm-E node.attr.resource_level=mdeiumbin/elasticsearch-d-p pid-E Node.name=node3-E node.max_local_storage_nodes=3-E path.data=node3_data-E path.logs=node3_logs-E node.attr.box_type=hot-E node.attr.resource_level=high View properties: enter the following command GET _ cat/indices?v in kibana to get the following results You can see the value of box_type and resource_level tags on each node node host ip attr valuenode3 127.0.1127.0.0.1 ml.machine_memory 17179869184node3 127.0.1127.0.0.1 ml.max_open_jobs 20node3 127.0.0.1 127.0.1127.0.0.1 box_type hotnode3 127.0.0.1127.0.0.1 xpack.installed truenode3 127.12 .0.0.1 127.0.0.1 resource_level highnode1 127.0.0.1 127.0.0.1 ml.machine_memory 17179869184node1 127.0.0.1 127.0.0.1 box_type warmnode1 127.0.0.1 127.0.0.1 xpack.installed truenode1 127.0.0.1 127.0.0.1 ml.max_open_jobs 20node1 127.0.0.1 127.0.0.1 resource_level highnode2 127.0.0.1 127.0.0.1 ml.machine_memory 17179869184node2 127.0.0.1 127.0.0.1 ml.max_open_jobs 20node2 127.0.0.1 127.0.0.1 box_type warmnode2 127.0.0.1 127.0.0.1 xpack.installed truenode2 127.0.0.1 127.0.0.1 resource_level mdeium2. Build an index

Assuming that the current time is September 1, 2019 and is stored on the hot node as the latest data, you only need to specify the allocation policy when indexing.

PUT api_log_2019-09-01 {"settings": {"number_of_shards": 3, "number_of_replicas": 0, "index.routing.allocation.require.box_type": "hot" >

It doesn't matter if this option is not configured when indexing, it can be modified dynamically.

PUT api_log_2019-09-01/_settings {"index.routing.allocation.require.box_type": "hot"} 3. Migrate index

The way to migrate historical indexes to warm nodes is also to dynamically modify requests.

PUT api_log_2019-09-01/_settings {"index.routing.allocation.require.box_type": "warm", "index.routing.allocation.include.resource_level": "mdeium"} We migrated api_log_2019-09-01 to a node whose box_type is warm,resource_level and mdeium That is, node2 queries the distribution of the index shards GET _ cat/shards/api_log_2019-09-01. The results are as follows: index shard prirep state docs store ip nodeapi_log_2019-09-01 1 p STARTED 4711 4.1mb 127.0.0.1 node2api_log_2019-09-01 2 p STARTED 4656 4mb 127.0.0.1 node2api_log_2019-09-01 0 p STARTED 4707 4.1mb 127.0.0.1 node2Rollover API

You should have noticed that the step of migrating the index is done manually. Is there a more intelligent way? the answer is yes. Rollover API can achieve this function very well.

1. Rollover

First, create an alias for the index. Since multiple index can correspond to an alias, in order to let es know which index to write to, mark one of the index is_write_index as true, and note that the index name is named in the form of horizontal bar + number ending. This is to let es generate the index automatically.

POST _ aliases {"actions": [{"remove": {"index": "api_log_2019-09-01", "alias": "api_logs", "is_write_index": true}}]} rollover API generates a new index POST api_logs/_rollover {"conditions": {"max_age": "1D" based on the set condition (conditions). "max_docs": 10000, "max_size": "5gb"} conditions detailed explanation: max_age: whether the index is created for more than 1 day max_docs: whether the number of index documents exceeds 10000max_size: whether the index size exceeds 5GBmax_size 's ongoing consolidation will result in a large number of temporary shard size growth When the merger ends, these growth will disappear, unstable, max_age every time may not be balanced, max_docs is more appropriate, that is, one of the above three conditions will automatically rollover2. The new index configuration rollover also supports index settings settings POST api_logs/_rollover {"conditions": {"max_age": "1D", "max_docs": 10000, "max_size": "5gb"}, "settings": {"index.number_of_shards": 2}} 3. Custom index name

The generated index name is api_log_2019-09-000002, which is named in the format of length 6, sequence number + 1, left filled with 0, but es supports custom names. You only need to add the name to the _ rollover endpoint.

POST api_logs/_rollover/api_log_2019-09-02 {.} on how to achieve automatic hot and cold separation of data in ElasticSearch is shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.