Shulou(Shulou.com)06/03 Report--

This article is about ES's data migration through snapshots. The editor thought it was very practical, so I shared it with you to learn. The following information is about the steps of data migration.

1. Install the oss plug-in to the es node of the computer room.

Aliyun provides the oss plug-in for es. The address https://github.com/aliyun/elasticsearch-repository-oss should note that installing this plug-in requires a rolling restart of the es cluster.

I have skipped the pre-operation of installing the oss plug-in here, there is nothing to say. Just refer to the scrolling restart procedure proposed in the official documentation.

2. Activate the bucket of Aliyun oss, and record the authorization information related to accesskey

3. Our snapshot data is directly sent to the private bucket of Aliyun oss through the private network (TIPS: after pulling the direct connect and going through the private network, you need to add a route to use the private network domain name address of oss)

The following is my full process of performing snapshots and restores:

Step1_idc_reg_oss.sh registers idc's es snapshot warehouse address to Aliyun oss

#! / bin/bash# pass parameter 1 to the lb address of the es cluster (or use the real backend node address) echo "function:" echo "register the snapshot address to Aliyun to oss warehouse" echo# to find the name of the cluster CLUSTER_NAME=$ (curl-s http://$1:9200/ | grep cluster_name | awk-F':'{print $2}'| awk-F'"'{print $2}') echo" ES cluster Name: "$CLUSTER_NAMEecho# Registration Snapshot Repository Take the direct connect oss private network address curl-s-XPUT http://$1:9200/_snapshot/$CLUSTER_NAME-d: {\ "type\":\ "oss\",\ "settings\": {\ "base_path\":\ "$CLUSTER_NAME\",\ "endpoint\":\ "http://oss-cn-hangzhou-internal.aliyuncs.com\",\"access_key_id\":\" thisismyaccesskeyid\ ",\" secret_access_key\ ":\" thisismykey\ " \ "bucket\":\ "snapshot-es\",\ "compress\": true} "# confirm whether the warehouse is okcurl-s-XGET http://$1:9200/_snapshot/$CLUSTER_NAME | jq.

Step2_first_snapshot.sh takes a full snapshot of idc's es to Aliyun oss.

#! / bin/bash# pass parameter 1 to the lb address of the es cluster (or use the real backend node address) echo "function:" echo "backend take a snapshot to store in Aliyun oss warehouse" echo# to find the name of the cluster CLUSTER_NAME=$ (curl-s http://$1:9200/ | grep cluster_name | awk-F':'{print $2}'| awk-F'"'{print $2}') echo" ES cluster name Say: "$CLUSTER_NAMEecho# takes a snapshot in the background echo" starts taking a snapshot in the background "curl-s-XPUT http://$1:9200/_snapshot/$CLUSTER_NAME/$(date +% Y_%m_%d_%H_%M) | jq .echoecho" check the progress of the snapshot: curl-XGET http://$1:9200/_snapshot/$CLUSTER_NAME/_all?pretty"echo

Note: after the step2 script runs successfully for the first time, we can add a cronjob to run every hour 1-2 days before cloud launch, so that the time required for incremental snapshots is very short during the final migration to the cloud. Significantly reduce unavailable time.

Step3_aliyun_reg_oss.sh registers Aliyun's es snapshot warehouse address to Aliyun oss

#! / bin/bash# Ref 1 is the LB address of the es cluster in the computer room # Ref 2 is the domain name echo of the Aliyun es cluster "function:" echo "registers Aliyun es to Aliyun's oss warehouse" echo# finds the name of the cluster CLUSTER_NAME=$ (curl-s http://$1:9200/ | grep cluster_name | awk-F':'{print $2}'| awk-F'"'{print $2}') Echo "ES Cluster name:" $CLUSTER_NAMEecho# Ali Cloud Associated oss Warehouse curl-s-XPUT http://$2:9200/_snapshot/$CLUSTER_NAME-d "{\" type\ ":\" oss\ " \ "settings\": {\ "base_path\":\ "$CLUSTER_NAME\",\ "endpoint\":\ "http://oss-cn-hangzhou-internal.aliyuncs.com\",\"access_key_id\":\" thisismyaccesskeyid\ ",\" secret_access_key\ ":\" thisismykey\ ",\" bucket\ ":\" snapshot-es\ " \ "compress\": true} "# confirm whether the snapshot repository is okcurl-s-XGET http://$2:9200/_snapshot/$CLUSTER_NAME?pretty | jq .echo" curl-s-XGET http://$2:9200/_snapshot/$CLUSTER_NAME?pretty | jq. "

Step4 starts and needs to be operated on the night of cutoff.

Step4_stop_write_idc_index.sh turns off the write permission of idc's es index and retains read-only permission

#! / bin/bash# pass parameter 1 to the lb address of the es cluster (or use the real backend node address) echo "function:" echo "closes the write permission of the es index in the computer room In order to perform the last snapshot operation later, "echo# finds out the name of the cluster CLUSTER_NAME=$ (curl-s http://$1:9200/ | grep cluster_name | awk-F':'{print $2}'| awk-F'"'{print $2}') echo $CLUSTER_NAME# closes write permissions for all indexes and retains read permissions echo "starts to close write permissions for all indexes. Read permission reserved... "curl-s-XPUT http://$1:9200/_all/_settings-d'{" index ": {" blocks ": {" write ":" true "}}'# official document https://www.elastic.co/guide/en/elasticsearch/reference/2.3/index-modules.html# index.blocks.read_only# Set to true to make the index and index metadata read only, false to allow writes and metadata changes. (set to true to prohibit the modification of the index and the index's metadata, more stringent control) # index.blocks.write#Set to true to disable write operations against the index. (set to true to disable writing to the index. But the metadata of the index can be written normally.) # Note: index.blocks.read_only and index.blocks.write cannot be used at the same time, only one of them can be selected.

Step5_increase_snapshot.sh manually executes the snapshot script again (this time it is an incremental snapshot, which will not take long)

#! / bin/bash# pass parameter 1 to the lb address of the es cluster (or use the real backend node address) echo "function:" echo "after turning off the es write permission in the computer room Take another snapshot at the backend and save it to Aliyun's oss warehouse. "echo# to find the name of the cluster CLUSTER_NAME=$ (curl-s http://$1:9200/ | grep cluster_name | awk-F':'{print $2}'| awk-F'"'{print $2}') echo "ES cluster name:" $CLUSTER_NAMEecho# takes a snapshot in the background echo "starts taking a snapshot in the background" echo "to check the progress of the snapshot: curl -XGET http://$1:9200/_snapshot/$CLUSTER_NAME/_all?pretty"curl-s-XPUT http://$1:9200/_snapshot/$CLUSTER_NAME/$(date +% Y_%m_%d_%H_%M) | jq.

Note: make sure that the snapshot progress of step5 is complete before performing the following step6 operation!

Step6_restore_into_aliyun.sh restores the latest snapshot above oss to Ali Cloud ES environment

#! / bin/bash# recovers data from Aliyun oss to Aliyun es# parameter 1 is the lb address of es of idc (without port number) # Parameter 2 is Aliyun es domain name (without port number) echo "function:" echo "recover data from oss warehouse to Aliyun es" echo# to find the name of the cluster CLUSTER_NAME=$ (curl-s http://$1:9200/ | grep cluster_name | Awk-F':'{print $2}'| awk-F'"'{print $2}') echo" ES cluster name: "$CLUSTER_NAMEecho# found the latest snapshot name: SNAPSHOT_NAME=$ (curl-s-XGET http://$2:9200/_snapshot/$CLUSTER_NAME/_all?pretty | egrep snapshot | tail-1 | awk-F':'{print $2}'| awk-F'"'{print $2}') echo "is the most The name of the latter snapshot: "${SNAPSHOT_NAME} # adjust Ali Cloud ES parameters to speed up recovery echo" adjust Ali Cloud ES parameters to speed up recovery speed "curl-s-XPUT http://$2:9200/_cluster/settings?flat_settings=true-d'{" transient ": {" indices.recovery.concurrent_streams ":" 10 " "indices.recovery.concurrent_small_file_streams": "10", "indices.recovery.max_bytes_per_sec": "100mb" "cluster.routing.allocation.node_concurrent_recoveries": "24"}'| jq. # execute Aliyun ES data recovery operation echo "start Aliyun ES data recovery" curl-s-XPOST http://$2:9200/_snapshot/$CLUSTER_NAME/$SNAPSHOT_NAME/_restore-d'{"index_settings": {"index.number_of_replicas": 0}} 'echoechoecho "to check the progress of recovery Curl-s-X GET http://$2:9200/_recovery/ | jq. "

After the Ali Cloud ES cluster turns green (observed via _ cat api or cerebro), the step7 operation begins.

Step7_aliyun_open_index_increase_replica.sh turns on the write function of Aliyun es and adjusts the copy of index to 1

#! / bin/bash# adjusts Aliyun es to increase the number of copies # Parameter 1 is the domain name of Aliyun es (without port number) echo "function:" echo "Aliyun es opens and writes And adjust the copy to 1 "echoecho" to open Aliyun ES's write permission "curl-s-XPUT http://$1:9200/_all/_settings-d'{" index ": {" blocks ": {" write ":" false "} 'sleep 1 echo" adjust the copy of all index to 1 "curl-s-XPUT http://$1:9200/_all/_settings-d' {" number_of_replicas ": 1}'

At this point, we can inform the developer that the test is released and then follow up the test.

Optional step

Step8_close_idc_index.sh recommends that the index of the old es be closed after the migration, so that if there is a connection to the old es, an error will be reported so that the problem can be found in time.

#! / bin/bash# pass parameter 1 to the lb address of the es cluster (or use the real backend node address) echo "function:" echo "closes the index of the es in the computer room" echo# lists the open index idx=$ (curl-s-XGET http://$1:9200/_cat/indices?h=index,status| grep-v close) indices=$ (echo $idx | sed's / /, / g'| sed's / Open//g') echo currently opens the following index: "echo $indicesechoechoecho" starts performing the close index operation. "curl-s-XPOST http://$1:9200/$indices/_close?pretty | jq.

So much for the data migration of ES through snapshots. I hope the above content can be helpful to you and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.