Shulou(Shulou.com)06/02 Report--

1. Problem: after the cluster red, all nodes restart and recover badly. Check the data directory under the node and find that there are no files in the corresponding index directory.

Refer to blog: http://www.wklken.me/posts/2015/05/23/elasticsearch-issues.html

(it is also said that one more node will be assigned automatically. I have added a node and found that this method does not work. After the operation of the following method, the cluster status will be green.)

Through a series of checks, it is found that 4 shards have not been assigned to nodes and cannot be allocated after restart. You can see the shards that cannot be assigned nodes through the head plug-in, and you can also see that 4 shards of unassigned shards cannot be allocated through the following command

Curl http://192.168.224.188:9200/_cluster/health\?pretty

{

"cluster_name": "gag-prod"

"status": "red"

"timed_out": false

"number_of_nodes": 3

"number_of_data_nodes": 3

"active_primary_shards": 233

Active_shards: 466

"relocating_shards": 0

"initializing_shards": 0

"unassigned_shards": 4,\ this is it

"delayed_unassigned_shards": 0

"number_of_pending_tasks": 0

"number_of_in_flight_fetch": 0

"task_max_waiting_in_queue_millis": 0

"active_shards_percent_as_number": 99.14893617021276

}

Curl http://192.168.224.188:9200/_cat/shards finds the index name of type UNASSIGNED here. (same as the remaining shard names seen in the head plug-in)

Items22 4 p STARTED 2273 571.1kb 192.168.224.187 gag-prod-node-187

Items22 4 r STARTED 2273 571.1kb 192.168.224.188 gag-prod-node-188

Items22 2 p UNASSIGNED

Items22 2 r UNASSIGNED

Items22 1 p STARTED 2284 555.2kb 192.168.224.187 gag-prod-node-187

Items22 1 r STARTED 2284 555.2kb 192.168.224.188 gag-prod-node-188

Items22 3 p STARTED 2276 641.5kb 192.168.224.187 gag-prod-node-187

Items22 3 r STARTED 2276 641.5kb 192.168.224.188 gag-prod-node-188

Items22 0 p UNASSIGNED

Items22 0 r UNASSIGNED

Shop_entity7 4 p STARTED 53 29.6kb 192.168.224.187 gag-prod-node-187

Curl http://192.168.224.188:9200_nodes/process?pretty views the unique identity of the master node

{

"cluster_name": "gag-prod"

"nodes": {

"tdp1G9DbRseQm8xS9v8jng": {\\ this is the unique identification of the 187node.

"name": "gag-prod-node-187"

"transport_address": "192.168.224.187VR 9300"

"host": "192.168.224.187"

"ip": "192.168.224.187"

"version": "2.3.2"

"build": "b9e4a6a"

"http_address": "192.168.224.187purl 9200"

"attributes": {

"master": "true"

}

"process": {

"refresh_interval_in_millis": 1000

"id": 10009

"mlockall": false

}

}

"a6tktPPYSCOGv4uw8uRclg": {\ this is the unique ID of the 186node.

"name": "gag-prod-node-186"

"transport_address": "192.168.224.186 purl 9300"

"host": "192.168.224.186"

"ip": "192.168.224.186"

"version": "2.3.2"

"build": "b9e4a6a"

"http_address": "192.168.224.186 purl 9200"

"attributes": {

"master": "false"

}

"process": {

"refresh_interval_in_millis": 1000

"id": 24049

"mlockall": false

}

}

"d5DvDdr6SLak8YCC099jRg": {\ this is the unique ID of the 188node.

"name": "gag-prod-node-188"

"transport_address": "192.168.224.188 purl 9300"

"host": "192.168.224.188"

"ip": "192.168.224.188"

"version": "2.3.2"

"build": "b9e4a6a"

"http_address": "192.168.224.188 purl 9200"

"attributes": {

"master": "true"

}

"process": {

"refresh_interval_in_millis": 1000

"id": 13058

"mlockall": false

}

}

}

}

Through the above operations, we have found the "problem shard" and "node unique identity", and now we can force the problem shard to be assigned to one of the nodes. Next, let's divide the question into pieces on gag-prod-node-187.

Edit script: (if you have a lot of unassigned shards, you can write a circular script)

#! / bin/bash

# forcibly assign items22 0 to gag-prod-node-187 (tdp1G9DbRseQm8xS9v8jng)

Curl-XPOST '192.168.224.187VR 9200 Swiss clusterUniverse Reroute'-d'{

"commands": [{

"allocate": {

"index": "items22"

"shard": 0

"node": "tdp1G9DbRseQm8xS9v8jng"

"allow_primary": true

}

}

]

}'

# forcibly assign items22 2 to gag-prod-node-187 (tdp1G9DbRseQm8xS9v8jng)

Curl-XPOST '192.168.224.187VR 9200 Swiss clusterUniverse Reroute'-d'{

"commands": [{

"allocate": {

"index": "items22"

"shard": 2

"node": "tdp1G9DbRseQm8xS9v8jng"

"allow_primary": true

}

}

]

}'

After running this script, check the status of the cluster, and it has been restored. After the shard is automatically backed up to another node, stop the gag-prod-node-187 node, and the shard can be shredded automatically.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.