Shulou(Shulou.com)06/02 Report--

Tags: restart K8S node part of pvc cannot be mounted normally

Restart K8S Node part pvc cannot be mounted normally

=

In this article, we focus on the solution that some pvc cannot be mounted normally after restarting the K8S node.

1. Failure occurs:

   one of our K8s computing nodes fails. After a forced restart, when a gitlab pod starts, the log viewed through kubectl describe pods gitlab always reports an error:

Multi-Attach error for volume "pvc-76b54b6c-df46-11e7-a2f0-005056b12f99" Volume is already exclusively attached to one node and can't be attached to another 2. Fault handling:

Check which rbd image corresponds to pv. This command is executed on the k8s node where the kubectl command is installed.

Kubectl get pv pvc-1557d839-714f-11e9-8d1a-5065f3457c8c-o go-template=' {{.spec.rbd.image}}'

Based on the output of the previous step, view the watcher,-p specified pool of rbd image on the ceph node

Sudo rbd status kubernetes-dynamic-pvc-199d4a26-714f-11e9-bb25-480fcf482f56-p kube

Based on the output, log in to that machine and check the rbd images mapped relationship

Sudo rbd showmapped | kubernetes-dynamic-pvc-199d4a26-714f-11e9-bb25-480fcf482f56

The last step is to uninstall rdb imags on this machine according to the result of the previous command

Sudo rbd unmap / dev/rbd8

Then restart pod and find that pvc can be mounted normally

For more details of the blog post, please follow my personal Wechat official account "Cloud era IT Operation and maintenance", which aims to share new technologies and trends of Internet operation and maintenance, including consultation from IT operation and maintenance industry and operation and maintenance technical document sharing. Focus on devops, jenkins, zabbix monitoring, kubernetes, ELK, the use of various middleware, such as redis, MQ, etc.; shell and python and other operation and maintenance programming languages; I have been engaged in IT operation and maintenance related work for more than ten years. Began to work full-time in Linux/Unix system operation and maintenance in 2008; have a certain degree of understanding of operation and maintenance related technology. All the blog posts in this official account are a summary of my actual work experience, and they are basically original blog posts. I am happy to share my accumulated experience, experience and technology with you! I hope to grow and make progress with you on the career path of IT operation and maintenance.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.