Shulou(Shulou.com)06/03 Report--

Recently, K8s has been applied to the company's test environment. Most containers use alpine-based images, and basic services such as MySQL and Redis are connected by domain names. However, there is a very strange phenomenon. Some applications in Pod report DNS timeout, and the error log probably goes like this (JAVA): io.netty.resolver.dns.DnsNameResolverTimeoutException: [/ 10.96.0.10 JAVA 53] query timed out after 5000 milliseconds (no stack trace available)

After searching on the Internet, it seems that this problem will occur when using alpine as a mirror image in K8s environment. Refer to this article: https://forum.choerodon.io/t/topic/1111, tries to solve the problem as follows

1. Use the modified alpine image that removes the parsing AAAA record, and installs glibc to support java. Image address: https://hub.docker.com/r/geekidea/alpine-a/, invalid

2. It is mentioned in the article that the problem of non-alpine images can be solved by modifying / etc/resolv.conf and adding options single-request-reopen, but not for alpine images because glibc is used for non-alpine images, while alpine uses musl libc by default. Thinking about whether the alpine we use already has glibc installed, can we try to modify it and add the following paragraph to YAML:

Lifecycle: postStart: exec: command:-/ bin/sh-- c-"/ bin/echo 'options single-request-reopen' > > / etc/resolv.conf"

Note: sed cannot be replaced directly here, it will report an error.

It's still invalid.

If you re-clarify your thinking, is it possible that there is a problem with the automatically generated / etc/resolv.conf, resulting in abnormal DNS? check the / etc/resolv.conf of Pod. The original file is as follows:

Nameserver 10.96.0.10search test1.svc.cluster.local svc.cluster.local cluster.localoptions ndots:5

Try ping test1.svc.cluster.local

Ping svc.cluster.local

Ping cluster.local

It is found that the latter two cannot communicate with ping at all, so do these two search fields cause YAML timeout, so I decided not to use the default generated / DNS, but to write it myself and view the official documents of K8s. A similar configuration was provided by the official. Add the following configuration to the spec section of DNS:

DnsPolicy: "None" dnsConfig: nameservers:-10.96.0.10 searches:-test1.svc.cluster.local options:-name: single-request-reopen

Note that nameservers is kube-dns 's ClusterIP,searches (search domain) according to your actual environment configuration

The generated / etc/resolv.conf is as follows:

Nameserver 10.96.0.10search test1.svc.cluster.localoptions single-request-reopen

Many attempts have been made to delete Pod and create Pod. The problem is no longer repeated. The original culprit lies in the search domain configuration of resolv.conf. The specific principle will be studied later, but there is no time to do so recently:)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.