Shulou(Shulou.com)05/31 Report--

Today, I will talk to you about how to sort out the troubleshooting ideas for the deployment of K8S, which may not be well understood by many people. in order to make you understand better, the editor has summarized the following contents for you. I hope you can get something according to this article.

The following will help you figure out how to debug deployment in Kubernetes. The following figure is a complete troubleshooting idea

When you want to deploy an application in Kubernetes, you usually need to define three components:

Deployment-- this is the way to create a copy of the application named Pods

Serivce-- internal load balancer to route traffic to Pods

Ingress-- can describe how traffic flows from outside the cluster to Service

Next, let's take a quick look at the picture.

In Kubernetes, your application is exposed through two layers of load balancers: internal and external.

The internal load balancer is called Service, while the external load balancer is called Ingress.

Pod is not deployed directly, so Deployment creates Pod and monitors them.

Suppose you want to deploy a simple Hello World application, and for such an application, the YAML file looks like this:

ApiVersion: apps/v1kind: Deploymentmetadata: name: my-deployment labels: track: canaryspec: selector: matchLabels: any-name: my-app template: metadata: labels: any-name: my-app spec: containers:-name: cont1 image: learnk8s/app:1.0.0 ports:-containerPort: 8080---apiVersion: v1kind: Servicemetadata: name: my-servicespec: Ports:-port: 80 targetPort: 8080 selector: name: app---apiVersion: networking.k8s.io/v1beta1kind: Ingressmetadata: name: my-ingressspec: rules:-http: paths:-backend: serviceName: app servicePort: 80 path: /

This definition is so long that it is easy to ignore the interrelationships between components.

For example:

When should you use port 80 and when should you use port 8080?

Should you create a new port for each service in case they conflict?

Is the name of the label important? Should it be the same everywhere?

Before debug, let's review how these three components relate to each other.

First, let's start with Deployment and Service.

Connect Deployment and Service

In fact, Deployment and Service are not connected at all. Instead, the Service points directly to Pod and skips Deployment altogether. So, you should focus on how Pod and Service relate to each other. You should remember three things:

Service selector matches at least one tag of Pod

The Serivce targetPort should match the containerPort of the container in Pod

Serviceport can be any number. Multiple Service can use the same port because they have been assigned different IP addresses

The following picture summarizes how to connect the port:

Consider pod exposed by Service

When you create a pod, you should define a port containerPort for each container in your Pod

When you create a Service, you can define a port and a targetPort. But which one should you connect to the container?

TargetPort and containerPort should match

If your container exposes port 3000, then targetPort should match that number.

If you check YAML, the tag should match ports or targerPort:

ApiVersion: apps/v1kind: Deploymentmetadata: name: my-deployment labels: track: canaryspec: selector: matchLabels: any-name: my-app template: metadata: labels: any-name: my-app spec: containers:-name: cont1 image: learnk8s/app:1.0.0 ports:-containerPort: 8080---apiVersion: v1kind: Servicemetadata: name: my-servicespec: Ports:-port: 80 targetPort: 8080 selector: any-name: my-app

What about the track: canary tag at the top of the Deployment? Should it also be a match?

That label belongs to deployment, and Service selector does not use it to route traffic. In other words, you can safely remove it or assign it a different value.

What about matchLabelsselector? It needs to match the Pod tag and Deployment uses it to track Pod.

Suppose you make a correct change, how should you test it? You can check that Pod has the correct label using the following command:

Kubectl get pods-show-labels

Or if you have Pod that belongs to multiple applications:

Kubectl get pods-selector any-name=my-app-show-labels

Where any-name=my-app is the label any-name: my-app. Is there still a problem? You can also connect to Pod. You can use the command port-forward in kubectl to connect to Serivce and test the connection.

Kubectl port-forward service/ 3000:80

Where:

Service/ is the name of serivce-- in the current YAML, it is "my-service".

3000 is the port you want to open on your computer.

80 is the port exposed by Service in the port field

If you can connect, then the settings are correct. If you can't connect, you probably have the wrong label or port mismatch.

Connect Service and Ingress

The next step in exposing the application is to configure Ingress. Ingress must know how to retrieve Service, and then retrieve Pod and route traffic to them. Ingress retrieves the correct Service by name and exposed port.

You should match two things in Ingress and Service:

The servicePort of Ingress should match the port of Service

The serviceName of Ingress should match the name of Service

The following picture summarizes how to connect the port:

You already know that the service has exposed a port.

Ingress has a field called servicePort.

Serviceport and IngressservicePort should match.

If you decide to assign port 80 to the service, you should also change the servicePort to 80

In practice, you need to look at these command lines:

ApiVersion: v1kind: Servicemetadata: name: my-servicespec: ports:-port: 80 targetPort: 8080 selector: any-name: my-app---apiVersion: networking.k8s.io/v1beta1kind: Ingressmetadata: name: my-ingressspec: rules:-http: paths:-backend: serviceName: my-service servicePort: 80 path: /

How should you test whether Ingress is working properly? You can use the same strategy as before, that is, kubectl port-forward, but not connect to service, but connect to Ingress controller.

First, retrieve the Pod name for Ingress controller using the following command:

Kubectl get pods-all-namespacesNAMESPACE NAME READY STATUSkube-system coredns-5644d7b6d9-jn7cq 1 Runningkube-system etcd-minikube 1 Runningkube-system kube-apiserver-minikube 1 Runningkube-system kube-controller-manager-minikube 1 Runningkube-system kube-proxy-zvf2h 1 Runningkube-system kube-scheduler-minikube 1/1 Runningkube-system nginx-ingress-controller-6fc5bcc 1/1 Running

Verify the Ingress Pod (possibly in a different namespace) and describe it to retrieve the port:

Kubectl describe pod nginx-ingress-controller-6fc5bcc\-namespace kube-system\ | grep PortsPorts: 80/TCP, 443/TCP, 18080/TCP

Finally, connect to Pod:

Kubectl port-forward nginx-ingress-controller-6fc5bcc 3000 80-- namespace kube-system

At this point, every time you access port 3000 on your computer, the request will be forwarded to port 80 on Ingress controller Pod.

If you visit http://localhost:3000, you should be able to find an application that provides web pages.

A brief review

Now, let's take a quick look at what ports and labels need to match:

Service selector should match the label of Pod

The ServicetargerPort should match the containerPort of the container in the Pod

The Service port can be any number. Multiple Service can use the same port because they have been assigned different IP addresses

The servicePort of Ingress should match the port in Service

The name of Service should match the field of serviceName in Ingress

Understanding how to construct a YAML is just the beginning. So, what happens when something goes wrong? Pod may fail to start or crash directly.

3-step troubleshooting of K8S Deployment

Before we delve into faulty deployment, we must have a well-defined model to understand how Kubernetes works.

Now that you have those three components in each deployment, you should debug them sequentially from the bottom.

You should make sure that your Pod is running

Focus on getting Service to route traffic to Pod

Check whether Ingress is configured correctly

You should start troubleshooting Deployment from the bottom. First, check that Pod is ready and running

If Pod is ready, you need to check to see if Service can allocate traffic to Pod.

Finally, you should check the connection between Service and Ingress.

1. Troubleshooting Pod

In most cases, the problem lies with Pod itself. So you should make sure that Pod is running and ready. How should I check it?

Kubectl get podsNAME READY STATUS RESTARTS AGEapp1 0 ImagePullBackOff 1 ImagePullBackOff 0 47happ2 0 76f9fcd46b-xbv4k 1 Error 0 47happ3-76f9fcd46b-xbv4k 1 Running 1 47h

In the above section, only the last Pod is running and ready, while the first two Pod have neither Running nor Ready. So, how should you locate what went wrong?

Here are four very useful commands to help you troubleshoot Pod:

Kubectl logs can help retrieve the container log of Pod

Kubectl describe pod can effectively retrieve the list of events related to Pod

Kubectl get pod is useful for extracting YAML definitions of Pod stored in Kubernetes

Kubectl exec-ti bash can be used to run an interactive command in one of the Pod containers

Which one should you use? In fact, no command is omnipotent, and you can use it together according to the actual situation.

Common Pod errors

Pod may have startup and run-time errors.

Startup errors include:

ImagePullBackoff

ImageInspectError

ErrImagePull

ErrImageNeverPull

RegistryUnavailable

InvalidImageName

Run-time errors include:

CrashLoopBackOff

RunContainerError

KillContainerError

VerifyNonRootError

RunInitContainerError

CreatePodSandboxError

ConfigPodSandboxError

KillPodSandboxError

SetupNetworkError

TeardownNetworkError

Some of these errors are more common than others. Here are the most common errors and how to fix them:

ImagePullBackOff

This error occurs when Kubernetes is unable to retrieve the container image of one of the Pod.

There are three common reasons:

The image name is not valid-for example, you misspelled the name or the image does not exist

You specified a tag that does not exist for this image.

The image you retrieved is in a private repository, and Kubernetes has no credentials to access it.

The first two reasons can be solved by correcting the image name and tag. Finally, you need to add the credentials to the private image repository in "Secret" and reference it in Pod.

Official documentation can make you more clear:

Https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

CrashLoopBackOff

If the container fails to start, the Kubernetes status displays a CrashLoopBackOff message.

Typically, the container cannot be started in the following scenarios:

There is an error in the application, which makes it impossible to start

You misconfigured the container.

Https://stackoverflow.com/questions/41604499/my-kubernetes-pods-keep-crashing-with-crashloopbackoff-but-i-cant-find-any-lo

Liveness probe failed too many times.

You should try and retrieve the container's log to determine the cause of the failure.

If you cannot view the log because your container restarts too quickly, you can use the following command:

Kubectl logs-previous

It prints an error message from the previous container.

RunContainerError

An error occurs when the container cannot be started, even before the application within the container starts.

This problem is usually caused by misconfiguration, such as:

Install a volume that does not exist, such as ConfigMap or Secret

Install read-only volume as read / write

You should use kubectl describe pod to collect and analyze errors.

Pod is in Pending state

When you create a Pod, the Pod remains in the Pending state. Why is that? Assuming that your scheduling component is running well, there are several reasons:

The cluster does not have enough resources to run Pod, such as CPU and memory

The current namespace has a ResourceQuota object and the Pod created will make the namespace exceed the resource quota

Pod is bound to a PersistentVolumeClaim of Pending state.

The best option, then, is to use the command kubectl describe to check for events:

Kubectl describe pod

For errors caused by ResourceQuotas, you can check the logs of the cluster using the following methods:

Kubectl get events-- sort-by=.metadata.creationTimestampPod is not in the Ready state

If Pod is running but not Ready, this means that the Readiness probe is faulty. When the Readiness probe fails, the Pod cannot be attached to the Service and the traffic cannot be forwarded to the instance.

Readiness probe failures are application-specific errors, so use kubectl describe to check the event section to verify the error.

2. Troubleshoot Service

If your Pod is running and ready, but you still can't receive a response from the application, you should check that the Service is configured correctly.

Service is designed to route traffic to Pod based on the label of pod. So first thing, you need to check how many Pod of Service target. You can do this by checking the Endpoint in Service:

Kubectl describe service | grep Endpoints

An endpoint is a pair of ```, and when Service (at least) target a pod. At least one pair.

If the "Endpoint" part is empty, there are two explanations:

Any running Pod does not have the correct label (hint: you need to check that you are in the correct namespace)

There are typos in the selector tag of Service

If you see the endpoint list but still can't access your application, then targetPort in your Service may be the culprit.

How should you test Service? Whatever the Service type is, you can connect to it using kubectl port-forward:

Kubectl port-forward service/ 3000:80

Where:

```is the name of Service

3000 is the port you want to open on your computer.

80 is the port exposed by Service

3. Troubleshoot Ingress

If you get to this part, it means:

Pod is running and ready

Service can distribute traffic to Pod

But you still can't receive a response from app. Then there is a good chance that there is an error in the configuration of Ingress.

Because the Ingress controller used is a third-party component in the cluster, it will be debugged by different techniques depending on the type of Ingress controller. But before delving into Ingress-specific tools, you can use some simple methods to check.

Ingress uses serviceName and servicePort to connect to Service. You should check that those are configured correctly. You can use the following command to check that Ingress is configured correctly:

Kubectl describe ingress

If the Backend column is empty, there must be an error in the configuration.

If you can see endpoint in the Backend column, but still cannot access the application, then the following may be the problem:

The way you expose Ingress to the public network

The way you expose your cluster to the public network

You can isolate infrastructure problems from Ingress by connecting directly to Ingress Pod.

First, retrieve the Pod for your Ingress Controller (possibly in a different namespace):

Kubectl get pods-all-namespacesNAMESPACE NAME READY STATUSkube-system coredns-5644d7b6d9-jn7cq 1 Runningkube-system etcd-minikube 1 Runningkube-system kube-apiserver-minikube 1 Runningkube-system kube-controller-manager-minikube 1 Runningkube-system kube-proxy-zvf2h 1 Runningkube-system kube-scheduler-minikube 1/1 Runningkube-system nginx-ingress-controller-6fc5bcc 1/1 Running

Describe it to retrieve the port:

Kubectl describe pod nginx-ingress-controller-6fc5bcc-- namespace kube-system\ | grep Ports

Finally, connect to Pod:

Kubectl port-forward nginx-ingress-controller-6fc5bcc 3000 80-- namespace kube-system

At this point, every time you access port 3000 on your computer, the request will be forwarded to port 80 on Pod.

So, is it working properly now?

If it works properly, the problem is the infrastructure. You should check how traffic is routed to your cluster.

If it doesn't work properly, the problem is Ingress controller. You should debug Ingress.

If you still cannot make Ingress controller work properly, you should start debugging it. There are many different versions of Ingress controller on the market. The popular ones include Nginx, HAProxy, Traefik and so on.

You should consult the Ingress controller documentation for troubleshooting guidelines.

Since Ingress Nginx is the most popular Ingress controller, we will introduce some related techniques in the next section.

Debug Ingress Nginx

Ingress-nginx has an official plug-in for kubectl. You can check it out at the following URL:

Https://kubernetes.github.io/ingress-nginx/kubectl-plugin/

You can use kubectl ingress-nginx to do the following:

Check logs, Backend, certificates, etc.

Connect to Ingress

Check the current configuration

You can also try the following three commands:

Kubectl ingress-nginx lint, this is used to check nginx.conf.

Kubectl ingress-nginx backend to check Backend (similar to kubectl describe ingress)

Kubectl ingress-nginx logs to check the log

Note that you need to use-- namespace to specify the correct namespace.

After reading the above, do you have any further understanding of how to sort out the troubleshooting ideas for K8S deployment? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.