Shulou(Shulou.com)05/31 Report--

In this issue, Xiaobian will bring you several ways to expose services on Kubernetes Engine TKE. The article is rich in content and analyzes and narrates from a professional perspective. After reading this article, I hope you can gain something.

Pre-knowledge 1. Service type on K8S

ClusterIP

Expose services through the internal IP of the cluster. Select this value. Services can only be accessed inside the cluster. This is also the default ServiceType.

NodePort

Services are exposed through IP and static ports on each Node. The NodePort service routes to the ClusterIP service, which is automatically created. A NodePort service can be accessed from outside the cluster by requesting: .

LoadBalancer

Using cloud providers 'load bureaus, services can be exposed externally. An external Load Balancer can route to NodePort services and ClusterIP services.

ExternalName

By returning CNAME and its value, you can map the service to the contents of the externalName field (for example, foo.bar.example.com). No proxy of any type was created.

Platform related basics

Tencent Cloud Kubernetes Engine (Tencent Kubernetes Engine, TKE) provides container-centric, highly scalable, high-performance container management services based on native Kubernetes, fully compatible with native Kubernetes API, and extends Kubernetes plug-ins such as Cloud Block Storage and Load Balancer of Tencent Cloud (Cloud) to provide a series of complete functions such as efficient deployment, resource scheduling, service discovery and dynamic scaling for containerized applications, solving the environmental consistency problem in user development, testing and O & M process. It improves the convenience of large-scale container cluster management, helps users reduce costs and improve efficiency.

2. Layer 4 network traffic exposure on TKE

By default, service-controller is used on TKE to manage Layer 4 listeners and rules on CLB. In this way, CLB binds the NodePort of each node at the backend. CLB receives external traffic, forwards it to the NodePort of one of the nodes, and then forwards it to Pod through Kubernetes 'internal Load Balancer using iptables or ipvs.

Request Details Process:

Request traffic into Load Balancer

Requests are forwarded by Load Balancer to the NodePort of a node

KubeProxy NAT forwards traffic from NodePort to a random pod

The request enters the container network and is forwarded to the corresponding node according to the Pod address.

Request comes to node where Pod belongs and forwards to Pod

The implementation results are as follows:

Reference document: cloud.tencent.com/document/product/457/45487

3. TKE layer 7 network traffic exposure

The default on TKE is l7-lb-controller as the Ingress controller, which manages layer 7 listeners and rules on CLB. The implementation principle and request details are the same as the four-layer implementation, but at the CLB level, it will be forwarded to different backend services according to the domain name and URL, and SSL offload can be performed.

Implementation details:

Using TKE's default Ingress creates a CLB for each Ingress resource and a Layer 7 listener rule of 80, 443.(HTTP/HTTPS), and for Ingress each location binding corresponding to each node of TKE a certain same NodePort as rs (Each location corresponds to a Service, and each Service exposes traffic through the same NodePort of each node). CLB forwards the traffic to the corresponding rs (i.e. NodePort) according to the matching location of the request. After the traffic reaches the NodePort, it will be forwarded to the corresponding backend Pod through iptables or ipvs of K8S.

The implementation results are as follows:

4. VPC-CNI on TKE

TKE usually uses the Global Router network mode (bridge scheme), and there is also a VPC-CNI (Elastic Network Interface (Network Interface) scheme). VPC-CNI is a new network mode on TKE. Each Pod is assigned an EIP of ENI Elastic Network Interface (Network Interface). Pods communicate directly through the Elastic Network Interface. It can be understood as: Each Pod is assigned an intranet IP.

Advantages: Each Pod can have an intranet IP

Disadvantages: Need to assign a separate free subnet

Reference document: cloud.tencent.com/document/product/457/41636

5. CLB Pass-Through Pod on TKE

CLB of TKE binds node IP and port by default. After VPC-CNI is used to provide independent intranet IP for Pod, CLB can bind Pod directly.

Request Details Process:

Request traffic into Load Balancer

The request is forwarded by Load Balancer to ENI Elastic Network Interface of a Pod

Reference document: cloud.tencent.com/document/product/457/41897

Reference document: mp.weixin.qq.com/s/fJtlm5Qjm2BfzekC4RegCQ

The implementation results are shown in the figure below. Note the ENI Elastic Network Interface and the actual port 80 of the Pod in the figure:

6. TKE uses an existing Load Balancer

Create CLB first

Add to the annotations of service:

service.kubernetes.io/tke-existed-lbid: lb-6swtxxxx

Reference document: cloud.tencent.com/document/product/457/45491

7. TKE uses intranet Load Balancer

This can be achieved by specifying the use of an existing intranet Load Balancer.

It can also be created dynamically by:

Add to the annotations of service:

service.kubernetes.io/qcloud-loadbalancer-internal-subnetid: subnet-xxxx # value Replace with one of the subnets id8. TKE Deploys Nginx Ingress

When TKE's default Ingress implementation (CLB's 7-layer rule) cannot meet business requirements, additional Nginx Ingress can be deployed (generally not used)

Reference document: cloud.tencent.com/document/product/457/47293

Best practices for real-world business scenarios 1. Exposed traffic within the cluster

[Priority] Level 4 agreement:

[Recommended] Use ClusterIP type Service, and access through the cluster domain name

[Optional] Use the four-layer rule of public network CLB

[Not recommended] Use the four-layer rule of intranet CLB

Seven-layer protocol:

[Recommended] Use ClusterIP type Service, and access through the cluster domain name, downgraded to a four-layer protocol

[Optional] Use the seven-layer rule of public network CLB

[Not recommended] Use the seven-layer rule of intranet CLB

The use of intranet CLB within a cluster requires attention to loopback issues, so it is not recommended to use intranet CLB within a cluster.

2. Exposed traffic outside the cluster

It is recommended that production environments use existing CLBs, i.e. manually create CLBs first, and then specify the CLB id in the relevant Service or Ingress.

The TKE default controller uses the following configuration in Service:

service.kubernetes.io/tke-existed-lbid: lb-6swtxxxx

At the same time, CLB enables "Enable default open", and the default open between CLB and CVM, so there is no need to adjust the security group on CVM according to NodePort, as shown in the following figure:

[Priority] Level 7 Agreement:

TKE comes with Ingress (3. TKE layer 7 network traffic exposure mode)

[Optional] Deploy Nginx Ingress by Yourself (8. TKE Deploys Nginx Ingress

Four-layer protocol:

TKE comes with LoadBalancer (2. TKE layer 4 network traffic exposure mode)

Using Istio:

Istio is a bit similar to Nginx Ingress, which is forwarded to NodePort by CLB layer 4 listener, and then forwarded to istio-ingressgateway-xx Pod through the rules defined by istio-ingressgateway service, and then forwarded to other Istio sidecars in the cluster.

The above is what are the several ways of service exposure on Kubernetes Engine TKE shared by Xiaobian. If there are similar doubts, please refer to the above analysis for understanding. If you want to know more about it, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.