Shulou(Shulou.com)05/31 Report--

This article will explain in detail how to use Unix Domain Socket to communicate with upstream clusters in Envoy. The content of the article is of high quality, so the editor shares it for you as a reference. I hope you will have some understanding of the relevant knowledge after reading this article.

In most cases, Envoy Proxy is deployed as a Sidecar in the same network environment as the application. Each application only needs to interact with Envoy (localhost) and does not need to know the address of other services. However, this is not the only use scenario for Envoy. It is a layer-7 proxy that implements core functions such as traffic governance and information monitoring through a modular structure. For example, traffic governance features include automatic reconnection, circuit breaker, global speed limit, traffic mirroring, anomaly detection and other advanced features. Therefore, Envoy is often used for edge proxies, such as Ingress Gateway of Istio and Ingress Controller based on Envoy (Contour, Ambassador, Gloo, etc.).

My blog is also deployed on a lightweight Kubernetes cluster (actually K3s). At first, I used Contour as an Ingress Controller to expose blogs, comments, and other services in the cluster. But the good times didn't last long, because I deployed all kinds of strange things in the cluster, some personalized configuration Contour could not meet my needs, after all, we all know that every level of abstraction will lose a lot of details. If you can't guarantee a Controller, you will encounter this kind of problem, so just use Envoy as an edge proxy directly, and you can only do YAML by hand.

Of course, it is not all hand masturbation. Although there is no so-called control plane, I still need a sense of ritual. I can dynamically update the configuration based on files. For specific methods, please refer to the Envoy basic tutorial: dynamically updating configurations based on file systems.

1. UDS introduction

After all that nonsense, let's get down to business. In order to improve the performance of the blog, I chose to deploy the blog on the same node as Envoy, and all use HostNetwork mode, and Envoy communicates with the Pod (Nginx) where the blog is located through localhost. To further improve performance, I set my sights on Unix Domain Socket (UDS,Unix domain socket), which is also called IPC (inter-process communication). To understand UDS, let's build a simple model.

In the real world, the whole process of information exchange between two people is called a communication (Communication), and both parties of communication are called endpoints (Endpoint). Different tool communication environment, between the endpoints can choose different tools for communication, close distance can be direct dialogue, long distance can choose to make a phone call, Wechat chat. These tools are called Socket.

Similarly, there is a similar concept in computers:

In Unix, a communication consists of two endpoints, such as a HTTP server and a HTTP client.

If you want to communicate between endpoints, you must use some tools to communicate with each other in Unix using Socket.

Socket was originally designed for network communication, but then an IPC mechanism, UDS, was developed on the framework of Socket. The benefits of using UDS are obvious: no need to go through the network protocol stack, no need to pack and unpack, calculate checksum, maintain sequence numbers and replies, etc., just copy application layer data from one process to another. This is because the IPC mechanism is essentially reliable communication, while network protocols are designed for unreliable communication.

The most obvious difference between UDS and network Socket is that the network Socket address is the IP address plus the port number, while the UDS address is the path of a file of type Socket in the file system, and the general name ends with .sock. The Socket file can be referenced by the system process, two processes can open a UDS to communicate at the same time, and this kind of communication will only occur in the system kernel and will not be propagated on the network. Let's take a look at how to let Envoy communicate with the upstream cluster Nginx through UDS. The communication model between them looks something like this:

2. Nginx listens to UDS

First of all, you need to modify the configuration of Nginx so that it listens on UDS, and as for the location of the Socket descriptor file, it's up to you. You need to modify the listen parameter to the following form:

Listen unix:/sock/hugo.sock

Of course, if you want to get faster communication speed, you can put it in the / dev/shm directory, which is called tmpfs, which is an area that RAM can use directly, so you can read and write very fast, which will be explained separately below.

3. Envoy-- > UDS-- > Nginx

By default, Envoy uses IP address and port number to communicate with the upstream cluster. If you want to use UDS to communicate with the upstream cluster, you first need to modify the type of service discovery and change type to static:

Type: static

At the same time, you need to define the endpoint as UDS:

-endpoint: address: pipe: path: "/ sock/hugo.sock"

The final Cluster configuration is as follows:

-"@ type": type.googleapis.com/envoy.api.v2.Cluster name: hugo connect_timeout: 15s type: static load_assignment: cluster_name: hugo endpoints:-lb_endpoints:-endpoint: address: pipe: path: "/ sock/hugo.sock"

Finally, to enable Envoy to access the Socket file of Nginx, the same emptyDir can be mounted into two Container in Kubernetes to achieve the purpose of sharing, of course, the biggest premise is that the Container in Pod is a shared IPC. The configuration is as follows:

Spec:... Template:... Spec: containers:-name: envoy. VolumeMounts:-mountPath: / sock name: hugo-socket.-name: hugo. VolumeMounts:-mountPath: / sock name: hugo-socket. Volumes:...-name: hugo-socket emptyDir: {}

Now you can happily visit my blog, check the Envoy log, and successfully forward the request to the upstream cluster via Socket:

[2020-04-27T02:49:47.943Z] "GET / posts/prometheus-histograms/ HTTP/1.1" 169949 10 "66.249.64.209" Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1) + http://www.google.com/bot.html)"9d490b2d-7c18-4dc7-b815-97f11bfc04d5"fuckcloudnative.io" / dev/shm/hugo.sock "

Hey, hey, Google's reptiles are also coming to join the party.

You might ask me: why is your Socket here in the / dev/shm/ directory? Don't worry, it's not over yet. Let's add a piece of background knowledge first.

4. Linux shared memory mechanism

Shared memory (shared memory) is a mechanism for interprocess communication (IPC) on Linux.

Inter-process communication can use pipes, Socket, signals, semaphores, message queues, etc., but these methods usually need to be copied between user mode and kernel state, and it is generally believed that there will be 4 copies; in contrast, shared memory maps memory directly to user state space, that is, multiple processes access the same memory, which has higher performance in theory. Hey hey, we can improve the above plan again.

There are two mechanisms for shared memory:

POSIX shared memory (shm_open (), shm_unlink ())

System V shared memory (shmget (), shmat (), shmdt ())

Among them, System V shared memory has a long history, general UNIX systems have this mechanism, while POSIX shared memory mechanism interface is more convenient to use, generally combined with memory mapping mmap.

The main differences between mmap and System V shared memory are:

System V shm is persistent and remains in memory until the system shuts down unless explicitly deleted by a process.

Mmap-mapped memory is not persistent, and if the process shuts down, the mapping fails unless it has been mapped to a file in advance.

/ dev/shm is the default mount point for sysv shared memory under Linux.

POSIX shared memory is implemented based on tmpfs. In fact, further, not only PSM (POSIX shared memory), but also SSM (System V shared memory) is implemented in the kernel based on tmpfs.

You can see from here that tmpfs has two main functions:

For System V shared memory, as well as anonymous memory mapping; this part is managed by the kernel and is not visible to users.

For POSIX shared memory, the user is responsible for mount, and generally mount to / dev/shm, depending on CONFIG_TMPFS.

Although both System V and POSIX share memory through tmpfs, they are subject to different limitations. That is, / proc/sys/kernel/shmmax only affects System V shared memory, and / dev/shm only affects POSIX shared memory. In fact, System V and POSIX share memory is already used by two different tmpfs instances.

The memory space that can be used by System V shared memory is limited only by / proc/sys/kernel/shmmax, while users mount / dev/shm, which defaults to 1 and 2 of physical memory.

To sum up:

Both POSIX shared memory and System V shared memory kernel are implemented through tmpfs, but corresponding to two different tmpfs instances, they are independent of each other.

The maximum value of System V shared memory can be limited with / proc/sys/kernel/shmmax, and the maximum value of POSIX shared memory can be limited with / dev/shm.

5. Kubernetes shared memory

A Pod created by Kubernetes whose shared memory defaults to 64MB and cannot be changed.

Why is this value? In fact, Kubernetes itself does not set the size of shared memory, 64MB is actually the default size of shared memory for Docker.

When Docker run, you can use-- shm-size to set the size of shared memory:

?? → docker run-- rm centos:7 df-h | grep shmshm 64M 064m 0% / dev/shm???? → docker run-- rm-- shm-size 128m centos:7 df-h | grep shmshm 128M 0128m 0% / dev/shm

However, Kubernetes does not provide a way to set the shm size. In this issue, the community has discussed for a long time whether to add a parameter to shm, but in the end there is no conclusion, but there is a workgroud solution: mount an emptyDir of type Memory to / dev/shm to solve the problem.

Kubernetes provides a special emptyDir: you can set the emptyDir.medium field to "Memory" to tell Kubernetes to use tmpfs (a RAM-based file system) as the media. Users can mount the emptyDir of Memory media to any directory, and then use this directory as a high-performance file system, or mount it to / dev/shm, which can solve the problem of insufficient shared memory.

Using emptyDir can solve the problem, but it also has disadvantages:

Users cannot be banned from using memory in a timely manner. Although Kubelet will crowd out Pod after 1-2 minutes, but during this time, there is still a risk to Node.

Affects Kubernetes scheduling because emptyDir does not involve Node's Resources, which causes Pod to "secretly" use Node's memory, but the scheduler is not aware of it.

The user cannot perceive that the memory is unavailable in time.

Since shared memory is also limited by Cgroup, we only need to set Memory limits for Pod. If you set the Memory limits of Pod to the size of shared memory, you will encounter a problem: when shared memory is exhausted, no command can be executed and can only be expelled by Kubelet after timeout.

This problem is also easy to solve, just set the size of shared memory to 50% of Memory limits. Based on the above analysis, the final design is as follows:

Mount the emptyDir of the Memory media to / dev/shm/.

Configure Memory limits for Pod.

The sizeLimit of configuring emptyDir is 50% of Memory limits.

6. Final configuration

According to the above design, the final configuration is as follows.

The configuration of Nginx is changed to:

Listen unix:/dev/shm/hugo.sock

The configuration of Envoy is changed to:

-"@ type": type.googleapis.com/envoy.api.v2.Cluster name: hugo connect_timeout: 15s type: static load_assignment: cluster_name: hugo endpoints:-lb_endpoints:-endpoint: address: pipe: path: "/ dev/shm/hugo.sock"

The manifest of Kubernetes is changed to:

Spec:... Template:... Spec: containers:-name: envoy resources: limits: memory: 256Mi. VolumeMounts:-mountPath: / dev/shm name: hugo-socket...-name: hugo resources: limits: memory: 256Mi. VolumeMounts:-mountPath: / dev/shm name: hugo-socket. Volumes:-name: hugo-socket emptyDir: medium: Memory sizeLimit: 128Mi so much about how to use Unix Domain Socket to communicate with the upstream cluster in Envoy foundation. I hope the above content can be helpful to you and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.