Shulou(Shulou.com)06/01 Report--

I believe many inexperienced people are at a loss about which Linux version to choose in the container scenario. Therefore, this article summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

The underlying implementation of the container deeply depends on many features of the kernel, such as overlay file system, namespace, cgroup and so on, so the function and stability of the kernel, to a large extent, determine the function and stability of the whole container PaaS platform. From the operation experience of tens of thousands of clusters and hundreds of thousands of nodes since TKE was launched more than three years ago, kernel problems account for about 1/3 of all node problems.

So which Linux version is the best for container scenarios? In Tencent Cloud, the answer is Tencent Linux.

At this time, you should have a lot of questions, the next FAQ will try to answer your questions.

Who maintains the Tencent Linux? Based on what release?

It is maintained by Tencent kernel and virtualization team. Tencent Linux 2.4 is based on CentOS 7, the user mode package is compatible with the latest version of CentOS 7, and the CentOS 7 version of the package can be used directly in Tencent Linux 2.4.

What's the difference between Tencent Linux and CentOS?

The key difference lies in the kernel version, which will be described in more detail later in this article. There are some adjustments in user mode, such as the configuration of YUM source. For more information, please refer to the official introduction document: https://cloud.tencent.com/document/product/213/38027.

What is the kernel version?

Tencent Linux 2.4 is currently the 4.14 kernel. The code and the rpm package are available at GitHub: https://github.com/Tencent/TencentOS-kernel. Version 5.4 will be released by the end of the year.

What is the relationship between Tencent Linux and TKE's Optimized image?

Their kernels are the same, but Tencent Linux 2.4 is the CVM public image and the TKE Optimized image is the market image.

TKE will use Tencent Linux2.4 instead of CentOS 7.6 TKE Optimized and Ubuntu18.04 TKE Optimized. Clusters that are already using CentOS 7.6 TKE Optimized and Ubuntu18.04 TKE Optimized can continue to be used, but these two mirrors will no longer be supported in the new cluster.

It is expected that from late November, TKE users can operate on the console to switch the OS image of the newly created node in the cluster to Tencent Linux2.4. The operation entry is as follows (modify the operating system to Tencent Linux2.4):

What are the advantages over distributions such as CentOS and Ubuntu?

The main advantages are as follows, which will be described in detail later:

After years of verification and polishing of a large number of internal business of Tencent.

Support from a team of top kernel experts.

Contains some key performance optimizations and customization features for container scenarios.

Years of verification and polishing of Tencent's internal business

Tencent Linux started research and development in 2010 and has been in operation within Tencent for 10 years, with a total deployment of one million, accounting for 99% of Tencent's internal Linux system, covering all Tencent businesses. At the same time, Tencent has the most diverse business ecology in China, from social networking, gaming, to financial payment, AI, security and so on, so it has stronger requirements for the stability, performance and compatibility of the underlying operating system.

For container scenarios, a large number of Tencent's core businesses have been almost partially or completely containerized. For example, all logical businesses of Wechat have been containerized, and a series of optimizations have been carried out according to the business characteristics of Wechat. This successfully ensures the red packet peak operation during the Spring Festival every year. At the same time, in terms of data security, it also works closely with Wechat to provide solutions.

Support from a team of kernel experts

There are currently more than 30 full-time kernel experts supporting this kernel version. There are kvm maintainers, as well as many kernel networking, storage, cgroup, scheduling and other subsystems experts.

Support is also reflected in the pace of version updates and hot patch services.

From the release record (https://github.com/Tencent/TencentOS-kernel/releases) can see that from July to October, the Tencent Linux 4.14 kernel series has released five versions. Most of the problems encountered by Tencent's internal business and Tencent Cloud's external customers can be located and fixed in time.

In addition, we will provide online kernel hot patches for some important fixes. Hot patch installation and effective, do not need to restart the machine, without interrupting the customer business, improve the customer business latency SLA.

For vulnerability repair, Tencent Linux has a full set of hot patches, including application-level hot patches, kernel-level hot patches, and so on. More than 100 hot patches are released every year. Most vulnerabilities are fixed within a week.

Performance optimization

According to the problems encountered by internal and external users in large-scale landing practice, Tecent Linux has also made a large number of performance optimizations for container scenarios, including but not limited to:

Solve the problem of connection exception caused by connection reuse in IPVS mode with high concurrency (# 81775).

Solve the problem that too many IPVS rules cause network burr in IPVS mode under high-configuration nodes (with a large number of cores).

Solve the problem of network burr caused by cAdvisor reading memcg trapped in kernel state for too long in container-intensive scenarios (there are a large number of containers on a single node).

To solve the problem of network burr caused by CPU load balancing in the scenario of large Pod (with a large number of cores and a high number of cores).

Solve the problem of periodic network jitter caused by TCP connection monitoring in high concurrency scenarios (such as deploying cAdvisor configuration to monitor TCP connections separately).

Optimize the network packet receiving soft interrupt to improve the network performance.

The optimization results for various container scenarios are very significant. Take point 3 as an example, the effect of ping delay monitoring is as follows (optimized after 11:00):

Container customization features container resource display isolation

The efficient operation of many golang and java programs depends on correctly obtaining the CPU and memory resources available to the process. However, what this kind of program gets from the container is the CPU and memory resources of the node, which does not match the resources allocated by the actual container, which often results in unreasonable thread pool and other parameters of the process, which leads to problems.

The mainstream solution of the community is to deploy LXCFS implemented by FUSE to achieve / proc/cpuinfo, / proc/meminfo and other resource presentation isolation by container. This scenario requires the deployment of the LXCFS file system on the node and the insertion of the relevant volume and mount point configuration into the POD sepc. For more information, please refer to Kubernetes Demystified: Using LXCFS to Improve Container Resource Visibility

LXCFS-like features are implemented in the Tencnet Linux kernel so that users do not need to deploy LXCFS file systems on nodes or modify POD spec. You only need to turn on a global switch (sysctl-w kernel.stats_isolated=1) in the node, and read / proc/cpuinfo, / proc/meminfo and other files in the container are isolated by container. It is as simple as that.

In addition, considering that some special containers, such as node monitoring components, may need to read node-level information. In order to solve this problem, the container-level switch kernel.container_stats_isolated is specially added. When the host-level switch is enabled, you only need to turn off the container-level switch (sysctl-w kernel.container_stats_isolated=0) in the container startup script, and then read / proc/cpuinfo, / proc/meminfo and other files in this container to obtain the information of the host. (note: the container level switch must be set in the container to take effect on this container)

Please refer to the detailed usage documentation: isolation of CPU, memory, process, disk and other information in the container.

Isolation of more kernel parameters

Net.ipv4.tcp_max_orphans

Net.ipv4.tcp_workaround_signed_windows

Net.ipv4.tcp_rmem

Net.ipv4.tcp_wmem

Vm.max_map_count

These kernel parameters are often customized by the business. However, there is no namespace isolation for these parameters in the community kernel. Changes made by a container to the above parameters will work on the host and all other containers. According to the needs of internal and external customers, Tencent Linux realizes the namespace isolation of these kernel parameters, and the business container can set these parameters without worrying about the interference to other businesses.

Optimization of default kernel parameters of container

In the case of high concurrency, packet loss may occur when the semi-connection queue is full. The problem can be alleviated by scaling up the net.core.somaxconn. But the default value of net.core.somaxconn in the container network namespace is only 128, and it is written dead by code. In the Tencent Linux kernel, we adjust this default to 4096 to reduce packet loss when semi-connection queues are full in high concurrency cases.

How to use Tencent Linux in TKE

If you want the nodes of the TKE cluster to use the Tencent Linux operating system, you need to select the Tencent Linux operating system when creating the TKE cluster:

Note: Tencent Linux not only supports ordinary CVM models, but also supports CPM and Nvidia GPU models.

After reading the above, do you know which Linux version of the container scene to choose? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.