Shulou(Shulou.com)06/02 Report--

ASP.NET Core application in Kubernetes memory utilization is too high analysis, I believe that many inexperienced people do not know what to do, so this paper summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

In the production environment, we deploy the asp.net core api application on Google Cloud (GCE-Google Container Engine) through Kubernetes. We found that the memory usage of most components (core applications) is unreasonable. We set the memory limit of the application to 500MB, and also found that many api application instances were constantly restarted by Kubernetes because they exceeded the memory limit (docker should be set-restart).

The following two pictures are two of our api, and when Kubernetes restarts them, you will find that they first keep growing and then reach the point of memory limit.

Aiming at this phenomenon, we spent a lot of time investigating this issue, during which we tried to analyze it by grasping dumps, but we didn't find the problem.

We also tried to recreate this problem in our development environment in a variety of ways:

In dev configuration in VS

On Windows with a production build

On Ubuntu with a production build

In Docker, using the actual production image

But in the above environment, none of them exceeded the memory usage of 500mb, and they all stopped when they grew to around 100-150mb.

During this period, in order to reduce the frequent restart of the container due to exceeding the maximum memory limit, we increased the memory limit from 500mb to 1000mb. After that, it is interesting that the memory usage becomes as shown in the following figure:

After testing, it is found that memory usage does not increase indefinitely, but it is also capped around 600mb, and this number is almost the same after different container instances and instances are restarted.

This phenomenon clearly shows that the application in our container does not have a memory leak, only that a piece of memory is allocated and not released. So I started to shift my focus to "how. Net programs running on Kubernetes limit memory."

In fact, Kubernetes ultimately runs the program in a docker container, and the docker container can limit memory usage through the docker run-memory parameter. So I suspect that maybe Kubernetes didn't pass any parameters about memory limits to the docker container instance, so. Net programs take it for granted that the current machine has a lot of memory available.

But this is not the case, we find the opposite (because the author suspects that Kubernetes does not pass memory-related parameters) in the documentation.

The spec.containers [] .resources.invents.memory is converted to an integer, and used as the value of the-- memory flag in the docker run command.

(this means that Kubernetes's spec.containers [] .resources.invents.memory automatically follows the integer value set by the-- memory parameter in docker run.)

This seems to be another dead end. I also tried to run the api program in docker on my computer, and passed a variety of memory limits through the-- memory parameter, but 1. I cannot reproduce the above 600mb memory usage scenario, which is only kept around 150mb, 2. 5%. It is also not observed that the container instance runs beyond its memory limit, and even if I specify a value less than 150mb through the-- memory parameter, the container instance can still run well under this smaller memory limit.

I also mentioned an issue about memory leaks associated with Kestrel (a new libuv-based server for core) on github a long time ago, and at this point, Tim Seaward sent an interesting suggestion about checking the number of cpu printed by my application in different environments, because cpu is a huge factor affecting memory usage.

The number I tried to print out in different environments through Environment.ProcessorCount in the code is as follows:

On my machine, just doing dotnet run, the value was 4.

On my machine, with Docker, it was 1.

On Google Cloud Kubernetes it was 8.

This finally gives me an explanation, because the number of cpu really affects the amount of memory used. The more cores of cpu, the more memory is used (for the author, he doesn't know exactly the type of gc, the number of cores of cpu, and the size of memory used by .net programs, although the link this post contains information about GC).

The final suggestion is to switch the GC mode from Server GC (server mode) to Workstation GC (mode), so as to achieve the optimization effect of low memory usage. You only need to do the following in the csproj project file:

False

After making this change, republish my api, and the result is as shown in the countryside (blue line):

The workstation gc pattern makes applications' use of memory more "conservative", and memory usage is reduced from about 600mb to between 100 and 150mb. Assuming that the workstation mode achieves this "600mb-to-150mb effect" at the expense of some performance and throughput (according to the official server mode is relatively better than the workstation mode in some scenarios), but so far I have not found any decline in api speed and throughput, although my api is not a very stringent performance requirement.

It is concluded from this story that OS, available memory, and the number of cpu cores are all key factors in locating memory problems, because they can greatly affect the GC of. Net. If you are stuck with a problem, please do not hesitate to throw it out, and there are many wonderful people in the .net community who will be happy to help others.

= split Lin

Little brother, the project under our company is also the same problem. At that time, it was perplexed for a long time. Why? because before, under windows, the memory occupation would not be "too obvious", because GC played a decisive role. But in the environment of core, plus docker+linux, it was suspected to be a docker problem at that time. At that time, we did not analyze this problem like this foreign friend. Through this question, I have learned the following:

1. I learned the step of this brother's positioning and thinking, and finally solved the problem from whether it is the problem of K8s-"multi-environmental problems -" github issue- "do analogies and speculate".

Additional knowledge points of 2.GC: this post

3. In addition, there are many knowledge points hidden: suggestion (I hope you will take a closer look at it)

4. I remember correctly that the configuration of GC was mentioned in the section on the project configuration file in the previous English official file.

At the beginning (the two types of GC are not known), normal people see that this true refers to the GC garbage collection that activates the application, but does not notice the results of the survey conducted by foreigners (true actually refers to the activation server GC mode, false does not mean not GC, but refers to the use of workstation GC mode) I can say whether Microsoft can be a little more "considerate" to point out the real difference between true and false (actually, our own .net research is not thorough enough, ), so that there will not be "doubts" like me, like this foreigner, about the excessive memory usage of core applications running in docker containers.

Our production has been changed to false, of course, true is no problem, but the server memory is "just eat not pull".

After reading the above, have you mastered the method of analyzing the problem of excessive memory utilization of ASP.NET Core applications on Kubernetes? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.