Shulou(Shulou.com)06/01 Report--

When I first started using mongodb, I didn't pay much attention to the memory use of mongodb, but through checking the data, I found that mongodb takes up a huge amount of memory. In the local test server, the 8G memory is actually occupied 45%. Sweat.

This article will analyze the specific use of memory in mongodb and the solution to the problem that mongodb takes up a lot of memory in the production environment.

First, take a look at the top command result of a MongoDB server.

Shell > top-p $(pidof mongod)

Mem: 32872124k total, 30065320k used, 2806804k free, 245020k buffers

Swap: 2097144k total, 100k used, 2097044k free, 26482048k cached

VIRT RES SHR MEM

1892g 21g 21g 69.6

Or top first, and then shift+m sorts the current entry according to how much memory is used. See how much memory your mongodb can take up.

Let's take a look at how linux manages memory:

In Linux (similar to other systems), there is physical memory and virtual memory in memory. There is no need to explain what physical memory is. Virtual memory is actually an abstraction of physical memory. In most cases, for convenience reasons, the program accesses the virtual memory address, and then the operating system translates it into physical memory address.

Many people will confuse virtual memory with Swap. In fact, Swap is just a technology derived from virtual memory: once the operating system runs out of physical memory, in order to free up memory space for new content, it will put the contents of the current physical memory into the swap partition and retrieve it later. It should be noted that the use of Swap may bring performance problems, so occasionally there is no need to be nervous. Unfortunately, data exchanges between physical memory and swap partitions occur frequently, which is called Swap bumps. Once this happens, it is necessary to know what causes it. If it is out of memory, it can be solved by adding memory. However, sometimes this problem may occur even if there is enough memory. For example, this may happen in MySQL. The solution is to limit the use of Swap:

Shell > sysctl-w vm.swappiness=0

The most common way to check memory is the free command:

Shell > free-m

Total used free shared buffers cached

Mem: 32101 29377 2723 0 239 25880

-/ + buffers/cache: 3258 28842

Swap: 2047 0 2047

Beginners see that the used column value is too large and the free column value is too small, so they tend to think that memory is running out. In fact, this is not the case, because whenever we operate files, Linux will cache the files in memory as much as possible, so that the next time we visit, we can get the results directly from memory, so the value in the cached column is very large, but don't worry, this part of memory is recyclable, the operating system will eliminate cold data according to LRU algorithm. In addition to cached, there is a buffers, which is similar to cached and is recyclable, but its focus is on alleviating blockages caused by inconsistent operating speeds of different devices, which will not be explained here.

Knowing the principle, we can infer that the memory available to the system is free + buffers + cached:

Shell > echo "2723 + 239 + 25880" | bc-l

28842

The memory actually used by the system is used-buffers-cached:

Shell > echo "29377-239-25880" | bc-l

3258

In addition to the free command, you can also use the sar command:

Shell > sar-r

Kbmemfree kbmemused memused kbbuffers kbcached

3224392 29647732 90.19 246116 26070160

3116324 29755800 90.52 245992 26157372

2959520 29912604 91.00 245556 26316396

2792248 30079876 91.51 245680 26485672

2718260 30153864 91.73 245684 26563540

Shell > sar-W

Pswpin/s pswpout/s

0.00 0.00

0.00 0.00

0.00 0.00

0.00 0.00

0.00 0.00

I hope you are not frightened by% memused. If you are unfortunately right, please refer to the explanation of the free command.

Then let's analyze how mongodb uses memory:

Currently, MongoDB uses a memory-mapped storage engine, which converts disk IO operations into memory operations. If it is a read operation, the data in memory acts as a cache. If it is a write operation, memory can also convert random write operations into sequential write operations, which can greatly improve performance. MongoDB does not interfere with memory management, but leaves these tasks to the virtual cache manager of the operating system. The advantage is that it simplifies the work of MongoDB, but the disadvantage is that you have no way to easily control how much memory MongoDB takes up. In fact, MongoDB will take up all available memory, so it is best not to put other services and MongoDB together.

Sometimes, even if MongoDB uses a 64-bit operating system, it may encounter the infamous OOM problem, which is mostly due to the limit on the size of virtual memory. You can view the current value like this:

Shell > ulimit-a | grep 'virtual'

Most operating systems set it to unlimited by default, and if your operating system is not, you can modify it like this:

Shell > ulimit-v unlimited

Note, however, that the use of ulimit is contextual and is best placed in the startup script of MongoDB.

Sometimes, for some reason, you may want to release the memory occupied by MongoDB, but as mentioned earlier, memory management is controlled by the virtual memory manager, so usually you can only release memory by restarting the service. You must be disdainful of this method, but fortunately you can use MongoDB's built-in closeAllDatabases command to achieve this goal:

Mongo > use admin

Mongo > db.runCommand ({closeAllDatabases:1})

In addition, the cache can also be freed by adjusting the kernel parameter drop_caches:

Shell > sysctl-w vm.drop_caches=1

Normally, you can monitor the memory usage of MongoDB through the mongo command line, as follows:

Mongo > db.serverStatus () .mem:

{

"resident": 22346

"virtual": 1938524

"mapped": 962283

}

You can also monitor the memory usage of MongoDB through the mongostat command, as shown below:

Shell > mongostat

Mapped vsize res faults

940g 1893g 21.9g 0

940g 1893g 21.9g 0

940g 1893g 21.9g 0

940g 1893g 21.9g 0

940g 1893g 21.9g 0

The meaning of memory-related fields is:

Mapped: data size mapped to memory

Visze: amount of virtual memory consumed

Res: the amount of memory actually used

Note: if the operation cannot be completed in memory, the value of the faults column will not be 0, and there may be performance problems depending on the size.

In the above results, vsize is twice the size of mapped, and mapped is equal to the size of the data file, so vsize is twice the size of the data file. This is because in this case, MongoDB turns on journal and needs to map the data file in memory one more time. If you turn off journal, vsize and mapped are roughly the same.

If you want to verify this, you can observe the file mapping through the pmap command after turning journal on or off:

Shell > pmap $(pidof mongod)

How much memory is appropriate for MongoDB? Generally speaking, × ×, if you want to be exact, it depends on your data and the size of the index. It is best if you can hold all the data and index in memory, but in many cases, the data will be larger than memory, such as the MongoDB instance mentioned in this article:

Mongo > db.stats ()

{

"dataSize": 1004862191980

"indexSize": 1335929664

}

In this example, the index is only more than 1G, and the memory can be installed completely, while the data file reaches 1T. It is estimated that it is difficult to find such a large memory. At this time, it is guaranteed that the memory can hold hot data. As for the number of hot data, this is a question of proportion, depending on the specific application. This makes the memory size clear: memory > index + hot data.

Based on the above analysis, we can draw several conclusions:

1. Mongodb uses the memory manager of the operating system to manage memory directly. The operating system uses LRU algorithm to eliminate cold data.

2. Mongodb can clean up the memory cache of mongodb by restarting the service, adjusting kernel parameters and mongodb internal syntax. The possible problem is that all of these cleanup methods are cleaned, so that mongodb's in-memory cache is invalidated.

3. Mongodb's use of memory can be monitored, and the data should be monitored regularly in the production environment.

4. Mongodb occupies memory so that it can be deployed separately from other memory-consuming businesses as far as possible, such as memcahe,sphinx,mysql.

5. The swap partition swap in the operating system, if operated frequently, will seriously reduce the efficiency of the system. To solve the problem, you can disable swapping, increase memory and do distribution.

6. The host where the mongodb is located in the production environment should have as much memory as possible.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.