Shulou(Shulou.com)06/03 Report--

How elasticsearch allocates memory:

1. The way to set variables: export ES_HEAP_SIZE=32G, this method is better.

2. Add startup parameters when starting es:-Xmx 32G-Xms 32G. The size of Xmx and Xms should be the same to prevent the program from prone to gc.

The reason why es allocates up to 32 GB of memory:

1. Memory is absolutely important for Elasticsearch to provide faster operations with more memory data. And there is also a big memory consumer-Lucene

Lucene is designed to cache data from the underlying OS into memory. Lucene segments are stored in a single file, these files will not change, so it is good for caching, at the same time, the operating system will also cache these segments for faster access.

The performance of Lucene depends on the interaction with OS, if you allocate all the memory to Elasticsearch, leaving nothing for Lucene, then your full-text search performance will be very poor.

The final standard recommendation is to give 50% of the memory to elasticsearch, the remaining 50% will not be useless, Lucene will quickly gobble up the rest of the memory. Do not exceed 32G

2. No large memory is allocated to Elasticsearch. In fact, jvm will use a memory object pointer compression technology when the memory is less than 32G.

In java, all objects are allocated on the heap, and then a pointer refers to it. The pointer to these objects is usually the size of the word length of CPU, either 32bit or 64bit, depending on your processor, and the pointer points to the exact location of your value.

For 32-bit systems, up to 4G of memory can be used. The 64 system can use more memory. But a 64-bit pointer means more waste because your pointer itself is bigger. Wasting memory doesn't count, and to make matters worse, larger pointers take up more bandwidth when moving data between main memory and buffers.

Java uses a technique called memory pointer compression to solve this problem. Its pointer no longer represents the exact location of the object in memory, but represents the offset. This means that a 32-bit pointer can refer to 4 billion objects instead of 4 billion bytes. In the end, that is, heap memory up to 32 gigabytes of physical memory can also be represented by a pointer to 32bit.

Once you cross that magical 30-32G boundary, the pointer will switch back to the pointer of a normal object, and each object's pointer will get longer, using more CPU memory bandwidth, which means you actually lose more memory. In fact, when the memory reaches 40-50GB, the effective memory is equivalent to 32 gigabytes of memory when using memory object pointer compression technology.

This description means: even if you have enough memory, try not to exceed 32 gigabytes, because it wastes memory, degrades the performance of CPU, and allows GC to cope with large memory.

Swapping is the tomb of performance

This is obvious, but it is necessary to make it clear that swapping memory to disk is fatal to server performance. Come to think of it, a memory operation must be fast.

If memory is swapped to disk, a 100-microsecond operation may become 10 milliseconds, and then think about the sum of 10-microsecond operation delays. It's not hard to see how terrible swapping is for performance.

The best way is to disable swapping completely in your operating system. This temporarily disables:

Sudo swapoff-a

To disable it permanently, you may need to modify the / etc/fstab file

Sed-I'sUnip. Swap.pickswap.Candle / etc/fstab

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.