Shulou(Shulou.com)06/01 Report--

This article introduces the knowledge of "what is the new memory model of Flink". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Flink New memory Model

There are various memory overflow problems when upgrading Flink from 1.7.2 to 1.10.0. Flink1.10 has proposed a new memory model, and it is necessary to use the new version of Flink to understand the new memory model. For the old flink, we usually only need to configure taskmanager.heap.size, for standalone cluster, this configuration is only configured with heap memory, while the external heap memory taskmanager.sh is allocated to a considerable number.

TM_MAX_OFFHEAP_SIZE= "8388607T" export JVM_ARGS= "${JVM_ARGS}-Xms$ {TM_HEAP_SIZE} M-Xmx$ {TM_HEAP_SIZE} M-XX:MaxDirectMemorySize=$ {TM_MAX_OFFHEAP_SIZE}"

For yarn cluster, this configuration item represents the memory of the entire container. Therefore, it is indeed necessary to collate and propose a new memory specification.

TaskManager

TaskManager total memory can be configured in two ways: total process memory and total flink memory, where total process memory = total flink memory + jvm runs on its own memory, such as metaspace. Total process memory is suitable for use in container environment, and the corresponding configuration item is taskmanager.memory.process.size, while total flink memory is suitable for use under standalone, and the corresponding configuration item is taskmanager.memory.flink.size.

Total flink memory = heap + off-heap. Heap is called heap memory, while off-heap here refers to memory that is not heap memory, and here it is not called out-of-heap or non-heap, to draw a line from these vague concepts.

Heap = task heap + framework heap, that is, heap memory used by user code and heap memory used by the flink framework itself. The jvm parameters for this part of memory end up being-Xmx and-Xms, so in fact task heap and framework heap are not isolated.

Off-heap=managed memory + direct memory. Managed memory is created by Flink with the Unsafe class and is not controlled by JVM. This part of the memory in the stream job can be used for the status back-end, such as rocksdb memory; the batch job memory can be used for sorting, caching intermediate results, and so on. Managed memory is the memory shared equally by slot (this is actually the only part of slot's memory isolation), so it is isolated by slot. Direct memory is what we usually call out-of-heap memory: java.nio.DirectByteBuffer, which is also created by the Unsafe class and is not supposed to be controlled by JVM, but-XX:MaxDirectMemorySize is used in JDK code to control its size. Size is limited in java.nio.Bits#tryReserveMemory. The JVM parameter for direct memory is-XX:MaxDirectMemorySize.

Private static boolean tryReserveMemory (long size, int cap) {/ /-XX:MaxDirectMemorySize limits the total capacity rather than the / / actual memory usage, which will differ when buffers are page / / aligned. Long totalCap; while (cap

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.