Shulou(Shulou.com)05/31 Report--

This article is to share with you about what has changed from flink1.9.1 to flink1.12.1. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Memory configuration

In version 1.10 and _ 1.11 _, Flink made significant changes to the memory configuration methods of TaskManager and JobManager, respectively. Some of the configuration parameters have been removed or semantically changed.

Jobmanager.heap.size

Jobmanager.heap.mb

Although these two parameters are named "Heap", they actually correspond exactly to JVM heap memory only in stand-alone deployment mode. In containerized deployment mode (Kubernetes and Yarn), the memory they specify also contains other parts of out-of-heap memory. The actual size of the JVM heap space is the amount left after the size specified by the parameter minus the container Cut-Off memory. The container excision memory has been completely removed in version 1.11 and above.

The above two parameters did not previously take effect on the Mesos deployment model. Flink does not set any JVM memory parameters when it starts the JobManager process on Mesos. Starting with version 1.11, Flink will set these parameters in the same way as the stand-alone deployment model.

These two configuration parameters are currently deprecated. If the above deprecated parameters are configured, but no corresponding new configuration parameters are configured, they will correspond to the new configuration parameters according to the following rules.

Independent deployment mode (Standalone Deployment), Mesos deployment mode: JVM heap memory (jobmanager.memory.heap.size)

Under containerized deployment mode (Containerized Deployement) (Kubernetes, Yarn): total process memory (jobmanager.memory.process.size)

It is recommended that you replace the enabled configuration parameters with new configuration parameters as soon as possible, and they may be completely removed in future releases.

If only Flink total memory or process total memory is configured, then JVM heap memory will be the rest of the total memory minus the rest of the memory. Please refer to how to configure total memory. Alternatively, you can directly specify JVM heap memory by configuring jobmanager.memory.heap.size.

The new memory configuration requires the user to specify at least one of the following configuration parameters (or combinations of parameters), otherwise Flink will not start.

TaskManager:JobManager:taskmanager.memory.flink.sizejobmanager.memory.flink.sizetaskmanager.memory.process.sizejobmanager.memory.process.sizetaskmanager.memory.task.heap.size and

Taskmanager.memory.managed.sizejobmanager.memory.heap.size

In the original memory configuration method, the configuration parameters used to specify JVM heap memory are:

Modify configuration

Jobmanager.heap.size: 1024m-- > jobmanager.memory.process.size: 1024m

Taskmanager.heap.size: 2048m-- > taskmanager.memory.flink.size: 2048m

Code modification

Version upgrade of flink: 1.9.1-- > 1.12.1

Kafka connector version upgrade: flink-connector-kafka-0.11_$ {scala.binary.version}-- > flink-connector-kafka_2.11

Waste removal method: env.getConfig (). DisableSysoutLogging ();-- > / / env.getConfig (). DisableSysoutLogging ()

Consumer: FlinkKafkaConsumer011 consumer = new FlinkKafkaConsumer011 (--> FlinkKafkaConsumer consumer = new FlinkKafkaConsumer (

Thank you for reading! This is the end of this article on "what is the change from flink1.9.1 to flink1.12.1". I hope the above content can be helpful to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.