How to call Jni Program in Spark

This article is about how to call the Jni program in Spark, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

Handle the resource destruction problem in the 1.jni call.

From saprk on yarn mode, every time a task is submitted, a group of processes are started and destroyed after the task is completed. Therefore, as long as there is only one copy of global resources, memory leaks and system crashes can be guaranteed.

Static objects can be shared among multiple methods in a 2.jni program.

The 3.corsegenbackend process is created when the task is submitted and destroyed after the task execution is completed. The number of this process is set through-- num-executors in yarn mode.

4. On spark, tasks may run on multiple processes. If the jni program contains C++ static objects, how to ensure that static objects are initialized and loaded only once?

Solution:

1. Written in mapparition, but being called too many times is a disadvantage.

two。 Writing in the static block ensures that each process calls the method only once.

The mappartitions method in 5.spark may be map only once if df.first () is called once in the background.

The above is how to call the Jni program in Spark. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.