Shulou(Shulou.com)06/01 Report--

This article mainly introduces the use of Flink Native Kubernetes, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

Review Flink Kubernetes

Flink Kubernetes and Flink Native Kubernetes are different overview, so let's review Flink Kubernetes:

As shown in the following figure, from version 1.2 to the latest 1.10 Flink, officials have given a plan to deploy and run Flink on Kubernetes:

There are two ways to run flink:session cluster and job cluster on kubernetes, where session cluster is a set of services that can submit multiple tasks, while job cluster is a set of services that only correspond to one task

The following figure shows a typical session cluster deployment operation. It can be seen that the key is to prepare yaml files for service, deployment and other resources, and then create them with the kubectl command:

About Flink Native Kubernetes

Comparing the official documents of version 1.9 and 1.10, as shown in the following figure and red and blue boxes, you can see that Flink Native Kubernetes is a new feature of version 1.10:

See how Native Kubernetes works, as shown in the following figure. The command to create session cluster comes from the Flink installation package:

More interestingly, the command to submit the task also comes from the Flink installation package, that is, we usually use the flink run command to submit the task, as shown below:

Combined with the official flow chart of submission and deployment, it is even clearer: Flink Master is deployed on kubernetes, and Flink Client submits session cluster and job requests:

The difference between Flink Kubernetes and Flink Native Kubernetes

At this point, you can summarize the difference between Flink Kubernetes and Flink Native Kubernetes:

Flink Kubernetes appeared for the first time since version 1.2, and Flink Native Kubernetes appeared for the first time since version 1.10.

Flink Kubernetes puts processes such as JobManager and TaskManager in containers, manages and runs in kubernetes, which is the same reason that we make java applications into docker images and then run on kubernetes. They all operate on kubernetes with kubectl.

Flink Native Kubernetes is a tool in the Flink installation package that can send requests to the Api Server of kubernetes, such as creating a Flink Master, and can communicate with Flink Master to submit tasks. We just need to use the tools in the Flink installation package without performing kubectl operations on kubernetes.

Inadequacies of Flink Native Kubernetes in Flink-1.10 version

Flink Native Kubernetes is only Beta version, belongs to the experimental nature (official words: still experimental), do not use in the production environment!

Only session cluster mode (one resident session performs multiple tasks) is supported, but Job clusters mode (one session for each task) is not supported.

Although it has not yet entered the Release stage, this mode of operation is still very friendly to developers who are not familiar with kubernetes. Let's experience it through actual combat.

Official request

In order to experience Native Kubernetes,flink, officials have put forward the following prerequisites:

Kubernetes version is not less than 1.9

The DNS of the kubernetes environment is normal

KubeConfig file, and this file has the right to add, delete, modify and query pod and service resources (the kubectl command has the right to operate on pod and service, but also because it uses the corresponding KubeConfig file), this file is generally in the kubernetes environment, full path: ~ / .kube/config

Pod executes as service account, and this service account has granted pod add and delete permissions through RBAC.

The first two points need you to ensure that you have met the requirements. The third and fourth points do not need to be concerned about now. There are detailed steps to complete them later.

Actual combat environment information

The actual combat environment is shown below, a kubernetes environment (version 1.15.3) and a CentOS7 computer on which flink-1.10 has been deployed (the deployment here means to decompress the installation package without starting any services): ready to start the actual combat ~

Brief introduction of actual combat content

This actual combat is to create a session cluster in the kubernetes environment, and then submit the task to this sessionc cluster to run. Different from the official tutorial, this actual combat uses custom namespace and service account. After all, the production environment generally does not allow the use of default as namespace and service account.

Actual combat

The root account is used when operating on the CetnOS7 computer

On the kubernetes node, make sure you have the right to execute the kubectl command to add, delete, modify and query pod and service, and copy the file ~ / .kube / config to the ~ / .kube / directory of the CentOS7 computer.

On the node of kubernetes, execute the following command to create a namespace named flink-session-cluster:

Kubectl create namespace flink-session-cluster

Execute the following command to create a serviceaccount named flink:

Kubectl create serviceaccount flink- n flink-session-cluster

Execute the following command to bind the serviceaccount to the role:

Kubectl create clusterrolebinding flink-role-binding-flink\-clusterrole=edit\-serviceaccount=flink-session-cluster:flink

SSH logs in to the CentOS7 computer where flink is deployed, and execute the following command under the flink directory to create a session cluster named session001, where the-Dkubernetes.namespace parameter specifies namespace, and also specifies that an TaskManager instance uses a CPU resource, 4G memory and 6 slot:

. / bin/kubernetes-session.sh\-Dkubernetes.namespace=flink-session-cluster\-Dkubernetes.jobmanager.service-account=flink\-Dkubernetes.cluster-id=session001\-Dtaskmanager.memory.process.size=8192m\-Dkubernetes.taskmanager.cpu=1\-Dtaskmanager.numberOfTaskSlots=4\-Dresourcemanager.taskmanager-timeout=3600000

As shown in the following figure, the console prompts that the creation is successful, and the red box indicates that the access address of flink web UI is http://192.168.50.135:31753:

It takes time to download the image and launch the container. You can use the kubectl get and kubectl describe commands to observe the status of the corresponding deployment and pod:

9. Visit flink web after pod starts successfully, as shown in the following figure. No TaskManager has been created at this time, so Slot is zero: 10. Go back to the CentOS7 computer and execute the following command under the flink directory to submit the official WindowJoin task to session cluster:

. / bin/flink run-d\-e kubernetes-session\-Dkubernetes.namespace=flink-session-cluster\-Dkubernetes.cluster-id=session001\ examples/streaming/WindowJoin.jar

The console prompts you to submit the task successfully:

The page will also synchronously show that one TaskManager has been added, corresponding to six slot, and one has already been used:

Submit the same task 5 times in a row, and use up the slot of this TaskManager:

When you submit another task at this time, you should add a TaskManager, but the number of TaskManager on the page is still 1, as shown in the following figure, and the red box shows that the new task is not working properly:

15. Check the pod situation in the kubernetes environment. As shown in the red box below, there is a newly created pod status of Pending. It seems that the seventh task cannot be executed because the newly created pod does not work properly: 16. If you look at the event notification for this namespace, as shown in the red box below, the pod named session001-taskmanager-1-2 has a notification message: pod creation failed due to insufficient CPU resources: 17. Too poor to have the money to configure a kubernetes environment, and can't even put together a core CPU:

18. I can't find any extra CPU resources at 01:30. The only thing I can do is to lower the CPU requirements of TaskManager. What I just configured is a TaskManager that uses one core CPU. I intend to reduce it by half, that is, 0.5 cores, so that it will be enough for two TaskManager.

19. You may wonder: why is there a configuration like 0. 5 CPU? This is related to the resource limit of kubernetes. The CPU limit granularity of kubernetes to pod is 1/1000 CPU, that is, in kubernetes, configuring 1000 units of CPU means using 1 core, while we configure 0.5 cores, only 500 units are configured (so I can be poorer.)

20. The next step is to stop the current session cluster, and then create a new one. The value of the parameter-Dkubernetes.taskmanager.cpu is changed from 1 to 0.5.

21. On the CentOS7 computer, execute the following command to stop session cluster and release all resources:

Echo 'stop' |\. / bin/kubernetes-session.sh\-Dkubernetes.namespace=flink-session-cluster\-Dkubernetes.cluster-id=session001\-Dexecution.attached=true

The console prompt operation was successful:

Wait a minute or so, and then check the pod, and find that they are all gone:

From the flink directory of the CentOS7 computer, execute the following command, and the only change compared to the previous one is the value of the-Dkubernetes.taskmanager.cpu parameter:

. / bin/kubernetes-session.sh\-Dkubernetes.namespace=flink-session-cluster\-Dkubernetes.jobmanager.service-account=flink\-Dkubernetes.cluster-id=session001\-Dtaskmanager.memory.process.size=4096m\-Dkubernetes.taskmanager.cpu=0.5\-Dtaskmanager.numberOfTaskSlots=6\-Dresourcemanager.taskmanager-timeout=3600000

Get the new flink web UI port value from the console prompt, and then visit the web page and find that the startup is successful:

Submit 7 tasks as before, this time smoothly. After submitting the seventh task, the new TaskManager is created successfully, and all 7 tasks are executed successfully:

Use the kubectl describe pod command to check the pod of TaskManager. As shown in the red box below, it can be seen that the CPU consumption of the pod is 500U, which is in line with the previous conjecture: again, reducing the amount of CPU means that the CPU execution time obtained by the processes in the pod is reduced, which will lead to slower task execution, so this method is not desirable. The right way of thinking is to make sure that the hardware resources meet the business needs (it is rare for me to be too poor to put together a single core of CPU.)

Clean up resources

If you have completed the Flink Native Kubernetes experience and want to completely clean up all previous resources, follow these steps:

Click Cancel Job to stop the running task on the web page, as shown in the red box below:

Stop session cluster on the CentOS7 computer:

Echo 'stop' |\. / bin/kubernetes-session.sh\-Dkubernetes.namespace=flink-session-cluster\-Dkubernetes.cluster-id=session001\-Dexecution.attached=true

Clean up service, clusterrolebinding, serviceaccount, namespace on the kubernetes node:

Kubectl delete service session001-n flink-session-clusterkubectl delete clusterrolebinding flink-role-binding-flinkkubectl delete serviceaccount flink- n flink-session-clusterkubectl delete namespace flink-session-cluster

All cluster session-related ConfigMap, Service, Deployment, Pod and other resources are associated with service through kubernetes's ownerReferences configuration, so once service is deleted, other resources are automatically cleaned up without processing.

Thank you for reading this article carefully. I hope the article "what's the use of Flink Native Kubernetes" shared by the editor will be helpful to everyone? at the same time, I also hope that you will support and pay attention to the industry information channel, and more related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.