Shulou(Shulou.com)06/02 Report--

This article mainly explains "the deployment and setup of Flink on Yarn". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "the deployment and setup of Flink on Yarn".

Execute ansible script to deploy CDH and Flink (ansible computer)

Enter the ~ / playbooks directory of the ansible computer. After the preparatory work of the previous article, the following should be shown in this directory:

Check whether the ansible remote operation of the CDH server is normal, and execute the command ansible deskmini-a "free-m". Normally, the memory information of the CDH server is displayed, as shown below:

Execute the command to start deployment: ansible-playbook cm6-cdh6-flink1.7-single-install.yml

The whole deployment process involves time-consuming operations such as online installation and file transfer, so please wait patiently (about half an hour). If you exit in error during deployment (such as network problems), you only need to repeat the above command. Ansible ensures that the operation is idempotent.

The successful deployment is shown in the following figure:

Restart the CDH server

Because the settings of selinux and swap have been modified, it is necessary to restart the operating system to take effect, so restart the CDH server

Execute the ansible script to start the CDH service (ansible computer)

Wait for the CDH server to restart successfully

Log in to the ansible computer and enter the ~ / playbooks directory

Execute the script to initialize the database and start CDH: ansible-playbook cdh-single-start.yml

After startup, the following information is output:

Ssh logs in to the CDH server and executes this command to observe the startup of the CDH service: tail-f / var/log/cloudera-scm-server/cloudera-scm-server.log. When you see the contents in the red box below, the startup is complete, and you can log in with the browser:

Settings (browser operation)

Now that the CDH service is started, you can operate it through the browser:

Browser access: http://192.168.50.134:7180, as shown below. The account password is all admin:

Next all the way, and select the 60-day trial version on the select version page:

Select the host page to see the CDH server (deskmini):

On the page where you select the CDH version, select 5.16.2-1 in the red box below:

Go to the page where Parcel is installed. Due to the offline parcle package uploaded in advance, the download progress has instantly become 100%. At this time, wait for the completion of allocation, decompression and activation:

Next, there are some recommended actions. Select the red box below to skip:

Next is the page for selecting services. I selected a custom service, and then selected HDFS, YARN and Zookeeper to meet the needs of running Flink:

On the page of selecting the host, select the CDH server:

Next is the page of database settings. The content you enter must be consistent with the following figure, that is, the database with hostname localhost,Activity Monitor, user, database with password of amon,Reports Manager, user and password are all rman. These contents are fixed in the ansible script, and the contents must be consistent here:

On the page of setting parameters, please set them according to the actual situation of your hard disk. There is plenty of space in my / home directory, so the storage location is changed to / home directory:

Wait for the service to start:

The startup of each service is complete:

YARN Settin

The default YARN parameters are very conservative, and you need to make some settings to perform Flink tasks smoothly:

Click the red box in the following figure to go to the YARN management page:

As shown in the following figure, check the value of the parameter yarn.nodemanager.resource.cpu-vcores, which must be greater than 1, otherwise YARN will not allocate resources to execute the task after submitting the Flink task. (if your CDH server is a virtual machine, this parameter will be set to 1 when the CPU has only a single core. The solution is to increase the number of virtual machine CPU cores first, and then modify this parameter):

Yarn.scheduler.minimum-allocation-mb: the minimum memory that can be applied for by a single container. I set it to 1G here.

Yarn.scheduler.maximum-allocation-mb: the maximum memory that can be applied for by a single container. I set it to 8G here.

Yarn.nodemanager.resource.memory-mb: the maximum available memory of the node, which is set to 8G here.

The values of the above three parameters are based on the background that my CDH server has 32 GB of memory. Please adjust them according to your own hardware resources.

Restart the YARN service after setup, as shown in the following figure:

At this point, the deployment and setup are complete, and the environment for Flink on Yarn is available.

Thank you for your reading, the above is the content of "deployment and setup of Flink on Yarn". After the study of this article, I believe you have a deeper understanding of the deployment and setup of Flink on Yarn, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.