Shulou(Shulou.com)06/01 Report--

This article mainly introduces how DataWorks uses Shell nodes to call custom resource group execution scripts, which has a certain reference value, and interested friends can refer to it. I hope you can learn a lot after reading this article.

1. Set custom resource groups and configure permissions ECS permissions

1.1 add custom resource groups under the corresponding space according to the document

Https://help.aliyun.com/document_detail/169950.html

1.2 View added custom resource groups

1.3 Log in to the ECS added as a custom resource group, establish the corresponding directory, and set permissions for admin users

The odps_haudong in the directory is the client of MaxCompute

The content of upfile.txt is

Chown-R amdin:admin / Home2 and DataWorks create new Shell nodes and edit scripts

2.1 create a temporary node and execute the following statement

Create table shell_test (key string); insert overwrite table shell_test values ('hadoop'), (' kafka'), ('habse'); select * from shell_test

View the current data display

2.2 establish manual business processes, configure Shell scripts, and add scheduling resource groups to custom resource groups

/ home/export/odps_huadong/bin/odpscmd-e "tunnel upload-overwrite true / home/export/upfile.txt shell_test" 3. Submit the shell node to the production environment and execute it

3.1 submit the business process

3.2 publish tasks to the production environment and execute them manually

Show success in executing the command

Fourth, check whether the command is executed

4.1 return the data development interface to execute the statement to confirm whether the data is overwritten and written

Select * from shell_test

You can see that the data was written successfully

Thank you for reading this article carefully. I hope the article "DataWorks how to use Shell nodes to call custom resource group execution scripts" shared by the editor will be helpful to everyone. At the same time, I also hope that you will support and follow the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.