Shulou(Shulou.com)06/01 Report--

This article mainly introduces how to compile and install datax, with a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

1.centos system maven installation reference https://www.cnblogs.com/laoayi/p/12867990.html

1. Download official website address: http://maven.apache.org/download.cgi

Curl-O https://mirror.bit.edu.cn/apache/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz

two。 Decompression

Tar-zxvf apache-maven-3.6.3-bin.tar.gz

3. Modify environment variabl

Vim / etc/profile

Export MAVEN_HOME=/opt/maven/apache-maven-3.6.3export PATH=$MAVEN_HOME/bin:$PATH

Modify maven image address

Vim / opt/maven/apache-maven-3.6.3/conf/settings.xml

Add the following code:

Alimaven aliyun maven http://maven.aliyun.com/nexus/content/repositories/central/ central

Source / etc/profile / / using environment variables takes effect

4. Check to see if the installation was successful

Mvn-version

[root@ambari-03 maven] # mvn-version

Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)

Maven home: / opt/maven/apache-maven-3.6.3

Java version: 1.8.0_181, vendor: Oracle Corporation, runtime: / usr/local/java/jdk/jdk1.8.0_181/jre

Default locale: en_US, platform encoding: UTF-8

OS name: "linux", version: "3.10.0-957.el7.x86_64", arch: "amd64", family: "unix"

two。 Download the datax source code

Install git

Yum install curl-devel expat-devel gettext-devel openssl-devel zlib-devel

Git-- version / / check whether the installation is successful

Git clone git@github.com:alibaba/DataX.git / / failed to download datax source code permission requires login user

Download the source code using another curl method

Curl-O https://gitee.com/jarynpl/DataX/repository/archive/master.zip

Unzip master.zip / / decompress

3. Packaged through maven:

$cd {DataX_source_code_home}

$mvn-U clean package assembly:assembly-Dmaven.test.skip=true

Problem handling of compilation failure: URL https://github.com/alibaba/datax/issues/676

[ERROR] Failed to execute goal on project clickhousewriter: Could not resolve dependencies for project com.alibaba.datax:clickhousewriter:jar:0.0.1-SNAPSHOT: Could not find artifact com.alibaba.datax:simulator:jar:0.0.1-SNAPSHOT in alimaven (http://maven.aliyun.com/nexus/content/repositories/central/)-> [Help 1])

There is a problem with Clickhousewriter. You can comment out the dependency in the module pom.xml file (the dependency is used as a test, and there are no unit tests in the code)

Com.alibaba.datax simulator ${datax-project-version} test

In addition, the 15 lines of ClickHouseType introduced in the ClickhouseWriter.java need to be removed, and the class is not recognized.

Import ru.yandex.clickhouse.ClickHouseTuple

Then try to compile successfully.

The package is successful, and the log is shown as follows:

[INFO] gdbwriter.. SUCCESS [2.537 s]

[INFO] cassandrawriter.. SUCCESS [1.323 s]

[INFO] clickhousewriter.. SUCCESS [1.416 s]

[INFO] hbase20xsqlreader.. SUCCESS [5.429 s]

[INFO] hbase20xsqlwriter.. SUCCESS [1.836 s]

[INFO]

[INFO] BUILD SUCCESS

[INFO]

[INFO] Total time: 04:07 min

[INFO] Finished at: 2021-01-22T19:24:49+08:00

[INFO]

The successfully packaged DataX package is located at {DataX_source_code_home} / target/datax/datax/

The structure is as follows:

$cd {DataX_source_code_home} $ls. / target/datax/datax/bin conf job lib log log_perf plugin script tmp Thank you for reading this article carefully. I hope the article "how to compile and install datax" shared by the editor will be helpful to you, and I also hope you will support it. Pay attention to the industry information channel, more related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.