How to realize offline data extraction by Datax in MySQL
This article will explain in detail how to extract offline data from Datax in MySQL. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.
Datax download address: https://github.com/alibaba/DataX
Datax MySQL read: https://github.com/alibaba/DataX/blob/master/mysqlreader/doc/mysqlreader.md
Datax MySQL writes: https://github.com/alibaba/DataX/blob/master/mysqlwriter/doc/mysqlwriter.md
1. Download and install Datax software (jdk version 1.8 or above must be installed)
[root@localhost ~] # tar xvf jdk-8u65-linux-x64.tar.gz
[root@localhost ~] # mv jdk1.8.0_151 / usr/local/jdk1.8.0_151
[root@localhost ~] # vim / etc/profile
Export PATH=$PATH:/usr/local/jdk1.8.0_151/bin
[root@localhost ~] # vim / etc/ld.so.conf.d/mysql-x86_64.conf
/ usr/local/jdk1.8.0_151/lib
[root@localhost ~] # tar xvf datax.tar.gz
[root@localhost ~] # cd datax/job/
two。 Edit configuration file
[root@localhost job] # vim job.json
{
"job": {
"setting": {
"speed": {
"channel": 5
}
}
"content": [
{
"reader": {
"name": "mysqlreader"
"parameter": {
"username": "taf"
"password": "taf2015"
Column: ["*"]
"connection": [
{
"table": ["T2"]
"jdbcUrl": ["jdbc:mysql://172.16.8.160:3306/test?useUnicode=true&characterEncoding=utf8"]
}
]
}
}
"writer": {
"name": "mysqlwriter"
"parameter": {
"writeMode": "update"
"username": "wangying"
"password": "wangying"
Column: ["*"]
"connection": [
{
"jdbcUrl": "jdbc:mysql://172.16.8.93:3306/db_stktag"
"table": ["T2"]
}
]
}
}
}
]
}
}
[root@localhost job] #
The results of the tables on both sides of the 3.mysql must arrive, and the data must be extracted using python2.
[root@localhost job] # python2 / root/datax/bin/datax.py job.json
DataX (DATAX-OPENSOURCE-3.0), From Alibaba!
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2019-01-29 11 VMInfo# operatingSystem class 1815 50.569 [main] INFO VMInfo-VMInfo# operatingSystem class = > sun.management.OperatingSystemImpl
2019-01-29 11 the machine info 18 18 15 [main] INFO Engine-the machine info = >
OsInfo:Oracle Corporation 1.8 25.151-b12
JvmInfo:Linux amd64 3.10.0-123.el7.x86_64
Cpu num:8
TotalPhysicalMemory:-0.00G
FreePhysicalMemory:-0.00G
MaxFileDescriptorCount:-1
CurrentOpenFileDescriptorCount:-1
. . .
. . .
Mission start time: 2019-01-29 11:18:50
End of mission: 2019-01-29 11:19:04
Total time spent on the task: 13s
Average task traffic: 1B/s
Record write speed: 0rec/s
Total number of readout records: 7
Total number of read and write failures: 0
[root@localhost job] #
This is the end of this article on "how to achieve offline data extraction in Datax in MySQL". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.
