Shulou(Shulou.com)05/31 Report--

This article mainly explains "what is the implementation method of Datax offline data extraction", interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Next, let the editor take you to learn "what is the implementation method of Datax offline data extraction"?

1. Download and install Datax software (jdk version 1.8 or above must be installed)

[root@localhost ~] # tar xvf jdk-8u65-linux-x64.tar.gz

[root@localhost ~] # mv jdk1.8.0_151 / usr/local/jdk1.8.0_151

[root@localhost ~] # vim / etc/profile

Export PATH=$PATH:/usr/local/jdk1.8.0_151/bin

[root@localhost ~] # vim / etc/ld.so.conf.d/mysql-x86_64.conf

/ usr/local/jdk1.8.0_151/lib

[root@localhost ~] # tar xvf datax.tar.gz

[root@localhost ~] # cd datax/job/

two。 Edit the configuration file (first synchronize the table structure with kettle)

[root@localhost job] # vim job1.json

{"job": {"setting": {"speed": {"channel": 5}}, "content": [{"reader": {"name": "oraclereader" "parameter": {"username": "upcenter", "password": "upcenter", "column": ["*"] "connection": [{"table": ["STOCK_CONC"] "jdbcUrl": ["jdbc:oracle:thin:@192.168.7.7:1521:upqc"]}, "writer": {"name": "mysqlwriter" "parameter": {"writeMode": "update", "username": "wangying", "password": "wangying", "column": ["*"] "connection": [{"jdbcUrl": "jdbc:mysql://172.16.8.93:3306/db_stktag" "table": ["T3"]}

[root@localhost job] #

3. Data extraction

[root@localhost job] # python2 / root/datax/bin/datax.py job1.json

DataX (DATAX-OPENSOURCE-3.0), From Alibaba!

Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.

2019-01-29 14 VMInfo# operatingSystem class 23 15 15 [main] INFO VMInfo-VMInfo# operatingSystem class = > sun.management.OperatingSystemImpl

2019-01-29 14 14 the machine info 23 53. 186 [main] INFO Engine-the machine info = >

OsInfo: Oracle Corporation 1.8 25.151-b12

JvmInfo: Linux amd64 3.10.0-123.el7.x86_64

Cpu num: 8

TotalPhysicalMemory:-0.00G

FreePhysicalMemory:-0.00G

MaxFileDescriptorCount:-1

CurrentOpenFileDescriptorCount:-1

. . .

. . .

2019-01-29 14 2404.012 [job-0] INFO JobContainer-

Mission start time: 2019-01-29 14:23:53

End of mission: 2019-01-29 14:24:04

Total task time: 10s

Average task traffic: 176.48KB/s

Record write speed: 4047rec/s

Total number of read records: 40475

Total number of read and write failures: 0

[root@localhost job] #

4. Validate data

Mysql > select count (1) from T3

+-+

| | count (1) | |

+-+

| | 40475 |

+-+

1 row in set (0.03 sec)

Mysql >

At this point, I believe you have a deeper understanding of "what is the implementation method of Datax offline data extraction". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.