Shulou(Shulou.com)06/01 Report--

1. Software version notes:

Goldengate 12c 12.2.0.1 forOracle (source side)

Goldengate 12c 12.2.0.1 for Bigdata (destination side)

Oracle 11g 11.2.0.4.0 (Note oracle database must be later than 11.2.0.4, previous oracle database versions do not support ogg 12c)

HBase 1.1.2

Java 1.8.0_91

2. Machine IP description

Source IP address (192.168.45.176) install Oracle 11g 11.2.0.4, Goldengate 12c

Destination IP address (192.168.45.175), installation hbase,Goldengate 12c, software installation

Hbase has been installed using ambari, the installation process is not detailed, in this main description of the installation directory of each software, as well as the port used.

The Oracle 12c for Oracle (192.168.45.176) installation directory is: / home/oracle/oggdb

The installation directory for Oracle 12c for bigdata (192.168.45.175) is: / ogg

Hbase installation directory: / usr/hdp/2.4.2.0-258/hbase

3. Configuration of source side (192.168.45.176)

# connecting to the database

Sqlplus / as sysdba

SQL > shutdownimmeidate

SQL > startup mount

# enable Archive Log

SQL > alter database archivelog

SQL > archive log list

# enable database-level logs

SQL > alter database force logging

# enable goldengate replication (Note: this parameter must be enabled for databases after oracle 11.2.0.4 to support this parameter)

SQL > alter system set enable_goldengate_replication=true scope=both

# close the garbage Recycle Bin

SQL > alter system set recyclebin=off scope=spfile

# Open the database

SQL > alter databaseopen

# create an ogg administrative user

SQL > create user oggidentified by ogg account unlock

SQL > grant execute on utl_file to ogg

SQL > grant restricted session to ogg

SQL > grant dba to ogg

Enter the source side (192.168.45.176) OGG directory and initialize the configuration data.

# cd / home/oracle/oggdb

# enter ogg

[oracle@sy-bd-061 oggdb] $. / ggsci

# configure source-side mgr process

GGSCI (oracle) 10 > edit params mgr

PORT 7809

# configure synchronization users

GGSCI (oracle) 7 > dblogin userid ogg,password ogg

Successfully logged into database.

# enable additional logs for all tables under scott users

GGSCI (oracle) 8 > add trandata scott.*

# configure initload extract process on the source side

GGSCI (oracle) 8 > add extract fafainih, sourceistable

EXTRACT added.

# configure extract process fafainie parameters

Extractfafainih

Useridogg,password ogg

Rmthost192.168.45.175,mgrport 7809

-- rmttaskreplicat,group fafainir

RMTFILE./dirdat/hh

Tablescott.*

4. The configuration of the target side. (192.168.45.175)

# configure the user's JAVA_HOME and LIB_PATH environment variables in ~ /. Bash_profile under the user's home directory.

ExportJAVA_HOME=/usr/java/jdk1.8.0_91/jre/

Export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/java/jdk1.8.0_91/jre/lib/amd64/libjsig.so:/usr/java/jdk1.8.0_91/jre/lib/amd64/server/libjvm.so:/usr/java/jdk1.8.0_91/jre/lib/amd64/server:/usr/java/jdk1.8.0_91/jre/lib/amd64

# make the newly added environment variable effective

Source / .bash_profile

# enter the OGG installation directory

Cd / ogg

# copy the configuration file corresponding to hbase.

Cp AdapterExamples/big-data/hbase/* dirprm/

Cd / ogg/dirprm

[root@sy-bd-060 dirprm] # vi hbase.props

Gg.handlerlist=hbase

Gg.handler.hbase.type=hbase

Gg.handler.hbase.hBaseColumnFamilyName=cf

Gg.handler.hbase.keyValueDelimiter=CDATA [=]

Gg.handler.hbase.keyValuePairDelimiter=CDATA [,]

Gg.handler.hbase.encoding=UTF-8

Gg.handler.hbase.pkUpdateHandling=abend

Gg.handler.hbase.nullValueRepresentation=CDATA [NULL]

Gg.handler.hbase.authType=none

Gg.handler.hbase.includeTokens=false

Gg.handler.hbase.mode=tx

Goldengate.userexit.timestamp=utc

Goldengate.userexit.writers=javawriter

Javawriter.stats.display=TRUE

Javawriter.stats.full=TRUE

Gg.log=log4j

Gg.log.level=INFO

Gg.report.time=30sec

# configure the path of the library used by hbase

Gg.classpath=/usr/hdp/2.4.2.0-258/hbase/lib/*:/usr/hdp/2.4.2.0-258/hbase/conf/:/usr/hdp/2.4.2.0-258Gash Hadoop-CPN *

Javawriter.bootoptions=-Xmx512m-Xms32m-Djava.class.path=ggjava/ggjava.jar

# enter ogg

Cd / ogg

. / ggsci

GGSCI (hbase) 2 > edit params mgr

Port 7809

# configure replication process

GGSCI (hbase) 5 > add replicat irhbase, specialrun

SPECIALRUN

END RUNTIME

EXTFILE. / dirdat/hh

TARGETDB LIBFILE libggjava.so SETproperty=dirprm/hbase.props

REPORTCOUNT EVERY 1 MINUTES, RATE

GROUPTRANSOPS 10000

MAP scott.*, TARGET BDTEST.*

5. Source end (192.168.45.176)

Cd / home/oracle/oggdb

. / ggsci

# start the fafainie process

GGSCI (oracle) 1 > start fafainih

# View the execution of the process

GGSCI (oracle) 2 > view report fafainih

6. Destination (192.168.45.175)

Cd / ogg

# start the replication process

. / replicat paramfile. / dirprm/irhbase.prm reportfile. / dirrpt/irhbase.rpt-p INITIALDATALOAD

# enter the hbase environment

Hbase shell

# View the data in the table

It is found that the data in the table already exists, indicating that the data of oracle is successfully loaded into hbase.

Dynamic crawling of configuration data on the source side (192.168.45.176)

Cd / home/oracle/oggdb

Sqlplus / as sysdba

SQL > @ marker_setup

# enter in the following prompt: ogg

Enter Oracle GoldenGate schema name:ogg

# create OGG tablespaces and grant appropriate permissions to OGG users

SQL > CREATE TABLESPACE TBS_OGG DATAFILE'/ home/oracle/app/oracle/oradata/oggdb/tbs_ogg_01.dbf'SIZE 2G AUTOEXTEND ON NEXT 50m MAXSIZE UNLIMITED

Tablespace created.

SQL > alter user ogg DEFAULT TABLESPACE TBS_OGG

User altered.

SQL > grant connect,resource,unlimited tablespace to ogg

Grant succeeded.

SQL > @ ddl_setup.sql

# in the following prompt: enter ogg

Enter Oracle GoldenGate schema name:ogg

SQL > @ role_setup

# in the following prompt: enter ogg

Enter GoldenGate schema name:ogg

SQL > GRANT GGS_GGSUSER_ROLE TO OGG

# configure OGG log crawling process

Cd / home/oracle/oggdb

. / ggsci

# create OGG extraction process on the source side

GGSCI (oracle) 1 > add extract scotthb, tranlog, begin now

# specify the location of the trail file in the extraction process

GGSCI (oracle) 2 > add rmttrail / ogg/dirdat/ha,extract scotthb

# Editing scotthb process parameters

GGSCI (oracle) 3 > edit params scotthb

Extract scotthb

Userid ogg,password ogg

Rmthost 192.168.45.175, mgrport 7809

Rmttrail / ogg/dirdat/ha

Discardfile / home/oracle/oggdb/dirrpt/trail.dsc,append,megabytes 100

Ddl include mapped

Table scott.*

7. The target side (192.168.45.175) configures the data replication process

Cd / ogg

. / ggsci

# configure global parameters

GGSCI (hbase) 1 > edit params. / GLOBALS

Ggschema ogg

Checkpointtable ogg.chkpnt_fafa

# configure the resolution process

GGSCI (hbase) 2 > add replicat rscott,exttrail / ogg/dirdat/ha

# Editing parsing process parameters

GGSCI (hbase) 3 > edit params rscott

REPLICAT rscott

Trail file for this example is located in "AdapterExamples/trail" directory

-- Command to add REPLICAT

-- add replicat rhbase, exttrail AdapterExamples/trail/tr

TARGETDB LIBFILE libggjava.so SET property=dirprm/hbase.props

REPORTCOUNT EVERY 1 MINUTES, RATE

GROUPTRANSOPS 10000

MAP scott.*, TARGET bdtest.*

8. The source side (192.168.45.176) starts the crawling process

Cd / home/oracle/ogdb

. / ggsci

GGSCI (oracle) 1 > start scotthb

9. The destination (192.168.45.173) starts the replication process

Cd / home/oracle/oggdb

. / ggsci

GGSCI (kafka) 1 > start rscott

At this point, the whole configuration process is completed, and after the configuration is completed, you can test whether the newly changed data can be synchronized to hbase by performing DML operations in the oracle database in the source side (192.168.45.176).

10. Test the source database (192.168.45.176)

Sqlplus / as sysdba

SQL > Alter user scott account unlock

SQL > conn scott/tiger

# insert a new piece of data into the scott.dept table

SQL > insert into scott.deptvalues (50 million DEVELOPPM SHANGHAI')

SQL > commit

11. View the data on the destination side (192.168.45.175)

Hbase shell

In hbase, the newly added data can already be queried.

12. Modify oracle data on the source side (192.168.45.176).

Sqlplus / as sysdba

SQL > update scott.dept set loc='HANGZHOU'where deptno=50

SQL > commit

13. View the data on the destination side (192.168.45.175)

Hbase shell

It is found that the record corresponding to the hbase data has been modified accordingly.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.