Shulou(Shulou.com)05/31 Report--

This article mainly introduces how to use OGG to synchronize ORACLE ACTIVE DATAGUARD data to CLOUDERA HBASE testing, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let Xiaobian take you to understand.

Environment introduction:

Machine name

Database version

Architecture

OGG version

IP

OS version

Uatdbrac1

Oracle 11.2.0.4

Rac

NULL

172.16.49.191

REDHAT6.4

Uatdbrac2

Oracle 11.2.0.4

Rac

NULL

172.16.49.193

REDHAT6.4

Uagracdg

Oracle 11.2.0.4

ADG

OGG 12.2.0.1

172.16.49.138

REDHAT6.4

Master1

Hbase

Cdh6.6.0

OGG FOR BIG DATA 12.2.0.1

172.16.49.35

REDHAT6.4

Purpose of the test:

Use ORACLE ACTIVE DATAGUARD as the data extraction source and use ORACLE GOLDENGATE to synchronize the change data in the ORACLE database to the HBASE cluster.

I. deploy the ACTIVE DATAGUARD of ORACLE RAC

Please refer to the previous ADG EXAMPLE documentation

II. Preparatory work on the main library RAC

1. Create GoldenGate tablespace

two。 Create GoldenGate users and authorize

3. Enable force logging (it has already been configured when doing Active Datagurad)

4. Enable additional logs

Alter database add supplemental log data

Alter system switch logfile

Select supplemental_log_data_min from v$database

5. Enable enable_goldengate_replication

Alter system set enable_goldengate_replication=true

III. Deploy goldengate 12.2 on ADG

1. Create a golden os user

two。 Create the ogg software installation directory / goldengate/gg12c

3. Edit golden user profile

Vi .bash _ profile adds the following:

Export ORACLE_BASE=/u01/app/oracle

Export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1

Export ORACLE_SID=uatracdg-ACTIVE DATAGUARD ORACLE SID

Export LD_LIBRARY_PATH=/goldengate/gg12c:$ORACLE_HOME/lib:$ORACLE_HOME/lib32

Export OGG_HOME=/goldengate/gg12c

PATH=$PATH:$HOME/bin:$ORACLE_HOME/bin:/goldengate/gg12c:.

Export PATH

Cd $OGG_HOME

4. Download goldengate 12.2 for linux 64bit version

The latest package name is V100692-01.zip

5. Extract the installation package

Unzip V100692-01.zip gets the following files:

Fbo_ggs_Linux_x64_shiphome

OGG-12.2.0.1.1-ReleaseNotes.pdf

OGG-12.2.0.1-README.txt

6. Edit silent installation parameter file

Cd / goldengate/fbo_ggs_Linux_x64_shiphome/Disk1/response

Edit the response file to change the following parameters

INSTALL_OPTION=ORA11g

SOFTWARE_LOCATION=/goldengate/gg12c

START_MANAGER=true

MANAGER_PORT=7801

DATABASE_LOCATION=/u01/app/oracle/product/11.2.0/dbhome_1

INVENTORY_LOCATION=/goldengate/gg12c/oraInventory

UNIX_GROUP_NAME=oinstall

7. Install OGG silently

Cd / goldengate/fbo_ggs_Linux_x64_shiphome/Disk1

. / runInstaller-silent-responseFile / goldengate/fbo_ggs_Linux_x64_shiphome/Disk1/response/oggcore.rsp

8. Add additional logs for the table

Because ADG is read-only mode, adding additional logs requires changing the data dictionary, so this operation needs to be done by connecting to the main library RAC on the ggsci command line.

Ggsci dblogin userid goldengate@uatrac passowrd

Add trandata HADP.OGGT1

Add trandata HADP.OGGT2

9. Add configuration extract process

Add extract ext01, tranlog, begin yyyy-MM-dd HH:mm:ss

Add exttrail. / dirdat/et, extract ext01

Edit param ext01

EXTRACT ext01

Setenv (NLS_LANG= "SIMPLIFIED CHINESE_CHINA.ZHS16GBK")

Setenv (ORACLE_SID=uatracdg)

Userid goldengate, password golden123

Exttrail. / dirdat/et

Gettruncates

-- dynamicresolution

Discardfile. / dirrpt/discardext01.txt, append, megabytes 50

-- DBOPTIONS ALLOWUNUSEDCOLUMN

-- TRANLOGOPTIONS CONVERTUCS2CLOBS

EOFDELAY 3

Getupdatebefores

TRANLOGOPTIONS MINEFROMACTIVEDG-FOR ADG

TABLE HADP.OGGT2

TABLE HADP.OGGT1

10. Add configuration PUMP process

Add extract pup1, exttrailsource. / dirdat/et

Add rmttrail. / dirdat/rt, extract pup1

Edit param pup1

Extract pup1

Setenv (NLS_LANG= "SIMPLIFIED CHINESE_CHINA.ZHS16GBK")

Userid goldengate password golden123

Rmthost 172.16.49.35, mgrport 7809, TCPBUFSIZE 5000000

Rmttrail. / dirdat/rt

DYNAMICRESOLUTION

NUMFILES 3000

ALLOCFILES 200

Passthru

TABLE HADP.OGGT2

TABLE HADP.OGGT1

IV. Deploy goldengate for big data 12.2 on CDH

1. Create a golden os user

two。 Create the ogg software installation directory / goldengate/gg12c

3. Edit golden user profile

Export JAVA_HOME=/usr/local/jdk1.7.0_80

PATH=$PATH:$HOME/bin

Export PATH

Export NLS_LANG=AMERICAN_AMERICA.ZHS16BGK

Export PATH=$JAVA_HOME/bin:$PATH:/goldengate/gg12c:/usr/local/apache-maven-3.1.1/bin:.

Export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64/server:/goldengate/gg12c:/goldengate/gg12c/ggjava/resources/ggplugins/hdfs:.

Export CLASSPATH=/goldengate/gg12c/ggjava/*:/goldengate/gg12c/dirprm/*:/goldengate/gg12c/lib/*:/etc/hadoop/conf/core-site.xml:/etc/hadoop/conf/hdfs-site.xml:/opt/cloudera/parcels/

CDH-5.6.0-1.cdh6.6.0.p0.45/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5.6.0-1.cdh6.6.0.p0.45/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.6.0-1.cdh6.6.0.p0.45/lib/hadoop/*:/

Opt/cloudera/parcels/CDH-5.6.0-1.cdh6.6.0.p0.45/lib/hadoop/client/*:/goldengate/gg12c/dirprm/*:/opt/cloudera/parcels/CDH-5.6.0-1.cdh 6.6.0.p0.45 Rhythm.

Export OGG_HOME=/goldengate/gg12c

4. Download ogg for big data

The latest software package V100447-01.zip

5. Install ogg for big data

Decompress it in the installation directory.

6. Create a subdirectory

Ggsci > create subdirs

7. Configuration Management process MGR

Edit param mgr

PORT 7809

Dynamicportlist 7820-7850

8. Add configuration replication process

Copy the file under / goldengate/gg12c/AdapterExamples/big-data/hbase to $OGG_HOME/dirprm

Edit hbase.props

Change the following parameters

Gg.handler.hbase.hBaseColumnFamilyName=ID

Gg.classpath=/etc/hbase/conf.cloudera.hbase:/opt/cloudera/parcels/CDH-5.6.0-1.cdh6.6.0.p0.45/lib/hbase/*:/opt/cloudera/parcels/CDH-5.6.0-1.cdh6.6.0.p0.45/lib/hbase/lib/*:/opt/cloudera/parcels/CDH-5.6.0-1.cdh6.6.0.p0.45/lib/hadoop/*:/opt/cloudera/ Parcels/CDH-5.6.0-1.cdh6.6.0.p0.45/lib/hadoop/client/*:/goldengate/gg12c/dirprm/*:/opt/cloudera/parcels/CDH-5.6.0-1.cdh 6.6.0.p0.45 Rhythm.

Ggsci > add replicat rhbase, nodbcheckpoint, exttrail. / dirdat/rt

Ggsci > Edit param rhbase

REPLICAT rhbase

SOURCEDEFS. / dirsql/hdfs.sql

Trail file for this example is located in "AdapterExamples/trail" directory

-- Command to add REPLICAT

-- add replicat rhbase, exttrail AdapterExamples/trail/tr

Setenv (NLS_LANG=AMERICAN_AMERICA.ZHS16GBK)

TARGETDB LIBFILE / goldengate/gg12c/libggjava.so SET property=/goldengate/gg12c/dirprm/hbase.props

REPORTCOUNT EVERY 1 MINUTES, RATE

GROUPTRANSOPS 10000

MAP HADP.OGGT2, TARGET HADP.OGGTEST

RESOLVECONFLICT (INSERTROWEXISTS, (DEFAULT, OVERWRITE))

COLMAP (ID=ID, COL1=COL1, COL2=COL2)

MAP HADP.OGGT1, TARGET HADP.OGGTEST

RESOLVECONFLICT (INSERTROWEXISTS, (DEFAULT, OVERWRITE))

COLMAP (ID=ID, COLA=COLA, COLB=COLB)

9. Log in to HBASE to create a test table

Create a namespace HADP:

Hbase shell

Create namespace 'HADP'

List_namespace

Describe_namespace 'HADP'

Create a table:

Create 'HADP:OGGTEST','ID','COL1','COL2','COLA','COLB'

View table data:

Scan 'HADP:OGGTEST'

10. View OGG FOR BIG DATA Log

Cd / goldengate/gg12c/dirrpt

More * hbase*.log

Thank you for reading this article carefully. I hope the article "how to use OGG to synchronize ORACLE ACTIVE DATAGUARD data to CLOUDERA HBASE testing" shared by the editor will be helpful to you. At the same time, I also hope you will support us and follow the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.