Purpose is to demonstrate how to create HA for Bi-Directional Replication Goldengate installed on ACFS with RAC cluster using XAG.
XAG simplifies the process since there are no requirements to create action scripts.
Please review REFERENCE section for versions used in test case and versions requirements.
Goldengate is installed on ACFS for simplicity; otherwise, at a minimum the following directories br, dirchk, dirdat, dirtmp will need to be on shared storage with symbolic links if installed on local storage. Keyword is minimum until you find out more directories are required.
Role separation was a huge PITA and do not attempt to perform chmod -R 775 /u01 as it will break since the setuid get unset.
Even with chmod 6751 oracle may prove to be ineffective and relink was done.
# id grid uid=54322(grid) gid=54321(oinstall) groups=54321(oinstall),54322(dba),493(vboxsf),54323(asmadmin),54324(asmdba) # id oracle uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),493(vboxsf),54324(asmdba) # id gguser uid=54323(gguser) gid=54321(oinstall) groups=54321(oinstall),54322(dba),493(vboxsf) $ ls -l $ORACLE_HOME/bin/oracle -rwsr-s--x. 1 grid oinstall 209914519 Dec 7 20:57 /u01/app/11.2.0.4/grid/bin/oracle
Last, DO NOT stop Goldengate using ggsci. For cluster-aware Goldengate, use $XAG_HOME/bin/agctl.
The configuration shown is for one cluster only and the same would have to be performed on the other clusters.
AS ROOT, CONFIGURE GOLDENGATE VIP (lax-ggate1-vip)
Also note below steps:
1. As root create app vip with user=gguser
2. Grant permissions on vip to oracle/grid
3. Grant permission on ACFS to gguser
Reference: Getting OGG-01091(error 13, Permission denied) with oracle GI XAG (Doc ID 1984100.1)
1. Determine the network number
# crsctl stat res -p|grep -ie .network -ie subnet|grep -ie name -ie ora_subnet NAME=ora.net1.network USR_ORA_SUBNET=192.168.56.0
2. Create GoldenGate VIP with naming Convention: LAX (closest airport code to data center), ggate1 (for Goldengate on network number 1), vip
Note IP address provided is 192.168.56.41
# appvipcfg create -network=1 -ip=192.168.56.41 -vipname=lax-ggate1-vip -user=gguser -group=oinstall Production Copyright 2007, 2008, Oracle.All rights reserved 2014-12-14 10:58:41: Creating Resource Type 2014-12-14 10:58:41: Executing /u01/app/11.2.0.4/grid/bin/crsctl add type app.appvip_net1.type -basetype ora.cluster_vip_net1.type -file /u01/app/11.2.0.4/grid/crs/template/appvip.type 2014-12-14 10:58:41: Executing cmd: /u01/app/11.2.0.4/grid/bin/crsctl add type app.appvip_net1.type -basetype ora.cluster_vip_net1.type -file /u01/app/11.2.0.4/grid/crs/template/appvip.type 2014-12-14 10:58:42: Create the Resource 2014-12-14 10:58:42: Executing /u01/app/11.2.0.4/grid/bin/crsctl add resource lax-ggate1-vip -type app.appvip_net1.type -attr "USR_ORA_VIP=192.168.56.41,START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network),STOP_DEPENDENCIES=hard(ora.net1.network),ACL='owner:root:rwx,pgrp:root:r-x,other::r--,group:oinstall:r-x,user:root:r-x',HOSTING_MEMBERS=rac01.localdomain,APPSVIP_FAILBACK=" 2014-12-14 10:58:42: Executing cmd: /u01/app/11.2.0.4/grid/bin/crsctl add resource lax-ggate1-vip -type app.appvip_net1.type -attr "USR_ORA_VIP=192.168.56.41,START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network),STOP_DEPENDENCIES=hard(ora.net1.network),ACL='owner:root:rwx,pgrp:root:r-x,other::r--,group:oinstall:r-x,user:root:r-x',HOSTING_MEMBERS=rac01.localdomain,APPSVIP_FAILBACK="
3. Set permission for users to start/stop VIP
# crsctl setperm resource lax-ggate1-vip -u user:oracle:r-x # crsctl setperm resource lax-ggate1-vip -u user:grid:r-x </del># crsctl setperm resource lax-ggate1-vip -u user:gguser:r-x</del> # crsctl start resource lax-ggate1-vip CRS-2672: Attempting to start 'lax-ggate1-vip' on 'rac02' CRS-2676: Start of 'lax-ggate1-vip' on 'rac02' succeeded
RETRIEVE INFORMATIONS TO CREATE GOLDENGATE AGENT
–nodes
$ olsnodes rac01 rac02
–filesystems
$ crsctl stat res -w "TYPE = ora.acfs.type" -p|grep '^NAME' NAME=ora.dg_acfs.vg_acfs.acfs NAME=ora.dg_acfs.vg_acfs.acfs
–databases
$ crsctl stat res -w "TYPE = ora.database.type" NAME=ora.emu.db TYPE=ora.database.type TARGET=ONLINE , ONLINE STATE=ONLINE on rac01, ONLINE on rac02
— extracts and replicats
GGSCI (rac02.localdomain) 1> info all Program Status Group Lag at Chkpt Time Since Chkpt MANAGER RUNNING EXTRACT RUNNING ELAX 00:00:00 00:00:02 EXTRACT RUNNING PLAX_DEN 00:00:00 00:00:03 REPLICAT RUNNING RDEN_LAX 00:00:00 00:00:06
AS GRID, CONFIGURE GOLDENGATE AGENT (lax_ggate)
AS GOLDENGATE USER, CONFIGURE GOLDENGATE AGENT (lax_ggate)
Options for –instance_type is source|target|dual and I am interpreting dual is bi-directional.
$XAG_HOME/bin/agctl add goldengate lax_ggate \ --gg_home /acfsmount/ggs112 \ --instance_type dual \ --nodes rac01,rac02 \ --vip_name lax-ggate1-vip \ --filesystems ora.dg_acfs.vg_acfs.acfs \ --databases ora.emu.db \ --oracle_home /u01/app/oracle/product/11.2.0.4/db_1 \ --monitor_extracts ELAX,PLAX_DEN \ --critical_extracts ELAX,PLAX_DEN \ --monitor_replicats RDEN_LAX \ --critical_replicats RDEN_LAX
VERIFY RESULTS
$XAG_HOME/bin/agctl config goldengate lax_ggate GoldenGate location is: /acfsmount/ggs112 GoldenGate instance type is: dual Configured to run on Nodes: rac01 rac02 ORACLE_HOME location is: /u01/app/oracle/product/11.2.0.4/db_1 Databases needed: ora.emu.db File System resources needed: ora.dg_acfs.vg_acfs.acfs Extracts to monitor: ELAX,PLAX_DEN Replicats to monitor: RDEN_LAX Critical extracts: ELAX,PLAX_DEN Critical replicats: RDEN_LAX Autostart on DataGuard role transition to PRIMARY: no Autostart JAgent: no
START GOLDENDATE
$XAG_HOME/bin/agctl start goldengate lax_ggate
CHECK GOLDENGATE STATUS
$XAG_HOME/bin/agctl status goldengate lax_ggate Goldengate instance 'lax_ggate' INTERMEDIATE on rac02 $XAG_HOME/bin/agctl status goldengate lax_ggate Goldengate instance 'lax_ggate' is running on rac02
$ crsstat Resource Name Resource Type Target State Node State Details ---------------------------------------- -------------- ------------ ------------ --------------- --------------- lax-ggate1-vip appvip_net1 ONLINE ONLINE rac02 ora.DATA2.dg diskgroup ONLINE ONLINE rac01 ora.DATA2.dg diskgroup ONLINE ONLINE rac02 ora.DG_ACFS.dg diskgroup ONLINE ONLINE rac01 ora.DG_ACFS.dg diskgroup ONLINE ONLINE rac02 ora.LISTENER.lsnr Listener ONLINE ONLINE rac01 ora.LISTENER.lsnr Listener ONLINE ONLINE rac02 ora.LISTENER_SCAN1.lsnr SCAN Listener ONLINE ONLINE rac01 ora.LISTENER_SCAN2.lsnr SCAN Listener ONLINE ONLINE rac02 ora.LISTENER_SCAN3.lsnr SCAN Listener ONLINE ONLINE rac02 ora.asm ASM ONLINE ONLINE rac01 Started ora.asm ASM ONLINE ONLINE rac02 Started ora.cvu cvu ONLINE ONLINE rac02 ora.dg_acfs.vg_acfs.acfs acfs ONLINE ONLINE rac01 mounted on /acfsmount ora.dg_acfs.vg_acfs.acfs acfs ONLINE ONLINE rac02 mounted on /acfsmount ora.emu.db database ONLINE ONLINE rac01 Open ora.emu.db database ONLINE ONLINE rac02 Open ora.gsd Gbl Svc Daemon OFFLINE OFFLINE ora.gsd Gbl Svc Daemon OFFLINE OFFLINE ora.net1.network Network (VIP) ONLINE ONLINE rac01 ora.net1.network Network (VIP) ONLINE ONLINE rac02 ora.oc4j OC4J ONLINE ONLINE rac02 ora.ons Ora Notif Svc ONLINE ONLINE rac01 ora.ons Ora Notif Svc ONLINE ONLINE rac02 ora.rac01.vip Cluster VIP ONLINE ONLINE rac01 ora.rac02.vip Cluster VIP ONLINE ONLINE rac02 ora.registry.acfs registry ONLINE ONLINE rac01 ora.registry.acfs registry ONLINE ONLINE rac02 ora.scan1.vip SCAN VIP ONLINE ONLINE rac01 ora.scan2.vip SCAN VIP ONLINE ONLINE rac02 ora.scan3.vip SCAN VIP ONLINE ONLINE rac02 xag.lax_ggate.goldengate goldengate ONLINE ONLINE rac02
CONNECT TO THE CORRECT NODE FOR GOLDENGATE
$ su - gguser Password: 0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ !!!!! PLEASE CONNECT TO NODE WHERE GOLDENGATE IS RUNNING !!!!! !!!!! !!!!! --+ Goldengate instance 'lax_ggate' is running on rac02 +-- !!!!! !!!!! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [gguser@rac01:/home/gguser] $ ssh rac02 Last login: Sun Dec 14 13:23:48 2014 from rac01 1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~ GOLDENGATE IS RUNNING ON THIS NODE - GOOD TO GO ~~~~~ --+ Goldengate instance 'lax_ggate' is running on rac02 +-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [gguser@rac02:/home/gguser] $ .bash_profile # User specific environment and startup programs ogg=`/u01/app/grid/xag/bin/agctl status goldengate lax_ggate` /u01/app/grid/xag/bin/agctl status goldengate lax_ggate|grep -ic `hostname -f` if [ "$?" != "0" ]; then clear echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" echo "!!!!! PLEASE CONNECT TO NODE WHERE GOLDENGATE IS RUNNING !!!!!" echo "!!!!! !!!!!" echo "--+ $ogg +--" echo "!!!!! !!!!!" echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" else clear echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" echo "~~~~~ GOLDENGATE IS RUNNING ON THIS NODE - GOOD TO GO ~~~~~" echo "--+ $ogg +--" echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" fi umask 0002
RELOCATE GOLDENGATE TO DIFFERENT NODE
[grid@rac01:+ASM1:/home/grid] $ crsstat Resource Name Resource Type Target State Node State Details ---------------------------------------- -------------- ------------ ------------ --------------- --------------- lax-ggate1-vip appvip_net1 ONLINE ONLINE rac02 ora.DATA2.dg diskgroup ONLINE ONLINE rac01 ora.DATA2.dg diskgroup ONLINE ONLINE rac02 ora.DG_ACFS.dg diskgroup ONLINE ONLINE rac01 ora.DG_ACFS.dg diskgroup ONLINE ONLINE rac02 ora.LISTENER.lsnr Listener ONLINE ONLINE rac01 ora.LISTENER.lsnr Listener ONLINE ONLINE rac02 ora.LISTENER_SCAN1.lsnr SCAN Listener ONLINE ONLINE rac01 ora.LISTENER_SCAN2.lsnr SCAN Listener ONLINE ONLINE rac02 ora.LISTENER_SCAN3.lsnr SCAN Listener ONLINE ONLINE rac02 ora.asm ASM ONLINE ONLINE rac01 Started ora.asm ASM ONLINE ONLINE rac02 Started ora.cvu cvu ONLINE ONLINE rac02 ora.dg_acfs.vg_acfs.acfs acfs ONLINE ONLINE rac01 mounted on /acfsmount ora.dg_acfs.vg_acfs.acfs acfs ONLINE ONLINE rac02 mounted on /acfsmount ora.emu.db database ONLINE ONLINE rac01 Open ora.emu.db database ONLINE ONLINE rac02 Open ora.gsd Gbl Svc Daemon OFFLINE OFFLINE ora.gsd Gbl Svc Daemon OFFLINE OFFLINE ora.net1.network Network (VIP) ONLINE ONLINE rac01 ora.net1.network Network (VIP) ONLINE ONLINE rac02 ora.oc4j OC4J ONLINE ONLINE rac02 ora.ons Ora Notif Svc ONLINE ONLINE rac01 ora.ons Ora Notif Svc ONLINE ONLINE rac02 ora.rac01.vip Cluster VIP ONLINE ONLINE rac01 ora.rac02.vip Cluster VIP ONLINE ONLINE rac02 ora.registry.acfs registry ONLINE ONLINE rac01 ora.registry.acfs registry ONLINE ONLINE rac02 ora.scan1.vip SCAN VIP ONLINE ONLINE rac01 ora.scan2.vip SCAN VIP ONLINE ONLINE rac02 ora.scan3.vip SCAN VIP ONLINE ONLINE rac02 xag.lax_ggate.goldengate goldengate ONLINE ONLINE rac02 [grid@rac01:+ASM1:/home/grid] $ $XAG_HOME/bin/agctl relocate goldengate lax_ggate --node rac01 [grid@rac01:+ASM1:/home/grid] $ crsstat Resource Name Resource Type Target State Node State Details ---------------------------------------- -------------- ------------ ------------ --------------- --------------- lax-ggate1-vip appvip_net1 ONLINE ONLINE rac01 ora.DATA2.dg diskgroup ONLINE ONLINE rac01 ora.DATA2.dg diskgroup ONLINE ONLINE rac02 ora.DG_ACFS.dg diskgroup ONLINE ONLINE rac01 ora.DG_ACFS.dg diskgroup ONLINE ONLINE rac02 ora.LISTENER.lsnr Listener ONLINE ONLINE rac01 ora.LISTENER.lsnr Listener ONLINE ONLINE rac02 ora.LISTENER_SCAN1.lsnr SCAN Listener ONLINE ONLINE rac01 ora.LISTENER_SCAN2.lsnr SCAN Listener ONLINE ONLINE rac02 ora.LISTENER_SCAN3.lsnr SCAN Listener ONLINE ONLINE rac02 ora.asm ASM ONLINE ONLINE rac01 Started ora.asm ASM ONLINE ONLINE rac02 Started ora.cvu cvu ONLINE ONLINE rac02 ora.dg_acfs.vg_acfs.acfs acfs ONLINE ONLINE rac01 mounted on /acfsmount ora.dg_acfs.vg_acfs.acfs acfs ONLINE ONLINE rac02 mounted on /acfsmount ora.emu.db database ONLINE ONLINE rac01 Open ora.emu.db database ONLINE ONLINE rac02 Open ora.gsd Gbl Svc Daemon OFFLINE OFFLINE ora.gsd Gbl Svc Daemon OFFLINE OFFLINE ora.net1.network Network (VIP) ONLINE ONLINE rac01 ora.net1.network Network (VIP) ONLINE ONLINE rac02 ora.oc4j OC4J ONLINE ONLINE rac02 ora.ons Ora Notif Svc ONLINE ONLINE rac01 ora.ons Ora Notif Svc ONLINE ONLINE rac02 ora.rac01.vip Cluster VIP ONLINE ONLINE rac01 ora.rac02.vip Cluster VIP ONLINE ONLINE rac02 ora.registry.acfs registry ONLINE ONLINE rac01 ora.registry.acfs registry ONLINE ONLINE rac02 ora.scan1.vip SCAN VIP ONLINE ONLINE rac01 ora.scan2.vip SCAN VIP ONLINE ONLINE rac02 ora.scan3.vip SCAN VIP ONLINE ONLINE rac02 xag.lax_ggate.goldengate goldengate ONLINE INTERMEDIATE rac01 ER(s) not running : ELAX,ELAX,RDEN_LAX [grid@rac01:+ASM1:/home/grid] $ crsstat Resource Name Resource Type Target State Node State Details ---------------------------------------- -------------- ------------ ------------ --------------- --------------- lax-ggate1-vip appvip_net1 ONLINE ONLINE rac01 ora.DATA2.dg diskgroup ONLINE ONLINE rac01 ora.DATA2.dg diskgroup ONLINE ONLINE rac02 ora.DG_ACFS.dg diskgroup ONLINE ONLINE rac01 ora.DG_ACFS.dg diskgroup ONLINE ONLINE rac02 ora.LISTENER.lsnr Listener ONLINE ONLINE rac01 ora.LISTENER.lsnr Listener ONLINE ONLINE rac02 ora.LISTENER_SCAN1.lsnr SCAN Listener ONLINE ONLINE rac01 ora.LISTENER_SCAN2.lsnr SCAN Listener ONLINE ONLINE rac02 ora.LISTENER_SCAN3.lsnr SCAN Listener ONLINE ONLINE rac02 ora.asm ASM ONLINE ONLINE rac01 Started ora.asm ASM ONLINE ONLINE rac02 Started ora.cvu cvu ONLINE ONLINE rac02 ora.dg_acfs.vg_acfs.acfs acfs ONLINE ONLINE rac01 mounted on /acfsmount ora.dg_acfs.vg_acfs.acfs acfs ONLINE ONLINE rac02 mounted on /acfsmount ora.emu.db database ONLINE ONLINE rac01 Open ora.emu.db database ONLINE ONLINE rac02 Open ora.gsd Gbl Svc Daemon OFFLINE OFFLINE ora.gsd Gbl Svc Daemon OFFLINE OFFLINE ora.net1.network Network (VIP) ONLINE ONLINE rac01 ora.net1.network Network (VIP) ONLINE ONLINE rac02 ora.oc4j OC4J ONLINE ONLINE rac02 ora.ons Ora Notif Svc ONLINE ONLINE rac01 ora.ons Ora Notif Svc ONLINE ONLINE rac02 ora.rac01.vip Cluster VIP ONLINE ONLINE rac01 ora.rac02.vip Cluster VIP ONLINE ONLINE rac02 ora.registry.acfs registry ONLINE ONLINE rac01 ora.registry.acfs registry ONLINE ONLINE rac02 ora.scan1.vip SCAN VIP ONLINE ONLINE rac01 ora.scan2.vip SCAN VIP ONLINE ONLINE rac02 ora.scan3.vip SCAN VIP ONLINE ONLINE rac02 xag.lax_ggate.goldengate goldengate ONLINE ONLINE rac01
STOP GOLDENDATE
[grid@rac01:+ASM1:/home/grid] $ $XAG_HOME/bin/agctl stop goldengate lax_ggate [grid@rac01:+ASM1:/home/grid]
DISCOVERY
It’s easy to know where everything when you are the architect, but what happens when you are not?
How to find the Goldengate VIP and is it in DNS?
$ crsctl stat res lax-ggate1-vip -p|grep USR_ORA_VIP GEN_USR_ORA_VIP= USR_ORA_VIP=192.168.56.41 $ nslookup 192.168.56.41 Server: 127.0.0.1 Address: 127.0.0.1#53 41.56.168.192.in-addr.arpa name = lax-ggate1-vip.localdomain. $ crsctl stat res -w "TYPE = ora.acfs.type" -p|grep '^MOUNT' MOUNTPOINT_PATH=/acfsmount MOUNTPOINT_PATH=/acfsmount
REFERENCE:
Oracle GoldenGate Best Practices: Configuring Oracle GoldenGate with Oracle Grid Infrastructure Bundled Agents (XAG) - (Document 1527310.1) Oracle Clusterware: http://oracle.com/goto/clusterware Oracle Grid Infrastructure (Bundled) Agents: http://www.oracle.com/technetwork/database/database-technologies/clusterware/downloads/index.html Oracle Grid Infrastructure Agents Version 5.1 11.2.0.3 and later 12.1.0.1 and later SRDC - Data to Collect on GoldenGate Issues Related to XAG (Doc ID 1913048.1) Patch 16762459: ORACLE GOLDENGATE V11.2.1.0.7 FOR ORACLE 11G How To Create A New ACFS Volume & Filesystem And Set The ACFS Filesystem Ownership To A Non-Grid/Oracle OS User (on Oracle Restart/SIHA/Standalone)? (Doc ID 1322383.1) Getting OGG-01091(error 13, Permission denied) with oracle GI XAG (Doc ID 1984100.1) [gguser@angel:/u01/app/ggs112] $ ./ggsci Oracle GoldenGate Command Interpreter for Oracle Version 11.2.1.0.7 16934304 OGGCORE_11.2.1.0.7_PLATFORMS_130709.1600.1_FBO Linux, x64, 64bit (optimized), Oracle 11g on Jul 18 2013 07:04:28 Copyright (C) 1995, 2013, Oracle and/or its affiliates. All rights reserved. GGSCI (angel.local) 1> dblogin userid ggadmin, password ggadmin Successfully logged into database. GGSCI (angel.local) 2> versions Operating System: Linux Version #1 SMP Fri Feb 22 18:16:18 PST 2013, Release 2.6.39-400.17.1.el6uek.x86_64 Node: angel.local Machine: x86_64 Database: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production PL/SQL Release 11.2.0.4.0 - Production CORE 11.2.0.4.0 Production TNS for Linux: Version 11.2.0.4.0 - Production NLSRTL Version 11.2.0.4.0 - Production
Thinking Out Loud 