Thinking Out Loud

June 11, 2017

GoldenGate 12.2 TROUBLESHOOTING REPLICAT LAG

Filed under: 12c,GoldenGate,oracle — mdinh @ 2:16 pm

Time Since Chkpt and Lag at Chkpt from replicat keep increasing

Program     Status      Group       Lag at Chkpt  Time Since Chkpt
REPLICAT    RUNNING     R_NEW12C    03:49:45      06:37:47    

This occurs for due to the following reasons:

Delivering a long running transaction
Waiting on a full table scan
Blocked by another sessions
Primary extract lag or pump lag keeps increasing

Is There a Way to Make Long-running Transactions Checkpoint? (Doc ID 969684.1)

The tradeoff with GROUPTRANSOPS is with efficiency. 
The tradeoff with MAXTRANSOPS is efficiency and transaction integrity. 

There are no long running transaction from the extract and is monitored using WARNLONGTRANS 15m, CHECKINTERVAL 3m.

grep "Long Running Transaction" dirrpt/E_OLD10G.rpt

My suspicion is FTS, but how to find out?

SQL> r
  1  select
  2  -- SQL_ID,PLAN_HASH_VALUE,
  3  OBJECT_OWNER,OBJECT_NAME, min(TIMESTAMP) min_ts, max(TIMESTAMP) max_ts, count(*) ct
  4  from DBA_HIST_SQL_PLAN
  5  where operation='TABLE ACCESS'
  6  and options='FULL'
  7  and NOT REGEXP_LIKE(object_owner,'SYS|SYSTEM|DBSNMP')
  8  and TIMESTAMP > TO_DATE('01-JUN-2017','DD-MON-YYYY')
  9  group by
 10  -- SQL_ID,PLAN_HASH_VALUE,
 11  OBJECT_OWNER,OBJECT_NAME
 12  order by count(*), OBJECT_OWNER,OBJECT_NAME
 13*
 
OBJECT_OWNER         OBJECT_NAME                    MIN_TS               MAX_TS                       CT
-------------------- ------------------------------ -------------------- -------------------- ----------
XXX                  THISISAREALLYLONGTABLENAME     01-JUN-2017 00:01:00 11-JUN-2017 02:55:36        114

SQL> select index_name from dba_indexes where table_name='THISISAREALLYLONGTABLENAME';
no rows selected
SQL> 

June 10, 2017

GoldenGate Debugging

Filed under: 12c,GoldenGate — mdinh @ 2:40 am

I was working on automating debug information to submit to Oracle Support and thought I share implementation for what was requested.

OGG_GROUP_NAME is from ggsci info all (case sensitive)
./debug_gg.sh: line 3: 1: —> USAGE: /debug_gg.sh OGG_GROUP_NAME
./debug_gg.sh E_LAX

#!/bin/sh -e
DN=`dirname $0`
BN=`basename $0`
OGG_GROUP_NAME=${1:?"---> USAGE: $DN/$BN "OGG_GROUP_NAME""}
set -x
ps -o pid,uname,cmd `pgrep -f "extract.*${OGG_GROUP_NAME}"`
pstack `pgrep -f "extract.*${OGG_GROUP_NAME}"`
pstack `pgrep -f "extract.*${OGG_GROUP_NAME}"`
pstack `pgrep -f "extract.*${OGG_GROUP_NAME}"`
set +x
$GG_HOME/ggsci << EOF
info ${OGG_GROUP_NAME} detail
send ${OGG_GROUP_NAME} status
sh sleep 2m
send ${OGG_GROUP_NAME} status
sh sleep 2m
send ${OGG_GROUP_NAME} status
send ${OGG_GROUP_NAME} report
exit
EOF
echo " "
ls -alrt $GG_HOME/dirrpt/${OGG_GROUP_NAME}*.rpt|tail -3
exit

Example output from pgrep and pstack

++ pgrep -f 'extract.*E_LAX'
+ ps -o pid,uname,cmd 40882
  PID USER     CMD
40882 ggsuser  /u01/gg/12.2.0/extract PARAMFILE /u01/gg/12.2.0/dirprm/e_lax.prm REPORTFILE /u01/gg/12.2.0/dirrpt/E_LAX.rpt PROCESSID E_LAX USESUBDIRS
++ pgrep -f 'extract.*E_LAX'
+ pstack 40882

Just realized does not scale since extract is hard coded.

Next step, learn how to read pstack output – YIKES!

June 8, 2017

GoldenGate Extract RBA Not Moving LAG Increasing Appears Hung

Filed under: GoldenGate — mdinh @ 10:55 am

OGG Extract RBA Not Moving And LAG Increasing And Appears Hung (Doc ID 964705.1)

Typically, when Goldengate is performing recovery:

In recovery[1] – Extract is recovering to its checkpoint in the transaction log.
In recovery[2] – Extract is recovering from its checkpoint to the end of the trail.
Recovery complete – The recovery is finished, and normal processing will resume.

Example:
Current status: In recovery[1]: Reading from data source

SEND EXTRACT

First time I have seen Processing data with empty data queue and is essentially the same process.

send e* status  
Current status: In recovery[1]: Processing data with empty data queue

send e* status
Current status: Recovery complete: Processing data with empty data queue    

The scary part is current redo vs Goldengate Checkpoint.

To prevent this, make sure there is no lag, no long running transactions before stopping extract.

select thread#,sequence# from v$log where status='CURRENT' order by 1;

THREAD#    SEQUENCE#
---------- ----------
        1     1198566
        2     1291021
        
info e*
Log Read Checkpoint  Oracle Redo Logs
                     2017-06-07 14:44:53  Thread 1, Seqno 1198492, RBA 112396
Log Read Checkpoint  Oracle Redo Logs
                     2017-06-07 14:45:33  Thread 2, Seqno 1290939, RBA 1060244424

What if integrated extract is used? Where’s my Seqno?

info e*
Log Read Checkpoint Oracle Integrated Redo Logs
2017-02-12 18:36:06
SCN 0.4928929 (4928929)

Read more about at Log sequence# and rba# of integrated extract checkpoint

May 15, 2017

DBFS and XAG for Goldengate P5

Filed under: 12c,GoldenGate — mdinh @ 11:49 pm
$ $GRID_HOME/crs/script/mount-dbfs.sh version
20160215

$ grep ^VERSION $GRID_HOME/crs/script/mount-dbfs.sh 
VERSION=20160215

The following is an option for customers that have removed the dos2unix tool:
1. vi /tmp/mount-dbfs.sh
2. :set ff=unix
3. :wq
4. repeat for /tmp/mount-dbfs.conf

May 14, 2017

DBFS and XAG for Goldengate P4

Filed under: 12c,GoldenGate — mdinh @ 3:13 pm

I encountered a situation where I was not able to start dbfs_mount to save my life.

2 people spent over 5 hours and was not able to resolve the issue.

1 person suggested to to restart clusterware because it looks like fuse group was added to oracle after oracle clusterware was up.

During configuration:
kernel-devel was missing, /etc/rc.modules did not exists, oracle was not part of fuse group

# rpm -qa |egrep 'fuse|kernel-devel'
# cat /etc/rc.modules
cat: /etc/rc.modules: No such file or directory
# grep fuse /etc/group
fuse:x:993:

Corrections made:

# cat /etc/rc.modules
/sbin/modprobe fuse

# grep fuse /etc/group
fuse:x:993:oracle
$ crsctl start res dbfs_mount
CRS-2672: Attempting to start 'dbfs_mount' on 'hawk1'
CRS-2672: Attempting to start 'dbfs_mount' on 'hawk2'
CRS-2674: Start of 'dbfs_mount' on 'hawk2' failed
CRS-2679: Attempting to clean 'dbfs_mount' on 'hawk2'
CRS-2674: Start of 'dbfs_mount' on 'hawk1' failed
CRS-2679: Attempting to clean 'dbfs_mount' on 'hawk1'
CRS-2681: Clean of 'dbfs_mount' on 'hawk1' succeeded
CRS-2681: Clean of 'dbfs_mount' on 'hawk2' succeeded
CRS-4000: Command Start failed, or completed with errors.

$ crsctl stat res dbfs_mount
NAME=dbfs_mount
TYPE=local_resource
TARGET=ONLINE , ONLINE
STATE=OFFLINE, OFFLINE

After restarting clusterware:

Note: this was started by root which I don't think is a good idea.
# crsctl start resource dbfs_mount
CRS-2672: Attempting to start 'dbfs_mount' on 'hawk1'
CRS-2676: Start of 'dbfs_mount' on 'hawk1' succeeded

Should have been started from grid user
$ crsctl start res dbfs_mount
CRS-2672: Attempting to start 'dbfs_mount' on 'hawk2'
CRS-2672: Attempting to start 'dbfs_mount' on 'hawk1'
CRS-2676: Start of 'dbfs_mount' on 'hawk1' succeeded
CRS-2676: Start of 'dbfs_mount' on 'hawk2' succeeded

$ crsctl stat res dbfs_mount
NAME=dbfs_mount
TYPE=local_resource
TARGET=ONLINE             , ONLINE
STATE=ONLINE on hawk1, ONLINE on hawk2

It really bothered me how this information was missed and where was it documented.

After hours of research, the information is not consistently documented.
Not Found:
White Paper: How To Setup 12c DBFS FileSystem. (Doc ID 1938421.1)

Found:
Configuring DBFS on Oracle Exadata Database Machine (Doc ID 1054431.1)

To pick up the additional group (fuse) membership for the oracle user on Linux or the workaround above on Solaris, Clusterware must be restarted.
For example, to restart Clusterware on all nodes at the same time (non-rolling), you can use the following commands as root:

Hopefully, you won’t have a much fun as I did.

May 9, 2017

DBFS and XAG for Goldengate P3

Filed under: 12c,GoldenGate — mdinh @ 12:47 am

Start Pump Extract at source failed as shown below.

INFO OGG-00993 Oracle GoldenGate Capture for Oracle, p_test.prm: EXTRACT P_TEST started.
ERROR OGG-01224 Oracle GoldenGate Capture for Oracle, p_test.prm: TCP/IP error 111 (Connection refused), endpoint: ggvip_hawk.local:7809.
ERROR OGG-01668 Oracle GoldenGate Capture for Oracle, p_test.prm: PROCESS ABENDING.

Hypothesis is ports are not opened from source to target and let’s verify configuration.

VIP name and address from DNS.

Name:    ggvip_hawk.local
Address: 10.10.10.201

Recall this is how goldengate instance was created.

# $GRID_HOME/bin/agctl add goldengate gg_xx \
--instance_type target \
--oracle_home /u01/app/oracle/product/12.1.0/db_1 \
--nodes hawk1,hawk2 \
--network 1 --ip 10.10.10.201 \
--user ggsuser --group dba \
--filesystems dbfs_mount \
--gg_home /u03/gg/12.2.0 

From Goldengate instance gg_xx,
Application VIP (gg_xx-vip) is created
using address (10.10.10.201).

Check for xag resource.

ggsuser@hawk1 ~ $ $GRID_HOME/bin/crsctl stat res -t|grep -A2 xag
xag.gg_xx-vip.vip
      1        ONLINE  ONLINE       hawk1                STABLE
xag.gg_xx.goldengate
      1        ONLINE  ONLINE       hawk1                STABLE

Verify Application VIP address assigned.

ggsuser@hawk1 /u03/app/gg/12.2.0 
$ $GRID_HOME/bin/crsctl stat res xag.gg_xx-vip.vip -p|grep USR_ORA_VIP
GEN_USR_ORA_VIP=
USR_ORA_VIP=10.10.10.201
ggsuser@hawk1 /u03/app/gg/12.2.0 
$ nslookup 10.10.10.201
Server:		10.80.107.101
Address:	10.80.107.101#53

name = ggvip_hawk.local
ggsuser@hawk1 /u03/app/gg/12.2.0 $ 
$GRID_HOME/bin/crsctl stat res xag.gg_xx-vip.vip -p
NAME=xag.gg_xx-vip.vip
TYPE=app.appvipx.type
ACL=owner:root:rwx,pgrp:root:r-x,other::r--,group:dba:r-x,user:ggsuser:r-x
ACTIONS=
ACTION_SCRIPT=
ACTION_TIMEOUT=60
ACTIVE_PLACEMENT=0
AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX%
APPSVIP_FAILBACK=0
AUTO_START=restore
CARDINALITY=1
CHECK_INTERVAL=1
CHECK_TIMEOUT=0
CLEAN_TIMEOUT=60
DEGREE=1
DELETE_TIMEOUT=60
DESCRIPTION=Application VIP
ENABLED=1
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
GEN_USR_ORA_STATIC_VIP=
GEN_USR_ORA_VIP=
HOSTING_MEMBERS=
INSTANCE_FAILOVER=1
INTERMEDIATE_TIMEOUT=0
LOAD=1
LOGGING_LEVEL=1
MODIFY_TIMEOUT=60
NLS_LANG=
OFFLINE_CHECK_INTERVAL=0
PLACEMENT=balanced
RELOCATE_BY_DEPENDENCY=1
RESTART_ATTEMPTS=0
SCRIPT_TIMEOUT=60
SERVER_CATEGORY=
SERVER_POOLS=*
START_CONCURRENCY=0
START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network)
START_TIMEOUT=0
STOP_CONCURRENCY=0
STOP_DEPENDENCIES=hard(ora.net1.network)
STOP_TIMEOUT=0
TYPE_VERSION=1.1
UPTIME_THRESHOLD=7d
USER_WORKLOAD=no
USE_STICKINESS=0
USR_ORA_ENV=
USR_ORA_VIP=10.10.10.201
VERSION=12.1.0.1.0

May 6, 2017

DBFS and XAG for Goldengate P2

Filed under: 12c,GoldenGate — mdinh @ 4:26 pm

In order to use agctl commands, we need to know goldengate instance_name.

Unfortunately, agctl does not work the same way as srvctl where it’s possible to determine what is configured.

ggsuser@hawk1 ~ $ $ORACLE_HOME/bin/srvctl config database
DBFS

ggsuser@hawk1 ~ $ $GRID_HOME/bin/agctl config goldengate
XAG-212: Instance '' is not yet registered.
ggsuser@hawk1 ~ $ 

How do we find out what the goldengate instance name is? IFF XAG is configured, then grep for it.

ggsuser@hawk1 ~ $ $GRID_HOME/bin/crsctl stat res -t|grep -A2 xag
xag.gg_xx-vip.vip
      1        ONLINE  ONLINE       hawk1                STABLE
xag.gg_xx.goldengate
      1        ONLINE  ONLINE       hawk1                STABLE
--------------------------------------------------------------------------------

ggsuser@hawk1 ~ $ $GRID_HOME/bin/agctl status goldengate gg_xx
Goldengate  instance 'gg_xx' is running on hawk1
ggsuser@hawk1 ~ $ 

Some other useful commands to gather configurations info.

ggsuser@hawk1 ~ $ $GRID_HOME/bin/crsctl stat res|grep xag
NAME=xag.gg_xx-vip.vip
NAME=xag.gg_xx.goldengate
TYPE=xag.goldengate.type
ggsuser@hawk1 ~ $ 

ggsuser@hawk1 ~ $ $GRID_HOME/bin/crsctl stat res|grep -i type|sort -u
TYPE=app.appvipx.type
TYPE=local_resource
TYPE=ora.asm.type
TYPE=ora.cluster_vip_net1.type
TYPE=ora.cvu.type
TYPE=ora.database.type
TYPE=ora.diskgroup.type
TYPE=ora.listener.type
TYPE=ora.mgmtdb.type
TYPE=ora.mgmtlsnr.type
TYPE=ora.network.type
TYPE=ora.oc4j.type
TYPE=ora.ons.type
TYPE=ora.scan_listener.type
TYPE=ora.scan_vip.type
TYPE=xag.goldengate.type

ggsuser@hawk1 ~ $ $GRID_HOME/bin/crsctl stat res -w "TYPE = xag.goldengate.type" -p
NAME=xag.gg_xx.goldengate
TYPE=xag.goldengate.type
ACL=owner:ggsuser:rwx,pgrp:dba:r-x,other::r--
ACTIONS=
ACTION_SCRIPT=%CRS_HOME%/bin/aggoldengateas
ACTION_TIMEOUT=60
ACTIVE_PLACEMENT=0
AGENT_FILENAME=%CRS_HOME%/bin/scriptagent
AUTO_START=restore
CARDINALITY=1
CHECK_INTERVAL=30
CHECK_TIMEOUT=0
CLEAN_TIMEOUT=60
CRITICAL_EXTRACTS=
CRITICAL_REPLICATS=
CRS_ATTRIBUTES=
DATABASES= (No DB dependencies - User Exits)
DATAGUARD_AUTOSTART=no
DB_SERVICES=
DEGREE=1
DELETE_TIMEOUT=60
DESCRIPTION="Oracle GoldenGate Clusterware Resource"
ENABLED=1
ENVIRONMENT_VARS=
FAILOVER_DELAY=0
FAILURE_INTERVAL=600
FAILURE_THRESHOLD=5
FILESYSTEMS=dbfs_mount
GG_HOME=/u03/gg/12.2.0
GG_INSTANCE_TYPE=target
HOSTING_MEMBERS=hawk1 hawk2
INSTANCE_FAILOVER=1
INTERMEDIATE_TIMEOUT=0
JAGENT_AUTOSTART=no
LOAD=1
LOGGING_LEVEL=1
MODIFY_TIMEOUT=60
MONITOR_EXTRACTS=
MONITOR_REPLICATS=
OFFLINE_CHECK_INTERVAL=0
ORACLE_CLIENT_HOME=
ORACLE_HOME=/u01/app/oracle/product/12.1.0/db_1
PLACEMENT=restricted
RELOCATE_BY_DEPENDENCY=1
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=60
SERVER_CATEGORY=
SERVER_POOLS=
START_CONCURRENCY=0
START_DEPENDENCIES=hard(xag.gg_xx-vip.vip,dbfs_mount) pullup(xag.gg_xx-vip.vip,dbfs_mount)
START_TIMEOUT=300
STOP_CONCURRENCY=0
STOP_DEPENDENCIES=hard(xag.gg_xx-vip.vip,intermediate:dbfs_mount)
STOP_TIMEOUT=300
UPTIME_THRESHOLD=10m
USER_WORKLOAD=no
USE_STICKINESS=0
VERSION=2
VIP_CREATED=1
VIP_NAME=xag.gg_xx-vip.vip
ggsuser@hawk1 ~ $ 

You might be thinking, if there are no dependencies for database, then why is it referencing Database Home?

ggsuser@hawk1 ::/u03/gg/12.2.0
$ ldd ggsci 
	linux-vdso.so.1 =>  (0x00007ffcaa8ff000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f6a02c5b000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f6a02a56000)
	libgglog.so => /u03/gg/12.2.0/./libgglog.so (0x00007f6a02630000)
	libggrepo.so => /u03/gg/12.2.0/./libggrepo.so (0x00007f6a023ba000)
	libdb-6.1.so => /u03/gg/12.2.0/./libdb-6.1.so (0x00007f6a01fd5000)
	libggperf.so => /u03/gg/12.2.0/./libggperf.so (0x00007f6a01da5000)
	libggparam.so => /u03/gg/12.2.0/./libggparam.so (0x00007f6a00c8d000)
	libicui18n.so.48 => /u03/gg/12.2.0/./libicui18n.so.48 (0x00007f6a0089d000)
	libicuuc.so.48 => /u03/gg/12.2.0/./libicuuc.so.48 (0x00007f6a0051c000)
	libicudata.so.48 => /u03/gg/12.2.0/./libicudata.so.48 (0x00007f69fed57000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f69feb3a000)
	libxerces-c.so.28 => /u03/gg/12.2.0/./libxerces-c.so.28 (0x00007f69fe574000)
	libantlr3c.so => /u03/gg/12.2.0/./libantlr3c.so (0x00007f69fe35b000)
	libnnz12.so => /u01/app/oracle/product/12.1.0/db_1/lib/libnnz12.so (0x00007f69fdc36000)
	libclntsh.so.12.1 => /u01/app/oracle/product/12.1.0/db_1/lib/libclntsh.so.12.1 (0x00007f69fabbf000)
	libons.so => /u01/app/oracle/product/12.1.0/db_1/lib/libons.so (0x00007f69fa97a000)
	libclntshcore.so.12.1 => /u01/app/oracle/product/12.1.0/db_1/lib/libclntshcore.so.12.1 (0x00007f69fa406000)
	libggnnzitp.so => /u03/gg/12.2.0/./libggnnzitp.so (0x00007f69f9922000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f69f9620000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f69f925e000)
	/lib64/ld-linux-x86-64.so.2 (0x00005624a8090000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f69f8f56000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f69f8d3f000)
	libmql1.so => /u01/app/oracle/product/12.1.0/db_1/lib/libmql1.so (0x00007f69f8ac8000)
	libipc1.so => /u01/app/oracle/product/12.1.0/db_1/lib/libipc1.so (0x00007f69f8750000)
	libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f69f8537000)
	libaio.so.1 => /lib64/libaio.so.1 (0x00007f69f8335000)

Would’t be much better if Goldengate installation is self contained without having to download and install 2 components!

May 5, 2017

DBFS and XAG for Goldengate P1

Filed under: 12c,GoldenGate — mdinh @ 9:53 pm

What’s the difference between the 2 GoldenGate configurations below.

$ $GRID_HOME/bin/agctl config goldengate gg_xx

GoldenGate location is: /u03/gg/12.2.0
GoldenGate instance type is: target
Configured to run on Nodes: arrow1 arrow2
ORACLE_HOME location is: /u01/app/oracle/product/12.1.0/db_1
Databases needed: ora.emu1.db
File System resources needed: dbfs_mount
Extracts to monitor: 
Replicats to monitor: 
Critical extracts: 
Critical replicats: 
Autostart on DataGuard role transition to PRIMARY: no
Autostart JAgent: no

$ $GRID_HOME/bin/agctl config goldengate gg_xx

GoldenGate location is: /u03/gg/12.2.0
GoldenGate instance type is: target
Configured to run on Nodes: hawk1 hawk2
ORACLE_HOME location is: /u01/app/oracle/product/12.1.0/db_1
File System resources needed: dbfs_mount
Extracts to monitor: 
Replicats to monitor: 
Critical extracts: 
Critical replicats: 
Autostart on DataGuard role transition to PRIMARY: no
Autostart JAgent: no

Here are how they are added:

# $GRID_HOME/bin/agctl add goldengate gg_xx \
--instance_type target \
--oracle_home /u01/app/oracle/product/12.1.0/db_1 \
--nodes hawk1,hawk2 \
--network 1 --ip 10.10.10.101 \
--user ggsuser --group dba \
--filesystems dbfs_mount \
--gg_home /u03/gg/12.2.0 \
--databases ora.emu1.db

# $GRID_HOME/bin/agctl add goldengate gg_xx \
--instance_type target \
--oracle_home /u01/app/oracle/product/12.1.0/db_1 \
--nodes arrow1,arrow2 \
--network 1 --ip 10.10.10.201 \
--user ggsuser --group dba \
--filesystems dbfs_mount \
--gg_home /u03/gg/12.2.0 

One is actually source from Database and the other is target User Exits.
But, but, but –instance_type is target for both.
That’s right (meaning your observation is correct) – hehe
Bad implementation – don’t do that.

instance_type – OGG source or OGG target (source, target) (dual is bi-directional)

$XAG_HOME/bin/agctl add goldengate lax_ggate \
--gg_home /acfsmount/ggs112 \
--instance_type dual \
--nodes rac01,rac02 \
--vip_name lax-ggate1-vip \
--filesystems ora.dg_acfs.vg_acfs.acfs \
--databases ora.emu.db \
--oracle_home /u01/app/oracle/product/11.2.0.4/db_1 \
--monitor_extracts ELAX,PLAX_DEN \
--critical_extracts ELAX,PLAX_DEN \
--monitor_replicats RDEN_LAX \
--critical_replicats RDEN_LAX

Why agctl is called from $GRID_HOME in one and $XAG_HOME in another?

$ $GRID_HOME/bin/agctl query releaseversion
The Oracle Grid Infrastructure Agents release version is 3.1.0

$ $GRID_HOME/bin/agctl query deployment
The Oracle Grid Infrastructure Agents deployment is bundled

agctl from GRID_HOME is bundled with install, but older version.
agctl from XAG_HOME is standalone, downloaded and installed from

Oracle Grid Infrastructure Standalone Agents for Oracle Clusterware 11g Rel. 2, 12c Rel. 1 and 12c Rel. 2
http://www.oracle.com/technetwork/database/database-technologies/clusterware/downloads/xag-agents-downloads-3636484.html

April 16, 2017

Be Prepared When Using DBFS

Filed under: 12c,GoldenGate,oracle — mdinh @ 9:18 pm

Goldegate 12c implementation for 2-node RAC with XAG from Grid Home using DBFS.

We are facing situation where DBFS has 41G free while DBFS_TS has 1.5G free and having to resize BIGFILE tablespace.

Tablespace and DBFS size is 45G.

Basically, 45G tablespace is used to maintain 4G Goldengate files.

Based on steps followed from NOTE:1453350.1 – How to Determine what storage is used in a LOBSEGMENT and should it be shrunk / reorganized?

Next step is to Shrink a SECUREFILE LOB using Online Redefinition (DBMS_REDEFINITION)? (Document  1394613.1)

At this point, does not look pretty.

Here are all the research I have gone through.

NOTE:1150157.1 - List of Critical Patches Required For Oracle 11.2 DBFS and DBFS Client
BUG:13824914 - DBFS FREE SPACE IS NOT RECLAIMED AFTER DELETIONS.
BUG:12662040 - SECUREFILE LOB SEGMENT KEEPS GROWING IN CASE OF PLENTY OF FREE SPACE

How DBFS Reclaims Free Space After Files Are Deleted (Doc ID 1438356.1) 
NOTE:162345.1 - LOBS - Storage, Read-consistency and Rollback
NOTE:386341.1 - How to determine the actual size of the LOB segments and how to free the deleted/unused space above/below the HWM
--------------------------------------------------------------------------------
NOTE:1453350.1 - How to Determine what storage is used in a LOBSEGMENT and should it be shrunk / reorganized?
3) Determine if the LOBSEGMENT is a candidate for shrink / reorganization

The following query will show the extents allocated for the LOBSEGMENT

SELECT BYTES, COUNT(*) FROM DBA_EXTENTS WHERE SEGMENT_NAME = ''  GROUP BY BYTES ORDER BY 2;

If NON Data Bytes is  one or more of the extent sizes from this query ...
     then this segment is a candiate for a shrink / reorganization as this process will likely be able to reduce the size of the LOBSEGMENT by at least one extent

4) If #3 is true ... then the method in the following note may be used to shrink / reorganize the lob segment

How to Shrink a SECUREFILE LOB using Online Redefinition (DBMS_REDEFINITION)? (Document  1394613.1)
--------------------------------------------------------------------------------
NOTE:66431.1 - LOBS - Storage, Redo and Performance Issues

April 1, 2017

Goldengate 12c Troubleshooting XAGENABLE

Filed under: 12c,GoldenGate — mdinh @ 2:52 pm

Environment:
Oracle RAC with DBFS running as oracle
Goldengate12c running as ggsuser using DBFS and XAGENABLE
Monitoring is running as monitor

ERROR: Transparent Integration with XAG is Enabled but CRS/XAG is not Available (Doc ID 2240440.1)

Modify GLOBALS per Doc ID 2240440.1

$ cat GLOBALS

setenv (GRID_HOME='/u01/app/12.1.0/grid')
XAGENABLE

$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 12.2.0.1.160823 OGGCORE_12.2.0.1.0OGGBP_PLATFORMS_160805.1058_FBO
Linux, x64, 64bit (optimized), Oracle 12c on Aug  5 2016 23:35:08
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2016, Oracle and/or its affiliates. All rights reserved.



2017-03-31 11:24:06  INFO    OGG-02095  Successfully set environment variable GRID_HOME=/u01/app/12.1.0/grid.

*** Could not open error log ggserr.log (error 13,Permission denied) ***

2017-03-31 11:24:06  INFO    OGG-02095  Successfully set environment variable GRID_HOME=/u01/app/12.1.0/grid.
ERROR: Transparent Integration with XAG is enabled but CRS/XAG is not available.

chmod 775 ggserr.log and set umask 002 in .bash_profile

$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 12.2.0.1.160823 OGGCORE_12.2.0.1.0OGGBP_PLATFORMS_160805.1058_FBO
Linux, x64, 64bit (optimized), Oracle 12c on Aug  5 2016 23:35:08
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2016, Oracle and/or its affiliates. All rights reserved.



2017-03-31 11:27:58  INFO    OGG-02095  Successfully set environment variable GRID_HOME=/u01/app/12.1.0/grid.

2017-03-31 11:27:58  INFO    OGG-02095  Successfully set environment variable GRID_HOME=/u01/app/12.1.0/grid.
ERROR: Transparent Integration with XAG is enabled but CRS/XAG is not available.

$ export GRID_HOME=/u01/app/12.1.0/grid
$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 12.2.0.1.160823 OGGCORE_12.2.0.1.0OGGBP_PLATFORMS_160805.1058_FBO
Linux, x64, 64bit (optimized), Oracle 12c on Aug  5 2016 23:35:08
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2016, Oracle and/or its affiliates. All rights reserved.



2017-03-31 11:29:40  INFO    OGG-02095  Successfully set environment variable GRID_HOME=/u01/app/12.1.0/grid.

2017-03-31 11:29:40  INFO    OGG-02095  Successfully set environment variable GRID_HOME=/u01/app/12.1.0/grid.

GGSCI 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt   XAG                           


MANAGER     RUNNING                                                  MANAGED/xag.ggs.goldengate  


GGSCI 2> exit

See anything wrong so far? Extracts are running and not shown.

As it turns out from strace of ggsci – no RW permission.
open(“./dirchk/*.cpe”, O_RDWR) = -1 EACCES (Permission denied)

Remembered my post about XAG many moons ago about role separation and what a PITA it was when implemented incorrectly?

Next Page »

Create a free website or blog at WordPress.com.