Thinking Out Loud

May 18, 2018

Using GoldenGate LogDump To Find Bad Data

Filed under: GoldenGate — mdinh @ 4:38 am

GoldenGate Primary Extract from source database captured data without any issues;

however, target was not able to consume the data since GoldenGate process would ABEND.

Unfortunately, I cannot provide all the details but high level.

Logdump 2433 >pos 0
+++ Starting with GoldenGate 12.2 TDR – Table Definition Record is in trail
+++ This will provide metadata for the table

Reading forward from RBA 0 
Logdump 2434 >SCANFORMETADATA
___________________________________________________________________ 
Hdr-Ind    :     E  (x45)     Partition  :     .  (x00)  
UndoFlag   :     .  (x00)     BeforeAfter:     A  (x41)  
RecLength  :    86  (x0056)   IO Time    : 2018/03/25 18:24:23.307.797   
IOType     :   170  (xaa)     OrigNode   :     1  (x01) 
TransInd   :     .  (x01)     FormatType :     R  (x52) 
SyskeyLen  :     0  (x00)     Incomplete :     .  (x00) 
DDR/TDR Idx:   (001, 000)     AuditPos   : 3277290592 
Continued  :     N  (x00)     RecCount   :     1  (x01) 

2018/03/25 18:24:23.307.797 Metadata             Len 86 RBA 1689 
Name:  
*
DDR Version: 1
Database type: ORACLE
Character set ID: CESU-8
National character set ID: UTF-16
Locale: neutral
Case sensitivity: 14 14 14 14 14 14 14 14 14 14 14 14 11 14 14 14
TimeZone: GMT-05:00
Global name: DBNAME
* 
Logdump 2435 >n
___________________________________________________________________ 
Hdr-Ind    :     E  (x45)     Partition  :     .  (x00)  
UndoFlag   :     .  (x00)     BeforeAfter:     A  (x41)  
RecLength  :   561  (x0231)   IO Time    : 2018/03/25 18:28:16.317.879   
IOType     :   170  (xaa)     OrigNode   :     2  (x02) 
TransInd   :     .  (x01)     FormatType :     R  (x52) 
SyskeyLen  :     0  (x00)     Incomplete :     .  (x00) 
DDR/TDR Idx:   (001, 001)     AuditPos   : 3352410292 
Continued  :     N  (x00)     RecCount   :     1  (x01) 

2018/03/25 18:28:16.317.879 Metadata             Len 561 RBA 1826 
Name: SCHEMA.TABLE 
*
 1)Name          2)Data Type        3)External Length  4)Fetch Offset      5)Scale         6)Level
 7)Null          8)Bump if Odd      9)Internal Length 10)Binary Length    11)Table Length 12)Most Sig DT
13)Least Sig DT 14)High Precision  15)Low Precision   16)Elementary Item  17)Occurs       18)Key Column
19)Sub DataType 20)Native DataType 21)Character Set   22)Character Length 23)LOB Type     24)Partial Type
*
TDR version: 1
Definition for table SCHEMA.TABLE
Record Length: 4298
Columns: 7
ID           64     50        0  0  0 1 0     50     50     50 0 0 0 0 1    0 1   2    2       -1      0 0 0
A            B      C         D  E  F G H     I      J      K  L M N O P    Q R

Position 18)Key Column identify column is Primary Key; hence, ID a primary key column.
I labeled each column using the alphabet for reference and R is the 18th letter of the alphabet.

Even though ID is the first column of the table, GoldenGate offset starts with 0

SQL> desc SCHEMA.TABLE
 Name                                      Null?    Type
 ----------------------------------------- -------- ----------------------------
0 ID                                        NOT NULL NUMBER
5 TEXT                                               VARCHAR2(4000)

SQL> r
  1  select b.column_name,a.constraint_type
  2  from dba_constraints a, dba_cons_columns b
  3  where a.table_name = b.table_name
  4  and a.constraint_name=b.constraint_name
  5  and a.constraint_type = 'P'
  6  and a.table_name='TABLE'
  7  and a.owner='SCHEMA'
  8*

COLUMN_NAME                    C
------------------------------ -
ID                             P

Logdump 2500 >n
___________________________________________________________________ 
Hdr-Ind    :     E  (x45)     Partition  :     .  (x0c)  
UndoFlag   :     .  (x00)     BeforeAfter:     A  (x41)  
RecLength  :   885  (x0375)   IO Time    : 2018/05/10 14:16:43.000.514   
IOType     :    15  (x0f)     OrigNode   :   255  (xff) 
TransInd   :     .  (x02)     FormatType :     R  (x52) 
SyskeyLen  :     0  (x00)     Incomplete :     .  (x00) 
AuditRBA   :      36261       AuditPos   : 2145515112 
Continued  :     N  (x00)     RecCount   :     1  (x01) 

2018/05/10 14:16:43.000.514 FieldComp            Len   885 RBA 88598006 
Name: SCHEMA.TABLE  (TDR Index: 1) 
After  Image:                                        ....................
 0000 000c 0000 0008 3239 3533 3432 3936 0001 0009 | ........29534296....  
 0000 0005 3937 3932 3600 0200 0500 0000 0131 0003 | ....................
 0005 0000 0001 3600 0400 0b00 0000 0731 3738 3337 | ....................
 3939 0005 0325 0000 0321 456e 6a6f 7920 796f 7572 | ....................  
 2073 7461 7920 696e 206f 7572 2068 6f74 656c 7320 | ....................
 7769 7468 2074 6865 206d 6f73 7420 6469 7363 6f75 | ....................  
 6e74 6564 2072 6174                               | ....................
Column     0 (x0000), Len    12 (x000c)  
 0000 0008 3239 3533 3432 3936                     | ....29534296 --- PRIMARY KEY VALUE
Column     1 (x0001), Len     9 (x0009)  
 0000 0005 3937 3932 36                            | ....97926  
Column     2 (x0002), Len     5 (x0005)  
 0000 0001 31                                      | ....1  
Column     3 (x0003), Len     5 (x0005)  
 0000 0001 36                                      | ....6  

SQL> select id, substr(TEXT,1,30) txt, vsize(TEXT), length(TEXT) from SCHEMA.TABLE where ID in (29534296);

        ID LTXT                           VSIZE(LTEXT) LENGTH(LTEXT)
---------- ------------------------------ ------------ -------------
  29534296 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx          801           800

SQL>

For database, NLS_LENGTH_SEMANTICS=BYTE; hence, 800 characters should equal 800 bytes. 
However, one character is most likely a multi-byte character which target cannot consume 
as it is not able to handle multi-byte.

		
Advertisements

May 15, 2018

Goldengate REPORTING P2

Filed under: GoldenGate — mdinh @ 2:35 am

Previous post for Goldengate REPORTING

Goldengate reporting has to be the least implemented functionality until it is needed and often in hindsight.

Here is an example of what I would normally implement.

STATOPTIONS RESETREPORTSTATS
REPORT AT 00:01
REPORTROLLOVER AT 00:01
REPORTCOUNT EVERY 15 MINUTES, RATE
DISCARDROLLOVER AT 00:01 ON SUNDAY

If business only cares about monthly data, then would not make sense to collect daily nor would it make sense for business to request such.

May 5, 2018

DBFS Nightmare

Filed under: 12c,GoldenGate — mdinh @ 4:19 pm
====================================================================================================
How to cleanup DBFS tablespace after removing files at DBFS filesystem (Doc ID 2331565.1)	
====================================================================================================
High level overview of the DBMS_DBFS_SFS.REORGANIZEFS procedure:
----------------------------------------------------------------------------------------------------
1) Create a NEW tablespace
2) Create a temporary filesystem with dbms_dbfs_sfs.createFilesystem in the new tablespace.
3) Run dbms_dbfs_sfs.reorganizeFS -->>
   EXEC DBMS_DBFS_SFS.REORGANIZEFS(SRCSTORE=>'FS_FS1', DSTSTORE=>'FS_TMP_FS');
4) The dbfs data is now in the smaller NEW tablespace.
5) Drop the temporay filesystem with dbms_dbfs_sfs.dropFilesystem
6) The OLD original tablespace is empty now.
----------------------------------------------------------------------------------------------------
To reuse the same tablespace again please follow the below steps
----------------------------------------------------------------------------------------------------
7) Create a temporary filesystem with dbms_dbfs_sfs.createFilesystem in the OLD ORIGINAL tablespace.
8) Run dbms_dbfs_sfs.reorganizeFS
9) The dbfs data is now in the smaller the ORIGINAL tablespace.
10) Drop the temporay filesystem with dbms_dbfs_sfs.dropFilesystem
11) The NEW small tablespace is empty now and can be dropped.

====================================================================================================
DBFS tablespace keep growing not using expired blocks (Doc ID 2327299.1)	
====================================================================================================
----------------------------------------------------------------------------------------------------
First perform the DBFS re-org to cleanup the tablespace after removing the files at file system level by following below document
----------------------------------------------------------------------------------------------------
How to cleanup DBFS tablespace after removing files at DBFS filesystem (Doc ID 2331565.1)

----------------------------------------------------------------------------------------------------
After the re-org, set the  below parameter to reuse the expired blocks
----------------------------------------------------------------------------------------------------
ALTER SYSTEM SET "_ENABLE_SPACE_PREALLOCATION" = 0;

====================================================================================================
ORA-1654 - DBFS FREE SPACE NOT RECLAIMED AFTER CLEARING FILES (Doc ID 1948305.1)	
====================================================================================================
----------------------------------------------------------------------------------------------------
a) Check Lob retention setting:
----------------------------------------------------------------------------------------------------
  ex. SQL> select 
  table_name,retention_type,retention_value,retention,securefile from dba_lobs 
  where table_name = 'T_FS1';

----------------------------------------------------------------------------------------------------  
b) Change LOB retention to none.
----------------------------------------------------------------------------------------------------
  ex. SQL> alter table DBFS_USER.T_FS1 modify lob (FILEDATA) (retention none);

After countless discussions, team found GOLD.

12.1 Shrinking and Reorganizing DBFS Filesystems

STEPS:

sqlplus /as sysdba
create bigfile tablespace DBFS_NEW datafile size 8G autoextend on next 1G maxsize 70G;
grant dba to dbfs_user;
alter user dbfs_user default role all;
exit

+++ REORGANIZE DBFS FILESYSTEM FS1 IN TABLESPACE DBFS_TS INTO A NEW TABLESPACE DBFS_NEW, 
+++ USING A TEMPORARY FILESYSTEM NAMED TMP_FS, WHERE ALL FILESYSTEMS BELONG TO DATABASE USER DBFS_USER
cd $ORACLE_HOME/rdbms/admin
sqlplus dbfs_user
@dbfs_create_filesystem DBFS_NEW TMP_FS
exec dbms_dbfs_sfs.reorganizefs('FS1','TMP_FS');
@dbfs_drop_filesystem TMP_FS
purge user_recyclebin;
exit

sqlplus / aa sysdba 
revoke dba from dbfs_user;
select count(*) from dba_extents where tablespace_name='DBFS_TS'; 
select count(*) from dba_extents where tablespace_name='DBFS_NEW'; 
-- DROP TABLESPACE HAVING ZERO EXTENTS
-- BE CAREFUL IN CASE USER AND DBFS ARE USING SAME TABLESPACE
Example:
create user dbfs_user identified by **** default tablespace dbfs_ts quota unlimited on dbfs_ts;
@$ORACLE_HOME/rdbms/admin/dbfs_create_filesystem.sql dbfs_ts FS1

TRUE CONFESSION:

I created an imperfect plan at first that might have saved me from a major catastrophe.

Create DBFS_TMP in a separate Disk Group since I was concern existing may not have enough storage.
Reorg from DBFS_TS to DBFS_TMP (bad idea to name tablespace as DBFS_TMP as LOB Segments migrated here).
Reorg from DBFS_TMP back to DBFS_TS.
Drop tablespace DBFS_TMP.

There’s 29G difference between old and new. How much if this space will be reclaimable due to HWM in data file?

Performing reorg twice is really not an option in prod.

RESULTS:

SEGMENT_NAME                   TABLESPACE_NAME                        MB    EXTENTS
------------------------------ ------------------------------ ---------- ----------
LOB_SFS$_FST_1                 DBFS_TS                        30647.1875       1851
LOB_SFS$_FST_6225924           DBFS_NEW                         1025.125        130

WARNINGS:

This has only been tested in a vacuum (non-prod env w very low activities) – YMMV.

April 18, 2018

Skip Goldengate Replicat Transaction

Filed under: GoldenGate — mdinh @ 12:22 am

Overview:

It’s easy to skip replicat transaction; however, there is a requirement to understand the result.

In the example provided, DELETE from PARENT table was not successful because there are ROWS from CHILD table.

Skip replicat transactions resulted in difference of 15273 rows for PARENT table between source and target.

In this case, there were 3 rows from 1 CHILD table causing the issue.

I know what you are thinking – Why was DELETE rows from CHILD table not replicate to begin with?

May be someone did the same – skip transaction and not synchronize the data?

That’s a story for another time. The problem is solved but not the cause.

====================================================================================================
ORA-02292: integrity constraint (OWNER.MARY_JOE_FK) violated - child record found (status = 2292). DELETE FROM "OWNER"."T_JOE"  WHERE "JOENUMMER" = :b0.
====================================================================================================

GGATE@SQL> select JOENUMMER from NEWS.T_MARY minus select JOENUMMER from NEWS.T_MARY@source;

JOENUMMER
------------
    50000402
    50000412
    50000422

GGATE@SQL> select JOENUMMER from NEWS.T_MARY@source minus select JOENUMMER from OWNER.T_MARY;

no rows selected

GGATE@SQL> 

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

GGATE@SQL> select JOENUMMER  from OWNER.T_MARY@source where JOENUMMER in (50000402,50000412,50000422);

no rows selected

GGATE@SQL> select JOENUMMER  from OWNER.T_MARY where JOENUMMER in (50000402,50000412,50000422);

JOENUMMER 
------------
    50000402
    50000412
    50000422

GGATE@SQL> delete from OWNER.T_MARY where JOENUMMER in (50000402,50000412,50000422);

3 rows deleted.

GGATE@SQL> commit;

Commit complete.

GGATE@SQL> 
====================================================================================================
Oracle GoldenGate Command Interpreter for Oracle
Version 11.2.1.0.15 17640173 OGGCORE_11.2.1.0.0OGGBP_PLATFORMS_131101.0605.2_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Nov 19 2013 03:18:45

Copyright (C) 1995, 2013, Oracle and/or its affiliates. All rights reserved.
====================================================================================================
ORA-02292: integrity constraint (OWNER.MARY_JOE_FK) violated - child record found (status = 2292). DELETE FROM "OWNER"."T_JOE"  WHERE "JOENUMMER" = :b0.
====================================================================================================

+++ SKIPTRANSACTION
GGSCI> start replicat REP1 SKIPTRANSACTION

+++ REVIEW PRM
[gguser]$ grep -i discard rep1.prm
--REPERROR (DEFAULT, DISCARD)
REPERROR (-1, DISCARD)
REPERROR (2291, DISCARD) 
DISCARDFILE ./discard/rep1.discard append, MEGABYTES 1024
DISCARDROLLOVER AT 00:01
[gguser]$ 

+++ REVIEW SKIPPING FROM DISCARD
[gguser]$ grep -c "Skipping delete from OWNER.T_JOE" rep1.discard
15276

[gguser]$ grep -A2 "Skipping delete from OWNER.T_JOE" ./discard/rep1.discard|head
Skipping delete from OWNER.T_JOE at seqno 4475 rba 87850906
*
JOENUMMER = 1
--
Skipping delete from OWNER.T_JOE at seqno 4475 rba 87851339
*
JOENUMMER = 2
--
Skipping delete from OWNER.T_JOE at seqno 4475 rba 87851735
*
[gguser@viz-cp-dc1-p11 oracle]$ grep -A2 "Skipping delete from OWNER.T_JOE" ./discard/rep1.discard|tail
*
JOENUMMER = 50093291
--
Skipping delete from OWNER.T_JOE at seqno 4475 rba 94033367
*
JOENUMMER = 50094681
--
Skipping delete from OWNER.T_JOE at seqno 4475 rba 94033767
*
JOENUMMER = 50094741

+++ REVIEW RBA FROM DISCARD
[gguser]$ grep rba rep1.discard|head -1
Aborting transaction on ./dirdat/nd beginning at seqno 4475 rba 87850906

[gguser]$ grep rba rep1.discard|tail -1
Skipping delete from OWNER.T_JOE at seqno 4475 rba 94033767
[gguser]$ 

+++ NOTICE MATCH WITH LOGDUMP
Logdump 23 >scanforendtrans
End of Transaction found at RBA 94033767 

====================================================================================================
GATHER DATA
====================================================================================================

GGATE@SQL> r
1 select count(*) from
2 (
3 (select JOENUMMER from OWNER.T_JOE minus select JOENUMMER from OWNER.T_JOE@source)
4 union all
5 (select JOENUMMER from OWNER.T_JOE@source minus select JOENUMMER from OWNER.T_JOE)
6 )
7*

COUNT(*)
———-
15273

GGATE@SQL>

+++ CREATE TEMPORARY TABLE

GGATE@SQL> create table T_JOE_DEL as select JOENUMMER from OWNER.T_JOE minus select JOENUMMER from OWNER.T_JOE@source;

+++ REVIEW DATA FROM TEMPORARY TABLE TO COMPARE WITH DISCARD

GGATE@SQL> r
1 select * from (
2 select JOENUMMER from T_JOE_DEL order by 1 asc
3* ) where rownum <11

JOE
————
1
2
3
21
23
24
25
26
27
28

10 rows selected.

GGATE@SQL>

GGATE@SQL> r
1 select * from (
2 select JOE from T_JOE_DEL order by 1 desc
3* ) where rownum <11

JOE
————
50094741
50094681
50093291
50093221
50093191
50093101
50092851
50092791
50092781
50092741

10 rows selected.

GGATE@SQL>

====================================================================================================
CORRECT DATA
====================================================================================================

GGATE@SQL> delete from OWNER.T_JOE where JOENUMMER in (select JOENUMMER from T_JOE_DEL);

15273 rows deleted.

GGATE@SQL> commit;

Commit complete.

====================================================================================================
VERIFY ROW COUNT
====================================================================================================

+++ USING COUNT MAY NOT BE THE BEST OPTION.
GGATE@SQL> select count(*) from OWNER.T_JOE;

  COUNT(*)
----------
      9939

GGATE@SQL> select count(*) from OWNER.T_JOE@source;

  COUNT(*)
----------
      9939

GGATE@SQL> 

====================================================================================================
REVIEW REPORT FILE
====================================================================================================

[gguser]$ grep SKIPTRANSACTION REP1*.rpt
rep1.rpt:2018-04-17 12:15:15  INFO    OGG-01370  User requested START SKIPTRANSACTION. The current transaction will be skipped. Transaction ID 22.30.1923599, position Seqno 4475, RBA 87850906.

[gguser]$ grep -i skip ggserr.log
2018-04-17 12:15:14  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (gguser): start replicat rep1 SKIPTRANSACTION.
2018-04-17 12:15:14  INFO    OGG-00963  Oracle GoldenGate Manager for Oracle, mgr.prm:  Command received from GGSCI on host 10.232.135.44:33310 (START REPLICAT rep1 SKIPTRANSACTION).
2018-04-17 12:15:15  INFO    OGG-01370  Oracle GoldenGate Delivery for Oracle, rep1.prm:  User requested START SKIPTRANSACTION. The current transaction will be skipped. Transaction ID 22.30.1923599, position Seqno 4475, RBA 87850906.
[gguser]$ 

====================================================================================================
LOGDUMP TO FIND END OF TRANSACTONS
====================================================================================================

Logdump 15 >open ./dirdat/nd004475
Current LogTrail is ./dirdat/nd004475 
Logdump 16 >detail on
Logdump 17 >fileheader detail
Logdump 18 >ghdr on
Logdump 19 >detail data
Logdump 20 >ggstoken detail
Logdump 21 >pos 87850906
Reading forward from RBA 87850906 
Logdump 22 >n
___________________________________________________________________ 
Hdr-Ind    :     E  (x45)     Partition  :     .  (x04)  
UndoFlag   :     .  (x00)     BeforeAfter:     B  (x42)  
RecLength  :   310  (x0136)   IO Time    : 2018/04/17 10:47:16.475.512   
IOType     :     3  (x03)     OrigNode   :   255  (xff) 
TransInd   :     .  (x00)     FormatType :     R  (x52) 
SyskeyLen  :     0  (x00)     Incomplete :     .  (x00) 
AuditRBA   :     167409       AuditPos   : 779280 
Continued  :     N  (x00)     RecCount   :     1  (x01) 

2018/04/17 10:47:16.475.512 Delete               Len   310 RBA 87850906 
Name: OWNER.T_JOE 
Before Image:                                             Partition 4   G  b   

GGS tokens: 
TokenID x52 'R' ORAROWID         Info x00  Length   20 
 4141 4148 6b55 4141 5441 4141 6264 7141 4159 0001 | AAAHkUAATAAAbdqAAY..  
TokenID x4c 'L' LOGCSN           Info x00  Length   10 
 3732 3833 3730 3834 3135                          | 7283708415  
TokenID x36 '6' TRANID           Info x00  Length   13 
 3232 2e33 302e 3139 3233 3539 39                  | 22.30.1923599  
   
Logdump 23 >scanforendtrans
End of Transaction found at RBA 94033767 
___________________________________________________________________ 
Hdr-Ind    :     E  (x45)     Partition  :     .  (x04)  
UndoFlag   :     .  (x00)     BeforeAfter:     B  (x42)  
RecLength  :   331  (x014b)   IO Time    : 2018/04/17 10:47:16.429.234   
IOType     :     3  (x03)     OrigNode   :   255  (xff) 
TransInd   :     .  (x02)     FormatType :     R  (x52) 
SyskeyLen  :     0  (x00)     Incomplete :     .  (x00) 
AuditRBA   :     167409       AuditPos   : 13903264 
Continued  :     N  (x00)     RecCount   :     1  (x01) 

2018/04/17 10:47:16.429.234 Delete               Len   331 RBA 94033767 
Name: OWNER.T_JOE 
Before Image:                                             Partition 4   G  e   
GGS tokens: 
TokenID x52 'R' ORAROWID         Info x00  Length   20 
 4141 4148 6b55 4141 5741 4141 4e6c 6c41 4177 0001 | AAAHkUAAWAAANllAAw..  
   
Logdump 24 >n
___________________________________________________________________ 
Hdr-Ind    :     E  (x45)     Partition  :     .  (x04)  
UndoFlag   :     .  (x00)     BeforeAfter:     B  (x42)  
RecLength  :   174  (x00ae)   IO Time    : 2018/04/17 10:47:24.429.491   
IOType     :    15  (x0f)     OrigNode   :   255  (xff) 
TransInd   :     .  (x00)     FormatType :     R  (x52) 
SyskeyLen  :     0  (x00)     Incomplete :     .  (x00) 
AuditRBA   :     167409       AuditPos   : 13947088 
Continued  :     N  (x00)     RecCount   :     1  (x01) 

2018/04/17 10:47:24.429.491 FieldComp            Len   174 RBA 94034190 
Name: OWNER.NEW_DATA 
Before Image:                                             Partition 4   G  b   
GGS tokens: 
TokenID x52 'R' ORAROWID         Info x00  Length   20 
 4141 4148 6a59 4141 5441 4142 794b 4541 412f 0001 | AAAHjYAATAAByKEAA/..  
TokenID x4c 'L' LOGCSN           Info x00  Length   10 
 3732 3833 3730 3834 3538                          | 7283708458  
TokenID x36 '6' TRANID           Info x00  Length   13 
 3132 2e31 362e 3330 3031 3139 37                  | 12.16.3001197  
   
Logdump 25 >open ./dirdat/nd004475
Current LogTrail is ./dirdat/nd004475 
Logdump 26 >count
LogTrail ./dirdat/nd004475 has 92822 records 
Total Data Bytes          92730182 
  Avg Bytes/Record             999 
Delete                       20937 
Insert                        5405 
FieldComp                      724 
LargeObject                  65755 
Others                           1 
Before Images                21163 
After Images                 71658 

Average of 1589 Transactions 
    Bytes/Trans .....      61161 
    Records/Trans ...         58 
    Files/Trans .....          5 
 
Logdump 27 >detail on
Logdump 28 >filter inc filename OWNER.T_JOE
Logdump 29 >count
Scanned     10000 records, RBA   12734577, 2018/04/17 07:25:42.524.558 
Scanned     20000 records, RBA   25670230, 2018/04/17 08:00:11.480.213 
Scanned     30000 records, RBA   38698934, 2018/04/17 08:30:24.488.669 
Scanned     40000 records, RBA   51436567, 2018/04/17 08:59:11.452.549 
Scanned     50000 records, RBA   63868041, 2018/04/17 09:43:10.477.605 
Scanned     60000 records, RBA   76010927, 2018/04/17 10:14:59.472.122 
Scanned     70000 records, RBA   94264594, 2018/04/17 10:47:31.447.436 
LogTrail ./dirdat/nd004475 has 15296 records 
Total Data Bytes           4757365 
  Avg Bytes/Record             311 
Delete                       15296 
Before Images                15296 
Filtering matched        15296 records 
          suppressed     77526 records 

Average of 2 Transactions 
    Bytes/Trans .....    2745786 
    Records/Trans ...       7648 
    Files/Trans .....        110 
 

OWNER.T_JOE                                      Partition 4 
Total Data Bytes           4757365 
  Avg Bytes/Record             311 
Delete                       15296 
Before Images                15296 
Logdump 30 >

 

March 31, 2018

Oracle Different Levels of Hell

Filed under: GoldenGate — mdinh @ 2:50 pm

Did not know there exists many levels of hell and Oracle certainly has them.

Would it be bad if someone is searching for hell and this blog is listed as top 10? :=(

Here’s the effort one needs to go through to determine what patch to apply for Goldengate software (binary) as part of quarterly Critical Patch Updates.

Hint: Goldengate does not participate in quarterly patch updates.

Find the latest patch available and patch using Opatch or install newer database compatible version using runInstaller.

Oracle GoldenGate — Oracle RDBMS Server Recommended Patches (Doc ID 1557031.1)
Latest RDBMS version Oracle Server 12.1.0.2

Let’s create a new doc for new version but don’t provide reference to it. Make the user find it.

Latest GoldenGate/Database (OGG/RDBMS) Patch recommendations (Doc ID 2193391.1)
Latest OGG Release 12.3
Latest RDBMS Release 12.2.0.1

OGG Release/OGG Patch
12.3/Recommended 12.3.0.1.2 or higher *** This is not a patch
12.2/Recommended 12.2.0.2.0 or higher *** This is not a patch
12.2/Minimum 12.2.0.1.170221 *** This is a patch

Master Note for Oracle GoldenGate Core Product Patch Sets (Doc ID 1645495.1)
There are no patch sets available for 12.3.0.1 at this time.
Latest OGG v12.2.0.1 Patch Set Availability Notes

February 11, 2018

Goldengate REPORTING

Filed under: GoldenGate — mdinh @ 2:52 pm

There were performance issues due to a new table being introduced to replication and I was asked to gather number of DMLs on table for 1 day.

Using DBA_TAB_MODIFICATIONS did not meet the requirements since statistics were gather about a week ago and over inflated.

Next thought process was why not use Goldengate since it captures all the changes and report on it.

This may or may not provide the required data if reporting is not properly configured.

Some will tell you to never rollover the report while others will rollover the report on a weekly basis.

From most recent experience, I would rollover report on daily basis because data can always be aggregated.

Here is an example of what I had to deal with.

$ grep -i report dirprm/e_hawk.prm

REPORTCOUNT EVERY 10 MINUTES, RATE

$ grep “activity since” dirtpt/E*.rpt|sort

E_HAWK0.rpt:Report at 2018-02-09 12:28:49 (activity since 2018-02-08 22:01:15)
--- How are you going to get daily activities accurately from aggregation over 2 months time frame?
E_HAWK1.rpt:Report at 2018-02-08 21:48:37 (activity since 2017-11-27 18:24:26)

Corresponding data from the report.
$ grep -A5 “From Table SCOTT.RATETEST” dirrpt/E_*rpt

E_HAWK0.rpt:Report at 2018-02-09 12:28:49 (activity since 2018-02-08 22:01:15)
dirrpt/E_HAWK0.rpt:From Table SCOTT.RATETEST:
dirrpt/E_HAWK0.rpt-       #                   inserts:       977
dirrpt/E_HAWK0.rpt-       #                   updates:  10439912
dirrpt/E_HAWK0.rpt-       #                   befores:  10439912
dirrpt/E_HAWK0.rpt-       #                   deletes:         0
dirrpt/E_HAWK0.rpt-       #                  discards:         0
--
E_HAWK1.rpt:Report at 2018-02-08 21:48:37 (activity since 2017-11-27 18:24:26)
dirrpt/E_HAWK1.rpt:From Table SCOTT.RATETEST:
dirrpt/E_HAWK1.rpt-       #                   inserts:     87063
dirrpt/E_HAWK1.rpt-       #                   updates: 821912582
dirrpt/E_HAWK1.rpt-       #                   befores: 821912582
dirrpt/E_HAWK1.rpt-       #                   deletes:         0
dirrpt/E_HAWK1.rpt-       #                  discards:         0

How did I end up getting the data?

From Goldengate report above (E_HAWK0.rpt) which is better as it provides the exact data processed by Goldengate vs trying to extract data from database.

I was lucky! Extract was restarted when table was added and removed.

Example of Better Configuration:

global_ggenv.inc

STATOPTIONS RESETREPORTSTATS
REPORT AT 00:01
REPORTROLLOVER AT 00:01
REPORTCOUNT EVERY 60 MINUTES, RATE
DISCARDROLLOVER AT 00:01

e_hawk.prm
EXTRACT E_HAWK

EXTRACT e_hawk
EXTTRAIL ./dirdat/bb
-- CHECKPARAMS
INCLUDE ./dirprm/global_macro.inc
INCLUDE ./dirprm/global_dbenv.inc
INCLUDE ./dirprm/global_ggenv.inc

Reference:

REPORTROLLOVER

Use the REPORTROLLOVER parameter to force report files to age on a regular schedule, instead of when a process starts. 
For long or continuous runs, setting an aging schedule controls the size of the active report file and provides a more predictable set of archives that can be included in your archiving routine.
Report statistics are carried over from one report to the other. To reset the statistics in the new report, use the STATOPTIONS parameter with the RESETREPORTSTATS option.

DBA_TAB_MODIFICATIONS

INSERTS/UPDATES/DELETES - Approximate number since the last time statistics were gathered.

February 3, 2018

PURGEOLDEXTRACTS Not Purging Trail Files Part2

Filed under: 12c,GoldenGate — mdinh @ 4:55 pm
If you read the post PURGEOLDEXTRACTS Not Purging Trail Files, you will find the solution is to replace syntax from mgr.prm as shown below:
Replace PURGEOLDEXTRACTS ./dirdat/*, USECHECKPOINTS, MINKEEPDAYS 1
With    PURGEOLDEXTRACTS /ggs/dirdat/*, USECHECKPOINTS, MINKEEPDAYS 2 

I found the solution by luck vs correct analysis; hence, the adage “Better to be lucky than good.”

Recently, the same issue occurred again for another environment and the solution was just the opposite.

PURGEOLDEXTRACTS dirdat/*, USECHECKPOINTS, MINKEEPHOURS 24, FREQUENCYMINUTES 30
-- PURGEOLDEXTRACTS /DBFS/ggs/dirdat/*, USECHECKPOINTS, MINKEEPHOURS 24, FREQUENCYMINUTES 30

Why is that!

No convention and inconsistency.

Here are the details.

--- How is manager configured?
GGSCI> send manager GETPURGEOLDEXTRACTS

Sending GETPURGEOLDEXTRACTS request to MANAGER ...

--- Manager is configured with trail pointing to /DBFS
PurgeOldExtracts Rules
Fileset                              MinHours MaxHours MinFiles MaxFiles UseCP
/DBFS/ggs/dirdat/*                   24       0        1        0        Y
OK	
--- Extract trail showing from $GG_HOME/dirdat
Extract Trails
Filename                        Oldest_Chkpt_Seqno  IsTable  IsVamTwoPhaseCommit
/u01/app/gg/12.2.0/dirdat/aa    16285

--- How was the extract created?
GGSCI> send e* status

Sending STATUS request to EXTRACT E_HAWK ...


EXTRACT E_HAWK (PID 40932)
  Current status: Recovery complete: At EOF

  Current read position:
  Sequence #: 16285
  RBA: 27233729
  Timestamp: 2018-01-25 21:01:35.000450
  Extract Trail: dirdat/aa --- This is how the trail is defined when extract was created

GGSCI> info e*

EXTRACT    E_HAWK    Last Started 2018-01-25 21:22   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:00:00 ago)
Process ID           40932
Log Read Checkpoint  File dirdat/aa000016286
                     2018-02-01 14:42:29.000124  RBA 29233729
GGSCI> exit

--- From $GG_HOME, dirdat is using symbolic link to /DBFS
$ ls -ld dir*
lrwxrwxrwx 1 ggsuser oinstall   23 Mar 18  2017 dirchk -> /DBFS/ggs/dirchk
lrwxrwxrwx 1 ggsuser oinstall   23 Mar 18  2017 dircrd -> /DBFS/ggs/dircrd
lrwxrwxrwx 1 ggsuser oinstall   23 Mar 18  2017 dirdat -> /DBFS/ggs/dirdat
lrwxrwxrwx 1 ggsuser oinstall   23 Mar 18  2017 dirdef -> /DBFS/ggs/dirdef
lrwxrwxrwx 1 ggsuser oinstall   23 Mar 18  2017 dirdmp -> /DBFS/ggs/dirdmp
lrwxrwxrwx 1 ggsuser oinstall   23 Mar 18  2017 dirout -> /DBFS/ggs/dirout
drwxr-xr-x 2 ggsuser oinstall 4096 Jan 26 13:49 dirpcs
lrwxrwxrwx 1 ggsuser oinstall   23 Mar 18  2017 dirprm -> /DBFS/ggs/dirprm
lrwxrwxrwx 1 ggsuser oinstall   23 Mar 18  2017 dirrpt -> /DBFS/ggs/dirrpt
lrwxrwxrwx 1 ggsuser oinstall   23 Mar 18  2017 dirsql -> /DBFS/ggs/dirsql
lrwxrwxrwx 1 ggsuser oinstall   23 Mar 18  2017 dirtmp -> /DBFS/ggs/dirtmp
lrwxrwxrwx 1 ggsuser oinstall   23 Mar 18  2017 dirwlt -> /DBFS/ggs/dirwlt
lrwxrwxrwx 1 ggsuser oinstall   23 Mar 18  2017 dirwww -> /DBFS/ggs/dirwww

In conclusion, PURGEOLDEXTRACTS location should be defined the same as the extract.

Isn’t that intuitive?

Oracle should make this a MOS Doc ;=)

January 31, 2018

Goldengate Tracing Network Ports

Filed under: GoldenGate — mdinh @ 4:30 am
--- Find mgr process
$ ps -ef|grep ./mgr
ggs  11823     1  0  2017 ?        00:29:13 ./mgr PARAMFILE /u01/app/ggs/dirprm/mgr.prm REPORTFILE /u01/app/ggs/dirrpt/MGR.rpt PROCESSID MGR
ggs  45054 30127  0 14:15 pts/0    00:00:00 grep --color=auto ./mgr

--- Find extract process
$ ps -ef|grep ./extract
ggs  17604 30127  0 14:15 pts/0    00:00:00 grep --color=auto ./extract
ggs  44306 11823  0 03:33 ?        00:01:28 /u01/app/ggs/extract PARAMFILE /u01/app/ggs/dirprm/e_hawk.prm REPORTFILE /u01/app/ggs/dirrpt/e_hawk.rpt PROCESSID e_hawk
ggs  44354 11823  0 03:33 ?        00:00:48 /u01/app/ggs/extract PARAMFILE /u01/app/ggs/dirprm/p_hawk.prm REPORTFILE /u01/app/ggs/dirrpt/p_hawk.rpt PROCESSID p_hawk

--- Find mgr port
$ lsof -i -P |grep 11823
mgr     11823 ggs    6u  IPv4 3492119103      0t0  TCP *:7809 (LISTEN)

--- Find extract port
$ lsof -i -P |grep 44306
extract 44306 ggs    6u  IPv4 2827659844      0t0  TCP *:7840 (LISTEN)

$ lsof -i -P |grep 44354
extract 44354 ggs    6u  IPv4 2827649786      0t0  TCP *:7841 (LISTEN)
extract 44354 ggs   10u  IPv4 2827610899      0t0  TCP hawk.local:21252->eagle.local:7822 (ESTABLISHED)

--- Use Goldengate to find the same info. However, it does only provides local port.
$ ./ggsci 

Oracle GoldenGate Command Interpreter for Oracle
Version 12.3.0.1.0 OGGCORE_12.3.0.1.0_PLATFORMS_170721.0154_FBO
Linux, x64, 64bit (optimized), Oracle 12c on Jul 21 2017 23:31:13
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2017, Oracle and/or its affiliates. All rights reserved.

GGSCI> send manager childstatus debug

Sending CHILDSTATUS request to MANAGER ...

Child Process Status - 2 Entries

ID      Group   Process Retry Retry Time          Start Time          Port
---- -------- --------- ----- ------------------- ------------------- ----
   0   e_hawk     44306     0 None                2017/11/20 19:49:14 7840
   1   p_hawk     44354     0 None                2017/11/20 19:55:43 7841

GGSCI>

December 2, 2017

Goldengate 12.3 Automatic CDR

Filed under: 12.2,GoldenGate,oracle — mdinh @ 11:51 pm

Automatic Conflict Detection and Resolution

Requirements: GoldenGate 12c (12.3.0.1) and Oracle Database 12c Release 2 (12.2) and later.

Automatic conflict detection and resolution does not require application changes for the following reasons:

  • Oracle Database automatically creates and maintains invisible timestamp columns.
  • Inserts, updates, and deletes use the delete tombstone log table to determine if a row was deleted.
  • LOB column conflicts can be detected.
  • Oracle Database automatically configures supplemental logging on required columns.

I have not had the chance to play with this yet and just only notice the documentation has been updated with details.

 

 

November 23, 2017

CRS-2674: Start of dbfs_mount failed

Filed under: 12c,GoldenGate,oracle,RAC — mdinh @ 1:04 am

$ crsctl start resource dbfs_mount
CRS-2672: Attempting to start ‘dbfs_mount’ on ‘node2’
CRS-2672: Attempting to start ‘dbfs_mount’ on ‘node1’
CRS-2674: Start of ‘dbfs_mount’ on ‘node1’ failed
CRS-2679: Attempting to clean ‘dbfs_mount’ on ‘node1’
CRS-2674: Start of ‘dbfs_mount’ on ‘node2’ failed
CRS-2679: Attempting to clean ‘dbfs_mount’ on ‘node2’
CRS-2681: Clean of ‘dbfs_mount’ on ‘node1’ succeeded
CRS-2681: Clean of ‘dbfs_mount’ on ‘node2’ succeeded
CRS-4000: Command Start failed, or completed with errors.

Check to make sure DBFS_USER password is not expired.

Next Page »

Create a free website or blog at WordPress.com.