Thinking Out Loud

March 7, 2015

restore validate archivelog

Filed under: 11g,RMAN — mdinh @ 7:06 pm

A common mistake I see in backup validation is not validating archivelog or Level 1 backup.

Here I will demonstrate various methods to validate achivelog.

Validate archivelog and does not list details for archivelog backup set. Too little information?

RMAN> restore validate archivelog from time “TO_DATE(‘2015-MAR-04 22:03:32′,’YYYY-MON-DD HH24:MI:SS’)”;

Starting restore at 2015-MAR-07 10:34:02
using channel ORA_DISK_1

channel ORA_DISK_1: scanning archived log /oradata/archivelog/hawklas/hawk_ba986d3b_1_871886678_245.arc
channel ORA_DISK_1: starting validation of archived log backup set
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/product/11.2.0/dbhome_1/dbs/3mq17ivt_1_1
channel ORA_DISK_1: piece handle=/u01/app/oracle/product/11.2.0/dbhome_1/dbs/3mq17ivt_1_1 tag=TAG20150307T095717
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: validation complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting validation of archived log backup set
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/product/11.2.0/dbhome_1/dbs/3nq17j0b_1_1
channel ORA_DISK_1: piece handle=/u01/app/oracle/product/11.2.0/dbhome_1/dbs/3nq17j0b_1_1 tag=TAG20150307T095717
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: validation complete, elapsed time: 00:00:01
Finished restore at 2015-MAR-07 10:34:05

Validate archivelog and list details for backupset. Too much information?

RMAN> restore validate preview archivelog from time “TO_DATE(‘2015-MAR-04 22:03:32′,’YYYY-MON-DD HH24:MI:SS’)”;

Starting restore at 2015-MAR-07 10:34:55
using channel ORA_DISK_1


List of Backup Sets
===================


BS Key  Size       Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ --------------------
117     45.54M     DISK        00:00:04     2015-MAR-07 09:57:21
        BP Key: 117   Status: AVAILABLE  Compressed: NO  Tag: TAG20150307T095717
        Piece Name: /u01/app/oracle/product/11.2.0/dbhome_1/dbs/3mq17ivt_1_1

  List of Archived Logs in backup set 117
  Thrd Seq     Low SCN    Low Time             Next SCN   Next Time
  ---- ------- ---------- -------------------- ---------- ---------
  1    149     351055     2015-MAR-04 22:03:32 371415     2015-MAR-05 14:54:39
  1    150     371415     2015-MAR-05 14:54:39 372594     2015-MAR-05 15:14:04
  1    151     372594     2015-MAR-05 15:14:04 373366     2015-MAR-05 15:34:04
  1    152     373366     2015-MAR-05 15:34:04 374012     2015-MAR-05 15:54:04
  1    153     374012     2015-MAR-05 15:54:04 374560     2015-MAR-05 16:14:05
  1    154     374560     2015-MAR-05 16:14:05 375084     2015-MAR-05 16:34:04
  1    155     375084     2015-MAR-05 16:34:04 375500     2015-MAR-05 16:54:05
  1    156     375500     2015-MAR-05 16:54:05 376122     2015-MAR-05 17:14:03
  1    157     376122     2015-MAR-05 17:14:03 376539     2015-MAR-05 17:34:05
  1    158     376539     2015-MAR-05 17:34:05 376952     2015-MAR-05 17:54:05
  1    159     376952     2015-MAR-05 17:54:05 377664     2015-MAR-05 18:14:04
  1    160     377664     2015-MAR-05 18:14:04 378236     2015-MAR-05 18:34:05
  1    161     378236     2015-MAR-05 18:34:05 378694     2015-MAR-05 18:54:05
  1    162     378694     2015-MAR-05 18:54:05 379347     2015-MAR-05 19:14:04
  1    163     379347     2015-MAR-05 19:14:04 379628     2015-MAR-06 07:16:37
  1    164     379628     2015-MAR-06 07:16:37 379909     2015-MAR-06 07:26:16
  1    165     379909     2015-MAR-06 07:26:16 380968     2015-MAR-06 07:46:16
  1    166     380968     2015-MAR-06 07:46:16 381650     2015-MAR-06 08:06:16
  1    167     381650     2015-MAR-06 08:06:16 382073     2015-MAR-06 08:26:16
  1    168     382073     2015-MAR-06 08:26:16 382491     2015-MAR-06 08:46:20
  1    169     382491     2015-MAR-06 08:46:20 382987     2015-MAR-06 09:06:22
  1    170     382987     2015-MAR-06 09:06:22 383474     2015-MAR-06 09:26:22
  1    171     383474     2015-MAR-06 09:26:22 383894     2015-MAR-06 09:46:21
  1    172     383894     2015-MAR-06 09:46:21 384457     2015-MAR-06 10:06:22
  1    173     384457     2015-MAR-06 10:06:22 384876     2015-MAR-06 10:26:22
  1    174     384876     2015-MAR-06 10:26:22 385294     2015-MAR-06 10:46:28
  1    175     385294     2015-MAR-06 10:46:28 385792     2015-MAR-06 11:06:28
  1    176     385792     2015-MAR-06 11:06:28 386280     2015-MAR-06 11:26:26
  1    177     386280     2015-MAR-06 11:26:26 386698     2015-MAR-06 11:46:27
  1    178     386698     2015-MAR-06 11:46:27 387196     2015-MAR-06 12:06:28
  1    179     387196     2015-MAR-06 12:06:28 387681     2015-MAR-06 12:26:28
  1    180     387681     2015-MAR-06 12:26:28 388098     2015-MAR-06 12:46:32
  1    181     388098     2015-MAR-06 12:46:32 388673     2015-MAR-06 13:06:33
  1    182     388673     2015-MAR-06 13:06:33 389092     2015-MAR-06 13:26:34
  1    183     389092     2015-MAR-06 13:26:34 389508     2015-MAR-06 13:46:33
  1    184     389508     2015-MAR-06 13:46:33 389985     2015-MAR-06 14:06:32
  1    185     389985     2015-MAR-06 14:06:32 390472     2015-MAR-06 14:26:33
  1    186     390472     2015-MAR-06 14:26:33 390888     2015-MAR-06 14:46:32
  1    187     390888     2015-MAR-06 14:46:32 391453     2015-MAR-06 15:06:32
  1    188     391453     2015-MAR-06 15:06:32 391878     2015-MAR-06 15:26:34
  1    189     391878     2015-MAR-06 15:26:34 392298     2015-MAR-06 15:46:40
  1    190     392298     2015-MAR-06 15:46:40 392809     2015-MAR-06 16:06:40
  1    191     392809     2015-MAR-06 16:06:40 393282     2015-MAR-06 16:26:40
  1    192     393282     2015-MAR-06 16:26:40 393699     2015-MAR-06 16:46:40
  1    193     393699     2015-MAR-06 16:46:40 394186     2015-MAR-06 17:06:41
  1    194     394186     2015-MAR-06 17:06:41 394671     2015-MAR-06 17:26:46
  1    195     394671     2015-MAR-06 17:26:46 395087     2015-MAR-06 17:46:45
  1    196     395087     2015-MAR-06 17:46:45 395649     2015-MAR-06 18:06:44
  1    197     395649     2015-MAR-06 18:06:44 396072     2015-MAR-06 18:26:44
  1    198     396072     2015-MAR-06 18:26:44 396489     2015-MAR-06 18:46:46
  1    199     396489     2015-MAR-06 18:46:46 396984     2015-MAR-06 19:06:46
  1    200     396984     2015-MAR-06 19:06:46 397481     2015-MAR-06 19:26:46
  1    201     397481     2015-MAR-06 19:26:46 397897     2015-MAR-06 19:46:45
  1    202     397897     2015-MAR-06 19:46:45 398392     2015-MAR-06 20:06:45
  1    203     398392     2015-MAR-06 20:06:45 398880     2015-MAR-06 20:26:44
  1    204     398880     2015-MAR-06 20:26:44 399299     2015-MAR-06 20:46:46
  1    205     399299     2015-MAR-06 20:46:46 399775     2015-MAR-06 21:06:46
  1    206     399775     2015-MAR-06 21:06:46 400258     2015-MAR-06 21:26:45
  1    207     400258     2015-MAR-06 21:26:45 400680     2015-MAR-06 21:46:44
  1    208     400680     2015-MAR-06 21:46:44 403781     2015-MAR-06 22:06:34

BS Key  Size       Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ --------------------
118     30.93M     DISK        00:00:02     2015-MAR-07 09:57:33
        BP Key: 118   Status: AVAILABLE  Compressed: NO  Tag: TAG20150307T095717
        Piece Name: /u01/app/oracle/product/11.2.0/dbhome_1/dbs/3nq17j0b_1_1

  List of Archived Logs in backup set 118
  Thrd Seq     Low SCN    Low Time             Next SCN   Next Time
  ---- ------- ---------- -------------------- ---------- ---------
  1    209     403781     2015-MAR-06 22:06:34 404216     2015-MAR-06 22:26:33
  1    210     404216     2015-MAR-06 22:26:33 404648     2015-MAR-06 22:46:32
  1    211     404648     2015-MAR-06 22:46:32 405189     2015-MAR-06 23:06:32
  1    212     405189     2015-MAR-06 23:06:32 405678     2015-MAR-06 23:26:35
  1    213     405678     2015-MAR-06 23:26:35 406110     2015-MAR-06 23:46:39
  1    214     406110     2015-MAR-06 23:46:39 406628     2015-MAR-07 00:06:38
  1    215     406628     2015-MAR-07 00:06:38 407191     2015-MAR-07 00:26:40
  1    216     407191     2015-MAR-07 00:26:40 407622     2015-MAR-07 00:46:39
  1    217     407622     2015-MAR-07 00:46:39 408298     2015-MAR-07 01:06:39
  1    218     408298     2015-MAR-07 01:06:39 408734     2015-MAR-07 01:26:38
  1    219     408734     2015-MAR-07 01:26:38 409167     2015-MAR-07 01:46:40
  1    220     409167     2015-MAR-07 01:46:40 409684     2015-MAR-07 02:06:39
  1    221     409684     2015-MAR-07 02:06:39 410318     2015-MAR-07 02:26:39
  1    222     410318     2015-MAR-07 02:26:39 410780     2015-MAR-07 02:46:38
  1    223     410780     2015-MAR-07 02:46:38 411462     2015-MAR-07 03:06:38
  1    224     411462     2015-MAR-07 03:06:38 411884     2015-MAR-07 03:26:40
  1    225     411884     2015-MAR-07 03:26:40 412300     2015-MAR-07 03:46:40
  1    226     412300     2015-MAR-07 03:46:40 412794     2015-MAR-07 04:06:39
  1    227     412794     2015-MAR-07 04:06:39 413315     2015-MAR-07 04:26:38
  1    228     413315     2015-MAR-07 04:26:38 413736     2015-MAR-07 04:46:38
  1    229     413736     2015-MAR-07 04:46:38 414223     2015-MAR-07 05:06:40
  1    230     414223     2015-MAR-07 05:06:40 414710     2015-MAR-07 05:26:38
  1    231     414710     2015-MAR-07 05:26:38 415134     2015-MAR-07 05:46:38
  1    232     415134     2015-MAR-07 05:46:38 417948     2015-MAR-07 06:06:26
  1    233     417948     2015-MAR-07 06:06:26 418380     2015-MAR-07 06:26:26
  1    234     418380     2015-MAR-07 06:26:26 418813     2015-MAR-07 06:46:27
  1    235     418813     2015-MAR-07 06:46:27 419405     2015-MAR-07 07:06:26
  1    236     419405     2015-MAR-07 07:06:26 419841     2015-MAR-07 07:26:28
  1    237     419841     2015-MAR-07 07:26:28 420275     2015-MAR-07 07:46:34
  1    238     420275     2015-MAR-07 07:46:34 420777     2015-MAR-07 08:06:33
  1    239     420777     2015-MAR-07 08:06:33 421312     2015-MAR-07 08:26:32
  1    240     421312     2015-MAR-07 08:26:32 421745     2015-MAR-07 08:46:32
  1    241     421745     2015-MAR-07 08:46:32 422279     2015-MAR-07 09:06:32
  1    242     422279     2015-MAR-07 09:06:32 422793     2015-MAR-07 09:26:34
  1    243     422793     2015-MAR-07 09:26:34 423233     2015-MAR-07 09:46:40
  1    244     423233     2015-MAR-07 09:46:40 423510     2015-MAR-07 09:57:16
List of Archived Log Copies for database with db_unique_name HAWKLAS
=====================================================================

Key     Thrd Seq     S Low Time
------- ---- ------- - --------------------
245     1    245     A 2015-MAR-07 09:57:16
        Name: /oradata/archivelog/hawklas/hawk_ba986d3b_1_871886678_245.arc


channel ORA_DISK_1: scanning archived log /oradata/archivelog/hawklas/hawk_ba986d3b_1_871886678_245.arc
channel ORA_DISK_1: starting validation of archived log backup set
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/product/11.2.0/dbhome_1/dbs/3mq17ivt_1_1
channel ORA_DISK_1: piece handle=/u01/app/oracle/product/11.2.0/dbhome_1/dbs/3mq17ivt_1_1 tag=TAG20150307T095717
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: validation complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting validation of archived log backup set
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/product/11.2.0/dbhome_1/dbs/3nq17j0b_1_1
channel ORA_DISK_1: piece handle=/u01/app/oracle/product/11.2.0/dbhome_1/dbs/3nq17j0b_1_1 tag=TAG20150307T095717
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: validation complete, elapsed time: 00:00:01
Finished restore at 2015-MAR-07 10:34:59

RMAN>

Validate archivelog and list summary for backupset. Just enough information?

RMAN> restore validate preview summary archivelog from time “TO_DATE(‘2015-MAR-04 22:03:32′,’YYYY-MON-DD HH24:MI:SS’)”;

Starting restore at 2015-MAR-07 10:36:58
using channel ORA_DISK_1


List of Backups
===============
Key     TY LV S Device Type Completion Time      #Pieces #Copies Compressed Tag
------- -- -- - ----------- -------------------- ------- ------- ---------- ---
117     B  A  A DISK        2015-MAR-07 09:57:21 1       1       NO         TAG20150307T095717
118     B  A  A DISK        2015-MAR-07 09:57:33 1       1       NO         TAG20150307T095717
List of Archived Log Copies for database with db_unique_name HAWKLAS
=====================================================================

Key     Thrd Seq     S Low Time
------- ---- ------- - --------------------
245     1    245     A 2015-MAR-07 09:57:16
        Name: /oradata/archivelog/hawklas/hawk_ba986d3b_1_871886678_245.arc


channel ORA_DISK_1: scanning archived log /oradata/archivelog/hawklas/hawk_ba986d3b_1_871886678_245.arc
channel ORA_DISK_1: starting validation of archived log backup set
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/product/11.2.0/dbhome_1/dbs/3mq17ivt_1_1
channel ORA_DISK_1: piece handle=/u01/app/oracle/product/11.2.0/dbhome_1/dbs/3mq17ivt_1_1 tag=TAG20150307T095717
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: validation complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting validation of archived log backup set
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/product/11.2.0/dbhome_1/dbs/3nq17j0b_1_1
channel ORA_DISK_1: piece handle=/u01/app/oracle/product/11.2.0/dbhome_1/dbs/3nq17j0b_1_1 tag=TAG20150307T095717
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: validation complete, elapsed time: 00:00:01
Finished restore at 2015-MAR-07 10:37:01

RMAN> list backup of archivelog from time “TO_DATE(‘2015-MAR-04 22:03:32′,’YYYY-MON-DD HH24:MI:SS’)” summary;

List of Backups
===============
Key     TY LV S Device Type Completion Time      #Pieces #Copies Compressed Tag
------- -- -- - ----------- -------------------- ------- ------- ---------- ---
115     B  A  A DISK        2015-MAR-06 07:16:40 1       1       YES        TAG20150306T071638
117     B  A  A DISK        2015-MAR-07 09:57:21 1       1       NO         TAG20150307T095717
118     B  A  A DISK        2015-MAR-07 09:57:33 1       1       NO         TAG20150307T095717

Why is backupset 115 not used in restore validate?
RMAN> list backupset 115; – contains backup for seq 149-163

List of Backup Sets
===================


BS Key  Size       Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ --------------------
115     3.42M      DISK        00:00:01     2015-MAR-06 07:16:40
        BP Key: 115   Status: AVAILABLE  Compressed: YES  Tag: TAG20150306T071638
        Piece Name: /u01/app/oracle/product/11.2.0/dbhome_1/dbs/3kq14l6n_1_1

  List of Archived Logs in backup set 115
  Thrd Seq     Low SCN    Low Time             Next SCN   Next Time
  ---- ------- ---------- -------------------- ---------- ---------
  1    149     351055     2015-MAR-04 22:03:32 371415     2015-MAR-05 14:54:39
  1    150     371415     2015-MAR-05 14:54:39 372594     2015-MAR-05 15:14:04
  1    151     372594     2015-MAR-05 15:14:04 373366     2015-MAR-05 15:34:04
  1    152     373366     2015-MAR-05 15:34:04 374012     2015-MAR-05 15:54:04
  1    153     374012     2015-MAR-05 15:54:04 374560     2015-MAR-05 16:14:05
  1    154     374560     2015-MAR-05 16:14:05 375084     2015-MAR-05 16:34:04
  1    155     375084     2015-MAR-05 16:34:04 375500     2015-MAR-05 16:54:05
  1    156     375500     2015-MAR-05 16:54:05 376122     2015-MAR-05 17:14:03
  1    157     376122     2015-MAR-05 17:14:03 376539     2015-MAR-05 17:34:05
  1    158     376539     2015-MAR-05 17:34:05 376952     2015-MAR-05 17:54:05
  1    159     376952     2015-MAR-05 17:54:05 377664     2015-MAR-05 18:14:04
  1    160     377664     2015-MAR-05 18:14:04 378236     2015-MAR-05 18:34:05
  1    161     378236     2015-MAR-05 18:34:05 378694     2015-MAR-05 18:54:05
  1    162     378694     2015-MAR-05 18:54:05 379347     2015-MAR-05 19:14:04
  1    163     379347     2015-MAR-05 19:14:04 379628     2015-MAR-06 07:16:37

RMAN>

Backup for seq 149-163 is in backupset 115 & 117

RMAN> list backup of archivelog from sequence 149 until sequence 163 summary;

List of Backups
===============
Key     TY LV S Device Type Completion Time      #Pieces #Copies Compressed Tag
------- -- -- - ----------- -------------------- ------- ------- ---------- ---
115     B  A  A DISK        2015-MAR-06 07:16:40 1       1       YES        TAG20150306T071638
117     B  A  A DISK        2015-MAR-07 09:57:21 1       1       NO         TAG20150307T095717

RMAN> list backupset 117; – verified

List of Backup Sets
===================

BS Key  Size       Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ --------------------
117     45.54M     DISK        00:00:04     2015-MAR-07 09:57:21
        BP Key: 117   Status: AVAILABLE  Compressed: NO  Tag: TAG20150307T095717
        Piece Name: /u01/app/oracle/product/11.2.0/dbhome_1/dbs/3mq17ivt_1_1

  List of Archived Logs in backup set 117
  Thrd Seq     Low SCN    Low Time             Next SCN   Next Time
  ---- ------- ---------- -------------------- ---------- ---------
  1    149     351055     2015-MAR-04 22:03:32 371415     2015-MAR-05 14:54:39
  1    150     371415     2015-MAR-05 14:54:39 372594     2015-MAR-05 15:14:04
  1    151     372594     2015-MAR-05 15:14:04 373366     2015-MAR-05 15:34:04
  1    152     373366     2015-MAR-05 15:34:04 374012     2015-MAR-05 15:54:04
  1    153     374012     2015-MAR-05 15:54:04 374560     2015-MAR-05 16:14:05
  1    154     374560     2015-MAR-05 16:14:05 375084     2015-MAR-05 16:34:04
  1    155     375084     2015-MAR-05 16:34:04 375500     2015-MAR-05 16:54:05
  1    156     375500     2015-MAR-05 16:54:05 376122     2015-MAR-05 17:14:03
  1    157     376122     2015-MAR-05 17:14:03 376539     2015-MAR-05 17:34:05
  1    158     376539     2015-MAR-05 17:34:05 376952     2015-MAR-05 17:54:05
  1    159     376952     2015-MAR-05 17:54:05 377664     2015-MAR-05 18:14:04
  1    160     377664     2015-MAR-05 18:14:04 378236     2015-MAR-05 18:34:05
  1    161     378236     2015-MAR-05 18:34:05 378694     2015-MAR-05 18:54:05
  1    162     378694     2015-MAR-05 18:54:05 379347     2015-MAR-05 19:14:04
  1    163     379347     2015-MAR-05 19:14:04 379628     2015-MAR-06 07:16:37
  1    164     379628     2015-MAR-06 07:16:37 379909     2015-MAR-06 07:26:16
  1    165     379909     2015-MAR-06 07:26:16 380968     2015-MAR-06 07:46:16
  1    166     380968     2015-MAR-06 07:46:16 381650     2015-MAR-06 08:06:16
  1    167     381650     2015-MAR-06 08:06:16 382073     2015-MAR-06 08:26:16
  1    168     382073     2015-MAR-06 08:26:16 382491     2015-MAR-06 08:46:20
  1    169     382491     2015-MAR-06 08:46:20 382987     2015-MAR-06 09:06:22
  1    170     382987     2015-MAR-06 09:06:22 383474     2015-MAR-06 09:26:22
  1    171     383474     2015-MAR-06 09:26:22 383894     2015-MAR-06 09:46:21
  1    172     383894     2015-MAR-06 09:46:21 384457     2015-MAR-06 10:06:22
  1    173     384457     2015-MAR-06 10:06:22 384876     2015-MAR-06 10:26:22
  1    174     384876     2015-MAR-06 10:26:22 385294     2015-MAR-06 10:46:28
  1    175     385294     2015-MAR-06 10:46:28 385792     2015-MAR-06 11:06:28
  1    176     385792     2015-MAR-06 11:06:28 386280     2015-MAR-06 11:26:26
  1    177     386280     2015-MAR-06 11:26:26 386698     2015-MAR-06 11:46:27
  1    178     386698     2015-MAR-06 11:46:27 387196     2015-MAR-06 12:06:28
  1    179     387196     2015-MAR-06 12:06:28 387681     2015-MAR-06 12:26:28
  1    180     387681     2015-MAR-06 12:26:28 388098     2015-MAR-06 12:46:32
  1    181     388098     2015-MAR-06 12:46:32 388673     2015-MAR-06 13:06:33
  1    182     388673     2015-MAR-06 13:06:33 389092     2015-MAR-06 13:26:34
  1    183     389092     2015-MAR-06 13:26:34 389508     2015-MAR-06 13:46:33
  1    184     389508     2015-MAR-06 13:46:33 389985     2015-MAR-06 14:06:32
  1    185     389985     2015-MAR-06 14:06:32 390472     2015-MAR-06 14:26:33
  1    186     390472     2015-MAR-06 14:26:33 390888     2015-MAR-06 14:46:32
  1    187     390888     2015-MAR-06 14:46:32 391453     2015-MAR-06 15:06:32
  1    188     391453     2015-MAR-06 15:06:32 391878     2015-MAR-06 15:26:34
  1    189     391878     2015-MAR-06 15:26:34 392298     2015-MAR-06 15:46:40
  1    190     392298     2015-MAR-06 15:46:40 392809     2015-MAR-06 16:06:40
  1    191     392809     2015-MAR-06 16:06:40 393282     2015-MAR-06 16:26:40
  1    192     393282     2015-MAR-06 16:26:40 393699     2015-MAR-06 16:46:40
  1    193     393699     2015-MAR-06 16:46:40 394186     2015-MAR-06 17:06:41
  1    194     394186     2015-MAR-06 17:06:41 394671     2015-MAR-06 17:26:46
  1    195     394671     2015-MAR-06 17:26:46 395087     2015-MAR-06 17:46:45
  1    196     395087     2015-MAR-06 17:46:45 395649     2015-MAR-06 18:06:44
  1    197     395649     2015-MAR-06 18:06:44 396072     2015-MAR-06 18:26:44
  1    198     396072     2015-MAR-06 18:26:44 396489     2015-MAR-06 18:46:46
  1    199     396489     2015-MAR-06 18:46:46 396984     2015-MAR-06 19:06:46
  1    200     396984     2015-MAR-06 19:06:46 397481     2015-MAR-06 19:26:46
  1    201     397481     2015-MAR-06 19:26:46 397897     2015-MAR-06 19:46:45
  1    202     397897     2015-MAR-06 19:46:45 398392     2015-MAR-06 20:06:45
  1    203     398392     2015-MAR-06 20:06:45 398880     2015-MAR-06 20:26:44
  1    204     398880     2015-MAR-06 20:26:44 399299     2015-MAR-06 20:46:46
  1    205     399299     2015-MAR-06 20:46:46 399775     2015-MAR-06 21:06:46
  1    206     399775     2015-MAR-06 21:06:46 400258     2015-MAR-06 21:26:45
  1    207     400258     2015-MAR-06 21:26:45 400680     2015-MAR-06 21:46:44
  1    208     400680     2015-MAR-06 21:46:44 403781     2015-MAR-06 22:06:34

RMAN>

March 6, 2015

RMAN-06023: no backup or copy of datafile # found to restore

Filed under: 11g,RMAN — mdinh @ 2:55 am

There’s a great note from MOS – Checklist for an RMAN Restore (Doc ID 1554636.1) but how many of you review this before performing a restore?

If you don’t then you are as guilty as I am.

RMAN> restore database until time "TO_DATE('2015-MAR-04 19:53:54','YYYY-MON-DD HH24:MI:SS')" preview summary;

Starting restore at 2015-MAR-05 18:03:28
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=9 device type=DISK

datafile 5 will be created automatically during restore operation
datafile 6 will be created automatically during restore operation
datafile 7 will be created automatically during restore operation
datafile 8 will be created automatically during restore operation
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 03/05/2015 18:03:28
RMAN-06026: some targets not found - aborting restore
RMAN-06023: no backup or copy of datafile 4 found to restore
RMAN-06023: no backup or copy of datafile 3 found to restore
RMAN-06023: no backup or copy of datafile 2 found to restore
RMAN-06023: no backup or copy of datafile 1 found to restore

Let’s check backup for datafile 5,6,7,8 – looks good

RMAN> list backup of datafile 5,6,7,8 summary;


List of Backups
===============
Key     TY LV S Device Type Completion Time      #Pieces #Copies Compressed Tag
------- -- -- - ----------- -------------------- ------- ------- ---------- ---
98      B  0  A DISK        2015-MAR-04 19:54:24 1       1       YES        LEVEL0
99      B  0  A DISK        2015-MAR-04 19:54:26 1       1       YES        LEVEL0
100     B  0  A DISK        2015-MAR-04 19:54:27 1       1       YES        LEVEL0
101     B  0  A DISK        2015-MAR-04 19:54:29 1       1       YES        LEVEL0
109     B  1  A DISK        2015-MAR-04 22:03:26 1       1       YES        LEVEL1
110     B  1  A DISK        2015-MAR-04 22:03:28 1       1       YES        LEVEL1
111     B  1  A DISK        2015-MAR-04 22:03:29 1       1       YES        LEVEL1
112     B  1  A DISK        2015-MAR-04 22:03:30 1       1       YES        LEVEL1

RMAN>
RMAN> list backup of datafile 1 summary;


List of Backups
===============
Key     TY LV S Device Type Completion Time      #Pieces #Copies Compressed Tag
------- -- -- - ----------- -------------------- ------- ------- ---------- ---
94      B  0  A DISK        2015-MAR-04 19:54:13 1       1       YES        LEVEL0
105     B  1  A DISK        2015-MAR-04 22:02:57 1       1       YES        LEVEL1

RMAN>

The until time was not far enough to include backup for data 1 ..

RMAN> restore database until time "TO_DATE('2015-MAR-04 19:54:30','YYYY-MON-DD HH24:MI:SS')" preview summary;

Starting restore at 2015-MAR-05 18:35:56
using channel ORA_DISK_1


List of Backups
===============
Key     TY LV S Device Type Completion Time      #Pieces #Copies Compressed Tag
------- -- -- - ----------- -------------------- ------- ------- ---------- ---
94      B  0  A DISK        2015-MAR-04 19:54:12 1       1       YES        LEVEL0
96      B  0  A DISK        2015-MAR-04 19:54:19 1       1       YES        LEVEL0
95      B  0  A DISK        2015-MAR-04 19:54:16 1       1       YES        LEVEL0
97      B  0  A DISK        2015-MAR-04 19:54:22 1       1       YES        LEVEL0
98      B  0  A DISK        2015-MAR-04 19:54:23 1       1       YES        LEVEL0
99      B  0  A DISK        2015-MAR-04 19:54:26 1       1       YES        LEVEL0
100     B  0  A DISK        2015-MAR-04 19:54:27 1       1       YES        LEVEL0
101     B  0  A DISK        2015-MAR-04 19:54:29 1       1       YES        LEVEL0


List of Backups
===============
Key     TY LV S Device Type Completion Time      #Pieces #Copies Compressed Tag
------- -- -- - ----------- -------------------- ------- ------- ---------- ---
102     B  A  A DISK        2015-MAR-04 19:54:30 1       1       YES        ARCHIVELOG
Media recovery start SCN is 342495
Recovery must be done beyond SCN 342508 to clear datafile fuzziness
Finished restore at 2015-MAR-05 18:35:56

Where on earth did I get the time “2015-MAR-04 19:54:30″ to begin with?

I used Low Time from ARCHIVELOG backup which was not sufficient for demonstration purpose.

RMAN> list backupset 102;


List of Backup Sets
===================


BS Key  Size       Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ --------------------
102     3.50K      DISK        00:00:00     2015-MAR-04 19:54:30
        BP Key: 102   Status: AVAILABLE  Compressed: YES  Tag: ARCHIVELOG
        Piece Name: /oradata/backup/arc_HAWK_3130551611_20150304_37q10orm_1_1

  List of Archived Logs in backup set 102
  Thrd Seq     Low SCN    Low Time             Next SCN   Next Time
  ---- ------- ---------- -------------------- ---------- ---------
  1    140     342477     2015-MAR-04 19:53:54 342514     2015-MAR-04 19:54:30

RMAN>

Instead, use “Next Time” which is also the same as “Completion Time” from list backup summary;

RMAN> list backup summary;


List of Backups
===============
Key     TY LV S Device Type Completion Time      #Pieces #Copies Compressed Tag
------- -- -- - ----------- -------------------- ------- ------- ---------- ---
90      B  A  A DISK        2015-MAR-04 19:53:58 1       1       YES        ARCHIVELOG
91      B  A  A DISK        2015-MAR-04 19:54:00 1       1       YES        ARCHIVELOG
92      B  A  A DISK        2015-MAR-04 19:54:02 1       1       YES        ARCHIVELOG
93      B  A  A DISK        2015-MAR-04 19:54:05 1       1       YES        ARCHIVELOG
94      B  0  A DISK        2015-MAR-04 19:54:13 1       1       YES        LEVEL0
95      B  0  A DISK        2015-MAR-04 19:54:16 1       1       YES        LEVEL0
96      B  0  A DISK        2015-MAR-04 19:54:20 1       1       YES        LEVEL0
97      B  0  A DISK        2015-MAR-04 19:54:22 1       1       YES        LEVEL0
98      B  0  A DISK        2015-MAR-04 19:54:24 1       1       YES        LEVEL0
99      B  0  A DISK        2015-MAR-04 19:54:26 1       1       YES        LEVEL0
100     B  0  A DISK        2015-MAR-04 19:54:27 1       1       YES        LEVEL0
101     B  0  A DISK        2015-MAR-04 19:54:29 1       1       YES        LEVEL0
102     B  A  A DISK        2015-MAR-04 19:54:30 1       1       YES        ARCHIVELOG
103     B  F  A DISK        2015-MAR-04 19:54:34 1       1       NO         TAG20150304T195432
104     B  A  A DISK        2015-MAR-04 22:02:49 1       1       YES        ARCHIVELOG
105     B  1  A DISK        2015-MAR-04 22:02:57 1       1       YES        LEVEL1
106     B  1  A DISK        2015-MAR-04 22:03:04 1       1       YES        LEVEL1
107     B  1  A DISK        2015-MAR-04 22:03:18 1       1       YES        LEVEL1
108     B  1  A DISK        2015-MAR-04 22:03:24 1       1       YES        LEVEL1
109     B  1  A DISK        2015-MAR-04 22:03:26 1       1       YES        LEVEL1
110     B  1  A DISK        2015-MAR-04 22:03:28 1       1       YES        LEVEL1
111     B  1  A DISK        2015-MAR-04 22:03:29 1       1       YES        LEVEL1
112     B  1  A DISK        2015-MAR-04 22:03:30 1       1       YES        LEVEL1
113     B  A  A DISK        2015-MAR-04 22:03:32 1       1       YES        ARCHIVELOG
114     B  F  A DISK        2015-MAR-04 22:03:35 1       1       NO         TAG20150304T220333

RMAN>

February 20, 2015

Oracle Manual Standby – Applying Log

Filed under: 11g,oracle,RMAN — mdinh @ 7:49 pm

If you are running Oracle EE, there are many books on Data Guard. However, for Oracle SE and manual standby, have fun searching.

I wanted to get a better understanding of registering logfile.

Depending on how the standby environment is monitored, not registering logfile may yield incorrect results when checking standby lag.

When logfile is not registered, v$archived_log is not updated.

STANDBY: check status

[oracle@armor:hawk:/home/oracle]
$ sqlplus / as sysdba @stby.sql

SQL*Plus: Release 11.2.0.4.0 Production on Fri Feb 20 10:12:58 2015

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Release 11.2.0.4.0 - 64bit Production

ARMOR:(SYS@hawk):PHYSICAL STANDBY> select al.thrd "Thread", almax "Last Seq Received", lhmax "Last Seq Applied"
  2  from
  3  (
  4    select thread# thrd, max(sequence#) almax from v$archived_log
  5    where resetlogs_change#=(select resetlogs_change# from v$database) and applied='YES'
  6    group by thread#
  7  ) al,
  8  (
  9    select thread# thrd, max(sequence#) lhmax
 10    from v$log_history
 11    where first_time=(select max(first_time) from v$log_history)
 12    group by thread#
 13  ) lh
 14  where al.thrd = lh.thrd
 15  ;

          Thread Last Seq Received Last Seq Applied
---------------- ----------------- ----------------
               1                44               44

ARMOR:(SYS@hawk):PHYSICAL STANDBY> select first_time,applied,archived,registrar,creator,thread#,sequence# from v$archived_log
  2  where resetlogs_change#=(select resetlogs_change# from v$database)
  3  and first_time>sysdate-3/24
  4  order by sequence# desc
  5  ;

NOTE: SRMN - RMAN at standby

FIRST_TIME          APPLIED   ARC REGISTR CREATOR          THREAD#        SEQUENCE#
------------------- --------- --- ------- ------- ---------------- ----------------
2015-02-20 10:09:09 YES       YES SRMN    SRMN                   1               44
2015-02-20 10:03:54 NO        YES SRMN    SRMN                   1               43
2015-02-20 09:58:33 YES       YES SRMN    SRMN                   1               42
2015-02-20 09:52:53 YES       YES SRMN    SRMN                   1               41
2015-02-20 09:47:17 YES       YES SRMN    SRMN                   1               40
2015-02-20 09:45:50 YES       YES SRMN    SRMN                   1               39
2015-02-20 09:41:36 YES       YES SRMN    SRMN                   1               38
2015-02-20 09:21:34 YES       YES SRMN    SRMN                   1               37
2015-02-20 09:01:33 YES       YES SRMN    SRMN                   1               36
2015-02-20 08:41:35 YES       YES SRMN    SRMN                   1               35
2015-02-20 08:21:34 YES       YES SRMN    SRMN                   1               34
2015-02-20 08:01:33 YES       YES SRMN    SRMN                   1               33
2015-02-20 07:41:30 YES       YES SRMN    SRMN                   1               32
2015-02-20 07:41:29 YES       YES SRMN    SRMN                   1               31
2015-02-20 07:41:29 YES       YES SRMN    SRMN                   1               30
2015-02-20 07:41:29 YES       YES SRMN    SRMN                   1               29
2015-02-20 07:41:26 YES       YES SRMN    SRMN                   1               28
2015-02-20 07:41:23 YES       YES SRMN    SRMN                   1               27
2015-02-20 07:24:52 YES       YES SRMN    SRMN                   1               26

19 rows selected.

ARMOR:(SYS@hawk):PHYSICAL STANDBY>

PRIMARY: create tablespace new1

ANGEL:(SYS@hawk):PRIMARY> create tablespace new1;

Tablespace created.

ANGEL:(SYS@hawk):PRIMARY> alter system archive log current;

System altered.

ANGEL:(SYS@hawk):PRIMARY> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            /oradata/archivelog/hawk
Oldest online log sequence     44
Next log sequence to archive   46
Current log sequence           46
ANGEL:(SYS@hawk):PRIMARY>

PRIMARY: transfer archive log to STANDBY

[oracle@angel:hawk:/oradata/archivelog/hawk]
$ rsync -avh /oradata/archivelog/hawk/ armor:/oradata/archivelog/hawk/
sending incremental file list
./
hawk_45_1_872097259.arc

sent 170.68K bytes  received 34 bytes  341.42K bytes/sec
total size is 78.17M  speedup is 457.90
[oracle@angel:hawk:/oradata/archivelog/hawk]
$

STANDBY: apply log

[oracle@armor:hawk:/home/oracle]
$ sqlplus / as sysdba @recoverauto.sql

SQL*Plus: Release 11.2.0.4.0 Production on Fri Feb 20 10:14:43 2015

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Release 11.2.0.4.0 - 64bit Production

ARMOR:(SYS@hawk):PHYSICAL STANDBY> recover automatic standby database;
ORA-00279: change 216448 generated at 02/20/2015 10:14:03 needed for thread 1
ORA-00289: suggestion : /oradata/archivelog/hawk/hawk_46_1_872097259.arc
ORA-00280: change 216448 for thread 1 is in sequence #46
ORA-00278: log file '/oradata/archivelog/hawk/hawk_46_1_872097259.arc' no longer needed for this recovery
ORA-00308: cannot open archived log '/oradata/archivelog/hawk/hawk_46_1_872097259.arc'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3


Specify log: {=suggested | filename | AUTO | CANCEL}
cancel
Media recovery cancelled.
ARMOR:(SYS@hawk):PHYSICAL STANDBY> @@stby.sql
ARMOR:(SYS@hawk):PHYSICAL STANDBY> set echo on numw 16 lines 200 pages 1000
ARMOR:(SYS@hawk):PHYSICAL STANDBY> select al.thrd "Thread", almax "Last Seq Received", lhmax "Last Seq Applied"
  2  from
  3  (
  4    select thread# thrd, max(sequence#) almax from v$archived_log
  5    where resetlogs_change#=(select resetlogs_change# from v$database) and applied='YES'
  6    group by thread#
  7  ) al,
  8  (
  9    select thread# thrd, max(sequence#) lhmax
 10    from v$log_history
 11    where first_time=(select max(first_time) from v$log_history)
 12    group by thread#
 13  ) lh
 14  where al.thrd = lh.thrd
 15  ;

          Thread Last Seq Received Last Seq Applied
---------------- ----------------- ----------------
               1                44               45

NOTE: sequence 45 is not registered with v$archived_log but has been applied

ARMOR:(SYS@hawk):PHYSICAL STANDBY> select first_time,applied,archived,registrar,creator,thread#,sequence# from v$archived_log
  2  where resetlogs_change#=(select resetlogs_change# from v$database)
  3  and first_time>sysdate-3/24
  4  order by sequence# desc
  5  ;

FIRST_TIME          APPLIED   ARC REGISTR CREATOR          THREAD#        SEQUENCE#
------------------- --------- --- ------- ------- ---------------- ----------------
2015-02-20 10:09:09 YES       YES SRMN    SRMN                   1               44
2015-02-20 10:03:54 NO        YES SRMN    SRMN                   1               43
2015-02-20 09:58:33 YES       YES SRMN    SRMN                   1               42
2015-02-20 09:52:53 YES       YES SRMN    SRMN                   1               41
2015-02-20 09:47:17 YES       YES SRMN    SRMN                   1               40
2015-02-20 09:45:50 YES       YES SRMN    SRMN                   1               39
2015-02-20 09:41:36 YES       YES SRMN    SRMN                   1               38
2015-02-20 09:21:34 YES       YES SRMN    SRMN                   1               37
2015-02-20 09:01:33 YES       YES SRMN    SRMN                   1               36
2015-02-20 08:41:35 YES       YES SRMN    SRMN                   1               35
2015-02-20 08:21:34 YES       YES SRMN    SRMN                   1               34
2015-02-20 08:01:33 YES       YES SRMN    SRMN                   1               33
2015-02-20 07:41:30 YES       YES SRMN    SRMN                   1               32
2015-02-20 07:41:29 YES       YES SRMN    SRMN                   1               31
2015-02-20 07:41:29 YES       YES SRMN    SRMN                   1               30
2015-02-20 07:41:29 YES       YES SRMN    SRMN                   1               29
2015-02-20 07:41:26 YES       YES SRMN    SRMN                   1               28
2015-02-20 07:41:23 YES       YES SRMN    SRMN                   1               27
2015-02-20 07:24:52 YES       YES SRMN    SRMN                   1               26

19 rows selected.

ARMOR:(SYS@hawk):PHYSICAL STANDBY> select name from v$tablespace;

NAME
------------------------------
SYSTEM
SYSAUX
UNDOTBS1
TEMP
USERS
NEW1

6 rows selected.

ARMOR:(SYS@hawk):PHYSICAL STANDBY>

PRIMARY: create tablespace new2

ANGEL:(SYS@hawk):PRIMARY> create tablespace new2;

Tablespace created.

ANGEL:(SYS@hawk):PRIMARY> alter system archive log current;

System altered.

ANGEL:(SYS@hawk):PRIMARY> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            /oradata/archivelog/hawk
Oldest online log sequence     45
Next log sequence to archive   47
Current log sequence           47
ANGEL:(SYS@hawk):PRIMARY>

PRIMARY: transfer archive log to STANDBY

[oracle@angel:hawk:/oradata/archivelog/hawk]
$ rsync -avh /oradata/archivelog/hawk/ armor:/oradata/archivelog/hawk/
sending incremental file list
./
hawk_46_1_872097259.arc

sent 144.59K bytes  received 34 bytes  289.24K bytes/sec
total size is 78.31M  speedup is 541.50
[oracle@angel:hawk:/oradata/archivelog/hawk]
$

STANDBY: register archive log using RMAN

[oracle@armor:hawk:/home/oracle]
$ rman @catalog_arc.rman

Recovery Manager: Release 11.2.0.4.0 - Production on Fri Feb 20 10:16:20 2015

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

RMAN> set echo on
2> connect target;
3> catalog start with '/oradata/archivelog/hawk/' noprompt;
4> exit
echo set on

connected to target database: HAWK (DBID=3130795691, not open)

using target database control file instead of recovery catalog
searching for all files that match the pattern /oradata/archivelog/hawk/

List of Files Unknown to the Database
=====================================
File Name: /oradata/archivelog/hawk/hawk_46_1_872097259.arc
File Name: /oradata/archivelog/hawk/hawk_45_1_872097259.arc
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: /oradata/archivelog/hawk/hawk_46_1_872097259.arc
File Name: /oradata/archivelog/hawk/hawk_45_1_872097259.arc

Recovery Manager complete.

STANDBY: apply log

[oracle@armor:hawk:/home/oracle]
$ sqlplus / as sysdba @recoverauto.sql

SQL*Plus: Release 11.2.0.4.0 Production on Fri Feb 20 10:16:34 2015

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Release 11.2.0.4.0 - 64bit Production

ARMOR:(SYS@hawk):PHYSICAL STANDBY> recover automatic standby database;
ORA-00279: change 216754 generated at 02/20/2015 10:15:39 needed for thread 1
ORA-00289: suggestion : /oradata/archivelog/hawk/hawk_47_1_872097259.arc
ORA-00280: change 216754 for thread 1 is in sequence #47
ORA-00278: log file '/oradata/archivelog/hawk/hawk_47_1_872097259.arc' no longer needed for this recovery
ORA-00308: cannot open archived log '/oradata/archivelog/hawk/hawk_47_1_872097259.arc'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3


Specify log: {=suggested | filename | AUTO | CANCEL}
cancel
Media recovery cancelled.
ARMOR:(SYS@hawk):PHYSICAL STANDBY> @@stby.sql
ARMOR:(SYS@hawk):PHYSICAL STANDBY> set echo on numw 16 lines 200 pages 1000
ARMOR:(SYS@hawk):PHYSICAL STANDBY> select al.thrd "Thread", almax "Last Seq Received", lhmax "Last Seq Applied"
  2  from
  3  (
  4    select thread# thrd, max(sequence#) almax from v$archived_log
  5    where resetlogs_change#=(select resetlogs_change# from v$database) and applied='YES'
  6    group by thread#
  7  ) al,
  8  (
  9    select thread# thrd, max(sequence#) lhmax
 10    from v$log_history
 11    where first_time=(select max(first_time) from v$log_history)
 12    group by thread#
 13  ) lh
 14  where al.thrd = lh.thrd
 15  ;

          Thread Last Seq Received Last Seq Applied
---------------- ----------------- ----------------
               1                46               46

ARMOR:(SYS@hawk):PHYSICAL STANDBY> select first_time,applied,archived,registrar,creator,thread#,sequence# from v$archived_log
  2  where resetlogs_change#=(select resetlogs_change# from v$database)
  3  and first_time>sysdate-3/24
  4  order by sequence# desc
  5  ;

FIRST_TIME          APPLIED   ARC REGISTR CREATOR          THREAD#        SEQUENCE#
------------------- --------- --- ------- ------- ---------------- ----------------
2015-02-20 10:14:03 YES       YES SRMN    SRMN                   1               46
2015-02-20 10:11:03 NO        YES SRMN    SRMN                   1               45
2015-02-20 10:09:09 YES       YES SRMN    SRMN                   1               44
2015-02-20 10:03:54 NO        YES SRMN    SRMN                   1               43
2015-02-20 09:58:33 YES       YES SRMN    SRMN                   1               42
2015-02-20 09:52:53 YES       YES SRMN    SRMN                   1               41
2015-02-20 09:47:17 YES       YES SRMN    SRMN                   1               40
2015-02-20 09:45:50 YES       YES SRMN    SRMN                   1               39
2015-02-20 09:41:36 YES       YES SRMN    SRMN                   1               38
2015-02-20 09:21:34 YES       YES SRMN    SRMN                   1               37
2015-02-20 09:01:33 YES       YES SRMN    SRMN                   1               36
2015-02-20 08:41:35 YES       YES SRMN    SRMN                   1               35
2015-02-20 08:21:34 YES       YES SRMN    SRMN                   1               34
2015-02-20 08:01:33 YES       YES SRMN    SRMN                   1               33
2015-02-20 07:41:30 YES       YES SRMN    SRMN                   1               32
2015-02-20 07:41:29 YES       YES SRMN    SRMN                   1               31
2015-02-20 07:41:29 YES       YES SRMN    SRMN                   1               30
2015-02-20 07:41:29 YES       YES SRMN    SRMN                   1               29
2015-02-20 07:41:26 YES       YES SRMN    SRMN                   1               28
2015-02-20 07:41:23 YES       YES SRMN    SRMN                   1               27
2015-02-20 07:24:52 YES       YES SRMN    SRMN                   1               26

21 rows selected.

STANDBY: open read only test

ARMOR:(SYS@hawk):PHYSICAL STANDBY> @openreadonly
ARMOR:(SYS@hawk):PHYSICAL STANDBY> set echo on numw 16 lines 200
ARMOR:(SYS@hawk):PHYSICAL STANDBY> alter database open read only;

Database altered.

ARMOR:(SYS@hawk):PHYSICAL STANDBY> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
ARMOR:(SYS@hawk):PHYSICAL STANDBY> startup nomount;
ORACLE instance started.

Total System Global Area       1068937216 bytes
Fixed Size                        2260088 bytes
Variable Size                   281019272 bytes
Database Buffers                780140544 bytes
Redo Buffers                      5517312 bytes
ARMOR:(SYS@hawk):PHYSICAL STANDBY> alter database mount standby database;

Database altered.

ARMOR:(SYS@hawk):PHYSICAL STANDBY> select controlfile_type,open_mode,database_role,db_unique_name,standby_became_primary_scn from v$database
  2  ;

CONTROL OPEN_MODE            DATABASE_ROLE    DB_UNIQUE_NAME                 STANDBY_BECAME_PRIMARY_SCN
------- -------------------- ---------------- ------------------------------ --------------------------
STANDBY MOUNTED              PHYSICAL STANDBY hawk_sfo                                                0

ARMOR:(SYS@hawk):PHYSICAL STANDBY> @@stby.sql
ARMOR:(SYS@hawk):PHYSICAL STANDBY> set echo on numw 16 lines 200 pages 1000
ARMOR:(SYS@hawk):PHYSICAL STANDBY> select al.thrd "Thread", almax "Last Seq Received", lhmax "Last Seq Applied"
  2  from
  3  (
  4    select thread# thrd, max(sequence#) almax from v$archived_log
  5    where resetlogs_change#=(select resetlogs_change# from v$database) and applied='YES'
  6    group by thread#
  7  ) al,
  8  (
  9    select thread# thrd, max(sequence#) lhmax
 10    from v$log_history
 11    where first_time=(select max(first_time) from v$log_history)
 12    group by thread#
 13  ) lh
 14  where al.thrd = lh.thrd
 15  ;

          Thread Last Seq Received Last Seq Applied
---------------- ----------------- ----------------
               1                46               46

ARMOR:(SYS@hawk):PHYSICAL STANDBY> select first_time,applied,archived,registrar,creator,thread#,sequence# from v$archived_log
  2  where resetlogs_change#=(select resetlogs_change# from v$database)
  3  and first_time>sysdate-3/24
  4  order by sequence# desc
  5  ;

FIRST_TIM APPLIED   ARC REGISTR CREATOR          THREAD#        SEQUENCE#
--------- --------- --- ------- ------- ---------------- ----------------
20-FEB-15 YES       YES SRMN    SRMN                   1               46
20-FEB-15 YES       YES SRMN    SRMN                   1               45
20-FEB-15 YES       YES SRMN    SRMN                   1               44
20-FEB-15 YES       YES SRMN    SRMN                   1               43
20-FEB-15 YES       YES SRMN    SRMN                   1               42
20-FEB-15 YES       YES SRMN    SRMN                   1               41
20-FEB-15 YES       YES SRMN    SRMN                   1               40
20-FEB-15 YES       YES SRMN    SRMN                   1               39
20-FEB-15 YES       YES SRMN    SRMN                   1               38
20-FEB-15 YES       YES SRMN    SRMN                   1               37
20-FEB-15 YES       YES SRMN    SRMN                   1               36
20-FEB-15 YES       YES SRMN    SRMN                   1               35
20-FEB-15 YES       YES SRMN    SRMN                   1               34
20-FEB-15 YES       YES SRMN    SRMN                   1               33
20-FEB-15 YES       YES SRMN    SRMN                   1               32
20-FEB-15 YES       YES SRMN    SRMN                   1               31
20-FEB-15 YES       YES SRMN    SRMN                   1               30
20-FEB-15 YES       YES SRMN    SRMN                   1               29
20-FEB-15 YES       YES SRMN    SRMN                   1               28
20-FEB-15 YES       YES SRMN    SRMN                   1               27
20-FEB-15 YES       YES SRMN    SRMN                   1               26

21 rows selected.

ARMOR:(SYS@hawk):PHYSICAL STANDBY> select name from v$tablespace;

NAME
------------------------------
SYSTEM
SYSAUX
UNDOTBS1
TEMP
USERS
NEW1
NEW2

7 rows selected.

ARMOR:(SYS@hawk):PHYSICAL STANDBY>

PRIMARY: configuration

ANGEL:(SYS@hawk):PRIMARY> show parameter name

NAME                      TYPE        VALUE
------------------------- ----------- ----------------------------------------
cell_offloadgroup_name    string
db_file_name_convert      string
db_name                   string      hawk
db_unique_name            string      hawk
global_names              boolean     FALSE
instance_name             string      hawk
lock_name_space           string
log_file_name_convert     string
processor_group_name      string
service_names             string      hawk
ANGEL:(SYS@hawk):PRIMARY> show parameter convert

NAME                      TYPE        VALUE
------------------------- ----------- ----------------------------------------
db_file_name_convert      string
log_file_name_convert     string
ANGEL:(SYS@hawk):PRIMARY> show parameter standby

NAME                      TYPE        VALUE
------------------------- ----------- ----------------------------------------
standby_archive_dest      string      ?/dbs/arch
standby_file_management   string      AUTO
ANGEL:(SYS@hawk):PRIMARY> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
/oradata/HAWK/datafile/o1_mf_system_bgf2mflo_.dbf
/oradata/HAWK/datafile/o1_mf_sysaux_bgf2mj87_.dbf
/oradata/HAWK/datafile/o1_mf_undotbs1_bgf2mkds_.dbf
/oradata/HAWK/datafile/o1_mf_users_bgf2mlf8_.dbf
/oradata/HAWK/datafile/o1_mf_new1_bggyc50z_.dbf
/oradata/HAWK/datafile/o1_mf_new2_bggyg5ky_.dbf

6 rows selected.

ANGEL:(SYS@hawk):PRIMARY>

STANDBY: configuration

ARMOR:(SYS@hawk):PHYSICAL STANDBY> show parameter name

NAME                      TYPE        VALUE
------------------------- ----------- ----------------------------------------
cell_offloadgroup_name    string
db_file_name_convert      string
db_name                   string      hawk
db_unique_name            string      hawk_sfo
global_names              boolean     FALSE
instance_name             string      hawk
lock_name_space           string
log_file_name_convert     string
processor_group_name      string
service_names             string      hawk
ARMOR:(SYS@hawk):PHYSICAL STANDBY> show parameter convert

NAME                      TYPE        VALUE
------------------------- ----------- ----------------------------------------
db_file_name_convert      string
log_file_name_convert     string
ARMOR:(SYS@hawk):PHYSICAL STANDBY> show parameter standby

NAME                      TYPE        VALUE
------------------------- ----------- ----------------------------------------
standby_archive_dest      string      ?/dbs/arch
standby_file_management   string      AUTO
ARMOR:(SYS@hawk):PHYSICAL STANDBY> select name from v$datafile;

NAME
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/oradata/HAWK_SFO/datafile/o1_mf_system_bgf4o5x7_.dbf
/oradata/HAWK_SFO/datafile/o1_mf_sysaux_bgf4on2c_.dbf
/oradata/HAWK_SFO/datafile/o1_mf_undotbs1_bgf4of05_.dbf
/oradata/HAWK_SFO/datafile/o1_mf_users_bgf4oq4b_.dbf
/oradata/HAWK_SFO/datafile/o1_mf_new1_bggydmsq_.dbf
/oradata/HAWK_SFO/datafile/o1_mf_new2_bggyj2n3_.dbf

6 rows selected.

ARMOR:(SYS@hawk):PHYSICAL STANDBY>

PRIMARY: drop tablespace and register/apply log to STANDBY – review alert log

[oracle@armor:hawk:/u01/app/oracle/diag/rdbms/hawk_sfo/hawk/trace]
$ cat alert_hawk.log
Fri Feb 20 11:23:18 2015
ALTER DATABASE RECOVER  automatic standby database
Media Recovery Start
 started logmerger process
Fri Feb 20 11:23:18 2015
Managed Standby Recovery not using Real Time Apply
Parallel Media Recovery started with 2 slaves
Media Recovery Log /oradata/archivelog/hawk/hawk_55_1_872097259.arc
Recovery deleting file #5:'/oradata/HAWK_SFO/datafile/o1_mf_new1_bggydmsq_.dbf' from controlfile.
Deleted Oracle managed file /oradata/HAWK_SFO/datafile/o1_mf_new1_bggydmsq_.dbf
Recovery dropped tablespace 'NEW1'
Recovery deleting file #6:'/oradata/HAWK_SFO/datafile/o1_mf_new2_bggyj2n3_.dbf' from controlfile.
Deleted Oracle managed file /oradata/HAWK_SFO/datafile/o1_mf_new2_bggyj2n3_.dbf
Recovery dropped tablespace 'NEW2'
Media Recovery Log /oradata/archivelog/hawk/hawk_56_1_872097259.arc
Errors with log /oradata/archivelog/hawk/hawk_56_1_872097259.arc
Errors in file /u01/app/oracle/diag/rdbms/hawk_sfo/hawk/trace/hawk_pr00_13085.trc:
ORA-00308: cannot open archived log '/oradata/archivelog/hawk/hawk_56_1_872097259.arc'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-279 signalled during: ALTER DATABASE RECOVER  automatic standby database  ...
ALTER DATABASE RECOVER    CANCEL
Media Recovery Canceled
Completed: ALTER DATABASE RECOVER    CANCEL
[oracle@armor:hawk:/u01/app/oracle/diag/rdbms/hawk_sfo/hawk/trace]
$

February 16, 2015

Using rsync to clone Goldengate installation

Filed under: GoldenGate — mdinh @ 3:15 pm

You may be thinking, why clone Goldengate and why now just download it?
The exact version and patch level might not be available.
Too lazy to search for it and many other reasons you can come up with.

Why use rsync and not tar – scp? I wanted to refresh memory of using rsync.

Commands used:

local source /u01/app/ggs01/ and remote target arrow:/u01/app/ggs03/

rsync -avh --delete --dry-run --exclude 'dirdatold' /u01/app/ggs01/ arrow:/u01/app/ggs03/
rsync -avh --delete --exclude 'dirdatold' /u01/app/ggs01/ arrow:/u01/app/ggs03/

Note:
/u01/app/ggs01/ means synch contents of directory to target
/u01/app/ggs01 means create ggs01 directory and sync contents to target

Demo:

Source: /u01/app/ggs01 and dirdata is symbolic link

oracle@arrow:las:/u01/app/ggs01
$ ls -ld dir*
drwxr-x---. 2 oracle oinstall 4096 Jan 13 13:12 dirchk
lrwxrwxrwx. 1 oracle oinstall   15 Feb 15 06:20 dirdat -> /oradata/backup
drwxr-x---. 2 oracle oinstall 4096 Jul 22  2014 dirdatold
drwxr-x---. 2 oracle oinstall 4096 Apr 26  2014 dirdef
drwxr-x---. 2 oracle oinstall 4096 Apr  4  2014 dirjar
drwxr-x---. 2 oracle oinstall 4096 Apr 26  2014 dirout
drwxr-x---. 2 oracle oinstall 4096 Feb 12 15:35 dirpcs
drwxr-x---. 2 oracle oinstall 4096 Jan 13 12:55 dirprm
drwxr-x---. 2 oracle oinstall 4096 Feb 12 15:36 dirrpt
drwxr-x---. 2 oracle oinstall 4096 Apr 26  2014 dirsql
drwxr-x---. 2 oracle oinstall 4096 Sep 25 08:56 dirtmp

Target: arrow:/u01/app/ggs03/

oracle@arrow:las:/u01/app/ggs01
$ ls -l /u01/app/ggs03/
total 0

Let’s do a dry run first.

oracle@arrow:las:/u01/app/ggs01

$ rsync -avh --delete --dry-run --exclude 'dirdatold' /u01/app/ggs01/ arrow:/u01/app/ggs03/
oracle@arrow's password:

sending incremental file list
./

.....

output ommited for brevity

dirout/
dirpcs/
dirprm/
dirprm/esan.prm
dirprm/jagent.prm
dirprm/mgr.prm
dirrpt/
dirrpt/ESAN.rpt
dirrpt/ESAN0.rpt
dirrpt/ESAN1.rpt
dirrpt/ESAN2.rpt
dirrpt/ESAN3.rpt
dirrpt/ESAN4.rpt
dirrpt/ESAN5.rpt
dirrpt/ESAN6.rpt
dirrpt/ESAN7.rpt
dirrpt/ESAN8.rpt
dirrpt/ESAN9.rpt
dirrpt/MGR.rpt
dirrpt/MGR0.rpt
dirrpt/MGR1.rpt
dirrpt/MGR2.rpt
dirrpt/MGR3.rpt
dirrpt/MGR4.rpt
dirrpt/MGR5.rpt
dirrpt/MGR6.rpt
dirrpt/MGR7.rpt
dirrpt/MGR8.rpt
dirrpt/MGR9.rpt
dirsql/
dirtmp/


sent 6.96K bytes  received 767 bytes  15.44K bytes/sec
total size is 237.10M  speedup is 30704.36 (DRY RUN)

oracle@arrow:las:/u01/app/ggs01
$ ls -l /u01/app/ggs03/
total 0

Perform actual rsync

oracle@arrow:las:/u01/app/ggs01

$ rsync -avh --delete --exclude 'dirdatold' /u01/app/ggs01/ arrow:/u01/app/ggs03/
oracle@arrow's password:

sending incremental file list
./

.....

output ommited for brevity

dirout/
dirpcs/
dirprm/
dirprm/esan.prm
dirprm/jagent.prm
dirprm/mgr.prm
dirrpt/
dirrpt/ESAN.rpt
dirrpt/ESAN0.rpt
dirrpt/ESAN1.rpt
dirrpt/ESAN2.rpt
dirrpt/ESAN3.rpt
dirrpt/ESAN4.rpt
dirrpt/ESAN5.rpt
dirrpt/ESAN6.rpt
dirrpt/ESAN7.rpt
dirrpt/ESAN8.rpt
dirrpt/ESAN9.rpt
dirrpt/MGR.rpt
dirrpt/MGR0.rpt
dirrpt/MGR1.rpt
dirrpt/MGR2.rpt
dirrpt/MGR3.rpt
dirrpt/MGR4.rpt
dirrpt/MGR5.rpt
dirrpt/MGR6.rpt
dirrpt/MGR7.rpt
dirrpt/MGR8.rpt
dirrpt/MGR9.rpt
dirsql/
dirtmp/

sent 237.14M bytes  received 4.40K bytes  31.62M bytes/sec
total size is 237.10M  speedup is 1.00

oracle@arrow:las:/u01/app/ggs01
$ ls -ld /u01/app/ggs03/dir*
drwxr-x---. 2 oracle oinstall 4096 Jan 13 13:12 /u01/app/ggs03/dirchk

lrwxrwxrwx. 1 oracle oinstall   15 Feb 15 06:20 /u01/app/ggs03/dirdat -> /oradata/backup

drwxr-x---. 2 oracle oinstall 4096 Apr 26  2014 /u01/app/ggs03/dirdef
drwxr-x---. 2 oracle oinstall 4096 Apr  4  2014 /u01/app/ggs03/dirjar
drwxr-x---. 2 oracle oinstall 4096 Apr 26  2014 /u01/app/ggs03/dirout
drwxr-x---. 2 oracle oinstall 4096 Feb 12 15:35 /u01/app/ggs03/dirpcs
drwxr-x---. 2 oracle oinstall 4096 Jan 13 12:55 /u01/app/ggs03/dirprm
drwxr-x---. 2 oracle oinstall 4096 Feb 12 15:36 /u01/app/ggs03/dirrpt
drwxr-x---. 2 oracle oinstall 4096 Apr 26  2014 /u01/app/ggs03/dirsql
drwxr-x---. 2 oracle oinstall 4096 Sep 25 08:56 /u01/app/ggs03/dirtmp
oracle@arrow:las:/u01/app/ggs01
$

Did it work?

oracle@arrow:las:/u01/app/ggs01
$ cd /u01/app/ggs03/
oracle@arrow:las:/u01/app/ggs03
$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 11.2.1.0.21 18343248 OGGCORE_11.2.1.0.0OGGBP_PLATFORMS_140404.1029_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Apr  4 2014 15:18:36

Copyright (C) 1995, 2014, Oracle and/or its affiliates. All rights reserved.



GGSCI (arrow.localdomain) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED
EXTRACT     STOPPED     ESAN        00:01:02      788:32:31
REPLICAT    STOPPED     RLAS_SAN    00:00:00      4989:14:29


GGSCI (arrow.localdomain) 2> exit

The above was from a neglected test environment
Delete details for extract/replicat at dirchk

oracle@arrow:las:/u01/app/ggs03
$ cd dirchk/
oracle@arrow:las:/u01/app/ggs03/dirchk
$ ll
total 8
-rw-r-----. 1 oracle oinstall 2048 Jan 13 13:12 ESAN.cpe
-rw-r-----. 1 oracle oinstall 2048 Jul 22  2014 RLAS_SAN.cpr
oracle@arrow:las:/u01/app/ggs03/dirchk
$ rm *
oracle@arrow:las:/u01/app/ggs03/dirchk
$ cd ../dirpcs/
oracle@arrow:las:/u01/app/ggs03/dirpcs
$ ll
total 0
oracle@arrow:las:/u01/app/ggs03/dirpcs
$ cd ..
oracle@arrow:las:/u01/app/ggs03
$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 11.2.1.0.21 18343248 OGGCORE_11.2.1.0.0OGGBP_PLATFORMS_140404.1029_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Apr  4 2014 15:18:36

Copyright (C) 1995, 2014, Oracle and/or its affiliates. All rights reserved.



GGSCI (arrow.localdomain) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED


GGSCI (arrow.localdomain) 2> start mgr

Manager started.


GGSCI (arrow.localdomain) 3> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED


GGSCI (arrow.localdomain) 4> exit

Now, what’s wrong?

oracle@arrow:las:/u01/app/ggs03
$ tail ggserr.log
2015-02-12 15:36:02  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): start mgr.
2015-02-12 15:36:02  ERROR   OGG-00664  Oracle GoldenGate Manager for Oracle, mgr.prm:  OCI Error during OCIServerAttach (status = 12541-ORA-12541: TNS:no listener).
2015-02-12 15:36:02  ERROR   OGG-01668  Oracle GoldenGate Manager for Oracle, mgr.prm:  PROCESS ABENDING.
2015-02-12 15:36:04  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): info all.
2015-02-15 09:44:51  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): info all.
2015-02-15 09:45:31  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): info all.
2015-02-15 09:45:41  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): start mgr.
2015-02-15 09:45:43  ERROR   OGG-00664  Oracle GoldenGate Manager for Oracle, mgr.prm:  OCI Error during OCIServerAttach (status = 12541-ORA-12541: TNS:no listener).
2015-02-15 09:45:43  ERROR   OGG-01668  Oracle GoldenGate Manager for Oracle, mgr.prm:  PROCESS ABENDING.
2015-02-15 09:45:44  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): info all.

oracle@arrow:las:/u01/app/ggs03
$ cat dirprm/mgr.prm
PORT 7901
DYNAMICPORTLIST 15100-15120

USERID ggs@san, PASSWORD *****

PURGEOLDEXTRACTS /u01/app/ggs01/dirdat/*, USECHECKPOINTS, MINKEEPDAYS 3
PURGEMARKERHISTORY MINKEEPDAYS 5, MAXKEEPDAYS 7, FREQUENCYHOURS 24
PURGEDDLHISTORY MINKEEPDAYS 5, MAXKEEPDAYS 7, FREQUENCYHOURS 24

AUTOSTART ER *
AUTORESTART ER *, RETRIES 5, WAITMINUTES 2, RESETMINUTES 60

CHECKMINUTES 1
LAGINFOMINUTES 0
LAGCRITICALMINUTES 1

oracle@arrow:las:/u01/app/ggs03
$ vi dirprm/mgr.prm

oracle@arrow:las:/u01/app/ggs03
$ cat dirprm/mgr.prm
PORT 7901
DYNAMICPORTLIST 15100-15120

-- USERID ggs@san, PASSWORD 888

PURGEOLDEXTRACTS /u01/app/ggs01/dirdat/*, USECHECKPOINTS, MINKEEPDAYS 3
PURGEMARKERHISTORY MINKEEPDAYS 5, MAXKEEPDAYS 7, FREQUENCYHOURS 24
PURGEDDLHISTORY MINKEEPDAYS 5, MAXKEEPDAYS 7, FREQUENCYHOURS 24

AUTOSTART ER *
AUTORESTART ER *, RETRIES 5, WAITMINUTES 2, RESETMINUTES 60

CHECKMINUTES 1
LAGINFOMINUTES 0
LAGCRITICALMINUTES 1
oracle@arrow:las:/u01/app/ggs03
$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 11.2.1.0.21 18343248 OGGCORE_11.2.1.0.0OGGBP_PLATFORMS_140404.1029_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Apr  4 2014 15:18:36

Copyright (C) 1995, 2014, Oracle and/or its affiliates. All rights reserved.



GGSCI (arrow.localdomain) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED


GGSCI (arrow.localdomain) 2> start mgr

Manager started.


GGSCI (arrow.localdomain) 3> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING


GGSCI (arrow.localdomain) 4> exit
oracle@arrow:las:/u01/app/ggs03
$

Don’t forget Oracle libraries are required to run Goldengate

oracle@arrow:las:/u01/app/ggs03
$ ldd ggsci
        linux-vdso.so.1 =>  (0x00007fff95ffa000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00000039e6000000)
        libgglog.so => /u01/app/ggs03/./libgglog.so (0x00007f862ca8d000)
        libggrepo.so => /u01/app/ggs03/./libggrepo.so (0x00007f862c923000)
        libdb-5.2.so => /u01/app/ggs03/./libdb-5.2.so (0x00007f862c688000)
        libicui18n.so.38 => /u01/app/ggs03/./libicui18n.so.38 (0x00007f862c327000)
        libicuuc.so.38 => /u01/app/ggs03/./libicuuc.so.38 (0x00007f862bfee000)
        libicudata.so.38 => /u01/app/ggs03/./libicudata.so.38 (0x00007f862b012000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00000039e6800000)
        libxerces-c.so.28 => /u01/app/ggs03/./libxerces-c.so.28 (0x00007f862aafa000)
        libantlr3c.so => /u01/app/ggs03/./libantlr3c.so (0x00007f862a9e4000)

        libnnz11.so => /u01/app/oracle/product/11.2.0/dbhome_1/lib/libnnz11.so (0x00007f862a616000)
        libclntsh.so.11.1 => /u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1 (0x00007f8627ba0000)

        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000039f1c00000)
        libm.so.6 => /lib64/libm.so.6 (0x00000039e7400000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000039f1800000)
        libc.so.6 => /lib64/libc.so.6 (0x00000039e6400000)
        /lib64/ld-linux-x86-64.so.2 (0x00000039e5c00000)
        libnsl.so.1 => /lib64/libnsl.so.1 (0x00000039f3400000)
        libaio.so.1 => /lib64/libaio.so.1 (0x00007f862799d000)

oracle@arrow:las:/u01/app/ggs03
$ env |egrep 'HOME|LD'
OLDPWD=/home/oracle
LD_LIBRARY_PATH=/u01/app/oracle/product/11.2.0/dbhome_1/lib:/lib:/usr/lib
HOME=/home/oracle
ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
oracle@arrow:las:/u01/app/ggs03
$ unset ORACLE_HOME
oracle@arrow:las:/u01/app/ggs03
$ export LD_LIBRARY_PATH=/lib:/usr/lib
oracle@arrow:las:/u01/app/ggs03
$ ./ggsci
./ggsci: error while loading shared libraries: libnnz11.so: cannot open shared object file: No such file or directory
oracle@arrow:las:/u01/app/ggs03
$

February 10, 2015

Goldengate – start replicat ATSCN or AFTERSCN ?

Filed under: DataPump,GoldenGate,RMAN — mdinh @ 3:36 am

When using Goldengate to instantiate target database from an Oracle source database, replicat process can be started to concide with extract based the method used for instantiation, e.g. RMAN or datapump.

ATSCN is used to start replicat if RMAN is used to instantiate target.
From Database Backup and Recovery Reference, UNTIL SCN specifies an SCN as an upper limit.
RMAN restore or recover up to but not including the specified SCN.

AFTERSCN is used to start replicat if datapump is used to instantiate target.
The export operation performed is consistent as of FLASHBACK_SCN.

Hope this helps to clear up when to use ATSCN versus AFTERSCN.

Referemce:
Oracle GoldenGate Best Practices: Instantiation from an Oracle Source Database –  Doc ID 1276058.1

December 14, 2014

GOLDENDATE HA MAA RAC ACFS XAG

Filed under: 11g,GoldenGate,RAC — mdinh @ 11:39 pm

Purpose is to demonstrate how to create HA for Bi-Directional Replication Goldengate installed on ACFS with RAC cluster using XAG.

XAG simplifies the process since there are no requirements to create action scripts.

Please review REFERENCE section for versions used in test case and versions requirements.

Goldengate is installed on ACFS for simplicity; otherwise, at a minimum the following directories br, dirchk, dirdat, dirtmp will need to be on shared storage with symbolic links if installed on local storage. Keyword is minimum until you find out more directories are required.

Role separation was a huge PITA and do not attempt to perform chmod -R 775 /u01 as it will break since the setuid get unset.
Even with chmod 6751 oracle may prove to be ineffective and relink was done.

# id grid
uid=54322(grid) gid=54321(oinstall) groups=54321(oinstall),54322(dba),493(vboxsf),54323(asmadmin),54324(asmdba)

# id oracle
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),493(vboxsf),54324(asmdba)

# id gguser
uid=54323(gguser) gid=54321(oinstall) groups=54321(oinstall),54322(dba),493(vboxsf)

$ ls -l $ORACLE_HOME/bin/oracle
-rwsr-s--x. 1 grid oinstall 209914519 Dec  7 20:57 /u01/app/11.2.0.4/grid/bin/oracle

Last, DO NOT stop Goldengate using ggsci. For cluster-aware Goldengate, use $XAG_HOME/bin/agctl.

The configuration shown is for one cluster only and the same would have to be performed on the other clusters.

AS ROOT, CONFIGURE GOLDENGATE VIP (lax-ggate1-vip)
1. Determine the network number

# crsctl stat res -p|grep -ie .network -ie subnet|grep -ie name -ie ora_subnet
NAME=ora.net1.network
USR_ORA_SUBNET=192.168.56.0

2. Create GoldenGate VIP with naming Convention: LAX (closest airport code to data center), ggate1 (for Goldengate on network number 1), vip
Note IP address provided is 192.168.56.41

# appvipcfg create -network=1 -ip=192.168.56.41 -vipname=lax-ggate1-vip -user=root -group=oinstall
Production Copyright 2007, 2008, Oracle.All rights reserved
2014-12-14 10:58:41: Creating Resource Type
2014-12-14 10:58:41: Executing /u01/app/11.2.0.4/grid/bin/crsctl add type app.appvip_net1.type -basetype ora.cluster_vip_net1.type -file /u01/app/11.2.0.4/grid/crs/template/appvip.type
2014-12-14 10:58:41: Executing cmd: /u01/app/11.2.0.4/grid/bin/crsctl add type app.appvip_net1.type -basetype ora.cluster_vip_net1.type -file /u01/app/11.2.0.4/grid/crs/template/appvip.type
2014-12-14 10:58:42: Create the Resource
2014-12-14 10:58:42: Executing /u01/app/11.2.0.4/grid/bin/crsctl add resource lax-ggate1-vip -type app.appvip_net1.type -attr "USR_ORA_VIP=192.168.56.41,START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network),STOP_DEPENDENCIES=hard(ora.net1.network),ACL='owner:root:rwx,pgrp:root:r-x,other::r--,group:oinstall:r-x,user:root:r-x',HOSTING_MEMBERS=rac01.localdomain,APPSVIP_FAILBACK="
2014-12-14 10:58:42: Executing cmd: /u01/app/11.2.0.4/grid/bin/crsctl add resource lax-ggate1-vip -type app.appvip_net1.type -attr "USR_ORA_VIP=192.168.56.41,START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network),STOP_DEPENDENCIES=hard(ora.net1.network),ACL='owner:root:rwx,pgrp:root:r-x,other::r--,group:oinstall:r-x,user:root:r-x',HOSTING_MEMBERS=rac01.localdomain,APPSVIP_FAILBACK="

3. Set permission for users to start/stop VIP

crsctl setperm resource lax-ggate1-vip -u user:oracle:r-x
# crsctl setperm resource lax-ggate1-vip -u user:grid:r-x
# crsctl setperm resource lax-ggate1-vip -u user:gguser:r-x
# crsctl start resource lax-ggate1-vip
CRS-2672: Attempting to start 'lax-ggate1-vip' on 'rac02'
CRS-2676: Start of 'lax-ggate1-vip' on 'rac02' succeeded

RETRIEVE INFORMATIONS TO CREATE GOLDENGATE AGENT
–nodes

$ olsnodes
rac01
rac02

–filesystems

$ crsctl stat res -w "TYPE = ora.acfs.type" -p|grep '^NAME'
NAME=ora.dg_acfs.vg_acfs.acfs
NAME=ora.dg_acfs.vg_acfs.acfs

–databases

$ crsctl stat res -w "TYPE = ora.database.type"
NAME=ora.emu.db
TYPE=ora.database.type
TARGET=ONLINE         , ONLINE
STATE=ONLINE on rac01, ONLINE on rac02

— extracts and replicats

GGSCI (rac02.localdomain) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     ELAX        00:00:00      00:00:02
EXTRACT     RUNNING     PLAX_DEN    00:00:00      00:00:03
REPLICAT    RUNNING     RDEN_LAX    00:00:00      00:00:06

AS GRID, CONFIGURE GOLDENGATE AGENT (lax_ggate)
Options for –instance_type is source|target|dual and I am interpreting dual is bi-directional.

$XAG_HOME/bin/agctl add goldengate lax_ggate \
--gg_home /acfsmount/ggs112 \
--instance_type dual \
--nodes rac01,rac02 \
--vip_name lax-ggate1-vip \
--filesystems ora.dg_acfs.vg_acfs.acfs \
--databases ora.emu.db \
--oracle_home /u01/app/oracle/product/11.2.0.4/db_1 \
--monitor_extracts ELAX,PLAX_DEN \
--critical_extracts ELAX,PLAX_DEN \
--monitor_replicats RDEN_LAX \
--critical_replicats RDEN_LAX

VERIFY RESULTS

$ $XAG_HOME/bin/agctl config goldengate lax_ggate
GoldenGate location is: /acfsmount/ggs112
GoldenGate instance type is: dual
Configured to run on Nodes: rac01 rac02
ORACLE_HOME location is: /u01/app/oracle/product/11.2.0.4/db_1
Databases needed: ora.emu.db
File System resources needed: ora.dg_acfs.vg_acfs.acfs
Extracts to monitor: ELAX,PLAX_DEN
Replicats to monitor: RDEN_LAX
Critical extracts: ELAX,PLAX_DEN
Critical replicats: RDEN_LAX
Autostart on DataGuard role transition to PRIMARY: no
Autostart JAgent: no

START GOLDENDATE

$XAG_HOME/bin/agctl start goldengate lax_ggate

CHECK GOLDENGATE STATUS

$XAG_HOME/bin/agctl status goldengate lax_ggate
Goldengate  instance 'lax_ggate' INTERMEDIATE on rac02

$XAG_HOME/bin/agctl status goldengate lax_ggate
Goldengate  instance 'lax_ggate' is running on rac02
$ crsstat

Resource Name                            Resource Type  Target       State        Node            State Details
---------------------------------------- -------------- ------------ ------------ --------------- ---------------
lax-ggate1-vip                           appvip_net1    ONLINE       ONLINE       rac02
ora.DATA2.dg                             diskgroup      ONLINE       ONLINE       rac01
ora.DATA2.dg                             diskgroup      ONLINE       ONLINE       rac02
ora.DG_ACFS.dg                           diskgroup      ONLINE       ONLINE       rac01
ora.DG_ACFS.dg                           diskgroup      ONLINE       ONLINE       rac02
ora.LISTENER.lsnr                        Listener       ONLINE       ONLINE       rac01
ora.LISTENER.lsnr                        Listener       ONLINE       ONLINE       rac02
ora.LISTENER_SCAN1.lsnr                  SCAN Listener  ONLINE       ONLINE       rac01
ora.LISTENER_SCAN2.lsnr                  SCAN Listener  ONLINE       ONLINE       rac02
ora.LISTENER_SCAN3.lsnr                  SCAN Listener  ONLINE       ONLINE       rac02
ora.asm                                  ASM            ONLINE       ONLINE       rac01           Started
ora.asm                                  ASM            ONLINE       ONLINE       rac02           Started
ora.cvu                                  cvu            ONLINE       ONLINE       rac02
ora.dg_acfs.vg_acfs.acfs                 acfs           ONLINE       ONLINE       rac01           mounted on /acfsmount
ora.dg_acfs.vg_acfs.acfs                 acfs           ONLINE       ONLINE       rac02           mounted on /acfsmount
ora.emu.db                               database       ONLINE       ONLINE       rac01           Open
ora.emu.db                               database       ONLINE       ONLINE       rac02           Open
ora.gsd                                  Gbl Svc Daemon OFFLINE      OFFLINE
ora.gsd                                  Gbl Svc Daemon OFFLINE      OFFLINE
ora.net1.network                         Network (VIP)  ONLINE       ONLINE       rac01
ora.net1.network                         Network (VIP)  ONLINE       ONLINE       rac02
ora.oc4j                                 OC4J           ONLINE       ONLINE       rac02
ora.ons                                  Ora Notif Svc  ONLINE       ONLINE       rac01
ora.ons                                  Ora Notif Svc  ONLINE       ONLINE       rac02
ora.rac01.vip                            Cluster VIP    ONLINE       ONLINE       rac01
ora.rac02.vip                            Cluster VIP    ONLINE       ONLINE       rac02
ora.registry.acfs                        registry       ONLINE       ONLINE       rac01
ora.registry.acfs                        registry       ONLINE       ONLINE       rac02
ora.scan1.vip                            SCAN VIP       ONLINE       ONLINE       rac01
ora.scan2.vip                            SCAN VIP       ONLINE       ONLINE       rac02
ora.scan3.vip                            SCAN VIP       ONLINE       ONLINE       rac02
xag.lax_ggate.goldengate                 goldengate     ONLINE       ONLINE       rac02

CONNECT TO THE CORRECT NODE FOR GOLDENGATE

$ su - gguser
Password:
0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
!!!!! PLEASE CONNECT TO NODE WHERE GOLDENGATE IS RUNNING !!!!!
!!!!!                                                    !!!!!
--+ Goldengate  instance 'lax_ggate' is running on rac02 +--
!!!!!                                                    !!!!!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[gguser@rac01:/home/gguser]
$ ssh rac02
Last login: Sun Dec 14 13:23:48 2014 from rac01
1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~ GOLDENGATE IS RUNNING ON THIS NODE - GOOD TO GO    ~~~~~
--+ Goldengate  instance 'lax_ggate' is running on rac02 +--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[gguser@rac02:/home/gguser]
$

.bash_profile
# User specific environment and startup programs
ogg=`/u01/app/grid/xag/bin/agctl status goldengate lax_ggate`
/u01/app/grid/xag/bin/agctl status goldengate lax_ggate|grep -ic `hostname -f`
if [ "$?" != "0" ]; then
  clear
  echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
  echo "!!!!! PLEASE CONNECT TO NODE WHERE GOLDENGATE IS RUNNING !!!!!"
  echo "!!!!!                                                    !!!!!"
  echo "--+ $ogg +--"
  echo "!!!!!                                                    !!!!!"
  echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
else
  clear
  echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
  echo "~~~~~ GOLDENGATE IS RUNNING ON THIS NODE - GOOD TO GO    ~~~~~"
  echo "--+ $ogg +--"
  echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
fi
umask 0002

RELOCATE GOLDENGATE TO DIFFERENT NODE

[grid@rac01:+ASM1:/home/grid]
$ crsstat


Resource Name                            Resource Type  Target       State        Node            State Details
---------------------------------------- -------------- ------------ ------------ --------------- ---------------
lax-ggate1-vip                           appvip_net1    ONLINE       ONLINE       rac02
ora.DATA2.dg                             diskgroup      ONLINE       ONLINE       rac01
ora.DATA2.dg                             diskgroup      ONLINE       ONLINE       rac02
ora.DG_ACFS.dg                           diskgroup      ONLINE       ONLINE       rac01
ora.DG_ACFS.dg                           diskgroup      ONLINE       ONLINE       rac02
ora.LISTENER.lsnr                        Listener       ONLINE       ONLINE       rac01
ora.LISTENER.lsnr                        Listener       ONLINE       ONLINE       rac02
ora.LISTENER_SCAN1.lsnr                  SCAN Listener  ONLINE       ONLINE       rac01
ora.LISTENER_SCAN2.lsnr                  SCAN Listener  ONLINE       ONLINE       rac02
ora.LISTENER_SCAN3.lsnr                  SCAN Listener  ONLINE       ONLINE       rac02
ora.asm                                  ASM            ONLINE       ONLINE       rac01           Started
ora.asm                                  ASM            ONLINE       ONLINE       rac02           Started
ora.cvu                                  cvu            ONLINE       ONLINE       rac02
ora.dg_acfs.vg_acfs.acfs                 acfs           ONLINE       ONLINE       rac01           mounted on /acfsmount
ora.dg_acfs.vg_acfs.acfs                 acfs           ONLINE       ONLINE       rac02           mounted on /acfsmount
ora.emu.db                               database       ONLINE       ONLINE       rac01           Open
ora.emu.db                               database       ONLINE       ONLINE       rac02           Open
ora.gsd                                  Gbl Svc Daemon OFFLINE      OFFLINE
ora.gsd                                  Gbl Svc Daemon OFFLINE      OFFLINE
ora.net1.network                         Network (VIP)  ONLINE       ONLINE       rac01
ora.net1.network                         Network (VIP)  ONLINE       ONLINE       rac02
ora.oc4j                                 OC4J           ONLINE       ONLINE       rac02
ora.ons                                  Ora Notif Svc  ONLINE       ONLINE       rac01
ora.ons                                  Ora Notif Svc  ONLINE       ONLINE       rac02
ora.rac01.vip                            Cluster VIP    ONLINE       ONLINE       rac01
ora.rac02.vip                            Cluster VIP    ONLINE       ONLINE       rac02
ora.registry.acfs                        registry       ONLINE       ONLINE       rac01
ora.registry.acfs                        registry       ONLINE       ONLINE       rac02
ora.scan1.vip                            SCAN VIP       ONLINE       ONLINE       rac01
ora.scan2.vip                            SCAN VIP       ONLINE       ONLINE       rac02
ora.scan3.vip                            SCAN VIP       ONLINE       ONLINE       rac02
xag.lax_ggate.goldengate                 goldengate     ONLINE       ONLINE       rac02

[grid@rac01:+ASM1:/home/grid]
$ $XAG_HOME/bin/agctl relocate goldengate lax_ggate --node rac01
[grid@rac01:+ASM1:/home/grid]
$ crsstat


Resource Name                            Resource Type  Target       State        Node            State Details
---------------------------------------- -------------- ------------ ------------ --------------- ---------------
lax-ggate1-vip                           appvip_net1    ONLINE       ONLINE       rac01
ora.DATA2.dg                             diskgroup      ONLINE       ONLINE       rac01
ora.DATA2.dg                             diskgroup      ONLINE       ONLINE       rac02
ora.DG_ACFS.dg                           diskgroup      ONLINE       ONLINE       rac01
ora.DG_ACFS.dg                           diskgroup      ONLINE       ONLINE       rac02
ora.LISTENER.lsnr                        Listener       ONLINE       ONLINE       rac01
ora.LISTENER.lsnr                        Listener       ONLINE       ONLINE       rac02
ora.LISTENER_SCAN1.lsnr                  SCAN Listener  ONLINE       ONLINE       rac01
ora.LISTENER_SCAN2.lsnr                  SCAN Listener  ONLINE       ONLINE       rac02
ora.LISTENER_SCAN3.lsnr                  SCAN Listener  ONLINE       ONLINE       rac02
ora.asm                                  ASM            ONLINE       ONLINE       rac01           Started
ora.asm                                  ASM            ONLINE       ONLINE       rac02           Started
ora.cvu                                  cvu            ONLINE       ONLINE       rac02
ora.dg_acfs.vg_acfs.acfs                 acfs           ONLINE       ONLINE       rac01           mounted on /acfsmount
ora.dg_acfs.vg_acfs.acfs                 acfs           ONLINE       ONLINE       rac02           mounted on /acfsmount
ora.emu.db                               database       ONLINE       ONLINE       rac01           Open
ora.emu.db                               database       ONLINE       ONLINE       rac02           Open
ora.gsd                                  Gbl Svc Daemon OFFLINE      OFFLINE
ora.gsd                                  Gbl Svc Daemon OFFLINE      OFFLINE
ora.net1.network                         Network (VIP)  ONLINE       ONLINE       rac01
ora.net1.network                         Network (VIP)  ONLINE       ONLINE       rac02
ora.oc4j                                 OC4J           ONLINE       ONLINE       rac02
ora.ons                                  Ora Notif Svc  ONLINE       ONLINE       rac01
ora.ons                                  Ora Notif Svc  ONLINE       ONLINE       rac02
ora.rac01.vip                            Cluster VIP    ONLINE       ONLINE       rac01
ora.rac02.vip                            Cluster VIP    ONLINE       ONLINE       rac02
ora.registry.acfs                        registry       ONLINE       ONLINE       rac01
ora.registry.acfs                        registry       ONLINE       ONLINE       rac02
ora.scan1.vip                            SCAN VIP       ONLINE       ONLINE       rac01
ora.scan2.vip                            SCAN VIP       ONLINE       ONLINE       rac02
ora.scan3.vip                            SCAN VIP       ONLINE       ONLINE       rac02
xag.lax_ggate.goldengate                 goldengate     ONLINE       INTERMEDIATE rac01           ER(s) not running : ELAX,ELAX,RDEN_LAX

[grid@rac01:+ASM1:/home/grid]
$ crsstat


Resource Name                            Resource Type  Target       State        Node            State Details
---------------------------------------- -------------- ------------ ------------ --------------- ---------------
lax-ggate1-vip                           appvip_net1    ONLINE       ONLINE       rac01
ora.DATA2.dg                             diskgroup      ONLINE       ONLINE       rac01
ora.DATA2.dg                             diskgroup      ONLINE       ONLINE       rac02
ora.DG_ACFS.dg                           diskgroup      ONLINE       ONLINE       rac01
ora.DG_ACFS.dg                           diskgroup      ONLINE       ONLINE       rac02
ora.LISTENER.lsnr                        Listener       ONLINE       ONLINE       rac01
ora.LISTENER.lsnr                        Listener       ONLINE       ONLINE       rac02
ora.LISTENER_SCAN1.lsnr                  SCAN Listener  ONLINE       ONLINE       rac01
ora.LISTENER_SCAN2.lsnr                  SCAN Listener  ONLINE       ONLINE       rac02
ora.LISTENER_SCAN3.lsnr                  SCAN Listener  ONLINE       ONLINE       rac02
ora.asm                                  ASM            ONLINE       ONLINE       rac01           Started
ora.asm                                  ASM            ONLINE       ONLINE       rac02           Started
ora.cvu                                  cvu            ONLINE       ONLINE       rac02
ora.dg_acfs.vg_acfs.acfs                 acfs           ONLINE       ONLINE       rac01           mounted on /acfsmount
ora.dg_acfs.vg_acfs.acfs                 acfs           ONLINE       ONLINE       rac02           mounted on /acfsmount
ora.emu.db                               database       ONLINE       ONLINE       rac01           Open
ora.emu.db                               database       ONLINE       ONLINE       rac02           Open
ora.gsd                                  Gbl Svc Daemon OFFLINE      OFFLINE
ora.gsd                                  Gbl Svc Daemon OFFLINE      OFFLINE
ora.net1.network                         Network (VIP)  ONLINE       ONLINE       rac01
ora.net1.network                         Network (VIP)  ONLINE       ONLINE       rac02
ora.oc4j                                 OC4J           ONLINE       ONLINE       rac02
ora.ons                                  Ora Notif Svc  ONLINE       ONLINE       rac01
ora.ons                                  Ora Notif Svc  ONLINE       ONLINE       rac02
ora.rac01.vip                            Cluster VIP    ONLINE       ONLINE       rac01
ora.rac02.vip                            Cluster VIP    ONLINE       ONLINE       rac02
ora.registry.acfs                        registry       ONLINE       ONLINE       rac01
ora.registry.acfs                        registry       ONLINE       ONLINE       rac02
ora.scan1.vip                            SCAN VIP       ONLINE       ONLINE       rac01
ora.scan2.vip                            SCAN VIP       ONLINE       ONLINE       rac02
ora.scan3.vip                            SCAN VIP       ONLINE       ONLINE       rac02
xag.lax_ggate.goldengate                 goldengate     ONLINE       ONLINE       rac01

STOP GOLDENDATE

[grid@rac01:+ASM1:/home/grid]
$ $XAG_HOME/bin/agctl stop goldengate lax_ggate
[grid@rac01:+ASM1:/home/grid]

DISCOVERY
It’s easy to know where everything when you are the architect, but what happens when you are not?
How to find the Goldengate VIP and is it in DNS?

$ crsctl stat res lax-ggate1-vip -p|grep USR_ORA_VIP
GEN_USR_ORA_VIP=
USR_ORA_VIP=192.168.56.41

$ nslookup 192.168.56.41
Server:         127.0.0.1
Address:        127.0.0.1#53

41.56.168.192.in-addr.arpa      name = lax-ggate1-vip.localdomain.

$ crsctl stat res -w "TYPE = ora.acfs.type" -p|grep '^MOUNT'
MOUNTPOINT_PATH=/acfsmount
MOUNTPOINT_PATH=/acfsmount

REFERENCE:

Oracle GoldenGate Best Practices: 
Configuring Oracle GoldenGate with Oracle Grid Infrastructure Bundled Agents (XAG) - (Document 1527310.1) 

Oracle Clusterware: 
http://oracle.com/goto/clusterware 

Oracle Grid Infrastructure (Bundled) Agents: 
http://www.oracle.com/technetwork/database/database-technologies/clusterware/downloads/index.html

Oracle Grid Infrastructure Agents Version 5.1
11.2.0.3 and later 
12.1.0.1 and later 
http://www.oracle.com/technetwork/database/database-technologies/clusterware/downloads/ogiba-2189738.pdf

SRDC - Data to Collect on GoldenGate Issues Related to XAG (Doc ID 1913048.1)	

Patch 16762459: ORACLE GOLDENGATE V11.2.1.0.7 FOR ORACLE 11G

[gguser@angel:/u01/app/ggs112]
$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 11.2.1.0.7 16934304 OGGCORE_11.2.1.0.7_PLATFORMS_130709.1600.1_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Jul 18 2013 07:04:28

Copyright (C) 1995, 2013, Oracle and/or its affiliates. All rights reserved.

GGSCI (angel.local) 1> dblogin userid ggadmin, password ggadmin
Successfully logged into database.

GGSCI (angel.local) 2> versions
Operating System:
Linux
Version #1 SMP Fri Feb 22 18:16:18 PST 2013, Release 2.6.39-400.17.1.el6uek.x86_64
Node: angel.local
Machine: x86_64

Database:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
PL/SQL Release 11.2.0.4.0 - Production
CORE    11.2.0.4.0      Production
TNS for Linux: Version 11.2.0.4.0 - Production
NLSRTL Version 11.2.0.4.0 - Production

December 5, 2014

GUI Be Gone

Filed under: oracle — mdinh @ 12:08 am

I am losing the luxury of using GUI since clients typically may not have X-Windows or VNC Server installed

Adapt or die and re-learning command line again.

Better to verify before installation than to clean up a problematic install.

Use runcluvfy to verify DB install:
runcluvfy.sh stage -pre dbinst -n rac01,rac02 -r 11gR2 -d /u01/app/oracle/product/11.2.0.4/db_1 -osdba dba -fixup -fixupdir /tmp -verbose

Use runInstaller -executePrereqs to verify responseFile for silent install and detect issues:
runInstaller -silent -executePrereqs -waitforcompletion -force -responseFile /media/sf_Linux/11.2.0.4/database/rac_db_swonly.rsp

Use grep to find results from installActions log:
grep -e ‘[[:upper:]]:’ installActions2014-12-04_02-49-27PM.log|cut -d “:” -f1|sort -u

DEMO:

[oracle@rac01:/media/sf_Linux/11.2.0.4/grid]
$ ./runcluvfy.sh stage -pre dbinst -n rac01,rac02 -r 11gR2 -d /u01/app/oracle/product/11.2.0.4/db_1 -osdba dba -fixup -fixupdir /tmp -verbose

Performing pre-checks for database installation

Checking node reachability...

Check: Node reachability from node "rac01"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rac02                                 yes
  rac01                                 yes
Result: Node reachability check passed from node "rac01"


Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Status
  ------------------------------------  ------------------------
  rac02                                 passed
  rac01                                 passed
Result: User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rac02                                 passed
  rac01                                 passed

Verification of the hosts config file successful


Interface information for node "rac02"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   10.0.2.15       10.0.2.0        0.0.0.0         10.0.2.2        08:00:27:87:91:11 1500
 eth1   192.168.56.12   192.168.56.0    0.0.0.0         10.0.2.2        08:00:27:4A:7B:27 1500
 eth1   192.168.56.33   192.168.56.0    0.0.0.0         10.0.2.2        08:00:27:4A:7B:27 1500
 eth1   192.168.56.32   192.168.56.0    0.0.0.0         10.0.2.2        08:00:27:4A:7B:27 1500
 eth1   192.168.56.22   192.168.56.0    0.0.0.0         10.0.2.2        08:00:27:4A:7B:27 1500
 eth2   10.10.10.12     10.0.0.0        0.0.0.0         10.0.2.2        08:00:27:E8:D6:21 1500
 eth2   169.254.82.236  169.254.0.0     0.0.0.0         10.0.2.2        08:00:27:E8:D6:21 1500


Interface information for node "rac01"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   10.0.2.15       10.0.2.0        0.0.0.0         10.0.2.2        08:00:27:02:B1:57 1500
 eth1   192.168.56.11   192.168.56.0    0.0.0.0         10.0.2.2        08:00:27:BD:66:A4 1500
 eth1   192.168.56.21   192.168.56.0    0.0.0.0         10.0.2.2        08:00:27:BD:66:A4 1500
 eth1   192.168.56.31   192.168.56.0    0.0.0.0         10.0.2.2        08:00:27:BD:66:A4 1500
 eth2   10.10.10.11     10.0.0.0        0.0.0.0         10.0.2.2        08:00:27:60:79:0F 1500
 eth2   169.254.34.109  169.254.0.0     0.0.0.0         10.0.2.2        08:00:27:60:79:0F 1500


Check: Node connectivity for interface "eth1"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac02[192.168.56.12]            rac02[192.168.56.33]            yes
  rac02[192.168.56.12]            rac02[192.168.56.32]            yes
  rac02[192.168.56.12]            rac02[192.168.56.22]            yes
  rac02[192.168.56.12]            rac01[192.168.56.11]            yes
  rac02[192.168.56.12]            rac01[192.168.56.21]            yes
  rac02[192.168.56.12]            rac01[192.168.56.31]            yes
  rac02[192.168.56.33]            rac02[192.168.56.32]            yes
  rac02[192.168.56.33]            rac02[192.168.56.22]            yes
  rac02[192.168.56.33]            rac01[192.168.56.11]            yes
  rac02[192.168.56.33]            rac01[192.168.56.21]            yes
  rac02[192.168.56.33]            rac01[192.168.56.31]            yes
  rac02[192.168.56.32]            rac02[192.168.56.22]            yes
  rac02[192.168.56.32]            rac01[192.168.56.11]            yes
  rac02[192.168.56.32]            rac01[192.168.56.21]            yes
  rac02[192.168.56.32]            rac01[192.168.56.31]            yes
  rac02[192.168.56.22]            rac01[192.168.56.11]            yes
  rac02[192.168.56.22]            rac01[192.168.56.21]            yes
  rac02[192.168.56.22]            rac01[192.168.56.31]            yes
  rac01[192.168.56.11]            rac01[192.168.56.21]            yes
  rac01[192.168.56.11]            rac01[192.168.56.31]            yes
  rac01[192.168.56.21]            rac01[192.168.56.31]            yes
Result: Node connectivity passed for interface "eth1"


Check: TCP connectivity of subnet "192.168.56.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac01:192.168.56.11             rac02:192.168.56.12             passed
  rac01:192.168.56.11             rac02:192.168.56.33             passed
  rac01:192.168.56.11             rac02:192.168.56.32             passed
  rac01:192.168.56.11             rac02:192.168.56.22             passed
  rac01:192.168.56.11             rac01:192.168.56.21             passed
  rac01:192.168.56.11             rac01:192.168.56.31             passed
Result: TCP connectivity check passed for subnet "192.168.56.0"


Check: Node connectivity for interface "eth2"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac02[10.10.10.12]              rac01[10.10.10.11]              yes
Result: Node connectivity passed for interface "eth2"


Check: TCP connectivity of subnet "10.0.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac01:10.10.10.11               rac02:10.10.10.12               passed
Result: TCP connectivity check passed for subnet "10.0.0.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.56.0".
Subnet mask consistency check passed for subnet "10.0.0.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.56.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.56.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "10.0.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "10.0.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         3.8674GB (4055296.0KB)    1GB (1048576.0KB)         passed
  rac01         3.8674GB (4055296.0KB)    1GB (1048576.0KB)         passed
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         2.6666GB (2796104.0KB)    50MB (51200.0KB)          passed
  rac01         2.7855GB (2920820.0KB)    50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         8GB (8388604.0KB)         3.8674GB (4055296.0KB)    passed
  rac01         8GB (8388604.0KB)         3.8674GB (4055296.0KB)    passed
Result: Swap space check passed

Check: Free disk space for "rac02:/u01/app/oracle/product/11.2.0.4/db_1,rac02:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /u01/app/oracle/product/11.2.0.4/db_1  rac02         /             45.459GB      6.7GB         passed
  /tmp              rac02         /             45.459GB      6.7GB         passed
Result: Free disk space check passed for "rac02:/u01/app/oracle/product/11.2.0.4/db_1,rac02:/tmp"

Check: Free disk space for "rac01:/u01/app/oracle/product/11.2.0.4/db_1,rac01:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /u01/app/oracle/product/11.2.0.4/db_1  rac01         /             45.4215GB     6.7GB         passed
  /tmp              rac01         /             45.4215GB     6.7GB         passed
Result: Free disk space check passed for "rac01:/u01/app/oracle/product/11.2.0.4/db_1,rac01:/tmp"

Check: User existence for "oracle"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rac02         passed                    exists(54321)
  rac01         passed                    exists(54321)

Checking for multiple users with UID value 54321
Result: Check for multiple users with UID value 54321 passed
Result: User existence check passed for "oracle"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rac02         passed                    exists
  rac01         passed                    exists
Result: Group existence check passed for "oinstall"

Check: Group existence for "dba"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rac02         passed                    exists
  rac01         passed                    exists
Result: Group existence check passed for "dba"

Check: Membership of user "oracle" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             yes           yes           yes           yes           passed
  rac01             yes           yes           yes           yes           passed
Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed

Check: Membership of user "oracle" in group "dba"
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rac02             yes           yes           yes           passed
  rac01             yes           yes           yes           passed
Result: Membership check for user "oracle" in group "dba" passed

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         5                         3,5                       passed
  rac01         5                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac02             hard          65536         65536         passed
  rac01             hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac02             soft          1024          1024          passed
  rac01             soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac02             hard          16384         16384         passed
  rac01             hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac02             soft          16384         2047          passed
  rac01             soft          16384         2047          passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         x86_64                    x86_64                    passed
  rac01         x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         2.6.39-400.17.1.el6uek.x86_64  2.6.32                    passed
  rac01         2.6.39-400.17.1.el6uek.x86_64  2.6.32                    passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             250           250           250           passed
  rac01             250           250           250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             32000         32000         32000         passed
  rac01             32000         32000         32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             100           100           100           passed
  rac01             100           100           100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             128           128           128           passed
  rac01             128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             4398046511104  4398046511104  2076311552    passed
  rac01             4398046511104  4398046511104  2076311552    passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             4096          4096          4096          passed
  rac01             4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             4294967296    4294967296    2097152       passed
  rac01             4294967296    4294967296    2097152       passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             6815744       6815744       6815744       passed
  rac01             6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
  rac01             between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             262144        262144        262144        passed
  rac01             262144        262144        262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             4194304       4194304       4194304       passed
  rac01             4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             262144        262144        262144        passed
  rac01             262144        262144        262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             1048576       1048576       1048576       passed
  rac01             1048576       1048576       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             1048576       1048576       1048576       passed
  rac01             1048576       1048576       1048576       passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
  rac01         binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "compat-libcap1"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         compat-libcap1-1.10-1     compat-libcap1-1.10       passed
  rac01         compat-libcap1-1.10-1     compat-libcap1-1.10       passed
Result: Package existence check passed for "compat-libcap1"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
  rac01         compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         libgcc(x86_64)-4.4.7-11.el6  libgcc(x86_64)-4.4.4      passed
  rac01         libgcc(x86_64)-4.4.7-11.el6  libgcc(x86_64)-4.4.4      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         libstdc++(x86_64)-4.4.7-11.el6  libstdc++(x86_64)-4.4.4   passed
  rac01         libstdc++(x86_64)-4.4.7-11.el6  libstdc++(x86_64)-4.4.4   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         libstdc++-devel(x86_64)-4.4.7-11.el6  libstdc++-devel(x86_64)-4.4.4  passed
  rac01         libstdc++-devel(x86_64)-4.4.7-11.el6  libstdc++-devel(x86_64)-4.4.4  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         sysstat-9.0.4-20.el6      sysstat-9.0.4             passed
  rac01         sysstat-9.0.4-20.el6      sysstat-9.0.4             passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "gcc"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         gcc-4.4.7-11.el6          gcc-4.4.4                 passed
  rac01         gcc-4.4.7-11.el6          gcc-4.4.4                 passed
Result: Package existence check passed for "gcc"

Check: Package existence for "gcc-c++"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         gcc-c++-4.4.7-11.el6      gcc-c++-4.4.4             passed
  rac01         gcc-c++-4.4.7-11.el6      gcc-c++-4.4.4             passed
Result: Package existence check passed for "gcc-c++"

Check: Package existence for "ksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         ksh-20120801-21.el6.1     ksh-20100621              passed
  rac01         ksh-20120801-21.el6.1     ksh-20100621              passed
Result: Package existence check passed for "ksh"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         make-3.81-20.el6          make-3.81                 passed
  rac01         make-3.81-20.el6          make-3.81                 passed
Result: Package existence check passed for "make"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         glibc(x86_64)-2.12-1.149.el6  glibc(x86_64)-2.12        passed
  rac01         glibc(x86_64)-2.12-1.149.el6  glibc(x86_64)-2.12        passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         glibc-devel(x86_64)-2.12-1.149.el6  glibc-devel(x86_64)-2.12  passed
  rac01         glibc-devel(x86_64)-2.12-1.149.el6  glibc-devel(x86_64)-2.12  passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
  rac01         libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
  rac01         libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  rac02                                 passed
  rac01                                 passed

Check for consistency of root user's primary group passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac02         0022                      0022                      passed
  rac01         0022                      0022                      passed
Result: Default user file creation mask check passed

Checking CRS integrity...

Clusterware version consistency passed
The Oracle Clusterware is healthy on node "rac02"
The Oracle Clusterware is healthy on node "rac01"

CRS integrity check passed

Checking Cluster manager integrity...


Checking CSS daemon...

  Node Name                             Status
  ------------------------------------  ------------------------
  rac02                                 running
  rac01                                 running

Oracle Cluster Synchronization Services appear to be online.

Cluster manager integrity check passed


Checking node application existence...

Checking existence of VIP node application (required)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac02         yes                       yes                       passed
  rac01         yes                       yes                       passed
VIP node application check passed

Checking existence of NETWORK node application (required)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac02         yes                       yes                       passed
  rac01         yes                       yes                       passed
NETWORK node application check passed

Checking existence of GSD node application (optional)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac02         no                        no                        exists
  rac01         no                        no                        exists
GSD node application is offline on nodes "rac02,rac01"

Checking existence of ONS node application (optional)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac02         no                        yes                       passed
  rac01         no                        yes                       passed
ONS node application check passed


Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
  Node Name                             Status
  ------------------------------------  ------------------------
  rac02                                 passed
  rac01                                 passed
Result: CTSS resource check passed


Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed

Check CTSS state started...
Check: CTSS state
  Node Name                             State
  ------------------------------------  ------------------------
  rac02                                 Active
  rac01                                 Active
CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Reference Time Offset Limit: 1000.0 msecs
Check: Reference Time Offset
  Node Name     Time Offset               Status
  ------------  ------------------------  ------------------------
  rac02         0.0                       passed
  rac01         0.0                       passed

Time offset is within the specified limits on the following set of nodes:
"[rac02, rac01]"
Result: Check of clock time offsets passed


Oracle Cluster Time Synchronization Services check passed
Checking consistency of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that search entry is "localdomain" as found on node "rac02"
All nodes of the cluster have same value for 'search'
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  rac02                                 passed
  rac01                                 passed
The DNS response time for an unreachable node is within acceptable limit on all nodes

File "/etc/resolv.conf" is consistent across nodes

Check: Time zone consistency
Result: Time zone consistency check passed

Checking Single Client Access Name (SCAN)...
  SCAN Name         Node          Running?      ListenerName  Port          Running?
  ----------------  ------------  ------------  ------------  ------------  ------------
  dinh-scan         rac01         true          LISTENER_SCAN1  1521          true
  dinh-scan         rac02         true          LISTENER_SCAN2  1521          true
  dinh-scan         rac02         true          LISTENER_SCAN3  1521          true

Checking TCP connectivity to SCAN Listeners...
  Node          ListenerName              TCP connectivity?
  ------------  ------------------------  ------------------------
  rac01         LISTENER_SCAN1            yes
  rac01         LISTENER_SCAN2            yes
  rac01         LISTENER_SCAN3            yes
TCP connectivity to SCAN Listeners exists on all cluster nodes

Checking name resolution setup for "dinh-scan"...

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Checking if "hosts" entry in file "/etc/nsswitch.conf" is consistent across nodes...
Checking file "/etc/nsswitch.conf" to make sure that only one "hosts" entry is defined
More than one "hosts" entry does not exist in any "/etc/nsswitch.conf" file
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed

  SCAN Name     IP Address                Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  dinh-scan     192.168.56.33             passed
  dinh-scan     192.168.56.31             passed
  dinh-scan     192.168.56.32             passed

Verification of SCAN VIP and Listener setup passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.

Checking Database and Clusterware version compatibility


Checking ASM and CRS version compatibility
ASM and CRS versions are compatible
Database version "11.2" is compatible with the Clusterware version "11.2.0.4.0".
Database Clusterware version compatibility passed

Pre-check for database installation was successful.
[oracle@rac01:/media/sf_Linux/11.2.0.4/grid]
$

[oracle@rac01:/media/sf_Linux/11.2.0.4/database]
$ ./runInstaller -silent -executePrereqs -showProgress -waitforcompletion -force -responseFile /media/sf_Linux/11.2.0.4/database/rac_db_swonly.rsp

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 44381 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 8191 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-12-04_02-49-27PM. Please wait ...
[oracle@rac01:/media/sf_Linux/11.2.0.4/database]
$ cd /u01/app/oraInventory/logs/
[oracle@rac01:/u01/app/oraInventory/logs]
$ ls -lrt
total 1068
-rw-r-----. 1 grid   oinstall     47 Nov 30 19:40 time2014-11-30_07-40-41PM.log
-rw-r-----. 1 grid   oinstall      0 Nov 30 19:42 oraInstall2014-11-30_07-40-41PM.err
-rw-r-----. 1 grid   oinstall      0 Nov 30 19:45 oraInstall2014-11-30_07-40-41PM.err.rac02
-rw-r-----. 1 grid   oinstall    121 Nov 30 19:46 oraInstall2014-11-30_07-40-41PM.out.rac02
-rw-r-----. 1 grid   oinstall   7650 Nov 30 19:46 AttachHome2014-11-30_07-40-41PM.log.rac02
-rw-r-----. 1 grid   oinstall    348 Nov 30 19:46 silentInstall2014-11-30_07-40-41PM.log
-rw-r-----. 1 grid   oinstall   1968 Nov 30 19:46 oraInstall2014-11-30_07-40-41PM.out
-rw-r-----. 1 grid   oinstall 975962 Nov 30 19:46 installActions2014-11-30_07-40-41PM.log
-rw-r-----. 1 grid   oinstall      0 Nov 30 20:12 oraInstall2014-11-30_08-12-19PM.err
-rw-r-----. 1 grid   oinstall      0 Nov 30 20:12 oraInstall2014-11-30_08-12-19PM.err.rac02
-rw-r-----. 1 grid   oinstall   7357 Nov 30 20:13 UpdateNodeList2014-11-30_08-12-19PM.log.rac02
-rw-r-----. 1 grid   oinstall     33 Nov 30 20:13 oraInstall2014-11-30_08-12-19PM.out.rac02
-rw-r-----. 1 grid   oinstall  11305 Nov 30 20:13 UpdateNodeList2014-11-30_08-12-19PM.log
-rw-r-----. 1 grid   oinstall     33 Nov 30 20:13 oraInstall2014-11-30_08-12-19PM.out
-rw-r--r--. 1 oracle oinstall     47 Dec  4 14:49 time2014-12-04_02-49-27PM.log
-rw-rw----. 1 oracle oinstall  56317 Dec  4 14:49 installActions2014-12-04_02-49-27PM.log
-rw-r--r--. 1 oracle oinstall      0 Dec  4 14:49 oraInstall2014-12-04_02-49-27PM.out
-rw-r--r--. 1 oracle oinstall      0 Dec  4 14:49 oraInstall2014-12-04_02-49-27PM.err

[oracle@rac01:/u01/app/oraInventory/logs]
$ tail -20 installActions2014-12-04_02-49-27PM.log

INFO: Actual Value:libaio-devel(x86_64)-0.3.107-10.el6
INFO: -----------------------------------------------
INFO: *********************************************
INFO: Users With Same UID: This test checks that multiple users do not exist with user id as "0".
INFO: Severity:CRITICAL
INFO: OverallStatus:SUCCESSFUL
INFO: -----------------------------------------------
INFO: Verification Result for Node:rac01
WARNING: Result values are not available for this verification task
INFO: *********************************************
INFO: Root user consistency: This test checks the consistency of the primary group of the root user across the cluster nodes
INFO: Severity:IGNORABLE
INFO: OverallStatus:SUCCESSFUL
INFO: -----------------------------------------------
INFO: Verification Result for Node:rac01
WARNING: Result values are not available for this verification task
INFO: All forked task are completed at state prepInstall
INFO: Exit Status is 0
INFO: Shutdown Oracle Database 11g Release 2 Installer
INFO: Unloading Setup Driver

[oracle@rac01:/u01/app/oraInventory/logs]
$ grep -e ‘[[:upper:]]:’ installActions2014-12-04_02-49-27PM.log|cut -d “:” -f1|sort -u

INFO
/tmp/OraInstall2014-12-04_02-49-27PM
WARNING
[oracle@rac01:/u01/app/oraInventory/logs]
$

December 1, 2014

Using runInstaller to check Prereqs with responseFile

Filed under: 11g,RAC — mdinh @ 3:38 am

The SILENT method

[grid@rac01:/media/sf_Linux/11.2.0.4/grid]
$ ./runInstaller -silent -executePrereqs -showProgress -waitforcompletion -force -responseFile /media/sf_Linux/11.2.0.4/grid/grid_crs_config.rsp
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 48807 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 8191 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-11-30_06-40-05PM. Please wait ...[grid@rac01:/media/sf_Linux/11.2.0.4/grid]
$ cd /u01/app/oraInventory/logs/
[grid@rac01:/u01/app/oraInventory/logs]
$ ls -lrt
total 84
-rw-r--r--. 1 grid oinstall    47 Nov 30 18:40 time2014-11-30_06-40-05PM.log
-rw-rw----. 1 grid oinstall 80257 Nov 30 18:41 installActions2014-11-30_06-40-05PM.log
-rw-r--r--. 1 grid oinstall     0 Nov 30 18:41 oraInstall2014-11-30_06-40-05PM.out
-rw-r--r--. 1 grid oinstall     0 Nov 30 18:41 oraInstall2014-11-30_06-40-05PM.err
[grid@rac01:/u01/app/oraInventory/logs]
$ tail installActions2014-11-30_06-40-05PM.log
INFO: Actual Value:0|2
INFO: -----------------------------------------------
INFO: -----------------------------------------------
INFO: Verification Result for Node:rac01
INFO: Expected Value:0|2
INFO: Actual Value:0|2
INFO: -----------------------------------------------
INFO: All forked task are completed at state init
INFO: Exit Status is 0
INFO: Shutdown Oracle Grid Infrastructure

The NOT SO SILENT method
executePrereqs

Avoid false errors with runcluvfy.sh for VirtualBox

Filed under: 11g,RAC — mdinh @ 2:32 am

My typical VirtualBox network configuration is: eth0 (NAT), eth1 (Host Only), eth2(Internal)

runcluvfy.sh stage -pre crsinst -n rac01,rac02 -r 11gR2 -fixup -fixupdir /tmp – FAILED

Node connectivity passed for subnet "10.0.2.0" with node(s) rac02,rac01

ERROR:
PRVF-7617 : Node connectivity between "rac01 : 10.0.2.15" and "rac02 : 10.0.2.15" failed
TCP connectivity check failed for subnet "10.0.2.0"

Node connectivity passed for subnet "192.168.56.0" with node(s) rac02,rac01
TCP connectivity check passed for subnet "192.168.56.0"

Node connectivity passed for subnet "10.0.0.0" with node(s) rac02,rac01
TCP connectivity check passed for subnet "10.0.0.0"

Pre-check for cluster services setup was unsuccessful.
Checks did not pass for the following node(s):
        rac01
[grid@rac01:/media/sf_Linux/11.2.0.4/grid]
$

I understand the error is ignorable but why create an ignorable error in the first place?
Next, I would like to look at the bottom line to know if the check passed for failed and if the failure is false positive, then it would require wasted time for investigation.

A better alternative is to exclude eth0 from the check using -networks eth1,eth2 to have successful setup.

Pre-check for cluster services setup was successful.

runcluvfy.sh stage -pre crsinst -n rac01,rac02 -r 11gR2 -fixup -fixupdir /tmp -networks eth1,eth2 -verbose

[grid@rac01:/media/sf_Linux/11.2.0.4/grid]
$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 08:00:27:02:B1:57
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe02:b157/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:578823 errors:0 dropped:0 overruns:0 frame:0
          TX packets:292879 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:477712790 (455.5 MiB)  TX bytes:15902513 (15.1 MiB)
[grid@rac01:/media/sf_Linux/11.2.0.4/grid]
$ ./runcluvfy.sh stage -pre crsinst -n rac01,rac02 -r 11gR2 -fixup -fixupdir /tmp -networks eth1,eth2 -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "rac01"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rac02                                 yes
  rac01                                 yes
Result: Node reachability check passed from node "rac01"


Checking user equivalence...

Check: User equivalence for user "grid"
  Node Name                             Status
  ------------------------------------  ------------------------
  rac02                                 passed
  rac01                                 passed
Result: User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rac02                                 passed
  rac01                                 passed

Verification of the hosts config file successful


Interface information for node "rac02"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   10.0.2.15       10.0.2.0        0.0.0.0         10.0.2.2        08:00:27:87:91:11 1500
 eth1   192.168.56.12   192.168.56.0    0.0.0.0         10.0.2.2        08:00:27:4A:7B:27 1500
 eth2   10.10.10.12     10.0.0.0        0.0.0.0         10.0.2.2        08:00:27:E8:D6:21 1500


Interface information for node "rac01"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   10.0.2.15       10.0.2.0        0.0.0.0         10.0.2.2        08:00:27:02:B1:57 1500
 eth1   192.168.56.11   192.168.56.0    0.0.0.0         10.0.2.2        08:00:27:BD:66:A4 1500
 eth2   10.10.10.11     10.0.0.0        0.0.0.0         10.0.2.2        08:00:27:60:79:0F 1500


Check: Node connectivity for interface "eth1"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac02[192.168.56.12]            rac01[192.168.56.11]            yes
Result: Node connectivity passed for interface "eth1"


Check: TCP connectivity of subnet "192.168.56.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac01:192.168.56.11             rac02:192.168.56.12             passed
Result: TCP connectivity check passed for subnet "192.168.56.0"


Check: Node connectivity for interface "eth2"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac02[10.10.10.12]              rac01[10.10.10.11]              yes
Result: Node connectivity passed for interface "eth2"


Check: TCP connectivity of subnet "10.0.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rac01:10.10.10.11               rac02:10.10.10.12               passed
Result: TCP connectivity check passed for subnet "10.0.0.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.56.0".
Subnet mask consistency check passed for subnet "10.0.0.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.56.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.56.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "10.0.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "10.0.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking ASMLib configuration.
  Node Name                             Status
  ------------------------------------  ------------------------
  rac02                                 passed
  rac01                                 passed
Result: Check for ASMLib configuration passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         3.8674GB (4055296.0KB)    1.5GB (1572864.0KB)       passed
  rac01         3.8674GB (4055296.0KB)    1.5GB (1572864.0KB)       passed
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         3.5583GB (3731180.0KB)    50MB (51200.0KB)          passed
  rac01         3.269GB (3427772.0KB)     50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         8GB (8388604.0KB)         3.8674GB (4055296.0KB)    passed
  rac01         8GB (8388604.0KB)         3.8674GB (4055296.0KB)    passed
Result: Swap space check passed

Check: Free disk space for "rac02:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rac02         /             52.2988GB     1GB           passed
Result: Free disk space check passed for "rac02:/tmp"

Check: Free disk space for "rac01:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rac01         /             49.9507GB     1GB           passed
Result: Free disk space check passed for "rac01:/tmp"

Check: User existence for "grid"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rac02         passed                    exists(54322)
  rac01         passed                    exists(54322)

Checking for multiple users with UID value 54322
Result: Check for multiple users with UID value 54322 passed
Result: User existence check passed for "grid"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rac02         passed                    exists
  rac01         passed                    exists
Result: Group existence check passed for "oinstall"

Check: Group existence for "dba"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rac02         passed                    exists
  rac01         passed                    exists
Result: Group existence check passed for "dba"

Check: Membership of user "grid" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             yes           yes           yes           yes           passed
  rac01             yes           yes           yes           yes           passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed

Check: Membership of user "grid" in group "dba"
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rac02             yes           yes           yes           passed
  rac01             yes           yes           yes           passed
Result: Membership check for user "grid" in group "dba" passed

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         5                         3,5                       passed
  rac01         5                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac02             hard          65536         65536         passed
  rac01             hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac02             soft          1024          1024          passed
  rac01             soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac02             hard          16384         16384         passed
  rac01             hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rac02             soft          16384         2047          passed
  rac01             soft          16384         2047          passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         x86_64                    x86_64                    passed
  rac01         x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         2.6.39-400.17.1.el6uek.x86_64  2.6.32                    passed
  rac01         2.6.39-400.17.1.el6uek.x86_64  2.6.32                    passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             250           250           250           passed
  rac01             250           250           250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             32000         32000         32000         passed
  rac01             32000         32000         32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             100           100           100           passed
  rac01             100           100           100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             128           128           128           passed
  rac01             128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             4398046511104  4398046511104  2076311552    passed
  rac01             4398046511104  4398046511104  2076311552    passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             4096          4096          4096          passed
  rac01             4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             4294967296    4294967296    2097152       passed
  rac01             4294967296    4294967296    2097152       passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             6815744       6815744       6815744       passed
  rac01             6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
  rac01             between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             262144        262144        262144        passed
  rac01             262144        262144        262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             4194304       4194304       4194304       passed
  rac01             4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             262144        262144        262144        passed
  rac01             262144        262144        262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             1048576       1048576       1048576       passed
  rac01             1048576       1048576       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             1048576       1048576       1048576       passed
  rac01             1048576       1048576       1048576       passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
  rac01         binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "compat-libcap1"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         compat-libcap1-1.10-1     compat-libcap1-1.10       passed
  rac01         compat-libcap1-1.10-1     compat-libcap1-1.10       passed
Result: Package existence check passed for "compat-libcap1"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
  rac01         compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         libgcc(x86_64)-4.4.7-11.el6  libgcc(x86_64)-4.4.4      passed
  rac01         libgcc(x86_64)-4.4.7-11.el6  libgcc(x86_64)-4.4.4      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         libstdc++(x86_64)-4.4.7-11.el6  libstdc++(x86_64)-4.4.4   passed
  rac01         libstdc++(x86_64)-4.4.7-11.el6  libstdc++(x86_64)-4.4.4   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         libstdc++-devel(x86_64)-4.4.7-11.el6  libstdc++-devel(x86_64)-4.4.4  passed
  rac01         libstdc++-devel(x86_64)-4.4.7-11.el6  libstdc++-devel(x86_64)-4.4.4  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         sysstat-9.0.4-20.el6      sysstat-9.0.4             passed
  rac01         sysstat-9.0.4-20.el6      sysstat-9.0.4             passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "gcc"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         gcc-4.4.7-11.el6          gcc-4.4.4                 passed
  rac01         gcc-4.4.7-11.el6          gcc-4.4.4                 passed
Result: Package existence check passed for "gcc"

Check: Package existence for "gcc-c++"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         gcc-c++-4.4.7-11.el6      gcc-c++-4.4.4             passed
  rac01         gcc-c++-4.4.7-11.el6      gcc-c++-4.4.4             passed
Result: Package existence check passed for "gcc-c++"

Check: Package existence for "ksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         ksh-20120801-21.el6.1     ksh-20100621              passed
  rac01         ksh-20120801-21.el6.1     ksh-20100621              passed
Result: Package existence check passed for "ksh"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         make-3.81-20.el6          make-3.81                 passed
  rac01         make-3.81-20.el6          make-3.81                 passed
Result: Package existence check passed for "make"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         glibc(x86_64)-2.12-1.149.el6  glibc(x86_64)-2.12        passed
  rac01         glibc(x86_64)-2.12-1.149.el6  glibc(x86_64)-2.12        passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         glibc-devel(x86_64)-2.12-1.149.el6  glibc-devel(x86_64)-2.12  passed
  rac01         glibc-devel(x86_64)-2.12-1.149.el6  glibc-devel(x86_64)-2.12  passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
  rac01         libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rac02         libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
  rac01         libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  rac02                                 passed
  rac01                                 passed

Check for consistency of root user's primary group passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "grid" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rac02         passed                    does not exist
  rac01         passed                    does not exist
Result: User "grid" is not part of "root" group. Check passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rac02         0022                      0022                      passed
  rac01         0022                      0022                      passed
Result: Default user file creation mask check passed
Checking consistency of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that search entry is "localdomain" as found on node "rac02"
All nodes of the cluster have same value for 'search'
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  rac02                                 passed
  rac01                                 passed
The DNS response time for an unreachable node is within acceptable limit on all nodes

File "/etc/resolv.conf" is consistent across nodes

Check: Time zone consistency
Result: Time zone consistency check passed

Starting check for Reverse path filter setting ...
Reverse path filter setting is correct for all private interconnect network interfaces on node "rac02.localdomain".
Reverse path filter setting is correct for all private interconnect network interfaces on node "rac01.localdomain".

Check for Reverse path filter setting passed

Pre-check for cluster services setup was successful.
[grid@rac01:/media/sf_Linux/11.2.0.4/grid]
$

October 31, 2014

Better Communication

Filed under: Uncategorized — mdinh @ 1:09 am

Sometimes I get excited over simple things. Today, I learned something new, something about human behaviors.

Basically, there are 4 quadrants for human behaviors.

If one does not understand the other person’s behavior, then communication can be very difficult.

To find more about human behaviors, read The DISC Model of Human Behavior – A Quick Overview.

Which quadrant are you in, versus your manager?

Next Page »

The Rubric Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 334 other followers