Thinking Out Loud

May 27, 2019

Choiceology with Katy Milkman

Filed under: Uncategorized — mdinh @ 8:15 pm

Good listening I thought about sharing with you.

RSS Feed:
Choiceology with Katy Milkman – Exposing the psychological traps that lead to expensive mistakes

You can listen and subscribe to Choiceology with Katy Milkman for free in any podcast player—such as Apple Podcasts, Google Podcasts or Spotify.
How Do I Listen to Choiceology?

May 19, 2019

Shocking opatchauto resume works after auto-logout

Filed under: 12c,opatchauto,RAC — mdinh @ 5:36 pm

WARNING: Please don’t try this at home or in production environment.

With that being said, patching was for DR production.

Oracle Interim Patch Installer version 12.2.0.1.16

Patching 2 nodes RAC cluster and node1 completed successfully.

Rationale for using -norestart because there was an issue at one time where datapatch was applied on the node1.

Don’t implement Active Data Guard and have database Start options: mount

# crsctl stat res -t -w '((TARGET != ONLINE) or (STATE != ONLINE)'
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.dbproddr.db
      2        ONLINE  INTERMEDIATE node2              Mounted (Closed),STABLE
ora.dbproddr.dbdr.svc
      2        ONLINE  OFFLINE                                          STABLE
--------------------------------------------------------------------------------

$ srvctl status database -d dbproddr -v
Instance dbproddr1 is running on node node1 with online services dbdr. Instance status: Open,Readonly.
Instance dbproddr2 is running on node node2. Instance status: Mounted (Closed).

Run opatchauto and ctrl-c from session is stuck.

node2 ~ # export PATCH_TOP_DIR=/u01/software/patches/Jan2019

node2 ~ # $GRID_HOME/OPatch/opatchauto apply $PATCH_TOP_DIR/28833531 -norestart

OPatchauto session is initiated at Thu May 16 20:20:24 2019

System initialization log file is /u02/app/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2019-05-16_08-20-26PM.log.

Session log file is /u02/app/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2019-05-16_08-20-47PM.log
The id for this session is K43Y

Executing OPatch prereq operations to verify patch applicability on home /u02/app/12.1.0/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/12.1.0/db
Patch applicability verified successfully on home /u01/app/oracle/product/12.1.0/db

Patch applicability verified successfully on home /u02/app/12.1.0/grid


Verifying SQL patch applicability on home /u01/app/oracle/product/12.1.0/db
"/bin/sh -c 'cd /u01/app/oracle/product/12.1.0/db; ORACLE_HOME=/u01/app/oracle/product/12.1.0/db ORACLE_SID=dbproddr2 /u01/app/oracle/product/12.1.0/db/OPatch/datapatch -prereq -verbose'" command failed with errors. Please refer to logs for more details. SQL changes, if any, can be analyzed by manually retrying the same command.

SQL patch applicability verified successfully on home /u01/app/oracle/product/12.1.0/db


Preparing to bring down database service on home /u01/app/oracle/product/12.1.0/db
Successfully prepared home /u01/app/oracle/product/12.1.0/db to bring down database service


Bringing down CRS service on home /u02/app/12.1.0/grid
Prepatch operation log file location: /u02/app/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_node2_2019-05-16_08-21-16PM.log
CRS service brought down successfully on home /u02/app/12.1.0/grid


Performing prepatch operation on home /u01/app/oracle/product/12.1.0/db
Perpatch operation completed successfully on home /u01/app/oracle/product/12.1.0/db


Start applying binary patch on home /u01/app/oracle/product/12.1.0/db
Binary patch applied successfully on home /u01/app/oracle/product/12.1.0/db


Performing postpatch operation on home /u01/app/oracle/product/12.1.0/db
Postpatch operation completed successfully on home /u01/app/oracle/product/12.1.0/db


Start applying binary patch on home /u02/app/12.1.0/grid

Binary patch applied successfully on home /u02/app/12.1.0/grid


Starting CRS service on home /u02/app/12.1.0/grid





*** Ctrl-C as shown below ***
^C
OPatchauto session completed at Thu May 16 21:41:58 2019
*** Time taken to complete the session 81 minutes, 34 seconds ***

opatchauto failed with error code 130

This is not good as session disconnected while troubleshooting in another session.

node2 ~ # timed out waiting for input: auto-logout

Even though opatchauto session was terminated cluster upgrade state is [NORMAL] vs cluster upgrade state is [ROLLING PATCH]

node2 ~ # crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [323461694].

node2 ~ # crsctl stat res -t -w '((TARGET != ONLINE) or (STATE != ONLINE)'
node2 ~ # crsctl stat res -t -w 'TYPE = ora.database.type'
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.dbproddr.db
      1        ONLINE  ONLINE       node1              Open,Readonly,STABLE
      2        ONLINE  ONLINE       node2              Open,Readonly,STABLE
--------------------------------------------------------------------------------

At this point, I was not sure what to do since everything looked good and online.

Colleague helping me with troubleshooting stated patch completed successfully and the main question if we need to try “opatchauto resume”

However, I was not comfortable with the outcome and tried opatchauto resume and it worked like magic.

Reconnect and opatchauto resume

mdinh@node2 ~ $ sudo su - 
~ # . /home/oracle/working/dinh/gi.env
The Oracle base has been set to /u01/app/oracle
ORACLE_SID=+ASM4
ORACLE_BASE=/u01/app/oracle
GRID_HOME=/u02/app/12.1.0/grid
ORACLE_HOME=/u02/app/12.1.0/grid
Oracle Instance alive for sid "+ASM4"
~ # export PATCH_TOP_DIR=/u01/software/patches/Jan2019/
~ # $GRID_HOME/OPatch/opatchauto resume

OPatchauto session is initiated at Thu May 16 22:03:09 2019
Session log file is /u02/app/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2019-05-16_10-03-10PM.log
Resuming existing session with id K43Y

Starting CRS service on home /u02/app/12.1.0/grid
Postpatch operation log file location: /u02/app/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_node2_2019-05-16_10-03-17PM.log
CRS service started successfully on home /u02/app/12.1.0/grid


Preparing home /u01/app/oracle/product/12.1.0/db after database service restarted

OPatchauto is running in norestart mode. PDB instances will not be checked for database on the current node.
No step execution required.........
 

Trying to apply SQL patch on home /u01/app/oracle/product/12.1.0/db
SQL patch applied successfully on home /u01/app/oracle/product/12.1.0/db

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:node2
RAC Home:/u01/app/oracle/product/12.1.0/db
Version:12.1.0.2.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/software/patches/Jan2019/28833531/26983807
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/software/patches/Jan2019/28833531/28729220
Reason: This patch is not applicable to this specified target type - "rac_database"


==Following patches were SUCCESSFULLY applied:

Patch: /u01/software/patches/Jan2019/28833531/28729213
Log: /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatchauto/core/opatch/opatch2019-05-16_20-22-06PM_1.log

Patch: /u01/software/patches/Jan2019/28833531/28731800
Log: /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatchauto/core/opatch/opatch2019-05-16_20-22-06PM_1.log


Host:node2
CRS Home:/u02/app/12.1.0/grid
Version:12.1.0.2.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/software/patches/Jan2019/28833531/26983807
Reason: This patch is already been applied, so not going to apply again.


==Following patches were SUCCESSFULLY applied:

Patch: /u01/software/patches/Jan2019/28833531/28729213
Log: /u02/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2019-05-16_20-23-32PM_1.log

Patch: /u01/software/patches/Jan2019/28833531/28729220
Log: /u02/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2019-05-16_20-23-32PM_1.log

Patch: /u01/software/patches/Jan2019/28833531/28731800
Log: /u02/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2019-05-16_20-23-32PM_1.log


Patching session reported following warning(s): 
_________________________________________________

[WARNING] The database instance 'drinstance2' from '/u01/app/oracle/product/12.1.0/db', in host'node2' is not running. SQL changes, if any,  will not be applied.
To apply. the SQL changes, bring up the database instance and run the command manually from any one node (run as oracle).
Refer to the readme to get the correct steps for applying the sql changes.

[WARNING] The database instances will not be brought up under the 'norestart' option. The database instance 'drinstance2' from '/u01/app/oracle/product/12.1.0/db', in host'node2' is not running. SQL changes, if any,  will not be applied.
To apply. the SQL changes, bring up the database instance and run the command manually from any one node (run as oracle).
Refer to the readme to get the correct steps for applying the sql changes.


OPatchauto session completed at Thu May 16 22:10:01 2019
Time taken to complete the session 6 minutes, 52 seconds
~ # 

Logs:

oracle@node2:/u02/app/12.1.0/grid/cfgtoollogs/crsconfig
> ls -alrt
total 508
drwxr-x--- 2 oracle oinstall   4096 Nov 23 02:15 oracle
-rwxrwxr-x 1 oracle oinstall 167579 Nov 23 02:15 rootcrs_node2_2018-11-23_02-07-58AM.log
drwxrwxr-x 9 oracle oinstall   4096 Apr 10 12:05 ..

opatchauto apply - Prepatch operation log file location: /u02/app/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_node2_2019-05-16_08-21-16PM.log
====================================================================================================
-rwxrwxr-x 1 oracle oinstall  33020 May 16 20:22 crspatch_node2_2019-05-16_08-21-16PM.log
====================================================================================================

Mysterious log file - Unknown where this log is from because it was not from my terminal output.
====================================================================================================
-rwxrwxr-x 1 oracle oinstall  86983 May 16 21:42 crspatch_node2_2019-05-16_08-27-35PM.log
====================================================================================================

-rwxrwxr-x 1 oracle oinstall  56540 May 16 22:06 srvmcfg1.log
-rwxrwxr-x 1 oracle oinstall  26836 May 16 22:06 srvmcfg2.log
-rwxrwxr-x 1 oracle oinstall  21059 May 16 22:06 srvmcfg3.log
-rwxrwxr-x 1 oracle oinstall  23032 May 16 22:08 srvmcfg4.log

opatchauto resume - Postpatch operation log file location: /u02/app/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_node2_2019-05-16_10-03-17PM.log
====================================================================================================
-rwxrwxr-x 1 oracle oinstall  64381 May 16 22:09 crspatch_node2_2019-05-16_10-03-17PM.log
====================================================================================================

Prepatch operation log file.

> tail -20 crspatch_node2_2019-05-16_08-21-16PM.log
2019-05-16 20:22:04: Running as user oracle: /u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -chkckpt -name ROOTCRS_POSTPATCH_OOP_REQSTEPS
2019-05-16 20:22:04: s_run_as_user2: Running /bin/su oracle -c ' echo CLSRSC_START; /u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -chkckpt -name ROOTCRS_POSTPATCH_OOP_REQSTEPS '
2019-05-16 20:22:04: Removing file /tmp/fileTChFoS
2019-05-16 20:22:04: Successfully removed file: /tmp/fileTChFoS
2019-05-16 20:22:04: pipe exit code: 0
2019-05-16 20:22:04: /bin/su successfully executed

2019-05-16 20:22:04: checkpoint ROOTCRS_POSTPATCH_OOP_REQSTEPS does not exist
2019-05-16 20:22:04: Done - Performing pre-pathching steps required for GI stack
2019-05-16 20:22:04: Resetting cluutil_trc_suff_pp to 0
2019-05-16 20:22:04: Invoking "/u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_PREPATCH -state SUCCESS"
2019-05-16 20:22:04: trace file=/u01/app/oracle/crsdata/node2/crsconfig/cluutil0.log
2019-05-16 20:22:04: Running as user oracle: /u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_PREPATCH -state SUCCESS
2019-05-16 20:22:04: s_run_as_user2: Running /bin/su oracle -c ' echo CLSRSC_START; /u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_PREPATCH -state SUCCESS '
2019-05-16 20:22:04: Removing file /tmp/fileDoYyQA
2019-05-16 20:22:04: Successfully removed file: /tmp/fileDoYyQA
2019-05-16 20:22:04: pipe exit code: 0
2019-05-16 20:22:04: /bin/su successfully executed

*** 2019-05-16 20:22:04: Succeeded in writing the checkpoint:'ROOTCRS_PREPATCH' with status:SUCCESS ***

Mysterious log file – crspatch_node2_2019-05-16_08-27-35PM.log

2019-05-16 21:42:00: Succeeded in writing the checkpoint:'ROOTCRS_STACK' with status:FAIL
2019-05-16 21:42:00: ###### Begin DIE Stack Trace ######
2019-05-16 21:42:00:     Package         File                 Line Calling   
2019-05-16 21:42:00:     --------------- -------------------- ---- ----------
2019-05-16 21:42:00:  1: main            rootcrs.pl            267 crsutils::dietrap
2019-05-16 21:42:00:  2: crsutils        crsutils.pm          1631 main::__ANON__
2019-05-16 21:42:00:  3: crsutils        crsutils.pm          1586 crsutils::system_cmd_capture_noprint
2019-05-16 21:42:00:  4: crsutils        crsutils.pm          9098 crsutils::system_cmd_capture
2019-05-16 21:42:00:  5: crspatch        crspatch.pm           988 crsutils::startFullStack
2019-05-16 21:42:00:  6: crspatch        crspatch.pm          1121 crspatch::performPostPatch
2019-05-16 21:42:00:  7: crspatch        crspatch.pm           212 crspatch::crsPostPatch
2019-05-16 21:42:00:  8: main            rootcrs.pl            276 crspatch::new
2019-05-16 21:42:00: ####### End DIE Stack Trace #######

2019-05-16 21:42:00: ROOTCRS_POSTPATCH checkpoint has failed
2019-05-16 21:42:00:      ckpt: -ckpt -oraclebase /u01/app/oracle -chkckpt -name ROOTCRS_POSTPATCH
2019-05-16 21:42:00: Invoking "/u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -chkckpt -name ROOTCRS_POSTPATCH"
2019-05-16 21:42:00: trace file=/u01/app/oracle/crsdata/node2/crsconfig/cluutil4.log
2019-05-16 21:42:00: Running as user oracle: /u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -chkckpt -name ROOTCRS_POSTPATCH
2019-05-16 21:42:00: s_run_as_user2: Running /bin/su oracle -c ' echo CLSRSC_START; /u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -chkckpt -name ROOTCRS_POSTPATCH '
2019-05-16 21:42:00: Removing file /tmp/filewniUim
2019-05-16 21:42:00: Successfully removed file: /tmp/filewniUim
2019-05-16 21:42:00: pipe exit code: 0
2019-05-16 21:42:00: /bin/su successfully executed

2019-05-16 21:42:00: Invoking "/u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -chkckpt -name ROOTCRS_POSTPATCH -status"
2019-05-16 21:42:00: trace file=/u01/app/oracle/crsdata/node2/crsconfig/cluutil5.log
2019-05-16 21:42:00: Running as user oracle: /u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -chkckpt -name ROOTCRS_POSTPATCH -status
2019-05-16 21:42:00: s_run_as_user2: Running /bin/su oracle -c ' echo CLSRSC_START; /u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -chkckpt -name ROOTCRS_POSTPATCH -status '
2019-05-16 21:42:00: Removing file /tmp/fileK1Tyw6
2019-05-16 21:42:00: Successfully removed file: /tmp/fileK1Tyw6
2019-05-16 21:42:00: pipe exit code: 0
2019-05-16 21:42:00: /bin/su successfully executed

2019-05-16 21:42:00: The 'ROOTCRS_POSTPATCH' status is FAILED
2019-05-16 21:42:00: ROOTCRS_POSTPATCH state is FAIL
2019-05-16 21:42:00: Invoking "/u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_POSTPATCH -state FAIL"
2019-05-16 21:42:00: trace file=/u01/app/oracle/crsdata/node2/crsconfig/cluutil6.log
2019-05-16 21:42:00: Running as user oracle: /u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_POSTPATCH -state FAIL
2019-05-16 21:42:00: s_run_as_user2: Running /bin/su oracle -c ' echo CLSRSC_START; /u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_POSTPATCH -state FAIL '
2019-05-16 21:42:00: Removing file /tmp/filej20epR
2019-05-16 21:42:00: Successfully removed file: /tmp/filej20epR
2019-05-16 21:42:00: pipe exit code: 0
2019-05-16 21:42:00: /bin/su successfully executed

2019-05-16 21:42:00: Succeeded in writing the checkpoint:'ROOTCRS_POSTPATCH' with status:FAIL
2019-05-16 21:42:00: Invoking "/u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_STACK -state FAIL"
2019-05-16 21:42:00: trace file=/u01/app/oracle/crsdata/node2/crsconfig/cluutil7.log
2019-05-16 21:42:00: Running as user oracle: /u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_STACK -state FAIL
2019-05-16 21:42:00: s_run_as_user2: Running /bin/su oracle -c ' echo CLSRSC_START; /u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_STACK -state FAIL '
2019-05-16 21:42:01: Removing file /tmp/filely834C
2019-05-16 21:42:01: Successfully removed file: /tmp/filely834C
2019-05-16 21:42:01: pipe exit code: 0
2019-05-16 21:42:01: /bin/su successfully executed

*** 2019-05-16 21:42:01: Succeeded in writing the checkpoint:'ROOTCRS_STACK' with status:FAIL ***

Postpatch operation log file.

> tail -20 crspatch_node2_2019-05-16_10-03-17PM.log
2019-05-16 22:09:59: Invoking "/u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_PREPATCH -state START"
2019-05-16 22:09:59: trace file=/u01/app/oracle/crsdata/node2/crsconfig/cluutil7.log
2019-05-16 22:09:59: Running as user oracle: /u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_PREPATCH -state START
2019-05-16 22:09:59: s_run_as_user2: Running /bin/su oracle -c ' echo CLSRSC_START; /u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_PREPATCH -state START '
2019-05-16 22:09:59: Removing file /tmp/file0IogVl
2019-05-16 22:09:59: Successfully removed file: /tmp/file0IogVl
2019-05-16 22:09:59: pipe exit code: 0
2019-05-16 22:09:59: /bin/su successfully executed

2019-05-16 22:09:59: Succeeded in writing the checkpoint:'ROOTCRS_PREPATCH' with status:START
2019-05-16 22:09:59: Invoking "/u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_POSTPATCH -state SUCCESS"
2019-05-16 22:09:59: trace file=/u01/app/oracle/crsdata/node2/crsconfig/cluutil8.log
2019-05-16 22:09:59: Running as user oracle: /u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_POSTPATCH -state SUCCESS
2019-05-16 22:09:59: s_run_as_user2: Running /bin/su oracle -c ' echo CLSRSC_START; /u02/app/12.1.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_POSTPATCH -state SUCCESS '
2019-05-16 22:09:59: Removing file /tmp/fileXDCkuM
2019-05-16 22:09:59: Successfully removed file: /tmp/fileXDCkuM
2019-05-16 22:09:59: pipe exit code: 0
2019-05-16 22:09:59: /bin/su successfully executed

*** 2019-05-16 22:09:59: Succeeded in writing the checkpoint:'ROOTCRS_POSTPATCH' with status:SUCCESS ***

Happy patching and hopefully patching primary to come will be seamlessly successful.

May 12, 2019

EM13.3 Directory Structures

Filed under: Grid Control — mdinh @ 2:41 pm

Currently, I am preparing POC to migrate OMS 13.3 from OEL6 to OEL7 and wanted a high level overview of the installation.

[oracle@MGOEM ~]$ cat .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

export PATH=$PATH:$HOME/bin
export DISPLAY=127.0.0.1:10.0

export ORACLE_BASE=/u01/app/oracle
export AGENT_BASE=$ORACLE_BASE/agent

export AGENT_HOME=$AGENT_BASE/agent_13.3.0.0.0
export EM_INSTANCE_BASE=$ORACLE_BASE/gc_inst
export OMS_INSTANCE_BASE=$EM_INSTANCE_BASE
export OHS=$EM_INSTANCE_BASE/user_projects/domains/GCDomain/servers/ohs1

### Starting from 13cR1, Oracle home (or OMS home) refers to the Middleware home.
export ORACLE_HOME=$ORACLE_BASE/middleware
export MW_HOME=$ORACLE_HOME
export OMS_HOME=$ORACLE_HOME
[oracle@MGOEM ~]$

Overview of the Directories Created for OMS Installation.
The OMS instance base directory (typically, gc_inst) is maintained outside the middleware home

[oracle@MGOEM ~]$ cd $MW_HOME; pwd; ls
/u01/app/oracle/middleware
allroot.sh   common               embip          ldap           OMSPatcher     plsql                root.sh     ucp
asr          create_header.log    gccompliance   lib            OPatch         plugins              slax        user_projects
bi           crs                  has            logs           oracle_common  plugins_common       soa         webgate
bin          css                  install        network        oracore        postjava_header.log  sqlplus     wlserver
bmp          disc                 instantclient  nls            oraInst.loc    precomp              srvm        xdk
cfgtoollogs  doc                  inventory      ocm            ord            rdbms                stage
clone        domain-registry.xml  jdbc           ohs            oui            relnotes             sysman
coherence    em                   jlib           omscarespfile  perl           response             thirdparty
[oracle@MGOEM middleware]$

Overview of the Directories Created for Management Agent Installation (Central Agent).
Agent base directory for the central agent (Management Agent installed with the OMS).

[oracle@MGOEM middleware]$ cd $AGENT_BASE; pwd; ls
/u01/app/oracle/agent
agent_13.3.0.0.0  agent_inst  agentInstall.rsp
[oracle@MGOEM agent]$

Agent home that is within the agent base directory.

[oracle@MGOEM agent]$ cd $AGENT_HOME; pwd; ls
/u01/app/oracle/agent/agent_13.3.0.0.0
agent.rsp    EMStage        jdbc  jythonLib  OPatch         perl     replacebins.sh           sbin    xsds
bin          install        jdk   ldap       oracle_common  plugins  replacebins.sh.template  stage
cfgtoollogs  instantclient  jlib  lib        oraInst.loc    prereqs  root.sh                  sysman
config       inventory      js    ocm        oui            rda      root.sh.template         ucp
[oracle@MGOEM agent_13.3.0.0.0]$

The OMS instance base directory (typically, gc_inst) is maintained outside the middleware home.

[oracle@MGOEM agent_13.3.0.0.0]$ cd $OMS_INSTANCE_BASE; pwd; ls
/u01/app/oracle/gc_inst
em  user_projects
[oracle@MGOEM gc_inst]$

ORACLE_BASE

[oracle@MGOEM gc_inst]$  cd $ORACLE_BASE; pwd; ls
/u01/app/oracle
agent  bip  gc_inst  middleware  swlib
[oracle@MGOEM oracle]$

Inventory and Patches:

[oracle@MGOEM ~]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2015, Oracle. All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>13.8.0.0.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="oms13c1" LOC="/u01/app/oracle/middleware" TYPE="O" IDX="1"/>
<HOME NAME="agent13c1" LOC="/u01/app/oracle/agent/agent_13.3.0.0.0" TYPE="O" IDX="2"/>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>
[oracle@MGOEM ~]$

[oracle@MGOEM ~]$ $AGENT_HOME/OPatch/opatch lspatches
27839641;One-off
27369653;One-off
27244723;One-off
27074880;OPSS Bundle Patch 12.1.3.0.171124
26933408;One-off
25832897;One-off
25412962;
23519804;One-off
20882747;One-off
20442348;One-off
19982906;One-off
19345252;One-off
18814458;One-off
28042003;One-off
27419391;WLS PATCH SET UPDATE 12.1.3.0.180417
23527146;One-off
20741228;JDBC 12.1.3.1 BP1

OPatch succeeded.
[oracle@MGOEM ~]$

[oracle@MGOEM ~]$ $ORACLE_HOME/OPatch/opatch lspatches
27839641;One-off
27369653;One-off
27244723;One-off
27074880;OPSS Bundle Patch 12.1.3.0.171124
26933408;One-off
25832897;One-off
25412962;
23519804;One-off
20882747;One-off
20442348;One-off
19982906;One-off
19345252;One-off
18814458;One-off
28042003;One-off
27419391;WLS PATCH SET UPDATE 12.1.3.0.180417
23527146;One-off
20741228;JDBC 12.1.3.1 BP1

OPatch succeeded.
[oracle@MGOEM ~]$

[oracle@MGOEM ~]$ $ORACLE_HOME/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 13.8.0.0.0
Copyright (c) 2019, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/middleware
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/middleware/oraInst.loc
OPatch version    : 13.8.0.0.0
OUI version       : 13.8.0.0.0
Log file location : /u01/app/oracle/middleware/cfgtoollogs/opatch/opatch2019-05-12_16-34-38PM_1.log


OPatch detects the Middleware Home as "/u01/app/oracle/middleware"

Lsinventory Output file location : /u01/app/oracle/middleware/cfgtoollogs/opatch/lsinv/lsinventory2019-05-12_16-34-38PM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: MGOEM
ARU platform id: 226
ARU platform description:: Linux_AMD64

[oracle@MGOEM ~]$ cat /etc/oraInst.loc
inventory_loc=/u01/app/oraInventory
inst_group=oinstall
[oracle@MGOEM ~]$

[oracle@MGOEM ~]$ cat /u01/app/oracle/middleware/oraInst.loc
#Oracle Installer Location File Location
#Fri May 10 16:53:18 CEST 2019
inst_group=oinstall
inventory_loc=/u01/app/oraInventory
[oracle@MGOEM ~]$

Reference:
DIRECTORY STRUCTURE AND LOCATIONS OF IMPORTANT TRACE AND LOG FILES OF ENTERPRISE MANAGER CLOUD CONTROL 13C

Overview of the Directories Created for an Enterprise Manager System

May 11, 2019

Create Mount Filesystem for Vagrant VirtualBox

Filed under: Uncategorized — mdinh @ 2:41 pm

Once again, I am using oravirt boxes.

If you just want to create the machine, and not run the provisioning step run this:

vagrant up

Since I don’t know ansible, it was much simpler to do the work manually.

Oracle Linux Server release 7.3

Review disks:

[root@MGOEM ~]# fdisk -l /dev/sd*

Disk /dev/sda: 52.4 GB, 52428800000 bytes, 102400000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000979b6

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   102399999    50150400   8e  Linux LVM

Disk /dev/sda1: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sda2: 51.4 GB, 51354009600 bytes, 100300800 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

### Disk is not partitioned since there are no # for device /dev/sdb and no Filesystem
Disk /dev/sdb: 187.9 GB, 187904819200 bytes, 367001600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
[root@MGOEM ~]#

Create partition:

[root@MGOEM ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x37a8a8de.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (2048-367001599, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-367001599, default 367001599):
Using default value 367001599
Partition 1 of type Linux and of size 175 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@MGOEM ~]#

Review disk: Linux System

[root@MGOEM ~]# fdisk -l /dev/sdb

Disk /dev/sdb: 187.9 GB, 187904819200 bytes, 367001600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x37a8a8de

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   367001599   183499776   83  Linux
[root@MGOEM ~]#

Create Filesystem:

[root@MGOEM ~]# mkfs.ext4 /dev/sdb1
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
11468800 inodes, 45874944 blocks
2293747 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2193620992
1400 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

[root@MGOEM ~]#

Create and mount /u01:

[root@MGOEM ~]# mkdir -p /u01
[root@MGOEM ~]# mount /dev/sdb1 /u01
[root@MGOEM ~]#
[root@MGOEM ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
devtmpfs              2.8G     0  2.8G   0% /dev
tmpfs                 2.8G     0  2.8G   0% /dev/shm
tmpfs                 2.8G  8.4M  2.8G   1% /run
tmpfs                 2.8G     0  2.8G   0% /sys/fs/cgroup
/dev/mapper/ol-root    46G  2.1G   44G   5% /
/dev/sda1            1014M  167M  848M  17% /boot
vagrant               932G  283G  650G  31% /vagrant
sf_working            420G  139G  281G  33% /sf_working
media_patch           3.7T  513G  3.2T  14% /media/patch
media_swrepo          3.7T  513G  3.2T  14% /media/swrepo
sf_OracleSoftware     3.7T  513G  3.2T  14% /sf_OracleSoftware
media_shared_storage  932G  283G  650G  31% /media/shared_storage
tmpfs                 571M     0  571M   0% /run/user/1000
/dev/sdb1             173G   61M  164G   1% /u01
[root@MGOEM ~]#

Update /etc/fstab:

[root@MGOEM ~]# tail /etc/fstab
# Created by anaconda on Tue Apr 18 08:50:14 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/ol-root     /                       xfs     defaults        0 0
UUID=ed2996e5-e077-4e23-83a5-10418226a725 /boot                   xfs     defaults        0 0
/dev/mapper/ol-swap     swap                    swap    defaults        0 0
/swapfile1              swap                    swap    defaults        0 0
/dev/sdb1               /u01                    ext4    defaults        1 1
[root@MGOEM ~]#

May 7, 2019

Remove GRID Home After Upgrade

Filed under: 12c,Grid Infrastructure,RAC — mdinh @ 9:53 pm

The environment started with a GRID 12.1.0.1 installation, upgraded to 18.3.0.0, and performed out-of-place patching (OOP) to 18.6.0.0.

As a result, there are three GRID homes and remove 12.1.0.1.

This demonstration will be for the last node from the cluster; however, the action performed will be the same for all nodes.

Review existing patch for Grid and Database homes:

[oracle@racnode-dc1-1 ~]$ ssh racnode-dc1-2 "/media/patch/lspatches.sh"
+ . /media/patch/gi.env
++ set +x
The Oracle base has been set to /u01/app/oracle
ORACLE_SID=+ASM2
ORACLE_BASE=/u01/app/oracle
GRID_HOME=/u01/18.3.0.0/grid_2
ORACLE_HOME=/u01/18.3.0.0/grid_2
Oracle Instance alive for sid "+ASM2"
+ /u01/18.3.0.0/grid_2/OPatch/opatch version
OPatch Version: 12.2.0.1.17

OPatch succeeded.
+ /u01/18.3.0.0/grid_2/OPatch/opatch lspatches
29302264;OCW RELEASE UPDATE 18.6.0.0.0 (29302264)
29301643;ACFS RELEASE UPDATE 18.6.0.0.0 (29301643)
29301631;Database Release Update : 18.6.0.0.190416 (29301631)
28547619;TOMCAT RELEASE UPDATE 18.0.0.0.0 (28547619)
28435192;DBWLM RELEASE UPDATE 18.0.0.0.0 (28435192)
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)

OPatch succeeded.
+ . /media/patch/hawk.env
++ set +x
The Oracle base remains unchanged with value /u01/app/oracle
ORACLE_UNQNAME=hawk
ORACLE_SID=hawk2
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/12.1.0.1/db1
Oracle Instance alive for sid "hawk2"
+ /u01/app/oracle/12.1.0.1/db1/OPatch/opatch version
OPatch Version: 12.2.0.1.17

OPatch succeeded.
+ /u01/app/oracle/12.1.0.1/db1/OPatch/opatch lspatches
28731800;Database Bundle Patch : 12.1.0.2.190115 (28731800)
28729213;OCW PATCH SET UPDATE 12.1.0.2.190115 (28729213)

OPatch succeeded.
+ exit
[oracle@racnode-dc1-1 ~]$

Notice that the GRID home is /u01/18.3.0.0/grid_2 because this was the suggestion from OOP process.
Based on experience, it might be better to name GRID home with the correct version, i.e. /u01/18.6.0.0/grid

Verify cluster state is [NORMAL]:

[oracle@racnode-dc1-1 ~]$ ssh racnode-dc1-2 "/media/patch/crs_Query.sh"
+ . /media/patch/gi.env
++ set +x
The Oracle base has been set to /u01/app/oracle
ORACLE_SID=+ASM2
ORACLE_BASE=/u01/app/oracle
GRID_HOME=/u01/18.3.0.0/grid_2
ORACLE_HOME=/u01/18.3.0.0/grid_2
Oracle Instance alive for sid "+ASM2"
+ crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [18.0.0.0.0]
+ crsctl query crs softwareversion
Oracle Clusterware version on node [racnode-dc1-2] is [18.0.0.0.0]
+ crsctl query crs softwarepatch
Oracle Clusterware patch level on node racnode-dc1-2 is [2056778364].
+ crsctl query crs releasepatch
Oracle Clusterware release patch level is [2056778364] and the complete list of patches [27908644 27923415 28090523 28090553 28090557 28256701 28435192 28547619 28822489 28864593 28864607 29301631 29301643 29302264 ] have been applied on the local node. The release patch string is [18.6.0.0.0].
+ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [18.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [2056778364].
+ exit
[oracle@racnode-dc1-1 ~]$

Check Oracle Inventory:

[oracle@racnode-dc1-2 ~]$ cat /etc/oraInst.loc
inventory_loc=/u01/app/oraInventory
inst_group=oinstall

[oracle@racnode-dc1-2 ~]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2019, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>12.2.0.4.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>

### GRID home (/u01/app/12.1.0.1/grid) to be removed.
========================================================================================
<HOME NAME="OraGI12Home1" LOC="/u01/app/12.1.0.1/grid" TYPE="O" IDX="1">
   <NODE_LIST>
      <NODE NAME="racnode-dc1-1"/>
      <NODE NAME="racnode-dc1-2"/>
   </NODE_LIST>
</HOME>
========================================================================================

<HOME NAME="OraDB12Home1" LOC="/u01/app/oracle/12.1.0.1/db1" TYPE="O" IDX="2">
   <NODE_LIST>
      <NODE NAME="racnode-dc1-1"/>
      <NODE NAME="racnode-dc1-2"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraGI18Home1" LOC="/u01/18.3.0.0/grid" TYPE="O" IDX="3"/>
<HOME NAME="OraHome1" LOC="/u01/18.3.0.0/grid_2" TYPE="O" IDX="4" CRS="true"/>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>
[oracle@racnode-dc1-2 ~]$

Remove GRID home (/u01/app/12.1.0.1/grid). Use -local flag to avoid any bug issues.

[oracle@racnode-dc1-2 ~]$ export ORACLE_HOME=/u01/app/12.1.0.1/grid
[oracle@racnode-dc1-2 ~]$ $ORACLE_HOME/oui/bin/runInstaller -detachHome -silent -local ORACLE_HOME=$ORACLE_HOME
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 16040 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'DetachHome' was successful.
[oracle@racnode-dc1-2 ~]$

Verify GRID home was removed:

[oracle@racnode-dc1-2 ~]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2014, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>12.1.0.2.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="OraDB12Home1" LOC="/u01/app/oracle/12.1.0.1/db1" TYPE="O" IDX="2">
   <NODE_LIST>
      <NODE NAME="racnode-dc1-1"/>
      <NODE NAME="racnode-dc1-2"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraGI18Home1" LOC="/u01/18.3.0.0/grid" TYPE="O" IDX="3"/>
<HOME NAME="OraHome1" LOC="/u01/18.3.0.0/grid_2" TYPE="O" IDX="4" CRS="true"/>

### GRID home (/u01/app/12.1.0.1/grid) removed.
================================================================================
<HOME NAME="OraGI12Home1" LOC="/u01/app/12.1.0.1/grid" TYPE="O" IDX="1" REMOVED="T"/>
================================================================================

</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>
[oracle@racnode-dc1-2 ~]$

Remove 12.1.0.1 directory:

[oracle@racnode-dc1-2 ~]$ sudo su -
Last login: Thu May  2 23:38:22 CEST 2019
[root@racnode-dc1-2 ~]# cd /u01/app/
[root@racnode-dc1-2 app]# ll
total 12
drwxr-xr-x  3 root   oinstall 4096 Apr 17 23:36 12.1.0.1
drwxrwxr-x 12 oracle oinstall 4096 Apr 30 18:05 oracle
drwxrwx---  5 oracle oinstall 4096 May  2 23:54 oraInventory
[root@racnode-dc1-2 app]# rm -rf 12.1.0.1/
[root@racnode-dc1-2 app]#

Check the cluster:

[root@racnode-dc1-2 app]# logout
[oracle@racnode-dc1-2 ~]$ . /media/patch/gi.env
The Oracle base has been set to /u01/app/oracle
ORACLE_SID=+ASM2
ORACLE_BASE=/u01/app/oracle
GRID_HOME=/u01/18.3.0.0/grid_2
ORACLE_HOME=/u01/18.3.0.0/grid_2
Oracle Instance alive for sid "+ASM2"
[oracle@racnode-dc1-2 ~]$ crsctl check cluster -all
**************************************************************
racnode-dc1-1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
racnode-dc1-2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[oracle@racnode-dc1-2 ~]$

Later, /u01/18.3.0.0/grid will be removed, too, if there are no issues with the most recent patch.

May 6, 2019

Notes: Troubleshooting GRID opatchauto and Sample GRID OOP Log

Filed under: opatchauto — mdinh @ 9:44 pm

Troubleshooting opatchauto Issues in Grid Infrastructure Environment (Doc ID 2467266.1)

Failure during prepatch execution
If any failure in pre-patch step, verify the logs under <oracle base>/crsdata/<hostname>/crsconfig/crspatch_<hostname>_<timestamp>.log

Failure during patching execution
If any failure in execution during patching, review the opatch execution logs under corresponding <oracle home>/cfgtoollogs/opatchauto location

Failure during post-patch execution
If any failure in post-patch execution, review the logs under <oracle base>/crsdata/<hostname>/crsconfig/crspatch_<hostname>_<timestamp>.log.

Generally, the error seen While Starting the Clusterware. In that situation, troubleshoot Grid Infrastructure issues referring Doc ID 1050908.1

How to debug opatchauto failures?
# export OPATCH_DEBUG=true
# opatchauto apply patch with location -loglevel FINEST

ACTUAL: Grid_Infrastructure_Out_of_Place_12.2 (GI/DB SAME version)

[root@racnode-dc2-1 ~]# export PATCH_HOME=/u01/stage/patch/Apr2019/29301687
[root@racnode-dc2-1 ~]# $GRID_HOME/OPatch/opatch version
OPatch Version: 12.2.0.1.17

[root@racnode-dc2-1 ~]# $GRID_HOME/OPatch/opatchauto apply $PATCH_HOME -prepare-clone -logLevel FINEST

System initialization log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchautodb/systemconfig2019-04-29_05-24-36PM.log.
Session log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/opatchauto2019-04-29_05-26-26PM.log
Prepatch operation log file location: /u01/app/oracle/crsdata/racnode-dc2-1/crsconfig/crspatch_racnode-dc2-1_2019-04-29_05-33-30PM.log

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:racnode-dc2-1
RAC Home:/u01/app/oracle/12.2.0.1/db1
Version:12.2.0.1.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/stage/patch/Apr2019/29301687/29301676
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/stage/patch/Apr2019/29301687/26839277
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/stage/patch/Apr2019/29301687/28566910
Reason: This patch is not applicable to this specified target type - "rac_database"


==Following patches were SUCCESSFULLY applied:

Patch: /u01/stage/patch/Apr2019/29301687/29314339
Log: /u01/app/oracle/12.2.0.1/db1_2/cfgtoollogs/opatchauto/core/opatch/opatch2019-04-29_17-42-54PM_1.log

Patch: /u01/stage/patch/Apr2019/29301687/29314424
Log: /u01/app/oracle/12.2.0.1/db1_2/cfgtoollogs/opatchauto/core/opatch/opatch2019-04-29_17-42-54PM_1.log


Host:racnode-dc2-1
CRS Home:/u01/app/12.2.0.1/grid
Version:12.2.0.1.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u01/stage/patch/Apr2019/29301687/26839277
Log: /u01/app/12.2.0.1/grid_2/cfgtoollogs/opatchauto/core/opatch/opatch2019-04-29_17-42-53PM_1.log

Patch: /u01/stage/patch/Apr2019/29301687/28566910
Log: /u01/app/12.2.0.1/grid_2/cfgtoollogs/opatchauto/core/opatch/opatch2019-04-29_17-42-53PM_1.log

Patch: /u01/stage/patch/Apr2019/29301687/29301676
Log: /u01/app/12.2.0.1/grid_2/cfgtoollogs/opatchauto/core/opatch/opatch2019-04-29_17-42-53PM_1.log

Patch: /u01/stage/patch/Apr2019/29301687/29314339
Log: /u01/app/12.2.0.1/grid_2/cfgtoollogs/opatchauto/core/opatch/opatch2019-04-29_17-42-53PM_1.log

Patch: /u01/stage/patch/Apr2019/29301687/29314424
Log: /u01/app/12.2.0.1/grid_2/cfgtoollogs/opatchauto/core/opatch/opatch2019-04-29_17-42-53PM_1.log


Out of place patching clone home(s) summary
____________________________________________
Host : racnode-dc2-1
Actual Home : /u01/app/oracle/12.2.0.1/db1
Version:12.2.0.1.0
Clone Home Path : /u01/app/oracle/12.2.0.1/db1_2

Host : racnode-dc2-1
Actual Home : /u01/app/12.2.0.1/grid
Version:12.2.0.1.0
Clone Home Path : /u01/app/12.2.0.1/grid_2


OPatchauto session completed at Mon Apr 29 18:03:48 2019
Time taken to complete the session 39 minutes, 16 seconds
[root@racnode-dc2-1 ~]#

ACTUAL: Grid_Infrastructure_Out_of_Place_18.6 (GI/DB DIFFERENT version)

+ /u01/18.3.0.0/grid_2/OPatch/opatch version
OPatch Version: 12.2.0.1.17

+ /u01/app/oracle/12.1.0.1/db1/OPatch/opatch version
OPatch Version: 12.2.0.1.17

[root@racnode-dc1-1 ~]# export PATCH_HOME=/u01/patch/Apr2019/29301682
[root@racnode-dc1-1 ~]# $GRID_HOME/OPatch/opatchauto apply $PATCH_HOME -prepare-clone -logLevel FINEST

System initialization log file is /u01/18.3.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2019-04-30_05-06-34PM.log.
Session log file is /u01/18.3.0.0/grid/cfgtoollogs/opatchauto/opatchauto2019-04-30_05-08-04PM.log

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:racnode-dc1-1
CRS Home:/u01/18.3.0.0/grid
Version:18.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/patch/Apr2019/29301682/28435192
Reason: This patch is already been applied, so not going to apply again.

Patch: /u01/patch/Apr2019/29301682/28547619
Reason: This patch is already been applied, so not going to apply again.


==Following patches were SUCCESSFULLY applied:

Patch: /u01/patch/Apr2019/29301682/29301631
Log: /u01/18.3.0.0/grid_2/cfgtoollogs/opatchauto/core/opatch/opatch2019-04-30_17-17-53PM_1.log

Patch: /u01/patch/Apr2019/29301682/29301643
Log: /u01/18.3.0.0/grid_2/cfgtoollogs/opatchauto/core/opatch/opatch2019-04-30_17-17-53PM_1.log

Patch: /u01/patch/Apr2019/29301682/29302264
Log: /u01/18.3.0.0/grid_2/cfgtoollogs/opatchauto/core/opatch/opatch2019-04-30_17-17-53PM_1.log


Out of place patching clone home(s) summary
____________________________________________
Host : racnode-dc1-1
Actual Home : /u01/18.3.0.0/grid
Version:18.0.0.0.0
Clone Home Path : /u01/18.3.0.0/grid_2


Following homes are skipped during patching as patches are not applicable:

/u01/app/oracle/12.1.0.1/db1

OPatchauto session completed at Tue Apr 30 17:27:21 2019
Time taken to complete the session 20 minutes, 52 seconds
[root@racnode-dc1-1 ~]#

May 5, 2019

What’s My Cluster Configuration

Filed under: 18c,Grid Infrastructure,RAC — mdinh @ 2:15 pm
[grid@ol7-183-node1 ~]$ . /media/patch/gi.env
The Oracle base has been set to /u01/app/grid
ORACLE_SID=+ASM1
ORACLE_BASE=/u01/app/grid
GRID_HOME=/u01/app/18.0.0.0/grid
ORACLE_HOME=/u01/app/18.0.0.0/grid
Oracle Instance alive for sid "+ASM1"

[grid@ol7-183-node1 ~]$ crsctl get cluster configuration
Name                : ol7-183-cluster
Configuration       : Cluster
Class               : Standalone Cluster
Type                : flex
The cluster is not extended.
--------------------------------------------------------------------------------
        MEMBER CLUSTER INFORMATION

      Name       Version        GUID                       Deployed Deconfigured
================================================================================
================================================================================

[grid@ol7-183-node1 ~]$ olsnodes -s -a -t
ol7-183-node1   Active  Hub     Unpinned
ol7-183-node2   Active  Hub     Unpinned

[grid@ol7-183-node1 ~]$ crsctl query crs releasepatch
Oracle Clusterware release patch level is [70732493] and the complete list of patches [27908644 27923415 28090523 28090553 28090557 28090564 28256701 ] have been applied on the local node. The release patch string is [18.3.0.0.0].

[grid@ol7-183-node1 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [18.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [70732493].
[grid@ol7-183-node1 ~]$

May 4, 2019

Updating vagrant-boxes/OracleRAC

Filed under: 18c,RAC,Vagrant,VirtualBox — mdinh @ 7:17 pm

I have been playing with and finally able to complete 18c RAC installation using oracle/vagrant-boxes/OracleRAC

Honestly, I am still fond of Mikael Sandström oravirt vagrant-boxes, but having some trouble with installations and thought to try something new.

Here are updates performed for oracle/vagrant-boxes/OracleRAC on all nodes and only showing one node as example.

/etc/oratab is not updated:

[oracle@ol7-183-node2 ~]$ ps -ef|grep pmon
grid      1155     1  0 14:00 ?        00:00:00 asm_pmon_+ASM2
oracle   18223 18079  0 14:43 pts/0    00:00:00 grep --color=auto pmon
oracle   31653     1  0 14:29 ?        00:00:00 ora_pmon_hawk2

[oracle@ol7-183-node2 ~]$ tail /etc/oratab
#   $ORACLE_SID:$ORACLE_HOME::
#
# The first and second fields are the system identifier and home
# directory of the database respectively.  The third field indicates
# to the dbstart utility that the database should , "Y", or should not,
# "N", be brought up at system boot time.
#
# Multiple entries with the same $ORACLE_SID are not allowed.
#
#

Update /etc/oratab [my framework works :=)]

[oracle@ol7-183-node2 ~]$ cat /etc/oratab
+ASM2:/u01/app/18.0.0.0/grid:N
hawk2:/u01/app/oracle/product/18.0.0.0/dbhome_1:N

[oracle@ol7-183-node2 ~]$ /media/patch/gi.env
The Oracle base has been set to /u01/app/grid
ORACLE_SID=+ASM2
ORACLE_BASE=/u01/app/grid
GRID_HOME=/u01/app/18.0.0.0/grid
ORACLE_HOME=/u01/app/18.0.0.0/grid
Oracle Instance not alive for sid "+ASM2"

[oracle@ol7-183-node2 ~]$ /media/patch/hawk.env
The Oracle base has been set to /u01/app/oracle
ORACLE_UNQNAME=hawk
ORACLE_SID=hawk2
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/product/18.0.0.0/dbhome_1
Oracle Instance alive for sid "hawk2"
[oracle@ol7-183-node2 ~]$

sudo for grid/oracle is not enabled:

[oracle@ol7-183-node2 ~]$ sudo /media/patch/findhomes.sh
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

[sudo] password for oracle:
oracle is not in the sudoers file.  This incident will be reported.
[oracle@ol7-183-node2 ~]$ exit

Enable sudo for grid/oracle: (shown as example for oracle and should be same for grid)

[vagrant@ol7-183-node2 ~]$ sudo su -
[root@ol7-183-node2 ~]# visudo
[root@ol7-183-node2 ~]# grep oracle /etc/sudoers
oracle  ALL=(ALL)       ALL
oracle  ALL=(ALL)       NOPASSWD: ALL
[root@ol7-183-node2 ~]# logout

[vagrant@ol7-183-node2 ~]$ sudo su - oracle
Last login: Sat May  4 14:43:46 -04 2019 on pts/0

[oracle@ol7-183-node2 ~]$ sudo /media/patch/findhomes.sh
   PID NAME                 ORACLE_HOME
  1155 asm_pmon_+asm2       /u01/app/18.0.0.0/grid/
 31653 ora_pmon_hawk2       /u01/app/oracle/product/18.0.0.0/dbhome_1/
[oracle@ol7-183-node2 ~]$

Login banner:

dinh@CMWPHV1 MINGW64 /c/vagrant-boxes/OracleRAC (master)
$ vagrant ssh node2
Last login: Sat May  4 14:43:40 2019 from 10.0.2.2

Welcome to Oracle Linux Server release 7.6 (GNU/Linux 4.14.35-1844.1.3.el7uek.x86_64)

The Oracle Linux End-User License Agreement can be viewed here:

    * /usr/share/eula/eula.en_US

For additional packages, updates, documentation and community help, see:

    * http://yum.oracle.com/

[vagrant@ol7-183-node2 ~]$

Remove login banner:

[root@ol7-183-node2 ~]# cp -v /etc/motd /etc/motd.bak
‘/etc/motd’ -> ‘/etc/motd.bak’
[root@ol7-183-node2 ~]# cat /dev/null > /etc/motd
[root@ol7-183-node2 ~]# logout
[vagrant@ol7-183-node2 ~]$ logout
Connection to 127.0.0.1 closed.

dinh@CMWPHV1 MINGW64 /c/vagrant-boxes/OracleRAC (master)
$ vagrant ssh node2
Last login: Sat May  4 15:00:06 2019 from 10.0.2.2
[vagrant@ol7-183-node2 ~]$

Mandatory GIMR is not installed:

    node1: -----------------------------------------------------------------
    node1: INFO: 2019-05-04 14:01:02: Make GI config command
    node1: -----------------------------------------------------------------
    node1: -----------------------------------------------------------------
    node1: INFO: 2019-05-04 14:01:02: Grid Infrastructure configuration as 'RAC'
    node1: INFO: 2019-05-04 14:01:02: - ASM library   : ASMLIB
    node1: INFO: 2019-05-04 14:01:02: - without MGMTDB: true
    node1: -----------------------------------------------------------------
    node1: Launching Oracle Grid Infrastructure Setup Wizard...

[oracle@ol7-183-node1 ~]$ ps -ef|grep pmon
grid      7294     1  0 13:53 ?        00:00:00 asm_pmon_+ASM1
oracle   10986     1  0 14:29 ?        00:00:00 ora_pmon_hawk1
oracle   28642 28586  0 15:12 pts/0    00:00:00 grep --color=auto pmon

[oracle@ol7-183-node1 ~]$ ssh ol7-183-node2
Last login: Sat May  4 14:48:20 2019
[oracle@ol7-183-node2 ~]$ ps -ef|grep pmon
grid      1155     1  0 14:00 ?        00:00:00 asm_pmon_+ASM2
oracle   29820 29711  0 15:12 pts/0    00:00:00 grep --color=auto pmon
oracle   31653     1  0 14:29 ?        00:00:00 ora_pmon_hawk2
[oracle@ol7-183-node2 ~]$

Create GMIR:
How to Move/Recreate GI Management Repository (GIMR / MGMTDB) to Different Shared Storage (Diskgroup, CFS or NFS etc) (Doc ID 1589394.1)
MDBUtil: GI Management Repository configuration tool (Doc ID 2065175.1)

[grid@ol7-183-node1 ~]$ ps -ef|grep pmon
grid      2286 27832  0 16:35 pts/0    00:00:00 grep --color=auto pmon
grid      7294     1  0 13:53 ?        00:00:00 asm_pmon_+ASM1
oracle   10986     1  0 14:29 ?        00:00:00 ora_pmon_hawk1

[grid@ol7-183-node1 ~]$ ll /tmp/mdbutil.*
-rwxr-xr-x. 1 grid oinstall 67952 May  4 16:02 /tmp/mdbutil.pl

[grid@ol7-183-node1 ~]$ /tmp/mdbutil.pl --status
mdbutil.pl version : 1.95
2019-05-04 16:35:44: I Checking CHM status...
2019-05-04 16:35:46: I Listener MGMTLSNR is configured and running on ol7-183-node1
2019-05-04 16:35:49: W MGMTDB is not configured on ol7-183-node1!
2019-05-04 16:35:49: W Cluster Health Monitor (CHM) is configured and not running on ol7-183-node1!

[grid@ol7-183-node1 ~]$ . /media/patch/gi.env
The Oracle base remains unchanged with value /u01/app/grid
ORACLE_SID=+ASM1
ORACLE_BASE=/u01/app/grid
GRID_HOME=/u01/app/18.0.0.0/grid
ORACLE_HOME=/u01/app/18.0.0.0/grid
Oracle Instance alive for sid "+ASM1"

[grid@ol7-183-node1 ~]$ asmcmd lsdg
State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512             512   4096  4194304     65520    63108                0           63108              0             Y  DATA/
MOUNTED  NORMAL  N         512             512   4096  4194304     16368    15260             4092            5584              0             N  RECO/

[grid@ol7-183-node1 ~]$ /tmp/mdbutil.pl --addmdb --target=+DATA -debug
mdbutil.pl version : 1.95
2019-05-04 16:36:57: D Executing: /u01/app/18.0.0.0/grid/bin/srvctl status diskgroup -g DATA
2019-05-04 16:36:58: D Exit code: 0
2019-05-04 16:36:58: D Output of last command execution:
Disk Group DATA is running on ol7-183-node1,ol7-183-node2
2019-05-04 16:36:58: I Starting To Configure MGMTDB at +DATA...
2019-05-04 16:36:58: D Executing: /u01/app/18.0.0.0/grid/bin/srvctl status mgmtlsnr
2019-05-04 16:36:59: D Exit code: 0
2019-05-04 16:36:59: D Output of last command execution:
Listener MGMTLSNR is enabled
2019-05-04 16:36:59: D Executing: /u01/app/18.0.0.0/grid/bin/srvctl status mgmtdb
2019-05-04 16:37:00: D Exit code: 1
2019-05-04 16:37:00: D Output of last command execution:
PRCD-1120 : The resource for database _mgmtdb could not be found.
2019-05-04 16:37:00: D Executing: /u01/app/18.0.0.0/grid/bin/srvctl status mgmtdb
2019-05-04 16:37:01: D Exit code: 1
2019-05-04 16:37:01: D Output of last command execution:
PRCD-1120 : The resource for database _mgmtdb could not be found.
2019-05-04 16:37:01: D Executing: /u01/app/18.0.0.0/grid/bin/srvctl stop mgmtlsnr
2019-05-04 16:37:05: D Exit code: 0
2019-05-04 16:37:05: D Output of last command execution:
2019-05-04 16:37:05: D Executing: /u01/app/18.0.0.0/grid/bin/crsctl query crs activeversion
2019-05-04 16:37:05: D Exit code: 0
2019-05-04 16:37:05: D Output of last command execution:
Oracle Clusterware active version on the cluster is [18.0.0.0.0]
2019-05-04 16:37:05: D Executing: /u01/app/18.0.0.0/grid/bin/srvctl enable qosmserver
2019-05-04 16:37:06: D Exit code: 2
2019-05-04 16:37:06: D Output of last command execution:
PRKF-1321 : QoS Management Server is already enabled.
2019-05-04 16:37:06: D Executing: /u01/app/18.0.0.0/grid/bin/srvctl start qosmserver
2019-05-04 16:37:07: D Exit code: 2
2019-05-04 16:37:07: D Output of last command execution:
PRCC-1014 : qosmserver was already running
2019-05-04 16:37:07: I Container database creation in progress... for GI 18.0.0.0.0
2019-05-04 16:37:07: D Executing: /u01/app/18.0.0.0/grid/bin/dbca  -silent -createDatabase -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName DATA -datafileJarLocation /u01/app/18.0.0.0/grid/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck
2019-05-04 16:55:03: D Exit code: 0
2019-05-04 16:55:03: D Output of last command execution:
Prepare for db operation
2019-05-04 16:55:03: I Plugable database creation in progress...
2019-05-04 16:55:03: D Executing: /u01/app/18.0.0.0/grid/bin/mgmtca -local
2019-05-04 16:59:32: D Exit code: 0
2019-05-04 16:59:32: D Output of last command execution:
2019-05-04 16:59:32: D Executing: scp /tmp/mdbutil.pl ol7-183-node1:/tmp/
2019-05-04 16:59:33: D Exit code: 0
2019-05-04 16:59:33: D Output of last command execution:
2019-05-04 16:59:33: I Executing "/tmp/mdbutil.pl --addchm" on ol7-183-node1 as root to configure CHM.
2019-05-04 16:59:33: D Executing: ssh root@ol7-183-node1 "/tmp/mdbutil.pl --addchm"
root@ol7-183-node1's password:
2019-05-04 16:59:42: D Exit code: 1
2019-05-04 16:59:42: D Output of last command execution:
mdbutil.pl version : 1.95
2019-05-04 16:59:42: W Not able to execute "/tmp/mdbutil.pl --addchm" on ol7-183-node1 as root to configure CHM.
2019-05-04 16:59:42: D Executing: scp /tmp/mdbutil.pl ol7-183-node2:/tmp/
2019-05-04 16:59:43: D Exit code: 0
2019-05-04 16:59:43: D Output of last command execution:
2019-05-04 16:59:43: I Executing "/tmp/mdbutil.pl --addchm" on ol7-183-node2 as root to configure CHM.
2019-05-04 16:59:43: D Executing: ssh root@ol7-183-node2 "/tmp/mdbutil.pl --addchm"
root@ol7-183-node2's password:
2019-05-04 16:59:51: D Exit code: 1
2019-05-04 16:59:51: D Output of last command execution:
mdbutil.pl version : 1.95
2019-05-04 16:59:51: W Not able to execute "/tmp/mdbutil.pl --addchm" on ol7-183-node2 as root to configure CHM.
2019-05-04 16:59:51: I MGMTDB & CHM configuration done!

[root@ol7-183-node1 ~]# . /media/patch/gi.env
The Oracle base has been set to /u01/app/grid
ORACLE_SID=+ASM1
ORACLE_BASE=/u01/app/grid
GRID_HOME=/u01/app/18.0.0.0/grid
ORACLE_HOME=/u01/app/18.0.0.0/grid
Oracle Instance alive for sid "+ASM1"

[root@ol7-183-node1 ~]# crsctl start res ora.crf -init
CRS-2501: Resource 'ora.crf' is disabled
CRS-4000: Command Start failed, or completed with errors.

[root@ol7-183-node1 ~]# crsctl modify res ora.crf -attr ENABLED=1 -init

[root@ol7-183-node1 ~]# crsctl start res ora.crf -init
CRS-2672: Attempting to start 'ora.crf' on 'ol7-183-node1'
CRS-2676: Start of 'ora.crf' on 'ol7-183-node1' succeeded

[root@ol7-183-node1 ~]# crsctl stat res ora.crf -init
NAME=ora.crf
TYPE=ora.crf.type
TARGET=ONLINE
STATE=ONLINE on ol7-183-node1

[root@ol7-183-node1 ~]# ll /tmp/mdbutil.pl
-rwxr-xr-x. 1 grid oinstall 67952 May  4 16:59 /tmp/mdbutil.pl
[root@ol7-183-node1 ~]# /tmp/mdbutil.pl --addchm
mdbutil.pl version : 1.95
2019-05-04 17:02:54: I Starting To Configure CHM...
2019-05-04 17:02:55: I CHM has already been configured!
2019-05-04 17:02:57: I CHM Configure Successfully Completed!
[root@ol7-183-node1 ~]#

[root@ol7-183-node1 ~]# ssh ol7-183-node2
Last login: Sat May  4 16:28:28 2019
[root@ol7-183-node2 ~]# . /media/patch/gi.env
The Oracle base has been set to /u01/app/grid
ORACLE_SID=+ASM2
ORACLE_BASE=/u01/app/grid
GRID_HOME=/u01/app/18.0.0.0/grid
ORACLE_HOME=/u01/app/18.0.0.0/grid
Oracle Instance alive for sid "+ASM2"
[root@ol7-183-node2 ~]# crsctl stat res ora.crf -init
NAME=ora.crf
TYPE=ora.crf.type
TARGET=OFFLINE
STATE=OFFLINE

[root@ol7-183-node2 ~]# crsctl modify res ora.crf -attr ENABLED=1 -init
[root@ol7-183-node2 ~]# crsctl start res ora.crf -init
CRS-2672: Attempting to start 'ora.crf' on 'ol7-183-node2'
CRS-2676: Start of 'ora.crf' on 'ol7-183-node2' succeeded
[root@ol7-183-node2 ~]# crsctl stat res ora.crf -init
NAME=ora.crf
TYPE=ora.crf.type
TARGET=ONLINE
STATE=ONLINE on ol7-183-node2

[root@ol7-183-node2 ~]# ll /tmp/mdbutil.pl
-rwxr-xr-x. 1 grid oinstall 67952 May  4 16:59 /tmp/mdbutil.pl
[root@ol7-183-node2 ~]# /tmp/mdbutil.pl --addchm
mdbutil.pl version : 1.95
2019-05-04 17:04:41: I Starting To Configure CHM...
2019-05-04 17:04:41: I CHM has already been configured!
2019-05-04 17:04:44: I CHM Configure Successfully Completed!

[root@ol7-183-node2 ~]# logout
Connection to ol7-183-node2 closed.
[root@ol7-183-node1 ~]# logout

[grid@ol7-183-node1 ~]$ /tmp/mdbutil.pl --status
mdbutil.pl version : 1.95
2019-05-04 17:04:54: I Checking CHM status...
2019-05-04 17:04:56: I Listener MGMTLSNR is configured and running on ol7-183-node1
2019-05-04 17:04:59: I Database MGMTDB is configured and running on ol7-183-node1
2019-05-04 17:05:00: I Cluster Health Monitor (CHM) is configured and running
--------------------------------------------------------------------------------
CHM Repository Path = +DATA/_MGMTDB/881717C3357B4146E0536538A8C05D2C/DATAFILE/sysmgmtdata.291.1007398657
MGMTDB space used on DG +DATA = 23628 Mb
--------------------------------------------------------------------------------
[grid@ol7-183-node1 ~]$

Due to role separation, fix broken script for lspatches.

[grid@ol7-183-node2 ~]$ /media/patch/lspatches.sh
+ . /media/patch/gi.env
++ set +x
+ /u01/app/18.0.0.0/grid/OPatch/opatch version
OPatch Version: 12.2.0.1.14

OPatch succeeded.
+ /u01/app/18.0.0.0/grid/OPatch/opatch lspatches
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
28256701;TOMCAT RELEASE UPDATE 18.3.0.0.0 (28256701)
28090564;DBWLM RELEASE UPDATE 18.3.0.0.0 (28090564)
28090557;ACFS RELEASE UPDATE 18.3.0.0.0 (28090557)
28090553;OCW RELEASE UPDATE 18.3.0.0.0 (28090553)
28090523;Database Release Update : 18.3.0.0.180717 (28090523)

OPatch succeeded.
+ . /media/patch/hawk.env
++ set +x
+ /u01/app/oracle/product/18.0.0.0/dbhome_1/OPatch/opatch version
OPatch Version: 12.2.0.1.14

OPatch succeeded.
+ /u01/app/oracle/product/18.0.0.0/dbhome_1/OPatch/opatch lspatches

====================================================================================================
OPatch could not create/open history file for writing.
====================================================================================================

27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
28090553;OCW RELEASE UPDATE 18.3.0.0.0 (28090553)
28090523;Database Release Update : 18.3.0.0.180717 (28090523)

OPatch succeeded.
+ exit
[grid@ol7-183-node2 ~]$

====================================================================================================

[root@ol7-183-node2 ~]# . /media/patch/gi.env
The Oracle base has been set to /u01/app/grid
ORACLE_SID=+ASM2
ORACLE_BASE=/u01/app/grid
GRID_HOME=/u01/app/18.0.0.0/grid
ORACLE_HOME=/u01/app/18.0.0.0/grid
Oracle Instance alive for sid "+ASM2"
[root@ol7-183-node2 ~]# chmod 775 -R $ORACLE_HOME/cfgtoollogs

[root@ol7-183-node2 ~]# . /media/patch/hawk.env
The Oracle base has been changed from /u01/app/grid to /u01/app/oracle
ORACLE_UNQNAME=hawk
ORACLE_SID=hawk2
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/product/18.0.0.0/dbhome_1
Oracle Instance alive for sid "hawk2"
[root@ol7-183-node2 ~]# chmod 775 -R $ORACLE_HOME/cfgtoollogs

====================================================================================================

[vagrant@ol7-183-node2 ~]$ sudo su - grid /media/patch/lspatches.sh
Last login: Sat May  4 18:16:38 -04 2019
+ /u01/app/18.0.0.0/grid/OPatch/opatch version
OPatch Version: 12.2.0.1.14

OPatch succeeded.
+ /u01/app/18.0.0.0/grid/OPatch/opatch lspatches
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
28256701;TOMCAT RELEASE UPDATE 18.3.0.0.0 (28256701)
28090564;DBWLM RELEASE UPDATE 18.3.0.0.0 (28090564)
28090557;ACFS RELEASE UPDATE 18.3.0.0.0 (28090557)
28090553;OCW RELEASE UPDATE 18.3.0.0.0 (28090553)
28090523;Database Release Update : 18.3.0.0.180717 (28090523)

OPatch succeeded.
+ . /media/patch/hawk.env
++ set +x
+ /u01/app/oracle/product/18.0.0.0/dbhome_1/OPatch/opatch version
OPatch Version: 12.2.0.1.14

OPatch succeeded.
+ /u01/app/oracle/product/18.0.0.0/dbhome_1/OPatch/opatch lspatches
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
28090553;OCW RELEASE UPDATE 18.3.0.0.0 (28090553)
28090523;Database Release Update : 18.3.0.0.180717 (28090523)

OPatch succeeded.
+ exit
[vagrant@ol7-183-node2 ~]$ sudo su - oracle /media/patch/lspatches.sh
Last login: Sat May  4 18:15:18 -04 2019 on pts/0
+ /u01/app/18.0.0.0/grid/OPatch/opatch version
OPatch Version: 12.2.0.1.14

OPatch succeeded.
+ /u01/app/18.0.0.0/grid/OPatch/opatch lspatches
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
28256701;TOMCAT RELEASE UPDATE 18.3.0.0.0 (28256701)
28090564;DBWLM RELEASE UPDATE 18.3.0.0.0 (28090564)
28090557;ACFS RELEASE UPDATE 18.3.0.0.0 (28090557)
28090553;OCW RELEASE UPDATE 18.3.0.0.0 (28090553)
28090523;Database Release Update : 18.3.0.0.180717 (28090523)

OPatch succeeded.
+ . /media/patch/hawk.env
++ set +x
+ /u01/app/oracle/product/18.0.0.0/dbhome_1/OPatch/opatch version
OPatch Version: 12.2.0.1.14

OPatch succeeded.
+ /u01/app/oracle/product/18.0.0.0/dbhome_1/OPatch/opatch lspatches
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
28090553;OCW RELEASE UPDATE 18.3.0.0.0 (28090553)
28090523;Database Release Update : 18.3.0.0.180717 (28090523)

OPatch succeeded.
+ exit
[vagrant@ol7-183-node2 ~]$

I will update post as I progress.

May 3, 2019

GRID Out Of Place (OOP) Rollback Disaster

Filed under: 18c,Grid Infrastructure,RAC — mdinh @ 4:45 pm

Now I understand the hesitation to use Oracle new features, especially any auto.

It may just be simpler and less stress to perform manual task having control and knowing what is being executed and validated.

GRID Out Of Place (OOP) patching completed successfully for 18.6.0.0.0.

GRID_HOME=/u01/18.3.0.0/grid_2
ORACLE_HOME=/u01/app/oracle/12.1.0.1/db1

Here is an example of inventory after patching.

+ /u01/18.3.0.0/grid_2/OPatch/opatch lspatches
29302264;OCW RELEASE UPDATE 18.6.0.0.0 (29302264)
29301643;ACFS RELEASE UPDATE 18.6.0.0.0 (29301643)
29301631;Database Release Update : 18.6.0.0.190416 (29301631)
28547619;TOMCAT RELEASE UPDATE 18.0.0.0.0 (28547619)
28435192;DBWLM RELEASE UPDATE 18.0.0.0.0 (28435192)
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)

+ /u01/app/oracle/12.1.0.1/db1/OPatch/opatch lspatches
28731800;Database Bundle Patch : 12.1.0.2.190115 (28731800)
28729213;OCW PATCH SET UPDATE 12.1.0.2.190115 (28729213)

Run cluvfy was successful too.

[oracle@racnode-dc1-1 ~]$ cluvfy stage -post crsinst -n racnode-dc1-1,racnode-dc1-2 -verbose

Post-check for cluster services setup was successful.

CVU operation performed:      stage -post crsinst
Date:                         Apr 30, 2019 8:17:49 PM
CVU home:                     /u01/18.3.0.0/grid_2/
User:                         oracle
[oracle@racnode-dc1-1 ~]$

GRID OOP Rollback Patching completed successfully for node1.

[root@racnode-dc1-1 ~]# crsctl check cluster -all
**************************************************************
racnode-dc1-1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
racnode-dc1-2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@racnode-dc1-1 ~]#
[root@racnode-dc1-1 ~]# echo $GRID_HOME
/u01/18.3.0.0/grid_2
[root@racnode-dc1-1 ~]# $GRID_HOME/OPatch/opatchauto rollback -switch-clone -logLevel FINEST

OPatchauto session is initiated at Fri May  3 01:06:47 2019

System initialization log file is /u01/18.3.0.0/grid_2/cfgtoollogs/opatchautodb/systemconfig2019-05-03_01-06-50AM.log.

Session log file is /u01/18.3.0.0/grid_2/cfgtoollogs/opatchauto/opatchauto2019-05-03_01-08-00AM.log
The id for this session is R47N

Update nodelist in the inventory for oracle home /u01/18.3.0.0/grid.
Update nodelist in the inventory is completed for oracle home /u01/18.3.0.0/grid.


Bringing down CRS service on home /u01/18.3.0.0/grid
CRS service brought down successfully on home /u01/18.3.0.0/grid


Starting CRS service on home /u01/18.3.0.0/grid
CRS service started successfully on home /u01/18.3.0.0/grid


Confirm that all resources have been started from home /u01/18.3.0.0/grid.
All resources have been started successfully from home /u01/18.3.0.0/grid.


OPatchAuto successful.

--------------------------------Summary--------------------------------
Out of place patching clone home(s) summary
____________________________________________
Host : racnode-dc1-1
Actual Home : /u01/18.3.0.0/grid_2
Version:18.0.0.0.0
Clone Home Path : /u01/18.3.0.0/grid


Following homes are skipped during patching as patches are not applicable:

/u01/app/oracle/12.1.0.1/db1

OPatchauto session completed at Fri May  3 01:14:25 2019
Time taken to complete the session 7 minutes, 38 seconds

[root@racnode-dc1-1 ~]# crsctl check cluster -all
**************************************************************
racnode-dc1-1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
racnode-dc1-2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

[root@racnode-dc1-1 ~]# /media/patch/findhomes.sh
   PID NAME                 ORACLE_HOME
 10486 asm_pmon_+asm1       /u01/18.3.0.0/grid/
 10833 apx_pmon_+apx1       /u01/18.3.0.0/grid/

[root@racnode-dc1-1 ~]# cat /etc/oratab
#Backup file is  /u01/app/oracle/12.1.0.1/db1/srvm/admin/oratab.bak.racnode-dc1-1 line added by Agent
#+ASM1:/u01/18.3.0.0/grid:N
hawk1:/u01/app/oracle/12.1.0.1/db1:N
hawk:/u01/app/oracle/12.1.0.1/db1:N             # line added by Agent
[root@racnode-dc1-1 ~]#

GRID OOP Rollback Patching completed successfully for node2.

[root@racnode-dc1-2 ~]# crsctl check cluster -all
**************************************************************
racnode-dc1-1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
racnode-dc1-2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@racnode-dc1-2 ~]#
[root@racnode-dc1-2 ~]# echo $GRID_HOME
/u01/18.3.0.0/grid_2
[root@racnode-dc1-2 ~]# $GRID_HOME/OPatch/opatchauto rollback -switch-clone -logLevel FINEST

OPatchauto session is initiated at Fri May  3 01:21:39 2019

System initialization log file is /u01/18.3.0.0/grid_2/cfgtoollogs/opatchautodb/systemconfig2019-05-03_01-21-41AM.log.

Session log file is /u01/18.3.0.0/grid_2/cfgtoollogs/opatchauto/opatchauto2019-05-03_01-22-46AM.log
The id for this session is 9RAT

Update nodelist in the inventory for oracle home /u01/18.3.0.0/grid.
Update nodelist in the inventory is completed for oracle home /u01/18.3.0.0/grid.


Bringing down CRS service on home /u01/18.3.0.0/grid
CRS service brought down successfully on home /u01/18.3.0.0/grid


Starting CRS service on home /u01/18.3.0.0/grid
CRS service started successfully on home /u01/18.3.0.0/grid


Confirm that all resources have been started from home /u01/18.3.0.0/grid.
All resources have been started successfully from home /u01/18.3.0.0/grid.


OPatchAuto successful.

--------------------------------Summary--------------------------------
Out of place patching clone home(s) summary
____________________________________________
Host : racnode-dc1-2
Actual Home : /u01/18.3.0.0/grid_2
Version:18.0.0.0.0
Clone Home Path : /u01/18.3.0.0/grid


Following homes are skipped during patching as patches are not applicable:

/u01/app/oracle/12.1.0.1/db1


OPatchauto session completed at Fri May  3 01:40:51 2019
Time taken to complete the session 19 minutes, 12 seconds
[root@racnode-dc1-2 ~]#

GRID OOP Rollback completed successfully for 18.5.0.0.0.

GRID_HOME=/u01/18.3.0.0/grid
ORACLE_HOME=/u01/app/oracle/12.1.0.1/db1

Here is an example of inventory after rollback.

+ /u01/18.3.0.0/grid/OPatch/opatch lspatches
28864607;ACFS RELEASE UPDATE 18.5.0.0.0 (28864607)
28864593;OCW RELEASE UPDATE 18.5.0.0.0 (28864593)
28822489;Database Release Update : 18.5.0.0.190115 (28822489)
28547619;TOMCAT RELEASE UPDATE 18.0.0.0.0 (28547619)
28435192;DBWLM RELEASE UPDATE 18.0.0.0.0 (28435192)
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)

+ /u01/app/oracle/12.1.0.1/db1/OPatch/opatch lspatches
28731800;Database Bundle Patch : 12.1.0.2.190115 (28731800)
28729213;OCW PATCH SET UPDATE 12.1.0.2.190115 (28729213)

Validation shows database is OFFLINE,

+ crsctl stat res -w '((TARGET != ONLINE) or (STATE != ONLINE)' -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.GHCHKPT.advm
               OFFLINE OFFLINE      racnode-dc1-1            STABLE
               OFFLINE OFFLINE      racnode-dc1-2            STABLE
ora.crs.ghchkpt.acfs
               OFFLINE OFFLINE      racnode-dc1-1            STABLE
               OFFLINE OFFLINE      racnode-dc1-2            STABLE
ora.helper
               OFFLINE OFFLINE      racnode-dc1-1            STABLE
               OFFLINE OFFLINE      racnode-dc1-2            IDLE,STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.hawk.db
      1        ONLINE  OFFLINE                               Instance Shutdown,STABLE
      2        ONLINE  OFFLINE                               Instance Shutdown,STABLE

Start database FAILED.

[oracle@racnode-dc1-2 ~]$ . /media/patch/hawk.env
The Oracle base has been set to /u01/app/oracle
ORACLE_UNQNAME=hawk
ORACLE_SID=hawk2
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/12.1.0.1/db1
Oracle Instance not alive for sid "hawk2"

[oracle@racnode-dc1-2 ~]$ srvctl status database -d $ORACLE_UNQNAME -v
Instance hawk1 is not running on node racnode-dc1-1
Instance hawk2 is not running on node racnode-dc1-2

[oracle@racnode-dc1-2 ~]$ srvctl start database -d $ORACLE_UNQNAME
PRCR-1079 : Failed to start resource ora.hawk.db
CRS-5017: The resource action "ora.hawk.db start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/hawk/spfilehawk.ora'
ORA-17503: ksfdopn:10 Failed to open file +DATA/hawk/spfilehawk.ora
ORA-27140: attach to post/wait facility failed
ORA-27300: OS system dependent operation:invalid_egid failed with status: 1
ORA-27301: OS failure message: Operation not permitted
ORA-27302: failure occurred at: skgpwinit6
ORA-27303: additional information: startup egid = 54321 (oinstall), current egid = 54322 (dba)
. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/racnode-dc1-1/crs/trace/crsd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.hawk.db' on 'racnode-dc1-1' failed
CRS-2632: There are no more servers to try to place resource 'ora.hawk.db' on that would satisfy its placement policy
CRS-5017: The resource action "ora.hawk.db start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/hawk/spfilehawk.ora'
ORA-17503: ksfdopn:10 Failed to open file +DATA/hawk/spfilehawk.ora
ORA-27140: attach to post/wait facility failed
ORA-27300: OS system dependent operation:invalid_egid failed with status: 1
ORA-27301: OS failure message: Operation not permitted
ORA-27302: failure occurred at: skgpwinit6
ORA-27303: additional information: startup egid = 54321 (oinstall), current egid = 54322 (dba)
. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/racnode-dc1-2/crs/trace/crsd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.hawk.db' on 'racnode-dc1-2' failed
[oracle@racnode-dc1-2 ~]$


[oracle@racnode-dc1-1 ~]$ . /media/patch/hawk.env
The Oracle base has been set to /u01/app/oracle
ORACLE_UNQNAME=hawk
ORACLE_SID=hawk1
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/12.1.0.1/db1
Oracle Instance not alive for sid "hawk1"

[oracle@racnode-dc1-1 ~]$ srvctl start database -d hawk
PRCR-1079 : Failed to start resource ora.hawk.db
CRS-5017: The resource action "ora.hawk.db start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/hawk/spfilehawk.ora'
ORA-17503: ksfdopn:10 Failed to open file +DATA/hawk/spfilehawk.ora
ORA-27140: attach to post/wait facility failed
ORA-27300: OS system dependent operation:invalid_egid failed with status: 1
ORA-27301: OS failure message: Operation not permitted
ORA-27302: failure occurred at: skgpwinit6
ORA-27303: additional information: startup egid = 54321 (oinstall), current egid = 54322 (dba)
. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/racnode-dc1-2/crs/trace/crsd_oraagent_oracle.trc".

CRS-5017: The resource action "ora.hawk.db start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/hawk/spfilehawk.ora'
ORA-17503: ksfdopn:10 Failed to open file +DATA/hawk/spfilehawk.ora
ORA-27140: attach to post/wait facility failed
ORA-27300: OS system dependent operation:invalid_egid failed with status: 1
ORA-27301: OS failure message: Operation not permitted
ORA-27302: failure occurred at: skgpwinit6
ORA-27303: additional information: startup egid = 54321 (oinstall), current egid = 54322 (dba)
. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/racnode-dc1-1/crs/trace/crsd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.hawk.db' on 'racnode-dc1-2' failed
CRS-2632: There are no more servers to try to place resource 'ora.hawk.db' on that would satisfy its placement policy
CRS-2674: Start of 'ora.hawk.db' on 'racnode-dc1-1' failed
[oracle@racnode-dc1-1 ~]$

Incorrect permissions for oracle library was the cause.
Change permissions for $GRID_HOME/bin/oracle (chmod 6751 $GRID_HOME/bin/oracle), stop and start CRS resolved the failure.

[oracle@racnode-dc1-1 dbs]$ ls -lhrt $ORACLE_HOME/bin/oracle
-rwsr-s--x 1 oracle dba 314M Apr 20 16:06 /u01/app/oracle/12.1.0.1/db1/bin/oracle

[oracle@racnode-dc1-1 dbs]$ ls -lhrt /u01/18.3.0.0/grid/bin/oracle
-rwxr-x--x 1 oracle oinstall 396M Apr 20 19:21 /u01/18.3.0.0/grid/bin/oracle

[oracle@racnode-dc1-1 dbs]$ cd /u01/18.3.0.0/grid/bin/
[oracle@racnode-dc1-1 bin]$ chmod 6751 oracle
[oracle@racnode-dc1-1 bin]$ ls -lhrt /u01/18.3.0.0/grid/bin/oracle
-rwsr-s--x 1 oracle oinstall 396M Apr 20 19:21 /u01/18.3.0.0/grid/bin/oracle

[root@racnode-dc1-1 ~]# . /media/patch/gi.env
The Oracle base has been set to /u01/app/oracle
ORACLE_SID=+ASM1
ORACLE_BASE=/u01/app/oracle
GRID_HOME=/u01/18.3.0.0/grid
ORACLE_HOME=/u01/18.3.0.0/grid
Oracle Instance alive for sid "+ASM1"
[root@racnode-dc1-1 ~]# crsctl stop crs

====================================================================================================

[root@racnode-dc1-2 ~]# . /media/patch/gi.env
The Oracle base has been set to /u01/app/oracle
ORACLE_SID=+ASM2
ORACLE_BASE=/u01/app/oracle
GRID_HOME=/u01/18.3.0.0/grid
ORACLE_HOME=/u01/18.3.0.0/grid
Oracle Instance alive for sid "+ASM2"

[root@racnode-dc1-2 ~]# ls -lhrt $GRID_HOME/bin/oracle
-rwxr-x--x 1 oracle oinstall 396M Apr 21 01:44 /u01/18.3.0.0/grid/bin/oracle

[root@racnode-dc1-2 ~]# chmod 6751 $GRID_HOME/bin/oracle
[root@racnode-dc1-2 ~]# ls -lhrt $GRID_HOME/bin/oracle
-rwsr-s--x 1 oracle oinstall 396M Apr 21 01:44 /u01/18.3.0.0/grid/bin/oracle

[root@racnode-dc1-2 ~]# crsctl stop crs

====================================================================================================

[root@racnode-dc1-2 ~]# crsctl start crs
[root@racnode-dc1-1 ~]# crsctl start crs

Reference: RAC Database Can’t Start: ORA-01565, ORA-17503: ksfdopn:10 Failed to open file +DATA/BPBL/spfileBPBL.ora (Doc ID 2316088.1)

May 1, 2019

VitualBox Cannot register the hard disk because a hard disk with UUID already exists

Filed under: Uncategorized — mdinh @ 1:03 pm

Here’s an issue I have faced multiple times and have finally able to find a reasonable resolution.

For the moment, it seems to be working and only time will tell as other predominant solutions on WWW did not work for me.

VBox version:

D:\VirtualBox>VBoxManage -version
6.0.4r128413

D:\VirtualBox>

Start VM failed after shutdown:

[oracle@racnode-dc2-1 dbca]$ logout
[vagrant@racnode-dc2-1 ~]$ logout
Connection to 127.0.0.1 closed.

dinh@CMWPHV1 MINGW64 /d/Vagrant2/vagrant-vbox-rac (master)
$ vagrant halt

==> racnode-dc2-1: Attempting graceful shutdown of VM...
==> racnode-dc2-1: Forcing shutdown of VM...
==> racnode-dc2-2: Unpausing the VM...
==> racnode-dc2-2: Attempting graceful shutdown of VM...
==> racnode-dc2-2: Forcing shutdown of VM...

dinh@CMWPHV1 MINGW64 /d/Vagrant2/vagrant-vbox-rac (master)
$ vagrant up
There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["showvminfo", "1d72cea4-f728-44d4-b69e-b2dd45064969"]

Stderr: VBoxManage.exe: error: Failed to create the VirtualBox object!
VBoxManage.exe: error: Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists
VBoxManage.exe: error: Details: code E_INVALIDARG (0x80070057), component VirtualBoxWrap, interface IVirtualBox

dinh@CMWPHV1 MINGW64 /d/Vagrant2/vagrant-vbox-rac (master)
$ vagrant status
There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["showvminfo", "1d72cea4-f728-44d4-b69e-b2dd45064969"]

Stderr: VBoxManage.exe: error: Failed to create the VirtualBox object!
VBoxManage.exe: error: Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists
VBoxManage.exe: error: Details: code E_INVALIDARG (0x80070057), component VirtualBoxWrap, interface IVirtualBox

dinh@CMWPHV1 MINGW64 /d/Vagrant2/vagrant-vbox-rac (master)
$ 

VBoxManage internalcommands sethduuid / clonevdi failed: UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists

D:\VirtualBox>VBoxManage.exe internalcommands sethduuid "D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk"
VBoxManage.exe: error: Failed to create the VirtualBox object!
VBoxManage.exe: error: Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35
-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists
VBoxManage.exe: error: Details: code E_INVALIDARG (0x80070057), component VirtualBoxWrap, interface IVirtualBox

D:\VirtualBox>VBoxManage clonevdi "D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk" "D:\VirtualBoxVM\vbox-rac-dc
2\racnode-dc2-2\packer-ol75-disk002.vmdk"
VBoxManage.exe: error: Failed to create the VirtualBox object!
VBoxManage.exe: error: Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35
-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c
748d54e-3cd2-4087-82dd-65324f4365f7} already exists
VBoxManage.exe: error: Details: code E_INVALIDARG (0x80070057), component VirtualBoxWrap, interface IVirtualBox

D:\VirtualBox>

Check UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} exists in VirtualBox.xml and VirtualBox.xml-prev:

dinh@CMWPHV1 MINGW64 ~/.VirtualBox
$ grep c748d54e-3cd2-4087-82dd-65324f4365f7 *
VBoxSVC.log:00:00:00.390000          ERROR [COM]: aRC=E_INVALIDARG (0x80070057) aIID={d0a0163f-e254-4e5b-a1f2-011cf991c38d} aComponent={VirtualBoxWrap} aText={Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists}, preserve=false aResultDetail=0
VBoxSVC.log.1:00:00:00.376400          ERROR [COM]: aRC=E_INVALIDARG (0x80070057) aIID={d0a0163f-e254-4e5b-a1f2-011cf991c38d} aComponent={VirtualBoxWrap} aText={Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists}, preserve=false aResultDetail=0
VBoxSVC.log.2:00:00:00.391000          ERROR [COM]: aRC=E_INVALIDARG (0x80070057) aIID={d0a0163f-e254-4e5b-a1f2-011cf991c38d} aComponent={VirtualBoxWrap} aText={Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists}, preserve=false aResultDetail=0
VBoxSVC.log.3:00:00:00.390000          ERROR [COM]: aRC=E_INVALIDARG (0x80070057) aIID={d0a0163f-e254-4e5b-a1f2-011cf991c38d} aComponent={VirtualBoxWrap} aText={Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists}, preserve=false aResultDetail=0
VBoxSVC.log.4:00:00:00.374400          ERROR [COM]: aRC=E_INVALIDARG (0x80070057) aIID={d0a0163f-e254-4e5b-a1f2-011cf991c38d} aComponent={VirtualBoxWrap} aText={Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists}, preserve=false aResultDetail=0
VBoxSVC.log.5:00:00:00.374400          ERROR [COM]: aRC=E_INVALIDARG (0x80070057) aIID={d0a0163f-e254-4e5b-a1f2-011cf991c38d} aComponent={VirtualBoxWrap} aText={Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists}, preserve=false aResultDetail=0
VBoxSVC.log.6:00:00:00.436800          ERROR [COM]: aRC=E_INVALIDARG (0x80070057) aIID={d0a0163f-e254-4e5b-a1f2-011cf991c38d} aComponent={VirtualBoxWrap} aText={Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists}, preserve=false aResultDetail=0

Entry Removed from VirtualBox.xml
------------------------------------------------------------------------------------------------------------------------
VirtualBox.xml:        <HardDisk uuid="{c748d54e-3cd2-4087-82dd-65324f4365f7}" location="D:/VirtualBoxVM/vbox-rac-dc2/racnode-dc2-2/packer-ol75-disk001.vmdk" format="VMDK" type="Normal">
------------------------------------------------------------------------------------------------------------------------

VirtualBox.xml-prev:        <HardDisk uuid="{c748d54e-3cd2-4087-82dd-65324f4365f7}" location="D:/VirtualBoxVM/vbox-rac-dc2/racnode-dc2-2/packer-ol75-disk001.vmdk" format="VMDK" type="Normal">

dinh@CMWPHV1 MINGW64 ~/.VirtualBox
$

Remove entry from VirtualBox.xml:

dinh@CMWPHV1 MINGW64 ~/.VirtualBox
$ grep c748d54e-3cd2-4087-82dd-65324f4365f7 *
VBoxSVC.log.2:00:00:00.391000          ERROR [COM]: aRC=E_INVALIDARG (0x80070057) aIID={d0a0163f-e254-4e5b-a1f2-011cf991c38d} aComponent={VirtualBoxWrap} aText={Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists}, preserve=false aResultDetail=0
VBoxSVC.log.3:00:00:00.390000          ERROR [COM]: aRC=E_INVALIDARG (0x80070057) aIID={d0a0163f-e254-4e5b-a1f2-011cf991c38d} aComponent={VirtualBoxWrap} aText={Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists}, preserve=false aResultDetail=0
VBoxSVC.log.4:00:00:00.376400          ERROR [COM]: aRC=E_INVALIDARG (0x80070057) aIID={d0a0163f-e254-4e5b-a1f2-011cf991c38d} aComponent={VirtualBoxWrap} aText={Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists}, preserve=false aResultDetail=0
VBoxSVC.log.5:00:00:00.391000          ERROR [COM]: aRC=E_INVALIDARG (0x80070057) aIID={d0a0163f-e254-4e5b-a1f2-011cf991c38d} aComponent={VirtualBoxWrap} aText={Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists}, preserve=false aResultDetail=0
VBoxSVC.log.6:00:00:00.390000          ERROR [COM]: aRC=E_INVALIDARG (0x80070057) aIID={d0a0163f-e254-4e5b-a1f2-011cf991c38d} aComponent={VirtualBoxWrap} aText={Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists}, preserve=false aResultDetail=0
VBoxSVC.log.7:00:00:00.374400          ERROR [COM]: aRC=E_INVALIDARG (0x80070057) aIID={d0a0163f-e254-4e5b-a1f2-011cf991c38d} aComponent={VirtualBoxWrap} aText={Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists}, preserve=false aResultDetail=0
VBoxSVC.log.8:00:00:00.374400          ERROR [COM]: aRC=E_INVALIDARG (0x80070057) aIID={d0a0163f-e254-4e5b-a1f2-011cf991c38d} aComponent={VirtualBoxWrap} aText={Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists}, preserve=false aResultDetail=0
VBoxSVC.log.9:00:00:00.436800          ERROR [COM]: aRC=E_INVALIDARG (0x80070057) aIID={d0a0163f-e254-4e5b-a1f2-011cf991c38d} aComponent={VirtualBoxWrap} aText={Cannot register the hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' {b507fc35-1c3c-46ab-9e0e-91f192c5b935} because a hard disk 'D:\VirtualBoxVM\vbox-rac-dc2\racnode-dc2-2\packer-ol75-disk001.vmdk' with UUID {c748d54e-3cd2-4087-82dd-65324f4365f7} already exists}, preserve=false aResultDetail=0

dinh@CMWPHV1 MINGW64 ~/.VirtualBox
$

Start VM:

dinh@CMWPHV1 MINGW64 /d/Vagrant2/vagrant-vbox-rac (master)
$ vagrant up
Bringing machine 'racnode-dc2-2' up with 'virtualbox' provider...
Bringing machine 'racnode-dc2-1' up with 'virtualbox' provider...

dinh@CMWPHV1 MINGW64 /d/Vagrant2/vagrant-vbox-rac (master)
$ vagrant status
Current machine states:

racnode-dc2-2             running (virtualbox)
racnode-dc2-1             running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

dinh@CMWPHV1 MINGW64 /d/Vagrant2/vagrant-vbox-rac (master)
$ vagrant ssh racnode-dc2-1
Last login: Wed May  1 14:40:57 2019 from 192.168.7.100

----------------------------------------
Welcome to racnode-dc2-1
OracleLinux 7.5 x86_64

FQDN: racnode-dc2-1.internal.lab
IP's:
enp0s3: 10.0.2.15
enp0s8: 192.168.7.100
enp0s9: 172.16.7.10

Processor: Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz
#CPU's:    2
Memory:    5708 MB
Kernel:    4.1.12-112.16.4.el7uek.x86_64

----------------------------------------
[vagrant@racnode-dc2-1 ~]$ sudo su - oracle
Last login: Wed May  1 14:42:47 CEST 2019
[oracle@racnode-dc2-1 ~]$ . /media/patch/gi.env
The Oracle base has been set to /u01/app/oracle
ORACLE_SID=+ASM1
ORACLE_BASE=/u01/app/oracle
GRID_HOME=/u01/app/12.2.0.1/grid
ORACLE_HOME=/u01/app/12.2.0.1/grid
Oracle Instance alive for sid "+ASM1"
[oracle@racnode-dc2-1 ~]$
[oracle@racnode-dc2-1 ~]$ ps -ef|grep [p]mon
oracle   17637     1  0 14:42 ?        00:00:00 asm_pmon_+ASM1
oracle   17987     1  0 14:43 ?        00:00:00 ora_pmon_hawk1
[oracle@racnode-dc2-1 ~]$ ssh racnode-dc2-2
Last login: Wed May  1 14:42:24 2019

----------------------------------------
Welcome to racnode-dc2-2
OracleLinux 7.5 x86_64

FQDN: racnode-dc2-2.internal.lab
IP's:
enp0s3: 10.0.2.15
enp0s8: 192.168.7.101
enp0s9: 172.16.7.11

Processor: Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz
#CPU's:    2
Memory:    5708 MB
Kernel:    4.1.12-112.16.4.el7uek.x86_64

----------------------------------------
[oracle@racnode-dc2-2 ~]$ ps -ef|grep pmon
oracle   16172     1  0 14:41 ?        00:00:00 asm_pmon_+ASM2
oracle   17229     1  0 14:41 ?        00:00:00 mdb_pmon_-MGMTDB
oracle   17368     1  0 14:41 ?        00:00:00 ora_pmon_hawk2
oracle   25901 25533  0 14:46 pts/0    00:00:00 grep --color=auto pmon
[oracle@racnode-dc2-2 ~]$

Check packer-ol75-disk001.vmdk from VirtualBox.xml:

dinh@CMWPHV1 MINGW64 ~/.VirtualBox
$ grep packer-ol75-disk001.vmdk VirtualBox.xml
        <HardDisk uuid="{b507fc35-1c3c-46ab-9e0e-91f192c5b935}" location="D:/VirtualBoxVM/vbox-rac-dc2/racnode-dc2-2/packer-ol75-disk001.vmdk" format="VMDK" type="Normal"/>
        <HardDisk uuid="{1e5667ab-69a7-49fc-aa76-7b6249aba862}" location="D:/VirtualBoxVM/vbox-rac-dc2/racnode-dc2-1/packer-ol75-disk001.vmdk" format="VMDK" type="Normal"/>

dinh@CMWPHV1 MINGW64 ~/.VirtualBox
$

Before and after:

dinh@CMWPHV1 MINGW64 /d/VirtualBoxVM/vbox-rac-dc2/racnode-dc2-2
$ ll
total 4566352
drwxr-xr-x 1 dinh 197121          0 Apr 30 20:04 Logs/
-rw-r--r-- 1 dinh 197121 4675928064 May  1 07:54 opacker-ol75-disk001.vmdk
-rw-r--r-- 1 dinh 197121       6433 May  1 07:54 racnode-dc2-2.vbox
-rw-r--r-- 1 dinh 197121       6433 Apr 30 20:04 racnode-dc2-2.vbox-prev

dinh@CMWPHV1 MINGW64 /d/VirtualBoxVM/vbox-rac-dc2/racnode-dc2-2
$ ll
total 4768592
drwxr-xr-x 1 dinh 197121          0 May  1 08:38 Logs/
-rw-r--r-- 1 dinh 197121 4883021824 May  1 08:43 packer-ol75-disk001.vmdk
-rw-r--r-- 1 dinh 197121       6433 May  1 08:38 racnode-dc2-2.vbox
-rw-r--r-- 1 dinh 197121       6433 May  1 08:38 racnode-dc2-2.vbox-prev

dinh@CMWPHV1 MINGW64 /d/VirtualBoxVM/vbox-rac-dc2/racnode-dc2-2
$

Create a free website or blog at WordPress.com.