Thinking Out Loud

November 12, 2017

opatchauto Hell

Filed under: 12c — mdinh @ 8:10 pm

I have been investigating new opatchauto apply for 12.1 RAC system.

Was overjoyed to see services relocated to instance2 when instance1 was being patched.

The joy became dismay to find relocated services on instance2 were stopped to perform datapatch.

I am definitely not an expert, but why shutdown services running on other instances?

What if services are manually relocated, would they still be shutdown?

From Oplan:

In-Place Patching: Apply Patch to Existing Oracle HomesIn this option, the patch will be applied directly to the specified Oracle Homes. 
This is the least time-consuming option. 
The procedures available for this option are outlined below.

    Rolling Mode: Apply Patch to Each Node In A Cluster Sequentially Resources running on each node in the cluster will be stopped, patched and then restarted, one at a time in sequence. 
      By patching only one node at a time, this approach ensures that multi-instance databases remain available during the patching process, but it will take longer to complete.

        Apply Patch In-Place using OPatch Auto Command in Rolling Mode:
            Advantages: No downtime while patching; fewer patching steps.
            Disadvantages: May be time consuming; less-than-optimal diagnosability of issues during patching; slower recovery from failures.
            Total number of steps required: 29
                During the full availability of services: 23
                During the partial availability of services: 6 

Step 2: Patch Apply Phase (Limited services will be available)

Instance orclcdb1 is running on node racnode-dc1-1 with online services DBA_TEST,testsvc11,testsvc12,testsvc13,testsvc14,testsvc15. Instance status: Open.

What does Oracle mean by limited? The services are shutdown and not restarted!

Update: services run on correct instance when restarted.

[oracle@racnode-dc1-1 ~]$ srvctl start service -d orclcdb
PRCD-1133 : Failed to start services for database
CRS-5702: Resource 'ora.orclcdb.testsvc26.svc' is already running on 'racnode-dc1-2'
CRS-5702: Resource 'ora.orclcdb.testsvc27.svc' is already running on 'racnode-dc1-2'
CRS-5702: Resource 'ora.orclcdb.testsvc28.svc' is already running on 'racnode-dc1-2'
CRS-5702: Resource 'ora.orclcdb.testsvc29.svc' is already running on 'racnode-dc1-2'

[oracle@racnode-dc1-1 ~]$ date; srvctl status database -d orclcdb -v
Sun Nov 12 21:17:52 CET 2017
Instance orclcdb1 is running on node racnode-dc1-1 with online services DBA_TEST,testsvc11,testsvc12,testsvc13,testsvc14,testsvc15. Instance status: Open.
Instance orclcdb2 is running on node racnode-dc1-2 with online services testsvc26,testsvc27,testsvc28,testsvc29. Instance status: Open.
[oracle@racnode-dc1-1 ~]$

[root@racnode-dc1-1 ~]# $GRID_HOME/OPatch/opatchauto apply /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880


OPatchauto session is initiated at Sun Nov 12 18:07:15 2017

System initialization log file is /u01/app/12.1.0.2/grid/cfgtoollogs/opatchautodb/systemconfig2017-11-12_06-08-14PM.log.

Session log file is /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/opatchauto2017-11-12_06-09-00PM.log
The id for this session is YKRN


Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.1.0.2/grid

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/12.1.0.2/db1
Patch applicability verified successfully on home /u01/app/oracle/12.1.0.2/db1

Patch applicability verified successfully on home /u01/app/12.1.0.2/grid


Verifying SQL patch applicability on home /u01/app/oracle/12.1.0.2/db1
SQL patch applicability verified successfully on home /u01/app/oracle/12.1.0.2/db1


Preparing to bring down database service on home /u01/app/oracle/12.1.0.2/db1
Successfully prepared home /u01/app/oracle/12.1.0.2/db1 to bring down database service


Bringing down CRS service on home /u01/app/12.1.0.2/grid
Prepatch operation log file location: /u01/app/12.1.0.2/grid/cfgtoollogs/crsconfig/crspatch_racnode-dc1-1_2017-11-12_06-20-39PM.log
CRS service brought down successfully on home /u01/app/12.1.0.2/grid


Performing prepatch operation on home /u01/app/oracle/12.1.0.2/db1
Perpatch operation completed successfully on home /u01/app/oracle/12.1.0.2/db1


Start applying binary patch on home /u01/app/oracle/12.1.0.2/db1
Binary patch applied successfully on home /u01/app/oracle/12.1.0.2/db1


Performing postpatch operation on home /u01/app/oracle/12.1.0.2/db1
Postpatch operation completed successfully on home /u01/app/oracle/12.1.0.2/db1


Start applying binary patch on home /u01/app/12.1.0.2/grid

Binary patch applied successfully on home /u01/app/12.1.0.2/grid


Starting CRS service on home /u01/app/12.1.0.2/grid
Postpatch operation log file location: /u01/app/12.1.0.2/grid/cfgtoollogs/crsconfig/crspatch_racnode-dc1-1_2017-11-12_07-20-43PM.log
CRS service started successfully on home /u01/app/12.1.0.2/grid


Preparing home /u01/app/oracle/12.1.0.2/db1 after database service restarted
Prepared home /u01/app/oracle/12.1.0.2/db1 successfully after database service restarted


Trying to apply SQL patch on home /u01/app/oracle/12.1.0.2/db1
SQL patch applied successfully on home /u01/app/oracle/12.1.0.2/db1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:racnode-dc1-1
RAC Home:/u01/app/oracle/12.1.0.2/db1
Summary:

==Following patches were SKIPPED:

Patch: /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880/21436941
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880/26392164
Reason: This patch is not applicable to this specified target type - "rac_database"


==Following patches were SUCCESSFULLY applied:

Patch: /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880/26392192
Log: /u01/app/oracle/12.1.0.2/db1/cfgtoollogs/opatchauto/core/opatch/opatch2017-11-12_18-22-11PM_1.log

Patch: /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880/26717470
Log: /u01/app/oracle/12.1.0.2/db1/cfgtoollogs/opatchauto/core/opatch/opatch2017-11-12_18-22-11PM_1.log


Host:racnode-dc1-1
CRS Home:/u01/app/12.1.0.2/grid
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880/21436941
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2017-11-12_18-46-41PM_1.log

Patch: /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880/26392164
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2017-11-12_18-46-41PM_1.log

Patch: /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880/26392192
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2017-11-12_18-46-41PM_1.log

Patch: /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880/26717470
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2017-11-12_18-46-41PM_1.log

OPatchauto session completed at Sun Nov 12 19:34:26 2017
Time taken to complete the session 87 minutes, 11 seconds
[root@racnode-dc1-1 ~]#

[root@racnode-dc1-1 opatchauto]# grep -A8 “Executing command as” opatchauto2017-11-12_06-09-00PM.log


Executing command as oracle:
 /u01/app/12.1.0.2/grid/OPatch/opatchauto  apply /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880 -oh /u01/app/12.1.0.2/grid -target_type cluster -binary -invPtrLoc /u01/app/12.1.0.2/grid/oraInst.loc -jre /u01/app/12.1.0.2/grid/OPatch/jre -persistresult /u01/app/12.1.0.2/grid/OPatch/auto/dbsessioninfo/sessionresult_analyze_racnode-dc1-1_crs.ser -analyze -online
2017-11-12 18:09:24,734 INFO  [53] com.oracle.glcm.patch.auto.db.integration.controller.action.OPatchAutoBinaryAction -
Executing command as oracle:
 /u01/app/oracle/12.1.0.2/db1/OPatch/opatchauto  apply /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880 -oh /u01/app/oracle/12.1.0.2/db1 -target_type rac_database -binary -invPtrLoc /u01/app/12.1.0.2/grid/oraInst.loc -jre /u01/app/12.1.0.2/grid/OPatch/jre -persistresult /u01/app/oracle/12.1.0.2/db1/OPatch/auto/dbsessioninfo/sessionresult_analyze_racnode-dc1-1_rac.ser -analyze -online
2017-11-12 18:12:45,263 INFO  [53] com.oracle.glcm.patch.auto.db.integration.controller.action.OPatchAutoBinaryAction - Opatchcore binary return code=0
2017-11-12 18:12:45,265 INFO  [53] com.oracle.glcm.patch.auto.db.integration.controller.action.OPatchAutoBinaryAction - Opatchcore binary output=Oracle Home : /u01/app/oracle/12.1.0.2/db1

OPatchAuto binary patching Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 13.9.3.0.0
--
Executing command as oracle:
 /bin/sh -c 'cd /u01/app/oracle/12.1.0.2/db1; ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 ORACLE_SID=orclcdb1 /u01/app/oracle/12.1.0.2/db1/OPatch/datapatch -prereq -verbose'
2017-11-12 18:19:52,146 INFO  [85] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'cd /u01/app/oracle/12.1.0.2/db1; ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 ORACLE_SID=orclcdb1 /u01/app/oracle/12.1.0.2/db1/OPatch/datapatch -prereq -verbose'
2017-11-12 18:20:20,415 INFO  [85] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 18:20:20,415 INFO  [85] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 18:20:20,415 INFO  [85] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 18:20:20,415 INFO  [85] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - SQL Patching tool version 12.1.0.2.0 Production on Sun Nov 12 18:19:52 2017
Copyright (c) 2012, 2017, Oracle.  All rights reserved.
--
Executing command as oracle:
 /bin/sh -c 'echo "DECLARE l_count NUMBER(4) :=0; BEGIN SELECT count(*) INTO l_count FROM CDB_PDBS; IF l_count >1 then EXECUTE IMMEDIATE '\''alter pluggable database ALL SAVE STATE'\''; END IF; commit; END;" > /tmp/OraDB12Home1_oracle_orclcdb1.sql'
2017-11-12 18:20:23,665 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'echo "DECLARE l_count NUMBER(4) :=0; BEGIN SELECT count(*) INTO l_count FROM CDB_PDBS; IF l_count >1 then EXECUTE IMMEDIATE '\''alter pluggable database ALL SAVE STATE'\''; END IF; commit; END;" > /tmp/OraDB12Home1_oracle_orclcdb1.sql'
2017-11-12 18:20:23,771 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 18:20:23,771 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 18:20:23,771 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 18:20:23,771 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
2017-11-12 18:20:23,771 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Command executed successfully.
--
Executing command as oracle:
 /bin/sh -c 'echo "/" >> /tmp/OraDB12Home1_oracle_orclcdb1.sql'
2017-11-12 18:20:23,772 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'echo "/" >> /tmp/OraDB12Home1_oracle_orclcdb1.sql'
2017-11-12 18:20:23,873 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 18:20:23,873 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 18:20:23,873 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 18:20:23,873 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
2017-11-12 18:20:23,873 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Command executed successfully.
--
Executing command as oracle:
 /bin/sh -c 'echo "EXIT;" >> /tmp/OraDB12Home1_oracle_orclcdb1.sql'
2017-11-12 18:20:23,873 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'echo "EXIT;" >> /tmp/OraDB12Home1_oracle_orclcdb1.sql'
2017-11-12 18:20:23,958 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 18:20:23,958 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 18:20:23,958 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 18:20:23,958 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
2017-11-12 18:20:23,958 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Command executed successfully.
--
Executing command as oracle:
 /bin/sh -c 'cd /u01/app/oracle/12.1.0.2/db1 ; ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 ORACLE_SID=orclcdb1 /u01/app/oracle/12.1.0.2/db1/bin/sqlplus / as sysdba @/tmp/OraDB12Home1_oracle_orclcdb1.sql'
2017-11-12 18:20:23,958 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'cd /u01/app/oracle/12.1.0.2/db1 ; ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 ORACLE_SID=orclcdb1 /u01/app/oracle/12.1.0.2/db1/bin/sqlplus / as sysdba @/tmp/OraDB12Home1_oracle_orclcdb1.sql'
2017-11-12 18:20:24,268 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 18:20:24,268 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 18:20:24,268 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 18:20:24,268 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
SQL*Plus: Release 12.1.0.2.0 Production on Sun Nov 12 18:20:24 2017
--

Executing command as oracle:
 /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl relocate service -d orclcdb -s DBA_TEST -i orclcdb1 -t orclcdb2'
2017-11-12 18:20:24,269 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl relocate service -d orclcdb -s DBA_TEST -i orclcdb1 -t orclcdb2'
2017-11-12 18:20:26,771 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 18:20:26,771 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 18:20:26,771 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 18:20:26,809 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
2017-11-12 18:20:26,809 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Command executed successfully.
--
Executing command as oracle:
 /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl relocate service -d orclcdb -s testsvc11 -i orclcdb1 -t orclcdb2'
2017-11-12 18:20:26,809 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl relocate service -d orclcdb -s testsvc11 -i orclcdb1 -t orclcdb2'
2017-11-12 18:20:28,931 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 18:20:28,932 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 18:20:28,932 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 18:20:28,932 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
2017-11-12 18:20:28,932 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Command executed successfully.
--
Executing command as oracle:
 /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl relocate service -d orclcdb -s testsvc12 -i orclcdb1 -t orclcdb2'
2017-11-12 18:20:28,932 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl relocate service -d orclcdb -s testsvc12 -i orclcdb1 -t orclcdb2'
2017-11-12 18:20:31,027 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 18:20:31,027 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 18:20:31,027 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 18:20:31,027 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
2017-11-12 18:20:31,027 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Command executed successfully.
--
Executing command as oracle:
 /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl relocate service -d orclcdb -s testsvc13 -i orclcdb1 -t orclcdb2'
2017-11-12 18:20:31,027 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl relocate service -d orclcdb -s testsvc13 -i orclcdb1 -t orclcdb2'
2017-11-12 18:20:33,346 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 18:20:33,346 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 18:20:33,346 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 18:20:33,346 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
2017-11-12 18:20:33,346 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Command executed successfully.
--
Executing command as oracle:
 /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl relocate service -d orclcdb -s testsvc14 -i orclcdb1 -t orclcdb2'
2017-11-12 18:20:33,346 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl relocate service -d orclcdb -s testsvc14 -i orclcdb1 -t orclcdb2'
2017-11-12 18:20:35,327 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 18:20:35,328 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 18:20:35,328 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 18:20:35,328 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
2017-11-12 18:20:35,328 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Command executed successfully.
--
Executing command as oracle:
 /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl relocate service -d orclcdb -s testsvc15 -i orclcdb1 -t orclcdb2'
2017-11-12 18:20:35,328 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl relocate service -d orclcdb -s testsvc15 -i orclcdb1 -t orclcdb2'
2017-11-12 18:20:37,885 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 18:20:37,885 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 18:20:37,885 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 18:20:37,885 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
2017-11-12 18:20:37,885 INFO  [125] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Command executed successfully.

--
Executing command as root:
 /u01/app/12.1.0.2/grid/perl/bin/perl -I/u01/app/12.1.0.2/grid/perl/lib -I/u01/app/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_racnode-dc1-1/patchwork/crs/install /u01/app/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_racnode-dc1-1/patchwork/crs/install/rootcrs.pl -prepatch
2017-11-12 18:20:38,384 INFO  [206] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /u01/app/12.1.0.2/grid/perl/bin/perl -I/u01/app/12.1.0.2/grid/perl/lib -I/u01/app/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_racnode-dc1-1/patchwork/crs/install /u01/app/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_racnode-dc1-1/patchwork/crs/install/rootcrs.pl -prepatch
2017-11-12 18:22:05,710 INFO  [206] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 18:22:05,711 INFO  [206] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 18:22:05,711 INFO  [206] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 18:22:05,711 INFO  [206] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Using configuration parameter file: /u01/app/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_racnode-dc1-1/patchwork/crs/install/crsconfig_params
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [0].
--
Executing command as oracle:
 /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880/26392192/custom/scripts/prepatch.sh -dbhome /u01/app/oracle/12.1.0.2/db1
2017-11-12 18:22:05,758 INFO  [224] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880/26392192/custom/scripts/prepatch.sh -dbhome /u01/app/oracle/12.1.0.2/db1
2017-11-12 18:22:05,793 INFO  [224] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 18:22:05,793 INFO  [224] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 18:22:05,793 INFO  [224] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 18:22:05,793 INFO  [224] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880/26392192/custom/scripts/prepatch.sh completed successfully.

--
Executing command as oracle:
 /u01/app/oracle/12.1.0.2/db1/OPatch/opatchauto  apply /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880 -oh /u01/app/oracle/12.1.0.2/db1 -target_type rac_database -binary -invPtrLoc /u01/app/12.1.0.2/grid/oraInst.loc -jre /u01/app/12.1.0.2/grid/OPatch/jre -persistresult /u01/app/oracle/12.1.0.2/db1/OPatch/auto/dbsessioninfo/sessionresult_racnode-dc1-1_rac.ser -analyzedresult /u01/app/oracle/12.1.0.2/db1/OPatch/auto/dbsessioninfo/sessionresult_analyze_racnode-dc1-1_rac.ser
2017-11-12 18:46:34,790 INFO  [238] com.oracle.glcm.patch.auto.db.integration.controller.action.OPatchAutoBinaryAction - Opatchcore binary return code=0
2017-11-12 18:46:34,801 INFO  [238] com.oracle.glcm.patch.auto.db.integration.controller.action.OPatchAutoBinaryAction - Opatchcore binary output=Oracle Home : /u01/app/oracle/12.1.0.2/db1

OPatchAuto binary patching Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 13.9.3.0.0
--
Executing command as oracle:
 /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880/26392192/custom/scripts/postpatch.sh -dbhome /u01/app/oracle/12.1.0.2/db1
2017-11-12 18:46:35,048 INFO  [281] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880/26392192/custom/scripts/postpatch.sh -dbhome /u01/app/oracle/12.1.0.2/db1
2017-11-12 18:46:35,337 INFO  [281] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 18:46:35,337 INFO  [281] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 18:46:35,337 INFO  [281] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 18:46:35,337 INFO  [281] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Reading /u01/app/oracle/12.1.0.2/db1/install/params.ora..
Reading /u01/app/oracle/12.1.0.2/db1/install/params.ora..
--
Executing command as oracle:
 /u01/app/12.1.0.2/grid/OPatch/opatchauto  apply /sf_OracleSoftware/12.1.0.2_PatchSet_Linux_x64/OCT2017PSU/26635880 -oh /u01/app/12.1.0.2/grid -target_type cluster -binary -invPtrLoc /u01/app/12.1.0.2/grid/oraInst.loc -jre /u01/app/12.1.0.2/grid/OPatch/jre -persistresult /u01/app/12.1.0.2/grid/OPatch/auto/dbsessioninfo/sessionresult_racnode-dc1-1_crs.ser -analyzedresult /u01/app/12.1.0.2/grid/OPatch/auto/dbsessioninfo/sessionresult_analyze_racnode-dc1-1_crs.ser
2017-11-12 19:20:43,366 INFO  [361] com.oracle.glcm.patch.auto.db.integration.controller.action.OPatchAutoBinaryAction - Opatchcore binary return code=0
2017-11-12 19:20:43,369 INFO  [361] com.oracle.glcm.patch.auto.db.integration.controller.action.OPatchAutoBinaryAction - Opatchcore binary output=Oracle Home : /u01/app/12.1.0.2/grid

OPatchAuto binary patching Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 13.9.3.0.0
--
Executing command as root:
 /u01/app/12.1.0.2/grid/rdbms/install/rootadd_rdbms.sh
2017-11-12 19:20:43,522 INFO  [414] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /u01/app/12.1.0.2/grid/rdbms/install/rootadd_rdbms.sh
2017-11-12 19:20:43,574 INFO  [414] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 19:20:43,574 INFO  [414] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 19:20:43,574 INFO  [414] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 19:20:43,574 INFO  [414] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
2017-11-12 19:20:43,574 INFO  [414] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Command executed successfully.
--
Executing command as root:
 /u01/app/12.1.0.2/grid/perl/bin/perl -I/u01/app/12.1.0.2/grid/perl/lib -I/u01/app/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_racnode-dc1-1/patchwork/crs/install /u01/app/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_racnode-dc1-1/patchwork/crs/install/rootcrs.pl -postpatch
2017-11-12 19:20:43,574 INFO  [414] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /u01/app/12.1.0.2/grid/perl/bin/perl -I/u01/app/12.1.0.2/grid/perl/lib -I/u01/app/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_racnode-dc1-1/patchwork/crs/install /u01/app/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_racnode-dc1-1/patchwork/crs/install/rootcrs.pl -postpatch
2017-11-12 19:33:19,090 INFO  [414] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 19:33:19,097 INFO  [414] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 19:33:19,098 INFO  [414] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 19:33:19,098 INFO  [414] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Using configuration parameter file: /u01/app/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_racnode-dc1-1/patchwork/crs/install/crsconfig_params
2017/11/12 19:20:51 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
--

Executing command as oracle:
 /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl stop service -d orclcdb -s DBA_TEST -i orclcdb2'
2017-11-12 19:33:19,393 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl stop service -d orclcdb -s DBA_TEST -i orclcdb2'
2017-11-12 19:33:20,976 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 19:33:20,977 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 19:33:20,980 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 19:33:20,980 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
2017-11-12 19:33:20,980 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Command executed successfully.
--
Executing command as oracle:
 /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl stop service -d orclcdb -s testsvc11 -i orclcdb2'
2017-11-12 19:33:20,980 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl stop service -d orclcdb -s testsvc11 -i orclcdb2'
2017-11-12 19:33:22,262 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 19:33:22,262 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 19:33:22,262 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 19:33:22,263 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
2017-11-12 19:33:22,263 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Command executed successfully.
--
Executing command as oracle:
 /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl stop service -d orclcdb -s testsvc12 -i orclcdb2'
2017-11-12 19:33:22,263 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl stop service -d orclcdb -s testsvc12 -i orclcdb2'
2017-11-12 19:33:23,539 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 19:33:23,539 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 19:33:23,539 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 19:33:23,539 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
2017-11-12 19:33:23,540 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Command executed successfully.
--
Executing command as oracle:
 /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl stop service -d orclcdb -s testsvc13 -i orclcdb2'
2017-11-12 19:33:23,540 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl stop service -d orclcdb -s testsvc13 -i orclcdb2'
2017-11-12 19:33:24,764 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 19:33:24,764 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 19:33:24,764 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 19:33:24,764 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
2017-11-12 19:33:24,764 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Command executed successfully.
--
Executing command as oracle:
 /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl stop service -d orclcdb -s testsvc14 -i orclcdb2'
2017-11-12 19:33:24,764 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl stop service -d orclcdb -s testsvc14 -i orclcdb2'
2017-11-12 19:33:26,003 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 19:33:26,003 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 19:33:26,003 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 19:33:26,003 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
2017-11-12 19:33:26,003 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Command executed successfully.
--
Executing command as oracle:
 /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl stop service -d orclcdb -s testsvc15 -i orclcdb2'
2017-11-12 19:33:26,004 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 /u01/app/oracle/12.1.0.2/db1/bin/srvctl stop service -d orclcdb -s testsvc15 -i orclcdb2'
2017-11-12 19:33:27,227 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 19:33:27,228 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 19:33:27,228 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 19:33:27,228 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor -
2017-11-12 19:33:27,228 INFO  [454] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Command executed successfully.

--
Executing command as oracle:
 /bin/sh -c 'cd /u01/app/oracle/12.1.0.2/db1;ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 ORACLE_SID=orclcdb1 /u01/app/oracle/12.1.0.2/db1/OPatch/datapatch -verbose'
2017-11-12 19:33:28,659 INFO  [524] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'cd /u01/app/oracle/12.1.0.2/db1;ORACLE_HOME=/u01/app/oracle/12.1.0.2/db1 ORACLE_SID=orclcdb1 /u01/app/oracle/12.1.0.2/db1/OPatch/datapatch -verbose'
2017-11-12 19:34:26,362 INFO  [524] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 0
2017-11-12 19:34:26,362 INFO  [524] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Successfully executed the above command.

2017-11-12 19:34:26,362 INFO  [524] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - Output from the command:
2017-11-12 19:34:26,362 INFO  [524] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - SQL Patching tool version 12.1.0.2.0 Production on Sun Nov 12 19:33:29 2017
Copyright (c) 2012, 2017, Oracle.  All rights reserved.
[root@racnode-dc1-1 opatchauto]#
Advertisements

November 5, 2017

Relocate Services Back To Instance Before Patching

Filed under: 12c,RAC — mdinh @ 1:16 pm

This will only work for 2-nodes RAC!

Prerequisite:
Patching starts at instance1, services failover to instance2.
Patching completed at instance1, restart instance1.
Patching starts at instance2, services failover to instance1.
Patching completed at instance2, restart instance2.
All services are now running at instance1.
Relocate services from instance2 back to where it belongs.

Save existing service configuration before patching.
[oracle@racnode-dc1-2 rac_relocate]$ ./save_service.sh

 

+ srvctl status database -d orclcdb -v
+ srvctl status database -d orclcdb -v
+ awk '-F ' '{print $2}'
+ cat /tmp/services.conf
Instance orclcdb1 is running on node racnode-dc1-1 with online services DBA_TEST,testsvc11,testsvc12,testsvc13,testsvc14,testsvc15. Instance status: Open.
Instance orclcdb2 is running on node racnode-dc1-2 with online services testsvc26,testsvc27,testsvc28,testsvc29. Instance status: Open.
+ cat /tmp/instance.conf
orclcdb1
orclcdb2
++ tail -1 /tmp/services.conf
++ awk '-F ' '{print $11}'
++ awk '{$0=substr($0,1,length($0)-1); print $0}'
+ svc=testsvc26,testsvc27,testsvc28,testsvc29
+ exit
[oracle@racnode-dc1-2 rac_relocate]$

 

Patching completed at instance1 and starting at instance2.
All services are running on instance1 after failover of instance2.

 

[oracle@racnode-dc1-2 rac_relocate]$ srvctl stop instance -db orclcdb -instance orclcdb2 -failover
[oracle@racnode-dc1-2 rac_relocate]$ srvctl status database -d orclcdb -v
Instance orclcdb1 is running on node racnode-dc1-1 with online services DBA_TEST,testsvc11,testsvc12,testsvc13,testsvc14,testsvc15,testsvc26,testsvc27,testsvc28,testsvc29. Instance status: Open.
Instance orclcdb2 is not running on node racnode-dc1-2
[oracle@racnode-dc1-2 rac_relocate]$

 

Patching completed at instance2, start instance2, all services running from instance1.

[oracle@racnode-dc1-2 rac_relocate]$ srvctl start instance -db orclcdb -instance orclcdb2
[oracle@racnode-dc1-2 rac_relocate]$ srvctl status database -d orclcdb -v
Instance orclcdb1 is running on node racnode-dc1-1 with online services DBA_TEST,testsvc11,testsvc12,testsvc13,testsvc14,testsvc15,testsvc26,testsvc27,testsvc28,testsvc29. Instance status: Open.
Instance orclcdb2 is running on node racnode-dc1-2. Instance status: Open.
[oracle@racnode-dc1-2 rac_relocate]$

Verify relocate services will work as intended by testing first – print but not execute commands.

[oracle@racnode-dc1-2 rac_relocate]$ ./test_relocate.sh
================================================================================
++++++ Saved Configuration
-rw-r--r-- 1 oracle oinstall  18 Nov  5 13:01 /tmp/instance.conf
-rw-r--r-- 1 oracle oinstall 291 Nov  5 13:01 /tmp/services.conf
Instance orclcdb1 is running on node racnode-dc1-1 with online services DBA_TEST,testsvc11,testsvc12,testsvc13,testsvc14,testsvc15. Instance status: Open.
Instance orclcdb2 is running on node racnode-dc1-2 with online services testsvc26,testsvc27,testsvc28,testsvc29. Instance status: Open.
orclcdb1
orclcdb2
================================================================================
++++++ Relocate Configuration
newinst=orclcdb2
oldinst=orclcdb1
svc=testsvc26,testsvc27,testsvc28,testsvc29
================================================================================
Instance orclcdb1 is running on node racnode-dc1-1 with online services DBA_TEST,testsvc11,testsvc12,testsvc13,testsvc14,testsvc15,testsvc26,testsvc27,testsvc28,testsvc29. Instance status: Open.
Instance orclcdb2 is running on node racnode-dc1-2. Instance status: Open.
srvctl relocate service -db orclcdb -service testsvc26 -oldinst orclcdb1 -newinst orclcdb2
srvctl relocate service -db orclcdb -service testsvc27 -oldinst orclcdb1 -newinst orclcdb2
srvctl relocate service -db orclcdb -service testsvc28 -oldinst orclcdb1 -newinst orclcdb2
srvctl relocate service -db orclcdb -service testsvc29 -oldinst orclcdb1 -newinst orclcdb2
[oracle@racnode-dc1-2 rac_relocate]$

Relocate services to orginal saved configuration.

[oracle@racnode-dc1-2 rac_relocate]$ ./relocate_service.sh
================================================================================
++++++ Saved Configuration
-rw-r--r-- 1 oracle oinstall  18 Nov  5 13:01 /tmp/instance.conf
-rw-r--r-- 1 oracle oinstall 291 Nov  5 13:01 /tmp/services.conf
Instance orclcdb1 is running on node racnode-dc1-1 with online services DBA_TEST,testsvc11,testsvc12,testsvc13,testsvc14,testsvc15. Instance status: Open.
Instance orclcdb2 is running on node racnode-dc1-2 with online services testsvc26,testsvc27,testsvc28,testsvc29. Instance status: Open.
orclcdb1
orclcdb2
================================================================================
++++++ Relocate Configuration
newinst=orclcdb2
oldinst=orclcdb1
svc=testsvc26,testsvc27,testsvc28,testsvc29
================================================================================
+ srvctl status database -d orclcdb -v
Instance orclcdb1 is running on node racnode-dc1-1 with online services DBA_TEST,testsvc11,testsvc12,testsvc13,testsvc14,testsvc15,testsvc26,testsvc27,testsvc28,testsvc29. Instance status: Open.
Instance orclcdb2 is running on node racnode-dc1-2. Instance status: Open.
+ IFS=,
+ for s in '${svc}'
+ srvctl relocate service -db orclcdb -service testsvc26 -oldinst orclcdb1 -newinst orclcdb2
+ for s in '${svc}'
+ srvctl relocate service -db orclcdb -service testsvc27 -oldinst orclcdb1 -newinst orclcdb2
+ for s in '${svc}'
+ srvctl relocate service -db orclcdb -service testsvc28 -oldinst orclcdb1 -newinst orclcdb2
+ for s in '${svc}'
+ srvctl relocate service -db orclcdb -service testsvc29 -oldinst orclcdb1 -newinst orclcdb2
+ srvctl status database -d orclcdb -v
Instance orclcdb1 is running on node racnode-dc1-1 with online services DBA_TEST,testsvc11,testsvc12,testsvc13,testsvc14,testsvc15. Instance status: Open.
Instance orclcdb2 is running on node racnode-dc1-2 with online services testsvc26,testsvc27,testsvc28,testsvc29. Instance status: Open.
+ exit
[oracle@racnode-dc1-2 rac_relocate]$

I have rant about hardcoding before.
YES! I hardcoded conf file location to provide a permanent and consistent location for all environments.

I don’t like to have to dig through code for find such information.
ex:
SCRIPT_DIR=/u01/app/oracle/scripts
LOG_DIR=$SCRIPT_DIR/log

save_service.sh


#!/bin/sh -x
srvctl status database -d ${db} -v > /tmp/services.conf
srvctl status database -d ${db} -v|awk -F" " '{print $2}' > /tmp/instance.conf
cat /tmp/services.conf
cat /tmp/instance.conf
svc=`tail -1 /tmp/services.conf | awk -F" " '{print $11}'|awk '{$0=substr($0,1,length($0)-1); print $0}'`
exit

 

test_relocate.sh


#!/bin/sh
echo "================================================================================"
echo "++++++ Saved Configuration"
ls -l /tmp/*.conf
cat /tmp/services.conf
cat /tmp/instance.conf
echo "================================================================================"
echo "++++++ Relocate Configuration"
export svc=`tail -1 /tmp/services.conf | awk -F" " '{print $11}'|awk '{$0=substr($0,1,length($0)-1); print $0}'`
export oldinst=`head -1 /tmp/instance.conf`
export newinst=`tail -1 /tmp/instance.conf`
env|egrep 'svc|inst'|sort
echo "================================================================================"
srvctl status database -d ${db} -v
IFS=","
for s in ${svc}
do
echo "srvctl relocate service -db ${db} -service ${s} -oldinst ${oldinst} -newinst ${newinst}"
done
exit

 

relocate_service.sh


#!/bin/sh
echo "================================================================================"
echo "++++++ Saved Configuration"
ls -l /tmp/*.conf
cat /tmp/services.conf
cat /tmp/instance.conf
echo "================================================================================"
echo "++++++ Relocate Configuration"
export svc=`tail -1 /tmp/services.conf | awk -F" " '{print $11}'|awk '{$0=substr($0,1,length($0)-1); print $0}'`
export oldinst=`head -1 /tmp/instance.conf`
export newinst=`tail -1 /tmp/instance.conf`
env|egrep 'svc|inst'|sort
echo "================================================================================"
set -x
srvctl status database -d ${db} -v
IFS=","
for s in ${svc}
do
srvctl relocate service -db ${db} -service ${s} -oldinst ${oldinst} -newinst ${newinst}
done
srvctl status database -d ${db} -v
exit

 

12.1 Improved Service Failover

Filed under: 12c,RAC — mdinh @ 12:51 pm

11gR2 Database Services and Instance Shutdown

The thought of having to manually relocate dozens of services was not very appealing.

As it turns out, there is no need to manually relocate services.

srvctl stop instance -db orclcdb -instance orclcdb1 -failover will do the trick.

Comparing the 2 commands, 12c is a lot clearer / cleaner.

12c:
srvctl add service -db orclcdb -service DBA_TEST -preferred orclcdb1 -available orclcdb2 -failovertype SELECT -tafpolicy BASIC

11g:
srvctl add service -d orclcdb -s DBA_TEST -P BASIC -e SELECT -r orclcdb1 -a orclcdb2

DEMO:

$ srvctl config service -d orclcdb -s DBA_TEST|egrep -i 'Service name|Preferred instances|Available instances|failover'

Service name: DBA_TEST
Failover type: SELECT
Failover method:
TAF failover retries:
TAF failover delay:
Preferred instances: orclcdb1
Available instances: orclcdb2

$ srvctl status database -d orclcdb

Instance orclcdb1 is running on node racnode-dc1-1
Instance orclcdb2 is running on node racnode-dc1-2

$ sqlplus mdinh/mdinh@dbatest @t.sql

SQL*Plus: Release 12.1.0.2.0 Production on Sun Nov 5 04:17:56 2017

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Last Successful login time: Sun Nov 05 2017 04:15:29 +01:00

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options


   INST_ID STARTUP_TIME
---------- -----------------------------
         1 05-NOV-2017 04:12:55
         2 05-NOV-2017 04:14:49


   INST_ID FAILOVER_TYPE FAILOVER_M FAI
---------- ------------- ---------- ---
         1 NONE          NONE       NO
         1 SELECT        BASIC      NO
         2 NONE          NONE       NO

04:17:57 MDINH @ dbatest:>host
[oracle@racnode-dc1-1 ~]$ srvctl stop instance -db orclcdb -instance orclcdb1 -failover;date
Sun Nov 5 04:18:34 CET 2017
[oracle@racnode-dc1-1 ~]$ exit
exit

04:18:37 MDINH @ dbatest:>@t.sql

   INST_ID STARTUP_TIME
---------- -----------------------------
         2 05-NOV-2017 04:14:49


   INST_ID FAILOVER_TYPE FAILOVER_M FAI
---------- ------------- ---------- ---
         2 SELECT        BASIC      YES

04:18:40 MDINH @ dbatest:>

October 27, 2017

KUP-04095: preprocessor command qopiprep.bat

Filed under: 12c — mdinh @ 12:46 pm

WARNING: Please open SR to confirm this is applicable to your environment as there are too many errors which look similar.

Applying Patch 26635880 – Database Proactive Bundle Patch 12.1.0.2.171017

# $GRID_HOME/OPatch/opatchauto apply $PATCHDIR/26635880

Starting database service on home /u01/oracle/db12c
Database service successfully started on home /u01/oracle/db12c

Preparing home /u01/oracle/db12c after database service restarted
No step execution required.........
Prepared home /u01/oracle/db12c successfully after database service restarted

Trying to apply SQL patch on home /u01/oracle/db12c
"/bin/sh -c 'ORACLE_HOME=/u01/oracle/db12c ORACLE_SID=db01
/u01/oracle/db12c/OPatch/datapatch'" command failed with errors. 
Please refer to logs for more details. SQL changes, if any, can be applied by manually retrying the same command.

SQL patch applied successfully on home /u01/oracle/db12c

Verifying patches applied on home /u01/oracle/gi12c
Patch verification completed with warning on home /u01/oracle/gi12c

Verifying patches applied on home /u01/oracle/db12c
Patch verification completed with warning on home /u01/oracle/db12c

OPatchAuto successful.

--------------------------------Summary--------------------------------
Output omitted for brevity.

This is the only results – no other errors.
SQL> select dbms_sqlpatch.verify_queryable_inventory from dual;

VERIFY_QUERYABLE_INVENTORY
--------------------------------------------------------------------------------
ORA-20001: Latest xml inventory is not loaded into table

SQL> select * from OPATCH_XML_INV;

ERROR:
ORA-29913: error in executing ODCIEXTTABLEFETCH callout
ORA-29400: data cartridge error
KUP-04095: preprocessor command /u01/oracle/db12c/QOpatch/qopiprep.bat
encountered error "Argument(s) Error... Cannot use file
"/u01/oracle/db12c/QOpatch/xml_file.xml" to generate XML output.
Specify path/filename and make sure filename doesn't"



no rows selected

SQL> host ls -l /u01/oracle/db12c/QOpatch/*
-rw-r--r--    1 oracle   dba          128227 Oct 24 16:24 /u01/oracle/db12c/QOpatch/qopatch_log.log
-r-xr-xr--    1 oracle   dba            1372 Apr 07 2013  /u01/oracle/db12c/QOpatch/qopiprep.bat
-rw-r--r--    1 oracle   dba            2349 Oct 24 16:24 /u01/oracle/db12c/QOpatch/stout.txt
-rw-r--r--    1 oracle   dba         2066473 Oct 24 16:24 /u01/oracle/db12c/QOpatch/xml_file.xml

SQL>

Oracle support provided solution as shown below:
Patch 20244108: QOPIPREP.BAT MODIFIES XML INVENTORY WHILE READING IT
Last Updated 03-Apr-2015 04:11 (2+ years ago)

What’s really FUBAR – patch is from 2+ years ago, so why wouldn’t it be included in BP?

October 24, 2017

DB 12.2 NF for Goldengate

Filed under: 12.2,GoldenGate — mdinh @ 12:30 pm

Logical Replication

Error Handling Improvements for Oracle GoldenGate Integrated Replicat
Database views can manage and display details about Oracle GoldenGate conflict resolution methods, 
DBA_APPLY_REPERROR_HANDLERS 
DBA_APPLY_HANDLECOLLISIONS
DBA_APPLY_DML_CONF_HANDLERS
DBA_APPLY_DML_CONF_COLUMNS

A new capture parameter, ANNOTATE_DDL_SIZE, 
is available for Oracle GoldenGate integrated capture to specify the size of the buffer that describes the DDL more fully.

October 10, 2017

Cloning Goldengate Integrated Capture and DB

Filed under: 12c,GoldenGate — mdinh @ 10:10 pm

Using DBMS_STREAMS_ADM To Cleanup GoldenGate

Let’s say you want to clone DB and Goldengate implementation from PROD to DEV, then you need to drop the capture that was registered with PROD database.

This is what happens when dependencies are introduced / created.

select capture_name from dba_capture;
exec DBMS_CAPTURE_ADM.DROP_CAPTURE ('&capture');

September 30, 2017

Mining Goldgate ggserr.log

Filed under: 12c,GoldenGate — mdinh @ 2:33 am

Can you imagine me running in circles shouting, “The sky is falling, the sky is falling?”

Replicat Lag at Chkpt: 03:21:45

Here are the trail files at target – look at how fast it is being created.

ls -alrt ./dirdat/aa*|tail -20
-rw-r----- 1 ggsuser ggsuser 499999845 Sep 29 11:09 ./dirdat/aa000020827
-rw-r----- 1 ggsuser ggsuser 499999575 Sep 29 11:09 ./dirdat/aa000020828
-rw-r----- 1 ggsuser ggsuser 499999929 Sep 29 11:10 ./dirdat/aa000020829
-rw-r----- 1 ggsuser ggsuser 499999771 Sep 29 11:11 ./dirdat/aa000020830
-rw-r----- 1 ggsuser ggsuser 499999941 Sep 29 11:11 ./dirdat/aa000020831
-rw-r----- 1 ggsuser ggsuser 499999858 Sep 29 11:12 ./dirdat/aa000020832
-rw-r----- 1 ggsuser ggsuser 499999571 Sep 29 11:12 ./dirdat/aa000020833
-rw-r----- 1 ggsuser ggsuser 499999874 Sep 29 11:13 ./dirdat/aa000020834
-rw-r----- 1 ggsuser ggsuser 499999782 Sep 29 11:14 ./dirdat/aa000020835
-rw-r----- 1 ggsuser ggsuser 499999975 Sep 29 11:14 ./dirdat/aa000020836

My hypothesis: lots of data being capture at source.

After all is said and done. The ggserr.log was mined.

Gather dates for the 10 highest number of trails created by day in 2017.

SOURCE:

grep "^2017" ggserr.log.dinh|grep "p_test.prm:  Rolling over remote file"|awk '{ print $1 }'|uniq -c|sort -nrk 1|head

    224 2017-09-29
    147 2017-06-02
    105 2017-02-28
    101 2017-05-31
    100 2017-03-01
     98 2017-06-01
     97 2017-05-18
     91 2017-05-26
     89 2017-07-25
     85 2017-01-26

TARGET:


grep "^2017" ggserr.log.dinh|grep "r_test.prm:  Switching to next trail file"|awk '{ print $1 }'|uniq -c|sort -nrk 1|head

    279 2017-09-29
    183 2017-02-28
    174 2017-03-01
    148 2017-06-02
    146 2017-02-24
    137 2017-08-29
    137 2017-03-24
    133 2017-08-11
    130 2017-08-16
    128 2017-03-02

Different count between source/target for 2017-09-29 is due to data being collected at different time.

September 29, 2017

Scheduler Jobs Do Not Run Automatically

Filed under: 12c,oracle — mdinh @ 2:09 am

After you have followed – IF: Jobs Do Not Run Automatically (Doc ID 2084527.1) – without any success,
then check to see if services have been created and are running.

RAC DB is 12.1.0.2.0 and was cloned from standby.

It just so happens, service as defined from the SQL below was not created:

select c.SERVICE
from dba_scheduler_jobs j, dba_scheduler_job_classes c
where j.JOB_CLASS=c.JOB_CLASS_NAME
and j.JOB_NAME=UPPER('&jobname')
;

To be honest, I was not able to find the issue and team mate did.

What I found very, very strange is manually running the job using exec dbms_scheduler.run_job is successful.

The manual job ran successfully without the service created and on the wrong node for where the service is defined
(the service is defined to run on node 2, while the manual run is from node 1).

Another unsolved mystery.

September 23, 2017

Grid Infrastructure 12.2.0.1 CRS_SWONLY Silent Install

Filed under: 12.2,Grid Infrastructure,oracle — mdinh @ 11:00 pm

Configuring Software Binaries for Oracle Grid Infrastructure for a Standalone Server

CREATE DIRECTORIES AND oraInst.loc:

[root@arrow1 ~]# id oracle
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),994(vboxsf)

[root@arrow1 ~]# mkdir /u01
[root@arrow1 ~]# chmod 775 /u01/
[root@arrow1 ~]# chown oracle:oinstall /u01/

[root@arrow1 ~]# vi /etc/oraInst.loc
[root@arrow1 ~]# chown oracle:oinstall /etc/oraInst.loc

[root@arrow1 ~]# cat /etc/oraInst.loc
inventory_loc=/u01/app/oraInventory
inst_group=oinstall

[root@arrow1 ~]# ll /etc/oraInst.loc
-rw-r--r-- 1 oracle oinstall 56 Sep 23 17:11 /etc/oraInst.loc
[root@arrow1 ~]# exit

[oracle@arrow1 ~]$ mkdir -p /u01/app/12.2.0.1/grid
[oracle@arrow1 ~]$ mkdir -p /u01/app/grid
[oracle@arrow1 ~]$ mkdir -p /u01/app/oraInventory

EXTRACT SOFTWARE DIRECTLY TO GRID HOME:

[oracle@arrow1 ~]$ ls -l /sf_OracleSoftware/12.2.0.1_Linux_64/
total 6297251
-rwxrwxrwx 1 vagrant vagrant 3453696911 Apr 12 09:44 linuxx64_12201_database.zip
-rwxrwxrwx 1 vagrant vagrant 2994687209 Apr 12 09:43 linuxx64_12201_grid_home.zip

[oracle@arrow1 ~]$ unzip -d /u01/app/12.2.0.1/grid -qo /sf_OracleSoftware/12.2.0.1_Linux_64/linuxx64_12201_grid_home.zip; echo $?
0

[oracle@arrow1 ~]$ ls /u01/app/12.2.0.1/grid
addnode     crs     deinstall    gpnp           inventory  lib      opmn     oui      qos       rootupgrade.sh  srvm      welcome.html
assistants  css     demo         gridSetup.sh   javavm     log      oracore  owm      racg      runcluvfy.sh    suptools  wlm
bin         cv      diagnostics  has            jdbc       md       ord      perl     rdbms     scheduler       tomcat    wwg
cdata       dbjava  dmu          hs             jdk        network  ordim    plsql    relnotes  slax            ucp       xag
cha         dbs     env.ora      install        jlib       nls      ords     precomp  rhp       sqlpatch        usm       xdk
clone       dc_ocm  evm          instantclient  ldap       OPatch   oss      QOpatch  root.sh   sqlplus         utl

[oracle@arrow1 ~]$ ls /u01/app/12.2.0.1/grid/install/response/
grid_2017-01-26_04-10-28PM.rsp  gridsetup.rsp  sample.ccf
[oracle@arrow1 ~]$

CREATE RESPONSE FILE FOR CRS_SWONLY – WITHOUT ASM:

CRS_SWONLY : To register Grid Infrastructure Software home (can be configured for cluster or stand alone server later)

oracle@arrow1::/u01/app/12.2.0.1/grid/install/response
$ cp -v gridsetup.rsp grid_crs_swonly.rsp
‘gridsetup.rsp’ -> ‘grid_crs_swonly.rsp’

oracle@arrow1::/u01/app/12.2.0.1/grid/install/response
$ diff -iwyB --suppress-common-lines -W 150 gridsetup.rsp grid_crs_swonly.rsp; echo
oracle.install.option=                                                    |     oracle.install.option=CRS_SWONLY
ORACLE_BASE=                                                              |     ORACLE_BASE=/u01/app/grid
oracle.install.asm.OSDBA=                                                 |     oracle.install.asm.OSDBA=dba
oracle.install.asm.OSOPER=                                                |     oracle.install.asm.OSOPER=dba
oracle.install.asm.OSASM=                                                 |     oracle.install.asm.OSASM=dba
oracle@arrow1::/u01/app/12.2.0.1/grid/install/response

RUNCLUVFY FOR HACFG:

oracle@arrow1::/u01/app/12.2.0.1/grid
$ ./runcluvfy.sh stage -pre hacfg

Verifying Physical Memory ...FAILED (PRVF-7530)
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...PASSED
Verifying Free Space: arrow1:/usr,arrow1:/var,arrow1:/etc,arrow1:/sbin,arrow1:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 54321 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying OS Kernel Parameter: panic_on_oops ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED

Pre-check for Oracle Restart configuration was unsuccessful.


Failures were encountered during execution of CVU verification request "stage -pre hacfg".

Verifying Physical Memory ...FAILED
arrow1: PRVF-7530 : Sufficient physical memory is not available on node
        "arrow1" [Required physical memory = 8GB (8388608.0KB)]


CVU operation performed:      stage -pre hacfg
Date:                         Sep 23, 2017 5:22:21 PM
CVU home:                     /u01/app/12.2.0.1/grid/
User:                         oracle
oracle@arrow1::/u01/app/12.2.0.1/grid
$

INSTALL GRID USING gridSetup.sh

oracle@arrow1::/u01/app/12.2.0.1/grid
$ ./gridSetup.sh -silent -waitforcompletion -skipPrereqs -responseFile /u01/app/12.2.0.1/grid/install/response/grid_crs_swonly.rsp
Launching Oracle Grid Infrastructure Setup Wizard...

[WARNING] [INS-41808] Possible invalid choice for OSASM Group.
   CAUSE: The name of the group you selected for the OSASM group is commonly used to grant other system privileges (For example: asmdba, asmoper, dba, oper).
   ACTION: Oracle recommends that you designate asmadmin as the OSASM group.
[WARNING] [INS-41809] Possible invalid choice for OSDBA Group.
   CAUSE: The group name you selected as the OSDBA for ASM group is commonly used for Oracle Database administrator privileges.
   ACTION: Oracle recommends that you designate asmdba as the OSDBA for ASM group, and that the group should not be the same group as an Oracle Database OSDBA group.
[WARNING] [INS-41810] Possible invalid choice for OSOPER Group.
   CAUSE: The group name you selected as the OSOPER for ASM group is commonly used for Oracle Database administrator privileges.
   ACTION: Oracle recommends that you designate asmoper as the OSOPER for ASM group, and that the group should not be the same group as an Oracle Database OSOPER group.
[WARNING] [INS-41813] OSDBA for ASM, OSOPER for ASM, and OSASM are the same OS group.
   CAUSE: The group you selected for granting the OSDBA for ASM group for database access, and the OSOPER for ASM group for startup and shutdown of Oracle ASM, is the same group as the OSASM group, whose members have SYSASM privileges on Oracle ASM.
   ACTION: Choose different groups as the OSASM, OSDBA for ASM, and OSOPER for ASM groups.
You can find the log of this install session at:
 /u01/app/oraInventory/logs/GridSetupActions2017-09-23_05-52-14PM/gridSetupActions2017-09-23_05-52-14PM.log

As a root user, execute the following script(s):
        1. /u01/app/12.2.0.1/grid/root.sh

Execute /u01/app/12.2.0.1/grid/root.sh on the following nodes:
[arrow1]


Successfully Setup Software.
oracle@arrow1::/u01/app/12.2.0.1/grid
$

RUN root.sh

[root@arrow1 ~]# /u01/app/12.2.0.1/grid/root.sh
Check /u01/app/12.2.0.1/grid/install/root_arrow1_2017-09-23_17-54-24-226719765.log for the output of root script

[root@arrow1 ~]# cat /u01/app/12.2.0.1/grid/install/root_arrow1_2017-09-23_17-54-24-226719765.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.2.0.1/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

To configure Grid Infrastructure for a Cluster or Grid Infrastructure for a Stand-Alone Server execute the following command as oracle user:
/u01/app/12.2.0.1/grid/gridSetup.sh
This command launches the Grid Infrastructure Setup Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.

[root@arrow1 ~]#

This is where Oracle has take a turn for the worse.
Instructions are not provided after running root.sh and are incorrect!

AS ROOT RUN roothas.pl

[root@arrow1 ~]# /u01/app/12.2.0.1/grid/perl/bin/perl -I /u01/app/12.2.0.1/grid/perl/lib -I /u01/app/12.2.0.1/grid/crs/install /u01/app/12.2.0.1/grid/crs/install/roothas.pl
Using configuration parameter file: /u01/app/12.2.0.1/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/arrow1/crsconfig/roothas_2017-09-23_06-11-34PM.log
2017/09/23 18:11:35 CLSRSC-363: User ignored prerequisites during installation
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node arrow1 successfully pinned.
2017/09/23 18:11:44 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'arrow1'
CRS-2673: Attempting to stop 'ora.evmd' on 'arrow1'
CRS-2677: Stop of 'ora.evmd' on 'arrow1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'arrow1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.

arrow1     2017/09/23 18:12:54     /u01/app/12.2.0.1/grid/cdata/arrow1/backup_20170923_181254.olr     0
2017/09/23 18:12:54 CLSRSC-327: Successfully configured Oracle Restart for a standalone server
[root@arrow1 ~]#

AS ORACLE runInstaller -updateNodeList

oracle@arrow1::/u01/app/12.2.0.1/grid
$ cd oui/bin/

oracle@arrow1::/u01/app/12.2.0.1/grid/oui/bin
$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/12.2.0.1/grid -defaultHomeName CLUSTER_NODES= CRS=TRUE
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4080 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.
oracle@arrow1::/u01/app/12.2.0.1/grid/oui/bin

CHECK PROCESESS

oracle@arrow1::/u01/app/12.2.0.1/grid/oui/bin
$ ps -ef|grep oracle
root      3796  1125  0 17:03 ?        00:00:00 sshd: oracle [priv]
oracle    3798  3796  0 17:03 ?        00:00:01 sshd: oracle@pts/2
oracle    3799  3798  0 17:03 pts/2    00:00:00 -bash
root     13572  1125  0 17:39 ?        00:00:00 sshd: oracle [priv]
oracle   13574 13572  0 17:39 ?        00:00:00 sshd: oracle@pts/0
oracle   13575 13574  0 17:39 pts/0    00:00:00 -bash
oracle   25084     1  0 18:12 ?        00:00:03 /u01/app/12.2.0.1/grid/bin/ohasd.bin reboot
oracle   25197     1  0 18:12 ?        00:00:02 /u01/app/12.2.0.1/grid/bin/oraagent.bin
oracle   25215     1  0 18:12 ?        00:00:01 /u01/app/12.2.0.1/grid/bin/evmd.bin
oracle   25269 25215  0 18:12 ?        00:00:01 /u01/app/12.2.0.1/grid/bin/evmlogger.bin -o /u01/app/12.2.0.1/grid/log/[HOSTNAME]/evmd/evmlogger.info -l /u01/app/12.2.0.1/grid/log/[HOSTNAME]/evmd/evmlogger.log
oracle   25404  3799  0 18:27 pts/2    00:00:00 ps -ef
oracle   25405  3799  0 18:27 pts/2    00:00:00 grep --color=auto oracle
oracle@arrow1::/u01/app/12.2.0.1/grid/oui/bin
$

CHECK INSTALL LOG

oracle@arrow1::/u01/app/12.2.0.1/grid/oui/bin
$ export LOG=/u01/app/oraInventory/logs/GridSetupActions2017-09-23_05-52-14PM/gridSetupActions2017-09-23_05-52-14PM.log

oracle@arrow1::/u01/app/12.2.0.1/grid/oui/bin
$ grep -e '[[:upper:]]: ' $LOG |cut -d ":" -f1|sort -u
   ACTION
   CAUSE
INFO
WARNING

oracle@arrow1::/u01/app/12.2.0.1/grid/oui/bin
$ grep -e '[[:upper:]]: ' $LOG|grep "^WARNING: "
WARNING:  [Sep 23, 2017 5:52:18 PM] Unable to find the namespace URI. Reason: Start of root element expected.
WARNING:  [Sep 23, 2017 5:52:18 PM] Unable to find the namespace URI. Reason: Start of root element expected.
WARNING:  [Sep 23, 2017 5:52:21 PM] Validation disabled for the state init
WARNING:  [Sep 23, 2017 5:52:26 PM] Validation disabled for the state flowDecider
WARNING:  [Sep 23, 2017 5:52:26 PM] Validation disabled for the state CRSNodeInformationUI
WARNING:  [Sep 23, 2017 5:52:26 PM] No Local Domain found, setting to whatever first domain is available
WARNING:  [Sep 23, 2017 5:52:26 PM] No Domain found, skipping Same Domain validation
WARNING:  [Sep 23, 2017 5:52:27 PM] [WARNING] [INS-41808] Possible invalid choice for OSASM Group.
WARNING:  [Sep 23, 2017 5:52:27 PM] [WARNING] [INS-41809] Possible invalid choice for OSDBA Group.
WARNING:  [Sep 23, 2017 5:52:27 PM] [WARNING] [INS-41810] Possible invalid choice for OSOPER Group.
WARNING:  [Sep 23, 2017 5:52:27 PM] [WARNING] [INS-41813] OSDBA for ASM, OSOPER for ASM, and OSASM are the same OS group.
WARNING:  [Sep 23, 2017 5:52:29 PM] Validation disabled for the state prereqExecutionDecider
WARNING:  [Sep 23, 2017 5:52:29 PM] Validation disabled for the state showSummary
WARNING:  [Sep 23, 2017 5:52:29 PM] Unable to find the namespace URI. Reason: /u01/app/12.2.0.1/grid/install/response/grid_2017-09-23_05-52-14PM.rsp (No such file or directory)
WARNING:  [Sep 23, 2017 5:53:50 PM] Validation disabled for the state finish
WARNING:  [Sep 23, 2017 5:53:50 PM] Could not create directory: /u01/app/oraInventory/logs/GridSetupActions2017-09-23_05-52-14PM
oracle@arrow1::/u01/app/12.2.0.1/grid/oui/bin
$

Bad Practice from Oracle – datapatch

Filed under: 12c,oracle — mdinh @ 1:18 pm

Not too long ago, I discovered:
dbca does hard coding

Now it gets even worse.

Create TEST environment.

SOURCE ORACLE_HOME:/u01/app/oracle/product/12.1.0/db_1
TARGET ORACLE_HOME:/u01/app/oracle/product/12.1.0/db

DB was cloned from Source to Target.

datapatch failed due to different OH and hard coding from dba_directories as shown below.

SOURCE:
OPATCH_LOG_DIR /u01/app/oracle/product/12.1.0/db_1/QOpatch
OPATCH_SCRIPT_DIR /u01/app/oracle/product/12.1.0/db_1/QOpatch
OPATCH_INST_DIR /u01/app/oracle/product/12.1.0/db_1/OPatch

TARGET:
OPATCH_LOG_DIR /u01/app/oracle/product/12.1.0/db/QOpatch
OPATCH_SCRIPT_DIR /u01/app/oracle/product/12.1.0/db/QOpatch
OPATCH_INST_DIR /u01/app/oracle/product/12.1.0/db/OPatch

I don’t get it! Why not use relative versus absolute path?

You might ask, “Why is the target different!”

To be honest, in this specific case, it should not have been.

BUT there may be scenarios where OH is already used and new location, i.e. db_2, is required.

Looking forward to Oracle AI – sarcasm.

 

Next Page »

Blog at WordPress.com.