Patching with OPLAN

From a time far, far way, I tweeted about Oracle Software Patching with OPLAN (Doc ID 1306814.1) and decided to give it a try.

First, you will need to configure X11 else error:
Can’t connect to X11 window server using ‘localhost:10.0’ as the value of the DISPLAY variable.

Second, you will need to using OPatch Version: 12.1.0.1.10, else error:
Caught exception: java.lang.ExceptionInInitializerError

If you like to see the results, then open and download Patch_Apply_Instructions_$PatchNumber.html from Google Drive

For some reason, opening does not work.

[grid@rac01:+ASM1:/home/grid]
$ /media/sf_Linux/patches/OPatch/oplan/oplan generateApplySteps /media/sf_Linux/patches/21744348/21523375/
from oplan /media/sf_Linux/patches/OPatch/oplan/../opatchauto-dir/opatchautocore/jlib/oracle.oplan.classpath.jar:/media/sf_Linux/patches/OPatch/oplan/../opatchauto-dir/opatchautocore/../opatchautodb/jlib/oplan_db.jar

Processing request...
Review the log messages captured in the following file: /u01/app/11.2.0.4/grid/cfgtoollogs/oplan/2015-12-19-10-32-11/log.txt
Success!

Follow the instructions outlined in the following Installation Instructions document and patch your system:

Apply Instructions (HTML) : /u01/app/11.2.0.4/grid/cfgtoollogs/oplan/2015-12-19-10-32-11/ApplyInstructions.html
Apply Instructions (TEXT) : /u01/app/11.2.0.4/grid/cfgtoollogs/oplan/2015-12-19-10-32-11/ApplyInstructions.txt

[grid@rac01:+ASM1:/u01/app/11.2.0.4/grid/cfgtoollogs/oplan]
$ cd /u01/app/11.2.0.4/grid/cfgtoollogs/oplan/2015-12-19-10-32-11/
[grid@rac01:+ASM1:/u01/app/11.2.0.4/grid/cfgtoollogs/oplan/2015-12-19-10-32-11]
$ ll
total 1284
-r--r-----. 1 grid oinstall 379154 Dec 19 10:32 ApplyInstructions.html
-r--r-----. 1 grid oinstall 8507 Dec 19 10:32 ApplyInstructions.txt
-r--r-----. 1 grid oinstall 9457 Dec 19 10:32 configuration.png
-r--r-----. 1 grid oinstall 42733 Dec 19 10:32 InplaceApplyNonRollingManual.txt
-r--r-----. 1 grid oinstall 36741 Dec 19 10:32 InplaceApplyRollingAuto.txt
-r--r-----. 1 grid oinstall 44548 Dec 19 10:32 InplaceApplyRollingManual.txt
-r--r-----. 1 grid oinstall 613286 Dec 19 10:32 log.txt
-r--r-----. 1 grid oinstall 0 Dec 19 10:32 log.txt.lck
dr-xr-x---. 3 grid oinstall 4096 Dec 19 10:32 machine-readable
-r--r-----. 1 grid oinstall 69478 Dec 19 10:32 OplaceApplyRolling.txt
-r--r-----. 1 grid oinstall 26608 Dec 19 10:32 OplaceSwitchbackRolling.txt
-r--r-----. 1 grid oinstall 353 Dec 19 10:32 README
-r--r-----. 1 grid oinstall 60991 Dec 19 10:32 README.html
[grid@rac01:+ASM1:/u01/app/11.2.0.4/grid/cfgtoollogs/oplan/2015-12-19-10-32-11]
$ 

Observation: OPLAN may need to be created after each patch is applied.

Combo of OJVM Component 11.2.0.4.5 DB PSU + GI PSU 11.2.0.4.8 (Oct2015) Patch contain the following patches:
Patch 21523375 – Database Grid Infrastructure Patch Set Update 11.2.0.4.8 (Oct2015) –> RAC-Rolling Installable
Patch 21555791 – Oracle JavaVM Component 11.2.0.4.5 Database PSU (OCT2015) –> Non RAC-Rolling Installable
Patch 19852360 – Oracle JavaVM Component 11.2.0.4.1 Database PSU – Generic JDBC Patch (OCT2014) –> RAC-Rolling Installable

Patch 19852360 is included as part of Patch 21555791 for the DATABASE.
Patch 19852360 instructions to apply patch for GRID is not available from OPLAN but is available from README.

Advertisement

Pet Peeve crs start/stop

When stopping crs, there are 50+ outputs and is Attempting really necessary?

Conversely, when starting crs, there are only 1 output and we all know the process has not completed since crsctl stat fails.

Wouldn’t it be nice if crsctl start provides some useful information as well and prompt the the all the processes are started?

What am I missing?

[root@rac01:/root]
# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac01'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac01'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rac01'
CRS-2673: Attempting to stop 'ora.emu.db' on 'rac01'
CRS-2673: Attempting to stop 'ora.dg_acfs.vg_acfs.acfs' on 'rac01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac01'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.rac01.vip' on 'rac01'
CRS-2677: Stop of 'ora.dg_acfs.vg_acfs.acfs' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.DG_ACFS.dg' on 'rac01'
CRS-2677: Stop of 'ora.registry.acfs' on 'rac01' succeeded
CRS-2677: Stop of 'ora.emu.db' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.DATA2.dg' on 'rac01'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac02'
CRS-2677: Stop of 'ora.rac01.vip' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.rac01.vip' on 'rac02'
CRS-2677: Stop of 'ora.DG_ACFS.dg' on 'rac01' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'rac02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac02'
CRS-2676: Start of 'ora.rac01.vip' on 'rac02' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac02' succeeded
CRS-2677: Stop of 'ora.DATA2.dg' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac01'
CRS-2677: Stop of 'ora.asm' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac01'
CRS-2677: Stop of 'ora.ons' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac01'
CRS-2677: Stop of 'ora.net1.network' on 'rac01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac01' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac01'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac01'
CRS-2673: Attempting to stop 'ora.asm' on 'rac01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac01'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac01'
CRS-2677: Stop of 'ora.ctssd' on 'rac01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac01' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac01' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac01' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac01'
CRS-2677: Stop of 'ora.cssd' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac01'
CRS-2677: Stop of 'ora.crf' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac01'
CRS-2677: Stop of 'ora.gipcd' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac01'
CRS-2677: Stop of 'ora.gpnpd' on 'rac01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac01' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@rac01:/root]
# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

[root@rac01:/root]
# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager

[root@rac01:/root]
# crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
[root@rac01:/root]
#

TFA installed as part of 21523375 (Oct2015 CPU)

+++ Patch 21523375 – Oracle Grid Infrastructure Patch Set Update 11.2.0.4.8 (Oct2015) (Includes Database PSU 11.2.0.4.8)
+++ Case 1: GI Home and the Database Homes that are not shared and ACFS file system is not configured.

After the completion of Patch 21523375, TFA was installed.

If you need to shutdown processes running from grid, TFA will need to be stopped as well (# /etc/init.d/init.tfa stop) since crsctl stop crs does not stop TFA.

I ***incorrectly*** tweeted that start crs starts TFA.

TFA was started from patch apply.

Interested to see what happens in the next patching cycle.

[root@rac02:/root]
# $ORACLE_HOME/OPatch/opatch auto /u01/app/grid/patches/21744348/21523375 -ocmrf /tmp/ocm.rsp
Executing /u01/app/11.2.0.4/grid/perl/bin/perl /u01/app/11.2.0.4/grid/OPatch/crs/patch11203.pl -patchdir /u01/app/grid/patches/21744348 -patchn 21523375 -ocmrf /tmp/ocm.rsp -paramfile /u01/app/11.2.0.4/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/11.2.0.4/grid/cfgtoollogs/opatchauto2015-12-18_08-18-46.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/11.2.0.4/grid/cfgtoollogs/opatchauto2015-12-18_08-18-46.report.log

2015-12-18 08:18:46: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/11.2.0.4/grid/crs/install/crsconfig_params

Stopping RAC /u01/app/oracle/product/11.2.0.4/db_1 ...
Stopped RAC /u01/app/oracle/product/11.2.0.4/db_1 successfully

patch /u01/app/grid/patches/21744348/21523375/21352635  apply successful for home  /u01/app/oracle/product/11.2.0.4/db_1
patch /u01/app/grid/patches/21744348/21523375/21352649/custom/server/21352649  apply successful for home  /u01/app/oracle/product/11.2.0.4/db_1

Stopping CRS...
Stopped CRS successfully

patch /u01/app/grid/patches/21744348/21523375/21352635  apply successful for home  /u01/app/11.2.0.4/grid
patch /u01/app/grid/patches/21744348/21523375/21352649  apply successful for home  /u01/app/11.2.0.4/grid
patch /u01/app/grid/patches/21744348/21523375/21352642  apply successful for home  /u01/app/11.2.0.4/grid

Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

Starting RAC /u01/app/oracle/product/11.2.0.4/db_1 ...
Started RAC /u01/app/oracle/product/11.2.0.4/db_1 successfully

opatch auto succeeded.
[root@rac02:/root]
#

[root@rac02:/root]
# ps -ef|grep tfa
root     11017     1  0 08:28 ?        00:00:00 /bin/sh /etc/init.d/init.tfa run
root     12980 12756  0 10:35 pts/0    00:00:00 grep tfa
root     26988     1  1 08:31 ?        00:01:33 /u01/app/11.2.0.4/grid/jdk/jre/bin/java -Xms128m -Xmx512m -classpath /u01/app/11.2.0.4/grid/tfa/rac02/tfa_home/jlib/RATFA.jar:/u01/app/11.2.0.4/grid/tfa/rac02/tfa_home/jlib/je-5.0.84.jar:/u01/app/11.2.0.4/grid/tfa/rac02/tfa_home/jlib/ojdbc5.jar:/u01/app/11.2.0.4/grid/tfa/rac02/tfa_home/jlib/commons-io-2.1.jar oracle.rat.tfa.TFAMain /u01/app/11.2.0.4/grid/tfa/rac02/tfa_home

[root@rac02:/root]
# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac02'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac02'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac02' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@rac02:/root]
# ps -ef|egrep 'oracle|grid|agent'
gdm       2968  2926  0 06:11 ?        00:00:00 /usr/libexec/polkit-gnome-authentication-agent-1
root     14811 12756  0 10:39 pts/0    00:00:00 egrep oracle|grid|agent
root     26988     1  1 08:31 ?        00:01:36 /u01/app/11.2.0.4/grid/jdk/jre/bin/java -Xms128m -Xmx512m -classpath /u01/app/11.2.0.4/grid/tfa/rac02/tfa_home/jlib/RATFA.jar:/u01/app/11.2.0.4/grid/tfa/rac02/tfa_home/jlib/je-5.0.84.jar:/u01/app/11.2.0.4/grid/tfa/rac02/tfa_home/jlib/ojdbc5.jar:/u01/app/11.2.0.4/grid/tfa/rac02/tfa_home/jlib/commons-io-2.1.jar oracle.rat.tfa.TFAMain /u01/app/11.2.0.4/grid/tfa/rac02/tfa_home

TFA needs to be stopped manually

[root@rac02:/root]
# /etc/init.d/init.tfa stop
Stopping TFA
TFA-00002 : Oracle Trace File Analyzer (TFA) is not running
TFAmain Force Stopped Successfully
Killing TFA running with pid 26988
. . .
Successfully stopped TFA..
[root@rac02:/root]
# ps -ef|egrep 'oracle|grid|agent'
gdm       2968  2926  0 06:11 ?        00:00:00 /usr/libexec/polkit-gnome-authentication-agent-1
root     15141 12756  0 10:40 pts/0    00:00:00 egrep oracle|grid|agent
[root@rac02:/root]
# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

[root@rac02:/root]
# ps -ef|grep -i tfa
root     11017     1  0 08:28 ?        00:00:00 /bin/sh /etc/init.d/init.tfa run
root     16558 12756  0 10:42 pts/0    00:00:00 grep -i tfa

TFA needs to be started manually

[root@rac02:/root]
# /etc/init.d/init.tfa start
Starting TFA..
start: Job is already running: oracle-tfa
Waiting up to 100 seconds for TFA to be started..
. . . . .
Successfully started TFA Process..
. . . . .
TFA Started and listening for commands
[root@rac02:/root]
# ps -ef|grep -i tfa
root     11017     1  0 08:28 ?        00:00:00 /bin/sh /etc/init.d/init.tfa run
root     16739     1 99 10:43 ?        00:00:13 /u01/app/11.2.0.4/grid/jdk/jre/bin/java -Xms128m -Xmx512m -classpath /u01/app/11.2.0.4/grid/tfa/rac02/tfa_home/jlib/RATFA.jar:/u01/app/11.2.0.4/grid/tfa/rac02/tfa_home/jlib/je-5.0.84.jar:/u01/app/11.2.0.4/grid/tfa/rac02/tfa_home/jlib/ojdbc5.jar:/u01/app/11.2.0.4/grid/tfa/rac02/tfa_home/jlib/commons-io-2.1.jar oracle.rat.tfa.TFAMain /u01/app/11.2.0.4/grid/tfa/rac02/tfa_home
root     16938 12756  0 10:43 pts/0    00:00:00 grep -i tfa
[root@rac02:/root]

TFA does not have a status and needs to be checked using ps -ef

[root@rac02:/root]
# /etc/init.d/init.tfa
Usage: /etc/init.d/init.tfa {stop|start|shutdown|restart}
[root@rac02:/root]
#

UPDATE: Dec 19 2015

[grid@rac01:+ASM1:/home/grid]
$ echo $ORACLE_BASE
/u01/app/grid
[grid@rac01:+ASM1:/home/grid]
$ cd $ORACLE_BASE

TFA directory beneath grid $ORACLE_BASE

[grid@rac01:+ASM1:/u01/app/grid]
$ ll
total 36
drwxrwxr-x. 4 grid   oinstall 4096 Nov 30  2014 cfgtoollogs
drwxrwxr-x. 2 grid   oinstall 4096 Nov 30  2014 checkpoints
drwxrwxr-x. 2 grid   oinstall 4096 Nov 30  2014 Clusterware
drwxrwxr-x. 4 grid   oinstall 4096 Nov 30  2014 diag
drwxrwxr-x. 3 oracle oinstall 4096 Dec  4  2014 oradiag_oracle
drwxrwxr-x. 3 root   root     4096 Dec  1  2014 oradiag_root
drwxrwxr-x. 3 grid   oinstall 4096 Nov 30  2014 rac01
drwxr-x--x. 4 root   root     4096 Nov 30  2014 tfa
drwxrwxr-x. 9 grid   oinstall 4096 Dec 18  2014 xag
[grid@rac01:+ASM1:/u01/app/grid]
$ cd tfa/
[grid@rac01:+ASM1:/u01/app/grid/tfa]
$ ll
ls: cannot open directory .: Permission denied
[grid@rac01:+ASM1:/u01/app/grid/tfa]