Thinking Out Loud

October 9, 2017

Upgrade 12.2 Journey – Failed GRID Upgrade from 11.2.0.4 to 12.2.0.1

Filed under: Upgrade 12.2 Journey — mdinh @ 12:05 am

Oracle Restart Upgrade Failed simply because CRS diskgroup is OFFLINE.

ora.CRS.dg
               OFFLINE OFFLINE      db-asm-1                 STABLE

Resolution coming up.

Upgrade Log

[oracle@db-asm-1 ~]$ crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.4.0]

[oracle@db-asm-1 ~]$ crsctl query has softwareversion
Oracle High Availability Services version on the local node is [11.2.0.4.0]
[oracle@db-asm-1 ~]$

00:58:42 SYS @ +ASM:>show parameter pfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +CRS/asm/asmparameterfile/regi
                                                 stry.253.956854045
00:58:47 SYS @ +ASM:>show parameter disk

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups                       string      DATA, FRA
asm_diskstring                       string      /dev/oracle/*
00:58:52 SYS @ +ASM:>

[oracle@db-asm-1 ~]$ srvctl config asm
ASM home: /u01/app/oracle/11.2.0.4/grid
ASM listener: LISTENER
Spfile: +CRS/asm/asmparameterfile/registry.253.956854045
ASM diskgroup discovery string: /dev/oracle/*

[oracle@db-asm-1 ~]$ asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  4194304     10236    10148                0           10148              0             N  CRS/
MOUNTED  EXTERN  N         512   4096  4194304     20472    18612                0           18612              0             N  DATA/
MOUNTED  EXTERN  N         512   4096  4194304     30708    30264                0           30264              0             N  FRA/
[oracle@db-asm-1 ~]$

[oracle@db-asm-1 ~]$ env|egrep -i 'oracle|home'
USER=oracle
LD_LIBRARY_PATH=/u01/app/oracle/11.2.0.4/grid/lib
ORACLE_SID=+ASM
ORACLE_BASE=/u01/app/oracle
MAIL=/var/spool/mail/oracle
PATH=/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/oracle/bin:/u01/app/oracle/11.2.0.4/grid/bin
PWD=/home/oracle
HOME=/home/oracle
GRID_HOME=/u01/app/oracle/11.2.0.4/grid
LOGNAME=oracle
LOG=/u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_ORCLCDB_APPLY_2017Oct08_21_41_02.log
ORACLE_HOME=/u01/app/oracle/11.2.0.4/grid
[oracle@db-asm-1 ~]$ unset ORACLE_SID ORACLE_BASE GRID_HOME ORACLE_HOME
[oracle@db-asm-1 ~]$ env|egrep -i 'oracle|home'
USER=oracle
LD_LIBRARY_PATH=/u01/app/oracle/11.2.0.4/grid/lib
MAIL=/var/spool/mail/oracle
PATH=/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/oracle/bin:/u01/app/oracle/11.2.0.4/grid/bin
PWD=/home/oracle
HOME=/home/oracle
LOGNAME=oracle
LOG=/u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_ORCLCDB_APPLY_2017Oct08_21_41_02.log
[oracle@db-asm-1 ~]$

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[oracle@db-asm-1 ~]$ cd /u01/app/12.2.0.1/grid/
[oracle@db-asm-1 grid]$ ./gridSetup.sh -silent -waitforcompletion -skipPrereqs -responseFile /u01/app/12.2.0.1/grid/install/response/grid_upgrade.rsp
Launching Oracle Grid Infrastructure Setup Wizard...

[WARNING] [INS-40436] Installer has detected an earlier version of Oracle Automatic Storage Management (ASM) Instance configured. Before proceeding with the upgrade you must shutdown all Oracle Databases that are using Oracle ASM for storage.

You may ignore this message and continue with the upgrade, if the database(s) are already in shutdown state.
[WARNING] [INS-41808] Possible invalid choice for OSASM Group.
   CAUSE: The name of the group you selected for the OSASM group is commonly used to grant other system privileges (For example: asmdba, asmoper, dba, oper).
   ACTION: Oracle recommends that you designate asmadmin as the OSASM group.
[WARNING] [INS-41809] Possible invalid choice for OSDBA Group.
   CAUSE: The group name you selected as the OSDBA for ASM group is commonly used for Oracle Database administrator privileges.
   ACTION: Oracle recommends that you designate asmdba as the OSDBA for ASM group, and that the group should not be the same group as an Oracle Database OSDBA group.
[WARNING] [INS-41810] Possible invalid choice for OSOPER Group.
   CAUSE: The group name you selected as the OSOPER for ASM group is commonly used for Oracle Database administrator privileges.
   ACTION: Oracle recommends that you designate asmoper as the OSOPER for ASM group, and that the group should not be the same group as an Oracle Database OSOPER group.
[WARNING] [INS-41812] OSDBA and OSASM are the same OS group.
   CAUSE: The chosen values for OSDBA group and the chosen value for OSASM group are the same.
   ACTION: Select an OS group that is unique for ASM administrators. The OSASM group should not be the same as the OS groups that grant privileges for Oracle ASM access, or for database administration.
[WARNING] [INS-32018] The selected Oracle home is outside of Oracle base.
   ACTION: Oracle recommends installing Oracle software within the Oracle base directory. Adjust the Oracle home or Oracle base accordingly.
You can find the log of this install session at:
 /u01/app/oraInventory/logs/GridSetupActions2017-10-09_01-03-02AM/gridSetupActions2017-10-09_01-03-02AM.log

As a root user, execute the following script(s):
        1. /u01/app/12.2.0.1/grid/rootupgrade.sh

Execute /u01/app/12.2.0.1/grid/rootupgrade.sh on the following nodes:
[db-asm-1]

Successfully Setup Software.
As install user, execute the following command to complete the configuration.
        /u01/app/12.2.0.1/grid/gridSetup.sh -executeConfigTools -responseFile /u01/app/12.2.0.1/grid/install/response/grid_upgrade.rsp [-silent]

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++        

[oracle@db-asm-1 grid]$ ps -ef|grep pmon
oracle    3333 26998  0 01:19 pts/2    00:00:00 grep pmon
oracle   14084     1  0 Oct08 ?        00:00:02 asm_pmon_+ASM
oracle   14346     1  0 Oct08 ?        00:00:02 ora_pmon_orclcdb
[oracle@db-asm-1 grid]$ asmcmd lsdg
Connected to an idle instance.
ASMCMD-8102: no connection to Oracle ASM; command requires Oracle ASM to run

[oracle@db-asm-1 grid]$ srvctl status asm
ASM is running on db-asm-1
[oracle@db-asm-1 grid]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
               ONLINE  ONLINE       db-asm-1
ora.DATA.dg
               ONLINE  ONLINE       db-asm-1
ora.FRA.dg
               ONLINE  ONLINE       db-asm-1
ora.LISTENER.lsnr
               ONLINE  ONLINE       db-asm-1
ora.asm
               ONLINE  ONLINE       db-asm-1                 Started
ora.ons
               OFFLINE OFFLINE      db-asm-1
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       db-asm-1
ora.diskmon
      1        OFFLINE OFFLINE
ora.evmd
      1        ONLINE  ONLINE       db-asm-1
ora.orclcdb.db
      1        ONLINE  ONLINE       db-asm-1                 Open

[oracle@db-asm-1 grid]$ env|egrep -i 'oracle|home'
OLDPWD=/home/oracle
USER=oracle
LD_LIBRARY_PATH=/u01/app/oracle/11.2.0.4/grid/lib
MAIL=/var/spool/mail/oracle
PATH=/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/oracle/bin:/u01/app/oracle/11.2.0.4/grid/bin
HOME=/home/oracle
LOGNAME=oracle
LOG=/u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_ORCLCDB_APPLY_2017Oct08_21_41_02.log
[oracle@db-asm-1 grid]$

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[oracle@db-asm-1 ~]$ su -
Password:
[root@db-asm-1 ~]# /u01/app/12.2.0.1/grid/rootupgrade.sh
Check /u01/app/12.2.0.1/grid/install/root_db-asm-1_2017-10-09_01-22-49-043808622.log for the output of root script

[root@db-asm-1 ~]# ps -ef|grep pmon
root      8648  3386  0 01:32 pts/1    00:00:00 grep pmon
[root@db-asm-1 ~]#

[oracle@db-asm-1 ~]$ cat /u01/app/12.2.0.1/grid/install/root_db-asm-1_2017-10-09_01-22-49-043808622.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.2.0.1/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.2.0.1/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/db-asm-1/crsconfig/roothas_2017-10-09_01-22-50AM.log
2017/10/09 01:23:00 CLSRSC-363: User ignored prerequisites during installation

ASM has been upgraded and started successfully.

Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node db-asm-1 successfully pinned.
2017/10/09 01:27:19 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'
CRS-4123: Oracle High Availability Services has been started.

2017/10/09 01:29:31 CLSRSC-482: Running command: 'srvctl upgrade model -s 11.2.0.4.0 -d 12.2.0.1.0 -p first'
2017/10/09 01:29:52 CLSRSC-482: Running command: 'srvctl upgrade model -s 11.2.0.4.0 -d 12.2.0.1.0 -p last'

db-asm-1     2017/10/09 01:29:57     /u01/app/12.2.0.1/grid/cdata/db-asm-1/backup_20171009_012957.olr     0

db-asm-1     2017/10/08 16:45:27     /u01/app/oracle/11.2.0.4/grid/cdata/db-asm-1/backup_20171008_164527.olr     -
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'db-asm-1'
CRS-2673: Attempting to stop 'ora.evmd' on 'db-asm-1'
CRS-2677: Stop of 'ora.evmd' on 'db-asm-1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'db-asm-1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/10/09 01:31:14 CLSRSC-327: Successfully configured Oracle Restart for a standalone server
[oracle@db-asm-1 ~]$

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[oracle@db-asm-1 grid]$ ps -ef|grep pmon
oracle    8652 26998  0 01:33 pts/2    00:00:00 grep pmon

[oracle@db-asm-1 grid]$ /u01/app/12.2.0.1/grid/gridSetup.sh -executeConfigTools -responseFile /u01/app/12.2.0.1/grid/install/response/grid_upgrade.rsp -silent
Launching Oracle Grid Infrastructure Setup Wizard...

You can find the logs of this session at:
/u01/app/oraInventory/logs/GridSetupActions2017-10-09_01-33-40AM

Successfully Configured Software.

[oracle@db-asm-1 grid]$ cd /u01/app/oraInventory/logs/GridSetupActions2017-10-09_01-33-40AM
[oracle@db-asm-1 GridSetupActions2017-10-09_01-33-40AM]$ ls -lrt
total 80
-rw-r-----. 1 oracle oinstall   129 Oct  9 01:33 installerPatchActions_2017-10-09_01-33-40AM.log
-rw-r-----. 1 oracle oinstall     0 Oct  9 01:33 gridSetupActions2017-10-09_01-33-40AM.err
-rw-r-----. 1 oracle oinstall   196 Oct  9 01:35 gridSetupActions2017-10-09_01-33-40AM.out
-rw-r-----. 1 oracle oinstall  2176 Oct  9 01:35 time2017-10-09_01-33-40AM.log
-rw-r-----. 1 oracle oinstall 67881 Oct  9 01:35 gridSetupActions2017-10-09_01-33-40AM.log

[oracle@db-asm-1 GridSetupActions2017-10-09_01-33-40AM]$ ps -ef|grep pmon
oracle    9335 26998  0 01:37 pts/2    00:00:00 grep pmon

[oracle@db-asm-1 GridSetupActions2017-10-09_01-33-40AM]$ tail /etc/oratab
# The first and second fields are the system identifier and home
# directory of the database respectively.  The third filed indicates
# to the dbstart utility that the database should , "Y", or should not,
# "N", be brought up at system boot time.
#
# Multiple entries with the same $ORACLE_SID are not allowed.
#
#
orclcdb:/u01/app/oracle/11.2.0.4/db1:N          # line added by Agent
+ASM:/u01/app/12.2.0.1/grid:N           # line added by Agent
[oracle@db-asm-1 GridSetupActions2017-10-

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[oracle@db-asm-1 ~]$ . oraenv <<< +ASM
ORACLE_SID = [oracle] ? The Oracle base has been set to /u01/app/oracle
[oracle@db-asm-1 ~]$ echo $ORACLE_HOME
/u01/app/12.2.0.1/grid
[oracle@db-asm-1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
               OFFLINE OFFLINE      db-asm-1                 STABLE
ora.DATA.dg
               ONLINE  OFFLINE      db-asm-1                 STABLE
ora.FRA.dg
               ONLINE  OFFLINE      db-asm-1                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       db-asm-1                 STABLE
ora.asm
               ONLINE  OFFLINE      db-asm-1                 STABLE
ora.ons
               OFFLINE OFFLINE      db-asm-1                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       db-asm-1                 STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       db-asm-1                 STABLE
ora.orclcdb.db
      1        ONLINE  OFFLINE                               Instance Shutdown,ST
                                                             ABLE
--------------------------------------------------------------------------------
[oracle@db-asm-1 ~]$ ps -ef|grep pmon
oracle    9360  4055  0 01:39 pts/3    00:00:00 grep pmon
[oracle@db-asm-1 ~]$ asmcmd lsdg
Connected to an idle instance.
ASMCMD-8102: no connection to Oracle ASM; command requires Oracle ASM to run

[oracle@db-asm-1 ~]$ crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [12.2.0.1.0]

[oracle@db-asm-1 ~]$ crsctl query has softwareversion
Oracle High Availability Services version on the local node is [12.2.0.1.0]
[oracle@db-asm-1 ~]$

Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: