Thinking Out Loud

July 31, 2020

I Will Buy You Lunch To Solve dbua -silent [FATAL] [DBT-20061]

Filed under: 19c,upgrade — mdinh @ 7:22 pm

Currently upgrading 11.2 to 19.3 database using dbua silent.

Run preupgrade.jar and no issues detected.

Run preupgrade_fixups.sql

Run dbua -silent error resulted with [FATAL] [DBT-20061]

Per Doc ID 2246770.1, ignore the error which may be hazardous.

Actually, I have already figured our the root cause; however, the method I used may not always be available for all environments.

Will post solution in a week.

HINT: probable cause is in the output which did not occur for first run.

$ env|grep HOME
OLD_HOME=/u01/app/oracle/product/11.2.0.4/dbhome_1
NEW_HOME=/u01/app/oracle/product/19.3.0.0/dbhome_1
HOME=/home/oracle
ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome_1

$ $OLD_HOME/jdk/bin/java -jar $NEW_HOME/rdbms/admin/preupgrade.jar TERMINAL TEXT FILE
==================
PREUPGRADE SUMMARY
==================
  /u01/app/oracle/cfgtoollogs/hawk_stby/preupgrade/preupgrade.log
  /u01/app/oracle/cfgtoollogs/hawk_stby/preupgrade/preupgrade_fixups.sql
  /u01/app/oracle/cfgtoollogs/hawk_stby/preupgrade/postupgrade_fixups.sql

Execute fixup scripts as indicated below:

Before upgrade:

Log into the database and execute the preupgrade fixups
@/u01/app/oracle/cfgtoollogs/hawk_stby/preupgrade/preupgrade_fixups.sql

After the upgrade:

Log into the database and execute the postupgrade fixups
@/u01/app/oracle/cfgtoollogs/hawk_stby/preupgrade/postupgrade_fixups.sql

Preupgrade complete: 2020-07-31T18:50:38
oracle@ol7-112-dg2:hawk:/home/oracle
$
	
*******************************************************

$ sqlplus / as sysdba @/u01/app/oracle/cfgtoollogs/hawk_stby/preupgrade/preupgrade_fixups.sql

SQL*Plus: Release 11.2.0.4.0 Production on Fri Jul 31 18:52:21 2020

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Executing Oracle PRE-Upgrade Fixup Script

Auto-Generated by:       Oracle Preupgrade Script
                         Version: 19.0.0.0.0 Build: 7
Generated on:            2020-07-31 18:50:36

For Source Database:     HAWK
Source Database Version: 11.2.0.4.0
For Upgrade to Version:  19.0.0.0.0

Preup                             Preupgrade
Action                            Issue Is
Number  Preupgrade Check Name     Remedied    Further DBA Action
------  ------------------------  ----------  --------------------------------
    1.  min_recovery_area_size    NO          Manual fixup required.
    2.  parameter_min_val         NO          Manual fixup recommended.
    3.  em_present                NO          Manual fixup recommended.
    4.  amd_exists                NO          Manual fixup recommended.
    5.  apex_manual_upgrade       NO          Manual fixup recommended.
    6.  tablespaces_info          NO          Informational only.
                                              Further action is optional.
    7.  exf_rul_exists            NO          Informational only.
                                              Further action is optional.
    8.  rman_recovery_version     NO          Informational only.
                                              Further action is optional.

The fixup scripts have been run and resolved what they can. However,
there are still issues originally identified by the preupgrade that
have not been remedied and are still present in the database.
Depending on the severity of the specific issue, and the nature of
the issue itself, that could mean that your database is not ready
for upgrade.  To resolve the outstanding issues, start by reviewing
the preupgrade_fixups.sql and searching it for the name of
the failed CHECK NAME or Preupgrade Action Number listed above.
There you will find the original corresponding diagnostic message
from the preupgrade which explains in more detail what still needs
to be done.
OL7-112-DG2:(SYS@hawk_stby:LOGICAL STANDBY>
	
*******************************************************
	
$ $NEW_HOME/bin/dbua -silent \
> -sid $ORACLE_SID \
> -oracleHome $ORACLE_HOME \
> -recompile_invalid_objects TRUE \
> -upgradeTimezone TRUE \
> -emConfiguration NONE \
> -upgrade_parallelism 2
Logs directory:  /u01/app/oracle/cfgtoollogs/dbua/upgrade2020-07-31_06-53-41PM
Performing Pre-Upgrade Checks...
============================
PRE- and POST- FIXUP ACTIONS
=============================
/u01/app/oracle/cfgtoollogs/dbua/upgrade2020-07-31_06-53-41PM/hawk/upgrade.xml
/u01/app/oracle/cfgtoollogs/dbua/upgrade2020-07-31_06-53-41PM/hawk/preupgrade_fixups.sql
/u01/app/oracle/cfgtoollogs/dbua/upgrade2020-07-31_06-53-41PM/hawk/postupgrade_fixups.sql
[FATAL] [DBT-20061] One or more of the pre-upgrade checks on the database have resulted into error conditions that must be addressed before proceeding.
   ACTION: Refer to the pre-upgrade results location for details: /u01/app/oracle/cfgtoollogs/dbua/upgrade2020-07-31_06-53-41PM/hawk

*******************************************************

$ ls -l /u01/app/oracle/cfgtoollogs/dbua/upgrade2020-07-31_06-53-41PM/hawk
total 692
-rw-r-----. 1 oracle oinstall      1 Jul 31 18:53 checksBuffer.tmp
-rw-r-----. 1 oracle oinstall  41134 Jul 31 18:53 components.properties
-rw-r-----. 1 oracle oinstall  15085 Jul 31 18:53 dbms_registry_extended.sql
drwxr-x---. 3 oracle oinstall     21 Jul 31 18:53 oracle
-rw-r-----. 1 oracle oinstall  14051 Jul 31 18:53 parameters.properties
-rw-r-----. 1 oracle oinstall   8584 Jul 31 18:53 postupgrade_fixups.sql
-rw-r-----. 1 oracle oinstall   7884 Jul 31 18:53 preupgrade_driver.sql
-rw-r-----. 1 oracle oinstall  13082 Jul 31 18:53 preupgrade_fixups.sql
-rw-r-----. 1 oracle oinstall 101702 Jul 31 18:53 preupgrade_messages.properties
-rw-r-----. 1 oracle oinstall 461660 Jul 31 18:53 preupgrade_package.sql
-rw-r-----. 1 oracle oinstall   5215 Jul 31 18:53 sqls.log
drwxr-x---. 3 oracle oinstall     24 Jul 31 18:53 upgrade
-rw-r-----. 1 oracle oinstall  15082 Jul 31 18:53 upgrade.xml

*******************************************************

DBUA Silent : Director Service Registration Error DBT-20060 DBT-08010 (Doc ID 2246770.1)	

Case 2 :
SEVERE: [FATAL] [DBT-20061] One or more of the pre-upgrade checks on the database have resulted into error conditions that must be addressed before proceeding.

./dbua -silent -sid <SID> -oracleHome $ORACLE_HOME -J-Doracle.install.dbua.validate.all=false

July 29, 2020

The Meaning Of Preupgrade Check Name

Filed under: 19c,upgrade — mdinh @ 7:32 pm

Upgrading 11.2.4 database with GoldenGate installed (don’t step on booby trap) to 19.3.

Does anyone know what the meaning of “Preupgrade Check Name” and wouldn’t be great of Oracle does not make you dig for it?

Please comment if you know the meaning of “Preupgrade Check Name”.

I will update post a week later with the info.

Run preupgrade_fixups.sql

oracle@db-fs-1:hawk:/home/oracle
$ sqlplus / as sysdba @/u01/app/oracle/cfgtoollogs/hawk/preupgrade/preupgrade_fixups.sql

SQL*Plus: Release 11.2.0.4.0 Production on Wed Jul 29 16:55:45 2020

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Executing Oracle PRE-Upgrade Fixup Script

Auto-Generated by:       Oracle Preupgrade Script
                         Version: 19.0.0.0.0 Build: 7
Generated on:            2020-07-29 16:54:11

For Source Database:     HAWK
Source Database Version: 11.2.0.4.0
For Upgrade to Version:  19.0.0.0.0

Preup                             Preupgrade
Action                            Issue Is
Number  Preupgrade Check Name     Remedied    Further DBA Action
------  ------------------------  ----------  --------------------------------
    1.  parameter_min_val         NO          Manual fixup recommended.
    2.  em_present                NO          Manual fixup recommended.
    3.  amd_exists                NO          Manual fixup recommended.
    4.  apex_manual_upgrade       NO          Manual fixup recommended.
    5.  streams_setup             NO          Manual fixup recommended.
    6.  trgowner_no_admndbtrg     YES         None.
    7.  pre_fixed_objects         YES         None.
    8.  tablespaces_info          NO          Informational only.
                                              Further action is optional.
    9.  exf_rul_exists            NO          Informational only.
                                              Further action is optional.
   10.  rman_recovery_version     NO          Informational only.
                                              Further action is optional.

The fixup scripts have been run and resolved what they can. However,
there are still issues originally identified by the preupgrade that
have not been remedied and are still present in the database.
Depending on the severity of the specific issue, and the nature of
the issue itself, that could mean that your database is not ready
for upgrade.  To resolve the outstanding issues, start by reviewing
the preupgrade_fixups.sql and searching it for the name of
the failed CHECK NAME or Preupgrade Action Number listed above.
There you will find the original corresponding diagnostic message
from the preupgrade which explains in more detail what still needs
to be done.

PL/SQL procedure successfully completed.

16:56:18 SYS @ hawk:>exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
oracle@db-fs-1:hawk:/home/oracle
$

Run postupgrade_fixups.sql


oracle@db-fs-1:hawk:/home/oracle
$ sqlplus / as sysdba @ /u01/app/oracle/cfgtoollogs/dbua/upgrade2020-07-29_07-50-43PM/hawk/postupgrade_fixups.sql

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jul 29 21:16:52 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0


Session altered.


PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


Package created.

No errors.

Package body created.


PL/SQL procedure successfully completed.

No errors.





Package created.

No errors.

Package body created.

No errors.
Executing Oracle POST-Upgrade Fixup Script

Auto-Generated by:       Oracle Preupgrade Script
                         Version: 19.0.0.0.0 Build: 7
Generated on:            2020-07-29 19:50:53

For Source Database:     HAWK
Source Database Version: 11.2.0.4.0
For Upgrade to Version:  19.0.0.0.0

Preup                             Preupgrade
Action                            Issue Is
Number  Preupgrade Check Name     Remedied    Further DBA Action
------  ------------------------  ----------  --------------------------------
    9.  old_time_zones_exist      YES         None.
   10.  dir_symlinks              YES         None.
   11.  post_dictionary           YES         None.
   12.  post_fixed_objects        NO          Informational only.
                                              Further action is optional.
   13.  upg_by_std_upgrd          NO          Informational only.
                                              Further action is optional.

The fixup scripts have been run and resolved what they can. However,
there are still issues originally identified by the preupgrade that
have not been remedied and are still present in the database.
Depending on the severity of the specific issue, and the nature of
the issue itself, that could mean that your database upgrade is not
fully complete.  To resolve the outstanding issues, start by reviewing
the postupgrade_fixups.sql and searching it for the name of
the failed CHECK NAME or Preupgrade Action Number listed above.
There you will find the original corresponding diagnostic message
from the preupgrade which explains in more detail what still needs
to be done.

PL/SQL procedure successfully completed.


Session altered.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
oracle@db-fs-1:hawk:/home/oracle
$

Let’s search for trgowner_no_admndbtrg – no luck!

oracle@db-fs-1:hawk:/u01/app/oracle/cfgtoollogs/hawk/preupgrade
$ grep trgowner_no_admndbtrg *.*
preupgrade_fixups.sql:    --    CHECK/FIXUP name: trgowner_no_admndbtrg
preupgrade_fixups.sql:    fixup_result := dbms_preup.run_fixup('trgowner_no_admndbtrg',6) AND fixup_result;
preupgrade_messages.properties:#    bspeckha    04/05/18 - update trgowner_no_admndbtrg_check action
preupgrade_package.sql:Rem    hvieyra     01/11/18 - Bug 27344799: Add fixup for check trgowner_no_admndbtrg_check.
preupgrade_package.sql:    FUNCTION trgowner_no_admndbtrg_check    (result_txt OUT CLOB) RETURN NUMBER;
preupgrade_package.sql:    FUNCTION trgowner_no_admndbtrg_fixup    (result_txt IN OUT VARCHAR2, pSqlcode IN OUT NUMBER) RETURN number;
preupgrade_package.sql:FUNCTION trgowner_no_admndbtrg_check (result_txt OUT CLOB) RETURN NUMBER
preupgrade_package.sql:        result_txt := get_failed_check_xml('trgowner_no_admndbtrg',
preupgrade_package.sql:END trgowner_no_admndbtrg_check;
preupgrade_package.sql:FUNCTION trgowner_no_admndbtrg_fixup (
preupgrade_package.sql:END trgowner_no_admndbtrg_fixup;
oracle@db-fs-1:hawk:/u01/app/oracle/cfgtoollogs/hawk/preupgrade
$

You can find more info for Check Name Database Preupgrade tool check list. (Doc ID 2380601.1)
For the benefit of doubt, info may be in preupgrade.log and may be grep -i ?

Example:

TRGOWNER_NO_ADMNDBTRG

The creation of database triggers must be done by users granted with ADMINISTER DATABASE TRIGGER privilege. Privilege must have been granted directly.

Directly grant ADMINISTER DATABASE TRIGGER privilege to the owner of the trigger or drop and re-create the trigger with a user that was granted directly with such. 
You can list those triggers using: 
SELECT OWNER, TRIGGER_NAME FROM DBA_TRIGGERS WHERE TRIM(BASE_OBJECT_TYPE)='DATABASE' AND OWNER NOT IN (SELECT GRANTEE FROM DBA_SYS_PRIVS WHERE PRIVILEGE='ADMINISTER DATABASE TRIGGER').

July 15, 2020

Create 19c RAC Standby Using RMAN

Filed under: 19c,Dataguard,RAC — mdinh @ 11:55 pm

See RAC_19c_rman_duplicate_standby_same_sid.log

Confirmed!

*** Oracle Data Guard Broker and Static Service Registration (Doc ID 1387859.1)
Note: Static “_DGMGRL” entries are no longer needed as of Oracle Database 12.1.0.2 in Oracle Data Guard Broker configurations
that are managed by Oracle Restart, RAC One Node or RAC as the Broker will use the clusterware to restart an instance.

July 11, 2020

Rename ASM DiskGroup 19c

Filed under: 19c,ASM — mdinh @ 4:17 pm

IMPORTANT: NO DB has been created !!!

Why is rename ASM DG?
1. Test RAC RMAN duplicate for standby similar to client environment.
2. Just because.

Rename Diskgroup name by renamedg command doesn't update the OCR (Doc ID 2392499.1)	

Bug 8740909  [11GR2-LNX-090726]OLD DISK GROUP RESOURCE NOT DELETED AFTER RENAMEDG

renamedg does not update resources, nor does renamedg update any file references within the database. 
Because of this, the original disk group resource is not automatically deleted. 
This resource can be manually deleted with Server Control Utility (SRVCTL).

==============================

[oracle@ol7-19-phx2 ~]$ asmcmd -V
asmcmd version 19.6.0.0.0
[oracle@ol7-19-phx2 ~]$

==============================

[oracle@ol7-19-phx1 ~]$ crsctl stat res -t -w "TYPE = ora.diskgroup.type"
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.CRS.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.DATA_PHX.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.RECO.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
--------------------------------------------------------------------------------
[oracle@ol7-19-phx1 ~]$


==============================

[oracle@ol7-19-phx1 ~]$ asmcmd umount RECO
[oracle@ol7-19-phx2 ~]$ asmcmd umount RECO

==============================

[oracle@ol7-19-phx1 ~]$ asmcmd lsdg -g
Inst_ID  State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
      1  MOUNTED  NORMAL  N         512             512   4096  1048576     98301    47769            32767            7501              0             Y  CRS/
      2  MOUNTED  NORMAL  N         512             512   4096  1048576     98301    47769            32767            7501              0             Y  CRS/
      1  MOUNTED  EXTERN  N         512             512   4096  1048576     32767    32663                0           32663              0             N  DATA_PHX/
      2  MOUNTED  EXTERN  N         512             512   4096  1048576     32767    32663                0           32663              0             N  DATA_PHX/

[oracle@ol7-19-phx1 ~]$ crsctl stat res -t -w "TYPE = ora.diskgroup.type"
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.CRS.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.DATA_PHX.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.RECO.dg(ora.asmgroup)
      1        OFFLINE OFFLINE                               STABLE
      2        OFFLINE OFFLINE                               STABLE
      3        OFFLINE OFFLINE                               STABLE
--------------------------------------------------------------------------------
[oracle@ol7-19-phx1 ~]$

==============================

[oracle@ol7-19-phx1 ~]$ renamedg dgname=RECO newdgname=FRA_PHX verbose=true

Parameters in effect:

         Old DG name       : RECO
         New DG name          : FRA_PHX
         Phases               :
                 Phase 1
                 Phase 2
         Discovery str        : (null)
         Clean              : TRUE
         Raw only           : TRUE
renamedg operation: dgname=RECO newdgname=FRA_PHX verbose=true
Executing phase 1
Discovering the group
Performing discovery with string:
ERROR: submiting READ on /dev/sdg failed due to kgfknm error
ERROR: -20(27041 /dev/sdg)ERROR: submiting READ on /dev/sda failed due to kgfknm error
ERROR: -20(27041 /dev/sda)ERROR: submiting READ on /dev/sdc failed due to kgfknm error
ERROR: -20(27041 /dev/sdc)ERROR: submiting READ on /dev/sdf failed due to kgfknm error
ERROR: -20(27041 /dev/sdf)ERROR: submiting READ on /dev/sde failed due to kgfknm error
ERROR: -20(27041 /dev/sde)ERROR: submiting READ on /dev/sdd failed due to kgfknm error
ERROR: -20(27041 /dev/sdd)ERROR: submiting READ on /dev/sdb failed due to kgfknm error
ERROR: -20(27041 /dev/sdb)ERROR: submiting READ on /dev/sda1 failed due to kgfknm error
ERROR: -20(27041 /dev/sda1)ERROR: submiting READ on /dev/sda2 failed due to kgfknm error
ERROR: -20(27041 /dev/sda2)ERROR: submiting READ on /dev/sdb1 failed due to kgfknm error
ERROR: -20(27041 /dev/sdb1)Identified disk UFS:/dev/sdg1 with disk number:0 and timestamp (33103213 1600008192)
Checking for hearbeat...
Re-discovering the group
Performing discovery with string:
ERROR: submiting READ on /dev/sdg failed due to kgfknm error
ERROR: -20(27041 /dev/sdg)ERROR: submiting READ on /dev/sda failed due to kgfknm error
ERROR: -20(27041 /dev/sda)ERROR: submiting READ on /dev/sdc failed due to kgfknm error
ERROR: -20(27041 /dev/sdc)ERROR: submiting READ on /dev/sdf failed due to kgfknm error
ERROR: -20(27041 /dev/sdf)ERROR: submiting READ on /dev/sde failed due to kgfknm error
ERROR: -20(27041 /dev/sde)ERROR: submiting READ on /dev/sdd failed due to kgfknm error
ERROR: -20(27041 /dev/sdd)ERROR: submiting READ on /dev/sdb failed due to kgfknm error
ERROR: -20(27041 /dev/sdb)ERROR: submiting READ on /dev/sda1 failed due to kgfknm error
ERROR: -20(27041 /dev/sda1)ERROR: submiting READ on /dev/sda2 failed due to kgfknm error
ERROR: -20(27041 /dev/sda2)ERROR: submiting READ on /dev/sdb1 failed due to kgfknm error
ERROR: -20(27041 /dev/sdb1)Identified disk UFS:/dev/sdg1 with disk number:0 and timestamp (33103213 1600008192)
Checking if the diskgroup is mounted or used by CSS
Checking disk number:0
Generating configuration file..
Completed phase 1
Executing phase 2
Looking for /dev/sdg1
Modifying the header
Completed phase 2
[oracle@ol7-19-phx1 ~]$

==============================

[oracle@ol7-19-phx1 ~]$ asmcmd mount FRA_PHX
[oracle@ol7-19-phx2 ~]$ asmcmd mount FRA_PHX

==============================

[oracle@ol7-19-phx1 ~]$ asmcmd lsdg FRA_PHX -g
Inst_ID  State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
      1  MOUNTED  EXTERN  N         512             512   4096  1048576     32767    32663                0           32663              0             N  FRA_PHX/
      2  MOUNTED  EXTERN  N         512             512   4096  1048576     32767    32663                0           32663              0             N  FRA_PHX/

[oracle@ol7-19-phx1 ~]$ crsctl stat res -t -w "TYPE = ora.diskgroup.type"
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.CRS.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.DATA_PHX.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.FRA_PHX.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.RECO.dg(ora.asmgroup)
      1        OFFLINE OFFLINE                               STABLE
      2        OFFLINE OFFLINE                               STABLE
      3        OFFLINE OFFLINE                               STABLE
--------------------------------------------------------------------------------
[oracle@ol7-19-phx1 ~]$

### For when DG is not online:
srvctl start diskgroup -diskgroup FRA_PHX -node "node_list"

==============================

[oracle@ol7-19-phx1 ~]$ srvctl remove diskgroup -g RECO
[oracle@ol7-19-phx1 ~]$ crsctl stat res -t -w "TYPE = ora.diskgroup.type"
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.CRS.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.DATA_PHX.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.FRA_PHX.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
--------------------------------------------------------------------------------
[oracle@ol7-19-phx1 ~]$

Q.E.D.

Blog at WordPress.com.