Thinking Out Loud

December 17, 2021

OEM Log4j Vulnerability

Filed under: Grid Control — mdinh @ 12:28 am

Surprise that Oracle did not automate the solution vs manual work.

Test case below is for EM 13.5 only.

Hopefully, I did it right and would be nice to have some sort of validations.

Security Alert For CVE-2021-44228 & CVE-2021-45046 Patch Availability Document for Oracle Enterprise Manager Cloud Control (Doc ID 2828296.1)

Applies to Oracle Enterprise Manager 13.5 & 13.4 and underlying Oracle Fusion Middleware 12.2.1.4 and 12.2.1.3 products using Log4j 2.X jars 

Components impacted with Log4j version 2 jars based on EM version
EM 13.5

FMW Component on OMS Home
DB Plugin Home
FMW Component on Agent Home

=====================================================
### Patch/Mitigate FMW component on OMS Home EM 13.5
=====================================================

Note : Perform these steps on all OMS homes in case of Multi OMS setup

find /u01/app/oracle/middleware -name setupinfo.txt
find /u01/app/oracle/middleware -name portlist.ini

--- Find FMW from ORACLE_BASE
[oracle@ol7-em135 ~]$ find /u01/app/oracle -name middleware
/u01/app/oracle/middleware
[oracle@ol7-em135 ~]$

--- Navigate to location 
[oracle@ol7-em135 ~]$ cd /u01/app/oracle/middleware/oracle_common/modules/thirdparty/

--- Run the below command
[oracle@ol7-em135 thirdparty]$ zip -q -d log4j-2.11.1.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

--- Verify removal of class on the LOG4J core jar
[oracle@ol7-em135 thirdparty]$ unzip -l log4j-2.11.1.jar | grep JndiLookup.class
[oracle@ol7-em135 thirdparty]$

--- Restart OMS server 
[oracle@ol7-em135 thirdparty]$ /u01/app/oracle/middleware/bin/emctl stop oms -all
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Stopping Oracle Management Server...
WebTier Successfully Stopped
Oracle Management Server Successfully Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down
JVMD Engine is Down

[oracle@ol7-em135 thirdparty]$ /u01/app/oracle/middleware/bin/emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
JVMD Engine is Up

[oracle@ol7-em135 thirdparty]$ /u01/app/oracle/middleware/bin/emctl status oms
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up
[oracle@ol7-em135 thirdparty]$


==================================================
### Patch/Mitigate Agent Home
==================================================

Note: These Steps have to be performed on each agent home

--- Find Agent Binaries
[oracle@ol7-em135 ~]$ ps -ef|grep [a]gent_inst
oracle   32531     1  0 22:53 pts/0    00:00:00 /u01/app/oracle/agent/agent_13.5.0.0.0/perl/bin/perl /u01/app/oracle/agent/agent_13.5.0.0.0/bin/emwd.pl agent /u01/app/oracle/agent/agent_inst/sysman/log/emagent.nohup
[oracle@ol7-em135 ~]$

--- Navigate to location  
[oracle@ol7-em135 ~]$ cd /u01/app/oracle/agent/agent_13.5.0.0.0/oracle_common/modules/thirdparty/
[oracle@ol7-em135 thirdparty]$

--- Run the below command
[oracle@ol7-em135 thirdparty]$ zip -q -d log4j-2.11.1.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
[oracle@ol7-em135 thirdparty]$

--- Verify removal of class on the LOG4J core jar
[oracle@ol7-em135 thirdparty]$ unzip -l log4j-2.11.1.jar | grep JndiLookup.class
[oracle@ol7-em135 thirdparty]$

--- Restart the Agent  
[oracle@ol7-em135 thirdparty]$ /u01/app/oracle/agent/agent_13.5.0.0.0/bin/emctl stop agent
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.

[oracle@ol7-em135 thirdparty]$ /u01/app/oracle/agent/agent_13.5.0.0.0/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Starting agent .............. started.

[oracle@ol7-em135 thirdparty]$ /u01/app/oracle/agent/agent_13.5.0.0.0/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.5.0.0.0
OMS Version            : 13.5.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/agent/agent_inst
Agent Log Directory    : /u01/app/oracle/agent/agent_inst/sysman/log
Agent Binaries         : /u01/app/oracle/agent/agent_13.5.0.0.0
Core JAR Location      : /u01/app/oracle/agent/agent_13.5.0.0.0/jlib
Agent Process ID       : 12927
Parent Process ID      : 12873
Agent URL              : https://ol7-em135.localdomain:3872/emd/main/
Local Agent URL in NAT : https://ol7-em135.localdomain:3872/emd/main/
Repository URL         : https://ol7-em135.localdomain:4903/empbs/upload
Started at             : 2021-12-16 23:34:05
Started by user        : oracle
Operating System       : Linux version 5.4.17-2136.300.7.el7uek.x86_64 (amd64)
Number of Targets      : 35
Last Reload            : (none)
Last successful upload                       : 2021-12-16 23:34:18
Last attempted upload                        : 2021-12-16 23:34:18
Total Megabytes of XML files uploaded so far : 0.02
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 55.66%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2021-12-16 23:34:11
Last successful heartbeat to OMS             : 2021-12-16 23:34:11
Next scheduled heartbeat to OMS              : 2021-12-16 23:35:13

---------------------------------------------------------------
Agent is Running and Ready
[oracle@ol7-em135 thirdparty]$

--- NOTE: thirdparty does not exists at agent_inst
[vagrant@ol7-em135 ~]$ cd /u01/app/oracle/agent/agent_inst
[vagrant@ol7-em135 agent_inst]$ ls
bin  diag  install  internal  oracle-dfw-0.tmp  sysman
[vagrant@ol7-em135 agent_inst]$


==================================================
### Patch/Mitigate DB Plug-in Home
==================================================

--- Find gc_inst
[oracle@ol7-em135 ~]$ find /u01/app/oracle -name gc_inst
/u01/app/oracle/gc_inst
[oracle@ol7-em135 ~]$ cd /u01/app/oracle/gc_inst

--- Locate log4j*.jar on your system using the following command
[oracle@ol7-em135 gc_inst]$ find . -name log4j*2.8.2*.jar -print
./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-api-2.8.2.jar
./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-core-2.8.2.jar
./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-web-2.8.2.jar
[oracle@ol7-em135 gc_inst]$

--- To identify the Log4j version use the below command 
--- (Ensure the log 4j version is indeed 2.8.2 )
[oracle@ol7-em135 gc_inst]$ unzip -p log4j-core-2.8.2.jar META-INF/MANIFEST.MF
[oracle@ol7-em135 gc_inst]$ unzip -p log4j-core-2.8.2.jar META-INF/MANIFEST.MF
[oracle@ol7-em135 gc_inst]$ unzip -p log4j-web-2.8.2.jar META-INF/MANIFEST.MF

--- Delete the following files
[oracle@ol7-em135 gc_inst]$ find . -name log4j*2.8.2*.jar -exec ls -l {} \;
-rw-r-----. 1 oracle oinstall 228154 May  4  2020 ./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-api-2.8.2.jar
-rw-r-----. 1 oracle oinstall 1407853 May  4  2020 ./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-core-2.8.2.jar
-rw-r-----. 1 oracle oinstall 32684 May  4  2020 ./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-web-2.8.2.jar
[oracle@ol7-em135 gc_inst]$

[oracle@ol7-em135 gc_inst]$ find . -name log4j*2.8.2*.jar -exec rm -fv {} \;
removed ‘./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-api-2.8.2.jar’
removed ‘./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-core-2.8.2.jar’
removed ‘./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-web-2.8.2.jar’
[oracle@ol7-em135 gc_inst]$

December 8, 2021

Find Statistics and SQL Using Index

Filed under: 11g,oracle,performance — mdinh @ 10:05 pm

Database is facing issues with enq: TX – allocate ITL entry.

Research to find Troubleshooting waits for ‘enq: TX – allocate ITL entry’ (Doc ID 1472175.1)

Index DINH.INFO_IX1 was rebuilt.

To find SQL using index DINH.INFO_IX1- https://github.com/jkstill/oracle-script-lib/blob/master/sql/find-index-sql.sql

After running find-index-sql.sql, get execution statistics and history for a SQL using Doc ID 1371778.1.

The document illustrates how to get execution statistics and history for a SQL using sql_id using Data Dictionary queries for both queries in memory and those in the AWR.

DEMO:

-- Find SQL where an index has been used

SQL> @find-index-sql.sql

old   2:    :b_index_owner := '&&v_index_owner';
new   2:    :b_index_owner := 'DINH';
old   3:    :b_index_name := '&&v_index_name';
new   3:    :b_index_name := 'INFO_IX1';

PL/SQL procedure successfully completed.

Elapsed: 00:00:00.00


SQL_ID        PLAN_HASH_VALUE
------------- ---------------
272kpc0q766tx      1690776192
7wd9njg9sy9mk      1635894828
gu9t5kgmy3pp6       698638001

Elapsed: 00:00:46.75

SQL>

======================================================================
--- sql_history.sql for SQL_ID: 272kpc0q766tx using index DINH.INFO_IX1
======================================================================

SQL> @sql_history.sql

--- From Memory

Enter value for sql_id: 272kpc0q766tx
old  16: where sql_id = '&sql_id'
new  16: where sql_id = '272kpc0q766tx'

SQL_ID        CHILD_NUMBER PLAN_HASH_VALUE FIRST_LOAD_TIME      LAST_LOAD_TIME       OUTLINE_CATEGORY     SQL_PROFILE                      EXECUTIONS
------------- ------------ --------------- -------------------- -------------------- -------------------- -------------------------------- ----------
272kpc0q766tx            0      1690776192 2018-10-23/11:20:35  2021-10-21/05:55:23                                                                29
272kpc0q766tx            1      1690776192 2018-10-23/11:20:35  2021-07-29/19:55:22                                                          29268833
272kpc0q766tx            3      1690776192 2018-10-23/11:20:35  2021-12-08/08:00:26                                                               441
272kpc0q766tx            4      1690776192 2018-10-23/11:20:35  2021-05-07/05:16:48                                                              3583
272kpc0q766tx            6      1690776192 2018-10-23/11:20:35  2021-12-01/21:18:11                                                            413747
272kpc0q766tx            8      1690776192 2018-10-23/11:20:35  2021-12-01/22:07:51                                                               164
272kpc0q766tx           37      1690776192 2018-10-23/11:20:35  2021-04-24/10:51:28                                                           3394967
272kpc0q766tx           38      1690776192 2018-10-23/11:20:35  2021-04-24/16:02:52                                                             43383
272kpc0q766tx           49      1690776192 2018-10-23/11:20:35  2021-07-21/18:27:05                                                            466811
272kpc0q766tx           51      1690776192 2018-10-23/11:20:35  2021-07-21/22:27:11                                                              5238
272kpc0q766tx           54      1690776192 2018-10-23/11:20:35  2021-08-07/22:48:03                                                           3286771
272kpc0q766tx           56      1690776192 2018-10-23/11:20:35  2021-08-08/05:01:11                                                              9835
272kpc0q766tx           67      1690776192 2018-10-23/11:20:35  2021-10-02/06:52:24                                                            126753
272kpc0q766tx           68      1690776192 2018-10-23/11:20:35  2021-10-02/10:41:18                                                               138

14 rows selected.

--- From AWR

Enter value for sql_id: 272kpc0q766tx
old  15: where sql_id = '&sql_id'
new  15: where sql_id = '272kpc0q766tx'

SQL_ID           SNAP_ID PLAN_HASH_VALUE SQL_PROFILE                      EXECUTIONS_TOTAL
------------- ---------- --------------- -------------------------------- ----------------
272kpc0q766tx     353952      1690776192                                          35878169
272kpc0q766tx     355246      1690776192                                          37443248

SQL>

======================================================================
--- sql_history.sql for SQL_ID: 7wd9njg9sy9mk using index DINH.INFO_IX1
======================================================================

SQL> @sql_history.sql

--- From Memory

Enter value for sql_id: 7wd9njg9sy9mk
old  16: where sql_id = '&sql_id'
new  16: where sql_id = '7wd9njg9sy9mk'

no rows selected

--- From AWR

Enter value for sql_id: 7wd9njg9sy9mk
old  15: where sql_id = '&sql_id'
new  15: where sql_id = '7wd9njg9sy9mk'

no rows selected

SQL>

======================================================================
--- sql_history.sql for SQL_ID: gu9t5kgmy3pp6 using index DINH.INFO_IX1
======================================================================

SQL> @sql_history.sql

--- From Memory

Enter value for sql_id: gu9t5kgmy3pp6
old  16: where sql_id = '&sql_id'
new  16: where sql_id = 'gu9t5kgmy3pp6'

SQL_ID        CHILD_NUMBER PLAN_HASH_VALUE FIRST_LOAD_TIME      LAST_LOAD_TIME       OUTLINE_CATEGORY     SQL_PROFILE                      EXECUTIONS
------------- ------------ --------------- -------------------- -------------------- -------------------- -------------------------------- ----------
gu9t5kgmy3pp6            0       698638001 2018-10-23/11:20:27  2021-12-08/08:05:10                                                               268
gu9t5kgmy3pp6            1       698638001 2018-10-23/11:20:27  2021-08-07/23:19:17                                                           5343273
gu9t5kgmy3pp6            5       698638001 2018-10-23/11:20:27  2021-12-01/18:42:33                                                            190004
gu9t5kgmy3pp6           23       698638001 2018-10-23/11:20:27  2021-04-24/14:27:12                                                           5512867
gu9t5kgmy3pp6           32       698638001 2018-10-23/11:20:27  2021-07-21/19:22:08                                                           2831328
gu9t5kgmy3pp6           39       698638001 2018-10-23/11:20:27  2021-10-02/04:16:50                                                            287105

6 rows selected.

--- From AWR

Enter value for sql_id: gu9t5kgmy3pp6
old  15: where sql_id = '&sql_id'
new  15: where sql_id = 'gu9t5kgmy3pp6'

no rows selected

SQL>

UPDATE for sql_id 7wd9njg9sy9mk:

sql_id 7wd9njg9sy9mk exist from gv$sql_plan and dba_hist_sql_plan (using find-index-sql.sql)

but not from v$sql and dba_hist_sqlstat (using Doc ID 1371778.1) which is strange.

Manually checking using DBMS_XPLAN.display_cursor and dbms_xplan.display_awr shows:

SQL> SELECT * FROM TABLE(DBMS_XPLAN.display_cursor(sql_id=>'7wd9njg9sy9mk',format=>'ALLSTATS LAST +cost +bytes')); 

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------
SQL_ID: 7wd9njg9sy9mk, child number: 0 cannot be found


Elapsed: 00:00:00.00

SQL>

SQL> SELECT * FROM TABLE(dbms_xplan.display_awr('7wd9njg9sy9mk'));

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------
SQL_ID 7wd9njg9sy9mk
--------------------
An uncaught error happened in prepare_sql_statement : ORA-01403: no data found

Plan hash value: 1635894828

-----------------------------------------------------------------------------------------------
| Id  | Operation           | Name                    | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------------
|   0 | DELETE STATEMENT    |                         |       |       |   538 (100)|          |
|   1 |  DELETE             | *******_UPDATE_INFO     |       |       |            |          |
|   2 |   COUNT STOPKEY     |                         |       |       |            |          |
|   3 |    NESTED LOOPS     |                         |    50M|   958M|   538 (100)| 00:00:07 |
|   4 |     INDEX FULL SCAN | SYS_C00137313           |    50M|   620M|     4   (0)| 00:00:01 |
|   5 |     INDEX RANGE SCAN| **************_INFO_IX1 |     1 |     7 |     1   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------------

Note
-----
   - dynamic sampling used for this statement (level=2)


21 rows selected.

Elapsed: 00:00:00.03

SQL>

December 4, 2021

How To Resolve Tablespace Created With Windows Path In Linux

Filed under: 11g,linux,oracle — mdinh @ 3:20 pm

Now I understand the rationale for not providing DBA privileges to inexperience developers.

Currently, I am uncertain if the current environment is production or not.

Here is demo for 11.2.0.4 to demonstrate the issue and resolution.

Reference:
https://docs.oracle.com/cd/E18283_01/server.112/e17120/dfiles005.htm#i1006478

[oracle@ol7-112-dg1 trace]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sat Dec 4 14:46:37 2021

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> show parameter db_create_file_dest

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_create_file_dest                  string      /u01/oradata

----------------------------------------
--- Windows path used for datafile:
----------------------------------------

SQL> create tablespace TEST datafile 'D:\Oracle\oradata\E1Local\TEST.dbf' SIZE 16M;

Tablespace created.

SQL> set lines 200
SQL> col name for a80
SQL> select file#,name from v$datafile order by 1;

     FILE# NAME
---------- --------------------------------------------------------------------------------
         1 /u01/oradata/hawk/system01.dbf
         2 /u01/oradata/hawk/sysaux01.dbf
         3 /u01/oradata/hawk/undotbs01.dbf
         4 /u01/oradata/hawk/users01.dbf
         5 /u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/D:OracleoradataE1LocalTEST.dbf

SQL> alter tablespace TEST offline normal;

Tablespace altered.

SQL> select TABLESPACE_NAME,STATUS from dba_tablespaces;

TABLESPACE_NAME                STATUS
------------------------------ ---------
SYSTEM                         ONLINE
SYSAUX                         ONLINE
UNDOTBS1                       ONLINE
TEMP                           ONLINE
USERS                          ONLINE
TEST                           OFFLINE

6 rows selected.

SQL> host

----------------------------------------
--- Rename datafile from OS
----------------------------------------

[oracle@ol7-112-dg1 trace]$ mv -fv /u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/D:OracleoradataE1LocalTEST.dbf /u01/oradata/test.dbf
‘/u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/D:OracleoradataE1LocalTEST.dbf’ -> ‘/u01/oradata/test.dbf’

[oracle@ol7-112-dg1 trace]$ exit
exit

SQL> alter tablespace TEST rename datafile '/u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/D:OracleoradataE1LocalTEST.dbf' TO '/u01/oradata/test.dbf';

Tablespace altered.

SQL> select TABLESPACE_NAME,STATUS from dba_tablespaces;

TABLESPACE_NAME                STATUS
------------------------------ ---------
SYSTEM                         ONLINE
SYSAUX                         ONLINE
UNDOTBS1                       ONLINE
TEMP                           ONLINE
USERS                          ONLINE
TEST                           OFFLINE

6 rows selected.

SQL> alter tablespace TEST online;

Tablespace altered.

SQL> select TABLESPACE_NAME,STATUS from dba_tablespaces;

TABLESPACE_NAME                STATUS
------------------------------ ---------
SYSTEM                         ONLINE
SYSAUX                         ONLINE
UNDOTBS1                       ONLINE
TEMP                           ONLINE
USERS                          ONLINE
TEST                           ONLINE

6 rows selected.

SQL> drop tablespace TEST including contents and datafiles;

Tablespace dropped.

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[oracle@ol7-112-dg1 trace]$

----------------------------------------
--- Alert Log: NO ERRORS!
----------------------------------------

[oracle@ol7-112-dg1 trace]$ tail -25 alert_hawk.log
LNS: Standby redo logfile selected for thread 1 sequence 126 for destination LOG_ARCHIVE_DEST_2
Sat Dec 04 14:47:17 2021
create tablespace TEST datafile 'D:\Oracle\oradata\E1Local\TEST.dbf' SIZE 16M
Completed: create tablespace TEST datafile 'D:\Oracle\oradata\E1Local\TEST.dbf' SIZE 16M
Sat Dec 04 14:48:08 2021
alter tablespace TEST offline normal
Completed: alter tablespace TEST offline normal
Sat Dec 04 14:48:08 2021
Starting background process SMCO
Sat Dec 04 14:48:08 2021
SMCO started with pid=36, OS id=5159
Sat Dec 04 14:48:19 2021
Checker run found 2 new persistent data failures
Sat Dec 04 14:49:33 2021
alter tablespace TEST rename datafile '/u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/D:OracleoradataE1LocalTEST.dbf' TO '/u01/oradata/test.dbf'
Completed: alter tablespace TEST rename datafile '/u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/D:OracleoradataE1LocalTEST.dbf' TO '/u01/oradata/test.dbf'
Sat Dec 04 14:49:42 2021
Checker run found 1 new persistent data failures
Sat Dec 04 14:50:00 2021
alter tablespace TEST online
Completed: alter tablespace TEST online
Sat Dec 04 14:50:33 2021
drop tablespace TEST including contents and datafiles
Deleted file /u01/oradata/test.dbf
Completed: drop tablespace TEST including contents and datafiles
[oracle@ol7-112-dg1 trace]$

----------------------------------------
--- Resolution for 19c with ASM
----------------------------------------

SQL> alter session set container=PDB;
SQL> ALTER DATABASE MOVE DATAFILE '/u02/app/oracle/product/19.0.0.0/dbhome_1/dbs/D:OracleoradataE1LocalTEST.dbf' TO '+DATAC1';

Blog at WordPress.com.