OEM Log4j Vulnerability

Surprise that Oracle did not automate the solution vs manual work.

Test case below is for EM 13.5 only.

Hopefully, I did it right and would be nice to have some sort of validations.

Security Alert For CVE-2021-44228 & CVE-2021-45046 Patch Availability Document for Oracle Enterprise Manager Cloud Control (Doc ID 2828296.1)

Applies to Oracle Enterprise Manager 13.5 & 13.4 and underlying Oracle Fusion Middleware 12.2.1.4 and 12.2.1.3 products using Log4j 2.X jars 

Components impacted with Log4j version 2 jars based on EM version
EM 13.5

FMW Component on OMS Home
DB Plugin Home
FMW Component on Agent Home

=====================================================
### Patch/Mitigate FMW component on OMS Home EM 13.5
=====================================================

Note : Perform these steps on all OMS homes in case of Multi OMS setup

find /u01/app/oracle/middleware -name setupinfo.txt
find /u01/app/oracle/middleware -name portlist.ini

--- Find FMW from ORACLE_BASE
[oracle@ol7-em135 ~]$ find /u01/app/oracle -name middleware
/u01/app/oracle/middleware
[oracle@ol7-em135 ~]$

--- Navigate to location 
[oracle@ol7-em135 ~]$ cd /u01/app/oracle/middleware/oracle_common/modules/thirdparty/

--- Run the below command
[oracle@ol7-em135 thirdparty]$ zip -q -d log4j-2.11.1.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

--- Verify removal of class on the LOG4J core jar
[oracle@ol7-em135 thirdparty]$ unzip -l log4j-2.11.1.jar | grep JndiLookup.class
[oracle@ol7-em135 thirdparty]$

--- Restart OMS server 
[oracle@ol7-em135 thirdparty]$ /u01/app/oracle/middleware/bin/emctl stop oms -all
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Stopping Oracle Management Server...
WebTier Successfully Stopped
Oracle Management Server Successfully Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down
JVMD Engine is Down

[oracle@ol7-em135 thirdparty]$ /u01/app/oracle/middleware/bin/emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
JVMD Engine is Up

[oracle@ol7-em135 thirdparty]$ /u01/app/oracle/middleware/bin/emctl status oms
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up
[oracle@ol7-em135 thirdparty]$


==================================================
### Patch/Mitigate Agent Home
==================================================

Note: These Steps have to be performed on each agent home

--- Find Agent Binaries
[oracle@ol7-em135 ~]$ ps -ef|grep [a]gent_inst
oracle   32531     1  0 22:53 pts/0    00:00:00 /u01/app/oracle/agent/agent_13.5.0.0.0/perl/bin/perl /u01/app/oracle/agent/agent_13.5.0.0.0/bin/emwd.pl agent /u01/app/oracle/agent/agent_inst/sysman/log/emagent.nohup
[oracle@ol7-em135 ~]$

--- Navigate to location  
[oracle@ol7-em135 ~]$ cd /u01/app/oracle/agent/agent_13.5.0.0.0/oracle_common/modules/thirdparty/
[oracle@ol7-em135 thirdparty]$

--- Run the below command
[oracle@ol7-em135 thirdparty]$ zip -q -d log4j-2.11.1.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
[oracle@ol7-em135 thirdparty]$

--- Verify removal of class on the LOG4J core jar
[oracle@ol7-em135 thirdparty]$ unzip -l log4j-2.11.1.jar | grep JndiLookup.class
[oracle@ol7-em135 thirdparty]$

--- Restart the Agent  
[oracle@ol7-em135 thirdparty]$ /u01/app/oracle/agent/agent_13.5.0.0.0/bin/emctl stop agent
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.

[oracle@ol7-em135 thirdparty]$ /u01/app/oracle/agent/agent_13.5.0.0.0/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Starting agent .............. started.

[oracle@ol7-em135 thirdparty]$ /u01/app/oracle/agent/agent_13.5.0.0.0/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.5.0.0.0
OMS Version            : 13.5.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/agent/agent_inst
Agent Log Directory    : /u01/app/oracle/agent/agent_inst/sysman/log
Agent Binaries         : /u01/app/oracle/agent/agent_13.5.0.0.0
Core JAR Location      : /u01/app/oracle/agent/agent_13.5.0.0.0/jlib
Agent Process ID       : 12927
Parent Process ID      : 12873
Agent URL              : https://ol7-em135.localdomain:3872/emd/main/
Local Agent URL in NAT : https://ol7-em135.localdomain:3872/emd/main/
Repository URL         : https://ol7-em135.localdomain:4903/empbs/upload
Started at             : 2021-12-16 23:34:05
Started by user        : oracle
Operating System       : Linux version 5.4.17-2136.300.7.el7uek.x86_64 (amd64)
Number of Targets      : 35
Last Reload            : (none)
Last successful upload                       : 2021-12-16 23:34:18
Last attempted upload                        : 2021-12-16 23:34:18
Total Megabytes of XML files uploaded so far : 0.02
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 55.66%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2021-12-16 23:34:11
Last successful heartbeat to OMS             : 2021-12-16 23:34:11
Next scheduled heartbeat to OMS              : 2021-12-16 23:35:13

---------------------------------------------------------------
Agent is Running and Ready
[oracle@ol7-em135 thirdparty]$

--- NOTE: thirdparty does not exists at agent_inst
[vagrant@ol7-em135 ~]$ cd /u01/app/oracle/agent/agent_inst
[vagrant@ol7-em135 agent_inst]$ ls
bin  diag  install  internal  oracle-dfw-0.tmp  sysman
[vagrant@ol7-em135 agent_inst]$


==================================================
### Patch/Mitigate DB Plug-in Home
==================================================

--- Find gc_inst
[oracle@ol7-em135 ~]$ find /u01/app/oracle -name gc_inst
/u01/app/oracle/gc_inst
[oracle@ol7-em135 ~]$ cd /u01/app/oracle/gc_inst

--- Locate log4j*.jar on your system using the following command
[oracle@ol7-em135 gc_inst]$ find . -name log4j*2.8.2*.jar -print
./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-api-2.8.2.jar
./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-core-2.8.2.jar
./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-web-2.8.2.jar
[oracle@ol7-em135 gc_inst]$

--- To identify the Log4j version use the below command 
--- (Ensure the log 4j version is indeed 2.8.2 )
[oracle@ol7-em135 gc_inst]$ unzip -p log4j-core-2.8.2.jar META-INF/MANIFEST.MF
[oracle@ol7-em135 gc_inst]$ unzip -p log4j-core-2.8.2.jar META-INF/MANIFEST.MF
[oracle@ol7-em135 gc_inst]$ unzip -p log4j-web-2.8.2.jar META-INF/MANIFEST.MF

--- Delete the following files
[oracle@ol7-em135 gc_inst]$ find . -name log4j*2.8.2*.jar -exec ls -l {} \;
-rw-r-----. 1 oracle oinstall 228154 May  4  2020 ./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-api-2.8.2.jar
-rw-r-----. 1 oracle oinstall 1407853 May  4  2020 ./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-core-2.8.2.jar
-rw-r-----. 1 oracle oinstall 32684 May  4  2020 ./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-web-2.8.2.jar
[oracle@ol7-em135 gc_inst]$

[oracle@ol7-em135 gc_inst]$ find . -name log4j*2.8.2*.jar -exec rm -fv {} \;
removed ‘./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-api-2.8.2.jar’
removed ‘./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-core-2.8.2.jar’
removed ‘./user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emdb/1danf1/database/jet/emsaasui/emcdbms-ui/ear/APP-INF/lib/log4j-web-2.8.2.jar’
[oracle@ol7-em135 gc_inst]$

Advertisement

Query OEM mgmt$(target|target_properties)

TARGET_TYPE oracle_database can be a database or an instance; however, rac_database is a database

select t.TARGET_TYPE, t.TYPE_QUALIFIER3, count(*)
from mgmt$target_properties p, mgmt$target t
where p.TARGET_GUID=t.TARGET_GUID
and p.PROPERTY_NAME='DataGuardStatus'
group by t.TARGET_TYPE, t.TYPE_QUALIFIER3
order by 2,1 desc
;

There is 1 single instance database, 105 RAC databases, 210 RAC instances.

TARGET_TYPE                    TYPE_QUALIFIER3        COUNT(*)
------------------------------ -------------------- ----------
rac_database                   DB                          105
oracle_database                DB                            1
oracle_database                RACINST                     210

DataGuardStatus does not mean Data Guard exist unless PROPERTY_VALUE is populated

select 
t.TARGET_TYPE, t.TYPE_QUALIFIER3, 
NVL(REGEXP_REPLACE(p.property_value,'[[:space:]]'),'NO DataGuard') PROPERTY_VALUE, count(*)
from mgmt$target_properties p, mgmt$target t
where p.TARGET_GUID=t.TARGET_GUID
and p.PROPERTY_NAME='DataGuardStatus'
group by t.TARGET_TYPE, t.TYPE_QUALIFIER3, PROPERTY_VALUE
order by 2,1 desc
;

There are 48 RAC Primary and 49 RAC Physical Standby because 1 RAC database has 2 Physical Standby.

TARGET_TYPE                    TYPE_QUALIFIER3      PROPERTY_VALUE                   COUNT(*)
------------------------------ -------------------- ------------------------------ ----------
rac_database                   DB                   NO DataGuard                            8
rac_database                   DB                   PhysicalStandby                        49
rac_database                   DB                   Primary                                48
oracle_database                DB                   NO DataGuard                            1
oracle_database                RACINST              NO DataGuard                           16
oracle_database                RACINST              PhysicalStandby                        98
oracle_database                RACINST              Primary                                96

Here’s how to determine the values for TYPE_QUALIFIER1-4

SQL> select distinct NVL(REGEXP_REPLACE(TYPE_QUALIFIER1,'[[:space:]]'),NULL) TYPE_QUALIFIER from mgmt$target order by 1;

SQL> c/TYPE_QUALIFIER1/TYPE_QUALIFIER2
SQL> c/TYPE_QUALIFIER2/TYPE_QUALIFIER3
SQL> c/TYPE_QUALIFIER3/TYPE_QUALIFIER4

SQL to gather primary and standby targets.

-- db.sql
set echo off lines 300 pages 500 trimsp on tab off
col HOST_NAME       for a30
col TARGET_TYPE     for a20
col TYPE1           for a9
col TYPE3           for a9
col TYPE4           for a9
col PROPERTY_VALUE  for a23
col PROPERTY_NAME   for a17
col TARGET_NAME     for a60
BREAK ON HOST_NAME SKIP 1 ON PROPERTY_VALUE ON TARGET_TYPE ON PROPERTY_NAME ON TYPE3
select
  REGEXP_SUBSTR(t.HOST_NAME,'[^.]+',1,1) host_name,
--  REGEXP_SUBSTR(t.TARGET_NAME,'[^.]+',1,1) target_name, t.TARGET_TYPE,
  t.TARGET_NAME, t.TARGET_TYPE,
  NVL(REGEXP_REPLACE(property_value,'[[:space:]]'), 'Primary: NO DataGuard') PROPERTY_VALUE,
  p.PROPERTY_NAME,
  TYPE_QUALIFIER1 type1, TYPE_QUALIFIER3 type3,
  (CASE TYPE_QUALIFIER4 WHEN 'FullLLFile+CDB' THEN 'CDB' WHEN 'FullLLFile' THEN 'DB' ELSE NULL END) type4
from mgmt$target_properties p, mgmt$target t
where p.TARGET_GUID=t.TARGET_GUID
and   p.PROPERTY_NAME='DataGuardStatus' -- Find Data Guard
and   t.TYPE_QUALIFIER3='DB'            -- Find Database
order by PROPERTY_VALUE desc, t.TARGET_TYPE, t.HOST_NAME, type1 ASC, type4
;

Port Forwarding Using SSH Config File

Here is a good reference SSH config file for OpenSSH client

From a secured server, I am able to connect to 2 different environments which seems counter intuitive but I digress.

Since there are 2 different environments, the same ~/.ssh/config cannot be used as there may be IP overlap.

One environment will use ~/.ssh/config and ssh emhost

Other environment will use ~/.ssh/cbconfig and ssh -F ~/.ssh/cbconfig emhost

The default EM port for both hosts is 7803.

Using firefox https://localhost:7803/em to access EM does not work well when saving username and password as they will be overwritten.

One solution to save sysman’s password is to use URL with different port.

Hence, config will have EM port 7803 forward to 7803 while cbconfig will have EM port 7803 forward to 7804.

========================================
This is on cloud and looks complicated. 
========================================
I did not create the configuration and don't know how many hours it took.
~/.ssh/config

Host emhost
     HostName 10.157.38.66
     LocalForward 7001 10.157.38.66:7001
     LocalForward 7102 10.157.38.66:7102
     LocalForward 7803 10.157.38.66:7803
     LocalForward 9803 10.157.38.66:9803
     LocalForward 9851 10.157.38.66:9851

# DEFAULTS:
Host *
User dinh


========================================
This is on premise and looks simpler. 
========================================
ssh -F ~/.ssh/cbconfig emhost

Host emhost
     HostName 10.10.72.254
     # Forward port need to use IP address.
     # Equivalent to ssh -L 7804:10.10.72.254:7803 mdinh@10.10.72.254
     LocalForward 7804 10.131.28.227:7803

# DEFAULTS:
Host *
User mdinh

EM13.3 Directory Structures

Currently, I am preparing POC to migrate OMS 13.3 from OEL6 to OEL7 and wanted a high level overview of the installation.

[oracle@MGOEM ~]$ cat .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

export PATH=$PATH:$HOME/bin
export DISPLAY=127.0.0.1:10.0

export ORACLE_BASE=/u01/app/oracle
export AGENT_BASE=$ORACLE_BASE/agent

export AGENT_HOME=$AGENT_BASE/agent_13.3.0.0.0
export EM_INSTANCE_BASE=$ORACLE_BASE/gc_inst
export OMS_INSTANCE_BASE=$EM_INSTANCE_BASE
export OHS=$EM_INSTANCE_BASE/user_projects/domains/GCDomain/servers/ohs1

### Starting from 13cR1, Oracle home (or OMS home) refers to the Middleware home.
export ORACLE_HOME=$ORACLE_BASE/middleware
export MW_HOME=$ORACLE_HOME
export OMS_HOME=$ORACLE_HOME
[oracle@MGOEM ~]$

Overview of the Directories Created for OMS Installation.
The OMS instance base directory (typically, gc_inst) is maintained outside the middleware home

[oracle@MGOEM ~]$ cd $MW_HOME; pwd; ls
/u01/app/oracle/middleware
allroot.sh   common               embip          ldap           OMSPatcher     plsql                root.sh     ucp
asr          create_header.log    gccompliance   lib            OPatch         plugins              slax        user_projects
bi           crs                  has            logs           oracle_common  plugins_common       soa         webgate
bin          css                  install        network        oracore        postjava_header.log  sqlplus     wlserver
bmp          disc                 instantclient  nls            oraInst.loc    precomp              srvm        xdk
cfgtoollogs  doc                  inventory      ocm            ord            rdbms                stage
clone        domain-registry.xml  jdbc           ohs            oui            relnotes             sysman
coherence    em                   jlib           omscarespfile  perl           response             thirdparty
[oracle@MGOEM middleware]$

Overview of the Directories Created for Management Agent Installation (Central Agent).
Agent base directory for the central agent (Management Agent installed with the OMS).

[oracle@MGOEM middleware]$ cd $AGENT_BASE; pwd; ls
/u01/app/oracle/agent
agent_13.3.0.0.0  agent_inst  agentInstall.rsp
[oracle@MGOEM agent]$

Agent home that is within the agent base directory.

[oracle@MGOEM agent]$ cd $AGENT_HOME; pwd; ls
/u01/app/oracle/agent/agent_13.3.0.0.0
agent.rsp    EMStage        jdbc  jythonLib  OPatch         perl     replacebins.sh           sbin    xsds
bin          install        jdk   ldap       oracle_common  plugins  replacebins.sh.template  stage
cfgtoollogs  instantclient  jlib  lib        oraInst.loc    prereqs  root.sh                  sysman
config       inventory      js    ocm        oui            rda      root.sh.template         ucp
[oracle@MGOEM agent_13.3.0.0.0]$

The OMS instance base directory (typically, gc_inst) is maintained outside the middleware home.

[oracle@MGOEM agent_13.3.0.0.0]$ cd $OMS_INSTANCE_BASE; pwd; ls
/u01/app/oracle/gc_inst
em  user_projects
[oracle@MGOEM gc_inst]$

ORACLE_BASE

[oracle@MGOEM gc_inst]$  cd $ORACLE_BASE; pwd; ls
/u01/app/oracle
agent  bip  gc_inst  middleware  swlib
[oracle@MGOEM oracle]$

Inventory and Patches:

[oracle@MGOEM ~]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2015, Oracle. All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>13.8.0.0.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="oms13c1" LOC="/u01/app/oracle/middleware" TYPE="O" IDX="1"/>
<HOME NAME="agent13c1" LOC="/u01/app/oracle/agent/agent_13.3.0.0.0" TYPE="O" IDX="2"/>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>
[oracle@MGOEM ~]$

[oracle@MGOEM ~]$ $AGENT_HOME/OPatch/opatch lspatches
27839641;One-off
27369653;One-off
27244723;One-off
27074880;OPSS Bundle Patch 12.1.3.0.171124
26933408;One-off
25832897;One-off
25412962;
23519804;One-off
20882747;One-off
20442348;One-off
19982906;One-off
19345252;One-off
18814458;One-off
28042003;One-off
27419391;WLS PATCH SET UPDATE 12.1.3.0.180417
23527146;One-off
20741228;JDBC 12.1.3.1 BP1

OPatch succeeded.
[oracle@MGOEM ~]$

[oracle@MGOEM ~]$ $ORACLE_HOME/OPatch/opatch lspatches
27839641;One-off
27369653;One-off
27244723;One-off
27074880;OPSS Bundle Patch 12.1.3.0.171124
26933408;One-off
25832897;One-off
25412962;
23519804;One-off
20882747;One-off
20442348;One-off
19982906;One-off
19345252;One-off
18814458;One-off
28042003;One-off
27419391;WLS PATCH SET UPDATE 12.1.3.0.180417
23527146;One-off
20741228;JDBC 12.1.3.1 BP1

OPatch succeeded.
[oracle@MGOEM ~]$

[oracle@MGOEM ~]$ $ORACLE_HOME/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 13.8.0.0.0
Copyright (c) 2019, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/middleware
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/middleware/oraInst.loc
OPatch version    : 13.8.0.0.0
OUI version       : 13.8.0.0.0
Log file location : /u01/app/oracle/middleware/cfgtoollogs/opatch/opatch2019-05-12_16-34-38PM_1.log


OPatch detects the Middleware Home as "/u01/app/oracle/middleware"

Lsinventory Output file location : /u01/app/oracle/middleware/cfgtoollogs/opatch/lsinv/lsinventory2019-05-12_16-34-38PM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: MGOEM
ARU platform id: 226
ARU platform description:: Linux_AMD64

[oracle@MGOEM ~]$ cat /etc/oraInst.loc
inventory_loc=/u01/app/oraInventory
inst_group=oinstall
[oracle@MGOEM ~]$

[oracle@MGOEM ~]$ cat /u01/app/oracle/middleware/oraInst.loc
#Oracle Installer Location File Location
#Fri May 10 16:53:18 CEST 2019
inst_group=oinstall
inventory_loc=/u01/app/oraInventory
[oracle@MGOEM ~]$

Reference:
DIRECTORY STRUCTURE AND LOCATIONS OF IMPORTANT TRACE AND LOG FILES OF ENTERPRISE MANAGER CLOUD CONTROL 13C

Overview of the Directories Created for an Enterprise Manager System

Where’s my RAC dbconsole

[oracle@rac01 ~]$ tail /etc/oratab

# The first and second fields are the system identifier and home
# directory of the database respectively.  The third filed indicates
# to the dbstart utility that the database should , "Y", or should not,
# "N", be brought up at system boot time.
#
# Multiple entries with the same $ORACLE_SID are not allowed.
#
#
+ASM1:/u01/app/11.2.0.3/grid:N          # line added by Agent
jay:/u01/app/oracle/product/11.2.0.3/dbhome_1:N         # line added by Agent
[oracle@rac01 ~]$ . oraenv
ORACLE_SID = [oracle] ? jay
The Oracle base has been set to /u01/app/oracle
[oracle@rac01 ~]$ emctl stsatus dbconsole
Environment variable ORACLE_UNQNAME not defined. Please set ORACLE_UNQNAME to database unique name.
[oracle@rac01 ~]$ export ORACLE_UNQNAME=jay
[oracle@rac01 ~]$ emctl status dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
https://rac01:1158/em/console/aboutApplication
Oracle Enterprise Manager 11g is running.
------------------------------------------------------------------
Logs are generated in directory /u01/app/oracle/product/11.2.0.3/dbhome_1/rac01_jay/sysman/log
[oracle@rac01 ~]$ nslookup rac01
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   rac01.localdomain
Address: 192.168.56.11

[oracle@rac01 ~]$

https://192.168.56.11:1158/em <--

Troubleshooting OEM 11g Grid Agent – root suid enabled

OEM 11g Grid Control on Solaris 10.

Not sure what I did wrong in response file, but I never saw the prompt to execute root.sh

Here is the error from the database page that displays IO through put – snmhsutl.c:executable nmhs should have root suid enabled

This is resolved by executing root.sh.

oracle@proddb01:agent11g:/u01/app/oracle/product/agent11g
> ls -l ./bin/nmhs*
-rwxr-xr-x   1 oracle   dba        95912 Dec 16 13:41 ./bin/nmhs
-rwxr-xr-x   1 oracle   dba        95912 Dec 16 13:41 ./bin/nmhs.0
-rwxr-xr-x   1 oracle   dba            0 May  4  2010 ./bin/nmhs0

oracle@proddb01:agent11g:/u01/app/oracle/product/agent11g
> ll root.sh
-rwxr-x---   1 oracle   dba        12237 Dec 16 13:41 root.sh

oracle@proddb01:agent11g:/u01/app/oracle/product/agent11g
> ls -l ./bin/nmhs*
-rws--x---   1 root     dba        95912 Dec 16 13:41 ./bin/nmhs
-rwxr-xr-x   1 oracle   dba        95912 Dec 16 13:41 ./bin/nmhs.0
-rwxr-xr-x   1 oracle   dba            0 May  4  2010 ./bin/nmhs0

oracle@proddb01:agent11g:/u01/app/oracle/product/agent11g

Reference: Grid Agent Configuration: storage_report_metrics.pl Reports Error “snmhsutl.c:executable nmhs should have root suid enabled” [ID 435793.1]

Troubleshooting OEM 11g Grid Crontrol – (Too many open files)

I am running OEM 11g Grid Control on Solaris 10.

Instead of seeing charts on the OEM pages, I was seeing red X.

First, the location of the logs can be found under $EM_INSTANCE_HOME

> echo $EM_INSTANCE_HOME
/u01/app/oracle/gc_inst/em/EMGC_OMS1

/u01/app/oracle/gc_inst/em/EMGC_OMS1/sysman/log
> ll
total 11216
drwxr-----   2 oracle   dba           96 Aug 31 12:57 pafLogs
-rw-r-----   1 oracle   dba            0 Aug 31 13:02 emovm.log.0
drwxr-xr-x   9 oracle   dba         8192 Aug 31 14:39 ..
-rw-r--r--   1 oracle   dba        15264 Sep 30 06:53 secure.log
drwxr-xr-x   3 oracle   dba         8192 Dec  6 11:29 .
-rw-r-----   1 oracle   dba            0 Dec  6 11:29 emovm.log.0.lck
-rw-r-----   1 oracle   dba       387329 Dec  6 11:32 emctl.log
-rw-r-----   1 oracle   dba      2640001 Dec  8 00:00 emoms.log
-rw-r-----   1 oracle   dba      2640001 Dec  8 00:00 emoms.trc

The following errors were in the emoms.log:

2010-12-06 07:55:04,717 [[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)']
WARN  chart.EmChartBean getChartImage.5105 - EXCEPTION while generating chart image [id]: D3B6716582D334A0D8F9F9D82AAD42C8.gif

java.io.FileNotFoundException:
/u01/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emgc/n47f6s/public/images/chartCache/chartD3B6716582D334A0D8F9F9D82AAD42C8.gif.gif
(Too many open files)

This was puzzling since I remembered changing open files from 1024 to 4096 and restarting oms.

> ulimit -an
core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
open files                    (-n) 4096
pipe size          (512 bytes, -p) 10
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) 29995
virtual memory        (kbytes, -v) unlimited

Let’s find WLS processes:

> ps -afe|grep oracle|grep java|grep -v grep
  oracle  7813  7754   0   Nov 02 ?         389:43 /usr/jdk/instances/jdk1.6.0_20/bin/sparcv9/java -client -Xms256m -Xmx1024m -XX:
  oracle  7191  3335   0   Nov 02 ?         127:43 /usr/jdk/instances/jdk1.6.0_20/jre/bin/sparcv9/java -classpath /usr/jdk/instanc
  oracle 26182 26159   0   Nov 12 ?         698:36 /usr/jdk/instances/jdk1.6.0_22/bin/sparcv9/java -client -Xms256m -Xmx1024m -XX:

Let’s look at the open files associated with each process:

> pfiles 7813|head
7813:   /usr/jdk/instances/jdk1.6.0_20/bin/sparcv9/java -client -Xms256m -Xmx1
  Current rlimit: 1024 file descriptors
   0: S_IFCHR mode:0666 dev:340,66 ino:36640 uid:0 gid:3 rdev:13,2
      O_RDONLY|O_LARGEFILE
      /dev/null
   1: S_IFREG mode:0640 dev:286,1000 ino:1599591 uid:15001 gid:15000 size:30922
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
   2: S_IFREG mode:0640 dev:286,1000 ino:1599591 uid:15001 gid:15000 size:30922
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
   3: S_IFCHR mode:0666 dev:340,66 ino:36699 uid:0 gid:3 rdev:13,12

> pfiles 7191|head
7191:   /usr/jdk/instances/jdk1.6.0_20/jre/bin/sparcv9/java -classpath /usr/jd
  Current rlimit: 1024 file descriptors
   0: S_IFIFO mode:0000 dev:322,0 ino:93191797 uid:15001 gid:15000 size:0
      O_RDWR
   1: S_IFIFO mode:0000 dev:322,0 ino:93191798 uid:15001 gid:15000 size:0
      O_RDWR
   2: S_IFIFO mode:0000 dev:322,0 ino:93191799 uid:15001 gid:15000 size:0
      O_RDWR
   3: S_IFCHR mode:0666 dev:340,66 ino:36699 uid:0 gid:3 rdev:13,12
      O_RDWR|O_LARGEFILE FD_CLOEXEC

> pfiles 26182|head
26182:  /usr/jdk/instances/jdk1.6.0_22/bin/sparcv9/java -client -Xms256m -Xmx1
  Current rlimit: 1024 file descriptors
   0: S_IFCHR mode:0666 dev:340,66 ino:36640 uid:0 gid:3 rdev:13,2
      O_RDONLY|O_LARGEFILE
      /dev/null
   1: S_IFREG mode:0640 dev:286,1000 ino:3232523 uid:15001 gid:15000 size:8965325
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
   2: S_IFREG mode:0640 dev:286,1000 ino:3232523 uid:15001 gid:15000 size:8965325
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
   3: S_IFCHR mode:0666 dev:340,66 ino:36699 uid:0 gid:3 rdev:13,12

It looks like WLS was started with only 1024 file descriptors.

Restart WLS after confirming open files (-n) 4096 and verify the process:

> ps -afe|grep oracle|grep java|grep -v grep
  oracle 11942 11766   0 11:27:20 ?           6:32 /usr/jdk/instances/jdk1.6.0_22/bin/sparcv9/java -client -Xms256m -Xmx1024m -XX:
  oracle 18722  3335   0 11:25:39 pts/4       0:12 /usr/jdk/instances/jdk1.6.0_22/jre/bin/sparcv9/java -classpath /usr/jdk/instanc
  oracle 20361 20266   0 11:25:51 ?           2:45 /usr/jdk/instances/jdk1.6.0_22/bin/sparcv9/java -client -Xms256m -Xmx1024m -XX:
 
> pfiles 11942|head
11942:  /usr/jdk/instances/jdk1.6.0_22/bin/sparcv9/java -client -Xms256m -Xmx1
  Current rlimit: 4096 file descriptors
   0: S_IFCHR mode:0666 dev:340,66 ino:36640 uid:0 gid:3 rdev:13,2
      O_RDONLY|O_LARGEFILE
      /dev/null
   1: S_IFREG mode:0640 dev:286,1000 ino:1609620 uid:15001 gid:15000 size:62670
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
   2: S_IFREG mode:0640 dev:286,1000 ino:1609620 uid:15001 gid:15000 size:62670
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
   3: S_IFCHR mode:0666 dev:340,66 ino:36699 uid:0 gid:3 rdev:13,12
 
> pfiles 18722|head
18722:  /usr/jdk/instances/jdk1.6.0_22/jre/bin/sparcv9/java -classpath /usr/jd
  Current rlimit: 4096 file descriptors
   0: S_IFIFO mode:0000 dev:322,0 ino:415173812 uid:15001 gid:15000 size:0
      O_RDWR
   1: S_IFIFO mode:0000 dev:322,0 ino:415173813 uid:15001 gid:15000 size:0
      O_RDWR
   2: S_IFIFO mode:0000 dev:322,0 ino:415173814 uid:15001 gid:15000 size:0
      O_RDWR
   3: S_IFCHR mode:0666 dev:340,66 ino:36699 uid:0 gid:3 rdev:13,12
      O_RDWR|O_LARGEFILE FD_CLOEXEC
 
> pfiles 20361|head
20361:  /usr/jdk/instances/jdk1.6.0_22/bin/sparcv9/java -client -Xms256m -Xmx1
  Current rlimit: 4096 file descriptors
   0: S_IFCHR mode:0666 dev:340,66 ino:36640 uid:0 gid:3 rdev:13,2
      O_RDONLY|O_LARGEFILE
      /dev/null
   1: S_IFREG mode:0640 dev:286,1000 ino:1610388 uid:15001 gid:15000 size:25426
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
   2: S_IFREG mode:0640 dev:286,1000 ino:1610388 uid:15001 gid:15000 size:25426
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
   3: S_IFCHR mode:0666 dev:340,66 ino:36699 uid:0 gid:3 rdev:13,12