Thinking Out Loud

May 12, 2019

EM13.3 Directory Structures

Filed under: Grid Control — mdinh @ 2:41 pm

Currently, I am preparing POC to migrate OMS 13.3 from OEL6 to OEL7 and wanted a high level overview of the installation.

[oracle@MGOEM ~]$ cat .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

export PATH=$PATH:$HOME/bin
export DISPLAY=127.0.0.1:10.0

export ORACLE_BASE=/u01/app/oracle
export AGENT_BASE=$ORACLE_BASE/agent

export AGENT_HOME=$AGENT_BASE/agent_13.3.0.0.0
export EM_INSTANCE_BASE=$ORACLE_BASE/gc_inst
export OMS_INSTANCE_BASE=$EM_INSTANCE_BASE
export OHS=$EM_INSTANCE_BASE/user_projects/domains/GCDomain/servers/ohs1

### Starting from 13cR1, Oracle home (or OMS home) refers to the Middleware home.
export ORACLE_HOME=$ORACLE_BASE/middleware
export MW_HOME=$ORACLE_HOME
export OMS_HOME=$ORACLE_HOME
[oracle@MGOEM ~]$

Overview of the Directories Created for OMS Installation.
The OMS instance base directory (typically, gc_inst) is maintained outside the middleware home

[oracle@MGOEM ~]$ cd $MW_HOME; pwd; ls
/u01/app/oracle/middleware
allroot.sh   common               embip          ldap           OMSPatcher     plsql                root.sh     ucp
asr          create_header.log    gccompliance   lib            OPatch         plugins              slax        user_projects
bi           crs                  has            logs           oracle_common  plugins_common       soa         webgate
bin          css                  install        network        oracore        postjava_header.log  sqlplus     wlserver
bmp          disc                 instantclient  nls            oraInst.loc    precomp              srvm        xdk
cfgtoollogs  doc                  inventory      ocm            ord            rdbms                stage
clone        domain-registry.xml  jdbc           ohs            oui            relnotes             sysman
coherence    em                   jlib           omscarespfile  perl           response             thirdparty
[oracle@MGOEM middleware]$

Overview of the Directories Created for Management Agent Installation (Central Agent).
Agent base directory for the central agent (Management Agent installed with the OMS).

[oracle@MGOEM middleware]$ cd $AGENT_BASE; pwd; ls
/u01/app/oracle/agent
agent_13.3.0.0.0  agent_inst  agentInstall.rsp
[oracle@MGOEM agent]$

Agent home that is within the agent base directory.

[oracle@MGOEM agent]$ cd $AGENT_HOME; pwd; ls
/u01/app/oracle/agent/agent_13.3.0.0.0
agent.rsp    EMStage        jdbc  jythonLib  OPatch         perl     replacebins.sh           sbin    xsds
bin          install        jdk   ldap       oracle_common  plugins  replacebins.sh.template  stage
cfgtoollogs  instantclient  jlib  lib        oraInst.loc    prereqs  root.sh                  sysman
config       inventory      js    ocm        oui            rda      root.sh.template         ucp
[oracle@MGOEM agent_13.3.0.0.0]$

The OMS instance base directory (typically, gc_inst) is maintained outside the middleware home.

[oracle@MGOEM agent_13.3.0.0.0]$ cd $OMS_INSTANCE_BASE; pwd; ls
/u01/app/oracle/gc_inst
em  user_projects
[oracle@MGOEM gc_inst]$

ORACLE_BASE

[oracle@MGOEM gc_inst]$  cd $ORACLE_BASE; pwd; ls
/u01/app/oracle
agent  bip  gc_inst  middleware  swlib
[oracle@MGOEM oracle]$

Inventory and Patches:

[oracle@MGOEM ~]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2015, Oracle. All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>13.8.0.0.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="oms13c1" LOC="/u01/app/oracle/middleware" TYPE="O" IDX="1"/>
<HOME NAME="agent13c1" LOC="/u01/app/oracle/agent/agent_13.3.0.0.0" TYPE="O" IDX="2"/>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>
[oracle@MGOEM ~]$

[oracle@MGOEM ~]$ $AGENT_HOME/OPatch/opatch lspatches
27839641;One-off
27369653;One-off
27244723;One-off
27074880;OPSS Bundle Patch 12.1.3.0.171124
26933408;One-off
25832897;One-off
25412962;
23519804;One-off
20882747;One-off
20442348;One-off
19982906;One-off
19345252;One-off
18814458;One-off
28042003;One-off
27419391;WLS PATCH SET UPDATE 12.1.3.0.180417
23527146;One-off
20741228;JDBC 12.1.3.1 BP1

OPatch succeeded.
[oracle@MGOEM ~]$

[oracle@MGOEM ~]$ $ORACLE_HOME/OPatch/opatch lspatches
27839641;One-off
27369653;One-off
27244723;One-off
27074880;OPSS Bundle Patch 12.1.3.0.171124
26933408;One-off
25832897;One-off
25412962;
23519804;One-off
20882747;One-off
20442348;One-off
19982906;One-off
19345252;One-off
18814458;One-off
28042003;One-off
27419391;WLS PATCH SET UPDATE 12.1.3.0.180417
23527146;One-off
20741228;JDBC 12.1.3.1 BP1

OPatch succeeded.
[oracle@MGOEM ~]$

[oracle@MGOEM ~]$ $ORACLE_HOME/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 13.8.0.0.0
Copyright (c) 2019, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/middleware
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/middleware/oraInst.loc
OPatch version    : 13.8.0.0.0
OUI version       : 13.8.0.0.0
Log file location : /u01/app/oracle/middleware/cfgtoollogs/opatch/opatch2019-05-12_16-34-38PM_1.log


OPatch detects the Middleware Home as "/u01/app/oracle/middleware"

Lsinventory Output file location : /u01/app/oracle/middleware/cfgtoollogs/opatch/lsinv/lsinventory2019-05-12_16-34-38PM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: MGOEM
ARU platform id: 226
ARU platform description:: Linux_AMD64

[oracle@MGOEM ~]$ cat /etc/oraInst.loc
inventory_loc=/u01/app/oraInventory
inst_group=oinstall
[oracle@MGOEM ~]$

[oracle@MGOEM ~]$ cat /u01/app/oracle/middleware/oraInst.loc
#Oracle Installer Location File Location
#Fri May 10 16:53:18 CEST 2019
inst_group=oinstall
inventory_loc=/u01/app/oraInventory
[oracle@MGOEM ~]$

Reference:
DIRECTORY STRUCTURE AND LOCATIONS OF IMPORTANT TRACE AND LOG FILES OF ENTERPRISE MANAGER CLOUD CONTROL 13C

Overview of the Directories Created for an Enterprise Manager System

Advertisements

May 18, 2013

Where’s my RAC dbconsole

Filed under: 11g,Grid Control,RAC — mdinh @ 11:33 pm

[oracle@rac01 ~]$ tail /etc/oratab

# The first and second fields are the system identifier and home
# directory of the database respectively.  The third filed indicates
# to the dbstart utility that the database should , "Y", or should not,
# "N", be brought up at system boot time.
#
# Multiple entries with the same $ORACLE_SID are not allowed.
#
#
+ASM1:/u01/app/11.2.0.3/grid:N          # line added by Agent
jay:/u01/app/oracle/product/11.2.0.3/dbhome_1:N         # line added by Agent
[oracle@rac01 ~]$ . oraenv
ORACLE_SID = [oracle] ? jay
The Oracle base has been set to /u01/app/oracle
[oracle@rac01 ~]$ emctl stsatus dbconsole
Environment variable ORACLE_UNQNAME not defined. Please set ORACLE_UNQNAME to database unique name.
[oracle@rac01 ~]$ export ORACLE_UNQNAME=jay
[oracle@rac01 ~]$ emctl status dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
https://rac01:1158/em/console/aboutApplication
Oracle Enterprise Manager 11g is running.
------------------------------------------------------------------
Logs are generated in directory /u01/app/oracle/product/11.2.0.3/dbhome_1/rac01_jay/sysman/log
[oracle@rac01 ~]$ nslookup rac01
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   rac01.localdomain
Address: 192.168.56.11

[oracle@rac01 ~]$

https://192.168.56.11:1158/em <--

December 19, 2010

Troubleshooting OEM 11g Grid Agent – root suid enabled

Filed under: 11g,Grid Control — mdinh @ 10:50 pm

OEM 11g Grid Control on Solaris 10.

Not sure what I did wrong in response file, but I never saw the prompt to execute root.sh

Here is the error from the database page that displays IO through put – snmhsutl.c:executable nmhs should have root suid enabled

This is resolved by executing root.sh.

oracle@proddb01:agent11g:/u01/app/oracle/product/agent11g
> ls -l ./bin/nmhs*
-rwxr-xr-x   1 oracle   dba        95912 Dec 16 13:41 ./bin/nmhs
-rwxr-xr-x   1 oracle   dba        95912 Dec 16 13:41 ./bin/nmhs.0
-rwxr-xr-x   1 oracle   dba            0 May  4  2010 ./bin/nmhs0

oracle@proddb01:agent11g:/u01/app/oracle/product/agent11g
> ll root.sh
-rwxr-x---   1 oracle   dba        12237 Dec 16 13:41 root.sh

oracle@proddb01:agent11g:/u01/app/oracle/product/agent11g
> ls -l ./bin/nmhs*
-rws--x---   1 root     dba        95912 Dec 16 13:41 ./bin/nmhs
-rwxr-xr-x   1 oracle   dba        95912 Dec 16 13:41 ./bin/nmhs.0
-rwxr-xr-x   1 oracle   dba            0 May  4  2010 ./bin/nmhs0

oracle@proddb01:agent11g:/u01/app/oracle/product/agent11g

Reference: Grid Agent Configuration: storage_report_metrics.pl Reports Error “snmhsutl.c:executable nmhs should have root suid enabled” [ID 435793.1]

December 9, 2010

Troubleshooting OEM 11g Grid Crontrol – (Too many open files)

Filed under: 11g,Grid Control — mdinh @ 3:58 am

I am running OEM 11g Grid Control on Solaris 10.

Instead of seeing charts on the OEM pages, I was seeing red X.

First, the location of the logs can be found under $EM_INSTANCE_HOME

> echo $EM_INSTANCE_HOME
/u01/app/oracle/gc_inst/em/EMGC_OMS1
/u01/app/oracle/gc_inst/em/EMGC_OMS1/sysman/log
> ll
total 11216
drwxr-----   2 oracle   dba           96 Aug 31 12:57 pafLogs
-rw-r-----   1 oracle   dba            0 Aug 31 13:02 emovm.log.0
drwxr-xr-x   9 oracle   dba         8192 Aug 31 14:39 ..
-rw-r--r--   1 oracle   dba        15264 Sep 30 06:53 secure.log
drwxr-xr-x   3 oracle   dba         8192 Dec  6 11:29 .
-rw-r-----   1 oracle   dba            0 Dec  6 11:29 emovm.log.0.lck
-rw-r-----   1 oracle   dba       387329 Dec  6 11:32 emctl.log
-rw-r-----   1 oracle   dba      2640001 Dec  8 00:00 emoms.log
-rw-r-----   1 oracle   dba      2640001 Dec  8 00:00 emoms.trc

The following errors were in the emoms.log:

2010-12-06 07:55:04,717 [[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)']
WARN  chart.EmChartBean getChartImage.5105 - EXCEPTION while generating chart image [id]: D3B6716582D334A0D8F9F9D82AAD42C8.gif

java.io.FileNotFoundException:
/u01/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user/emgc/n47f6s/public/images/chartCache/chartD3B6716582D334A0D8F9F9D82AAD42C8.gif.gif
(Too many open files)

This was puzzling since I remembered changing open files from 1024 to 4096 and restarting oms.

> ulimit -an
core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
open files                    (-n) 4096
pipe size          (512 bytes, -p) 10
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) 29995
virtual memory        (kbytes, -v) unlimited

Let’s find WLS processes:

> ps -afe|grep oracle|grep java|grep -v grep
  oracle  7813  7754   0   Nov 02 ?         389:43 /usr/jdk/instances/jdk1.6.0_20/bin/sparcv9/java -client -Xms256m -Xmx1024m -XX:
  oracle  7191  3335   0   Nov 02 ?         127:43 /usr/jdk/instances/jdk1.6.0_20/jre/bin/sparcv9/java -classpath /usr/jdk/instanc
  oracle 26182 26159   0   Nov 12 ?         698:36 /usr/jdk/instances/jdk1.6.0_22/bin/sparcv9/java -client -Xms256m -Xmx1024m -XX:

Let’s look at the open files associated with each process:

> pfiles 7813|head
7813:   /usr/jdk/instances/jdk1.6.0_20/bin/sparcv9/java -client -Xms256m -Xmx1
  Current rlimit: 1024 file descriptors
   0: S_IFCHR mode:0666 dev:340,66 ino:36640 uid:0 gid:3 rdev:13,2
      O_RDONLY|O_LARGEFILE
      /dev/null
   1: S_IFREG mode:0640 dev:286,1000 ino:1599591 uid:15001 gid:15000 size:30922
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
   2: S_IFREG mode:0640 dev:286,1000 ino:1599591 uid:15001 gid:15000 size:30922
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
   3: S_IFCHR mode:0666 dev:340,66 ino:36699 uid:0 gid:3 rdev:13,12

> pfiles 7191|head
7191:   /usr/jdk/instances/jdk1.6.0_20/jre/bin/sparcv9/java -classpath /usr/jd
  Current rlimit: 1024 file descriptors
   0: S_IFIFO mode:0000 dev:322,0 ino:93191797 uid:15001 gid:15000 size:0
      O_RDWR
   1: S_IFIFO mode:0000 dev:322,0 ino:93191798 uid:15001 gid:15000 size:0
      O_RDWR
   2: S_IFIFO mode:0000 dev:322,0 ino:93191799 uid:15001 gid:15000 size:0
      O_RDWR
   3: S_IFCHR mode:0666 dev:340,66 ino:36699 uid:0 gid:3 rdev:13,12
      O_RDWR|O_LARGEFILE FD_CLOEXEC

> pfiles 26182|head
26182:  /usr/jdk/instances/jdk1.6.0_22/bin/sparcv9/java -client -Xms256m -Xmx1
  Current rlimit: 1024 file descriptors
   0: S_IFCHR mode:0666 dev:340,66 ino:36640 uid:0 gid:3 rdev:13,2
      O_RDONLY|O_LARGEFILE
      /dev/null
   1: S_IFREG mode:0640 dev:286,1000 ino:3232523 uid:15001 gid:15000 size:8965325
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
   2: S_IFREG mode:0640 dev:286,1000 ino:3232523 uid:15001 gid:15000 size:8965325
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
   3: S_IFCHR mode:0666 dev:340,66 ino:36699 uid:0 gid:3 rdev:13,12

It looks like WLS was started with only 1024 file descriptors.

Restart WLS after confirming open files (-n) 4096 and verify the process:

> ps -afe|grep oracle|grep java|grep -v grep
  oracle 11942 11766   0 11:27:20 ?           6:32 /usr/jdk/instances/jdk1.6.0_22/bin/sparcv9/java -client -Xms256m -Xmx1024m -XX:
  oracle 18722  3335   0 11:25:39 pts/4       0:12 /usr/jdk/instances/jdk1.6.0_22/jre/bin/sparcv9/java -classpath /usr/jdk/instanc
  oracle 20361 20266   0 11:25:51 ?           2:45 /usr/jdk/instances/jdk1.6.0_22/bin/sparcv9/java -client -Xms256m -Xmx1024m -XX:
 
> pfiles 11942|head
11942:  /usr/jdk/instances/jdk1.6.0_22/bin/sparcv9/java -client -Xms256m -Xmx1
  Current rlimit: 4096 file descriptors
   0: S_IFCHR mode:0666 dev:340,66 ino:36640 uid:0 gid:3 rdev:13,2
      O_RDONLY|O_LARGEFILE
      /dev/null
   1: S_IFREG mode:0640 dev:286,1000 ino:1609620 uid:15001 gid:15000 size:62670
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
   2: S_IFREG mode:0640 dev:286,1000 ino:1609620 uid:15001 gid:15000 size:62670
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
   3: S_IFCHR mode:0666 dev:340,66 ino:36699 uid:0 gid:3 rdev:13,12
 
> pfiles 18722|head
18722:  /usr/jdk/instances/jdk1.6.0_22/jre/bin/sparcv9/java -classpath /usr/jd
  Current rlimit: 4096 file descriptors
   0: S_IFIFO mode:0000 dev:322,0 ino:415173812 uid:15001 gid:15000 size:0
      O_RDWR
   1: S_IFIFO mode:0000 dev:322,0 ino:415173813 uid:15001 gid:15000 size:0
      O_RDWR
   2: S_IFIFO mode:0000 dev:322,0 ino:415173814 uid:15001 gid:15000 size:0
      O_RDWR
   3: S_IFCHR mode:0666 dev:340,66 ino:36699 uid:0 gid:3 rdev:13,12
      O_RDWR|O_LARGEFILE FD_CLOEXEC
 
> pfiles 20361|head
20361:  /usr/jdk/instances/jdk1.6.0_22/bin/sparcv9/java -client -Xms256m -Xmx1
  Current rlimit: 4096 file descriptors
   0: S_IFCHR mode:0666 dev:340,66 ino:36640 uid:0 gid:3 rdev:13,2
      O_RDONLY|O_LARGEFILE
      /dev/null
   1: S_IFREG mode:0640 dev:286,1000 ino:1610388 uid:15001 gid:15000 size:25426
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
   2: S_IFREG mode:0640 dev:286,1000 ino:1610388 uid:15001 gid:15000 size:25426
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
   3: S_IFCHR mode:0666 dev:340,66 ino:36699 uid:0 gid:3 rdev:13,12

Blog at WordPress.com.