Thinking Out Loud

My RAC Mistakes and Notes on Virtual Box

No, this is not another post on how to install RAC on Virtual Box. Instead, it is a post on my mistakes and notes so that you don’t make the same.

Adapter 1 – eth0 (DHCP) / Adapter 2 – eth1 (STATIC IP) / Adapter 3 (STATIC IP) (Private)

dinh-vm

Hosts

[root@rac02 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.56.11  rac01 rac01.localdomain
192.168.56.12  rac02 rac02.localdomain
192.168.56.13  rac03 rac03.localdomain

192.168.56.21  rac01-vip rac01-vip.localdomain
192.168.56.22  rac02-vip rac02-vip.localdomain
192.168.56.23  rac03-vip rac03-vip.localdomain

10.10.10.11    rac01-priv rac02-priv.localdomain
10.10.10.12    rac02-priv rac02-priv.localdomain
10.10.10.13    rac03-priv rac03-priv.localdomain

Setup same eth interface across all nodes

[root@rac02 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
NAME="System eth0"
TYPE="Ethernet"
DEVICE="eth0"
BOOTPROTO=dhcp
NM_CONTROLLED="yes"
ONBOOT=yes
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
HWADDR=08:00:27:DE:4D:20
PEERDNS=yes
PEERROUTES=yes

[root@rac02 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE="eth1"
BOOTPROTO=none
NM_CONTROLLED="yes"
ONBOOT=yes
TYPE="Ethernet"
IPADDR=192.168.56.12
PREFIX=24
GATEWAY=192.168.56.1
DNS1=8.8.8.8
DOMAIN=localdomain
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth1"
HWADDR=08:00:27:EE:48:CF

[root@rac02 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth2
DEVICE="eth2"
BOOTPROTO=none
NM_CONTROLLED="yes"
ONBOOT=yes
TYPE="Ethernet"
IPADDR=10.10.10.12
PREFIX=8
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth2"
HWADDR=08:00:27:FC:83:39

Use generic name for volume group. Otherwise, rac02 clone will have vg-rac01.

[root@rac02 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg00-lv_root
                       12G  2.0G  8.9G  18% /
tmpfs                1004M   88K 1004M   1% /dev/shm
/dev/sda1             485M   55M  405M  12% /boot
Linux                 466G   45G  422G  10% /media/sf_Linux

DNS configuration – dynamic and no change needed when cloning

[root@rac02 ~]# cat /etc/dnsmasq.conf
listen-address=127.0.0.1
interface=eth1
strict-order
domain-needed
bogus-priv
no-poll
no-hosts
local=/localdomain/
addn-hosts=/etc/racdns
cache-size=150

[root@rac02 ~]# cat /etc/racdns
192.168.56.11  rac01.localdomain
192.168.56.12  rac02.localdomain
192.168.56.13  rac03.localdomain
192.168.56.31  dinh-scan.localdomain
192.168.56.32  dinh-scan.localdomain
192.168.56.33  dinh-scan.localdomain

Prevent DHCP from modifying /etc/resolv.conf

[root@rac02 ~]# chattr +i /etc/resolv.conf

Multiple DNS

[root@rac02 ~]# cat /etc/resolv.conf
nameserver 127.0.0.1
nameserver 8.8.8.8
nameserver 172.16.0.1
search gateway.2wire.net localdomain

Script – I get tired of having to type the same thing over and over again

[root@rac02 ~]# cat t.sh
ping -c 3 rac01
ping -c 3 rac02
ping -c 3 rac03
ping -c 3 www.google.com

nslookup rac01
nslookup rac02
nslookup rac03

nslookup www.google.com
nslookup dinh-scan
nslookup dinh-scan
nslookup dinh-scan

Verify Network – ping and nslookup

+ ping -c 3 rac01
[root@rac02 ~]# sh -x ./t.sh
PING rac01 (192.168.56.11) 56(84) bytes of data.
64 bytes from rac01 (192.168.56.11): icmp_seq=1 ttl=64 time=0.508 ms
64 bytes from rac01 (192.168.56.11): icmp_seq=2 ttl=64 time=0.471 ms
64 bytes from rac01 (192.168.56.11): icmp_seq=3 ttl=64 time=0.565 ms

--- rac01 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.471/0.514/0.565/0.046 ms
+ ping -c 3 rac02
PING rac02 (192.168.56.12) 56(84) bytes of data.
64 bytes from rac02 (192.168.56.12): icmp_seq=1 ttl=64 time=0.038 ms
64 bytes from rac02 (192.168.56.12): icmp_seq=2 ttl=64 time=0.021 ms
64 bytes from rac02 (192.168.56.12): icmp_seq=3 ttl=64 time=0.042 ms

--- rac02 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.021/0.033/0.042/0.011 ms
+ ping -c 3 rac03
PING rac03 (192.168.56.13) 56(84) bytes of data.
64 bytes from rac03 (192.168.56.13): icmp_seq=1 ttl=64 time=0.536 ms
64 bytes from rac03 (192.168.56.13): icmp_seq=2 ttl=64 time=0.438 ms
64 bytes from rac03 (192.168.56.13): icmp_seq=3 ttl=64 time=0.471 ms

--- rac03 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.438/0.481/0.536/0.047 ms
+ ping -c 3 www.google.com
PING www.google.com (74.125.224.178) 56(84) bytes of data.
64 bytes from lax02s01-in-f18.1e100.net (74.125.224.178): icmp_seq=1 ttl=51 time=29.5 ms
64 bytes from lax02s01-in-f18.1e100.net (74.125.224.178): icmp_seq=2 ttl=49 time=28.6 ms
64 bytes from lax02s01-in-f18.1e100.net (74.125.224.178): icmp_seq=3 ttl=51 time=27.5 ms

--- www.google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2030ms
rtt min/avg/max/mdev = 27.559/28.571/29.517/0.824 ms
+ nslookup rac01
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   rac01.localdomain
Address: 192.168.56.11

+ nslookup rac02
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   rac02.localdomain
Address: 192.168.56.12

+ nslookup rac03
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   rac03.localdomain
Address: 192.168.56.13

+ nslookup www.google.com
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
Name:   www.google.com
Address: 74.125.224.179
Name:   www.google.com
Address: 74.125.224.177
Name:   www.google.com
Address: 74.125.224.180
Name:   www.google.com
Address: 74.125.224.176
Name:   www.google.com
Address: 74.125.224.178

+ nslookup dinh-scan
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   dinh-scan.localdomain
Address: 192.168.56.33
Name:   dinh-scan.localdomain
Address: 192.168.56.31
Name:   dinh-scan.localdomain
Address: 192.168.56.32

+ nslookup dinh-scan
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   dinh-scan.localdomain
Address: 192.168.56.31
Name:   dinh-scan.localdomain
Address: 192.168.56.32
Name:   dinh-scan.localdomain
Address: 192.168.56.33

+ nslookup dinh-scan
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   dinh-scan.localdomain
Address: 192.168.56.32
Name:   dinh-scan.localdomain
Address: 192.168.56.33
Name:   dinh-scan.localdomain
Address: 192.168.56.31

[root@rac02 ~]#

!!! BACKUP!!!
!!! BACKUP!!! I FUBAR the install where ASM disks were listed as MEMBER versus CANDIDATE. !!! BACKUP!!!
!!! BACKUP!!!

Now’s a good time since everything looks good. Next step is to create ASM disks.

Best way to backup, shutdown VM copy ALL folders to new directory. To restore, copy everything back. Does not get easier.

Some times I feel like GUI, some times, I don’t.

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\falcon>d:

D:\>cd "VirtualBox VMs"

D:\VirtualBox VMs>dir
 Volume in drive D is SOFTWARE
 Volume Serial Number is 8A61-FD0C

 Directory of D:\VirtualBox VMs

04/24/2013  06:27 PM              .
04/24/2013  06:27 PM              ..
04/24/2013  07:51 AM              asm
04/24/2013  07:51 AM              GOLD
04/24/2013  07:37 PM              rac01
04/24/2013  07:37 PM              rac02
04/24/2013  07:37 PM              rac03
               0 File(s)              0 bytes
               7 Dir(s)  658,327,556,096 bytes free

D:\VirtualBox VMs>cd asm

D:\VirtualBox VMs\asm>dir
 Volume in drive D is SOFTWARE
 Volume Serial Number is 8A61-FD0C

 Directory of D:\VirtualBox VMs\asm

04/24/2013  07:51 AM              .
04/24/2013  07:51 AM              ..
               0 File(s)              0 bytes
               2 Dir(s)  658,327,556,096 bytes free

D:\VirtualBox VMs\asm>VBoxManage createhd --filename asm1.vdi --size 5120 --format VDI --variant Fixed
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Disk image created. UUID: 65b57fff-26d4-4691-9d29-54924b7a870f

D:\VirtualBox VMs\asm>VBoxManage createhd --filename asm2.vdi --size 5120 --format VDI --variant Fixed
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Disk image created. UUID: 345917d0-d033-4ef3-af85-f4a4cd82669f

D:\VirtualBox VMs\asm>VBoxManage createhd --filename asm3.vdi --size 5120 --format VDI --variant Fixed
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Disk image created. UUID: 40064c7e-5795-4544-908d-bb19d9545b02

D:\VirtualBox VMs\asm>VBoxManage createhd --filename asm4.vdi --size 5120 --format VDI --variant Fixed
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Disk image created. UUID: dc1320c0-34b7-46b7-afca-1ec2b5319aeb

D:\VirtualBox VMs\asm>VBoxManage createhd --filename asm5.vdi --size 5120 --format VDI --variant Fixed
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Disk image created. UUID: 6fb56b8d-76e5-43a9-865c-48b5aed5cdac

D:\VirtualBox VMs\asm>VBoxManage storageattach rac01 --storagectl "SATA" --port 1 --device 0 --type hdd --medium asm1.vdi --mtype shareable

D:\VirtualBox VMs\asm>VBoxManage storageattach rac01 --storagectl "SATA" --port 2 --device 0 --type hdd --medium asm2.vdi --mtype shareable

D:\VirtualBox VMs\asm>VBoxManage storageattach rac01 --storagectl "SATA" --port 3 --device 0 --type hdd --medium asm3.vdi --mtype shareable

D:\VirtualBox VMs\asm>VBoxManage storageattach rac01 --storagectl "SATA" --port 4 --device 0 --type hdd --medium asm4.vdi --mtype shareable

D:\VirtualBox VMs\asm>VBoxManage storageattach rac01 --storagectl "SATA" --port 5 --device 0 --type hdd --medium asm5.vdi --mtype shareable

D:\VirtualBox VMs\asm>VBoxManage modifyhd asm1.vdi --type shareable

D:\VirtualBox VMs\asm>VBoxManage modifyhd asm2.vdi --type shareable

D:\VirtualBox VMs\asm>VBoxManage modifyhd asm3.vdi --type shareable

D:\VirtualBox VMs\asm>VBoxManage modifyhd asm4.vdi --type shareable

D:\VirtualBox VMs\asm>VBoxManage modifyhd asm5.vdi --type shareable

D:\VirtualBox VMs\asm>

Verify partitions

[root@rac01 ~]# /sbin/partprobe /dev/sdb
[root@rac01 ~]# /sbin/partprobe /dev/sdc
[root@rac01 ~]# /sbin/partprobe /dev/sdd
[root@rac01 ~]# /sbin/partprobe /dev/sde
[root@rac01 ~]# /sbin/partprobe /dev/sdf
[root@rac01 ~]# ll /dev/sd*
brw-rw---- 1 root disk 8,  0 Apr 24 19:44 /dev/sda
brw-rw---- 1 root disk 8,  1 Apr 24 19:44 /dev/sda1
brw-rw---- 1 root disk 8,  2 Apr 24 19:44 /dev/sda2
brw-rw---- 1 root disk 8, 16 Apr 24 19:49 /dev/sdb
brw-rw---- 1 root disk 8, 17 Apr 24 19:49 /dev/sdb1
brw-rw---- 1 root disk 8, 32 Apr 24 19:49 /dev/sdc
brw-rw---- 1 root disk 8, 33 Apr 24 19:49 /dev/sdc1
brw-rw---- 1 root disk 8, 48 Apr 24 19:49 /dev/sdd
brw-rw---- 1 root disk 8, 49 Apr 24 19:49 /dev/sdd1
brw-rw---- 1 root disk 8, 64 Apr 24 19:49 /dev/sde
brw-rw---- 1 root disk 8, 65 Apr 24 19:49 /dev/sde1
brw-rw---- 1 root disk 8, 80 Apr 24 19:49 /dev/sdf
brw-rw---- 1 root disk 8, 81 Apr 24 19:49 /dev/sdf1
[root@rac01 ~]# vi /etc/udev/rules.d/99-oracle-asmdevices.rules
[root@rac01 ~]# /sbin/udevadm control --reload-rules
[root@rac01 ~]# /sbin/start_udev
Starting udev:                                             [  OK  ]
[root@rac01 ~]# ls -al /dev/asm*
brw-rw---- 1 oracle dba 8, 17 Apr 24 19:54 /dev/asm-disk1
brw-rw---- 1 oracle dba 8, 33 Apr 24 19:54 /dev/asm-disk2
brw-rw---- 1 oracle dba 8, 49 Apr 24 19:54 /dev/asm-disk3
brw-rw---- 1 oracle dba 8, 65 Apr 24 19:54 /dev/asm-disk4
brw-rw---- 1 oracle dba 8, 81 Apr 24 19:54 /dev/asm-disk5

OOPS! Need to shutdown rac01

D:\VirtualBox VMs\asm>VBoxManage storageattach rac02 --storagectl "SATA" --port 1 --device 0 --type hdd --medium asm1.vdi --mtype shareable
VBoxManage.exe: error: Medium 'D:\VirtualBox VMs\asm\asm1.vdi' is locked for reading by another task
VBoxManage.exe: error: Details: code VBOX_E_INVALID_OBJECT_STATE (0x80bb0007), component Medium, interface IMedium, callee IUnknown
VBoxManage.exe: error: Context: "COMSETTER(Type)(mediumType)" at line 678 of file VBoxManageStorageController.cpp
VBoxManage.exe: error: Failed to set the medium type

Attach shared disks to VM

D:\VirtualBox VMs\asm>VBoxManage storageattach rac02 --storagectl "SATA" --port 1 --device 0 --type hdd --medium asm1.vdi --mtype shareable
D:\VirtualBox VMs\asm>VBoxManage storageattach rac02 --storagectl "SATA" --port 2 --device 0 --type hdd --medium asm2.vdi --mtype shareable
D:\VirtualBox VMs\asm>VBoxManage storageattach rac02 --storagectl "SATA" --port 3 --device 0 --type hdd --medium asm3.vdi --mtype shareable
D:\VirtualBox VMs\asm>VBoxManage storageattach rac02 --storagectl "SATA" --port 4 --device 0 --type hdd --medium asm4.vdi --mtype shareable
D:\VirtualBox VMs\asm>VBoxManage storageattach rac02 --storagectl "SATA" --port 5 --device 0 --type hdd --medium asm5.vdi --mtype shareable

D:\VirtualBox VMs\asm>VBoxManage storageattach rac03 --storagectl "SATA" --port 1 --device 0 --type hdd --medium asm1.vdi --mtype shareable
D:\VirtualBox VMs\asm>VBoxManage storageattach rac03 --storagectl "SATA" --port 2 --device 0 --type hdd --medium asm2.vdi --mtype shareable
D:\VirtualBox VMs\asm>VBoxManage storageattach rac03 --storagectl "SATA" --port 3 --device 0 --type hdd --medium asm3.vdi --mtype shareable
D:\VirtualBox VMs\asm>VBoxManage storageattach rac03 --storagectl "SATA" --port 4 --device 0 --type hdd --medium asm4.vdi --mtype shareable
D:\VirtualBox VMs\asm>VBoxManage storageattach rac03 --storagectl "SATA" --port 5 --device 0 --type hdd --medium asm5.vdi --mtype shareable

cluster

VERIFY ASM DISKS FOR ALL NODES – TRUST ME! YOU DON’T WANT TO FIND OUT AFTER THE FACT.

[oracle@rac01:]/home/oracle
$ ls -al /dev/asm*
brw-rw---- 1 oracle dba 8, 17 Apr 25 20:34 /dev/asm-disk1
brw-rw---- 1 oracle dba 8, 33 Apr 25 20:34 /dev/asm-disk2
brw-rw---- 1 oracle dba 8, 49 Apr 25 20:34 /dev/asm-disk3
brw-rw---- 1 oracle dba 8, 65 Apr 25 20:34 /dev/asm-disk4
brw-rw---- 1 oracle dba 8, 81 Apr 25 20:34 /dev/asm-disk5

[oracle@rac02:]/home/oracle
$ ls -al /dev/asm*
brw-rw---- 1 oracle dba 8, 17 Apr 25 20:34 /dev/asm-disk1
brw-rw---- 1 oracle dba 8, 33 Apr 25 20:34 /dev/asm-disk2
brw-rw---- 1 oracle dba 8, 49 Apr 25 20:34 /dev/asm-disk3
brw-rw---- 1 oracle dba 8, 65 Apr 25 20:34 /dev/asm-disk4
brw-rw---- 1 oracle dba 8, 81 Apr 25 20:34 /dev/asm-disk5

[oracle@rac03:]/home/oracle
$ ls -al /dev/asm*
brw-rw---- 1 oracle dba 8, 17 Apr 25 20:34 /dev/asm-disk1
brw-rw---- 1 oracle dba 8, 33 Apr 25 20:34 /dev/asm-disk2
brw-rw---- 1 oracle dba 8, 49 Apr 25 20:34 /dev/asm-disk3
brw-rw---- 1 oracle dba 8, 65 Apr 25 20:34 /dev/asm-disk4
brw-rw---- 1 oracle dba 8, 81 Apr 25 20:34 /dev/asm-disk5

NOTE: I tried to install everything under oracle and was struggling.
If I am going to create a environment, then why not create it production like!
Using grid and NOT oracle for grid infrastructure instal.

[grid@rac01:]/home/grid
$ ll /dev/asm*;id
brw-rw---- 1 grid asmadmin 8, 17 Apr 28 14:49 /dev/asm-disk1
brw-rw---- 1 grid asmadmin 8, 33 Apr 28 14:49 /dev/asm-disk2
brw-rw---- 1 grid asmadmin 8, 49 Apr 28 14:49 /dev/asm-disk3
brw-rw---- 1 grid asmadmin 8, 65 Apr 28 14:44 /dev/asm-disk4
brw-rw---- 1 grid asmadmin 8, 81 Apr 28 14:44 /dev/asm-disk5
uid=54322(grid) gid=54322(dba) groups=54322(dba),54321(oinstall),54323(vboxsf),54324(asmoper),54325(asmadmin),54326(asmdba)

[grid@rac02:]/home/grid
$ ll /dev/asm*;id
brw-rw---- 1 grid asmadmin 8, 17 Apr 28 14:49 /dev/asm-disk1
brw-rw---- 1 grid asmadmin 8, 33 Apr 28 14:49 /dev/asm-disk2
brw-rw---- 1 grid asmadmin 8, 49 Apr 28 14:49 /dev/asm-disk3
brw-rw---- 1 grid asmadmin 8, 65 Apr 28 14:44 /dev/asm-disk4
brw-rw---- 1 grid asmadmin 8, 81 Apr 28 14:44 /dev/asm-disk5
uid=54322(grid) gid=54322(dba) groups=54322(dba),54321(oinstall),54323(vboxsf),54324(asmoper),54325(asmadmin),54326(asmdba)

[grid@rac03:]/home/grid
$ ll /dev/asm*;id
brw-rw---- 1 grid asmadmin 8, 17 Apr 28 14:49 /dev/asm-disk1
brw-rw---- 1 grid asmadmin 8, 33 Apr 28 14:49 /dev/asm-disk2
brw-rw---- 1 grid asmadmin 8, 49 Apr 28 14:49 /dev/asm-disk3
brw-rw---- 1 grid asmadmin 8, 65 Apr 28 14:44 /dev/asm-disk4
brw-rw---- 1 grid asmadmin 8, 81 Apr 28 14:44 /dev/asm-disk5
uid=54322(grid) gid=54322(dba) groups=54322(dba),54321(oinstall),54323(vboxsf),54324(asmoper),54325(asmadmin),54326(asmdba)

Just don’t cut and paste and follow. Do some digging and googling.
Pretty cool features, eh.

[oracle@rac01:]/media/sf_Linux/grid/sshsetup
$ ./sshUserSetup.sh
Please specify a valid and existing cluster configuration file.
Either user name or host information is missing
Usage ./sshUserSetup.sh -user  [ -hosts "" | -hostfile  ] [ -advanced ]  [ -verify] [ -exverify ] [ -logfile  ] [-confirm] [-shared] [-help] [-usePassphrase] [-noPromptPassphrase]
[oracle@rac01:]/media/sf_Linux/grid/sshsetup

$ /media/sf_Linux/grid/sshsetup/sshUserSetup.sh -user oracle -hosts "rac01 rac02 rac03" -noPromptPassphrase -confirm -advanced -exverify
The output of this script is also logged into /tmp/sshUserSetup_2013-04-25-20-47-55.log
Hosts are rac01 rac02 rac03
user is oracle
Platform:- Linux

/media/sf_Linux/grid/runcluvfy.sh stage -pre crsinst -n rac01,rac02,rac03 = That’s how I found out about

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
NTP Configuration file check passed
No NTP Daemons or Services were found to be running
PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s):
rac03,rac02,rac01
Clock synchronization check using Network Time Protocol(NTP) failed

mv /etc/ntp.conf /etc/ntp.conf.org

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
No NTP Daemons or Services were found to be running

Clock synchronization check using Network Time Protocol(NTP) passed

Face it. Not every system is going to be perfect.
cluvfy is your friend http://docs.oracle.com/cd/E11882_01/rac.112/e16794/cvu.htm

Script it

[root@rac03 ~]# cat crdir.sh
mkdir -p /u01/app/oraInventory
chown -R grid:oinstall /u01/app/oraInventory

mkdir -p /u01/app/grid
chown -R grid:oinstall /u01/app/grid
mkdir -p /u01/app/11.2.0.3/grid
chown -R grid:oinstall /u01/app/11.2.0.3

mkdir -p /u01/app/oracle
mkdir -p /u01/app/oracle/cfgtoollogs
mkdir -p /u01/app/oracle/product/11.2.0.3/dbhome_1
chown -R oracle:oinstall /u01/app/oracle

chmod -R 775 /u01

!!! Primary group for grid must be dba, not oinstall for my 11.2.0.3 !!!
!!! Must let root.sh complete on the first node, then parallel !!!

SAVE the file

$ You can find the log of this install session at:
 /u01/app/oraInventory/logs/installActions2013-04-26_08-50-45PM.log

That’s messed up! I build cluster as rac03,02,01
But ran root.sh in order rac03,01,02 – evident by time for /usr/local/bin

[root@rac03 ~]# cat /etc/oratab
#Backup file is  /u01/app/11.2.0.3/grid/srvm/admin/oratab.bak.rac03 line added by Agent
+ASM1:/u01/app/11.2.0.3/grid:N          # line added by Agent

$ cat /etc/oratab
#Backup file is  /u01/app/11.2.0.3/grid/srvm/admin/oratab.bak.rac01 line added by Agent
+ASM2:/u01/app/11.2.0.3/grid:N          # line added by Agent
[grid@rac01:]/home/grid
$ ll /usr/local/bin/
total 20
-rwxr-xr-x 1 grid root 5778 Apr 26 21:13 coraenv
-rwxr-xr-x 1 grid root 2415 Apr 26 21:13 dbhome
-rwxr-xr-x 1 grid root 6183 Apr 26 21:13 oraenv

[root@rac02 ~]# cat /etc/oratab
#Backup file is  /u01/app/11.2.0.3/grid/srvm/admin/oratab.bak.rac02 line added by Agent
+ASM3:/u01/app/11.2.0.3/grid:N          # line added by Agent
[root@rac02 ~]# ll /usr/local/bin/
total 20
-rwxr-xr-x 1 grid root 5778 Apr 26 21:14 coraenv
-rwxr-xr-x 1 grid root 2415 Apr 26 21:14 dbhome
-rwxr-xr-x 1 grid root 6183 Apr 26 21:14 oraenv

What’s my cluster and scan listener?

$ olsnodes -help

Usage: olsnodes [ [-n] [-i] [-s] [-t] [ | -l [-p]] | [-c] ] [-g] [-v]
        where
                -n print node number with the node name
                -p print private interconnect address for the local node
                -i print virtual IP address with the node name
                 print information for the specified node
                -l print information for the local node
                -s print node status - active or inactive
                -t print node type - pinned or unpinned
                -g turn on logging
                -v Run in debug mode; use at direction of Oracle Support only.
                -c print clusterware name

[grid@rac01:+ASM2]/u01/app/11.2.0.3/grid/bin
$ olsnodes
rac03
rac01
rac02

[grid@rac01:+ASM2]/u01/app/11.2.0.3/grid/bin
$ olsnodes -c
dinh-cluster

[grid@rac01:+ASM2]/u01/app/11.2.0.3/grid/bin
$ olsnodes -n -i -s -t
rac03   1       rac03-vip       Active  Unpinned
rac01   2       rac01-vip       Active  Unpinned
rac02   3       rac02-vip       Active  Unpinned

[grid@rac01:+ASM2]/u01/app/11.2.0.3/grid/bin
$ olsnodes -n -i -s -t -l -p
rac01   2       10.10.10.11     rac01-vip       Active  Unpinned

$ srvctl config -help

The SRVCTL config command displays the configuration for the object stored in the OCR.

Usage: srvctl config database [-d  [-a] ] [-v]
Usage: srvctl config service -d  [-s ] [-v]
Usage: srvctl config nodeapps [-a] [-g] [-s]
Usage: srvctl config vip { -n  | -i  }
Usage: srvctl config network [-k ]
Usage: srvctl config asm [-a]
Usage: srvctl config listener [-l ] [-a]
Usage: srvctl config scan [-i ]
Usage: srvctl config scan_listener [-i ]
Usage: srvctl config srvpool [-g ]
Usage: srvctl config oc4j
Usage: srvctl config filesystem -d 
Usage: srvctl config gns [-a] [-d] [-k] [-m] [-n ] [-p] [-s] [-V] [-q ] [-l] [-v]
Usage: srvctl config cvu
For detailed help on each command and object and its options use:
  srvctl   -h
Usage (for backward compatibility): srvctl config
Usage (for backward compatibility): srvctl config -p  -n 
Usage (for backward compatibility): srvctl config -V
[grid@rac01:+ASM2]/u01/app/11.2.0.3/grid/bin

$ srvctl config nodeapps
Network exists: 1/192.168.56.0/255.255.255.0/eth1, type static
VIP exists: /rac01-vip/192.168.56.21/192.168.56.0/255.255.255.0/eth1, hosting node rac01
VIP exists: /rac02-vip/192.168.56.22/192.168.56.0/255.255.255.0/eth1, hosting node rac02
VIP exists: /rac03-vip/192.168.56.23/192.168.56.0/255.255.255.0/eth1, hosting node rac03
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016

[grid@rac01:+ASM2]/u01/app/11.2.0.3/grid/bin
$ srvctl config network
Network exists: 1/192.168.56.0/255.255.255.0/eth1, type static

[grid@rac01:+ASM2]/u01/app/11.2.0.3/grid/bin
$ srvctl config asm
ASM home: /u01/app/11.2.0.3/grid
ASM listener: LISTENER

[grid@rac01:+ASM2]/u01/app/11.2.0.3/grid/bin
$ srvctl config listener
Name: LISTENER
Network: 1, Owner: grid
Home: 
End points: TCP:1521

[grid@rac01:+ASM2]/u01/app/11.2.0.3/grid/bin
$ srvctl config scan
SCAN name: dinh-scan, Network: 1/192.168.56.0/255.255.255.0/eth1
SCAN VIP name: scan1, IP: /dinh-scan/192.168.56.33
SCAN VIP name: scan2, IP: /dinh-scan/192.168.56.31
SCAN VIP name: scan3, IP: /dinh-scan/192.168.56.32

[grid@rac01:+ASM2]/u01/app/11.2.0.3/grid/bin
$ srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1551
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1551
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1551

$ srvctl config srvpool
Server pool name: Free
Importance: 0, Min: 0, Max: -1
Candidate server names:
Server pool name: Generic
Importance: 0, Min: 0, Max: -1
Candidate server names:

[grid@rac01:+ASM2]/u01/app/11.2.0.3/grid/bin
$ srvctl config cvu
CVU is configured to run once every 360 minutes

Where’s my telnet? – yum install telnet -y
Notice that scan tried all 3 addresses

[root@rac02 ~]# telnet dinh-scan
Trying 192.168.56.31...
telnet: connect to address 192.168.56.31: Connection refused
Trying 192.168.56.33...
telnet: connect to address 192.168.56.33: Connection refused
Trying 192.168.56.32...
telnet: connect to address 192.168.56.32: Connection refused

[root@rac02 ~]# telnet dinh-scan 1551
Trying 192.168.56.31...
Connected to dinh-scan.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

Shutting down cluster – Good Night

[root@rac03 ~]# /u01/app/11.2.0.3/grid/bin/crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'rac03'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac03'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac03'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rac03'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rac03'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac03'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac03'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac03' succeeded
CRS-2673: Attempting to stop 'ora.rac03.vip' on 'rac03'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rac03' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rac03'
CRS-2677: Stop of 'ora.scan3.vip' on 'rac03' succeeded
CRS-2677: Stop of 'ora.rac03.vip' on 'rac03' succeeded
CRS-2673: Attempting to stop 'ora.crsd' on 'rac02'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rac02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac01'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac01'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac02'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.rac02.vip' on 'rac02'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac01' succeeded
CRS-2677: Stop of 'ora.rac02.vip' on 'rac02' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.rac01.vip' on 'rac01'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rac02'
CRS-2677: Stop of 'ora.rac01.vip' on 'rac01' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rac02' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rac03' succeeded
CRS-2677: Stop of 'ora.cvu' on 'rac03' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rac03' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac03'
CRS-2677: Stop of 'ora.asm' on 'rac03' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac01'
CRS-2677: Stop of 'ora.DATA.dg' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac02'
CRS-2677: Stop of 'ora.asm' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac01'
CRS-2677: Stop of 'ora.asm' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac02'
CRS-2677: Stop of 'ora.ons' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac01'
CRS-2677: Stop of 'ora.net1.network' on 'rac01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac01' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac01'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac01'
CRS-2673: Attempting to stop 'ora.asm' on 'rac01'
CRS-2677: Stop of 'ora.ons' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac02'
CRS-2677: Stop of 'ora.net1.network' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac03'
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac02' has completed
CRS-2677: Stop of 'ora.ons' on 'rac03' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac03'
CRS-2677: Stop of 'ora.net1.network' on 'rac03' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac03' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac02'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac02'
CRS-2673: Attempting to stop 'ora.asm' on 'rac02'
CRS-2677: Stop of 'ora.crsd' on 'rac03' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac03'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac03'
CRS-2673: Attempting to stop 'ora.asm' on 'rac03'
CRS-2677: Stop of 'ora.evmd' on 'rac01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac02' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac03' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac01' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac02' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac03' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac01'
CRS-2677: Stop of 'ora.asm' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac02'
CRS-2677: Stop of 'ora.asm' on 'rac03' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac03'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac02'
CRS-2677: Stop of 'ora.cssd' on 'rac01' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac03' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac03'
CRS-2677: Stop of 'ora.cssd' on 'rac02' succeeded
CRS-2677: Stop of 'ora.cssd' on 'rac03' succeeded
[root@rac03 ~]#

CRAP! CRAP! CRAP!
I proceed to install database software and received insufficient space error.
There was only 4GB left and that not enough since software requires 4.7GB if memory serves me correctly.
After some digging, yum was taking up some space.
Next time, allocate 16GB and not 12GB

yum clean all

[root@rac01 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg00-lv_root
                       12G  6.9G  4.0G  64% /
tmpfs                1004M  212M  792M  22% /dev/shm
/dev/sda1             485M   55M  405M  12% /boot
Linux                 466G   45G  422G  10% /media/sf_Linux
[root@rac01 ~]# cd /var/cache
[root@rac01 cache]# du -sh *
4.0K    abrt-di
12K     cups
72K     fontconfig
4.0K    foomatic
16K     gdm
328K    hald
40K     ldconfig
544K    man
96K     PackageKit
12K     rpcbind
1.4G    yum
[root@rac01 cache]# yum clean all
Loaded plugins: refresh-packagekit, security
Cleaning repos: ol6_UEK_latest ol6_latest
Cleaning up Everything
[root@rac01 cache]# du -sh *
4.0K    abrt-di
12K     cups
72K     fontconfig
4.0K    foomatic
16K     gdm
328K    hald
40K     ldconfig
544K    man
96K     PackageKit
12K     rpcbind
36K     yum
[root@rac01 cache]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg00-lv_root
                       12G  5.6G  5.3G  52% /
tmpfs                1004M  212M  792M  22% /dev/shm
/dev/sda1             485M   55M  405M  12% /boot
Linux                 466G   45G  422G  10% /media/sf_Linux
[root@rac01 cache]#

dba

From Metalink Step By Step guide for installing Oracle RAC 11gR2 (11.2.0.2) on Linux
#/usr/sbin/useradd -u 501 -g oinstall -G asmadmin,asmdba,asmoper grid
#/usr/sbin/useradd -u 502 -g oinstall -G dba,asmdba oracle

Even Document 1442768.1 – RACGuide: 11.2.0.3 RAC Installation on Linux [Video] add oinstall as primary.

PRIMARY GROUP NEEDS TO BE DBA for my 11.2.0.3 – VERY STRANGE.

[oracle@rac01:]/home/oracle
$ id
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54323(vboxsf),54326(asmdba)
[oracle@rac01:]/home/oracle
$ logout
[root@rac01 cache]# /tmp/CVU_11.2.0.3.0_oracle/runfixup.sh
Response file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.response
Enable file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.enable
Log file location: /tmp/CVU_11.2.0.3.0_oracle/orarun.log
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54323(vboxsf),54326(asmdba)
[root@rac01 cache]# su - oracle
[oracle@rac01:]/home/oracle
$ id
uid=54321(oracle) gid=54322(dba) groups=54322(dba),54321(oinstall),54323(vboxsf),54326(asmdba)

[oracle@rac02:]/home/oracle
$ id
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54323(vboxsf),54326(asmdba)
[oracle@rac02:]/home/oracle
$ logout
[root@rac02 ~]# /tmp/CVU_11.2.0.3.0_oracle/runfixup.sh
Response file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.response
Enable file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.enable
Log file location: /tmp/CVU_11.2.0.3.0_oracle/orarun.log
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54323(vboxsf),54326(asmdba)
[root@rac02 ~]# su - oracle
[oracle@rac02:]/home/oracle
$ id
uid=54321(oracle) gid=54322(dba) groups=54322(dba),54321(oinstall),54323(vboxsf),54326(asmdba)

[oracle@rac03:]/home/oracle
$ id
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54323(vboxsf),54326(asmdba)
[oracle@rac03:]/home/oracle
$ logout
[root@rac03 ~]# /tmp/CVU_11.2.0.3.0_oracle/runfixup.sh
Response file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.response
Enable file being used is :/tmp/CVU_11.2.0.3.0_oracle/fixup.enable
Log file location: /tmp/CVU_11.2.0.3.0_oracle/orarun.log
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54323(vboxsf),54326(asmdba)
[root@rac03 ~]# su - oracle
[oracle@rac03:]/home/oracle
$ id
uid=54321(oracle) gid=54322(dba) groups=54322(dba),54321(oinstall),54323(vboxsf),54326(asmdba)

LOGOUT and LOGIN again; otherwise, error as shown below.
oinstall

When cloning VM, Don’t forget to change hostname and add to /etc/sysconfig/network
NOZEROCONF=yes

Run raccheck and modify accordingly.

WARNING => avahi-daemon process is running
CSSD Fails to Join the Cluster After Private Network Recovered if avahi Daemon is up and Running [ID 1501093.1] Modified:Dec 5, 2012
# /etc/init.d/avahi-daemon stop
# /sbin/chkconfig avahi-daemon off

Don’t forget to change hostname and add to /etc/sysconfig/network
NOZEROCONF=yes

WARNING => kernel parameter rp_filter is set to 1
rp_filter for multiple private interconnects and Linux Kernel 2.6.32+ [ID 1286796.1] Modified:Dec 3, 2012

# grep rp_filter /etc/sysctl.conf
net.ipv4.conf.default.rp_filter = 1

Changed to net.ipv4.conf.default.rp_filter = 0

WARNING => kernel.shmmax parameter is NOT configured according to recommendation
# grep kernel.shmmax /etc/sysctl.conf
# oracle-rdbms-server-11gR2-preinstall setting for kernel.shmmax is 4398046511104 on x86_64
# oracle-rdbms-server-11gR2-preinstall setting for kernel.shmmax is 4294967295 on i386
kernel.shmmax = 4398046511104

To be continued …

8 Comments »

  1. Hi,
    I have followed this wonderful post to get a 2-node RAC working but after a week, I am still stuck at being able to ping the other node.
    Firewall is disabled on both nodes but I am still getting:

    [root@dap-rac2 etc]# ping -c 3 dap-rac1
    PING dap-rac1.localdomain (169.254.36.101) 56(84) bytes of data.
    From dap-rac2.localdomain (169.254.36.102) icmp_seq=1 Destination Host Unreachable
    From dap-rac2.localdomain (169.254.36.102) icmp_seq=2 Destination Host Unreachable
    From dap-rac2.localdomain (169.254.36.102) icmp_seq=3 Destination Host Unreachable

    — dap-rac1.localdomain ping statistics —
    3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3000ms
    pipe 3

    Please help!

    Comment by dapogbite — September 18, 2013 @ 2:02 pm | Reply

    • Please check your gateway. Are you able to ping dap-rac2 from dap-rac1?

      Comment by mdinh — September 23, 2013 @ 1:33 am | Reply

  2. Could you please seme me the details of the ifconfig -a from your nodes. I reckon it is an issue with nework config but I can’t figure out what the problems is.

    Thanks and best regards

    Comment by dapo gbite — October 3, 2013 @ 12:45 pm | Reply

    • I hope this helps you.

      Adapter 1 (NAT) – eth0 (DHCP)
      Adapter 2 (Host Only) – eth1 (STATIC IP)
      Adapter 3 (Internal) – eth2 (STATIC IP) (Private)

      [root@rac01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
      DEVICE=”eth0″
      NAME=”System eth0″
      BOOTPROTO=”dhcp”
      TYPE=”Ethernet”
      NM_CONTROLLED=”yes”
      ONBOOT=yes
      IPV4_FAILURE_FATAL=yes
      IPV6INIT=no
      DEFROUTE=yes
      PEERDNS=yes
      PEERROUTES=yes
      HWADDR=08:00:27:14:4F:98

      [root@rac01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
      DEVICE=”eth1″
      NAME=”System eth1″
      TYPE=”Ethernet”
      BOOTPROTO=none
      NM_CONTROLLED=”yes”
      ONBOOT=yes
      IPV4_FAILURE_FATAL=yes
      IPV6INIT=no
      DEFROUTE=yes
      PREFIX=24
      GATEWAY=192.168.56.1
      HWADDR=08:00:27:F9:E0:54
      IPADDR=192.168.56.11

      [root@rac01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth2
      DEVICE=”eth2″
      NAME=”System eth2″
      TYPE=”Ethernet”
      BOOTPROTO=none
      NM_CONTROLLED=”yes”
      ONBOOT=yes
      IPV4_FAILURE_FATAL=yes
      IPV6INIT=no
      DEFROUTE=yes
      PREFIX=8
      HWADDR=08:00:27:1E:77:37
      IPADDR=10.10.10.11

      [root@rac01 ~]# ifconfig -a
      eth0 Link encap:Ethernet HWaddr 08:00:27:14:4F:98
      inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0
      inet6 addr: fe80::a00:27ff:fe14:4f98/64 Scope:Link
      UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
      RX packets:4 errors:0 dropped:0 overruns:0 frame:0
      TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000
      RX bytes:2360 (2.3 KiB) TX bytes:2100 (2.0 KiB)

      eth1 Link encap:Ethernet HWaddr 08:00:27:F9:E0:54
      inet addr:192.168.56.11 Bcast:192.168.56.255 Mask:255.255.255.0
      inet6 addr: fe80::a00:27ff:fef9:e054/64 Scope:Link
      UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
      RX packets:369 errors:0 dropped:0 overruns:0 frame:0
      TX packets:220 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000
      RX bytes:36591 (35.7 KiB) TX bytes:28065 (27.4 KiB)

      eth2 Link encap:Ethernet HWaddr 08:00:27:1E:77:37
      inet addr:10.10.10.11 Bcast:10.255.255.255 Mask:255.0.0.0
      inet6 addr: fe80::a00:27ff:fe1e:7737/64 Scope:Link
      UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
      RX packets:4 errors:0 dropped:0 overruns:0 frame:0
      TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000
      RX bytes:240 (240.0 b) TX bytes:830 (830.0 b)

      lo Link encap:Local Loopback
      inet addr:127.0.0.1 Mask:255.0.0.0
      inet6 addr: ::1/128 Scope:Host
      UP LOOPBACK RUNNING MTU:16436 Metric:1
      RX packets:6 errors:0 dropped:0 overruns:0 frame:0
      TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:382 (382.0 b) TX bytes:382 (382.0 b)

      [root@rac01 ~]#

      [root@rac02 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
      DEVICE=”eth0″
      NAME=”System eth0″
      BOOTPROTO=”dhcp”
      TYPE=”Ethernet”
      NM_CONTROLLED=”yes”
      ONBOOT=yes
      IPV4_FAILURE_FATAL=yes
      IPV6INIT=no
      DEFROUTE=yes
      PEERDNS=yes
      PEERROUTES=yes
      HWADDR=08:00:27:36:3A:C9

      [root@rac02 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
      DEVICE=”eth1″
      NAME=”System eth1″
      TYPE=”Ethernet”
      BOOTPROTO=none
      NM_CONTROLLED=”yes”
      ONBOOT=yes
      IPV4_FAILURE_FATAL=yes
      IPV6INIT=no
      DEFROUTE=yes
      PREFIX=24
      GATEWAY=192.168.56.1
      HWADDR=08:00:27:BA:72:9B
      IPADDR=192.168.56.12

      [root@rac02 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth2
      DEVICE=”eth2″
      NAME=”System eth2″
      TYPE=”Ethernet”
      BOOTPROTO=none
      NM_CONTROLLED=”yes”
      ONBOOT=yes
      IPV4_FAILURE_FATAL=yes
      IPV6INIT=no
      DEFROUTE=yes
      PREFIX=8
      HWADDR=08:00:27:39:87:19
      IPADDR=10.10.10.12

      [root@rac02 ~]# ifconfig -a
      eth0 Link encap:Ethernet HWaddr 08:00:27:36:3A:C9
      inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0
      inet6 addr: fe80::a00:27ff:fe36:3ac9/64 Scope:Link
      UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
      RX packets:4 errors:0 dropped:0 overruns:0 frame:0
      TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000
      RX bytes:2360 (2.3 KiB) TX bytes:2100 (2.0 KiB)

      eth1 Link encap:Ethernet HWaddr 08:00:27:BA:72:9B
      inet addr:192.168.56.12 Bcast:192.168.56.255 Mask:255.255.255.0
      inet6 addr: fe80::a00:27ff:feba:729b/64 Scope:Link
      UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
      RX packets:277 errors:0 dropped:0 overruns:0 frame:0
      TX packets:167 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000
      RX bytes:27515 (26.8 KiB) TX bytes:22591 (22.0 KiB)

      eth2 Link encap:Ethernet HWaddr 08:00:27:39:87:19
      inet addr:10.10.10.12 Bcast:10.255.255.255 Mask:255.0.0.0
      inet6 addr: fe80::a00:27ff:fe39:8719/64 Scope:Link
      UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
      RX packets:6 errors:0 dropped:0 overruns:0 frame:0
      TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000
      RX bytes:360 (360.0 b) TX bytes:746 (746.0 b)

      lo Link encap:Local Loopback
      inet addr:127.0.0.1 Mask:255.0.0.0
      inet6 addr: ::1/128 Scope:Host
      UP LOOPBACK RUNNING MTU:16436 Metric:1
      RX packets:6 errors:0 dropped:0 overruns:0 frame:0
      TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:382 (382.0 b) TX bytes:382 (382.0 b)

      [root@rac02 ~]#

      [root@rac01 ~]# ping -c 3 rac01
      PING rac01 (192.168.56.11) 56(84) bytes of data.
      64 bytes from rac01 (192.168.56.11): icmp_seq=1 ttl=64 time=0.028 ms
      64 bytes from rac01 (192.168.56.11): icmp_seq=2 ttl=64 time=0.041 ms
      64 bytes from rac01 (192.168.56.11): icmp_seq=3 ttl=64 time=0.039 ms

      — rac01 ping statistics —
      3 packets transmitted, 3 received, 0% packet loss, time 2011ms
      rtt min/avg/max/mdev = 0.028/0.036/0.041/0.005 ms
      [root@rac01 ~]# ping -c 3 rac02
      PING rac02 (192.168.56.12) 56(84) bytes of data.
      64 bytes from rac02 (192.168.56.12): icmp_seq=1 ttl=64 time=0.360 ms
      64 bytes from rac02 (192.168.56.12): icmp_seq=2 ttl=64 time=0.577 ms
      64 bytes from rac02 (192.168.56.12): icmp_seq=3 ttl=64 time=0.510 ms

      — rac02 ping statistics —
      3 packets transmitted, 3 received, 0% packet loss, time 2008ms
      rtt min/avg/max/mdev = 0.360/0.482/0.577/0.092 ms
      [root@rac01 ~]#

      [root@rac02 ~]# ping -c 3 rac01
      PING rac01 (192.168.56.11) 56(84) bytes of data.
      64 bytes from rac01 (192.168.56.11): icmp_seq=1 ttl=64 time=0.525 ms
      64 bytes from rac01 (192.168.56.11): icmp_seq=2 ttl=64 time=0.451 ms
      64 bytes from rac01 (192.168.56.11): icmp_seq=3 ttl=64 time=0.447 ms

      — rac01 ping statistics —
      3 packets transmitted, 3 received, 0% packet loss, time 2053ms
      rtt min/avg/max/mdev = 0.447/0.474/0.525/0.040 ms
      [root@rac02 ~]# ping -c 3 rac02
      PING rac02 (192.168.56.12) 56(84) bytes of data.
      64 bytes from rac02 (192.168.56.12): icmp_seq=1 ttl=64 time=0.112 ms
      64 bytes from rac02 (192.168.56.12): icmp_seq=2 ttl=64 time=0.039 ms
      64 bytes from rac02 (192.168.56.12): icmp_seq=3 ttl=64 time=0.038 ms

      — rac02 ping statistics —
      3 packets transmitted, 3 received, 0% packet loss, time 2001ms
      rtt min/avg/max/mdev = 0.038/0.063/0.112/0.034 ms
      [root@rac02 ~]#

      Comment by mdinh — October 4, 2013 @ 10:56 pm | Reply

      • Thanks and I am really grateful for your time. I have compared and with the little change I made, I can now ping the node’s public IPs but I cannot ping the private IPs
        I have noticed this difference on my rac2 but don’t know where to change or where it went wrong:

        eth0 Link encap:Ethernet HWaddr 08:00:27:BD:5C:66
        inet6 addr: fe80::a00:27ff:febd:5c66/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
        RX packets:12 errors:0 dropped:0 overruns:0 frame:0
        TX packets:28 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:1000
        RX bytes:720 (720.0 b) TX bytes:5508 (5.3 KiB)

        This line is missing:
        inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0

        I reckon that may be causing the problem

        Comment by dapo gbite — October 8, 2013 @ 1:12 pm

  3. I don’t think private IPs are ping-able. Don’t have access to my system at the moment.

    [root@rac02 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth# to review eth configuration.

    Comment by mdinh — October 13, 2013 @ 12:53 pm | Reply

  4. Thanks so much.
    I made a 3rd clone and that works. I could have messed up the 2nd node somehow
    With your help and others mainly oraclebase.com, I have now installed my 2 node rac which I can now explore and hopefully, it will boost my chance for getting another job.
    I followed different notes which didn’t work for me for reasons which are network related but yours….having NAT,Host-only and internal adapter with the IPs appears to be the one that worked for me.

    Regards

    Comment by dapo gbite — October 16, 2013 @ 3:10 pm | Reply

    • Thanks for the feedback and I am glad that it helped.

      Comment by mdinh — October 21, 2013 @ 1:43 pm | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: