Solve the mystery – Database crash without a clue

March 19, 2013

Solve the mystery – Database crash without a clue

Filed under: oracle — mdinh @ 4:14 am

So there I was, troubleshooting root cause for database crash without a clue.

There are no trace files and no core dumps. The alert log is shown below?

What do you think happened?

Mar 18 20:50:08 2013
Thread 1 advanced to log sequence 5 (LGWR switch)
  Current log# 2 seq# 5 mem# 0: /oracle/flashrecovery/LAX_DB01/onlinelog/o1_mf_2_8nctxzw9_.log
Mon Mar 18 20:50:09 2013
Archived Log entry 56 added for thread 1 sequence 4 ID 0x55ed081e dest 1:
Mon Mar 18 20:55:18 2013
Adjusting the default value of parameter parallel_max_servers
from 40 to 35 due to the value of parameter processes (50)
Starting ORACLE instance (normal)
Mon Mar 18 20:55:27 2013
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
Started redo scan
Completed redo scan
 read 49 KB redo, 38 data blocks need recovery
Started redo application at
 Thread 1: logseq 4, block 3
Recovery of Online Redo Log: Thread 1 Group 1 Seq 4 Reading mem 0
  Mem# 0: /oracle/flashrecovery/LAX_DB01/onlinelog/o1_mf_1_8nctxrsl_.log
Recovery of Online Redo Log: Thread 1 Group 2 Seq 5 Reading mem 0
  Mem# 0: /oracle/flashrecovery/LAX_DB01/onlinelog/o1_mf_2_8nctxzw9_.log
Completed redo application of 0.04MB
Completed crash recovery at
 Thread 1: logseq 5, block 3, scn 642127
 38 data blocks read, 38 data blocks written, 49 redo k-bytes read
LGWR: STARTING ARCH PROCESSES

[oracle@lax:db01]/u01/app/oracle/diag/rdbms/lax_db01/db01/trace
$ last reboot|head -1
reboot system boot 2.6.39-200.24.1. Mon Mar 18 20:53 – 21:12 (00:19)

The system was rebooted !!!

About these ads

Comments (4)

4 Comments »

Use splunk and combine alert.log, audit files, ASM alert log, grid alert log, messages, listener.log, etc. for a broader view. If it’s a reboot, it’s visible. If it’s hardware related, and you’ve got a BMC, it can swallow SNMP too to.

From what I see, the DB did not shut down, so it started and did crash recovery. Which is consistent with a hard reboot, power fail, hardware crash etc.

Comment by Frits Hoogland — March 19, 2013 @ 7:38 am | Reply
- Thanks and hearing good things about splunk.
  
  Comment by mdinh — March 19, 2013 @ 3:11 pm | Reply
Hi,
looks like no ASM and ‘from 40 to 35 due to the value of parameter processes (50)’ processes is only 50 ?
What kind of DB is that ?
Regards
GregG

Comment by goryszewskig — March 19, 2013 @ 8:36 am | Reply
- Test DB on VM to reproduce the issue and last reboot command from Linux.
  
  Comment by mdinh — March 19, 2013 @ 3:10 pm | Reply