昨天处理了一起ora-00600的错误，其中也经历了各种曲折，真是雾里看花，看透了之后发现很多问题都是有原因的。起初是开发说有一个
昨天处理了一起ora-00600的错误，其中也经历了各种曲折，真是雾里看花，看透了之后发现很多问题都是有原因的。起初是开发说有一个job运行的时候报错了，oracle数据库版本是11.2.0.2.0
等到问题提交到我这，客户已经检查了一些信息了。但是还是没有结论。
对于这个问题，我还是照例开始检查数据库日志。
在那个时间段内里出现了ora-00600的错误。
wed jun 10 13:47:17 2015
errors in file /opt/app/oracle/dbccbspr1/diag/rdbms/prodb/prodb/trace/prodb_p070_1200.trc (incident=2124332):
ora-00600: internal error code, arguments: [srsnext_3], [], [], [], [], [], [], [], [], [], [], []
incident details in: /opt/app/oracle/dbccbspr1/diag/rdbms/prodb/prodb/incident/incdir_2124332/prodb_p070_1200_i2124332.trc
wed jun 10 13:48:04 2015
use adrci or support workbench to package the incident.
see note 411.1 at my oracle support for error and packaging details.
在这个ora-00600错误前后又碰到了几个奇怪的ora错误,我把错误日志按照时间先后来排列一下。
thread 1 advanced to log sequence 82829 (lgwr switch)
current log# 1 seq# 82829 mem# 0: /dbccbpr1/oracle/prodb/redolog_a1/redo/redo01a.log
current log# 1 seq# 82829 mem# 1: /dbccbpr1/oracle/prodb/redolog_b1/redo/redo01b.log
archived log entry 82894 added for thread 1 sequence 82828 id 0xb8c6d509 dest 1:
wed jun 10 07:10:17 2015
ora-00060: deadlock detected. more info in file /opt/app/oracle/dbccbspr1/diag/rdbms/prodb/prodb/trace/prodb_ora_18508.trc.
wed jun 10 07:10:21 2015
ora-00060: deadlock detected. more info in file /opt/app/oracle/dbccbspr1/diag/rdbms/prodb/prodb/trace/prodb_ora_18593.trc.
wed jun 10 07:12:14 2015
ora-00060: deadlock detected. more info in file /opt/app/oracle/dbccbspr1/diag/rdbms/prodb/prodb/trace/prodb_ora_16505.trc.
wed jun 10 07:26:54 2015
ora-00060: deadlock detected. more info in file /opt/app/oracle/dbccbspr1/diag/rdbms/prodb/prodb/trace/prodb_ora_18558.trc.
wed jun 10 07:52:47 2015
thread 1 advanced to log sequence 82830 (lgwr switch)
current log# 2 seq# 82830 mem# 0: /dbccbpr1/oracle/prodb/redolog_a2/redo/redo02a.log
current log# 2 seq# 82830 mem# 1: /dbccbpr1/oracle/prodb/redolog_b2/redo/redo02b.log
wed jun 10 12:09:40 2015
exception [type: sigsegv, address not mapped to object] [addr:0x210] [pc:0x193f4df, kxfpqrclb()+71] [flags: 0x0, count: 1]
errors in file /opt/app/oracle/dbccbspr1/diag/rdbms/prodb/prodb/trace/prodb_p135_15651.trc (incident=2116204):
ora-07445: exception encountered: core dump [kxfpqrclb()+71] [sigsegv] [addr:0x210] [pc:0x193f4df] [address not mapped to object] []
ora-10382: parallel query server interrupt (reset)
incident details in: /opt/app/oracle/dbccbspr1/diag/rdbms/prodb/prodb/incident/incdir_2116204/prodb_p135_15651_i2116204.trc
use adrci or support workbench to package the incident.
see note 411.1 at my oracle support for error and packaging details.
wed jun 10 12:09:41 2015
dumping diagnostic data in directory=[cdmp_20150610120941], requested by (instance=1, osid=15651 (p135)), summary=[incident=2116204].
wed jun 10 12:09:44 2015
sweep [inc][2116204]: completed
sweep [inc2][2116204]: completed
wed jun 10 12:09:59 2015
wed jun 10 13:46:27 2015
archived log entry 82967 added for thread 1 sequence 82901 id 0xb8c6d509 dest 1:
wed jun 10 13:47:17 2015
errors in file /opt/app/oracle/dbccbspr1/diag/rdbms/prodb/prodb/trace/prodb_p070_1200.trc (incident=2124332):
ora-00600: internal error code, arguments: [srsnext_3], [], [], [], [], [], [], [], [], [], [], []
incident details in: /opt/app/oracle/dbccbspr1/diag/rdbms/prodb/prodb/incident/incdir_2124332/prodb_p070_1200_i2124332.trc
wed jun 10 13:48:04 2015
archived log entry 82993 added for thread 1 sequence 82927 id 0xb8c6d509 dest 1:
wed jun 10 14:38:53 2015
errors in file /opt/app/oracle/dbccbspr1/diag/rdbms/prodb/prodb/trace/prodb_ora_15548.trc (incident=2110436):
ora-04030: out of process memory when trying to allocate 16328 bytes (koh-kghu sessi,pmucalm coll)
incident details in: /opt/app/oracle/dbccbspr1/diag/rdbms/prodb/prodb/incident/incdir_2110436/prodb_ora_15548_i2110436.trc
use adrci or support workbench to package the incident.
see note 411.1 at my oracle support for error and packaging details.
为了更加清晰，我把对应的ora错误和数据库负载联系在一起。
对于deadlock的错误，很可能是应用死锁造成的，简单查看了下trace日志，做了基本确认，就交给开发去分析这部分了。
从后续的错误情况来看，似乎和后续的问题没有直接关系。我们暂时先放下这个deaklock的错误。所以在图中没有标注出来。
其它三个都用红色标注出来。可以看出在负载开始增加的几个时间点里，依次发生了几个ora错误。