整理一下前段时间遇到的ora-15064错误。有一套测试环境是11gr2的rac使用asm，应用在测试时发现一个节点宕机了，检查了一下alert日
整理一下前段时间遇到的ora-15064错误。
有一套测试环境是11gr2的rac使用asm，应用在测试时发现一个节点宕机了，检查了一下alert日志如下错误：
mon jun 04 23:27:32 2012
note: asmb terminating
errors in file /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc:
ora-15064: communication failure with asm instance
ora-03113: end-of-file on communication channel
process id:
session id: 23 serial number: 1161
errors in file /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc:
ora-15064: communication failure with asm instance
ora-03113: end-of-file on communication channel
process id:
session id: 23 serial number: 1161
asmb (ospid: 13828564): terminating the instance due to error 15064
termination issued to instance processes. waiting for the processes to exit
mon jun 04 23:27:43 2012
instance termination failed to kill one or more processes
instance terminated by asmb, pid = 13828564
通过错误可知，改错无与asm有关，，应该是asm意外关闭才导致的instance关闭。
错误的trace文件内容如下：
[pt1rac2@root]# cat /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc
trace file /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc
oracle database 11g enterprise edition release 11.2.0.2.0 - 64bit production
with the partitioning, real application clusters, olap, data mining
and real application testing options
oracle_home = /soft/oracle/product/11.2.0.2/dbh
system name: aix
node name: pt1rac2
release: 1
version: 6
machine: 00f770e14c00
instance name: pt1db2
redo thread mounted by this instance: 0
oracle process number: 27
unix process pid: 13828564, image: oracle@pt1rac2 (asmb)
*** 2012-06-04 16:19:14.483
*** session id:(2998.1) 2012-06-04 16:19:14.483
*** client id:() 2012-06-04 16:19:14.483
*** service name:() 2012-06-04 16:19:14.483
*** module name:() 2012-06-04 16:19:14.483
*** action name:() 2012-06-04 16:19:14.483
note: initiating mark startup
*** 2012-06-04 16:19:18.290
mode of disk 0 (data_0000) changing from 0x7 to 0x7
*** 2012-06-04 23:07:12.286
mode of disk 0 (data_0000) changing from 0x7 to 0x7
*** 2012-06-04 23:27:32.492
note: asmb terminating
ora-15064: communication failure with asm instance
ora-03113: end-of-file on communication channel
process id:
session id: 23 serial number: 1161
error 15064 detected in background process
ora-15064: communication failure with asm instance
ora-03113: end-of-file on communication channel
process id:
session id: 23 serial number: 1161
kjzduptcctx: notifying diag for crash event
----- abridged call stack trace -----
ksedsts()+644
----- end of abridged call stack trace -----
*** 2012-06-04 23:27:32.854
asmb (ospid: 13828564): terminating the instance due to error 15064
*** 2012-06-04 23:27:43.387
instance termination failed to kill one or more processes
可知的确是由于asm意外关闭才导致的instance关闭，asm意外关闭的原因继续查。
准备生产一下awr报告看一下，结果发现由于数据关闭了，没有awr可看。
一下数据说明数据库被重启过几次，且经过与应用人员确认，这些重启的时间点他们都在做压力测试，因此初步怀疑是和性能有关，继续查。
snap
instance db name snap id snap started level
------------ ------------ --------- ------------------ -----
pt1db2 pt1db 503 02 jun 2012 16:00 1
504 02 jun 2012 17:00 1
505 04 jun 2012 09:43 1
506 04 jun 2012 11:00 1
507 04 jun 2012 12:00 1
508 04 jun 2012 13:00 1
509 04 jun 2012 14:00 1
510 04 jun 2012 15:00 1
511 04 jun 2012 16:00 1
512 04 jun 2012 17:00 1
513 04 jun 2012 18:00 1
514 04 jun 2012 19:00 1
515 04 jun 2012 20:00 1
516 04 jun 2012 21:00 1
517 04 jun 2012 22:00 1
518 04 jun 2012 23:00 1
522 05 jun 2012 01:20 1
523 05 jun 2012 01:30 1
524 05 jun 2012 01:45 1
先查看ocssd.log日志
2012-06-04 23:26:02.274: [ cssd][1029]clssscmonitorthreads clssnmvdiskpingthread not scheduled for 113598 msecs
2012-06-04 23:26:03.126: [ cssd][1029]clssscselect: cookie accept request 110abb440
2012-06-04 23:26:03.126: [ cssd][1029]clssgmallocproc: (11328bcd0) allocated
2012-06-04 23:26:03.127: [ cssd][1029]clssgmclientconnectmsg: properties of cmproc 11328bcd0 - 0,1,2,3,4
2012-06-04 23:26:03.127: [ cssd][1029]clssgmclientconnectmsg: connect from con(2ea97f) proc(11328bcd0) pid(15335558) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:03.127: [ cssd][1029]clssgmclientconnectmsg: msg flags 0x0000
2012-06-04 23:26:03.274: [ cssd][3862]clssnmsendingthread: sending status msg to all nodes
2012-06-04 23:26:03.274: [ cssd][3862]clssnmsendingthread: sent 4 status msgs to all nodes
2012-06-04 23:26:03.309: [ cssd][1029]clssscselect: cookie accept request 110abb440
2012-06-04 23:26:03.309: [ cssd][1029]clssgmallocproc: (1132d30d0) allocated
2012-06-04 23:26:03.310: [ cssd][1029]clssgmclientconnectmsg: properties of cmproc 1132d30d0 - 0,1,2,3,4
2012-06-04 23:26:03.310: [ cssd][1029]clssgmclientconnectmsg: connect from con(2ea9fa) proc(1132d30d0) pid(9306434/9306434) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:03.310: [ cssd][1029]clssgmclientconnectmsg: msg flags 0x0000
2012-06-04 23:26:03.315: [ cssd][1029]clssscselect: cookie accept request 1132d30d0
2012-06-04 23:26:03.315: [ cssd][1029]clssscevtypshrcon: getting client with cmproc 1132d30d0
2012-06-04 23:26:03.316: [ cssd][1029]clssgmregisterclient: proc(57/1132d30d0), client(1/11313ef70)
2012-06-04 23:26:03.316: [ cssd][1029]clssgmregistershared: grp dg_data, mbr 1, type 1
。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。
2012-06-04 23:26:29.658: [ cssd][1029]clssgmdeadproc: proc 112f32d30
2012-06-04 23:26:29.658: [ cssd][1029]clssgmdestroyproc: cleaning up proc(112f32d30) con(2ea049) skgpid ospid 35520704 with 0 clients, refcount 0
2012-06-04 23:26:29.658: [ cssd][1029]clssgmdiscendpcl: gipcdestroy 2ea049
2012-06-04 23:26:29.675: [ cssd][1029]clssscselect: cookie accept request 110abb440
2012-06-04 23:26:29.675: [ cssd][1029]clssgmallocproc: (113242130) allocated
2012-06-04 23:26:29.676: [ cssd][1029]clssgmclientconnectmsg: properties of cmproc 113242130 - 0,1,2,3,4
2012-06-04 23:26:29.676: [ cssd][1029]clssgmclientconnectmsg: connect from con(2eadef) proc(113242130) pid(10879472) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:29.676: [ cssd][1029]clssgmclientconnectmsg: msg flags 0x0000
2012-06-04 23:26:30.226: [ cssd][1029]clssscselect: cookie accept request 110abb440
2012-06-04 23:26:30.226: [ cssd][1029]clssgmallocproc: (1132a3390) allocated
2012-06-04 23:26:30.227: [ cssd][1029]clssgmclientconnectmsg: properties of cmproc 1132a3390 - 0,1,2,3,4
2012-06-04 23:26:30.227: [ cssd][1029]clssgmclientconnectmsg: connect from con(2eae64) proc(1132a3390) pid(35520706) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.228: [ cssd][1029]clssgmclientconnectmsg: msg flags 0x0000
2012-06-04 23:26:30.542: [ cssd][1029]clssscselect: cookie accept request 110abb440
2012-06-04 23:26:30.542: [ cssd][1029]clssgmallocproc: (1132ad030) allocated
2012-06-04 23:26:30.544: [ cssd][1029]clssgmclientconnectmsg: properties of cmproc 1132ad030 - 0,1,2,3,4
2012-06-04 23:26:30.544: [ cssd][1029]clssgmclientconnectmsg: connect from con(2eaede) proc(1132ad030) pid(8585490/8585490) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.544: [ cssd][1029]clssgmclientconnectmsg: msg flags 0x0000
2012-06-04 23:26:30.553: [ cssd][1029]clssscselect: cookie accept request 1132ad030
2012-06-04 23:26:30.553: [ cssd][1029]clssscevtypshrcon: getting client with cmproc 1132ad030
2012-06-04 23:26:30.553: [ cssd][1029]clssgmregisterclient: proc(63/1132ad030), client(1/1132ad370)
2012-06-04 23:26:30.554: [ cssd][1029]clssgmregistershared: grp dg_data, mbr 1, type 1
2012-06-04 23:26:30.554: [ cssd][1029]clssgmqueueshare: (1132ad510) target global grock dg_data member 1 type 1 queued from client (1132ad370), global grock dg_data, refcount 15
2012-06-04 23:26:30.554: [ cssd][1029]clssgmregistershared: global grock dg_data member 1 share type 1, refcount 15
2012-06-04 23:26:30.557: [ cssd][1029]clssscselect: cookie accept request 1132ad030
2012-06-04 23:26:30.557: [ cssd][1029]clssscevtypshrcon: getting client with cmproc 1132ad030
2012-06-04 23:26:30.557: [ cssd][1029]clssgmregisterclient: proc(63/1132ad030), client(2/1132a3d90)
2012-06-04 23:26:30.558: [ cssd][1029]clssgmregistershared: grp db+asm, mbr 1, type 1
2012-06-04 23:26:30.558: [ cssd][1029]clssgmqueueshare: (1132ad910) target global grock db+asm member 1 type 1 queued from client (1132a3d90), global grock db+asm, refcount 17
2012-06-04 23:26:30.558: [ cssd][1029]clssgmregistershared: global grock db+asm member 1 share type 1, refcount 17
2012-06-04 23:26:30.636: [ cssd][1029]clssscselect: cookie accept request 110abb440
2012-06-04 23:26:30.637: [ cssd][1029]clssgmallocproc: (11333ba70) allocated
2012-06-04 23:26:30.638: [ cssd][1029]clssgmclientconnectmsg: properties of cmproc 11333ba70 - 0,1,2,3,4
2012-06-04 23:26:30.638: [ cssd][1029]clssgmclientconnectmsg: connect from con(2eaf6f) proc(11333ba70) pid(8913188/8913188) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.638: [ cssd][1029]clssgmclientconnectmsg: msg flags 0x0000
2012-06-04 23:26:30.650: [ cssd][1029]clssscselect: cookie accept request 11333ba70
2012-06-04 23:26:30.650: [ cssd][1029]clssscevtypshrcon: getting client with cmproc 11333ba70
2012-06-04 23:26:30.650: [ cssd][1029]clssgmregisterclient: proc(64/11333ba70), client(1/112e8e550)
2012-06-04 23:26:30.652: [ cssd][1029]clssgmregistershared: grp dg_data, mbr 1, type 1
2012-06-04 23:26:30.652: [ cssd][1029]clssgmqueueshare: (11328ed90) target global grock dg_data member 1 type 1 queued from client (112e8e550), global grock dg_data, refcount 16
2012-06-04 23:26:30.652: [ cssd][1029]clssgmregistershared: global grock dg_data member 1 share type 1, refcount 16
2012-06-04 23:26:30.766: [ cssd][1029]clssscselect: cookie accept request 11333ba70
2012-06-04 23:26:30.766: [ cssd][1029]clssscevtypshrcon: getting client with cmproc 11333ba70
2012-06-04 23:26:30.766: [ cssd][1029]clssgmregisterclient: proc(64/11333ba70), client(2/11328eff0)
2012-06-04 23:26:30.772: [ cssd][1029]clssgmregistershared: grp db+asm, mbr 1, type 1
2012-06-04 23:26:30.772: [ cssd][1029]clssgmqueueshare: (11328f190) target global grock db+asm member 1 type 1 queued from client (11328eff0), global grock db+asm, refcount 18
2012-06-04 23:26:30.773: [ cssd][1029]clssgmregistershared: global grock db+asm member 1 share type 1, refcount 18
2012-06-04 23:26:31.382: [ cssd][3862]clssnmsendingthread: sending status msg to all nodes
2012-06-04 23:26:31.382: [ cssd][3862]clssnmsendingthread: sent 4 status msgs to all nodes
2012-06-04 23:26:34.394: [ cssd][1029]clssscmonitorthreads clssnmvdiskpingthread not scheduled for 145718 msecs
2012-06-04 23:26:35.392: [ cssd][3862]clssnmsendingthread: sending status msg to all nodes
2012-06-04 23:26:35.392: [ cssd][3862]clssnmsendingthread: sent 4 status msgs to all nodes
2012-06-04 23:26:36.396: [ cssd][1029]clssscmonitorthreads clssnmvworkerthread not scheduled for 47678 msecs
2012-06-04 23:26:38.571: [ cssd][2834]clssnmvdiskcheck: (/dev/rhdisk1) no i/o completed after 75% maximum time, 200000 ms, will be considered unusable in 49104 ms
2012-06-04 23:26:39.408: [ cssd][3862]clssnmsendingthread: sending status msg to all nodes
2012-06-04 23:26:39.409: [ cssd][3862]clssnmsendingthread: sent 4 status msgs to all nodes
2012-06-04 23:26:42.413: [ cssd][1029]clssscmonitorthreads clssnmvdiskpingthread not scheduled for 153738 msecs
2012-06-04 23:26:43.413: [ cssd][3862]clssnmsendingthread: sending status msg to all nodes
2012-06-04 23:26:43.413: [ cssd][3862]clssnmsendingthread: sent 4 status msgs to all nodes
2012-06-04 23:26:44.414: [ cssd][1029]clssscmonitorthreads clssnmvworkerthread not scheduled for 55696 msecs
2012-06-04 23:26:47.421: [ cssd][3862]clssnmsendingthread: sending status msg to all nodes
2012-06-04 23:26:47.421: [ cssd][3862]clssnmsendingthread: sent 4 status msgs to all nodes
2012-06-04 23:26:50.433: [ cssd][1029]clssscmonitorthreads clssnmvdiskpingthread not scheduled for 161758 msecs
2012-06-04 23:26:51.431: [ cssd][3862]clssnmsendingthread: sending status msg to all nodes
2012-06-04 23:26:51.431: [ cssd][3862]clssnmsendingthread: sent 4 status msgs to all nodes
2012-06-04 23:26:52.439: [ cssd][1029]clssscmonitorthreads clssnmvworkerthread not scheduled for 63721 msecs
2012-06-04 23:26:55.442: [ cssd][3862]clssnmsendingthread: sending status msg to all nodes
2012-06-04 23:26:55.442: [ cssd][3862]clssnmsendingthread: sent 4 status msgs to all nodes
2012-06-04 23:26:58.445: [ cssd][1029]clssscmonitorthreads clssnmvdiskpingthread not scheduled for 169771 msecs
2012-06-04 23:26:59.455: [ cssd][3862]clssnmsendingthread: sending status msg to all nodes
2012-06-04 23:26:59.455: [ cssd][3862]clssnmsendingthread: sent 4 status msgs to all nodes
2012-06-04 23:27:00.493: [ cssd][1029]clssscmonitorthreads clssnmvworkerthread not scheduled for 71776 msecs
2012-06-04 23:27:03.470: [ cssd][3862]clssnmsendingthread: sending status msg to all nodes
2012-06-04 23:27:03.470: [ cssd][3862]clssnmsendingthread: sent 4 status msgs to all nodes
2012-06-04 23:27:06.473: [ cssd][1029]clssscmonitorthreads clssnmvdiskpingthread not scheduled for 177799 msecs
2012-06-04 23:27:07.475: [ cssd][3862]clssnmsendingthread: sending status msg to all nodes
2012-06-04 23:27:07.476: [ cssd][3862]clssnmsendingthread: sent 4 status msgs to all nodes
2012-06-04 23:27:08.192: [ cssd][1029]clssgmdeadproc: proc 112f35f50
2012-06-04 23:27:08.192: [ cssd][1029]clssgmdestroyproc: cleaning up proc(112f35f50) con(2ea340) skgpid ospid 10551654 with 0 clients, refcount 0
2012-06-04 23:27:08.192: [ cssd][1029]clssgmdiscendpcl: gipcdestroy 2ea340
2012-06-04 23:27:08.192: [ cssd][1029]clssscmonitorthreads clssnmvworkerthread not scheduled for 79475 msecs
2012-06-04 23:27:08.498: [ cssd][1029]clssgmdeadproc: proc 1131b4b10
2012-06-04 23:27:08.498: [ cssd][1029]clssgmdestroyproc: cleaning up proc(1131b4b10) con(2ea3c4) skgpid ospid 10355074 with 0 clients, refcount 0
2012-06-04 23:27:08.498: [ cssd][1029]clssgmdiscendpcl: gipcdestroy 2ea3c4
2012-06-04 23:27:08.569: [ cssd][1029]clssscselect: cookie accept request 110abb440
2012-06-04 23:27:08.569: [ cssd][1029]clssgmallocproc: (1131fe670) allocated
2012-06-04 23:27:08.571: [ cssd][1029]clssgmclientconnectmsg: properties of cmproc 1131fe670 - 0,1,2,3,4
2012-06-04 23:27:08.571: [ cssd][1029]clssgmclientconnectmsg: connect from con(2eb1f8) proc(1131fe670) pid(10551656) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.571: [ cssd][1029]clssgmclientconnectmsg: msg flags 0x0000
2012-06-04 23:27:08.637: [ cssd][2834]clssnmvdiskcheck: (/dev/rhdisk1) no i/o completed after 90% maximum time, 200000 ms, will be considered unusable in 19038 ms
2012-06-04 23:27:08.809: [ cssd][1029]clssscselect: cookie accept request 110abb440
2012-06-04 23:27:08.810: [ cssd][1029]clssgmallocproc: (1132593f0) allocated
2012-06-04 23:27:08.811: [ cssd][1029]clssgmclientconnectmsg: properties of cmproc 1132593f0 - 0,1,2,3,4
2012-06-04 23:27:08.811: [ cssd][1029]clssgmclientconnectmsg: connect from con(2eb278) proc(1132593f0) pid(10355076) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.811: [ cssd][1029]clssgmclientconnectmsg: msg flags 0x0000
2012-06-04 23:27:08.855: [ cssd][1029]clssscselect: cookie accept request 110abb440
2012-06-04 23:27:08.855: [ cssd][1029]clssgmallocproc: (113259730) allocated
2012-06-04 23:27:08.857: [ cssd][1029]clssgmclientconnectmsg: properties of cmproc 113259730 - 0,1,2,3,4
2012-06-04 23:27:08.858: [ cssd][1029]clssgmclientconnectmsg: connect from con(2eb2ec) proc(113259730) pid(8192508/8192508) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.858: [ cssd][1029]clssgmclientconnectmsg: msg flags 0x0000
2012-06-04 23:27:08.867: [ cssd][1029]clssscselect: cookie accept request 113259730
2012-06-04 23:27:08.867: [ cssd][1029]clssscevtypshrcon: getting client with cmproc 113259730
2012-06-04 23:27:08.867: [ cssd][1029]clssgmregisterclient: proc(65/113259730), client(1/112f35f50)
2012-06-04 23:27:08.868: [ cssd][1029]clssgmregistershared: grp dg_data, mbr 1, type 1
2012-06-04 23:27:08.868: [ cssd][1029]clssgmqueueshare: (11334b430) target global grock dg_data member 1 type 1 queued from client (112f35f50), global grock dg_data, refcount 17
2012-06-04 23:27:08.868: [ cssd][1029]clssgmregistershared: global grock dg_data member 1 share type 1, refcount 17
2012-06-04 23:27:08.872: [ cssd][1029]clssscselect: cookie accept request 113259730
2012-06-04 23:27:08.872: [ cssd][1029]clssscevtypshrcon: getting client with cmproc 113259730
2012-06-04 23:27:08.872: [ cssd][1029]clssgmregisterclient: proc(65/113259730), client(2/113215ed0)
2012-06-04 23:27:08.874: [ cssd][1029]clssgmregistershared: grp db+asm, mbr 1, type 1
2012-06-04 23:27:08.874: [ cssd][1029]clssgmqueueshare: (11334b690) target global grock db+asm member 1 type 1 queued from client (113215ed0), global grock db+asm, refcount 19
2012-06-04 23:27:08.874: [ cssd][1029]clssgmregistershared: global grock db+asm member 1 share type 1, refcount 19
2012-06-04 23:27:09.000: [ cssd][1029]clssscselect: cookie accept request 110abb440
2012-06-04 23:27:09.000: [ cssd][1029]clssgmallocproc: (11334b8f0) allocated
2012-06-04 23:27:09.001: [ cssd][1029]clssgmclientconnectmsg: properties of cmproc 11334b8f0 - 0,1,2,3,4
2012-06-04 23:27:09.001: [ cssd][1029]clssgmclientconnectmsg: connect from con(2eb37d) proc(11334b8f0) pid(8061184/8061184) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:09.001: [ cssd][1029]clssgmclientconnectmsg: msg flags 0x0000
2012-06-04 23:27:09.010: [ cssd][1029]clssscselect: cookie accept request 11334b8f0
2012-06-04 23:27:09.010: [ cssd][1029]clssscevtypshrcon: getting client with cmproc 11334b8f0
2012-06-04 23:27:09.010: [ cssd][1029]clssgmregisterclient: proc(66/11334b8f0), client(1/1131b5310)
2012-06-04 23:27:09.011: [ cssd][1029]clssgmregistershared: grp dg_data, mbr 1, type 1
2012-06-04 23:27:09.011: [ cssd][1029]clssgmqueueshare: (11334bc30) target global grock dg_data member 1 type 1 queued from client (1131b5310), global grock dg_data, refcount 18
2012-06-04 23:27:09.011: [ cssd][1029]clssgmregistershared: global grock dg_data member 1 share type 1, refcount 18
2012-06-04 23:27:09.014: [ cssd][1029]clssscselect: cookie accept request 11334b8f0
2012-06-04 23:27:09.014: [ cssd][1029]clssscevtypshrcon: getting client with cmproc 11334b8f0
2012-06-04 23:27:09.014: [ cssd][1029]clssgmregisterclient: proc(66/11334b8f0), client(2/11334be90)
2012-06-04 23:27:09.015: [ cssd][1029]clssgmregistershared: grp db+asm, mbr 1, type 1
2012-06-04 23:27:09.015: [ cssd][1029]clssgmqueueshare: (113350230) target global grock db+asm member 1 type 1 queued from client (11334be90), global grock db+asm, refcount 20
2012-06-04 23:27:09.015: [ cssd][1029]clssgmregistershared: global grock db+asm member 1 share type 1, refcount 20
2012-06-04 23:27:11.480: [ cssd][3862]clssnmsendingthread: sending status msg to all nodes
2012-06-04 23:27:11.480: [ cssd][3862]clssnmsendingthread: sent 4 status msgs to all nodes
2012-06-04 23:27:14.494: [ cssd][1029]clssscmonitorthreads clssnmvdiskpingthread not scheduled for 185820 msecs
2012-06-04 23:27:15.500: [ cssd][3862]clssnmsendingthread: sending status msg to all nodes
2012-06-04 23:27:15.500: [ cssd][3862]clssnmsendingthread: sent 4 status msgs to all nodes
2012-06-04 23:27:16.504: [ cssd][1029]clssscmonitorthreads clssnmvworkerthread not scheduled for 87787 msecs
2012-06-04 23:27:19.519: [ cssd][3862]clssnmsendingthread: sending status msg to all nodes
2012-06-04 23:27:19.519: [ cssd][3862]clssnmsendingthread: sent 4 status msgs to all nodes
2012-06-04 23:27:20.285: [ cssd][1029]clssscselect: cookie accept request 110abb440
2012-06-04 23:27:20.285: [ cssd][1029]clssgmallocproc: (1132b9290) allocated
2012-06-04 23:27:20.294: [ cssd][1029]clssgmclientconnectmsg: properties of cmproc 1132b9290 - 0,1,2,3,4
2012-06-04 23:27:20.294: [ cssd][1029]clssgmclientconnectmsg: connect from con(2eb49d) proc(1132b9290) pid(8257826) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:20.294: [ cssd][1029]clssgmclientconnectmsg: msg flags 0x0000
2012-06-04 23:27:20.299: [ cssd][1029]clssnmgetnodenumber: pt1rac2
2012-06-04 23:27:20.457: [ cssd][1029]clssgmdeadproc: proc 1132b9290
2012-06-04 23:27:20.457: [ cssd][1029]clssgmdestroyproc: cleaning up proc(1132b9290) con(2eb49d) skgpid ospid 8257826 with 0 clients, refcount 0
2012-06-04 23:27:20.457: [ cssd][1029]clssgmdiscendpcl: gipcdestroy 2eb49d
2012-06-04 23:27:20.999: [ cssd][1029]clssscselect: cookie accept request 110abb440
2012-06-04 23:27:20.999: [ cssd][1029]clssgmallocproc: (1132b2650) allocated
2012-06-04 23:27:21.000: [ cssd][1029]clssgmclientconnectmsg: properties of cmproc 1132b2650 - 0,1,2,3,4
2012-06-04 23:27:21.000: [ cssd][1029]clssgmclientconnectmsg: connect from con(2eb51a) proc(1132b2650) pid(7733742) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:21.000: [ cssd][1029]clssgmclientconnectmsg: msg flags 0x0000
2012-06-04 23:27:21.005: [ cssd][1029]clssnmgetnodenumber: pt1rac1
2012-06-04 23:27:21.289: [ cssd][1029]clssgmdeadproc: proc 1132b2650
2012-06-04 23:27:21.290: [ cssd][1029]clssgmdestroyproc: cleaning up proc(1132b2650) con(2eb51a) skgpid ospid 7733742 with 0 clients, refcount 0
2012-06-04 23:27:21.290: [ cssd][1029]clssgmdiscendpcl: gipcdestroy 2eb51a
2012-06-04 23:27:22.001: [ cssd][1029]clssscmonitorthreads clssnmvdiskpingthread not scheduled for 193327 msecs
2012-06-04 23:27:23.533: [ cssd][3862]clssnmsendingthread: sending status msg to all nodes
2012-06-04 23:27:23.534: [ cssd][3862]clssnmsendingthread: sent 4 status msgs to all nodes
2012-06-04 23:27:24.945: [ cssd][1029]clssgmdeadproc: proc 113248030
2012-06-04 23:27:24.945: [ cssd][1029]clssgmdestroyproc: cleaning up proc(113248030) con(2ea626) skgpid ospid 9699686 with 0 clients, refcount 0
2012-06-04 23:27:24.945: [ cssd][1029]clssgmdiscendpcl: gipcdestroy 2ea626
2012-06-04 23:27:25.142: [ cssd][1029]clssscselect: cookie accept request 110abb440
2012-06-04 23:27:25.143: [ cssd][1029]clssgmallocproc: (1132b2650) allocated
2012-06-04 23:27:25.143: [ cssd][1029]clssscmonitorthreads clssnmvworkerthread not scheduled for 96425 msecs
2012-06-04 23:27:25.144: [ cssd][1029]clssgmclientconnectmsg: properties of cmproc 1132b2650 - 1,2,3,4
2012-06-04 23:27:25.144: [ cssd][1029]clssgmclientconnectmsg: connect from con(2eb5c3) proc(1132b2650) pid(9699688) version 11:2:1:4, properties: 1,2,3,4
2012-06-04 23:27:25.144: [ cssd][1029]clssgmclientconnectmsg: msg flags 0x0000
2012-06-04 23:27:25.323: [ cssd][1029]clssscselect: cookie accept request 110abb440
2012-06-04 23:27:25.324: [ cssd][1029]clssgmallocproc: (1132b7e50) allocated
2012-06-04 23:27:25.326: [ cssd][1029]clssgmclientconnectmsg: properties of cmproc 1132b7e50 - 0,1,2,3,4
2012-06-04 23:27:25.326: [ cssd][1029]clssgmclientconnectmsg: connect from con(2eb63a) proc(1132b7e50) pid(7733746/7733746) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:25.326: [ cssd][1029]clssgmclientconnectmsg: msg flags 0x0000
2012-06-04 23:27:25.485: [ cssd][1029]clssscselect: cookie accept request 1132b7e50
2012-06-04 23:27:25.485: [ cssd][1029]clssscevtypshrcon: getting client with cmproc 1132b7e50
2012-06-04 23:27:25.485: [ cssd][1029]clssgmregisterclient: proc(67/1132b7e50), client(1/113153d90)
2012-06-04 23:27:25.487: [ cssd][1029]clssgmregistershared: grp dg_data, mbr 1, type 1
2012-06-04 23:27:25.487: [ cssd][1029]clssgmqueueshare: (113338450) target global grock dg_data member 1 type 1 queued from client (113153d90), global grock dg_data, refcount 19
2012-06-04 23:27:25.487: [ cssd][1029]clssgmregistershared: global grock dg_data member 1 share type 1, refcount 19
2012-06-04 23:27:25.495: [ cssd][1029]clssscselect: cookie accept request 1132b7e50
2012-06-04 23:27:25.495: [ cssd][1029]clssscevtypshrcon: getting client with cmproc 1132b7e50
2012-06-04 23:27:25.495: [ cssd][1029]clssgmregisterclient: proc(67/1132b7e50), client(2/1132b42b0)
2012-06-04 23:27:25.497: [ cssd][1029]clssgmregistershared: grp db+asm, mbr 1, type 1
2012-06-04 23:27:25.497: [ cssd][1029]clssgmqueueshare: (11334c0f0) target global grock db+asm member 1 type 1 queued from client (1132b42b0), global grock db+asm, refcount 21
2012-06-04 23:27:25.497: [ cssd][1029]clssgmregistershared: global grock db+asm member 1 share type 1, refcount 21
2012-06-04 23:27:27.542: [ cssd][3862]clssnmsendingthread: sending status msg to all nodes
2012-06-04 23:27:27.542: [ cssd][3862]clssnmsendingthread: sent 4 status msgs to all nodes
2012-06-04 23:27:27.680: [ cssd][2834](:cssnm00058:)clssnmvdiskcheck: no i/o completions for 200006 ms for voting file /dev/rhdisk1)
2012-06-04 23:27:27.693: [ cssd][2834]clssnmvdiskavailabilitychange: voting file /dev/rhdisk1 now offline
2012-06-04 23:27:27.693: [ cssd][2834](:cssnm00018:)clssnmvdiskcheck: aborting, 0 of 1 configured voting disks available, need 1
2012-06-04 23:27:27.694: [ cssd][2834]###################################
2012-06-04 23:27:27.694: [ cssd][2834]clssscexit: cssd aborting from thread clssnmvdiskpingmonitorthread
2012-06-04 23:27:27.694: [ cssd][2834]###################################
2012-06-04 23:27:27.694: [ cssd][2834](:csssc00012:)clssscexit: a fatal error occurred and the css daemon is terminating abnormally
2012-06-04 23:27:27.695: [ cssd][2834]
----- call stack trace -----
2012-06-04 23:27:27.723: [ cssd][2834]calling call entry argument values in hex
2012-06-04 23:27:27.723: [ cssd][2834]location type point (? means dubious value)
2012-06-04 23:27:27.723: [ cssd][2834]-------------------- -------- -------------------- ----------------------------
2012-06-04 23:27:27.724: [ cssd][2834]clssscexit()+692 bl 100067114 67800000678 ? 1000eb40c ?
2012-06-04 23:27:27.724: [ cssd][2834] 110091190 ? 900000000ba4d14 ?
2012-06-04 23:27:27.724: [ cssd][2834] 000000000 ? 013960031 ?
2012-06-04 23:27:27.724: [ cssd][2834] 013960031 ? 000000000 ?
2012-06-04 23:27:27.724: [ cssd][2834]clssnmvdiskcheck()+ call clssscexit() 111e38b90 ? 200000002 ?
2012-06-04 23:27:27.724: [ cssd][2834]2576 000000001 ? 1000f52a0 ?
2012-06-04 23:27:27.724: [ cssd][2834] 110091190 ? 013960031 ?
2012-06-04 23:27:27.724: [ cssd][2834] 013960031 ? 000000000 ?
2012-06-04 23:27:27.724: [ cssd][2834]clssnmvdiskpingmoni call clssnmvdiskcheck() 111e38b90 ? 110bf7fb0 ?
2012-06-04 23:27:27.724: [ cssd][2834]torthread()+392 112468680 ? 000000000 ?
2012-06-04 23:27:27.724: [ cssd][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ cssd][2834] 013960031 ? 000000000 ?
2012-06-04 23:27:27.725: [ cssd][2834]clssscthrdmain()+20 call clssnmvdiskpingmoni 111e38b90 ? 000000000 ?
2012-06-04 23:27:27.725: [ cssd][2834]8 torthread() 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ cssd][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ cssd][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ cssd][2834]_pthread_body()+240 call clssscthrdmain() 111e38b90 ? 000000000 ?
2012-06-04 23:27:27.725: [ cssd][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ cssd][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ cssd][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ cssd][2834]fffffffffffffffc call _pthread_body() 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ cssd][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ cssd][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ cssd][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.726: [ cssd][2834]
2012-06-04 23:27:27.726: [ cssd][2834]--------------------- binary stack dump ---------------------
2012-06-04 23:27:27.726: [ cssd][2834]
2012-06-04 23:27:27.726: [ cssd][2834]========== frame [1] (clssscexit()+692 -> 100067114) ==========
2012-06-04 23:27:27.726: [ cssd][2834]defined by frame pointers 0x1124683f0 and 0x112468260
2012-06-04 23:27:27.726: [ cssd][2834]call type: bl error signaled: no caller: clssscexit
2012-06-04 23:27:27.726: [ cssd][2834]
dump of memory from 0x112468260 to 0x1124683f0
2012-06-04 23:27:27.726: [ cssd][2834]112468260 00000001 124683f0 44000224 000003d7 [.....f..d..$....]
2012-06-04 23:27:27.726: [ cssd][2834]112468270 00000001 000031d8 00000000 00000083 [......1.........]
2012-06-04 23:27:27.726: [ cssd][2834]112468280 00000000 00000029 00000001 100182b0 [.......)........]
2012-06-04 23:27:27.726: [ cssd][2834]112468290 00000678 00000678 00000001 000eb40c [...x...x........]
2012-06-04 23:27:27.726: [ cssd][2834]1124682a0 00000001 10091190 09000000 00ba4d14 [..............m.]
2012-06-04 23:27:27.726: [ cssd][2834]1124682b0 00000000 00000000 00000000 13960031 [...............1]
2012-06-04 23:27:27.727: [ cssd][2834]1124682c0 00000000 13960031 00000000 00000000 [.......1........]
2012-06-04 23:27:27.727: [ cssd][2834]1124682d0 00000001 1008ff50 00000001 1002bd70 [.......p.......p]
2012-06-04 23:27:27.727: [ cssd][2834]1124682e0 00000000 00000001 43525300 12468368 [........crs..f.h]
2012-06-04 23:27:27.727: [ cssd][2834]1124682f0 00000001 00000000 00000001 100181f0 [................]
2012-06-04 23:27:27.727: [ cssd][2834]112468300 00000001 10018208 00000001 100181f0 [................]
2012-06-04 23:27:27.727: [ cssd][2834]112468310 00000001 10018208 00000001 11e38b90 [................]
2012-06-04 23:27:27.727: [ cssd][2834]112468320 00000001 11e38b90 00000000 00000000 [................]
2012-06-04 23:27:27.727: [ cssd][2834]112468330 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.728: [ cssd][2834] repeat 3 times
2012-06-04 23:27:27.728: [ cssd][2834]112468370 00000000 00000000 00000000 13960031 [...............1]
2012-06-04 23:27:27.728: [ cssd][2834]112468380 09000000 00c27160 00000000 00000000 [......q`........]
2012-06-04 23:27:27.728: [ cssd][2834]112468390 00000001 1002bd70 00000001 117b6590 [.......p.....{e.]
2012-06-04 23:27:27.728: [ cssd][2834]1124683a0 00000001 00000001 00000000 00000000 [................]
2012-06-04 23:27:27.728: [ cssd][2834]1124683b0 00000000 00000000 00000646 00000646 [...........f...f]
2012-06-04 23:27:27.728: [ cssd][2834]1124683c0 00000001 12468428 00000001 117c6c84 [.....f.(.....|l.]
2012-06-04 23:27:27.728: [ cssd][2834]1124683d0 00000001 117c6c84 00000001 1008ff50 [.....|l........p]
2012-06-04 23:27:27.728: [ cssd][2834]1124683e0 00000000 00000000 00000001 000f24e8 [..............$.]
2012-06-04 23:27:27.728: [ cssd][2834]========== frame [2] (clssnmvdiskcheck()+2576 -> clssscexit()) ==========
2012-06-04 23:27:27.728: [ cssd][2834]defined by frame pointers 0x112468600 and 0x1124683f0
2012-06-04 23:27:27.729: [ cssd][2834]call type: call error signaled: no caller: clssnmvdiskcheck
2012-06-04 23:27:27.729: [ cssd][2834]
dump of memory from 0x1124683f0 to 0x112468600
2012-06-04 23:27:27.729: [ cssd][2834]1124683f0 00000001 12468600 09001000 a0688640 [.....f.......h.@]
2012-06-04 23:27:27.729: [ cssd][2834]112468400 00000001 0002cd94 4400b22b 00000000 [........d..+....]
2012-06-04 23:27:27.729: [ cssd][2834]112468410 00000001 001058b8 00000001 100182b0 [......x.........]
2012-06-04 23:27:27.729: [ cssd][2834]112468420 00000001 11e38b90 00000002 00000002 [................]
2012-06-04 23:27:27.729: [ cssd][2834]112468430 00000000 00000001 00000001 000f52a0 [..............r.]
2012-06-04 23:27:27.729: [ cssd][2834]112468440 00000001 10091190 00000000 13960031 [...............1]
2012-06-04 23:27:27.729: [ cssd][2834]112468450 00000000 13960031 00000000 00000000 [.......1........]
2012-06-04 23:27:27.730: [ cssd][2834]112468460 00000001 1008ff50 00000001 118508f0 [.......p........]
2012-06-04 23:27:27.730: [ cssd][2834]112468470 00000001 117b6590 8c91440c 00030d46 [.....{e...d....f]
2012-06-04 23:27:27.730: [ cssd][2834]112468480 00000001 12c508b0 00000000 00030d40 [...............@]
2012-06-04 23:27:27.730: [ cssd][2834]112468490 00010001 9dd81012 d0e54f97 bfee0f2c [..........o....,]
2012-06-04 23:27:27.730: [ cssd][2834]1124684a0 af8aa88c 00000000 2200088b 00000000 [...............]
2012-06-04 23:27:27.730: [ cssd][2834]1124684b0 09000000 014f0814 09001000 a0688640 [.....o.......h.@]
2012-06-04 23:27:27.730: [ cssd][2834]1124684c0 00030d40 1045f530 00000004 00000004 [...@.e.0........]
2012-06-04 23:27:27.730: [ cssd][2834]1124684d0 00000001 00105de8 00000001 000f2bd4 [......].......+.]
2012-06-04 23:27:27.730: [ cssd][2834]1124684e0 00000001 000f2bd4 00030d40 000f24e8 [......+....@..$.]
2012-06-04 23:27:27.731: [ cssd][2834]1124684f0 00000001 00000001 2200022b 00000000 [..........+....]
2012-06-04 23:27:27.731: [ cssd][2834]112468500 00000000 00000000 fffffff5 000f24e8 [..............$.]
2012-06-04 23:27:27.731: [ cssd][2834]112468510 00000001 12468600 09001000 a0688640 [.....f.......h.@]
2012-06-04 23:27:27.731: [ cssd][2834]112468520 00000001 00070dfc 00000001 117c6c84 [.............|l.]
2012-06-04 23:27:27.731: [ cssd][2834]112468530 00000001 117c6c84 00000001 100182b0 [.....|l.........]
2012-06-04 23:27:27.731: [ cssd][2834]112468540 00000001 1045f530 00000004 00000004 [.....e.0........]
2012-06-04 23:27:27.731: [ cssd][2834]112468550 00000001 00105de8 00000000 00000009 [......].........]
2012-06-04 23:27:27.731: [ cssd][2834]112468560 00000000 000003e8 00000000 000003e8 [................]
2012-06-04 23:27:27.731: [ cssd][2834]112468570 00000000 13960031 00000000 00000040 [.......1.......@]
2012-06-04 23:27:27.732: [ cssd][2834]112468580 00000001 1008ff50 00000001 117b6590 [.......p.....{e.]
2012-06-04 23:27:27.732: [ cssd][2834]112468590 00000001 1002bd70 00000001 117b65a0 [.......p.....{e.]
2012-06-04 23:27:27.732: [ cssd][2834]1124685a0 00000001 117b66a0 ffffffb2 000003d4 [.....{f.........]
2012-06-04 23:27:27.732: [ cssd][2834]1124685b0 00000001 117c6c84 00000001 117c6c84 [.....|l......|l.]
2012-06-04 23:27:27.732: [ cssd][2834] repeat 1 times
2012-06-04 23:27:27.732: [ cssd][2834]1124685d0 00000001 000f2bd4 00000001 000f2bd4 [......+.......+.]
2012-06-04 23:27:27.732: [ cssd][2834]1124685e0 00000000 00000000 00000001 12c508b0 [................]
2012-06-04 23:27:27.732: [ cssd][2834]1124685f0 00000000 00000000 00000001 000f24e8 [..............$.]
2012-06-04 23:27:27.732: [ cssd][2834]========== frame [3] (clssnmvdiskpingmonitorthread()+392 -> clssnmvdiskcheck()) ==========
2012-06-04 23:27:27.732: [ cssd][2834]defined by frame pointers 0x1124686f0 and 0x112468600
2012-06-04 23:27:27.732: [ cssd][2834]call type: call error signaled: no caller: clssnmvdiskpingmonitorthread
2012-06-04 23:27:27.732: [ cssd][2834]
dump of memory from 0x112468600 to 0x1124686f0
2012-06-04 23:27:27.733: [ cssd][2834]112468600 00000001 124686f0 00000000 00000000 [.....f..........]
2012-06-04 23:27:27.733: [ cssd][2834]112468610 00000001 0002cfac 09001000 a0688640 [.............h.@]
2012-06-04 23:27:27.733: [ cssd][2834]112468620 00000000 00000000 00000001 100182b0 [................]
2012-06-04 23:27:27.733: [ cssd][2834]112468630 00000001 11e38b90 00000001 10bf7fb0 [................]
2012-06-04 23:27:27.733: [ cssd][2834]112468640 00000001 12468680 00000000 00000000 [.....f..........]
2012-06-04 23:27:27.733: [ cssd][2834]112468650 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.733: [ cssd][2834]112468660 00000000 13960031 00000000 00000000 [.......1........]
2012-06-04 23:27:27.733: [ cssd][2834]112468670 00000001 10bf7fb0 00000001 117b6590 [.............{e.]
2012-06-04 23:27:27.733: [ cssd][2834]112468680 00030d40 8c914024 000003e8 100182b0 [...@..@$........]
2012-06-04 23:27:27.733: [ cssd][2834]112468690 09000000 014f0814 00000000 00000000 [.....o..........]
2012-06-04 23:27:27.734: [ cssd][2834]1124686a0 00000000 00000000 00000000 00000001 [................]
2012-06-04 23:27:27.734: [ cssd][2834]1124686b0 000003e8 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.734: [ cssd][2834]1124686c0 00000000 00000000 000003e8 000003e8 [................]
2012-06-04 23:27:27.734: [ cssd][2834]1124686d0 000003e8 1008ff50 00000001 1002bd70 [.......p.......p]
2012-06-04 23:27:27.734: [ cssd][2834]1124686e0 00000001 10bf97d0 00000001 12468a78 [.............f.x]
2012-06-04 23:27:27.734: [ cssd][2834]========== frame [4] (clssscthrdmain()+208 -> clssnmvdiskpingmonitorthread()) ==========
2012-06-04 23:27:27.734: [ cssd][2834]defined by frame pointers 0x112468790 and 0x1124686f0
2012-06-04 23:27:27.734: [ cssd][2834]call type: call error signaled: no caller: clssscthrdmain
2012-06-04 23:27:27.734: [ cssd][2834]
dump of memory from 0x1124686f0 to 0x112468790
2012-06-04 23:27:27.734: [ cssd][2834]1124686f0 00000001 12468790 00000000 00000000 [.....f..........]
2012-06-04 23:27:27.734: [ cssd][2834]112468700 00000001 0000a294 00000000 00000000 [................]
2012-06-04 23:27:27.735: [ cssd][2834]112468710 00000000 00000000 00000001 100182b0 [................]
2012-06-04 23:27:27.735: [ cssd][2834]112468720 00000001 11e38b90 00000000 00000000 [................]
2012-06-04 23:27:27.735: [ cssd][2834]112468730 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.735: [ cssd][2834] repeat 2 times
2012-06-04 23:27:27.735: [ cssd][2834]112468760 00000001 1008ff50 00000001 1002bd70 [.......p.......p]
2012-06-04 23:27:27.735: [ cssd][2834]112468770 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.735: [ cssd][2834]112468780 00000000 00000000 00000001 1008ff50 [...............p]
2012-06-04 23:27:27.735: [ cssd][2834]========== frame [5] (_pthread_body()+240 -> clssscthrdmain()) ==========
2012-06-04 23:27:27.735: [ cssd][2834]defined by frame pointers 0x112468810 and 0x112468790
2012-06-04 23:27:27.735: [ cssd][2834]call type: call error signaled: no caller: _pthread_body
2012-06-04 23:27:27.735: [ cssd][2834]
dump of memory from 0x112468790 to 0x112468890
2012-06-04 23:27:27.736: [ cssd][2834]112468790 00000001 12468810 00000000 00000000 [.....f..........]
2012-06-04 23:27:27.736: [ cssd][2834]1124687a0 09000000 00b7bd54 00000000 00000000 [.......t........]
2012-06-04 23:27:27.736: [ cssd][2834]1124687b0 00000000 00000000 09001000 a0288350 [.............(.p]
2012-06-04 23:27:27.736: [ cssd][2834]1124687c0 00000001 11e38b90 00000000 00000000 [................]
2012-06-04 23:27:27.736: [ cssd][2834]1124687d0 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.737: [ cssd][2834] repeat 11 times
2012-06-04 23:27:27.737: [ cssd][2834]========== frame [6] (fffffffffffffffc -> _pthread_body()) ==========
2012-06-04 23:27:27.737: [ cssd][2834]defined by frame pointers 0x0 and 0x112468810
2012-06-04 23:27:27.737: [ cssd][2834]call type: call error signaled: no caller: fffffffffffffffc
2012-06-04 23:27:27.737: [ cssd][2834]
----- argument/register address dump -----
2012-06-04 23:27:27.737: [ cssd][2834]argument/register addr=0x1000eb40c.
dump of memory from 0x1000eb3cc to 0x1000eb50c
2012-06-04 23:27:27.737: [ cssd][2834]1000eb3c0 6c206572 [l er]
2012-06-04 23:27:27.737: [ cssd][2834]1000eb3d0 726f7220 6f636375 72726564 20616e64 [ror occurred and]
2012-06-04 23:27:27.737: [ cssd][2834]1000eb3e0 20746865 20435353 20646165 6d6f6e20 [ the css daemon ]
2012-06-04 23:27:27.737: [ cssd][2834]1000eb3f0 69732074 65726d69 6e617469 6e672061 [is terminating a]
2012-06-04 23:27:27.737: [ cssd][2834]1000eb400 626e6f72 6d616c6c 79004942 283a4353 [bnormally.ib(:cs]
2012-06-04 23:27:27.737: [ cssd][2834]1000eb410 53534330 30303132 3a290049 636c7373 [ssc00012:).iclss]
2012-06-04 23:27:27.738: [ cssd][2834]1000eb420 73634578 69743a20 53746172 74696e67 [scexit: starting]
2012-06-04 23:27:27.738: [ cssd][2834]1000eb430 20435253 4420636c 65616e75 70004942 [ crsd cleanup.ib]
2012-06-04 23:27:27.738: [ cssd][2834]1000eb440 636c7373 73634578 69743a20 43525344 [clssscexit: crsd]
2012-06-04 23:27:27.738: [ cssd][2834]1000eb450 20636c65 616e7570 20737461 74757320 [ cleanup status ]
2012-06-04 23:27:27.738: [ cssd][2834]1000eb460 25640049 636c7373 73634578 69743a20 [%d.iclssscexit: ]
2012-06-04 23:27:27.738: [ cssd][2834]1000eb470 43525344 20636c65 616e7570 20666169 [crsd cleanup fai]
2012-06-04 23:27:27.738: [ cssd][2834]1000eb480 6c656420 77697468 20256400 636c7373 [led with %d.clss]
2012-06-04 23:27:27.738: [ cssd][2834]1000eb490 73634578 69743a20 43525344 20636c65 [scexit: crsd cle]
2012-06-04 23:27:27.738: [ cssd][2834]1000eb4a0 616e7570 20737563 63657373 66756c6c [anup successfull]
2012-06-04 23:27:27.739: [ cssd][2834]1000eb4b0 7920636f 6d706c65 74656400 636c7373 [y completed.clss]
2012-06-04 23:27:27.739: [ cssd][2834]1000eb4c0 73634578 69743a20 61626f72 7420616c [scexit: abort al]
2012-06-04 23:27:27.739: [ cssd][2834]1000eb4d0 72656164 79207365 74202564 0049424d [ready set %d.ibm]
2012-06-04 23:27:27.739: [ cssd][2834]1000eb4e0 41535345 52542025 73202564 0049424d [assert %s %d.ibm]
2012-06-04 23:27:27.739: [ cssd][2834]1000eb4f0 636c7373 73632e63 0049424d 283a4353 [clsssc.c.ibm(:cs]
2012-06-04 23:27:27.739: [ cssd][2834]1000eb500 53534330 30303133 3a29636c [ssc00013:)cl]
2012-06-04 23:27:27.739: [ cssd][2834]argument/register addr=0x110091190.
dump of memory from 0x110091150 to 0x110091290
2012-06-04 23:27:27.739: [ cssd][2834]110091150 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.739: [ cssd][2834] repeat 2 times
2012-06-04 23:27:27.739: [ cssd][2834]110091180 00000000 00000000 00000001 00000001 [................]
2012-06-04 23:27:27.740: [ cssd][2834]110091190 2f736f66 742f6772 69642f70 726f6475 [/soft/grid/produ]
2012-06-04 23:27:27.740: [ cssd][2834]1100911a0 63742f31 312e322e 302e322f 6769682f [ct/11.2.0.2/gih/]
2012-06-04 23:27:27.740: [ cssd][2834]1100911b0 6c6f672f 70743172 6163322f 63737364 [log/pt1rac2/cssd]
2012-06-04 23:27:27.740: [ cssd][2834]1100911c0 2f6f6373 73642e6c 6f670000 00000000 [/ocssd.log......]
2012-06-04 23:27:27.740: [ cssd][2834]1100911d0 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.741: [ cssd][2834] repeat 11 times
2012-06-04 23:27:27.741: [ cssd][2834]argument/register addr=0x900000000ba4d14.
dump of memory from 0x900000000ba4cd4 to 0x900000000ba4e14
2012-06-04 23:27:27.741: [ cssd][2834]900000000ba4cd0 38600000 4bfd5608 00000000 [8`..k.v.....]
2012-06-04 23:27:27.741: [ cssd][2834]900000000ba4ce0 000c0000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.741: [ cssd][2834]900000000ba4cf0 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.741: [ cssd][2834]900000000ba4d00 348d8278 41810008 38800000 e8630000 [4..xa...8....c..]
2012-06-04 23:27:27.741: [ cssd][2834]900000000ba4d10 38a00008 81840230 7c0004ac 398cffff [8......0|...9...]
2012-06-04 23:27:27.741: [ cssd][2834]900000000ba4d20 e9040110 38e00000 65081000 7d251828 [....8...e...}%.(]
2012-06-04 23:27:27.741: [ cssd][2834]900000000ba4d30 7c894000 40860014 7ce5192d 40a2fff0 [|.@.@...|..-@...]
2012-06-04 23:27:27.742: [ cssd][2834]900000000ba4d40 91840230 4e800020 7c632a14 4bfd4694 [...0n.. |c*.k.f.]
2012-06-04 23:27:27.742: [ cssd][2834]900000000ba4d50 00000000 000c0000 00000000 00000000 [................]
2012-06-04 23:27:27.742: [ cssd][2834]900000000ba4d60 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.742: [ cssd][2834] repeat 1 times
2012-06-04 23:27:27.742: [ cssd][2834]900000000ba4d80 7d251828 64871000 2c890000 40860014 [}%.(d...,...@...]
2012-06-04 23:27:27.742: [ cssd][2834]900000000ba4d90 7ce5192d 4c00012c 4d820020 4bffffe4 [|..-l..,m.. k...]
2012-06-04 23:27:27.742: [ cssd][2834]900000000ba4da0 7c862378 7c851a14 7cc53378 4bfd5534 [|.#x|...|.3xk.u4]
2012-06-04 23:27:27.742: [ cssd][2834]900000000ba4db0 00000000 000c0000 00000000 00000000 [................]
2012-06-04 23:27:27.742: [ cssd][2834]900000000ba4dc0 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.743: [ cssd][2834] repeat 3 times
2012-06-04 23:27:27.743: [ cssd][2834]900000000ba4e00 7d251828 64871000 2c890000 40860014 [}%.(d...,...@...]
2012-06-04 23:27:27.743: [ cssd][2834]900000000ba4e10 7ce5192d [|..-]
2012-06-04 23:27:27.743: [ cssd][2834]argument/register addr=0x0.
dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.743: [ cssd][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.744: [ cssd][2834] repeat 15 times
2012-06-04 23:27:27.744: [ cssd][2834]argument/register addr=0x0.
dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.744: [ cssd][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.745: [ cssd][2834] repeat 15 times
2012-06-04 23:27:27.745: [ cssd][2834]argument/register addr=0x111e38b90.
dump of memory from 0x111e38b50 to 0x111e38c90
2012-06-04 23:27:27.745: [ cssd][2834]111e38b50 00000001 00000001 8c8fc179 000003e8 [...........y....]
2012-06-04 23:27:27.745: [ cssd][2834]111e38b60 8c914038 8c913a22 0000000b 00000000 [..@8..:........]
2012-06-04 23:27:27.745: [ cssd][2834]111e38b70 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.745: [ cssd][2834]111e38b80 5b5b5b5b 00000000 00000000 000001e0 [[[[[............]
2012-06-04 23:27:27.745: [ cssd][2834]111e38b90 00000001 1008ff50 00000001 10bf7fb0 [.......p........]
2012-06-04 23:27:27.746: [ cssd][2834]111e38ba0 00000001 11e3a320 00000001 11e389a0 [....... ........]
2012-06-04 23:27:27.746: [ cssd][2834]111e38bb0 00000b12 00000b12 636c7373 6e6d7644 [........clssnmvd]
2012-06-04 23:27:27.746: [ cssd][2834]111e38bc0 69736b50 696e674d 6f6e6974 6f725468 [iskpingmonitorth]
2012-06-04 23:27:27.746: [ cssd][2834]111e38bd0 72656164 00000000 00000000 00000000 [read............]
2012-06-04 23:27:27.746: [ cssd][2834]111e38be0 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.746: [ cssd][2834]111e38bf0 00000000 00000000 00020000 00000000 [................]
2012-06-04 23:27:27.746: [ cssd][2834]111e38c00 00000001 1045f530 00000001 11e38b90 [.....e.0........]
2012-06-04 23:27:27.746: [ cssd][2834]111e38c10 00000001 100179b0 00000000 00000000 [......y.........]
2012-06-04 23:27:27.746: [ cssd][2834]111e38c20 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.747: [ cssd][2834] repeat 4 times
2012-06-04 23:27:27.747: [ cssd][2834]111e38c70 00000000 00000001 00000000 00000005 [................]
2012-06-04 23:27:27.747: [ cssd][2834]111e38c80 00000000 00001677 00000000 00000000 [.......w........]
2012-06-04 23:27:27.747: [ cssd][2834]argument/register addr=0x1.
dump of memory from 0x0 to 0x101
2012-06-04 23:27:27.747: [ cssd][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.748: [ cssd][2834] repeat 15 times
2012-06-04 23:27:27.748: [ cssd][2834]000000100 7c7243a6 [|rc.]
2012-06-04 23:27:27.748: [ cssd][2834]argument/register addr=0x1000f52a0.
dump of memory from 0x1000f5260 to 0x1000f53a0
2012-06-04 23:27:27.748: [ cssd][2834]1000f5260 6b436865 636b3a20 41626f72 74696e67 [kcheck: aborting]
2012-06-04 23:27:27.748: [ cssd][2834]1000f5270 2c202564 206f6620 25642025 7320766f [, %d of %d %s vo]
2012-06-04 23:27:27.748: [ cssd][2834]1000f5280 74696e67 20646973 6b732061 7661696c [ting disks avail]
2012-06-04 23:27:27.749: [ cssd][2834]1000f5290 61626c65 2c206e65 65642025 75004942 [able, need %u.ib]
2012-06-04 23:27:27.749: [ cssd][2834]1000f52a0 283a4353 534e4d30 30303138 3a290049 [(:cssnm00018:).i]
2012-06-04 23:27:27.749: [ cssd][2834]1000f52b0 636c7373 6e6d7644 69736b50 696e674d [clssnmvdiskpingm]
2012-06-04 23:27:27.749: [ cssd][2834]1000f52c0 6f6e6974 6f725468 72656164 3a20206e [onitorthread: n]
2012-06-04 23:27:27.749: [ cssd][2834]1000f52d0 6f20766f 74696e67 20646973 6b2c0049 [o voting disk,.i]
2012-06-04 23:27:27.749: [ cssd][2834]1000f52e0 636c7373 6e6d7644 69736b50 4d543a20 [clssnmvdiskpmt: ]
2012-06-04 23:27:27.749: [ cssd][2834]1000f52f0 736c6565 70696e67 20666f72 20256420 [sleeping for %d ]
2012-06-04 23:27:27.749: [ cssd][2834]1000f5300 6d730049 636c7373 6e6d7653 63616e43 [ms.iclssnmvscanc]
2012-06-04 23:27:27.749: [ cssd][2834]1000f5310 6f6d706c 6574696f 6e733a20 4661696c [ompletions: fail]
2012-06-04 23:27:27.749: [ cssd][2834]1000f5320 75726520 2564206e 6f746966 79696e67 [ure %d notifying]
2012-06-04 23:27:27.750: [ cssd][2834]1000f5330 20636c69 656e7473 206f6620 776f726b [ clients of work]
2012-06-04 23:27:27.750: [ cssd][2834]1000f5340 20636f6d 706c6574 696f6e2e 0049424d [ completion..ibm]
2012-06-04 23:27:27.750: [ cssd][2834]1000f5350 636c7373 6e6d7653 63616e43 6f6d706c [clssnmvscancompl]
2012-06-04 23:27:27.750: [ cssd][2834]1000f5360 6574696f 6e733a20 636f6d70 6c657465 [etions: complete]
2012-06-04 23:27:27.750: [ cssd][2834]1000f5370 64202564 20697465 6d730049 636c7373 [d %d items.iclss]
2012-06-04 23:27:27.750: [ cssd][2834]1000f5380 6e6d7652 6561644c 65617365 3a20656e [nmvreadlease: en]
2012-06-04 23:27:27.750: [ cssd][2834]1000f5390 74657200 636c7373 6e6d7652 6561644c &nbs