rac上是vmware vspere的虚拟机上有两个oel6.3的虚拟机，上面跑的库是11.2.0.4
rac上是vmware vspere的虚拟机上有两个oel6.3的虚拟机，，上面跑的库是11.2.0.4
发现节点2挂掉了
1.检查节点1的alert
[root@racnode1 racnode1]# pwd
/u01/apps/grid/gridhome/11.2.0/grid/log/racnode1
[root@racnode1 racnode1]# tail -1000 alertracnode1.log
2014-04-15 09:41:20.815:
[crsd(27311)]crs-2765:resource 'ora.net1.network' has failed on server 'racnode1'.
2014-04-15 09:41:42.760:
[cssd(26972)]crs-1612:network communication with node racnode2 (2) missing for 50% of timeout interval. removal of this node from cluster in 15.000 seconds
2014-04-15 09:41:50.763:
[cssd(26972)]crs-1611:network communication with node racnode2 (2) missing for 75% of timeout interval. removal of this node from cluster in 7.000 seconds
2014-04-15 09:41:54.764:
[cssd(26972)]crs-1610:network communication with node racnode2 (2) missing for 90% of timeout interval. removal of this node from cluster in 3.000 seconds
2014-04-15 09:41:57.766:
[cssd(26972)]crs-1607:node racnode2 is being evicted in cluster incarnation 291818318; details at (:cssnm00007:) in /u01/apps/grid/gridhome/11.2.0/grid/log/racnode1/cssd/ocssd.log.
2014-04-15 09:42:06.052:
[cssd(26972)]crs-1625:node racnode2, number 2, was manually shut down
2014-04-15 09:42:06.059:
[cssd(26972)]crs-1601:cssd reconfiguration complete. active nodes are racnode1 .
2014-04-15 09:42:06.950:
[crsd(27311)]crs-5504:node down event reported for node 'racnode2'.
2014-04-15 09:42:24.882:
[crsd(27311)]crs-2773:server 'racnode2' has been removed from pool 'generic'.
2014-04-15 09:42:24.882:
[crsd(27311)]crs-2773:server 'racnode2' has been removed from pool 'ora.pera'.
[root@racnode1 racnode1]#
节点2被驱逐时间：2014-04-15 09:41:57
2.查看ocssd.log
# more /u01/apps/grid/gridhome/11.2.0/grid/log/racnode1/cssd/ocssd.log |grep 2014-04-15 09:41
2014-04-15 09:41:23.620: [ cssd][906479360]clssnmsendingthread: sending status msg to all nodes
2014-04-15 09:41:23.621: [ cssd][906479360]clssnmsendingthread: sent 4 status msgs to all nodes
2014-04-15 09:41:28.758: [ cssd][906479360]clssnmsendingthread: sending status msg to all nodes
2014-04-15 09:41:28.758: [ cssd][906479360]clssnmsendingthread: sent 5 status msgs to all nodes
2014-04-15 09:41:33.145: [gipchgen][919283456] gipchainterfacefail: marking interface failing 0x7fc01821d4c0 { host '', haname 'css_racnode-cluster', local (nil), ip '172.168.1.11:52955', subnet '172.168.1.0', mask '255.255.255.0', mac '00-50-56-a1-7b-e2', ifname 'eth1', numref 1, numfail 0, idxboot 0, flags 0x184d }
2014-04-15 09:41:33.760: [ cssd][906479360]clssnmsendingthread: sending status msg to all nodes
2014-04-15 09:41:33.760: [gipchgen][920860416] gipchainterfacefail: marking interface failing 0x7fc024040a20 { host 'racnode2', haname 'css_racnode-cluster', local 0x7fc01821d4c0, ip '172.168.1.12:60678', subnet '172.168.1.0', mask '255.255.255.0', mac '', ifname '', numref 0, numfail 0, idxboot 4, flags 0x6 }
2014-04-15 09:41:33.760: [ cssd][906479360]clssnmsendingthread: sent 5 status msgs to all nodes
2014-04-15 09:41:33.760: [gipchgen][920860416] gipchainterfacedisable: disabling interface 0x7fc01821d4c0 { host '', haname 'css_racnode-cluster', local (nil), ip '172.168.1.11:52955', subnet '172.168.1.0', mask '255.255.255.0', mac '00-50-56-a1-7b-e2', ifname 'eth1', numref 0, numfail 1, idxboot 0, flags 0x19cd }
2014-04-15 09:41:33.760: [gipchgen][920860416] gipchainterfacedisable: disabling interface 0x7fc024040a20 { host 'racnode2', haname 'css_racnode-cluster', local 0x7fc01821d4c0, ip '172.168.1.12:60678', subnet '172.168.1.0', mask '255.255.255.0', mac '', ifname '', numref 0, numfail 0, idxboot 4, flags 0x86 }
2014-04-15 09:41:33.761: [gipchalo][920860416] gipchalowercleaninterfaces: performing cleanup of disabled interface 0x7fc024040a20 { host 'racnode2', haname 'css_racnode-cluster', local 0x7fc01821d4c0, ip '172.168.1.12:60678', subnet '172.168.1.0', mask '255.255.255.0', mac '', ifname '', numref 0, numfail 0, idxboot 4, flags 0xa6 }
2014-04-15 09:41:33.885: [gipchdem][920860416] gipchaworkercleaninterface: performing cleanup of disabled interface 0x7fc01821d4c0 { host '', haname 'css_racnode-cluster', local (nil), ip '172.168.1.11:52955', subnet '172.168.1.0', mask '255.255.255.0', mac '00-50-56-a1-7b-e2', ifname 'eth1', numref 0, numfail 0, idxboot 0, flags 0x19ed }
2014-04-15 09:41:34.145: [gipchdem][919283456] gipchadaemoninfrequest: sent local interfacerequest, hctx 0x1e22410 [0000000000000010] { gipchacontext : host 'racnode1', name 'css_racnode-cluster', luid '8bbba732-00000000', numnode 1, numinf 0, usrflags 0x0, flags 0x63 } to gipcd
2014-04-15 09:41:37.993: [gipchalo][920860416] gipchalowerprocessnode: no valid interfaces found to node for 5240 ms, node 0x7fc018226540 { host 'racnode2', haname 'css_racnode-cluster', srcluid 8bbba732-d4368a71, dstluid 0fc9914c-ec29f220 numinf 0, contigseq 1787047, lastack 1786995, lastvalidack 1787046, sendseq [1786996 : 1787014], createtime 1683284, sentregister 1, localmonitor 1, flags 0x2408 }
2014-04-15 09:41:38.762: [ cssd][906479360]clssnmsendingthread: sending status msg to all nodes
2014-04-15 09:41:38.762: [ cssd][906479360]clssnmsendingthread: sent 5 status msgs to all nodes
2014-04-15 09:41:40.146: [gipchdem][919283456] gipchadaemoninfrequest: sent local interfacerequest, hctx 0x1e22410 [0000000000000010] { gipchacontext : host 'racnode1', name 'css_racnode-cluster', luid '8bbba732-00000000', numnode 1, numinf 0, usrflags 0x0, flags 0x63 } to gipcd
2014-04-15 09:41:42.760: [ cssd][908056320]clssnmpollingthread: node racnode2 (2) at 50% heartbeat fatal, removal in 15.000 seconds
2014-04-15 09:41:42.760: [ cssd][908056320]clssnmpollingthread: node racnode2 (2) is impending reconfig, flag 2294796, misstime 15000
2014-04-15 09:41:42.760: [ cssd][908056320]clssnmpollingthread: local disktimeout set to 27000 ms, remote disk timeout set to 27000, impending reconfig status(1)
2014-04-15 09:41:42.764: [ cssd][906479360]clssnmsendingthread: sending status msg to all nodes
2014-04-15 09:41:42.764: [ cssd][906479360]clssnmsendingthread: sent 4 status msgs to all nodes
2014-04-15 09:41:43.764: [gipchalo][920860416] gipchalowerprocessnode: no valid interfaces found to node for 11010 ms, node 0x7fc018226540 { host 'racnode2', haname 'css_racnode-cluster', srcluid 8bbba732-d4368a71, dstluid 0fc9914c-ec29f220 numinf 0, contigseq 1787047, lastack 1786995, lastvalidack 1787046, sendseq [1786996 : 1787025], createtime 1683284, sentregister 1, localmonitor 1, flags 0x2408 }
2014-04-15 09:41:43.822: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526103/1203750934
2014-04-15 09:41:43.906: [ cssd][914388736]clssnmvdhbvalidatencopy: node 2, racnode2, has a disk hb, but no network hb, dhb has rcfg 291818318, wrtcnt, 1193997, lats 1203751024, lastseqno 1175771, uniqueness 1396323841, timestamp 1397526102/1203310914
2014-04-15 09:41:45.102: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526105/1203752214
2014-04-15 09:41:45.772: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526105/1203752884
2014-04-15 09:41:46.147: [gipchdem][919283456] gipchadaemoninfrequest: sent local interfacerequest, hctx 0x1e22410 [0000000000000010] { gipchacontext : host 'racnode1', name 'css_racnode-cluster', luid '8bbba732-00000000', numnode 1, numinf 0, usrflags 0x0, flags 0x63 } to gipcd
2014-04-15 09:41:46.181: [ cssd][914388736]clssnmvdhbvalidatencopy: node 2, racnode2, has a disk hb, but no network hb, dhb has rcfg 291818318, wrtcnt, 1193999, lats 1203753294, lastseqno 1193997, uniqueness 1396323841, timestamp 1397526104/1203313164
2014-04-15 09:41:46.782: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526106/1203753894
2014-04-15 09:41:47.120: [ cssd][914388736]clssnmvdhbvalidatencopy: node 2, racnode2, has a disk hb, but no network hb, dhb has rcfg 291818318, wrtcnt, 1194001, lats 1203754234, lastseqno 1193999, uniqueness 1396323841, timestamp 1397526106/1203314834
2014-04-15 09:41:47.622: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526107/1203754734
2014-04-15 09:41:47.765: [ cssd][906479360]clssnmsendingthread: sending status msg to all nodes
2014-04-15 09:41:47.765: [ cssd][906479360]clssnmsendingthread: sent 5 status msgs to all nodes
2014-04-15 09:41:48.004: [ cssd][914388736]clssnmvdhbvalidatencopy: node 2, racnode2, has a disk hb, but no network hb, dhb has rcfg 291818318, wrtcnt, 1194002, lats 1203755124, lastseqno 1194001, uniqueness 1396323841, timestamp 1397526107/1203315684
2014-04-15 09:41:48.507: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526108/1203755624
2014-04-15 09:41:48.774: [ cssd][914388736]clssnmvdhbvalidatencopy: node 2, racnode2, has a disk hb, but no network hb, dhb has rcfg 291818318, wrtcnt, 1194003, lats 1203755894, lastseqno 1194002, uniqueness 1396323841, timestamp 1397526108/1203316514
2014-04-15 09:41:49.213: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526109/1203756324
2014-04-15 09:41:49.766: [gipchalo][920860416] gipchalowerprocessnode: no valid interfaces found to node for 17010 ms, node 0x7fc018226540 { host 'racnode2', haname 'css_racnode-cluster', srcluid 8bbba732-d4368a71, dstluid 0fc9914c-ec29f220 numinf 0, contigseq 1787047, lastack 1786995, lastvalidack 1787046, sendseq [1786996 : 1787037], createtime 1683284, sentregister 1, localmonitor 1, flags 0x2408 }
2014-04-15 09:41:49.893: [ cssd][914388736]clssnmvdhbvalidatencopy: node 2, racnode2, has a disk hb, but no network hb, dhb has rcfg 291818318, wrtcnt, 1194004, lats 1203757004, lastseqno 1194003, uniqueness 1396323841, timestamp 1397526109/1203317334
2014-04-15 09:41:49.903: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526109/1203757014
2014-04-15 09:41:50.528: [ cssd][914388736]clssnmvdhbvalidatencopy: node 2, racnode2, has a disk hb, but no network hb, dhb has rcfg 291818318, wrtcnt, 1194005, lats 1203757644, lastseqno 1194004, uniqueness 1396323841, timestamp 1397526109/1203317964
2014-04-15 09:41:50.548: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526110/1203757664
2014-04-15 09:41:50.763: [ cssd][908056320]clssnmpollingthread: node racnode2 (2) at 75% heartbeat fatal, removal in 7.000 seconds
2014-04-15 09:41:50.875: [ cssd][914388736]clssnmvdhbvalidatencopy: node 2, racnode2, has a disk hb, but no network hb, dhb has rcfg 291818318, wrtcnt, 1194006, lats 1203757994, lastseqno 1194005, uniqueness 1396323841, timestamp 1397526110/1203318664
2014-04-15 09:41:51.113: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526111/1203758224
2014-04-15 09:41:51.396: [ cssd][914388736]clssnmvdhbvalidatencopy: node 2, racnode2, has a disk hb, but no network hb, dhb has rcfg 291818318, wrtcnt, 1194007, lats 1203758514, lastseqno 1194006, uniqueness 1396323841, timestamp 1397526110/1203319204
2014-04-15 09:41:51.692: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526111/1203758804
2014-04-15 09:41:52.148: [gipchdem][919283456] gipchadaemoninfrequest: sent local interfacerequest, hctx 0x1e22410 [0000000000000010] { gipchacontext : host 'racnode1', name 'css_racnode-cluster', luid '8bbba732-00000000', numnode 1, numinf 0, usrflags 0x0, flags 0x63 } to gipcd
2014-04-15 09:41:52.294: [ cssd][914388736]clssnmvdhbvalidatencopy: node 2, racnode2, has a disk hb, but no network hb, dhb has rcfg 291818318, wrtcnt, 1194008, lats 1203759404, lastseqno 1194007, uniqueness 1396323841, timestamp 1397526111/1203319754
2014-04-15 09:41:52.305: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526112/1203759424
2014-04-15 09:41:52.767: [ cssd][906479360]clssnmsendingthread: sending status msg to all nodes
2014-04-15 09:41:52.767: [ cssd][906479360]clssnmsendingthread: sent 5 status msgs to all nodes
2014-04-15 09:41:52.772: [ cssd][914388736]clssnmvdhbvalidatencopy: node 2, racnode2, has a disk hb, but no network hb, dhb has rcfg 291818318, wrtcnt, 1194009, lats 1203759884, lastseqno 1194008, uniqueness 1396323841, timestamp 1397526112/1203320364
2014-04-15 09:41:52.974: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526112/1203760084
2014-04-15 09:41:53.793: [ cssd][914388736]clssnmvdhbvalidatencopy: node 2, racnode2, has a disk hb, but no network hb, dhb has rcfg 291818318, wrtcnt, 1194010, lats 1203760904, lastseqno 1194009, uniqueness 1396323841, timestamp 1397526112/1203321034
2014-04-15 09:41:53.934: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526113/1203761044
2014-04-15 09:41:54.401: [ cssd][914388736]clssnmvdhbvalidatencopy: node 2, racnode2, has a disk hb, but no network hb, dhb has rcfg 291818318, wrtcnt, 1194012, lats 1203761514, lastseqno 1194010, uniqueness 1396323841, timestamp 1397526114/1203322404
2014-04-15 09:41:54.494: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526114/1203761604
2014-04-15 09:41:54.764: [ cssd][908056320]clssnmpollingthread: node racnode2 (2) at 90% heartbeat fatal, removal in 3.000 seconds, seedhbimpd 1
2014-04-15 09:41:55.044: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526115/1203762154
2014-04-15 09:41:55.134: [ cssd][914388736]clssnmvdhbvalidatencopy: node 2, racnode2, has a disk hb, but no network hb, dhb has rcfg 291818318, wrtcnt, 1194013, lats 1203762254, lastseqno 1194012, uniqueness 1396323841, timestamp 1397526114/1203322964
2014-04-15 09:41:55.640: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526115/1203762754
2014-04-15 09:41:55.768: [gipchalo][920860416] gipchalowerprocessnode: no valid interfaces found to node for 23010 ms, node 0x7fc018226540 { host 'racnode2', haname 'css_racnode-cluster', srcluid 8bbba732-d4368a71, dstluid 0fc9914c-ec29f220 numinf 0, contigseq 1787047, lastack 1786995, lastvalidack 1787046, sendseq [1786996 : 1787048], createtime 1683284, sentregister 1, localmonitor 1, flags 0x2408 }
2014-04-15 09:41:56.498: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526116/1203763614
2014-04-15 09:41:57.145: [ cssd][914388736]clssnmvdhbvalidatencopy: node 2, racnode2, has a disk hb, but no network hb, dhb has rcfg 291818318, wrtcnt, 1194015, lats 1203764254, lastseqno 1194013, uniqueness 1396323841, timestamp 1397526116/1203324554
2014-04-15 09:41:57.645: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526117/1203764764
2014-04-15 09:41:57.764: [ cssd][908056320]clssnmpollingthread: removal started for node racnode2 (2), flags 0x23040c, state 3, wt4c 0
2014-04-15 09:41:57.764: [ cssd][908056320]clssnmmarknodeforremoval: node 2, racnode2 marked for removal
2014-04-15 09:41:57.765: [ cssd][908056320]clssnmdischelper: racnode2, node(2) connection failed, endp (0x3938), probe(0x7fc000000000), ninf->endp 0x3938
2014-04-15 09:41:57.765: [ cssd][908056320]clssnmdischelper: node 2 clean up, endp (0x3938), init state 5, cur state 5
2014-04-15 09:41:57.765: [gipcxcpt][908056320] gipcinternaldissociate: obj 0x7fc024056a70 [0000000000003938] { gipcendpoint : localaddr 'gipcha://racnode1:nm2_racnode-cluster/da4b-8718-50e0-b51', remoteaddr 'gipcha://racnode2:7b5c-4672-87e8-565', numpend 1, numready 0, numdone 0, numdead 0, numtransfer 0, objflags 0x0, pidpeer 0, readyref (nil), ready 1, wobj 0x7fc02402dd90, sendp (nil)flags 0x138606, usrflags 0x0 } not associated with any container, ret gipcretfail (1)
2014-04-15 09:41:57.765: [gipcxcpt][908056320] gipcdissociatef [clssnmdischelper : clssnm.c : 3485]: exception[ ret gipcretfail (1) ] failed to dissociate obj 0x7fc024056a70 [0000000000003938] { gipcendpoint : localaddr 'gipcha://racnode1:nm2_racnode-cluster/da4b-8718-50e0-b51', remoteaddr 'gipcha://racnode2:7b5c-4672-87e8-565', numpend 1, numready 0, numdone 0, numdead 0, numtransfer 0, objflags 0x0, pidpeer 0, readyref (nil), ready 1, wobj 0x7fc02402dd90, sendp (nil)flags 0x138606, usrflags 0x0 }, flags 0x0
2014-04-15 09:41:57.765: [ cssd][904902400]clssnmdosyncupdate: initiating sync 291818318
2014-04-15 09:41:57.765: [ cssd][904902400]clsssccompareswapeventvalue: changed nmreconfiginprogress val 1, from -1, changes 7
2014-04-15 09:41:57.765: [ cssd][904902400]clssnmdosyncupdate: local disk timeout set to 27000 ms, remote disk timeout set to 27000
2014-04-15 09:41:57.765: [ cssd][904902400]clssnmdosyncupdate: new values for local disk timeout and remote disk timeout will take effect when the sync is completed.
2014-04-15 09:41:57.765: [ cssd][903325440]clssnmdiscendp: gipcdestroy 0x3938
2014-04-15 09:41:57.765: [ cssd][904902400]clssnmdosyncupdate: starting cluster reconfig with incarnation 291818318
2014-04-15 09:41:57.765: [ cssd][904902400]clssnmsetupackwait: ack message type (11)
2014-04-15 09:41:57.765: [ cssd][904902400]clssnmsetupackwait: node(1) is alive
2014-04-15 09:41:57.765: [ cssd][904902400]clssnmsendsync: syncseqno(291818318), indicating exadata fence initialization complete
2014-04-15 09:41:57.765: [ cssd][904902400]list of nodes that have acked my sync: null
2014-04-15 09:41:57.765: [ cssd][904902400]clssnmsendsync: syncseqno(291818318)
2014-04-15 09:41:57.765: [ cssd][903325440]clssnmhandlesync: node racnode1, number 1, is exadata fence capable
2014-04-15 09:41:57.765: [ cssd][904902400]clssnmwaitforacks: ack message type(11), ackcount(1)
2014-04-15 09:41:57.765: [ cssd][903325440]clssscupdateeventvalue: nmreconfiginprogress val 1, changes 8
2014-04-15 09:41:57.765: [ cssd][903325440]clssnmhandlesync: local disk timeout set to 27000 ms, remote disk timeout set to 27000
2014-04-15 09:41:57.765: [ cssd][903325440]clssnmhandlesync: initleader 1 newleader 1
2014-04-15 09:41:57.765: [ cssd][903325440]clssnmqueueclientevent: sending event(2), type 2, incarn 291818317
2014-04-15 09:41:57.765: [ cssd][903325440]clssnmqueueclientevent: node[1] state = 3, birth = 291818316, unique = 1396323475
2014-04-15 09:41:57.765: [ cssd][903325440]clssnmqueueclientevent: node[2] state = 5, birth = 291818317, unique = 1396323841
2014-04-15 09:41:57.765: [ cssd][903325440]clssnmhandlesync: acknowledging sync: src[1] srcname[racnode1] seq[11] sync[291818318]
2014-04-15 09:41:57.765: [ cssd][903325440]clssnmsendack: node 1, racnode1, syncseqno(291818318) type(11)
2014-04-15 09:41:57.765: [ cssd][903325440]clssnmhandleack: received ack type 11 from node racnode1, number 1, with seq 0 for sync 291818318, waiting for 0 acks
2014-04-15 09:41:57.765: [ cssd][904902400]clssnmsendsync: syncseqno(291818318), indicating exadata fence initialization complete
2014-04-15 09:41:57.765: [ cssd][941274880]clssgmstartnmmon: node 1 active, birth 291818316
2014-04-15 09:41:57.765: [ cssd][941274880]clssgmstartnmmon: node 2 active, birth 291818317
2014-04-15 09:41:57.765: [ cssd][941274880]nmevent_suspend [00][00][00][06]
2014-04-15 09:41:57.765: [ cssd][941274880]clssgmcompareswapeventvalue: changed cminfo state val 5, from 11, changes 21
2014-04-15 09:41:57.765: [ cssd][941274880]clssgmsuspendallgrocks: issue suspend
2014-04-15 09:41:57.765: [ cssd][904902400]list of nodes that have acked my sync: 1
2014-04-15 09:41:57.765: [ cssd][904902400]clssnmwaitforacks: done, syncseq(291818318), msg type(11)
2014-04-15 09:41:57.765: [ cssd][941274880]clssgmqueuegrockevent: groupname(igperaperaxdb) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.765: [ cssd][904902400]clssnmsetminmaxversion:node1 product/protocol (11.2/1.4)
2014-04-15 09:41:57.766: [ cssd][904902400]clssnmsetminmaxversion: properties common to all nodes: 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17
2014-04-15 09:41:57.766: [ cssd][904902400]clssnmsetminmaxversion: min product/protocol (11.2/1.4)
2014-04-15 09:41:57.766: [ cssd][904902400]clssnmsetminmaxversion: max product/protocol (11.2/1.4)
2014-04-15 09:41:57.766: [ cssd][941274880]clssgmqueuegrockevent: groupname(ig+asmsys$users) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.766: [ cssd][904902400]clssnmneedconfreq: no configuration to change
2014-04-15 09:41:57.766: [ cssd][904902400]clssnmdosyncupdate: terminating node 2, racnode2, misstime(30000) state(5)
2014-04-15 09:41:57.766: [ cssd][904902400]clssnmdosyncupdate: wait for 0 vote ack(s)
2014-04-15 09:41:57.766: [ cssd][904902400]clssnmcheckdskinfo: checking disk info...
2014-04-15 09:41:57.766: [ cssd][941274880]clssgmqueuegrockevent: groupname(igperasys$users) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.766: [ cssd][904902400]clssnmchecksplit: node 2, racnode2, is alive, dhb (1397526116, 1203324554) more than disk timeout of 27000 after the last nhb (1397526085, 1203293884)
2014-04-15 09:41:57.766: [ cssd][941274880]clssgmqueuegrockevent: groupname(crs_version) count(3) master(0) event(2), incarn 3, mbrc 3, to member 0, events 0x20, state 0x0
2014-04-15 09:41:57.766: [ cssd][904902400]clssnmcheckdskinfo: my cohort: 1
2014-04-15 09:41:57.766: [ cssd][904902400]clssnmremove: start
2014-04-15 09:41:57.766: [ cssd][904902400](:cssnm00007:)clssnmrremovenode: evicting node 2, racnode2, from the cluster in incarnation 291818318, node birth incarnation 291818317, death incarnation 291818318, stateflags 0x234000 uniqueness value 1396323841
2014-04-15 09:41:57.766: [ cssd][941274880]clssgmqueuegrockevent: groupname(crs_version) count(3) master(0) event(2), incarn 3, mbrc 3, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.767: [ cssd][941274880]clssgmqueuegrockevent: groupname(crf-) count(4) master(0) event(2), incarn 4, mbrc 4, to member 0, events 0x38, state 0x0
2014-04-15 09:41:57.767: [ cssd][941274880]clssgmqueuegrockevent: groupname(crf-) count(4) master(0) event(2), incarn 4, mbrc 4, to member 1, events 0x38, state 0x0
2014-04-15 09:41:57.767: [ default][904902400]kgzf_gen_node_reid2: generated reid cid=a2998261b3ccff8abf36841a04ffe27b,icin=291818316,nmn=2,lnid=291818317,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100
2014-04-15 09:41:57.767: [ cssd][941274880]clssgmqueuegrockevent: groupname(clsn.onsproc.master) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0xa0, state 0x0
2014-04-15 09:41:57.767: [ cssd][904902400]clssnmrfencesage: fenced node racnode2, number 2, with exadata, handle 0
2014-04-15 09:41:57.767: [ cssd][904902400]clssnmsendshutdown: req to node 2, kill time 1203764884
2014-04-15 09:41:57.767: [ cssd][904902400]clssnmsendmsg: not connected to node 2
2014-04-15 09:41:57.767: [ cssd][904902400]clssnmsendshutdown: send to node 2 failed
2014-04-15 09:41:57.767: [ cssd][904902400]clssnmwaitonevictions: start
2014-04-15 09:41:57.767: [ cssd][941274880]clssgmqueuegrockevent: groupname(db+asm) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x68, state 0x0
2014-04-15 09:41:57.767: [ cssd][904902400]clssnmwaitonevictions: node 2, undead 1, exadata fence handle 0 kill reqest id 0, last dhb (1397526116, 1203324554, 1201720), seedhbimpd true
2014-04-15 09:41:57.767: [ cssd][941274880]clssgmqueuegrockevent: groupname(dg+asm) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x0, state 0x0
2014-04-15 09:41:57.767: [ cssd][941274880]clssgmqueuegrockevent: groupname(ig+asmsys$background) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.767: [ cssd][941274880]clssgmqueuegrockevent: groupname(dbpera) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x68, state 0x0
2014-04-15 09:41:57.767: [ cssd][941274880]clssgmqueuegrockevent: groupname(vt+asm) count(2) master(1) event(2), incarn 6, mbrc 2, to member 1, events 0x60, state 0x0
2014-04-15 09:41:57.767: [ cssd][941274880]clssgmqueuegrockevent: groupname(igperasys$background) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.767: [ cssd][941274880]clssgmqueuegrockevent: groupname(dg_fra) count(2) master(1) event(2), incarn 5, mbrc 2, to member 1, events 0x4, state 0x0
2014-04-15 09:41:57.767: [ cssd][941274880]clssgmqueuegrockevent: groupname(dg+asm0) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x0, state 0x0
2014-04-15 09:41:57.768: [ cssd][941274880]clssgmqueuegrockevent: groupname(gr+gcr1) count(4) master(0) event(2), incarn 26, mbrc 4, to member 0, events 0x280, state 0x0
2014-04-15 09:41:57.768: [ cssd][941274880]clssgmqueuegrockevent: groupname(gr+gcr1) count(4) master(0) event(2), incarn 26, mbrc 4, to member 2, events 0x280, state 0x0
2014-04-15 09:41:57.768: [ cssd][941274880]clssgmqueuegrockevent: groupname(dg_crs) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x4, state 0x0
2014-04-15 09:41:57.768: [ cssd][941274880]clssgmqueuegrockevent: groupname(dgpera-) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x0, state 0x0
2014-04-15 09:41:57.768: [ cssd][941274880]clssgmqueuegrockevent: groupname(dgpera0) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x0, state 0x0
2014-04-15 09:41:57.768: [ cssd][941274880]clssgmqueuegrockevent: groupname(dg_crs1) count(2) master(1) event(2), incarn 5, mbrc 2, to member 1, events 0x4, state 0x0
2014-04-15 09:41:57.768: [ cssd][941274880]clssgmqueuegrockevent: groupname(dg_data) count(2) master(1) event(2), incarn 5, mbrc 2, to member 1, events 0x4, state 0x0
2014-04-15 09:41:57.768: [ cssd][941274880]clssgmqueuegrockevent: groupname(clsn.rlb.pera.master) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0xa0, state 0x0
2014-04-15 09:41:57.768: [ cssd][941274880]clssgmqueuegrockevent: groupname(clsframe) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
2014-04-15 09:41:57.768: [ cssd][941274880]clssgmqueuegrockevent: groupname(evmdmain) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
2014-04-15 09:41:57.768: [ cssd][941274880]clssgmqueuegrockevent: groupname(crsdmain) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
2014-04-15 09:41:57.768: [ cssd][941274880]clssgmqueuegrockevent: groupname(evmdmain2) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
2014-04-15 09:41:57.769: [ cssd][941274880]clssgmqueuegrockevent: groupname(igperaall) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.769: [ cssd][941274880]clssgmqueuegrockevent: groupname(ctssgroup) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
2014-04-15 09:41:57.769: [ cssd][941274880]clssgmqueuegrockevent: groupname(clsn.aqproc.pera.master) count(2) master(2) event(2), incarn 2, mbrc 2, to member 1, events 0xa0, state 0x0
2014-04-15 09:41:57.769: [ cssd][941274880]clssgmqueuegrockevent: groupname(igperapera) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.769: [ cssd][941274880]clssgmqueuegrockevent: groupname(ocr_racnode-cluster) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x78, state 0x0
2014-04-15 09:41:57.769: [ cssd][941274880]clssgmsuspendallgrocks: done
2014-04-15 09:41:57.769: [ cssd][941274880]clssgmcompareswapeventvalue: changed cminfo state val 2, from 5, changes 22
2014-04-15 09:41:57.769: [ cssd][941274880]clssgmupdateeventvalue: connectednodes val 291818317, changes 7
2014-04-15 09:41:57.769: [ cssd][941274880]clssgmcleanupnodecontexts(): cleaning up nodes, rcfg(291818317)
2014-04-15 09:41:57.769: [ cssd][941274880]clssgmcleanupnodecontexts(): successful cleanup of nodes rcfg(291818317)
2014-04-15 09:41:57.769: [gipchaup][920860416] gipchaupperdisconnect: initiated discconnect umsg 0x7fc024054c60 { msg 0x7fc0240565c8, ret gipcretrequestpending (15), flags 0x2 }, msg 0x7fc0240565c8 { type gipchamsgtypedisconnect (5), srccid 00000000-000038ed, dstcid 00000000-0000059b }, endp 0x7fc0240553a0 [00000000000038ed] { gipchaendpoint : port 'nm2_racnode-cluster/da4b-8718-50e0-b51b', peer 'racnode2:7b5c-4672-87e8-565a', srccid 00000000-000038ed, dstcid 00000000-0000059b, numsend 29, maxsend 100, grouplisttype 2, hagroup 0x2021010, usrflags 0x4000, flags 0x21c }
2014-04-15 09:41:57.769: [ cssd][941274880]clssgmstartnmmon: completed node cleanup
2014-04-15 09:41:57.769: [ cssd][909633280]clssgmupdateeventvalue: holdrequest val 1, changes 5
2014-04-15 09:41:57.775: [ cssd][906479360]clssnmsendingthread: sending status msg to all nodes
2014-04-15 09:41:57.775: [ cssd][906479360]clssnmsendingthread: sent 5 status msgs to all nodes
2014-04-15 09:41:57.867: [ cssd][904902400]clssnmwaitonevictions: node 2, undead 1, exadata fence handle 0 kill reqest id 0, last dhb (1397526116, 1203324554, 1201720), seedhbimpd true
2014-04-15 09:41:57.934: [ cssd][914388736]clssnmvdhbvalidatencopy: node 2, racnode2, has a disk hb, but no network hb, dhb has rcfg 291818318, wrtcnt, 1194016, lats 1203765044, lastseqno 1194015, uniqueness 1396323841, timestamp 1397526117/1203325474
2014-04-15 09:41:57.934: [ cssd][914388736]clssnmvdiskevict: kill block write, file /dev/asm-diskb flags 0x00010004, kill block unique 1396323841, stamp 1203764884/1203764884
2014-04-15 09:41:57.934: [ cssd][904902400]clssnmwaitonevictions: node 2, undead 1, exadata fence handle 0 kill reqest id 0, last dhb (1397526117, 1203325474, 1201721), seedhbimpd true
2014-04-15 09:41:58.035: [ cssd][904902400]clssnmwaitonevictions: node 2, undead 1, exadata fence handle 0 kill reqest id 0, last dhb (1397526117, 1203325474, 1201721), seedhbimpd true
2014-04-15 09:41:58.135: [ cssd][904902400]clssnmwaitonevictions: node 2, undead 1, exadata fence handle 0 kill reqest id 0, last dhb (1397526117, 1203325474, 1201721), seedhbimpd true
2014-04-15 09:41:58.149: [gipchdem][919283456] gipchadaemoninfrequest: sent local interfacerequest, hctx 0x1e22410 [0000000000000010] { gipchacontext : host 'racnode1', name 'css_racnode-cluster', luid '8bbba732-00000000', numnode 1, numinf 0, usrflags 0x0, flags 0x63 } to gipcd
2014-04-15 09:41:58.205: [ cssd][917706496]clssnmvdiskping: writing with status 0x3, timestamp 1397526118/1203765314
2014-04-15 09:41:58.235: [ cssd][904902400]clssnmwaitonevictions: node 2, undead 1, exadata fence handle 0 kill reqest id 0, last dhb (1397526117, 1203325474, 1201721), seedhbimpd true
2014-04-15 09:41:58.335: [ cssd][904902400]clssnmwaitonevictions: node 2, undead 1, exadata fence handle 0 kill reqest id 0, last dhb (1397526117, 1203325474, 1201721), seedhbimpd true
2014-04-15 09:41:58.435: [ cssd][904902400]clssnmwaitonevictions: node 2, undead 1, exadata fence handle 0 kill reqest id 0, last dhb (1397526117, 1203325474, 1201721), seedhbimpd true
2014-04-15 09:41:58.535: [ cssd][904902400]clssnmwaitonevictions: node 2, undead 1, exadata fence handle 0 kill reqest id 0, last dhb (1397526117, 1203325474, 1201721), seedhbimpd true
2014-04-15 09:41:58.551: [ cssd][916129536]clssnmvdiskkillcheck: not evicted, file /dev/asm-diskb flags 0x00000000, kill block unique 0, my unique 1396323475
2014-04-15 09:41:58.636: [ cssd][904902400]clssnmwaitonevictions: node 2, undead 1, exadata fence handle 0 kill reqest id 0, last dhb (1397526117, 1203325474, 1201721), seedhbimpd true
更多详情见请继续阅读下一页的精彩内容：