最近在vmware中搭建了一个oracle10g rac的双节点实验平台并将oracle rac从10.2.0.1升级到10.2.0.5，后来发现两台linux经常自动重
1)         背景介绍：
最近在vmware中搭建了一个oracle10g rac的双节点实验平台并将oracle rac从10.2.0.1升级到10.2.0.5，后来发现两台linux经常自动重启；
2)         平台信息：
vmware7 + oel5.7x64 + asmlib2.0 + oracle10.2.0.5
3)         /var/log/message日志：
node1:linux1
apr 18 20:44:18 linux1 syslogd 1.4.1: restart.
apr 18 20:44:18 linux1 kernel: klogd 1.4.1, log source = /proc/kmsg started.
apr 18 20:44:18 linux1 kernel: initializing cgroup subsys cpuset
apr 18 20:44:18 linux1 kernel: initializing cgroup subsys cpu
apr 18 20:44:18 linux1 kernel: linux version 2.6.32-200.13.1.el5uek (mockbuild@ca-build9.us.oracle.com) (gcc version 4.1.2 20080704 (red hat 4.1.2-50)) #1 smp wed jul 27 21:02:33 edt 2011
apr 18 20:44:18 linux1 kernel: command line: ro root=/dev/volgroup00/logvol00 rhgb quiet
apr 18 20:44:18 linux1 kernel: kernel supported cpus:
apr 18 20:44:18 linux1 kernel:   intel genuineintel
apr 18 20:44:18 linux1 kernel:   amd authenticamd
apr 18 20:44:18 linux1 kernel:   centaur centaurhauls
apr 18 20:44:18 linux1 kernel: bios-provided physical ram map:
apr 18 20:44:18 linux1 kernel: bios-e820: 0000000000000000 - 000000000009f800 (usable)
apr 18 20:44:18 linux1 kernel: bios-e820: 000000000009f800 - 00000000000a0000 (reserved)
apr 18 20:44:18 linux1 kernel: bios-e820: 00000000000ca000 - 00000000000cc000 (reserved)
apr 18 20:44:18 linux1 kernel: bios-e820: 00000000000dc000 - 00000000000e4000 (reserved)
apr 18 20:44:18 linux1 kernel: bios-e820: 00000000000e8000 - 0000000000100000 (reserved)
apr 18 20:44:18 linux1 kernel: bios-e820: 0000000000100000 - 00000000bfef0000 (usable)
apr 18 20:44:18 linux1 kernel: bios-e820: 00000000bfef0000 - 00000000bfeff000 (acpi data)
apr 18 20:44:18 linux1 kernel: bios-e820: 00000000bfeff000 - 00000000bff00000 (acpi nvs)
apr 18 20:44:18 linux1 kernel: bios-e820: 00000000bff00000 - 00000000c0000000 (usable)
apr 18 20:44:18 linux1 kernel: bios-e820: 00000000e0000000 - 00000000f0000000 (reserved)
apr 18 20:44:18 linux1 kernel: bios-e820: 00000000fec00000 - 00000000fec10000 (reserved)
apr 18 20:44:18 linux1 kernel: bios-e820: 00000000fee00000 - 00000000fee01000 (reserved)
apr 18 20:44:18 linux1 kernel: bios-e820: 00000000fffe0000 - 0000000100000000 (reserved)
apr 18 20:44:18 linux1 kernel: bios-e820: 0000000100000000 - 0000000140000000 (usable)
apr 18 20:44:18 linux1 kernel: dmi present.
node2:linux2
apr 18 20:43:35 linux2 kernel: o2net: connection to node linux1 (num 0) at 192.168.3.131:7777 has been idle for 30.0 seconds, shutting it down.
apr 18 20:43:35 linux2 kernel: (swapper,0,0):o2net_idle_timer:1498 here are some times that might help debug the situation: (tmr 1334752985.559806 now 1334753015.306532 dr 1334752985.559360 adv 1334752985.559806:1334752985.559807 func (b651ea27:504) 1334752951.27068:1334752951.27323)
apr 18 20:43:35 linux2 kernel: o2net: no longer connected to node linux1 (num 0) at 192.168.3.131:7777
apr 18 20:43:56 linux2 kernel: o2net: connection to node linux1 (num 0) at 192.168.3.131:7777 shutdown, state 7
apr 18 20:44:05 linux2 kernel: (o2net,3480,0):o2net_connect_expired:1659 error: no connection established with node 0 after 30.0 seconds, giving up and returning errors.
apr 18 20:44:24 linux2 avahi-daemon[4341]: registering new address record for 192.168.0.136 on eth0.
apr 18 20:44:26 linux2 kernel: o2net: connection to node linux1 (num 0) at 192.168.3.131:7777 shutdown, state 7
apr 18 20:44:28 linux2 last message repeated 2 times
apr 18 20:44:28 linux2 kernel: (o2hb-9938799a41,3564,1):o2dlm_eviction_cb:267 o2dlm has evicted node 0 from group 9938799a418642218a66fe77029de473
apr 18 20:44:28 linux2 kernel: (ocfs2rec,19793,1):ocfs2_replay_journal:1605 recovering node 0 from slot 0 on device (8,65)
apr 18 20:44:30 linux2 kernel: o2net: connection to node linux1 (num 0) at 192.168.3.131:7777 shutdown, state 8
apr 18 20:44:31 linux2 kernel: (ocfs2rec,19793,0):ocfs2_begin_quota_recovery:407 beginning quota recovery in slot 0
apr 18 20:44:31 linux2 kernel: (ocfs2_wq,3567,1):ocfs2_finish_quota_recovery:598 finishing quota recovery in slot 0
apr 18 20:44:31 linux2 kernel: (dlm_reco_thread,3573,0):dlm_get_lock_resource:836 9938799a418642218a66fe77029de473:$recovery: at least one node (0) to recover before lock mastery can begin
apr 18 20:44:31 linux2 kernel: (dlm_reco_thread,3573,0):dlm_get_lock_resource:870 9938799a418642218a66fe77029de473: recovery map is not empty, but must master $recovery lock now
apr 18 20:44:31 linux2 kernel: (dlm_reco_thread,3573,0):dlm_do_recovery:523 (3573) node 1 is the recovery master for the dead node 0 for domain 9938799a418642218a66fe77029de473
以上信息在两台机器中会交换出现，说明并不是总是固定的一台机器对另外一台超时。
4)         根据message信息报错，应该是o2cb的idle时间超限导致的，，系统中o2cb服务的状态为：
[oracle@linux1]service o2cb status
driver for configfs: loaded
filesystem configfs: mounted
stack glue driver: loaded
stack plugin o2cb: loaded
driver for ocfs2_dlmfs: loaded
filesystem ocfs2_dlmfs: mounted
checking o2cb cluster ocfs2: online
heartbeat dead threshold = 301
 network idle timeout: 30000                            /此处单位为毫秒，正式message中报的30秒
 network keepalive delay: 2000
 network reconnect delay: 2000
checking o2cb heartbeat: active