oracle 11g rac asm磁盘全部丢失后的恢复，oracle 11.2.0.3 rac on oracle linux 6 x86_64，只有一个asm外部冗余磁盘组mdash;m
一、环境描述
（1）oracle 11.2.0.3 rac on oracle linux 6 x86_64，只有一个asm外部冗余磁盘组——data；
（2）ocr，votedisk，datafile，controlfile，spfile全部位于这个磁盘组上； 
二、故障描述（1）存储故障导致asm磁盘丢失。
（2）crs因为ocr和votedisk的丢失，除了ohas还联机外，clusterware服务都已经停止。
三、备份情况
（1）rman备份：包括controlfile，，database，spfile，archivelog，
（2）ocr备份：没有进行过人工备份，在$crs_home/cdata目录下有crs自动备份文件。
四、操作步骤说明：准使用crs自动备份的文件恢复ocr，使用rman备份来恢复数据库；准备恢复数据的同时，调整asm磁盘组，将ocr，votedisk同数据库文件分开存放。
推荐阅读：
oracle 11g从入门到精通 pdf+光盘源代码  
ubuntu 12.04(amd64)安装完oracle 11gr2后各种问题解决方法
4.1 恢复ocr和votedisk(1) 在所有rac节点上停止crs服务
[root@rac1 ~]# crsctl stop has -fcrs-2791: starting shutdown of oracle high availability services-managed resources on 'rac1'crs-2673: attempting to stop 'ora.mdnsd' on 'rac1'crs-2673: attempting to stop 'ora.crf' on 'rac1'crs-2677: stop of 'ora.mdnsd' on 'rac1' succeededcrs-2677: stop of 'ora.crf' on 'rac1' succeededcrs-2673: attempting to stop 'ora.gipcd' on 'rac1'crs-2677: stop of 'ora.gipcd' on 'rac1' succeededcrs-2673: attempting to stop 'ora.gpnpd' on 'rac1'crs-2677: stop of 'ora.gpnpd' on 'rac1' succeededcrs-2793: shutdown of oracle high availability services-managed resources on 'rac1' has completedcrs-4133: oracle high availability services has been stopped.
[root@rac2 ~]# crsctl stop has -fcrs-2791: starting shutdown of oracle high availability services-managed resources on 'rac2'crs-2673: attempting to stop 'ora.mdnsd' on 'rac2'crs-2673: attempting to stop 'ora.crf' on 'rac2'crs-2677: stop of 'ora.mdnsd' on 'rac2' succeededcrs-2677: stop of 'ora.crf' on 'rac2' succeededcrs-2673: attempting to stop 'ora.gipcd' on 'rac2'crs-2677: stop of 'ora.gipcd' on 'rac2' succeededcrs-2673: attempting to stop 'ora.gpnpd' on 'rac2'crs-2677: stop of 'ora.gpnpd' on 'rac2' succeededcrs-2793: shutdown of oracle high availability services-managed resources on 'rac2' has completedcrs-4133: oracle high availability services has been stopped. 
(2) 在一个节点上以nocrs方式启动crs，此操作会启动asm实例。
[root@rac1 ~]# crsctl start crs -excl -nocrscrs-4123: oracle high availability services has been started.crs-2672: attempting to start 'ora.mdnsd' on 'rac1'crs-2676: start of 'ora.mdnsd' on 'rac1' succeededcrs-2672: attempting to start 'ora.gpnpd' on 'rac1'crs-2676: start of 'ora.gpnpd' on 'rac1' succeededcrs-2672: attempting to start 'ora.cssdmonitor' on 'rac1'crs-2672: attempting to start 'ora.gipcd' on 'rac1'crs-2676: start of 'ora.cssdmonitor' on 'rac1' succeededcrs-2676: start of 'ora.gipcd' on 'rac1' succeededcrs-2672: attempting to start 'ora.cssd' on 'rac1'crs-2672: attempting to start 'ora.diskmon' on 'rac1'crs-2676: start of 'ora.diskmon' on 'rac1' succeededcrs-2676: start of 'ora.cssd' on 'rac1' succeededcrs-2679: attempting to clean 'ora.cluster_interconnect.haip' on 'rac1'crs-2672: attempting to start 'ora.ctssd' on 'rac1'crs-2681: clean of 'ora.cluster_interconnect.haip' on 'rac1' succeededcrs-2672: attempting to start 'ora.cluster_interconnect.haip' on 'rac1'crs-2676: start of 'ora.ctssd' on 'rac1' succeededcrs-2676: start of 'ora.cluster_interconnect.haip' on 'rac1' succeededcrs-2672: attempting to start 'ora.asm' on 'rac1'crs-2676: start of 'ora.asm' on 'rac1' succeeded 
(3) 新添加了三块磁盘，已经使用udev进行了绑定，查看磁盘状态。
[root@rac1 ~]# su - grid[grid@rac1 ~]$ sqlplus / as sysasmsql*plus: release 11.2.0.3.0 production on fri jul 5 17:41:49 2013copyright (c) 1982, 2011, oracle. all rights reserved.connected to:oracle database 11g enterprise edition release 11.2.0.3.0 - 64bit productionwith the real application clusters and automatic storage management optionssql> select group_number group#, disk_number disk#, os_mb, state, path, header_status from v$asm_disk order by 1,2;group#disk#os_mb statepathheader_status---------- ---------- ---------- ---------- -------------------- ----------------------001024 normal/dev/asm-diskccandidate015120 normal/dev/asm-diskdcandidate0220480 normal/dev/asm-diskbcandidate 
(4) 创建三个磁盘组，systemdg给crs使用，用于存放ocr，votedisk和asm实例的spfile。其余两个给oracle使用，datadg用于存放datafile，controlfile，redolog，spfile；arclogdg存放archivelog。
sql> create diskgroup systemdg external redundancy 2 disk '/dev/asm-diskc' 3 attribute 'compatible.rdbms' = '11.2','compatible.asm' = '11.2';diskgroup created.sql> create diskgroup datadg external redundancy 2 disk '/dev/asm-diskb' 3 attribute 'compatible.rdbms' = '11.2','compatible.asm' = '11.2';diskgroup created.sql> create diskgroup arclogdg external redundancy 2 disk '/dev/asm-diskd' 3 attribute 'compatible.rdbms' = '11.2','compatible.asm' = '11.2';diskgroup created. 
(5) 准备恢复ocr和votedisk，/etc/oracle/ocr.loc中记录了ocr路径，修改ocrconfig_loc的值，以便将ocr恢复到新的磁盘组中。
[root@rac1 ~]# more /etc/oracle/ocr.lococrconfig_loc=+datalocal_only=false[root@rac1 ~]# vi /etc/oracle/ocr.lococrconfig_loc=+systemdglocal_only=false 
(6) 恢复ocr