errors in file /ora10g/admin/dqb/bdump/dqb1_smon_185150.trc: ora-00376: message 376 not found; no message file for product=rdbms, facility=ora; arguments: [113] ora-01110: message 1110 not found; no message file for product=rdbms, facility=
errors in file /ora10g/admin/dqb/bdump/dqb1_smon_185150.trc:
ora-00376: message 376 not found; no message file for product=rdbms, facility=ora; arguments: [113]
ora-01110: message 1110 not found; no message file for product=rdbms, facility=ora; arguments: [113] [/dev/rrlv_20g_data18]
oracle instance dqb1 (pid = 17) - error 376 encountered while recovering transaction (65, 12) on object 2854119.
tue jan 20 16:23:22 beist 2015
errors in file /ora10g/admin/dqb/bdump/dqb1_smon_185150.trc:
ora-00376: message 376 not found; no message file for product=rdbms, facility=ora; arguments: [112]
ora-01110: message 1110 not found; no message file for product=rdbms, facility=ora; arguments: [112] [/dev/rrlv_20g_data16]
oracle instance dqb1 (pid = 17) - error 376 encountered while recovering transaction (87, 27) on object 3036334.
tue jan 20 16:23:23 beist 2015
errors in file /ora10g/admin/dqb/bdump/dqb1_smon_185150.trc:
ora-00376: message 376 not found; no message file for product=rdbms, facility=ora; arguments: [113]
ora-01110: message 1110 not found; no message file for product=rdbms, facility=ora; arguments: [113] [/dev/rrlv_20g_data18]
oracle instance dqb1 (pid = 17) - error 376 encountered while recovering transaction (65, 12) on object 2854119.
tue jan 20 16:23:24 beist 2015
errors in file /ora10g/admin/dqb/bdump/dqb1_smon_185150.trc:
ora-00376: message 376 not found; no message file for product=rdbms, facility=ora; arguments: [112]
ora-01110: message 1110 not found; no message file for product=rdbms, facility=ora; arguments: [112] [/dev/rrlv_20g_data16]
oracle instance dqb1 (pid = 17) - error 376 encountered while recovering transaction (87, 27) on object 3036334.
tue jan 20 16:23:25 beist 2015
errors in file /ora10g/admin/dqb/bdump/dqb1_smon_185150.trc:
ora-00376: message 376 not found; no message file for product=rdbms, facility=ora; arguments: [113]
ora-01110: message 1110 not found; no message file for product=rdbms, facility=ora; arguments: [113] [/dev/rrlv_20g_data18]
oracle instance dqb1 (pid = 17) - error 376 encountered while recovering transaction (65, 12) on object 2854119.
tue jan 20 16:23:26 beist 2015
errors in file /ora10g/admin/dqb/bdump/dqb1_smon_185150.trc:
ora-00376: message 376 not found; no message file for product=rdbms, facility=ora; arguments: [112]
ora-01110: message 1110 not found; no message file for product=rdbms, facility=ora; arguments: [112] [/dev/rrlv_20g_data16]
tue jan 20 16:23:27 beist 2015
oracle instance dqb1 (pid = 17) - error 376 encountered while recovering transaction (87, 27) on object 3036334.
tue jan 20 16:23:27 beist 2015
errors in file /ora10g/admin/dqb/bdump/dqb1_smon_185150.trc:
ora-00376: message 376 not found; no message file for product=rdbms, facility=ora; arguments: [113]
ora-01110: message 1110 not found; no message file for product=rdbms, facility=ora; arguments: [113] [/dev/rrlv_20g_data18]
oracle instance dqb1 (pid = 17) - error 376 encountered while recovering transaction (65, 12) on object 2854119.
tue jan 20 16:23:28 beist 2015
errors in file /ora10g/admin/dqb/bdump/dqb1_smon_185150.trc:
ora-00376: message 376 not found; no message file for product=rdbms, facility=ora; arguments: [112]
ora-01110: message 1110 not found; no message file for product=rdbms, facility=ora; arguments: [112] [/dev/rrlv_20g_data16]
oracle instance dqb1 (pid = 17) - error 376 encountered while recovering transaction (87, 27) on object 3036334.
tue jan 20 16:23:30 beist 2015
errors in file /ora10g/admin/dqb/bdump/dqb1_smon_185150.trc:
ora-00376: message 376 not found; no message file for product=rdbms, facility=ora; arguments: [113]
ora-01110: message 1110 not found; no message file for product=rdbms, facility=ora; arguments: [113] [/dev/rrlv_20g_data18]
oracle instance dqb1 (pid = 17) - error 376 encountered while recovering transaction (65, 12) on object 2854119.
tue jan 20 16:23:31 beist 2015
errors in file /ora10g/admin/dqb/bdump/dqb1_smon_185150.trc:
ora-00376: message 376 not found; no message file for product=rdbms, facility=ora; arguments: [112]
ora-01110: message 1110 not found; no message file for product=rdbms, facility=ora; arguments: [112] [/dev/rrlv_20g_data16]
oracle instance dqb1 (pid = 17) - error 376 encountered while recovering transaction (87, 27) on object 3036334.
上述错误信息一直在后台报错，查询部分表报错，查询数据库的数据文件状态很多是offline状态。
sql> select name ,status from v$datafile where status'online';
name
--------------------------------------------------------------------------------
status
-------
/dev/rrac_system_1g
system
/dev/rrlv_20g_data01
offline
/dev/rrlv_20g_data03
offline
name
--------------------------------------------------------------------------------
status
-------
/dev/rrlv_20g_data04
offline
/dev/rrlv_20g_data05
offline
/dev/rrlv_20g_data06
offline
name
--------------------------------------------------------------------------------
status
-------
/dev/rrlv_20g_data07
offline
/dev/rrlv_20g_data09
offline
/dev/rrlv_20g_data10
offline
name
--------------------------------------------------------------------------------
status
-------
/dev/rrlv_20g_data11
offline
/dev/rrlv_20g_data12
offline
/dev/rrlv_20g_data13
offline
name
--------------------------------------------------------------------------------
status
-------
/dev/rrlv_20g_data14
offline
/dev/rrac_20g20
system
/dev/rrac_20g21
system
name
--------------------------------------------------------------------------------
status
-------
/dev/rrlv_20g_data15
offline
/dev/rrlv_20g_data17
offline
/dev/rrlv_20g_data16
offline
name
--------------------------------------------------------------------------------
status
-------
/dev/rrlv_20g_data18
offline
19 rows selected.
仔细查看数据库日志发现，在1月12日做了表空间增加数据文件操作。
查看两个节点的数据文件权限
两个节点的裸设备都是这个样子
brw-rw----    1 root     system       83,  1 feb 20 2014  /dev/rlv_20g_data01
brw-rw----    1 root     system       83,  2 feb 20 2014  /dev/rlv_20g_data02
brw-rw----    1 root     system       83,  3 feb 20 2014  /dev/rlv_20g_data03
brw-rw----    1 root     system       83,  4 feb 20 2014  /dev/rlv_20g_data04
brw-rw----    1 root     system       83,  5 feb 20 2014  /dev/rlv_20g_data05
brw-rw----    1 root     system       83,  6 feb 20 2014  /dev/rlv_20g_data06
brw-rw----    1 root     system       83,  7 feb 20 2014  /dev/rlv_20g_data07
brw-rw----    1 root     system       83,  8 feb 20 2014  /dev/rlv_20g_data08
brw-rw----    1 root     system       83,  9 feb 20 2014  /dev/rlv_20g_data09
brw-rw----    1 root     system       83, 10 feb 20 2014  /dev/rlv_20g_data10
brw-rw----    1 root     system       83, 11 feb 20 2014  /dev/rlv_20g_data11
brw-rw----    1 root     system       83, 12 feb 20 2014  /dev/rlv_20g_data12
brw-rw----    1 root     system       83, 13 feb 20 2014  /dev/rlv_20g_data13
brw-rw----    1 root     system       83, 14 feb 20 2014  /dev/rlv_20g_data14
brw-rw----    1 root     system       83, 15 feb 20 2014  /dev/rlv_20g_data15
brw-rw----    1 root     system       83, 16 feb 20 2014  /dev/rlv_20g_data16
brw-rw----    1 root     system       83, 17 feb 20 2014  /dev/rlv_20g_data17
brw-rw----    1 root     system       83, 18 feb 20 2014  /dev/rlv_20g_data18
brw-rw----    1 root     system       83, 19 feb 20 2014  /dev/rlv_20g_data19
brw-rw----    1 root     system       83, 20 feb 20 2014  /dev/rlv_20g_data20
正确是以下：
crw-rw----    1 ora10g   dba          83,  1 jan 20 08:19 /dev/rrlv_20g_data01
crw-rw----    1 ora10g   dba          83,  2 jan 20 20:28 /dev/rrlv_20g_data02
crw-rw----    1 ora10g   dba          83,  3 jan 20 08:19 /dev/rrlv_20g_data03
crw-rw----    1 ora10g   dba          83,  4 jan 20 08:19 /dev/rrlv_20g_data04
crw-rw----    1 ora10g   dba          83,  5 jan 20 08:19 /dev/rrlv_20g_data05
crw-rw----    1 ora10g   dba          83,  6 jan 20 08:19 /dev/rrlv_20g_data06
crw-rw----    1 ora10g   dba          83,  7 jan 20 08:19 /dev/rrlv_20g_data07
crw-rw----    1 ora10g   dba          83,  8 jan 20 20:28 /dev/rrlv_20g_data08
crw-rw----    1 ora10g   dba          83,  9 jan 20 08:19 /dev/rrlv_20g_data09
crw-rw----    1 ora10g   dba          83, 10 jan 20 08:19 /dev/rrlv_20g_data10
crw-rw----    1 ora10g   dba          83, 11 jan 20 08:19 /dev/rrlv_20g_data11
crw-rw----    1 ora10g   dba          83, 12 jan 20 08:19 /dev/rrlv_20g_data12
crw-rw----    1 ora10g   dba          83, 13 jan 20 08:19 /dev/rrlv_20g_data13
crw-rw----    1 ora10g   dba          83, 14 jan 20 08:19 /dev/rrlv_20g_data14
crw-rw----    1 ora10g   dba          83, 15 jan 20 08:19 /dev/rrlv_20g_data15
crw-rw----    1 ora10g   dba          83, 16 jan 20 08:19 /dev/rrlv_20g_data16
crw-rw----    1 ora10g   dba          83, 17 jan 20 08:19 /dev/rrlv_20g_data17
crw-rw----    1 ora10g   dba          83, 18 jan 20 08:19 /dev/rrlv_20g_data18
crw-rw----    1 ora10g   dba          83, 19 jan 20 20:28 /dev/rrlv_20g_data19
crw-rw----    1 ora10g   dba          83, 20 jan 20 20:28 /dev/rrlv_20g_data20
这样子问题就清楚了，执行权限修改，后执行数据文件恢复
1、chown -r oracle;dba /dev/rrlv_20g_data*
2、停掉两个节点的数据库，启动第一个节点到mount，执行recover操作
[begin] 2015-1-21 10:43:09
ora-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92773_727466139.dbf
ora-00280: change 14242883477423 for thread 2 is in sequence #92773
ora-00278: log file '+arch/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
log applied.
media recovery complete.
sql> recover    datafile       14;
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92772_727466139.dbf
ora-00280: change 14242805827356 for thread 2 is in sequence #92772
specify log: {=suggested | filename | auto | cancel}
auto
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ora-00289: suggestion : +arch/archive/arch1/arch1_30832_727466139.dbf
ora-00280: change 14242805827356 for thread 1 is in sequence #30832
ora-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92773_727466139.dbf
ora-00280: change 14242883477423 for thread 2 is in sequence #92773
ora-00278: log file '+arch/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
log applied.
media recovery complete.
sql> recover   datafile      17;
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92772_727466139.dbf
ora-00280: change 14242805827356 for thread 2 is in sequence #92772
specify log: {=suggested | filename | auto | cancel}
auto
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ora-00289: suggestion : +arch/archive/arch1/arch1_30832_727466139.dbf
ora-00280: change 14242805827356 for thread 1 is in sequence #30832
ora-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92773_727466139.dbf
ora-00280: change 14242883477423 for thread 2 is in sequence #92773
ora-00278: log file '+arch/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
log applied.
media recovery complete.
sql> recover    datafile     18;
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92772_727466139.dbf
ora-00280: change 14242805827356 for thread 2 is in sequence #92772
specify log: {=suggested | filename | auto | cancel}
auto
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ora-00289: suggestion : +arch/archive/arch1/arch1_30832_727466139.dbf
ora-00280: change 14242805827356 for thread 1 is in sequence #30832
ora-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92773_727466139.dbf
ora-00280: change 14242883477423 for thread 2 is in sequence #92773
ora-00278: log file '+arch/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
log applied.
media recovery complete.
sql> recover     datafile      19;
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92772_727466139.dbf
ora-00280: change 14242805827356 for thread 2 is in sequence #92772
specify log: {=suggested | filename | auto | cancel}
auto
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ora-00289: suggestion : +arch/archive/arch1/arch1_30832_727466139.dbf
ora-00280: change 14242805827356 for thread 1 is in sequence #30832
ora-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92773_727466139.dbf
ora-00280: change 14242883477423 for thread 2 is in sequence #92773
ora-00278: log file '+arch/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
log applied.
media recovery complete.
sql> recover    datafile    21;
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92772_727466139.dbf
ora-00280: change 14242805827356 for thread 2 is in sequence #92772
specify log: {=suggested | filename | auto | cancel}
auto
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ora-00289: suggestion : +arch/archive/arch1/arch1_30832_727466139.dbf
ora-00280: change 14242805827356 for thread 1 is in sequence #30832
ora-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92773_727466139.dbf
ora-00280: change 14242883477423 for thread 2 is in sequence #92773
ora-00278: log file '+arch/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
log applied.
media recovery complete.
sql> recover    datafile    22;
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92772_727466139.dbf
ora-00280: change 14242805827356 for thread 2 is in sequence #92772
specify log: {=suggested | filename | auto | cancel}
auto
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ora-00289: suggestion : +arch/archive/arch1/arch1_30832_727466139.dbf
ora-00280: change 14242805827356 for thread 1 is in sequence #30832
ora-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92773_727466139.dbf
ora-00280: change 14242883477423 for thread 2 is in sequence #92773
ora-00278: log file '+arch/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
log applied.
media recovery complete.
sql> recover    datafile    23;
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92772_727466139.dbf
ora-00280: change 14242805827356 for thread 2 is in sequence #92772
specify log: {=suggested | filename | auto | cancel}
auto
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ora-00289: suggestion : +arch/archive/arch1/arch1_30832_727466139.dbf
ora-00280: change 14242805827356 for thread 1 is in sequence #30832
ora-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92773_727466139.dbf
ora-00280: change 14242883477423 for thread 2 is in sequence #92773
ora-00278: log file '+arch/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
log applied.
media recovery complete.
sql> recover    datafile    26;
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92772_727466139.dbf
ora-00280: change 14242805827356 for thread 2 is in sequence #92772
specify log: {=suggested | filename | auto | cancel}
auto
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ora-00289: suggestion : +arch/archive/arch1/arch1_30832_727466139.dbf
ora-00280: change 14242805827356 for thread 1 is in sequence #30832
ora-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92773_727466139.dbf
ora-00280: change 14242883477423 for thread 2 is in sequence #92773
ora-00278: log file '+arch/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
log applied.
media recovery complete.
sql> recover    datafile    29;
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92772_727466139.dbf
ora-00280: change 14242805827356 for thread 2 is in sequence #92772
specify log: {=suggested | filename | auto | cancel}
auto
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ora-00289: suggestion : +arch/archive/arch1/arch1_30832_727466139.dbf
ora-00280: change 14242805827356 for thread 1 is in sequence #30832
ora-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92773_727466139.dbf
ora-00280: change 14242883477423 for thread 2 is in sequence #92773
ora-00278: log file '+arch/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
log applied.
media recovery complete.
sql> recover    datafile    55;
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92772_727466139.dbf
ora-00280: change 14242805827356 for thread 2 is in sequence #92772
specify log: {=suggested | filename | auto | cancel}
auto
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ora-00289: suggestion : +arch/archive/arch1/arch1_30832_727466139.dbf
ora-00280: change 14242805827356 for thread 1 is in sequence #30832
ora-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92773_727466139.dbf
ora-00280: change 14242883477423 for thread 2 is in sequence #92773
ora-00278: log file '+arch/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
log applied.
media recovery complete.
sql> recover    datafile    103;
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92772_727466139.dbf
ora-00280: change 14242805827356 for thread 2 is in sequence #92772
specify log: {=suggested | filename | auto | cancel}
auto
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ora-00289: suggestion : +arch/archive/arch1/arch1_30832_727466139.dbf
ora-00280: change 14242805827356 for thread 1 is in sequence #30832
ora-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92773_727466139.dbf
ora-00280: change 14242883477423 for thread 2 is in sequence #92773
ora-00278: log file '+arch/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
log applied.
media recovery complete.
sql> recover    datafile    104;
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92772_727466139.dbf
ora-00280: change 14242805827356 for thread 2 is in sequence #92772
specify log: {=suggested | filename | auto | cancel}
auto
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ora-00289: suggestion : +arch/archive/arch1/arch1_30832_727466139.dbf
ora-00280: change 14242805827356 for thread 1 is in sequence #30832
ora-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92773_727466139.dbf
ora-00280: change 14242883477423 for thread 2 is in sequence #92773
ora-00278: log file '+arch/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
log applied.
media recovery complete.
sql> recover    datafile    112;
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92772_727466139.dbf
ora-00280: change 14242805827356 for thread 2 is in sequence #92772
specify log: {=suggested | filename | auto | cancel}
auto
ora-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ora-00289: suggestion : +arch/archive/arch1/arch1_30832_727466139.dbf
ora-00280: change 14242805827356 for thread 1 is in sequence #30832
ora-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ora-00289: suggestion : +arch/archive/arch1/arch2_92773_727466139.dbf
ora-00280: change 14242883477423 for thread 2 is in sequence #92773
ora-00278: log file '+arch/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
log applied.
media recovery complete.
sql> alter database open;
最后成功执行。