存储故障时的ORA-7445错误
上一篇 / 下一篇 2008-06-02 23:50:21 / 个人分类:Bug
今天由于存储的磁盘故障,导致了一台服务器的DOWN机,没想到Oracle在出现硬件错误的时候也会报ORA-7445错误。
在alert文件中,可以找到下面的错误信息:
Mon Jun 2 16:05:40 2008
[
Njl)h8r5ae0Errors in file /u1/oracle/admin/repdb01/udump/repdb01_ora_8648.trc:
-Ie_lSJy5B8n(u0ORA-07445: exception encountered: core dump [00000001004530E8] [SIGBUS] [Object specific hardware error] [0x103319670] [] []
7HUC,[#l%]%y!b:R0Mon Jun 2 16:05:40 2008
2gB)B Kv4Wf0KCF: write/open error block=0x1557 nline=1
5FW/\]-X0 file=32 /data/oradata/repdb01/undo/UNDOTBS2_1.dbf
#hjSE m-^pj0{0 error=27063 txt: 'SVR4 Error: 5: I/O errorITPUB个人空间TK
g8rD
Additional information: -1ITPUB个人空间Nu7W"MR4n!l3k.m
Additional information: 8192'ITPUB个人空间kz;Rm/O#Cjs
Mon Jun 2 16:05:40 2008
*A1t} O;|0KCF: write/open error block=0x1558 nline=1ITPUB个人空间+S9nB*X+x!{O"a
file=32 /data/oradata/repdb01/undo/UNDOTBS2_1.dbf
#I} x
T_%[t1l0 error=27063 txt: 'SVR4 Error: 5: I/O errorITPUB个人空间:yC)J)O'LL
Additional information: -1ITPUB个人空间-UKm+Y(Z+y QrU e
Additional information: 8192'ITPUB个人空间M.a"d$shQ4?mz
Mon Jun 2 16:05:41 2008ITPUB个人空间jQM,{;Fo!u
W
Errors in file /u1/oracle/admin/repdb01/bdump/repdb01_dbw2_540.trc:
x%q#I+@B%Q;}$x0ORA-07445: exception encountered: core dump [0000000101262540] [SIGBUS] [Object specific hardware error] [0x101262540] [] []
*bZ@(s+axb
D? ik0ORA-27063: Message 27063 not found; product=RDBMS; facility=ORA
?xeBYX0SVR4 Error: 5: I/O errorITPUB个人空间C}'?*y#X1Z?,U
Additional information: -1
cgB6p@)H;\^x0Additional information: 8192ITPUB个人空间L8w-[3~2V9U
Mon Jun 2 16:05:42 2008ITPUB个人空间H o)W:J2O"F9B(@
Errors in file /u1/oracle/admin/repdb01/bdump/repdb01_dbw0_536.trc:
9J,x#xY7I|:j7q
A5V0ORA-07445: exception encountered: core dump [0000000101262540] [SIGBUS] [Object specific hardware error] [0x101262540] [] []ITPUB个人空间(SM)qn-o)A$z+|L
ORA-27063: Message 27063 not found; product=RDBMS; facility=ORA
c+S#s#v,SA1Djgt0SVR4 Error: 5: I/O error
c4B_Cg0Additional information: -1ITPUB个人空间#s1X0q?Gn8i.s
Additional information: 8192ITPUB个人空间 A0V9u:Z~Oe3r
Mon Jun 2 16:05:42 2008
(nX.i4yf0Errors in file /u1/oracle/admin/repdb01/bdump/repdb01_lgwr_544.trc:ITPUB个人空间rOhB.DX
ORA-00345: Message 345 not found; product=RDBMS; facility=ORAITPUB个人空间(pvr4aJ-U5Kd
; arguments: [840190] [2]ITPUB个人空间T8[EaT(Q$c
ORA-00312: Message 312 not found; product=RDBMS; facility=ORA
,J&~wlQr2w3l9]0; arguments: [3] [1] [/data/oradata/repdb01/log/log3.ora]ITPUB个人空间U3XAn4tE [
ORA-27063: Message 27063 not found; product=RDBMS; facility=ORAITPUB个人空间3u]i~c s
SVR4 Error: 5: I/O error
K'ng,Q%~c0Additional information: -1
@4j?SV4k$E0Additional information: 1024
%vPE cw&g!o/z0Mon Jun 2 16:05:51 2008ITPUB个人空间b$G9@u9b
Errors in file /u1/oracle/admin/repdb01/udump/repdb01_ora_8668.trc:ITPUB个人空间Jy4\ YwB"fv,c~
ORA-00449: Message 449 not found; product=RDBMS; facility=ORA
Z\Q)x:W-N9F0; arguments: [LGWR] [340]ITPUB个人空间!U/A @.CD
ORA-00340: Message 340 not found; product=RDBMS; facility=ORA
!z&~#J!E-[#rI_E0Mon Jun 2 16:05:51 2008ITPUB个人空间-l \)F)h*D[&gL!}0xP(M
Errors in file /u1/oracle/admin/repdb01/bdump/repdb01_lgwr_544.trc:ITPUB个人空间#xWHL
yZ6hTNq
ORA-00340: Message 340 not found; product=RDBMS; facility=ORAITPUB个人空间`%C
X&F;}1B
D%]
; arguments: [3] [1]
.iw2o8sF0ORA-00345: Message 345 not found; product=RDBMS; facility=ORA
HFc;n"~0; arguments: [840190] [2]
Q%{ g%s+u\K1\0ORA-00312: Message 312 not found; product=RDBMS; facility=ORAITPUB个人空间,Eoi H~
; arguments: [3] [1] [/data/oradata/repdb01/log/log3.ora]ITPUB个人空间r#iI&U@L
ORA-27063: Message 27063 not found; product=RDBMS; facility=ORAITPUB个人空间M+X0h-c7g0x)B$@@6|
SVR4 Error: 5: I/O errorITPUB个人空间0sm4C.S A4yM
Additional information: -1ITPUB个人空间 e i B
eB;CK \p
[\|
Additional information: 1024ITPUB个人空间:tCz/S&Vr*an[7X
Mon Jun 2 16:05:51 2008ITPUB个人空间:})b|0Z:dRa7z
LGWR: terminating instance due to error 340ITPUB个人空间.S1w?c$Rn\C
Instance terminated by LGWR, pid = 544
从错误信息不难发现,问题出现在IO系统上,由于Oracle无法执行正常的IO操作,导致数据库最终被LGWR进程中止。
先来看看集中错误信息:
ORA-27063 skgfospo: number of bytes read/written is incorrect
Iq)v
dty%b0Cause: The number of bytes read/written as returned by AIOWAIT does not match the original number, additional information indicates both these numbers.ITPUB个人空间+K&j ?0S!B JCz
Action: Check errno.
ORA-00345 redo log write error block string count string
%qAX$? na*Sf\UY0Cause: An I/O error occurred while writing the log.
5}`GVL$sm0Action: Correct the cause of the error, and then restart the system. If the log is lost, apply media/incomplete recovery.
ORA-00340 IO error processing online log string of thread string
R&|+l_V0Cause: An I/O error occurred on the named online log.ITPUB个人空间:R\1N0Dh8[
Action: Restore accessibility to the file or restore the file from backup.
ORA-00312 online log string thread string: 'string'ITPUB个人空间QG dV}&Y0l:^.T
Cause: This message reports the file name for details of another message.
f}jqb#oX(ik S0Action: Other messages will accompany this message. See the associated messages for the appropriate action to take.
ORA-00449 background process 'string' unexpectedly terminated with error string
\;KOyj^'A(^\0Cause: A foreground process needing service from a background process has discovered the process died.
1Q1]-G1^&F[&r0^0Action: Refer to the message code given in the message and the trace file for the foreground and the background processes.
可以看到,这些错误都是和IO错误相关,更具体一点,基本上都是由于LGWR进程无法正确的写入数据造成的。
而上面的7445错误就比较奇怪了,检查一下ORA-7445对应的TRACE文件:
bash-2.03$ more /u1/oracle/admin/repdb01/udump/repdb01_ora_8648.trcITPUB个人空间m2B5LN;oI*B0\V
/u1/oracle/admin/repdb01/udump/repdb01_ora_8648.trcITPUB个人空间)Wl"d
x"C6i!hz6}
Oracle9i Enterprise Edition Release 9.2.0.4.0 - 64bit Production
r"NSV!y g0With the Partitioning, OLAP and Oracle Data Mining optionsITPUB个人空间U&X1]1lw#g{
?BY!{
JServer Release 9.2.0.4.0 - ProductionITPUB个人空间 ls4U9q:?E
ORACLE_HOME = /data/oracle/product/920
4Ju6l2ny+Z/BHw7u0System name: SunOS
6n*\:@T(N0Node name: newreport