ALERT: Database Hang Likely on Pre 2.4.9-e.57 Redhat Linux Kernels

Applies to:

Oracle Server - Enterprise Edition - Version: 9.2.0.5 to 9.2.0.99
Linux x86
Versions Affected
~~~~~~~~~~~~~~~~~
This issue is known to affect Oracle Database 9.2.0.5 with the one-off patch for Bug 3865774 as well as 9.2.0.6 and greater.

Platforms Affected
~~~~~~~~~~~~~~~~~~
If you are on RedHat AS 2.1 with 2.4.9 and on an errata lower than e57 it is likely you will hit this problem.

Description

The symptoms of this problem vary depending on the process that experiences the problem. The full description of the kernel defect can be found via bugzilla entry:

130926 read() sometimes crashes in the second pthread_set_canceltype call

Likelihood of Occurrence

If you are on RedHat AS 2.1 with 2.4.9 and on an errata lower than e57 it is likely you will hit this problem.

Possible Symptoms

Possible symptoms include, but are not restricted to:

• Background and foreground processes in a slow spin. It is not uncommon
  for a process hitting this issue to utilize 25% or more of one CPU.
• Database hangs due to the resources held by the spinning process.
• Empty call stacks generated by oradebug commands.
• Oradebug commands hang
• pstack outputs empty stacks or core dumps
• Attaching a debugger to the process will show a recursive stacks with no symbols. It is possible to see "killpg" at the bottom
  of the stacks if it is caught early enough:

(gdb) bt
#0 0x4046cb15 in ?? ()
#1 <signal handler called>
#2 0x4046cb15 in ?? ()
#3 <signal handler called>
...
0x40473b15: pthread_sighandler_rt + 0x15 (b, 40657c30, 40657cb0, b, 0, 80) + 368
0x404c3950: killpg + 0x4c (b, 40657fb8, 40658038, b, 0, 80) + 368
0x404c3950: killpg + 0x4c (b, 40658340, 406583c0, b, 0, 80) + 368
0x404c3950: killpg + 0x4c (b, 406586c8, 40658748, b, 0, 80) + 368
0x404c3950: killpg + 0x4c (b, 40658a50, 40658ad0, b, 0, 80) + 368

• In a RAC configuration a clusterwide database hang is common.

Workaround

There is no workaround. The new kernel is mandatory.

Patches

This a RedHat Kernel issue and is fixed by applying 2.4.9-e.57 or newer.

Modification History

References

Bug 3865774 - Ora-600 [4000] And Other Memory Corruptions Cause 1 Rac Instance To Crash
Note 295403.1 - Spinning job queue processes after upgrade to 9.2.0.6 on Redhat AS 2.1