How to debug random bus errors?
We're seeing programs die with "bus error" (SIGBUS) randomly on a dual processor Opteron machine. I've run memtest86+ and cpuburn stress tests with no failure. gdb on a core file seems uninteresting. Is there some way to trace the kernel to try to get more insight?
Thanks!
Signature · 5 lines
Orion Poplawski System Administrator 303-415-9701 x222 NWRA/CoRA Division FAX: 303-415-9702 3380 Mitchell Lane [email protected] Boulder, CO 80301 http://www.cora.nwra.com
Quoted 5 lines · Orion Poplawski
We're seeing programs die with "bus error" (SIGBUS) randomly on a dual processor Opteron machine. I've run memtest86+ and cpuburn stress tests with no failure. gdb on a core file seems uninteresting. Is there some way to trace the kernel to try to get more insight?
Which kernel?
Opterons in SMP environments had a TLB flushing bug if the tlb flush filter was enabled.
We've had that fixed for a long time, but if you have a kernel older that 2.6.14-rc2 or so (but still new enough to use the four-level page tables:
it's not a big window), you can hit the bug. We fixed it pretty much exactly a year go ("Sat Sep 17, 2005").
Other than that, does it tend to happen to the same particular program? It might just be a user space bug that needs a specific set of things to happen. Some of those bugs go away when you disable address space randomization etc, since they can depend on just pure luck (or rather, lack there-of).
If it happens to lots of different programs, and you sometimes see SIGSEGV's and occasionally perhaps data corruption, that does tend to indicate memory problems (or wild kernel pointers, but on the whole memory problems tend to be more common).
Linus
Quoted 9 lines · Linus Torvalds
Quoted 4 lines · Orion Poplawski
We're seeing programs die with "bus error" (SIGBUS) randomly on a dual processor Opteron machine. I've run memtest86+ and cpuburn stress tests with no failure. gdb on a core file seems uninteresting. Is there some way to trace the kernel to try to get more insight?
Which kernel?
Sorry, 2.6.17-1.2142_FC4smp, so fairly recent.
Quoted 1 line
Other than that, does it tend to happen to the same particular program? It might just be a user space bug that needs a specific set of things to happen. Some of those bugs go away when you disable address space randomization etc, since they can depend on just pure luck (or rather, lack there-of).
So far just one program. Forgot about address space randomization, may need to look into turning that off to help debugging.
Thanks!
Signature · 5 lines
Orion Poplawski System Administrator 303-415-9701 x222 NWRA/CoRA Division FAX: 303-415-9702 3380 Mitchell Lane [email protected] Boulder, CO 80301 http://www.cora.nwra.com