Skip to contentSkip to conversation

How to debug random bus errors?

DISCUSSION Orion Poplawski 2 participants 3 messages Sep 22, 2006

lore
Share & tools
lore
#0 lore raw

We're seeing programs die with "bus error" (SIGBUS) randomly on a dual processor Opteron machine. I've run memtest86+ and cpuburn stress tests with no failure. gdb on a core file seems uninteresting. Is there some way to trace the kernel to try to get more insight?

Thanks!

Signature · 5 lines
Orion Poplawski
System Administrator                  303-415-9701 x222
NWRA/CoRA Division                    FAX: 303-415-9702
3380 Mitchell Lane                  [email protected]
Boulder, CO 80301              http://www.cora.nwra.com
#1 lore raw
Quoted 5 lines · Orion Poplawski

We're seeing programs die with "bus error" (SIGBUS) randomly on a dual processor Opteron machine. I've run memtest86+ and cpuburn stress tests with no failure. gdb on a core file seems uninteresting. Is there some way to trace the kernel to try to get more insight?

Which kernel?

Opterons in SMP environments had a TLB flushing bug if the tlb flush filter was enabled.

We've had that fixed for a long time, but if you have a kernel older that 2.6.14-rc2 or so (but still new enough to use the four-level page tables:
it's not a big window), you can hit the bug. We fixed it pretty much exactly a year go ("Sat Sep 17, 2005").

Other than that, does it tend to happen to the same particular program? It might just be a user space bug that needs a specific set of things to happen. Some of those bugs go away when you disable address space randomization etc, since they can depend on just pure luck (or rather, lack there-of).

If it happens to lots of different programs, and you sometimes see SIGSEGV's and occasionally perhaps data corruption, that does tend to indicate memory problems (or wild kernel pointers, but on the whole memory problems tend to be more common).

Linus

#2 lore raw
Quoted 9 lines · Linus Torvalds
Quoted 4 lines · Orion Poplawski

We're seeing programs die with "bus error" (SIGBUS) randomly on a dual processor Opteron machine. I've run memtest86+ and cpuburn stress tests with no failure. gdb on a core file seems uninteresting. Is there some way to trace the kernel to try to get more insight?

Which kernel?

Sorry, 2.6.17-1.2142_FC4smp, so fairly recent.

Quoted 1 line

Other than that, does it tend to happen to the same particular program? It might just be a user space bug that needs a specific set of things to happen. Some of those bugs go away when you disable address space randomization etc, since they can depend on just pure luck (or rather, lack there-of).

So far just one program. Forgot about address space randomization, may need to look into turning that off to help debugging.

Thanks!

Signature · 5 lines
Orion Poplawski
System Administrator                  303-415-9701 x222
NWRA/CoRA Division                    FAX: 303-415-9702
3380 Mitchell Lane                  [email protected]
Boulder, CO 80301              http://www.cora.nwra.com