Skip to contentSkip to conversation

How to go about debuging a system lockup?

DISCUSSION Lennart Sorensen 4 participants 6 messages Nov 16, 2006 – Nov 21, 2006

lore
Share & tools
lore
#0 lore raw

We have a router with a Geode SC1200 cpu, with 4 AMD 972 ethernet ports (pcnet32) behind a PLX 6152 PCI-PCI bridge, which quite regularly locks up completely if we try to do simultanius traffic on all 4 ports (our test case sends data from port 1 to port 2, and back and from port 3 to port 4 and back at a rate of 8000 packets per second using 1500byte packets). We usually manage to run the test for about 1 minute before the system hangs. This happens on every one of the systems we have tried so far. If we only run 2 ports, it seems to never die, and with 3 ports we haven't seen any failures yet, although maybe we just haven't tested long enough. If we just receive the packets but don't forward them out again, then we never crash, so it seems to be related to simultanious transmit on the pcnet32s.

So far I have tried printing a message everytime the pcnet32 driver enables and disables interrupts to find out if it hangs somewhere with interrupts disabled, but that didn't seem to indicate anything meaningful.

So far I have tried this with 2.6.8, 2.6.16.22, and 2.6.18.2 and no difference so far. I can't think of what kind of even could cause the system to just hang with no further console output or a kernel panic or oops or anything. Usually most errors produce some kind of message.

Does anyone have any suggestions for where I go from here to find out what is happening and where to look? I don't even know if I should suspect the hardware or the software at this point. I want to know if the program counter is still changing, or if the cpu is simply hung or something, but I have no idea how to get at that.

--
Len Sorensen

#1 lore raw
Quoted 23 lines · Lennart Sorensen

We have a router with a Geode SC1200 cpu, with 4 AMD 972 ethernet ports (pcnet32) behind a PLX 6152 PCI-PCI bridge, which quite regularly locks up completely if we try to do simultanius traffic on all 4 ports (our test case sends data from port 1 to port 2, and back and from port 3 to port 4 and back at a rate of 8000 packets per second using 1500byte packets). We usually manage to run the test for about 1 minute before the system hangs. This happens on every one of the systems we have tried so far. If we only run 2 ports, it seems to never die, and with 3 ports we haven't seen any failures yet, although maybe we just haven't tested long enough. If we just receive the packets but don't forward them out again, then we never crash, so it seems to be related to simultanious transmit on the pcnet32s.

So far I have tried printing a message everytime the pcnet32 driver enables and disables interrupts to find out if it hangs somewhere with interrupts disabled, but that didn't seem to indicate anything meaningful.

So far I have tried this with 2.6.8, 2.6.16.22, and 2.6.18.2 and no difference so far. I can't think of what kind of even could cause the system to just hang with no further console output or a kernel panic or oops or anything. Usually most errors produce some kind of message.

Quoted 6 lines

Does anyone have any suggestions for where I go from here to find out what is happening and where to look? I don't even know if I should suspect the hardware or the software at this point. I want to know if the program counter is still changing, or if the cpu is simply hung or something, but I have no idea how to get at that.

Well, I have a few ideas that are hopefully useul.

- If you have not done so already, then go in to the "Kernel Hacking" section of the kernel configuration and enable some (all?) of the debug options and see if that produces anything that will help you track down the problem.

- You could enable 'magic sysrq' and see if you can manage to get a backtrace with it when it hangs (see Documentation/sysrq.txt) (ohh and raise the console log level so you get all messages, including debug ones).

- You could also try kdb (http://oss.sgi.com/projects/kdb/) or kgdb
(http://kgdb.linsyssoft.com/). That might help you pinpoint the failure.
See also : http://kerneltrap.org/node/112

- If you have (or can identify) an older, working, kernel version and you are confident that you can reproduce the problem reliably, then doing a git bisection search starting with your newest "known good" and oldest "known bad" kernel versions, should help you pinpoint the commit causing the breakage.

Hope some of that helps :)

Signature · 3 lines
Jesper Juhl <[email protected]>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html
#2 lore raw
Quoted 6 lines · Jesper Juhl

Well, I have a few ideas that are hopefully useul.

- If you have not done so already, then go in to the "Kernel Hacking" section of the kernel configuration and enable some (all?) of the debug options and see if that produces anything that will help you track down the problem.

I enabled the things that sounded useful. I will try enabling the rest.

Quoted 4 lines

- You could enable 'magic sysrq' and see if you can manage to get a backtrace with it when it hangs (see Documentation/sysrq.txt) (ohh and raise the console log level so you get all messages, including debug ones).

Yeah I did that. No response to sysrq (at least not on the serial console. Maybe I should get a keyboard connector put on.) Normally we run without VGA/keyboard/etc, and just serial console. Of course the serial console requires working interrupts. Not sure about the keyboard driver.

Quoted 3 lines

- You could also try kdb (http://oss.sgi.com/projects/kdb/) or kgdb
(http://kgdb.linsyssoft.com/). That might help you pinpoint the failure.

Can I run that remotely somehow? I never really looked at kdb or kgdb before.

Quoted 7 lines

See also : http://kerneltrap.org/node/112

- If you have (or can identify) an older, working, kernel version and you are confident that you can reproduce the problem reliably, then doing a git bisection search starting with your newest "known good" and oldest "known bad" kernel versions, should help you pinpoint the commit causing the breakage.

I don't know of a good version yet. I so far don't know if there ever was one. This could even be a bug in the PCI hardware, or the way the BIOS on this system on a board configured the PCI controller. Maybe I should go back and try a 2.4 kernel.

Quoted 1 line

Hope some of that helps :)

Well hopefully.

--
Len Sorensen

#3 lore raw
Quoted 1 line · Lennart Sorensen

On Thu, Nov 16, 2006 at 09:49:06PM +0100, Jesper Juhl wrote:

...

Quoted 7 lines
Quoted 3 lines

- You could also try kdb (http://oss.sgi.com/projects/kdb/) or kgdb
(http://kgdb.linsyssoft.com/). That might help you pinpoint the failure.

Can I run that remotely somehow? I never really looked at kdb or kgdb before.

Yes you can. kgdb is run on a separate machine from the one you are debugging.

Quoted 11 lines
Quoted 5 lines

- If you have (or can identify) an older, working, kernel version and you are confident that you can reproduce the problem reliably, then doing a git bisection search starting with your newest "known good" and oldest "known bad" kernel versions, should help you pinpoint the commit causing the breakage.

I don't know of a good version yet. I so far don't know if there ever was one. This could even be a bug in the PCI hardware, or the way the BIOS on this system on a board configured the PCI controller. Maybe I should go back and try a 2.4 kernel.

Or just try a few random older 2.6 kernels like 2.6.14, 2.6.9, 2.6.whatever (of course it needs to be a version that git knows about).

Signature · 3 lines
Jesper Juhl <[email protected]>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html
#4 lore raw
Quoted 3 lines · Jesper Juhl

Or just try a few random older 2.6 kernels like 2.6.14, 2.6.9, 2.6.whatever (of course it needs to be a version that git knows about).

One can also do "bisect" manually, works with all kernels.

Signature · 1 line
Krzysztof Halasa
#5 lore raw
In reply to Lennart Sorensen
Quoted 21 lines · Lennart Sorensen (on Thu, 16 Nov 2006 16:21:40 -0500) wrote:
Quoted 6 lines · Jesper Juhl

Well, I have a few ideas that are hopefully useul.

- If you have not done so already, then go in to the "Kernel Hacking" section of the kernel configuration and enable some (all?) of the debug options and see if that produces anything that will help you track down the problem.

I enabled the things that sounded useful. I will try enabling the rest.

Quoted 4 lines

- You could enable 'magic sysrq' and see if you can manage to get a backtrace with it when it hangs (see Documentation/sysrq.txt) (ohh and raise the console log level so you get all messages, including debug ones).

Yeah I did that. No response to sysrq (at least not on the serial console. Maybe I should get a keyboard connector put on.) Normally we run without VGA/keyboard/etc, and just serial console. Of course the serial console requires working interrupts. Not sure about the keyboard driver.

Quoted 6 lines
Quoted 3 lines

- You could also try kdb (http://oss.sgi.com/projects/kdb/) or kgdb
(http://kgdb.linsyssoft.com/). That might help you pinpoint the failure.

Can I run that remotely somehow? I never really looked at kdb or kgdb before.

kgdb can only be run remotely. kdb can be run on the local keyboard/console or over a serial console.