x86-64 sporadic hang in 2.6.23rc7 and 2.6.22
The two kernels mentioned hangs occationally. Typically when I compile something and pass the time by surfing the web.
A few minutes and then I notice that the mouse (and everything else in X) stops. kbd LEDs does not react to numlock/capslock.
The only thing that still works is sysrq+B
So far this has happened while running X, so no messages.
I have gone back to 2.6.22rc4, which seems to work.
This is a single opteron, although on a dual-slot board.
Helge Hafting
Quoted 4 lines · Helge Hafting
The two kernels mentioned hangs occationally. Typically when I compile something and pass the time by surfing the web.
Quoted 8 lines
A few minutes and then I notice that the mouse (and everything else in X) stops. kbd LEDs does not react to numlock/capslock.
The only thing that still works is sysrq+B
So far this has happened while running X, so no messages.I have gone back to 2.6.22rc4, which seems to work.
This is a single opteron, although on a dual-slot board.
Can you switch to serial console, so we can get some information out of that box? Sysrq-B is working, so we can get info from other sysrq functions as well.
tglx
Quoted 11 lines · Thomas Gleixner
Quoted 9 lines · Helge Hafting
The two kernels mentioned hangs occationally. Typically when I compile something and pass the time by surfing the web.
A few minutes and then I notice that the mouse (and everything else in X) stops. kbd LEDs does not react to numlock/capslock.
The only thing that still works is sysrq+B
So far this has happened while running X, so no messages.
Quoted 9 lines
Quoted 4 lines
I have gone back to 2.6.22rc4, which seems to work.
This is a single opteron, although on a dual-slot board.
Can you switch to serial console, so we can get some information out of that box? Sysrq-B is working, so we can get info from other sysrq functions as well.
I didn't need the serial - it crashes during console work too.
I think a "make clean" was in progress at the time. There must be work going on in order to crash.
This time 2.6.22rc4 died on me with a general protection fault
I got two reports, the first one scrolled partially off screen but the whole trace was there:
shrink_dcache_memoryshrink_slab
kswapdautoremove_wake_functionthread_returntrace_hardirqs_on
kswapd
kswapd
kthtreadchild_riprestore_args
kthreadchild_rip
Then I got:
spinlock lockup on cpu #0, kswapd 0/212 _raw_spin_lockshrink_dcache_parentshrink_dcache_parentproc_flush_taskrelease_taskdo_exit
dieerror_exitprune_dcache
[From here on, it continues exactly like the first report:] shrink_dcache_memoryshrink_slab
kswapdautoremove_wake_functionthread_returntrace_hardirqs_on
kswapd
kswapd
kthtreadchild_riprestore_args
kthreadchild_rip
sysrq P says:
cpu 0
pid 212 comm: kswapd0 not tainted 2.6.22-rc4 #18
RIP: __delay
I took a picture of the screen, in case the register dumps are interesting. Wonder what this is - dcache trouble? swap trouble?
Helge Hafting
Quoted 10 lines · Helge Hafting
Quoted 9 lines · Thomas Gleixner
Quoted 4 lines
I have gone back to 2.6.22rc4, which seems to work.
This is a single opteron, although on a dual-slot board.
Can you switch to serial console, so we can get some information out of that box? Sysrq-B is working, so we can get info from other sysrq functions as well.
Quoted 8 lines
I didn't need the serial - it crashes during console work too.
I think a "make clean" was in progress at the time. There must be work going on in order to crash.This time 2.6.22rc4 died on me with a general protection fault
I got two reports, the first one scrolled partially off screen but the whole trace was there:
That's why I asked for a serial console. That way we can get all the information from the reports including the register dumps ....
Quoted 2 lines
Then I got:
spinlock lockup on cpu #0, kswapd 0/212
That's probably caused by the previous one.
tglx
Quoted 18 lines · Thomas Gleixner
Quoted 16 lines · Helge Hafting
Quoted 10 lines · Thomas Gleixner
Quoted 5 lines
I have gone back to 2.6.22rc4, which seems to work.
This is a single opteron, although on a dual-slot board.
Can you switch to serial console, so we can get some information out of that box? Sysrq-B is working, so we can get info from other sysrq functions as well.
I didn't need the serial - it crashes during console work too.
I think a "make clean" was in progress at the time. There must be work going on in order to crash.
Quoted 9 lines
Quoted 5 lines
This time 2.6.22rc4 died on me with a general protection fault
I got two reports, the first one scrolled partially off screen but the whole trace was there:
That's why I asked for a serial console. That way we can get all the information from the reports including the register dumps ...
Sure. But I can't get a cable right now. Was the registers necessary in this case? Often, the trace turns out to be enough.
Helge Hafting
Quoted 18 lines · Thomas Gleixner
Quoted 16 lines · Helge Hafting
Quoted 10 lines · Thomas Gleixner
Quoted 5 lines
I have gone back to 2.6.22rc4, which seems to work.
This is a single opteron, although on a dual-slot board.
Can you switch to serial console, so we can get some information out of that box? Sysrq-B is working, so we can get info from other sysrq functions as well.
I didn't need the serial - it crashes during console work too.
I think a "make clean" was in progress at the time. There must be work going on in order to crash.
Quoted 9 lines
Quoted 5 lines
This time 2.6.22rc4 died on me with a general protection fault
I got two reports, the first one scrolled partially off screen but the whole trace was there:
That's why I asked for a serial console. That way we can get all the information from the reports including the register dumps ....
I got another crash - with a full dump. I have also discovered files with lots of single-bit errors, so this is probably just some kind of hw problem. :-(
Replace mermory or the motherboard with everything on it . . . :-(
Helge Hafting
Quoted 2 lines · Helge Hafting
shrink_dcache_memory
That usually means random memory corruption from somewhere -- dcache tends to use a lot of memory and when it is corrupted anywhere these functions tend to crash while walking the lists.
Unfortunately memory corruption is hard to track down because the messenger is usually not the one to blame.
Perhaps enable slab debugging and see if it turns something up. Could be also broken hardware. Does an older kernel run stable? If yes and if it can be reproduced bisecting would be good.
-Andi
Quoted 8 lines · Andi Kleen
Quoted 2 lines · Helge Hafting
shrink_dcache_memoryThat usually means random memory corruption from somewhere -- dcache
tends to use a lot of memory and when it is corrupted anywhere these functions tend to crash while walking the lists.
Quoted 8 lines
Unfortunately memory corruption is hard to track down because the messenger is usually not the one to blame.
Perhaps enable slab debugging and see if it turns something up. Could be also broken hardware. Does an older kernel run stable? If yes and if it can be reproduced bisecting would be good.
2.6.18 had no problem compiling stuff without crashing. Looks like I have some work to do then.
Helge Hafting
Quoted 8 lines · Andi Kleen
Quoted 2 lines · Helge Hafting
shrink_dcache_memoryThat usually means random memory corruption from somewhere -- dcache
tends to use a lot of memory and when it is corrupted anywhere these functions tend to crash while walking the lists.
Quoted 8 lines
Unfortunately memory corruption is hard to track down because the messenger is usually not the one to blame.
Perhaps enable slab debugging and see if it turns something up. Could be also broken hardware. Does an older kernel run stable? If yes and if it can be reproduced bisecting would be good.
I attempted bisecting - and failed. The problem is that 2.6.23rc7 seems very unstable, but 2.6.22rc4 is much better but not perfect. 2.6.22rc4 only crashed once - it can compile for hours and swap lots and keep running. But it died at least once.
I'll try running recent kernels with more debugging instead. I think I used SLUB instead of SLAB - either way I can switch that over to see if it changes things.
Helge Hafting