Hard lockup with 2.6.24.7-rt26 on x86
Dear Linux rt users
On two different desktop PC (both Core 2 Duo) and on a notebook (Core Duo) the kernel (2.6.24.7-rt26, CONFIG_X86_32) locks up hard under a certain rt load. The failure can be reproduced reliably with the following command:
sudo ./cyclictest -p99 -t10 -n -i250
During the test I made some additional observations:
- The lockup is really hard: there is no output on a netconsole and even the magic sysrequest keys to reboot the system (Alt+SysRQ+b) do not work anymore.
- with a lower realtime priority the system seems to run at least more stable (e.g. sudo ./cyclictest -p95 -t10 -n -i250)
- with the vanilla 2.6.26.6-rt11 kernel I observed probably the same crash
- the problem also appears on Ubuntu Hardy Heron with the latest official kernel (2.6.24.23.25 with rt21 patch)
- the problem did not appear on Ubuntu Hardy Heron with older rt-kernel versions (also 2.6.24 but based on rt3 patch)
So at the moment I am really clueless and I would be happy for any hint on how I can further debug this crash.
Best regards
Matthias
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 13 lines · On Mon, 2 Feb 2009 21:55:36 +0100 [email protected] wrote:
Dear Linux rt users
On two different desktop PC (both Core 2 Duo) and on a notebook (Core Duo) the kernel (2.6.24.7-rt26,
CONFIG_X86_32) locks up hard under a certain rt load. The failure can be reproduced reliably with the following command:sudo ./cyclictest -p99 -t10 -n -i250
During the test I made some additional observations:
- The lockup is really hard: there is no output on a netconsole and even the magic sysrequest keys to reboot the system (Alt+SysRQ+b) do not work anymore.
- with a lower realtime priority the system seems to run at least more stable (e.g. sudo ./cyclictest -p95 -t10 -n -i250)
- with the vanilla 2.6.26.6-rt11 kernel I observed probably the same crash
- the problem also appears on Ubuntu Hardy Heron with the latest official kernel (2.6.24.23.25 with rt21 patch)
- the problem did not appear on Ubuntu Hardy Heron with older rt-kernel versions (also 2.6.24 but based on rt3 patch)
Quoted 4 lines
So at the moment I am really clueless and I would be happy for any hint on how I can further debug this crash.
Best regards
Matthias
My guess would be that since you're running 10 threads on two processors at the highest available priority, you're starving all the hard IRQ threads (as well as soft irq and other kernel threads).
This is one of those power-tool moments where you can lock up the system with the wrong workload/priority combination.
Can you lock up the system running at -p49 (I suspect all your [IRQ-X] threads are running at priority 50)?
Clark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (GNU/Linux)
iEYEARECAAYFAkmIXCcACgkQHyuj/+TTEp0nqgCfdoddZvpxUwcpN5EXRAE90kmf hegAn18D1Ji5cGqeJwgHNk3Xb8fxMrCe
=F04E
-----END PGP SIGNATURE-----
Clark,
Quoted 10 lines
Quoted 5 lines
On two different desktop PC (both Core 2 Duo) and on a notebook (Core Duo) the kernel (2.6.24.7-rt26,
CONFIG_X86_32) locks up hard under a certain rt load. The failure can be reproduced reliably with the following command:
sudo ./cyclictest -p99 -t10 -n -i250My guess would be that since you're running 10 threads on two processors at the highest available priority, you're starving all the hard IRQ threads (as well as soft irq and other kernel threads). This is one of those power-tool moments where you can lock up the system with the wrong workload/priority combination.
Hmm, I would agree immediately, if the tasks were using all of the CPU power. But this is not the case. In the present case, running
# cyclictest -p99 -t10 -n -i250
results in about 10% CPU load. So there is plenty of time in between to let the system respond. I can reproduce the problem here (2.6.24-rt, 2.6.26-rt, various configurations). The system does not crash every time when cyclictest is started, only once in about 5 trials or so. If it crashes, then it does so immediately after being started. Whenever cyclictest survives for several seconds, then it never crashes at a later time. I tend to believe that we have a bug here, not a regular behavior.
Carsten.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 17 lines · On Tue, 03 Feb 2009 16:44:10 +0100 Carsten Emde <[email protected]> wrote:
Clark,
Quoted 10 lines
Quoted 5 lines
On two different desktop PC (both Core 2 Duo) and on a notebook (Core Duo) the kernel (2.6.24.7-rt26,
CONFIG_X86_32) locks up hard under a certain rt load. The failure can be reproduced reliably with the following command:
sudo ./cyclictest -p99 -t10 -n -i250My guess would be that since you're running 10 threads on two processors at the highest available priority, you're starving all the hard IRQ threads (as well as soft irq and other kernel threads). This is one of those power-tool moments where you can lock up the system with the wrong workload/priority combination.
Hmm, I would agree immediately, if the tasks were using all of the CPU power. But this is not the case. In the present case, running
# cyclictest -p99 -t10 -n -i250
Quoted 9 lines
results in about 10% CPU load. So there is plenty of time in between to let the system respond. I can reproduce the problem here (2.6.24-rt, 2.6.26-rt, various configurations). The system does not crash every time when cyclictest is started, only once in about 5 trials or so. If it crashes, then it does so immediately after being started. Whenever cyclictest survives for several seconds, then it never crashes at a later time. I tend to believe that we have a bug here, not a regular behavior.
That's what I get for firing off a response with out running it first :)
I don't have a 2.6.26 test box handy, but I'll set that up this afternoon. I agree with your results that this is a bug, since I haven't been able to kill my 2.6.24.7-based RT test system and if it was just starvation I should be able to do that easily.
Clark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (GNU/Linux)
iEYEARECAAYFAkmIcDMACgkQHyuj/+TTEp21DgCfUihH/6BUUS5o3SwlrnK57dJC /HQAoMEZp8bYuOEBiS6wVtbGgA0vi0MS
=X1jB
-----END PGP SIGNATURE-----
Hello Matthias,
2009/2/2 <[email protected]>:
Quoted 4 lines
Dear Linux rt users
On two different desktop PC (both Core 2 Duo) and on a notebook (Core Duo) the kernel (2.6.24.7-rt26,
CONFIG_X86_32) locks up hard under a certain rt load. The failure can be reproduced reliably with the following command:
In noticed some hangups too while using 2.6.24.7-rt26, while 2.6.24.7-rt20 was running fine.
I can reproduce it here on an ARM11 core, when CONFIG_DEBUG_LOCKING_API_SELFTESTS is set the system hangs during boot on rt26, while it boots properly on rt20. So, there is a regression here.
I did not find the time yet to bisect it to the offending RT-patch.
Kind regards,
Remy
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hello Denis,
Please keep the mailinglist on the CC.
Kind Regards,
Remy
2009/2/4 Denis Borisevich <[email protected]>:
Quoted 14 lines
2009/2/3 Remy Bohmer <[email protected]>:
Quoted 6 lines
In noticed some hangups too while using 2.6.24.7-rt26, while 2.6.24.7-rt20 was running fine.
I can reproduce it here on an ARM11 core, whenCONFIG_DEBUG_LOCKING_API_SELFTESTSis set the system hangs during boot on rt26, while it boots properly on rt20. So, there is a regression here.I second that. With
CONFIG_DEBUG_LOCKING_API_SELFTESTsystem hangs on boot with -rt26.--
Denis
Hi all,
Quoted 20 lines · Remy Bohmer
Hello Denis,
Please keep the mailinglist on the CC.
Kind Regards,
Remy
2009/2/4 Denis Borisevich <[email protected]>:
Quoted 11 lines
2009/2/3 Remy Bohmer <[email protected]>:
Quoted 6 lines
In noticed some hangups too while using 2.6.24.7-rt26, while 2.6.24.7-rt20 was running fine.
I can reproduce it here on an ARM11 core, whenCONFIG_DEBUG_LOCKING_API_SELFTESTSis set the system hangs during boot on rt26, while it boots properly on rt20. So, there is a regression here.I second that. With
CONFIG_DEBUG_LOCKING_API_SELFTESTsystem hangs on boot with -rt26.
Quoted 7 lines
Quoted 2 lines
--
Denis--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
For your info:
i tried to reproduce this with 2.6.24.7-rt21, and i cant reproduce the problem here.
Signature · 8 lines
Mit freundlichen Gruessen / Best regards Claus Gindhart SW R&D Kontron Modular Computers phone :++49 (0)8341-803-374 mailto:[email protected] http://www.kontron.com