2.6.17-1.2145_FC5 mmap-related soft lockup
A test program which allocates about 256M of MAP_ANONYMOUS mmap memory, and then spawns 4 processess, where each process i writes to 1/4 of the mapped memory, and then reads the memory written by
the process (i + 1)%4, triggers a soft lockup, when exiting. Hardware:
dual core dual Opteron 275 (Tyan motherboard, 4G physical memory) has been rock solid reliable.
BUG: soft lockup detected on CPU#3!
Call Trace: <IRQ> <ffffffff802b2fb5>{softlockup_tick+219}
<ffffffff8029708e>{update_process_times+66}
<ffffffff8027a3ed>{smp_local_timer_interrupt+35}
<ffffffff8027aa95>{smp_apic_timer_interrupt+65}
<ffffffff80263acb>{apic_timer_interrupt+135} <EOI>
<ffffffff8020e578>{__set_page_dirty_nobuffers+0}
<ffffffff8026a128>{_write_unlock_irq+11}
<ffffffff8020e62d>{__set_page_dirty_nobuffers+181}
<ffffffff80207af6>{unmap_vmas+1037}
<ffffffff8023c7d9>{exit_mmap+120} <ffffffff8023eda8>{mmput+44}
<ffffffff80215ece>{do_exit+599}
<ffffffff8024cacd>{debug_mutex_init+0}
<ffffffff80262f01>{tracesys+209}
BUG: soft lockup detected on CPU#0!
Call Trace: <IRQ> <ffffffff802b2fb5>{softlockup_tick+219}
<ffffffff8029708e>{update_process_times+66}
<ffffffff8027a3ed>{smp_local_timer_interrupt+35}
<ffffffff8027aa95>{smp_apic_timer_interrupt+65}
<ffffffff80263acb>{apic_timer_interrupt+135} <EOI>
<ffffffff8020e578>{__set_page_dirty_nobuffers+0}
<ffffffff8026a128>{_write_unlock_irq+11}
<ffffffff8020e62d>{__set_page_dirty_nobuffers+181}
<ffffffff80207af6>{unmap_vmas+1037}
<ffffffff8023c7d9>{exit_mmap+120} <ffffffff8023eda8>{mmput+44}
<ffffffff80215ece>{do_exit+599}
<ffffffff8024cacd>{debug_mutex_init+0}
<ffffffff80262f01>{tracesys+209}
BUG: soft lockup detected on CPU#2!
Call Trace: <IRQ> <ffffffff802b2fb5>{softlockup_tick+219}
<ffffffff8029708e>{update_process_times+66}
<ffffffff8027a3ed>{smp_local_timer_interrupt+35}
<ffffffff8027aa95>{smp_apic_timer_interrupt+65}
<ffffffff80263acb>{apic_timer_interrupt+135} <EOI>
<ffffffff8020e578>{__set_page_dirty_nobuffers+0}
<ffffffff8026a128>{_write_unlock_irq+11}
<ffffffff8020e62d>{__set_page_dirty_nobuffers+181}
<ffffffff80207af6>{unmap_vmas+1037}
<ffffffff8023c7d9>{exit_mmap+120} <ffffffff8023eda8>{mmput+44}
<ffffffff80215ece>{do_exit+599}
<ffffffff8024cacd>{debug_mutex_init+0}
<ffffffff80262f01>{tracesys+209}
BUG: soft lockup detected on CPU#1!
Call Trace: <IRQ> <ffffffff802b2fb5>{softlockup_tick+219}
<ffffffff8029708e>{update_process_times+66}
<ffffffff8027a3ed>{smp_local_timer_interrupt+35}
<ffffffff8027aa95>{smp_apic_timer_interrupt+65}
<ffffffff80263acb>{apic_timer_interrupt+135} <EOI>
<ffffffff8020e578>{__set_page_dirty_nobuffers+0}
<ffffffff8026a128>{_write_unlock_irq+11}
<ffffffff8020e62d>{__set_page_dirty_nobuffers+181}
<ffffffff80207af6>{unmap_vmas+1037}
<ffffffff8023c7d9>{exit_mmap+120} <ffffffff8023eda8>{mmput+44}
<ffffffff80215ece>{do_exit+599}
<ffffffff8024cacd>{debug_mutex_init+0}
<ffffffff80262f01>{tracesys+209}The test program runs successfully, but hangs several seconds upon exit.
The hardware and software configuration has been solid for several months, but
we have seen timer-related synchronization issues with recent kernels (where ntp has to force a re-sync for example, and an occasional lost ticks message).
The test program mentioned above is more complicated than described, and can't easily be reproduced in source form, but the binary could be made available.
Quoted 35 lines · On Sat, 15 Jul 2006 10:07:26 -0700 "Gary Funck" <[email protected]> wrote:
A test program which allocates about 256M of
MAP_ANONYMOUSmmap memory, and then spawns 4 processess, where each process i writes to 1/4 of the mapped memory, and then reads the memory written by
the process (i + 1)%4, triggers a soft lockup, when exiting. Hardware:
dual core dual Opteron 275 (Tyan motherboard, 4G physical memory) has been rock solid reliable.BUG: soft lockup detected on CPU#3! Call Trace: <IRQ> <ffffffff802b2fb5>{softlockup_tick+219} <ffffffff8029708e>{update_process_times+66} <ffffffff8027a3ed>{smp_local_timer_interrupt+35} <ffffffff8027aa95>{smp_apic_timer_interrupt+65} <ffffffff80263acb>{apic_timer_interrupt+135} <EOI> <ffffffff8020e578>{__set_page_dirty_nobuffers+0} <ffffffff8026a128>{_write_unlock_irq+11} <ffffffff8020e62d>{__set_page_dirty_nobuffers+181} <ffffffff80207af6>{unmap_vmas+1037} <ffffffff8023c7d9>{exit_mmap+120} <ffffffff8023eda8>{mmput+44} <ffffffff80215ece>{do_exit+599} <ffffffff8024cacd>{debug_mutex_init+0} <ffffffff80262f01>{tracesys+209}..
The test program runs successfully, but hangs several seconds upon exit.
The hardware and software configuration has been solid for several months, but
we have seen timer-related synchronization issues with recent kernels (where ntp has to force a re-sync for example, and an occasional lost ticks message).
Quoted 3 lines
The test program mentioned above is more complicated than described, and can't easily be reproduced in source form, but the binary could be made available.
ah-hah. This sounds like the write_lock(tree_lock) starvation bug.
Are you able to confirm that setting CONFIG_DEBUG_SPINLOCK=n fixes it?
And are you able to get us a copy of that test app?
Thanks.
Quoted 5 lines
Are you able to confirm that setting
CONFIG_DEBUG_SPINLOCK=nfixes it?And are you able to get us a copy of that test app?
Yes, I just ran the test with 2.6.17.6. With CONFIG_DEBUG_SPINLOCK=y the test fails and the soft lockup situation often results. However, when built with CONFIG_DEBUG_SPINLOCK=n, the test passes, and runs rather quickly in comparison to when it fails.
I've attached a slightly updated version of the test case. It is a little more carefully crafted and prints some output so that you have some idea that it is working.
Quoted 6 lines · On Tue, 18 Jul 2006 19:25:16 -0700 "Gary Funck" <[email protected]> wrote:
Quoted 5 lines
Are you able to confirm that setting
CONFIG_DEBUG_SPINLOCK=nfixes it?And are you able to get us a copy of that test app?
Quoted 9 lines
Yes, I just ran the test with 2.6.17.6. With
CONFIG_DEBUG_SPINLOCK=ythe test fails and the soft lockup situation often results. However, when built withCONFIG_DEBUG_SPINLOCK=n, the test passes, and runs rather quickly in comparison to when it fails.I've attached a slightly updated version of the test case. It is a little more carefully crafted and prints some output so that you have some idea that it is working.
That's great, thanks. That pretty much confirms that this long-standing box-killing rwlock starvation bug is specific to Opterons. Neither Ingo nor I have Opteron machines so a fix will take a little longer than one would expect.
Meanwhile, an appropriate workaround is to disable CONFIG_DEBUG_SPINLOCK.
Quoted 13 lines
From: Gary Funck
Sent: Saturday, July 15, 2006 10:07 AMA test program which allocates about 256M of
MAP_ANONYMOUSmmap memory, and then spawns 4 processess, where each process i writes to 1/4 of the mapped memory, and then reads the memory written by
the process (i + 1)%4, triggers a soft lockup, when exiting. Hardware:
dual core dual Opteron 275 (Tyan motherboard, 4G physical memory) has been rock solid reliable.
Quoted 1 line
BUG: soft lockup detected on CPU#3!
Follow up, the attached test program, when compiled on FC5 with the 2.6.17-1.2145 kernel will cause lost timer ticks, lost RS-232 interrupts, and often will lead to the soft lockup situation shown below:
BUG: soft lockup detected on CPU#0!
Call Trace: <IRQ> <ffffffff802b2fb5>{softlockup_tick+219}
<ffffffff8029708e>{update_process_times+66}
<ffffffff8027a3ed>{smp_local_timer_interrupt+35}
<ffffffff8027aa95>{smp_apic_timer_interrupt+65}
<ffffffff80263acb>{apic_timer_interrupt+135} <EOI>
<ffffffff8020e578>{__set_page_dirty_nobuffers+0}
<ffffffff8020a7ab>{release_pages+111}
<ffffffff80267b76>{thread_return+0}
<ffffffff80267bd4>{thread_return+94}
<ffffffff8020e578>{__set_page_dirty_nobuffers+0}
<ffffffff8020e578>{__set_page_dirty_nobuffers+0}
<ffffffff8020e09e>{free_pages_and_swap_cache+115}
<ffffffff80207b62>{unmap_vmas+1145}
<ffffffff8023c7d9>{exit_mmap+120} <ffffffff8023eda8>{mmput+44}
<ffffffff80215ece>{do_exit+599}
<ffffffff8024cacd>{debug_mutex_init+0}
<ffffffff80262f01>{tracesys+209}After compiling the code, A continuous loop like the following, seems to eventually lead to the soft lockup situation (FC5, x86_64) shown above:
while true; do
a.out
done