Skip to contentSkip to conversation

2.6.17-1.2145_FC5 mmap-related soft lockup

DISCUSSION Gary Funck 2 participants 5 messages Jul 15, 2006 – Jul 19, 2006

lore
Share & tools
lore

Gary Funck

author
#0 lore raw

A test program which allocates about 256M of MAP_ANONYMOUS mmap memory, and then spawns 4 processess, where each process i writes to 1/4 of the mapped memory, and then reads the memory written by
the process (i + 1)%4, triggers a soft lockup, when exiting. Hardware:
dual core dual Opteron 275 (Tyan motherboard, 4G physical memory) has been rock solid reliable.

BUG: soft lockup detected on CPU#3!

Call Trace: <IRQ> <ffffffff802b2fb5>{softlockup_tick+219}
       <ffffffff8029708e>{update_process_times+66}
<ffffffff8027a3ed>{smp_local_timer_interrupt+35}
       <ffffffff8027aa95>{smp_apic_timer_interrupt+65}
<ffffffff80263acb>{apic_timer_interrupt+135} <EOI>
       <ffffffff8020e578>{__set_page_dirty_nobuffers+0}
<ffffffff8026a128>{_write_unlock_irq+11}
       <ffffffff8020e62d>{__set_page_dirty_nobuffers+181}
<ffffffff80207af6>{unmap_vmas+1037}
       <ffffffff8023c7d9>{exit_mmap+120} <ffffffff8023eda8>{mmput+44}
       <ffffffff80215ece>{do_exit+599}
<ffffffff8024cacd>{debug_mutex_init+0}
       <ffffffff80262f01>{tracesys+209}
BUG: soft lockup detected on CPU#0!

Call Trace: <IRQ> <ffffffff802b2fb5>{softlockup_tick+219}
       <ffffffff8029708e>{update_process_times+66}
<ffffffff8027a3ed>{smp_local_timer_interrupt+35}
       <ffffffff8027aa95>{smp_apic_timer_interrupt+65}
<ffffffff80263acb>{apic_timer_interrupt+135} <EOI>
       <ffffffff8020e578>{__set_page_dirty_nobuffers+0}
<ffffffff8026a128>{_write_unlock_irq+11}
       <ffffffff8020e62d>{__set_page_dirty_nobuffers+181}
<ffffffff80207af6>{unmap_vmas+1037}
       <ffffffff8023c7d9>{exit_mmap+120} <ffffffff8023eda8>{mmput+44}
       <ffffffff80215ece>{do_exit+599}
<ffffffff8024cacd>{debug_mutex_init+0}
       <ffffffff80262f01>{tracesys+209}
BUG: soft lockup detected on CPU#2!

Call Trace: <IRQ> <ffffffff802b2fb5>{softlockup_tick+219}
       <ffffffff8029708e>{update_process_times+66}
<ffffffff8027a3ed>{smp_local_timer_interrupt+35}
       <ffffffff8027aa95>{smp_apic_timer_interrupt+65}
<ffffffff80263acb>{apic_timer_interrupt+135} <EOI>
       <ffffffff8020e578>{__set_page_dirty_nobuffers+0}
<ffffffff8026a128>{_write_unlock_irq+11}
       <ffffffff8020e62d>{__set_page_dirty_nobuffers+181}
<ffffffff80207af6>{unmap_vmas+1037}
       <ffffffff8023c7d9>{exit_mmap+120} <ffffffff8023eda8>{mmput+44}
       <ffffffff80215ece>{do_exit+599}
<ffffffff8024cacd>{debug_mutex_init+0}
       <ffffffff80262f01>{tracesys+209}
BUG: soft lockup detected on CPU#1!

Call Trace: <IRQ> <ffffffff802b2fb5>{softlockup_tick+219}
       <ffffffff8029708e>{update_process_times+66}
<ffffffff8027a3ed>{smp_local_timer_interrupt+35}
       <ffffffff8027aa95>{smp_apic_timer_interrupt+65}
<ffffffff80263acb>{apic_timer_interrupt+135} <EOI>
       <ffffffff8020e578>{__set_page_dirty_nobuffers+0}
<ffffffff8026a128>{_write_unlock_irq+11}
       <ffffffff8020e62d>{__set_page_dirty_nobuffers+181}
<ffffffff80207af6>{unmap_vmas+1037}
       <ffffffff8023c7d9>{exit_mmap+120} <ffffffff8023eda8>{mmput+44}
       <ffffffff80215ece>{do_exit+599}
<ffffffff8024cacd>{debug_mutex_init+0}
       <ffffffff80262f01>{tracesys+209}

The test program runs successfully, but hangs several seconds upon exit.

The hardware and software configuration has been solid for several months, but
we have seen timer-related synchronization issues with recent kernels (where ntp has to force a re-sync for example, and an occasional lost ticks message).

The test program mentioned above is more complicated than described, and can't easily be reproduced in source form, but the binary could be made available.

#1 lore raw
Quoted 35 lines · On Sat, 15 Jul 2006 10:07:26 -0700 "Gary Funck" <[email protected]> wrote:

A test program which allocates about 256M of MAP_ANONYMOUS mmap memory, and then spawns 4 processess, where each process i writes to 1/4 of the mapped memory, and then reads the memory written by
the process (i + 1)%4, triggers a soft lockup, when exiting. Hardware:
dual core dual Opteron 275 (Tyan motherboard, 4G physical memory) has been rock solid reliable.

BUG: soft lockup detected on CPU#3!

Call Trace: <IRQ> <ffffffff802b2fb5>{softlockup_tick+219}
       <ffffffff8029708e>{update_process_times+66}
<ffffffff8027a3ed>{smp_local_timer_interrupt+35}
       <ffffffff8027aa95>{smp_apic_timer_interrupt+65}
<ffffffff80263acb>{apic_timer_interrupt+135} <EOI>
       <ffffffff8020e578>{__set_page_dirty_nobuffers+0}
<ffffffff8026a128>{_write_unlock_irq+11}
       <ffffffff8020e62d>{__set_page_dirty_nobuffers+181}
<ffffffff80207af6>{unmap_vmas+1037}
       <ffffffff8023c7d9>{exit_mmap+120} <ffffffff8023eda8>{mmput+44}
       <ffffffff80215ece>{do_exit+599}
<ffffffff8024cacd>{debug_mutex_init+0}
       <ffffffff80262f01>{tracesys+209}

..

The test program runs successfully, but hangs several seconds upon exit.

The hardware and software configuration has been solid for several months, but
we have seen timer-related synchronization issues with recent kernels (where ntp has to force a re-sync for example, and an occasional lost ticks message).

Quoted 3 lines

The test program mentioned above is more complicated than described, and can't easily be reproduced in source form, but the binary could be made available.

ah-hah. This sounds like the write_lock(tree_lock) starvation bug.

Are you able to confirm that setting CONFIG_DEBUG_SPINLOCK=n fixes it?

And are you able to get us a copy of that test app?

Thanks.

author
#2 lore raw
Quoted 5 lines

Are you able to confirm that setting CONFIG_DEBUG_SPINLOCK=n fixes it?

And are you able to get us a copy of that test app?

Yes, I just ran the test with 2.6.17.6. With CONFIG_DEBUG_SPINLOCK=y the test fails and the soft lockup situation often results. However, when built with CONFIG_DEBUG_SPINLOCK=n, the test passes, and runs rather quickly in comparison to when it fails.

I've attached a slightly updated version of the test case. It is a little more carefully crafted and prints some output so that you have some idea that it is working.

#3 lore raw
Quoted 6 lines · On Tue, 18 Jul 2006 19:25:16 -0700 "Gary Funck" <[email protected]> wrote:
Quoted 5 lines

Are you able to confirm that setting CONFIG_DEBUG_SPINLOCK=n fixes it?

And are you able to get us a copy of that test app?

Quoted 9 lines

Yes, I just ran the test with 2.6.17.6. With CONFIG_DEBUG_SPINLOCK=y the test fails and the soft lockup situation often results. However, when built with CONFIG_DEBUG_SPINLOCK=n, the test passes, and runs rather quickly in comparison to when it fails.

I've attached a slightly updated version of the test case. It is a little more carefully crafted and prints some output so that you have some idea that it is working.

That's great, thanks. That pretty much confirms that this long-standing box-killing rwlock starvation bug is specific to Opterons. Neither Ingo nor I have Opteron machines so a fix will take a little longer than one would expect.

Meanwhile, an appropriate workaround is to disable CONFIG_DEBUG_SPINLOCK.

author
#4 lore raw
In reply to Gary Funck
Quoted 13 lines

From: Gary Funck
Sent: Saturday, July 15, 2006 10:07 AM

A test program which allocates about 256M of MAP_ANONYMOUS mmap memory, and then spawns 4 processess, where each process i writes to 1/4 of the mapped memory, and then reads the memory written by
the process (i + 1)%4, triggers a soft lockup, when exiting. Hardware:
dual core dual Opteron 275 (Tyan motherboard, 4G physical memory) has been rock solid reliable.

Quoted 1 line
BUG: soft lockup detected on CPU#3!

Follow up, the attached test program, when compiled on FC5 with the 2.6.17-1.2145 kernel will cause lost timer ticks, lost RS-232 interrupts, and often will lead to the soft lockup situation shown below:

BUG: soft lockup detected on CPU#0!

Call Trace: <IRQ> <ffffffff802b2fb5>{softlockup_tick+219}
       <ffffffff8029708e>{update_process_times+66}
<ffffffff8027a3ed>{smp_local_timer_interrupt+35}
       <ffffffff8027aa95>{smp_apic_timer_interrupt+65}
<ffffffff80263acb>{apic_timer_interrupt+135} <EOI>
       <ffffffff8020e578>{__set_page_dirty_nobuffers+0}
<ffffffff8020a7ab>{release_pages+111}
       <ffffffff80267b76>{thread_return+0}
<ffffffff80267bd4>{thread_return+94}
       <ffffffff8020e578>{__set_page_dirty_nobuffers+0}
<ffffffff8020e578>{__set_page_dirty_nobuffers+0}
       <ffffffff8020e09e>{free_pages_and_swap_cache+115}
<ffffffff80207b62>{unmap_vmas+1145}
       <ffffffff8023c7d9>{exit_mmap+120} <ffffffff8023eda8>{mmput+44}
       <ffffffff80215ece>{do_exit+599}
<ffffffff8024cacd>{debug_mutex_init+0}
       <ffffffff80262f01>{tracesys+209}

After compiling the code, A continuous loop like the following, seems to eventually lead to the soft lockup situation (FC5, x86_64) shown above:

while true; do
a.out
done