Softlockup detected with 2.6.23-rc2-mm1
Hi,
I get call trace, when the file system stress is run on the 2.6.23-rc2-mm1 kernel on a Dual Core AMD Opteron (processor 270)
============================================\BUG: spinlock bad magic on CPU#1, fsx-linux/19721
lock: ffff8100028cef48, .magic: 00000000, .owner: <none>/-1, .owner_cpu: 0
Call Trace: [<ffffffff803359a6>] _raw_spin_lock+0x22/0xf6 [<ffffffff804e4c13>] _spin_lock_irqsave+0x9/0xe [<ffffffff803303fe>] prop_norm_single+0x40/0x9a [<ffffffff8026ac1f>] set_page_dirty+0x8d/0xc9 [<ffffffff8026bc09>] set_page_dirty_balance+0x9/0x39 [<ffffffff80271f14>] __do_fault+0x37a/0x395 [<ffffffff802738d7>] handle_mm_fault+0x342/0x6c3 [<ffffffff804e6ac6>] do_page_fault+0x3e5/0x7ab [<ffffffff802117d3>] arch_get_unmapped_area+0x184/0x1f9 [<ffffffff804e4c13>] _spin_lock_irqsave+0x9/0xe [<ffffffff803318cc>] __up_write+0x21/0x10d [<ffffffff804e500d>] error_exit+0x0/0x84 BUG: spinlock lockup on CPU#1, fsx-linux/19721, ffff8100028cef48 Call Trace: [<ffffffff80335a53>] _raw_spin_lock+0xcf/0xf6 [<ffffffff804e4c13>] _spin_lock_irqsave+0x9/0xe [<ffffffff803303fe>] prop_norm_single+0x40/0x9a [<ffffffff8026ac1f>] set_page_dirty+0x8d/0xc9 [<ffffffff8026bc09>] set_page_dirty_balance+0x9/0x39 [<ffffffff80271f14>] __do_fault+0x37a/0x395 [<ffffffff802738d7>] handle_mm_fault+0x342/0x6c3 [<ffffffff804e6ac6>] do_page_fault+0x3e5/0x7ab [<ffffffff802117d3>] arch_get_unmapped_area+0x184/0x1f9 [<ffffffff804e4c13>] _spin_lock_irqsave+0x9/0xe [<ffffffff803318cc>] __up_write+0x21/0x10d [<ffffffff804e500d>] error_exit+0x0/0x84 BUG: soft lockup - CPU#2 stuck for 11s! [events/2:17]
CPU 2:
Modules linked in: ipv6 hidp rfcomm l2cap bluetooth sunrpc battery ac lp parport_pc parport nvram amd_rng rng_core i2c_amd756 i2c_core button
Pid: 17, comm: events/2 Not tainted 2.6.23-rc2-mm1-autokern1 #1
RIP: 0010:[<ffffffff8021a4a4>] [<ffffffff8021a4a4>] __smp_call_function+0x63/0x84
RSP: 0018:ffff810001727e00 EFLAGS: 00000297
RAX: 00000000000008fc RBX: 0000000000000003 RCX: 0000000000000000
RDX: 00000000000008fc RSI: ffff810001727de0 RDI: 00000000000000fc
RBP: 0000000000000246 R08: 0000000000000003 R09: 0000000000000005
R10: 0000000000000010 R11: 0000000000000246 R12: 0000000000000400
R13: 0000000000000400 R14: 0000000000000000 R15: ffff81000102d980
FS: 00002b8de51be6f0(0000) GS:ffff8100016123c0(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000036d4b938a0 CR3: 000000001659f000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace: [<ffffffff80214c8d>] mcheck_check_cpu+0x0/0x30 [<ffffffff8021a4f7>] smp_call_function+0x32/0x49 [<ffffffff80214c8d>] mcheck_check_cpu+0x0/0x30 [<ffffffff8023aca7>] on_each_cpu+0x10/0x22 [<ffffffff80214532>] mcheck_timer+0x0/0x7c [<ffffffff8021454f>] mcheck_timer+0x1d/0x7c [<ffffffff804e4be1>] _spin_unlock_irq+0x9/0xc [<ffffffff80244c93>] run_workqueue+0x8d/0x11a [<ffffffff802454e2>] worker_thread+0x0/0xe4 [<ffffffff802455bc>] worker_thread+0xda/0xe4 [<ffffffff8024846b>] autoremove_wake_function+0x0/0x2e [<ffffffff80248360>] kthread+0x47/0x73 [<ffffffff8020ca78>] child_rip+0xa/0x12 [<ffffffff80248319>] kthread+0x0/0x73 [<ffffffff8020ca6e>] child_rip+0x0/0x12 BUG: soft lockup - CPU#2 stuck for 11s! [events/2:17]
CPU 2:
Modules linked in: ipv6 hidp rfcomm l2cap bluetooth sunrpc battery ac lp parport_pc parport nvram amd_rng rng_core i2c_amd756 i2c_core button
Pid: 17, comm: events/2 Not tainted 2.6.23-rc2-mm1-autokern1 #1
RIP: 0010:[<ffffffff8021a4a4>] [<ffffffff8021a4a4>] __smp_call_function+0x63/0x84
RSP: 0018:ffff810001727e00 EFLAGS: 00000297
RAX: 00000000000008fc RBX: 0000000000000003 RCX: 0000000000000000
RDX: 00000000000008fc RSI: ffff810001727de0 RDI: 00000000000000fc
RBP: 0000000000000246 R08: 0000000000000003 R09: 0000000000000005
R10: 0000000000000010 R11: 0000000000000246 R12: 0000000000000400
R13: 0000000000000400 R14: 0000000000000000 R15: ffff81000102d980
FS: 00002b8de51be6f0(0000) GS:ffff8100016123c0(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000036d4b938a0 CR3: 000000001659f000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace: [<ffffffff80214c8d>] mcheck_check_cpu+0x0/0x30 [<ffffffff8021a4f7>] smp_call_function+0x32/0x49 [<ffffffff80214c8d>] mcheck_check_cpu+0x0/0x30 [<ffffffff8023aca7>] on_each_cpu+0x10/0x22 [<ffffffff80214532>] mcheck_timer+0x0/0x7c [<ffffffff8021454f>] mcheck_timer+0x1d/0x7c [<ffffffff804e4be1>] _spin_unlock_irq+0x9/0xc [<ffffffff80244c93>] run_workqueue+0x8d/0x11a [<ffffffff802454e2>] worker_thread+0x0/0xe4 [<ffffffff802455bc>] worker_thread+0xda/0xe4 [<ffffffff8024846b>] autoremove_wake_function+0x0/0x2e [<ffffffff80248360>] kthread+0x47/0x73 [<ffffffff8020ca78>] child_rip+0xa/0x12 [<ffffffff80248319>] kthread+0x0/0x73 [<ffffffff8020ca6e>] child_rip+0x0/0x12 BUG: soft lockup - CPU#2 stuck for 11s! [events/2:17]
CPU 2:
Modules linked in: ipv6 hidp rfcomm l2cap bluetooth sunrpc battery ac lp parport_pc parport nvram amd_rng rng_core i2c_amd756 i2c_core button
Pid: 17, comm: events/2 Not tainted 2.6.23-rc2-mm1-autokern1 #1
RIP: 0010:[<ffffffff8021a4a4>] [<ffffffff8021a4a4>] __smp_call_function+0x63/0x84
RSP: 0018:ffff810001727e00 EFLAGS: 00000297
RAX: 00000000000008fc RBX: 0000000000000003 RCX: 0000000000000000
RDX: 00000000000008fc RSI: ffff810001727de0 RDI: 00000000000000fc
RBP: 0000000000000246 R08: 0000000000000003 R09: 0000000000000005
R10: 0000000000000010 R11: 0000000000000246 R12: 0000000000000400
R13: 0000000000000400 R14: 0000000000000000 R15: ffff81000102d980
FS: 00002b8de51be6f0(0000) GS:ffff8100016123c0(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000036d4b938a0 CR3: 0000000000201000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace: [<ffffffff80214c8d>] mcheck_check_cpu+0x0/0x30 [<ffffffff8021a4f7>] smp_call_function+0x32/0x49 [<ffffffff80214c8d>] mcheck_check_cpu+0x0/0x30 [<ffffffff8023aca7>] on_each_cpu+0x10/0x22 [<ffffffff80214532>] mcheck_timer+0x0/0x7c [<ffffffff8021454f>] mcheck_timer+0x1d/0x7c [<ffffffff804e4be1>] _spin_unlock_irq+0x9/0xc [<ffffffff80244c93>] run_workqueue+0x8d/0x11a [<ffffffff802454e2>] worker_thread+0x0/0xe4 [<ffffffff802455bc>] worker_thread+0xda/0xe4 [<ffffffff8024846b>] autoremove_wake_function+0x0/0x2e [<ffffffff80248360>] kthread+0x47/0x73 [<ffffffff8020ca78>] child_rip+0xa/0x12 [<ffffffff80248319>] kthread+0x0/0x73 [<ffffffff8020ca6e>] child_rip+0x0/0x12 BUG: soft lockup - CPU#2 stuck for 11s! [events/2:17]
CPU 2:
Modules linked in: ipv6 hidp rfcomm l2cap bluetooth sunrpc battery ac lp parport_pc parport nvram amd_rng rng_core i2c_amd756 i2c_core button
Pid: 17, comm: events/2 Not tainted 2.6.23-rc2-mm1-autokern1 #1
RIP: 0010:[<ffffffff8021a4a6>] [<ffffffff8021a4a6>] __smp_call_function+0x65/0x84
RSP: 0018:ffff810001727e00 EFLAGS: 00000297
RAX: 00000000000008fc RBX: 0000000000000003 RCX: 0000000000000000
RDX: 00000000000008fc RSI: ffff810001727de0 RDI: 00000000000000fc
RBP: 0000000000000246 R08: 0000000000000003 R09: 0000000000000005
R10: 0000000000000010 R11: 0000000000000246 R12: 0000000000000400
R13: 0000000000000400 R14: 0000000000000000 R15: ffff81000102d980
FS: 00002b8de51be6f0(0000) GS:ffff8100016123c0(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000036d4b938a0 CR3: 0000000000201000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace: [<ffffffff80214c8d>] mcheck_check_cpu+0x0/0x30 [<ffffffff8021a4f7>] smp_call_function+0x32/0x49 [<ffffffff80214c8d>] mcheck_check_cpu+0x0/0x30 [<ffffffff8023aca7>] on_each_cpu+0x10/0x22 [<ffffffff80214532>] mcheck_timer+0x0/0x7c [<ffffffff8021454f>] mcheck_timer+0x1d/0x7c [<ffffffff804e4be1>] _spin_unlock_irq+0x9/0xc [<ffffffff80244c93>] run_workqueue+0x8d/0x11a [<ffffffff802454e2>] worker_thread+0x0/0xe4 [<ffffffff802455bc>] worker_thread+0xda/0xe4 [<ffffffff8024846b>] autoremove_wake_function+0x0/0x2e [<ffffffff80248360>] kthread+0x47/0x73 [<ffffffff8020ca78>] child_rip+0xa/0x12 [<ffffffff80248319>] kthread+0x0/0x73 [<ffffffff8020ca6e>] child_rip+0x0/0x12 BUG: soft lockup - CPU#2 stuck for 11s! [events/2:17]
CPU 2:
Modules linked in: ipv6 hidp rfcomm l2cap bluetooth sunrpc battery ac lp parport_pc parport nvram amd_rng rng_core i2c_amd756 i2c_core button
Pid: 17, comm: events/2 Not tainted 2.6.23-rc2-mm1-autokern1 #1
RIP: 0010:[<ffffffff8021a4a4>] [<ffffffff8021a4a4>] __smp_call_function+0x63/0x84
RSP: 0018:ffff810001727e00 EFLAGS: 00000297
RAX: 00000000000008fc RBX: 0000000000000003 RCX: 0000000000000000
RDX: 00000000000008fc RSI: ffff810001727de0 RDI: 00000000000000fc
RBP: 0000000000000246 R08: 0000000000000003 R09: 0000000000000005
R10: 0000000000000010 R11: 0000000000000246 R12: 0000000000000400
R13: 0000000000000400 R14: 0000000000000000 R15: ffff81000102d980
FS: 00002b8de51be6f0(0000) GS:ffff8100016123c0(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000036d4b938a0 CR3: 0000000000201000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace: [<ffffffff80214c8d>] mcheck_check_cpu+0x0/0x30 [<ffffffff8021a4f7>] smp_call_function+0x32/0x49 [<ffffffff80214c8d>] mcheck_check_cpu+0x0/0x30 [<ffffffff8023aca7>] on_each_cpu+0x10/0x22 [<ffffffff80214532>] mcheck_timer+0x0/0x7c [<ffffffff8021454f>] mcheck_timer+0x1d/0x7c [<ffffffff804e4be1>] _spin_unlock_irq+0x9/0xc [<ffffffff80244c93>] run_workqueue+0x8d/0x11a [<ffffffff802454e2>] worker_thread+0x0/0xe4 [<ffffffff802455bc>] worker_thread+0xda/0xe4 [<ffffffff8024846b>] autoremove_wake_function+0x0/0x2e [<ffffffff80248360>] kthread+0x47/0x73 [<ffffffff8020ca78>] child_rip+0xa/0x12 [<ffffffff80248319>] kthread+0x0/0x73 [<ffffffff8020ca6e>] child_rip+0x0/0x12
Thanks,
Kamalesh Babulal.
Quoted 6 lines · On Fri, 10 Aug 2007 11:46:25 +0530 Kamalesh Babulal…
I get call trace, when the file system stress is run on the 2.6.23-rc2-mm1 kernel on a Dual Core AMD Opteron (processor 270)
============================================\BUG: spinlock bad magic on CPU#1, fsx-linux/19721
Yes, sorry, mm-dirty-balancing-for-tasks.patch had a bug. Those patches were removed from 2.6.23-rc2-mm2 - please test that instead.
Quoted 10 lines · Andrew Morton
Quoted 6 lines · On Fri, 10 Aug 2007 11:46:25 +0530 Kamalesh Babulal…
I get call trace, when the file system stress is run on the 2.6.23-rc2-mm1 kernel on a Dual Core AMD Opteron (processor 270)
============================================\BUG: spinlock bad magic on CPU#1, fsx-linux/19721
Quoted 8 lines
Yes, sorry, mm-dirty-balancing-for-tasks.patch had a bug. Those patches were removed from 2.6.23-rc2-mm2 - please test that instead.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
I get different call trace on AMD Opteron(tm) Processor 844 machine , I am not sure where it is related to the same patch
=====================================================
BUG: soft lockup - CPU#3 stuck for 11s! [pdflush:272] CPU 3: Modules linked in: Pid: 272, comm: pdflush Not tainted 2.6.23-rc2-mm1-autokern1 #1 RIP: 0010:[<ffffffff8021a9c3>] [<ffffffff8021a9c3>] flush_tlb_others+0x69/0x95 RSP: 0000:ffff810001f15a90 EFLAGS: 00000202 RAX: 0000000000000003 RBX: ffff810001f15ac0 RCX: 0000000000000008 RDX: 00000000000008f3 RSI: 00000000000000f3 RDI: 0000000000000002 RBP: 0000000000000000 R08: ffff810082f05210 R09: ffffffff802e60c1 R10: ffff8100815e6e70 R11: 0000000000000000 R12: ffff8101ffc38080 R13: ffffffff80358b47 R14: ffff810001f15a40 R15: ffff810081d73208 FS: 0000000000000000(0000) GS:ffff810180724280(0000) knlGS:00000000f7f75b80 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000000f7e20494 CR3: 00000000029f0000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [<ffffffff8021abd0>] flush_tlb_page+0x8f/0x97 [<ffffffff8026daee>] page_mkclean+0x120/0x171 [<ffffffff802e6227>] ext3_ordered_writepage+0x13f/0x16c [<ffffffff8025ed09>] clear_page_dirty_for_io+0x52/0xba [<ffffffff8025f002>] write_cache_pages+0x1b2/0x33a [<ffffffff8022a149>] update_curr+0xd9/0xf8 [<ffffffff8025e9ca>] __writepage+0x0/0x2a [<ffffffff8025f1a9>] generic_writepages+0x1f/0x25 [<ffffffff8025f1db>] do_writepages+0x2c/0x35 [<ffffffff8029b453>] __writeback_single_inode+0x1c9/0x346 [<ffffffff8023b809>] try_to_del_timer_sync+0x55/0x60 [<ffffffff8023b826>] del_timer_sync+0x12/0x1f [<ffffffff8022a149>] update_curr+0xd9/0xf8 [<ffffffff8022a446>] dequeue_entity+0x7d/0x92 [<ffffffff8029ba20>] generic_sync_sb_inodes+0x216/0x372 [<ffffffff8029bb99>] sync_sb_inodes+0x1d/0x1f [<ffffffff8029bdd9>] writeback_inodes+0x83/0xd6 [<ffffffff8025e82b>] wb_kupdate+0xa0/0x113 [<ffffffff8025f658>] pdflush+0x156/0x206 [<ffffffff8025e78b>] wb_kupdate+0x0/0x113 [<ffffffff8025f502>] pdflush+0x0/0x206 [<ffffffff80245660>] kthread+0x44/0x6d [<ffffffff8020c5e8>] child_rip+0xa/0x12 [<ffffffff8024561c>] kthread+0x0/0x6d [<ffffffff8020c5de>] child_rip+0x0/0x12
Thanks
Kamalesh Babulal.
Quoted 25 lines · Kamalesh Babulal
Quoted 12 lines · Andrew Morton
Quoted 7 lines · On Fri, 10 Aug 2007 11:46:25 +0530 Kamalesh Babulal…
I get call trace, when the file system stress is run on the 2.6.23-rc2-mm1 kernel on a Dual Core AMD Opteron (processor 270)
============================================\BUG: spinlock bad magic on CPU#1, fsx-linux/19721
Quoted 9 lines
Yes, sorry, mm-dirty-balancing-for-tasks.patch had a bug. Those patches were
removed from 2.6.23-rc2-mm2 - please test that instead.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/I get different call trace on AMD Opteron(tm) Processor 844 machine , I am not sure where it is related to the same patch
Quoted 7 lines
=====================================================
BUG: soft lockup - CPU#3 stuck for 11s! [pdflush:272] CPU 3: Modules linked in: Pid: 272, comm: pdflush Not tainted 2.6.23-rc2-mm1-autokern1 #1 RIP: 0010:[<ffffffff8021a9c3>] [<ffffffff8021a9c3>] flush_tlb_others+0x69/0x95
Cannot be 100% sure but of the group of machines showing your original problem one showed this form. Dropping the patches indicated by Andrew seemed to fix both symptoms. So I think it is highly likely the same thing.
-apw
Quoted 10 lines · Andrew Morton
Quoted 6 lines · On Fri, 10 Aug 2007 11:46:25 +0530 Kamalesh Babulal…
I get call trace, when the file system stress is run on the 2.6.23-rc2-mm1 kernel on a Dual Core AMD Opteron (processor 270)
============================================\BUG: spinlock bad magic on CPU#1, fsx-linux/19721
Quoted 8 lines
Yes, sorry, mm-dirty-balancing-for-tasks.patch had a bug. Those patches were removed from 2.6.23-rc2-mm2 - please test that instead.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
The Call trace is not reproducible in the 2.6.23-rc2-mm2.