Assertion failed: atomic_read(&pag->pag_ref) == 0
Folks,
Since themainline merge, I've been getting unmount failures during shutdown that look like:
Unmounting local filesystems...done.
Shutting down LVM Volume Groups[ 7088.820123] Assertion failed: atomic_read(&pag->pag_ref) == 0, file: fs/xfs/xfs_mount.c, line: 259
[ 7088.821811] ------------[ cut here ]------------ [ 7088.822594] kernel BUG at fs/xfs/support/debug.c:108! [ 7088.823383] invalid opcode: 0000 [#1] SMP [ 7088.824019] last sysfs file: /sys/devices/system/node/node0/cpumap [ 7088.824045] CPU 1 [ 7088.824045] Modules linked in: [ 7088.824045] [ 7088.824045] Pid: 0, comm: kworker/0:0 Not tainted 2.6.36-dgc+ #587 /Bochs [ 7088.824045] RIP: 0010:[<ffffffff814b74cf>] [<ffffffff814b74cf>] assfail+0x1f/0x30 [ 7088.824045] RSP: 0018:ffff8800df003e50 EFLAGS: 00010286 [ 7088.824045] RAX: 0000000000000069 RBX: ffff88011760a400 RCX: 0000000000000001 [ 7088.824045] RDX: ffff88011b7742c0 RSI: 0000000000000001 RDI: 0000000000000246 [ 7088.824045] RBP: ffff8800df003e50 R08: 0000000000000001 R09: 0000000000000001 [ 7088.824045] R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff81ef8f00 [ 7088.824045] R13: ffff880117118df8 R14: ffff8800df1cecf0 R15: ffff880116ebf6e8 [ 7088.824045] FS: 0000000000000000(0000) GS:ffff8800df000000(0000) knlGS:0000000000000000 [ 7088.824045] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 7088.824045] CR2: 00007ffd8c8b6990 CR3: 0000000001edb000 CR4: 00000000000006e0 [ 7088.824045] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7088.824045] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7088.824045] Process kworker/0:0 (pid: 0, threadinfo ffff88011b776000, task ffff88011b7742c0) [ 7088.824045] Stack: [ 7088.824045] ffff8800df003e70 ffffffff81499007 ffff8800df003e70 ffff8800df1cecc0 [ 7088.824045] <0> ffff8800df003ed0 ffffffff810e900a 0000000000000001 000000000000000a [ 7088.824045] <0> ffff880100000006 0000000000000202 0000000000000100 0000000000000048 [ 7088.824045] Call Trace: [ 7088.824045] <IRQ> [ 7088.824045] [<ffffffff81499007>] __xfs_free_perag+0x37/0x50 [ 7088.824045] [<ffffffff810e900a>] __rcu_process_callbacks+0x13a/0x3e0 [ 7088.824045] [<ffffffff810e92d8>] rcu_process_callbacks+0x28/0x50 [ 7088.824045] [<ffffffff8108848d>] __do_softirq+0xcd/0x290 [ 7088.824045] [<ffffffff810a8808>] ? hrtimer_interrupt+0x138/0x250 [ 7088.824045] [<ffffffff81037f5c>] call_softirq+0x1c/0x50 [ 7088.824045] [<ffffffff810398dd>] do_softirq+0x9d/0xd0 [ 7088.824045] [<ffffffff810881e5>] irq_exit+0x95/0xa0 [ 7088.824045] [<ffffffff81b06380>] smp_apic_timer_interrupt+0x70/0x9b [ 7088.824045] [<ffffffff81037a13>] apic_timer_interrupt+0x13/0x20 [ 7088.824045] <EOI> [ 7088.824045] [<ffffffff81060f6b>] ? native_safe_halt+0xb/0x10 [ 7088.824045] [<ffffffff810baded>] ? trace_hardirqs_on+0xd/0x10 [ 7088.824045] [<ffffffff8103fd70>] default_idle+0x50/0xb0 [ 7088.824045] [<ffffffff81035e28>] cpu_idle+0x78/0x100 [ 7088.824045] [<ffffffff81af627b>] start_secondary+0x1ac/0x1b1 [ 7088.824045] Code: 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f 44 00 00 31 c0 89 d1 48 89 f2 48 89 fe 48 c7 c7 08 38 df 81 e8 7b 34 64 00 <0f> 0b eb fe 66 66 66 66 2e [ 7088.824045] RIP [<ffffffff814b74cf>] assfail+0x1f/0x30 [ 7088.824045] RSP <ffff8800df003e50> [ 7088.863091] ---[ end trace ec76f8135c3adba9 ]---
I'm not seeing failures during xfstests runs, it seems that dbench may be the trigger. Is anyone else seeing reference counting problems like this on the current linus tree?
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines · Dave Chinner
Folks,
Since themainline merge, I've been getting unmount failures during shutdown that look like:
I've done quite a few mainline runs, but haven't seen anything like that. On the other hand I see completely silent hangs in 076 once in a while.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Christoph Hellwig
Quoted 4 lines · Dave Chinner
Folks,
Since themainline merge, I've been getting unmount failures during shutdown that look like:
I've done quite a few mainline runs, but haven't seen anything like that. On the other hand I see completely silent hangs in 076 once in a while.
I can't reproduce that anymore since upgrading to a newer Linus' tree. But I've hit the following twice now:
070 7s ...
[ 1208.818651] Assertion failed: dp->i_d.di_forkoff, file: fs/xfs/xfs_attr_leaf.c, line: 373 [ 1208.823852] ------------[ cut here ]------------ [ 1208.825880] kernel BUG at fs/xfs/support/debug.c:108! [ 1208.827647] invalid opcode: 0000 [#1] SMP [ 1208.827724] last sysfs file: /sys/devices/virtio-pci/virtio1/block/vdb/removable [ 1208.827724] Modules linked in: [ 1208.827724] [ 1208.827724] Pid: 4422, comm: fsstress Not tainted 2.6.36-xfs+ #68 /Bochs [ 1208.827724] EIP: 0060:[<c04eba6e>] EFLAGS: 00010282 CPU: 0 [ 1208.827724] EIP is at assfail+0x1e/0x30 [ 1208.827724] EAX: 00000060 EBX: f4c8d530 ECX: ffffffa0 EDX: 016e0000 [ 1208.827724] ESI: 00000051 EDI: 00000000 EBP: ece99d20 ESP: ece99d10 [ 1208.827724] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 [ 1208.827724] Process fsstress (pid: 4422, ti=ece98000 task=f6310040 task.ti=ece98000) [ 1208.827724] Stack: [ 1208.827724] c0b9dbb8 c0b5fa7e c0b5f912 00000175 ece99d58 c04848a9 00000001 f65cea88 [ 1208.827724] <0> 02e99d44 f5533258 ece99d78 f4c8d530 00000002 00000003 00000051 f4c8d530 [ 1208.827724] <0> 00000000 0000000a ece99df8 c04807d5 00000000 00000004 00000003 ece99d78 [ 1208.827724] Call Trace: [ 1208.827724] [<c04848a9>] ? xfs_attr_shortform_remove+0x159/0x270 [ 1208.827724] [<c04807d5>] ? xfs_attr_remove_int+0x225/0x280 [ 1208.827724] [<c04b512a>] ? xfs_iunlock+0xaa/0x160 [ 1208.827724] [<c04808cc>] ? xfs_attr_remove+0x9c/0xc0 [ 1208.827724] [<c04eb769>] ? xfs_xattr_set+0x89/0x90 [ 1208.827724] [<c02283ec>] ? generic_removexattr+0x8c/0xa0 [ 1208.827724] [<c022884e>] ? vfs_removexattr+0x7e/0xf0 [ 1208.827724] [<c01f0e46>] ? might_fault+0x46/0xa0 [ 1208.827724] [<c02288fb>] ? removexattr+0x3b/0x60 [ 1208.827724] [<c0206d8e>] ? kfree_debugcheck+0xe/0x30 [ 1208.827724] [<c02071bc>] ? cache_free_debugcheck+0x17c/0x250 [ 1208.827724] [<c067f994>] ? debug_check_no_obj_freed+0x124/0x180 [ 1208.827724] [<c0195d16>] ? debug_check_no_locks_freed+0xb6/0x140 [ 1208.827724] [<c0207345>] ? kmem_cache_free+0xb5/0x120 [ 1208.827724] [<c0195c5b>] ? trace_hardirqs_on+0xb/0x10 [ 1208.827724] [<c0219d4a>] ? user_path_at+0x4a/0x80 [ 1208.827724] [<c02120d2>] ? sys_stat64+0x22/0x30 [ 1208.827724] [<c02289fb>] ? sys_lremovexattr+0x6b/0x80 [ 1208.827724] [<c0911f2d>] ? syscall_call+0x7/0xb
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 55 lines · Dave Chinner
Folks,
Since themainline merge, I've been getting unmount failures during shutdown that look like:
Unmounting local filesystems...done.
Shutting down LVM Volume Groups[ 7088.820123] Assertion failed:atomic_read(&pag->pag_ref)== 0, file:fs/xfs/xfs_mount.c, line: 259[ 7088.821811] ------------[ cut here ]------------ [ 7088.822594] kernel BUG at fs/xfs/support/debug.c:108! [ 7088.823383] invalid opcode: 0000 [#1] SMP [ 7088.824019] last sysfs file: /sys/devices/system/node/node0/cpumap [ 7088.824045] CPU 1 [ 7088.824045] Modules linked in: [ 7088.824045] [ 7088.824045] Pid: 0, comm: kworker/0:0 Not tainted 2.6.36-dgc+ #587 /Bochs [ 7088.824045] RIP: 0010:[<ffffffff814b74cf>] [<ffffffff814b74cf>] assfail+0x1f/0x30 [ 7088.824045] RSP: 0018:ffff8800df003e50 EFLAGS: 00010286 [ 7088.824045] RAX: 0000000000000069 RBX: ffff88011760a400 RCX: 0000000000000001 [ 7088.824045] RDX: ffff88011b7742c0 RSI: 0000000000000001 RDI: 0000000000000246 [ 7088.824045] RBP: ffff8800df003e50 R08: 0000000000000001 R09: 0000000000000001 [ 7088.824045] R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff81ef8f00 [ 7088.824045] R13: ffff880117118df8 R14: ffff8800df1cecf0 R15: ffff880116ebf6e8 [ 7088.824045] FS: 0000000000000000(0000) GS:ffff8800df000000(0000) knlGS:0000000000000000 [ 7088.824045] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 7088.824045] CR2: 00007ffd8c8b6990 CR3: 0000000001edb000 CR4: 00000000000006e0 [ 7088.824045] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7088.824045] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7088.824045] Process kworker/0:0 (pid: 0, threadinfo ffff88011b776000, task ffff88011b7742c0) [ 7088.824045] Stack: [ 7088.824045] ffff8800df003e70 ffffffff81499007 ffff8800df003e70 ffff8800df1cecc0 [ 7088.824045] <0> ffff8800df003ed0 ffffffff810e900a 0000000000000001 000000000000000a [ 7088.824045] <0> ffff880100000006 0000000000000202 0000000000000100 0000000000000048 [ 7088.824045] Call Trace: [ 7088.824045] <IRQ> [ 7088.824045] [<ffffffff81499007>] __xfs_free_perag+0x37/0x50 [ 7088.824045] [<ffffffff810e900a>] __rcu_process_callbacks+0x13a/0x3e0 [ 7088.824045] [<ffffffff810e92d8>] rcu_process_callbacks+0x28/0x50 [ 7088.824045] [<ffffffff8108848d>] __do_softirq+0xcd/0x290 [ 7088.824045] [<ffffffff810a8808>] ? hrtimer_interrupt+0x138/0x250 [ 7088.824045] [<ffffffff81037f5c>] call_softirq+0x1c/0x50 [ 7088.824045] [<ffffffff810398dd>] do_softirq+0x9d/0xd0 [ 7088.824045] [<ffffffff810881e5>] irq_exit+0x95/0xa0 [ 7088.824045] [<ffffffff81b06380>] smp_apic_timer_interrupt+0x70/0x9b [ 7088.824045] [<ffffffff81037a13>] apic_timer_interrupt+0x13/0x20 [ 7088.824045] <EOI> [ 7088.824045] [<ffffffff81060f6b>] ? native_safe_halt+0xb/0x10 [ 7088.824045] [<ffffffff810baded>] ? trace_hardirqs_on+0xd/0x10 [ 7088.824045] [<ffffffff8103fd70>] default_idle+0x50/0xb0 [ 7088.824045] [<ffffffff81035e28>] cpu_idle+0x78/0x100 [ 7088.824045] [<ffffffff81af627b>] start_secondary+0x1ac/0x1b1 [ 7088.824045] Code: 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f 44 00 00 31 c0 89 d1 48 89 f2 48 89 fe 48 c7 c7 08 38 df 81 e8 7b 34 64 00 <0f> 0b eb fe 66 66 66 66 2e [ 7088.824045] RIP [<ffffffff814b74cf>] assfail+0x1f/0x30 [ 7088.824045] RSP <ffff8800df003e50> [ 7088.863091] ---[ end trace ec76f8135c3adba9 ]---
Quoted 3 lines
I'm not seeing failures during xfstests runs, it seems that dbench may be the trigger. Is anyone else seeing reference counting problems like this on the current linus tree?
Ok, found the bug - it's in the reclaim scalability patchset that was merged into .37-rc1 - when the shrinker skips a locked AG it misseѕ a xfs_perag_put() call. I'll push out a patch soon.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs