xfstest 179 ASSERT

OSS sources with the xfs: fix buffer shudown reference count mismatch patch and xfstest 179.

xfstest 179 started to have various asserts starting with the "move the workers" series, but mostly the b_hold count is zero assert.

Now that the b_hold count is fixed, the asert is:

XFS: Assertion failed: atomic_read(&pag->pag_ref) == 0,
file: /root/xfs/fs/xfs/xfs_mount.c, line: 273

PID: 6741 TASK: ffff880268f46540 CPU: 0 COMMAND: "umount"
#0 [ffff88034bba1a70] machine_kexec at ffffffff8102a71d #1 [ffff88034bba1ae0] crash_kexec at ffffffff810a4703 #2 [ffff88034bba1bb0] oops_end at ffffffff81432098 #3 [ffff88034bba1be0] die at ffffffff81005763 #4 [ffff88034bba1c10] do_trap at ffffffff814319b3 #5 [ffff88034bba1c70] do_invalid_op at ffffffff81002f50 #6 [ffff88034bba1d10] invalid_op at ffffffff8143a09e

    [exception RIP: assfail+29]
    RIP: ffffffffa038e7dd  RSP: ffff88034bba1dc8  RFLAGS: 00010292
    RAX: 0000000000000065  RBX: 0000000000000000  RCX: 0000000000000de5
    RDX: 0000000000002013  RSI: 0000000000000092  RDI: 0000000000000246
    RBP: ffff88034bba1dc8   R8: 0000000000000000   R9: 0000000000000000
    R10: 0000000000000352  R11: 0000000000000351  R12: ffff88034d8af1f8
    R13: ffff88034d8af000  R14: ffff88034d8af1e8  R15: ffff88034fbf28c0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018

#7 [ffff88034bba1dd0] xfs_free_perag at ffffffffa03de5d8 [xfs] #8 [ffff88034bba1e10] xfs_unmountfs at ffffffffa03e08bc [xfs] #9 [ffff88034bba1e60] xfs_fs_put_super at ffffffffa0390e80 [xfs]
#10 [ffff88034bba1e80] generic_shutdown_super at ffffffff8114f37d #11 [ffff88034bba1eb0] kill_block_super at ffffffff8114f42b #12 [ffff88034bba1ed0] deactivate_locked_super at ffffffff8114f9e7 #13 [ffff88034bba1ef0] deactivate_super at ffffffff811502d9 #14 [ffff88034bba1f10] mntput_no_expire at ffffffff8116a821 #15 [ffff88034bba1f40] sys_umount at ffffffff8116bd21 #16 [ffff88034bba1f80] system_call_fastpath at ffffffff814390a9

    RIP: 00007f64a30cf5d7  RSP: 00007ffffe2f3798  RFLAGS: 00010202
    RAX: 00000000000000a6  RBX: ffffffff814390a9  RCX: 000000000000c0c8
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 00007f64a3c0e0f0
    RBP: 00007f64a3c0dff0   R8: 0000000000000000   R9: 00007f64a3a05435
    R10: 00007ffffe2f35c0  R11: 0000000000000246  R12: 00007f64a3c0e0f0
    R13: 00007f64a3c0e0d0  R14: 00007f64a3c0e0f0  R15: 0000000000000000
    ORIG_RAX: 00000000000000a6  CS: 0033  SS: 002b

mount perag information:

  m_perag_tree = {
    height = 0x1,
    gfp_mask = 0x20,
    rnode = 0xffff8803124bb4b1

},
crash> radix_tree_node ffff8803124bb4b0
struct radix_tree_node {
height = 0x1,
count = 0x3,

  {
    parent = 0x0,
    callback_head = {
      next = 0x0,
      func = 0xffffffff812384b0 <radix_tree_node_rcu_free>
    }

},
slots = {0x0, 0xffff88034fd6f540, 0xffff88034fd6f840,
0xffff88034fd6f240, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
tags = {{0x0}, {0x0}, {0x0}}
}

The inodes at 0xffff88034fd6f540, 0xffff88034fd6f840, 0xffff88034fd6f240 don't look valid.

I haven't spent much time on this.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

The only way you are going to track this down is through tracing perag gets and puts, and finding the object that is missing a put. There are already tracepoints for these and they tell you the caller function, so that should be sufficient to find what function has a get without a put...

The other possibility is that there are buffers that have not be freed properly (or leaked), but IIRC this failure pre-exists attaching the perag to buffers....

Those aren't inodes based on the addresses - they are all in the same page, so the size is at most 0x300 bytes (768 bytes). An XFS inode is larger than this, but a radix tree node is smaller (560 bytes and fits 7 to a page on my x86_64 machines), so I'm not sure what you have there. how big does /proc/slabinfo tell you a radix tree node is?

Cheers,

Dave.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

I have been spending too much time in earlier versions of XFS.

Last night, I had a good laugh at my mistake when I realized the perag was not in an array...duh me, this is the radix tree of the perag structures. I could not do a crash "kmem" command, that would have given me a clue.

I *think* Ben is also seeing this assert on his 32 bit test machine. Thanks for the advice, I will do some investigation this weekend.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs