list_add corruption after "xfs: mode di_mode to vfs inode"
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 11 lines · Joe Lawrence
Hi Dave,
Upon loading 4.6-rc1, I noticed a few linked list corruption messages in dmesg shortly after boot up. I bisected the kernel, landing on:
[
c19b3b05ae440de50fffe2ac2a9b27392a7448e9] xfs: modedi_modeto vfs inodeIf I revert
c19b3b05ae44from 4.6-rc1, the warnings stop.WARNING: CPU: 35 PID: 6715 at lib/list_debug.c:29 __list_add+0x65/0xc0
list_addcorruption.next->prevshould be prev (ffff882030928a00), but wasffff88103f00c300. (next=ffff88100fde5ce8).
.....
Quoted 14 lines
[<
ffffffff812488f0>] ?bdev_test+0x20/0x20[<ffffffff813551a5>] __list_add+0x65/0xc0 [<ffffffff81249bd8>] bd_acquire+0xc8/0xd0 [<ffffffff8124aa59>] blkdev_open+0x39/0x70 [<ffffffff8120bc27>] do_dentry_open+0x227/0x320 [<ffffffff8124aa20>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff8120d057>] vfs_open+0x57/0x60 [<ffffffff8121c9fa>] path_openat+0x1ba/0x1340 [<ffffffff8121eff1>] do_filp_open+0x91/0x100 [<ffffffff8122c806>] ? __alloc_fd+0x46/0x180 [<ffffffff8120d3b4>] do_sys_open+0x124/0x210 [<ffffffff8120d4be>] SyS_open+0x1e/0x20 [<ffffffff81003c12>] do_syscall_64+0x62/0x110 [<ffffffff8169ade1>] entry_SYSCALL64_slow_path+0x25/0x25
....
Quoted 2 lines
According to the
bd_acquire+0xc8offset, we're inbd_acquire()attempting the list add:
....
Quoted 5 lines
713 bdev =
bdget(inode->i_rdev);
714 if (bdev) {
715spin_lock(&bdev_lock);
716 if (!inode->i_bdev) {
717 /*
Quoted 9 lines
718 * We take an additional reference to bd_inode, 719 * and it's released in clear_inode() of inode. 720 * So, we can access it via ->i_mapping always 721 * without igrab(). 722 */ 723 bdgrab(bdev); 724 inode->i_bdev = bdev; 725 inode->i_mapping = bdev->bd_inode->i_mapping; 726 list_add(&inode->i_devices, &bdev->bd_inodes);
So the bdev->bd_inodes list is corrupt, and this call trace is just the messenger.
Quoted 53 lines
crash> ps -a | grep mdadm
...
PID: 6715 TASK:ffff882033ac2d40CPU: 35 COMMAND: "mdadm"
ARG:/sbin/mdadm--detail --export/var/opt/ft/osm/osm_temporary_md_device_node
...I traced the proprietary-driver-dependent user program to figure out what it was doing and boiled that down to a repro that hits the same corruption when running *stock* 4.6-rc1. (Note /tmp is hosted on an XFS volume):
--
MD=/dev/md1
LOOP_A=/dev/loop0
LOOP_B=/dev/loop1
TMP_A=/tmp/diska
TMP_B=/tmp/diskbecho
echo Setting up ...dd if=/dev/zero of=$TMP_A bs=1M count=200
dd if=/dev/zero of=$TMP_B bs=1M count=200
losetup $LOOP_A $TMP_A
losetup $LOOP_B $TMP_Bmdadm --create $MD \ --metadata=1 \ --level=1 \ --raid-devices=2 \ --bitmap=internal \ $LOOP_A $LOOP_BMAJOR=$(stat -c %t $MD)
MINOR=$(stat -c %T $MD)echo
echo Testing major: $MAJOR minor: $MINOR ...for i in {0..100}; do
mknod --mode=0600/tmp/tmp_nodeb $MAJOR $MINOR
mdadm --detail --export/tmp/tmp_node
rm -f/tmp/tmp_node
doneecho
echo Cleanup ...mdadm --stop $MD
losetup -d $LOOP_A $LOOP_B
rm -f $TMP_A $TMP_B
Quoted 8 lines
echo
echo Done.--
I'm not really sure why the bisect landed on
c19b3b05ae44"xfs: modedi_modeto vfs inode", but as I mentioned, reverting it made the list warnings go away.
Neither am I at this point as it's the bdev inode (not an xfs inode) that has a corrupted list. I'll have to try to reproduce this.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 48 lines · Dave Chinner
Quoted 11 lines · Joe Lawrence
Hi Dave,
Upon loading 4.6-rc1, I noticed a few linked list corruption messages in dmesg shortly after boot up. I bisected the kernel, landing on:
[
c19b3b05ae440de50fffe2ac2a9b27392a7448e9] xfs: modedi_modeto vfs inodeIf I revert
c19b3b05ae44from 4.6-rc1, the warnings stop.WARNING: CPU: 35 PID: 6715 at lib/list_debug.c:29 __list_add+0x65/0xc0
list_addcorruption.next->prevshould be prev (ffff882030928a00), but wasffff88103f00c300. (next=ffff88100fde5ce8)......
Quoted 14 lines
[<
ffffffff812488f0>] ?bdev_test+0x20/0x20[<ffffffff813551a5>] __list_add+0x65/0xc0 [<ffffffff81249bd8>] bd_acquire+0xc8/0xd0 [<ffffffff8124aa59>] blkdev_open+0x39/0x70 [<ffffffff8120bc27>] do_dentry_open+0x227/0x320 [<ffffffff8124aa20>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff8120d057>] vfs_open+0x57/0x60 [<ffffffff8121c9fa>] path_openat+0x1ba/0x1340 [<ffffffff8121eff1>] do_filp_open+0x91/0x100 [<ffffffff8122c806>] ? __alloc_fd+0x46/0x180 [<ffffffff8120d3b4>] do_sys_open+0x124/0x210 [<ffffffff8120d4be>] SyS_open+0x1e/0x20 [<ffffffff81003c12>] do_syscall_64+0x62/0x110 [<ffffffff8169ade1>] entry_SYSCALL64_slow_path+0x25/0x25....
Quoted 2 lines
According to the
bd_acquire+0xc8offset, we're inbd_acquire()attempting the list add:....
Quoted 5 lines
713 bdev =
bdget(inode->i_rdev);
714 if (bdev) {
715spin_lock(&bdev_lock);
716 if (!inode->i_bdev) {
717 /*Quoted 9 lines
718 * We take an additional reference to bd_inode, 719 * and it's released in clear_inode() of inode. 720 * So, we can access it via ->i_mapping always 721 * without igrab(). 722 */ 723 bdgrab(bdev); 724 inode->i_bdev = bdev; 725 inode->i_mapping = bdev->bd_inode->i_mapping; 726 list_add(&inode->i_devices, &bdev->bd_inodes);So the
bdev->bd_inodeslist is corrupt, and this call trace is just the messenger.
....
Quoted 6 lines
Quoted 3 lines
I'm not really sure why the bisect landed on
c19b3b05ae44"xfs: modedi_modeto vfs inode", but as I mentioned, reverting it made the list warnings go away.Neither am I at this point as it's the bdev inode (not an xfs inode) that has a corrupted list. I'll have to try to reproduce this.
Patch below should fix the problem. Smoke tested only at this point.
Cheers,
Dave.
Signature · 49 lines
Dave Chinner [email protected] xfs: we don't need no steekin ->evict_inode From: Dave Chinner <[email protected]> Joe Lawrence reported a list_add corruption with 4.6-rc1 when testing some custom md administration code that made it's own block device nodes for the md array. The simple test loop of: for i in {0..100}; do mknod --mode=0600 $tmp/tmp_node b $MAJOR $MINOR mdadm --detail --export $tmp/tmp_node > /dev/null rm -f $tmp/tmp_node done Would produce this warning in bd_acquire() when mdadm opened the device node: list_add double add: new=ffff88043831c7b8, prev=ffff8804380287d8, next=ffff88043831c7b8. And then produce this from bd_forget from kdevtmpfs evicting a block dev inode: list_del corruption. prev->next should be ffff8800bb83eb10, but was ffff88043831c7b8 This is a regression caused by commit c19b3b05 ("xfs: mode di_mode to vfs inode"). The issue is that xfs_inactive() frees the unlinked inode, and the above commit meant that this freeing zeroed the mode in the struct inode. The problem is that after evict() has called ->evict_inode, it expects the i_mode to be intact so that it can call bd_forget() or cd_forget() to drop the reference to the block device inode attached to the XFS inode. In reality, the only thing we do in xfs_fs_evict_inode() that is not generic is call xfs_inactive(). We can move the xfs_inactive() call to xfs_fs_destroy_inode() without any problems at all, and this will leave the VFS inode intact until it is completely done with it. So, remove xfs_fs_evict_inode(), and do the work it used to do in ->destroy_inode instead. Reported-by: Joe Lawrence <[email protected]> Signed-off-by: Dave Chinner <[email protected]> --- fs/xfs/xfs_super.c | 28 +++++++--------------------- 1 file changed, 7 insertions(+), 21 deletions(-)
fs/xfs/xfs_super.c +7 -21
@@ -928,7 +928,7 @@ xfs_fs_alloc_inode(/* * Now that the generic code is guaranteed not to be accessing -* the linux inode, we can reclaim the inode. +* the linux inode, we can inactivate and reclaim the inode. */ STATIC void xfs_fs_destroy_inode(
@@ -938,9 +938,14 @@ xfs_fs_destroy_inode(trace_xfs_destroy_inode (ip); -XFS_STATS_INC (ip->i_mount, vn_reclaim); +ASSERT (!rwsem_is_locked (&ip->i_iolock.mr_lock)); +XFS_STATS_INC (ip->i_mount, vn_rele); +XFS_STATS_INC (ip->i_mount, vn_remove); + +xfs_inactive (ip);ASSERT (XFS_FORCED_SHUTDOWN (ip->i_mount) || ip->i_delayed_blks ==0 ); +XFS_STATS_INC (ip->i_mount, vn_reclaim);/* * We should never get here with one of the reclaim flags already set.
@@ -987,24 +992,6 @@ xfs_fs_inode_init_once( "xfsino", ip->i_ino); } -STATIC void -xfs_fs_evict_inode ( -struct inode *inode) -{ -xfs_inode_t *ip =XFS_I (inode); - -ASSERT (!rwsem_is_locked (&ip->i_iolock.mr_lock)); - -trace_xfs_evict_inode (ip); - -truncate_inode_pages_final (&inode->i_data); -clear_inode (inode); -XFS_STATS_INC (ip->i_mount, vn_rele); -XFS_STATS_INC (ip->i_mount, vn_remove); - -xfs_inactive (ip); -} -/* * We do an unlocked check for XFS_IDONTCACHE here because we are already * serialised against cache hits here via the inode->i_lock and igrab() in
@@ -1673,7 +1660,6 @@ xfs_fs_free_cached_objects(static const struct super_operations xfs_super_operations = { .alloc_inode = xfs_fs_alloc_inode, .destroy_inode = xfs_fs_destroy_inode, - .evict_inode = xfs_fs_evict_inode, .drop_inode = xfs_fs_drop_inode, .put_super = xfs_fs_put_super, .sync_fs = xfs_fs_sync_fs,
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 50 lines · Dave Chinner
Quoted 48 lines · Dave Chinner
Quoted 11 lines · Joe Lawrence
Hi Dave,
Upon loading 4.6-rc1, I noticed a few linked list corruption messages in dmesg shortly after boot up. I bisected the kernel, landing on:
[
c19b3b05ae440de50fffe2ac2a9b27392a7448e9] xfs: modedi_modeto vfs inodeIf I revert
c19b3b05ae44from 4.6-rc1, the warnings stop.WARNING: CPU: 35 PID: 6715 at lib/list_debug.c:29 __list_add+0x65/0xc0
list_addcorruption.next->prevshould be prev (ffff882030928a00), but wasffff88103f00c300. (next=ffff88100fde5ce8)......
Quoted 14 lines
[<
ffffffff812488f0>] ?bdev_test+0x20/0x20[<ffffffff813551a5>] __list_add+0x65/0xc0 [<ffffffff81249bd8>] bd_acquire+0xc8/0xd0 [<ffffffff8124aa59>] blkdev_open+0x39/0x70 [<ffffffff8120bc27>] do_dentry_open+0x227/0x320 [<ffffffff8124aa20>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff8120d057>] vfs_open+0x57/0x60 [<ffffffff8121c9fa>] path_openat+0x1ba/0x1340 [<ffffffff8121eff1>] do_filp_open+0x91/0x100 [<ffffffff8122c806>] ? __alloc_fd+0x46/0x180 [<ffffffff8120d3b4>] do_sys_open+0x124/0x210 [<ffffffff8120d4be>] SyS_open+0x1e/0x20 [<ffffffff81003c12>] do_syscall_64+0x62/0x110 [<ffffffff8169ade1>] entry_SYSCALL64_slow_path+0x25/0x25....
Quoted 2 lines
According to the
bd_acquire+0xc8offset, we're inbd_acquire()attempting the list add:....
Quoted 5 lines
713 bdev =
bdget(inode->i_rdev);
714 if (bdev) {
715spin_lock(&bdev_lock);
716 if (!inode->i_bdev) {
717 /*Quoted 9 lines
718 * We take an additional reference to bd_inode, 719 * and it's released in clear_inode() of inode. 720 * So, we can access it via ->i_mapping always 721 * without igrab(). 722 */ 723 bdgrab(bdev); 724 inode->i_bdev = bdev; 725 inode->i_mapping = bdev->bd_inode->i_mapping; 726 list_add(&inode->i_devices, &bdev->bd_inodes);So the
bdev->bd_inodeslist is corrupt, and this call trace is just the messenger.....
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
I'm not really sure why the bisect landed on
c19b3b05ae44"xfs: modedi_modeto vfs inode", but as I mentioned, reverting it made the list warnings go away.Neither am I at this point as it's the bdev inode (not an xfs inode) that has a corrupted list. I'll have to try to reproduce this.
Patch below should fix the problem. Smoke tested only at this point.
Thanks Dave, this looks good on both the QEMU and on the originating hardware instances.
Let me know if there are any additional tests that I can run, otherwise consider this Tested-by.
BTW, cuda.sgi.com is rejecting my mail to the list (even though I subscribed), so apologies for this not making it out to [email protected].
Thanks again,
-- Joe
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Joe,
I'll forward your email to our IT group to update the spam filters. Sorry for the inconvenience.
-Troy
________________________________________
From: [email protected] [[email protected]] on behalf of Joe Lawrence [[email protected]]
Sent: Wednesday, April 13, 2016 12:16 PM
To: [email protected]; [email protected]
Subject: Re: list_add corruption after "xfs: mode di_mode to vfs inode"
Quoted 50 lines · Dave Chinner
Quoted 48 lines · Dave Chinner
Quoted 11 lines · Joe Lawrence
Hi Dave,
Upon loading 4.6-rc1, I noticed a few linked list corruption messages in dmesg shortly after boot up. I bisected the kernel, landing on:
[
c19b3b05ae440de50fffe2ac2a9b27392a7448e9] xfs: modedi_modeto vfs inodeIf I revert
c19b3b05ae44from 4.6-rc1, the warnings stop.WARNING: CPU: 35 PID: 6715 at lib/list_debug.c:29 __list_add+0x65/0xc0
list_addcorruption.next->prevshould be prev (ffff882030928a00), but wasffff88103f00c300. (next=ffff88100fde5ce8)......
Quoted 14 lines
[<
ffffffff812488f0>] ?bdev_test+0x20/0x20[<ffffffff813551a5>] __list_add+0x65/0xc0 [<ffffffff81249bd8>] bd_acquire+0xc8/0xd0 [<ffffffff8124aa59>] blkdev_open+0x39/0x70 [<ffffffff8120bc27>] do_dentry_open+0x227/0x320 [<ffffffff8124aa20>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff8120d057>] vfs_open+0x57/0x60 [<ffffffff8121c9fa>] path_openat+0x1ba/0x1340 [<ffffffff8121eff1>] do_filp_open+0x91/0x100 [<ffffffff8122c806>] ? __alloc_fd+0x46/0x180 [<ffffffff8120d3b4>] do_sys_open+0x124/0x210 [<ffffffff8120d4be>] SyS_open+0x1e/0x20 [<ffffffff81003c12>] do_syscall_64+0x62/0x110 [<ffffffff8169ade1>] entry_SYSCALL64_slow_path+0x25/0x25....
Quoted 2 lines
According to the
bd_acquire+0xc8offset, we're inbd_acquire()attempting the list add:....
Quoted 5 lines
713 bdev =
bdget(inode->i_rdev);
714 if (bdev) {
715spin_lock(&bdev_lock);
716 if (!inode->i_bdev) {
717 /*Quoted 9 lines
718 * We take an additional reference to bd_inode, 719 * and it's released in clear_inode() of inode. 720 * So, we can access it via ->i_mapping always 721 * without igrab(). 722 */ 723 bdgrab(bdev); 724 inode->i_bdev = bdev; 725 inode->i_mapping = bdev->bd_inode->i_mapping; 726 list_add(&inode->i_devices, &bdev->bd_inodes);So the
bdev->bd_inodeslist is corrupt, and this call trace is just the messenger.....
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
I'm not really sure why the bisect landed on
c19b3b05ae44"xfs: modedi_modeto vfs inode", but as I mentioned, reverting it made the list warnings go away.Neither am I at this point as it's the bdev inode (not an xfs inode) that has a corrupted list. I'll have to try to reproduce this.
Patch below should fix the problem. Smoke tested only at this point.
Thanks Dave, this looks good on both the QEMU and on the originating hardware instances.
Let me know if there are any additional tests that I can run, otherwise consider this Tested-by.
BTW, cuda.sgi.com is rejecting my mail to the list (even though I subscribed), so apologies for this not making it out to [email protected].
Thanks again,
-- Joe
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs