btrfs regression since 4.X kernel NULL pointer dereference
Subsystems (2)
Hello,
today i experienced the following btrfs bug:
Aug 20 11:59:18 debian-build kernel: [ 325.170036] BUG: unable to handle kernel NULL pointer dereference at 0000000000000330 Aug 20 11:59:18 debian-build kernel: [ 325.170144] IP: [<ffffffff813204c0>] blk_get_backing_dev_info+0x10/0x20 Aug 20 11:59:18 debian-build kernel: [ 325.170216] PGD 74f57067 PUD 74f51067 PMD 0 Aug 20 11:59:18 debian-build kernel: [ 325.170282] Oops: 0000 [#1] SMP Aug 20 11:59:18 debian-build kernel: [ 325.170330] Modules linked in: dm_mod netconsole xt_multiport iptable_filter ip_tables x_tab les cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative ext2 loop shpchp i2c_piix4 i2c_core virtio_balloon acpi_c pufreq button btrfs xor lzo_compress usbhid raid6_pq ata_generic sg sd_mod virtio_net virtio_scsi floppy uhci_hcd ehci_hcd ata_piix usbcore usb_common virtio_pci Aug 20 11:59:18 debian-build kernel: [ 325.170783] CPU: 4 PID: 13323 Comm: btrfs Not tainted 4.1.6+17-ph #1 Aug 20 11:59:18 debian-build kernel: [ 325.170842] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.1-0-g4adadbd-20150316_085822-nilsson.home.kraxel.org 04/01/2014 Aug 20 11:59:18 debian-build kernel: [ 325.170952] task: ffff88022d6bbae0 ti: ffff8800748e0000 task.ti: ffff8800748e0000 Aug 20 11:59:18 debian-build kernel: [ 325.171017] RIP: 0010:[<ffffffff813204c0>] [<ffffffff813204c0>] blk_get_backing_dev_info+0x10/0x20 Aug 20 11:59:18 debian-build kernel: [ 325.171096] RSP: 0018:ffff8800748e39a8 EFLAGS: 00010202 Aug 20 11:59:18 debian-build kernel: [ 325.171148] RAX: 0000000000000000 RBX: ffff880234680770 RCX: 0000000000000001 Aug 20 11:59:18 debian-build kernel: [ 325.171210] RDX: 7fffffffffffffff RSI: 0000000000000000 RDI: ffff880234680680 Aug 20 11:59:18 debian-build kernel: [ 325.171271] RBP: ffff8800748e39a8 R08: 7fffffffffffffff R09: 0000000000000246 Aug 20 11:59:18 debian-build kernel: [ 325.171333] R10: ffffffffa0158bdc R11: 0000000000000000 R12: ffff880237019000 Aug 20 11:59:18 debian-build kernel: [ 325.171393] R13: 7fffffffffffffff R14: ffff880092df07fc R15: 7fffffffffffffff Aug 20 11:59:18 debian-build kernel: [ 325.171455] FS: 00007fb05f0ba880(0000) GS:ffff88023fd00000(0000) knlGS:0000000000000000 Aug 20 11:59:18 debian-build kernel: [ 325.171522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 20 11:59:18 debian-build kernel: [ 325.171577] CR2: 0000000000000330 CR3: 0000000074ce4000 CR4: 00000000000006e0 Aug 20 11:59:18 debian-build kernel: [ 325.171669] Stack: Aug 20 11:59:18 debian-build kernel: [ 325.171706] ffff8800748e39c8 ffffffff811e6d60 ffff8802346808c0 0000000000000000 Aug 20 11:59:18 debian-build kernel: [ 325.171811] ffff8800748e3a18 ffffffff8114e232 ffff880212f93910 7fffffffffffffff Aug 20 11:59:18 debian-build kernel: [ 325.171923] 0000000000000000 0000000000000000 7fffffffffffffff 0000000000000001 Aug 20 11:59:18 debian-build kernel: [ 325.172078] Call Trace: Aug 20 11:59:18 debian-build kernel: [ 325.172132] [<ffffffff811e6d60>] inode_to_bdi+0x60/0x70 Aug 20 11:59:18 debian-build kernel: [ 325.172221] [<ffffffff8114e232>] __filemap_fdatawrite_range+0x42/0x70 Aug 20 11:59:18 debian-build kernel: [ 325.172319] [<ffffffff8114eea3>] filemap_fdatawrite_range+0x13/0x20 Aug 20 11:59:18 debian-build kernel: [ 325.172418] [<ffffffffa0157c2b>] btrfs_fdatawrite_range+0x2b/0x70 [btrfs] Aug 20 11:59:18 debian-build kernel: [ 325.172493] [<ffffffffa015d57c>] btrfs_wait_ordered_range+0x4c/0x130 [btrfs] Aug 20 11:59:18 debian-build kernel: [ 325.174258] [<ffffffffa0155075>] ? btrfs_drop_extent_cache+0x355/0x420 [btrfs] Aug 20 11:59:18 debian-build kernel: [ 325.175688] [<ffffffffa014dde6>] btrfs_evict_inode+0x226/0x550 [btrfs] Aug 20 11:59:18 debian-build kernel: [ 325.177252] [<ffffffff811e726d>] ? __inode_wait_for_writeback+0x6d/0xc0 Aug 20 11:59:18 debian-build kernel: [ 325.179214] [<ffffffff811d9058>] evict+0xb8/0x190 Aug 20 11:59:18 debian-build kernel: [ 325.180619] [<ffffffff811d986b>] iput+0x18b/0x1f0 Aug 20 11:59:18 debian-build kernel: [ 325.182034] [<ffffffff811d4f28>] __dentry_kill+0x198/0x200 Aug 20 11:59:18 debian-build kernel: [ 325.183559] [<ffffffff811d50ad>] shrink_dentry_list+0x11d/0x2b0 Aug 20 11:59:18 debian-build kernel: [ 325.184981] [<ffffffff811d56c8>] d_invalidate+0xd8/0x100 Aug 20 11:59:18 debian-build kernel: [ 325.186394] [<ffffffffa017757b>] btrfs_ioctl_snap_destroy+0x50b/0x6e0 [btrfs] Aug 20 11:59:18 debian-build kernel: [ 325.187832] [<ffffffffa017abca>] btrfs_ioctl+0x131a/0x2a30 [btrfs] Aug 20 11:59:18 debian-build kernel: [ 325.189239] [<ffffffff8115ab2b>] ? lru_cache_add_active_or_unevictable+0x2b/0xa0 Aug 20 11:59:18 debian-build kernel: [ 325.190668] [<ffffffff8117970a>] ? handle_mm_fault+0x2ba/0x1860 Aug 20 11:59:18 debian-build kernel: [ 325.192062] [<ffffffff81181566>] ? mmap_region+0x316/0x630 Aug 20 11:59:18 debian-build kernel: [ 325.193453] [<ffffffff81116ecc>] ? acct_account_cputime+0x1c/0x20 Aug 20 11:59:18 debian-build kernel: [ 325.194851] [<ffffffff810ae3f9>] ? account_user_time+0x99/0xb0 Aug 20 11:59:18 debian-build kernel: [ 325.196241] [<ffffffff811d0bd3>] do_vfs_ioctl+0x83/0x550 Aug 20 11:59:18 debian-build kernel: [ 325.197584] [<ffffffff8114be23>] ? context_tracking_user_exit+0x13/0x20 Aug 20 11:59:18 debian-build kernel: [ 325.198913] [<ffffffff81012558>] ? syscall_trace_enter_phase1+0xf8/0x160 Aug 20 11:59:18 debian-build kernel: [ 325.200229] [<ffffffff811d10ec>] SyS_ioctl+0x4c/0x90 Aug 20 11:59:18 debian-build kernel: [ 325.201548] [<ffffffff8163442e>] system_call_fastpath+0x12/0x71 Aug 20 11:59:18 debian-build kernel: [ 325.202836] Code: e9 23 ff ff ff b8 01 00 00 00 45 31 e4 eb d5 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 48 8b 87 98 00 00 00 55 48 89 e5 <48> 8b 80 30 03 00 00 5d 48 05 98 01 00 00 c3 90 0f 1f 44 00 00 Aug 20 11:59:18 debian-build kernel: [ 325.205694] RIP [<ffffffff813204c0>] blk_get_backing_dev_info+0x10/0x20 Aug 20 11:59:18 debian-build kernel: [ 325.206983] RSP <ffff8800748e39a8> Aug 20 11:59:18 debian-build kernel: [ 325.208233] CR2: 0000000000000330 Aug 20 11:59:18 debian-build kernel: [ 325.209467] ---[ end trace 9dd28134a31aacc4 ]---
It was introduced by:
| commit de1414a654e66b81b5348dbc5259ecf2fb61655e | Author: Christoph Hellwig <[email protected]> | Date: Wed Jan 14 10:42:36 2015 +0100 | | fs: export inode_to_bdi and use it in favor of mapping->backing_dev_info
More details and a reproducer from a 3rd person can be found here:
https://bugzilla.kernel.org/show_bug.cgi?id=100911
Greets,
Stefan
I think this is btrfs using a struct block_device that doesn't have a valid queue pointer in it's gendisk for ->s_bdev. And there are some fishy looking ->s_bdev assignments in the code which I suspect are related to it:
fs/btrfs/dev-replace.c: if (fs_info->sb->s_bdev == src_device->bdev) fs/btrfs/dev-replace.c: fs_info->sb->s_bdev = tgt_device->bdev; fs/btrfs/volumes.c: if (device->bdev == root->fs_info->sb->s_bdev) fs/btrfs/volumes.c: root->fs_info->sb->s_bdev = next_device->bdev; fs/btrfs/volumes.c: if (tgtdev->bdev == fs_info->sb->s_bdev) fs/btrfs/volumes.c: fs_info->sb->s_bdev = next_device->bdev;
Am 25.08.2015 um 11:00 schrieb Christoph Hellwig:
Quoted 5 lines
I think this is btrfs using a struct
block_devicethat doesn't have a valid queue pointer in it's gendisk for ->s_bdev. And there are some fishy looking ->s_bdev assignments in the code which I suspect are related to it:
Quoted 7 lines
fs/btrfs/dev-replace.c: if (fs_info->sb->s_bdev == src_device->bdev) fs/btrfs/dev-replace.c: fs_info->sb->s_bdev = tgt_device->bdev; fs/btrfs/volumes.c: if (device->bdev == root->fs_info->sb->s_bdev) fs/btrfs/volumes.c: root->fs_info->sb->s_bdev = next_device->bdev; fs/btrfs/volumes.c: if (tgtdev->bdev == fs_info->sb->s_bdev) fs/btrfs/volumes.c: fs_info->sb->s_bdev = next_device->bdev;
Would be very nice if anyone from btrfs can pick this up.
Stefan
Quoted 11 lines · Christoph Hellwig
I think this is btrfs using a struct
block_devicethat doesn't have a valid queue pointer in it's gendisk for ->s_bdev. And there are some fishy looking ->s_bdev assignments in the code which I suspect are related to it:fs/btrfs/dev-replace.c: if (fs_info->sb->s_bdev == src_device->bdev) fs/btrfs/dev-replace.c: fs_info->sb->s_bdev = tgt_device->bdev; fs/btrfs/volumes.c: if (device->bdev == root->fs_info->sb->s_bdev) fs/btrfs/volumes.c: root->fs_info->sb->s_bdev = next_device->bdev; fs/btrfs/volumes.c: if (tgtdev->bdev == fs_info->sb->s_bdev) fs/btrfs/volumes.c: fs_info->sb->s_bdev = next_device->bdev;
We've had trouble with this in the past, I'll take a look.
-chris
Quoted 8 lines
Am 25.08.2015 um 15:51 schrieb Chris Mason <[email protected]>:
Quoted 6 lines · Christoph Hellwig
I think this is btrfs using a struct
block_devicethat doesn't have a valid queue pointer in it's gendisk for ->s_bdev. And there are some fishy looking ->s_bdev assignments in the code which I suspect are related to it:
Quoted 8 lines
Quoted 6 lines
fs/btrfs/dev-replace.c: if (fs_info->sb->s_bdev == src_device->bdev) fs/btrfs/dev-replace.c: fs_info->sb->s_bdev = tgt_device->bdev; fs/btrfs/volumes.c: if (device->bdev == root->fs_info->sb->s_bdev) fs/btrfs/volumes.c: root->fs_info->sb->s_bdev = next_device->bdev; fs/btrfs/volumes.c: if (tgtdev->bdev == fs_info->sb->s_bdev) fs/btrfs/volumes.c: fs_info->sb->s_bdev = next_device->bdev;We've had trouble with this in the past, I'll take a look.
Any news?
Quoted 3 lines
-chris
Quoted 15 lines · Stefan Priebe - Profihost AG
Quoted 15 lines
Am 25.08.2015 um 15:51 schrieb Chris Mason <[email protected]>:
Quoted 12 lines · Christoph Hellwig
I think this is btrfs using a struct
block_devicethat doesn't have a valid queue pointer in it's gendisk for ->s_bdev. And there are some fishy looking ->s_bdev assignments in the code which I suspect are related to it:fs/btrfs/dev-replace.c: if (fs_info->sb->s_bdev == src_device->bdev) fs/btrfs/dev-replace.c: fs_info->sb->s_bdev = tgt_device->bdev; fs/btrfs/volumes.c: if (device->bdev == root->fs_info->sb->s_bdev) fs/btrfs/volumes.c: root->fs_info->sb->s_bdev = next_device->bdev; fs/btrfs/volumes.c: if (tgtdev->bdev == fs_info->sb->s_bdev) fs/btrfs/volumes.c: fs_info->sb->s_bdev = next_device->bdev;
Quoted 3 lines
Quoted 1 line
We've had trouble with this in the past, I'll take a look.
Any news?
Haven't been able to reproduce yet, I'll try again in the morning.
-chris
Am 01.09.2015 um 02:06 schrieb Chris Mason:
Quoted 16 lines · Stefan Priebe - Profihost AG
Quoted 15 lines
Quoted 15 lines
Am 25.08.2015 um 15:51 schrieb Chris Mason <[email protected]>:
Quoted 12 lines · Christoph Hellwig
I think this is btrfs using a struct
block_devicethat doesn't have a valid queue pointer in it's gendisk for ->s_bdev. And there are some fishy looking ->s_bdev assignments in the code which I suspect are related to it:fs/btrfs/dev-replace.c: if (fs_info->sb->s_bdev == src_device->bdev) fs/btrfs/dev-replace.c: fs_info->sb->s_bdev = tgt_device->bdev; fs/btrfs/volumes.c: if (device->bdev == root->fs_info->sb->s_bdev) fs/btrfs/volumes.c: root->fs_info->sb->s_bdev = next_device->bdev; fs/btrfs/volumes.c: if (tgtdev->bdev == fs_info->sb->s_bdev) fs/btrfs/volumes.c: fs_info->sb->s_bdev = next_device->bdev;
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
We've had trouble with this in the past, I'll take a look.
Any news?
Haven't been able to reproduce yet, I'll try again in the morning.
Thanks. We're using schroot like the user in this bugreport:
https://bugzilla.kernel.org/show_bug.cgi?id=100911
But he also claims he found another way to reproduce using vfcgbackup (last comment).
Stefan
Quoted 3 lines
-chris
Stefan Priebe wrote...
Quoted 5 lines
Thanks. We're using schroot like the user in this bugreport:
https://bugzilla.kernel.org/show_bug.cgi?id=100911But he also claims he found another way to reproduce using vfcgbackup (last comment).
FWIW, this was still the same thing, just stripped down to get closer to the actual cause. It began with building a private Debian package that has lvm2 as build dependency. Daemons are disabled in the build chroot (via policy-rc.d) but the vfcgbackup invocation in lvm2's postinst is still run.
Also, a quick test of Jeff's update to the Bugzilla ticket looks very good. More tests will follow.
Christoph
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 19 lines · Chris Mason
Quoted 17 lines · Stefan Priebe - Profihost AG
Quoted 17 lines
Am 25.08.2015 um 15:51 schrieb Chris Mason <[email protected]>:
Quoted 14 lines
On Tue, Aug 25, 2015 at 11:00:30AM +0200, Christoph Hellwig
wrote: I think this is btrfs using a structblock_devicethat doesn't have a valid queue pointer in it's gendisk for ->s_bdev. And there are some fishy looking ->s_bdev assignments in the code which I suspect are related to it:
fs/btrfs/dev-replace.c: if (fs_info->sb->s_bdev==src_device->bdev)fs/btrfs/dev-replace.c:fs_info->sb->s_bdev=tgt_device->bdev;fs/btrfs/volumes.c: if (device->bdev==root->fs_info->sb->s_bdev)fs/btrfs/volumes.c:root->fs_info->sb->s_bdev=next_device->bdev;fs/btrfs/volumes.c: if (tgtdev->bdev==fs_info->sb->s_bdev)fs/btrfs/volumes.c:fs_info->sb->s_bdev=next_device->bdev;
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
We've had trouble with this in the past, I'll take a look.
Any news?
Haven't been able to reproduce yet, I'll try again in the morning.
I'm unable to reproduce it as well and I'm hearing about people hitting it in different forums.
Can you describe your storage configuration? There's a gendisk without a queue sneaking in somehow. There are some spots in dm where the mddev gets a new queue assigned to it but the old queue associated with the gendisk is left behind so that's not it. If you're able to reproduce it reliably with this configuration, that's certainly interesting. I'm wondering if the case is that vgcfgbackup is iterating over devices and causing something that might not have been previously scanned to be scanned and added to the btrfs device list. Are there any messages above the oops indicating when a device is added? Could you post your complete dmesg somewhere?
Thanks,
- -Jeff
- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.19 (Darwin)
iQIcBAEBAgAGBQJV8gJ2AAoJEB57S2MheeWyEP4QALvIXZGIqkXhUBZEYvLC+rnD Q79chLpbt8x52xtFoav78i3nG24SMXV5NbhFlGNnr6xt/CQtKSzBuDyJ+nigkyy+ bTtHbwkMqNddrqKJ3r928nKwz1IOyU4J58l7hGyC0owSA3mDxyI7+yS16OXFwc1S 6lxtV4ljf6nqobA+rVyC3n3BU6CF1aL45V2FpM/m4OHL/Xqv97Xg2tGlysnk8RPD dRvz5fs2T75B0eBsK4xWHKEgKKWdVYaKqlySRXUJQpnNJTlJltW5HzL8RnR47brt LLaOCD4jcLlEtGhTi6wEN6BlEYbgDx/1yjQs5zxLG0vbTUi5ZcTeT82d5+Hh4WMD HyLfE4pwFUZXTHvcIoQ51wPLS/tWMVJXnleQKBrDT+WcsKPr5yuwUZpwfrOBfjol Qan13/mIcmvFsz2yVWnoZJWU5osnhe+frBsVFlIFLNXp50QVXk+WvruPDOsoY2eQ J8L20LhY5e0dgJseXWkAAy68mJ0/rrkX0iAdiED/WEsBdjy7QuOomJqPyQQEPplm d+tIkDR77Biajket2tNEXK1m14f7QvFUFojI0NwegcmZT3iJ7BBWD6wjxIvh3KqE LqHIcTaECU9XSg9x9E3N416hCbclC9xeosxaJB1iN+EvpaD7CBA+V+onkNMN6Fu4 J0aVUYlZuLcqkEs/BuNL
=BRtR
-----END PGP SIGNATURE-----
Am 11.09.2015 um 00:21 schrieb Jeff Mahoney:
Quoted 29 lines
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 19 lines · Chris Mason
Quoted 17 lines · Stefan Priebe - Profihost AG
Quoted 17 lines
Am 25.08.2015 um 15:51 schrieb Chris Mason <[email protected]>:
Quoted 14 lines
On Tue, Aug 25, 2015 at 11:00:30AM +0200, Christoph Hellwig
wrote: I think this is btrfs using a structblock_devicethat doesn't have a valid queue pointer in it's gendisk for ->s_bdev. And there are some fishy looking ->s_bdev assignments in the code which I suspect are related to it:
fs/btrfs/dev-replace.c: if (fs_info->sb->s_bdev==src_device->bdev)fs/btrfs/dev-replace.c:fs_info->sb->s_bdev=tgt_device->bdev;fs/btrfs/volumes.c: if (device->bdev==root->fs_info->sb->s_bdev)fs/btrfs/volumes.c:root->fs_info->sb->s_bdev=next_device->bdev;fs/btrfs/volumes.c: if (tgtdev->bdev==fs_info->sb->s_bdev)fs/btrfs/volumes.c:fs_info->sb->s_bdev=next_device->bdev;Quoted 5 lines
Quoted 3 lines
Quoted 1 line
We've had trouble with this in the past, I'll take a look.
Any news?
Haven't been able to reproduce yet, I'll try again in the morning.
Quoted 7 lines
I'm unable to reproduce it as well and I'm hearing about people hitting it in different forums.
Can you describe your storage configuration? There's a gendisk
without a queue sneaking in somehow. There are some spots in dm where the mddev gets a new queue assigned to it but the old queue associated with the gendisk is left behind so that's not it.
I'm hitting this inside a debian buil VM using schroot. So the disk is a simple plain qemu emulated scsi disk (virtio-pci-scsi).
Quoted 5 lines
If you're able to
reproduce it reliably with this configuration, that's certainly interesting. I'm wondering if the case is that vgcfgbackup is iterating over devices and causing something that might not have been previously scanned to be scanned and added to the btrfs device list.
I'm not the guy from the kernel bug report - so i'm only using debian schroot not vgcfgbackup.
I can trigger it 100% by using schroot with sbuild using debian or ubuntu.
Quoted 2 lines
Are there any messages above the oops indicating when a device is added? Could you post your complete dmesg somewhere?
Sure:
http://pastebin.com/raw.php?i=bG6bZUPA
Thanks!
Greets,
Stefan
Quoted 25 lines
Thanks,
- -Jeff
- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.19 (Darwin)iQIcBAEBAgAGBQJV8gJ2AAoJEB57S2MheeWyEP4QALvIXZGIqkXhUBZEYvLC+rnD Q79chLpbt8x52xtFoav78i3nG24SMXV5NbhFlGNnr6xt/CQtKSzBuDyJ+nigkyy+ bTtHbwkMqNddrqKJ3r928nKwz1IOyU4J58l7hGyC0owSA3mDxyI7+yS16OXFwc1S 6lxtV4ljf6nqobA+rVyC3n3BU6CF1aL45V2FpM/m4OHL/Xqv97Xg2tGlysnk8RPD dRvz5fs2T75B0eBsK4xWHKEgKKWdVYaKqlySRXUJQpnNJTlJltW5HzL8RnR47brt LLaOCD4jcLlEtGhTi6wEN6BlEYbgDx/1yjQs5zxLG0vbTUi5ZcTeT82d5+Hh4WMD HyLfE4pwFUZXTHvcIoQ51wPLS/tWMVJXnleQKBrDT+WcsKPr5yuwUZpwfrOBfjol Qan13/mIcmvFsz2yVWnoZJWU5osnhe+frBsVFlIFLNXp50QVXk+WvruPDOsoY2eQ J8L20LhY5e0dgJseXWkAAy68mJ0/rrkX0iAdiED/WEsBdjy7QuOomJqPyQQEPplm d+tIkDR77Biajket2tNEXK1m14f7QvFUFojI0NwegcmZT3iJ7BBWD6wjxIvh3KqE LqHIcTaECU9XSg9x9E3N416hCbclC9xeosxaJB1iN+EvpaD7CBA+V+onkNMN6Fu4 J0aVUYlZuLcqkEs/BuNL
=BRtR
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 5 lines · Christoph Hellwig
I think this is btrfs using a struct
block_devicethat doesn't have a valid queue pointer in it's gendisk for ->s_bdev. And there are some fishy looking ->s_bdev assignments in the code which I suspect are related to it:
Quoted 7 lines
fs/btrfs/dev-replace.c: if (fs_info->sb->s_bdev==src_device->bdev)fs/btrfs/dev-replace.c:fs_info->sb->s_bdev=tgt_device->bdev;fs/btrfs/volumes.c: if
(device->bdev==root->fs_info->sb->s_bdev)fs/btrfs/volumes.c:root->fs_info->sb->s_bdev=next_device->bdev;fs/btrfs/volumes.c:
if (tgtdev->bdev==fs_info->sb->s_bdev)fs/btrfs/volumes.c:fs_info->sb->s_bdev=next_device->bdev;
The report at https://bugzilla.kernel.org/show_bug.cgi?id=100911
tracks it down a bit further and it's bdev->bd_disk == NULL instead of the queue in the gendisk. I don't think that the s_bdev stuff is related, though I'd certainly love to see that bit go away.
If we're calling blk_get_backing_dev_info, that means we're already using an inode that has blockdev_superblock and the btrfs superblock isn't even involved.
We're getting there because btrfs_evict_inode -> btrfs_wait_ordered_range -> btrfs_fdatawrite_range -> filemap_fdatawrite_range gets called with inode->i_mapping. That mapping gets passed down through __filemap_fdatawrite_range to wbc_attach_fdatawrite_inode where the inode passed is mapping->host -- which will be the block device inode rather than the btrfs device node inode. That inode is the one ultimately checked in inode_to_bdi.
So it looks like we're causing writeback on an unrelated block device that was opened using a device node hosted on btrfs, which is obviously wrong.
I don't think snapshot removal is even a requirement to trigger this.
I expect it's possible to trigger with two device nodes for the same
block device where one is getting closed and cleaned up while the eviction of the other happens. The device nodes wouldn't even need to be on the same fs.
Other file systems use &inode->i_data in eviction. Is it that simple here?
- -Jeff
- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.19 (Darwin)
iQIcBAEBAgAGBQJV8yOVAAoJEB57S2MheeWysvMP/0cIPCytKGzQqkNpzjfcBk4b a4s3xM3xnxZ0BayvAWIpSrCLp/5OR5N30Eu326LNZKIEnC7jbkQHePFLIftnhtJ/ eGWlFe9kOsHGWtdA2HyZO9s6V/Nnh0t7vXKUBfqTjV71T66VL/FP9cfRVJ4Ov5Zb 99dK58glhDuF0tOQhePdfaqw4zym+3YHkD+CJjTUKO9YnpTgr4CQFJ+6v6itGbIt QRY7qVY0S1nz0w/s8AsKu2g76thILtBvmwsEMik3TYSJI5gHxLgSpR0btk64o67+ N50AGsO/TMJs6u9p8Ad4zMFF8AfylAgTV3g8uH6v2QLI3ILVMhjtqgOwWlT78Aca dmceWAfhBAdRizYqKQC6ZKq26Qf9GTSEoM0L/3TuBqN5scKtGYx0mvoDzj080i7p nmPJ955pWwxa2tsmo8wRoPXVjvOXegIyguyHvqTg0wrwzfm4aPtZGTtr7RU65lp2 83fl2KJXan8V1vkOwmZ9n4e1G1g8Gggb+qCMAiv9cLWkfTus2HFdh5GNEZ+jSCJ1 2+QzIjFzLqx0N3wQmneBfkdDiWpQkAbQJjJLPdJykivo4WytV/6Vtvcqbv39JCJj 1awM2EpqB9rKV24BGDH86MiErvVT3HBLjSEEpIa41T8PlBXEsQOH1hsXTZSzyP9o iO8qclZgSIIUgiN4feV3
=xPuq
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 36 lines · Jeff Mahoney
Quoted 5 lines · Christoph Hellwig
I think this is btrfs using a struct
block_devicethat doesn't have a valid queue pointer in it's gendisk for ->s_bdev. And there are some fishy looking ->s_bdev assignments in the code which I suspect are related to it:Quoted 7 lines
fs/btrfs/dev-replace.c: if (fs_info->sb->s_bdev==src_device->bdev)fs/btrfs/dev-replace.c:fs_info->sb->s_bdev=tgt_device->bdev;fs/btrfs/volumes.c: if (device->bdev==root->fs_info->sb->s_bdev)fs/btrfs/volumes.c:root->fs_info->sb->s_bdev=next_device->bdev;fs/btrfs/volumes.c: if (tgtdev->bdev==fs_info->sb->s_bdev)fs/btrfs/volumes.c:fs_info->sb->s_bdev=next_device->bdev;The report at https://bugzilla.kernel.org/show_bug.cgi?id=100911
tracks it down a bit further and it'sbdev->bd_disk== NULL instead of the queue in the gendisk. I don't think that thes_bdevstuff is related, though I'd certainly love to see that bit go away.If we're calling
blk_get_backing_dev_info, that means we're already using an inode that hasblockdev_superblockand the btrfs superblock isn't even involved.We're getting there because
btrfs_evict_inode->btrfs_wait_ordered_range->btrfs_fdatawrite_range->filemap_fdatawrite_rangegets called withinode->i_mapping. That mapping gets passed down through__filemap_fdatawrite_rangetowbc_attach_fdatawrite_inodewhere the inode passed ismapping->host-- which will be the block device inode rather than the btrfs device node inode. That inode is the one ultimately checked ininode_to_bdi.So it looks like we're causing writeback on an unrelated block device that was opened using a device node hosted on btrfs, which is obviously wrong.
Quoted 8 lines
I don't think snapshot removal is even a requirement to trigger this. I expect it's possible to trigger with two device nodes for the same block device where one is getting closed and cleaned up while the eviction of the other happens. The device nodes wouldn't even need to be on the same fs.
Other file systems use
&inode->i_datain eviction. Is it that simple here?
Incidentally, this explanation also covers why I was unable to reproduce it locally. SLES systems use devtmpfs and I just bind mounted it into my chroot environment like I normally would. When I cp'd /dev into the test environment, I was able to reproduce immediately .
- -Jeff
- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.19 (Darwin)
iQIcBAEBAgAGBQJV8yYAAAoJEB57S2MheeWy2BwP+QGdpsErIfyHJcx95LLrvsxu n0kBoI4Jd5yfNxp8m+Ll3xgUdsd6rKHJV2Muq8aRdNEdzf1E0DFrRcE0d1W5UrJy lPzrA8QxCVaLf5jFysFp0xygKbLKHGmOAv2KnAGYFw6exIjb344UnZb6aiw5Uekm DqrTmEq+0Yb/mE04GVpWMylK6pkDOhgkOzFVZa1Pff0eKY4E61G5GtmA2kNAUP9v CsoZ0FO1WdF2Fc9ONSPjq7FdZLKH+OmIVakHnaELa8EEM3W7NU+mxLRabznBV25e L/KPjr+awzkhV1ieyAAww/dddE3bN5nmDOq+OgvA9WPgaRvvwne2tHVTFaxHoiHg d8oHDLkC1/Z1MqINLi5dZNsSuIWMvRhIMV9Th5F2rdWxrBCSRvID7N+Z2HHh6mJC Q9rgSOyYKclTam6IF7yX8lDWIqkAnoA6OxvOKRccgr3hS/u4DzVtRmWHO9RblEi+ a9dF2FCP+v+Lgdb8C5n7XUixrtF5H6BWHhmArgjmxD6iyeXOmphyGrgqmSLdY1s9 sakvLrSB9i3O27CKoup2OHyF6MOdgsaa90FZLPLt6BDrCTWAscd0LDy8MbaKgKCR kjfSTiwNydzZfkJixH71U/1mGbuB9nqf6jrNWCQdE5f57MSCEwiFqQvaD1KK+Uug ZW2Bz1VQxkOvGbYiJ4HV
=ic4+
-----END PGP SIGNATURE-----
Am 11.09.2015 um 21:05 schrieb Jeff Mahoney:
Quoted 48 lines
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 44 lines · Jeff Mahoney
Quoted 5 lines · Christoph Hellwig
I think this is btrfs using a struct
block_devicethat doesn't have a valid queue pointer in it's gendisk for ->s_bdev. And there are some fishy looking ->s_bdev assignments in the code which I suspect are related to it:Quoted 7 lines
fs/btrfs/dev-replace.c: if (fs_info->sb->s_bdev==src_device->bdev)fs/btrfs/dev-replace.c:fs_info->sb->s_bdev=tgt_device->bdev;fs/btrfs/volumes.c: if (device->bdev==root->fs_info->sb->s_bdev)fs/btrfs/volumes.c:root->fs_info->sb->s_bdev=next_device->bdev;fs/btrfs/volumes.c: if (tgtdev->bdev==fs_info->sb->s_bdev)fs/btrfs/volumes.c:fs_info->sb->s_bdev=next_device->bdev;The report at https://bugzilla.kernel.org/show_bug.cgi?id=100911
tracks it down a bit further and it'sbdev->bd_disk== NULL instead of the queue in the gendisk. I don't think that thes_bdevstuff is related, though I'd certainly love to see that bit go away.If we're calling
blk_get_backing_dev_info, that means we're already using an inode that hasblockdev_superblockand the btrfs superblock isn't even involved.We're getting there because
btrfs_evict_inode->btrfs_wait_ordered_range->btrfs_fdatawrite_range->filemap_fdatawrite_rangegets called withinode->i_mapping. That mapping gets passed down through__filemap_fdatawrite_rangetowbc_attach_fdatawrite_inodewhere the inode passed ismapping->host-- which will be the block device inode rather than the btrfs device node inode. That inode is the one ultimately checked ininode_to_bdi.So it looks like we're causing writeback on an unrelated block device that was opened using a device node hosted on btrfs, which is obviously wrong.
I don't think snapshot removal is even a requirement to trigger this. I expect it's possible to trigger with two device nodes for the same block device where one is getting closed and cleaned up while the eviction of the other happens. The device nodes wouldn't even need to be on the same fs.
Other file systems use
&inode->i_datain eviction. Is it that simple here?
Your patch works fine here. Did some simple tests already.
Thanks!
Stefan
Quoted 30 lines
Incidentally, this explanation also covers why I was unable to reproduce it locally. SLES systems use devtmpfs and I just bind mounted it into my chroot environment like I normally would. When I cp'd /dev into the test environment, I was able to reproduce immediately .
- -Jeff
- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.19 (Darwin)iQIcBAEBAgAGBQJV8yYAAAoJEB57S2MheeWy2BwP+QGdpsErIfyHJcx95LLrvsxu n0kBoI4Jd5yfNxp8m+Ll3xgUdsd6rKHJV2Muq8aRdNEdzf1E0DFrRcE0d1W5UrJy lPzrA8QxCVaLf5jFysFp0xygKbLKHGmOAv2KnAGYFw6exIjb344UnZb6aiw5Uekm DqrTmEq+0Yb/mE04GVpWMylK6pkDOhgkOzFVZa1Pff0eKY4E61G5GtmA2kNAUP9v CsoZ0FO1WdF2Fc9ONSPjq7FdZLKH+OmIVakHnaELa8EEM3W7NU+mxLRabznBV25e L/KPjr+awzkhV1ieyAAww/dddE3bN5nmDOq+OgvA9WPgaRvvwne2tHVTFaxHoiHg d8oHDLkC1/Z1MqINLi5dZNsSuIWMvRhIMV9Th5F2rdWxrBCSRvID7N+Z2HHh6mJC Q9rgSOyYKclTam6IF7yX8lDWIqkAnoA6OxvOKRccgr3hS/u4DzVtRmWHO9RblEi+ a9dF2FCP+v+Lgdb8C5n7XUixrtF5H6BWHhmArgjmxD6iyeXOmphyGrgqmSLdY1s9 sakvLrSB9i3O27CKoup2OHyF6MOdgsaa90FZLPLt6BDrCTWAscd0LDy8MbaKgKCR kjfSTiwNydzZfkJixH71U/1mGbuB9nqf6jrNWCQdE5f57MSCEwiFqQvaD1KK+Uug ZW2Bz1VQxkOvGbYiJ4HV
=ic4+
-----END PGP SIGNATURE-----
Quoted 38 lines · Jeff Mahoney
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 5 lines · Christoph Hellwig
I think this is btrfs using a struct
block_devicethat doesn't have a valid queue pointer in it's gendisk for ->s_bdev. And there are some fishy looking ->s_bdev assignments in the code which I suspect are related to it:Quoted 7 lines
fs/btrfs/dev-replace.c: if (fs_info->sb->s_bdev==src_device->bdev)fs/btrfs/dev-replace.c:fs_info->sb->s_bdev=tgt_device->bdev;fs/btrfs/volumes.c: if
(device->bdev==root->fs_info->sb->s_bdev)fs/btrfs/volumes.c:root->fs_info->sb->s_bdev=next_device->bdev;fs/btrfs/volumes.c:
if (tgtdev->bdev==fs_info->sb->s_bdev)fs/btrfs/volumes.c:fs_info->sb->s_bdev=next_device->bdev;The report at https://bugzilla.kernel.org/show_bug.cgi?id=100911
tracks it down a bit further and it'sbdev->bd_disk== NULL instead of the queue in the gendisk. I don't think that thes_bdevstuff is related, though I'd certainly love to see that bit go away.If we're calling
blk_get_backing_dev_info, that means we're already using an inode that hasblockdev_superblockand the btrfs superblock isn't even involved.We're getting there because
btrfs_evict_inode->btrfs_wait_ordered_range->btrfs_fdatawrite_range->filemap_fdatawrite_rangegets called withinode->i_mapping. That mapping gets passed down through__filemap_fdatawrite_rangetowbc_attach_fdatawrite_inodewhere the inode passed ismapping->host-- which will be the block device inode rather than the btrfs device node inode. That inode is the one ultimately checked ininode_to_bdi.So it looks like we're causing writeback on an unrelated block device that was opened using a device node hosted on btrfs, which is obviously wrong.
Quoted 8 lines
I don't think snapshot removal is even a requirement to trigger this.
I expect it's possible to trigger with two device nodes for the same
block device where one is getting closed and cleaned up while the eviction of the other happens. The device nodes wouldn't even need to be on the same fs.Other file systems use
&inode->i_datain eviction. Is it that simple here?
Oh, ok I'm following now. This really should explain it. Jeff mentioned that he's working on a patch to skip the wait_ordered_range dance based on i_mode. Thanks Jeff!
-chris