md related oops triggered in bdev_inode_switch_bdi
Hi Christoph et. al.,
My testing recently triggered an oops in bdi_lock_two called from bdev_inode_switch_bdi.
The bdi and the request_queue that contains it had been freed.
This happens with md which can free the md device and request queue immediately after last close.
It seems that this is caused by your patch f758eeabeb96f8. Prior to that the 'old' bdi was never dereferenced in bdev_inode_switch_bdi. Now it is.
I think we can fix that by simply moving the call to bdev_inode_switch_bdi before the call to ->release as in the patch below.
Do you see any problem with this patch?
Thanks,
NeilBrown
Quoted 1 line
From
bcc5851cbb6876c97cce214feddd0ec092f7d71cMon Sep 17 00:00:00 2001
From: NeilBrown <[email protected]>
Date: Wed, 31 Aug 2011 15:37:22 +1000
Subject: [PATCH] Avoid dereferencing a 'request_queue' after last close.
On the last close of an 'md' device which as been stopped, the device is destroyed and in particular the request_queue is freed. The free is done in a separate thread so it might happen a short time later.
__blkdev_put calls bdev_inode_switch_bdi *after* ->release has been called.
Since commit f758eeabeb96f878c860e8f110f94ec8820822a9 bdev_inode_switch_bdi will dereference the 'old' bdi, which lives inside a request_queue, to get a spin lock. This causes the last close on an md device to sometime take a spin_lock which lives in freed memory - which results in an oops.
So move the called to bdev_inode_switch_bdi before the call to ->release.
fs/block_dev.c +6 -2
@@ -1430,6 +1430,12 @@ static int __blkdev_put(struct block_device *bdev, fmode_t mode, int for_part)sync_blockdev (bdev);kill_bdev (bdev); } +if (!bdev->bd_openers) +/* ->release can cause the old bdi to disappear, +* so must switch it out first +*/ +bdev_inode_switch_bdi (bdev->bd_inode, + &default_backing_dev_info);if (bdev->bd_contains == bdev) {if (disk->fops->release) ret = disk->fops->release (disk, mode);
@@ -1442,8 +1448,6 @@ static int __blkdev_put(struct block_device *bdev, fmode_t mode, int for_part)disk_put_part (bdev->bd_part); bdev->bd_part =NULL ; bdev->bd_disk =NULL ; -bdev_inode_switch_bdi (bdev->bd_inode, - &default_backing_dev_info);if (bdev != bdev->bd_contains) victim = bdev->bd_contains; bdev->bd_contains =NULL ;
Wu Fengguang
Reviewed-byHi Neil,
Quoted 1 line
Subject: [PATCH] Avoid dereferencing a '
request_queue' after last close.
with comments below.
Quoted 5 lines
fs/block_dev.c +0
-0@@ -1430,6 +1430,12 @@ static int __blkdev_put(struct block_device *bdev, fmode_t mode, int for_part)sync_blockdev (bdev);kill_bdev (bdev);
fs/block_dev.c static int __blkdev_put(struct block_device *bdev, fmode_t mode, int for_part) · line 1441
Quoted 10 lines
fs/block_dev.c +6
-0} +if (!bdev->bd_openers) +/* ->release can cause the old bdi to disappear, +* so must switch it out first +*/ +bdev_inode_switch_bdi (bdev->bd_inode, + &default_backing_dev_info);if (bdev->bd_contains == bdev) {if (disk->fops->release) ret = disk->fops->release (disk, mode);
The bdev_inode_switch_bdi() call can be further moved into the previous if block, like this:
if (!--bdev->bd_openers) {
WARN_ON_ONCE(bdev->bd_holders);
sync_blockdev(bdev);
kill_bdev(bdev);+
+ /* ->release can cause the old bdi to disappear,
+ * so must switch it out first
+ */
+ bdev_inode_switch_bdi(bdev->bd_inode,
+ &default_backing_dev_info);
}Then it's obvious that kill_bdev() will truncate all inode pages and there won't be further interactions with dirty writes.
Although there are dozens of disk->fops->release functions, however it's very unlikely they need to access some inode on top of the disk (which is illogical thing).
So I don't see any problems. It makes sense to push it to next for broader test ASAP. Will you do it, or me?
Thanks,
Fengguang
Quoted 5 lines · On Thu, 1 Sep 2011 11:30:56 +0800 Wu Fengguang <[email protected]> wrote:
Hi Neil,
Quoted 1 line
Subject: [PATCH] Avoid dereferencing a '
request_queue' after last close.Reviewed-by Wu Fengguang <[email protected]>
Thanks.
Quoted 26 lines
with comments below.
Quoted 5 lines
fs/block_dev.c +0
-0@@ -1430,6 +1430,12 @@ static int __blkdev_put(struct block_device *bdev, fmode_t mode, int for_part)sync_blockdev (bdev);kill_bdev (bdev);fs/block_dev.c static int __blkdev_put(struct block_device *bdev, fmode_t mode, int for_part) · line 1441
Quoted 10 lines
fs/block_dev.c +6
-0} +if (!bdev->bd_openers) +/* ->release can cause the old bdi to disappear, +* so must switch it out first +*/ +bdev_inode_switch_bdi (bdev->bd_inode, + &default_backing_dev_info);if (bdev->bd_contains == bdev) {if (disk->fops->release) ret = disk->fops->release (disk, mode);The
bdev_inode_switch_bdi()call can be further moved into the previous if block, like this:if (!--bdev->bd_openers) { WARN_ON_ONCE(bdev->bd_holders); sync_blockdev(bdev); kill_bdev(bdev);
fs/block_dev.c static int __blkdev_put(struct block_device *bdev, fmode_t mode, int for_part)
Quoted 7 lines
fs/block_dev.c +6
-0+ +/* ->release can cause the old bdi to disappear, +* so must switch it out first +*/ +bdev_inode_switch_bdi (bdev->bd_inode, + &default_backing_dev_info); }
Yes, and obvious improvement now that you have pointed it out - thanks.
Quoted 10 lines
Then it's obvious that
kill_bdev()will truncate all inode pages and there won't be further interactions with dirty writes.Although there are dozens of
disk->fops->releasefunctions, however it's very unlikely they need to access some inode on top of the disk (which is illogical thing).So I don't see any problems. It makes sense to push it to next for broader test ASAP. Will you do it, or me?
I've just push it into my for-next. If I heard nothing else by mid next week I'll push it to Linus
Thanks,
NeilBrown
Quoted 3 lines
Thanks,
Fengguang
Quoted 56 lines · NeilBrown
Quoted 5 lines · On Thu, 1 Sep 2011 11:30:56 +0800 Wu Fengguang <[email protected]> wrote:
Hi Neil,
Quoted 1 line
Subject: [PATCH] Avoid dereferencing a '
request_queue' after last close.Reviewed-by Wu Fengguang <[email protected]>Thanks.
Quoted 22 lines
with comments below.
Quoted 5 lines
fs/block_dev.c +0
-0@@ -1430,6 +1430,12 @@ static int __blkdev_put(struct block_device *bdev, fmode_t mode, int for_part) sync_blockdev(bdev); kill_bdev(bdev);fs/block_dev.c static int __blkdev_put(struct block_device *bdev, fmode_t mode, int for_part) · line 1441
Quoted 10 lines
fs/block_dev.c +6
-0} + if (!bdev->bd_openers) + /* ->release can cause the old bdi to disappear, + * so must switch it out first + */ + bdev_inode_switch_bdi(bdev->bd_inode, + &default_backing_dev_info); if (bdev->bd_contains == bdev) { if (disk->fops->release) ret = disk->fops->release(disk, mode);The
bdev_inode_switch_bdi()call can be further moved into the previous if block, like this:Quoted 11 lines
if (!--bdev->bd_openers) {
WARN_ON_ONCE(bdev->bd_holders);sync_blockdev(bdev);
kill_bdev(bdev);
+
+ /* ->release can cause the old bdi to disappear,
+ * so must switch it out first
+ */
+ bdev_inode_switch_bdi(bdev->bd_inode,
+&default_backing_dev_info); }Yes, and obvious improvement now that you have pointed it out - thanks.
Quoted 8 lines
Then it's obvious that
kill_bdev()will truncate all inode pages and there won't be further interactions with dirty writes.Although there are dozens of
disk->fops->releasefunctions, however it's very unlikely they need to access some inode on top of the disk (which is illogical thing).
Quoted 5 lines
Quoted 2 lines
So I don't see any problems. It makes sense to push it to next for broader test ASAP. Will you do it, or me?
I've just push it into my for-next. If I heard nothing else by mid next week I'll push it to Linus
Just FYI,
this patch also fixes below bug which can be reproduced by unplug the usb disk before umount.
BUG blkdev_queue: Poison overwritten
-----------------------------------------------------------------------------
INFO: 0xffff880106640308-0xffff880106640309. First byte 0x6c instead of 0x6b
INFO: Allocated in blk_alloc_queue_node+0x1f/0x1a9 age=5794 cpu=0 pid=3449
__slab_alloc+0x503/0x56e
kmem_cache_alloc_node+0x61/0x18e
blk_alloc_queue_node+0x1f/0x1a9
blk_init_queue_node+0x24/0x61
blk_init_queue+0xc/0xe
__scsi_alloc_queue+0x21/0x145
scsi_alloc_queue+0x18/0x64
scsi_alloc_sdev+0x1bb/0x282
scsi_probe_and_add_lun+0x13b/0xb6a
__scsi_scan_target+0xc5/0x60c
scsi_scan_channel+0x58/0x80
scsi_scan_host_selected+0xed/0x136
do_scsi_scan_host+0x6b/0x70
scsi_scan_host+0x1a0/0x1c5
usb_stor_scan_thread+0x16d/0x17b
kthread+0x7d/0x85
INFO: Freed in blk_release_queue+0x60/0x65 age=2267 cpu=1 pid=3484
__slab_free+0x2c/0x30d
kmem_cache_free+0xbc/0x127
blk_release_queue+0x60/0x65
kobject_release+0x51/0x67
kref_put+0x43/0x4d
kobject_put+0x47/0x4b
blk_put_queue+0x10/0x12
scsi_device_dev_release_usercontext+0xbf/0x10a
execute_in_process_context+0x2a/0x61
scsi_device_dev_release+0x17/0x19
device_release+0x49/0x7e
kobject_release+0x51/0x67
kref_put+0x43/0x4d
kobject_put+0x47/0x4b
put_device+0x12/0x14
scsi_device_put+0x3d/0x42Thanks,
Lin Ming
Sitsofe Wheeler
Tested-byQuoted 7 lines · Lin Ming
Quoted 6 lines · NeilBrown
Quoted 4 lines · On Thu, 1 Sep 2011 11:30:56 +0800 Wu Fengguang <[email protected]> wrote:
Hi Neil,
Quoted 1 line
Subject: [PATCH] Avoid dereferencing a '
request_queue' after last close.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Reviewed-by Wu Fengguang <[email protected]>Thanks.
Just FYI,
this patch also fixes below bug which can be reproduced by unplug the usb disk before umount.
Agreed - this fixes the oops I was seeing pulling out a USB key with a mounted FAT32 filesystem (https://lkml.org/lkml/2011/9/5/339 ). Thanks to Yang RuiRui and akpm for point me to this patch.
Signature · 1 line
Sitsofe | http://sucs.org/~sits/