use rcu_barrier() to wait for bdev puts at unmount
PATCH 1/1 use rcu_barrier() to wait for bdev puts at unmount
Doing this would reliably fail with -EBUSY for me:
# mount /dev/sdb2 /mnt/scratch; umount /mnt/scratch; mkfs.btrfs -f /dev/sdb2 ... unable to open /dev/sdb2: Device or resource busy
because mkfs.btrfs tries to open the device O_EXCL, and somebody still has it.
Using systemtap to track bdev gets & puts shows a kworker thread doing a blkdev put after mkfs attempts a get; this is left over from the unmount.
Adding an rcu_barrier() to btrfs_close_devices() causes unmount to wait until all blkdev_put()s are done, and the device is truly free once unmount completes.
---
p.s. I debated putting it into close_ctree(); I don't know if there' anything else to wait for. Thoughts?
fs/btrfs/volumes.c +1 -0
@@ -680,6 +680,7 @@ int btrfs_close_devices(struct btrfs_fs_devices *fs_devices)__btrfs_close_devices (fs_devices);free_fs_devices (fs_devices); } +rcu_barrier ();return ret; }
Quoted 6 lines · Eric Sandeen
Doing this would reliably fail with
-EBUSYfor me:# mount /dev/sdb2 /mnt/scratch; umount /mnt/scratch; mkfs.btrfs -f /dev/sdb2 ... unable to open /dev/sdb2: Device or resource busy
Quoted 8 lines
because mkfs.btrfs tries to open the device
O_EXCL, and somebody still has it.Using systemtap to track bdev gets & puts shows a kworker thread doing a blkdev put after mkfs attempts a get; this is left over from the unmount.
Adding an
rcu_barrier()tobtrfs_close_devices()causes unmount to wait until all blkdev_put()s are done, and the device is truly free once unmount completes.
Thanks for tracking this down Eric. Is this kworker triggered by btrfs or is this something we should be doing for the other filesystems too?
I'd move it down to close_ctree, but I don't really have a good reason.
-chris
Quoted 9 lines · Chris Mason
Quoted 8 lines · Eric Sandeen
Doing this would reliably fail with
-EBUSYfor me:# mount /dev/sdb2 /mnt/scratch; umount /mnt/scratch; mkfs.btrfs -f /dev/sdb2 ... unable to open /dev/sdb2: Device or resource busybecause mkfs.btrfs tries to open the device
O_EXCL, and somebody still has it.
Quoted 8 lines
Quoted 6 lines
Using systemtap to track bdev gets & puts shows a kworker thread doing a blkdev put after mkfs attempts a get; this is left over from the unmount.
Adding an
rcu_barrier()tobtrfs_close_devices()causes unmount to wait until all blkdev_put()s are done, and the device is truly free once unmount completes.Thanks for tracking this down Eric.
Sure thing, sorry it took so long.
Quoted 2 lines
Is this kworker triggered by btrfs
or is this something we should be doing for the other filesystems too?
It's all btrfs ;)
btrfs_close_devices
__btrfs_close_devices call_rcu(&device->rcu, free_device); free_device INIT_WORK(&device->rcu_work, __free_device); schedule_work(&device->rcu_work);
The behavior came from:
commit 1f78160ce1b1b8e657e2248118c4d91f881763f0
Author: Xiao Guangrong <[email protected]> Date: Wed Apr 20 10:09:16 2011 +0000
Btrfs: using rcu lock in the reader side of devices list
Anyway, I can send V2 in close_ctree if you like. Thinking about it more though, btrfs_close_devices is closer to the action, so now I think I'd leave it there. :)
I probably should have put a comment in to say what the heck it's for, too. Feel free to fix on merge or I can send another patch.
Thanks,
-Eric
Quoted 4 lines
I'd move it down to
close_ctree, but I don't really have a good reason.-chris
Quoted 24 lines · Eric Sandeen
Quoted 9 lines · Chris Mason
Quoted 8 lines · Eric Sandeen
Doing this would reliably fail with
-EBUSYfor me:# mount /dev/sdb2 /mnt/scratch; umount /mnt/scratch; mkfs.btrfs -f /dev/sdb2 ... unable to open /dev/sdb2: Device or resource busybecause mkfs.btrfs tries to open the device
O_EXCL, and somebody still has it.Quoted 8 lines
Quoted 6 lines
Using systemtap to track bdev gets & puts shows a kworker thread doing a blkdev put after mkfs attempts a get; this is left over from the unmount.
Adding an
rcu_barrier()tobtrfs_close_devices()causes unmount to wait until all blkdev_put()s are done, and the device is truly free once unmount completes.Thanks for tracking this down Eric.
Sure thing, sorry it took so long.
Quoted 2 lines
Is this kworker triggered by btrfs
or is this something we should be doing for the other filesystems too?
Quoted 9 lines
It's all btrfs ;)
btrfs_close_devices__btrfs_close_devices call_rcu(&device->rcu, free_device); free_device INIT_WORK(&device->rcu_work, __free_device); schedule_work(&device->rcu_work);
Great, that makes a ton more sense. I'm a little confused on why we're seeing it so much more now than in the past.
Quoted 7 lines
The behavior came from:
commit
1f78160ce1b1b8e657e2248118c4d91f881763f0Author: Xiao Guangrong <[email protected]> Date: Wed Apr 20 10:09:16 2011 +0000
Quoted 8 lines
Btrfs: using rcu lock in the reader side of devices list
Anyway, I can send V2 in
close_ctreeif you like. Thinking about it more though,btrfs_close_devicesis closer to the action, so now I think I'd leave it there. :)I probably should have put a comment in to say what the heck it's for, too. Feel free to fix on merge or I can send another patch.
Ok, please add the comment and a cc to stable. Thanks again.
-chris
PATCH v2 1/1 btrfs: use rcu_barrier() to wait for bdev puts at unmount
In reply to Eric SandeenDoing this would reliably fail with -EBUSY for me:
# mount /dev/sdb2 /mnt/scratch; umount /mnt/scratch; mkfs.btrfs -f /dev/sdb2 ... unable to open /dev/sdb2: Device or resource busy
because mkfs.btrfs tries to open the device O_EXCL, and somebody still has it.
Using systemtap to track bdev gets & puts shows a kworker thread doing a blkdev put after mkfs attempts a get; this is left over from the unmount path:
btrfs_close_devices
__btrfs_close_devices call_rcu(&device->rcu, free_device); free_device INIT_WORK(&device->rcu_work, __free_device); schedule_work(&device->rcu_work);
so unmount might complete before __free_device fires & does its blkdev_put.
Adding an rcu_barrier() to btrfs_close_devices() causes unmount to wait until all blkdev_put()s are done, and the device is truly free once unmount completes.
---
V2: expand commit msg, add code comment, cc: stable
fs/btrfs/volumes.c +6 -0
@@ -680,6 +680,12 @@ int btrfs_close_devices(struct btrfs_fs_devices *fs_devices)__btrfs_close_devices (fs_devices);free_fs_devices (fs_devices); } +/* +* Wait for rcu kworkers under __btrfs_close_devices +* to finish all blkdev_puts so device is really +* free when umount is done. +*/ +rcu_barrier ();return ret; }