Allow race-free block device handling
Subsystems (3)
cover Allow race-free block device handling
This work aims to allow userspace to create and destroy block devices in a race-free and leak-free way, and to allow them to be exposed to other Xen VMs via blkback without leaks or races. It’s marked as RFC for a few reasons:
- The code has been only lightly tested. It might be unstable or insecure.
- The DM_DEV_CREATE ioctl gains a new flag. Unknown flags were previously ignored, so this could theoretically break buggy userspace tools.
- I have no idea if I got the block device reference counting and locking correct.
Demi Marie Obenour (7):
block: Support creating a struct file from a block device Allow userspace to get an FD to a newly-created DM device Implement diskseq checks in blkback
Increment diskseq when releasing a loop device If autoclear is set, delete a no-longer-used loop device Minor blkback cleanups
xen/blkback: Inform userspace that device has been opened
block/bdev.c | 77 +++++++++++-- block/genhd.c | 1 + drivers/block/loop.c | 17 ++- drivers/block/xen-blkback/blkback.c | 8 +- drivers/block/xen-blkback/xenbus.c | 171 ++++++++++++++++++++++------ drivers/md/dm-ioctl.c | 67 +++++++++-- include/linux/blkdev.h | 5 + include/uapi/linux/dm-ioctl.h | 16 ++- 8 files changed, 298 insertions(+), 64 deletions(-)
Signature · 3 lines
Sincerely, Demi Marie Obenour (she/her/hers) Invisible Things Lab
RFC PATCH 1/7 block: Support creating a struct file from a block device
The newly added blkdev_get_file() function allows kernel code to create a struct file for any block device. The main use-case is for the struct file to be exposed to userspace as a file descriptor. A future patch will modify the DM_DEV_CREATE_CREATE ioctl to allow userspace to get a file descriptor to the newly created block device, avoiding nasty race conditions.
--- block/bdev.c | 77 +++++++++++++++++++++++++++++++++++------- include/linux/blkdev.h | 5 +++ 2 files changed, 70 insertions(+), 12 deletions(-)
block/bdev.c +65 -12
@@ -459,10 +459,33 @@ static struct file_system_type bd_type = {struct super_block *blockdev_superblock__read_mostly ;EXPORT_SYMBOL_GPL (blockdev_superblock); +static struct vfsmount *bd_mnt__read_mostly ; + +struct file * +blkdev_get_file (struct block_device *bdev,fmode_t flags,void *holder) +{ +struct inode *inode; +struct file *filp; +int ret; + + ret =blkdev_do_open (bdev, flags, holder); +if (ret) +return ERR_PTR (ret); + inode = bdev->bd_inode; + filp =alloc_file_pseudo (inode, bd_mnt,"[block]" , flags |O_CLOEXEC , &def_blk_fops); +if (IS_ERR (filp)) { +blkdev_put (bdev, flags); + }else { + filp->f_mapping = inode->i_mapping; + filp->f_wb_err =filemap_sample_wb_err (filp->f_mapping); + } +return filp; +} +EXPORT_SYMBOL (blkdev_get_file); +void __init bdev_cache_init (void ) {int err; -static struct vfsmount *bd_mnt; bdev_cachep = kmem_cache_create("bdev_cache", sizeof(struct bdev_inode), 0, (SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|
@@ -775,7 +798,7 @@ void blkdev_put_no_open(struct block_device *bdev)* * Use this interface ONLY if you really do not have anything better - i.e. when * you are behind a truly sucky interface and all you are given is a device -* number. Everything else should use blkdev_get_by_path(). +* number. Everything else should use blkdev_get_by_path() or blkdev_do_open(). * * CONTEXT: * Might sleep.
@@ -785,9 +808,7 @@ void blkdev_put_no_open(struct block_device *bdev)*/ struct block_device *blkdev_get_by_dev (dev_t dev,fmode_t mode,void *holder) { -bool unblock_events = true;struct block_device *bdev; -struct gendisk *disk;int ret; ret = devcgroup_check_permission(DEVCG_DEV_BLOCK,
@@ -800,18 +821,52 @@ struct block_device *blkdev_get_by_dev(dev_t dev, fmode_t mode, void *holder) bdev =blkdev_get_no_open (dev);if (!bdev)return ERR_PTR (-ENXIO ); - disk = bdev->bd_disk; + + ret =blkdev_do_open (bdev, mode, holder); +if (ret) { +blkdev_put_no_open (bdev); +return ERR_PTR (ret); + } + +return bdev; +} +EXPORT_SYMBOL (blkdev_get_by_dev); + +/** +* blkdev_do_open - open a block device by device pointer +* @bdev: pointer to the device to open +* @mode: FMODE_* mask +* @holder: exclusive holder identifier +* +* Open the block device pointed to by @bdev. If @mode includes +* %FMODE_EXCL, the block device is opened with exclusive access. Specifying +* %FMODE_EXCL with a %NULL @holder is invalid. Exclusive opens may nest for +* the same @holder. +* +* Unlike blkdev_get_by_dev() and bldev_get_by_path(), this function does not +* do any permission checks. The most common use-case is where the device +* was freshly created by userspace. +* +* CONTEXT: +* Might sleep. +* +* RETURNS: +* Reference 0 on success, -errno on failure. +*/ +int blkdev_do_open (struct block_device *bdev,fmode_t mode,void *holder) { +struct gendisk *disk = bdev->bd_disk; +int ret = -ENXIO ; +bool unblock_events = true;if (mode &FMODE_EXCL ) { ret =bd_prepare_to_claim (bdev, holder);if (ret) - goto put_blkdev; +return ret; }disk_block_events (disk);mutex_lock (&disk->open_mutex); - ret = -ENXIO ;if (!disk_live (disk)) goto abort_claiming;if (!try_module_get (disk->fops->owner))
@@ -842,7 +897,7 @@ struct block_device *blkdev_get_by_dev(dev_t dev, fmode_t mode, void *holder)if (unblock_events)disk_unblock_events (disk); -return bdev; +return 0 ; put_module:module_put (disk->fops->owner); abort_claiming:
@@ -850,11 +905,9 @@ struct block_device *blkdev_get_by_dev(dev_t dev, fmode_t mode, void *holder)bd_abort_claiming (bdev, holder);mutex_unlock (&disk->open_mutex);disk_unblock_events (disk); -put_blkdev: -blkdev_put_no_open (bdev); -return ERR_PTR (ret); +return ret; } -EXPORT_SYMBOL (blkdev_get_by_dev); +EXPORT_SYMBOL (blkdev_do_open);/** * blkdev_get_by_path - open a block device by name
include/linux/blkdev.h +5 -0
@@ -325,6 +325,11 @@ typedef int (*report_zones_cb)(struct blk_zone *zone, unsigned int idx,void disk_set_zoned (struct gendisk *disk,enum blk_zoned_model model); +struct file * +blkdev_get_file (struct block_device *bdev,fmode_t flags,void *holder); + +int blkdev_do_open (struct block_device *bdev,fmode_t flags,void *holder); +#ifdef CONFIG_BLK_DEV_ZONED #define BLK_ALL_ZONES ((unsigned int)-1)
Signature · 3 lines
Sincerely, Demi Marie Obenour (she/her/hers) Invisible Things Lab
Re: [RFC PATCH 1/7] block: Support creating a struct file from a block device
Quoted 6 lines · Demi Marie Obenour
The newly added
blkdev_get_file()function allows kernel code to create a struct file for any block device. The main use-case is for the struct file to be exposed to userspace as a file descriptor. A future patch will modify theDM_DEV_CREATE_CREATEioctl to allow userspace to get a file descriptor to the newly created block device, avoiding nasty race conditions.
NAK. Do not add wierd side-way interfaces to the block layer.
Re: [RFC PATCH 1/7] block: Support creating a struct file from a block device
Quoted 9 lines · Christoph Hellwig
Quoted 6 lines · Demi Marie Obenour
The newly added
blkdev_get_file()function allows kernel code to create a struct file for any block device. The main use-case is for the struct file to be exposed to userspace as a file descriptor. A future patch will modify theDM_DEV_CREATE_CREATEioctl to allow userspace to get a file descriptor to the newly created block device, avoiding nasty race conditions.NAK. Do not add wierd side-way interfaces to the block layer.
What do you recommend instead? This solves a real problem for device-mapper users and I am not aware of a better solution.
Signature · 3 lines
Sincerely, Demi Marie Obenour (she/her/hers) Invisible Things Lab
Re: [RFC PATCH 1/7] block: Support creating a struct file from a block device
Quoted 2 lines · Demi Marie Obenour
What do you recommend instead? This solves a real problem for device-mapper users and I am not aware of a better solution.
You could start with explaining the problem and what other methods you tried that failed. In the end it's not my job to fix your problem. I generally gladly help, but this kind of attitude doesn't get very far.
Re: [RFC PATCH 1/7] block: Support creating a struct file from a block device
Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Demi Marie Obenour
What do you recommend instead? This solves a real problem for device-mapper users and I am not aware of a better solution.
You could start with explaining the problem and what other methods you tried that failed. In the end it's not my job to fix your problem.
I’m working on a “block not-script” (Xen block device hotplug script written in C) for Qubes OS. The current hotplug script is a shell script that takes a global lock, which serializes all invocations and significantly slows down VM creation and destruction. My C program avoids this problem.
One of the goals of the not-script is to never leak resources, even if it dies with SIGKILL or is never called with the “remove” argument to destroy the devices it created. Therefore, whenever possible, it relies on automatic destruction of devices that are no longer used. I have managed to make this work for loop devices, provided that the Xen blkback driver is patched to accept a diskseq in the physical-device Xenstore node. I have *not* managed to make this work for device-mapper devices, however. One of the problems is that there is no way to atomically create a device-mapper device and obtain a file descriptor to it such that the device will be destroyed when no longer used. To solve this problem, I added a new flag (DM_FILE_DESCRIPTOR_FLAG) that asks the device-mapper driver to provide userspace a file descriptor for the device that was just created. The uAPI will likely change in future versions of the patch, but the general idea will not.
While it is easy to provide userspace with an FD to any struct file, it is *not* easy to obtain a struct file for a given struct block_device. I could have had device-mapper implement everything itself, but that would have duplicated a large amount of code already in the block layer. Instead, I decided to refactor the block layer to provide a function that does exactly what was needed. The result was this patch. In the future, I would like to add an ioctl for /dev/loop-control that creates a loop device and returns a file descriptor to the loop device. I could also see iSCSI supporting this, with the socket file descriptor being passed in from userspace.
blkdev_do_open() does not solve any problem for me at this time. Instead, it represents the code shared by blkdev_get_by_dev() and blkdev_get_file(). I decided to export it because it could be of independent use to others. In particular, it could potentially simplify disk_scan_partitions() in block/genhd.c, pkt_new_dev() in pktcdvd, backing_dev_store() in zram, and f2fs_scan_devices() in f2fs.
I hope this is enough information. If it is not, feel free to ask for more.
Signature · 3 lines
Sincerely, Demi Marie Obenour (she/her/hers) Invisible Things Lab
Re: [RFC PATCH 1/7] block: Support creating a struct file from a block device
Quoted 10 lines · Demi Marie Obenour
While it is easy to provide userspace with an FD to any struct file, it is *not* easy to obtain a struct file for a given struct
block_device. I could have had device-mapper implement everything itself, but that would have duplicated a large amount of code already in the block layer. Instead, I decided to refactor the block layer to provide a function that does exactly what was needed. The result was this patch. In the future, I would like to add an ioctl for/dev/loop-controlthat creates a loop device and returns a file descriptor to the loop device. I could also see iSCSI supporting this, with the socket file descriptor being passed in from userspace.
And it is somewhat intentional that you can't. Block device inodes have interesting life times and are never directly exposed to userspace at all. They are internal, and only f_mapping of a file system inode delegates to them or I/O. Your patch now magically exposes them to userspace. And it then bypasses all pathname and inode permission based access checks and auditing. So we can't just do it.
Quoted 6 lines
blkdev_do_open()does not solve any problem for me at this time. Instead, it represents the code shared byblkdev_get_by_dev()andblkdev_get_file(). I decided to export it because it could be of independent use to others. In particular, it could potentially simplifydisk_scan_partitions()inblock/genhd.c,pkt_new_dev()in pktcdvd,backing_dev_store()in zram, andf2fs_scan_devices()in f2fs.
All thse need to actually open the underlying device as they do I/O. Doing I/O without opening the device is a no-go.
Re: [RFC PATCH 1/7] block: Support creating a struct file from a block device
Quoted 12 lines · Christoph Hellwig
Quoted 10 lines · Demi Marie Obenour
While it is easy to provide userspace with an FD to any struct file, it is *not* easy to obtain a struct file for a given struct
block_device. I could have had device-mapper implement everything itself, but that would have duplicated a large amount of code already in the block layer. Instead, I decided to refactor the block layer to provide a function that does exactly what was needed. The result was this patch. In the future, I would like to add an ioctl for/dev/loop-controlthat creates a loop device and returns a file descriptor to the loop device. I could also see iSCSI supporting this, with the socket file descriptor being passed in from userspace.
Quoted 5 lines
And it is somewhat intentional that you can't. Block device inodes have interesting life times and are never directly exposed to userspace at all. They are internal, and only
f_mappingof a file system inode delegates to them or I/O. Your patch now magically exposes them to userspace.
The intention is that the file descriptor is equvalent to what one would get by first creating the device and then opening it. If it is not, that is a bug in one of my patches.
Quoted 2 lines
And it then bypasses all pathname and inode permission based access checks and auditing. So we can't just do it.
Accessing /dev/mapper/control is already enough to panic the kernel, so presumably only fully trusted userspace can make the ioctl to begin with. Furthermore, this only allows a userspace process to get a file descriptor to the device-mapper device it itself created.
Quoted 9 lines
Quoted 6 lines
blkdev_do_open()does not solve any problem for me at this time. Instead, it represents the code shared byblkdev_get_by_dev()andblkdev_get_file(). I decided to export it because it could be of independent use to others. In particular, it could potentially simplifydisk_scan_partitions()inblock/genhd.c,pkt_new_dev()in pktcdvd,backing_dev_store()in zram, andf2fs_scan_devices()in f2fs.All thse need to actually open the underlying device as they do I/O. Doing I/O without opening the device is a no-go.
blkdev_do_open() *does* open the device. If it doesn’t, that’s a bug. In v2 I will add the same access control checks that blkdev_get_by_dev() does. Is this sufficient?
Signature · 3 lines
Sincerely, Demi Marie Obenour (she/her/hers) Invisible Things Lab
Re: [RFC PATCH 1/7] block: Support creating a struct file from a block device
In reply to Demi Marie ObenourQuoted 8 lines · Demi Marie Obenour
Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Demi Marie Obenour
What do you recommend instead? This solves a real problem for device-mapper users and I am not aware of a better solution.
You could start with explaining the problem and what other methods you tried that failed. In the end it's not my job to fix your problem.
Quoted 8 lines
I’m working on a “block not-script” (Xen block device hotplug script written in C) for Qubes OS. The current hotplug script is a shell script that takes a global lock, which serializes all invocations and significantly slows down VM creation and destruction. My C program avoids this problem.
One of the goals of the not-script is to never leak resources, even if it dies with SIGKILL or is never called with the “remove” argument to
If it dies, you still can restart one new instance for handling the device leak by running one simple daemon to monitor if not-script is live.
Quoted 2 lines
destroy the devices it created. Therefore, whenever possible, it relies on automatic destruction of devices that are no longer used. I have
This automatic destruction of devices is supposed to be done in userspace, cause only userspace knows when device is needed, when it is needed.
So not sure this kind of work should be involved in kernel.
Thanks,
Ming
Re: [RFC PATCH 1/7] block: Support creating a struct file from a block device
Quoted 15 lines · Ming Lei
Quoted 14 lines · Demi Marie Obenour
Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Demi Marie Obenour
What do you recommend instead? This solves a real problem for device-mapper users and I am not aware of a better solution.
You could start with explaining the problem and what other methods you tried that failed. In the end it's not my job to fix your problem.
I’m working on a “block not-script” (Xen block device hotplug script written in C) for Qubes OS. The current hotplug script is a shell script that takes a global lock, which serializes all invocations and significantly slows down VM creation and destruction. My C program avoids this problem.
Quoted 5 lines
Quoted 2 lines
One of the goals of the not-script is to never leak resources, even if it dies with SIGKILL or is never called with the “remove” argument to
If it dies, you still can restart one new instance for handling the device leak by running one simple daemon to monitor if not-script is live.
This requires userspace to maintain state that persists across process restarts, and is also non-compositional. If there was a userspace daemon that was responsible for all block device management in the system, this would be more reasonable, but no such daemon exists. Furthermore, the amount of code required in userspace dwarfs the amount of code my patches add to the kernel, both in size and complexity.
Quoted 6 lines
Quoted 2 lines
destroy the devices it created. Therefore, whenever possible, it relies on automatic destruction of devices that are no longer used. I have
This automatic destruction of devices is supposed to be done in userspace, cause only userspace knows when device is needed, when it is needed.
In my use-case, the last reference to the device is held by the blkback driver in the kernel. More generally, any case where a device is created for a single purpose and should be destroyed when no longer used will benefit from this. Encrypted swap devices are a simple example, as they can be destroyed with a single “swapoff” command.
Signature · 3 lines
Sincerely, Demi Marie Obenour (she/her/hers) Invisible Things Lab
RFC PATCH 2/7 Allow userspace to get an FD to a newly-created DM device
In reply to Demi Marie ObenourThis allows creating a device-mapper device, opening it, and setting it to be deleted when unused in a single atomic operation.
--- drivers/md/dm-ioctl.c | 67 +++++++++++++++++++++++++++++------ include/uapi/linux/dm-ioctl.h | 16 ++++++++- 2 files changed, 72 insertions(+), 11 deletions(-)
drivers/md/dm-ioctl.c +57 -10
@@ -853,9 +853,21 @@ static void __dev_status(struct mapped_device *md, struct dm_ioctl *param)static int dev_create (struct file *filp,struct dm_ioctl *param,size_t param_size) { -int r, m =DM_ANY_MINOR ; +int r, m =DM_ANY_MINOR , fd;struct mapped_device *md; +/* Do not allow unknown flags */ +if (param->flags > (2 *DM_FILE_DESCRIPTOR_FLAG -1 )) +return -EINVAL ; + +/* +* Do not allow creating a device that would just be destroyed +* before the ioctl returns. +*/ +if ((param->flags &DM_DEFERRED_REMOVE ) && + !(param->flags &DM_FILE_DESCRIPTOR_FLAG )) +return -EINVAL ; + r =check_name (param->name);if (r)return r;
@@ -867,20 +879,55 @@ static int dev_create(struct file *filp, struct dm_ioctl *param, size_t param_siif (r)return r; - r =dm_hash_insert (param->name, *param->uuid ? param->uuid :NULL , md); -if (r) { -dm_put (md); -dm_destroy (md); -return r; - } - param->flags &= ~DM_INACTIVE_PRESENT_FLAG ; + r =dm_hash_insert (param->name, *param->uuid ? param->uuid :NULL , md); +if (r) + goto out_put; + +if (param->flags &DM_FILE_DESCRIPTOR_FLAG ) { +struct block_device *bdev =dm_disk (md)->part0; +struct file *file; + + fd =get_unused_fd_flags (O_RDWR |O_CLOEXEC ); +if (fd <0 ) { + r = fd; + goto out_put; + } + + file =blkdev_get_file (bdev,O_RDWR |O_CLOEXEC ,NULL ); +if (IS_ERR (file)) { + r =PTR_ERR (file); + goto out_put_fd; + } + +/* +* Simulate opening the device. The other checks in +* dm_blk_open() are not necessary becuase we have a reference +* to the `struct md`. +*/ +atomic_inc (&md->open_count); +fd_install (fd, file); + param->file_descriptor = fd; + } + +/* +* If userspace requests it, automatically delete the device +* when it is no longer used +*/ +if (param->flags &DM_DEFERRED_REMOVE ) +set_bit (DMF_DEFERRED_REMOVE , &md->flags); +__dev_status (md, param); -dm_put (md); -return 0 ; + +out_put_fd: +put_unused_fd (fd); +out_put: +dm_put (md); +dm_destroy (md); +return r; }/*
include/uapi/linux/dm-ioctl.h +15 -1
@@ -136,7 +136,13 @@ struct dm_ioctl {* For output, the ioctls return the event number, not the cookie. */ __u32 event_nr;/* in/out */ -__u32 padding; + +union { +/* Padding for named devices */ +__u32 padding; +/* For anonymous devices, this is a file descriptor. */ +__u32 file_descriptor; + };__u64 dev;/* in/out */
@@ -382,4 +388,12 @@ enum {*/ #define DM_IMA_MEASUREMENT_FLAG (1 << 19)/* In */ +/* +* If set in a DM_DEV_CREATE ioctl(), sets the file_descriptor field +* to a valid file descriptor. This can be combined with DM_DEFERRED_REMOVE +* to cause the device to be destroyed when the file descriptor is closed +* and is otherwise unused. +*/ +#define DM_FILE_DESCRIPTOR_FLAG (1 << 20)/* In */ +#endif /* _LINUX_DM_IOCTL_H */
Signature · 3 lines
Sincerely, Demi Marie Obenour (she/her/hers) Invisible Things Lab
RFC PATCH 3/7 Implement diskseq checks in blkback
In reply to Demi Marie ObenourFrom: Demi Marie Obenour <[email protected]>
This allows specifying a disk sequence number in XenStore. If it does not match the disk sequence number of the underlying device, the device will not be exported and a warning will be logged. Userspace can use this to eliminate race conditions due to major/minor number reuse. Older kernels will ignore this, so it is safe for userspace to set it unconditionally.
This also makes physical-device parsing stricter. I do not believe this will break any extant userspace tools.
--- drivers/block/xen-blkback/xenbus.c | 137 +++++++++++++++++++++-------- 1 file changed, 100 insertions(+), 37 deletions(-)
drivers/block/xen-blkback/xenbus.c +100 -37
@@ -24,6 +24,7 @@ struct backend_info {struct xenbus_watch backend_watch;unsigned major;unsigned minor; +unsigned long long diskseq;char *mode; };
@@ -479,7 +480,7 @@ static void xen_vbd_free(struct xen_vbd *vbd)static int xen_vbd_create (struct xen_blkif *blkif,blkif_vdev_t handle,unsigned major,unsigned minor,int readonly, -int cdrom) +bool cdrom,u64 diskseq) {struct xen_vbd *vbd;struct block_device *bdev;
@@ -507,6 +508,25 @@ static int xen_vbd_create(struct xen_blkif *blkif, blkif_vdev_t handle,xen_vbd_free (vbd);return -ENOENT ; } + +if (diskseq) { +struct gendisk *disk = bdev->bd_disk; +if (unlikely (disk ==NULL )) { +pr_err ("xen_vbd_create: device %08x has no gendisk\n" , + vbd->pdevice); +xen_vbd_free (vbd); +return -EFAULT ; + } + +if (unlikely (disk->diskseq != diskseq)) { +pr_warn ("xen_vbd_create: device %08x has incorrect sequence " +"number 0x%llx (expected 0x%llx)\n" , + vbd->pdevice, disk->diskseq, diskseq); +xen_vbd_free (vbd); +return -ENODEV ; + } + } + vbd->size =vbd_sz (vbd);if (cdrom ||disk_to_cdi (vbd->bdev->bd_disk))
@@ -690,6 +710,55 @@ static int xen_blkbk_probe(struct xenbus_device *dev,return err; } +static bool read_physical_device (struct xenbus_device *dev, +unsigned long long *diskseq, +unsigned *major,unsigned *minor) +{ +char *physical_device, *problem; +int i, physical_device_length; +char junk; + + physical_device =xenbus_read (XBT_NIL , dev->nodename,"physical-device" , + &physical_device_length); + +if (IS_ERR (physical_device)) { +int err =PTR_ERR (physical_device); +/* +* Since this watch will fire once immediately after it is +* registered, we expect "does not exist" errors. Ignore +* them and wait for the hotplug scripts. +*/ +if (unlikely (!XENBUS_EXIST_ERR (err))) +xenbus_dev_fatal (dev, err,"reading physical-device" ); +return false; + } + +for (i =0 ; i < physical_device_length; ++i) +if (unlikely (physical_device[i] <=0x20 || physical_device[i] >=0x7F )) { + problem ="bad byte in physical-device" ; + goto fail; + } + +if (sscanf (physical_device,"%16llx@%8x:%8x%c" , + diskseq, major, minor, &junk) ==3 ) { +if (*diskseq ==0 ) { + problem ="diskseq 0 is invalid" ; + goto fail; + } + }else if (sscanf (physical_device,"%8x:%8x%c" , major, minor, &junk) ==2 ) { + *diskseq =0 ; + }else { + problem ="invalid physical-device" ; + goto fail; + } +kfree (physical_device); +return true; +fail: +kfree (physical_device); +xenbus_dev_fatal (dev, -EINVAL , problem); +return false; +} +/* * Callback received when the hotplug scripts have placed the physical-device * node. Read it and the mode node, and create a vbd. If the frontend is
@@ -707,28 +776,17 @@ static void backend_changed(struct xenbus_watch *watch,int cdrom =0 ;unsigned long handle;char *device_type; +unsigned long long diskseq;pr_debug ("%s %p %d\n" , __func__, dev, dev->otherend_id); - - err =xenbus_scanf (XBT_NIL , dev->nodename,"physical-device" ,"%x:%x" , - &major, &minor); -if (XENBUS_EXIST_ERR (err)) { -/* -* Since this watch will fire once immediately after it is -* registered, we expect this. Ignore it, and wait for the -* hotplug scripts. -*/ +if (!read_physical_device (dev, &diskseq, &major, &minor))return ; - } -if (err !=2 ) { -xenbus_dev_fatal (dev, err,"reading physical-device" ); -return ; - } -if (be->major | be->minor) { -if (be->major != major || be->minor != minor) -pr_warn ("changing physical device (from %x:%x to %x:%x) not supported.\n" , - be->major, be->minor, major, minor); +if (be->major | be->minor | be->diskseq) { +if (be->major != major || be->minor != minor || be->diskseq != diskseq) +pr_warn ("changing physical device (from %x:%x:%llx to %x:%x:%llx)" +" not supported.\n" , + be->major, be->minor, be->diskseq, major, minor, diskseq);return ; }
@@ -756,29 +814,34 @@ static void backend_changed(struct xenbus_watch *watch, be->major = major; be->minor = minor; + be->diskseq = diskseq; err =xen_vbd_create (be->blkif, handle, major, minor, - !strchr (be->mode,'w' ), cdrom); - -if (err) -xenbus_dev_fatal (dev, err,"creating vbd structure" ); -else { - err =xenvbd_sysfs_addif (dev); -if (err) { -xen_vbd_free (&be->blkif->vbd); -xenbus_dev_fatal (dev, err,"creating sysfs entries" ); - } - } + !strchr (be->mode,'w' ), cdrom, diskseq);if (err) { -kfree (be->mode); - be->mode =NULL ; - be->major =0 ; - be->minor =0 ; - }else { -/* We're potentially connected now */ -xen_update_blkif_status (be->blkif); +xenbus_dev_fatal (dev, err,"creating vbd structure" ); + goto fail; } + + err =xenvbd_sysfs_addif (dev); +if (err) { +xenbus_dev_fatal (dev, err,"creating sysfs entries" ); + goto free_vbd; + } + +/* We're potentially connected now */ +xen_update_blkif_status (be->blkif); +return ; + +free_vbd: +xen_vbd_free (&be->blkif->vbd); +fail: +kfree (be->mode); + be->mode =NULL ; + be->major =0 ; + be->minor =0 ; + be->diskseq =0 ; }/*
Signature · 3 lines
Sincerely, Demi Marie Obenour (she/her/hers) Invisible Things Lab
RFC PATCH 4/7 Increment diskseq when releasing a loop device
In reply to Demi Marie ObenourThis ensures that userspace is aware that the device may now point to something else.
--- block/genhd.c | 1 + drivers/block/loop.c | 6 ++++++ 2 files changed, 7 insertions(+)
block/genhd.c +1 -0
@@ -1490,3 +1490,4 @@ void inc_diskseq(struct gendisk *disk) { disk->diskseq =atomic64_inc_return (&diskseq); } +EXPORT_SYMBOL (inc_diskseq);
drivers/block/loop.c +6 -0
@@ -1205,6 +1205,12 @@ static void __loop_clr_fd(struct loop_device *lo, bool release)if (!part_shift)set_bit (GD_SUPPRESS_PART_SCAN , &lo->lo_disk->state);mutex_lock (&lo->lo_mutex); + +/* +* Increment the disk sequence number, so that userspace knows this +* device now points to something else. +*/ +inc_diskseq (lo->lo_disk); lo->lo_state = Lo_unbound;mutex_unlock (&lo->lo_mutex);
Signature · 3 lines
Sincerely, Demi Marie Obenour (she/her/hers) Invisible Things Lab
Re: [RFC PATCH 4/7] Increment diskseq when releasing a loop device
Quoted 2 lines · Demi Marie Obenour
This ensures that userspace is aware that the device may now point to something else.
The subject is wong, this also does two things to two different subystems, not of which is mentioned in the subject.
RFC PATCH 6/7 Minor blkback cleanups
In reply to Demi Marie ObenourNo functional change intended.
--- drivers/block/xen-blkback/blkback.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
drivers/block/xen-blkback/blkback.c +5 -3
@@ -1238,6 +1238,8 @@ static int dispatch_rw_block_io(struct xen_blkif_ring *ring, nseg = req->operation ==BLKIF_OP_INDIRECT ? req->u.indirect.nr_segments : req->u.rw.nr_segments; + BUILD_BUG_ON(offsetof(struct blkif_request, u.rw.id) != 8); + BUILD_BUG_ON(offsetof(struct blkif_request, u.indirect.id) != 8); if (unlikely(nseg == 0 && operation_flags != REQ_PREFLUSH) || unlikely((req->operation != BLKIF_OP_INDIRECT) && (nseg > BLKIF_MAX_SEGMENTS_PER_REQUEST)) ||
@@ -1261,13 +1263,13 @@ static int dispatch_rw_block_io(struct xen_blkif_ring *ring, preq.sector_number = req->u.rw.sector_number;for (i =0 ; i < nseg; i++) { pages[i]->gref = req->u.rw.seg[i].gref; - seg[i].nsec = req->u.rw.seg[i].last_sect - - req->u.rw.seg[i].first_sect +1 ; - seg[i].offset = (req->u.rw.seg[i].first_sect <<9 );if ((req->u.rw.seg[i].last_sect >= (XEN_PAGE_SIZE >>9 )) || (req->u.rw.seg[i].last_sect < req->u.rw.seg[i].first_sect)) goto fail_response; + seg[i].nsec = req->u.rw.seg[i].last_sect - + req->u.rw.seg[i].first_sect +1 ; + seg[i].offset = (req->u.rw.seg[i].first_sect <<9 ); preq.nr_sects += seg[i].nsec; } }else {
Signature · 3 lines
Sincerely, Demi Marie Obenour (she/her/hers) Invisible Things Lab
RFC PATCH 7/7 xen/blkback: Inform userspace that device has been opened
In reply to Demi Marie ObenourThis allows userspace to use block devices with delete-on-close behavior, which is necessary to ensure virtual devices (such as loop or device-mapper devices) are cleaned up automatically. Protocol details are included in comments.
--- drivers/block/xen-blkback/xenbus.c | 34 ++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
drivers/block/xen-blkback/xenbus.c +34 -0
@@ -3,6 +3,19 @@Copyright (C) 2005 Rusty Russell <[email protected]> Copyright (C) 2005 XenSource Ltd +In addition to the XenStore nodes required by the Xen block device +specification, this implementation of blkback uses a new XenStore +node: "opened". blkback sets "opened" to "0" before the hotplug script +is called. Once the device node has been opened, blkback sets "opened" +to "1". + +"opened" is used exclusively by userspace. It serves two purposes: + +1. It tells userspace that diskseq@major:minor syntax for "physical-device" is +supported. +2. It tells userspace that it can wait for "opened" to be set to 1. Once +"opened" is 1, blkback has a reference to the device, so userspace doesn't +need to keep one. */
@@ -698,6 +711,14 @@ static int xen_blkbk_probe(struct xenbus_device *dev,if (err)pr_warn ("%s write out 'max-ring-page-order' failed\n" , __func__); +/* +* This informs userspace that the "opened" node will be set to "1" when +* the device has been opened successfully. +*/ + err =xenbus_write (XBT_NIL , dev->nodename,"opened" ,"0" ); +if (err) + goto fail; + err =xenbus_switch_state (dev, XenbusStateInitWait);if (err) goto fail;
@@ -824,6 +845,19 @@ static void backend_changed(struct xenbus_watch *watch, goto fail; } +/* +* Tell userspace that the device has been opened and that blkback has a +* reference to it. Userspace can then close the device or mark it as +* delete-on-close, knowing that blkback will keep the device open as +* long as necessary. +*/ + err =xenbus_write (XBT_NIL , dev->nodename,"opened" ,"1" ); +if (err) { +xenbus_dev_fatal (dev, err,"%s: notifying userspace device has been opened" , + dev->nodename); + goto free_vbd; + } + err =xenvbd_sysfs_addif (dev);if (err) {xenbus_dev_fatal (dev, err,"creating sysfs entries" );
Signature · 3 lines
Sincerely, Demi Marie Obenour (she/her/hers) Invisible Things Lab
Mike Snitzer
maintainerQuoted 2 lines · Demi Marie Obenour
This work aims to allow userspace to create and destroy block devices in a race-free and leak-free way,
"race-free and leak-free way" implies there both races and leaks in existing code. You're making claims that are likely very specific to your Xen use-case. Please explain more carefully.
Quoted 10 lines
and to allow them to be exposed to
other Xen VMs via blkback without leaks or races. It’s marked as RFC for a few reasons:- The code has been only lightly tested. It might be unstable or insecure.
- The
DM_DEV_CREATEioctl gains a new flag. Unknown flags were previously ignored, so this could theoretically break buggy userspace tools.
Not seeing a reason that type of DM change is needed. If you feel strongly about it send a separate patch and we can discuss it.
Quoted 2 lines
- I have no idea if I got the block device reference counting and locking correct.
Your headers and justifcation for this line of work are really way too terse. Please take the time to clearly make the case for your changes in both the patch headers and code.
Mike
--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
Quoted 9 lines · Mike Snitzer
Quoted 2 lines · Demi Marie Obenour
This work aims to allow userspace to create and destroy block devices in a race-free and leak-free way,
"race-free and leak-free way" implies there both races and leaks in existing code. You're making claims that are likely very specific to your Xen use-case. Please explain more carefully.
Will do in v2.
Quoted 7 lines
Quoted 7 lines
and to allow them to be exposed to
other Xen VMs via blkback without leaks or races. It’s marked as RFC for a few reasons:- The code has been only lightly tested. It might be unstable or insecure.
Quoted 6 lines
Quoted 3 lines
- The
DM_DEV_CREATEioctl gains a new flag. Unknown flags were previously ignored, so this could theoretically break buggy userspace tools.Not seeing a reason that type of DM change is needed. If you feel strongly about it send a separate patch and we can discuss it.
Patch 2/7 is the diskseq change. v2 will contain a revised and tested version with a greatly expanded commit message.
Quoted 6 lines
Quoted 2 lines
- I have no idea if I got the block device reference counting and locking correct.
Your headers and justifcation for this line of work are really way too terse. Please take the time to clearly make the case for your changes in both the patch headers and code.
I will expand the commit message in v2, but I am not sure what you want me to add to the code comments. Would you mind explaining?
Signature · 3 lines
Sincerely, Demi Marie Obenour (she/her/hers) Invisible Things Lab
Mike Snitzer
maintainerQuoted 24 lines · Demi Marie Obenour
Quoted 9 lines · Mike Snitzer
Quoted 2 lines · Demi Marie Obenour
This work aims to allow userspace to create and destroy block devices in a race-free and leak-free way,
"race-free and leak-free way" implies there both races and leaks in existing code. You're making claims that are likely very specific to your Xen use-case. Please explain more carefully.
Will do in v2.
Quoted 11 lines
Quoted 10 lines
and to allow them to be exposed to
other Xen VMs via blkback without leaks or races. It’s marked as RFC for a few reasons:- The code has been only lightly tested. It might be unstable or insecure.
- The
DM_DEV_CREATEioctl gains a new flag. Unknown flags were previously ignored, so this could theoretically break buggy userspace tools.
Quoted 5 lines
Quoted 2 lines
Not seeing a reason that type of DM change is needed. If you feel strongly about it send a separate patch and we can discuss it.
Patch 2/7 is the diskseq change. v2 will contain a revised and tested version with a greatly expanded commit message.
I'm aware that 2/7 is where you make the DM change to disallow unknown flags, what I'm saying is I don't see a reason for that change.
Certainly doesn't look to be a requirement for everything else in that patch.
So send a separate patch, but I'm inclined to _not_ accept it because it does potentially break some userspace.
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
- I have no idea if I got the block device reference counting and locking correct.
Your headers and justifcation for this line of work are really way too terse. Please take the time to clearly make the case for your changes in both the patch headers and code.
I will expand the commit message in v2, but I am not sure what you want me to add to the code comments. Would you mind explaining?
Nothing specific about code, was just a general reminder (based on how terse the 2/7 header was).
Mike
--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
Quoted 27 lines · Mike Snitzer
Quoted 24 lines · Demi Marie Obenour
Quoted 9 lines · Mike Snitzer
Quoted 2 lines · Demi Marie Obenour
This work aims to allow userspace to create and destroy block devices in a race-free and leak-free way,
"race-free and leak-free way" implies there both races and leaks in existing code. You're making claims that are likely very specific to your Xen use-case. Please explain more carefully.
Will do in v2.
Quoted 11 lines
Quoted 10 lines
and to allow them to be exposed to
other Xen VMs via blkback without leaks or races. It’s marked as RFC for a few reasons:- The code has been only lightly tested. It might be unstable or insecure.
- The
DM_DEV_CREATEioctl gains a new flag. Unknown flags were previously ignored, so this could theoretically break buggy userspace tools.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Not seeing a reason that type of DM change is needed. If you feel strongly about it send a separate patch and we can discuss it.
Patch 2/7 is the diskseq change. v2 will contain a revised and tested version with a greatly expanded commit message.
I'm aware that 2/7 is where you make the DM change to disallow unknown flags, what I'm saying is I don't see a reason for that change.
Thanks for the clarification.
Quoted 2 lines
Certainly doesn't look to be a requirement for everything else in that patch.
Indeed it is not. I will make it a separate patch.
Quoted 2 lines
So send a separate patch, but I'm inclined to _not_ accept it because it does potentially break some userspace.
Is it okay to add DM_FILE_DESCRIPTOR_FLAG (with the same meaning as in 2/7) _without_ rejecting unknown flags? The same patch would bump the minor version number, so userspace would still be able to tell if the kernel supported DM_FILE_DESCRIPTOR_FLAG. If you wanted, I could ignore DM_FILE_DESCRIPTOR_FLAG unless the minor number passed by userspace is sufficiently recent.
Another option would be to make userspace opt-in to strict parameter checking by passing 5 as the major version instead of 4. Userspace programs that passed 4 would get the old behavior, while userspace programs that passed 5 would get strict parameter checking and be able to use new features such as DM_FILE_DESCRIPTOR_FLAG.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
- I have no idea if I got the block device reference counting and locking correct.
Your headers and justifcation for this line of work are really way too terse. Please take the time to clearly make the case for your changes in both the patch headers and code.
Quoted 7 lines
Quoted 2 lines
I will expand the commit message in v2, but I am not sure what you want me to add to the code comments. Would you mind explaining?
Nothing specific about code, was just a general reminder (based on how terse the 2/7 header was).
Mike
Thanks for the feedback!
Signature · 3 lines
Sincerely, Demi Marie Obenour (she/her/hers) Invisible Things Lab