loop: defer the queue limits clear to a workqueue
PATCH v3 1/1 loop: defer the queue limits clear to a workqueue
From: Tao Cui <[email protected]>
loop_clear_limits() calls queue_limits_commit_update() directly from the loop workqueue that processes the request. That does a non-atomic struct assignment to q->limits without freezing the queue, which races with lockless readers of q->limits on other CPUs - bio splitting reads max_hw_sectors, the discard path reads max_hw_discard_sectors - and can let them observe torn values. The trigger is a discard or write-zeroes request on a loop device whose backing file does not support the corresponding fallocate operation.
The code already has an XXX comment saying this should move to a workqueue. Do that: schedule a work item on the system workqueue, where it is safe to freeze the queue around the limits update. Accumulate pending modes in lo->clear_limits_mode so that failures between scheduling and execution of the work item are not lost. A rebinding of the device drops the accumulated modes and invalidates an already scheduled work item, so a stale clear cannot hit the new backing file. The work item is cancelled before the device is freed.
---
Changes since v2:
- Skip the clear when the device was rebound since the work item was scheduled: the modes used to be captured before the freeze, so a LOOP_CHANGE_FD completing in between could apply the old modes to the new backing file. The rebind generation is now checked with the queue frozen, which excludes loop_change_fd() because it assigns the new backing file under the same freeze, so a rebind cannot slip in between the check and the commit. Cancelling from loop_assign_backing_file() would instead deadlock on the freeze held by loop_change_fd().
- Also bump the rebind generation from __loop_clr_fd(): unbinding does not go through loop_assign_backing_file(), so a work item scheduled before the last close could otherwise commit a stale clear to the queue limits of the unbound device.
Changes since v1:
- Make clear_limits_mode atomic_t: loop workers can hit the |= concurrently.
- Reset clear_limits_mode when assigning a new backing file, so stale modes do not clear limits of the new file.
- Cancel the work item from loop_remove() before del_gendisk():
the queue can already be in RCU-delayed freeing when lo_free_disk() cancels it.
Tested on x86-64 (qemu, vfat-backed loop device): 30s discard and reconfigure loop exercises the clear path 577 times, no torn sysfs reads, no difference against the unpatched kernel.
--- drivers/block/loop.c | 64 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 50 insertions(+), 14 deletions(-)
drivers/block/loop.c +50 -14
@@ -67,6 +67,10 @@ struct loop_device {struct list_head rootcg_cmd_list;struct list_head idle_worker_list;struct rb_root worker_tree; +struct work_struct clear_limits_work; +atomic_t clear_limits_mode; +atomic_t rebind_gen; +atomic_t clear_limits_gen;struct timer_list timer;bool sysfs_inited;
@@ -222,26 +226,44 @@ static void loop_set_size(struct loop_device *lo, loff_t size)kobject_uevent (&disk_to_dev (lo->lo_disk)->kobj,KOBJ_CHANGE ); } -static void loop_clear_limits (struct loop_device *lo,int mode) +static void loop_clear_limits_workfn (struct work_struct *work) { +struct loop_device *lo = + container_of(work, struct loop_device, clear_limits_work);struct queue_limits lim =queue_limits_start_update (lo->lo_queue); - -if (mode &FALLOC_FL_ZERO_RANGE ) - lim.max_write_zeroes_sectors =0 ; - -if (mode &FALLOC_FL_PUNCH_HOLE ) { - lim.max_hw_discard_sectors =0 ; - lim.discard_granularity =0 ; - } +unsigned int memflags; +int mode =0 ;/* -* XXX: this updates the queue limits without freezing the queue, which -* is against the locking protocol and dangerous. But we can't just -* freeze the queue as we're inside the ->queue_rq method here. So this -* should move out into a workqueue unless we get the file operations to -* advertise if they support specific fallocate operations. +* Commit the unmodified limits if the device was rebound since +* the work item was scheduled. The freeze excludes a rebind +* through loop_change_fd(), which assigns the new backing file +* under the freeze; the other rebinding paths, loop_configure() +* and __loop_clr_fd(), do not freeze the queue, but they bump +* rebind_gen and reset clear_limits_mode themselves, so the +* generation check and the atomic_xchg() below cover them. */ + memflags =blk_mq_freeze_queue (lo->lo_queue); +if (atomic_read (&lo->clear_limits_gen) ==atomic_read (&lo->rebind_gen)) { + mode =atomic_xchg (&lo->clear_limits_mode,0 ); + +if (mode &FALLOC_FL_ZERO_RANGE ) + lim.max_write_zeroes_sectors =0 ; + +if (mode &FALLOC_FL_PUNCH_HOLE ) { + lim.max_hw_discard_sectors =0 ; + lim.discard_granularity =0 ; + } + }queue_limits_commit_update (lo->lo_queue, &lim); +blk_mq_unfreeze_queue (lo->lo_queue, memflags); +} + +static void loop_clear_limits (struct loop_device *lo,int mode) +{ +atomic_set (&lo->clear_limits_gen,atomic_read (&lo->rebind_gen)); +atomic_or (mode, &lo->clear_limits_mode); +schedule_work (&lo->clear_limits_work); } static int lo_fallocate(struct loop_device *lo, struct request *rq, loff_t pos,
@@ -516,6 +538,8 @@ static int loop_validate_file(struct file *file, struct block_device *bdev)static void loop_assign_backing_file (struct loop_device *lo,struct file *file) { lo->lo_backing_file = file; +atomic_inc (&lo->rebind_gen); +atomic_set (&lo->clear_limits_mode,0 ); lo->old_gfp_mask =mapping_gfp_mask (file->f_mapping);mapping_set_gfp_mask (file->f_mapping, lo->old_gfp_mask & ~(__GFP_IO | __GFP_FS));
@@ -1144,6 +1168,12 @@ static void __loop_clr_fd(struct loop_device *lo)spin_lock_irq (&lo->lo_lock); filp = lo->lo_backing_file; lo->lo_backing_file =NULL ; +/* +* Invalidate any pending clear that was scheduled against the old +* backing file, like loop_assign_backing_file() does on rebind. +*/ +atomic_inc (&lo->rebind_gen); +atomic_set (&lo->clear_limits_mode,0 );spin_unlock_irq (&lo->lo_lock); lo->lo_device =NULL ;
@@ -1781,6 +1811,7 @@ static void lo_free_disk(struct gendisk *disk)destroy_workqueue (lo->workqueue);loop_free_idle_workers (lo, true);timer_shutdown_sync (&lo->timer); +cancel_work_sync (&lo->clear_limits_work);mutex_destroy (&lo->lo_mutex);kfree (lo); }
@@ -2100,6 +2131,7 @@ static int loop_add(int i)spin_lock_init (&lo->lo_lock);spin_lock_init (&lo->lo_work_lock);INIT_WORK (&lo->rootcg_work, loop_rootcg_workfn); +INIT_WORK (&lo->clear_limits_work, loop_clear_limits_workfn);INIT_LIST_HEAD (&lo->rootcg_cmd_list); disk->major =LOOP_MAJOR ; disk->first_minor = i << part_shift;
@@ -2138,6 +2170,10 @@ static int loop_add(int i)static void loop_remove (struct loop_device *lo) { +/* Cancel early: the queue may be in RCU-delayed freeing +* by the time lo_free_disk() runs. */ +cancel_work_sync (&lo->clear_limits_work); +/* Make this loop device unreachable from pathname. */ del_gendisk (lo->lo_disk);blk_mq_free_tag_set (&lo->tag_set);
Signature · 1 line
2.43.0
drivers/block/loop.c struct loop_device { · line 73
Quoted 4 lines · Tao Cui
drivers/block/loop.c +4
-0+struct work_struct clear_limits_work; +atomic_t clear_limits_mode; +atomic_t rebind_gen; +atomic_t clear_limits_gen;
Three new atomic variables? That's too much because it makes it much harder than necessary to verify the code changes. Please change the atomic variables to non-atomic and use a new mutex to serialize accesses to the new variables.
Thanks,
Bart.