fuse: fix io-uring background queue dispatch on request completion
Subsystems (4)
Joanne Koong
maintainerPATCH v2 1/1 fuse: fix io-uring background queue dispatch on request completion
When a background request completes via the io_uring path, the background queue gets flushed to dispatch pending background requests, but this is done before the connection-level background counters (fc->num_background, fc->active_background) are properly accounted, which may reduce effective queue depth to one.
The connection-level counters are decremented in fuse_request_end(), but flush_bg_queue() flushes the /dev/fuse path queue (fc->bg_queue), not the io_uring per-queue bg one, which means pending uring background requests on the queue are never dispatched in this path.
Fix this by accounting the connection-level background counters first before flushing the queue's background queue. Since fuse_request_bg_finish() clears FR_BACKGROUND, fuse_request_end() will skip the background cleanup branch entirely, which avoids any double-decrements; it will call the wake_up(&req->waitq) branch but this is effectively a no-op as background requests have no waiters on req->waitq.
---
v1: https://lore.kernel.org/linux-fsdevel/[email protected]/
v1 -> v2:
* change commit message wording (Bernd)
--- fs/fuse/dev.c | 41 ++++++++++++++++++++++++----------------- fs/fuse/dev_uring.c | 1 + fs/fuse/fuse_dev_i.h | 1 + 3 files changed, 26 insertions(+), 17 deletions(-)
fs/fuse/dev.c +24 -17
@@ -447,6 +447,29 @@ static void flush_bg_queue(struct fuse_conn *fc) } } +void fuse_request_bg_finish (struct fuse_conn *fc,struct fuse_req *req) +{ +lockdep_assert_held (&fc->bg_lock); + +clear_bit (FR_BACKGROUND , &req->flags); +if (fc->num_background == fc->max_background) { + fc->blocked =0 ; +wake_up (&fc->blocked_waitq); + }else if (!fc->blocked) { +/* +* Wake up next waiter, if any. It's okay to use +* waitqueue_active(), as we've already synced up +* fc->blocked with waiters with the wake_up() call +* above. +*/ +if (waitqueue_active (&fc->blocked_waitq)) +wake_up (&fc->blocked_waitq); + } + + fc->num_background--; + fc->active_background--; +} +/* * This function is called when a request is finished. Either a reply * has arrived or it was aborted (and not yet sent) or some error
@@ -479,23 +502,7 @@ void fuse_request_end(struct fuse_req *req)WARN_ON (test_bit (FR_SENT , &req->flags));if (test_bit (FR_BACKGROUND , &req->flags)) {spin_lock (&fc->bg_lock); -clear_bit (FR_BACKGROUND , &req->flags); -if (fc->num_background == fc->max_background) { - fc->blocked =0 ; -wake_up (&fc->blocked_waitq); - }else if (!fc->blocked) { -/* -* Wake up next waiter, if any. It's okay to use -* waitqueue_active(), as we've already synced up -* fc->blocked with waiters with the wake_up() call -* above. -*/ -if (waitqueue_active (&fc->blocked_waitq)) -wake_up (&fc->blocked_waitq); - } - - fc->num_background--; - fc->active_background--; +fuse_request_bg_finish (fc, req);flush_bg_queue (fc);spin_unlock (&fc->bg_lock); }else {
fs/fuse/dev_uring.c +1 -0
@@ -90,6 +90,7 @@ static void fuse_uring_req_end(struct fuse_ring_ent *ent, struct fuse_req *req,if (test_bit (FR_BACKGROUND , &req->flags)) { queue->active_background--;spin_lock (&fc->bg_lock); +fuse_request_bg_finish (fc, req);fuse_uring_flush_bg (queue);spin_unlock (&fc->bg_lock); }
fs/fuse/fuse_dev_i.h +1 -0
@@ -59,6 +59,7 @@ unsigned int fuse_req_hash(u64 unique);struct fuse_req *fuse_request_find (struct fuse_pqueue *fpq,u64 unique);void fuse_dev_end_requests (struct list_head *head); +void fuse_request_bg_finish (struct fuse_conn *fc,struct fuse_req *req);void fuse_copy_init (struct fuse_copy_state *cs,bool write,struct iov_iter *iter);
Signature · 1 line
2.52.0
Miklos Szeredi
maintainerQuoted 12 lines · Joanne Koong
When a background request completes via the
io_uringpath, the background queue gets flushed to dispatch pending background requests, but this is done before the connection-level background counters (fc->num_background,fc->active_background) are properly accounted, which may reduce effective queue depth to one.The connection-level counters are decremented in
fuse_request_end(), butflush_bg_queue()flushes the/dev/fusepath queue (fc->bg_queue), not theio_uringper-queue bg one, which means pending uring background requests on the queue are never dispatched in this path.
Quoted 7 lines
Fix this by accounting the connection-level background counters first before flushing the queue's background queue. Since
fuse_request_bg_finish()clearsFR_BACKGROUND,fuse_request_end()will skip the background cleanup branch entirely, which avoids any double-decrements; it will call thewake_up(&req->waitq)branch but this is effectively a no-op as background requests have no waiters onreq->waitq.
Does this guarantee progress if there are still requests on fc->bg_queue at the point when ring becomes ready?
Seems so, because there must be at least one background request on the regular request queues if bg_queue is non-empty, and when that is finished, a new one will be put on the pending queue, and so on until bg_queue becomes empty.
Maybe add a comment about this subtlety?
Thanks,
Miklos
Bernd Schubert
maintainerQuoted 21 lines · Miklos Szeredi
Quoted 12 lines · Joanne Koong
When a background request completes via the
io_uringpath, the background queue gets flushed to dispatch pending background requests, but this is done before the connection-level background counters (fc->num_background,fc->active_background) are properly accounted, which may reduce effective queue depth to one.The connection-level counters are decremented in
fuse_request_end(), butflush_bg_queue()flushes the/dev/fusepath queue (fc->bg_queue), not theio_uringper-queue bg one, which means pending uring background requests on the queue are never dispatched in this path.Quoted 7 lines
Fix this by accounting the connection-level background counters first before flushing the queue's background queue. Since
fuse_request_bg_finish()clearsFR_BACKGROUND,fuse_request_end()will skip the background cleanup branch entirely, which avoids any double-decrements; it will call thewake_up(&req->waitq)branch but this is effectively a no-op as background requests have no waiters onreq->waitq.
Quoted 2 lines
Does this guarantee progress if there are still requests on
fc->bg_queueat the point when ring becomes ready?
Request allocation is still blocked until the ring is set to ready - there should be no background requests at all.
There were a couple of issues with dynamic switching from /dev/fuse to fuse-io-uring, a really hard one was a lock order change. I think once we have reduced queues merged, I'm to create an additional patch set which allow allow distribution of requests between queues. The initial series had that, but Joanne had concerns. Anyway, once we have distribution between queues, we can avoid the global bg lock and global bg limit and set that per queue. If I remember right, that would solve the lock order issue. Although I forgot the exact details. Wish I would have creates notes or a blog - too late now :(
Quoted 5 lines
Seems so, because there must be at least one background request on the regular request queues if
bg_queueis non-empty, and when that is finished, a new one will be put on the pending queue, and so on untilbg_queuebecomes empty.
Yeah, in order to avoid stalls, ring queues always allow one bg request, ignoring the global limit.
Quoted 2 lines
Maybe add a comment about this subtlety?
Where should that comment be?
Thanks,
Bernd
Quoted 20 lines · Joanne Koong
When a background request completes via the
io_uringpath, the background queue gets flushed to dispatch pending background requests, but this is done before the connection-level background counters (fc->num_background,fc->active_background) are properly accounted, which may reduce effective queue depth to one.The connection-level counters are decremented in
fuse_request_end(), butflush_bg_queue()flushes the/dev/fusepath queue (fc->bg_queue), not theio_uringper-queue bg one, which means pending uring background requests on the queue are never dispatched in this path.Fix this by accounting the connection-level background counters first before flushing the queue's background queue. Since
fuse_request_bg_finish()clearsFR_BACKGROUND,fuse_request_end()will skip the background cleanup branch entirely, which avoids any double-decrements; it will call thewake_up(&req->waitq)branch but this is effectively a no-op as background requests have no waiters onreq->waitq.
Quoted 4 lines
Reviewed-by Bernd Schubert <[email protected]>Fixes 857b0263f30e ("fuse: Allow to queue bg requests through io-uring")Signed-off-by Joanne Koong <[email protected]>
Applied, thanks.
Miklos