io_uring: ensure REQ_F_ISREG is set async offload
Jens Axboe
maintainerPATCH 1/1 io_uring: ensure REQ_F_ISREG is set async offload
If we're offloading requests directly to io-wq because IOSQE_ASYNC was set in the sqe, we can miss hashing writes appropriately because we haven't set REQ_F_ISREG yet. This can cause a performance regression with buffered writes, as io-wq then no longer correctly serializes writes to that file.
Ensure that we set the flags in io_prep_async_work(), which will cause the io-wq work item to be hashed appropriately.
---
Yin, this is the 5.20 version. Once this lands upstream, I'll get the 5.19/18 variant sent to stable as well. Thanks!
io_uring/io_uring.c +3 -0
@@ -406,6 +406,9 @@ static void io_prep_async_work(struct io_kiocb *req)if (req->flags &REQ_F_FORCE_ASYNC ) req->work.flags |=IO_WQ_WORK_CONCURRENT ; +if (req->file && !io_req_ffs_set (req)) + req->flags |=io_file_get_flags (req->file) <<REQ_F_SUPPORT_NOWAIT_BIT ; +if (req->flags &REQ_F_ISREG ) {if (def->hash_reg_file || (ctx->flags &IORING_SETUP_IOPOLL ))io_wq_hash_work (&req->work,file_inode (req->file));
io_uring/io_uring.h +5 -0
@@ -41,6 +41,11 @@ struct file *io_file_get_normal(struct io_kiocb *req, int fd);struct file *io_file_get_fixed (struct io_kiocb *req,int fd,unsigned issue_flags); +static inline bool io_req_ffs_set (struct io_kiocb *req) +{ +return req->flags &REQ_F_FIXED_FILE ; +} +bool io_is_uring_fops (struct file *file);bool io_alloc_async_data (struct io_kiocb *req);void io_req_task_work_add (struct io_kiocb *req);
io_uring/rw.c +0 -5
@@ -647,11 +647,6 @@ static bool need_read_all(struct io_kiocb *req)S_ISBLK (file_inode (req->file)->i_mode); } -static inline bool io_req_ffs_set (struct io_kiocb *req) -{ -return req->flags &REQ_F_FIXED_FILE ; -} -static int io_rw_init_file (struct io_kiocb *req,fmode_t mode) {struct io_rw *rw =io_kiocb_to_cmd (req);
Signature · 1 line
Jens Axboe
Quoted 6 lines · Jens Axboe
If we're offloading requests directly to io-wq because
IOSQE_ASYNCwas set in the sqe, we can miss hashing writes appropriately because we haven't setREQ_F_ISREGyet. This can cause a performance regression with buffered writes, as io-wq then no longer correctly serializes writes to that file.
Quoted 6 lines
Ensure that we set the flags in
io_prep_async_work(), which will cause the io-wq work item to be hashed appropriately.Fixes 584b0180f0f4 ("io_uring: move read/write file prep state into actual opcode handler")Reported-and-tested-by Yin Fengwei <[email protected]>
This issue is reported by (from the original report):
If you fix the issue, kindly add following tag
Regards
Yin, Fengwei
Quoted 54 lines
Signed-off-by Jens Axboe <[email protected]>---
Yin, this is the 5.20 version. Once this lands upstream, I'll get the 5.19/18 variant sent to stable as well. Thanks!
io_uring/io_uring.c +3
-0@@ -406,6 +406,9 @@ static void io_prep_async_work(struct io_kiocb *req)if (req->flags &REQ_F_FORCE_ASYNC ) req->work.flags |=IO_WQ_WORK_CONCURRENT ; +if (req->file && !io_req_ffs_set (req)) + req->flags |=io_file_get_flags (req->file) <<REQ_F_SUPPORT_NOWAIT_BIT ; +if (req->flags &REQ_F_ISREG ) {if (def->hash_reg_file || (ctx->flags &IORING_SETUP_IOPOLL ))io_wq_hash_work (&req->work,file_inode (req->file));io_uring/io_uring.h +5
-0@@ -41,6 +41,11 @@ struct file *io_file_get_normal(struct io_kiocb *req, int fd);struct file *io_file_get_fixed (struct io_kiocb *req,int fd,unsigned issue_flags); +static inline bool io_req_ffs_set (struct io_kiocb *req) +{ +return req->flags &REQ_F_FIXED_FILE ; +} +bool io_is_uring_fops (struct file *file);bool io_alloc_async_data (struct io_kiocb *req);void io_req_task_work_add (struct io_kiocb *req);io_uring/rw.c +0
-5@@ -647,11 +647,6 @@ static bool need_read_all(struct io_kiocb *req)S_ISBLK (file_inode (req->file)->i_mode); } -static inline bool io_req_ffs_set (struct io_kiocb *req) -{ -return req->flags &REQ_F_FIXED_FILE ; -} -static int io_rw_init_file (struct io_kiocb *req,fmode_t mode) {struct io_rw *rw =io_kiocb_to_cmd (req);
Jens Axboe
maintainerQuoted 11 lines · Yin Fengwei
Quoted 9 lines · Jens Axboe
If we're offloading requests directly to io-wq because
IOSQE_ASYNCwas set in the sqe, we can miss hashing writes appropriately because we haven't setREQ_F_ISREGyet. This can cause a performance regression with buffered writes, as io-wq then no longer correctly serializes writes to that file.Ensure that we set the flags in
io_prep_async_work(), which will cause the io-wq work item to be hashed appropriately.
Quoted 7 lines
Quoted 3 lines
Fixes 584b0180f0f4 ("io_uring: move read/write file prep state into actual opcode handler")Reported-and-tested-by Yin Fengwei <[email protected]>This issue is reported by (from the original report):
If you fix the issue, kindly add following tag
Reported-by kernel test robot <[email protected]>
Sorry, I missed that in the original. I'll be rebasing this branch this weekend anyway, I'll try and remember to make the edit.
Signature · 1 line
Jens Axboe