Fix locking for MSG_RING and IOPOLL targets
Jens Axboe
maintainercover Fix locking for MSG_RING and IOPOLL targets
Hi,
First patch is just a prep patch, patch two adds conditional locking for posting CQEs to an IOPOLL based target.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 1/2 io_uring/msg_ring: move double lock/unlock helpers higher up
In preparation for needing them somewhere else, move them and get rid of the unused 'issue_flags' for the unlock side.
No functional changes in this patch.
--- io_uring/msg_ring.c | 47 ++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 24 deletions(-)
io_uring/msg_ring.c +23 -24
@@ -25,6 +25,28 @@ struct io_msg {u32 flags; }; +static void io_double_unlock_ctx (struct io_ring_ctx *octx) +{ +mutex_unlock (&octx->uring_lock); +} + +static int io_double_lock_ctx (struct io_ring_ctx *octx, +unsigned int issue_flags) +{ +/* +* To ensure proper ordering between the two ctxs, we can only +* attempt a trylock on the target. If that fails and we already have +* the source ctx lock, punt to io-wq. +*/ +if (!(issue_flags &IO_URING_F_UNLOCKED )) { +if (!mutex_trylock (&octx->uring_lock)) +return -EAGAIN ; +return 0 ; + } +mutex_lock (&octx->uring_lock); +return 0 ; +} +void io_msg_ring_cleanup (struct io_kiocb *req) { struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg);
@@ -77,29 +99,6 @@ static int io_msg_ring_data(struct io_kiocb *req)return -EOVERFLOW ; } -static void io_double_unlock_ctx (struct io_ring_ctx *octx, -unsigned int issue_flags) -{ -mutex_unlock (&octx->uring_lock); -} - -static int io_double_lock_ctx (struct io_ring_ctx *octx, -unsigned int issue_flags) -{ -/* -* To ensure proper ordering between the two ctxs, we can only -* attempt a trylock on the target. If that fails and we already have -* the source ctx lock, punt to io-wq. -*/ -if (!(issue_flags &IO_URING_F_UNLOCKED )) { -if (!mutex_trylock (&octx->uring_lock)) -return -EAGAIN ; -return 0 ; - } -mutex_lock (&octx->uring_lock); -return 0 ; -} -static struct file *io_msg_grab_file (struct io_kiocb *req,unsigned int issue_flags) { struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg);
@@ -148,7 +147,7 @@ static int io_msg_install_complete(struct io_kiocb *req, unsigned int issue_flagif (!io_post_aux_cqe (target_ctx, msg->user_data, msg->len,0 )) ret = -EOVERFLOW ; out_unlock: -io_double_unlock_ctx (target_ctx, issue_flags); +io_double_unlock_ctx (target_ctx);return ret; }
Signature · 1 line
2.39.0
Jens Axboe
maintainerPATCH 2/2 io_uring/msg_ring: fix missing lock on overflow for IOPOLL
In reply to Jens AxboeIf the target ring is configured with IOPOLL, then we always need to hold the target ring uring_lock before posting CQEs. We could just grab it unconditionally, but since we don't expect many target rings to be of this type, make grabbing the uring_lock conditional on the ring type.
--- io_uring/msg_ring.c | 39 ++++++++++++++++++++++++++++++--------- 1 file changed, 30 insertions(+), 9 deletions(-)
io_uring/msg_ring.c +30 -9
@@ -65,20 +65,33 @@ static void io_msg_tw_complete(struct callback_head *head)struct io_ring_ctx *target_ctx = req->file->private_data;int ret =0 ; -if (current->flags &PF_EXITING ) +if (current->flags &PF_EXITING ) { ret = -EOWNERDEAD ; -else if (!io_post_aux_cqe (target_ctx, msg->user_data, msg->len,0 )) - ret = -EOVERFLOW ; + }else { +/* +* If the target ring is using IOPOLL mode, then we need to be +* holding the uring_lock for posting completions. Other ring +* types rely on the regular completion locking, which is +* handled while posting. +*/ +if (target_ctx->flags &IORING_SETUP_IOPOLL ) +mutex_lock (&target_ctx->uring_lock); +if (!io_post_aux_cqe (target_ctx, msg->user_data, msg->len,0 )) + ret = -EOVERFLOW ; +if (target_ctx->flags &IORING_SETUP_IOPOLL ) +mutex_unlock (&target_ctx->uring_lock); + }if (ret <0 )req_set_fail (req);io_req_queue_tw_complete (req, ret); } -static int io_msg_ring_data (struct io_kiocb *req) +static int io_msg_ring_data (struct io_kiocb *req,unsigned int issue_flags) {struct io_ring_ctx *target_ctx = req->file->private_data; struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); +int ret;if (msg->src_fd || msg->dst_fd || msg->flags)return -EINVAL ;
@@ -93,10 +106,18 @@ static int io_msg_ring_data(struct io_kiocb *req)return IOU_ISSUE_SKIP_COMPLETE ; } -if (io_post_aux_cqe (target_ctx, msg->user_data, msg->len,0 )) -return 0 ; - -return -EOVERFLOW ; + ret = -EOVERFLOW ; +if (target_ctx->flags &IORING_SETUP_IOPOLL ) { +if (unlikely (io_double_lock_ctx (target_ctx, issue_flags))) +return -EAGAIN ; +if (io_post_aux_cqe (target_ctx, msg->user_data, msg->len,0 )) + ret =0 ; +io_double_unlock_ctx (target_ctx); + }else { +if (io_post_aux_cqe (target_ctx, msg->user_data, msg->len,0 )) + ret =0 ; + } +return ret; }static struct file *io_msg_grab_file (struct io_kiocb *req,unsigned int issue_flags)
@@ -223,7 +244,7 @@ int io_msg_ring(struct io_kiocb *req, unsigned int issue_flags)switch (msg->cmd) {case IORING_MSG_DATA : - ret =io_msg_ring_data (req); + ret =io_msg_ring_data (req, issue_flags);break ;case IORING_MSG_SEND_FD : ret =io_msg_send_fd (req, issue_flags);
Signature · 1 line
2.39.0
cover Fix locking for MSG_RING and IOPOLL targets
In reply to Jens AxboeHello,
Does this patchset need to be backported to 6.1?
Thank you
Quoted 3 lines · Oleksandr Tymoshenko
Hello,
Does this patchset need to be backported to 6.1?
Yes. Seems to be missing in linux-6.1.y and it is a security fix. Do you want to send a backport?
Signature · 1 line
Gabriel Krisman Bertazi
Jens Axboe
maintainerQuoted 3 lines · Oleksandr Tymoshenko
Hello,
Does this patchset need to be backported to 6.1?
Yeah it probably should, guessing it didn't get picked as there are dependencies. I'll take a look.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 7 lines · Jens Axboe
Quoted 3 lines · Oleksandr Tymoshenko
Hello,
Does this patchset need to be backported to 6.1?
Yeah it probably should, guessing it didn't get picked as there are dependencies. I'll take a look.
Here they are - I'll send them in for stable.
Signature · 1 line
Jens Axboe
From c47db9ca937e232606060077bb09cff49e75dbb7 Mon Sep 17 00:00:00 2001
From: Jens Axboe <[email protected]>
Date: Tue, 22 Aug 2023 18:00:02 -0600
Subject: [PATCH 4/4] io_uring/msg_ring: fix missing lock on overflow for
IOPOLL
Commit e12d7a46f65ae4b7d58a5e0c1cbfa825cf8d830d upstream.
If the target ring is configured with IOPOLL, then we always need to hold the target ring uring_lock before posting CQEs. We could just grab it unconditionally, but since we don't expect many target rings to be of this type, make grabbing the uring_lock conditional on the ring type.
--- io_uring/msg_ring.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)
io_uring/msg_ring.c +15 -5
@@ -57,20 +57,30 @@ void io_msg_ring_cleanup(struct io_kiocb *req) msg->src_file =NULL ; } -static int io_msg_ring_data (struct io_kiocb *req) +static int io_msg_ring_data (struct io_kiocb *req,unsigned int issue_flags) {struct io_ring_ctx *target_ctx = req->file->private_data; struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); +int ret;if (msg->src_fd || msg->dst_fd || msg->flags)return -EINVAL ;if (target_ctx->flags &IORING_SETUP_R_DISABLED )return -EBADFD ; -if (io_post_aux_cqe (target_ctx, msg->user_data, msg->len,0 , true)) -return 0 ; + ret = -EOVERFLOW ; +if (target_ctx->flags &IORING_SETUP_IOPOLL ) { +if (unlikely (io_double_lock_ctx (target_ctx, issue_flags))) +return -EAGAIN ; +if (io_post_aux_cqe (target_ctx, msg->user_data, msg->len,0 , true)) + ret =0 ; +io_double_unlock_ctx (target_ctx); + }else { +if (io_post_aux_cqe (target_ctx, msg->user_data, msg->len,0 , true)) + ret =0 ; + } -return -EOVERFLOW ; +return ret; }static struct file *io_msg_grab_file (struct io_kiocb *req,unsigned int issue_flags)
@@ -175,7 +185,7 @@ int io_msg_ring(struct io_kiocb *req, unsigned int issue_flags)switch (msg->cmd) {case IORING_MSG_DATA : - ret =io_msg_ring_data (req); + ret =io_msg_ring_data (req, issue_flags);break ;case IORING_MSG_SEND_FD : ret =io_msg_send_fd (req, issue_flags);
Signature · 1 line
2.40.1
From 5c3ffa8ec64e0726dcbfe37b5d126701301011ed Mon Sep 17 00:00:00 2001
From: Jens Axboe <[email protected]>
Date: Thu, 19 Jan 2023 09:01:27 -0700
Subject: [PATCH 3/4] io_uring/msg_ring: move double lock/unlock helpers higher
up
Commit 423d5081d0451faa59a707e57373801da5b40141 upstream.
In preparation for needing them somewhere else, move them and get rid of the unused 'issue_flags' for the unlock side.
No functional changes in this patch.
--- io_uring/msg_ring.c | 47 ++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 24 deletions(-)
io_uring/msg_ring.c +23 -24
65 lines not on this page · raw message
Signature · 1 line
2.40.1
From ef7b782465bd6943fbcacb4af4fbe790137a5820 Mon Sep 17 00:00:00 2001
From: Pavel Begunkov <[email protected]>
Date: Wed, 7 Dec 2022 03:53:35 +0000
Subject: [PATCH 2/4] io_uring: extract a io_msg_install_complete helper
Commit 172113101641cf1f9628c528ec790cb809f2b704 upstream.
Extract a helper called io_msg_install_complete() from io_msg_send_fd(), will be used later.
--- io_uring/msg_ring.c | 38 +++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-)
io_uring/msg_ring.c +21 -17
@@ -94,40 +94,25 @@ static struct file *io_msg_grab_file(struct io_kiocb *req, unsigned int issue_flreturn file; } -static int io_msg_send_fd (struct io_kiocb *req,unsigned int issue_flags) +static int io_msg_install_complete (struct io_kiocb *req,unsigned int issue_flags) {struct io_ring_ctx *target_ctx = req->file->private_data; struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); -struct io_ring_ctx *ctx = req->ctx;struct file *src_file = msg->src_file;int ret; -if (msg->len) -return -EINVAL ; -if (target_ctx == ctx) -return -EINVAL ; -if (target_ctx->flags &IORING_SETUP_R_DISABLED ) -return -EBADFD ; -if (!src_file) { - src_file =io_msg_grab_file (req, issue_flags); -if (!src_file) -return -EBADF ; - msg->src_file = src_file; - req->flags |=REQ_F_NEED_CLEANUP ; - } -if (unlikely (io_double_lock_ctx (target_ctx, issue_flags)))return -EAGAIN ; ret =__io_fixed_fd_install (target_ctx, src_file, msg->dst_fd);if (ret <0 ) goto out_unlock; + msg->src_file =NULL ; req->flags &= ~REQ_F_NEED_CLEANUP ;if (msg->flags &IORING_MSG_RING_CQE_SKIP ) goto out_unlock; -/* * If this fails, the target still received the file descriptor but * wasn't notified of the fact. This means that if this request
@@ -141,6 +126,25 @@ static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags)return ret; } +static int io_msg_send_fd (struct io_kiocb *req,unsigned int issue_flags) +{ +struct io_ring_ctx *target_ctx = req->file->private_data; + struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); +struct io_ring_ctx *ctx = req->ctx; +struct file *src_file = msg->src_file; + +if (target_ctx == ctx) +return -EINVAL ; +if (!src_file) { + src_file =io_msg_grab_file (req, issue_flags); +if (!src_file) +return -EBADF ; + msg->src_file = src_file; + req->flags |=REQ_F_NEED_CLEANUP ; + } +return io_msg_install_complete (req, issue_flags); +} +int io_msg_ring_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) { struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg);
Signature · 1 line
2.40.1
From f10ba483becae0b8c7de0e9fe3e0f6d07a405b7a Mon Sep 17 00:00:00 2001
From: Pavel Begunkov <[email protected]>
Date: Wed, 7 Dec 2022 03:53:34 +0000
Subject: [PATCH 1/4] io_uring: get rid of double locking
Commit 11373026f2960390d5e330df4e92735c4265c440 upstream.
We don't need to take both uring_locks at once, msg_ring can be split in two parts, first getting a file from the filetable of the first ring and then installing it into the second one.
--- io_uring/msg_ring.c | 85 ++++++++++++++++++++++++++------------------- io_uring/msg_ring.h | 1 + io_uring/opdef.c | 1 + 3 files changed, 51 insertions(+), 36 deletions(-)
io_uring/msg_ring.c +49 -36
144 lines not on this page · raw message
io_uring/msg_ring.h +1 -0
@@ -2,3 +2,4 @@int io_msg_ring_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe);int io_msg_ring (struct io_kiocb *req,unsigned int issue_flags); +void io_msg_ring_cleanup (struct io_kiocb *req);
io_uring/opdef.c +1 -0
@@ -445,6 +445,7 @@ const struct io_op_def io_op_defs[] = { .name = "MSG_RING", .prep = io_msg_ring_prep, .issue = io_msg_ring, + .cleanup = io_msg_ring_cleanup, }, [IORING_OP_FSETXATTR] = { .needs_file = 1,
Signature · 1 line
2.40.1