io_uring fixes for 7.3-stable

Hi Greg,

Here are io_uring fixes for 7.3-stable. They were waiting on the io_uring pull for 7.4-rc, which Linus has now merged.

They apply on top of v7.3.3 with the current queue-7.3 applied (stable-queue 0ae08d01c389). They don't depend on anything in the queue, and apply to v7.3.3 alone as well.

The io_uring series I sent earlier this week is already in v7.3.3, these go on top of that.

Please take patches 1-3 together. Patch 2 closes a race that patch 1 opens up, and patch 3 fixes a WARN_ON_ONCE() that patch 2 introduces on SQPOLL thread exit.

Patch 1 needed changes to apply to 7.3, the [axboe: ...] note at the end of it says what was changed.

Chris Mason (1):
io_uring/cancel: drop sqd->lock while SQPOLL waits

Jens Axboe (3):
io_uring/sqpoll: don't attach to a thread that is canceling io_uring/poll: don't poll a writer for a read side hangup io_uring/poll: don't poll a writer for urgent data

Shakeel Butt (1):
io_uring/sqpoll: do the return-to-userspace work in the SQPOLL thread

Shang En Sim (2):
io_uring/cancel: count all requests that a CANCEL_ALL cancels io_uring/poll: cancel all matching requests in one pass

Shuai Liu (1):
io_uring/sqpoll: don't set IO_SQ_THREAD_SHOULD_STOP on thread exit

Yuansheng Yang (1):
io_uring/rsrc: hold uring_lock across index-alloc put_user

 io_uring/cancel.c | 26 ++++++++++++++++++--
 io_uring/poll.c   | 62 +++++++++++++++++++++++++++++++++++++++++------
 io_uring/rsrc.c   | 21 ++++++++++++----
 io_uring/sqpoll.c | 26 +++++++++++++++++++-
 4 files changed, 120 insertions(+), 15 deletions(-)

2.56.0

From: Chris Mason <[email protected]>

Commit cb0306c52ffae083a86fb283de67875ae7a9500e upstream.

An exiting SQPOLL thread calls io_uring_cancel_generic() with sqd->lock held, and keeps it across cond_resched() and the sleep until nothing is in flight anymore. A zero-copy send notification can't be canceled and stays pending as long as the peer doesn't read, so the lock is held for as long as a user cares to. io_ring_exit_work() needs it to park the thread, and so do IORING_REGISTER_IOWQ_AFF and IOWQ_MAX_WORKERS. With 64 such rings sharing the thread, every iou_exit slot sleeps on the lock and no ring gets freed for anyone.

Drop the lock around cond_resched() and the sleep, like the main loop of the thread does. Each walk of ctx_list is still under one hold, and nothing is kept across the unlock. The thread drops its io-wq, frees its tctx and clears sqd->thread under one hold after the loop, so whoever holds the lock and finds sqd->thread set can still use them.

---
 io_uring/cancel.c | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

index 75a21ecfe69c..c7ed51377dea 100644
@@ -650,7 +650,16 @@ __cold void io_uring_cancel_generic(bool cancel_all, struct io_sq_data *sqd)
 		}
 
 		if (loop) {
+			/*
+			 * A request that runs in io-wq and doesn't end makes
+			 * each round come here. Don't hold sqd->lock for that
+			 * long either.
+			 */
+			if (sqd)
+				mutex_unlock(&sqd->lock);
 			cond_resched();
+			if (sqd)
+				mutex_lock(&sqd->lock);
 			continue;
 		}
 

@@ -678,8 +687,20 @@ __cold void io_uring_cancel_generic(bool cancel_all, struct io_sq_data *sqd)
 		 * avoids a race where a completion comes in before we did
 		 * prepare_to_wait().
 		 */
-		if (inflight == tctx_inflight(tctx, !cancel_all))
+		if (inflight == tctx_inflight(tctx, !cancel_all)) {
+			/*
+			 * A zero-copy notification can't be canceled, and a
+			 * peer that doesn't read keeps it pending. Don't hold
+			 * sqd->lock while waiting for it, or whoever wants the
+			 * lock sleeps as long. No ring is used across this,
+			 * the next round walks ->ctx_list anew.
+			 */
+			if (sqd)
+				mutex_unlock(&sqd->lock);
 			schedule();
+			if (sqd)
+				mutex_lock(&sqd->lock);
+		}
 end_wait:
 		finish_wait(&tctx->wait, &wait);
 	} while (1);

2.56.0

Commit 22d9db9fd64a493525d692b32c556875b7254665 upstream.

io_sq_offload_create() refuses to attach a ring to an SQPOLL thread that has already gone, by checking sqd->thread under sqd->lock. Since the thread drops the lock while io_uring_cancel_generic() waits for its requests, that check can run while the thread is on its way out and still see it. The new ring is then added to ctx_list, io_uring_setup() succeeds, the thread finishes canceling and exits without ever having looked at the ring, and io_uring_enter() fails with -EOWNERDEAD.

Only a thread that exits on a fatal signal can be caught like this, as the stop bit is set once the last ring has let go of the sqd, when no fd is left to attach through. A fatal signal is group wide and attaching needs the same thread group, so the attacher is on its way out as well. Still, make the check see it: set the stop bit before canceling, whatever the reason for the exit, and test it when attaching.

---
 io_uring/sqpoll.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

index 49c52284ecdd..c833e776320d 100644
@@ -433,6 +433,12 @@ static int io_sq_thread(void *data)
 	if (io_sq_tw_pending())
 		io_sq_tw(UINT_MAX);
 
+	/*
+	 * Cancelation drops sqd->lock while it waits, and a ring could attach
+	 * to a thread that is on its way out. A fatal signal doesn't set the
+	 * bit, so set it here for io_sq_offload_create() to see.
+	 */
+	set_bit(IO_SQ_THREAD_SHOULD_STOP, &sqd->state);
 	io_uring_cancel_generic(true, sqd);
 	rcu_assign_pointer(sqd->thread, NULL);
 	put_task_struct(current);

@@ -502,7 +508,10 @@ __cold int io_sq_offload_create(struct io_ring_ctx *ctx,
 		list_add(&ctx->sqd_list, &sqd->ctx_list);
 		io_sqd_update_thread_idle(sqd);
 		/* don't attach to a dying SQPOLL thread, would be racy */
-		ret = (attached && !sqd->thread) ? -ENXIO : 0;
+		ret = 0;
+		if (attached && (!sqd->thread ||
+		    test_bit(IO_SQ_THREAD_SHOULD_STOP, &sqd->state)))
+			ret = -ENXIO;
 		io_sq_thread_unpark(sqd);
 
 		if (ret < 0)

2.56.0

From: Shuai Liu <[email protected]>

Commit fca3ff9e2eee7d285585daf51942e0b5e2be354a upstream.

io_sq_thread_stop() carries a WARN_ON_ONCE() asserting that IO_SQ_THREAD_SHOULD_STOP isn't set yet, i.e. that stopping the SQPOLL thread is never attempted twice. Since commit 22d9db9fd64a ("io_uring/sqpoll: don't attach to a thread that is canceling"), the SQPOLL thread itself sets that bit on its way out, so that io_sq_offload_create() won't attach a new ring to a thread that is canceling.

When the thread exits on a fatal signal, the bit is thus set before the ring teardown gets to io_sq_thread_stop(), which now triggers the warning on every such exit:

  WARNING: io_uring/sqpoll.c:88 at io_sq_thread_stop
  Workqueue: iou_exit io_ring_exit_work
  Call Trace:
   io_put_sq_data
   io_sq_thread_finish
   io_ring_ctx_free
   io_ring_exit_work

Introduce a separate IO_SQ_THREAD_EXITING bit for the thread marking itself as on its way out, and check it in io_sq_offload_create() along with IO_SQ_THREAD_SHOULD_STOP. IO_SQ_THREAD_SHOULD_STOP regains its original meaning, "a stop has been requested", and the WARN_ON_ONCE() holds again.

Reproduced with an SQPOLL ring and an immediate fatal signal: the warning fires on v7.3.0+ (22d9db9fd64a present) and is gone with this change.

---
 io_uring/sqpoll.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

index c833e776320d..e86658ede44b 100644
@@ -29,6 +29,7 @@
 enum {
 	IO_SQ_THREAD_SHOULD_STOP = 0,
 	IO_SQ_THREAD_SHOULD_PARK,
+	IO_SQ_THREAD_EXITING,
 };
 
 void io_sq_thread_unpark(struct io_sq_data *sqd)

@@ -436,9 +437,10 @@ static int io_sq_thread(void *data)
 	/*
 	 * Cancelation drops sqd->lock while it waits, and a ring could attach
 	 * to a thread that is on its way out. A fatal signal doesn't set the
-	 * bit, so set it here for io_sq_offload_create() to see.
+	 * stop bit, so mark the thread as exiting here for
+	 * io_sq_offload_create() to see.
 	 */
-	set_bit(IO_SQ_THREAD_SHOULD_STOP, &sqd->state);
+	set_bit(IO_SQ_THREAD_EXITING, &sqd->state);
 	io_uring_cancel_generic(true, sqd);
 	rcu_assign_pointer(sqd->thread, NULL);
 	put_task_struct(current);

@@ -510,7 +512,8 @@ __cold int io_sq_offload_create(struct io_ring_ctx *ctx,
 		/* don't attach to a dying SQPOLL thread, would be racy */
 		ret = 0;
 		if (attached && (!sqd->thread ||
-		    test_bit(IO_SQ_THREAD_SHOULD_STOP, &sqd->state)))
+		    test_bit(IO_SQ_THREAD_SHOULD_STOP, &sqd->state) ||
+		    test_bit(IO_SQ_THREAD_EXITING, &sqd->state)))
 			ret = -ENXIO;
 		io_sq_thread_unpark(sqd);
 

2.56.0

From: Shakeel Butt <[email protected]>

Commit f013b695506491c5202a412a6568313549c81f50 upstream.

The SQPOLL thread never returns to userspace. io_sq_tw() runs its task_work, but the rest of resume_user_mode_work(), like memory.high reclaim and block cgroup throttling, only runs when the thread exits. So memory.high reclaim only happens inside a charge, with sqd->lock and ctx->uring_lock held, and not at all if the charges can't block.

This used to work. The loop called io_run_task_work(), which does this work since commit b5d3ae202fbf ("io_uring: handle TIF_NOTIFY_RESUME when checking for task_work"). Commit af5d68f8892f ("io_uring/sqpoll:
manage task_work privately") replaced that call with io_sq_tw().

Call resume_user_mode_work() at the top of the loop when TIF_NOTIFY_RESUME is set, with sqd->lock dropped. It runs once per pass, so a pass that charges two or more batches still reclaims inside the charge, as before.

Tested in a VM with memory.high=64M and swap on zram. An SQPOLL ring reads 256M from /dev/zero into untouched memory, 64K at a time. Before, the thread made 337 calls to __mem_cgroup_handle_over_high(), all from inside the charge. After, it makes 704, all from the top of the loop. memory.peak stays at 64M. With 1M reads, 296 of its 385 calls still come from inside the charge. Both runs take about 50% longer, and the extra time is in the handler, which now runs more often. Without memory.high pressure, the test never calls resume_user_mode_work() and the times do not change. Lockdep showed no warnings, also with io_uring_register() racing for sqd->lock.

---
 io_uring/sqpoll.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

index e86658ede44b..d387488db2de 100644
@@ -349,6 +349,18 @@ static int io_sq_thread(void *data)
 			timeout = jiffies + sqd->sq_thread_idle;
 		}
 
+		/*
+		 * The SQPOLL thread never returns to userspace, so do the
+		 * return-to-userspace work, like memory.high reclaim, here.
+		 * It can sleep for a long time, so drop sqd->lock.
+		 */
+		if (test_thread_flag(TIF_NOTIFY_RESUME)) {
+			mutex_unlock(&sqd->lock);
+			resume_user_mode_work(NULL);
+			mutex_lock(&sqd->lock);
+			sqd->sq_cpu = raw_smp_processor_id();
+		}
+
 		cap_entries = !list_is_singular(&sqd->ctx_list);
 		list_for_each_entry(ctx, &sqd->ctx_list, sqd_list) {
 			int ret;

2.56.0

From: Shang En Sim <[email protected]>

Commit 21ddbb14f7c3baab50a2a6c13ee375edc2bbe031 upstream.

With IORING_ASYNC_CANCEL_ALL, __io_async_cancel() counts one for each io_try_cancel() call that cancels something. io_futex_cancel() and io_waitid_cancel() cancel every match in one call and return how many, so four futex waits cancelled with CANCEL_ALL give a result of 1.

Add the number that they return.

---
 io_uring/cancel.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

index c7ed51377dea..a34543cb423c 100644
@@ -198,7 +198,8 @@ static int __io_async_cancel(struct io_cancel_data *cd,
 			break;
 		if (!all)
 			return ret;
-		nr++;
+		/* futex and waitid return how many they cancelled */
+		nr += max(ret, 1);
 	} while (1);
 
 	/* slow path, try all io-wq's */

2.56.0

From: Shang En Sim <[email protected]>

Commit 9eff9c07e927dcd9c57150cd6fb3d4ec76f00480 upstream.

With IORING_ASYNC_CANCEL_ALL, __io_async_cancel() calls io_try_cancel() until nothing is found, and io_poll_cancel() cancels one request per call, starting each lookup at the beginning of the hash. A cancelled request stays hashed until its task work runs, so every lookup walks past the ones already cancelled. Cancelling K of N pending polls visits about K*N/2 requests in one io_uring_enter(), with uring_lock held.

A single CANCEL_FD|ALL on 80000 polls on one eventfd takes 17.06s, 0.02s with this change. With voluntary preemption the soft lockup watchdog fires:

  watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [repro:100]
  RIP: 0010:io_cancel_req_match+0x19/0xa0
   io_poll_cancel+0x73/0x170
   io_try_cancel+0x7b/0xf0
   __io_async_cancel+0x41/0x110

Walk the hash once and cancel every match, as io_cancel_remove() does for futex and waitid. Cancel by user_data only needs its own bucket. __io_async_cancel() still loops until nothing is found, so a request that io-wq parks in the poll hash meanwhile is still cancelled.

A cancel that matches one request now walks the whole hash twice instead of stopping at the match.

---
 io_uring/cancel.c |  2 +-
 io_uring/poll.c   | 39 +++++++++++++++++++++++++++++++++++++--
 2 files changed, 38 insertions(+), 3 deletions(-)

index a34543cb423c..6be224ab53dc 100644
@@ -198,7 +198,7 @@ static int __io_async_cancel(struct io_cancel_data *cd,
 			break;
 		if (!all)
 			return ret;
-		/* futex and waitid return how many they cancelled */
+		/* poll, futex and waitid return how many they cancelled */
 		nr += max(ret, 1);
 	} while (1);
 

index 4f04bda42f7f..9b73b0d88a0e 100644
@@ -827,12 +827,47 @@ static int io_poll_disarm(struct io_kiocb *req)
 	return 0;
 }
 
+/*
+ * Cancel every request matching @cd in one pass. Cancelled requests stay
+ * hashed until their task work runs, so finding them one at a time would
+ * walk past all the earlier ones on each lookup.
+ */
+static int io_poll_cancel_all(struct io_ring_ctx *ctx, struct io_cancel_data *cd)
+{
+	unsigned int nr_buckets = 1U << ctx->cancel_table.hash_bits;
+	unsigned int i = 0;
+	struct hlist_node *tmp;
+	struct io_kiocb *req;
+	int nr = 0;
+
+	if (!(cd->flags & (IORING_ASYNC_CANCEL_FD | IORING_ASYNC_CANCEL_OP |
+			   IORING_ASYNC_CANCEL_ANY))) {
+		/* keyed by user_data, so only one bucket can match */
+		i = hash_long(cd->data, ctx->cancel_table.hash_bits);
+		nr_buckets = i + 1;
+	}
+
+	for (; i < nr_buckets; i++) {
+		struct io_hash_bucket *hb = &ctx->cancel_table.hbs[i];
+
+		hlist_for_each_entry_safe(req, tmp, &hb->list, hash_node) {
+			if (!io_cancel_req_match(req, cd))
+				continue;
+			io_poll_cancel_req(req);
+			nr++;
+		}
+	}
+	return nr ?: -ENOENT;
+}
+
 static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd)
 {
 	struct io_kiocb *req;
 
-	if (cd->flags & (IORING_ASYNC_CANCEL_FD | IORING_ASYNC_CANCEL_OP |
-			 IORING_ASYNC_CANCEL_ANY))
+	if (cd->flags & (IORING_ASYNC_CANCEL_ALL | IORING_ASYNC_CANCEL_ANY))
+		return io_poll_cancel_all(ctx, cd);
+
+	if (cd->flags & (IORING_ASYNC_CANCEL_FD | IORING_ASYNC_CANCEL_OP))
 		req = io_poll_file_find(ctx, cd);
 	else
 		req = io_poll_find(ctx, false, cd);

2.56.0

Commit 74c1f1048e4401d6a953ab2299b14ace1ad6a899 upstream.

io_init_poll_iocb() adds EPOLLRDHUP to every poll, including the async poll of a send that waits for buffer space. Once the peer has shut down its sending side, the socket keeps reporting EPOLLRDHUP, so each re-arm in __io_arm_poll_handler() finds a ready mask and the request is issued again. The send still can't make progress, the request burns through APOLL_MAX_RETRY in a few milliseconds and ends up in io-wq.

A send in io-wq blocks until the peer reads, but sendmsg and writev go through ____sys_sendmsg() and sock_write_iter(), which turn O_NONBLOCK into MSG_DONTWAIT, and the application gets -EAGAIN for a request that never asked for a non-blocking send.

Only add EPOLLRDHUP for polls that want to read. A peer that stopped sending is no event for a writer, and a poll that asks for it explicitly still gets it.

---
 io_uring/poll.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

index 9b73b0d88a0e..9703a2fdfb44 100644
@@ -128,12 +128,24 @@ static void io_poll_req_insert(struct io_kiocb *req)
 	hlist_add_head(&req->hash_node, &table->hbs[index].list);
 }
 
+#define IO_POLL_UNMASK	(EPOLLERR|EPOLLHUP|EPOLLNVAL)
+
+static __poll_t io_poll_unmask(__poll_t events)
+{
+	/*
+	 * A peer that stopped sending is an event for a reader, but a write
+	 * only poll must stay armed until the file is writable again.
+	 */
+	if (events & (EPOLLIN | EPOLLRDNORM | EPOLLRDBAND))
+		return IO_POLL_UNMASK | EPOLLRDHUP;
+	return IO_POLL_UNMASK;
+}
+
 static void io_init_poll_iocb(struct io_poll *poll, __poll_t events)
 {
 	poll->head = NULL;
-#define IO_POLL_UNMASK	(EPOLLERR|EPOLLHUP|EPOLLNVAL|EPOLLRDHUP)
 	/* mask in events that we always want/need */
-	poll->events = events | IO_POLL_UNMASK;
+	poll->events = events | io_poll_unmask(events);
 	INIT_LIST_HEAD(&poll->wait.entry);
 	init_waitqueue_func_entry(&poll->wait, io_poll_wake);
 }

@@ -1006,7 +1018,7 @@ int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags)
 
 			poll->events &= ~0xffff;
 			poll->events |= poll_update->events & 0xffff;
-			poll->events |= IO_POLL_UNMASK;
+			poll->events |= io_poll_unmask(poll->events);
 		}
 		if (poll_update->update_user_data)
 			preq->cqe.user_data = poll_update->new_user_data;

2.56.0

Commit edeac4546a71b6c1db4ba006a2dbf7c863f54e9a upstream.

io_arm_poll_handler() starts the mask of every async poll with EPOLLPRI, for a reader and a writer alike. A TCP socket reports EPOLLPRI for as long as urgent data sits unread, so once the peer has sent a byte with MSG_OOB, the async poll of a send that waits for buffer space is ready on every re-arm. The request spins through task_work and __io_arm_poll_handler() until APOLL_MAX_RETRY is used up and io-wq takes it, the same spin that EPOLLRDHUP caused for a half-closed peer.

While it spins, the request is neither in the poll hash nor in io-wq. A cancel issued right after the send gets -ENOENT and the send stays pending, until a later cancel finds it in io-wq.

Only add EPOLLPRI for polls that want to read. Urgent data is an event for a recv, not for a writer.

---
 io_uring/poll.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

index 9703a2fdfb44..fc1800c7097b 100644
@@ -739,7 +739,7 @@ int io_arm_apoll(struct io_kiocb *req, unsigned issue_flags, __poll_t mask)
 int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags)
 {
 	const struct io_issue_def *def = &io_issue_defs[req->opcode];
-	__poll_t mask = POLLPRI | POLLERR;
+	__poll_t mask = EPOLLERR;
 
 	if (!def->pollin && !def->pollout)
 		return IO_APOLL_ABORTED;

@@ -747,7 +747,8 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags)
 		return IO_APOLL_ABORTED;
 
 	if (def->pollin) {
-		mask |= EPOLLIN | EPOLLRDNORM;
+		/* urgent data is an event for a reader only */
+		mask |= EPOLLIN | EPOLLRDNORM | EPOLLPRI;
 
 		/* If reading from MSG_ERRQUEUE using recvmsg, ignore POLLIN */
 		if (req->flags & REQ_F_CLEAR_POLLIN)

2.56.0

From: Yuansheng Yang <[email protected]>

Commit d7c7037253f6935af243570635d71b4920b5a4ea upstream.

io_files_update_with_index_alloc() installs a fixed file via io_fixed_fd_install(), which drops ->uring_lock when the request runs on io-wq, then put_user()s the allocated slot back to userspace. If that faults, the error path calls __io_close_fixed() by slot index after re-taking the lock.

The slot can be closed and reused between the install and that cleanup:
another fixed CLOSE plus open/accept/pipe/socket, or a FILES_UPDATE, can put an unrelated file into the same slot. The error path then removes that file instead of the one this request installed, ending an unrelated file's fixed-table reference early. This is a correctness/DoS issue, not a use-after-free: fixed-file requests hold an io_rsrc_node reference, so the node and file stay alive until they are done.

Hold ->uring_lock across __io_fixed_fd_install() and put_user(), and do the failure cleanup with io_fixed_fd_remove() under the same lock. This is the same class as the io_pipe_fixed() fix in a3d7ee11289f. Faulting while holding ->uring_lock is fine, the registration paths already do it.

---
 io_uring/rsrc.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

index d69baa47c7e7..c1ab760c4591 100644
@@ -521,11 +521,12 @@ static int io_files_update_with_index_alloc(struct io_kiocb *req,
 {
 	struct io_rsrc_update *up = io_kiocb_to_cmd(req, struct io_rsrc_update);
 	__s32 __user *fds = u64_to_user_ptr(up->arg);
+	struct io_ring_ctx *ctx = req->ctx;
 	unsigned int done;
 	struct file *file;
 	int ret, fd;
 
-	if (!req->ctx->file_table.data.nr)
+	if (!ctx->file_table.data.nr)
 		return -ENXIO;
 
 	for (done = 0; done < up->nr_args; done++) {

@@ -539,15 +540,25 @@ static int io_files_update_with_index_alloc(struct io_kiocb *req,
 			ret = -EBADF;
 			break;
 		}
-		ret = io_fixed_fd_install(req, issue_flags, file,
-					  IORING_FILE_INDEX_ALLOC);
-		if (ret < 0)
+		/*
+		 * Hold the lock across the put_user(), if it's dropped the
+		 * slot can be closed and reused before the error path runs.
+		 */
+		io_ring_submit_lock(ctx, issue_flags);
+		ret = __io_fixed_fd_install(ctx, file,
+					    IORING_FILE_INDEX_ALLOC);
+		if (ret < 0) {
+			io_ring_submit_unlock(ctx, issue_flags);
+			fput(file);
 			break;
+		}
 		if (put_user(ret, &fds[done])) {
-			__io_close_fixed(req->ctx, issue_flags, ret);
+			io_fixed_fd_remove(ctx, ret);
+			io_ring_submit_unlock(ctx, issue_flags);
 			ret = -EFAULT;
 			break;
 		}
+		io_ring_submit_unlock(ctx, issue_flags);
 	}
 
 	if (done)

2.56.0