Skip to contentSkip to conversation

io_uring/register: preserve SQ array entries on resize

PATCH Applied guzebing 2 participants 3 messages Jun 8 – Jun 16 io_uring

lore
Share & tools
lore

guzebing

author
#0 lore raw

PATCH 1/1 io_uring/register: preserve SQ array entries on resize

Ring resizing copies pending SQEs from the old SQE array into the new one so submissions queued before the resize can still be consumed afterwards.

That copy currently walks the SQ head/tail range directly. This is only correct when there is no SQ array indirection. With a regular SQ array, each pending SQ entry contains an index into the SQE array. After resize, ctx->sq_array is repointed at the newly allocated array, so pending entries lose their old logical-to-physical mapping and may submit the wrong SQE.

Remember the old and new SQ arrays while migrating pending SQ entries. For each pending entry, copy the SQE selected by the old array into the new destination slot and rebuild the new array entry to point at the copied SQE. Keep invalid user-provided entries invalid so the normal submission path still drops them after resize.

Fixes 79cfe9e59c2a1 ("io_uring/register: add IORING_REGISTER_RESIZE_RINGS")
---
 io_uring/register.c | 31 +++++++++++++++++++++----------
 1 file changed, 21 insertions(+), 10 deletions(-)
io_uring/register.c +21 -10
index dce5e2f9cf770..02bc103bcc9d5 100644
@@ -503,6 +503,7 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg)
 	unsigned i, tail, old_head;
 	struct io_uring_params *p = &config.p;
 	struct io_rings_layout *rl = &config.layout;
+	u32 *o_sq_array, *n_sq_array = NULL;
 	int ret;
 
 	memset(&config, 0, sizeof(config));
@@ -589,6 +590,9 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg)
 	ctx->rings = NULL;
 	o.sq_sqes = ctx->sq_sqes;
 	ctx->sq_sqes = NULL;
+	o_sq_array = ctx->sq_array;
+	if (!(ctx->flags & IORING_SETUP_NO_SQARRAY))
+		n_sq_array = (u32 *)((char *)n.rings + rl->sq_array_offset);
 
 	/*
 	 * Now copy SQ and CQ entries, if any. If either of the destination
@@ -599,20 +603,27 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg)
 	if (tail - old_head > p->sq_entries)
 		goto overflow;
 	for (i = old_head; i < tail; i++) {
-		unsigned index, dst_mask, src_mask;
+		unsigned int dst, src;
 		size_t sq_size;
 
-		index = i;
+		dst = i & (p->sq_entries - 1);
+		src = i & (ctx->sq_entries - 1);
+		if (n_sq_array) {
+			src = READ_ONCE(o_sq_array[src]);
+			if (unlikely(src >= ctx->sq_entries)) {
+				WRITE_ONCE(n_sq_array[dst], UINT_MAX);
+				continue;
+			}
+			WRITE_ONCE(n_sq_array[dst], dst);
+		}
+
 		sq_size = sizeof(struct io_uring_sqe);
-		src_mask = ctx->sq_entries - 1;
-		dst_mask = p->sq_entries - 1;
 		if (ctx->flags & IORING_SETUP_SQE128) {
-			index <<= 1;
+			dst <<= 1;
+			src <<= 1;
 			sq_size <<= 1;
-			src_mask = (ctx->sq_entries << 1) - 1;
-			dst_mask = (p->sq_entries << 1) - 1;
 		}
-		memcpy(&n.sq_sqes[index & dst_mask], &o.sq_sqes[index & src_mask], sq_size);
+		memcpy(&n.sq_sqes[dst], &o.sq_sqes[src], sq_size);
 	}
 	WRITE_ONCE(n.rings->sq.head, old_head);
 	WRITE_ONCE(n.rings->sq.tail, tail);
@@ -655,8 +666,8 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg)
 	WRITE_ONCE(n.rings->cq_overflow, READ_ONCE(o.rings->cq_overflow));
 
 	/* all done, store old pointers and assign new ones */
-	if (!(ctx->flags & IORING_SETUP_NO_SQARRAY))
-		ctx->sq_array = (u32 *)((char *)n.rings + rl->sq_array_offset);
+	if (n_sq_array)
+		ctx->sq_array = n_sq_array;
 
 	ctx->sq_entries = p->sq_entries;
 	ctx->cq_entries = p->cq_entries;
Signature · 1 line
2.20.1
author
#1 lore raw

Hi Jens,

A gentle ping on this patch.

Please allow me to add some details about the root cause and the reproducer.

There appear to be two related parts to how the resize path migrates pending SQ entries in regular SQ-array mode.

1. It copies SQEs by walking the logical SQ head/tail range directly. For a pending SQ entry in regular SQ-array mode, the resize path should first resolve the old sq_array[] entry to find the source physical SQE slot, and then copy that SQE into the new SQE array.

2. The old resize path switches ctx->sq_array to the SQ array in the
new ring, but it does not initialize the SQ array entries for the pending submissions.

And I reproduced this with a small test that:

1. creates a ring with IORING_SETUP_SINGLE_ISSUER | IORING_SETUP_DEFER_TASKRUN,
2. writes two NOP SQEs into physical slots 1 and 2,
3. queues two pending entries by setting sq_array[0] = 1 and sq_array[1] = 2 and advancing the SQ tail,
4. resizes the ring,
5. calls io_uring_enter() to submit those pending SQ entries.

Without the fix, the subsequent submission path can consume the wrong SQE after resize and return a CQE with the wrong user_data. With the patch applied, pending SQ entries are migrated using the old SQ-array mapping, so they still refer to the intended SQEs after resize.

Could you please take a look when you have a chance? I would also appreciate feedback if you think this is not the right direction, or if there is a better way to preserve pending SQ-array entries during resize.

Thanks,
guzebing

maintainer Applied
#2 lore raw
In reply to guzebing
Quoted 4 lines · guzebing

Ring resizing copies pending SQEs from the old SQE array into the new one so submissions queued before the resize can still be consumed afterwards.

Quoted 8 lines

That copy currently walks the SQ head/tail range directly. This is only correct when there is no SQ array indirection. With a regular SQ array, each pending SQ entry contains an index into the SQE array. After resize, ctx->sq_array is repointed at the newly allocated array, so pending entries lose their old logical-to-physical mapping and may submit the wrong SQE.

[...]

Applied, thanks!

[1/1] io_uring/register: preserve SQ array entries on resize

      commit: 1fe703cc708f19209ae8e6261247483db723c221

Best regards,

Signature · 1 line
Jens Axboe