io_uring/register: preserve SQ array entries on resize
PATCH 1/1 io_uring/register: preserve SQ array entries on resize
Ring resizing copies pending SQEs from the old SQE array into the new one so submissions queued before the resize can still be consumed afterwards.
That copy currently walks the SQ head/tail range directly. This is only correct when there is no SQ array indirection. With a regular SQ array, each pending SQ entry contains an index into the SQE array. After resize, ctx->sq_array is repointed at the newly allocated array, so pending entries lose their old logical-to-physical mapping and may submit the wrong SQE.
Remember the old and new SQ arrays while migrating pending SQ entries. For each pending entry, copy the SQE selected by the old array into the new destination slot and rebuild the new array entry to point at the copied SQE. Keep invalid user-provided entries invalid so the normal submission path still drops them after resize.
--- io_uring/register.c | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-)
io_uring/register.c +21 -10
@@ -503,6 +503,7 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg)unsigned i, tail, old_head;struct io_uring_params *p = &config.p;struct io_rings_layout *rl = &config.layout; +u32 *o_sq_array, *n_sq_array =NULL ;int ret;memset (&config,0 ,sizeof (config));
@@ -589,6 +590,9 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) ctx->rings =NULL ; o.sq_sqes = ctx->sq_sqes; ctx->sq_sqes =NULL ; + o_sq_array = ctx->sq_array; +if (!(ctx->flags &IORING_SETUP_NO_SQARRAY )) + n_sq_array = (u32 *)((char *)n.rings + rl->sq_array_offset);/* * Now copy SQ and CQ entries, if any. If either of the destination
@@ -599,20 +603,27 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg)if (tail - old_head > p->sq_entries) goto overflow;for (i = old_head; i < tail; i++) { -unsigned index, dst_mask, src_mask; +unsigned int dst, src;size_t sq_size; - index = i; + dst = i & (p->sq_entries -1 ); + src = i & (ctx->sq_entries -1 ); +if (n_sq_array) { + src =READ_ONCE (o_sq_array[src]); +if (unlikely (src >= ctx->sq_entries)) { +WRITE_ONCE (n_sq_array[dst],UINT_MAX ); +continue ; + } +WRITE_ONCE (n_sq_array[dst], dst); + } + sq_size =sizeof (struct io_uring_sqe ); - src_mask = ctx->sq_entries -1 ; - dst_mask = p->sq_entries -1 ;if (ctx->flags &IORING_SETUP_SQE128 ) { - index <<=1 ; + dst <<=1 ; + src <<=1 ; sq_size <<=1 ; - src_mask = (ctx->sq_entries <<1 ) -1 ; - dst_mask = (p->sq_entries <<1 ) -1 ; } -memcpy (&n.sq_sqes[index & dst_mask], &o.sq_sqes[index & src_mask], sq_size); +memcpy (&n.sq_sqes[dst], &o.sq_sqes[src], sq_size); }WRITE_ONCE (n.rings->sq.head, old_head);WRITE_ONCE (n.rings->sq.tail, tail);
@@ -655,8 +666,8 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg)WRITE_ONCE (n.rings->cq_overflow,READ_ONCE (o.rings->cq_overflow));/* all done, store old pointers and assign new ones */ -if (!(ctx->flags &IORING_SETUP_NO_SQARRAY )) - ctx->sq_array = (u32 *)((char *)n.rings + rl->sq_array_offset); +if (n_sq_array) + ctx->sq_array = n_sq_array; ctx->sq_entries = p->sq_entries; ctx->cq_entries = p->cq_entries;
Signature · 1 line
2.20.1
Hi Jens,
A gentle ping on this patch.
Please allow me to add some details about the root cause and the reproducer.
There appear to be two related parts to how the resize path migrates pending SQ entries in regular SQ-array mode.
1. It copies SQEs by walking the logical SQ head/tail range directly. For a pending SQ entry in regular SQ-array mode, the resize path should first resolve the old sq_array[] entry to find the source physical SQE slot, and then copy that SQE into the new SQE array.
2. The old resize path switches ctx->sq_array to the SQ array in the
new ring, but it does not initialize the SQ array entries for the pending submissions.
And I reproduced this with a small test that:
1. creates a ring with IORING_SETUP_SINGLE_ISSUER | IORING_SETUP_DEFER_TASKRUN,
2. writes two NOP SQEs into physical slots 1 and 2,
3. queues two pending entries by setting sq_array[0] = 1 and sq_array[1] = 2 and advancing the SQ tail,
4. resizes the ring,
5. calls io_uring_enter() to submit those pending SQ entries.
Without the fix, the subsequent submission path can consume the wrong SQE after resize and return a CQE with the wrong user_data. With the patch applied, pending SQ entries are migrated using the old SQ-array mapping, so they still refer to the intended SQEs after resize.
Could you please take a look when you have a chance? I would also appreciate feedback if you think this is not the right direction, or if there is a better way to preserve pending SQ-array entries during resize.
Thanks,
guzebing
Quoted 4 lines · guzebing
Ring resizing copies pending SQEs from the old SQE array into the new one so submissions queued before the resize can still be consumed afterwards.
Quoted 8 lines
That copy currently walks the SQ head/tail range directly. This is only correct when there is no SQ array indirection. With a regular SQ array, each pending SQ entry contains an index into the SQE array. After resize,
ctx->sq_arrayis repointed at the newly allocated array, so pending entries lose their old logical-to-physical mapping and may submit the wrong SQE.[...]
Applied, thanks!
[1/1] io_uring/register: preserve SQ array entries on resize
commit: 1fe703cc708f19209ae8e6261247483db723c221
Best regards,
Signature · 1 line
Jens Axboe