kbuf fixes
Subsystems (2)
Jens Axboe
maintainercover kbuf fixes
Hi,
First patch just kills an unused member in struct io_buffer_list, and then patch 2 fixes up an issue with incrementally consumed buffers where the application cannot inform the kernel of how much space should be left before moving on to the next buffer. This can cause spurious -EFAULT with multishot recvmsg, when the current buffer falls below the limit needed for the headers.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 1/2 io_uring/kbuf: kill dead struct io_buffer_list 'nr_entries' member
This is only ever assigned, never used. The only used part is the calculated mask, which is used for indexing. Kill 'nr_entries'.
--- io_uring/kbuf.c | 1 - io_uring/kbuf.h | 1 - 2 files changed, 2 deletions(-)
io_uring/kbuf.c +0 -1
@@ -680,7 +680,6 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg) }#endif - bl->nr_entries = reg.ring_entries; bl->mask = reg.ring_entries -1 ; bl->flags |=IOBL_BUF_RING ; bl->buf_ring = br;
io_uring/kbuf.h +0 -1
@@ -27,7 +27,6 @@ struct io_buffer_list {__u16 bgid;/* below is for ring provided buffers */ -__u16 nr_entries;__u16 head;__u16 mask;
Signature · 1 line
2.53.0
Jens Axboe
maintainerPATCH 2/2 io_uring/kbuf: support min length left for incremental buffers
In reply to Jens AxboeFrom: Martin Michaelis <[email protected]>
Incrementally consumed buffer rings are generally fully consumed, but it's quite possible that the application has a minimum size it needs to meet to avoid truncation. Currently that minimum limit is 1 byte, but this should be a setting that is the hands of the application. For recvmsg multishot, a prime use case for incrementally consumed buffers, the application may get spurious -EFAULT returned at the end of an incrementally consumed buffer, as less space is available than the headers need.
Grab a u32 field in struct io_uring_buf_reg, which the application can use to inform the kernel of the minimum size that should be available in an incrementally consumed buffer. If less than that is available, the current buffer is fully processed and the next one will be picked.
--- include/uapi/linux/io_uring.h | 3 ++- io_uring/kbuf.c | 8 +++++++- io_uring/kbuf.h | 7 +++++++ 3 files changed, 16 insertions(+), 2 deletions(-)
include/uapi/linux/io_uring.h +2 -1
@@ -905,7 +905,8 @@ struct io_uring_buf_reg {__u32 ring_entries;__u16 bgid;__u16 flags; -__u64 resv[3 ]; +__u32 min_left; +__u32 resv[5 ]; };/* argument for IORING_REGISTER_PBUF_STATUS */
io_uring/kbuf.c +7 -1
@@ -47,7 +47,7 @@ static bool io_kbuf_inc_commit(struct io_buffer_list *bl, int len) this_len =min_t (u32, len, buf_len); buf_len -= this_len;/* Stop looping for invalid buffer length of 0 */ -if (buf_len || !this_len) { +if (buf_len > bl->min_left_sub_one || !this_len) {WRITE_ONCE (buf->addr,READ_ONCE (buf->addr) + this_len);WRITE_ONCE (buf->len, buf_len);return false;
@@ -637,6 +637,10 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg)if (reg.ring_entries >=65536 )return -EINVAL ; +/* minimum left byte count is a property of incremental buffers */ +if (!(reg.flags &IOU_PBUF_RING_INC ) && reg.min_left) +return -EINVAL ; + bl =io_buffer_get_list (ctx, reg.bgid);if (bl) {/* if mapped buffer ring OR classic exists, don't allow */
@@ -683,6 +687,8 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg) bl->mask = reg.ring_entries -1 ; bl->flags |=IOBL_BUF_RING ; bl->buf_ring = br; +if (reg.min_left) + bl->min_left_sub_one = reg.min_left -1 ;if (reg.flags &IOU_PBUF_RING_INC ) bl->flags |=IOBL_INC ; ret =io_buffer_add_list (ctx, bl, reg.bgid);
io_uring/kbuf.h +7 -0
@@ -32,6 +32,13 @@ struct io_buffer_list {__u16 flags; +/* +* minimum required amount to be left to reuse an incrementally +* consumed buffer. If less than this is left at consumption time, +* buffer is done and head is incremented to the next buffer. +*/ +__u32 min_left_sub_one; +struct io_mapped_region region; };
Signature · 1 line
2.53.0
Re: [PATCH 2/2] io_uring/kbuf: support min length left for incremental buffers
Quoted 33 lines · Jens Axboe
From: Martin Michaelis <[email protected]>
Incrementally consumed buffer rings are generally fully consumed, but it's quite possible that the application has a minimum size it needs to meet to avoid truncation. Currently that minimum limit is 1 byte, but this should be a setting that is the hands of the application. For recvmsg multishot, a prime use case for incrementally consumed buffers, the application may get spurious
-EFAULTreturned at the end of an incrementally consumed buffer, as less space is available than the headers need.Grab a u32 field in struct
io_uring_buf_reg, which the application can use to inform the kernel of the minimum size that should be available in an incrementally consumed buffer. If less than that is available, the current buffer is fully processed and the next one will be picked.Fixes ae98dbf43d75 ("io_uring/kbuf: add support for incremental buffer consumption")Signed-off-by Martin Michaelis <[email protected]>[axboe: write commit message, change io_buffer_list member name]Signed-off-by Jens Axboe <[email protected]>--- include/uapi/linux/io_uring.h | 3 ++- io_uring/kbuf.c | 8 +++++++- io_uring/kbuf.h | 7 +++++++ 3 files changed, 16 insertions(+), 2 deletions(-)include/uapi/linux/io_uring.h +0
-0
include/uapi/linux/io_uring.h struct io_uring_buf_reg { · line 909
Quoted 6 lines
include/uapi/linux/io_uring.h +2
-1@@ -905,7 +905,8 @@ struct io_uring_buf_reg {__u32 ring_entries;__u16 bgid;__u16 flags; -__u64 resv[3 ]; +__u32 min_left; +__u32 resv[5 ];
Honest question, isn't this a property of the specific operation and/or fd being operated, instead of the buffer_reg?
io_uring/kbuf.c static bool io_kbuf_inc_commit(struct io_buffer_list *bl, int len) · line 50
Quoted 13 lines
include/uapi/linux/io_uring.h +0
-0};/* argument for IORING_REGISTER_PBUF_STATUS */ io_uring/kbuf.c +1
-1@@ -47,7 +47,7 @@ static bool io_kbuf_inc_commit(struct io_buffer_list *bl, int len) this_len =min_t (u32, len, buf_len); buf_len -= this_len;/* Stop looping for invalid buffer length of 0 */ -if (buf_len || !this_len) { +if (buf_len > bl->min_left_sub_one || !this_len) {
Cosmetic, but perhaps store min_left_sub_one instead of min_left itself? the buf_len must be >= min_left, and that is easier to read. (buf_len && buf_len >= min_left || !this_len)
Quoted 40 lines
io_uring/kbuf.c +6
-0WRITE_ONCE (buf->addr,READ_ONCE (buf->addr) + this_len);WRITE_ONCE (buf->len, buf_len);return false;@@ -637,6 +637,10 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg)if (reg.ring_entries >=65536 )return -EINVAL ; +/* minimum left byte count is a property of incremental buffers */ +if (!(reg.flags &IOU_PBUF_RING_INC ) && reg.min_left) +return -EINVAL ; + bl =io_buffer_get_list (ctx, reg.bgid);if (bl) {/* if mapped buffer ring OR classic exists, don't allow */ @@ -683,6 +687,8 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg) bl->mask = reg.ring_entries -1 ; bl->flags |=IOBL_BUF_RING ; bl->buf_ring = br; +if (reg.min_left) + bl->min_left_sub_one = reg.min_left -1 ;if (reg.flags &IOU_PBUF_RING_INC ) bl->flags |=IOBL_INC ; ret =io_buffer_add_list (ctx, bl, reg.bgid);io_uring/kbuf.h +7
-0@@ -32,6 +32,13 @@ struct io_buffer_list {__u16 flags; +/* +* minimum required amount to be left to reuse an incrementally +* consumed buffer. If less than this is left at consumption time, +* buffer is done and head is incremented to the next buffer. +*/ +__u32 min_left_sub_one; +struct io_mapped_region region; };
Signature · 1 line
Gabriel Krisman Bertazi
Re: [PATCH 1/2] io_uring/kbuf: kill dead struct io_buffer_list 'nr_entries' member
In reply to Jens AxboeQuoted 4 lines · Jens Axboe
This is only ever assigned, never used. The only used part is the calculated mask, which is used for indexing. Kill '
nr_entries'.Signed-off-by Jens Axboe <[email protected]>
Quoted 28 lines
--- io_uring/kbuf.c | 1 - io_uring/kbuf.h | 1 - 2 files changed, 2 deletions(-)io_uring/kbuf.c +0
-1@@ -680,7 +680,6 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg) }#endif - bl->nr_entries = reg.ring_entries; bl->mask = reg.ring_entries -1 ; bl->flags |=IOBL_BUF_RING ; bl->buf_ring = br;io_uring/kbuf.h +0
-1@@ -27,7 +27,6 @@ struct io_buffer_list {__u16 bgid;/* below is for ring provided buffers */ -__u16 nr_entries;__u16 head;__u16 mask;
Signature · 1 line
Gabriel Krisman Bertazi
Jens Axboe
maintainerRe: [PATCH 2/2] io_uring/kbuf: support min length left for incremental buffers
In reply to Gabriel Krisman BertaziQuoted 30 lines · Gabriel Krisman Bertazi
Quoted 28 lines · Jens Axboe
From: Martin Michaelis <[email protected]>
Incrementally consumed buffer rings are generally fully consumed, but it's quite possible that the application has a minimum size it needs to meet to avoid truncation. Currently that minimum limit is 1 byte, but this should be a setting that is the hands of the application. For recvmsg multishot, a prime use case for incrementally consumed buffers, the application may get spurious
-EFAULTreturned at the end of an incrementally consumed buffer, as less space is available than the headers need.Grab a u32 field in struct
io_uring_buf_reg, which the application can use to inform the kernel of the minimum size that should be available in an incrementally consumed buffer. If less than that is available, the current buffer is fully processed and the next one will be picked.Fixes ae98dbf43d75 ("io_uring/kbuf: add support for incremental buffer consumption")Signed-off-by Martin Michaelis <[email protected]>[axboe: write commit message, change io_buffer_list member name]Signed-off-by Jens Axboe <[email protected]>--- include/uapi/linux/io_uring.h | 3 ++- io_uring/kbuf.c | 8 +++++++- io_uring/kbuf.h | 7 +++++++ 3 files changed, 16 insertions(+), 2 deletions(-)
Quoted 14 lines
include/uapi/linux/io_uring.h struct io_uring_buf_reg { · line 909
Quoted 11 lines
include/uapi/linux/io_uring.h +2
-1@@ -905,7 +905,8 @@ struct io_uring_buf_reg {__u32 ring_entries;__u16 bgid;__u16 flags; -__u64 resv[3 ]; +__u32 min_left; +__u32 resv[5 ];Honest question, isn't this a property of the specific operation and/or fd being operated, instead of the
buffer_reg?
It kind of is, in that some users may not care. But it's not currently possible to pass this in on a per-op basis, and while I did hack that up initially, it's almost impossible as you end up with layering violations. In practice, this is really mostly a recvmsg multishot issue, because we need to store the headers. Hence the solution to stuff it in the io_uring_buf_reg instead, and make it a fixed property of the buffer group. In practice, you may even want a larger min_left than what the recvmsg requires, as you don't want a tiny truncated transfer at the end, regardless of what type of recv or read operation this is. Hence it works generically as well.
Also see the linked GH issue, that's where most of the discussion around this have happened already.
Quoted 7 lines
Quoted 7 lines
include/uapi/linux/io_uring.h +0
-0/* argument for IORING_REGISTER_PBUF_STATUS */ io_uring/kbuf.c +0
-0@@ -47,7 +47,7 @@ static bool io_kbuf_inc_commit(struct io_buffer_list *bl, int len) this_len =min_t (u32, len, buf_len);
Quoted 8 lines
io_uring/kbuf.c static bool io_kbuf_inc_commit(struct io_buffer_list *bl, int len) · line 50
Quoted 4 lines
io_uring/kbuf.c +1
-1buf_len -= this_len;/* Stop looping for invalid buffer length of 0 */ -if (buf_len || !this_len) { +if (buf_len > bl->min_left_sub_one || !this_len) {Cosmetic, but perhaps store
min_left_sub_oneinstead ofmin_leftitself? thebuf_lenmust be >=min_left, and that is easier to read. (buf_len&&buf_len>=min_left|| !this_len)
Also see GH issue.
Signature · 1 line
Jens Axboe
Gabriel Krisman Bertazi
Reviewed-byRe: [PATCH 2/2] io_uring/kbuf: support min length left for incremental buffers
Quoted 17 lines · Jens Axboe
Quoted 2 lines
Honest question, isn't this a property of the specific operation and/or fd being operated, instead of the
buffer_reg?It kind of is, in that some users may not care. But it's not currently possible to pass this in on a per-op basis, and while I did hack that up initially, it's almost impossible as you end up with layering violations. In practice, this is really mostly a recvmsg multishot issue, because we need to store the headers. Hence the solution to stuff it in the
io_uring_buf_reginstead, and make it a fixed property of the buffer group. In practice, you may even want a largermin_leftthan what the recvmsg requires, as you don't want a tiny truncated transfer at the end, regardless of what type of recv or read operation this is. Hence it works generically as well.Also see the linked GH issue, that's where most of the discussion around this have happened already.
Quoted 8 lines
Quoted 6 lines
io_uring/kbuf.c static bool io_kbuf_inc_commit(struct io_buffer_list *bl, int len) · line 50
Quoted 2 lines
io_uring/kbuf.c +1
-1-if (buf_len || !this_len) { +if (buf_len > bl->min_left_sub_one || !this_len) {Cosmetic, but perhaps store
min_left_sub_oneinstead ofmin_leftitself? thebuf_lenmust be >=min_left, and that is easier to read. (buf_len&&buf_len>=min_left|| !this_len)Also see GH issue.
Ack. Thanks. Feel free to add:
Signature · 1 line
Gabriel Krisman Bertazi