io_uring zerocopy send
Subsystems (7)
v5 cover io_uring zerocopy send
NOTE: Not to be picked directly. After getting necessary acks, I'll be
working out merging with Jakub and Jens.
The patchset implements io_uring zerocopy send. It works with both registered and normal buffers, mixing is allowed but not recommended. Apart from usual request completions, just as with MSG_ZEROCOPY, io_uring separately notifies the userspace when buffers are freed and can be reused (see API design below), which is delivered into io_uring's Completion Queue. Those "buffer-free" notifications are not necessarily per request, but the userspace has control over it and should explicitly attaching a number of requests to a single notification. The series also adds some internal optimisations when used with registered buffers like removing page referencing.
From the kernel networking perspective there are two main changes. The first one is passing ubuf_info into the network layer from io_uring (inside of an in kernel struct msghdr). This allows extra optimisations, e.g. ubuf_info caching on the io_uring side, but also helps to avoid cross-referencing and synchronisation problems. The second part is an optional optimisation removing page referencing for requests with registered buffers.
Benchmarking UDP with an optimised version of the selftest (see [1]), which sends a bunch of requests, waits for completions and repeats. "+ flush" column posts one additional "buffer-free" notification per request, and just "zc" doesn't post buffer notifications at all.
NIC (requests / second):
IO size | non-zc | zc | zc + flush 4000 | 495134 | 606420 (+22%) | 558971 (+12%) 1500 | 551808 | 577116 (+4.5%) | 565803 (+2.5%) 1000 | 584677 | 592088 (+1.2%) | 560885 (-4%) 600 | 596292 | 598550 (+0.4%) | 555366 (-6.7%)
dummy (requests / second):
IO size | non-zc | zc | zc + flush 8000 | 1299916 | 2396600 (+84%) | 2224219 (+71%) 4000 | 1869230 | 2344146 (+25%) | 2170069 (+16%) 1200 | 2071617 | 2361960 (+14%) | 2203052 (+6%) 600 | 2106794 | 2381527 (+13%) | 2195295 (+4%)
Previously it also brought a massive performance speedup compared to the msg_zerocopy tool (see [3]), which is probably not super interesting. There is also an additional bunch of refcounting optimisations that was omitted from the series for simplicity and as they don't change the picture drastically, they will be sent as follow up, as well as flushing optimisations closing the performance gap b/w two last columns.
For TCP on localhost (with hacks enabling localhost zerocopy) and including additional overhead for receive:
IO size | non-zc | zc 1200 | 4174 | 4148 4096 | 7597 | 11228
Using a real NIC 1200 bytes, zc is worse than non-zc ~5-10%, maybe the omitted optimisations will somewhat help, should look better for 4000, but couldn't test properly because of setup problems.
Links:
liburing (benchmark + tests):
[1] https://github.com/isilence/liburing/tree/zc_v4
kernel repo:
[2] https://github.com/isilence/linux/tree/zc_v4
RFC v1:
[3] https://lore.kernel.org/io-uring/[email protected]/
RFC v2:
https://lore.kernel.org/io-uring/[email protected]/
Net patches based:
[email protected]:isilence/linux.git zc_v4-net-base or
https://github.com/isilence/linux/tree/zc_v4-net-base
API design overview:
The series introduces an io_uring concept of notifactors. From the userspace perspective it's an entity to which it can bind one or more requests and then requesting to flush it. Flushing a notifier makes it impossible to attach new requests to it, and instructs the notifier to post a completion once all requests attached to it are completed and the kernel doesn't need the buffers anymore.
Notifications are stored in notification slots, which should be registered as an array in io_uring. Each slot stores only one notifier at any particular moment. Flushing removes it from the slot and the slot automatically replaces it with a new notifier. All operations with notifiers are done by specifying an index of a slot it's currently in.
When registering a notification the userspace specifies a u64 tag for each slot, which will be copied in notification completion entries as cqe::user_data. cqe::res is 0 and cqe::flags is equal to wrap around u32 sequence number counting notifiers of a slot.
Changelog:
v4 -> v5
remove ubuf_info checks from custom iov_iter callbacks to avoid disabling the page refs optimisations for TCP
v3 -> v4
custom iov_iter handling
RFC v2 -> v3:
mem accounting for non-registered buffers allow mixing registered and normal requests per notifier notification flushing via IORING_OP_RSRC_UPDATE TCP support
fix buffer indexing
fix io-wq ->uring_lock locking
fix bugs when mixing with MSG_ZEROCOPY
fix managed refs bugs in skbuff.c
RFC -> RFC v2:
remove additional overhead for non-zc from skb_release_data() avoid msg propagation, hide extra bits of non-zc overhead task_work based "buffer free" notifications
improve io_uring's notification refcounting
added 5/19, (no pfmemalloc tracking)
added 8/19 and 9/19 preventing small copies with zc misc small changes
David Ahern (1):
net: Allow custom iter handler in msghdr
Pavel Begunkov (26):
ipv4: avoid partial copy for zc
ipv6: avoid partial copy for zc
skbuff: don't mix ubuf_info from different sources
skbuff: add SKBFL_DONT_ORPHAN flag
skbuff: carry external ubuf_info in msghdr
net: introduce managed frags infrastructure
net: introduce __skb_fill_page_desc_noacc ipv4/udp: support externally provided ubufs ipv6/udp: support externally provided ubufs
tcp: support externally provided ubufs
io_uring: initialise msghdr::msg_ubuf
io_uring: export io_put_task()
io_uring: add zc notification infrastructure
io_uring: cache struct io_notif
io_uring: complete notifiers in tw
io_uring: add rsrc referencing for notifiers
io_uring: add notification slot registration
io_uring: wire send zc request type
io_uring: account locked pages for non-fixed zc
io_uring: allow to pass addr into sendzc
io_uring: sendzc with fixed buffers
io_uring: flush notifiers after sendzc
io_uring: rename IORING_OP_FILES_UPDATE
io_uring: add zc notification flush requests
io_uring: enable managed frags with register buffers selftests/io_uring: test zerocopy send
include/linux/io_uring_types.h | 37 ++ include/linux/skbuff.h | 66 +- include/linux/socket.h | 5 + include/uapi/linux/io_uring.h | 45 +- io_uring/Makefile | 2 +- io_uring/io_uring.c | 42 +- io_uring/io_uring.h | 22 + io_uring/net.c | 187 ++++++ io_uring/net.h | 4 + io_uring/notif.c | 215 +++++++ io_uring/notif.h | 87 +++ io_uring/opdef.c | 24 +- io_uring/rsrc.c | 55 +- io_uring/rsrc.h | 16 +- io_uring/tctx.h | 26 - net/compat.c | 1 + net/core/datagram.c | 14 +- net/core/skbuff.c | 37 +- net/ipv4/ip_output.c | 50 +- net/ipv4/tcp.c | 32 +- net/ipv6/ip6_output.c | 49 +- net/socket.c | 3 + tools/testing/selftests/net/Makefile | 1 + .../selftests/net/io_uring_zerocopy_tx.c | 605 ++++++++++++++++++ .../selftests/net/io_uring_zerocopy_tx.sh | 131 ++++ 25 files changed, 1628 insertions(+), 128 deletions(-) create mode 100644 io_uring/notif.c create mode 100644 io_uring/notif.h create mode 100644 tools/testing/selftests/net/io_uring_zerocopy_tx.c create mode 100755 tools/testing/selftests/net/io_uring_zerocopy_tx.sh
Signature · 1 line
2.37.0
PATCH v5 1/27 ipv4: avoid partial copy for zc
Even when zerocopy transmission is requested and possible, __ip_append_data() will still copy a small chunk of data just because it allocated some extra linear space (e.g. 148 bytes). It wastes CPU cycles on copy and iter manipulations and also misalignes potentially aligned data. Avoid such coies. And as a bonus we can allocate smaller skb.
--- net/ipv4/ip_output.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
net/ipv4/ip_output.c +6 -2
@@ -969,7 +969,6 @@ static int __ip_append_data(struct sock *sk,struct inet_sock *inet =inet_sk (sk);struct ubuf_info *uarg =NULL ;struct sk_buff *skb; -struct ip_options *opt = cork->opt;int hh_len;int exthdrlen;
@@ -977,6 +976,7 @@ static int __ip_append_data(struct sock *sk,int copy;int err;int offset =0 ; +bool zc = false;unsigned int maxfraglen, fragheaderlen, maxnonfragsize;int csummode =CHECKSUM_NONE ;struct rtable *rt = (struct rtable *)cork->dst;
@@ -1025,6 +1025,7 @@ static int __ip_append_data(struct sock *sk,if (rt->dst.dev->features &NETIF_F_SG && csummode ==CHECKSUM_PARTIAL ) { paged = true; + zc = true; }else { uarg->zerocopy =0 ;skb_zcopy_set (skb, uarg, &extra_uref);
@@ -1091,9 +1092,12 @@ static int __ip_append_data(struct sock *sk, (fraglen + alloc_extra <SKB_MAX_ALLOC || !(rt->dst.dev->features & NETIF_F_SG))) alloclen = fraglen; - else { + else if (!zc) { alloclen =min_t (int, fraglen,MAX_HEADER ); pagedlen = fraglen - alloclen; + } else { + alloclen = fragheaderlen + transhdrlen; + pagedlen = datalen - transhdrlen; } alloclen += alloc_extra;
Signature · 1 line
2.37.0
Jakub Kicinski
maintainerRe: [PATCH net-next v5 01/27] ipv4: avoid partial copy for zc
Quoted 5 lines · On Tue, 12 Jul 2022 21:52:25 +0100 Pavel Begunkov wrote:
Even when zerocopy transmission is requested and possible,
__ip_append_data()will still copy a small chunk of data just because it allocated some extra linear space (e.g. 148 bytes). It wastes CPU cycles on copy and iter manipulations and also misalignes potentially aligned data. Avoid such coies. And as a bonus we can allocate smaller skb.
s/coies/copies/ can fix when applying
Quoted 35 lines
Signed-off-by Pavel Begunkov <[email protected]>--- net/ipv4/ip_output.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)net/ipv4/ip_output.c +2
-1@@ -969,7 +969,6 @@ static int __ip_append_data(struct sock *sk,struct inet_sock *inet =inet_sk (sk);struct ubuf_info *uarg =NULL ;struct sk_buff *skb; -struct ip_options *opt = cork->opt;int hh_len;int exthdrlen;@@ -977,6 +976,7 @@ static int __ip_append_data(struct sock *sk,int copy;int err;int offset =0 ; +bool zc = false;unsigned int maxfraglen, fragheaderlen, maxnonfragsize;int csummode =CHECKSUM_NONE ;struct rtable *rt = (struct rtable *)cork->dst;@@ -1025,6 +1025,7 @@ static int __ip_append_data(struct sock *sk,if (rt->dst.dev->features &NETIF_F_SG && csummode ==CHECKSUM_PARTIAL ) { paged = true; + zc = true; }else { uarg->zerocopy =0 ;skb_zcopy_set (skb, uarg, &extra_uref);
net/ipv4/ip_output.c static int __ip_append_data(struct sock *sk, · line 1096
Quoted 6 lines
net/ipv4/ip_output.c +1
-1@@ -1091,9 +1092,12 @@ static int __ip_append_data(struct sock *sk, (fraglen + alloc_extra < SKB_MAX_ALLOC || !(rt->dst.dev->features & NETIF_F_SG))) alloclen = fraglen; - else { + else if (!zc) { alloclen =min_t (int, fraglen,MAX_HEADER );
Willem, I think this came in with your GSO work, is there a reason we use MAX_HEADER here? I thought MAX_HEADER is for headers (i.e. more or less to be reserved) not for the min amount of data to be included.
I wanna make sure we're not missing something about GSO here.
Otherwise I don't think we need the extra branch but that can be a follow up.
Quoted 7 lines
net/ipv4/ip_output.c +3
-0pagedlen = fraglen - alloclen; + }else { + alloclen = fragheaderlen + transhdrlen; + pagedlen = datalen - transhdrlen; } alloclen += alloc_extra;
Willem de Bruijn
maintainerRe: [PATCH net-next v5 01/27] ipv4: avoid partial copy for zc
Quoted 56 lines · Jakub Kicinski
Quoted 5 lines · On Tue, 12 Jul 2022 21:52:25 +0100 Pavel Begunkov wrote:
Even when zerocopy transmission is requested and possible,
__ip_append_data()will still copy a small chunk of data just because it allocated some extra linear space (e.g. 148 bytes). It wastes CPU cycles on copy and iter manipulations and also misalignes potentially aligned data. Avoid such coies. And as a bonus we can allocate smaller skb.s/coies/copies/ can fix when applying
Quoted 35 lines
Signed-off-by Pavel Begunkov <[email protected]>--- net/ipv4/ip_output.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)net/ipv4/ip_output.c +2
-1@@ -969,7 +969,6 @@ static int __ip_append_data(struct sock *sk,struct inet_sock *inet =inet_sk (sk);struct ubuf_info *uarg =NULL ;struct sk_buff *skb; -struct ip_options *opt = cork->opt;int hh_len;int exthdrlen;@@ -977,6 +976,7 @@ static int __ip_append_data(struct sock *sk,int copy;int err;int offset =0 ; +bool zc = false;unsigned int maxfraglen, fragheaderlen, maxnonfragsize;int csummode =CHECKSUM_NONE ;struct rtable *rt = (struct rtable *)cork->dst;@@ -1025,6 +1025,7 @@ static int __ip_append_data(struct sock *sk,if (rt->dst.dev->features &NETIF_F_SG && csummode ==CHECKSUM_PARTIAL ) { paged = true; + zc = true; }else { uarg->zerocopy =0 ;skb_zcopy_set (skb, uarg, &extra_uref);net/ipv4/ip_output.c static int __ip_append_data(struct sock *sk, · line 1096
Quoted 6 lines
net/ipv4/ip_output.c +1
-1@@ -1091,9 +1092,12 @@ static int __ip_append_data(struct sock *sk, (fraglen + alloc_extra < SKB_MAX_ALLOC || !(rt->dst.dev->features & NETIF_F_SG))) alloclen = fraglen; - else { + else if (!zc) { alloclen =min_t (int, fraglen,MAX_HEADER );Willem, I think this came in with your GSO work, is there a reason we use
MAX_HEADERhere? I thoughtMAX_HEADERis for headers (i.e. more or less to be reserved) not for the min amount of data to be included.
Quoted 4 lines
I wanna make sure we're not missing something about GSO here.
Otherwise I don't think we need the extra branch but that can be a follow up.
The change was introduced for UDP GSO, to avoid copying most payload on software segmentation:
"
commit 15e36f5b8e982debe43e425d2e12d34e022d51e9
Author: Willem de Bruijn <[email protected]> Date: Thu Apr 26 13:42:19 2018 -0400
udp: paged allocation with gso
When sending large datagrams that are later segmented, store data in page frags to avoid copying from linear in skb_segment.
"
and in code
- else
- alloclen = datalen + fragheaderlen;
+ else if (!paged)
+ alloclen = fraglen;
+ else {
+ alloclen = min_t(int, fraglen, MAX_HEADER);
+ pagedlen = fraglen - alloclen;
+ }
MAX_HEADER was a short-hand for the exact header length. "alloclen = fragheaderlen + transhdrlen;" is probably a better choice indeed.
Whether with branch or without, the same change needs to be made to __ip6_append_data, just as in the referenced commit. Let's keep the stacks in sync.
This is tricky code. If in doubt, run the msg_zerocopy and udp_gso tests from tools/testing/selftests/net, ideally with KASAN.
Re: [PATCH net-next v5 01/27] ipv4: avoid partial copy for zc
Quoted 57 lines · Willem de Bruijn
Quoted 56 lines · Jakub Kicinski
Quoted 5 lines · On Tue, 12 Jul 2022 21:52:25 +0100 Pavel Begunkov wrote:
Even when zerocopy transmission is requested and possible,
__ip_append_data()will still copy a small chunk of data just because it allocated some extra linear space (e.g. 148 bytes). It wastes CPU cycles on copy and iter manipulations and also misalignes potentially aligned data. Avoid such coies. And as a bonus we can allocate smaller skb.s/coies/copies/ can fix when applying
Quoted 35 lines
Signed-off-by Pavel Begunkov <[email protected]>--- net/ipv4/ip_output.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)net/ipv4/ip_output.c +2
-1@@ -969,7 +969,6 @@ static int __ip_append_data(struct sock *sk,struct inet_sock *inet =inet_sk (sk);struct ubuf_info *uarg =NULL ;struct sk_buff *skb; -struct ip_options *opt = cork->opt;int hh_len;int exthdrlen;@@ -977,6 +976,7 @@ static int __ip_append_data(struct sock *sk,int copy;int err;int offset =0 ; +bool zc = false;unsigned int maxfraglen, fragheaderlen, maxnonfragsize;int csummode =CHECKSUM_NONE ;struct rtable *rt = (struct rtable *)cork->dst;@@ -1025,6 +1025,7 @@ static int __ip_append_data(struct sock *sk,if (rt->dst.dev->features &NETIF_F_SG && csummode ==CHECKSUM_PARTIAL ) { paged = true; + zc = true; }else { uarg->zerocopy =0 ;skb_zcopy_set (skb, uarg, &extra_uref);net/ipv4/ip_output.c static int __ip_append_data(struct sock *sk, · line 1096
Quoted 6 lines
net/ipv4/ip_output.c +1
-1@@ -1091,9 +1092,12 @@ static int __ip_append_data(struct sock *sk, (fraglen + alloc_extra < SKB_MAX_ALLOC || !(rt->dst.dev->features & NETIF_F_SG))) alloclen = fraglen; - else { + else if (!zc) { alloclen =min_t (int, fraglen,MAX_HEADER );Willem, I think this came in with your GSO work, is there a reason we use
MAX_HEADERhere? I thoughtMAX_HEADERis for headers (i.e. more or less to be reserved) not for the min amount of data to be included.
Quoted 4 lines
Quoted 4 lines
I wanna make sure we're not missing something about GSO here.
Otherwise I don't think we need the extra branch but that can be a follow up.
I brought it up before but left it for later as I don't know workloads and there might be perf implications. I'll send a follow up.
Quoted 26 lines
The change was introduced for UDP GSO, to avoid copying most payload on software segmentation:
"
commit15e36f5b8e982debe43e425d2e12d34e022d51e9Author: Willem de Bruijn <[email protected]> Date: Thu Apr 26 13:42:19 2018 -0400udp: paged allocation with gso
When sending large datagrams that are later segmented, store data in page frags to avoid copying from linear in
skb_segment.
"and in code
(quoted diff) +6
-2- else - alloclen = datalen + fragheaderlen; + else if (!paged) + alloclen = fraglen; + else { + alloclen = min_t(int, fraglen, MAX_HEADER); + pagedlen = fraglen - alloclen; + }
Quoted 2 lines
MAX_HEADERwas a short-hand for the exact header length. "alloclen = fragheaderlen + transhdrlen;" is probably a better choice indeed.
Great, thanks for taking a look!
Quoted 4 lines
Whether with branch or without, the same change needs to be made to
__ip6_append_data, just as in the referenced commit. Let's keep the stacks in sync.
__ip6_append_data() is changed as well but in the following patch. I had doubts whether it's preferable to keep ipv4 and ipv6 changes separately.
Quoted 2 lines
This is tricky code. If in doubt, run the
msg_zerocopyandudp_gsotests fromtools/testing/selftests/net, ideally with KASAN.
Signature · 1 line
Pavel Begunkov
PATCH v5 2/27 ipv6: avoid partial copy for zc
In reply to Pavel BegunkovEven when zerocopy transmission is requested and possible, __ip_append_data() will still copy a small chunk of data just because it allocated some extra linear space (e.g. 128 bytes). It wastes CPU cycles on copy and iter manipulations and also misalignes potentially aligned data. Avoid such coies. And as a bonus we can allocate smaller skb.
--- net/ipv6/ip6_output.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
net/ipv6/ip6_output.c +6 -1
@@ -1464,6 +1464,7 @@ static int __ip6_append_data(struct sock *sk,int copy;int err;int offset =0 ; +bool zc = false;u32 tskey =0 ;struct rt6_info *rt = (struct rt6_info *)cork->dst;struct ipv6_txoptions *opt = v6_cork->opt;
@@ -1549,6 +1550,7 @@ static int __ip6_append_data(struct sock *sk,if (rt->dst.dev->features &NETIF_F_SG && csummode ==CHECKSUM_PARTIAL ) { paged = true; + zc = true; }else { uarg->zerocopy =0 ;skb_zcopy_set (skb, uarg, &extra_uref);
@@ -1630,9 +1632,12 @@ static int __ip6_append_data(struct sock *sk, (fraglen + alloc_extra <SKB_MAX_ALLOC || !(rt->dst.dev->features & NETIF_F_SG))) alloclen = fraglen; - else { + else if (!zc) { alloclen =min_t (int, fraglen,MAX_HEADER ); pagedlen = fraglen - alloclen; + } else { + alloclen = fragheaderlen + transhdrlen; + pagedlen = datalen - transhdrlen; } alloclen += alloc_extra;
Signature · 1 line
2.37.0
PATCH v5 3/27 skbuff: don't mix ubuf_info from different sources
In reply to Pavel BegunkovWe should not append MSG_ZEROCOPY requests to skbuff with non MSG_ZEROCOPY ubuf_info, they might be not compatible.
--- net/core/skbuff.c | 4 ++++ 1 file changed, 4 insertions(+)
net/core/skbuff.c +4 -0
@@ -1212,6 +1212,10 @@ struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size,const u32 byte_limit =1 <<19 ;/* limit to a few TSO */ u32 bytelen, next; +/* there might be non MSG_ZEROCOPY users */ +if (uarg->callback != msg_zerocopy_callback) +return NULL ; +/* realloc only when socket is locked (TCP, UDP cork), * so uarg->len and sk_zckey access is serialized */
Signature · 1 line
2.37.0
PATCH v5 4/27 skbuff: add SKBFL_DONT_ORPHAN flag
In reply to Pavel BegunkovWe don't want to list every single ubuf_info callback in skb_orphan_frags(), add a flag controlling the behaviour.
--- include/linux/skbuff.h | 8 +++++--- net/core/skbuff.c | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-)
include/linux/skbuff.h +5 -3
@@ -686,10 +686,13 @@ enum {* charged to the kernel memory. */ SKBFL_PURE_ZEROCOPY =BIT (2 ), + + SKBFL_DONT_ORPHAN = BIT(3), };#define SKBFL_ZEROCOPY_FRAG (SKBFL_ZEROCOPY_ENABLE | SKBFL_SHARED_FRAG) -#define SKBFL_ALL_ZEROCOPY (SKBFL_ZEROCOPY_FRAG | SKBFL_PURE_ZEROCOPY) +#define SKBFL_ALL_ZEROCOPY (SKBFL_ZEROCOPY_FRAG | SKBFL_PURE_ZEROCOPY | \ + SKBFL_DONT_ORPHAN)/* * The callback notifies userspace to release buffers when skb DMA is done in
@@ -3182,8 +3185,7 @@ static inline int skb_orphan_frags(struct sk_buff *skb, gfp_t gfp_mask) {if (likely (!skb_zcopy (skb)))return 0 ; -if (!skb_zcopy_is_nouarg (skb) && -skb_uarg (skb)->callback == msg_zerocopy_callback) +if (skb_shinfo (skb)->flags &SKBFL_DONT_ORPHAN )return 0 ;return skb_copy_ubufs (skb, gfp_mask); }
net/core/skbuff.c +1 -1
@@ -1193,7 +1193,7 @@ static struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size) uarg->len =1 ; uarg->bytelen = size; uarg->zerocopy =1 ; - uarg->flags =SKBFL_ZEROCOPY_FRAG ; + uarg->flags =SKBFL_ZEROCOPY_FRAG |SKBFL_DONT_ORPHAN ;refcount_set (&uarg->refcnt,1 );sock_hold (sk);
Signature · 1 line
2.37.0
PATCH v5 5/27 skbuff: carry external ubuf_info in msghdr
In reply to Pavel BegunkovMake possible for network in-kernel callers like io_uring to pass in a custom ubuf_info by setting it in a new field of struct msghdr.
--- include/linux/socket.h | 1 + net/compat.c | 1 + net/socket.c | 3 +++ 3 files changed, 5 insertions(+)
include/linux/socket.h +1 -0
@@ -69,6 +69,7 @@ struct msghdr {unsigned int msg_flags;/* flags on received message */ __kernel_size_t msg_controllen;/* ancillary data buffer length */ struct kiocb *msg_iocb;/* ptr to iocb for async requests */ +struct ubuf_info *msg_ubuf; };struct user_msghdr {
net/compat.c +1 -0
@@ -80,6 +80,7 @@ int __get_compat_msghdr(struct msghdr *kmsg,return -EMSGSIZE ; kmsg->msg_iocb =NULL ; + kmsg->msg_ubuf =NULL ; *ptr = msg.msg_iov; *len = msg.msg_iovlen;return 0 ;
net/socket.c +3 -0
@@ -2106,6 +2106,7 @@ int __sys_sendto(int fd, void __user *buff, size_t len, unsigned int flags, msg.msg_control =NULL ; msg.msg_controllen =0 ; msg.msg_namelen =0 ; + msg.msg_ubuf =NULL ;if (addr) { err =move_addr_to_kernel (addr, addr_len, &address);if (err <0 )
@@ -2171,6 +2172,7 @@ int __sys_recvfrom(int fd, void __user *ubuf, size_t size, unsigned int flags, msg.msg_namelen =0 ; msg.msg_iocb =NULL ; msg.msg_flags =0 ; + msg.msg_ubuf =NULL ;if (sock->file->f_flags &O_NONBLOCK ) flags |=MSG_DONTWAIT ; err =sock_recvmsg (sock, &msg, flags);
@@ -2409,6 +2411,7 @@ int __copy_msghdr_from_user(struct msghdr *kmsg,return -EMSGSIZE ; kmsg->msg_iocb =NULL ; + kmsg->msg_ubuf =NULL ; *uiov = msg.msg_iov; *nsegs = msg.msg_iovlen;return 0 ;
Signature · 1 line
2.37.0
PATCH v5 6/27 net: Allow custom iter handler in msghdr
In reply to Pavel BegunkovFrom: David Ahern <[email protected]>
Add support for custom iov_iter handling to msghdr. The idea is that in-kernel subsystems want control over how an SG is split.
--- include/linux/skbuff.h | 7 ++++--- include/linux/socket.h | 4 ++++ net/core/datagram.c | 14 ++++++++++---- net/core/skbuff.c | 2 +- 4 files changed, 19 insertions(+), 8 deletions(-)
include/linux/skbuff.h +4 -3
@@ -1776,13 +1776,14 @@ void msg_zerocopy_put_abort(struct ubuf_info *uarg, bool have_uref);void msg_zerocopy_callback (struct sk_buff *skb,struct ubuf_info *uarg,bool success); -int __zerocopy_sg_from_iter (struct sock *sk,struct sk_buff *skb, -struct iov_iter *from,size_t length); +int __zerocopy_sg_from_iter (struct msghdr *msg,struct sock *sk, +struct sk_buff *skb,struct iov_iter *from, +size_t length);static inline int skb_zerocopy_iter_dgram (struct sk_buff *skb,struct msghdr *msg,int len) { -return __zerocopy_sg_from_iter (skb->sk, skb, &msg->msg_iter, len); +return __zerocopy_sg_from_iter (msg, skb->sk, skb, &msg->msg_iter, len); } int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb,
include/linux/socket.h +4 -0
@@ -14,6 +14,8 @@ struct file;struct pid ;struct cred ;struct socket ; +struct sock ; +struct sk_buff ;#define __sockaddr_check_size (size) \ BUILD_BUG_ON(((size) > sizeof(struct __kernel_sockaddr_storage)))
@@ -70,6 +72,8 @@ struct msghdr {__kernel_size_t msg_controllen;/* ancillary data buffer length */ struct kiocb *msg_iocb;/* ptr to iocb for async requests */ struct ubuf_info *msg_ubuf; +int (*sg_from_iter)(struct sock *sk,struct sk_buff *skb, +struct iov_iter *from,size_t length); };struct user_msghdr {
net/core/datagram.c +10 -4
@@ -613,10 +613,16 @@ int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, }EXPORT_SYMBOL (skb_copy_datagram_from_iter); -int __zerocopy_sg_from_iter (struct sock *sk,struct sk_buff *skb, -struct iov_iter *from,size_t length) +int __zerocopy_sg_from_iter (struct msghdr *msg,struct sock *sk, +struct sk_buff *skb,struct iov_iter *from, +size_t length) { -int frag =skb_shinfo (skb)->nr_frags; +int frag; + +if (msg && msg->sg_from_iter) +return msg->sg_from_iter (sk, skb, from, length); + + frag =skb_shinfo (skb)->nr_frags;while (length &&iov_iter_count (from)) {struct page *pages[MAX_SKB_FRAGS ];
@@ -702,7 +708,7 @@ int zerocopy_sg_from_iter(struct sk_buff *skb, struct iov_iter *from)if (skb_copy_datagram_from_iter (skb,0 , from, copy))return -EFAULT ; -return __zerocopy_sg_from_iter (NULL , skb, from, ~0U ); +return __zerocopy_sg_from_iter (NULL ,NULL , skb, from, ~0U ); }EXPORT_SYMBOL (zerocopy_sg_from_iter);
net/core/skbuff.c +1 -1
@@ -1358,7 +1358,7 @@ int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb,if (orig_uarg && uarg != orig_uarg)return -EEXIST ; - err =__zerocopy_sg_from_iter (sk, skb, &msg->msg_iter, len); + err =__zerocopy_sg_from_iter (msg, sk, skb, &msg->msg_iter, len);if (err == -EFAULT || (err == -EMSGSIZE && skb->len == orig_len)) {struct sock *save_sk = skb->sk;
Signature · 1 line
2.37.0
PATCH v5 7/27 net: introduce managed frags infrastructure
In reply to Pavel BegunkovSome users like io_uring can do page pinning more efficiently, so we want a way to delegate referencing to other subsystems. For that add a new flag called SKBFL_MANAGED_FRAG_REFS. When set, skb doesn't hold page references and upper layers are responsivle to managing page lifetime.
It's allowed to convert skbs from managed to normal by calling skb_zcopy_downgrade_managed(). The function will take all needed page references and clear the flag. It's needed, for instance, to avoid mixing managed modes.
--- include/linux/skbuff.h | 25 +++++++++++++++++++++++-- net/core/skbuff.c | 29 +++++++++++++++++++++++++++-- 2 files changed, 50 insertions(+), 4 deletions(-)
include/linux/skbuff.h +23 -2
@@ -688,11 +688,16 @@ enum {SKBFL_PURE_ZEROCOPY =BIT (2 ),SKBFL_DONT_ORPHAN =BIT (3 ), + +/* page references are managed by the ubuf_info, so it's safe to +* use frags only up until ubuf_info is released +*/ + SKBFL_MANAGED_FRAG_REFS = BIT(4), };#define SKBFL_ZEROCOPY_FRAG (SKBFL_ZEROCOPY_ENABLE | SKBFL_SHARED_FRAG)#define SKBFL_ALL_ZEROCOPY (SKBFL_ZEROCOPY_FRAG | SKBFL_PURE_ZEROCOPY | \ - SKBFL_DONT_ORPHAN) + SKBFL_DONT_ORPHAN | SKBFL_MANAGED_FRAG_REFS)/* * The callback notifies userspace to release buffers when skb DMA is done in
@@ -1810,6 +1815,11 @@ static inline bool skb_zcopy_pure(const struct sk_buff *skb)return skb_shinfo (skb)->flags &SKBFL_PURE_ZEROCOPY ; } +static inline bool skb_zcopy_managed (const struct sk_buff *skb) +{ +return skb_shinfo (skb)->flags &SKBFL_MANAGED_FRAG_REFS ; +} +static inline bool skb_pure_zcopy_same (const struct sk_buff *skb1,const struct sk_buff *skb2) {
@@ -1884,6 +1894,14 @@ static inline void skb_zcopy_clear(struct sk_buff *skb, bool zerocopy_success) } } +void __skb_zcopy_downgrade_managed (struct sk_buff *skb); + +static inline void skb_zcopy_downgrade_managed (struct sk_buff *skb) +{ +if (unlikely (skb_zcopy_managed (skb))) +__skb_zcopy_downgrade_managed (skb); +} +static inline void skb_mark_not_on_list (struct sk_buff *skb) { skb->next =NULL ;
@@ -3499,7 +3517,10 @@ static inline void __skb_frag_unref(skb_frag_t *frag, bool recycle)*/ static inline void skb_frag_unref (struct sk_buff *skb,int f) { -__skb_frag_unref (&skb_shinfo (skb)->frags[f], skb->pp_recycle); +struct skb_shared_info *shinfo =skb_shinfo (skb); + +if (!skb_zcopy_managed (skb)) +__skb_frag_unref (&shinfo->frags[f], skb->pp_recycle); }/**
net/core/skbuff.c +27 -2
@@ -666,11 +666,18 @@ static void skb_release_data(struct sk_buff *skb) &shinfo->dataref)) goto exit; -skb_zcopy_clear (skb, true); +if (skb_zcopy (skb)) { +bool skip_unref = shinfo->flags &SKBFL_MANAGED_FRAG_REFS ; + +skb_zcopy_clear (skb, true); +if (skip_unref) + goto free_head; + }for (i =0 ; i < shinfo->nr_frags; i++)__skb_frag_unref (&shinfo->frags[i], skb->pp_recycle); +free_head:if (shinfo->frag_list)kfree_skb_list (shinfo->frag_list);
@@ -895,7 +902,10 @@ EXPORT_SYMBOL(skb_dump);*/ void skb_tx_error (struct sk_buff *skb) { -skb_zcopy_clear (skb, true); +if (skb) { +skb_zcopy_downgrade_managed (skb); +skb_zcopy_clear (skb, true); + } }EXPORT_SYMBOL (skb_tx_error);
@@ -1375,6 +1385,16 @@ int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb, }EXPORT_SYMBOL_GPL (skb_zerocopy_iter_stream); +void __skb_zcopy_downgrade_managed (struct sk_buff *skb) +{ +int i; + +skb_shinfo (skb)->flags &= ~SKBFL_MANAGED_FRAG_REFS ; +for (i =0 ; i <skb_shinfo (skb)->nr_frags; i++) +skb_frag_ref (skb, i); +} +EXPORT_SYMBOL_GPL (__skb_zcopy_downgrade_managed); +static int skb_zerocopy_clone (struct sk_buff *nskb,struct sk_buff *orig,gfp_t gfp_mask) {
@@ -1692,6 +1712,8 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,BUG_ON (skb_shared (skb)); +skb_zcopy_downgrade_managed (skb); + size =SKB_DATA_ALIGN (size);if (skb_pfmemalloc (skb))
@@ -3488,6 +3510,8 @@ void skb_split(struct sk_buff *skb, struct sk_buff *skb1, const u32 len)int pos =skb_headlen (skb);const int zc_flags =SKBFL_SHARED_FRAG |SKBFL_PURE_ZEROCOPY ; +skb_zcopy_downgrade_managed (skb); +skb_shinfo (skb1)->flags |=skb_shinfo (skb)->flags & zc_flags;skb_zerocopy_clone (skb1, skb,0 );if (len < pos)/* Split line is inside header. */
@@ -3841,6 +3865,7 @@ int skb_append_pagefrags(struct sk_buff *skb, struct page *page,if (skb_can_coalesce (skb, i, page, offset)) {skb_frag_size_add (&skb_shinfo (skb)->frags[i -1 ], size); }else if (i <MAX_SKB_FRAGS ) { +skb_zcopy_downgrade_managed (skb);get_page (page);skb_fill_page_desc (skb, i, page, offset, size); }else {
Signature · 1 line
2.37.0
PATCH v5 8/27 net: introduce __skb_fill_page_desc_noacc
In reply to Pavel BegunkovManaged pages contain pinned userspace pages and controlled by upper layers, there is no need in tracking skb->pfmemalloc for them. Introduce a helper for filling frags but ignoring page tracking, it'll be needed later.
--- include/linux/skbuff.h | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-)
include/linux/skbuff.h +17 -11
@@ -2550,6 +2550,22 @@ static inline unsigned int skb_pagelen(const struct sk_buff *skb)return skb_headlen (skb) +__skb_pagelen (skb); } +static inline void __skb_fill_page_desc_noacc (struct skb_shared_info *shinfo, +int i,struct page *page, +int off,int size) +{ +skb_frag_t *frag = &shinfo->frags[i]; + +/* +* Propagate page pfmemalloc to the skb if we can. The problem is +* that not all callers have unique ownership of the page but rely +* on page_is_pfmemalloc doing the right thing(tm). +*/ + frag->bv_page = page; + frag->bv_offset = off; +skb_frag_size_set (frag, size); +} +/** * __skb_fill_page_desc - initialise a paged fragment in an skb * @skb: buffer containing fragment to be initialised
@@ -2566,17 +2582,7 @@ static inline unsigned int skb_pagelen(const struct sk_buff *skb)static inline void __skb_fill_page_desc (struct sk_buff *skb,int i,struct page *page,int off,int size) { -skb_frag_t *frag = &skb_shinfo (skb)->frags[i]; - -/* -* Propagate page pfmemalloc to the skb if we can. The problem is -* that not all callers have unique ownership of the page but rely -* on page_is_pfmemalloc doing the right thing(tm). -*/ - frag->bv_page = page; - frag->bv_offset = off; -skb_frag_size_set (frag, size); - +__skb_fill_page_desc_noacc (skb_shinfo (skb), i, page, off, size); page =compound_head (page);if (page_is_pfmemalloc (page)) skb->pfmemalloc = true;
Signature · 1 line
2.37.0
PATCH v5 9/27 ipv4/udp: support externally provided ubufs
In reply to Pavel BegunkovTeach ipv4/udp how to use external ubuf_info provided in msghdr and also prepare it for managed frags by sprinkling skb_zcopy_downgrade_managed() when it could mix managed and not managed frags.
--- net/ipv4/ip_output.c | 44 +++++++++++++++++++++++++++++++------------- 1 file changed, 31 insertions(+), 13 deletions(-)
net/ipv4/ip_output.c +31 -13
@@ -1017,18 +1017,35 @@ static int __ip_append_data(struct sock *sk, (!exthdrlen || (rt->dst.dev->features & NETIF_F_HW_ESP_TX_CSUM))) csummode =CHECKSUM_PARTIAL ; -if (flags &MSG_ZEROCOPY && length &&sock_flag (sk,SOCK_ZEROCOPY )) { - uarg =msg_zerocopy_realloc (sk, length,skb_zcopy (skb)); -if (!uarg) -return -ENOBUFS ; - extra_uref = !skb_zcopy (skb);/* only ref on new uarg */ -if (rt->dst.dev->features &NETIF_F_SG && - csummode ==CHECKSUM_PARTIAL ) { - paged = true; - zc = true; - }else { - uarg->zerocopy =0 ; -skb_zcopy_set (skb, uarg, &extra_uref); +if ((flags &MSG_ZEROCOPY ) && length) { +struct msghdr *msg = from; + +if (getfrag == ip_generic_getfrag && msg->msg_ubuf) { +if (skb_zcopy (skb) && msg->msg_ubuf !=skb_zcopy (skb)) +return -EINVAL ; + +/* Leave uarg NULL if can't zerocopy, callers should +* be able to handle it. +*/ +if ((rt->dst.dev->features &NETIF_F_SG ) && + csummode ==CHECKSUM_PARTIAL ) { + paged = true; + zc = true; + uarg = msg->msg_ubuf; + } + }else if (sock_flag (sk,SOCK_ZEROCOPY )) { + uarg =msg_zerocopy_realloc (sk, length,skb_zcopy (skb)); +if (!uarg) +return -ENOBUFS ; + extra_uref = !skb_zcopy (skb);/* only ref on new uarg */ +if (rt->dst.dev->features &NETIF_F_SG && + csummode ==CHECKSUM_PARTIAL ) { + paged = true; + zc = true; + }else { + uarg->zerocopy =0 ; +skb_zcopy_set (skb, uarg, &extra_uref); + } } }
@@ -1192,13 +1209,14 @@ static int __ip_append_data(struct sock *sk, err = -EFAULT ; goto error; } - }else if (!uarg || !uarg->zerocopy) { + }else if (!zc) {int i =skb_shinfo (skb)->nr_frags; err = -ENOMEM ;if (!sk_page_frag_refill (sk, pfrag)) goto error; +skb_zcopy_downgrade_managed (skb);if (!skb_can_coalesce (skb, i, pfrag->page, pfrag->offset)) { err = -EMSGSIZE ;
Signature · 1 line
2.37.0
PATCH v5 10/27 ipv6/udp: support externally provided ubufs
In reply to Pavel BegunkovTeach ipv6/udp how to use external ubuf_info provided in msghdr and also prepare it for managed frags by sprinkling skb_zcopy_downgrade_managed() when it could mix managed and not managed frags.
--- net/ipv6/ip6_output.c | 44 ++++++++++++++++++++++++++++++------------- 1 file changed, 31 insertions(+), 13 deletions(-)
net/ipv6/ip6_output.c +31 -13
@@ -1542,18 +1542,35 @@ static int __ip6_append_data(struct sock *sk, rt->dst.dev->features & (NETIF_F_IPV6_CSUM | NETIF_F_HW_CSUM)) csummode =CHECKSUM_PARTIAL ; -if (flags &MSG_ZEROCOPY && length &&sock_flag (sk,SOCK_ZEROCOPY )) { - uarg =msg_zerocopy_realloc (sk, length,skb_zcopy (skb)); -if (!uarg) -return -ENOBUFS ; - extra_uref = !skb_zcopy (skb);/* only ref on new uarg */ -if (rt->dst.dev->features &NETIF_F_SG && - csummode ==CHECKSUM_PARTIAL ) { - paged = true; - zc = true; - }else { - uarg->zerocopy =0 ; -skb_zcopy_set (skb, uarg, &extra_uref); +if ((flags &MSG_ZEROCOPY ) && length) { +struct msghdr *msg = from; + +if (getfrag == ip_generic_getfrag && msg->msg_ubuf) { +if (skb_zcopy (skb) && msg->msg_ubuf !=skb_zcopy (skb)) +return -EINVAL ; + +/* Leave uarg NULL if can't zerocopy, callers should +* be able to handle it. +*/ +if ((rt->dst.dev->features &NETIF_F_SG ) && + csummode ==CHECKSUM_PARTIAL ) { + paged = true; + zc = true; + uarg = msg->msg_ubuf; + } + }else if (sock_flag (sk,SOCK_ZEROCOPY )) { + uarg =msg_zerocopy_realloc (sk, length,skb_zcopy (skb)); +if (!uarg) +return -ENOBUFS ; + extra_uref = !skb_zcopy (skb);/* only ref on new uarg */ +if (rt->dst.dev->features &NETIF_F_SG && + csummode ==CHECKSUM_PARTIAL ) { + paged = true; + zc = true; + }else { + uarg->zerocopy =0 ; +skb_zcopy_set (skb, uarg, &extra_uref); + } } }
@@ -1747,13 +1764,14 @@ static int __ip6_append_data(struct sock *sk, err = -EFAULT ; goto error; } - }else if (!uarg || !uarg->zerocopy) { + }else if (!zc) {int i =skb_shinfo (skb)->nr_frags; err = -ENOMEM ;if (!sk_page_frag_refill (sk, pfrag)) goto error; +skb_zcopy_downgrade_managed (skb);if (!skb_can_coalesce (skb, i, pfrag->page, pfrag->offset)) { err = -EMSGSIZE ;
Signature · 1 line
2.37.0
PATCH v5 11/27 tcp: support externally provided ubufs
In reply to Pavel BegunkovTeach tcp how to use external ubuf_info provided in msghdr and also prepare it for managed frags by sprinkling skb_zcopy_downgrade_managed() when it could mix managed and not managed frags.
--- net/ipv4/tcp.c | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-)
net/ipv4/tcp.c +20 -12
@@ -1223,17 +1223,23 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) flags = msg->msg_flags; -if (flags &MSG_ZEROCOPY && size &&sock_flag (sk,SOCK_ZEROCOPY )) { +if ((flags &MSG_ZEROCOPY ) && size) { skb =tcp_write_queue_tail (sk); - uarg =msg_zerocopy_realloc (sk, size,skb_zcopy (skb)); -if (!uarg) { - err = -ENOBUFS ; - goto out_err; - } - zc = sk->sk_route_caps &NETIF_F_SG ; -if (!zc) - uarg->zerocopy =0 ; +if (msg->msg_ubuf) { + uarg = msg->msg_ubuf; +net_zcopy_get (uarg); + zc = sk->sk_route_caps &NETIF_F_SG ; + }else if (sock_flag (sk,SOCK_ZEROCOPY )) { + uarg =msg_zerocopy_realloc (sk, size,skb_zcopy (skb)); +if (!uarg) { + err = -ENOBUFS ; + goto out_err; + } + zc = sk->sk_route_caps &NETIF_F_SG ; +if (!zc) + uarg->zerocopy =0 ; + } } if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect) &&
@@ -1356,9 +1362,11 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) copy =min_t (int, copy, pfrag->size - pfrag->offset); -if (tcp_downgrade_zcopy_pure (sk, skb)) - goto wait_for_space; - +if (unlikely (skb_zcopy_pure (skb) ||skb_zcopy_managed (skb))) { +if (tcp_downgrade_zcopy_pure (sk, skb)) + goto wait_for_space; +skb_zcopy_downgrade_managed (skb); + } copy =tcp_wmem_schedule (sk, copy);if (!copy) goto wait_for_space;
Signature · 1 line
2.37.0
PATCH v5 12/27 io_uring: initialise msghdr::msg_ubuf
In reply to Pavel BegunkovInitialise newly added ->msg_ubuf in io_recv() and io_send().
--- io_uring/net.c | 2 ++ 1 file changed, 2 insertions(+)
io_uring/net.c +2 -0
@@ -255,6 +255,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) msg.msg_control =NULL ; msg.msg_controllen =0 ; msg.msg_namelen =0 ; + msg.msg_ubuf =NULL ; flags = sr->msg_flags;if (issue_flags &IO_URING_F_NONBLOCK )
@@ -601,6 +602,7 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) msg.msg_flags =0 ; msg.msg_controllen =0 ; msg.msg_iocb =NULL ; + msg.msg_ubuf =NULL ; flags = sr->msg_flags;if (force_nonblock)
Signature · 1 line
2.37.0
PATCH v5 13/27 io_uring: export io_put_task()
In reply to Pavel BegunkovMake io_put_task() available to non-core parts of io_uring, we'll need it for notification infrastructure.
--- include/linux/io_uring_types.h | 25 +++++++++++++++++++++++++ io_uring/io_uring.c | 11 +---------- io_uring/io_uring.h | 10 ++++++++++ io_uring/tctx.h | 26 -------------------------- 4 files changed, 36 insertions(+), 36 deletions(-)
include/linux/io_uring_types.h +25 -0
@@ -4,6 +4,7 @@#include <linux/blkdev.h> #include <linux/task_work.h> #include <linux/bitmap.h> +#include <linux/llist.h> #include <uapi/linux/io_uring.h> struct io_wq_work_node {
@@ -43,6 +44,30 @@ struct io_hash_table {unsigned hash_bits; }; +/* +* Arbitrary limit, can be raised if need be +*/ +#define IO_RINGFD_REG_MAX 16 + +struct io_uring_task { +/* submission side */ +int cached_refs; +const struct io_ring_ctx *last; +struct io_wq *io_wq; +struct file *registered_rings[IO_RINGFD_REG_MAX ]; + +struct xarray xa; +struct wait_queue_head wait; +atomic_t in_idle; +atomic_t inflight_tracked; +struct percpu_counter inflight; + +struct {/* task_work */ +struct llist_head task_list; +struct callback_head task_work; + }____cacheline_aligned_in_smp ; +}; +struct io_uring {u32 head____cacheline_aligned_in_smp ;u32 tail____cacheline_aligned_in_smp ;
io_uring/io_uring.c +1 -10
@@ -602,7 +602,7 @@ static bool io_cqring_overflow_flush(struct io_ring_ctx *ctx)return ret; } -static void __io_put_task (struct task_struct *task,int nr) +void __io_put_task (struct task_struct *task,int nr) {struct io_uring_task *tctx = task->io_uring;
@@ -612,15 +612,6 @@ static void __io_put_task(struct task_struct *task, int nr)put_task_struct_many (task, nr); } -/* must to be called somewhat shortly after putting a request */ -static inline void io_put_task (struct task_struct *task,int nr) -{ -if (likely (task == current)) - task->io_uring->cached_refs += nr; -else -__io_put_task (task, nr); -} -static void io_task_refs_refill (struct io_uring_task *tctx) {unsigned int refill = -tctx->cached_refs +IO_TCTX_REFS_CACHE_NR ;
io_uring/io_uring.h +10 -0
@@ -66,6 +66,7 @@ void io_wq_submit_work(struct io_wq_work *work);void io_free_req (struct io_kiocb *req);void io_queue_next (struct io_kiocb *req); +void __io_put_task (struct task_struct *task,int nr);bool io_match_task_safe (struct io_kiocb *head,struct task_struct *task,bool cancel_all);
@@ -253,4 +254,13 @@ static inline void io_commit_cqring_flush(struct io_ring_ctx *ctx) __io_commit_cqring_flush(ctx); } +/* must to be called somewhat shortly after putting a request */ +static inline void io_put_task (struct task_struct *task,int nr) +{ +if (likely (task == current)) + task->io_uring->cached_refs += nr; +else +__io_put_task (task, nr); +} +#endif
io_uring/tctx.h +0 -26
@@ -1,31 +1,5 @@// SPDX-License-Identifier: GPL-2.0 -#include <linux/llist.h> - -/* -* Arbitrary limit, can be raised if need be -*/ -#define IO_RINGFD_REG_MAX 16 - -struct io_uring_task { -/* submission side */ -int cached_refs; -const struct io_ring_ctx *last; -struct io_wq *io_wq; -struct file *registered_rings[IO_RINGFD_REG_MAX ]; - -struct xarray xa; -struct wait_queue_head wait; -atomic_t in_idle; -atomic_t inflight_tracked; -struct percpu_counter inflight; - -struct {/* task_work */ -struct llist_head task_list; -struct callback_head task_work; - }____cacheline_aligned_in_smp ; -}; -struct io_tctx_node {struct list_head ctx_node;struct task_struct *task;
Signature · 1 line
2.37.0
PATCH v5 14/27 io_uring: add zc notification infrastructure
In reply to Pavel BegunkovAdd internal part of send zerocopy notifications. There are two main structures, the first one is struct io_notif, which carries inside struct ubuf_info and maps 1:1 to it. io_uring will be binding a number of zerocopy send requests to it and ask to complete (aka flush) it. When flushed and all attached requests and skbs complete, it'll generate one and only one CQE. There are intended to be passed into the network layer as struct msghdr::msg_ubuf.
The second concept is notification slots. The userspace will be able to register an array of slots and subsequently addressing them by the index in the array. Slots are independent of each other. Each slot can have only one notifier at a time (called active notifier) but many notifiers during the lifetime. When active, a notifier not going to post any completion but the userspace can attach requests to it by specifying the corresponding slot while issueing send zc requests. Eventually, the userspace will want to "flush" the notifier losing any way to attach new requests to it, however it can use the next atomatically added notifier of this slot or of any other slot.
When the network layer is done with all enqueued skbs attached to a notifier and doesn't need the specified in them user data, the flushed notifier will post a CQE.
--- include/linux/io_uring_types.h | 5 ++ io_uring/Makefile | 2 +- io_uring/io_uring.c | 8 ++- io_uring/io_uring.h | 2 + io_uring/notif.c | 102 +++++++++++++++++++++++++++++++++ io_uring/notif.h | 64 +++++++++++++++++++++ 6 files changed, 179 insertions(+), 4 deletions(-) create mode 100644 io_uring/notif.c create mode 100644 io_uring/notif.h
include/linux/io_uring_types.h +5 -0
@@ -34,6 +34,9 @@ struct io_file_table {unsigned int alloc_hint; }; +struct io_notif ; +struct io_notif_slot ; +struct io_hash_bucket {spinlock_t lock;struct hlist_head list;
@@ -232,6 +235,8 @@ struct io_ring_ctx {unsigned nr_user_files;unsigned nr_user_bufs;struct io_mapped_ubuf **user_bufs; +struct io_notif_slot *notif_slots; +unsigned nr_notif_slots;struct io_submit_state submit_state;
io_uring/Makefile +1 -1
@@ -7,5 +7,5 @@ obj-$(CONFIG_IO_URING) += io_uring.o xattr.o nop.o fs.o splice.o \ openclose.o uring_cmd.o epoll.o \ statx.o net.o msg_ring.o timeout.o \ sqpoll.o fdinfo.o tctx.o poll.o \ - cancel.o kbuf.o rsrc.o rw.o opdef.o + cancel.o kbuf.o rsrc.o rw.o opdef.o notif.o obj-$(CONFIG_IO_WQ) += io-wq.o
io_uring/io_uring.c +5 -3
@@ -89,6 +89,7 @@#include "kbuf.h" #include "rsrc.h" #include "cancel.h" +#include "notif.h" #include "timeout.h" #include "poll.h"
@@ -726,9 +727,8 @@ struct io_uring_cqe *__io_get_cqe(struct io_ring_ctx *ctx)return &rings->cqes[off]; } -static bool io_fill_cqe_aux (struct io_ring_ctx *ctx, -u64 user_data,s32 res,u32 cflags, -bool allow_overflow) +bool io_fill_cqe_aux (struct io_ring_ctx *ctx,u64 user_data,s32 res,u32 cflags, +bool allow_overflow) {struct io_uring_cqe *cqe;
@@ -2496,6 +2496,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) }#endif WARN_ON_ONCE (!list_empty (&ctx->ltimeout_list)); +WARN_ON_ONCE (ctx->notif_slots || ctx->nr_notif_slots);io_mem_free (ctx->rings);io_mem_free (ctx->sq_sqes);
@@ -2672,6 +2673,7 @@ static __cold void io_ring_ctx_wait_and_kill(struct io_ring_ctx *ctx)io_unregister_personality (ctx, index);if (ctx->rings)io_poll_remove_all (ctx,NULL , true); +io_notif_unregister (ctx);mutex_unlock (&ctx->uring_lock);/* failed during ring init, it couldn't have issued any requests */
io_uring/io_uring.h +2 -0
@@ -33,6 +33,8 @@ void io_req_complete_post(struct io_kiocb *req);void __io_req_complete_post (struct io_kiocb *req);bool io_post_aux_cqe (struct io_ring_ctx *ctx,u64 user_data,s32 res,u32 cflags,bool allow_overflow); +bool io_fill_cqe_aux (struct io_ring_ctx *ctx,u64 user_data,s32 res,u32 cflags, +bool allow_overflow);void __io_commit_cqring_flush (struct io_ring_ctx *ctx);struct page **io_pin_pages (unsigned long ubuf,unsigned long len,int *npages);
io_uring/notif.c new file +102 -0
103 lines not on this page · raw message
io_uring/notif.h new file +64 -0
64 lines not on this page · raw message
Signature · 1 line
2.37.0
PATCH v5 15/27 io_uring: cache struct io_notif
In reply to Pavel Begunkovkmalloc'ing struct io_notif is too expensive when done frequently, cache them as many other resources in io_uring. Keep two list, the first one is from where we're getting notifiers, it's protected by ->uring_lock. The second is protected by ->completion_lock, to which we queue released notifiers. Then we splice one list into another when needed.
--- include/linux/io_uring_types.h | 7 +++++ io_uring/io_uring.c | 3 ++ io_uring/notif.c | 57 +++++++++++++++++++++++++++++----- io_uring/notif.h | 5 +++ 4 files changed, 65 insertions(+), 7 deletions(-)
include/linux/io_uring_types.h +7 -0
@@ -244,6 +244,9 @@ struct io_ring_ctx {struct xarray io_bl_xa;struct list_head io_buffers_cache; +/* struct io_notif cache, protected by uring_lock */ +struct list_head notif_list; +struct io_hash_table cancel_table_locked;struct list_head cq_overflow_list;struct list_head apoll_cache;
@@ -255,6 +258,10 @@ struct io_ring_ctx {struct io_wq_work_list locked_free_list;unsigned int locked_free_nr; +/* struct io_notif cache protected by completion_lock */ +struct list_head notif_list_locked; +unsigned int notif_locked_nr; +const struct cred *sq_creds;/* cred used for __io_sq_thread() */ struct io_sq_data *sq_data;/* if using sq thread polling */
io_uring/io_uring.c +3 -0
@@ -318,6 +318,8 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)INIT_WQ_LIST (&ctx->locked_free_list);INIT_DELAYED_WORK (&ctx->fallback_work, io_fallback_req_func);INIT_WQ_LIST (&ctx->submit_state.compl_reqs); +INIT_LIST_HEAD (&ctx->notif_list); +INIT_LIST_HEAD (&ctx->notif_list_locked);return ctx; err:kfree (ctx->dummy_ubuf);
@@ -2498,6 +2500,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx)WARN_ON_ONCE (!list_empty (&ctx->ltimeout_list));WARN_ON_ONCE (ctx->notif_slots || ctx->nr_notif_slots); +io_notif_cache_purge (ctx);io_mem_free (ctx->rings);io_mem_free (ctx->sq_sqes);
io_uring/notif.c +50 -7
81 lines not on this page · raw message
io_uring/notif.h +5 -0
@@ -5,6 +5,8 @@#include <net/sock.h> #include <linux/nospec.h> +#define IO_NOTIF_SPLICE_BATCH 32 +struct io_notif {struct ubuf_info uarg;struct io_ring_ctx *ctx;
@@ -13,6 +15,8 @@ struct io_notif {u64 tag;/* see struct io_notif_slot::seq */ u32 seq; +/* hook into ctx->notif_list and ctx->notif_list_locked */ +struct list_head cache_node;union {struct callback_head task_work;
@@ -41,6 +45,7 @@ struct io_notif_slot { };int io_notif_unregister (struct io_ring_ctx *ctx); +void io_notif_cache_purge (struct io_ring_ctx *ctx);struct io_notif *io_alloc_notif (struct io_ring_ctx *ctx,struct io_notif_slot *slot);
Signature · 1 line
2.37.0
PATCH v5 16/27 io_uring: complete notifiers in tw
In reply to Pavel BegunkovWe need a task context to post CQEs but using wq is too expensive. Try to complete notifiers using task_work and fall back to wq if fails.
--- io_uring/notif.c | 22 +++++++++++++++++++--- io_uring/notif.h | 3 +++ 2 files changed, 22 insertions(+), 3 deletions(-)
io_uring/notif.c +19 -3
@@ -13,6 +13,11 @@ static void __io_notif_complete_tw(struct callback_head *cb) struct io_notif *notif = container_of(cb, struct io_notif, task_work);struct io_ring_ctx *ctx = notif->ctx; +if (likely (notif->task)) { +io_put_task (notif->task,1 ); + notif->task =NULL ; + } +io_cq_lock (ctx);io_fill_cqe_aux (ctx, notif->tag,0 , notif->seq, true);
@@ -43,6 +48,14 @@ static void io_uring_tx_zerocopy_callback(struct sk_buff *skb,if (!refcount_dec_and_test (&uarg->refcnt))return ; + +if (likely (notif->task)) { +init_task_work (¬if->task_work, __io_notif_complete_tw); +if (likely (!task_work_add (notif->task, ¬if->task_work, +TWA_SIGNAL ))) +return ; + } +INIT_WORK (¬if->commit_work, io_notif_complete_wq);queue_work (system_unbound_wq, ¬if->commit_work); }
@@ -134,12 +147,15 @@ __cold int io_notif_unregister(struct io_ring_ctx *ctx)for (i =0 ; i < ctx->nr_notif_slots; i++) {struct io_notif_slot *slot = &ctx->notif_slots[i]; -if (slot->notif) -io_notif_slot_flush (slot); +if (!slot->notif) +continue ; +if (WARN_ON_ONCE (slot->notif->task)) + slot->notif->task =NULL ; +io_notif_slot_flush (slot); }kvfree (ctx->notif_slots); ctx->notif_slots =NULL ; ctx->nr_notif_slots =0 ;return 0 ; -} +}
io_uring/notif.h +3 -0
@@ -11,6 +11,9 @@ struct io_notif {struct ubuf_info uarg;struct io_ring_ctx *ctx; +/* complete via tw if ->task is non-NULL, fallback to wq otherwise */ +struct task_struct *task; +/* cqe->user_data, io_notif_slot::tag if not overridden */ u64 tag;/* see struct io_notif_slot::seq */
Signature · 1 line
2.37.0
PATCH v5 17/27 io_uring: add rsrc referencing for notifiers
In reply to Pavel BegunkovIn preparation to zerocopy sends with fixed buffers make notifiers to reference the rsrc node to protect the used fixed buffers. We can't just grab it for a send request as notifiers can likely outlive requests that used it.
--- io_uring/notif.c | 5 +++++ io_uring/notif.h | 1 + io_uring/rsrc.h | 12 +++++++++--- 3 files changed, 15 insertions(+), 3 deletions(-)
io_uring/notif.c +5 -0
@@ -7,10 +7,12 @@#include "io_uring.h" #include "notif.h" +#include "rsrc.h" static void __io_notif_complete_tw (struct callback_head *cb) { struct io_notif *notif = container_of(cb, struct io_notif, task_work); +struct io_rsrc_node *rsrc_node = notif->rsrc_node;struct io_ring_ctx *ctx = notif->ctx;if (likely (notif->task)) {
@@ -25,6 +27,7 @@ static void __io_notif_complete_tw(struct callback_head *cb) ctx->notif_locked_nr++;io_cq_unlock_post (ctx); +io_rsrc_put_node (rsrc_node,1 );percpu_ref_put (&ctx->refs); }
@@ -119,6 +122,8 @@ struct io_notif *io_alloc_notif(struct io_ring_ctx *ctx,/* master ref owned by io_notif_slot, will be dropped on flush */ refcount_set (¬if->uarg.refcnt,1 );percpu_ref_get (&ctx->refs); + notif->rsrc_node = ctx->rsrc_node; +io_charge_rsrc_node (ctx);return notif; }
io_uring/notif.h +1 -0
@@ -10,6 +10,7 @@struct io_notif {struct ubuf_info uarg;struct io_ring_ctx *ctx; +struct io_rsrc_node *rsrc_node;/* complete via tw if ->task is non-NULL, fallback to wq otherwise */ struct task_struct *task;
io_uring/rsrc.h +9 -3
@@ -135,6 +135,13 @@ static inline void io_req_put_rsrc_locked(struct io_kiocb *req, } } +static inline void io_charge_rsrc_node (struct io_ring_ctx *ctx) +{ + ctx->rsrc_cached_refs--; +if (unlikely (ctx->rsrc_cached_refs <0 )) +io_rsrc_refs_refill (ctx); +} +static inline void io_req_set_rsrc_node (struct io_kiocb *req,struct io_ring_ctx *ctx,unsigned int issue_flags)
@@ -144,9 +151,8 @@ static inline void io_req_set_rsrc_node(struct io_kiocb *req,if (!(issue_flags &IO_URING_F_UNLOCKED )) {lockdep_assert_held (&ctx->uring_lock); - ctx->rsrc_cached_refs--; -if (unlikely (ctx->rsrc_cached_refs <0 )) -io_rsrc_refs_refill (ctx); + +io_charge_rsrc_node (ctx); }else {percpu_ref_get (&req->rsrc_node->refs); }
Signature · 1 line
2.37.0
PATCH v5 18/27 io_uring: add notification slot registration
In reply to Pavel BegunkovLet the userspace to register and unregister notification slots.
--- include/uapi/linux/io_uring.h | 17 ++++++++++++++ io_uring/io_uring.c | 9 ++++++++ io_uring/notif.c | 43 +++++++++++++++++++++++++++++++++++ io_uring/notif.h | 3 +++ 4 files changed, 72 insertions(+)
include/uapi/linux/io_uring.h +17 -0
@@ -454,6 +454,10 @@ enum {/* register a range of fixed file slots for automatic slot allocation */ IORING_REGISTER_FILE_ALLOC_RANGE =25 , +/* zerocopy notification API */ +IORING_REGISTER_NOTIFIERS =26 , +IORING_UNREGISTER_NOTIFIERS =27 , +/* this goes last */ IORING_REGISTER_LAST };
@@ -500,6 +504,19 @@ struct io_uring_rsrc_update2 {__u32 resv2; }; +struct io_uring_notification_slot { +__u64 tag; +__u64 resv[3 ]; +}; + +struct io_uring_notification_register { +__u32 nr_slots; +__u32 resv; +__u64 resv2; +__u64 data; +__u64 resv3; +}; +/* Skip updating fd indexes set to this value in the fd table */ #define IORING_REGISTER_FILES_SKIP (-2)
io_uring/io_uring.c +9 -0
@@ -3875,6 +3875,15 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,break ; ret =io_register_file_alloc_range (ctx, arg);break ; +case IORING_REGISTER_NOTIFIERS : + ret =io_notif_register (ctx, arg, nr_args); +break ; +case IORING_UNREGISTER_NOTIFIERS : + ret = -EINVAL ; +if (arg || nr_args) +break ; + ret =io_notif_unregister (ctx); +break ;default : ret = -EINVAL ;break ;
io_uring/notif.c +43 -0
48 lines not on this page · raw message
io_uring/notif.h +3 -0
@@ -6,6 +6,7 @@#include <linux/nospec.h> #define IO_NOTIF_SPLICE_BATCH 32 +#define IORING_MAX_NOTIF_SLOTS (1U << 10)struct io_notif {struct ubuf_info uarg;
@@ -48,6 +49,8 @@ struct io_notif_slot {u32 seq; }; +int io_notif_register (struct io_ring_ctx *ctx, +void __user *arg,unsigned int size);int io_notif_unregister (struct io_ring_ctx *ctx);void io_notif_cache_purge (struct io_ring_ctx *ctx);
Signature · 1 line
2.37.0
PATCH v5 19/27 io_uring: wire send zc request type
In reply to Pavel BegunkovAdd a new io_uring opcode IORING_OP_SENDZC. The main distinction from IORING_OP_SEND is that the user should specify a notification slot index in sqe::notification_idx and the buffers are safe to reuse only when the used notification is flushed and completes.
--- include/uapi/linux/io_uring.h | 5 ++ io_uring/net.c | 94 +++++++++++++++++++++++++++++++++++ io_uring/net.h | 4 ++ io_uring/opdef.c | 15 ++++++ 4 files changed, 118 insertions(+)
include/uapi/linux/io_uring.h +5 -0
@@ -63,6 +63,10 @@ struct io_uring_sqe {union {__s32 splice_fd_in;__u32 file_index; +struct { +__u16 notification_idx; +__u16 __pad; + }; }; union { struct {
@@ -194,6 +198,7 @@ enum io_uring_op {IORING_OP_GETXATTR ,IORING_OP_SOCKET ,IORING_OP_URING_CMD , +IORING_OP_SENDZC_NOTIF ,/* this goes last, obviously */ IORING_OP_LAST,
io_uring/net.c +94 -0
112 lines not on this page · raw message
io_uring/net.h +4 -0
@@ -40,4 +40,8 @@ int io_socket(struct io_kiocb *req, unsigned int issue_flags);int io_connect_prep_async (struct io_kiocb *req);int io_connect_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe);int io_connect (struct io_kiocb *req,unsigned int issue_flags); + +int io_sendzc (struct io_kiocb *req,unsigned int issue_flags); +int io_sendzc_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe); +#endif
io_uring/opdef.c +15 -0
@@ -470,6 +470,21 @@ const struct io_op_def io_op_defs[] = { .issue = io_uring_cmd, .prep_async = io_uring_cmd_prep_async, }, + [IORING_OP_SENDZC_NOTIF] = { + .name = "SENDZC_NOTIF", + .needs_file = 1, + .unbound_nonreg_file = 1, + .pollout = 1, + .audit_skip = 1, + .ioprio = 1, +#if defined(CONFIG_NET ) + .prep = io_sendzc_prep, + .issue = io_sendzc, +#else + .prep = io_eopnotsupp_prep, +#endif + + }, };const char *io_uring_get_opcode (u8 opcode)
Signature · 1 line
2.37.0
PATCH v5 20/27 io_uring: account locked pages for non-fixed zc
In reply to Pavel BegunkovFixed buffers are RLIMIT_MEMLOCK accounted, however it doesn't cover iovec based zerocopy sends. Do the accounting on the io_uring side.
--- io_uring/net.c | 1 + io_uring/notif.c | 6 ++++++ 2 files changed, 7 insertions(+)
io_uring/net.c +1 -0
@@ -724,6 +724,7 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) ret =import_single_range (WRITE , zc->buf, zc->len, &iov, &msg.msg_iter);if (unlikely (ret))return ret; +mm_account_pinned_pages (¬if->uarg.mmp, zc->len); msg_flags = zc->msg_flags |MSG_ZEROCOPY ;if (issue_flags &IO_URING_F_NONBLOCK )
io_uring/notif.c +6 -0
@@ -14,7 +14,13 @@ static void __io_notif_complete_tw(struct callback_head *cb) struct io_notif *notif = container_of(cb, struct io_notif, task_work);struct io_rsrc_node *rsrc_node = notif->rsrc_node;struct io_ring_ctx *ctx = notif->ctx; +struct mmpin *mmp = ¬if->uarg.mmp; +if (mmp->user) { +atomic_long_sub (mmp->num_pg, &mmp->user->locked_vm); +free_uid (mmp->user); + mmp->user =NULL ; + }if (likely (notif->task)) {io_put_task (notif->task,1 ); notif->task =NULL ;
Signature · 1 line
2.37.0
PATCH v5 21/27 io_uring: allow to pass addr into sendzc
In reply to Pavel BegunkovAllow to specify an address to zerocopy sends making it more like sendto(2).
--- include/uapi/linux/io_uring.h | 2 +- io_uring/net.c | 18 ++++++++++++++++-- 2 files changed, 17 insertions(+), 3 deletions(-)
include/uapi/linux/io_uring.h +1 -1
@@ -65,7 +65,7 @@ struct io_uring_sqe {__u32 file_index;struct {__u16 notification_idx; -__u16 __pad; +__u16 addr_len; }; };union {
io_uring/net.c +16 -2
@@ -66,6 +66,8 @@ struct io_sendzc {u16 slot_idx;unsigned msg_flags;unsigned flags; +unsigned addr_len; +void __user *addr; };#define IO_APOLL_MULTI_POLLED (REQ_F_APOLL_MULTISHOT | REQ_F_POLLED)
@@ -666,8 +668,7 @@ int io_sendzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) {struct io_sendzc *zc =io_kiocb_to_cmd (req); -if (READ_ONCE (sqe->addr2) ||READ_ONCE (sqe->__pad2[0 ]) || -READ_ONCE (sqe->addr3)) +if (READ_ONCE (sqe->__pad2[0 ]) ||READ_ONCE (sqe->addr3))return -EINVAL ; zc->flags =READ_ONCE (sqe->ioprio);
@@ -680,6 +681,10 @@ int io_sendzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) zc->slot_idx =READ_ONCE (sqe->notification_idx);if (zc->msg_flags &MSG_DONTWAIT ) req->flags |=REQ_F_NOWAIT ; + + zc->addr =u64_to_user_ptr (READ_ONCE (sqe->addr2)); + zc->addr_len =READ_ONCE (sqe->addr_len); +#ifdef CONFIG_COMPAT if (req->ctx->compat) zc->msg_flags |=MSG_CMSG_COMPAT ;
@@ -689,6 +694,7 @@ int io_sendzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)int io_sendzc (struct io_kiocb *req,unsigned int issue_flags) { +struct sockaddr_storage address;struct io_ring_ctx *ctx = req->ctx;struct io_sendzc *zc =io_kiocb_to_cmd (req);struct io_notif_slot *notif_slot;
@@ -726,6 +732,14 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags)return ret;mm_account_pinned_pages (¬if->uarg.mmp, zc->len); +if (zc->addr) { + ret =move_addr_to_kernel (zc->addr, zc->addr_len, &address); +if (unlikely (ret <0 )) +return ret; + msg.msg_name = (struct sockaddr *)&address; + msg.msg_namelen = zc->addr_len; + } + msg_flags = zc->msg_flags |MSG_ZEROCOPY ;if (issue_flags &IO_URING_F_NONBLOCK ) msg_flags |=MSG_DONTWAIT ;
Signature · 1 line
2.37.0
PATCH v5 22/27 io_uring: sendzc with fixed buffers
In reply to Pavel BegunkovAllow zerocopy sends to use fixed buffers. There is an optimisation for this case, the network layer don't need to reference the pages, see SKBFL_MANAGED_FRAG_REFS, so io_uring have to ensure validity of fixed buffers until the notifier is released.
--- include/uapi/linux/io_uring.h | 6 +++++- io_uring/net.c | 29 ++++++++++++++++++++++++----- 2 files changed, 29 insertions(+), 6 deletions(-)
include/uapi/linux/io_uring.h +5 -1
@@ -269,9 +269,13 @@ enum io_uring_op {* IORING_RECV_MULTISHOT Multishot recv. Sets IORING_CQE_F_MORE if * the handler will continue to report * CQEs on behalf of the same SQE. +* +* IORING_RECVSEND_FIXED_BUF Use registered buffers, the index is stored in +* the buf_index field. */ #define IORING_RECVSEND_POLL_FIRST (1U << 0) -#define IORING_RECV_MULTISHOT (1U << 1) +#define IORING_RECV_MULTISHOT (1U << 1) +#define IORING_RECVSEND_FIXED_BUF (1U << 2)/* * accept flags stored in sqe->ioprio
io_uring/net.c +24 -5
@@ -14,6 +14,7 @@#include "kbuf.h" #include "net.h" #include "notif.h" +#include "rsrc.h" #if defined(CONFIG_NET ) struct io_shutdown {
@@ -667,13 +668,23 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags)int io_sendzc_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) {struct io_sendzc *zc =io_kiocb_to_cmd (req); +struct io_ring_ctx *ctx = req->ctx;if (READ_ONCE (sqe->__pad2[0 ]) ||READ_ONCE (sqe->addr3))return -EINVAL ; zc->flags =READ_ONCE (sqe->ioprio); -if (zc->flags & ~IORING_RECVSEND_POLL_FIRST ) +if (zc->flags & ~(IORING_RECVSEND_POLL_FIRST |IORING_RECVSEND_FIXED_BUF ))return -EINVAL ; +if (zc->flags &IORING_RECVSEND_FIXED_BUF ) { +unsigned idx =READ_ONCE (sqe->buf_index); + +if (unlikely (idx >= ctx->nr_user_bufs)) +return -EFAULT ; + idx =array_index_nospec (idx, ctx->nr_user_bufs); + req->imu =READ_ONCE (ctx->user_bufs[idx]); +io_req_set_rsrc_node (req, ctx,0 ); + } zc->buf =u64_to_user_ptr (READ_ONCE (sqe->addr)); zc->len =READ_ONCE (sqe->len);
@@ -727,10 +738,18 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) msg.msg_controllen =0 ; msg.msg_namelen =0 ; - ret =import_single_range (WRITE , zc->buf, zc->len, &iov, &msg.msg_iter); -if (unlikely (ret)) -return ret; -mm_account_pinned_pages (¬if->uarg.mmp, zc->len); +if (zc->flags &IORING_RECVSEND_FIXED_BUF ) { + ret =io_import_fixed (WRITE , &msg.msg_iter, req->imu, + (u64 )zc->buf, zc->len); +if (unlikely (ret)) +return ret; + }else { + ret =import_single_range (WRITE , zc->buf, zc->len, &iov, + &msg.msg_iter); +if (unlikely (ret)) +return ret; +mm_account_pinned_pages (¬if->uarg.mmp, zc->len); + }if (zc->addr) { ret =move_addr_to_kernel (zc->addr, zc->addr_len, &address);
Signature · 1 line
2.37.0
PATCH v5 23/27 io_uring: flush notifiers after sendzc
In reply to Pavel BegunkovAllow to flush notifiers as a part of sendzc request by setting IORING_SENDZC_FLUSH flag. When the sendzc request succeedes it will flush the used [active] notifier.
--- include/uapi/linux/io_uring.h | 4 ++++ io_uring/io_uring.c | 11 +---------- io_uring/io_uring.h | 10 ++++++++++ io_uring/net.c | 5 ++++- io_uring/notif.c | 2 +- io_uring/notif.h | 11 +++++++++++ 6 files changed, 31 insertions(+), 12 deletions(-)
include/uapi/linux/io_uring.h +4 -0
@@ -272,10 +272,14 @@ enum io_uring_op {* * IORING_RECVSEND_FIXED_BUF Use registered buffers, the index is stored in * the buf_index field. +* +* IORING_RECVSEND_NOTIF_FLUSH Flush a notification after a successful +* successful. Only for zerocopy sends. */ #define IORING_RECVSEND_POLL_FIRST (1U << 0)#define IORING_RECV_MULTISHOT (1U << 1)#define IORING_RECVSEND_FIXED_BUF (1U << 2) +#define IORING_RECVSEND_NOTIF_FLUSH (1U << 3)/* * accept flags stored in sqe->ioprio
io_uring/io_uring.c +1 -10
@@ -615,7 +615,7 @@ void __io_put_task(struct task_struct *task, int nr)put_task_struct_many (task, nr); } -static void io_task_refs_refill (struct io_uring_task *tctx) +void io_task_refs_refill (struct io_uring_task *tctx) {unsigned int refill = -tctx->cached_refs +IO_TCTX_REFS_CACHE_NR ;
@@ -624,15 +624,6 @@ static void io_task_refs_refill(struct io_uring_task *tctx) tctx->cached_refs += refill; } -static inline void io_get_task_refs (int nr) -{ -struct io_uring_task *tctx = current->io_uring; - - tctx->cached_refs -= nr; -if (unlikely (tctx->cached_refs <0 )) -io_task_refs_refill (tctx); -} -static __cold void io_uring_drop_tctx_refs (struct task_struct *task) {struct io_uring_task *tctx = task->io_uring;
io_uring/io_uring.h +10 -0
@@ -69,6 +69,7 @@ void io_wq_submit_work(struct io_wq_work *work);void io_free_req (struct io_kiocb *req);void io_queue_next (struct io_kiocb *req);void __io_put_task (struct task_struct *task,int nr); +void io_task_refs_refill (struct io_uring_task *tctx);bool io_match_task_safe (struct io_kiocb *head,struct task_struct *task,bool cancel_all);
@@ -265,4 +266,13 @@ static inline void io_put_task(struct task_struct *task, int nr) __io_put_task(task, nr); } +static inline void io_get_task_refs (int nr) +{ +struct io_uring_task *tctx = current->io_uring; + + tctx->cached_refs -= nr; +if (unlikely (tctx->cached_refs <0 )) +io_task_refs_refill (tctx); +} +#endif
io_uring/net.c +4 -1
@@ -674,7 +674,8 @@ int io_sendzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)return -EINVAL ; zc->flags =READ_ONCE (sqe->ioprio); -if (zc->flags & ~(IORING_RECVSEND_POLL_FIRST |IORING_RECVSEND_FIXED_BUF )) +if (zc->flags & ~(IORING_RECVSEND_POLL_FIRST | +IORING_RECVSEND_FIXED_BUF |IORING_RECVSEND_NOTIF_FLUSH ))return -EINVAL ;if (zc->flags &IORING_RECVSEND_FIXED_BUF ) {unsigned idx =READ_ONCE (sqe->buf_index);
@@ -776,6 +777,8 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags)return ret == -ERESTARTSYS ? -EINTR : ret; } +if (zc->flags &IORING_RECVSEND_NOTIF_FLUSH ) +io_notif_slot_flush_submit (notif_slot,0 );io_req_set_res (req, ret,0 );return IOU_OK ; }
io_uring/notif.c +1 -1
@@ -133,7 +133,7 @@ struct io_notif *io_alloc_notif(struct io_ring_ctx *ctx,return notif; } -static void io_notif_slot_flush (struct io_notif_slot *slot) +void io_notif_slot_flush (struct io_notif_slot *slot) __must_hold(&ctx->uring_lock) {struct io_notif *notif = slot->notif;
io_uring/notif.h +11 -0
@@ -54,6 +54,7 @@ int io_notif_register(struct io_ring_ctx *ctx,int io_notif_unregister (struct io_ring_ctx *ctx);void io_notif_cache_purge (struct io_ring_ctx *ctx); +void io_notif_slot_flush (struct io_notif_slot *slot);struct io_notif *io_alloc_notif (struct io_ring_ctx *ctx,struct io_notif_slot *slot);
@@ -74,3 +75,13 @@ static inline struct io_notif_slot *io_get_notif_slot(struct io_ring_ctx *ctx, idx =array_index_nospec (idx, ctx->nr_notif_slots);return &ctx->notif_slots[idx]; } + +static inline void io_notif_slot_flush_submit (struct io_notif_slot *slot, +unsigned int issue_flags) +{ +if (!(issue_flags &IO_URING_F_UNLOCKED )) { + slot->notif->task = current; +io_get_task_refs (1 ); + } +io_notif_slot_flush (slot); +}
Signature · 1 line
2.37.0
PATCH v5 24/27 io_uring: rename IORING_OP_FILES_UPDATE
In reply to Pavel BegunkovIORING_OP_FILES_UPDATE will be a more generic opcode serving different resource types, rename it into IORING_OP_RSRC_UPDATE and add subtype handling.
--- include/uapi/linux/io_uring.h | 12 +++++++++++- io_uring/opdef.c | 9 +++++---- io_uring/rsrc.c | 17 +++++++++++++++-- io_uring/rsrc.h | 4 ++-- 4 files changed, 33 insertions(+), 9 deletions(-)
include/uapi/linux/io_uring.h +11 -1
@@ -171,7 +171,8 @@ enum io_uring_op {IORING_OP_FALLOCATE ,IORING_OP_OPENAT ,IORING_OP_CLOSE , -IORING_OP_FILES_UPDATE , +IORING_OP_RSRC_UPDATE , +IORING_OP_FILES_UPDATE =IORING_OP_RSRC_UPDATE ,IORING_OP_STATX ,IORING_OP_READ , IORING_OP_WRITE,
@@ -220,6 +221,7 @@ enum io_uring_op {#define IORING_TIMEOUT_ETIME_SUCCESS (1U << 5)#define IORING_TIMEOUT_CLOCK_MASK (IORING_TIMEOUT_BOOTTIME | IORING_TIMEOUT_REALTIME)#define IORING_TIMEOUT_UPDATE_MASK (IORING_TIMEOUT_UPDATE | IORING_LINK_TIMEOUT_UPDATE) +/* * sqe->splice_flags * extends splice(2) flags
@@ -286,6 +288,14 @@ enum io_uring_op {*/ #define IORING_ACCEPT_MULTISHOT (1U << 0) + +/* +* IORING_OP_RSRC_UPDATE flags +*/ +enum { +IORING_RSRC_UPDATE_FILES , +}; +/* * IORING_OP_MSG_RING command types, stored in sqe->addr */
io_uring/opdef.c +5 -4
@@ -246,12 +246,13 @@ const struct io_op_def io_op_defs[] = { .prep = io_close_prep, .issue = io_close, }, - [IORING_OP_FILES_UPDATE] = { + [IORING_OP_RSRC_UPDATE] = { .audit_skip = 1, .iopoll = 1, - .name = "FILES_UPDATE", - .prep = io_files_update_prep, - .issue = io_files_update, + .name = "RSRC_UPDATE", + .prep = io_rsrc_update_prep, + .issue = io_rsrc_update, + .ioprio = 1, }, [IORING_OP_STATX] = { .audit_skip = 1,
io_uring/rsrc.c +15 -2
@@ -21,6 +21,7 @@ struct io_rsrc_update {u64 arg;u32 nr_args;u32 offset; +int type; }; static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov,
@@ -658,7 +659,7 @@ __cold int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg,return -EINVAL ; } -int io_files_update_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) +int io_rsrc_update_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) {struct io_rsrc_update *up =io_kiocb_to_cmd (req);
@@ -672,6 +673,7 @@ int io_files_update_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)if (!up->nr_args)return -EINVAL ; up->arg =READ_ONCE (sqe->addr); + up->type =READ_ONCE (sqe->ioprio);return 0 ; }
@@ -711,7 +713,7 @@ static int io_files_update_with_index_alloc(struct io_kiocb *req,return ret; } -int io_files_update (struct io_kiocb *req,unsigned int issue_flags) +static int io_files_update (struct io_kiocb *req,unsigned int issue_flags) {struct io_rsrc_update *up =io_kiocb_to_cmd (req);struct io_ring_ctx *ctx = req->ctx;
@@ -740,6 +742,17 @@ int io_files_update(struct io_kiocb *req, unsigned int issue_flags)return IOU_OK ; } +int io_rsrc_update (struct io_kiocb *req,unsigned int issue_flags) +{ +struct io_rsrc_update *up =io_kiocb_to_cmd (req); + +switch (up->type) { +case IORING_RSRC_UPDATE_FILES : +return io_files_update (req, issue_flags); + } +return -EINVAL ; +} +int io_queue_rsrc_removal (struct io_rsrc_data *data,unsigned idx,struct io_rsrc_node *node,void *rsrc) {
io_uring/rsrc.h +2 -2
@@ -167,6 +167,6 @@ static inline u64 *io_get_tag_slot(struct io_rsrc_data *data, unsigned int idx) return &data->tags[table_idx][off]; } -int io_files_update (struct io_kiocb *req,unsigned int issue_flags); -int io_files_update_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe); +int io_rsrc_update (struct io_kiocb *req,unsigned int issue_flags); +int io_rsrc_update_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe);#endif
Signature · 1 line
2.37.0
PATCH v5 25/27 io_uring: add zc notification flush requests
In reply to Pavel BegunkovOverlay notification control onto IORING_OP_RSRC_UPDATE (former IORING_OP_FILES_UPDATE). It allows to flush a range of zc notifications from slots with indexes [sqe->off, sqe->off+sqe->len). If sqe->arg is not zero, it also copies sqe->arg as a new tag for all flushed notifications.
Note, it doesn't flush a notification of a slot if there was no requests attached to it (since last flush or registration).
--- include/uapi/linux/io_uring.h | 1 + io_uring/rsrc.c | 38 +++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)
include/uapi/linux/io_uring.h +1 -0
@@ -294,6 +294,7 @@ enum io_uring_op {*/ enum {IORING_RSRC_UPDATE_FILES , +IORING_RSRC_UPDATE_NOTIF , };/*
io_uring/rsrc.c +38 -0
@@ -15,6 +15,7 @@#include "io_uring.h" #include "openclose.h" #include "rsrc.h" +#include "notif.h" struct io_rsrc_update {struct file *file;
@@ -742,6 +743,41 @@ static int io_files_update(struct io_kiocb *req, unsigned int issue_flags)return IOU_OK ; } +static int io_notif_update (struct io_kiocb *req,unsigned int issue_flags) +{ +struct io_rsrc_update *up =io_kiocb_to_cmd (req); +struct io_ring_ctx *ctx = req->ctx; +unsigned len = up->nr_args; +unsigned idx_end, idx = up->offset; +int ret =0 ; + +io_ring_submit_lock (ctx, issue_flags); +if (unlikely (check_add_overflow (idx, len, &idx_end))) { + ret = -EOVERFLOW ; + goto out; + } +if (unlikely (idx_end > ctx->nr_notif_slots)) { + ret = -EINVAL ; + goto out; + } + +for (; idx < idx_end; idx++) { +struct io_notif_slot *slot = &ctx->notif_slots[idx]; + +if (!slot->notif) +continue ; +if (up->arg) + slot->tag = up->arg; +io_notif_slot_flush_submit (slot, issue_flags); + } +out: +io_ring_submit_unlock (ctx, issue_flags); +if (ret <0 ) +req_set_fail (req); +io_req_set_res (req, ret,0 ); +return IOU_OK ; +} +int io_rsrc_update (struct io_kiocb *req,unsigned int issue_flags) {struct io_rsrc_update *up =io_kiocb_to_cmd (req);
@@ -749,6 +785,8 @@ int io_rsrc_update(struct io_kiocb *req, unsigned int issue_flags)switch (up->type) {case IORING_RSRC_UPDATE_FILES :return io_files_update (req, issue_flags); +case IORING_RSRC_UPDATE_NOTIF : +return io_notif_update (req, issue_flags); }return -EINVAL ; }
Signature · 1 line
2.37.0
PATCH v5 26/27 io_uring: enable managed frags with register buffers
In reply to Pavel Begunkovio_uring's registered buffers infra has a good performant way of pinning pages, so let's use SKBFL_MANAGED_FRAG_REFS when our requests are purely register buffer backed.
--- io_uring/net.c | 56 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 55 insertions(+), 1 deletion(-)
io_uring/net.c +55 -1
@@ -704,6 +704,60 @@ int io_sendzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)return 0 ; } +static int io_sg_from_iter (struct sock *sk,struct sk_buff *skb, +struct iov_iter *from,size_t length) +{ +struct skb_shared_info *shinfo =skb_shinfo (skb); +int frag = shinfo->nr_frags; +int ret =0 ; +struct bvec_iter bi; +ssize_t copied =0 ; +unsigned long truesize =0 ; + +if (!shinfo->nr_frags) + shinfo->flags |=SKBFL_MANAGED_FRAG_REFS ; + +if (!skb_zcopy_managed (skb) || !iov_iter_is_bvec (from)) { +skb_zcopy_downgrade_managed (skb); +return __zerocopy_sg_from_iter (NULL , sk, skb, from, length); + } + + bi.bi_size =min (from->count, length); + bi.bi_bvec_done = from->iov_offset; + bi.bi_idx =0 ; + +while (bi.bi_size && frag <MAX_SKB_FRAGS ) { +struct bio_vec v =mp_bvec_iter_bvec (from->bvec, bi); + + copied += v.bv_len; + truesize +=PAGE_ALIGN (v.bv_len + v.bv_offset); +__skb_fill_page_desc_noacc (shinfo, frag++, v.bv_page, + v.bv_offset, v.bv_len); +bvec_iter_advance_single (from->bvec, &bi, v.bv_len); + } +if (bi.bi_size) + ret = -EMSGSIZE ; + + shinfo->nr_frags = frag; + from->bvec += bi.bi_idx; + from->nr_segs -= bi.bi_idx; + from->count = bi.bi_size; + from->iov_offset = bi.bi_bvec_done; + + skb->data_len += copied; + skb->len += copied; + skb->truesize += truesize; + +if (sk && sk->sk_type ==SOCK_STREAM ) { +sk_wmem_queued_add (sk, truesize); +if (!skb_zcopy_pure (skb)) +sk_mem_charge (sk, truesize); + }else { +refcount_add (truesize, &skb->sk->sk_wmem_alloc); + } +return ret; +} +int io_sendzc (struct io_kiocb *req,unsigned int issue_flags) {struct sockaddr_storage address;
@@ -768,7 +822,7 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) msg.msg_flags = msg_flags; msg.msg_ubuf = ¬if->uarg; - msg.sg_from_iter =NULL ; + msg.sg_from_iter = io_sg_from_iter; ret =sock_sendmsg (sock, &msg);if (unlikely (ret < min_ret)) {
Signature · 1 line
2.37.0
PATCH v5 27/27 selftests/io_uring: test zerocopy send
In reply to Pavel BegunkovAdd selftests for io_uring zerocopy sends and io_uring's notification infrastructure. It's largely influenced by msg_zerocopy and uses it on the receive side.
--- tools/testing/selftests/net/Makefile | 1 + .../selftests/net/io_uring_zerocopy_tx.c | 605 ++++++++++++++++++ .../selftests/net/io_uring_zerocopy_tx.sh | 131 ++++ 3 files changed, 737 insertions(+) create mode 100644 tools/testing/selftests/net/io_uring_zerocopy_tx.c create mode 100755 tools/testing/selftests/net/io_uring_zerocopy_tx.sh
tools/testing/selftests/net/Makefile +1 -0
@@ -59,6 +59,7 @@ TEST_GEN_FILES += toeplitz TEST_GEN_FILES += cmsg_sender TEST_GEN_FILES += stress_reuseport_listen TEST_PROGS += test_vxlan_vnifiltering.sh +TEST_GEN_FILES += io_uring_zerocopy_tx TEST_FILES := settings
tools/testing/selftests/net/io_uring_zerocopy_tx.c new file +605 -0
605 lines not on this page · raw message
tools/testing/selftests/net/io_uring_zerocopy_tx.sh new file +131 -0
131 lines not on this page · raw message
Signature · 1 line
2.37.0
Re: [PATCH net-next v5 27/27] selftests/io_uring: test zerocopy send
Quoted 73 lines · Pavel Begunkov
Add selftests for
io_uringzerocopy sends and io_uring's notification infrastructure. It's largely influenced bymsg_zerocopyand uses it on the receive side.Signed-off-by Pavel Begunkov <[email protected]>---
tools/testing/selftests/net/Makefile | 1 + .../selftests/net/io_uring_zerocopy_tx.c | 605 ++++++++++++++++++ .../selftests/net/io_uring_zerocopy_tx.sh | 131 ++++ 3 files changed, 737 insertions(+)create mode 100644
tools/testing/selftests/net/io_uring_zerocopy_tx.ccreate mode 100755tools/testing/selftests/net/io_uring_zerocopy_tx.shtools/testing/selftests/net/Makefile +0
-0@@ -59,6 +59,7 @@ TEST_GEN_FILES += toeplitz
TEST_GEN_FILES+=cmsg_senderTEST_GEN_FILES+=stress_reuseport_listenTEST_PROGS+=test_vxlan_vnifiltering.sh
+TEST_GEN_FILES +=io_uring_zerocopy_tx
TEST_FILES:= settingstools/testing/selftests/net/io_uring_zerocopy_tx.c new file +42
-0@@ -0,0 +1,605 @@ +/* SPDX-License-Identifier: MIT */ +/* based on linux-kernel/tools/testing/selftests/net/msg_zerocopy.c */ +#include <assert.h> +#include <errno.h> +#include <error.h> +#include <fcntl.h> +#include <limits.h> +#include <stdbool.h> +#include <stdint.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <unistd.h> + +#include <arpa/inet.h> +#include <linux/errqueue.h> +#include <linux/if_packet.h> +#include <linux/io_uring.h> +#include <linux/ipv6.h> +#include <linux/socket.h> +#include <linux/sockios.h> +#include <net/ethernet.h> +#include <net/if.h> +#include <netinet/in.h> +#include <netinet/ip.h> +#include <netinet/ip6.h> +#include <netinet/tcp.h> +#include <netinet/udp.h> +#include <sys/ioctl.h> +#include <sys/mman.h> +#include <sys/resource.h> +#include <sys/socket.h> +#include <sys/stat.h> +#include <sys/time.h> +#include <sys/types.h> +#include <sys/un.h> +#include <sys/wait.h> + +#define NOTIF_TAG 0xfffffffULL +#define NONZC_TAG 0 +#define ZC_TAG 1 +
<...>
tools/testing/selftests/net/io_uring_zerocopy_tx.c line 504
Quoted 13 lines
tools/testing/selftests/net/io_uring_zerocopy_tx.c new file +13
-0+static void do_test (int domain,int type,int protocol) +{ +int i; + +for (i =0 ; i <IP_MAXPACKET ; i++) + payload[i] ='a' + (i %26 ); +do_tx (domain, type, protocol); +} + +static void usage (const char *filepath) +{ +error (1 ,0 ,"Usage: %s [-f] [-n<N>] [-z0] [-s<payload size>] " +"(-4|-6) [-t<time s>] -D<dst_ip> udp" , filepath);
A small flaw, the usage here doesn't match the real options in parse_opts().
Thanks
Quoted 240 lines
tools/testing/selftests/net/io_uring_zerocopy_tx.c new file +101
-0+} + +static void parse_opts (int argc,char **argv) +{ +const int max_payload_len =sizeof (payload) - +sizeof (struct ipv6hdr ) - +sizeof (struct tcphdr ) - +40 /* max tcp options */ ; +struct sockaddr_in6 *addr6 = (void *) &cfg_dst_addr; +struct sockaddr_in *addr4 = (void *) &cfg_dst_addr; +char *daddr =NULL ; +int c; + +if (argc <=1 ) +usage (argv[0 ]); + cfg_payload_len = max_payload_len; + +while ((c =getopt (argc, argv,"46D:p:s:t:n:fc:m:" )) !=-1 ) { +switch (c) { +case '4' : +if (cfg_family !=PF_UNSPEC ) +error (1 ,0 ,"Pass one of -4 or -6" ); + cfg_family =PF_INET ; + cfg_alen =sizeof (struct sockaddr_in ); +break ; +case '6' : +if (cfg_family !=PF_UNSPEC ) +error (1 ,0 ,"Pass one of -4 or -6" ); + cfg_family =PF_INET6 ; + cfg_alen =sizeof (struct sockaddr_in6 ); +break ; +case 'D' : + daddr = optarg; +break ; +case 'p' : + cfg_port =strtoul (optarg,NULL ,0 ); +break ; +case 's' : + cfg_payload_len =strtoul (optarg,NULL ,0 ); +break ; +case 't' : + cfg_runtime_ms =200 +strtoul (optarg,NULL ,10 ) *1000 ; +break ; +case 'n' : + cfg_nr_reqs =strtoul (optarg,NULL ,0 ); +break ; +case 'f' : + cfg_flush =1 ; +break ; +case 'c' : + cfg_cork =strtol (optarg,NULL ,0 ); +break ; +case 'm' : + cfg_mode =strtol (optarg,NULL ,0 ); +break ; + } + } + +switch (cfg_family) { +case PF_INET : +memset (addr4,0 ,sizeof (*addr4)); + addr4->sin_family =AF_INET ; + addr4->sin_port =htons (cfg_port); +if (daddr && +inet_pton (AF_INET , daddr, &(addr4->sin_addr)) !=1 ) +error (1 ,0 ,"ipv4 parse error: %s" , daddr); +break ; +case PF_INET6 : +memset (addr6,0 ,sizeof (*addr6)); + addr6->sin6_family =AF_INET6 ; + addr6->sin6_port =htons (cfg_port); +if (daddr && +inet_pton (AF_INET6 , daddr, &(addr6->sin6_addr)) !=1 ) +error (1 ,0 ,"ipv6 parse error: %s" , daddr); +break ; +default : +error (1 ,0 ,"illegal domain" ); + } + +if (cfg_payload_len > max_payload_len) +error (1 ,0 ,"-s: payload exceeds max (%d)" , max_payload_len); +if (cfg_mode ==MODE_NONZC && cfg_flush) +error (1 ,0 ,"-f: only zerocopy modes support notifications" ); +if (optind != argc -1 ) +usage (argv[0 ]); +} + +int main (int argc,char **argv) +{ +const char *cfg_test = argv[argc -1 ]; + +parse_opts (argc, argv); + +if (!strcmp (cfg_test,"tcp" )) +do_test (cfg_family,SOCK_STREAM ,0 ); +else if (!strcmp (cfg_test,"udp" )) +do_test (cfg_family,SOCK_DGRAM ,0 ); +else +error (1 ,0 ,"unknown cfg_test %s" , cfg_test); +return 0 ; +}tools/testing/selftests/net/io_uring_zerocopy_tx.sh new file +131
-0@@ -0,0 +1,131 @@ +#!/bin/bash +# +# Send data between two processes across namespaces +# Run twice: once without and once with zerocopy + +set -e + +readonly DEV="veth0" +readonly DEV_MTU=65535 +readonly BIN_TX="./io_uring_zerocopy_tx" +readonly BIN_RX="./msg_zerocopy" + +readonly RAND="$(mktemp -u XXXXXX)" +readonly NSPREFIX="ns-${RAND}" +readonly NS1="${NSPREFIX}1" +readonly NS2="${NSPREFIX}2" + +readonly SADDR4='192.168.1.1' +readonly DADDR4='192.168.1.2' +readonly SADDR6='fd::1' +readonly DADDR6='fd::2' + +readonly path_sysctl_mem="net.core.optmem_max" + +# No arguments: automated test +if [[ "$#" -eq "0" ]]; then + IPs=( "4" "6" ) + protocols=( "tcp" "udp" ) + + for IP in "${IPs[@]}"; do + for proto in "${protocols[@]}"; do + for mode in $(seq 1 3); do + $0 "$IP" "$proto" -m "$mode" -t 1 -n 32 + $0 "$IP" "$proto" -m "$mode" -t 1 -n 32 -f + $0 "$IP" "$proto" -m "$mode" -t 1 -n 32 -c -f + done + done + done + + echo "OK. All tests passed" + exit 0 +fi + +# Argument parsing +if [[ "$#" -lt "2" ]]; then + echo "Usage: $0 [4|6] [tcp|udp|raw|raw_hdrincl|packet|packet_dgram] <args>" + exit 1 +fi + +readonly IP="$1" +shift +readonly TXMODE="$1" +shift +readonly EXTRA_ARGS="$@" + +# Argument parsing: configure addresses +if [[ "${IP}" == "4" ]]; then + readonly SADDR="${SADDR4}" + readonly DADDR="${DADDR4}" +elif [[ "${IP}" == "6" ]]; then + readonly SADDR="${SADDR6}" + readonly DADDR="${DADDR6}" +else + echo "Invalid IP version ${IP}" + exit 1 +fi + +# Argument parsing: select receive mode +# +# This differs from send mode for +# - packet: use raw recv, because packet receives skb clones +# - raw_hdrinc: use raw recv, because hdrincl is a tx-only option +case "${TXMODE}" in +'packet' | 'packet_dgram' | 'raw_hdrincl') + RXMODE='raw' + ;; +*) + RXMODE="${TXMODE}" + ;; +esac + +# Start of state changes: install cleanup handler +save_sysctl_mem="$(sysctl -n ${path_sysctl_mem})" + +cleanup() { + ip netns del "${NS2}" + ip netns del "${NS1}" + sysctl -w -q "${path_sysctl_mem}=${save_sysctl_mem}" +} + +trap cleanup EXIT + +# Configure system settings +sysctl -w -q "${path_sysctl_mem}=1000000" + +# Create virtual ethernet pair between network namespaces +ip netns add "${NS1}" +ip netns add "${NS2}" + +ip link add "${DEV}" mtu "${DEV_MTU}" netns "${NS1}" type veth \ + peer name "${DEV}" mtu "${DEV_MTU}" netns "${NS2}" + +# Bring the devices up +ip -netns "${NS1}" link set "${DEV}" up +ip -netns "${NS2}" link set "${DEV}" up + +# Set fixed MAC addresses on the devices +ip -netns "${NS1}" link set dev "${DEV}" address 02:02:02:02:02:02 +ip -netns "${NS2}" link set dev "${DEV}" address 06:06:06:06:06:06 + +# Add fixed IP addresses to the devices +ip -netns "${NS1}" addr add 192.168.1.1/24 dev "${DEV}" +ip -netns "${NS2}" addr add 192.168.1.2/24 dev "${DEV}" +ip -netns "${NS1}" addr add fd::1/64 dev "${DEV}" nodad +ip -netns "${NS2}" addr add fd::2/64 dev "${DEV}" nodad + +# Optionally disable sg or csum offload to test edge cases +# ip netns exec "${NS1}" ethtool -K "${DEV}" sg off + +do_test() { + local readonly ARGS="$1" + + echo "ipv${IP} ${TXMODE} ${ARGS}" + ip netns exec "${NS2}" "${BIN_RX}" "-${IP}" -t 2 -C 2 -S "${SADDR}" -D "${DADDR}" -r "${RXMODE}" & + sleep 0.2 + ip netns exec "${NS1}" "${BIN_TX}" "-${IP}" -t 1 -D "${DADDR}" ${ARGS} "${TXMODE}" + wait +} + +do_test "${EXTRA_ARGS}" +echo okSignature · 1 line
2.37.0
Re: [PATCH net-next v5 27/27] selftests/io_uring: test zerocopy send
Quoted 7 lines · dust.li
Quoted 7 lines
tools/testing/selftests/net/io_uring_zerocopy_tx.c new file +7
-0+static void do_test (int domain,int type,int protocol) +{ +int i; + +for (i =0 ; i <IP_MAXPACKET ; i++) + payload[i] ='a' + (i %26 ); +do_tx (domain, type, protocol);
Quoted 8 lines
tools/testing/selftests/net/io_uring_zerocopy_tx.c line 504
Quoted 6 lines
tools/testing/selftests/net/io_uring_zerocopy_tx.c new file +6
-0+} + +static void usage (const char *filepath) +{ +error (1 ,0 ,"Usage: %s [-f] [-n<N>] [-z0] [-s<payload size>] " +"(-4|-6) [-t<time s>] -D<dst_ip> udp" , filepath);A small flaw, the usage here doesn't match the real options in
parse_opts().
Indeed. I'll adjust it, thanks!
Signature · 1 line
Pavel Begunkov
Re: (subset) [PATCH net-next v5 00/27] io_uring zerocopy send
In reply to Pavel BegunkovQuoted 13 lines · Pavel Begunkov
NOTE: Not to be picked directly. After getting necessary acks, I'll be
working out merging with Jakub and Jens.The patchset implements
io_uringzerocopy send. It works with both registered and normal buffers, mixing is allowed but not recommended. Apart from usual request completions, just as withMSG_ZEROCOPY,io_uringseparately notifies the userspace when buffers are freed and can be reused (see API design below), which is delivered into io_uring's Completion Queue. Those "buffer-free" notifications are not necessarily per request, but the userspace has control over it and should explicitly attaching a number of requests to a single notification. The series also adds some internal optimisations when used with registered buffers like removing page referencing.
Quoted 1 line
[...]
Applied, thanks!
[12/27] io_uring: initialise msghdr::msg_ubuf
commit: 06f241e2bf4ba2a3e77269be25d21c0196a57a4f
[13/27] io_uring: export io_put_task()
commit: ba64c07a6ef9a05ca9eb09e13b70df7500e78cf8
[14/27] io_uring: add zc notification infrastructure
commit: 6f322c753daee4b9d4ad494d4e8b05da610d804c
[15/27] io_uring: cache struct io_notif
commit: cf49e2d47c49e547d4bc370efe73785fc82354e5
[16/27] io_uring: complete notifiers in tw
commit: 9cc16ae447db07d210175d2ad2419784dd20f784
[17/27] io_uring: add rsrc referencing for notifiers
commit: e133e289093ea35c1f7f940fe4c0ceb62037dc59
[18/27] io_uring: add notification slot registration
commit: f20b817fd29b64ef6de24b83ef23e1f3fb273967
[19/27] io_uring: wire send zc request type
commit: 480ec5ff9a5a75d68423c0bd02e57a9ee6325320
[20/27] io_uring: account locked pages for non-fixed zc
commit: fcb98e61d0232cff7dd14ae85ad1c88d68f98273
[21/27] io_uring: allow to pass addr into sendzc
commit: 7ab12997edc9aa3e2be4169f929c50a1fcd41004
[22/27] io_uring: sendzc with fixed buffers
commit: bb4019de9ea11d21137b4a8ff01d9e338071d633
[23/27] io_uring: flush notifiers after sendzc
commit: 95a70c191696da64a6ae235d52132a5c17866dae
[24/27] io_uring: rename IORING_OP_FILES_UPDATE
commit: d488e605a45192f9f60c7624d46ba0b8c4d93aab
[25/27] io_uring: add zc notification flush requests
commit: cb155defb9bf20a647c8825a085695f3f94fdb60
[26/27] io_uring: enable managed frags with register buffers
commit: 04ae3dbe8a027cf10ab759456ffc4fb119486f74
[27/27] selftests/io_uring: test zerocopy send
commit: 0c450de20ce7d6bc8a2f97c98387baf910454477Best regards,
Signature · 1 line
Jens Axboe
Quoted 13 lines · Pavel Begunkov
NOTE: Not to be picked directly. After getting necessary acks, I'll be
working out merging with Jakub and Jens.The patchset implements
io_uringzerocopy send. It works with both registered and normal buffers, mixing is allowed but not recommended. Apart from usual request completions, just as withMSG_ZEROCOPY,io_uringseparately notifies the userspace when buffers are freed and can be reused (see API design below), which is delivered into io_uring's Completion Queue. Those "buffer-free" notifications are not necessarily per request, but the userspace has control over it and should explicitly attaching a number of requests to a single notification. The series also adds some internal optimisations when used with registered buffers like removing page referencing.
Quoted 8 lines
Quoted 1 line
From the kernel networking perspective there are two main changes. The first
one is passing
ubuf_infointo the network layer fromio_uring(inside of an in kernel struct msghdr). This allows extra optimisations, e.g.ubuf_infocaching on theio_uringside, but also helps to avoid cross-referencing and synchronisation problems. The second part is an optional optimisation removing page referencing for requests with registered buffers.Benchmarking UDP with an optimised version of the selftest (see [1]), which
Hi, Pavel, I'm interested in zero copy sending of io_uring, but I can't reproduce its performance using zerocopy send selftest test case, such as "bash io_uring_zerocopy_tx.sh 6 udp -m 0/1/2/3 -n 64", even baseline performance may be the best.
MB/s
NONZC 8379 ZC 5910 ZC_FIXED 6294 MIXED 6350
And the zero-copy example in [1] does not seem to work because the kernel is modified by following commit:
https://lore.kernel.org/all/[email protected]/
Can you help me reproduce this performance test result? Is it necessary to configure better parameters to reproduce the problem?
Quoted 165 lines
sends a bunch of requests, waits for completions and repeats. "+ flush" column posts one additional "buffer-free" notification per request, and just "zc" doesn't post buffer notifications at all.
NIC (requests / second):
IO size | non-zc | zc | zc + flush 4000 | 495134 | 606420 (+22%) | 558971 (+12%) 1500 | 551808 | 577116 (+4.5%) | 565803 (+2.5%) 1000 | 584677 | 592088 (+1.2%) | 560885 (-4%) 600 | 596292 | 598550 (+0.4%) | 555366 (-6.7%)dummy (requests / second):
IO size | non-zc | zc | zc + flush 8000 | 1299916 | 2396600 (+84%) | 2224219 (+71%) 4000 | 1869230 | 2344146 (+25%) | 2170069 (+16%) 1200 | 2071617 | 2361960 (+14%) | 2203052 (+6%) 600 | 2106794 | 2381527 (+13%) | 2195295 (+4%)Previously it also brought a massive performance speedup compared to the
msg_zerocopytool (see [3]), which is probably not super interesting. There is also an additional bunch of refcounting optimisations that was omitted from the series for simplicity and as they don't change the picture drastically, they will be sent as follow up, as well as flushing optimisations closing the performance gap b/w two last columns.For TCP on localhost (with hacks enabling localhost zerocopy) and including additional overhead for receive:
IO size | non-zc | zc 1200 | 4174 | 4148 4096 | 7597 | 11228Using a real NIC 1200 bytes, zc is worse than non-zc ~5-10%, maybe the omitted optimisations will somewhat help, should look better for 4000, but couldn't test properly because of setup problems.
Links:
liburing (benchmark + tests):
[1] https://github.com/isilence/liburing/tree/zc_v4kernel repo:
[2] https://github.com/isilence/linux/tree/zc_v4RFC v1:
[3] https://lore.kernel.org/io-uring/[email protected]/RFC v2:
https://lore.kernel.org/io-uring/[email protected]/Net patches based:
[email protected]:isilence/linux.git zc_v4-net-base or
https://github.com/isilence/linux/tree/zc_v4-net-baseAPI design overview:
The series introduces an
io_uringconcept of notifactors. From the userspace perspective it's an entity to which it can bind one or more requests and then requesting to flush it. Flushing a notifier makes it impossible to attach new requests to it, and instructs the notifier to post a completion once all requests attached to it are completed and the kernel doesn't need the buffers anymore.Notifications are stored in notification slots, which should be registered as an array in
io_uring. Each slot stores only one notifier at any particular moment. Flushing removes it from the slot and the slot automatically replaces it with a new notifier. All operations with notifiers are done by specifying an index of a slot it's currently in.When registering a notification the userspace specifies a u64 tag for each slot, which will be copied in notification completion entries as
cqe::user_data.cqe::resis 0 andcqe::flagsis equal to wrap around u32 sequence number counting notifiers of a slot.Changelog:
v4 -> v5
removeubuf_infochecks from customiov_itercallbacks to avoid disabling the page refs optimisations for TCPv3 -> v4
customiov_iterhandlingRFC v2 -> v3:
mem accounting for non-registered buffers allow mixing registered and normal requests per notifier notification flushing viaIORING_OP_RSRC_UPDATETCP support
fix buffer indexing
fix io-wq ->uring_lock locking
fix bugs when mixing withMSG_ZEROCOPY
fix managed refs bugs inskbuff.cRFC -> RFC v2:
remove additional overhead for non-zc fromskb_release_data()avoid msg propagation, hide extra bits of non-zc overheadtask_workbased "buffer free" notifications
improve io_uring's notification refcounting
added 5/19, (no pfmemalloc tracking)
added 8/19 and 9/19 preventing small copies with zc misc small changesDavid Ahern (1):
net: Allow custom iter handler in msghdrPavel Begunkov (26):
ipv4: avoid partial copy for zc
ipv6: avoid partial copy for zc
skbuff: don't mixubuf_infofrom different sources
skbuff: addSKBFL_DONT_ORPHANflag
skbuff: carry externalubuf_infoin msghdr
net: introduce managed frags infrastructure
net: introduce__skb_fill_page_desc_noaccipv4/udp: support externally provided ubufs ipv6/udp: support externally provided ubufs
tcp: support externally provided ubufs
io_uring: initialisemsghdr::msg_ubuf
io_uring: exportio_put_task()
io_uring: add zc notification infrastructure
io_uring: cache structio_notif
io_uring: complete notifiers in tw
io_uring: add rsrc referencing for notifiers
io_uring: add notification slot registration
io_uring: wire send zc request type
io_uring: account locked pages for non-fixed zc
io_uring: allow to pass addr into sendzc
io_uring: sendzc with fixed buffers
io_uring: flush notifiers after sendzc
io_uring: renameIORING_OP_FILES_UPDATE
io_uring: add zc notification flush requests
io_uring: enable managed frags with register buffers selftests/io_uring: test zerocopy sendinclude/linux/io_uring_types.h | 37 ++ include/linux/skbuff.h | 66 +- include/linux/socket.h | 5 + include/uapi/linux/io_uring.h | 45 +- io_uring/Makefile | 2 +- io_uring/io_uring.c | 42 +- io_uring/io_uring.h | 22 + io_uring/net.c | 187 ++++++ io_uring/net.h | 4 + io_uring/notif.c | 215 +++++++ io_uring/notif.h | 87 +++ io_uring/opdef.c | 24 +- io_uring/rsrc.c | 55 +- io_uring/rsrc.h | 16 +- io_uring/tctx.h | 26 - net/compat.c | 1 + net/core/datagram.c | 14 +- net/core/skbuff.c | 37 +- net/ipv4/ip_output.c | 50 +- net/ipv4/tcp.c | 32 +- net/ipv6/ip6_output.c | 49 +- net/socket.c | 3 + tools/testing/selftests/net/Makefile | 1 + .../selftests/net/io_uring_zerocopy_tx.c | 605 ++++++++++++++++++ .../selftests/net/io_uring_zerocopy_tx.sh | 131 ++++ 25 files changed, 1628 insertions(+), 128 deletions(-) create mode 100644 io_uring/notif.c create mode 100644 io_uring/notif.h create mode 100644 tools/testing/selftests/net/io_uring_zerocopy_tx.c create mode 100755 tools/testing/selftests/net/io_uring_zerocopy_tx.sh
Quoted 28 lines · Jinjie Ruan
Quoted 13 lines · Pavel Begunkov
NOTE: Not to be picked directly. After getting necessary acks, I'll be
working out merging with Jakub and Jens.The patchset implements
io_uringzerocopy send. It works with both registered and normal buffers, mixing is allowed but not recommended. Apart from usual request completions, just as withMSG_ZEROCOPY,io_uringseparately notifies the userspace when buffers are freed and can be reused (see API design below), which is delivered into io_uring's Completion Queue. Those "buffer-free" notifications are not necessarily per request, but the userspace has control over it and should explicitly attaching a number of requests to a single notification. The series also adds some internal optimisations when used with registered buffers like removing page referencing.Quoted 8 lines
Quoted 1 line
From the kernel networking perspective there are two main changes. The first
one is passing
ubuf_infointo the network layer fromio_uring(inside of an in kernel struct msghdr). This allows extra optimisations, e.g.ubuf_infocaching on theio_uringside, but also helps to avoid cross-referencing and synchronisation problems. The second part is an optional optimisation removing page referencing for requests with registered buffers.Benchmarking UDP with an optimised version of the selftest (see [1]), which
Hi, Pavel, I'm interested in zero copy sending of
io_uring, but I can't reproduce its performance using zerocopy send selftest test case, such as "bashio_uring_zerocopy_tx.sh6 udp -m 0/1/2/3 -n 64", even baseline performance may be the best.
Quoted 5 lines
MB/s
NONZC 8379 ZC 5910 ZC_FIXED 6294 MIXED 6350
It's using veth, and zerocopy is effectively disabled for most of virtual devices, or to be specific "for paths that may loop packets to receive sockets".
https://lore.kernel.org/netdev/[email protected]/
So that's the worst of the two, it copies data but also incurs the overhead for notifications. You can use a dummy device as a sink with no receiver, but you'll get more realistic numbers if you use a real device (that supports features required for zerocopy).
Quoted 4 lines
And the zero-copy example in [1] does not seem to work because the kernel is modified by following commit:
The right version was merged long ago and sits in
liburing/examples/send-zerocopy.c
It's brushed up more than the selftest version, so I'd suggest using that one. Arguments are a bit different, but it prints help.
./send-zerocopy -6 udp -D <ip> -t 10 -n 1 -l0 -b1 -d -z1
Signature · 1 line
Pavel Begunkov