io_uring zerocopy send
Subsystems (7)
v4 cover io_uring zerocopy send
NOTE: Not be picked directly. After getting necessary acks, I'll be working
out merging with Jakub and Jens.
The patchset implements io_uring zerocopy send. It works with both registered and normal buffers, mixing is allowed but not recommended. Apart from usual request completions, just as with MSG_ZEROCOPY, io_uring separately notifies the userspace when buffers are freed and can be reused (see API design below), which is delivered into io_uring's Completion Queue. Those "buffer-free" notifications are not necessarily per request, but the userspace has control over it and should explicitly attaching a number of requests to a single notification. The series also adds some internal optimisations when used with registered buffers like removing page referencing.
From the kernel networking perspective there are two main changes. The first one is passing ubuf_info into the network layer from io_uring (inside of an in kernel struct msghdr). This allows extra optimisations, e.g. ubuf_info caching on the io_uring side, but also helps to avoid cross-referencing and synchronisation problems. The second part is an optional optimisation removing page referencing for requests with registered buffers.
Benchmarking with an optimised version of the selftest (see [1]), which sends a bunch of requests, waits for completions and repeats. "+ flush" column posts one additional "buffer-free" notification per request, and just "zc" doesn't post buffer notifications at all.
NIC (requests / second):
IO size | non-zc | zc | zc + flush 4000 | 495134 | 606420 (+22%) | 558971 (+12%) 1500 | 551808 | 577116 (+4.5%) | 565803 (+2.5%) 1000 | 584677 | 592088 (+1.2%) | 560885 (-4%) 600 | 596292 | 598550 (+0.4%) | 555366 (-6.7%)
dummy (requests / second):
IO size | non-zc | zc | zc + flush 8000 | 1299916 | 2396600 (+84%) | 2224219 (+71%) 4000 | 1869230 | 2344146 (+25%) | 2170069 (+16%) 1200 | 2071617 | 2361960 (+14%) | 2203052 (+6%) 600 | 2106794 | 2381527 (+13%) | 2195295 (+4%)
Previously it also brought a massive performance speedup compared to the msg_zerocopy tool (see [3]), which is probably not super interesting.
There is an additional bunch of refcounting optimisations that was omitted from the series for simplicity and as they don't change the picture drastically, they will be sent as follow up, as well as flushing optimisations closing the performance gap b/w two last columns.
Note: the series is based on net-next + for-5.20/io_uring, but as vanilla net-next fails for me the repo (see [2]) is on top of for-5.20/io_uring.
Links:
liburing (benchmark + tests):
[1] https://github.com/isilence/liburing/tree/zc_v4
kernel repo:
[2] https://github.com/isilence/linux/tree/zc_v4
RFC v1:
[3] https://lore.kernel.org/io-uring/[email protected]/
RFC v2:
https://lore.kernel.org/io-uring/[email protected]/
Net patches based [email protected]:isilence/linux.git zc_v4-net-base
API design overview:
The series introduces an io_uring concept of notifactors. From the userspace perspective it's an entity to which it can bind one or more requests and then requesting to flush it. Flushing a notifier makes it impossible to attach new requests to it, and instructs the notifier to post a completion once all requests attached to it are completed and the kernel doesn't need the buffers anymore.
Notifications are stored in notification slots, which should be registered as an array in io_uring. Each slot stores only one notifier at any particular moment. Flushing removes it from the slot and the slot automatically replaces it with a new notifier. All operations with notifiers are done by specifying an index of a slot it's currently in.
When registering a notification the userspace specifies a u64 tag for each slot, which will be copied in notification completion entries as cqe::user_data. cqe::res is 0 and cqe::flags is equal to wrap around u32 sequence number counting notifiers of a slot.
Changelog:
v3 -> v4
custom iov_iter handling
RFC v2 -> v3:
mem accounting for non-registered buffers allow mixing registered and normal requests per notifier notification flushing via IORING_OP_RSRC_UPDATE TCP support
fix buffer indexing
fix io-wq ->uring_lock locking
fix bugs when mixing with MSG_ZEROCOPY
fix managed refs bugs in skbuff.c
RFC -> RFC v2:
remove additional overhead for non-zc from skb_release_data() avoid msg propagation, hide extra bits of non-zc overhead task_work based "buffer free" notifications
improve io_uring's notification refcounting
added 5/19, (no pfmemalloc tracking)
added 8/19 and 9/19 preventing small copies with zc misc small changes
David Ahern (1):
net: Allow custom iter handler in msghdr
Pavel Begunkov (26):
ipv4: avoid partial copy for zc
ipv6: avoid partial copy for zc
skbuff: don't mix ubuf_info from different sources
skbuff: add SKBFL_DONT_ORPHAN flag
skbuff: carry external ubuf_info in msghdr
net: introduce managed frags infrastructure
net: introduce __skb_fill_page_desc_noacc ipv4/udp: support externally provided ubufs ipv6/udp: support externally provided ubufs
tcp: support externally provided ubufs
io_uring: initialise msghdr::msg_ubuf
io_uring: export io_put_task()
io_uring: add zc notification infrastructure
io_uring: cache struct io_notif
io_uring: complete notifiers in tw
io_uring: add rsrc referencing for notifiers
io_uring: add notification slot registration
io_uring: wire send zc request type
io_uring: account locked pages for non-fixed zc
io_uring: allow to pass addr into sendzc
io_uring: sendzc with fixed buffers
io_uring: flush notifiers after sendzc
io_uring: rename IORING_OP_FILES_UPDATE
io_uring: add zc notification flush requests
io_uring: enable managed frags with register buffers selftests/io_uring: test zerocopy send
include/linux/io_uring_types.h | 37 ++ include/linux/skbuff.h | 66 +- include/linux/socket.h | 5 + include/uapi/linux/io_uring.h | 45 +- io_uring/Makefile | 2 +- io_uring/io_uring.c | 42 +- io_uring/io_uring.h | 22 + io_uring/net.c | 187 ++++++ io_uring/net.h | 4 + io_uring/notif.c | 215 +++++++ io_uring/notif.h | 87 +++ io_uring/opdef.c | 24 +- io_uring/rsrc.c | 55 +- io_uring/rsrc.h | 16 +- io_uring/tctx.h | 26 - net/compat.c | 1 + net/core/datagram.c | 14 +- net/core/skbuff.c | 37 +- net/ipv4/ip_output.c | 50 +- net/ipv4/tcp.c | 32 +- net/ipv6/ip6_output.c | 49 +- net/socket.c | 3 + tools/testing/selftests/net/Makefile | 1 + .../selftests/net/io_uring_zerocopy_tx.c | 605 ++++++++++++++++++ .../selftests/net/io_uring_zerocopy_tx.sh | 131 ++++ 25 files changed, 1628 insertions(+), 128 deletions(-) create mode 100644 io_uring/notif.c create mode 100644 io_uring/notif.h create mode 100644 tools/testing/selftests/net/io_uring_zerocopy_tx.c create mode 100755 tools/testing/selftests/net/io_uring_zerocopy_tx.sh
Signature · 1 line
2.36.1
PATCH v4 1/27 ipv4: avoid partial copy for zc
Even when zerocopy transmission is requested and possible, __ip_append_data() will still copy a small chunk of data just because it allocated some extra linear space (e.g. 148 bytes). It wastes CPU cycles on copy and iter manipulations and also misalignes potentially aligned data. Avoid such coies. And as a bonus we can allocate smaller skb.
--- net/ipv4/ip_output.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
net/ipv4/ip_output.c +6 -2
@@ -969,7 +969,6 @@ static int __ip_append_data(struct sock *sk,struct inet_sock *inet =inet_sk (sk);struct ubuf_info *uarg =NULL ;struct sk_buff *skb; -struct ip_options *opt = cork->opt;int hh_len;int exthdrlen;
@@ -977,6 +976,7 @@ static int __ip_append_data(struct sock *sk,int copy;int err;int offset =0 ; +bool zc = false;unsigned int maxfraglen, fragheaderlen, maxnonfragsize;int csummode =CHECKSUM_NONE ;struct rtable *rt = (struct rtable *)cork->dst;
@@ -1025,6 +1025,7 @@ static int __ip_append_data(struct sock *sk,if (rt->dst.dev->features &NETIF_F_SG && csummode ==CHECKSUM_PARTIAL ) { paged = true; + zc = true; }else { uarg->zerocopy =0 ;skb_zcopy_set (skb, uarg, &extra_uref);
@@ -1091,9 +1092,12 @@ static int __ip_append_data(struct sock *sk, (fraglen + alloc_extra <SKB_MAX_ALLOC || !(rt->dst.dev->features & NETIF_F_SG))) alloclen = fraglen; - else { + else if (!zc) { alloclen =min_t (int, fraglen,MAX_HEADER ); pagedlen = fraglen - alloclen; + } else { + alloclen = fragheaderlen + transhdrlen; + pagedlen = datalen - transhdrlen; } alloclen += alloc_extra;
Signature · 1 line
2.36.1
PATCH v4 2/27 ipv6: avoid partial copy for zc
In reply to Pavel BegunkovEven when zerocopy transmission is requested and possible, __ip_append_data() will still copy a small chunk of data just because it allocated some extra linear space (e.g. 128 bytes). It wastes CPU cycles on copy and iter manipulations and also misalignes potentially aligned data. Avoid such coies. And as a bonus we can allocate smaller skb.
--- net/ipv6/ip6_output.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
net/ipv6/ip6_output.c +6 -1
@@ -1464,6 +1464,7 @@ static int __ip6_append_data(struct sock *sk,int copy;int err;int offset =0 ; +bool zc = false;u32 tskey =0 ;struct rt6_info *rt = (struct rt6_info *)cork->dst;struct ipv6_txoptions *opt = v6_cork->opt;
@@ -1549,6 +1550,7 @@ static int __ip6_append_data(struct sock *sk,if (rt->dst.dev->features &NETIF_F_SG && csummode ==CHECKSUM_PARTIAL ) { paged = true; + zc = true; }else { uarg->zerocopy =0 ;skb_zcopy_set (skb, uarg, &extra_uref);
@@ -1630,9 +1632,12 @@ static int __ip6_append_data(struct sock *sk, (fraglen + alloc_extra <SKB_MAX_ALLOC || !(rt->dst.dev->features & NETIF_F_SG))) alloclen = fraglen; - else { + else if (!zc) { alloclen =min_t (int, fraglen,MAX_HEADER ); pagedlen = fraglen - alloclen; + } else { + alloclen = fragheaderlen + transhdrlen; + pagedlen = datalen - transhdrlen; } alloclen += alloc_extra;
Signature · 1 line
2.36.1
PATCH v4 3/27 skbuff: don't mix ubuf_info from different sources
In reply to Pavel BegunkovWe should not append MSG_ZEROCOPY requests to skbuff with non MSG_ZEROCOPY ubuf_info, they might be not compatible.
--- net/core/skbuff.c | 4 ++++ 1 file changed, 4 insertions(+)
net/core/skbuff.c +4 -0
@@ -1212,6 +1212,10 @@ struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size,const u32 byte_limit =1 <<19 ;/* limit to a few TSO */ u32 bytelen, next; +/* there might be non MSG_ZEROCOPY users */ +if (uarg->callback != msg_zerocopy_callback) +return NULL ; +/* realloc only when socket is locked (TCP, UDP cork), * so uarg->len and sk_zckey access is serialized */
Signature · 1 line
2.36.1
PATCH v4 4/27 skbuff: add SKBFL_DONT_ORPHAN flag
In reply to Pavel BegunkovWe don't want to list every single ubuf_info callback in skb_orphan_frags(), add a flag controlling the behaviour.
--- include/linux/skbuff.h | 8 +++++--- net/core/skbuff.c | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-)
include/linux/skbuff.h +5 -3
@@ -686,10 +686,13 @@ enum {* charged to the kernel memory. */ SKBFL_PURE_ZEROCOPY =BIT (2 ), + + SKBFL_DONT_ORPHAN = BIT(3), };#define SKBFL_ZEROCOPY_FRAG (SKBFL_ZEROCOPY_ENABLE | SKBFL_SHARED_FRAG) -#define SKBFL_ALL_ZEROCOPY (SKBFL_ZEROCOPY_FRAG | SKBFL_PURE_ZEROCOPY) +#define SKBFL_ALL_ZEROCOPY (SKBFL_ZEROCOPY_FRAG | SKBFL_PURE_ZEROCOPY | \ + SKBFL_DONT_ORPHAN)/* * The callback notifies userspace to release buffers when skb DMA is done in
@@ -3182,8 +3185,7 @@ static inline int skb_orphan_frags(struct sk_buff *skb, gfp_t gfp_mask) {if (likely (!skb_zcopy (skb)))return 0 ; -if (!skb_zcopy_is_nouarg (skb) && -skb_uarg (skb)->callback == msg_zerocopy_callback) +if (skb_shinfo (skb)->flags &SKBFL_DONT_ORPHAN )return 0 ;return skb_copy_ubufs (skb, gfp_mask); }
net/core/skbuff.c +1 -1
@@ -1193,7 +1193,7 @@ static struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size) uarg->len =1 ; uarg->bytelen = size; uarg->zerocopy =1 ; - uarg->flags =SKBFL_ZEROCOPY_FRAG ; + uarg->flags =SKBFL_ZEROCOPY_FRAG |SKBFL_DONT_ORPHAN ;refcount_set (&uarg->refcnt,1 );sock_hold (sk);
Signature · 1 line
2.36.1
PATCH v4 5/27 skbuff: carry external ubuf_info in msghdr
In reply to Pavel BegunkovMake possible for network in-kernel callers like io_uring to pass in a custom ubuf_info by setting it in a new field of struct msghdr.
--- include/linux/socket.h | 1 + net/compat.c | 1 + net/socket.c | 3 +++ 3 files changed, 5 insertions(+)
include/linux/socket.h +1 -0
@@ -69,6 +69,7 @@ struct msghdr {unsigned int msg_flags;/* flags on received message */ __kernel_size_t msg_controllen;/* ancillary data buffer length */ struct kiocb *msg_iocb;/* ptr to iocb for async requests */ +struct ubuf_info *msg_ubuf; };struct user_msghdr {
net/compat.c +1 -0
@@ -80,6 +80,7 @@ int __get_compat_msghdr(struct msghdr *kmsg,return -EMSGSIZE ; kmsg->msg_iocb =NULL ; + kmsg->msg_ubuf =NULL ; *ptr = msg.msg_iov; *len = msg.msg_iovlen;return 0 ;
net/socket.c +3 -0
@@ -2106,6 +2106,7 @@ int __sys_sendto(int fd, void __user *buff, size_t len, unsigned int flags, msg.msg_control =NULL ; msg.msg_controllen =0 ; msg.msg_namelen =0 ; + msg.msg_ubuf =NULL ;if (addr) { err =move_addr_to_kernel (addr, addr_len, &address);if (err <0 )
@@ -2171,6 +2172,7 @@ int __sys_recvfrom(int fd, void __user *ubuf, size_t size, unsigned int flags, msg.msg_namelen =0 ; msg.msg_iocb =NULL ; msg.msg_flags =0 ; + msg.msg_ubuf =NULL ;if (sock->file->f_flags &O_NONBLOCK ) flags |=MSG_DONTWAIT ; err =sock_recvmsg (sock, &msg, flags);
@@ -2409,6 +2411,7 @@ int __copy_msghdr_from_user(struct msghdr *kmsg,return -EMSGSIZE ; kmsg->msg_iocb =NULL ; + kmsg->msg_ubuf =NULL ; *uiov = msg.msg_iov; *nsegs = msg.msg_iovlen;return 0 ;
Signature · 1 line
2.36.1
PATCH v4 6/27 net: Allow custom iter handler in msghdr
In reply to Pavel BegunkovFrom: David Ahern <[email protected]>
Add support for custom iov_iter handling to msghdr. The idea is that in-kernel subsystems want control over how an SG is split.
--- include/linux/skbuff.h | 7 ++++--- include/linux/socket.h | 4 ++++ net/core/datagram.c | 14 ++++++++++---- net/core/skbuff.c | 2 +- 4 files changed, 19 insertions(+), 8 deletions(-)
include/linux/skbuff.h +4 -3
@@ -1776,13 +1776,14 @@ void msg_zerocopy_put_abort(struct ubuf_info *uarg, bool have_uref);void msg_zerocopy_callback (struct sk_buff *skb,struct ubuf_info *uarg,bool success); -int __zerocopy_sg_from_iter (struct sock *sk,struct sk_buff *skb, -struct iov_iter *from,size_t length); +int __zerocopy_sg_from_iter (struct msghdr *msg,struct sock *sk, +struct sk_buff *skb,struct iov_iter *from, +size_t length);static inline int skb_zerocopy_iter_dgram (struct sk_buff *skb,struct msghdr *msg,int len) { -return __zerocopy_sg_from_iter (skb->sk, skb, &msg->msg_iter, len); +return __zerocopy_sg_from_iter (msg, skb->sk, skb, &msg->msg_iter, len); } int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb,
include/linux/socket.h +4 -0
@@ -14,6 +14,8 @@ struct file;struct pid ;struct cred ;struct socket ; +struct sock ; +struct sk_buff ;#define __sockaddr_check_size (size) \ BUILD_BUG_ON(((size) > sizeof(struct __kernel_sockaddr_storage)))
@@ -70,6 +72,8 @@ struct msghdr {__kernel_size_t msg_controllen;/* ancillary data buffer length */ struct kiocb *msg_iocb;/* ptr to iocb for async requests */ struct ubuf_info *msg_ubuf; +int (*sg_from_iter)(struct sock *sk,struct sk_buff *skb, +struct iov_iter *from,size_t length); };struct user_msghdr {
net/core/datagram.c +10 -4
@@ -613,10 +613,16 @@ int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, }EXPORT_SYMBOL (skb_copy_datagram_from_iter); -int __zerocopy_sg_from_iter (struct sock *sk,struct sk_buff *skb, -struct iov_iter *from,size_t length) +int __zerocopy_sg_from_iter (struct msghdr *msg,struct sock *sk, +struct sk_buff *skb,struct iov_iter *from, +size_t length) { -int frag =skb_shinfo (skb)->nr_frags; +int frag; + +if (msg && msg->sg_from_iter && msg->msg_ubuf ==skb_zcopy (skb)) +return msg->sg_from_iter (sk, skb, from, length); + + frag =skb_shinfo (skb)->nr_frags;while (length &&iov_iter_count (from)) {struct page *pages[MAX_SKB_FRAGS ];
@@ -702,7 +708,7 @@ int zerocopy_sg_from_iter(struct sk_buff *skb, struct iov_iter *from)if (skb_copy_datagram_from_iter (skb,0 , from, copy))return -EFAULT ; -return __zerocopy_sg_from_iter (NULL , skb, from, ~0U ); +return __zerocopy_sg_from_iter (NULL ,NULL , skb, from, ~0U ); }EXPORT_SYMBOL (zerocopy_sg_from_iter);
net/core/skbuff.c +1 -1
@@ -1358,7 +1358,7 @@ int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb,if (orig_uarg && uarg != orig_uarg)return -EEXIST ; - err =__zerocopy_sg_from_iter (sk, skb, &msg->msg_iter, len); + err =__zerocopy_sg_from_iter (msg, sk, skb, &msg->msg_iter, len);if (err == -EFAULT || (err == -EMSGSIZE && skb->len == orig_len)) {struct sock *save_sk = skb->sk;
Signature · 1 line
2.36.1
Re: [PATCH net-next v4 06/27] net: Allow custom iter handler in msghdr
Quoted 67 lines · Pavel Begunkov
From: David Ahern <[email protected]>
Add support for custom
iov_iterhandling to msghdr. The idea is that in-kernel subsystems want control over how an SG is split.Signed-off-by David Ahern <[email protected]>[pavel: move callback into msghdr]Signed-off-by Pavel Begunkov <[email protected]>--- include/linux/skbuff.h | 7 ++++--- include/linux/socket.h | 4 ++++ net/core/datagram.c | 14 ++++++++++---- net/core/skbuff.c | 2 +- 4 files changed, 19 insertions(+), 8 deletions(-)include/linux/skbuff.h +4
-3@@ -1776,13 +1776,14 @@ void msg_zerocopy_put_abort(struct ubuf_info *uarg, bool have_uref);void msg_zerocopy_callback (struct sk_buff *skb,struct ubuf_info *uarg,bool success); -int __zerocopy_sg_from_iter (struct sock *sk,struct sk_buff *skb, -struct iov_iter *from,size_t length); +int __zerocopy_sg_from_iter (struct msghdr *msg,struct sock *sk, +struct sk_buff *skb,struct iov_iter *from, +size_t length);static inline int skb_zerocopy_iter_dgram (struct sk_buff *skb,struct msghdr *msg,int len) { -return __zerocopy_sg_from_iter (skb->sk, skb, &msg->msg_iter, len); +return __zerocopy_sg_from_iter (msg, skb->sk, skb, &msg->msg_iter, len); } int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb,include/linux/socket.h +4
-0@@ -14,6 +14,8 @@ struct file;struct pid ;struct cred ;struct socket ; +struct sock ; +struct sk_buff ;#define __sockaddr_check_size (size) \ BUILD_BUG_ON(((size) > sizeof(struct __kernel_sockaddr_storage)))@@ -70,6 +72,8 @@ struct msghdr {__kernel_size_t msg_controllen;/* ancillary data buffer length */ struct kiocb *msg_iocb;/* ptr to iocb for async requests */ struct ubuf_info *msg_ubuf; +int (*sg_from_iter)(struct sock *sk,struct sk_buff *skb, +struct iov_iter *from,size_t length); };struct user_msghdr {net/core/datagram.c +0
-0@@ -613,10 +613,16 @@ int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, }EXPORT_SYMBOL (skb_copy_datagram_from_iter);
net/core/datagram.c int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, · line 622
Quoted 10 lines
net/core/datagram.c +6
-3-int __zerocopy_sg_from_iter (struct sock *sk,struct sk_buff *skb, -struct iov_iter *from,size_t length) +int __zerocopy_sg_from_iter (struct msghdr *msg,struct sock *sk, +struct sk_buff *skb,struct iov_iter *from, +size_t length) { -int frag =skb_shinfo (skb)->nr_frags; +int frag; + +if (msg && msg->sg_from_iter && msg->msg_ubuf ==skb_zcopy (skb))
I'm killing "msg->msg_ubuf == skb_zcopy(skb)", which I added with an intention to make it less fragile, but it disables the optimisation for TCP because skb_zerocopy_iter_stream() assigns ubuf to the skb only after calling __zerocopy_sg_from_iter().
Quoted 28 lines
net/core/datagram.c +4
-1+return msg->sg_from_iter (sk, skb, from, length); + + frag =skb_shinfo (skb)->nr_frags;while (length &&iov_iter_count (from)) {struct page *pages[MAX_SKB_FRAGS ];@@ -702,7 +708,7 @@ int zerocopy_sg_from_iter(struct sk_buff *skb, struct iov_iter *from)if (skb_copy_datagram_from_iter (skb,0 , from, copy))return -EFAULT ; -return __zerocopy_sg_from_iter (NULL , skb, from, ~0U ); +return __zerocopy_sg_from_iter (NULL ,NULL , skb, from, ~0U ); }EXPORT_SYMBOL (zerocopy_sg_from_iter);net/core/skbuff.c +1
-1@@ -1358,7 +1358,7 @@ int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb,if (orig_uarg && uarg != orig_uarg)return -EEXIST ; - err =__zerocopy_sg_from_iter (sk, skb, &msg->msg_iter, len); + err =__zerocopy_sg_from_iter (msg, sk, skb, &msg->msg_iter, len);if (err == -EFAULT || (err == -EMSGSIZE && skb->len == orig_len)) {struct sock *save_sk = skb->sk;
Signature · 1 line
Pavel Begunkov
PATCH v4 7/27 net: introduce managed frags infrastructure
In reply to Pavel BegunkovSome users like io_uring can do page pinning more efficiently, so we want a way to delegate referencing to other subsystems. For that add a new flag called SKBFL_MANAGED_FRAG_REFS. When set, skb doesn't hold page references and upper layers are responsivle to managing page lifetime.
It's allowed to convert skbs from managed to normal by calling skb_zcopy_downgrade_managed(). The function will take all needed page references and clear the flag. It's needed, for instance, to avoid mixing managed modes.
--- include/linux/skbuff.h | 25 +++++++++++++++++++++++-- net/core/skbuff.c | 29 +++++++++++++++++++++++++++-- 2 files changed, 50 insertions(+), 4 deletions(-)
include/linux/skbuff.h +23 -2
@@ -688,11 +688,16 @@ enum {SKBFL_PURE_ZEROCOPY =BIT (2 ),SKBFL_DONT_ORPHAN =BIT (3 ), + +/* page references are managed by the ubuf_info, so it's safe to +* use frags only up until ubuf_info is released +*/ + SKBFL_MANAGED_FRAG_REFS = BIT(4), };#define SKBFL_ZEROCOPY_FRAG (SKBFL_ZEROCOPY_ENABLE | SKBFL_SHARED_FRAG)#define SKBFL_ALL_ZEROCOPY (SKBFL_ZEROCOPY_FRAG | SKBFL_PURE_ZEROCOPY | \ - SKBFL_DONT_ORPHAN) + SKBFL_DONT_ORPHAN | SKBFL_MANAGED_FRAG_REFS)/* * The callback notifies userspace to release buffers when skb DMA is done in
@@ -1810,6 +1815,11 @@ static inline bool skb_zcopy_pure(const struct sk_buff *skb)return skb_shinfo (skb)->flags &SKBFL_PURE_ZEROCOPY ; } +static inline bool skb_zcopy_managed (const struct sk_buff *skb) +{ +return skb_shinfo (skb)->flags &SKBFL_MANAGED_FRAG_REFS ; +} +static inline bool skb_pure_zcopy_same (const struct sk_buff *skb1,const struct sk_buff *skb2) {
@@ -1884,6 +1894,14 @@ static inline void skb_zcopy_clear(struct sk_buff *skb, bool zerocopy_success) } } +void __skb_zcopy_downgrade_managed (struct sk_buff *skb); + +static inline void skb_zcopy_downgrade_managed (struct sk_buff *skb) +{ +if (unlikely (skb_zcopy_managed (skb))) +__skb_zcopy_downgrade_managed (skb); +} +static inline void skb_mark_not_on_list (struct sk_buff *skb) { skb->next =NULL ;
@@ -3499,7 +3517,10 @@ static inline void __skb_frag_unref(skb_frag_t *frag, bool recycle)*/ static inline void skb_frag_unref (struct sk_buff *skb,int f) { -__skb_frag_unref (&skb_shinfo (skb)->frags[f], skb->pp_recycle); +struct skb_shared_info *shinfo =skb_shinfo (skb); + +if (!skb_zcopy_managed (skb)) +__skb_frag_unref (&shinfo->frags[f], skb->pp_recycle); }/**
net/core/skbuff.c +27 -2
@@ -666,11 +666,18 @@ static void skb_release_data(struct sk_buff *skb) &shinfo->dataref)) goto exit; -skb_zcopy_clear (skb, true); +if (skb_zcopy (skb)) { +bool skip_unref = shinfo->flags &SKBFL_MANAGED_FRAG_REFS ; + +skb_zcopy_clear (skb, true); +if (skip_unref) + goto free_head; + }for (i =0 ; i < shinfo->nr_frags; i++)__skb_frag_unref (&shinfo->frags[i], skb->pp_recycle); +free_head:if (shinfo->frag_list)kfree_skb_list (shinfo->frag_list);
@@ -895,7 +902,10 @@ EXPORT_SYMBOL(skb_dump);*/ void skb_tx_error (struct sk_buff *skb) { -skb_zcopy_clear (skb, true); +if (skb) { +skb_zcopy_downgrade_managed (skb); +skb_zcopy_clear (skb, true); + } }EXPORT_SYMBOL (skb_tx_error);
@@ -1375,6 +1385,16 @@ int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb, }EXPORT_SYMBOL_GPL (skb_zerocopy_iter_stream); +void __skb_zcopy_downgrade_managed (struct sk_buff *skb) +{ +int i; + +skb_shinfo (skb)->flags &= ~SKBFL_MANAGED_FRAG_REFS ; +for (i =0 ; i <skb_shinfo (skb)->nr_frags; i++) +skb_frag_ref (skb, i); +} +EXPORT_SYMBOL_GPL (__skb_zcopy_downgrade_managed); +static int skb_zerocopy_clone (struct sk_buff *nskb,struct sk_buff *orig,gfp_t gfp_mask) {
@@ -1692,6 +1712,8 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,BUG_ON (skb_shared (skb)); +skb_zcopy_downgrade_managed (skb); + size =SKB_DATA_ALIGN (size);if (skb_pfmemalloc (skb))
@@ -3488,6 +3510,8 @@ void skb_split(struct sk_buff *skb, struct sk_buff *skb1, const u32 len)int pos =skb_headlen (skb);const int zc_flags =SKBFL_SHARED_FRAG |SKBFL_PURE_ZEROCOPY ; +skb_zcopy_downgrade_managed (skb); +skb_shinfo (skb1)->flags |=skb_shinfo (skb)->flags & zc_flags;skb_zerocopy_clone (skb1, skb,0 );if (len < pos)/* Split line is inside header. */
@@ -3841,6 +3865,7 @@ int skb_append_pagefrags(struct sk_buff *skb, struct page *page,if (skb_can_coalesce (skb, i, page, offset)) {skb_frag_size_add (&skb_shinfo (skb)->frags[i -1 ], size); }else if (i <MAX_SKB_FRAGS ) { +skb_zcopy_downgrade_managed (skb);get_page (page);skb_fill_page_desc (skb, i, page, offset, size); }else {
Signature · 1 line
2.36.1
PATCH v4 8/27 net: introduce __skb_fill_page_desc_noacc
In reply to Pavel BegunkovManaged pages contain pinned userspace pages and controlled by upper layers, there is no need in tracking skb->pfmemalloc for them. Introduce a helper for filling frags but ignoring page tracking, it'll be needed later.
--- include/linux/skbuff.h | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-)
include/linux/skbuff.h +17 -11
@@ -2550,6 +2550,22 @@ static inline unsigned int skb_pagelen(const struct sk_buff *skb)return skb_headlen (skb) +__skb_pagelen (skb); } +static inline void __skb_fill_page_desc_noacc (struct skb_shared_info *shinfo, +int i,struct page *page, +int off,int size) +{ +skb_frag_t *frag = &shinfo->frags[i]; + +/* +* Propagate page pfmemalloc to the skb if we can. The problem is +* that not all callers have unique ownership of the page but rely +* on page_is_pfmemalloc doing the right thing(tm). +*/ + frag->bv_page = page; + frag->bv_offset = off; +skb_frag_size_set (frag, size); +} +/** * __skb_fill_page_desc - initialise a paged fragment in an skb * @skb: buffer containing fragment to be initialised
@@ -2566,17 +2582,7 @@ static inline unsigned int skb_pagelen(const struct sk_buff *skb)static inline void __skb_fill_page_desc (struct sk_buff *skb,int i,struct page *page,int off,int size) { -skb_frag_t *frag = &skb_shinfo (skb)->frags[i]; - -/* -* Propagate page pfmemalloc to the skb if we can. The problem is -* that not all callers have unique ownership of the page but rely -* on page_is_pfmemalloc doing the right thing(tm). -*/ - frag->bv_page = page; - frag->bv_offset = off; -skb_frag_size_set (frag, size); - +__skb_fill_page_desc_noacc (skb_shinfo (skb), i, page, off, size); page =compound_head (page);if (page_is_pfmemalloc (page)) skb->pfmemalloc = true;
Signature · 1 line
2.36.1
PATCH v4 9/27 ipv4/udp: support externally provided ubufs
In reply to Pavel BegunkovTeach ipv4/udp how to use external ubuf_info provided in msghdr and also prepare it for managed frags by sprinkling skb_zcopy_downgrade_managed() when it could mix managed and not managed frags.
--- net/ipv4/ip_output.c | 44 +++++++++++++++++++++++++++++++------------- 1 file changed, 31 insertions(+), 13 deletions(-)
net/ipv4/ip_output.c +31 -13
@@ -1017,18 +1017,35 @@ static int __ip_append_data(struct sock *sk, (!exthdrlen || (rt->dst.dev->features & NETIF_F_HW_ESP_TX_CSUM))) csummode =CHECKSUM_PARTIAL ; -if (flags &MSG_ZEROCOPY && length &&sock_flag (sk,SOCK_ZEROCOPY )) { - uarg =msg_zerocopy_realloc (sk, length,skb_zcopy (skb)); -if (!uarg) -return -ENOBUFS ; - extra_uref = !skb_zcopy (skb);/* only ref on new uarg */ -if (rt->dst.dev->features &NETIF_F_SG && - csummode ==CHECKSUM_PARTIAL ) { - paged = true; - zc = true; - }else { - uarg->zerocopy =0 ; -skb_zcopy_set (skb, uarg, &extra_uref); +if ((flags &MSG_ZEROCOPY ) && length) { +struct msghdr *msg = from; + +if (getfrag == ip_generic_getfrag && msg->msg_ubuf) { +if (skb_zcopy (skb) && msg->msg_ubuf !=skb_zcopy (skb)) +return -EINVAL ; + +/* Leave uarg NULL if can't zerocopy, callers should +* be able to handle it. +*/ +if ((rt->dst.dev->features &NETIF_F_SG ) && + csummode ==CHECKSUM_PARTIAL ) { + paged = true; + zc = true; + uarg = msg->msg_ubuf; + } + }else if (sock_flag (sk,SOCK_ZEROCOPY )) { + uarg =msg_zerocopy_realloc (sk, length,skb_zcopy (skb)); +if (!uarg) +return -ENOBUFS ; + extra_uref = !skb_zcopy (skb);/* only ref on new uarg */ +if (rt->dst.dev->features &NETIF_F_SG && + csummode ==CHECKSUM_PARTIAL ) { + paged = true; + zc = true; + }else { + uarg->zerocopy =0 ; +skb_zcopy_set (skb, uarg, &extra_uref); + } } }
@@ -1192,13 +1209,14 @@ static int __ip_append_data(struct sock *sk, err = -EFAULT ; goto error; } - }else if (!uarg || !uarg->zerocopy) { + }else if (!zc) {int i =skb_shinfo (skb)->nr_frags; err = -ENOMEM ;if (!sk_page_frag_refill (sk, pfrag)) goto error; +skb_zcopy_downgrade_managed (skb);if (!skb_can_coalesce (skb, i, pfrag->page, pfrag->offset)) { err = -EMSGSIZE ;
Signature · 1 line
2.36.1
PATCH v4 10/27 ipv6/udp: support externally provided ubufs
In reply to Pavel BegunkovTeach ipv6/udp how to use external ubuf_info provided in msghdr and also prepare it for managed frags by sprinkling skb_zcopy_downgrade_managed() when it could mix managed and not managed frags.
--- net/ipv6/ip6_output.c | 44 ++++++++++++++++++++++++++++++------------- 1 file changed, 31 insertions(+), 13 deletions(-)
net/ipv6/ip6_output.c +31 -13
@@ -1542,18 +1542,35 @@ static int __ip6_append_data(struct sock *sk, rt->dst.dev->features & (NETIF_F_IPV6_CSUM | NETIF_F_HW_CSUM)) csummode =CHECKSUM_PARTIAL ; -if (flags &MSG_ZEROCOPY && length &&sock_flag (sk,SOCK_ZEROCOPY )) { - uarg =msg_zerocopy_realloc (sk, length,skb_zcopy (skb)); -if (!uarg) -return -ENOBUFS ; - extra_uref = !skb_zcopy (skb);/* only ref on new uarg */ -if (rt->dst.dev->features &NETIF_F_SG && - csummode ==CHECKSUM_PARTIAL ) { - paged = true; - zc = true; - }else { - uarg->zerocopy =0 ; -skb_zcopy_set (skb, uarg, &extra_uref); +if ((flags &MSG_ZEROCOPY ) && length) { +struct msghdr *msg = from; + +if (getfrag == ip_generic_getfrag && msg->msg_ubuf) { +if (skb_zcopy (skb) && msg->msg_ubuf !=skb_zcopy (skb)) +return -EINVAL ; + +/* Leave uarg NULL if can't zerocopy, callers should +* be able to handle it. +*/ +if ((rt->dst.dev->features &NETIF_F_SG ) && + csummode ==CHECKSUM_PARTIAL ) { + paged = true; + zc = true; + uarg = msg->msg_ubuf; + } + }else if (sock_flag (sk,SOCK_ZEROCOPY )) { + uarg =msg_zerocopy_realloc (sk, length,skb_zcopy (skb)); +if (!uarg) +return -ENOBUFS ; + extra_uref = !skb_zcopy (skb);/* only ref on new uarg */ +if (rt->dst.dev->features &NETIF_F_SG && + csummode ==CHECKSUM_PARTIAL ) { + paged = true; + zc = true; + }else { + uarg->zerocopy =0 ; +skb_zcopy_set (skb, uarg, &extra_uref); + } } }
@@ -1747,13 +1764,14 @@ static int __ip6_append_data(struct sock *sk, err = -EFAULT ; goto error; } - }else if (!uarg || !uarg->zerocopy) { + }else if (!zc) {int i =skb_shinfo (skb)->nr_frags; err = -ENOMEM ;if (!sk_page_frag_refill (sk, pfrag)) goto error; +skb_zcopy_downgrade_managed (skb);if (!skb_can_coalesce (skb, i, pfrag->page, pfrag->offset)) { err = -EMSGSIZE ;
Signature · 1 line
2.36.1
PATCH v4 11/27 tcp: support externally provided ubufs
In reply to Pavel BegunkovTeach ipv4/udp how to use external ubuf_info provided in msghdr and also prepare it for managed frags by sprinkling skb_zcopy_downgrade_managed() when it could mix managed and not managed frags.
--- net/ipv4/tcp.c | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-)
net/ipv4/tcp.c +20 -12
@@ -1223,17 +1223,23 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) flags = msg->msg_flags; -if (flags &MSG_ZEROCOPY && size &&sock_flag (sk,SOCK_ZEROCOPY )) { +if ((flags &MSG_ZEROCOPY ) && size) { skb =tcp_write_queue_tail (sk); - uarg =msg_zerocopy_realloc (sk, size,skb_zcopy (skb)); -if (!uarg) { - err = -ENOBUFS ; - goto out_err; - } - zc = sk->sk_route_caps &NETIF_F_SG ; -if (!zc) - uarg->zerocopy =0 ; +if (msg->msg_ubuf) { + uarg = msg->msg_ubuf; +net_zcopy_get (uarg); + zc = sk->sk_route_caps &NETIF_F_SG ; + }else if (sock_flag (sk,SOCK_ZEROCOPY )) { + uarg =msg_zerocopy_realloc (sk, size,skb_zcopy (skb)); +if (!uarg) { + err = -ENOBUFS ; + goto out_err; + } + zc = sk->sk_route_caps &NETIF_F_SG ; +if (!zc) + uarg->zerocopy =0 ; + } } if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect) &&
@@ -1356,9 +1362,11 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) copy =min_t (int, copy, pfrag->size - pfrag->offset); -if (tcp_downgrade_zcopy_pure (sk, skb)) - goto wait_for_space; - +if (unlikely (skb_zcopy_pure (skb) ||skb_zcopy_managed (skb))) { +if (tcp_downgrade_zcopy_pure (sk, skb)) + goto wait_for_space; +skb_zcopy_downgrade_managed (skb); + } copy =tcp_wmem_schedule (sk, copy);if (!copy) goto wait_for_space;
Signature · 1 line
2.36.1
David Ahern
maintainerRe: [PATCH net-next v4 11/27] tcp: support externally provided ubufs
Quoted 1 line · Pavel Begunkov
Teach ipv4/udp how to use external
ubuf_infoprovided in msghdr and
ipv4/tcp?
Quoted 51 lines
also prepare it for managed frags by sprinkling
skb_zcopy_downgrade_managed()when it could mix managed and not managed frags.Signed-off-by Pavel Begunkov <[email protected]>--- net/ipv4/tcp.c | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-)net/ipv4/tcp.c +15
-10@@ -1223,17 +1223,23 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) flags = msg->msg_flags; -if (flags &MSG_ZEROCOPY && size &&sock_flag (sk,SOCK_ZEROCOPY )) { +if ((flags &MSG_ZEROCOPY ) && size) { skb =tcp_write_queue_tail (sk); - uarg =msg_zerocopy_realloc (sk, size,skb_zcopy (skb)); -if (!uarg) { - err = -ENOBUFS ; - goto out_err; - } - zc = sk->sk_route_caps &NETIF_F_SG ; -if (!zc) - uarg->zerocopy =0 ; +if (msg->msg_ubuf) { + uarg = msg->msg_ubuf; +net_zcopy_get (uarg); + zc = sk->sk_route_caps &NETIF_F_SG ; + }else if (sock_flag (sk,SOCK_ZEROCOPY )) { + uarg =msg_zerocopy_realloc (sk, size,skb_zcopy (skb)); +if (!uarg) { + err = -ENOBUFS ; + goto out_err; + } + zc = sk->sk_route_caps &NETIF_F_SG ; +if (!zc) + uarg->zerocopy =0 ; + } } if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect) &&@@ -1356,9 +1362,11 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) copy =min_t (int, copy, pfrag->size - pfrag->offset); -if (tcp_downgrade_zcopy_pure (sk, skb))
net/ipv4/tcp.c int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) · line 1372
Quoted 10 lines
net/ipv4/tcp.c +5
-2- goto wait_for_space; - +if (unlikely (skb_zcopy_pure (skb) ||skb_zcopy_managed (skb))) { +if (tcp_downgrade_zcopy_pure (sk, skb)) + goto wait_for_space; +skb_zcopy_downgrade_managed (skb); + } copy =tcp_wmem_schedule (sk, copy);if (!copy) goto wait_for_space;
You dropped the msg->msg_ubuf checks on jump labels. Removing the one you had at 'out_nopush' I agree with based on my tests (i.e, it is not needed).
The one at 'out_err' seems like it is needed - but it has been a few weeks since I debugged that case. I believe the error path I was hitting was sk_stream_wait_memory with MSG_DONTWAIT flag set meaning timeout is 0 and it jumps there with EPIPE.
Re: [PATCH net-next v4 11/27] tcp: support externally provided ubufs
Quoted 4 lines · David Ahern
Quoted 1 line · Pavel Begunkov
Teach ipv4/udp how to use external
ubuf_infoprovided in msghdr andipv4/tcp?
Ehh, just tcp. Thanks, I updated the branches
Quoted 50 lines
Quoted 50 lines
also prepare it for managed frags by sprinkling
skb_zcopy_downgrade_managed()when it could mix managed and not managed frags.Signed-off-by Pavel Begunkov <[email protected]>--- net/ipv4/tcp.c | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-)net/ipv4/tcp.c +15
-9@@ -1223,17 +1223,23 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) flags = msg->msg_flags; -if (flags &MSG_ZEROCOPY && size &&sock_flag (sk,SOCK_ZEROCOPY )) { +if ((flags &MSG_ZEROCOPY ) && size) { skb =tcp_write_queue_tail (sk); - uarg =msg_zerocopy_realloc (sk, size,skb_zcopy (skb)); -if (!uarg) { - err = -ENOBUFS ; - goto out_err; - } - zc = sk->sk_route_caps &NETIF_F_SG ; -if (!zc) - uarg->zerocopy =0 ; +if (msg->msg_ubuf) { + uarg = msg->msg_ubuf; +net_zcopy_get (uarg); + zc = sk->sk_route_caps &NETIF_F_SG ; + }else if (sock_flag (sk,SOCK_ZEROCOPY )) { + uarg =msg_zerocopy_realloc (sk, size,skb_zcopy (skb)); +if (!uarg) { + err = -ENOBUFS ; + goto out_err; + } + zc = sk->sk_route_caps &NETIF_F_SG ; +if (!zc) + uarg->zerocopy =0 ; + } } if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect) &&@@ -1356,9 +1362,11 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) copy =min_t (int, copy, pfrag->size - pfrag->offset);
Quoted 15 lines
net/ipv4/tcp.c int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) · line 1372
Quoted 11 lines
net/ipv4/tcp.c +5
-3-if (tcp_downgrade_zcopy_pure (sk, skb)) - goto wait_for_space; - +if (unlikely (skb_zcopy_pure (skb) ||skb_zcopy_managed (skb))) { +if (tcp_downgrade_zcopy_pure (sk, skb)) + goto wait_for_space; +skb_zcopy_downgrade_managed (skb); + } copy =tcp_wmem_schedule (sk, copy);if (!copy) goto wait_for_space;You dropped the
msg->msg_ubufchecks on jump labels. Removing the one you had at 'out_nopush' I agree with based on my tests (i.e, it is not needed).
It was an optimisation, which I dropped for simplicity. Will be sending it and couple more afterwards.
Quoted 4 lines
The one at '
out_err' seems like it is needed - but it has been a few weeks since I debugged that case. I believe the error path I was hitting wassk_stream_wait_memorywithMSG_DONTWAITflag set meaning timeout is 0 and it jumps there with EPIPE.
Currently, it's consistent with MSG_ZEROCOPY ubuf_info, we grab a ubuf_info reference at the beginning (msg_zerocopy_realloc() for MSG_ZEROCOPY and net_zcopy_get() for msg_ubuf), and then release it at the end with net_zcopy_put() or net_zcopy_put_abort().
All users, e.g. skb_zerocopy_iter_stream(), have to grab a new reference, skb_zcopy_set() -> net_zcopy_get().
Not sure I see any issue, and if there is it sounds that it should also be affecting MSG_ZEROCOPY.
Signature · 1 line
Pavel Begunkov
David Ahern
maintainerRe: [PATCH net-next v4 11/27] tcp: support externally provided ubufs
Quoted 29 lines · Pavel Begunkov
Quoted 11 lines
Quoted 11 lines
net/ipv4/tcp.c +4
-2@@ -1356,9 +1362,11 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) copy = min_t(int, copy, pfrag->size - pfrag->offset); - if (tcp_downgrade_zcopy_pure(sk, skb)) - goto wait_for_space; - + if (unlikely(skb_zcopy_pure(skb) ||skb_zcopy_managed (skb))) { + if (tcp_downgrade_zcopy_pure(sk, skb)) + goto wait_for_space; + skb_zcopy_downgrade_managed(skb);Quoted 8 lines
Quoted 4 lines
net/ipv4/tcp.c +1
-0+ } copy = tcp_wmem_schedule(sk, copy); if (!copy)goto
wait_for_space;You dropped the
msg->msg_ubufchecks on jump labels. Removing the one you had at 'out_nopush' I agree with based on my tests (i.e, it is not needed).It was an optimisation, which I dropped for simplicity. Will be sending it and couple more afterwards.
Quoted 4 lines
The one at '
out_err' seems like it is needed - but it has been a few weeks since I debugged that case. I believe the error path I was hitting wassk_stream_wait_memorywithMSG_DONTWAITflag set meaning timeout is 0 and it jumps there with EPIPE.
Quoted 5 lines
Currently, it's consistent with
MSG_ZEROCOPYubuf_info, we grab aubuf_inforeference at the beginning (msg_zerocopy_realloc()forMSG_ZEROCOPYandnet_zcopy_get()formsg_ubuf), and then release it at the end withnet_zcopy_put()ornet_zcopy_put_abort().
my fault; I somehow dropped a line in the port to the 5.13 kernel.
PATCH v4 12/27 io_uring: initialise msghdr::msg_ubuf
In reply to Pavel BegunkovInitialise newly added ->msg_ubuf in io_recv() and io_send().
--- io_uring/net.c | 2 ++ 1 file changed, 2 insertions(+)
io_uring/net.c +2 -0
@@ -255,6 +255,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) msg.msg_control =NULL ; msg.msg_controllen =0 ; msg.msg_namelen =0 ; + msg.msg_ubuf =NULL ; flags = sr->msg_flags;if (issue_flags &IO_URING_F_NONBLOCK )
@@ -601,6 +602,7 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) msg.msg_flags =0 ; msg.msg_controllen =0 ; msg.msg_iocb =NULL ; + msg.msg_ubuf =NULL ; flags = sr->msg_flags;if (force_nonblock)
Signature · 1 line
2.36.1
PATCH v4 13/27 io_uring: export io_put_task()
In reply to Pavel BegunkovMake io_put_task() available to non-core parts of io_uring, we'll need it for notification infrastructure.
--- include/linux/io_uring_types.h | 25 +++++++++++++++++++++++++ io_uring/io_uring.c | 11 +---------- io_uring/io_uring.h | 10 ++++++++++ io_uring/tctx.h | 26 -------------------------- 4 files changed, 36 insertions(+), 36 deletions(-)
include/linux/io_uring_types.h +25 -0
@@ -4,6 +4,7 @@#include <linux/blkdev.h> #include <linux/task_work.h> #include <linux/bitmap.h> +#include <linux/llist.h> #include <uapi/linux/io_uring.h> struct io_wq_work_node {
@@ -43,6 +44,30 @@ struct io_hash_table {unsigned hash_bits; }; +/* +* Arbitrary limit, can be raised if need be +*/ +#define IO_RINGFD_REG_MAX 16 + +struct io_uring_task { +/* submission side */ +int cached_refs; +const struct io_ring_ctx *last; +struct io_wq *io_wq; +struct file *registered_rings[IO_RINGFD_REG_MAX ]; + +struct xarray xa; +struct wait_queue_head wait; +atomic_t in_idle; +atomic_t inflight_tracked; +struct percpu_counter inflight; + +struct {/* task_work */ +struct llist_head task_list; +struct callback_head task_work; + }____cacheline_aligned_in_smp ; +}; +struct io_uring {u32 head____cacheline_aligned_in_smp ;u32 tail____cacheline_aligned_in_smp ;
io_uring/io_uring.c +1 -10
@@ -602,7 +602,7 @@ static bool io_cqring_overflow_flush(struct io_ring_ctx *ctx)return ret; } -static void __io_put_task (struct task_struct *task,int nr) +void __io_put_task (struct task_struct *task,int nr) {struct io_uring_task *tctx = task->io_uring;
@@ -612,15 +612,6 @@ static void __io_put_task(struct task_struct *task, int nr)put_task_struct_many (task, nr); } -/* must to be called somewhat shortly after putting a request */ -static inline void io_put_task (struct task_struct *task,int nr) -{ -if (likely (task == current)) - task->io_uring->cached_refs += nr; -else -__io_put_task (task, nr); -} -static void io_task_refs_refill (struct io_uring_task *tctx) {unsigned int refill = -tctx->cached_refs +IO_TCTX_REFS_CACHE_NR ;
io_uring/io_uring.h +10 -0
@@ -66,6 +66,7 @@ void io_wq_submit_work(struct io_wq_work *work);void io_free_req (struct io_kiocb *req);void io_queue_next (struct io_kiocb *req); +void __io_put_task (struct task_struct *task,int nr);bool io_match_task_safe (struct io_kiocb *head,struct task_struct *task,bool cancel_all);
@@ -253,4 +254,13 @@ static inline void io_commit_cqring_flush(struct io_ring_ctx *ctx) __io_commit_cqring_flush(ctx); } +/* must to be called somewhat shortly after putting a request */ +static inline void io_put_task (struct task_struct *task,int nr) +{ +if (likely (task == current)) + task->io_uring->cached_refs += nr; +else +__io_put_task (task, nr); +} +#endif
io_uring/tctx.h +0 -26
@@ -1,31 +1,5 @@// SPDX-License-Identifier: GPL-2.0 -#include <linux/llist.h> - -/* -* Arbitrary limit, can be raised if need be -*/ -#define IO_RINGFD_REG_MAX 16 - -struct io_uring_task { -/* submission side */ -int cached_refs; -const struct io_ring_ctx *last; -struct io_wq *io_wq; -struct file *registered_rings[IO_RINGFD_REG_MAX ]; - -struct xarray xa; -struct wait_queue_head wait; -atomic_t in_idle; -atomic_t inflight_tracked; -struct percpu_counter inflight; - -struct {/* task_work */ -struct llist_head task_list; -struct callback_head task_work; - }____cacheline_aligned_in_smp ; -}; -struct io_tctx_node {struct list_head ctx_node;struct task_struct *task;
Signature · 1 line
2.36.1
PATCH v4 14/27 io_uring: add zc notification infrastructure
In reply to Pavel BegunkovAdd internal part of send zerocopy notifications. There are two main structures, the first one is struct io_notif, which carries inside struct ubuf_info and maps 1:1 to it. io_uring will be binding a number of zerocopy send requests to it and ask to complete (aka flush) it. When flushed and all attached requests and skbs complete, it'll generate one and only one CQE. There are intended to be passed into the network layer as struct msghdr::msg_ubuf.
The second concept is notification slots. The userspace will be able to register an array of slots and subsequently addressing them by the index in the array. Slots are independent of each other. Each slot can have only one notifier at a time (called active notifier) but many notifiers during the lifetime. When active, a notifier not going to post any completion but the userspace can attach requests to it by specifying the corresponding slot while issueing send zc requests. Eventually, the userspace will want to "flush" the notifier losing any way to attach new requests to it, however it can use the next atomatically added notifier of this slot or of any other slot.
When the network layer is done with all enqueued skbs attached to a notifier and doesn't need the specified in them user data, the flushed notifier will post a CQE.
--- include/linux/io_uring_types.h | 5 ++ io_uring/Makefile | 2 +- io_uring/io_uring.c | 8 ++- io_uring/io_uring.h | 2 + io_uring/notif.c | 102 +++++++++++++++++++++++++++++++++ io_uring/notif.h | 64 +++++++++++++++++++++ 6 files changed, 179 insertions(+), 4 deletions(-) create mode 100644 io_uring/notif.c create mode 100644 io_uring/notif.h
include/linux/io_uring_types.h +5 -0
@@ -34,6 +34,9 @@ struct io_file_table {unsigned int alloc_hint; }; +struct io_notif ; +struct io_notif_slot ; +struct io_hash_bucket {spinlock_t lock;struct hlist_head list;
@@ -232,6 +235,8 @@ struct io_ring_ctx {unsigned nr_user_files;unsigned nr_user_bufs;struct io_mapped_ubuf **user_bufs; +struct io_notif_slot *notif_slots; +unsigned nr_notif_slots;struct io_submit_state submit_state;
io_uring/Makefile +1 -1
@@ -7,5 +7,5 @@ obj-$(CONFIG_IO_URING) += io_uring.o xattr.o nop.o fs.o splice.o \ openclose.o uring_cmd.o epoll.o \ statx.o net.o msg_ring.o timeout.o \ sqpoll.o fdinfo.o tctx.o poll.o \ - cancel.o kbuf.o rsrc.o rw.o opdef.o + cancel.o kbuf.o rsrc.o rw.o opdef.o notif.o obj-$(CONFIG_IO_WQ) += io-wq.o
io_uring/io_uring.c +5 -3
@@ -89,6 +89,7 @@#include "kbuf.h" #include "rsrc.h" #include "cancel.h" +#include "notif.h" #include "timeout.h" #include "poll.h"
@@ -726,9 +727,8 @@ struct io_uring_cqe *__io_get_cqe(struct io_ring_ctx *ctx)return &rings->cqes[off]; } -static bool io_fill_cqe_aux (struct io_ring_ctx *ctx, -u64 user_data,s32 res,u32 cflags, -bool allow_overflow) +bool io_fill_cqe_aux (struct io_ring_ctx *ctx,u64 user_data,s32 res,u32 cflags, +bool allow_overflow) {struct io_uring_cqe *cqe;
@@ -2496,6 +2496,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) }#endif WARN_ON_ONCE (!list_empty (&ctx->ltimeout_list)); +WARN_ON_ONCE (ctx->notif_slots || ctx->nr_notif_slots);io_mem_free (ctx->rings);io_mem_free (ctx->sq_sqes);
@@ -2672,6 +2673,7 @@ static __cold void io_ring_ctx_wait_and_kill(struct io_ring_ctx *ctx)io_unregister_personality (ctx, index);if (ctx->rings)io_poll_remove_all (ctx,NULL , true); +io_notif_unregister (ctx);mutex_unlock (&ctx->uring_lock);/* failed during ring init, it couldn't have issued any requests */
io_uring/io_uring.h +2 -0
@@ -33,6 +33,8 @@ void io_req_complete_post(struct io_kiocb *req);void __io_req_complete_post (struct io_kiocb *req);bool io_post_aux_cqe (struct io_ring_ctx *ctx,u64 user_data,s32 res,u32 cflags,bool allow_overflow); +bool io_fill_cqe_aux (struct io_ring_ctx *ctx,u64 user_data,s32 res,u32 cflags, +bool allow_overflow);void __io_commit_cqring_flush (struct io_ring_ctx *ctx);struct page **io_pin_pages (unsigned long ubuf,unsigned long len,int *npages);
io_uring/notif.c new file +102 -0
103 lines not on this page · raw message
io_uring/notif.h new file +64 -0
64 lines not on this page · raw message
Signature · 1 line
2.36.1
PATCH v4 15/27 io_uring: cache struct io_notif
In reply to Pavel Begunkovkmalloc'ing struct io_notif is too expensive when done frequently, cache them as many other resources in io_uring. Keep two list, the first one is from where we're getting notifiers, it's protected by ->uring_lock. The second is protected by ->completion_lock, to which we queue released notifiers. Then we splice one list into another when needed.
--- include/linux/io_uring_types.h | 7 +++++ io_uring/io_uring.c | 3 ++ io_uring/notif.c | 57 +++++++++++++++++++++++++++++----- io_uring/notif.h | 5 +++ 4 files changed, 65 insertions(+), 7 deletions(-)
include/linux/io_uring_types.h +7 -0
@@ -244,6 +244,9 @@ struct io_ring_ctx {struct xarray io_bl_xa;struct list_head io_buffers_cache; +/* struct io_notif cache, protected by uring_lock */ +struct list_head notif_list; +struct io_hash_table cancel_table_locked;struct list_head cq_overflow_list;struct list_head apoll_cache;
@@ -255,6 +258,10 @@ struct io_ring_ctx {struct io_wq_work_list locked_free_list;unsigned int locked_free_nr; +/* struct io_notif cache protected by completion_lock */ +struct list_head notif_list_locked; +unsigned int notif_locked_nr; +const struct cred *sq_creds;/* cred used for __io_sq_thread() */ struct io_sq_data *sq_data;/* if using sq thread polling */
io_uring/io_uring.c +3 -0
@@ -318,6 +318,8 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)INIT_WQ_LIST (&ctx->locked_free_list);INIT_DELAYED_WORK (&ctx->fallback_work, io_fallback_req_func);INIT_WQ_LIST (&ctx->submit_state.compl_reqs); +INIT_LIST_HEAD (&ctx->notif_list); +INIT_LIST_HEAD (&ctx->notif_list_locked);return ctx; err:kfree (ctx->dummy_ubuf);
@@ -2498,6 +2500,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx)WARN_ON_ONCE (!list_empty (&ctx->ltimeout_list));WARN_ON_ONCE (ctx->notif_slots || ctx->nr_notif_slots); +io_notif_cache_purge (ctx);io_mem_free (ctx->rings);io_mem_free (ctx->sq_sqes);
io_uring/notif.c +50 -7
81 lines not on this page · raw message
io_uring/notif.h +5 -0
@@ -5,6 +5,8 @@#include <net/sock.h> #include <linux/nospec.h> +#define IO_NOTIF_SPLICE_BATCH 32 +struct io_notif {struct ubuf_info uarg;struct io_ring_ctx *ctx;
@@ -13,6 +15,8 @@ struct io_notif {u64 tag;/* see struct io_notif_slot::seq */ u32 seq; +/* hook into ctx->notif_list and ctx->notif_list_locked */ +struct list_head cache_node;union {struct callback_head task_work;
@@ -41,6 +45,7 @@ struct io_notif_slot { };int io_notif_unregister (struct io_ring_ctx *ctx); +void io_notif_cache_purge (struct io_ring_ctx *ctx);struct io_notif *io_alloc_notif (struct io_ring_ctx *ctx,struct io_notif_slot *slot);
Signature · 1 line
2.36.1
PATCH v4 16/27 io_uring: complete notifiers in tw
In reply to Pavel BegunkovWe need a task context to post CQEs but using wq is too expensive. Try to complete notifiers using task_work and fall back to wq if fails.
--- io_uring/notif.c | 22 +++++++++++++++++++--- io_uring/notif.h | 3 +++ 2 files changed, 22 insertions(+), 3 deletions(-)
io_uring/notif.c +19 -3
@@ -13,6 +13,11 @@ static void __io_notif_complete_tw(struct callback_head *cb) struct io_notif *notif = container_of(cb, struct io_notif, task_work);struct io_ring_ctx *ctx = notif->ctx; +if (likely (notif->task)) { +io_put_task (notif->task,1 ); + notif->task =NULL ; + } +io_cq_lock (ctx);io_fill_cqe_aux (ctx, notif->tag,0 , notif->seq, true);
@@ -43,6 +48,14 @@ static void io_uring_tx_zerocopy_callback(struct sk_buff *skb,if (!refcount_dec_and_test (&uarg->refcnt))return ; + +if (likely (notif->task)) { +init_task_work (¬if->task_work, __io_notif_complete_tw); +if (likely (!task_work_add (notif->task, ¬if->task_work, +TWA_SIGNAL ))) +return ; + } +INIT_WORK (¬if->commit_work, io_notif_complete_wq);queue_work (system_unbound_wq, ¬if->commit_work); }
@@ -134,12 +147,15 @@ __cold int io_notif_unregister(struct io_ring_ctx *ctx)for (i =0 ; i < ctx->nr_notif_slots; i++) {struct io_notif_slot *slot = &ctx->notif_slots[i]; -if (slot->notif) -io_notif_slot_flush (slot); +if (!slot->notif) +continue ; +if (WARN_ON_ONCE (slot->notif->task)) + slot->notif->task =NULL ; +io_notif_slot_flush (slot); }kvfree (ctx->notif_slots); ctx->notif_slots =NULL ; ctx->nr_notif_slots =0 ;return 0 ; -} +}
io_uring/notif.h +3 -0
@@ -11,6 +11,9 @@ struct io_notif {struct ubuf_info uarg;struct io_ring_ctx *ctx; +/* complete via tw if ->task is non-NULL, fallback to wq otherwise */ +struct task_struct *task; +/* cqe->user_data, io_notif_slot::tag if not overridden */ u64 tag;/* see struct io_notif_slot::seq */
Signature · 1 line
2.36.1
PATCH v4 17/27 io_uring: add rsrc referencing for notifiers
In reply to Pavel BegunkovIn preparation to zerocopy sends with fixed buffers make notifiers to reference the rsrc node to protect the used fixed buffers. We can't just grab it for a send request as notifiers can likely outlive requests that used it.
--- io_uring/notif.c | 5 +++++ io_uring/notif.h | 1 + io_uring/rsrc.h | 12 +++++++++--- 3 files changed, 15 insertions(+), 3 deletions(-)
io_uring/notif.c +5 -0
@@ -7,10 +7,12 @@#include "io_uring.h" #include "notif.h" +#include "rsrc.h" static void __io_notif_complete_tw (struct callback_head *cb) { struct io_notif *notif = container_of(cb, struct io_notif, task_work); +struct io_rsrc_node *rsrc_node = notif->rsrc_node;struct io_ring_ctx *ctx = notif->ctx;if (likely (notif->task)) {
@@ -25,6 +27,7 @@ static void __io_notif_complete_tw(struct callback_head *cb) ctx->notif_locked_nr++;io_cq_unlock_post (ctx); +io_rsrc_put_node (rsrc_node,1 );percpu_ref_put (&ctx->refs); }
@@ -119,6 +122,8 @@ struct io_notif *io_alloc_notif(struct io_ring_ctx *ctx,/* master ref owned by io_notif_slot, will be dropped on flush */ refcount_set (¬if->uarg.refcnt,1 );percpu_ref_get (&ctx->refs); + notif->rsrc_node = ctx->rsrc_node; +io_charge_rsrc_node (ctx);return notif; }
io_uring/notif.h +1 -0
@@ -10,6 +10,7 @@struct io_notif {struct ubuf_info uarg;struct io_ring_ctx *ctx; +struct io_rsrc_node *rsrc_node;/* complete via tw if ->task is non-NULL, fallback to wq otherwise */ struct task_struct *task;
io_uring/rsrc.h +9 -3
@@ -135,6 +135,13 @@ static inline void io_req_put_rsrc_locked(struct io_kiocb *req, } } +static inline void io_charge_rsrc_node (struct io_ring_ctx *ctx) +{ + ctx->rsrc_cached_refs--; +if (unlikely (ctx->rsrc_cached_refs <0 )) +io_rsrc_refs_refill (ctx); +} +static inline void io_req_set_rsrc_node (struct io_kiocb *req,struct io_ring_ctx *ctx,unsigned int issue_flags)
@@ -144,9 +151,8 @@ static inline void io_req_set_rsrc_node(struct io_kiocb *req,if (!(issue_flags &IO_URING_F_UNLOCKED )) {lockdep_assert_held (&ctx->uring_lock); - ctx->rsrc_cached_refs--; -if (unlikely (ctx->rsrc_cached_refs <0 )) -io_rsrc_refs_refill (ctx); + +io_charge_rsrc_node (ctx); }else {percpu_ref_get (&req->rsrc_node->refs); }
Signature · 1 line
2.36.1
PATCH v4 18/27 io_uring: add notification slot registration
In reply to Pavel BegunkovLet the userspace to register and unregister notification slots.
--- include/uapi/linux/io_uring.h | 17 ++++++++++++++ io_uring/io_uring.c | 9 ++++++++ io_uring/notif.c | 43 +++++++++++++++++++++++++++++++++++ io_uring/notif.h | 3 +++ 4 files changed, 72 insertions(+)
include/uapi/linux/io_uring.h +17 -0
@@ -454,6 +454,10 @@ enum {/* register a range of fixed file slots for automatic slot allocation */ IORING_REGISTER_FILE_ALLOC_RANGE =25 , +/* zerocopy notification API */ +IORING_REGISTER_NOTIFIERS =26 , +IORING_UNREGISTER_NOTIFIERS =27 , +/* this goes last */ IORING_REGISTER_LAST };
@@ -500,6 +504,19 @@ struct io_uring_rsrc_update2 {__u32 resv2; }; +struct io_uring_notification_slot { +__u64 tag; +__u64 resv[3 ]; +}; + +struct io_uring_notification_register { +__u32 nr_slots; +__u32 resv; +__u64 resv2; +__u64 data; +__u64 resv3; +}; +/* Skip updating fd indexes set to this value in the fd table */ #define IORING_REGISTER_FILES_SKIP (-2)
io_uring/io_uring.c +9 -0
@@ -3875,6 +3875,15 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,break ; ret =io_register_file_alloc_range (ctx, arg);break ; +case IORING_REGISTER_NOTIFIERS : + ret =io_notif_register (ctx, arg, nr_args); +break ; +case IORING_UNREGISTER_NOTIFIERS : + ret = -EINVAL ; +if (arg || nr_args) +break ; + ret =io_notif_unregister (ctx); +break ;default : ret = -EINVAL ;break ;
io_uring/notif.c +43 -0
48 lines not on this page · raw message
io_uring/notif.h +3 -0
@@ -6,6 +6,7 @@#include <linux/nospec.h> #define IO_NOTIF_SPLICE_BATCH 32 +#define IORING_MAX_NOTIF_SLOTS (1U << 10)struct io_notif {struct ubuf_info uarg;
@@ -48,6 +49,8 @@ struct io_notif_slot {u32 seq; }; +int io_notif_register (struct io_ring_ctx *ctx, +void __user *arg,unsigned int size);int io_notif_unregister (struct io_ring_ctx *ctx);void io_notif_cache_purge (struct io_ring_ctx *ctx);
Signature · 1 line
2.36.1
PATCH v4 19/27 io_uring: wire send zc request type
In reply to Pavel BegunkovAdd a new io_uring opcode IORING_OP_SENDZC. The main distinction from IORING_OP_SEND is that the user should specify a notification slot index in sqe::notification_idx and the buffers are safe to reuse only when the used notification is flushed and completes.
--- include/uapi/linux/io_uring.h | 5 ++ io_uring/net.c | 94 +++++++++++++++++++++++++++++++++++ io_uring/net.h | 4 ++ io_uring/opdef.c | 15 ++++++ 4 files changed, 118 insertions(+)
include/uapi/linux/io_uring.h +5 -0
@@ -63,6 +63,10 @@ struct io_uring_sqe {union {__s32 splice_fd_in;__u32 file_index; +struct { +__u16 notification_idx; +__u16 __pad; + }; }; union { struct {
@@ -194,6 +198,7 @@ enum io_uring_op {IORING_OP_GETXATTR ,IORING_OP_SOCKET ,IORING_OP_URING_CMD , +IORING_OP_SENDZC ,/* this goes last, obviously */ IORING_OP_LAST,
io_uring/net.c +94 -0
112 lines not on this page · raw message
io_uring/net.h +4 -0
@@ -40,4 +40,8 @@ int io_socket(struct io_kiocb *req, unsigned int issue_flags);int io_connect_prep_async (struct io_kiocb *req);int io_connect_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe);int io_connect (struct io_kiocb *req,unsigned int issue_flags); + +int io_sendzc (struct io_kiocb *req,unsigned int issue_flags); +int io_sendzc_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe); +#endif
io_uring/opdef.c +15 -0
@@ -470,6 +470,21 @@ const struct io_op_def io_op_defs[] = { .issue = io_uring_cmd, .prep_async = io_uring_cmd_prep_async, }, + [IORING_OP_SENDZC] = { + .name = "SENDZC", + .needs_file = 1, + .unbound_nonreg_file = 1, + .pollout = 1, + .audit_skip = 1, + .ioprio = 1, +#if defined(CONFIG_NET ) + .prep = io_sendzc_prep, + .issue = io_sendzc, +#else + .prep = io_eopnotsupp_prep, +#endif + + }, };const char *io_uring_get_opcode (u8 opcode)
Signature · 1 line
2.36.1
PATCH v4 20/27 io_uring: account locked pages for non-fixed zc
In reply to Pavel BegunkovFixed buffers are RLIMIT_MEMLOCK accounted, however it doesn't cover iovec based zerocopy sends. Do the accounting on the io_uring side.
--- io_uring/net.c | 1 + io_uring/notif.c | 6 ++++++ 2 files changed, 7 insertions(+)
io_uring/net.c +1 -0
@@ -724,6 +724,7 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) ret =import_single_range (WRITE , zc->buf, zc->len, &iov, &msg.msg_iter);if (unlikely (ret))return ret; +mm_account_pinned_pages (¬if->uarg.mmp, zc->len); msg_flags = zc->msg_flags |MSG_ZEROCOPY ;if (issue_flags &IO_URING_F_NONBLOCK )
io_uring/notif.c +6 -0
@@ -14,7 +14,13 @@ static void __io_notif_complete_tw(struct callback_head *cb) struct io_notif *notif = container_of(cb, struct io_notif, task_work);struct io_rsrc_node *rsrc_node = notif->rsrc_node;struct io_ring_ctx *ctx = notif->ctx; +struct mmpin *mmp = ¬if->uarg.mmp; +if (mmp->user) { +atomic_long_sub (mmp->num_pg, &mmp->user->locked_vm); +free_uid (mmp->user); + mmp->user =NULL ; + }if (likely (notif->task)) {io_put_task (notif->task,1 ); notif->task =NULL ;
Signature · 1 line
2.36.1
PATCH v4 21/27 io_uring: allow to pass addr into sendzc
In reply to Pavel BegunkovAllow to specify an address to zerocopy sends making it more like sendto(2).
--- include/uapi/linux/io_uring.h | 2 +- io_uring/net.c | 18 ++++++++++++++++-- 2 files changed, 17 insertions(+), 3 deletions(-)
include/uapi/linux/io_uring.h +1 -1
@@ -65,7 +65,7 @@ struct io_uring_sqe {__u32 file_index;struct {__u16 notification_idx; -__u16 __pad; +__u16 addr_len; }; };union {
io_uring/net.c +16 -2
@@ -66,6 +66,8 @@ struct io_sendzc {u16 slot_idx;unsigned msg_flags;unsigned flags; +unsigned addr_len; +void __user *addr; };#define IO_APOLL_MULTI_POLLED (REQ_F_APOLL_MULTISHOT | REQ_F_POLLED)
@@ -666,8 +668,7 @@ int io_sendzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) {struct io_sendzc *zc =io_kiocb_to_cmd (req); -if (READ_ONCE (sqe->addr2) ||READ_ONCE (sqe->__pad2[0 ]) || -READ_ONCE (sqe->addr3)) +if (READ_ONCE (sqe->__pad2[0 ]) ||READ_ONCE (sqe->addr3))return -EINVAL ; zc->flags =READ_ONCE (sqe->ioprio);
@@ -680,6 +681,10 @@ int io_sendzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) zc->slot_idx =READ_ONCE (sqe->notification_idx);if (zc->msg_flags &MSG_DONTWAIT ) req->flags |=REQ_F_NOWAIT ; + + zc->addr =u64_to_user_ptr (READ_ONCE (sqe->addr2)); + zc->addr_len =READ_ONCE (sqe->addr_len); +#ifdef CONFIG_COMPAT if (req->ctx->compat) zc->msg_flags |=MSG_CMSG_COMPAT ;
@@ -689,6 +694,7 @@ int io_sendzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)int io_sendzc (struct io_kiocb *req,unsigned int issue_flags) { +struct sockaddr_storage address;struct io_ring_ctx *ctx = req->ctx;struct io_sendzc *zc =io_kiocb_to_cmd (req);struct io_notif_slot *notif_slot;
@@ -726,6 +732,14 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags)return ret;mm_account_pinned_pages (¬if->uarg.mmp, zc->len); +if (zc->addr) { + ret =move_addr_to_kernel (zc->addr, zc->addr_len, &address); +if (unlikely (ret <0 )) +return ret; + msg.msg_name = (struct sockaddr *)&address; + msg.msg_namelen = zc->addr_len; + } + msg_flags = zc->msg_flags |MSG_ZEROCOPY ;if (issue_flags &IO_URING_F_NONBLOCK ) msg_flags |=MSG_DONTWAIT ;
Signature · 1 line
2.36.1
PATCH v4 22/27 io_uring: sendzc with fixed buffers
In reply to Pavel BegunkovAllow zerocopy sends to use fixed buffers. There is an optimisation for this case, the network layer don't need to reference the pages, see SKBFL_MANAGED_FRAG_REFS, so io_uring have to ensure validity of fixed buffers until the notifier is released.
--- include/uapi/linux/io_uring.h | 6 +++++- io_uring/net.c | 29 ++++++++++++++++++++++++----- 2 files changed, 29 insertions(+), 6 deletions(-)
include/uapi/linux/io_uring.h +5 -1
@@ -269,9 +269,13 @@ enum io_uring_op {* IORING_RECV_MULTISHOT Multishot recv. Sets IORING_CQE_F_MORE if * the handler will continue to report * CQEs on behalf of the same SQE. +* +* IORING_RECVSEND_FIXED_BUF Use registered buffers, the index is stored in +* the buf_index field. */ #define IORING_RECVSEND_POLL_FIRST (1U << 0) -#define IORING_RECV_MULTISHOT (1U << 1) +#define IORING_RECV_MULTISHOT (1U << 1) +#define IORING_RECVSEND_FIXED_BUF (1U << 2)/* * accept flags stored in sqe->ioprio
io_uring/net.c +24 -5
@@ -14,6 +14,7 @@#include "kbuf.h" #include "net.h" #include "notif.h" +#include "rsrc.h" #if defined(CONFIG_NET ) struct io_shutdown {
@@ -667,13 +668,23 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags)int io_sendzc_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) {struct io_sendzc *zc =io_kiocb_to_cmd (req); +struct io_ring_ctx *ctx = req->ctx;if (READ_ONCE (sqe->__pad2[0 ]) ||READ_ONCE (sqe->addr3))return -EINVAL ; zc->flags =READ_ONCE (sqe->ioprio); -if (zc->flags & ~IORING_RECVSEND_POLL_FIRST ) +if (zc->flags & ~(IORING_RECVSEND_POLL_FIRST |IORING_RECVSEND_FIXED_BUF ))return -EINVAL ; +if (zc->flags &IORING_RECVSEND_FIXED_BUF ) { +unsigned idx =READ_ONCE (sqe->buf_index); + +if (unlikely (idx >= ctx->nr_user_bufs)) +return -EFAULT ; + idx =array_index_nospec (idx, ctx->nr_user_bufs); + req->imu =READ_ONCE (ctx->user_bufs[idx]); +io_req_set_rsrc_node (req, ctx,0 ); + } zc->buf =u64_to_user_ptr (READ_ONCE (sqe->addr)); zc->len =READ_ONCE (sqe->len);
@@ -727,10 +738,18 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) msg.msg_controllen =0 ; msg.msg_namelen =0 ; - ret =import_single_range (WRITE , zc->buf, zc->len, &iov, &msg.msg_iter); -if (unlikely (ret)) -return ret; -mm_account_pinned_pages (¬if->uarg.mmp, zc->len); +if (zc->flags &IORING_RECVSEND_FIXED_BUF ) { + ret =io_import_fixed (WRITE , &msg.msg_iter, req->imu, + (u64 )zc->buf, zc->len); +if (unlikely (ret)) +return ret; + }else { + ret =import_single_range (WRITE , zc->buf, zc->len, &iov, + &msg.msg_iter); +if (unlikely (ret)) +return ret; +mm_account_pinned_pages (¬if->uarg.mmp, zc->len); + }if (zc->addr) { ret =move_addr_to_kernel (zc->addr, zc->addr_len, &address);
Signature · 1 line
2.36.1
PATCH v4 23/27 io_uring: flush notifiers after sendzc
In reply to Pavel BegunkovAllow to flush notifiers as a part of sendzc request by setting IORING_SENDZC_FLUSH flag. When the sendzc request succeedes it will flush the used [active] notifier.
--- include/uapi/linux/io_uring.h | 4 ++++ io_uring/io_uring.c | 11 +---------- io_uring/io_uring.h | 10 ++++++++++ io_uring/net.c | 5 ++++- io_uring/notif.c | 2 +- io_uring/notif.h | 11 +++++++++++ 6 files changed, 31 insertions(+), 12 deletions(-)
include/uapi/linux/io_uring.h +4 -0
@@ -272,10 +272,14 @@ enum io_uring_op {* * IORING_RECVSEND_FIXED_BUF Use registered buffers, the index is stored in * the buf_index field. +* +* IORING_RECVSEND_NOTIF_FLUSH Flush a notification after a successful +* successful. Only for zerocopy sends. */ #define IORING_RECVSEND_POLL_FIRST (1U << 0)#define IORING_RECV_MULTISHOT (1U << 1)#define IORING_RECVSEND_FIXED_BUF (1U << 2) +#define IORING_RECVSEND_NOTIF_FLUSH (1U << 3)/* * accept flags stored in sqe->ioprio
io_uring/io_uring.c +1 -10
@@ -615,7 +615,7 @@ void __io_put_task(struct task_struct *task, int nr)put_task_struct_many (task, nr); } -static void io_task_refs_refill (struct io_uring_task *tctx) +void io_task_refs_refill (struct io_uring_task *tctx) {unsigned int refill = -tctx->cached_refs +IO_TCTX_REFS_CACHE_NR ;
@@ -624,15 +624,6 @@ static void io_task_refs_refill(struct io_uring_task *tctx) tctx->cached_refs += refill; } -static inline void io_get_task_refs (int nr) -{ -struct io_uring_task *tctx = current->io_uring; - - tctx->cached_refs -= nr; -if (unlikely (tctx->cached_refs <0 )) -io_task_refs_refill (tctx); -} -static __cold void io_uring_drop_tctx_refs (struct task_struct *task) {struct io_uring_task *tctx = task->io_uring;
io_uring/io_uring.h +10 -0
@@ -69,6 +69,7 @@ void io_wq_submit_work(struct io_wq_work *work);void io_free_req (struct io_kiocb *req);void io_queue_next (struct io_kiocb *req);void __io_put_task (struct task_struct *task,int nr); +void io_task_refs_refill (struct io_uring_task *tctx);bool io_match_task_safe (struct io_kiocb *head,struct task_struct *task,bool cancel_all);
@@ -265,4 +266,13 @@ static inline void io_put_task(struct task_struct *task, int nr) __io_put_task(task, nr); } +static inline void io_get_task_refs (int nr) +{ +struct io_uring_task *tctx = current->io_uring; + + tctx->cached_refs -= nr; +if (unlikely (tctx->cached_refs <0 )) +io_task_refs_refill (tctx); +} +#endif
io_uring/net.c +4 -1
@@ -674,7 +674,8 @@ int io_sendzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)return -EINVAL ; zc->flags =READ_ONCE (sqe->ioprio); -if (zc->flags & ~(IORING_RECVSEND_POLL_FIRST |IORING_RECVSEND_FIXED_BUF )) +if (zc->flags & ~(IORING_RECVSEND_POLL_FIRST | +IORING_RECVSEND_FIXED_BUF |IORING_RECVSEND_NOTIF_FLUSH ))return -EINVAL ;if (zc->flags &IORING_RECVSEND_FIXED_BUF ) {unsigned idx =READ_ONCE (sqe->buf_index);
@@ -776,6 +777,8 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags)return ret == -ERESTARTSYS ? -EINTR : ret; } +if (zc->flags &IORING_RECVSEND_NOTIF_FLUSH ) +io_notif_slot_flush_submit (notif_slot,0 );io_req_set_res (req, ret,0 );return IOU_OK ; }
io_uring/notif.c +1 -1
@@ -133,7 +133,7 @@ struct io_notif *io_alloc_notif(struct io_ring_ctx *ctx,return notif; } -static void io_notif_slot_flush (struct io_notif_slot *slot) +void io_notif_slot_flush (struct io_notif_slot *slot) __must_hold(&ctx->uring_lock) {struct io_notif *notif = slot->notif;
io_uring/notif.h +11 -0
@@ -54,6 +54,7 @@ int io_notif_register(struct io_ring_ctx *ctx,int io_notif_unregister (struct io_ring_ctx *ctx);void io_notif_cache_purge (struct io_ring_ctx *ctx); +void io_notif_slot_flush (struct io_notif_slot *slot);struct io_notif *io_alloc_notif (struct io_ring_ctx *ctx,struct io_notif_slot *slot);
@@ -74,3 +75,13 @@ static inline struct io_notif_slot *io_get_notif_slot(struct io_ring_ctx *ctx, idx =array_index_nospec (idx, ctx->nr_notif_slots);return &ctx->notif_slots[idx]; } + +static inline void io_notif_slot_flush_submit (struct io_notif_slot *slot, +unsigned int issue_flags) +{ +if (!(issue_flags &IO_URING_F_UNLOCKED )) { + slot->notif->task = current; +io_get_task_refs (1 ); + } +io_notif_slot_flush (slot); +}
Signature · 1 line
2.36.1
PATCH v4 24/27 io_uring: rename IORING_OP_FILES_UPDATE
In reply to Pavel BegunkovIORING_OP_FILES_UPDATE will be a more generic opcode serving different resource types, rename it into IORING_OP_RSRC_UPDATE and add subtype handling.
--- include/uapi/linux/io_uring.h | 12 +++++++++++- io_uring/opdef.c | 9 +++++---- io_uring/rsrc.c | 17 +++++++++++++++-- io_uring/rsrc.h | 4 ++-- 4 files changed, 33 insertions(+), 9 deletions(-)
include/uapi/linux/io_uring.h +11 -1
@@ -171,7 +171,8 @@ enum io_uring_op {IORING_OP_FALLOCATE ,IORING_OP_OPENAT ,IORING_OP_CLOSE , -IORING_OP_FILES_UPDATE , +IORING_OP_RSRC_UPDATE , +IORING_OP_FILES_UPDATE =IORING_OP_RSRC_UPDATE ,IORING_OP_STATX ,IORING_OP_READ , IORING_OP_WRITE,
@@ -220,6 +221,7 @@ enum io_uring_op {#define IORING_TIMEOUT_ETIME_SUCCESS (1U << 5)#define IORING_TIMEOUT_CLOCK_MASK (IORING_TIMEOUT_BOOTTIME | IORING_TIMEOUT_REALTIME)#define IORING_TIMEOUT_UPDATE_MASK (IORING_TIMEOUT_UPDATE | IORING_LINK_TIMEOUT_UPDATE) +/* * sqe->splice_flags * extends splice(2) flags
@@ -286,6 +288,14 @@ enum io_uring_op {*/ #define IORING_ACCEPT_MULTISHOT (1U << 0) + +/* +* IORING_OP_RSRC_UPDATE flags +*/ +enum { +IORING_RSRC_UPDATE_FILES , +}; +/* * IORING_OP_MSG_RING command types, stored in sqe->addr */
io_uring/opdef.c +5 -4
@@ -246,12 +246,13 @@ const struct io_op_def io_op_defs[] = { .prep = io_close_prep, .issue = io_close, }, - [IORING_OP_FILES_UPDATE] = { + [IORING_OP_RSRC_UPDATE] = { .audit_skip = 1, .iopoll = 1, - .name = "FILES_UPDATE", - .prep = io_files_update_prep, - .issue = io_files_update, + .name = "RSRC_UPDATE", + .prep = io_rsrc_update_prep, + .issue = io_rsrc_update, + .ioprio = 1, }, [IORING_OP_STATX] = { .audit_skip = 1,
io_uring/rsrc.c +15 -2
@@ -21,6 +21,7 @@ struct io_rsrc_update {u64 arg;u32 nr_args;u32 offset; +int type; }; static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov,
@@ -658,7 +659,7 @@ __cold int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg,return -EINVAL ; } -int io_files_update_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) +int io_rsrc_update_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) {struct io_rsrc_update *up =io_kiocb_to_cmd (req);
@@ -672,6 +673,7 @@ int io_files_update_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)if (!up->nr_args)return -EINVAL ; up->arg =READ_ONCE (sqe->addr); + up->type =READ_ONCE (sqe->ioprio);return 0 ; }
@@ -711,7 +713,7 @@ static int io_files_update_with_index_alloc(struct io_kiocb *req,return ret; } -int io_files_update (struct io_kiocb *req,unsigned int issue_flags) +static int io_files_update (struct io_kiocb *req,unsigned int issue_flags) {struct io_rsrc_update *up =io_kiocb_to_cmd (req);struct io_ring_ctx *ctx = req->ctx;
@@ -740,6 +742,17 @@ int io_files_update(struct io_kiocb *req, unsigned int issue_flags)return IOU_OK ; } +int io_rsrc_update (struct io_kiocb *req,unsigned int issue_flags) +{ +struct io_rsrc_update *up =io_kiocb_to_cmd (req); + +switch (up->type) { +case IORING_RSRC_UPDATE_FILES : +return io_files_update (req, issue_flags); + } +return -EINVAL ; +} +int io_queue_rsrc_removal (struct io_rsrc_data *data,unsigned idx,struct io_rsrc_node *node,void *rsrc) {
io_uring/rsrc.h +2 -2
@@ -167,6 +167,6 @@ static inline u64 *io_get_tag_slot(struct io_rsrc_data *data, unsigned int idx) return &data->tags[table_idx][off]; } -int io_files_update (struct io_kiocb *req,unsigned int issue_flags); -int io_files_update_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe); +int io_rsrc_update (struct io_kiocb *req,unsigned int issue_flags); +int io_rsrc_update_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe);#endif
Signature · 1 line
2.36.1
PATCH v4 25/27 io_uring: add zc notification flush requests
In reply to Pavel BegunkovOverlay notification control onto IORING_OP_RSRC_UPDATE (former IORING_OP_FILES_UPDATE). It allows to flush a range of zc notifications from slots with indexes [sqe->off, sqe->off+sqe->len). If sqe->arg is not zero, it also copies sqe->arg as a new tag for all flushed notifications.
Note, it doesn't flush a notification of a slot if there was no requests attached to it (since last flush or registration).
--- include/uapi/linux/io_uring.h | 1 + io_uring/rsrc.c | 38 +++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)
include/uapi/linux/io_uring.h +1 -0
@@ -294,6 +294,7 @@ enum io_uring_op {*/ enum {IORING_RSRC_UPDATE_FILES , +IORING_RSRC_UPDATE_NOTIF , };/*
io_uring/rsrc.c +38 -0
@@ -15,6 +15,7 @@#include "io_uring.h" #include "openclose.h" #include "rsrc.h" +#include "notif.h" struct io_rsrc_update {struct file *file;
@@ -742,6 +743,41 @@ static int io_files_update(struct io_kiocb *req, unsigned int issue_flags)return IOU_OK ; } +static int io_notif_update (struct io_kiocb *req,unsigned int issue_flags) +{ +struct io_rsrc_update *up =io_kiocb_to_cmd (req); +struct io_ring_ctx *ctx = req->ctx; +unsigned len = up->nr_args; +unsigned idx_end, idx = up->offset; +int ret =0 ; + +io_ring_submit_lock (ctx, issue_flags); +if (unlikely (check_add_overflow (idx, len, &idx_end))) { + ret = -EOVERFLOW ; + goto out; + } +if (unlikely (idx_end > ctx->nr_notif_slots)) { + ret = -EINVAL ; + goto out; + } + +for (; idx < idx_end; idx++) { +struct io_notif_slot *slot = &ctx->notif_slots[idx]; + +if (!slot->notif) +continue ; +if (up->arg) + slot->tag = up->arg; +io_notif_slot_flush_submit (slot, issue_flags); + } +out: +io_ring_submit_unlock (ctx, issue_flags); +if (ret <0 ) +req_set_fail (req); +io_req_set_res (req, ret,0 ); +return IOU_OK ; +} +int io_rsrc_update (struct io_kiocb *req,unsigned int issue_flags) {struct io_rsrc_update *up =io_kiocb_to_cmd (req);
@@ -749,6 +785,8 @@ int io_rsrc_update(struct io_kiocb *req, unsigned int issue_flags)switch (up->type) {case IORING_RSRC_UPDATE_FILES :return io_files_update (req, issue_flags); +case IORING_RSRC_UPDATE_NOTIF : +return io_notif_update (req, issue_flags); }return -EINVAL ; }
Signature · 1 line
2.36.1
PATCH v4 26/27 io_uring: enable managed frags with register buffers
In reply to Pavel Begunkovio_uring's registered buffers infra has a good performant way of pinning pages, so let's use SKBFL_MANAGED_FRAG_REFS when our requests are purely register buffer backed.
--- io_uring/net.c | 56 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 55 insertions(+), 1 deletion(-)
io_uring/net.c +55 -1
@@ -704,6 +704,60 @@ int io_sendzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)return 0 ; } +static int io_sg_from_iter (struct sock *sk,struct sk_buff *skb, +struct iov_iter *from,size_t length) +{ +struct skb_shared_info *shinfo =skb_shinfo (skb); +int frag = shinfo->nr_frags; +int ret =0 ; +struct bvec_iter bi; +ssize_t copied =0 ; +unsigned long truesize =0 ; + +if (!shinfo->nr_frags) + shinfo->flags |=SKBFL_MANAGED_FRAG_REFS ; + +if (!skb_zcopy_managed (skb) || !iov_iter_is_bvec (from)) { +skb_zcopy_downgrade_managed (skb); +return __zerocopy_sg_from_iter (NULL , sk, skb, from, length); + } + + bi.bi_size =min (from->count, length); + bi.bi_bvec_done = from->iov_offset; + bi.bi_idx =0 ; + +while (bi.bi_size && frag <MAX_SKB_FRAGS ) { +struct bio_vec v =mp_bvec_iter_bvec (from->bvec, bi); + + copied += v.bv_len; + truesize +=PAGE_ALIGN (v.bv_len + v.bv_offset); +__skb_fill_page_desc_noacc (shinfo, frag++, v.bv_page, + v.bv_offset, v.bv_len); +bvec_iter_advance_single (from->bvec, &bi, v.bv_len); + } +if (bi.bi_size) + ret = -EMSGSIZE ; + + shinfo->nr_frags = frag; + from->bvec += bi.bi_idx; + from->nr_segs -= bi.bi_idx; + from->count = bi.bi_size; + from->iov_offset = bi.bi_bvec_done; + + skb->data_len += copied; + skb->len += copied; + skb->truesize += truesize; + +if (sk && sk->sk_type ==SOCK_STREAM ) { +sk_wmem_queued_add (sk, truesize); +if (!skb_zcopy_pure (skb)) +sk_mem_charge (sk, truesize); + }else { +refcount_add (truesize, &skb->sk->sk_wmem_alloc); + } +return ret; +} +int io_sendzc (struct io_kiocb *req,unsigned int issue_flags) {struct sockaddr_storage address;
@@ -768,7 +822,7 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) msg.msg_flags = msg_flags; msg.msg_ubuf = ¬if->uarg; - msg.sg_from_iter =NULL ; + msg.sg_from_iter = io_sg_from_iter; ret =sock_sendmsg (sock, &msg);if (unlikely (ret < min_ret)) {
Signature · 1 line
2.36.1
PATCH v4 27/27 selftests/io_uring: test zerocopy send
In reply to Pavel BegunkovAdd selftests for io_uring zerocopy sends and io_uring's notification infrastructure. It's largely influenced by msg_zerocopy and uses it on the receive side.
--- tools/testing/selftests/net/Makefile | 1 + .../selftests/net/io_uring_zerocopy_tx.c | 605 ++++++++++++++++++ .../selftests/net/io_uring_zerocopy_tx.sh | 131 ++++ 3 files changed, 737 insertions(+) create mode 100644 tools/testing/selftests/net/io_uring_zerocopy_tx.c create mode 100755 tools/testing/selftests/net/io_uring_zerocopy_tx.sh
tools/testing/selftests/net/Makefile +1 -0
@@ -59,6 +59,7 @@ TEST_GEN_FILES += toeplitz TEST_GEN_FILES += cmsg_sender TEST_GEN_FILES += stress_reuseport_listen TEST_PROGS += test_vxlan_vnifiltering.sh +TEST_GEN_FILES += io_uring_zerocopy_tx TEST_FILES := settings
tools/testing/selftests/net/io_uring_zerocopy_tx.c new file +605 -0
605 lines not on this page · raw message
tools/testing/selftests/net/io_uring_zerocopy_tx.sh new file +131 -0
131 lines not on this page · raw message
Signature · 1 line
2.36.1
David Ahern
maintainerQuoted 33 lines · Pavel Begunkov
NOTE: Not be picked directly. After getting necessary acks, I'll be working
out merging with Jakub and Jens.The patchset implements
io_uringzerocopy send. It works with both registered and normal buffers, mixing is allowed but not recommended. Apart from usual request completions, just as withMSG_ZEROCOPY,io_uringseparately notifies the userspace when buffers are freed and can be reused (see API design below), which is delivered into io_uring's Completion Queue. Those "buffer-free" notifications are not necessarily per request, but the userspace has control over it and should explicitly attaching a number of requests to a single notification. The series also adds some internal optimisations when used with registered buffers like removing page referencing.From the kernel networking perspective there are two main changes. The first one is passing
ubuf_infointo the network layer fromio_uring(inside of an in kernel struct msghdr). This allows extra optimisations, e.g.ubuf_infocaching on theio_uringside, but also helps to avoid cross-referencing and synchronisation problems. The second part is an optional optimisation removing page referencing for requests with registered buffers.Benchmarking with an optimised version of the selftest (see [1]), which sends a bunch of requests, waits for completions and repeats. "+ flush" column posts one additional "buffer-free" notification per request, and just "zc" doesn't post buffer notifications at all.
NIC (requests / second):
IO size | non-zc | zc | zc + flush 4000 | 495134 | 606420 (+22%) | 558971 (+12%) 1500 | 551808 | 577116 (+4.5%) | 565803 (+2.5%) 1000 | 584677 | 592088 (+1.2%) | 560885 (-4%) 600 | 596292 | 598550 (+0.4%) | 555366 (-6.7%)dummy (requests / second):
Quoted 9 lines
IO size | non-zc | zc | zc + flush 8000 | 1299916 | 2396600 (+84%) | 2224219 (+71%) 4000 | 1869230 | 2344146 (+25%) | 2170069 (+16%) 1200 | 2071617 | 2361960 (+14%) | 2203052 (+6%) 600 | 2106794 | 2381527 (+13%) | 2195295 (+4%)Previously it also brought a massive performance speedup compared to the
msg_zerocopytool (see [3]), which is probably not super interesting.
can you add a comment that the above results are for UDP.
You dropped comments about TCP testing; any progress there? If not, can you relay any issues you are hitting?
Quoted 40 lines · David Ahern
Quoted 39 lines · Pavel Begunkov
NOTE: Not be picked directly. After getting necessary acks, I'll be working
out merging with Jakub and Jens.The patchset implements
io_uringzerocopy send. It works with both registered and normal buffers, mixing is allowed but not recommended. Apart from usual request completions, just as withMSG_ZEROCOPY,io_uringseparately notifies the userspace when buffers are freed and can be reused (see API design below), which is delivered into io_uring's Completion Queue. Those "buffer-free" notifications are not necessarily per request, but the userspace has control over it and should explicitly attaching a number of requests to a single notification. The series also adds some internal optimisations when used with registered buffers like removing page referencing.From the kernel networking perspective there are two main changes. The first
one is passingubuf_infointo the network layer fromio_uring(inside of an in kernel struct msghdr). This allows extra optimisations, e.g.ubuf_infocaching on theio_uringside, but also helps to avoid cross-referencing and synchronisation problems. The second part is an optional optimisation removing page referencing for requests with registered buffers.Benchmarking with an optimised version of the selftest (see [1]), which sends a bunch of requests, waits for completions and repeats. "+ flush" column posts one additional "buffer-free" notification per request, and just "zc" doesn't post buffer notifications at all.
NIC (requests / second):
IO size | non-zc | zc | zc + flush 4000 | 495134 | 606420 (+22%) | 558971 (+12%) 1500 | 551808 | 577116 (+4.5%) | 565803 (+2.5%) 1000 | 584677 | 592088 (+1.2%) | 560885 (-4%) 600 | 596292 | 598550 (+0.4%) | 555366 (-6.7%)dummy (requests / second):
IO size | non-zc | zc | zc + flush 8000 | 1299916 | 2396600 (+84%) | 2224219 (+71%) 4000 | 1869230 | 2344146 (+25%) | 2170069 (+16%) 1200 | 2071617 | 2361960 (+14%) | 2203052 (+6%) 600 | 2106794 | 2381527 (+13%) | 2195295 (+4%)
Quoted 5 lines
Quoted 3 lines
Previously it also brought a massive performance speedup compared to the
msg_zerocopytool (see [3]), which is probably not super interesting.can you add a comment that the above results are for UDP.
Oh, right, forgot to add it
Quoted 2 lines
You dropped comments about TCP testing; any progress there? If not, can you relay any issues you are hitting?
Not really a problem, but for me it's bottle necked at NIC bandwidth (~3GB/s) for both zc and non-zc and doesn't even nearly saturate a CPU. Was actually benchmarked by my colleague quite a while ago, but can't find numbers. Probably need to at least add localhost numbers or grab a better server.
Signature · 1 line
Pavel Begunkov
Quoted 50 lines · Pavel Begunkov
Quoted 40 lines · David Ahern
Quoted 39 lines · Pavel Begunkov
NOTE: Not be picked directly. After getting necessary acks, I'll be working out merging with Jakub and Jens.
The patchset implements
io_uringzerocopy send. It works with both registered and normal buffers, mixing is allowed but not recommended. Apart from usual request completions, just as withMSG_ZEROCOPY,io_uringseparately notifies the userspace when buffers are freed and can be reused (see API design below), which is delivered into io_uring's Completion Queue. Those "buffer-free" notifications are not necessarily per request, but the userspace has control over it and should explicitly attaching a number of requests to a single notification. The series also adds some internal optimisations when used with registered buffers like removing page referencing.From the kernel networking perspective there are two main changes. The first one is passing
ubuf_infointo the network layer fromio_uring(inside of an in kernel struct msghdr). This allows extra optimisations, e.g.ubuf_infocaching on theio_uringside, but also helps to avoid cross-referencing and synchronisation problems. The second part is an optional optimisation removing page referencing for requests with registered buffers.Benchmarking with an optimised version of the selftest (see [1]), which sends a bunch of requests, waits for completions and repeats. "+ flush" column posts one additional "buffer-free" notification per request, and just "zc" doesn't post buffer notifications at all.
NIC (requests / second):
IO size | non-zc | zc | zc + flush4000 | 495134 | 606420 (+22%) | 558971 (+12%) 1500 | 551808 | 577116 (+4.5%) | 565803 (+2.5%) 1000 | 584677 | 592088 (+1.2%) | 560885 (-4%)600 | 596292 | 598550 (+0.4%) | 555366 (-6.7%)
dummy (requests / second):
IO size | non-zc | zc | zc + flush8000 | 1299916 | 2396600 (+84%) | 2224219 (+71%) 4000 | 1869230 | 2344146 (+25%) | 2170069 (+16%) 1200 | 2071617 | 2361960 (+14%) | 2203052 (+6%)600 | 2106794 | 2381527 (+13%) | 2195295 (+4%)
Quoted 5 lines
Quoted 3 lines
Previously it also brought a massive performance speedup compared to the
msg_zerocopytool (see [3]), which is probably not super interesting.can you add a comment that the above results are for UDP.
Oh, right, forgot to add it
Quoted 8 lines
Quoted 2 lines
You dropped comments about TCP testing; any progress there? If not, can you relay any issues you are hitting?
Not really a problem, but for me it's bottle necked at NIC bandwidth (~3GB/s) for both zc and non-zc and doesn't even nearly saturate a CPU. Was actually benchmarked by my colleague quite a while ago, but can't find numbers. Probably need to at least add localhost numbers or grab a better server.
Testing localhost TCP with a hack (see below), it doesn't include refcounting optimisations I was testing UDP with and that will be sent afterwards. Numbers are in MB/s
IO size | non-zc | zc 1200 | 4174 | 4148 4096 | 7597 | 11228
Because it's localhost, we also spend cycles here for the recv side. Using a real NIC 1200 bytes, zc is worse than non-zc ~5-10%, maybe the omitted optimisations will somewhat help. I don't consider it to be a blocker. but would be interesting to poke into later. One thing helping non-zc is that it squeezes a number of requests into a single page whenever zerocopy adds a new frag for every request.
Can't say anything new for larger payloads, I'm still NIC-bound but looking at CPU utilisation zc doesn't drain as much cycles as non-zc. Also, I don't remember if mentioned before, but another catch is that with TCP it expects users to not be flushing notifications too much, because it forces it to allocate a new skb and lose a good chunk of benefits from using TCP.
include/linux/skbuff.h +1 -4
@@ -3218,9 +3218,7 @@ static inline int skb_orphan_frags(struct sk_buff *skb, gfp_t gfp_mask)/* Frags must be orphaned, even if refcounted, if skb might loop to rx path */ static inline int skb_orphan_frags_rx (struct sk_buff *skb,gfp_t gfp_mask) { -if (likely (!skb_zcopy (skb))) -return 0 ; -return skb_copy_ubufs (skb, gfp_mask); +return skb_orphan_frags (skb, gfp_mask); } --
Pavel Begunkov
David Ahern
maintainerQuoted 77 lines · Pavel Begunkov
Quoted 67 lines · Pavel Begunkov
Quoted 56 lines · David Ahern
Quoted 55 lines · Pavel Begunkov
NOTE: Not be picked directly. After getting necessary acks, I'll be working
out merging with Jakub and Jens.The patchset implements
io_uringzerocopy send. It works with both registered
and normal buffers, mixing is allowed but not recommended. Apart from usual
request completions, just as withMSG_ZEROCOPY,io_uringseparately notifies
the userspace when buffers are freed and can be reused (see API design below),
which is delivered into io_uring's Completion Queue. Those "buffer-free"
notifications are not necessarily per request, but the userspace has control
over it and should explicitly attaching a number of requests to a single
notification. The series also adds some internal optimisations when used with
registered buffers like removing page referencing.From the kernel networking perspective there are two main changes. The first
one is passingubuf_infointo the network layer fromio_uring(inside of an
in kernel struct msghdr). This allows extra optimisations, e.g.ubuf_info
caching on theio_uringside, but also helps to avoid cross-referencing and synchronisation problems. The second part is an optional optimisation
removing page referencing for requests with registered buffers.Benchmarking with an optimised version of the selftest (see [1]), which sends
a bunch of requests, waits for completions and repeats. "+ flush" column posts
one additional "buffer-free" notification per request, and just "zc" doesn't
post buffer notifications at all.NIC (requests / second):
IO size | non-zc | zc | zc + flush4000 | 495134 | 606420 (+22%) | 558971 (+12%) 1500 | 551808 | 577116 (+4.5%) | 565803 (+2.5%) 1000 | 584677 | 592088 (+1.2%) | 560885 (-4%)600 | 596292 | 598550 (+0.4%) | 555366 (-6.7%)
dummy (requests / second):
IO size | non-zc | zc | zc + flush8000 | 1299916 | 2396600 (+84%) | 2224219 (+71%) 4000 | 1869230 | 2344146 (+25%) | 2170069 (+16%) 1200 | 2071617 | 2361960 (+14%) | 2203052 (+6%)600 | 2106794 | 2381527 (+13%) | 2195295 (+4%)
Quoted 6 lines
Quoted 4 lines
Previously it also brought a massive performance speedup compared to the
msg_zerocopytool (see [3]), which is probably not super interesting.can you add a comment that the above results are for UDP.
Oh, right, forgot to add it
Quoted 8 lines
Quoted 2 lines
You dropped comments about TCP testing; any progress there? If not, can you relay any issues you are hitting?
Not really a problem, but for me it's bottle necked at NIC bandwidth (~3GB/s) for both zc and non-zc and doesn't even nearly saturate a CPU. Was actually benchmarked by my colleague quite a while ago, but can't find numbers. Probably need to at least add localhost numbers or grab a better server.
Quoted 7 lines
Testing localhost TCP with a hack (see below), it doesn't include refcounting optimisations I was testing UDP with and that will be sent afterwards. Numbers are in MB/s
IO size | non-zc | zc
1200 | 4174 | 4148 4096 | 7597 | 11228
I am surprised by the low numbers; you should be able to saturate a 100G link with TCP and ZC TX API.
Quoted 8 lines
Because it's localhost, we also spend cycles here for the recv side. Using a real NIC 1200 bytes, zc is worse than non-zc ~5-10%, maybe the omitted optimisations will somewhat help. I don't consider it to be a blocker. but would be interesting to poke into later. One thing helping non-zc is that it squeezes a number of requests into a single page whenever zerocopy adds a new frag for every request.
Quoted 6 lines
Can't say anything new for larger payloads, I'm still NIC-bound but looking at CPU utilisation zc doesn't drain as much cycles as non-zc. Also, I don't remember if mentioned before, but another catch is that with TCP it expects users to not be flushing notifications too much, because it forces it to allocate a new skb and lose a good chunk of benefits from using TCP.
I had issues with TCP sockets and io_uring at the end of 2020:
https://www.spinics.net/lists/io-uring/msg05125.html
have not tried anything recent (from 2022).
Quoted 82 lines · David Ahern
Quoted 81 lines · Pavel Begunkov
Quoted 67 lines · Pavel Begunkov
Quoted 56 lines · David Ahern
Quoted 55 lines · Pavel Begunkov
NOTE: Not be picked directly. After getting necessary acks, I'll be working
out merging with Jakub and Jens.The patchset implements
io_uringzerocopy send. It works with both registered
and normal buffers, mixing is allowed but not recommended. Apart from usual
request completions, just as withMSG_ZEROCOPY,io_uringseparately notifies
the userspace when buffers are freed and can be reused (see API design below),
which is delivered into io_uring's Completion Queue. Those "buffer-free"
notifications are not necessarily per request, but the userspace has control
over it and should explicitly attaching a number of requests to a single
notification. The series also adds some internal optimisations when used with
registered buffers like removing page referencing.From the kernel networking perspective there are two main changes.
The first
one is passingubuf_infointo the network layer fromio_uring(inside of an
in kernel struct msghdr). This allows extra optimisations, e.g.ubuf_info
caching on theio_uringside, but also helps to avoid cross-referencing and synchronisation problems. The second part is an optional optimisation
removing page referencing for requests with registered buffers.Benchmarking with an optimised version of the selftest (see [1]), which sends
a bunch of requests, waits for completions and repeats. "+ flush" column posts
one additional "buffer-free" notification per request, and just "zc" doesn't
post buffer notifications at all.NIC (requests / second):
IO size | non-zc | zc | zc + flush4000 | 495134 | 606420 (+22%) | 558971 (+12%) 1500 | 551808 | 577116 (+4.5%) | 565803 (+2.5%) 1000 | 584677 | 592088 (+1.2%) | 560885 (-4%)600 | 596292 | 598550 (+0.4%) | 555366 (-6.7%)
dummy (requests / second):
IO size | non-zc | zc | zc + flush8000 | 1299916 | 2396600 (+84%) | 2224219 (+71%) 4000 | 1869230 | 2344146 (+25%) | 2170069 (+16%) 1200 | 2071617 | 2361960 (+14%) | 2203052 (+6%)600 | 2106794 | 2381527 (+13%) | 2195295 (+4%)
Quoted 6 lines
Quoted 4 lines
Previously it also brought a massive performance speedup compared to the
msg_zerocopytool (see [3]), which is probably not super interesting.can you add a comment that the above results are for UDP.
Oh, right, forgot to add it
Quoted 8 lines
Quoted 2 lines
You dropped comments about TCP testing; any progress there? If not, can you relay any issues you are hitting?
Not really a problem, but for me it's bottle necked at NIC bandwidth (~3GB/s) for both zc and non-zc and doesn't even nearly saturate a CPU. Was actually benchmarked by my colleague quite a while ago, but can't find numbers. Probably need to at least add localhost numbers or grab a better server.
Testing localhost TCP with a hack (see below), it doesn't include refcounting optimisations I was testing UDP with and that will be sent afterwards. Numbers are in MB/s
Quoted 6 lines
Quoted 3 lines
IO size | non-zc | zc
1200 | 4174 | 4148 4096 | 7597 | 11228I am surprised by the low numbers; you should be able to saturate a 100G link with TCP and ZC TX API.
It was a quick test with my laptop, not a super fast CPU, preemptible kernel, etc., and considering that the fact that it processes receives from in the same send syscall roughly doubles the overhead, 87Gb/s looks ok. It's not like MSG_ZEROCOPY would look much different, even more to that all sends here will be executed sequentially in io_uring, so no extra parallelism or so. As for 1200, I think 4GB/s is reasonable, it's just the kernel overhead per byte is too high, should be same with just send(2).
Quoted 14 lines
Quoted 7 lines
Because it's localhost, we also spend cycles here for the recv side. Using a real NIC 1200 bytes, zc is worse than non-zc ~5-10%, maybe the omitted optimisations will somewhat help. I don't consider it to be a blocker. but would be interesting to poke into later. One thing helping non-zc is that it squeezes a number of requests into a single page whenever zerocopy adds a new frag for every request.
Quoted 6 lines
Can't say anything new for larger payloads, I'm still NIC-bound but looking at CPU utilisation zc doesn't drain as much cycles as non-zc. Also, I don't remember if mentioned before, but another catch is that with TCP it expects users to not be flushing notifications too much, because it forces it to allocate a new skb and lose a good chunk of benefits from using TCP.
Quoted 4 lines
I had issues with TCP sockets and
io_uringat the end of 2020:
https://www.spinics.net/lists/io-uring/msg05125.htmlhave not tried anything recent (from 2022).
Haven't seen it back then. In general io_uring doesn't stop submitting requests if one request fails, at least because we're trying to execute requests asynchronously. And in general, requests can get executed out of order, so most probably submitting a bunch of requests to a single TCP sock without any ordering on io_uring side is likely a bug.
You can link io_uring requests, i.e. IOSQE_IO_LINK, guaranteeing execution ordering. And if you meant links in the message, I agree that it was not the best decision to consider len < sqe->len not an error and not breaking links, but it was later added that MSG_WAITALL would also change the success condition to len==sqe->len. But all that is relevant if you was using linking.
Signature · 1 line
Pavel Begunkov
David Ahern
maintainerQuoted 21 lines · Pavel Begunkov
Quoted 14 lines
Quoted 14 lines
Quoted 9 lines
Quoted 3 lines
You dropped comments about TCP testing; any progress there? If not, can
you relay any issues you are hitting?Not really a problem, but for me it's bottle necked at NIC bandwidth (~3GB/s) for both zc and non-zc and doesn't even nearly saturate a CPU. Was actually benchmarked by my colleague quite a while ago, but can't find numbers. Probably need to at least add localhost numbers or grab a better server.
Testing localhost TCP with a hack (see below), it doesn't include refcounting optimisations I was testing UDP with and that will be sent afterwards. Numbers are in MB/s
Quoted 6 lines
Quoted 3 lines
IO size | non-zc | zc
1200 | 4174 | 4148 4096 | 7597 | 11228I am surprised by the low numbers; you should be able to saturate a 100G link with TCP and ZC TX API.
Quoted 8 lines
It was a quick test with my laptop, not a super fast CPU, preemptible kernel, etc., and considering that the fact that it processes receives from in the same send syscall roughly doubles the overhead, 87Gb/s looks ok. It's not like
MSG_ZEROCOPYwould look much different, even more to that all sends here will be executed sequentially inio_uring, so no extra parallelism or so. As for 1200, I think 4GB/s is reasonable, it's just the kernel overhead per byte is too high, should be same with just send(2).
?
It's a stream socket so those sends are coalesced into MTU sized packets.
Quoted 18 lines
Quoted 18 lines
Quoted 8 lines
Because it's localhost, we also spend cycles here for the recv side. Using a real NIC 1200 bytes, zc is worse than non-zc ~5-10%, maybe the omitted optimisations will somewhat help. I don't consider it to be a blocker. but would be interesting to poke into later. One thing helping non-zc is that it squeezes a number of requests into a single page whenever zerocopy adds a new frag for every request.
Quoted 6 lines
Can't say anything new for larger payloads, I'm still NIC-bound but looking at CPU utilisation zc doesn't drain as much cycles as non-zc. Also, I don't remember if mentioned before, but another catch is that with TCP it expects users to not be flushing notifications too much, because it forces it to allocate a new skb and lose a good chunk of benefits from using TCP.
I had issues with TCP sockets and
io_uringat the end of 2020:
https://www.spinics.net/lists/io-uring/msg05125.html
Quoted 7 lines
Quoted 1 line
have not tried anything recent (from 2022).
Haven't seen it back then. In general
io_uringdoesn't stop submitting requests if one request fails, at least because we're trying to execute requests asynchronously. And in general, requests can get executed out of order, so most probably submitting a bunch of requests to a single TCP sock without any ordering onio_uringside is likely a bug.
TCP socket buffer fills resulting in a partial send (i.e, for a given sqe submission only part of the write/send succeeded). io_uring was not handling that case.
I'll try to find some time to resurrect the iperf3 patch and try top of tree kernel.
Quoted 8 lines
You can link
io_uringrequests, i.e.IOSQE_IO_LINK, guaranteeing execution ordering. And if you meant links in the message, I agree that it was not the best decision to consider len <sqe->lennot an error and not breaking links, but it was later added thatMSG_WAITALLwould also change the success condition to len==sqe->len. But all that is relevant if you was using linking.
Quoted 31 lines · David Ahern
Quoted 21 lines · Pavel Begunkov
Quoted 14 lines
Quoted 14 lines
Quoted 9 lines
Quoted 3 lines
You dropped comments about TCP testing; any progress there? If not, can
you relay any issues you are hitting?Not really a problem, but for me it's bottle necked at NIC bandwidth (~3GB/s) for both zc and non-zc and doesn't even nearly saturate a CPU. Was actually benchmarked by my colleague quite a while ago, but can't find numbers. Probably need to at least add localhost numbers or grab a better server.
Testing localhost TCP with a hack (see below), it doesn't include refcounting optimisations I was testing UDP with and that will be sent afterwards. Numbers are in MB/s
Quoted 6 lines
Quoted 3 lines
IO size | non-zc | zc
1200 | 4174 | 4148 4096 | 7597 | 11228I am surprised by the low numbers; you should be able to saturate a 100G link with TCP and ZC TX API.
Quoted 8 lines
It was a quick test with my laptop, not a super fast CPU, preemptible kernel, etc., and considering that the fact that it processes receives from in the same send syscall roughly doubles the overhead, 87Gb/s looks ok. It's not like
MSG_ZEROCOPYwould look much different, even more to that all sends here will be executed sequentially inio_uring, so no extra parallelism or so. As for 1200, I think 4GB/s is reasonable, it's just the kernel overhead per byte is too high, should be same with just send(2).
Quoted 2 lines
?
It's a stream socket so those sends are coalesced into MTU sized packets.
That leaves syscall and io_uring overhead, locking the socket, etc., which still requires more cycles than just copying 1200 bytes. And the used CPU is not blazingly fast, could be that a better CPU/setup will saturate 100G
Quoted 25 lines
Quoted 17 lines
Quoted 17 lines
Quoted 7 lines
Because it's localhost, we also spend cycles here for the recv side. Using a real NIC 1200 bytes, zc is worse than non-zc ~5-10%, maybe the omitted optimisations will somewhat help. I don't consider it to be a blocker. but would be interesting to poke into later. One thing helping non-zc is that it squeezes a number of requests into a single page whenever zerocopy adds a new frag for every request.
Quoted 6 lines
Can't say anything new for larger payloads, I'm still NIC-bound but looking at CPU utilisation zc doesn't drain as much cycles as non-zc. Also, I don't remember if mentioned before, but another catch is that with TCP it expects users to not be flushing notifications too much, because it forces it to allocate a new skb and lose a good chunk of benefits from using TCP.
I had issues with TCP sockets and
io_uringat the end of 2020:
https://www.spinics.net/lists/io-uring/msg05125.htmlQuoted 7 lines
Quoted 1 line
have not tried anything recent (from 2022).
Haven't seen it back then. In general
io_uringdoesn't stop submitting requests if one request fails, at least because we're trying to execute requests asynchronously. And in general, requests can get executed out of order, so most probably submitting a bunch of requests to a single TCP sock without any ordering onio_uringside is likely a bug.
Quoted 3 lines
TCP socket buffer fills resulting in a partial send (i.e, for a given sqe submission only part of the write/send succeeded).
io_uringwas not handling that case.
Shouldn't have been different from send(2) with MSG_NOWAIT, can be short and the user should handle it. Also I believe Jens pushed just recently in-kernel retries on the io_uring side for TCP in such cases.
Quoted 2 lines
I'll try to find some time to resurrect the iperf3 patch and try top of tree kernel.
Awesome
Quoted 6 lines
Quoted 6 lines
You can link
io_uringrequests, i.e.IOSQE_IO_LINK, guaranteeing execution ordering. And if you meant links in the message, I agree that it was not the best decision to consider len <sqe->lennot an error and not breaking links, but it was later added thatMSG_WAITALLwould also change the success condition to len==sqe->len. But all that is relevant if you was using linking.
Signature · 1 line
Pavel Begunkov
David Ahern
maintainerQuoted 7 lines · David Ahern
Quoted 6 lines
Haven't seen it back then. In general
io_uringdoesn't stop submitting requests if one request fails, at least because we're trying to execute requests asynchronously. And in general, requests can get executed out of order, so most probably submitting a bunch of requests to a single TCP sock without any ordering onio_uringside is likely a bug.
Quoted 6 lines
TCP socket buffer fills resulting in a partial send (i.e, for a given sqe submission only part of the write/send succeeded).
io_uringwas not handling that case.I'll try to find some time to resurrect the iperf3 patch and try top of tree kernel.
With your zc_v5 branch (plus the init fix on using msg->sg_from_iter), iperf3 with io_uring support (non-ZC case) no longer shows completions with incomplete sends. So that is good improvement over the last time I tried it.
However, adding in the ZC support and that problem resurfaces - a lot of completions are for an incomplete size.
liburing comes from your tree, zc_v4 branch. Upstream does not have support for notifications yet, so I can not move to it.
Changes to iperf3 are here:
https://github.com/dsahern/iperf mods-3.10-io_uring
Quoted 15 lines · David Ahern
Quoted 7 lines · David Ahern
Quoted 6 lines
Haven't seen it back then. In general
io_uringdoesn't stop submitting requests if one request fails, at least because we're trying to execute requests asynchronously. And in general, requests can get executed out of order, so most probably submitting a bunch of requests to a single TCP sock without any ordering onio_uringside is likely a bug.Quoted 6 lines
TCP socket buffer fills resulting in a partial send (i.e, for a given sqe submission only part of the write/send succeeded).
io_uringwas not handling that case.I'll try to find some time to resurrect the iperf3 patch and try top of tree kernel.
Quoted 7 lines
With your
zc_v5branch (plus the init fix on usingmsg->sg_from_iter), iperf3 withio_uringsupport (non-ZC case) no longer shows completions with incomplete sends. So that is good improvement over the last time I tried it.However, adding in the ZC support and that problem resurfaces - a lot of completions are for an incomplete size.
Makes sense, it explicitly retries with normal sends but I didn't implement it for zc. Might be a good thing to add.
Quoted 2 lines
liburing comes from your tree,
zc_v4branch. Upstream does not have support for notifications yet, so I can not move to it.
Upstreamed it
Quoted 2 lines
Changes to iperf3 are here:
https://github.com/dsahern/iperf mods-3.10-io_uring
Signature · 1 line
Pavel Begunkov
David Ahern
maintainerQuoted 11 lines · Pavel Begunkov
Quoted 7 lines
With your
zc_v5branch (plus the init fix on usingmsg->sg_from_iter), iperf3 withio_uringsupport (non-ZC case) no longer shows completions with incomplete sends. So that is good improvement over the last time I tried it.However, adding in the ZC support and that problem resurfaces - a lot of completions are for an incomplete size.
Makes sense, it explicitly retries with normal sends but I didn't implement it for zc. Might be a good thing to add.
Yes, before this goes it. It will be confusing to users to get incomplete completions when using the ZC option.
Quoted 20 lines · David Ahern
Quoted 7 lines · David Ahern
Quoted 6 lines
Haven't seen it back then. In general
io_uringdoesn't stop submitting requests if one request fails, at least because we're trying to execute requests asynchronously. And in general, requests can get executed out of order, so most probably submitting a bunch of requests to a single TCP sock without any ordering onio_uringside is likely a bug.Quoted 6 lines
TCP socket buffer fills resulting in a partial send (i.e, for a given sqe submission only part of the write/send succeeded).
io_uringwas not handling that case.I'll try to find some time to resurrect the iperf3 patch and try top of tree kernel.
With your
zc_v5branch (plus the init fix on usingmsg->sg_from_iter), iperf3 withio_uringsupport (non-ZC case) no longer shows completions with incomplete sends. So that is good improvement over the last time I tried it.
Quoted 8 lines
However, adding in the ZC support and that problem resurfaces - a lot of completions are for an incomplete size.
liburing comes from your tree,
zc_v4branch. Upstream does not have support for notifications yet, so I can not move to it.Changes to iperf3 are here:
https://github.com/dsahern/iperf mods-3.10-io_uring
Tried it out, the branch below fixes a small problem, adds a couple of extra optimisations and now it actually uses registered buffers.
https://github.com/isilence/iperf iou-sendzc
Still, the submission loop looked a bit weird, i.e. it submits I/O to io_uring only when it exhausts sqes instead of sending right away with some notion of QD and/or sending in batches. The approach is good for batching (SQ size =16 here), but not so for latency.
I also see some CPU cycles being burnt in select(2). io_uring wait would be more natural and perhaps more performant, but I didn't spend enough time with iperf to say for sure.
Signature · 1 line
Pavel Begunkov
David Ahern
maintainerQuoted 4 lines · Pavel Begunkov
Tried it out, the branch below fixes a small problem, adds a couple of extra optimisations and now it actually uses registered buffers.
https://github.com/isilence/iperf iou-sendzc
thanks for the patch; will it pull it in.
Quoted 9 lines
Still, the submission loop looked a bit weird, i.e. it submits I/O to
io_uringonly when it exhausts sqes instead of sending right away with some notion of QD and/or sending in batches. The approach is good for batching (SQ size =16 here), but not so for latency.I also see some CPU cycles being burnt in select(2).
io_uringwait would be more natural and perhaps more performant, but I didn't spend enough time with iperf to say for sure.
ok. It will be a while before I have time to come back to it. In the meantime it seems like some io_uring changes happened between your dev branch and what was merged into liburing (compile worked on your branch but fails with upstream). Is the ZC support in liburing now?
Quoted 13 lines · David Ahern
Quoted 4 lines · Pavel Begunkov
Tried it out, the branch below fixes a small problem, adds a couple of extra optimisations and now it actually uses registered buffers.
https://github.com/isilence/iperf iou-sendzc
thanks for the patch; will it pull it in.
Quoted 5 lines
Still, the submission loop looked a bit weird, i.e. it submits I/O to
io_uringonly when it exhausts sqes instead of sending right away with some notion of QD and/or sending in batches. The approach is good for batching (SQ size =16 here), but not so for latency.
Quoted 8 lines
Quoted 3 lines
I also see some CPU cycles being burnt in select(2).
io_uringwait would be more natural and perhaps more performant, but I didn't spend enough time with iperf to say for sure.ok. It will be a while before I have time to come back to it. In the meantime it seems like some
io_uringchanges happened between your dev branch and what was merged into liburing (compile worked on your branch but fails with upstream). Is the ZC support in liburing now?
It is. I forgot to put a note that I also adapted your patches to uapi changes.No more notification slots but a zc send request now can post a second CQE if IORING_CQE_F_MORE is set in the first one. Better described in io_uring_enter(2) man, e.g.
https://git.kernel.dk/cgit/liburing/tree/man/io_uring_enter.2#n1063
Signature · 1 line
Pavel Begunkov