socket sendmsg MSG_ZEROCOPY
Subsystems (11)
Willem de Bruijn
maintainerv2 cover socket sendmsg MSG_ZEROCOPY
From: Willem de Bruijn <[email protected]>
RFCv2:
I have received a few requests for status and rebased code of this feature. We have been running this code internally, discovering and fixing various bugs. With net-next closed, now seems like a good time to share an updated patchset with fixes. The rebase from RFCv1/v4.2 was mostly straightforward: mainly iov_iter changes. Full changelog:
RFC -> RFCv2:
- review comment: do not loop skb with zerocopy frags onto rx:
add skb_orphan_frags_rx to orphan even refcounted frags call this in __netif_receive_skb_core, deliver_skb and tun:
the same as 1080e512d44d ("net: orphan frags on receive")
- fix: hold an explicit sk reference on each notification skb. previously relied on the reference (or wmem) held by the data skb that would trigger notification, but this breaks on skb_orphan.
- fix: when aborting a send, do not inc the zerocopy counter this caused gaps in the notification chain
- fix: in packet with SOCK_DGRAM, pull ll headers before calling zerocopy_sg_from_iter
- fix: if sock_zerocopy_realloc does not allow coalescing, do not fail, just allocate a new ubuf
- fix: in tcp, check return value of second allocation attempt
- chg: allocate notification skbs from optmem
to avoid affecting tcp write queue accounting (TSQ)
- chg: limit #locked pages (ulimit) per user instead of per process
- chg: grow notification ids from 16 to 32 bit
- pass range [lo, hi] through 32 bit fields ee_info and ee_data
- chg: rebased to davem-net-next on top of v4.10-rc7
- add: limit notification coalescing
sharing ubufs limits overhead, but delays notification until the last packet is released, possibly unbounded. Add a cap.
- tests: add snd_zerocopy_lo pf_packet test
- tests: two bugfixes (add do_flush_tcp, ++sent not only in debug)
The change to allocate notification skbuffs from optmem requires ensuring that net.core.optmem is at least a few 100KB. To experiment, run
sysctl -w net.core.optmem_max=1048576
The snd_zerocopy_lo benchmarks reported in the individual patches were rerun for RFCv2. To make them work, calls to skb_orphan_frags_rx were replaced with skb_orphan_frags to allow looping to local sockets. The netperf results below are also rerun with v2.
In application load, copy avoidance shows a roughly 5% systemwide reduction in cycles when streaming large flows and a 4-8% reduction in wall clock time on early tensorflow test workloads.
Overview (from original RFC):
Add zerocopy socket sendmsg() support with flag MSG_ZEROCOPY. Implement the feature for TCP, UDP, RAW and packet sockets. This is a generalization of a previous packet socket RFC patch
http://patchwork.ozlabs.org/patch/413184/
On a send call with MSG_ZEROCOPY, the kernel pins the user pages and creates skbuff fragments directly from these pages. On tx completion, it notifies the socket owner that it is safe to modify memory by queuing a completion notification onto the socket error queue.
The kernel already implements such copy avoidance with vmsplice plus splice and with ubuf_info for tun and virtio. Extend the second with features required by TCP and others: reference counting to support cloning (retransmit queue) and shared fragments (GSO) and notification coalescing to handle corking.
Notifications are queued onto the socket error queue as a range range [N, N+m], where N is a per-socket counter incremented on each successful zerocopy send call.
* Performance
The below table shows cycles reported by perf for a netperf process sending a single 10 Gbps TCP_STREAM. The first three columns show Mcycles spent in the netperf process context. The second three columns show time spent systemwide (-a -C A,B) on the two cpus that run the process and interrupt handler. Reported is the median of at least 3 runs. std is a standard netperf, zc uses zerocopy and % is the ratio. Netperf is pinned to cpu 2, network interrupts to cpu3, rps and rfs are disabled and the kernel is booted with idle=halt.
NETPERF=./netperf -t TCP_STREAM -H $host -T 2 -l 30 -- -m $size
perf stat -e cycles $NETPERF
perf stat -C 2,3 -a -e cycles $NETPERF
--process cycles-- ----cpu cycles----
std zc % std zc %
4K 27,609 11,217 41 49,217 39,175 79
16K 21,370 3,823 18 43,540 29,213 67
64K 20,557 2,312 11 42,189 26,910 64
256K 21,110 2,134 10 43,006 27,104 63
1M 20,987 1,610 8 42,759 25,931 61Perf record indicates the main source of these differences. Process cycles only at 1M writes (perf record; perf report -n):
std:
Samples: 42K of event 'cycles', Event count (approx.): 21258597313
79.41% 33884 netperf [kernel.kallsyms] [k] copy_user_generic_string 3.27% 1396 netperf [kernel.kallsyms] [k] tcp_sendmsg 1.66% 694 netperf [kernel.kallsyms] [k] get_page_from_freelist 0.79% 325 netperf [kernel.kallsyms] [k] tcp_ack 0.43% 188 netperf [kernel.kallsyms] [k] __alloc_skb
zc:
Samples: 1K of event 'cycles', Event count (approx.): 1439509124
30.36% 584 netperf.zerocop [kernel.kallsyms] [k] gup_pte_range 14.63% 284 netperf.zerocop [kernel.kallsyms] [k] __zerocopy_sg_from_iter 8.03% 159 netperf.zerocop [kernel.kallsyms] [k] skb_zerocopy_add_frags_iter 4.84% 96 netperf.zerocop [kernel.kallsyms] [k] __alloc_skb 3.10% 60 netperf.zerocop [kernel.kallsyms] [k] kmem_cache_alloc_node
* Safety
The number of pages that can be pinned on behalf of a user with MSG_ZEROCOPY is bound by the locked memory ulimit.
While the kernel holds process memory pinned, a process cannot safely reuse those pages for other purposes. Packets looped onto the receive stack and queued to a socket can be held indefinitely. Avoid unbounded notification latency by restricting user pages to egress paths only. skb_orphan_frags_rx() will create a private copy of pages even for refcounted packets when these are looped, as did skb_orphan_frags for the original tun zerocopy implementation.
Pages are not remapped read-only. Processes can modify packet contents while packets are in flight in the kernel path. Bytes on which kernel control flow depends (headers) are copied to avoid TOCTTOU attacks. Datapath integrity does not otherwise depend on payload, with three
exceptions: checksums, optional sk_filter/tc u32/.. and device + driver logic. The effect of wrong checksums is limited to the misbehaving process. TC filters that access contents may have to be excluded by adding an skb_orphan_frags_rx.
Processes can also safely avoid OOM conditions by bounding the number of bytes passed with MSG_ZEROCOPY and by removing shared pages after transmission from their own memory map.
* Limitations / Known Issues
- PF_INET6 is not yet supported.
- TCP does not build max GSO packets, especially for small send buffers (< 4 KB)
Willem de Bruijn (12):
sock: allocate skbs from optmem
sock: skb_copy_ubufs support for compound pages
sock: add generic socket zerocopy
sock: enable sendmsg zerocopy
sock: sendmsg zerocopy notification coalescing
sock: sendmsg zerocopy ulimit
sock: sendmsg zerocopy limit bytes per notification
tcp: enable sendmsg zerocopy
udp: enable sendmsg zerocopy
raw: enable sendmsg zerocopy with IP_HDRINCL packet: enable sendmsg zerocopy test: add sendmsg zerocopy tests
drivers/net/tun.c | 2 +- drivers/vhost/net.c | 1 + include/linux/sched.h | 2 +- include/linux/skbuff.h | 94 +++- include/linux/socket.h | 1 + include/net/sock.h | 4 + include/uapi/linux/errqueue.h | 1 + net/core/datagram.c | 35 +- net/core/dev.c | 4 +- net/core/skbuff.c | 327 ++++++++++++-- net/core/sock.c | 29 ++ net/ipv4/ip_output.c | 34 +- net/ipv4/raw.c | 27 +- net/ipv4/tcp.c | 37 +- net/packet/af_packet.c | 52 ++- tools/testing/selftests/net/.gitignore | 2 + tools/testing/selftests/net/Makefile | 1 + tools/testing/selftests/net/snd_zerocopy.c | 354 +++++++++++++++ tools/testing/selftests/net/snd_zerocopy_lo.c | 596 ++++++++++++++++++++++++++ 19 files changed, 1536 insertions(+), 67 deletions(-) create mode 100644 tools/testing/selftests/net/snd_zerocopy.c create mode 100644 tools/testing/selftests/net/snd_zerocopy_lo.c
Signature · 1 line
2.11.0.483.g087da7b7c-goog
Willem de Bruijn
maintainerRFC PATCH v2 1/12 sock: allocate skbs from optmem
From: Willem de Bruijn <[email protected]>
Add sock_omalloc and sock_ofree to be able to allocate control skbs, for instance for looping errors onto sk_error_queue.
The transmit budget (sk_wmem_alloc) is involved in transmit skb shaping, most notably in TCP Small Queues. Using this budget for control packets would impact transmission.
--- include/net/sock.h | 2 ++ net/core/sock.c | 27 +++++++++++++++++++++++++++ 2 files changed, 29 insertions(+)
include/net/sock.h +2 -0
@@ -1531,6 +1531,8 @@ struct sk_buff *sock_wmalloc(struct sock *sk, unsigned long size, int force, gfp_t priority);void __sock_wfree (struct sk_buff *skb);void sock_wfree (struct sk_buff *skb); +struct sk_buff *sock_omalloc (struct sock *sk,unsigned long size, +gfp_t priority);void skb_orphan_partial (struct sk_buff *skb);void sock_rfree (struct sk_buff *skb);void sock_efree (struct sk_buff *skb);
net/core/sock.c +27 -0
@@ -1772,6 +1772,33 @@ struct sk_buff *sock_wmalloc(struct sock *sk, unsigned long size, int force, }EXPORT_SYMBOL (sock_wmalloc); +static void sock_ofree (struct sk_buff *skb) +{ +struct sock *sk = skb->sk; + +atomic_sub (skb->truesize, &sk->sk_omem_alloc); +} + +struct sk_buff *sock_omalloc (struct sock *sk,unsigned long size, +gfp_t priority) +{ +struct sk_buff *skb; + +/* small safe race: SKB_TRUESIZE may differ from final skb->truesize */ +if (atomic_read (&sk->sk_omem_alloc) +SKB_TRUESIZE (size) > + sysctl_optmem_max) +return NULL ; + + skb =alloc_skb (size, priority); +if (!skb) +return NULL ; + +atomic_add (skb->truesize, &sk->sk_omem_alloc); + skb->sk = sk; + skb->destructor = sock_ofree; +return skb; +} +/* * Allocate a memory block from the socket's option memory buffer. */
Signature · 1 line
2.11.0.483.g087da7b7c-goog
Willem de Bruijn
maintainerRFC PATCH v2 2/12 sock: skb_copy_ubufs support for compound pages
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Refine skb_copy_ubufs to support compount pages. With upcoming TCP and UDP zerocopy sendmsg, such fragments may appear.
These skbuffs can have both kernel and zerocopy fragments, e.g., when corking. Avoid unnecessary copying of fragments that have no userspace reference.
It is not safe to modify skb frags when the skbuff is shared. This should not happen. Fail loudly if we find an unexpected edge case.
--- net/core/skbuff.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)
net/core/skbuff.c +23 -1
@@ -944,6 +944,9 @@ EXPORT_SYMBOL_GPL(skb_morph);* If this function is called from an interrupt gfp_mask() must be * %GFP_ATOMIC. * +* skb_shinfo(skb) can only be safely modified when not accessed +* concurrently. Fail if the skb is shared or cloned. +* * Returns 0 on success or a negative error code on failure * to allocate kernel memory to copy to. */
@@ -954,11 +957,29 @@ int skb_copy_ubufs(struct sk_buff *skb, gfp_t gfp_mask)struct page *page, *head =NULL ;struct ubuf_info *uarg =skb_shinfo (skb)->destructor_arg; +if (skb_shared (skb) ||skb_cloned (skb)) { +WARN_ON_ONCE (1 ); +return -EINVAL ; + } +for (i =0 ; i < num_frags; i++) {u8 *vaddr; +unsigned int order =0 ; +gfp_t mask = gfp_mask;skb_frag_t *f = &skb_shinfo (skb)->frags[i]; - page =alloc_page (gfp_mask); + page =skb_frag_page (f); +if (page_count (page) ==1 ) { +skb_frag_ref (skb, i); + goto copy_done; + } + +if (f->size >PAGE_SIZE ) { + order =get_order (f->size); + mask |= __GFP_COMP; + } + + page =alloc_pages (mask, order);if (!page) {while (head) {struct page *next = (struct page *)page_private (head);
@@ -971,6 +992,7 @@ int skb_copy_ubufs(struct sk_buff *skb, gfp_t gfp_mask)memcpy (page_address (page), vaddr + f->page_offset,skb_frag_size (f));kunmap_atomic (vaddr); +copy_done:set_page_private (page, (unsigned long )head); head = page; }
Signature · 1 line
2.11.0.483.g087da7b7c-goog
Re: [PATCH RFC v2 02/12] sock: skb_copy_ubufs support for compound pages
Quoted 40 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
Refine
skb_copy_ubufsto support compount pages. With upcoming TCP and UDP zerocopy sendmsg, such fragments may appear.These skbuffs can have both kernel and zerocopy fragments, e.g., when corking. Avoid unnecessary copying of fragments that have no userspace reference.
It is not safe to modify skb frags when the skbuff is shared. This should not happen. Fail loudly if we find an unexpected edge case.
Signed-off-by Willem de Bruijn <[email protected]>--- net/core/skbuff.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)net/core/skbuff.c +8
-0@@ -944,6 +944,9 @@ EXPORT_SYMBOL_GPL(skb_morph);* If this function is called from an interrupt gfp_mask() must be * %GFP_ATOMIC. * +* skb_shinfo(skb) can only be safely modified when not accessed +* concurrently. Fail if the skb is shared or cloned. +* * Returns 0 on success or a negative error code on failure * to allocate kernel memory to copy to. */ @@ -954,11 +957,29 @@ int skb_copy_ubufs(struct sk_buff *skb, gfp_t gfp_mask)struct page *page, *head =NULL ;struct ubuf_info *uarg =skb_shinfo (skb)->destructor_arg; +if (skb_shared (skb) ||skb_cloned (skb)) { +WARN_ON_ONCE (1 ); +return -EINVAL ; + } +
net/core/skbuff.c int skb_copy_ubufs(struct sk_buff *skb, gfp_t gfp_mask) · line 973
Quoted 10 lines
net/core/skbuff.c +5
-1for (i =0 ; i < num_frags; i++) {u8 *vaddr; +unsigned int order =0 ; +gfp_t mask = gfp_mask;skb_frag_t *f = &skb_shinfo (skb)->frags[i]; - page =alloc_page (gfp_mask); + page =skb_frag_page (f); +if (page_count (page) ==1 ) { +skb_frag_ref (skb, i);
This could be : get_page(page);
net/core/skbuff.c int skb_copy_ubufs(struct sk_buff *skb, gfp_t gfp_mask) · line 979
Quoted 6 lines
net/core/skbuff.c +6
-0+ goto copy_done; + } + +if (f->size >PAGE_SIZE ) { + order =get_order (f->size); + mask |= __GFP_COMP;
Note that this would probably fail under memory pressure.
We could instead try to explode the few segments into order-0 only pages.
Hopefully this case should not be frequent.
Willem de Bruijn
maintainerRe: [PATCH RFC v2 02/12] sock: skb_copy_ubufs support for compound pages
Quoted 7 lines
net/core/skbuff.c int skb_copy_ubufs(struct sk_buff *skb, gfp_t gfp_mask) · line 973
Quoted 5 lines
net/core/skbuff.c +3
-1- page =alloc_page (gfp_mask); + page =skb_frag_page (f); +if (page_count (page) ==1 ) { +skb_frag_ref (skb, i);This could be :
get_page(page);
Ah, indeed. Thanks.
Quoted 8 lines
net/core/skbuff.c int skb_copy_ubufs(struct sk_buff *skb, gfp_t gfp_mask) · line 979
Quoted 7 lines
net/core/skbuff.c +6
-0+ goto copy_done; + } + +if (f->size >PAGE_SIZE ) { + order =get_order (f->size); + mask |= __GFP_COMP;
Quoted 4 lines
Note that this would probably fail under memory pressure.
We could instead try to explode the few segments into order-0 only pages.
Good point. I'll revise to use only order-0 here.
Willem de Bruijn
maintainerRFC PATCH v2 3/12 sock: add generic socket zerocopy
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
The kernel supports zerocopy sendmsg in virtio and tap. Expand the infrastructure to support other socket types. Introduce a completion notification channel over the socket error queue. Notifications are returned with ee_origin SO_EE_ORIGIN_ZEROCOPY. ee_errno is 0 to avoid blocking the send/recv path on receiving notifications.
Add reference counting, to support the skb split, merge, resize and clone operations possible with SOCK_STREAM and other socket types.
The patch does not yet modify any datapaths.
--- include/linux/skbuff.h | 46 ++++++++++++++++ include/linux/socket.h | 1 + include/net/sock.h | 2 + include/uapi/linux/errqueue.h | 1 + net/core/datagram.c | 35 ++++++++---- net/core/skbuff.c | 120 ++++++++++++++++++++++++++++++++++++++++++ net/core/sock.c | 2 + 7 files changed, 196 insertions(+), 11 deletions(-)
include/linux/skbuff.h +46 -0
@@ -390,6 +390,7 @@ enum { SKBTX_SCHED_TSTAMP = 1 << 6, }; +#define SKBTX_ZEROCOPY_FRAG (SKBTX_DEV_ZEROCOPY | SKBTX_SHARED_FRAG)#define SKBTX_ANY_SW_TSTAMP (SKBTX_SW_TSTAMP | \ SKBTX_SCHED_TSTAMP)#define SKBTX_ANY_TSTAMP (SKBTX_HW_TSTAMP | SKBTX_ANY_SW_TSTAMP)
@@ -406,8 +407,27 @@ struct ubuf_info {void (*callback)(struct ubuf_info *,bool zerocopy_success);void *ctx;unsigned long desc; +atomic_t refcnt; }; +#define skb_uarg (SKB ) ((struct ubuf_info *)(skb_shinfo(SKB)->destructor_arg)) + +struct ubuf_info *sock_zerocopy_alloc (struct sock *sk,size_t size); + +static inline void sock_zerocopy_get (struct ubuf_info *uarg) +{ +atomic_inc (&uarg->refcnt); +} + +void sock_zerocopy_put (struct ubuf_info *uarg); + +void sock_zerocopy_callback (struct ubuf_info *uarg,bool success); + +bool skb_zerocopy_alloc (struct sk_buff *skb,size_t size); +int skb_zerocopy_add_frags_iter (struct sock *sk,struct sk_buff *skb, +struct iov_iter *iter,int len, +struct ubuf_info *uarg); +/* This data is invariant across clones and lives at * the end of the header data, ie. at skb->end. */
@@ -1230,6 +1250,32 @@ static inline struct skb_shared_hwtstamps *skb_hwtstamps(struct sk_buff *skb)return &skb_shinfo (skb)->hwtstamps; } +static inline struct ubuf_info *skb_zcopy (struct sk_buff *skb) +{ +bool is_zcopy = skb &&skb_shinfo (skb)->tx_flags &SKBTX_DEV_ZEROCOPY ; + +return is_zcopy ?skb_uarg (skb) :NULL ; +} + +static inline void skb_zcopy_set (struct sk_buff *skb,struct ubuf_info *uarg) +{ +if (uarg) { +sock_zerocopy_get (uarg); +skb_shinfo (skb)->destructor_arg = uarg; +skb_shinfo (skb)->tx_flags |=SKBTX_ZEROCOPY_FRAG ; + } +} + +static inline void skb_zcopy_clear (struct sk_buff *skb) +{ +struct ubuf_info *uarg =skb_zcopy (skb); + +if (uarg) { +sock_zerocopy_put (uarg); +skb_shinfo (skb)->tx_flags &= ~SKBTX_DEV_ZEROCOPY ; + } +} +/** * skb_queue_empty - check if a queue is empty * @list: queue head
include/linux/socket.h +1 -0
@@ -287,6 +287,7 @@ struct ucred {#define MSG_BATCH 0x40000/* sendmmsg(): more messages coming */ #define MSG_EOF MSG_FIN +#define MSG_ZEROCOPY 0x4000000/* Use user data in kernel path */ #define MSG_FASTOPEN 0x20000000/* Send data in TCP SYN */ #define MSG_CMSG_CLOEXEC 0x40000000/* Set close_on_exec for file descriptor received through
include/net/sock.h +2 -0
@@ -288,6 +288,7 @@ struct sock_common {* @sk_stamp: time stamp of last packet received * @sk_tsflags: SO_TIMESTAMPING socket options * @sk_tskey: counter to disambiguate concurrent tstamp requests +* @sk_zckey: counter to order MSG_ZEROCOPY notifications * @sk_socket: Identd and reporting IO signals * @sk_user_data: RPC layer private data * @sk_frag: cached page frag
@@ -455,6 +456,7 @@ struct sock {u16 sk_tsflags;u8 sk_shutdown;u32 sk_tskey; +atomic_t sk_zckey;struct socket *sk_socket;void *sk_user_data;#ifdef CONFIG_SECURITY
include/uapi/linux/errqueue.h +1 -0
@@ -18,6 +18,7 @@ struct sock_extended_err {#define SO_EE_ORIGIN_ICMP 2#define SO_EE_ORIGIN_ICMP6 3#define SO_EE_ORIGIN_TXSTATUS 4 +#define SO_EE_ORIGIN_ZEROCOPY 5#define SO_EE_ORIGIN_TIMESTAMPING SO_EE_ORIGIN_TXSTATUS#define SO_EE_OFFENDER (ee) ((struct sockaddr*)((ee)+1))
net/core/datagram.c +24 -11
@@ -564,17 +564,12 @@ EXPORT_SYMBOL(skb_copy_datagram_from_iter);* * Returns 0, -EFAULT or -EMSGSIZE. */ -int zerocopy_sg_from_iter (struct sk_buff *skb,struct iov_iter *from) +int __zerocopy_sg_from_iter (struct sock *sk,struct sk_buff *skb, +struct iov_iter *from,size_t length) { -int len =iov_iter_count (from); -int copy =min_t (int,skb_headlen (skb), len); -int frag =0 ; - -/* copy up to skb headlen */ -if (skb_copy_datagram_from_iter (skb,0 , from, copy)) -return -EFAULT ; +int frag =skb_shinfo (skb)->nr_frags; -while (iov_iter_count (from)) { +while (length &&iov_iter_count (from)) {struct page *pages[MAX_SKB_FRAGS ];size_t start;ssize_t copied;
@@ -584,18 +579,24 @@ int zerocopy_sg_from_iter(struct sk_buff *skb, struct iov_iter *from)if (frag ==MAX_SKB_FRAGS )return -EMSGSIZE ; - copied =iov_iter_get_pages (from, pages, ~0U , + copied =iov_iter_get_pages (from, pages, length,MAX_SKB_FRAGS - frag, &start);if (copied <0 )return -EFAULT ;iov_iter_advance (from, copied); + length -= copied; truesize =PAGE_ALIGN (copied + start); skb->data_len += copied; skb->len += copied; skb->truesize += truesize; -atomic_add (truesize, &skb->sk->sk_wmem_alloc); +if (sk && sk->sk_type ==SOCK_STREAM ) { + sk->sk_wmem_queued += truesize; +sk_mem_charge (sk, truesize); + }else { +atomic_add (truesize, &skb->sk->sk_wmem_alloc); + }while (copied) {int size =min_t (int, copied,PAGE_SIZE - start);skb_fill_page_desc (skb, frag++, pages[n], start, size);
@@ -606,6 +607,18 @@ int zerocopy_sg_from_iter(struct sk_buff *skb, struct iov_iter *from) }return 0 ; } +EXPORT_SYMBOL (__zerocopy_sg_from_iter); + +int zerocopy_sg_from_iter (struct sk_buff *skb,struct iov_iter *from) +{ +int copy =min_t (int,skb_headlen (skb),iov_iter_count (from)); + +/* copy up to skb headlen */ +if (skb_copy_datagram_from_iter (skb,0 , from, copy)) +return -EFAULT ; + +return __zerocopy_sg_from_iter (NULL , skb, from, ~0U ); +}EXPORT_SYMBOL (zerocopy_sg_from_iter); static int skb_copy_and_csum_datagram(const struct sk_buff *skb, int offset,
net/core/skbuff.c +120 -0
126 lines not on this page · raw message
net/core/sock.c +2 -0
@@ -1526,6 +1526,7 @@ struct sock *sk_clone_lock(const struct sock *sk, const gfp_t priority)atomic_set (&newsk->sk_drops,0 ); newsk->sk_send_head =NULL ; newsk->sk_userlocks = sk->sk_userlocks & ~SOCK_BINDPORT_LOCK ; +atomic_set (&newsk->sk_zckey,0 );sock_reset_flag (newsk,SOCK_DONE );skb_queue_head_init (&newsk->sk_error_queue);
@@ -2524,6 +2525,7 @@ void sock_init_data(struct socket *sock, struct sock *sk) sk->sk_sndtimeo =MAX_SCHEDULE_TIMEOUT ; sk->sk_stamp =ktime_set (-1L ,0 ); +atomic_set (&sk->sk_zckey,0 );#ifdef CONFIG_NET_RX_BUSY_POLL sk->sk_napi_id =0 ;
Signature · 1 line
2.11.0.483.g087da7b7c-goog
Willem de Bruijn
maintainerRFC PATCH v2 4/12 sock: enable sendmsg zerocopy
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Prepare the datapath for refcounted ubuf_info. Clone ubuf_info with skb_zerocopy_clone() wherever needed due to skb split, merge, resize or clone.
Split skb_orphan_frags into two variants. The split, merge, .. paths support reference counted zerocopy buffers, so do not do a deep copy. Add skb_orphan_frags_rx for paths that may loop packets to receive sockets. That is not allowed, as it may cause unbounded latency. Deep copy all zerocopy copy buffers, ref-counted or not, in this path.
The exact locations to modify were chosen by exhaustively searching through all code that might modify skb_frag references and/or the the SKBTX_DEV_ZEROCOPY tx_flags bit.
The changes err on the safe side, in two ways.
(1) legacy ubuf_info paths virtio and tap are not modified. They keep a 1:1 ubuf_info to sk_buff relationship. Calls to skb_orphan_frags still call skb_copy_ubufs and thus copy frags in this case.
(2) not all copies deep in the stack are addressed yet. skb_shift, skb_split and skb_try_coalesce can be refined to avoid copying. These are not in the hot path and this patch is hairy enough as is, so that is left for future refinement.
--- drivers/net/tun.c | 2 +- drivers/vhost/net.c | 1 + include/linux/skbuff.h | 16 ++++++++++++++-- net/core/dev.c | 4 ++-- net/core/skbuff.c | 52 +++++++++++++++++++++----------------------------- 5 files changed, 40 insertions(+), 35 deletions(-)
drivers/net/tun.c +1 -1
@@ -880,7 +880,7 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev) sk_filter(tfile->socket.sk, skb)) goto drop; -if (unlikely (skb_orphan_frags (skb,GFP_ATOMIC ))) +if (unlikely (skb_orphan_frags_rx (skb,GFP_ATOMIC ))) goto drop;skb_tx_timestamp (skb);
drivers/vhost/net.c +1 -0
@@ -454,6 +454,7 @@ static void handle_tx(struct vhost_net *net) ubuf->callback = vhost_zerocopy_callback; ubuf->ctx = nvq->ubufs; ubuf->desc = nvq->upend_idx; +atomic_set (&ubuf->refcnt,1 ); msg.msg_control = ubuf; msg.msg_controllen =sizeof (ubuf); ubufs = nvq->ubufs;
include/linux/skbuff.h +14 -2
@@ -2448,7 +2448,7 @@ static inline void skb_orphan(struct sk_buff *skb) }/** -* skb_orphan_frags - orphan the frags contained in a buffer +* skb_orphan_frags - make a local copy of non-refcounted user frags * @skb: buffer to orphan frags from * @gfp_mask: allocation mask for replacement pages *
@@ -2458,7 +2458,17 @@ static inline void skb_orphan(struct sk_buff *skb)*/ static inline int skb_orphan_frags (struct sk_buff *skb,gfp_t gfp_mask) { -if (likely (!(skb_shinfo (skb)->tx_flags &SKBTX_DEV_ZEROCOPY ))) +if (likely (!skb_zcopy (skb))) +return 0 ; +if (skb_uarg (skb)->callback == sock_zerocopy_callback) +return 0 ; +return skb_copy_ubufs (skb, gfp_mask); +} + +/* Frags must be orphaned, even if refcounted, if skb might loop to rx path */ +static inline int skb_orphan_frags_rx (struct sk_buff *skb,gfp_t gfp_mask) +{ +if (likely (!skb_zcopy (skb)))return 0 ;return skb_copy_ubufs (skb, gfp_mask); }
@@ -2890,6 +2900,8 @@ static inline int skb_add_data(struct sk_buff *skb,static inline bool skb_can_coalesce (struct sk_buff *skb,int i,const struct page *page,int off) { +if (skb_zcopy (skb)) +return false;if (i) {const struct skb_frag_struct *frag = &skb_shinfo (skb)->frags[i -1 ];
net/core/dev.c +2 -2
@@ -1801,7 +1801,7 @@ static inline int deliver_skb(struct sk_buff *skb, struct packet_type *pt_prev, struct net_device *orig_dev) { - if (unlikely(skb_orphan_frags(skb, GFP_ATOMIC))) + if (unlikely(skb_orphan_frags_rx(skb, GFP_ATOMIC))) return -ENOMEM;atomic_inc (&skb->users);return pt_prev->func (skb, skb->dev, pt_prev, orig_dev);
@@ -4173,7 +4173,7 @@ static int __netif_receive_skb_core(struct sk_buff *skb, bool pfmemalloc) }if (pt_prev) { -if (unlikely (skb_orphan_frags (skb,GFP_ATOMIC ))) +if (unlikely (skb_orphan_frags_rx (skb,GFP_ATOMIC ))) goto drop;else ret = pt_prev->func (skb, skb->dev, pt_prev, orig_dev);
net/core/skbuff.c +22 -30
149 lines not on this page · raw message
Signature · 1 line
2.11.0.483.g087da7b7c-goog
Willem de Bruijn
maintainerRFC PATCH v2 5/12 sock: sendmsg zerocopy notification coalescing
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
In the simple case, each sendmsg() call generates data and eventually a zerocopy ready notification N, where N indicates the Nth successful invocation of sendmsg() with the MSG_ZEROCOPY flag on this socket.
TCP and corked sockets can cause sendmsg() calls to append to a single sk_buff and ubuf_info. Modify the notification path to return an inclusive range of notifications [N..N+m].
Add skb_zerocopy_realloc() to reuse ubuf_info across sendmsg() calls and modify the notification path to return a range.
For the case of reliable ordered transmission (TCP), only the upper value of the range to be read, as the lower value is guaranteed to be 1 above the last read notification.
Additionally, coalesce notifications in this common case: if an skb_uarg [1, 1] is queued while [0, 0] is already on the queue, just modify the head of the queue to read [0, 1].
--- include/linux/skbuff.h | 21 +++++++++++- net/core/skbuff.c | 92 +++++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 107 insertions(+), 6 deletions(-)
include/linux/skbuff.h +20 -1
@@ -406,13 +406,21 @@ enum {struct ubuf_info {void (*callback)(struct ubuf_info *,bool zerocopy_success);void *ctx; -unsigned long desc; +union { +unsigned long desc; +struct { +u32 id; +u16 len; + }; + };atomic_t refcnt; };#define skb_uarg (SKB ) ((struct ubuf_info *)(skb_shinfo(SKB)->destructor_arg))struct ubuf_info *sock_zerocopy_alloc (struct sock *sk,size_t size); +struct ubuf_info *sock_zerocopy_realloc (struct sock *sk,size_t size, +struct ubuf_info *uarg);static inline void sock_zerocopy_get (struct ubuf_info *uarg) {
@@ -420,6 +428,7 @@ static inline void sock_zerocopy_get(struct ubuf_info *uarg) }void sock_zerocopy_put (struct ubuf_info *uarg); +void sock_zerocopy_put_abort (struct ubuf_info *uarg);void sock_zerocopy_callback (struct ubuf_info *uarg,bool success);
@@ -1276,6 +1285,16 @@ static inline void skb_zcopy_clear(struct sk_buff *skb) } } +static inline void skb_zcopy_abort (struct sk_buff *skb) +{ +struct ubuf_info *uarg =skb_zcopy (skb); + +if (uarg) { +sock_zerocopy_put_abort (uarg); +skb_shinfo (skb)->tx_flags &= ~SKBTX_DEV_ZEROCOPY ; + } +} +/** * skb_queue_empty - check if a queue is empty * @list: queue head
net/core/skbuff.c +87 -5
124 lines not on this page · raw message
Signature · 1 line
2.11.0.483.g087da7b7c-goog
Willem de Bruijn
maintainerRFC PATCH v2 6/12 sock: sendmsg zerocopy ulimit
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Bound the number of pages that a user may pin.
Follow the lead of perf tools to maintain a per-user bound on memory locked pages commit 789f90fcf6b0 ("perf_counter: per user mlock gift")
--- include/linux/sched.h | 2 +- include/linux/skbuff.h | 5 +++++ net/core/skbuff.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 54 insertions(+), 1 deletion(-)
include/linux/sched.h +1 -1
@@ -905,7 +905,7 @@ struct user_struct {struct hlist_node uidhash_node;kuid_t uid; -#if defined(CONFIG_PERF_EVENTS ) || defined(CONFIG_BPF_SYSCALL ) +#if defined(CONFIG_PERF_EVENTS ) || defined(CONFIG_BPF_SYSCALL ) || defined(CONFIG_NET )atomic_long_t locked_vm;#endif };
include/linux/skbuff.h +5 -0
@@ -414,6 +414,11 @@ struct ubuf_info { }; };atomic_t refcnt; + +struct mmpin { +struct user_struct *user; +int num_pg; + } mmp; };#define skb_uarg (SKB ) ((struct ubuf_info *)(skb_shinfo(SKB)->destructor_arg))
net/core/skbuff.c +48 -0
72 lines not on this page · raw message
Signature · 1 line
2.11.0.483.g087da7b7c-goog
Willem de Bruijn
maintainerRFC PATCH v2 7/12 sock: sendmsg zerocopy limit bytes per notification
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Zerocopy can coalesce notifications of up to 65535 send calls. Excessive coalescing increases notification latency and process working set size.
Experiments showed trains of 75 syscalls holding around 8 MB of data per notification. On servers with many slower clients, this causes many GB of user data waiting for acknowledgment and many seconds of latency between send and notification reception.
Introduce a notification byte limit.
Implementation notes:
- Due to space constraints in struct ubuf_info, the internal calculation is approximate, in Kilobytes and capped to 64MB.
- The field is accessed only on initial allocation of ubuf_info, when the struct is private, or under the tcp lock.
- When breaking a chain, we create a new notification structure uarg. A chain can be broken in the middle of a large sendmsg. Each skbuff can only point to a single uarg, so skb_zerocopy_add_frags_iter will fail after breaking a chain. The (next) TCP patch is changed in v2 to detect failure (EEXIST) and jump to new_segment to create a new skbuff that can point to the new uarg. As a result, packetization of the bytestream may differ from a send without zerocopy.
--- include/linux/skbuff.h | 1 + net/core/skbuff.c | 11 ++++++++++- 2 files changed, 11 insertions(+), 1 deletion(-)
include/linux/skbuff.h +1 -0
@@ -411,6 +411,7 @@ struct ubuf_info {struct {u32 id;u16 len; +u16 kbytelen; }; };atomic_t refcnt;
net/core/skbuff.c +10 -1
@@ -974,6 +974,7 @@ struct ubuf_info *sock_zerocopy_alloc(struct sock *sk, size_t size) uarg->callback = sock_zerocopy_callback; uarg->id = ((u32 )atomic_inc_return (&sk->sk_zckey)) -1 ; uarg->len =1 ; + uarg->kbytelen =min_t (size_t,DIV_ROUND_UP (size,1024u ),USHRT_MAX );atomic_set (&uarg->refcnt,0 );sock_hold (sk);
@@ -990,6 +991,8 @@ struct ubuf_info *sock_zerocopy_realloc(struct sock *sk, size_t size, struct ubuf_info *uarg) {if (uarg) { +const size_t limit_kb =512 ;/* consider a sysctl */ +size_t kbytelen;u32 next;/* realloc only when socket is locked (TCP, UDP cork),
@@ -997,8 +1000,13 @@ struct ubuf_info *sock_zerocopy_realloc(struct sock *sk, size_t size,*/ BUG_ON (!sock_owned_by_user (sk)); + kbytelen = uarg->kbytelen +DIV_ROUND_UP (size,1024u ); +if (unlikely (kbytelen > limit_kb)) + goto new_alloc; + uarg->kbytelen = kbytelen; +if (unlikely (uarg->len ==USHRT_MAX -1 )) -return NULL ; + goto new_alloc; next = (u32 )atomic_read (&sk->sk_zckey);if ((u32)(uarg->id + uarg->len) == next) {
@@ -1010,6 +1018,7 @@ struct ubuf_info *sock_zerocopy_realloc(struct sock *sk, size_t size, } } +new_alloc:return sock_zerocopy_alloc (sk, size); }EXPORT_SYMBOL_GPL (sock_zerocopy_realloc);
Signature · 1 line
2.11.0.483.g087da7b7c-goog
Willem de Bruijn
maintainerRFC PATCH v2 8/12 tcp: enable sendmsg zerocopy
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Enable support for MSG_ZEROCOPY to the TCP stack. Data that is sent to a remote host will be zerocopy. TSO and GSO are supported.
Tested:
A 10x TCP_STREAM between two hosts showed a reduction in netserver process cycles by up to 70%, depending on packet size. Systemwide, savings are of course much less pronounced, at up to 20% best case.
loopback test snd_zerocopy_lo -t -z produced:
without zerocopy (-t):
rx=102852 (6418 MB) tx=102852 txc=0
rx=213216 (13305 MB) tx=213216 txc=0
rx=325266 (20298 MB) tx=325266 txc=0
rx=437082 (27275 MB) tx=437082 txc=0with zerocopy (-t -z):
rx=238446 (14880 MB) tx=238446 txc=238434
rx=500076 (31207 MB) tx=500076 txc=500060
rx=763728 (47660 MB) tx=763728 txc=763706
rx=1028184 (64163 MB) tx=1028184 txc=1028156 This test opens a pair of local sockets, one one calls sendmsg with 64KB and optionally MSG_ZEROCOPY and on the other reads the initial bytes. The receiver truncates, so this is strictly an upper bound on what is achievable. It is more representative of sending data out of a physical NIC (when payload is not touched, either).
--- net/ipv4/tcp.c | 37 ++++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-)
net/ipv4/tcp.c +34 -3
@@ -1051,13 +1051,17 @@ static int linear_payload_sz(bool first_skb)return 0 ; } -static int select_size (const struct sock *sk,bool sg,bool first_skb) +static int select_size (const struct sock *sk,bool sg,bool first_skb, +bool zerocopy) {const struct tcp_sock *tp =tcp_sk (sk);int tmp = tp->mss_cache;if (sg) {if (sk_can_gso (sk)) { +if (zerocopy) +return 0 ; + tmp =linear_payload_sz (first_skb); }else {int pgbreak =SKB_MAX_HEAD (MAX_TCP_HEADER );
@@ -1121,6 +1125,7 @@ int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)struct tcp_sock *tp =tcp_sk (sk);struct sk_buff *skb;struct sockcm_cookie sockc; +struct ubuf_info *uarg =NULL ;int flags, err, copied =0 ;int mss_now =0 , size_goal, copied_syn =0 ;bool process_backlog = false;
@@ -1190,6 +1195,21 @@ int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) sg = !!(sk->sk_route_caps &NETIF_F_SG ); +if (sg && (flags &MSG_ZEROCOPY ) && size && !uarg) { + skb =tcp_send_head (sk) ?tcp_write_queue_tail (sk) :NULL ; + uarg =sock_zerocopy_realloc (sk, size,skb_zcopy (skb)); +if (!uarg) { +if ((err =sk_stream_wait_memory (sk, &timeo)) !=0 ) + goto out_err; + uarg =sock_zerocopy_realloc (sk, size,skb_zcopy (skb)); +if (!uarg) { + err = -ENOBUFS ; + goto out_err; + } + } +sock_zerocopy_get (uarg); + } +while (msg_data_left (msg)) {int copy =0 ;int max = size_goal;
@@ -1217,7 +1237,7 @@ int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) } first_skb =skb_queue_empty (&sk->sk_write_queue); skb =sk_stream_alloc_skb (sk, -select_size (sk, sg, first_skb), +select_size (sk, sg, first_skb, uarg), sk->sk_allocation, first_skb);if (!skb)
@@ -1253,7 +1273,7 @@ int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) err =skb_add_data_nocache (sk, skb, &msg->msg_iter, copy);if (err) goto do_fault; - }else { + }else if (!uarg) {bool merge = true;int i =skb_shinfo (skb)->nr_frags;struct page_frag *pfrag =sk_page_frag (sk);
@@ -1291,6 +1311,15 @@ int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)page_ref_inc (pfrag->page); } pfrag->offset += copy; + }else { + err =skb_zerocopy_add_frags_iter (sk, skb, + &msg->msg_iter, + copy, uarg); +if (err == -EMSGSIZE || err == -EEXIST ) + goto new_segment; +if (err <0 ) + goto do_error; + copy = err; }if (!copied)
@@ -1337,6 +1366,7 @@ int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)tcp_push (sk, flags, mss_now, tp->nonagle, size_goal); } out_nopush: +sock_zerocopy_put (uarg);release_sock (sk);return copied + copied_syn;
@@ -1354,6 +1384,7 @@ int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)if (copied + copied_syn) goto out; out_err: +sock_zerocopy_put_abort (uarg); err =sk_stream_error (sk, flags, err);/* make sure we wake any epoll edge trigger waiter */ if (unlikely(skb_queue_len(&sk->sk_write_queue) == 0 &&
Signature · 1 line
2.11.0.483.g087da7b7c-goog
Willem de Bruijn
maintainerRFC PATCH v2 9/12 udp: enable sendmsg zerocopy
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Add MSG_ZEROCOPY support to inet/dgram. This includes udplite.
Tested:
loopback test snd_zerocopy_lo -u -z produces
without zerocopy (-u):
rx=173940 (10854 MB) tx=173940 txc=0
rx=367026 (22904 MB) tx=367026 txc=0
rx=564078 (35201 MB) tx=564078 txc=0
rx=756588 (47214 MB) tx=756588 txc=0with zerocopy (-u -z):
rx=377994 (23588 MB) tx=377994 txc=377980
rx=792654 (49465 MB) tx=792654 txc=792632
rx=1209582 (75483 MB) tx=1209582 txc=1209552
rx=1628376 (101618 MB) tx=1628376 txc=1628338 loopback test currently fails with corking, due to CHECKSUM_PARTIAL being disabled with UDP_CORK after commit d749c9cbffd6 ("ipv4: no CHECKSUM_PARTIAL on MSG_MORE corked sockets")
I will suggest to allow it on NETIF_F_LOOPBACK.
--- include/linux/skbuff.h | 5 +++++ net/ipv4/ip_output.c | 34 +++++++++++++++++++++++++++++----- 2 files changed, 34 insertions(+), 5 deletions(-)
include/linux/skbuff.h +5 -0
@@ -424,6 +424,11 @@ struct ubuf_info {#define skb_uarg (SKB ) ((struct ubuf_info *)(skb_shinfo(SKB)->destructor_arg)) +#define sock_can_zerocopy (sk, rt, csummode) \ + ((rt->dst.dev->features & NETIF_F_SG) && \ + ((sk->sk_type == SOCK_RAW) || \ + (sk->sk_type == SOCK_DGRAM && csummode & CHECKSUM_UNNECESSARY))) +struct ubuf_info *sock_zerocopy_alloc (struct sock *sk,size_t size);struct ubuf_info *sock_zerocopy_realloc (struct sock *sk,size_t size,struct ubuf_info *uarg);
net/ipv4/ip_output.c +29 -5
@@ -919,7 +919,7 @@ static int __ip_append_data(struct sock *sk, {struct inet_sock *inet =inet_sk (sk);struct sk_buff *skb; - +struct ubuf_info *uarg =NULL ;struct ip_options *opt = cork->opt;int hh_len;int exthdrlen;
@@ -963,9 +963,16 @@ static int __ip_append_data(struct sock *sk, !exthdrlen) csummode =CHECKSUM_PARTIAL ; +if (flags &MSG_ZEROCOPY && length && +sock_can_zerocopy (sk, rt, skb ? skb->ip_summed : csummode)) { + uarg =sock_zerocopy_realloc (sk, length,skb_zcopy (skb)); +if (!uarg) +return -ENOBUFS ; + } + cork->length += length;if ((((length + fragheaderlen) > mtu) || (skb &&skb_is_gso (skb))) && - (sk->sk_protocol ==IPPROTO_UDP ) && + (sk->sk_protocol ==IPPROTO_UDP ) && !uarg && (rt->dst.dev->features &NETIF_F_UFO ) && !rt->dst.header_len && (sk->sk_type ==SOCK_DGRAM ) && !sk->sk_no_check_tx) { err = ip_ufo_append_data(sk, queue, getfrag, from, length,
@@ -1017,6 +1024,8 @@ static int __ip_append_data(struct sock *sk,if ((flags &MSG_MORE ) && !(rt->dst.dev->features&NETIF_F_SG )) alloclen = mtu; +else if (uarg) + alloclen =min_t (int, fraglen,MAX_HEADER );else alloclen = fraglen;
@@ -1059,11 +1068,12 @@ static int __ip_append_data(struct sock *sk, cork->tx_flags =0 ;skb_shinfo (skb)->tskey = tskey; tskey =0 ; +skb_zcopy_set (skb, uarg);/* * Find where to start putting bytes. */ - data =skb_put (skb, fraglen + exthdrlen); + data =skb_put (skb, alloclen);skb_set_network_header (skb, exthdrlen); skb->transport_header = (skb->network_header + fragheaderlen);
@@ -1079,7 +1089,9 @@ static int __ip_append_data(struct sock *sk,pskb_trim_unique (skb_prev, maxfraglen); } - copy = datalen - transhdrlen - fraggap; + copy =min (datalen, + alloclen - exthdrlen - fragheaderlen); + copy -= transhdrlen - fraggap;if (copy >0 &&getfrag (from, data + transhdrlen, offset, copy, fraggap, skb) <0 ) { err = -EFAULT ;kfree_skb (skb);
@@ -1087,7 +1099,7 @@ static int __ip_append_data(struct sock *sk, } offset += copy; - length -= datalen - fraggap; + length -= copy + transhdrlen; transhdrlen =0 ; exthdrlen =0 ; csummode =CHECKSUM_NONE ;
@@ -1115,6 +1127,17 @@ static int __ip_append_data(struct sock *sk, err = -EFAULT ; goto error; } + }else if (uarg) { +struct iov_iter *iter; + +if (sk->sk_type ==SOCK_RAW ) + iter = &((struct msghdr **)from)[0 ]->msg_iter; +else + iter = &((struct msghdr *)from)->msg_iter; + err =skb_zerocopy_add_frags_iter (sk, skb, iter, copy, uarg); +if (err <0 ) + goto error; + copy = err; }else {int i =skb_shinfo (skb)->nr_frags;
@@ -1155,6 +1178,7 @@ static int __ip_append_data(struct sock *sk, error_efault: err = -EFAULT ; error: +sock_zerocopy_put_abort (uarg); cork->length -= length;IP_INC_STATS (sock_net (sk),IPSTATS_MIB_OUTDISCARDS );return err;
Signature · 1 line
2.11.0.483.g087da7b7c-goog
Willem de Bruijn
maintainerRFC PATCH v2 10/12 raw: enable sendmsg zerocopy with IP_HDRINCL
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Tested:
raw loopback test snd_zerocopy_lo -r -z produces:
without zerocopy (-r):
rx=97632 (6092 MB) tx=97632 txc=0
rx=208194 (12992 MB) tx=208194 txc=0
rx=318714 (19889 MB) tx=318714 txc=0
rx=429126 (26779 MB) tx=429126 txc=0with zerocopy (-r -z):
rx=326160 (20353 MB) tx=326160 txc=326144
rx=689244 (43012 MB) tx=689244 txc=689220
rx=1049352 (65484 MB) tx=1049352 txc=1049320
rx=1408782 (87914 MB) tx=1408782 txc=1408744 raw hdrincl loopback test snd_zerocopy_lo -R -z produces:
without zerocopy (-R):
rx=167328 (10442 MB) tx=167328 txc=0
rx=354942 (22150 MB) tx=354942 txc=0
rx=542400 (33848 MB) tx=542400 txc=0
rx=716442 (44709 MB) tx=716442 txc=0with zerocopy (-R -z):
rx=340116 (21224 MB) tx=340116 txc=340102
rx=712746 (44478 MB) tx=712746 txc=712726
rx=1083732 (67629 MB) tx=1083732 txc=1083704
rx=1457856 (90976 MB) tx=1457856 txc=1457820--- net/ipv4/raw.c | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-)
net/ipv4/raw.c +23 -4
@@ -351,7 +351,7 @@ static int raw_send_hdrinc(struct sock *sk, struct flowi4 *fl4,unsigned int iphlen;int err;struct rtable *rt = *rtp; -int hlen, tlen; +int hlen, tlen, linear;if (length > rt->dst.dev->mtu) { ip_local_error(sk, EMSGSIZE, fl4->daddr, inet->inet_dport,
@@ -363,8 +363,14 @@ static int raw_send_hdrinc(struct sock *sk, struct flowi4 *fl4, hlen =LL_RESERVED_SPACE (rt->dst.dev); tlen = rt->dst.dev->needed_tailroom; + linear = length; + +if (flags &MSG_ZEROCOPY && length && +sock_can_zerocopy (sk, rt,CHECKSUM_UNNECESSARY )) + linear =min_t (int, length,MAX_HEADER ); + skb =sock_alloc_send_skb (sk, - length + hlen + tlen +15 , + linear + hlen + tlen +15 , flags &MSG_DONTWAIT , &err);if (!skb) goto error;
@@ -377,7 +383,7 @@ static int raw_send_hdrinc(struct sock *sk, struct flowi4 *fl4,skb_reset_network_header (skb); iph =ip_hdr (skb); -skb_put (skb, length); +skb_put (skb, linear); skb->ip_summed =CHECKSUM_NONE ;
@@ -388,7 +394,7 @@ static int raw_send_hdrinc(struct sock *sk, struct flowi4 *fl4, skb->transport_header = skb->network_header; err = -EFAULT ; -if (memcpy_from_msg (iph, msg, length)) +if (memcpy_from_msg (iph, msg, linear)) goto error_free; iphlen = iph->ihl *4 ;
@@ -404,6 +410,17 @@ static int raw_send_hdrinc(struct sock *sk, struct flowi4 *fl4,if (iphlen > length) goto error_free; +if (length != linear) { +size_t datalen = length - linear; + +if (!skb_zerocopy_alloc (skb, datalen)) + goto error_zcopy; + err =skb_zerocopy_add_frags_iter (sk, skb, &msg->msg_iter, + datalen,skb_uarg (skb)); +if (err != datalen) + goto error_zcopy; + } +if (iphlen >=sizeof (*iph)) {if (!iph->saddr) iph->saddr = fl4->saddr;
@@ -430,6 +447,8 @@ static int raw_send_hdrinc(struct sock *sk, struct flowi4 *fl4, out:return 0 ; +error_zcopy: +sock_zerocopy_put_abort (skb_zcopy (skb)); error_free:kfree_skb (skb); error:
Signature · 1 line
2.11.0.483.g087da7b7c-goog
Willem de Bruijn
maintainerRFC PATCH v2 11/12 packet: enable sendmsg zerocopy
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Support MSG_ZEROCOPY on PF_PACKET transmission.
Tested:
pf_packet loopback test snd_zerocopy_lo -p -z produces:
without zerocopy (-p):
rx=0 (0 MB) tx=221696 txc=0
rx=0 (0 MB) tx=443880 txc=0
rx=0 (0 MB) tx=661056 txc=0
rx=0 (0 MB) tx=877152 txc=0with zerocopy (-p -z):
rx=0 (0 MB) tx=528548 txc=528544
rx=0 (0 MB) tx=1052364 txc=1052360
rx=0 (0 MB) tx=1571956 txc=1571952
rx=0 (0 MB) tx=2094144 txc=2094140Packets do not arrive at the Rx socket due to a martian test:
IPv4: martian destination 127.0.0.1 from 127.0.0.1, dev lo
I'll need to revise snd_zerocopy_lo to bypass that.
--- net/packet/af_packet.c | 52 ++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 42 insertions(+), 10 deletions(-)
net/packet/af_packet.c +42 -10
@@ -2754,28 +2754,55 @@ static int tpacket_snd(struct packet_sock *po, struct msghdr *msg)static struct sk_buff *packet_alloc_skb (struct sock *sk,size_t prepad,size_t reserve,size_t len, -size_t linear,int noblock, +size_t linear,int flags,int *err) {struct sk_buff *skb; +size_t data_len; -/* Under a page? Don't bother with paged skb. */ -if (prepad + len <PAGE_SIZE || !linear) - linear = len; +if (flags &MSG_ZEROCOPY ) { +/* Minimize linear, but respect header lower bound */ + linear = reserve +min (len,max_t (size_t, linear,MAX_HEADER )); + data_len =0 ; + }else { +/* Under a page? Don't bother with paged skb. */ +if (prepad + len <PAGE_SIZE || !linear) + linear = len; + data_len = len - linear; + } - skb =sock_alloc_send_pskb (sk, prepad + linear, len - linear, noblock, - err,0 ); + skb =sock_alloc_send_pskb (sk, prepad + linear, data_len, + flags &MSG_DONTWAIT , err,0 );if (!skb)return NULL ;skb_reserve (skb, reserve);skb_put (skb, linear); - skb->data_len = len - linear; - skb->len += len - linear; + skb->data_len = data_len; + skb->len += data_len;return skb; } +static int packet_zerocopy_sg_from_iovec (struct sk_buff *skb, +struct msghdr *msg, +int offset,size_t size) +{ +int ret; + +/* if SOCK_DGRAM, head room was alloc'ed and holds ll-headers */ +__skb_pull (skb, offset); + ret =zerocopy_sg_from_iter (skb, &msg->msg_iter); +__skb_push (skb, offset); +if (unlikely (ret)) +return ret == -EMSGSIZE ? ret : -EIO ; + +if (!skb_zerocopy_alloc (skb, size)) +return -ENOMEM ; + +return 0 ; +} +static int packet_snd (struct socket *sock,struct msghdr *msg,size_t len) {struct sock *sk = sock->sk;
@@ -2853,7 +2880,7 @@ static int packet_snd(struct socket *sock, struct msghdr *msg, size_t len) linear =__virtio16_to_cpu (vio_le (), vnet_hdr.hdr_len); linear =max (linear,min_t (int, len, dev->hard_header_len)); skb =packet_alloc_skb (sk, hlen + tlen, hlen, len, linear, - msg->msg_flags &MSG_DONTWAIT , &err); + msg->msg_flags, &err);if (skb ==NULL ) goto out_unlock;
@@ -2867,7 +2894,11 @@ static int packet_snd(struct socket *sock, struct msghdr *msg, size_t len) }/* Returns -EFAULT on error */ - err =skb_copy_datagram_from_iter (skb, offset, &msg->msg_iter, len); +if (msg->msg_flags &MSG_ZEROCOPY ) + err =packet_zerocopy_sg_from_iovec (skb, msg, offset, len); +else + err =skb_copy_datagram_from_iter (skb, offset, &msg->msg_iter, + len);if (err) goto out_free;
@@ -2913,6 +2944,7 @@ static int packet_snd(struct socket *sock, struct msghdr *msg, size_t len)return len; out_free: +skb_zcopy_abort (skb);kfree_skb (skb); out_unlock:if (dev)
Signature · 1 line
2.11.0.483.g087da7b7c-goog
Willem de Bruijn
maintainerRFC PATCH v2 12/12 test: add sendmsg zerocopy tests
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
Introduce the tests uses to verify MSG_ZEROCOPY behavior:
snd_zerocopy:
send zerocopy fragments out over the default route.
snd_zerocopy_lo:
send data between a pair of local sockets and report throughput.
These tests are not suitable for inclusion in /tools/testing/selftest as is, as they do not return a pass/fail verdict. Including them in this RFC for demonstration, only.
--- tools/testing/selftests/net/.gitignore | 2 + tools/testing/selftests/net/Makefile | 1 + tools/testing/selftests/net/snd_zerocopy.c | 354 +++++++++++++++ tools/testing/selftests/net/snd_zerocopy_lo.c | 596 ++++++++++++++++++++++++++ 4 files changed, 953 insertions(+) create mode 100644 tools/testing/selftests/net/snd_zerocopy.c create mode 100644 tools/testing/selftests/net/snd_zerocopy_lo.c
tools/testing/selftests/net/.gitignore +2 -0
@@ -5,3 +5,5 @@ reuseport_bpf reuseport_bpf_cpu reuseport_bpf_numa reuseport_dualstack +snd_zerocopy +snd_zerocopy_lo
tools/testing/selftests/net/Makefile +1 -0
@@ -7,6 +7,7 @@ NET_PROGS = socket NET_PROGS += psock_fanout psock_tpacket NET_PROGS += reuseport_bpf reuseport_bpf_cpu reuseport_bpf_numa NET_PROGS += reuseport_dualstack +NET_PROGS += snd_zerocopy snd_zerocopy_lo all: $(NET_PROGS) reuseport_bpf_numa: LDFLAGS += -lnuma
tools/testing/selftests/net/snd_zerocopy.c new file +354 -0
354 lines not on this page · raw message
tools/testing/selftests/net/snd_zerocopy_lo.c new file +596 -0
596 lines not on this page · raw message
Signature · 1 line
2.11.0.483.g087da7b7c-goog
David Miller
maintainerFrom: Willem de Bruijn <[email protected]>
Date: Wed, 22 Feb 2017 11:38:49 -0500
Quoted 9 lines
From: Willem de Bruijn <[email protected]>
RFCv2:
I have received a few requests for status and rebased code of this feature. We have been running this code internally, discovering and fixing various bugs. With net-next closed, now seems like a good time to share an updated patchset with fixes. The rebase from RFCv1/v4.2 was mostly straightforward: mainly
iov_iterchanges. Full changelog:
I've been over this series once or twice and generally speaking it looks fine to me.
Quoted 4 lines · Willem de Bruijn
* Limitations / Known Issues
-
PF_INET6is not yet supported.
we struggled so far to make it work in our setups which are ipv6 only. Looking at patches it seems the code should just work.
What particularly is missing ?
Great stuff. Looking forward to net-next reopening :)
Willem de Bruijn
maintainerQuoted 11 lines · Alexei Starovoitov
Quoted 4 lines · Willem de Bruijn
* Limitations / Known Issues
-
PF_INET6is not yet supported.we struggled so far to make it work in our setups which are ipv6 only. Looking at patches it seems the code should just work.
What particularly is missing ?Great stuff. Looking forward to net-next reopening :)
Thanks for taking the feature for a spin!
The udp and raw paths require separate ipv6 patches. TCP should indeed just work. I just ran a slightly modified snd_zerocopy_lo with good results as well as a hacked netperf to another host.
I should have had ipv6 from the start, of course. Will add it before resubmitting when net-next opens. For now, quick hack to snd_zerocopy_lo.c:
(quoted diff) +0 -0
b/tools/testing/selftests/net/snd_zerocopy_lo.c
index 309b016a4fd5..38a165e2af64 100644
tools/testing/selftests/net/snd_zerocopy_lo.c +1 -1
@@ -453,7 +453,7 @@ static int do_setup_rx(int domain, int type, int protocol)static void do_setup_and_run (int domain,int type,int protocol) { -struct sockaddr_in addr; +struct sockaddr_in6 addr;socklen_t alen;int fdr, fdt, ret;
@@ -468,8 +468,8 @@ static void do_setup_and_run(int domain, int type,
int protocol)
if (domain != PF_PACKET) {
memset(&addr, 0, sizeof(addr));
- addr.sin_family = AF_INET;
- addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
+ addr.sin6_family = AF_INET6;
+ addr.sin6_addr = in6addr_loopback;
alen = sizeof(addr);
if (bind(fdr, (void *) &addr, sizeof(addr)))
@@ -589,7 +589,7 @@ int main(int argc, char **argv)
if (cfg_test_raw_hdrincl)
do_setup_and_run(PF_INET, SOCK_RAW, IPPROTO_RAW);
if (cfg_test_tcp)- do_setup_and_run(PF_INET, SOCK_STREAM, 0);
+ do_setup_and_run(PF_INET6, SOCK_STREAM, 0);
Loopback zerocopy is disabled in RFCv2, so to use snd_zerocopy_lo to verify the feature requires this hack in skb_orphan_frags_rx:
static inline int skb_orphan_frags_rx(struct sk_buff *skb, gfp_t gfp_mask) {
- if (likely(!skb_zcopy(skb)))
- return 0;
- return skb_copy_ubufs(skb, gfp_mask);
+ return skb_orphan_frags(skb, gfp_mask);
}
With this change, I see
$ ./snd_zerocopy_lo_ipv6 -t test socket(10, 1, 0) cpu: 23 rx=106364 (6637 MB) tx=106364 txc=0 rx=209736 (13088 MB) tx=209736 txc=0 rx=314524 (19627 MB) tx=314524 txc=0 rx=419424 (26174 MB) tx=419424 txc=0 OK. All tests passed
$ ./snd_zerocopy_lo_ipv6 -t -z test socket(10, 1, 0) cpu: 23 rx=239792 (14964 MB) tx=239792 txc=239786 rx=477376 (29790 MB) tx=477376 txc=477370 rx=715016 (44620 MB) tx=715016 txc=715010 rx=952820 (59460 MB) tx=952820 txc=952814 OK. All tests passed
In comparison, the same without the change
$ ./snd_zerocopy_lo_ipv6 -t test socket(10, 1, 0) cpu: 23 rx=109908 (6858 MB) tx=109908 txc=0 rx=217100 (13548 MB) tx=217100 txc=0 rx=326584 (20380 MB) tx=326584 txc=0 rx=429568 (26807 MB) tx=429568 txc=0 OK. All tests passed
$ ./snd_zerocopy_lo_ipv6 -t -z test socket(10, 1, 0) cpu: 23 rx=87636 (5468 MB) tx=87636 txc=87630 rx=174328 (10878 MB) tx=174328 txc=174322 rx=260360 (16247 MB) tx=260360 txc=260354 rx=346512 (21623 MB) tx=346512 txc=346506
Here the sk_buff hits the deep copy in skb_copy_ubufs called from __netif_receive_skb_core, which actually degrades performance versus copying as part of the sendmsg() syscall.
The netperf change is to add MSG_ZEROCOPY to send() in send_tcp_stream and also adding a recvmsg(send_socket, &msg, MSG_ERRQUEUE) to the same function, preferably called only once every N iterations. This does not take any additional explicit references on the send_ring element while data is in flight, so is really a hack, but ring contents should be static throughout the test. I did not modify the omni tests, so this requires building with --no-omni.
[CC += [email protected]]
Hi Willem
This is a change to the kernel-user-space API. Please CC [email protected] on any future iterations of this patch.
Thanks,
Michael
Quoted 194 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
RFCv2:
I have received a few requests for status and rebased code of this feature. We have been running this code internally, discovering and fixing various bugs. With net-next closed, now seems like a good time to share an updated patchset with fixes. The rebase from RFCv1/v4.2 was mostly straightforward: mainly
iov_iterchanges. Full changelog:RFC -> RFCv2:
- review comment: do not loop skb with zerocopy frags onto rx:
addskb_orphan_frags_rxto orphan even refcounted frags call this in__netif_receive_skb_core,deliver_skband tun:
the same as1080e512d44d("net: orphan frags on receive")
- fix: hold an explicit sk reference on each notification skb. previously relied on the reference (or wmem) held by the data skb that would trigger notification, but this breaks onskb_orphan.
- fix: when aborting a send, do not inc the zerocopy counter this caused gaps in the notification chain
- fix: in packet withSOCK_DGRAM, pull ll headers before callingzerocopy_sg_from_iter
- fix: ifsock_zerocopy_reallocdoes not allow coalescing, do not fail, just allocate a new ubuf
- fix: in tcp, check return value of second allocation attempt
- chg: allocate notification skbs from optmem
to avoid affecting tcp write queue accounting (TSQ)
- chg: limit #locked pages (ulimit) per user instead of per process
- chg: grow notification ids from 16 to 32 bit
- pass range [lo, hi] through 32 bit fieldsee_infoandee_data
- chg: rebased to davem-net-next on top of v4.10-rc7
- add: limit notification coalescing
sharing ubufs limits overhead, but delays notification until the last packet is released, possibly unbounded. Add a cap.
- tests: addsnd_zerocopy_lopf_packettest
- tests: two bugfixes (adddo_flush_tcp, ++sent not only in debug)The change to allocate notification skbuffs from optmem requires ensuring that net.core.optmem is at least a few 100KB. To experiment, run
sysctl -w net.core.optmem_max=1048576
The
snd_zerocopy_lobenchmarks reported in the individual patches were rerun for RFCv2. To make them work, calls toskb_orphan_frags_rxwere replaced withskb_orphan_fragsto allow looping to local sockets. The netperf results below are also rerun with v2.In application load, copy avoidance shows a roughly 5% systemwide reduction in cycles when streaming large flows and a 4-8% reduction in wall clock time on early tensorflow test workloads.
Overview (from original RFC):
Add zerocopy socket
sendmsg()support with flagMSG_ZEROCOPY. Implement the feature for TCP, UDP, RAW and packet sockets. This is a generalization of a previous packet socket RFC patchhttp://patchwork.ozlabs.org/patch/413184/
On a send call with
MSG_ZEROCOPY, the kernel pins the user pages and creates skbuff fragments directly from these pages. On tx completion, it notifies the socket owner that it is safe to modify memory by queuing a completion notification onto the socket error queue.The kernel already implements such copy avoidance with vmsplice plus splice and with
ubuf_infofor tun and virtio. Extend the second with features required by TCP and others: reference counting to support cloning (retransmit queue) and shared fragments (GSO) and notification coalescing to handle corking.Notifications are queued onto the socket error queue as a range range [N, N+m], where N is a per-socket counter incremented on each successful zerocopy send call.
* Performance
The below table shows cycles reported by perf for a netperf process sending a single 10 Gbps
TCP_STREAM. The first three columns show Mcycles spent in the netperf process context. The second three columns show time spent systemwide (-a -C A,B) on the two cpus that run the process and interrupt handler. Reported is the median of at least 3 runs. std is a standard netperf, zc uses zerocopy and % is the ratio. Netperf is pinned to cpu 2, network interrupts to cpu3, rps and rfs are disabled and the kernel is booted with idle=halt.NETPERF=./netperf -t
TCP_STREAM-H $host -T 2 -l 30 -- -m $sizeperf stat -e cycles $NETPERF
perf stat -C 2,3 -a -e cycles $NETPERF--process cycles-- ----cpu cycles---- std zc % std zc % 4K 27,609 11,217 41 49,217 39,175 79 16K 21,370 3,823 18 43,540 29,213 67 64K 20,557 2,312 11 42,189 26,910 64 256K 21,110 2,134 10 43,006 27,104 63 1M 20,987 1,610 8 42,759 25,931 61Perf record indicates the main source of these differences. Process cycles only at 1M writes (perf record; perf report -n):
std:
Samples: 42K of event 'cycles', Event count (approx.): 2125859731379.41% 33884 netperf [kernel.kallsyms] [k] copy_user_generic_string 3.27% 1396 netperf [kernel.kallsyms] [k] tcp_sendmsg 1.66% 694 netperf [kernel.kallsyms] [k] get_page_from_freelist 0.79% 325 netperf [kernel.kallsyms] [k] tcp_ack 0.43% 188 netperf [kernel.kallsyms] [k] __alloc_skbzc:
Samples: 1K of event 'cycles', Event count (approx.): 143950912430.36% 584 netperf.zerocop [kernel.kallsyms] [k] gup_pte_range 14.63% 284 netperf.zerocop [kernel.kallsyms] [k] __zerocopy_sg_from_iter 8.03% 159 netperf.zerocop [kernel.kallsyms] [k] skb_zerocopy_add_frags_iter 4.84% 96 netperf.zerocop [kernel.kallsyms] [k] __alloc_skb 3.10% 60 netperf.zerocop [kernel.kallsyms] [k] kmem_cache_alloc_node* Safety
The number of pages that can be pinned on behalf of a user with
MSG_ZEROCOPYis bound by the locked memory ulimit.While the kernel holds process memory pinned, a process cannot safely reuse those pages for other purposes. Packets looped onto the receive stack and queued to a socket can be held indefinitely. Avoid unbounded notification latency by restricting user pages to egress paths only.
skb_orphan_frags_rx()will create a private copy of pages even for refcounted packets when these are looped, as didskb_orphan_fragsfor the original tun zerocopy implementation.Pages are not remapped read-only. Processes can modify packet contents while packets are in flight in the kernel path. Bytes on which kernel control flow depends (headers) are copied to avoid TOCTTOU attacks. Datapath integrity does not otherwise depend on payload, with three
exceptions: checksums, optional sk_filter/tc u32/.. and device + driver logic. The effect of wrong checksums is limited to the misbehaving process. TC filters that access contents may have to be excluded by adding anskb_orphan_frags_rx.Processes can also safely avoid OOM conditions by bounding the number of bytes passed with
MSG_ZEROCOPYand by removing shared pages after transmission from their own memory map.* Limitations / Known Issues
-
PF_INET6is not yet supported.
- TCP does not build max GSO packets, especially for small send buffers (< 4 KB)Willem de Bruijn (12):
sock: allocate skbs from optmem
sock:skb_copy_ubufssupport for compound pages
sock: add generic socket zerocopy
sock: enable sendmsg zerocopy
sock: sendmsg zerocopy notification coalescing
sock: sendmsg zerocopy ulimit
sock: sendmsg zerocopy limit bytes per notification
tcp: enable sendmsg zerocopy
udp: enable sendmsg zerocopyraw: enable sendmsg zerocopy with IP_HDRINCL packet: enable sendmsg zerocopy test: add sendmsg zerocopy testsdrivers/net/tun.c | 2 +- drivers/vhost/net.c | 1 + include/linux/sched.h | 2 +- include/linux/skbuff.h | 94 +++- include/linux/socket.h | 1 + include/net/sock.h | 4 + include/uapi/linux/errqueue.h | 1 + net/core/datagram.c | 35 +- net/core/dev.c | 4 +- net/core/skbuff.c | 327 ++++++++++++-- net/core/sock.c | 29 ++ net/ipv4/ip_output.c | 34 +- net/ipv4/raw.c | 27 +- net/ipv4/tcp.c | 37 +- net/packet/af_packet.c | 52 ++- tools/testing/selftests/net/.gitignore | 2 + tools/testing/selftests/net/Makefile | 1 + tools/testing/selftests/net/snd_zerocopy.c | 354 +++++++++++++++ tools/testing/selftests/net/snd_zerocopy_lo.c | 596 ++++++++++++++++++++++++++ 19 files changed, 1536 insertions(+), 67 deletions(-) create mode 100644 tools/testing/selftests/net/snd_zerocopy.c create mode 100644 tools/testing/selftests/net/snd_zerocopy_lo.c--
2.11.0.483.g087da7b7c-goog
Signature · 3 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Author of "The Linux Programming Interface", http://blog.man7.org/
Quoted 9 lines · Michael Kerrisk
[CC += [email protected]]
Hi Willem
Quoted 4 lines
On a send call with
MSG_ZEROCOPY, the kernel pins the user pages and creates skbuff fragments directly from these pages. On tx completion, it notifies the socket owner that it is safe to modify memory by queuing a completion notification onto the socket error queue.
What happens if the user writes to the pages while it's not safe?
How about if you're writing to an interface or a route that has crypto involved and a malicious user can make the data change in the middle of a crypto operation, thus perhaps leaking the entire key? (I wouldn't be at all surprised if a lot of provably secure AEAD constructions are entirely compromised if an attacker can get the ciphertext and tag computed from a message that changed during the computation.
I can see this working if you have a special type of skb that indicates that the data might be concurrently written and have all the normal skb APIs (including, especially, anything that clones it) make a copy first.
--Andy
Willem de Bruijn
maintainerQuoted 14 lines · Andy Lutomirski
Quoted 9 lines · Michael Kerrisk
[CC += [email protected]]
Hi Willem
Quoted 4 lines
On a send call with
MSG_ZEROCOPY, the kernel pins the user pages and creates skbuff fragments directly from these pages. On tx completion, it notifies the socket owner that it is safe to modify memory by queuing a completion notification onto the socket error queue.What happens if the user writes to the pages while it's not safe?
Quoted 7 lines
How about if you're writing to an interface or a route that has crypto involved and a malicious user can make the data change in the middle of a crypto operation, thus perhaps leaking the entire key? (I wouldn't be at all surprised if a lot of provably secure AEAD constructions are entirely compromised if an attacker can get the ciphertext and tag computed from a message that changed during the computation.
Operations that read or write payload, such as this crypto example, but also ebpf in tc or iptables, for instance, demand a deep copy using skb_copy_ubufs before the operation.
This blacklist approach requires caution, but these paths should be few and countable. It is not possible to predict at the socket layer whether a packet will encounter any such operation, so white-listing a subset of end-to-end paths is not practical.
Quoted 4 lines
I can see this working if you have a special type of skb that indicates that the data might be concurrently written and have all the normal skb APIs (including, especially, anything that clones it) make a copy first.
Support for cloned skbs is required for TCP, both at tcp_transmit_skb and segmentation offload. Patch 4 especially adds reference counting of shared pages across clones and other sk_buff operations like pskb_expand_head. This still allows for deep copy (skb_copy_ubufs) on clones in specific datapaths like the above.
A message that is not in the archive (sent off-list or not yet received): CAF=yD-K_0zO3pMeXf-UKGTsD4sNOdyN9KJkUb5MnCO_J5pisrA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 23 lines · Willem de Bruijn
Quoted 14 lines · Andy Lutomirski
Quoted 9 lines · Michael Kerrisk
[CC += [email protected]]
Hi Willem
Quoted 4 lines
On a send call with
MSG_ZEROCOPY, the kernel pins the user pages and creates skbuff fragments directly from these pages. On tx completion, it notifies the socket owner that it is safe to modify memory by queuing a completion notification onto the socket error queue.What happens if the user writes to the pages while it's not safe?
Quoted 7 lines
How about if you're writing to an interface or a route that has crypto involved and a malicious user can make the data change in the middle of a crypto operation, thus perhaps leaking the entire key? (I wouldn't be at all surprised if a lot of provably secure AEAD constructions are entirely compromised if an attacker can get the ciphertext and tag computed from a message that changed during the computation.
Quoted 8 lines
Operations that read or write payload, such as this crypto example, but also ebpf in tc or iptables, for instance, demand a deep copy using
skb_copy_ubufsbefore the operation.This blacklist approach requires caution, but these paths should be few and countable. It is not possible to predict at the socket layer whether a packet will encounter any such operation, so white-listing a subset of end-to-end paths is not practical.
How about hardware that malfunctions if the packet changes out from under it? A whitelist seems quite a bit safer.
--Andy
David Miller
maintainerFrom: Andy Lutomirski <[email protected]>
Date: Tue, 28 Feb 2017 13:06:49 -0800
Quoted 29 lines
Quoted 27 lines · Willem de Bruijn
Quoted 14 lines · Andy Lutomirski
Quoted 9 lines · Michael Kerrisk
[CC += [email protected]]
Hi Willem
Quoted 4 lines
On a send call with
MSG_ZEROCOPY, the kernel pins the user pages and creates skbuff fragments directly from these pages. On tx completion, it notifies the socket owner that it is safe to modify memory by queuing a completion notification onto the socket error queue.What happens if the user writes to the pages while it's not safe?
Quoted 7 lines
How about if you're writing to an interface or a route that has crypto involved and a malicious user can make the data change in the middle of a crypto operation, thus perhaps leaking the entire key? (I wouldn't be at all surprised if a lot of provably secure AEAD constructions are entirely compromised if an attacker can get the ciphertext and tag computed from a message that changed during the computation.
Operations that read or write payload, such as this crypto example, but also ebpf in tc or iptables, for instance, demand a deep copy using
skb_copy_ubufsbefore the operation.
Quoted 7 lines
Quoted 4 lines
This blacklist approach requires caution, but these paths should be few and countable. It is not possible to predict at the socket layer whether a packet will encounter any such operation, so white-listing a subset of end-to-end paths is not practical.
How about hardware that malfunctions if the packet changes out from under it? A whitelist seems quite a bit safer.
These device are already choking, because as I stated this can already be done via sendfile().
Networking card wise this isn't an issue, chips bring the entire packet into their FIFO, compute checksums on the fly mid-stream, and then write the 16-bit checksum field before starting to write the packet onto the wire.
I think this is completely a non-issue, and we thought about this right from the start when sendfile() support was added nearly two decades ago. If network cards from back then didn't crap out in this situation I think the ones out there now are probably ok.
Quoted 8 lines · David Miller
These device are already choking, because as I stated this can already be done via
sendfile().Networking card wise this isn't an issue, chips bring the entire packet into their FIFO, compute checksums on the fly mid-stream, and then write the 16-bit checksum field before starting to write the packet onto the wire.
Quoted 4 lines
I think this is completely a non-issue, and we thought about this right from the start when
sendfile()support was added nearly two decades ago. If network cards from back then didn't crap out in this situation I think the ones out there now are probably ok.
Well, we had to fix one issue with GSO fall back about 4 years ago.
So extra scrutiny would be nice.
Zero copy is incredibly hard to get right.
Quoted 48 lines · David Miller
From: Andy Lutomirski <[email protected]>
Date: Tue, 28 Feb 2017 13:06:49 -0800Quoted 29 lines
Quoted 27 lines · Willem de Bruijn
Quoted 14 lines · Andy Lutomirski
Quoted 9 lines · Michael Kerrisk
[CC += [email protected]]
Hi Willem
Quoted 4 lines
On a send call with
MSG_ZEROCOPY, the kernel pins the user pages and creates skbuff fragments directly from these pages. On tx completion, it notifies the socket owner that it is safe to modify memory by queuing a completion notification onto the socket error queue.What happens if the user writes to the pages while it's not safe?
Quoted 7 lines
How about if you're writing to an interface or a route that has crypto involved and a malicious user can make the data change in the middle of a crypto operation, thus perhaps leaking the entire key? (I wouldn't be at all surprised if a lot of provably secure AEAD constructions are entirely compromised if an attacker can get the ciphertext and tag computed from a message that changed during the computation.
Operations that read or write payload, such as this crypto example, but also ebpf in tc or iptables, for instance, demand a deep copy using
skb_copy_ubufsbefore the operation.Quoted 7 lines
Quoted 4 lines
This blacklist approach requires caution, but these paths should be few and countable. It is not possible to predict at the socket layer whether a packet will encounter any such operation, so white-listing a subset of end-to-end paths is not practical.
How about hardware that malfunctions if the packet changes out from under it? A whitelist seems quite a bit safer.
These device are already choking, because as I stated this can already be done via
sendfile().Networking card wise this isn't an issue, chips bring the entire packet into their FIFO, compute checksums on the fly mid-stream, and then write the 16-bit checksum field before starting to write the packet onto the wire.
Quoted 4 lines
I think this is completely a non-issue, and we thought about this right from the start when
sendfile()support was added nearly two decades ago. If network cards from back then didn't crap out in this situation I think the ones out there now are probably ok.
Fair enough.
Quoted 11 lines · Willem de Bruijn
Quoted 5 lines
I can see this working if you have a special type of skb that indicates that the data might be concurrently written and have all the normal skb APIs (including, especially, anything that clones it) make a copy first.
Support for cloned skbs is required for TCP, both at
tcp_transmit_skband segmentation offload. Patch 4 especially adds reference counting of shared pages across clones and othersk_buffoperations likepskb_expand_head. This still allows for deep copy (skb_copy_ubufs) on clones in specific datapaths like the above.
Does this mean that a user program that does a zerocopy send can cause a retransmitted segment to contain different data than the original segment? If so, is that okay?
Willem de Bruijn
maintainerQuoted 11 lines
Quoted 10 lines
Quoted 4 lines
I can see this working if you have a special type of skb that indicates that the data might be concurrently written and have all the normal skb APIs (including, especially, anything that clones it) make a copy first.
Support for cloned skbs is required for TCP, both at
tcp_transmit_skband segmentation offload. Patch 4 especially adds reference counting of shared pages across clones and othersk_buffoperations likepskb_expand_head. This still allows for deep copy (skb_copy_ubufs) on clones in specific datapaths like the above.
Quoted 3 lines
Does this mean that a user program that does a zerocopy send can cause a retransmitted segment to contain different data than the original segment? If so, is that okay?
That is possible, indeed. The bytestream at the receiver is then likely undefined. Though integrity of the tcp receive stack should not be affected. A valid question is whether the stack should protect against such users. The pattern is reminiscent of evasion attacks. In the limited case, privileged users already can generate this data pattern, of course.
Quoted 3 lines · Andy Lutomirski
Does this mean that a user program that does a zerocopy send can cause a retransmitted segment to contain different data than the original segment? If so, is that okay?
Same remark applies to sendfile() already, or other zero copy modes (vmsplice() + splice() )
A message that is not in the archive (sent off-list or not yet received): 1488318476.9415.270.camel-XN9IlZ5yJG9HTL0Zs8A6p+yfmBU6pStAUsxypvmhUTTZJqsBc5GL+g@public.gmane.org
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Andy Lutomirski
Does this mean that a user program that does a zerocopy send can cause a retransmitted segment to contain different data than the original segment? If so, is that okay?
Same remark applies to
sendfile()already
True.
Quoted 2 lines
, or other zero copy modes
(vmsplice()+splice())
I hate vmsplice(). I thought I remembered it being essentially disabled at some point due to security problems.
Quoted 1 line
A message that is not in the archive (sent off-list or not yet received): CALCETrVQj1AEsLEGGkWW1zApGz6_x2rDmE0wz4ft+O5h07f_Ug-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 9 lines · Andy Lutomirski
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Andy Lutomirski
Does this mean that a user program that does a zerocopy send can cause a retransmitted segment to contain different data than the original segment? If so, is that okay?
Same remark applies to
sendfile()already
Quoted 7 lines
True.
Quoted 2 lines
, or other zero copy modes
(vmsplice()+splice())I hate
vmsplice(). I thought I remembered it being essentially disabled at some point due to security problems.
Right, zero copy is hard ;)
vmsplice() is not disabled in current kernels, unless I missed something.
Quoted 15 lines · Eric Dumazet
Quoted 14 lines · Andy Lutomirski
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Andy Lutomirski
Does this mean that a user program that does a zerocopy send can cause a retransmitted segment to contain different data than the original segment? If so, is that okay?
Same remark applies to
sendfile()alreadyTrue.
Quoted 2 lines
, or other zero copy modes
(vmsplice()+splice())
Quoted 8 lines
Quoted 2 lines
I hate
vmsplice(). I thought I remembered it being essentially disabled at some point due to security problems.Right, zero copy is hard ;)
vmsplice()is not disabled in current kernels, unless I missed something.
I think you're right. That being said, from the man page:
The user pages are a gift to the kernel. The application may not modify this memory ever, otherwise the page cache and on-disk data may differ.
This is just not okay IMO.
Quoted 5 lines · Andy Lutomirski
The user pages are a gift to the kernel. The application may not modify this memory ever, otherwise the page cache and on-disk data may differ.
This is just not okay IMO.
TCP works just fine in this case.
TX checksum will be computed by the NIC after/while data is copied.
If really the application changes the data, that will not cause any problems, other than user side consistency.
This is why we require a copy (for all buffers that came from zero-copy) if network stack hits a device that can not offload TX checksum.
Even pwrite() does not guarantee consistency if multiple threads are using it on overlapping regions.
A message that is not in the archive (sent off-list or not yet received): 1488324131.9415.278.camel-XN9IlZ5yJG9HTL0Zs8A6p+yfmBU6pStAUsxypvmhUTTZJqsBc5GL+g@public.gmane.org
Quoted 12 lines · Eric Dumazet
Quoted 5 lines · Andy Lutomirski
The user pages are a gift to the kernel. The application may not modify this memory ever, otherwise the page cache and on-disk data may differ.
This is just not okay IMO.
TCP works just fine in this case.
TX checksum will be computed by the NIC after/while data is copied.
Quoted 9 lines
If really the application changes the data, that will not cause any problems, other than user side consistency.
This is why we require a copy (for all buffers that came from zero-copy) if network stack hits a device that can not offload TX checksum.
Even
pwrite()does not guarantee consistency if multiple threads are using it on overlapping regions.
The Mellanox team working on TLS offload pointed out to us that if data is changed for a retransmit then it becomes trivial for someone snooping to break the encryption. Sounds pretty scary and it would be a shame if we couldn't use zero-copy in that use case :-( Hopefully we can find a solution...
Tom
Quoted 2 lines
A message that is not in the archive (sent off-list or not yet received): CALx6S357ssnbEu7CMrczEjiX25QYBJh3WG=w8KuAoxGQS4aKLA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 5 lines · Tom Herbert
The Mellanox team working on TLS offload pointed out to us that if data is changed for a retransmit then it becomes trivial for someone snooping to break the encryption. Sounds pretty scary and it would be a shame if we couldn't use zero-copy in that use case :-( Hopefully we can find a solution...
Right, this is why offloading encryption over TCP is also hard ;)
Quoted 19 lines · Tom Herbert
Quoted 18 lines · Eric Dumazet
Quoted 5 lines · Andy Lutomirski
The user pages are a gift to the kernel. The application may not modify this memory ever, otherwise the page cache and on-disk data may differ.
This is just not okay IMO.
TCP works just fine in this case.
TX checksum will be computed by the NIC after/while data is copied.
If really the application changes the data, that will not cause any problems, other than user side consistency.
This is why we require a copy (for all buffers that came from zero-copy) if network stack hits a device that can not offload TX checksum.
Quoted 9 lines
Quoted 3 lines
Even
pwrite()does not guarantee consistency if multiple threads are using it on overlapping regions.The Mellanox team working on TLS offload pointed out to us that if data is changed for a retransmit then it becomes trivial for someone snooping to break the encryption. Sounds pretty scary and it would be a shame if we couldn't use zero-copy in that use case :-( Hopefully we can find a solution...
This requires collusion by the process initiating the zerocopy send to help the entity snooping the link. That could be an attack on admin configured tunnels, but user-directed encryption offload like AF_TLS can still use zerocopy.
Quoted 30 lines · Willem de Bruijn
Quoted 19 lines · Tom Herbert
Quoted 18 lines · Eric Dumazet
Quoted 5 lines · Andy Lutomirski
The user pages are a gift to the kernel. The application may not modify this memory ever, otherwise the page cache and on-disk data may differ.
This is just not okay IMO.
TCP works just fine in this case.
TX checksum will be computed by the NIC after/while data is copied.
If really the application changes the data, that will not cause any problems, other than user side consistency.
This is why we require a copy (for all buffers that came from zero-copy) if network stack hits a device that can not offload TX checksum.
Quoted 9 lines
Quoted 3 lines
Even
pwrite()does not guarantee consistency if multiple threads are using it on overlapping regions.The Mellanox team working on TLS offload pointed out to us that if data is changed for a retransmit then it becomes trivial for someone snooping to break the encryption. Sounds pretty scary and it would be a shame if we couldn't use zero-copy in that use case :-( Hopefully we can find a solution...
Quoted 4 lines
This requires collusion by the process initiating the zerocopy send to help the entity snooping the link. That could be an attack on admin configured tunnels, but user-directed encryption offload like
AF_TLScan still use zerocopy.
Yes, but we can't trust the user to always understand or correctly implement the semantic nuances when security is involved. If we can't provide a robust API then the only recourse is to not allow zero copy in that case. We could suggest COW to solve all problems, but I think we know where the conversation will go ;-)
Tom
David Miller
maintainerFrom: Andy Lutomirski <[email protected]>
Date: Tue, 28 Feb 2017 11:46:23 -0800
Quoted 7 lines
Quoted 4 lines · Michael Kerrisk
[CC += [email protected]]
Hi Willem
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
On a send call with
MSG_ZEROCOPY, the kernel pins the user pages and creates skbuff fragments directly from these pages. On tx completion, it notifies the socket owner that it is safe to modify memory by queuing a completion notification onto the socket error queue.What happens if the user writes to the pages while it's not safe?
Just want to mention that this ability to write to data behind a network send's back is not a new thing added by MSG_ZEROCOPY.
All of this is already possible with sendfile(). The pages can be written to completely asynchronously to the data being pulled out of the page cache into the transmit path.
The crypto case is interesting, but that is a seperate discussion about an existing problem rather than something specifically new to the MSG_ZEROCOPY changes.
Thanks.