mptcp: add MSG_ZEROCOPY support
Subsystems (3)
Geliang Tang
reviewerv5 cover mptcp: add MSG_ZEROCOPY support
From: Geliang Tang <[email protected]>
This series adds MSG_ZEROCOPY support for MPTCP sockets, allowing userspace to transmit data without intermediate kernel copies, and provides corresponding selftests to exercise the new path.
Patch 1, patch 2 and patch 3 are small cleanups along the mptcp_sendmsg path.
Patch 4 implements the core MSG_ZEROCOPY support. The design follows TCP's reference-counting model, with a single ubuf_info per sendmsg, tracked by three reference buckets: one held by sendmsg itself, one per MPTCP dfrag in the retransmission queue, and one per subflow SKB. Completion is reported only after all bytes are acknowledged at both the MPTCP and subflow levels. The feature is silently downgraded to a regular copy in cases where zero-copy cannot be safely performed (MSG_FASTOPEN, fallback mode, or memory pressure).
Patch 5 handles SO_ZEROCOPY in setsockopt.
Patch 6 and patch 7 add a new 'zerocopy' I/O mode to the mptcp_connect selftest, along with a wrapper script to run it as part of the regular test suite, ensuring the zero-copy path is exercised in CI.
Patch 8 is a small cleanup for mptcp_connect.c in the selftests.
v5:
- A new patch to align struct mptcp_data_frag to 8 bytes on 32-bit.
- Prevent coalescing of dfrags with different ubuf instances into the same subflow skb to avoid premature completion notifications.
- Return -EFAULT instead of -EAGAIN when iov_iter_get_pages2() returns 0 to properly signal iterator exhaustion.
- Account for dfrag metadata overhead in sk_wmem_queued_add() to prevent bypassing socket memory limits.
- Widen data_len field from u16 to u32 to prevent truncation on 64KB-page kernels (ARM64, PPC64).
- In selftests, use batched polling (200ms intervals with 5s total timeout) to drain completion notifications for large transfers that may generate multiple completion events.
v4:
- two more cleanups, patch 2 and patch 7.
- net/mptcp/protocol.c, mptcp_sendmsg_frag(): pull the type-mismatch
check out of the if (can_coalesce) block and use the standard
skb_zcopy_pure() helper; mptcp_sendmsg_zerocopy_iter() now
calls iov_iter_revert() before put_page() in the kzalloc failure
path so a retry starts from the same iterator offset.
- tools/testing/selftests/net/mptcp/mptcp_connect.c, copyfd_io_zc():
reset msg_controllen each iteration of the inner reap loop using a
do {} while block; use >= instead of > in the MPTFO size
adjustment so file_size reaches 0 when the whole file is sent via
fast open.
- https://patchwork.kernel.org/project/mptcp/cover/[email protected]/
v3:
- Force fresh skb when mixing zerocopy and kernel-copy fragments.
- Simplify zero-copy availability check by using SOCK_ZEROCOPY flag
directly.
- Validate SO_ZEROCOPY value in setsockopt and avoid affecting
fallback subflow.
- In selftests, drain MPTFO partial data, use poll+recvmsg for completion notifications with timeout, and verify getsockopt round-trip.
- https://patchwork.kernel.org/project/mptcp/cover/[email protected]/
v2:
- patch 2, never mix PURE_ZEROCOPY frags with kernel-copy frags in the
same skb; handle fallback, return values.
- patch 3, a new patch to handle SO_ZEROCOPY in setsockopt.
- patch 4, set SO_ZEROCOPY, handle listen_mode, error queue.
- https://patchwork.kernel.org/project/mptcp/cover/[email protected]/
v1:
- https://patchwork.kernel.org/project/mptcp/cover/[email protected]/
Geliang Tang (8):
mptcp: use local variable tp in sendmsg_frag
mptcp: align struct mptcp_data_frag to 8 bytes on 32-bit
mptcp: remove redundant orig_offset in carve_data_frag
mptcp: add MSG_ZEROCOPY support
mptcp: handle SO_ZEROCOPY in setsockopt
selftests: mptcp: connect: add zerocopy io mode
selftests: mptcp: connect: cover zerocopy mode
selftests: mptcp: connect: close listensock deterministically
net/mptcp/protocol.c | 134 ++++++++++++++-- net/mptcp/protocol.h | 3 +- net/mptcp/sockopt.c | 17 +- tools/testing/selftests/net/mptcp/Makefile | 1 + .../selftests/net/mptcp/mptcp_connect.c | 146 +++++++++++++++++- .../net/mptcp/mptcp_connect_zerocopy.sh | 5 + 6 files changed, 280 insertions(+), 26 deletions(-) create mode 100755 tools/testing/selftests/net/mptcp/mptcp_connect_zerocopy.sh
Signature · 1 line
2.53.0
Geliang Tang
reviewerPATCH v5 1/8 mptcp: use local variable tp in sendmsg_frag
From: Geliang Tang <[email protected]>
In mptcp_sendmsg_frag(), tcp_sk(ssk) is called multiple times. Introduce a local variable 'tp' to hold the tcp_sock pointer and use it throughout the function. This avoids repeated dereferencing and improves code readability, with no functional change.
--- net/mptcp/protocol.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
net/mptcp/protocol.c +4 -3
@@ -1375,6 +1375,7 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,u64 data_seq = dfrag->data_seq + info->sent;int offset = dfrag->offset + info->sent;struct mptcp_sock *msk =mptcp_sk (sk); +struct tcp_sock *tp =tcp_sk (ssk);bool zero_window_probe = false;struct mptcp_ext *mpext =NULL ;bool can_coalesce = false;
@@ -1410,14 +1411,14 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk, mpext =mptcp_get_ext (skb);if (!mptcp_skb_can_collapse_to (data_seq, skb, mpext)) {TCP_SKB_CB (skb)->eor =1 ; -tcp_mark_push (tcp_sk (ssk), skb); +tcp_mark_push (tp, skb); goto alloc_skb; } i =skb_shinfo (skb)->nr_frags; can_coalesce =skb_can_coalesce (skb, i, dfrag->page, offset);if (!can_coalesce && i >=READ_ONCE (net_hotdata.sysctl_max_skb_frags)) { -tcp_mark_push (tcp_sk (ssk), skb); +tcp_mark_push (tp, skb); goto alloc_skb; }
@@ -1471,7 +1472,7 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk, skb->truesize += copy;sk_wmem_queued_add (ssk, copy);sk_mem_charge (ssk, copy); -WRITE_ONCE (tcp_sk (ssk)->write_seq,tcp_sk (ssk)->write_seq + copy); +WRITE_ONCE (tp->write_seq, tp->write_seq + copy);TCP_SKB_CB (skb)->end_seq += copy;tcp_skb_pcount_set (skb,0 );
Signature · 1 line
2.53.0
Geliang Tang
reviewerPATCH v5 2/8 mptcp: align struct mptcp_data_frag to 8 bytes on 32-bit
In reply to Geliang TangFrom: Geliang Tang <[email protected]>
mptcp_carve_data_frag() places a struct mptcp_data_frag inside a page-frag region, using ALIGN(orig_offset, sizeof(long)) to compute the offset where the struct lives. On 64-bit Linux sizeof(long) is 8 and this works as intended. On 32-bit Linux sizeof(long) is 4, which only guarantees 4-byte alignment for the struct.
struct mptcp_data_frag contains a u64 data_seq member, which requires 8-byte alignment. 4-byte-aligned 64-bit loads fault on 32-bit architectures that do not handle unaligned 64-bit accesses in hardware (armv7, mips, sparc, ...). The in-source comment near BUILD_BUG_ON that states "ALIGN(1, sizeof(long)) - 1, so 8-1" is misleading because it implicitly assumes sizeof(long) is always 8.
Switch both the ALIGN() and the matching BUILD_BUG_ON to sizeof(u64) so the 8-byte alignment is enforced on every architecture, and the overhead u8 budget still fits the worst-case 7-byte padding plus the struct size. This is a no-op on 64-bit and a correctness fix on 32-bit.
--- net/mptcp/protocol.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
net/mptcp/protocol.c +3 -3
@@ -1237,7 +1237,7 @@ static struct mptcp_data_frag * mptcp_carve_data_frag(const struct mptcp_sock *msk, struct page_frag *pfrag, int orig_offset) { - int offset = ALIGN(orig_offset, sizeof(long)); + int offset = ALIGN(orig_offset, sizeof(u64));struct mptcp_data_frag *dfrag; dfrag = (struct mptcp_data_frag *)(page_to_virt (pfrag->page) + offset);
@@ -4789,9 +4789,9 @@ void __init mptcp_proto_init(void) BUILD_BUG_ON(sizeof(struct mptcp_skb_cb) > sizeof_field(struct sk_buff, cb));/* struct mptcp_data_frag: 'overhead' corresponds to the alignment -* (ALIGN(1, sizeof(long)) - 1, so 8-1) + the struct's size +* (ALIGN(1, sizeof(u64)) - 1, so 8-1) + the struct's size */ -BUILD_BUG_ON (ALIGN (1 ,sizeof (long )) -1 +sizeof (struct mptcp_data_frag ) +BUILD_BUG_ON (ALIGN (1 ,sizeof (u64)) -1 +sizeof (struct mptcp_data_frag ) >U8_MAX ); }
Signature · 1 line
2.53.0
Geliang Tang
reviewerPATCH v5 3/8 mptcp: remove redundant orig_offset in carve_data_frag
In reply to Geliang TangFrom: Geliang Tang <[email protected]>
mptcp_carve_data_frag() already receives struct page_frag via its 'pfrag' parameter, making the separate 'orig_offset' parameter redundant since it merely duplicates pfrag->offset.
Simplify the function signature by removing this unnecessary parameter and directly accessing the offset from struct pfrag internally.
--- net/mptcp/protocol.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
net/mptcp/protocol.c +5 -4
@@ -1234,12 +1234,13 @@ static bool mptcp_page_frag_refill(struct sock *sk, struct page_frag *pfrag) }static struct mptcp_data_frag * -mptcp_carve_data_frag (const struct mptcp_sock *msk,struct page_frag *pfrag, -int orig_offset) +mptcp_carve_data_frag (const struct mptcp_sock *msk,struct page_frag *pfrag) { -int offset =ALIGN (orig_offset,sizeof (u64)); +int orig_offset = pfrag->offset;struct mptcp_data_frag *dfrag; +int offset; + offset =ALIGN (orig_offset,sizeof (u64)); dfrag = (struct mptcp_data_frag *)(page_to_virt (pfrag->page) + offset); dfrag->data_len =0 ; dfrag->data_seq = msk->write_seq;
@@ -2031,7 +2032,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)if (!mptcp_page_frag_refill (sk, pfrag)) goto wait_for_memory; - dfrag =mptcp_carve_data_frag (msk, pfrag, pfrag->offset); + dfrag =mptcp_carve_data_frag (msk, pfrag); frag_truesize = dfrag->overhead; }
Signature · 1 line
2.53.0
Geliang Tang
reviewerPATCH v5 4/8 mptcp: add MSG_ZEROCOPY support
In reply to Geliang TangFrom: Geliang Tang <[email protected]>
This patch enables MSG_ZEROCOPY support for MPTCP sockets, allowing user-space to transmit data without intermediate kernel copies.
The implementation follows the same lifecycle model as TCP:
A single ubuf_info_msgzc is allocated per sendmsg via msg_zerocopy_realloc(). Its reference count consists of three parts:
one 'caller' reference held until sendmsg returns, one reference per MPTCP dfrag in the retransmission queue (released when the dfrag is acked and cleared), and one reference per subflow skb that consumes dfrag bytes (released by TCP's skb_zcopy_clear() when the skb is acked). Completion is reported only after all bytes are acknowledged at both MPTCP and subflow levels.
User pages are pinned via iov_iter_get_pages2() and stored directly in a newly allocated mptcp_data_frag. The dfrag takes one reference on the ubuf_info and one on the page. Subflow skbs take additional page references via get_page() and are marked with skb_zcopy_set() and SKBFL_PURE_ZEROCOPY, mirroring TCP's behavior and bypassing kernel memory accounting for those pages.
The feature is silently downgraded to regular copy when MSG_FASTOPEN is set or the application has not opted in via setsockopt(SO_ZEROCOPY). If msg_zerocopy_realloc() fails (e.g. RLIMIT_MEMLOCK pressure), sendmsg(2) aborts with -ENOBUFS.
The implementation prevents coalescing of dfrags with different ubuf instances into the same subflow skb to avoid premature completion notifications. The data_len field is widened from u16 to u32 to prevent truncation on 64KB-page kernels (ARM64, PPC64). The dfrag metadata overhead is accounted for in sk_wmem_queued_add() to prevent bypassing socket memory limits. When iov_iter_get_pages2() returns 0, the function returns -EFAULT to properly signal iterator exhaustion.
--- net/mptcp/protocol.c | 114 ++++++++++++++++++++++++++++++++++++++++--- net/mptcp/protocol.h | 3 +- 2 files changed, 109 insertions(+), 8 deletions(-)
net/mptcp/protocol.c +107 -7
@@ -1118,9 +1118,11 @@ static bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk, df->data_seq + df->data_len == msk->write_seq; } -static void dfrag_uncharge (struct sock *sk,int len) +static void dfrag_uncharge (struct sock *sk,int len, +struct mptcp_data_frag *dfrag) { -sk_mem_uncharge (sk, len); +if (!dfrag->ubuf) +sk_mem_uncharge (sk, len);sk_wmem_queued_add (sk, -len); }
@@ -1129,7 +1131,13 @@ static void dfrag_clear(struct sock *sk, struct mptcp_data_frag *dfrag)int len = dfrag->data_len + dfrag->overhead;list_del (&dfrag->list); -dfrag_uncharge (sk, len); +dfrag_uncharge (sk, len, dfrag); +if (dfrag->ubuf) { +net_zcopy_put (dfrag->ubuf); +put_page (dfrag->page); +kfree (dfrag); +return ; + }put_page (dfrag->page); }
@@ -1174,7 +1182,7 @@ static void __mptcp_clean_una(struct sock *sk) dfrag->data_len -= delta; dfrag->already_sent -= delta; -dfrag_uncharge (sk, delta); +dfrag_uncharge (sk, delta, dfrag); }/* all retransmitted data acked, recovery completed */
@@ -1249,6 +1257,7 @@ mptcp_carve_data_frag(const struct mptcp_sock *msk, struct page_frag *pfrag) dfrag->already_sent =0 ; dfrag->page = pfrag->page; dfrag->eor =0 ; + dfrag->ubuf =NULL ;return dfrag; }
@@ -1418,6 +1427,17 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk, i =skb_shinfo (skb)->nr_frags; can_coalesce =skb_can_coalesce (skb, i, dfrag->page, offset); +if (skb_zcopy_pure (skb) != !!dfrag->ubuf) { + can_coalesce = false; +tcp_mark_push (tp, skb); + goto alloc_skb; + } +if (can_coalesce && dfrag->ubuf &&skb_zcopy (skb) && +skb_zcopy (skb) != dfrag->ubuf) { + can_coalesce = false; +tcp_mark_push (tp, skb); + goto alloc_skb; + }if (!can_coalesce && i >=READ_ONCE (net_hotdata.sysctl_max_skb_frags)) {tcp_mark_push (tp, skb); goto alloc_skb;
@@ -1456,7 +1476,7 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk, } copy =min_t (size_t, copy, info->limit - info->sent); -if (!sk_wmem_schedule (ssk, copy)) { +if (!dfrag->ubuf && !sk_wmem_schedule (ssk, copy)) {tcp_remove_empty_skb (ssk);return -ENOMEM ; }
@@ -1466,13 +1486,19 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk, }else {get_page (dfrag->page);skb_fill_page_desc (skb, i, dfrag->page, offset, copy); + +if (dfrag->ubuf && !skb_zcopy (skb)) { +skb_zcopy_set (skb, dfrag->ubuf,NULL ); +skb_shinfo (skb)->flags |=SKBFL_PURE_ZEROCOPY ; + } } skb->len += copy; skb->data_len += copy; skb->truesize += copy;sk_wmem_queued_add (ssk, copy); -sk_mem_charge (ssk, copy); +if (!skb_zcopy_pure (skb)) +sk_mem_charge (ssk, copy);WRITE_ONCE (tp->write_seq, tp->write_seq + copy);TCP_SKB_CB (skb)->end_seq += copy;tcp_skb_pcount_set (skb,0 );
@@ -1969,22 +1995,81 @@ static void mptcp_rps_record_subflows(const struct mptcp_sock *msk) } } +static int mptcp_sendmsg_zerocopy_iter (struct sock *sk,struct msghdr *msg, +struct ubuf_info *ubuf,u32 copy_limit, +ssize_t *copied,long *timeo) +{ +struct mptcp_sock *msk =mptcp_sk (sk); +struct mptcp_data_frag *dfrag; +struct page *pages[1 ] = {}; +size_t pg_off =0 ; +ssize_t pg_len; + + pg_len =iov_iter_get_pages2 (&msg->msg_iter, pages, + copy_limit,1 , &pg_off); +if (pg_len <0 ) +return pg_len; +/* pg_len == 0 means the iterator is exhausted */ +if (pg_len ==0 ) +return -EFAULT ; + + dfrag =kzalloc_obj (*dfrag,GFP_KERNEL_ACCOUNT ); +if (!dfrag) { +iov_iter_revert (&msg->msg_iter, pg_len); +put_page (pages[0 ]); +return -ENOMEM ; + } + dfrag->data_len = pg_len; + dfrag->data_seq = msk->write_seq; + dfrag->offset = pg_off; + dfrag->page = pages[0 ]; + dfrag->ubuf = ubuf; + dfrag->overhead =sizeof (struct mptcp_data_frag ); + +/* one ref for the dfrag; released in dfrag_clear */ +net_zcopy_get (ubuf); + + *copied += pg_len; +WRITE_ONCE (msk->write_seq, msk->write_seq + pg_len); +sk_wmem_queued_add (sk, pg_len + dfrag->overhead); + +list_add_tail (&dfrag->list, &msk->rtx_queue); +if (!msk->first_pending) + msk->first_pending = dfrag; + +return 0 ; +} +static int mptcp_sendmsg (struct sock *sk,struct msghdr *msg,size_t len) {struct mptcp_sock *msk =mptcp_sk (sk); +struct ubuf_info *ubuf =NULL ;struct page_frag *pfrag;size_t copied =0 ; +bool zc = false;int ret =0 ;long timeo;/* silently ignore everything else */ msg->msg_flags &=MSG_MORE |MSG_DONTWAIT |MSG_NOSIGNAL | -MSG_FASTOPEN |MSG_EOR ; +MSG_FASTOPEN |MSG_EOR |MSG_ZEROCOPY ;lock_sock (sk);mptcp_rps_record_subflows (msk); +if (msg->msg_flags &MSG_FASTOPEN || !sock_flag (sk,SOCK_ZEROCOPY )) + msg->msg_flags &= ~MSG_ZEROCOPY ; + +if ((msg->msg_flags &MSG_ZEROCOPY ) && len) { + ubuf =msg_zerocopy_realloc (sk, len,NULL , false); +if (!ubuf) { + ret = -ENOBUFS ; + goto do_error; + } + zc = true; + } +if (unlikely (inet_test_bit (DEFER_CONNECT , sk) || msg->msg_flags &MSG_FASTOPEN )) {int copied_syn =0 ;
@@ -2023,6 +2108,15 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)if (!copy_limit) goto wait_for_memory; +if (zc) { + ret =mptcp_sendmsg_zerocopy_iter (sk, msg, + ubuf, copy_limit, + &copied, &timeo); +if (ret) + goto do_error; +continue ; + } +/* reuse tail pfrag, if possible, or carve a new one from the * page allocator */
@@ -2098,6 +2192,8 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) } out: +if (zc) +net_zcopy_put (ubuf);release_sock (sk);return copied;
@@ -2106,6 +2202,10 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) goto out; copied =sk_stream_error (sk, msg->msg_flags, ret); +if (zc) { +net_zcopy_put_abort (ubuf, true); + zc = false; + } goto out; }
net/mptcp/protocol.h +2 -1
@@ -262,12 +262,13 @@ struct mptcp_pm_addr_entry {struct mptcp_data_frag {struct list_head list;u64 data_seq; -u16 data_len; +u32 data_len;/* u16 truncates on 64KB-page kernels */ u16 offset;u8 overhead;u8 eor;/* currently using 1 bit */ u16 already_sent;struct page *page; +struct ubuf_info *ubuf; };/* Arbitrary compromise between as low as possible to react timely to subflow
Signature · 1 line
2.53.0
Geliang Tang
reviewerPATCH v5 5/8 mptcp: handle SO_ZEROCOPY in setsockopt
In reply to Geliang TangFrom: Geliang Tang <[email protected]>
SO_ZEROCOPY is currently rejected by MPTCP setsockopt with -EOPNOTSUPP, even though the matching MSG_ZEROCOPY path in mptcp_sendmsg() is wired up. As a result MSG_ZEROCOPY silently falls back to a copy on MPTCP sockets because the opt-in flag is never set.
Expose SO_ZEROCOPY through the SOL_SOCKET setsockopt dispatcher. The new mptcp_setsockopt_sol_socket_zerocopy() helper validates that val is 0 or 1 (matching standard TCP sk_setsockopt) and flips SOCK_ZEROCOPY on the msk only. mptcp_sendmsg() consults the same flag directly, so opt-in is honoured on every data path without any cross-socket propagation.
--- net/mptcp/sockopt.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
net/mptcp/sockopt.c +16 -1
@@ -175,6 +175,19 @@ static int mptcp_setsockopt_sol_socket_tstamp(struct mptcp_sock *msk, int optnamreturn 0 ; } +static int mptcp_setsockopt_sol_socket_zerocopy (struct mptcp_sock *msk,int val) +{ +struct sock *sk = (struct sock *)msk; + +if (val !=0 && val !=1 ) +return -EINVAL ; + +lock_sock (sk); +sock_valbool_flag (sk,SOCK_ZEROCOPY , val); +release_sock (sk); +return 0 ; +} +static int mptcp_setsockopt_sol_socket_int (struct mptcp_sock *msk,int optname,sockptr_t optval,unsigned int optlen)
@@ -203,6 +216,8 @@ static int mptcp_setsockopt_sol_socket_int(struct mptcp_sock *msk, int optname,case SO_TIMESTAMPNS_OLD :case SO_TIMESTAMPNS_NEW :return mptcp_setsockopt_sol_socket_tstamp (msk, optname, val); +case SO_ZEROCOPY : +return mptcp_setsockopt_sol_socket_zerocopy (msk, val); }return -ENOPROTOOPT ;
@@ -342,6 +357,7 @@ static int mptcp_setsockopt_sol_socket(struct mptcp_sock *msk, int optname,case SO_TIMESTAMP_NEW :case SO_TIMESTAMPNS_OLD :case SO_TIMESTAMPNS_NEW : +case SO_ZEROCOPY :return mptcp_setsockopt_sol_socket_int (msk, optname, optval, optlen);case SO_TIMESTAMPING_OLD :
@@ -387,7 +403,6 @@ static int mptcp_setsockopt_sol_socket(struct mptcp_sock *msk, int optname,* SO_CNX_ADVICE is currently unsupported, could possibly be relevant, * but likely needs careful design * -* SO_ZEROCOPY is currently unsupported, TODO in sndmsg * SO_TXTIME is currently unsupported */
Signature · 1 line
2.53.0
Geliang Tang
reviewerPATCH v5 6/8 selftests: mptcp: connect: add zerocopy io mode
In reply to Geliang TangFrom: Geliang Tang <[email protected]>
Add a new I/O mode 'zerocopy' to the mptcp_connect selftest, which uses sendmsg() with the MSG_ZEROCOPY flag to transmit data. This enables testing and performance validation of the zero-copy send path for both TCP and MPTCP connections.
This mode is useful for benchmarking and for verifying correctness of MSG_ZEROCOPY handling in the MPTCP stack.
The completion notification draining uses batched polling to handle large transfers that may generate multiple completion events across different TCP segments.
--- .../selftests/net/mptcp/mptcp_connect.c | 122 +++++++++++++++++- 1 file changed, 121 insertions(+), 1 deletion(-)
tools/testing/selftests/net/mptcp/mptcp_connect.c +121 -1
@@ -33,6 +33,7 @@#include <linux/tcp.h> #include <linux/time_types.h> #include <linux/sockios.h> +#include <linux/errqueue.h> #include <linux/compiler.h> extern int optind;
@@ -53,6 +54,7 @@ enum cfg_mode {CFG_MODE_MMAP ,CFG_MODE_SENDFILE ,CFG_MODE_SPLICE , + CFG_MODE_ZEROCOPY, }; enum cfg_peek {
@@ -125,7 +127,7 @@ static void die_usage(void)fprintf (stderr,"\t-j -- add additional sleep at connection start and tear down " "-- for MPJ tests\n" );fprintf (stderr,"\t-l -- listens mode, accepts incoming connection\n" ); -fprintf (stderr,"\t-m [poll|mmap|sendfile|splice] -- use poll(default)/mmap+write/sendfile/splice\n" ); +fprintf (stderr,"\t-m [poll|mmap|sendfile|splice|zerocopy] -- use poll(default)/mmap+write/sendfile/splice/zerocopy\n" );fprintf (stderr,"\t-M mark -- set socket packet mark\n" );fprintf (stderr,"\t-o option -- test sockopt <option>\n" );fprintf (stderr,"\t-p num -- use port num\n" );
@@ -998,6 +1000,110 @@ static int copyfd_io_splice(int infd, int peerfd, int outfd, unsigned int size,return err; } +static int copyfd_io_zc (int infd,int peerfd,int outfd,unsigned int size, +bool *in_closed_after_out,struct wstate *winfo) +{ +size_t ctl_len =CMSG_SPACE (sizeof (struct sock_extended_err )); +struct pollfd pfd = { .fd = peerfd, .events =POLLERR }; +struct msghdr errmsg = {}, msg = {}; +int sndbuf = size *2 ; +int total_ms =5000 ; +int batch_ms =200 ; +struct iovec iov; +char *buf, *ctl; +socklen_t len; +int on =1 ; +int err; + +if (spool_buf (peerfd, winfo) <0 ) { +perror ("spool_buf" ); +return 1 ; + } + + ctl =malloc (ctl_len); +if (!ctl) { +perror ("malloc ctl" ); +return 1 ; + } + + errmsg.msg_control = ctl; + errmsg.msg_controllen = ctl_len; + + buf =malloc (size); +if (!buf) { +perror ("malloc" ); +free (ctl); +return 1 ; + } + +if (read (infd, buf, size) != size) { +perror ("read input" ); + err =1 ; + goto free_bufs; + } + + iov.iov_base = buf; + iov.iov_len = size; + msg.msg_iov = &iov; + msg.msg_iovlen =1 ; + +if (setsockopt (peerfd,SOL_SOCKET ,SO_SNDBUF , &sndbuf,sizeof (sndbuf))) +perror ("setsockopt SO_SNDBUF" ); + + len =sizeof (on); +if (setsockopt (peerfd,SOL_SOCKET ,SO_ZEROCOPY , &on, len)) { +perror ("setsockopt SO_ZEROCOPY" ); + err =1 ; + goto free_bufs; + } + + on =-1 ; +if (getsockopt (peerfd,SOL_SOCKET ,SO_ZEROCOPY , &on, &len) || + on !=1 ) { +perror ("getsockopt SO_ZEROCOPY" ); + err =1 ; + goto free_bufs; + } + +if (listen_mode) { +if (do_recvfile (peerfd, outfd) <0 ) { + err =1 ; + goto free_bufs; + } + } + + err =sendmsg (peerfd, &msg,MSG_ZEROCOPY ); +if (err <0 || (unsigned int )err != size) { +perror ("sendmsg MSG_ZEROCOPY" ); + err =1 ; + goto free_bufs; + } + err =0 ; + +if (!listen_mode) { +shut_wr (peerfd); +if (do_recvfile (peerfd, outfd) <0 ) + err =1 ; + *in_closed_after_out = true; + } + +while (total_ms >0 ) { +int n =poll (&pfd,1 , batch_ms); + +if (n <=0 ) +break ; +do { + errmsg.msg_controllen = ctl_len; + }while (recvmsg (peerfd, &errmsg,MSG_ERRQUEUE ) >=0 ); + total_ms -= batch_ms; + } + +free_bufs: +free (buf); +free (ctl); +return err; +} +static int copyfd_io (int infd,int peerfd,int outfd,bool close_peerfd,struct wstate *winfo) {bool in_closed_after_out = false;
@@ -1038,6 +1144,17 @@ static int copyfd_io(int infd, int peerfd, int outfd, bool close_peerfd, struct &in_closed_after_out, winfo);break ; +case CFG_MODE_ZEROCOPY : + file_size =get_infd_size (infd); +if (file_size <0 ) +return file_size; +if (cfg_sockopt_types.mptfo && winfo->total_len && + file_size >= winfo->total_len) + file_size -= winfo->total_len; + ret =copyfd_io_zc (infd, peerfd, outfd, file_size, + &in_closed_after_out, winfo); +break ; +default :fprintf (stderr,"Invalid mode %d\n" , cfg_mode);
@@ -1453,6 +1570,8 @@ int parse_mode(const char *mode)return CFG_MODE_SENDFILE ;if (!strcasecmp (mode,"splice" ))return CFG_MODE_SPLICE ; +if (!strcasecmp (mode,"zerocopy" )) +return CFG_MODE_ZEROCOPY ;fprintf (stderr,"Unknown test mode: %s\n" , mode);fprintf (stderr,"Supported modes are:\n" );
@@ -1460,6 +1579,7 @@ int parse_mode(const char *mode)fprintf (stderr,"\t\t\"mmap\" - send entire input file (mmap+write), then read response (-l will read input first)\n" );fprintf (stderr,"\t\t\"sendfile\" - send entire input file (sendfile), then read response (-l will read input first)\n" );fprintf (stderr,"\t\t\"splice\" - send entire input file (splice), then read response (-l will read input first)\n" ); +fprintf (stderr,"\t\t\"zerocopy\" - send entire input file (zerocopy), then read response (-l will read input first)\n" );die_usage ();
Signature · 1 line
2.53.0
Geliang Tang
reviewerPATCH v5 7/8 selftests: mptcp: connect: cover zerocopy mode
In reply to Geliang TangFrom: Geliang Tang <[email protected]>
Add a new test script mptcp_connect_zerocopy.sh to run the mptcp_connect selftest with the zerocopy I/O mode. This ensures the MSG_ZEROCOPY path is exercised in the regular test suite.
The script follows the same pattern as the existing mmap, sendfile, and splice test wrappers, invoking mptcp_connect.sh with the -m zerocopy option.
--- tools/testing/selftests/net/mptcp/Makefile | 1 + tools/testing/selftests/net/mptcp/mptcp_connect_zerocopy.sh | 5 +++++ 2 files changed, 6 insertions(+) create mode 100755 tools/testing/selftests/net/mptcp/mptcp_connect_zerocopy.sh
tools/testing/selftests/net/mptcp/Makefile +1 -0
@@ -12,6 +12,7 @@ TEST_PROGS := \ mptcp_connect_mmap.sh \ mptcp_connect_sendfile.sh \ mptcp_connect_splice.sh \ + mptcp_connect_zerocopy.sh \ mptcp_join.sh \ mptcp_sockopt.sh \ pm_netlink.sh \
tools/testing/selftests/net/mptcp/mptcp_connect_zerocopy.sh new file +5 -0
@@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +MPTCP_LIB_KSFT_TEST="$(basename "${0}" .sh)" \ + "$(dirname "${0}")/mptcp_connect.sh" -m zerocopy "${@}"
Signature · 1 line
2.53.0
Geliang Tang
reviewerPATCH v5 8/8 selftests: mptcp: connect: close listensock deterministically
In reply to Geliang TangFrom: Geliang Tang <[email protected]>
In main_loop_s(), listensock was closed nondeterministically inside maybe_close() during the accept loop, and error paths closed it without tracking, leading to double-close or leaks across repeated iterations (via goto again).
Fix by making maybe_close() return a bool indicating whether it closed the fd, and track that per iteration. Funnel poll/accept error paths through a single 'goto out' label and close the socket there only if not already closed. Reset 'closed' at the 'again:' label so each iteration starts with a fresh tracking state. This ensures exactly one close per iteration in all cases.
--- .../selftests/net/mptcp/mptcp_connect.c | 24 +++++++++++++------ 1 file changed, 17 insertions(+), 7 deletions(-)
tools/testing/selftests/net/mptcp/mptcp_connect.c +17 -7
@@ -1283,12 +1283,15 @@ static void check_getpeername_connect(int fd) cfg_host, a, cfg_port, b); } -static void maybe_close (int fd) +static bool maybe_close (int fd) {unsigned int r =rand (); -if (!(cfg_join || cfg_remove || cfg_repeat >1 ) && (r &1 )) +if (!(cfg_join || cfg_remove || cfg_repeat >1 ) && (r &1 )) {close (fd); +return true; + } +return false; }int main_loop_s (int listensock)
@@ -1298,27 +1301,30 @@ int main_loop_s(int listensock)struct pollfd polls;socklen_t salen;int remotesock; +bool closed;int err =0 ;int fd =0 ; again: + closed = false; polls.fd = listensock; polls.events =POLLIN ;switch (poll (&polls,1 , poll_timeout)) {case -1 :perror ("poll" ); -return 1 ; + err =1 ; + goto out;case 0 :fprintf (stderr,"%s: timed out\n" , __func__); -close (listensock); -return 2 ; + err =2 ; + goto out; } salen =sizeof (ss); remotesock =accept (listensock, (struct sockaddr *)&ss, &salen);if (remotesock >=0 ) { -maybe_close (listensock); + closed =maybe_close (listensock);check_sockaddr (pf, &ss, salen);check_getpeername (remotesock, &ss, salen);
@@ -1334,7 +1340,8 @@ int main_loop_s(int listensock) err =copyfd_io (fd, remotesock,1 , true, &winfo); }else {perror ("accept" ); -return 1 ; + err =1 ; + goto out; }if (cfg_input)
@@ -1343,6 +1350,9 @@ int main_loop_s(int listensock)if (!err && --cfg_repeat >0 ) goto again; +out: +if (!closed) +close (listensock);return err; }
Signature · 1 line
2.53.0
Hi Geliang,
Thank you for your modifications, that's great!
Our CI did some validations and here is its report:
- KVM Validation: normal (except selftest_mptcp_join): Success! ✅
- KVM Validation: normal (only selftest_mptcp_join): Success! ✅
- KVM Validation: debug (except selftest_mptcp_join): Success! ✅
- KVM Validation: debug (only selftest_mptcp_join): Success! ✅
- KVM Validation: btf-normal (only bpftest_all): Success! ✅
- KVM Validation: btf-debug (only bpftest_all): Success! ✅
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/29411868709
Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/b5ef0fe4ad2d
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=1128077
If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:
$ cd [kernel source code]
$ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
--pull always mptcp/mptcp-upstream-virtme-docker:latest \
auto-normalFor more details:
https://github.com/multipath-tcp/mptcp-upstream-virtme-docker
Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)
Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)