Skip to contentSkip to conversation

mptcp: add MSG_ZEROCOPY support

PATCH In review Geliang Tang 2 participants 11 messages Jul 17

Subsystems (4)
lore
Share & tools
lore

Geliang Tang

author reviewer
#0 lore raw

v7 cover mptcp: add MSG_ZEROCOPY support

From: Geliang Tang <[email protected]>

This series adds MSG_ZEROCOPY support for MPTCP sockets, allowing userspace to transmit data without intermediate kernel copies, and provides corresponding selftests to exercise the new path.

Patch 1 fixes an uninitialized memory leak in the IPv4 error queue handling for zerocopy completion notifications by adding an early check for SO_EE_ORIGIN_ZEROCOPY.

Patch 2 is a small cleanup along the mptcp_sendmsg path.

Patch 3 aligns struct mptcp_data_frag to 8 bytes on 32-bit architectures to prevent unaligned 64-bit access faults on ARM, MIPS, and SPARC.

Patch 4 removes the redundant orig_offset parameter from mptcp_carve_data_frag() since it merely duplicates pfrag->offset.

Patch 5 implements the core MSG_ZEROCOPY support. The design follows TCP's reference-counting model, with a single ubuf_info per sendmsg, tracked by three reference buckets: one held by sendmsg itself, one per MPTCP dfrag in the retransmission queue, and one per subflow SKB. Completion is reported only after all bytes are acknowledged at both the MPTCP and subflow levels. The feature is silently downgraded to a regular copy in cases where zero-copy cannot be safely performed (MSG_FASTOPEN, fallback mode, or memory pressure). The new mptcp_sendmsg_zerocopy_iter() helper caps pg_len to U16_MAX (reverting the excess from the iov_iter on overflow) and returns -EFAULT when pg_off exceeds U16_MAX (which can happen on 256KB-page kernels like PowerPC and Hexagon), so the u16 dfrag->data_len and dfrag->offset fields cannot wrap. The caller accumulates the per-iteration byte count via the ssize_t return value.

Patch 6 handles SO_ZEROCOPY in setsockopt.

Patch 7 adds a new 'zerocopy' I/O mode to the mptcp_connect selftest to enable testing and performance validation of the zero-copy send path.

Patch 8 adds a wrapper script to run the zerocopy selftest as part of the regular test suite, ensuring the zero-copy path is exercised in CI.

Patch 9 is a cleanup for mptcp_connect.c that closes listensock deterministically to avoid double-close or leaks across repeated iterations.

v7:
- Add the pg_len / pg_off defensive truncation directly inside mptcp_sendmsg_zerocopy_iter() (the new helper in patch 5). When iov_iter_get_pages2() returns more than U16_MAX bytes, the excess is reverted from the iov_iter so the next iteration picks it up cleanly. When the page offset exceeds U16_MAX (on 256KB-page kernels like PowerPC and Hexagon), the call returns -EFAULT instead of silently truncating. The non-zerocopy path is left untouched.
- Refactor mptcp_sendmsg_zerocopy_iter() to return the per-iteration byte count as ssize_t so the caller can simply accumulate via copied += ret; this drops a ssize_t *copied out-parameter.
- Clean up and fix the zerocopy completion path in mptcp_connect.c

v6:
- Add patch 2 to widen offset field in struct mptcp_data_frag from u16 to
u32 to support page sizes up to 4GB on architectures with large page
sizes (256KB on PowerPC and Hexagon). This prevents offset wrap-around
when page fragment offset exceeds 65535.
- Add patch 5 to fix uninitialized memory leak in IPv4 error queue
handling for zerocopy completion notifications. Add early check for SO_EE_ORIGIN_ZEROCOPY in ipv4_datagram_support_cmsg() to prevent
reading uninitialized memory from ip_hdr(skb)->saddr.
- Refactor zerocopy completion notification draining logic in selftests.
Extract process_zc_cmsg() helper to process individual control messages. Extract drain_errqueue() helper to handle error queue draining.
Simplify wait_for_zc_completions() main loop from 6 levels to 3 levels.
Move variables to their actual usage scope for better maintainability.
- Use batched polling (200ms intervals with 5s total timeout) to drain completion notifications for large transfers that may generate multiple completion events.
- Parse error queue messages to verify all sent data has been ACKed before breaking the completion loop.
- Reset msg_controllen each iteration of the inner reap loop to prevent truncation of control messages.
- Use >= instead of > in the MPTFO size adjustment so file_size reaches 0
when the whole file is sent via fast open.
- https://patchwork.kernel.org/project/mptcp/cover/[email protected]/

v5:
- A new patch to align struct mptcp_data_frag to 8 bytes on 32-bit.
- Prevent coalescing of dfrags with different ubuf instances into the same subflow skb to avoid premature completion notifications.
- Return -EFAULT instead of -EAGAIN when iov_iter_get_pages2() returns 0
to properly signal iterator exhaustion.
- Account for dfrag metadata overhead in sk_wmem_queued_add() to prevent bypassing socket memory limits.
- Widen data_len field from u16 to u32 to prevent truncation on 64KB-page kernels (ARM64, PPC64).
- In selftests, use batched polling (200ms intervals with 5s total timeout)
to drain completion notifications for large transfers that may generate multiple completion events.
- https://patchwork.kernel.org/project/mptcp/cover/[email protected]/

v4:
- two more cleanups, patch 2 and patch 7.
- net/mptcp/protocol.c, mptcp_sendmsg_frag(): pull the type-mismatch
check out of the if (can_coalesce) block and use the standard
skb_zcopy_pure() helper; mptcp_sendmsg_zerocopy_iter() now
calls iov_iter_revert() before put_page() in the kzalloc failure
path so a retry starts from the same iterator offset.
- tools/testing/selftests/net/mptcp/mptcp_connect.c, copyfd_io_zc():
reset msg_controllen each iteration of the inner reap loop using a
do {} while block; use >= instead of > in the MPTFO size
adjustment so file_size reaches 0 when the whole file is sent via
fast open.
- https://patchwork.kernel.org/project/mptcp/cover/[email protected]/

v3:
- Force fresh skb when mixing zerocopy and kernel-copy fragments.
- Simplify zero-copy availability check by using SOCK_ZEROCOPY flag
directly.
- Validate SO_ZEROCOPY value in setsockopt and avoid affecting
fallback subflow.
- In selftests, drain MPTFO partial data, use poll+recvmsg for completion notifications with timeout, and verify getsockopt round-trip.
- https://patchwork.kernel.org/project/mptcp/cover/[email protected]/

v2:
- patch 2, never mix PURE_ZEROCOPY frags with kernel-copy frags in the
same skb; handle fallback, return values.
- patch 3, a new patch to handle SO_ZEROCOPY in setsockopt.
- patch 4, set SO_ZEROCOPY, handle listen_mode, error queue.
- https://patchwork.kernel.org/project/mptcp/cover/[email protected]/

v1:
- https://patchwork.kernel.org/project/mptcp/cover/[email protected]/

Geliang Tang (9):
ipv4: fix uninitialized memory in zerocopy cmsg
mptcp: use local variable tp in sendmsg_frag
mptcp: align struct mptcp_data_frag to 8 bytes on 32-bit
mptcp: remove redundant orig_offset in carve_data_frag
mptcp: add MSG_ZEROCOPY support
mptcp: handle SO_ZEROCOPY in setsockopt
selftests: mptcp: connect: add zerocopy io mode
selftests: mptcp: connect: cover zerocopy mode
selftests: mptcp: connect: close listensock deterministically

 net/ipv4/ip_sockglue.c                        |   6 +
 net/mptcp/protocol.c                          | 152 ++++++++++++--
 net/mptcp/protocol.h                          |   1 +
 net/mptcp/sockopt.c                           |  17 +-
 tools/testing/selftests/net/mptcp/Makefile    |   1 +
 .../selftests/net/mptcp/mptcp_connect.c       | 186 +++++++++++++++++-
 .../net/mptcp/mptcp_connect_zerocopy.sh       |   5 +
 7 files changed, 343 insertions(+), 25 deletions(-)
 create mode 100755 tools/testing/selftests/net/mptcp/mptcp_connect_zerocopy.sh
Signature · 1 line
2.53.0
author reviewer
#1 lore raw

PATCH v7 1/9 ipv4: fix uninitialized memory in zerocopy cmsg

From: Geliang Tang <[email protected]>

When SOF_TIMESTAMPING_OPT_CMSG is enabled and applications read MSG_ZEROCOPY completion notifications from the error queue, the ipv4_datagram_support_cmsg() function attempts to read ip_hdr(skb)->saddr to populate IP_PKTINFO cmsg data. However, zerocopy completion notification skbs don't have IP headers, causing this read to access uninitialized memory which is then leaked to userspace via put_cmsg().

This is because the function checks PKTINFO_SKB_CB(skb)->ipi_ifindex (which aliases the ops pointer in ubuf_info_msgzc stored in skb->cb), and if non-NULL, proceeds to read ip_hdr(skb)->saddr without verifying that the skb actually has an IP header.

Fix this by adding an early check for SO_EE_ORIGIN_ZEROCOPY and returning false, since zerocopy completion notifications don't have IP headers and IP_PKTINFO cmsg is not applicable to them.

Fixes 829ae9d61165 ("net-timestamp: allow reading recv cmsg on errqueue with origin tstamp")
---
 net/ipv4/ip_sockglue.c | 6 ++++++
 1 file changed, 6 insertions(+)
net/ipv4/ip_sockglue.c +6 -0
index a55ef327ec93..59513f15e522 100644
@@ -504,6 +504,12 @@ static bool ipv4_datagram_support_cmsg(const struct sock *sk,
 	if (ee_origin == SO_EE_ORIGIN_LOCAL)
 		return false;
 
+	/* Zerocopy completion notifications don't have IP headers,
+	 * so IP_PKTINFO cmsg is not applicable.
+	 */
+	if (ee_origin == SO_EE_ORIGIN_ZEROCOPY)
+		return false;
+
 	/* Support IP_PKTINFO on tstamp packets if requested, to correlate
 	 * timestamp with egress dev. Not possible for packets without iif
 	 * or without payload (SOF_TIMESTAMPING_OPT_TSONLY).
Signature · 1 line
2.53.0
author reviewer
#2 lore raw

PATCH v7 2/9 mptcp: use local variable tp in sendmsg_frag

In reply to Geliang Tang

From: Geliang Tang <[email protected]>

In mptcp_sendmsg_frag(), tcp_sk(ssk) is called multiple times. Introduce a local variable 'tp' to hold the tcp_sock pointer and use it throughout the function. This avoids repeated dereferencing and improves code readability, with no functional change.

---
 net/mptcp/protocol.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
net/mptcp/protocol.c +4 -3
index ffcf5a1788f6..1cab07b1b9b1 100644
@@ -1375,6 +1375,7 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	u64 data_seq = dfrag->data_seq + info->sent;
 	int offset = dfrag->offset + info->sent;
 	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct tcp_sock *tp = tcp_sk(ssk);
 	bool zero_window_probe = false;
 	struct mptcp_ext *mpext = NULL;
 	bool can_coalesce = false;
@@ -1410,14 +1411,14 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 		mpext = mptcp_get_ext(skb);
 		if (!mptcp_skb_can_collapse_to(data_seq, skb, mpext)) {
 			TCP_SKB_CB(skb)->eor = 1;
-			tcp_mark_push(tcp_sk(ssk), skb);
+			tcp_mark_push(tp, skb);
 			goto alloc_skb;
 		}
 
 		i = skb_shinfo(skb)->nr_frags;
 		can_coalesce = skb_can_coalesce(skb, i, dfrag->page, offset);
 		if (!can_coalesce && i >= READ_ONCE(net_hotdata.sysctl_max_skb_frags)) {
-			tcp_mark_push(tcp_sk(ssk), skb);
+			tcp_mark_push(tp, skb);
 			goto alloc_skb;
 		}
 
@@ -1471,7 +1472,7 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	skb->truesize += copy;
 	sk_wmem_queued_add(ssk, copy);
 	sk_mem_charge(ssk, copy);
-	WRITE_ONCE(tcp_sk(ssk)->write_seq, tcp_sk(ssk)->write_seq + copy);
+	WRITE_ONCE(tp->write_seq, tp->write_seq + copy);
 	TCP_SKB_CB(skb)->end_seq += copy;
 	tcp_skb_pcount_set(skb, 0);
 
Signature · 1 line
2.53.0
author reviewer
#3 lore raw

PATCH v7 3/9 mptcp: align struct mptcp_data_frag to 8 bytes on 32-bit

In reply to Geliang Tang

From: Geliang Tang <[email protected]>

mptcp_carve_data_frag() places a struct mptcp_data_frag inside a page-frag region, using ALIGN(orig_offset, sizeof(long)) to compute the offset where the struct lives. On 64-bit Linux sizeof(long) is 8 and this works as intended. On 32-bit Linux sizeof(long) is 4, which only guarantees 4-byte alignment for the struct.

struct mptcp_data_frag contains a u64 data_seq member, which requires 8-byte alignment. 4-byte-aligned 64-bit loads fault on 32-bit architectures that do not handle unaligned 64-bit accesses in hardware (armv7, mips, sparc, ...). The in-source comment near BUILD_BUG_ON that states "ALIGN(1, sizeof(long)) - 1, so 8-1" is misleading because it implicitly assumes sizeof(long) is always 8.

Switch both the ALIGN() and the matching BUILD_BUG_ON to sizeof(u64) so the 8-byte alignment is enforced on every architecture, and the overhead u8 budget still fits the worst-case 7-byte padding plus the struct size. This is a no-op on 64-bit and a correctness fix on 32-bit.

---
 net/mptcp/protocol.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
net/mptcp/protocol.c +3 -3
index 1cab07b1b9b1..ca0c02f3bfd3 100644
@@ -1237,7 +1237,7 @@ static struct mptcp_data_frag *
 mptcp_carve_data_frag(const struct mptcp_sock *msk, struct page_frag *pfrag,
 		      int orig_offset)
 {
-	int offset = ALIGN(orig_offset, sizeof(long));
+	int offset = ALIGN(orig_offset, sizeof(u64));
 	struct mptcp_data_frag *dfrag;
 
 	dfrag = (struct mptcp_data_frag *)(page_to_virt(pfrag->page) + offset);
@@ -4789,9 +4789,9 @@ void __init mptcp_proto_init(void)
 	BUILD_BUG_ON(sizeof(struct mptcp_skb_cb) > sizeof_field(struct sk_buff, cb));
 
 	/* struct mptcp_data_frag: 'overhead' corresponds to the alignment
-	 * (ALIGN(1, sizeof(long)) - 1, so 8-1) + the struct's size
+	 * (ALIGN(1, sizeof(u64)) - 1, so 8-1) + the struct's size
 	 */
-	BUILD_BUG_ON(ALIGN(1, sizeof(long)) - 1 + sizeof(struct mptcp_data_frag)
+	BUILD_BUG_ON(ALIGN(1, sizeof(u64)) - 1 + sizeof(struct mptcp_data_frag)
 		     > U8_MAX);
 }
 
Signature · 1 line
2.53.0
author reviewer
#4 lore raw

PATCH v7 4/9 mptcp: remove redundant orig_offset in carve_data_frag

In reply to Geliang Tang

From: Geliang Tang <[email protected]>

mptcp_carve_data_frag() already receives struct page_frag via its 'pfrag' parameter, making the separate 'orig_offset' parameter redundant since it merely duplicates pfrag->offset.

Simplify the function signature by removing this unnecessary parameter and directly accessing the offset from struct pfrag internally.

---
 net/mptcp/protocol.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
net/mptcp/protocol.c +5 -4
index ca0c02f3bfd3..6a9daa187dad 100644
@@ -1234,12 +1234,13 @@ static bool mptcp_page_frag_refill(struct sock *sk, struct page_frag *pfrag)
 }
 
 static struct mptcp_data_frag *
-mptcp_carve_data_frag(const struct mptcp_sock *msk, struct page_frag *pfrag,
-		      int orig_offset)
+mptcp_carve_data_frag(const struct mptcp_sock *msk, struct page_frag *pfrag)
 {
-	int offset = ALIGN(orig_offset, sizeof(u64));
+	int orig_offset = pfrag->offset;
 	struct mptcp_data_frag *dfrag;
+	int offset;
 
+	offset = ALIGN(orig_offset, sizeof(u64));
 	dfrag = (struct mptcp_data_frag *)(page_to_virt(pfrag->page) + offset);
 	dfrag->data_len = 0;
 	dfrag->data_seq = msk->write_seq;
@@ -2031,7 +2032,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 			if (!mptcp_page_frag_refill(sk, pfrag))
 				goto wait_for_memory;
 
-			dfrag = mptcp_carve_data_frag(msk, pfrag, pfrag->offset);
+			dfrag = mptcp_carve_data_frag(msk, pfrag);
 			frag_truesize = dfrag->overhead;
 		}
 
Signature · 1 line
2.53.0
author reviewer
#5 lore raw

PATCH v7 5/9 mptcp: add MSG_ZEROCOPY support

In reply to Geliang Tang

From: Geliang Tang <[email protected]>

This patch enables MSG_ZEROCOPY support for MPTCP sockets, allowing user-space to transmit data without intermediate kernel copies.

The implementation follows the same lifecycle model as TCP:

A single ubuf_info_msgzc is allocated per sendmsg via msg_zerocopy_realloc(). Its reference count consists of three parts:
one 'caller' reference held until sendmsg returns, one reference per MPTCP dfrag in the retransmission queue (released when the dfrag is acked and cleared), and one reference per subflow skb that consumes dfrag bytes (released by TCP's skb_zcopy_clear() when the skb is acked). Completion is reported only after all bytes are acknowledged at both MPTCP and subflow levels.

User pages are pinned via iov_iter_get_pages2() and stored directly in a newly allocated mptcp_data_frag. The dfrag takes one reference on the ubuf_info and one on the page. Subflow skbs take additional page references via get_page() and are marked with skb_zcopy_set() and SKBFL_PURE_ZEROCOPY, mirroring TCP's behavior and bypassing kernel memory accounting for those pages.

The feature is silently downgraded to regular copy when MSG_FASTOPEN is set or the application has not opted in via setsockopt(SO_ZEROCOPY). If msg_zerocopy_realloc() fails (e.g. RLIMIT_MEMLOCK pressure), sendmsg(2) aborts with -ENOBUFS.

The implementation prevents coalescing of dfrags with different ubuf instances into the same subflow skb to avoid premature completion
notifications: a coalescing break is taken when the skb's pure-zerocopy state differs from the incoming dfrag, and another when both dfrags have ubuf instances but they differ. The new mptcp_sendmsg_zerocopy_iter() helper caps pg_len to U16_MAX (reverting the excess from the iov_iter when iov_iter_get_pages2() returns more than that) and returns -EFAULT when pg_off exceeds U16_MAX (which can happen on 256KB-page kernels like PowerPC and Hexagon), so the u16 dfrag->data_len and dfrag->offset fields cannot wrap. The caller accumulates the per-iteration byte count via the ssize_t return value. The non-zerocopy path is left untouched.

When iov_iter_get_pages2() returns 0, the function returns -EFAULT to properly signal iterator exhaustion.

---
 net/mptcp/protocol.c | 132 ++++++++++++++++++++++++++++++++++++++++---
 net/mptcp/protocol.h |   1 +
 2 files changed, 126 insertions(+), 7 deletions(-)
net/mptcp/protocol.c +125 -7
index 6a9daa187dad..8b58309566b8 100644
@@ -1118,9 +1118,11 @@ static bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk,
 		df->data_seq + df->data_len == msk->write_seq;
 }
 
-static void dfrag_uncharge(struct sock *sk, int len)
+static void dfrag_uncharge(struct sock *sk, int len,
+			   struct mptcp_data_frag *dfrag)
 {
-	sk_mem_uncharge(sk, len);
+	if (!dfrag->ubuf)
+		sk_mem_uncharge(sk, len);
 	sk_wmem_queued_add(sk, -len);
 }
 
@@ -1129,7 +1131,13 @@ static void dfrag_clear(struct sock *sk, struct mptcp_data_frag *dfrag)
 	int len = dfrag->data_len + dfrag->overhead;
 
 	list_del(&dfrag->list);
-	dfrag_uncharge(sk, len);
+	dfrag_uncharge(sk, len, dfrag);
+	if (dfrag->ubuf) {
+		net_zcopy_put(dfrag->ubuf);
+		put_page(dfrag->page);
+		kfree(dfrag);
+		return;
+	}
 	put_page(dfrag->page);
 }
 
@@ -1174,7 +1182,7 @@ static void __mptcp_clean_una(struct sock *sk)
 		dfrag->data_len -= delta;
 		dfrag->already_sent -= delta;
 
-		dfrag_uncharge(sk, delta);
+		dfrag_uncharge(sk, delta, dfrag);
 	}
 
 	/* all retransmitted data acked, recovery completed */
@@ -1249,6 +1257,7 @@ mptcp_carve_data_frag(const struct mptcp_sock *msk, struct page_frag *pfrag)
 	dfrag->already_sent = 0;
 	dfrag->page = pfrag->page;
 	dfrag->eor = 0;
+	dfrag->ubuf = NULL;
 
 	return dfrag;
 }
@@ -1416,6 +1425,16 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 			goto alloc_skb;
 		}
 
+		if (skb_zcopy_pure(skb) != !!dfrag->ubuf) {
+			tcp_mark_push(tp, skb);
+			goto alloc_skb;
+		}
+		if (dfrag->ubuf && skb_zcopy(skb) &&
+		    skb_zcopy(skb) != dfrag->ubuf) {
+			tcp_mark_push(tp, skb);
+			goto alloc_skb;
+		}
+
 		i = skb_shinfo(skb)->nr_frags;
 		can_coalesce = skb_can_coalesce(skb, i, dfrag->page, offset);
 		if (!can_coalesce && i >= READ_ONCE(net_hotdata.sysctl_max_skb_frags)) {
@@ -1456,7 +1475,7 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	}
 
 	copy = min_t(size_t, copy, info->limit - info->sent);
-	if (!sk_wmem_schedule(ssk, copy)) {
+	if (!dfrag->ubuf && !sk_wmem_schedule(ssk, copy)) {
 		tcp_remove_empty_skb(ssk);
 		return -ENOMEM;
 	}
@@ -1466,13 +1485,19 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	} else {
 		get_page(dfrag->page);
 		skb_fill_page_desc(skb, i, dfrag->page, offset, copy);
+
+		if (dfrag->ubuf && !skb_zcopy(skb)) {
+			skb_zcopy_set(skb, dfrag->ubuf, NULL);
+			skb_shinfo(skb)->flags |= SKBFL_PURE_ZEROCOPY;
+		}
 	}
 
 	skb->len += copy;
 	skb->data_len += copy;
 	skb->truesize += copy;
 	sk_wmem_queued_add(ssk, copy);
-	sk_mem_charge(ssk, copy);
+	if (!skb_zcopy_pure(skb))
+		sk_mem_charge(ssk, copy);
 	WRITE_ONCE(tp->write_seq, tp->write_seq + copy);
 	TCP_SKB_CB(skb)->end_seq += copy;
 	tcp_skb_pcount_set(skb, 0);
@@ -1969,22 +1994,99 @@ static void mptcp_rps_record_subflows(const struct mptcp_sock *msk)
 	}
 }
 
+static ssize_t mptcp_sendmsg_zerocopy_iter(struct sock *sk, struct msghdr *msg,
+					   struct ubuf_info *ubuf,
+					   u32 copy_limit)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct mptcp_data_frag *dfrag;
+	struct page *pages[1] = {};
+	size_t pg_off = 0;
+	ssize_t pg_len;
+
+	pg_len = iov_iter_get_pages2(&msg->msg_iter, pages,
+				     copy_limit, 1, &pg_off);
+	if (pg_len < 0)
+		return pg_len;
+	/* pg_len == 0 means the iterator is exhausted */
+	if (pg_len == 0)
+		return -EFAULT;
+
+	if (pg_len > U16_MAX) {
+		iov_iter_revert(&msg->msg_iter, pg_len - U16_MAX);
+		pg_len = U16_MAX;
+	}
+
+	if (pg_off > U16_MAX) {
+		iov_iter_revert(&msg->msg_iter, pg_len);
+		put_page(pages[0]);
+		return -EFAULT;
+	}
+
+	if (!sk_wmem_schedule(sk, pg_len + sizeof(*dfrag))) {
+		iov_iter_revert(&msg->msg_iter, pg_len);
+		put_page(pages[0]);
+		return -EAGAIN;
+	}
+
+	dfrag = kzalloc_obj(*dfrag, GFP_KERNEL_ACCOUNT);
+	if (!dfrag) {
+		iov_iter_revert(&msg->msg_iter, pg_len);
+		put_page(pages[0]);
+		sk_forward_alloc_add(sk, -(pg_len + sizeof(*dfrag)));
+		return -ENOMEM;
+	}
+
+	dfrag->data_len = pg_len;
+	dfrag->data_seq = msk->write_seq;
+	dfrag->offset = pg_off;
+	dfrag->page = pages[0];
+	dfrag->ubuf = ubuf;
+	dfrag->overhead = sizeof(struct mptcp_data_frag);
+
+	/* one ref for the dfrag; released in dfrag_clear */
+	net_zcopy_get(ubuf);
+
+	WRITE_ONCE(msk->write_seq, msk->write_seq + pg_len);
+	sk_wmem_queued_add(sk, pg_len + dfrag->overhead);
+
+	list_add_tail(&dfrag->list, &msk->rtx_queue);
+	if (!msk->first_pending)
+		msk->first_pending = dfrag;
+
+	return pg_len;
+}
+
 static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct ubuf_info *ubuf = NULL;
 	struct page_frag *pfrag;
 	size_t copied = 0;
+	bool zc = false;
 	int ret = 0;
 	long timeo;
 
 	/* silently ignore everything else */
 	msg->msg_flags &= MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL |
-			  MSG_FASTOPEN | MSG_EOR;
+			  MSG_FASTOPEN | MSG_EOR | MSG_ZEROCOPY;
 
 	lock_sock(sk);
 
 	mptcp_rps_record_subflows(msk);
 
+	if (msg->msg_flags & MSG_FASTOPEN || !sock_flag(sk, SOCK_ZEROCOPY))
+		msg->msg_flags &= ~MSG_ZEROCOPY;
+
+	if ((msg->msg_flags & MSG_ZEROCOPY) && len) {
+		ubuf = msg_zerocopy_realloc(sk, len, NULL, false);
+		if (!ubuf) {
+			ret = -ENOBUFS;
+			goto do_error;
+		}
+		zc = true;
+	}
+
 	if (unlikely(inet_test_bit(DEFER_CONNECT, sk) ||
 		     msg->msg_flags & MSG_FASTOPEN)) {
 		int copied_syn = 0;
@@ -2023,6 +2125,16 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		if (!copy_limit)
 			goto wait_for_memory;
 
+		if (zc) {
+			ret = mptcp_sendmsg_zerocopy_iter(sk, msg, ubuf,
+							  copy_limit);
+			if (ret < 0)
+				goto do_error;
+
+			copied += ret;
+			continue;
+		}
+
 		/* reuse tail pfrag, if possible, or carve a new one from the
 		 * page allocator
 		 */
@@ -2098,6 +2210,8 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 	}
 
 out:
+	if (zc)
+		net_zcopy_put(ubuf);
 	release_sock(sk);
 	return copied;
 
@@ -2106,6 +2220,10 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		goto out;
 
 	copied = sk_stream_error(sk, msg->msg_flags, ret);
+	if (zc) {
+		net_zcopy_put_abort(ubuf, true);
+		zc = false;
+	}
 	goto out;
 }
 
net/mptcp/protocol.h +1 -0
index da40c6f3705f..f4e69ea51a94 100644
@@ -268,6 +268,7 @@ struct mptcp_data_frag {
 	u8 eor;			/* currently using 1 bit */
 	u16 already_sent;
 	struct page *page;
+	struct ubuf_info *ubuf;
 };
 
 /* Arbitrary compromise between as low as possible to react timely to subflow
Signature · 1 line
2.53.0
author reviewer
#6 lore raw

PATCH v7 6/9 mptcp: handle SO_ZEROCOPY in setsockopt

In reply to Geliang Tang

From: Geliang Tang <[email protected]>

SO_ZEROCOPY is currently rejected by MPTCP setsockopt with -EOPNOTSUPP, even though the matching MSG_ZEROCOPY path in mptcp_sendmsg() is wired up. As a result MSG_ZEROCOPY silently falls back to a copy on MPTCP sockets because the opt-in flag is never set.

Expose SO_ZEROCOPY through the SOL_SOCKET setsockopt dispatcher. The new mptcp_setsockopt_sol_socket_zerocopy() helper validates that val is 0 or 1 (matching standard TCP sk_setsockopt) and flips SOCK_ZEROCOPY on the msk only. mptcp_sendmsg() consults the same flag directly, so opt-in is honoured on every data path without any cross-socket propagation.

---
 net/mptcp/sockopt.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
net/mptcp/sockopt.c +16 -1
index fcf6feb2a9eb..abcc7d4452b4 100644
@@ -175,6 +175,19 @@ static int mptcp_setsockopt_sol_socket_tstamp(struct mptcp_sock *msk, int optnam
 	return 0;
 }
 
+static int mptcp_setsockopt_sol_socket_zerocopy(struct mptcp_sock *msk, int val)
+{
+	struct sock *sk = (struct sock *)msk;
+
+	if (val != 0 && val != 1)
+		return -EINVAL;
+
+	lock_sock(sk);
+	sock_valbool_flag(sk, SOCK_ZEROCOPY, val);
+	release_sock(sk);
+	return 0;
+}
+
 static int mptcp_setsockopt_sol_socket_int(struct mptcp_sock *msk, int optname,
 					   sockptr_t optval,
 					   unsigned int optlen)
@@ -203,6 +216,8 @@ static int mptcp_setsockopt_sol_socket_int(struct mptcp_sock *msk, int optname,
 	case SO_TIMESTAMPNS_OLD:
 	case SO_TIMESTAMPNS_NEW:
 		return mptcp_setsockopt_sol_socket_tstamp(msk, optname, val);
+	case SO_ZEROCOPY:
+		return mptcp_setsockopt_sol_socket_zerocopy(msk, val);
 	}
 
 	return -ENOPROTOOPT;
@@ -342,6 +357,7 @@ static int mptcp_setsockopt_sol_socket(struct mptcp_sock *msk, int optname,
 	case SO_TIMESTAMP_NEW:
 	case SO_TIMESTAMPNS_OLD:
 	case SO_TIMESTAMPNS_NEW:
+	case SO_ZEROCOPY:
 		return mptcp_setsockopt_sol_socket_int(msk, optname, optval,
 						       optlen);
 	case SO_TIMESTAMPING_OLD:
@@ -387,7 +403,6 @@ static int mptcp_setsockopt_sol_socket(struct mptcp_sock *msk, int optname,
 	 * SO_CNX_ADVICE is currently unsupported, could possibly be relevant,
 	 * but likely needs careful design
 	 *
-	 * SO_ZEROCOPY is currently unsupported, TODO in sndmsg
 	 * SO_TXTIME is currently unsupported
 	 */
 
Signature · 1 line
2.53.0
author reviewer
#7 lore raw

PATCH v7 7/9 selftests: mptcp: connect: add zerocopy io mode

In reply to Geliang Tang

From: Geliang Tang <[email protected]>

Add a new I/O mode 'zerocopy' to the mptcp_connect selftest, which uses sendmsg() with the MSG_ZEROCOPY flag to transmit data. This enables testing and performance validation of the zero-copy send path for both TCP and MPTCP connections.

This mode is useful for benchmarking and for verifying correctness of MSG_ZEROCOPY handling in the MPTCP stack.

The completion notification draining uses batched polling to handle large transfers that may generate multiple completion events across different TCP segments.

---
 .../selftests/net/mptcp/mptcp_connect.c       | 162 +++++++++++++++++-
 1 file changed, 161 insertions(+), 1 deletion(-)
tools/testing/selftests/net/mptcp/mptcp_connect.c +161 -1
index ea4cb6c1bd5e..f6ebdc5bf745 100644
@@ -33,6 +33,7 @@
 #include <linux/tcp.h>
 #include <linux/time_types.h>
 #include <linux/sockios.h>
+#include <linux/errqueue.h>
 #include <linux/compiler.h>
 
 extern int optind;
@@ -53,6 +54,7 @@ enum cfg_mode {
 	CFG_MODE_MMAP,
 	CFG_MODE_SENDFILE,
 	CFG_MODE_SPLICE,
+	CFG_MODE_ZEROCOPY,
 };
 
 enum cfg_peek {
@@ -125,7 +127,7 @@ static void die_usage(void)
 	fprintf(stderr, "\t-j     -- add additional sleep at connection start and tear down "
 		"-- for MPJ tests\n");
 	fprintf(stderr, "\t-l     -- listens mode, accepts incoming connection\n");
-	fprintf(stderr, "\t-m [poll|mmap|sendfile|splice] -- use poll(default)/mmap+write/sendfile/splice\n");
+	fprintf(stderr, "\t-m [poll|mmap|sendfile|splice|zerocopy] -- use poll(default)/mmap+write/sendfile/splice/zerocopy\n");
 	fprintf(stderr, "\t-M mark -- set socket packet mark\n");
 	fprintf(stderr, "\t-o option -- test sockopt <option>\n");
 	fprintf(stderr, "\t-p num -- use port num\n");
@@ -998,6 +1000,150 @@ static int copyfd_io_splice(int infd, int peerfd, int outfd, unsigned int size,
 	return err;
 }
 
+static bool process_cmsg_zc(struct msghdr *errmsg)
+{
+	struct sock_extended_err *serr;
+	struct cmsghdr *cm;
+
+	for (cm = CMSG_FIRSTHDR(errmsg); cm; cm = CMSG_NXTHDR(errmsg, cm)) {
+		if (cm->cmsg_level != SOL_IP && cm->cmsg_level != SOL_IPV6)
+			continue;
+		if (cm->cmsg_type != IP_RECVERR &&
+		    cm->cmsg_type != IPV6_RECVERR)
+			continue;
+
+		serr = (struct sock_extended_err *)CMSG_DATA(cm);
+		if (serr->ee_origin == SO_EE_ORIGIN_ZEROCOPY)
+			return true;
+	}
+
+	return false;
+}
+
+static bool drain_errqueue_zc(int peerfd)
+{
+	size_t ctl_len = CMSG_SPACE(sizeof(struct sock_extended_err));
+	struct msghdr errmsg = {};
+	bool found = false;
+	char *ctl;
+
+	ctl = malloc(ctl_len);
+	if (!ctl) {
+		perror("malloc ctl");
+		return false;
+	}
+
+	errmsg.msg_control = ctl;
+
+	while (!found) {
+		errmsg.msg_controllen = ctl_len;
+		if (recvmsg(peerfd, &errmsg, MSG_ERRQUEUE) < 0)
+			break;
+
+		found = process_cmsg_zc(&errmsg);
+	}
+
+	free(ctl);
+	return found;
+}
+
+static void wait_for_completions_zc(int peerfd)
+{
+	struct pollfd pfd = { .fd = peerfd, .events = POLLERR };
+	int i, timeout_ms = 5000;
+
+	if (cfg_time > 0)
+		timeout_ms = cfg_time;
+
+	for (i = 0; i < timeout_ms; i += 200) {
+		if (poll(&pfd, 1, 200) < 0)
+			break;
+
+		if (drain_errqueue_zc(peerfd))
+			break;
+	}
+}
+
+static int copyfd_io_zc(int infd, int peerfd, int outfd, unsigned int size,
+			bool *in_closed_after_out, struct wstate *winfo)
+{
+	struct msghdr msg = {};
+	struct iovec iov;
+	socklen_t len;
+	int on = 1;
+	char *buf;
+	int err;
+
+	if (spool_buf(peerfd, winfo) < 0) {
+		perror("spool_buf");
+		return 1;
+	}
+
+	set_sndbuf(peerfd, size * 2);
+
+	len = sizeof(on);
+	if (setsockopt(peerfd, SOL_SOCKET, SO_ZEROCOPY, &on, len)) {
+		perror("setsockopt SO_ZEROCOPY");
+		return 1;
+	}
+
+	on = -1;
+	if (getsockopt(peerfd, SOL_SOCKET, SO_ZEROCOPY, &on, &len) ||
+	    on != 1) {
+		perror("getsockopt SO_ZEROCOPY");
+		return 1;
+	}
+
+	buf = malloc(size);
+	if (!buf) {
+		perror("malloc");
+		return 1;
+	}
+
+	if (read(infd, buf, size) != size) {
+		perror("read input");
+		err = 1;
+		goto free_bufs;
+	}
+
+	iov.iov_base = buf;
+	iov.iov_len = size;
+	msg.msg_iov = &iov;
+	msg.msg_iovlen = 1;
+
+	if (listen_mode) {
+		if (do_recvfile(peerfd, outfd) < 0) {
+			err = 1;
+			goto free_bufs;
+		}
+	}
+
+	err = sendmsg(peerfd, &msg, MSG_ZEROCOPY);
+	if (err < 0) {
+		perror("sendmsg MSG_ZEROCOPY");
+		err = 1;
+		goto free_bufs;
+	}
+	if ((unsigned int)err != size) {
+		fprintf(stderr, "sendmsg short: %d/%u bytes queued\n",
+			err, size);
+	}
+	err = 0;
+
+	if (!listen_mode) {
+		shut_wr(peerfd);
+		if (do_recvfile(peerfd, outfd) < 0)
+			err = 1;
+		*in_closed_after_out = true;
+	}
+
+	wait_for_completions_zc(peerfd);
+
+free_bufs:
+	free(buf);
+	return err;
+}
+
 static int copyfd_io(int infd, int peerfd, int outfd, bool close_peerfd, struct wstate *winfo)
 {
 	bool in_closed_after_out = false;
@@ -1038,6 +1184,17 @@ static int copyfd_io(int infd, int peerfd, int outfd, bool close_peerfd, struct
 				       &in_closed_after_out, winfo);
 		break;
 
+	case CFG_MODE_ZEROCOPY:
+		file_size = get_infd_size(infd);
+		if (file_size < 0)
+			return file_size;
+		if (cfg_sockopt_types.mptfo && winfo->total_len &&
+		    file_size >= winfo->total_len)
+			file_size -= winfo->total_len;
+		ret = copyfd_io_zc(infd, peerfd, outfd, file_size,
+				   &in_closed_after_out, winfo);
+		break;
+
 	default:
 		fprintf(stderr, "Invalid mode %d\n", cfg_mode);
 
@@ -1453,6 +1610,8 @@ int parse_mode(const char *mode)
 		return CFG_MODE_SENDFILE;
 	if (!strcasecmp(mode, "splice"))
 		return CFG_MODE_SPLICE;
+	if (!strcasecmp(mode, "zerocopy"))
+		return CFG_MODE_ZEROCOPY;
 
 	fprintf(stderr, "Unknown test mode: %s\n", mode);
 	fprintf(stderr, "Supported modes are:\n");
@@ -1460,6 +1619,7 @@ int parse_mode(const char *mode)
 	fprintf(stderr, "\t\t\"mmap\" - send entire input file (mmap+write), then read response (-l will read input first)\n");
 	fprintf(stderr, "\t\t\"sendfile\" - send entire input file (sendfile), then read response (-l will read input first)\n");
 	fprintf(stderr, "\t\t\"splice\" - send entire input file (splice), then read response (-l will read input first)\n");
+	fprintf(stderr, "\t\t\"zerocopy\" - send entire input file (zerocopy), then read response (-l will read input first)\n");
 
 	die_usage();
 
Signature · 1 line
2.53.0
author reviewer
#8 lore raw

PATCH v7 8/9 selftests: mptcp: connect: cover zerocopy mode

In reply to Geliang Tang

From: Geliang Tang <[email protected]>

Add a new test script mptcp_connect_zerocopy.sh to run the mptcp_connect selftest with the zerocopy I/O mode. This ensures the MSG_ZEROCOPY path is exercised in the regular test suite.

The script follows the same pattern as the existing mmap, sendfile, and splice test wrappers, invoking mptcp_connect.sh with the -m zerocopy option.

---
 tools/testing/selftests/net/mptcp/Makefile                  | 1 +
 tools/testing/selftests/net/mptcp/mptcp_connect_zerocopy.sh | 5 +++++
 2 files changed, 6 insertions(+)
 create mode 100755 tools/testing/selftests/net/mptcp/mptcp_connect_zerocopy.sh
tools/testing/selftests/net/mptcp/Makefile +1 -0
index 22ba0da2adb8..9e47ecfe58e2 100644
@@ -12,6 +12,7 @@ TEST_PROGS := \
 	mptcp_connect_mmap.sh \
 	mptcp_connect_sendfile.sh \
 	mptcp_connect_splice.sh \
+	mptcp_connect_zerocopy.sh \
 	mptcp_join.sh \
 	mptcp_sockopt.sh \
 	pm_netlink.sh \
tools/testing/selftests/net/mptcp/mptcp_connect_zerocopy.sh new file +5 -0
new file mode 100755
index 000000000000..0220d29ad68e
@@ -0,0 +1,5 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+MPTCP_LIB_KSFT_TEST="$(basename "${0}" .sh)" \
+	"$(dirname "${0}")/mptcp_connect.sh" -m zerocopy "${@}"
Signature · 1 line
2.53.0
author reviewer
#9 lore raw

PATCH v7 9/9 selftests: mptcp: connect: close listensock deterministically

In reply to Geliang Tang

From: Geliang Tang <[email protected]>

In main_loop_s(), listensock was closed nondeterministically inside maybe_close() during the accept loop, and error paths closed it without tracking, leading to double-close or leaks across repeated iterations (via goto again).

Fix by making maybe_close() return a bool indicating whether it closed the fd, and track that per iteration. Funnel poll/accept error paths through a single 'goto out' label and close the socket there only if not already closed. Reset 'closed' at the 'again:' label so each iteration starts with a fresh tracking state. This ensures exactly one close per iteration in all cases.

---
 .../selftests/net/mptcp/mptcp_connect.c       | 24 +++++++++++++------
 1 file changed, 17 insertions(+), 7 deletions(-)
tools/testing/selftests/net/mptcp/mptcp_connect.c +17 -7
index f6ebdc5bf745..b04c99605518 100644
@@ -1323,12 +1323,15 @@ static void check_getpeername_connect(int fd)
 			cfg_host, a, cfg_port, b);
 }
 
-static void maybe_close(int fd)
+static bool maybe_close(int fd)
 {
 	unsigned int r = rand();
 
-	if (!(cfg_join || cfg_remove || cfg_repeat > 1) && (r & 1))
+	if (!(cfg_join || cfg_remove || cfg_repeat > 1) && (r & 1)) {
 		close(fd);
+		return true;
+	}
+	return false;
 }
 
 int main_loop_s(int listensock)
@@ -1338,27 +1341,30 @@ int main_loop_s(int listensock)
 	struct pollfd polls;
 	socklen_t salen;
 	int remotesock;
+	bool closed;
 	int err = 0;
 	int fd = 0;
 
 again:
+	closed = false;
 	polls.fd = listensock;
 	polls.events = POLLIN;
 
 	switch (poll(&polls, 1, poll_timeout)) {
 	case -1:
 		perror("poll");
-		return 1;
+		err = 1;
+		goto out;
 	case 0:
 		fprintf(stderr, "%s: timed out\n", __func__);
-		close(listensock);
-		return 2;
+		err = 2;
+		goto out;
 	}
 
 	salen = sizeof(ss);
 	remotesock = accept(listensock, (struct sockaddr *)&ss, &salen);
 	if (remotesock >= 0) {
-		maybe_close(listensock);
+		closed = maybe_close(listensock);
 		check_sockaddr(pf, &ss, salen);
 		check_getpeername(remotesock, &ss, salen);
 
@@ -1374,7 +1380,8 @@ int main_loop_s(int listensock)
 		err = copyfd_io(fd, remotesock, 1, true, &winfo);
 	} else {
 		perror("accept");
-		return 1;
+		err = 1;
+		goto out;
 	}
 
 	if (cfg_input)
@@ -1383,6 +1390,9 @@ int main_loop_s(int listensock)
 	if (!err && --cfg_repeat > 0)
 		goto again;
 
+out:
+	if (!closed)
+		close(listensock);
 	return err;
 }
 
Signature · 1 line
2.53.0
#10 lore raw
In reply to Geliang Tang

Hi Geliang,

Thank you for your modifications, that's great!

Our CI did some validations and here is its report:

- KVM Validation: normal (except selftest_mptcp_join): Unstable: 1 failed test(s): packetdrill_add_addr ⚠️
- KVM Validation: normal (only selftest_mptcp_join): Success! ✅
- KVM Validation: debug (except selftest_mptcp_join): Success! ✅
- KVM Validation: debug (only selftest_mptcp_join): Success! ✅
- KVM Validation: btf-normal (only bpftest_all): Success! ✅
- KVM Validation: btf-debug (only bpftest_all): Success! ✅
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/29556502423

Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/ad26cdbf6ab5
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=1129243

If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:

    $ cd [kernel source code]
    $ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
        --pull always mptcp/mptcp-upstream-virtme-docker:latest \
        auto-normal

For more details:

https://github.com/multipath-tcp/mptcp-upstream-virtme-docker

Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)