Skip to contentSkip to conversation

sendmsg: fix pending issues.

PATCH In review Paolo Abeni 3 participants 7 messages Dec 4, 2019 – Dec 6, 2019

Subsystems (2)
lore
Share & tools
lore

Paolo Abeni

author
#0 lore raw

v3 cover sendmsg: fix pending issues.

This includes the change suggested by Mat. Bumped revision 3 to avoid confusion with intermediated 2/3 v2.

The patch 1 should be inserted just after "tcp: clean ext on tx recycle"

The patches 2 and 3 have explicit individual squash-to tags, but will casue a lot of conflicts, the uneasy ones to solve are on patch "mptcp: rework mptcp_sendmsg_frag to accept optional dfrag"

A tree rebased with the above patches applied is available at:

https://github.com/pabeni/mptcp/tree/mptcp_net-next-FIN_v1_recvmsg_2

The original cover letter was as follow
---

This series addresses the known pending issue in mptcp_sendmsg, specifically:

* the crash repored by matt (patch 3/3)
* the possible data corruption on ext allocation failure - the "TODO"" in sendmsg_frag (patches 1,2/3)

uunfortunately none of the above is actually tested, since I can't reproduce the issues - still the problems look real.

This is not squashed yet, as it will conflict with most patches, I'm just trying to collect feedback.

Open points:

The patch 2/3 introduces a per socket cache of a single skb_ext. As per recent upstream changes related to TCP rx/tx skb cache, that is debatable, could have subtle performance and memory usage implications. We could avoid such cache allways allocating the extension for each sendmsg_frag() call, and than free it if unused. Both options looks sub-optiomal to me do you have any preference? - I do have some for the selected impl ;)

The new API exposed by patch 1/3 is open to misusage. e.g. the caller could allocate a skb_ext, set arbitrary ext->offset[id] values and than call __skb_ext_add(). That will lead to memory leak or GPF. I did no try to address the issue, the '__' prefix should guide the users to extra care. Can we do any better?

(this is a pre-existing topic) the sk_stream_wait_memory() calls in sk_stream_wait_memory(), waits on the subflow. Should we wait on the mptcp socket instead? Otherwise should we release the mptcp socket lock, before such call?

Paolo Abeni (1):
skb: add helpers to allocate ext independently from sk_buff
mptcp: avoid data stream corruption on allocation failure.
mptcp: properly detect skb collapsing

 include/linux/skbuff.h |    3 ++
 net/core/skbuff.c      |   35 ++++++++++++++++++++++++++++++--
 net/mptcp/protocol.c   |   53 +++++++++++++++++++++++++++++++------------------
 net/mptcp/protocol.h   |    1 
 4 files changed, 71 insertions(+), 21 deletions(-)
Signature · 1 line
2.21.0
author
#1 lore raw

PATCH v3 1/3 skb: add helpers to allocate ext independently from sk_buff

Currently we can allocate the extension only after the skb, this change allows the user to do the opposite, will simplify allocation failure handling from MPTCP.

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>

---
v1 -> v3:
- renamed the helpers, add comment, set explicitly the active_extensions mask (Mat)

---
 include/linux/skbuff.h |  3 +++
 net/core/skbuff.c      | 35 +++++++++++++++++++++++++++++++++--
 2 files changed, 36 insertions(+), 2 deletions(-)
include/linux/skbuff.h +3 -0
index b4ef8b758828..bf2eee2b2b02 100644
@@ -4114,6 +4114,9 @@ struct skb_ext {
 	char data[0] __aligned(8);
 };
 
+struct skb_ext *__skb_ext_alloc(void);
+void *__skb_ext_set(struct sk_buff *skb, enum skb_ext_id id,
+		    struct skb_ext *ext);
 void *skb_ext_add(struct sk_buff *skb, enum skb_ext_id id);
 void __skb_ext_del(struct sk_buff *skb, enum skb_ext_id id);
 void __skb_ext_put(struct skb_ext *ext);
net/core/skbuff.c +33 -2
index 0aba2cd69881..2218d700e371 100644
@@ -5981,7 +5981,14 @@ static void *skb_ext_get_ptr(struct skb_ext *ext, enum skb_ext_id id)
 	return (void *)ext + (ext->offset[id] * SKB_EXT_ALIGN_VALUE);
 }
 
-static struct skb_ext *skb_ext_alloc(void)
+/**
+ * __skb_ext_alloc - allocate a new skb extensions storage
+ *
+ * Returns the newly allocated pointer. The pointer can later attached to a
+ * skb via __skb_ext_set().
+ * Note: caller must handle the skb_ext as an opaque data.
+ */
+struct skb_ext *__skb_ext_alloc(void)
 {
 	struct skb_ext *new = kmem_cache_alloc(skbuff_ext_cache, GFP_ATOMIC);
 
@@ -6021,6 +6028,30 @@ static struct skb_ext *skb_ext_maybe_cow(struct skb_ext *old,
 	return new;
 }
 
+/**
+ * __skb_ext_set - attach the specified extension storage to this skb
+ * @skb: buffer
+ * @id: extension id
+ * @ext: extension storage previously allocated via __skb_ext_alloc()
+ *
+ * Existing extensions, if any, are cleared.
+ *
+ * Returns the pointer to the extension.
+ */
+void *__skb_ext_set(struct sk_buff *skb, enum skb_ext_id id,
+		    struct skb_ext *ext)
+{
+	unsigned int newlen, newoff = SKB_EXT_CHUNKSIZEOF(*ext);
+
+	skb_ext_put(skb);
+	newlen = newoff + skb_ext_type_len[id];
+	ext->chunks = newlen;
+	ext->offset[id] = newoff;
+	skb->extensions = ext;
+	skb->active_extensions = 1 << id;
+	return skb_ext_get_ptr(ext, id);
+}
+
 /**
  * skb_ext_add - allocate space for given extension, COW if needed
  * @skb: buffer
@@ -6054,7 +6085,7 @@ void *skb_ext_add(struct sk_buff *skb, enum skb_ext_id id)
 	} else {
 		newoff = SKB_EXT_CHUNKSIZEOF(*new);
 
-		new = skb_ext_alloc();
+		new = __skb_ext_alloc();
 		if (!new)
 			return NULL;
 	}
Signature · 1 line
2.21.0
author
#2 lore raw

PATCH v3 2/3 mptcp: avoid data stream corruption on allocation failure.

In reply to Paolo Abeni

Currently, when if mpext allocation fails, the data stream is corrupted, as we can add the required DSS nor roll-back the TCP status.

Allocate a skb_ext before the actually skb_buff creation/TCP status update: in case of memory allocation failure we can bail early, avoiding the above condition.

Additionally, if the ext is unused to to skb collapsing, cache the ext in the msk for the next send.

Squash-to: "mptcp: Write MPTCP DSS headers to outgoing data packets"
Signed-off-by: Paolo Abeni <pabeni(a)redhat.com>
---
v2 -> v3:
- rebased after skb ext helpers rename change

v1 -> v2:
- properly convert to bool mptcp_ext_cache_refill() return value

---
 net/mptcp/protocol.c | 39 +++++++++++++++++++++++++--------------
 net/mptcp/protocol.h |  1 +
 2 files changed, 26 insertions(+), 14 deletions(-)
net/mptcp/protocol.c +25 -14
index 3180202e56b4..1808df510cb3 100644
@@ -55,6 +55,14 @@ static struct sock *mptcp_subflow_get(const struct mptcp_sock *msk)
 	return NULL;
 }
 
+static bool mptcp_ext_cache_refill(struct mptcp_sock *msk)
+{
+	if (!msk->cached_ext)
+		msk->cached_ext = __skb_ext_alloc();
+
+	return !!msk->cached_ext;
+}
+
 static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 			      struct msghdr *msg, long *timeo)
 {
@@ -69,7 +77,8 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	 * from one substream to another, but do per subflow memory accounting
 	 */
 	pfrag = sk_page_frag(sk);
-	while (!sk_page_frag_refill(ssk, pfrag)) {
+	while (!sk_page_frag_refill(ssk, pfrag) ||
+	       !mptcp_ext_cache_refill(msk)) {
 		ret = sk_stream_wait_memory(ssk, timeo);
 		if (ret)
 			return ret;
@@ -103,19 +112,19 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 		iov_iter_revert(&msg->msg_iter, psize - ret);
 
 	skb = tcp_write_queue_tail(ssk);
-	mpext = skb_ext_add(skb, SKB_EXT_MPTCP);
-	if (mpext) {
-		memset(mpext, 0, sizeof(*mpext));
-		mpext->data_seq = msk->write_seq;
-		mpext->subflow_seq = mptcp_subflow_ctx(ssk)->rel_write_seq;
-		mpext->data_len = ret;
-		mpext->use_map = 1;
-		mpext->dsn64 = 1;
-
-		pr_debug("data_seq=%llu subflow_seq=%u data_len=%u dsn64=%d",
-			 mpext->data_seq, mpext->subflow_seq, mpext->data_len,
-			 mpext->dsn64);
-	} /* TODO: else fallback */
+	mpext = __skb_ext_set(skb, SKB_EXT_MPTCP, msk->cached_ext);
+	msk->cached_ext = NULL;
+
+	memset(mpext, 0, sizeof(*mpext));
+	mpext->data_seq = msk->write_seq;
+	mpext->subflow_seq = mptcp_subflow_ctx(ssk)->rel_write_seq;
+	mpext->data_len = ret;
+	mpext->use_map = 1;
+	mpext->dsn64 = 1;
+
+	pr_debug("data_seq=%llu subflow_seq=%u data_len=%u dsn64=%d",
+		 mpext->data_seq, mpext->subflow_seq, mpext->data_len,
+		 mpext->dsn64);
 
 	pfrag->offset += ret;
 	msk->write_seq += ret;
@@ -242,6 +251,8 @@ static void mptcp_close(struct sock *sk, long timeout)
 		sock_release(mptcp_subflow_tcp_socket(subflow));
 	}
 
+	if (msk->cached_ext)
+		__skb_ext_put(msk->cached_ext);
 	release_sock(sk);
 	sk_common_release(sk);
 }
net/mptcp/protocol.h +1 -0
index 412718c1ae5c..3b8d1548dda1 100644
@@ -58,6 +58,7 @@ struct mptcp_sock {
 	u64		write_seq;
 	u64		ack_seq;
 	u32		token;
+	struct	skb_ext *cached_ext;	/* for the next sendmsg */
 	struct list_head conn_list;
 	struct socket	*subflow; /* outgoing connect/listener/!mp_capable */
 };
Signature · 1 line
2.21.0
author
#3 lore raw

PATCH v3 3/3 mptcp: properly detect skb collapsing

In reply to Paolo Abeni

Currently we can misdetect skb collapsing, which will lead to a kernel warning and crash.

Changing a bit the detection logic for skb collapsing we can address the issue.

Squash-to: mptcp: allow collapsing consecutive sendpages on the same substream
Signed-off-by: Paolo Abeni <pabeni(a)redhat.com>
--
v1 -> v3:
- cleanup the 'collapsed' check (Mat)

---
 net/mptcp/protocol.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
net/mptcp/protocol.c +9 -5
index d83569672129..8fb93cc3d43f 100644
@@ -105,10 +105,10 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 {
 	int mss_now, avail_size, size_goal, ret;
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	bool collapsed, can_collapse = false;
 	struct mptcp_ext *mpext = NULL;
+	struct sk_buff *skb, *tail;
+	bool can_collapse = false;
 	struct page_frag *pfrag;
-	struct sk_buff *skb;
 	size_t psize;
 
 	/* use the mptcp page cache so that we can easily move the data
@@ -165,10 +165,14 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	if (unlikely(ret < psize))
 		iov_iter_revert(&msg->msg_iter, psize - ret);
 
-	collapsed = skb == tcp_write_queue_tail(ssk);
-	if (collapsed) {
+	/* if the tail skb extension is still the cached one, collapsing
+	 * really happened. Note: we can't check for 'same skb' as the sk_buff
+	 * hdr on tail can be transmitted, freed and re-allocated by the
+	 * do_tcp_sendpages() call
+	 */
+	tail = tcp_write_queue_tail(ssk);
+	if (mpext && tail && mpext == skb_ext_find(tail, SKB_EXT_MPTCP)) {
 		WARN_ON_ONCE(!can_collapse);
-		/* when collapsing mpext always exists */
 		mpext->data_len += ret;
 		goto out;
 	}
Signature · 1 line
2.21.0
#4 lore raw
In reply to Paolo Abeni
Quoted 2 lines · Paolo Abeni

This includes the change suggested by Mat. Bumped revision 3 to avoid confusion with intermediated 2/3 v2.

Thanks, Paolo. v3 looks good to me.

Mat

Quoted 53 lines

The patch 1 should be inserted just after "tcp: clean ext on tx recycle"

The patches 2 and 3 have explicit individual squash-to tags, but will casue a lot of conflicts, the uneasy ones to solve are on patch "mptcp: rework mptcp_sendmsg_frag to accept optional dfrag"

A tree rebased with the above patches applied is available at:

https://github.com/pabeni/mptcp/tree/mptcp_net-next-FIN_v1_recvmsg_2

The original cover letter was as follow
---

This series addresses the known pending issue in mptcp_sendmsg, specifically:

* the crash repored by matt (patch 3/3)
* the possible data corruption on ext allocation failure - the "TODO"" in sendmsg_frag (patches 1,2/3)

uunfortunately none of the above is actually tested, since I can't reproduce the issues - still the problems look real.

This is not squashed yet, as it will conflict with most patches, I'm just trying to collect feedback.

Open points:

The patch 2/3 introduces a per socket cache of a single skb_ext. As per recent upstream changes related to TCP rx/tx skb cache, that is debatable, could have subtle performance and memory usage implications. We could avoid such cache allways allocating the extension for each sendmsg_frag() call, and than free it if unused. Both options looks sub-optiomal to me do you have any preference? - I do have some for the selected impl ;)

The new API exposed by patch 1/3 is open to misusage. e.g. the caller could allocate a skb_ext, set arbitrary ext->offset[id] values and than call __skb_ext_add(). That will lead to memory leak or GPF. I did no try to address the issue, the '__' prefix should guide the users to extra care. Can we do any better?

(this is a pre-existing topic) the sk_stream_wait_memory() calls in sk_stream_wait_memory(), waits on the subflow. Should we wait on the mptcp socket instead? Otherwise should we release the mptcp socket lock, before such call?

Paolo Abeni (1):
skb: add helpers to allocate ext independently from sk_buff
mptcp: avoid data stream corruption on allocation failure.
mptcp: properly detect skb collapsing

Quoted 12 lines
include/linux/skbuff.h |    3 ++
net/core/skbuff.c      |   35 ++++++++++++++++++++++++++++++--
net/mptcp/protocol.c   |   53 +++++++++++++++++++++++++++++++------------------
net/mptcp/protocol.h   |    1
4 files changed, 71 insertions(+), 21 deletions(-)
Signature · 4 lines
2.21.0
_______________________________________________
mptcp mailing list -- mptcp(a)lists.01.org
To unsubscribe send an email to mptcp-leave(a)lists.01.org

--
Mat Martineau
Intel

#5 lore raw

Hi Paolo, Mat,

Quoted 7 lines · Mat Martineau
Quoted 2 lines · Paolo Abeni

This includes the change suggested by Mat. Bumped revision 3 to avoid confusion with intermediated 2/3 v2.

Thanks, Paolo. v3 looks good to me.

Thank you for the patches and the review!

- patch 1/3 has been added just before: "mptcp: Add MPTCP socket stubs"
- no conflicts
- 14eb86607211: "squashed" patch 2/3 in "mptcp: Write MPTCP DSS headers to outgoing data packets"
- 6e357ac6f397: mptcp: fix alignement in protocol.h (tab/space)
- 59a474684481: conflict in t/mptcp-recvmsg-can-drain-data-from-multiple-subflows
- dd7d2904f7c4: conflict in t/mptcp-allow-collapsing-consecutive-sendpages-on-the-same-substream
- d41ac2e4911d: "squashed" patch 3/3 in "mptcp: allow collapsing consecutive sendpages on the same substream"
- 67b1e688acac: conflict in t/mptcp-queue-data-for-mptcp-level-retransmission (note: compared to the ref branch, I had to remove 'collapse' variable, not used ; and "can_collapse" was declared twice)
- 47890b23de5d: conflict in t/mptcp-implement-memory-accounting-for-mptcp-rtx-queue
- 46d9c6a121dc: conflict in t/mptcp-rework-mptcp_sendmsg_frag-to-accept-optional-dfrag
- 1cc43731a90d..d6c1939a16c5: result

Tests are still the same. They all passed except the one without packets re-ordering due to a timeout (known issue).

Cheers,
Matt

Signature · 5 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
author
#6 lore raw
Quoted 24 lines · Matthieu Baerts

Hi Paolo, Mat,

Quoted 6 lines · Mat Martineau
Quoted 2 lines · Paolo Abeni

This includes the change suggested by Mat. Bumped revision 3 to avoid confusion with intermediated 2/3 v2.

Thanks, Paolo. v3 looks good to me.

Thank you for the patches and the review!

- patch 1/3 has been added just before: "mptcp: Add MPTCP socket stubs"
- no conflicts
- 14eb86607211: "squashed" patch 2/3 in "mptcp: Write MPTCP DSS headers to outgoing data packets"
- 6e357ac6f397: mptcp: fix alignement in protocol.h (tab/space)
- 59a474684481: conflict in t/mptcp-recvmsg-can-drain-data-from-multiple-subflows
- dd7d2904f7c4: conflict in t/mptcp-allow-collapsing-consecutive-sendpages-on-the-same-substream
- d41ac2e4911d: "squashed" patch 3/3 in "mptcp: allow collapsing consecutive sendpages on the same substream"
- 67b1e688acac: conflict in

Quoted 8 lines

t/mptcp-queue-data-for-mptcp-level-retransmission (note: compared to the ref branch, I had to remove 'collapse' variable, not used ; and "can_collapse" was declared twice)
- 47890b23de5d: conflict in t/mptcp-implement-memory-accounting-for-mptcp-rtx-queue
- 46d9c6a121dc: conflict in t/mptcp-rework-mptcp_sendmsg_frag-to-accept-optional-dfrag
- 1cc43731a90d..d6c1939a16c5: result

Conflicts resolution LGTM, thanks!

46d9c6a121dc was a tricky one :(

Cheers,

Paolo