Skip to contentSkip to conversation

mptcp: partial recvmsg() refactor

PATCH In review Paolo Abeni 3 participants 5 messages Sep 18, 2019 – Sep 21, 2019

Subsystems (2)
lore
Share & tools
lore

Paolo Abeni

author
#0 lore raw

v2 cover mptcp: partial recvmsg() refactor

This 2 patches start the recvmsg() refactor discussed in last mtg. The explicit 'duplicate data' handling in 1/2 fix some stream corruption I observe with re-injection, and the 2nd one fix another stream corruption at the end of the stream, still triggered by re-injection.

With the above 2 patches I still stream corruptions with re-injection, are almost gone - even if I can still hit them with in at least a very rare corner case.

Only minor changes since v1, to address Mat's feedback, details in the individual changelog

Paolo Abeni (2):
mptcp: refactor recvmsg()
mptcp: recvmsg() should really try to deliver 'len' bytes to
user-space

 net/mptcp/protocol.c | 120 +++++++++++++++++++------------------------
 1 file changed, 54 insertions(+), 66 deletions(-)
Signature · 1 line
2.21.0
author
#1 lore raw

PATCH v2 1/2 mptcp: refactor recvmsg()

The current implementation tries to handle exceptional sitations (no mapping, bad subflow sequence number) without baling out, but that makes debugging existing bugs very difficult.

Also duplicate MPTCP data due to retransmissions are hard to track, because we discard the duplicate mapping and skip the following pkts lacking the map without any fourther control.

This changeset refactor the main loop to return an error code to the user space for unhandled exceptional condition. It also track explicitly the number of duplicate MPTCP data bytes to be discarded: duplicate mapping are still considered valid, and we just discard the bytes belonging to such mapping.

This makes debugging easier, and also clean-up a bit the existing code

v1 -> v2:
- rewrite warn and debug message to fit a single line

Signed-off-by: Paolo Abeni <pabeni(a)redhat.com>
--
I'm ok to squash this into commit 786f07cf97f1 ("mptcp: Implement MPTCP receive path")

---
 net/mptcp/protocol.c | 113 ++++++++++++++++++-------------------------
 1 file changed, 47 insertions(+), 66 deletions(-)
net/mptcp/protocol.c +47 -66
index fba7e00df4ff..dc59baeca0c7 100644
@@ -452,17 +452,6 @@ static int mptcp_read_actor(read_descriptor_t *desc, struct sk_buff *skb,
 	return copy_len;
 }
 
-static int mptcp_flush_actor(read_descriptor_t *desc, struct sk_buff *skb,
-			     unsigned int offset, size_t len)
-{
-	pr_debug("Flushing one skb with %zu of %zu bytes remaining",
-		 len, len + offset);
-
-	desc->count = 0;
-
-	return len;
-}
-
 enum mapping_status {
 	MAPPING_ADDED,
 	MAPPING_MISSING,
@@ -568,6 +557,12 @@ static void mptcp_wait_data(struct sock *sk, long *timeo)
 	remove_wait_queue(sk_sleep(sk), &wait);
 }
 
+static void warn_bad_map(struct subflow_context *subflow, u32 ssn)
+{
+	WARN_ONCE(1, "Bad mapping: ssn=%d map_seq=%d map_data_len=%d",
+		  ssn, subflow->map_subflow_seq, subflow->map_data_len);
+}
+
 static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 			 int nonblock, int flags, int *addr_len)
 {
@@ -611,7 +606,6 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 
 	while (copied < len) {
 		enum mapping_status status;
-		size_t discard_len = 0;
 		u32 map_remaining;
 		int bytes_read;
 		u64 ack_seq;
@@ -627,20 +621,27 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 		if (status == MAPPING_ADDED) {
 			/* Common case, but nothing to do here */
 		} else if (status == MAPPING_MISSING) {
-			if (!subflow->map_valid) {
-				pr_debug("Mapping missing, trying next skb");
-
-				arg.msg = NULL;
-				desc.count = SIZE_MAX;
-
-				bytes_read = tcp_read_sock(ssk, &desc,
-							   mptcp_flush_actor);
-
-				if (bytes_read < 0)
-					break;
+			struct sk_buff *skb = skb_peek(&ssk->sk_receive_queue);
 
+			if (!skb->len) {
+				/* the TCP stack deliver 0 len FIN pkt
+				 * to the receive queue, that is the only
+				 * 0len pkts ever expected here, and we can
+				 * admit no mapping only for 0 len pkts
+				 */
+				if (!(TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN))
+					WARN_ONCE(1, "0len seq %d:%d flags %x",
+						  TCP_SKB_CB(skb)->seq,
+						  TCP_SKB_CB(skb)->end_seq,
+						  TCP_SKB_CB(skb)->tcp_flags);
+				sk_eat_skb(sk, skb);
 				continue;
 			}
+			if (!subflow->map_valid) {
+				WARN_ONCE(1, "corrupted stream: missing mapping");
+				copied = -EBADFD;
+				break;
+			}
 		} else if (status == MAPPING_EMPTY) {
 			goto wait_for_data;
 		} else if (status == MAPPING_DATA_FIN) {
@@ -658,60 +659,39 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 
 		if (unlikely(before(ssn, subflow->map_subflow_seq))) {
 			/* Mapping covers data later in the subflow stream,
-			 * discard unmapped data.
+			 * currently unsupported.
 			 */
-			pr_debug("Mapping covers data later in stream");
-			discard_len = subflow->map_subflow_seq - ssn;
+			warn_bad_map(subflow, ssn);
+			copied = -EBADFD;
+			break;
 		} else if (unlikely(!before(ssn, (subflow->map_subflow_seq +
 						  subflow->map_data_len)))) {
 			/* Mapping ends earlier in the subflow stream.
-			 * Invalidate the mapping and try again.
+			 * Invalid
 			 */
-			subflow->map_valid = 0;
-			pr_debug("Invalid mapping ssn=%d map_seq=%d map_data_len=%d",
-				 ssn, subflow->map_subflow_seq,
-				 subflow->map_data_len);
-			continue;
-		} else {
-			ack_seq = get_mapped_dsn(subflow);
-
-			if (before64(ack_seq, old_ack)) {
-				/* Mapping covers data already received,
-				 * discard data in the current mapping
-				 * and invalidate the map
-				 */
-				u64 map_end_dsn = subflow->map_seq +
-					subflow->map_data_len;
-				discard_len = min(map_end_dsn - ack_seq,
-						  old_ack - ack_seq);
-				subflow->map_valid = 0;
-				pr_debug("Duplicate MPTCP data found");
-			}
+			warn_bad_map(subflow, ssn);
+			copied = -EBADFD;
+			break;
 		}
 
-		if (discard_len) {
-			/* Discard data for the current mapping.
-			 */
-			pr_debug("Discard %zu bytes", discard_len);
+		ack_seq = get_mapped_dsn(subflow);
+		map_remaining = subflow->map_data_len - get_map_offset(subflow);
 
+		if (before64(ack_seq, old_ack)) {
+			/* Mapping covers data already received, discard data
+			 * in the current mapping
+			 */
 			arg.msg = NULL;
-			desc.count = discard_len;
-
-			bytes_read = tcp_read_sock(ssk, &desc,
-						   mptcp_read_actor);
-
-			if (bytes_read < 0)
-				break;
-			else if (bytes_read == discard_len)
-				continue;
-			else
-				goto wait_for_data;
+			desc.count = min_t(size_t, old_ack - ack_seq,
+					   map_remaining);
+			pr_debug("Dup data, map len=%d acked=%lld dropped=%ld",
+				 map_remaining, old_ack - ack_seq, desc.count);
+		} else {
+			arg.msg = msg;
+			desc.count = min_t(size_t, len - copied, map_remaining);
 		}
 
 		/* Read mapped data */
-		map_remaining = subflow->map_data_len - get_map_offset(subflow);
-		desc.count = min_t(size_t, len - copied, map_remaining);
-		arg.msg = msg;
 		bytes_read = tcp_read_sock(ssk, &desc, mptcp_read_actor);
 		if (bytes_read < 0)
 			break;
@@ -730,7 +710,8 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 				 subflow->map_data_len);
 		}
 
-		copied += bytes_read;
+		if (arg.msg)
+			copied += bytes_read;
 
 wait_for_data:
 		if (copied)
Signature · 1 line
2.21.0
author
#2 lore raw

PATCH v2 2/2 mptcp: recvmsg() should really try to deliver 'len' bytes to user-space

In reply to Paolo Abeni

Currently we break the recvmsg() main loop as soon as we have copied any amount of data or the ssk is closed.

With reinjection we can receive duplicate data and reach the 'wait_for_data' label with 'copied == 0'.
The subflow sk could be already in closed status, if the local socket has already received all the data from the peer (and the data is sitting into the receive queue).

When we meet the above scenario, currently recvmsg() terminates with return code '0' and the user space interpret that as EoF, even if there is potentially a lot of data pending.

This change addresses the issue checking for pending data before going through the 'wait_for_data' code path.

v1 -> v2:
- fix type in comment

Signed-off-by: Paolo Abeni <pabeni(a)redhat.com>
--
I'm ok with squashing to commit 786f07cf97f1 ("mptcp: Implement MPTCP receive path")

---
 net/mptcp/protocol.c | 7 +++++++
 1 file changed, 7 insertions(+)
net/mptcp/protocol.c +7 -0
index dc59baeca0c7..19bbe16993e9 100644
@@ -713,6 +713,13 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 		if (arg.msg)
 			copied += bytes_read;
 
+		/* The 'wait_for_data' code path can terminate the receive loop
+		 * in a number of scenarios: check if more data is pending
+		 * before giving up.
+		 */
+		if (!skb_queue_empty(&ssk->sk_receive_queue))
+			continue;
+
 wait_for_data:
 		if (copied)
 			break;
Signature · 1 line
2.21.0
#3 lore raw
In reply to Paolo Abeni
Quoted 12 lines · Paolo Abeni

This 2 patches start the recvmsg() refactor discussed in last mtg. The explicit 'duplicate data' handling in 1/2 fix some stream corruption I observe with re-injection, and the 2nd one fix another stream corruption at the end of the stream, still triggered by re-injection.

With the above 2 patches I still stream corruptions with re-injection, are almost gone - even if I can still hit them with in at least a very rare corner case.

Only minor changes since v1, to address Mat's feedback, details in the individual changelog

Quoted 10 lines

Paolo Abeni (2):
mptcp: refactor recvmsg()
mptcp: recvmsg() should really try to deliver 'len' bytes to
user-space

net/mptcp/protocol.c | 120 +++++++++++++++++++------------------------

1 file changed, 54 insertions(+), 66 deletions(-)
Signature · 1 line
2.21.0

Thanks for the v2. These look good to merge.

--
Mat Martineau
Intel

#4 lore raw

Hi Paolo, Mat,

Quoted 23 lines · Mat Martineau
Quoted 21 lines · Paolo Abeni

This 2 patches start the recvmsg() refactor discussed in last mtg. The explicit 'duplicate data' handling in 1/2 fix some stream corruption I
observe with re-injection, and the 2nd one fix another stream corruption at
the end of the stream, still triggered  by re-injection.

With the above 2 patches I still stream corruptions with re-injection, are
almost gone - even if I can still hit them with in at least a very rare corner
case.

Only minor changes since v1, to address Mat's feedback, details in the individual changelog

Paolo Abeni (2):
 mptcp: refactor recvmsg()
 mptcp: recvmsg() should really try to deliver 'len' bytes to user-space

Quoted 7 lines
Quoted 5 lines

net/mptcp/protocol.c | 120 +++++++++++++++++++------------------------

1 file changed, 54 insertions(+), 66 deletions(-)

--
2.21.0

Thanks for the v2. These look good to merge.

Thank you for the patches and the reviews!

- ae3e015a67e3: "squashed" the first one in "mptcp: Implement MPTCP receive path"
- signed off was already there
- eebee22fe45e: "squashed" the second one in "mptcp: Implement MPTCP receive path" as well.
- 380876e851ad: conflict in t/mptcp-Make-MPTCP-socket-block-wakeup-ignore-sk_receive_queue
- 782be479bf43..0341ab0068b3: result

This fixed all the previous failed tests, that's good :)

Cheers,
Matt

Signature · 5 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium