mptcp: partial recvmsg() refactor
Subsystems (2)
Paolo Abeni
v2 cover mptcp: partial recvmsg() refactor
This 2 patches start the recvmsg() refactor discussed in last mtg. The explicit 'duplicate data' handling in 1/2 fix some stream corruption I observe with re-injection, and the 2nd one fix another stream corruption at the end of the stream, still triggered by re-injection.
With the above 2 patches I still stream corruptions with re-injection, are almost gone - even if I can still hit them with in at least a very rare corner case.
Only minor changes since v1, to address Mat's feedback, details in the individual changelog
Paolo Abeni (2):
mptcp: refactor recvmsg()
mptcp: recvmsg() should really try to deliver 'len' bytes to
user-space
net/mptcp/protocol.c | 120 +++++++++++++++++++------------------------ 1 file changed, 54 insertions(+), 66 deletions(-)
Signature · 1 line
2.21.0
Paolo Abeni
PATCH v2 1/2 mptcp: refactor recvmsg()
The current implementation tries to handle exceptional sitations (no mapping, bad subflow sequence number) without baling out, but that makes debugging existing bugs very difficult.
Also duplicate MPTCP data due to retransmissions are hard to track, because we discard the duplicate mapping and skip the following pkts lacking the map without any fourther control.
This changeset refactor the main loop to return an error code to the user space for unhandled exceptional condition. It also track explicitly the number of duplicate MPTCP data bytes to be discarded: duplicate mapping are still considered valid, and we just discard the bytes belonging to such mapping.
This makes debugging easier, and also clean-up a bit the existing code
v1 -> v2:
- rewrite warn and debug message to fit a single line
Signed-off-by: Paolo Abeni <pabeni(a)redhat.com>
--
I'm ok to squash this into commit 786f07cf97f1 ("mptcp: Implement MPTCP receive path")
--- net/mptcp/protocol.c | 113 ++++++++++++++++++------------------------- 1 file changed, 47 insertions(+), 66 deletions(-)
net/mptcp/protocol.c +47 -66
@@ -452,17 +452,6 @@ static int mptcp_read_actor(read_descriptor_t *desc, struct sk_buff *skb,return copy_len; } -static int mptcp_flush_actor (read_descriptor_t *desc,struct sk_buff *skb, -unsigned int offset,size_t len) -{ -pr_debug ("Flushing one skb with %zu of %zu bytes remaining" , - len, len + offset); - - desc->count =0 ; - -return len; -} -enum mapping_status {MAPPING_ADDED ,MAPPING_MISSING ,
@@ -568,6 +557,12 @@ static void mptcp_wait_data(struct sock *sk, long *timeo)remove_wait_queue (sk_sleep (sk), &wait); } +static void warn_bad_map (struct subflow_context *subflow,u32 ssn) +{ +WARN_ONCE (1 ,"Bad mapping: ssn=%d map_seq=%d map_data_len=%d" , + ssn, subflow->map_subflow_seq, subflow->map_data_len); +} +static int mptcp_recvmsg (struct sock *sk,struct msghdr *msg,size_t len,int nonblock,int flags,int *addr_len) {
@@ -611,7 +606,6 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,while (copied < len) {enum mapping_status status; -size_t discard_len =0 ;u32 map_remaining;int bytes_read;u64 ack_seq;
@@ -627,20 +621,27 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,if (status ==MAPPING_ADDED ) {/* Common case, but nothing to do here */ }else if (status ==MAPPING_MISSING ) { -if (!subflow->map_valid) { -pr_debug ("Mapping missing, trying next skb" ); - - arg.msg =NULL ; - desc.count =SIZE_MAX ; - - bytes_read =tcp_read_sock (ssk, &desc, - mptcp_flush_actor); - -if (bytes_read <0 ) -break ; +struct sk_buff *skb =skb_peek (&ssk->sk_receive_queue); +if (!skb->len) { +/* the TCP stack deliver 0 len FIN pkt +* to the receive queue, that is the only +* 0len pkts ever expected here, and we can +* admit no mapping only for 0 len pkts +*/ +if (!(TCP_SKB_CB (skb)->tcp_flags &TCPHDR_FIN )) +WARN_ONCE (1 ,"0len seq %d:%d flags %x" , +TCP_SKB_CB (skb)->seq, +TCP_SKB_CB (skb)->end_seq, +TCP_SKB_CB (skb)->tcp_flags); +sk_eat_skb (sk, skb);continue ; } +if (!subflow->map_valid) { +WARN_ONCE (1 ,"corrupted stream: missing mapping" ); + copied = -EBADFD ; +break ; + } }else if (status ==MAPPING_EMPTY ) { goto wait_for_data; }else if (status ==MAPPING_DATA_FIN ) {
@@ -658,60 +659,39 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,if (unlikely (before (ssn, subflow->map_subflow_seq))) {/* Mapping covers data later in the subflow stream, -* discard unmapped data. +* currently unsupported. */ -pr_debug ("Mapping covers data later in stream" ); - discard_len = subflow->map_subflow_seq - ssn; +warn_bad_map (subflow, ssn); + copied = -EBADFD ; +break ; }else if (unlikely (!before (ssn, (subflow->map_subflow_seq + subflow->map_data_len)))) {/* Mapping ends earlier in the subflow stream. -* Invalidate the mapping and try again. +* Invalid */ - subflow->map_valid =0 ; -pr_debug ("Invalid mapping ssn=%d map_seq=%d map_data_len=%d" , - ssn, subflow->map_subflow_seq, - subflow->map_data_len); -continue ; - }else { - ack_seq =get_mapped_dsn (subflow); - -if (before64 (ack_seq, old_ack)) { -/* Mapping covers data already received, -* discard data in the current mapping -* and invalidate the map -*/ -u64 map_end_dsn = subflow->map_seq + - subflow->map_data_len; - discard_len =min (map_end_dsn - ack_seq, - old_ack - ack_seq); - subflow->map_valid =0 ; -pr_debug ("Duplicate MPTCP data found" ); - } +warn_bad_map (subflow, ssn); + copied = -EBADFD ; +break ; } -if (discard_len) { -/* Discard data for the current mapping. -*/ -pr_debug ("Discard %zu bytes" , discard_len); + ack_seq =get_mapped_dsn (subflow); + map_remaining = subflow->map_data_len -get_map_offset (subflow); +if (before64 (ack_seq, old_ack)) { +/* Mapping covers data already received, discard data +* in the current mapping +*/ arg.msg =NULL ; - desc.count = discard_len; - - bytes_read =tcp_read_sock (ssk, &desc, - mptcp_read_actor); - -if (bytes_read <0 ) -break ; -else if (bytes_read == discard_len) -continue ; -else - goto wait_for_data; + desc.count =min_t (size_t, old_ack - ack_seq, + map_remaining); +pr_debug ("Dup data, map len=%d acked=%lld dropped=%ld" , + map_remaining, old_ack - ack_seq, desc.count); + }else { + arg.msg = msg; + desc.count =min_t (size_t, len - copied, map_remaining); }/* Read mapped data */ - map_remaining = subflow->map_data_len -get_map_offset (subflow); - desc.count =min_t (size_t, len - copied, map_remaining); - arg.msg = msg; bytes_read =tcp_read_sock (ssk, &desc, mptcp_read_actor);if (bytes_read <0 )break ;
@@ -730,7 +710,8 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, subflow->map_data_len); } - copied += bytes_read; +if (arg.msg) + copied += bytes_read; wait_for_data:if (copied)
Signature · 1 line
2.21.0
Paolo Abeni
PATCH v2 2/2 mptcp: recvmsg() should really try to deliver 'len' bytes to user-space
In reply to Paolo AbeniCurrently we break the recvmsg() main loop as soon as we have copied any amount of data or the ssk is closed.
With reinjection we can receive duplicate data and reach the 'wait_for_data' label with 'copied == 0'.
The subflow sk could be already in closed status, if the local socket has already received all the data from the peer (and the data is sitting into the receive queue).
When we meet the above scenario, currently recvmsg() terminates with return code '0' and the user space interpret that as EoF, even if there is potentially a lot of data pending.
This change addresses the issue checking for pending data before going through the 'wait_for_data' code path.
v1 -> v2:
- fix type in comment
Signed-off-by: Paolo Abeni <pabeni(a)redhat.com>
--
I'm ok with squashing to commit 786f07cf97f1 ("mptcp: Implement MPTCP receive path")
--- net/mptcp/protocol.c | 7 +++++++ 1 file changed, 7 insertions(+)
net/mptcp/protocol.c +7 -0
@@ -713,6 +713,13 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,if (arg.msg) copied += bytes_read; +/* The 'wait_for_data' code path can terminate the receive loop +* in a number of scenarios: check if more data is pending +* before giving up. +*/ +if (!skb_queue_empty (&ssk->sk_receive_queue)) +continue ; + wait_for_data:if (copied)break ;
Signature · 1 line
2.21.0
Mat Martineau
Quoted 12 lines · Paolo Abeni
This 2 patches start the
recvmsg()refactor discussed in last mtg. The explicit 'duplicate data' handling in 1/2 fix some stream corruption I observe with re-injection, and the 2nd one fix another stream corruption at the end of the stream, still triggered by re-injection.With the above 2 patches I still stream corruptions with re-injection, are almost gone - even if I can still hit them with in at least a very rare corner case.
Only minor changes since v1, to address Mat's feedback, details in the individual changelog
Quoted 10 lines
Paolo Abeni (2):
mptcp: refactorrecvmsg()
mptcp:recvmsg()should really try to deliver 'len' bytes to
user-space
net/mptcp/protocol.c| 120 +++++++++++++++++++------------------------1 file changed, 54 insertions(+), 66 deletions(-)Signature · 1 line
2.21.0
Thanks for the v2. These look good to merge.
--
Mat Martineau
Intel
Matthieu Baerts
Hi Paolo, Mat,
Quoted 23 lines · Mat Martineau
Quoted 21 lines · Paolo Abeni
This 2 patches start the
recvmsg()refactor discussed in last mtg. The explicit 'duplicate data' handling in 1/2 fix some stream corruption I
observe with re-injection, and the 2nd one fix another stream corruption at
the end of the stream, still triggered by re-injection.With the above 2 patches I still stream corruptions with re-injection, are
almost gone - even if I can still hit them with in at least a very rare corner
case.Only minor changes since v1, to address Mat's feedback, details in the individual changelog
Paolo Abeni (2):
mptcp: refactorrecvmsg()
mptcp:recvmsg()should really try to deliver 'len' bytes to user-space
Quoted 7 lines
Quoted 5 lines
net/mptcp/protocol.c| 120 +++++++++++++++++++------------------------1 file changed, 54 insertions(+), 66 deletions(-)--
2.21.0Thanks for the v2. These look good to merge.
Thank you for the patches and the reviews!
- ae3e015a67e3: "squashed" the first one in "mptcp: Implement MPTCP receive path"
- signed off was already there
- eebee22fe45e: "squashed" the second one in "mptcp: Implement MPTCP receive path" as well.
- 380876e851ad: conflict in t/mptcp-Make-MPTCP-socket-block-wakeup-ignore-sk_receive_queue
- 782be479bf43..0341ab0068b3: result
This fixed all the previous failed tests, that's good :)
Cheers,
Matt
Signature · 5 lines
Matthieu Baerts | R&D Engineer matthieu.baerts(a)tessares.net Tessares SA | Hybrid Access Solutions www.tessares.net 1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium