mptcp: address stall under memory pressure
Subsystems (3)
Paolo Abeni
maintainerv3 cover mptcp: address stall under memory pressure
This an attempt to fix the data transfer stall reported by Geliang and Gang more carefully enforcing memory constraints at the MPTCP level.
Patch 1/10 moves the bound check before entering the TCP socket. Patch 2, 3, 4 and 5 are cleanups/refactors finalized to safely re-using TCP helpers on MPTCP skbs.
Patch 6 makes TCP pruning related helpers available to MPTCP and patch 7 makes use of them. Patch 8 addresses an edge scenario that could still lead to transfer stall under memory pressure.
Finally patch 9 and 10 improve the MPTCP-level retransmission schema to make recovery from memory pressure significanly faster.
Note that the diffstat is biases by the quite large patch 4/9, which contains mechanical transformation of existing code; "real" changes are noticiable smaller.
Tested successfully vs the test cases proposed by Geliang and Gang and vs the selftests.
---
Some notes on each patch WRT ignored or false positive issues noticed by sashiko so far.
Paolo Abeni (10):
mptcp: move checks vs rcvbuf size earlier in the RX path
mptcp: drop the mptcp_ooo_try_coalesce() helper
mptcp: drop the cant_coalesce CB field
mptcp: remove CB offset field
mptcp: sync mptcp skb cb layout with tcp one
tcp: expose the tcp_collapse_ofo_queue() helper to mptcp usage, too
mptcp: implemented OoO queue pruning
mptcp: track prune recovery status
mptcp: move the retrans loop to a separate helper
mptcp: let the retrans scheduler do its job.
include/net/tcp.h | 8 + net/ipv4/tcp_input.c | 55 +++--- net/mptcp/fastopen.c | 17 +- net/mptcp/mib.c | 3 + net/mptcp/mib.h | 3 + net/mptcp/options.c | 64 ++++++- net/mptcp/protocol.c | 399 ++++++++++++++++++++++++++++--------------- net/mptcp/protocol.h | 24 ++- net/mptcp/subflow.c | 11 ++ 9 files changed, 414 insertions(+), 170 deletions(-)
Signature · 1 line
2.54.0
Paolo Abeni
maintainerPATCH v3 2/10 mptcp: drop the mptcp_ooo_try_coalesce() helper
It's used to save an additional comparison for in-order skbs, but is also a barrier to remove CB offset. Remove the helper, let __mptcp_try_coalesce() always perform the sequence check and remove duplicate checks from the callers.
--- net/mptcp/protocol.c | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-)
net/mptcp/protocol.c +6 -15
@@ -159,7 +159,8 @@ static bool __mptcp_try_coalesce(struct sock *sk, struct sk_buff *to, {int limit =READ_ONCE (sk->sk_rcvbuf); -if (unlikely (MPTCP_SKB_CB (to)->cant_coalesce) || +if (MPTCP_SKB_CB (from)->map_seq !=MPTCP_SKB_CB (to)->end_seq || +unlikely (MPTCP_SKB_CB (to)->cant_coalesce) ||MPTCP_SKB_CB (from)->offset || ((to->len + from->len) > (limit >>3 )) || !skb_try_coalesce (to, from, fragstolen, delta))
@@ -192,15 +193,6 @@ static bool mptcp_try_coalesce(struct sock *sk, struct sk_buff *to,return true; } -static bool mptcp_ooo_try_coalesce (struct mptcp_sock *msk,struct sk_buff *to, -struct sk_buff *from) -{ -if (MPTCP_SKB_CB (from)->map_seq !=MPTCP_SKB_CB (to)->end_seq) -return false; - -return mptcp_try_coalesce ((struct sock *)msk, to, from); -} -/* "inspired" by tcp_rcvbuf_grow(), main difference: * - mptcp does not maintain a msk-level window clamp * - returns true when the receive buffer is actually updated
@@ -275,7 +267,7 @@ static void mptcp_data_queue_ofo(struct mptcp_sock *msk, struct sk_buff *skb)/* with 2 subflows, adding at end of ooo queue is quite likely * Use of ooo_last_skb avoids the O(Log(N)) rbtree lookup. */ -if (mptcp_ooo_try_coalesce (msk, msk->ooo_last_skb, skb)) { +if (mptcp_try_coalesce (sk, msk->ooo_last_skb, skb)) {MPTCP_INC_STATS (sock_net (sk),MPTCP_MIB_OFOMERGE );MPTCP_INC_STATS (sock_net (sk),MPTCP_MIB_OFOQUEUETAIL );return ;
@@ -321,7 +313,7 @@ static void mptcp_data_queue_ofo(struct mptcp_sock *msk, struct sk_buff *skb)MPTCP_INC_STATS (sock_net (sk),MPTCP_MIB_DUPDATA ); goto merge_right; } - }else if (mptcp_ooo_try_coalesce (msk, skb1, skb)) { + }else if (mptcp_try_coalesce (sk, skb1, skb)) {MPTCP_INC_STATS (sock_net (sk),MPTCP_MIB_OFOMERGE );return ; }
@@ -672,8 +664,7 @@ static void __mptcp_add_backlog(struct sock *sk,if (!list_empty (&msk->backlog_list)) tail = list_last_entry(&msk->backlog_list, struct sk_buff, list); -if (tail &&MPTCP_SKB_CB (skb)->map_seq ==MPTCP_SKB_CB (tail)->end_seq && - ssk == tail->sk && +if (tail && ssk == tail->sk &&__mptcp_try_coalesce (sk, tail, skb, &fragstolen, &delta)) { skb->truesize -= delta;kfree_skb_partial (skb, fragstolen);
@@ -797,7 +788,7 @@ static bool __mptcp_ofo_queue(struct mptcp_sock *msk) end_seq =MPTCP_SKB_CB (skb)->end_seq; tail =skb_peek_tail (&sk->sk_receive_queue); -if (!tail || !mptcp_ooo_try_coalesce (msk, tail, skb)) { +if (!tail || !mptcp_try_coalesce (sk, tail, skb)) {int delta = msk->ack_seq -MPTCP_SKB_CB (skb)->map_seq;/* skip overlapping data, if any */
Signature · 1 line
2.54.0
Paolo Abeni
maintainerPATCH v3 1/10 mptcp: move checks vs rcvbuf size earlier in the RX path
In reply to Paolo AbeniCurrently the enforcement of the rcvbuf constraint is implemented when moving the skbs into the msk receive or OoO queue.
Under significant memory pressure the above can cause permanent data transfer stalls. Move the checks early on, before landing even in the subflow queues.
---
v1 -> v2:
- deal correctly with tcp fin and zero win probe
RFC -> v1:
- limit vs actual buffer size
- use CB info instead of skb->len
Note that:
- this needs the follow-up patches to really fix the stall
- sashiko can assume ZWP carries unacked data and may be silently dropped. AFAIK that is false.
- the memory comparison is intentionally very rough, as
the msk socket lock is not currently held where the condition is now enforced. This should require some refinement, shared as-is to avoid more latency on my side
--- net/mptcp/options.c | 25 +++++++++++++++++++++++-- net/mptcp/protocol.c | 10 ++-------- 2 files changed, 25 insertions(+), 10 deletions(-)
net/mptcp/options.c +23 -2
@@ -1158,8 +1158,23 @@ static bool add_addr_hmac_valid(struct mptcp_sock *msk,return hmac == mp_opt->ahmac; } -/* Return false in case of error (or subflow has been reset), -* else return true. +static bool mptcp_over_limit (struct sock *sk,const struct sock *ssk, +const struct sk_buff *skb) +{ +if (likely (sk_rmem_alloc_get (sk) <=READ_ONCE (sk->sk_rcvbuf))) +return false; + +/* Avoid silently dropping pure acks, fin or zero win probes. */ +if (TCP_SKB_CB (skb)->seq ==TCP_SKB_CB (skb)->end_seq || +TCP_SKB_CB (skb)->tcp_flags &TCPHDR_FIN || + !after (TCP_SKB_CB (skb)->end_seq,tcp_sk (ssk)->rcv_nxt)) +return false; + +return true; +} + +/* Return false when the caller must drop the packet, i.e. in case of error, +* subflow has been reset, or over memory limits. */ bool mptcp_incoming_options (struct sock *sk,struct sk_buff *skb) {
@@ -1185,6 +1200,9 @@ bool mptcp_incoming_options(struct sock *sk, struct sk_buff *skb)__mptcp_data_acked (subflow->conn);mptcp_data_unlock (subflow->conn); + +if (mptcp_over_limit (subflow->conn, sk, skb)) +return false;return true; }
@@ -1263,6 +1281,9 @@ bool mptcp_incoming_options(struct sock *sk, struct sk_buff *skb)return true; } +if (mptcp_over_limit (subflow->conn, sk, skb)) +return false; + mpext =skb_ext_add (skb,SKB_EXT_MPTCP );if (!mpext)return false;
net/mptcp/protocol.c +2 -8
@@ -739,7 +739,7 @@ static bool __mptcp_move_skbs_from_subflow(struct mptcp_sock *msk,mptcp_init_skb (ssk, skb, offset, len); -if (own_msk &&sk_rmem_alloc_get (sk) < sk->sk_rcvbuf) { +if (own_msk) {mptcp_subflow_lend_fwdmem (subflow, skb); ret |=__mptcp_move_skb (sk, skb); }else {
@@ -2197,10 +2197,6 @@ static bool __mptcp_move_skbs(struct sock *sk, struct list_head *skbs, u32 *delt *delta =0 ;while (1 ) { -/* If the msk recvbuf is full stop, don't drop */ -if (sk_rmem_alloc_get (sk) > sk->sk_rcvbuf) -break ; -prefetch (skb->next);list_del (&skb->list); *delta += skb->truesize;
@@ -2228,9 +2224,7 @@ static bool mptcp_can_spool_backlog(struct sock *sk, struct list_head *skbs)DEBUG_NET_WARN_ON_ONCE (msk->backlog_unaccounted && sk->sk_socket &&mem_cgroup_from_sk (sk)); -/* Don't spool the backlog if the rcvbuf is full. */ -if (list_empty (&msk->backlog_list) || -sk_rmem_alloc_get (sk) > sk->sk_rcvbuf) +if (list_empty (&msk->backlog_list))return false;INIT_LIST_HEAD (skbs);
Signature · 1 line
2.54.0
Paolo Abeni
maintainerPATCH v3 3/10 mptcp: drop the cant_coalesce CB field
In reply to Paolo AbeniSuch field is used to ensure in-sequence processing in case of fastopen. Instead let's perform synchronization of the fastopen skb sequence when the IASN becomes available with the 3rd ack.
When the cant_coalesce field has been introduced, commit f03afb3aeb9d ("mptcp: drop __mptcp_fastopen_gen_msk_ackseq()") noted that updating the already queued skb for passive fastopen socket at 3rd ack time would be difficult and race prone. The main point is that such update don't need to be synchronously performed at 3rd ack time, but is sufficient to perform it before the next segment is introduced into the msk.
To such extent, add an explicit test in __mptcp_move_skb(). Performance wise this trades a conditional in the fast path - in __mptcp_try_coalesce()
- with a similar one in __mptcp_move_skb() and a couple more in slow paths.
After this change the user-space will always observe consistent sequence numbers in the receive queue, even in the TFO dummy mapping case.
--- net/mptcp/fastopen.c | 2 +- net/mptcp/protocol.c | 28 ++++++++++++++++++++++++++-- net/mptcp/protocol.h | 4 +++- net/mptcp/subflow.c | 7 +++++++ 4 files changed, 37 insertions(+), 4 deletions(-)
net/mptcp/fastopen.c +1 -1
@@ -48,11 +48,11 @@ void mptcp_fastopen_subflow_synack_set_params(struct mptcp_subflow_context *subfMPTCP_SKB_CB (skb)->end_seq =0 ;MPTCP_SKB_CB (skb)->offset =0 ;MPTCP_SKB_CB (skb)->has_rxtstamp = has_rxtstamp; -MPTCP_SKB_CB (skb)->cant_coalesce =1 ;mptcp_data_lock (sk);DEBUG_NET_WARN_ON_ONCE (sock_owned_by_user_nocheck (sk)); +mptcp_sk (sk)->rcvd_dummy_seq = true;mptcp_borrow_fwdmem (sk, skb);skb_set_owner_r (skb, sk);__skb_queue_tail (&sk->sk_receive_queue, skb);
net/mptcp/protocol.c +26 -2
@@ -160,7 +160,6 @@ static bool __mptcp_try_coalesce(struct sock *sk, struct sk_buff *to,int limit =READ_ONCE (sk->sk_rcvbuf);if (MPTCP_SKB_CB (from)->map_seq !=MPTCP_SKB_CB (to)->end_seq || -unlikely (MPTCP_SKB_CB (to)->cant_coalesce) ||MPTCP_SKB_CB (from)->offset || ((to->len + from->len) > (limit >>3 )) || !skb_try_coalesce (to, from, fragstolen, delta))
@@ -357,7 +356,6 @@ static void mptcp_init_skb(struct sock *ssk, struct sk_buff *skb, int offset,MPTCP_SKB_CB (skb)->end_seq =MPTCP_SKB_CB (skb)->map_seq + copy_len;MPTCP_SKB_CB (skb)->offset = offset;MPTCP_SKB_CB (skb)->has_rxtstamp = has_rxtstamp; -MPTCP_SKB_CB (skb)->cant_coalesce =0 ;__skb_unlink (skb, &ssk->sk_receive_queue);
@@ -365,6 +363,24 @@ static void mptcp_init_skb(struct sock *ssk, struct sk_buff *skb, int offset,skb_dst_drop (skb); } +void __mptcp_sync_rcv_sequence (struct sock *sk) +{ +struct mptcp_sock *msk =mptcp_sk (sk); +struct sk_buff *skb; + +if (likely (!msk->rcvd_dummy_seq)) +return ; + +/* User space can have already received the TFO skb. */ + msk->rcvd_dummy_seq = false; + skb =skb_peek_tail (&sk->sk_receive_queue); +if (!skb) +return ; + +MPTCP_SKB_CB (skb)->map_seq = msk->ack_seq - skb->len; +MPTCP_SKB_CB (skb)->end_seq = msk->ack_seq; +} +static bool __mptcp_move_skb (struct sock *sk,struct sk_buff *skb) {u64 copy_len =MPTCP_SKB_CB (skb)->end_seq -MPTCP_SKB_CB (skb)->map_seq;
@@ -373,6 +389,12 @@ static bool __mptcp_move_skb(struct sock *sk, struct sk_buff *skb)mptcp_borrow_fwdmem (sk, skb); +/* Be sure to sync the eventual fastopen dummy mapping before any other +* skb lands into the msk. +*/ +if (unlikely (msk->rcvd_dummy_seq)) +__mptcp_sync_rcv_sequence (sk); +if (MPTCP_SKB_CB (skb)->map_seq == msk->ack_seq) {/* in sequence */ msk->bytes_received += copy_len;
@@ -3682,6 +3704,8 @@ static void mptcp_release_cb(struct sock *sk)__mptcp_error_report (sk);if (__test_and_clear_bit (MPTCP_SYNC_SNDBUF , &msk->cb_flags))__mptcp_sync_sndbuf (sk); +if (__test_and_clear_bit (MPTCP_SYNC_SEQ , &msk->cb_flags)) +__mptcp_sync_rcv_sequence (sk); } }
net/mptcp/protocol.h +3 -1
@@ -124,13 +124,13 @@#define MPTCP_FLUSH_JOIN_LIST 5#define MPTCP_SYNC_STATE 6#define MPTCP_SYNC_SNDBUF 7 +#define MPTCP_SYNC_SEQ 8struct mptcp_skb_cb {u64 map_seq;u64 end_seq;u32 offset;u8 has_rxtstamp; -u8 cant_coalesce; };#define MPTCP_SKB_CB (__skb) ((struct mptcp_skb_cb *)&((__skb)->cb[0]))
@@ -310,6 +310,7 @@ struct mptcp_sock {u32 token;unsigned long flags;unsigned long cb_flags; +bool rcvd_dummy_seq;bool recovery;/* closing subflow write queue reinjected */ bool can_ack;bool fully_established;
@@ -1172,6 +1173,7 @@ void mptcp_event_pm_listener(const struct sock *ssk, enum mptcp_event_type event);bool mptcp_userspace_pm_active (const struct mptcp_sock *msk); +void __mptcp_sync_rcv_sequence (struct sock *sk);void mptcp_fastopen_subflow_synack_set_params (struct mptcp_subflow_context *subflow,struct request_sock *req); int mptcp_pm_genl_fill_addr(struct sk_buff *msg,
net/mptcp/subflow.c +7 -0
@@ -478,6 +478,8 @@ static void subflow_set_remote_key(struct mptcp_sock *msk,struct mptcp_subflow_context *subflow, const struct mptcp_options_received *mp_opt) { + struct sock *sk = (struct sock *)msk; +/* active MPC subflow will reach here multiple times: * at subflow_finish_connect() time and at 4th ack time */
@@ -496,6 +498,11 @@ static void subflow_set_remote_key(struct mptcp_sock *msk,WRITE_ONCE (msk->ack_seq, subflow->iasn);WRITE_ONCE (msk->can_ack, true);atomic64_set (&msk->rcv_wnd_sent, subflow->iasn); + +if (!sock_owned_by_user (sk)) +__mptcp_sync_rcv_sequence (sk); +else +__set_bit (MPTCP_SYNC_SEQ , &msk->cb_flags); } static void mptcp_propagate_state(struct sock *sk, struct sock *ssk,
Signature · 1 line
2.54.0
Paolo Abeni
maintainerPATCH v3 4/10 mptcp: remove CB offset field
In reply to Paolo AbeniInstead, use a new msk-level field to track the bytes already consumed inside each skb, carrying the amount of bytes already copied to user-space, alike what TCP is already doing.
copied_seq is always accessed under the msk socket lock, delegating the synchronization with IASN to the msk release cb, when the socket is owned by the user-space at remote key reception time. Such synchronization preserves any partial progress (copy) made on the TFO packet.
Prior to this patch, the TFO skb dummy mapping was always ignored, now it affects the copied_seq initial update: be sure to extends the sign correctly of such mapping initialization time.
Overall this simplify a bit the __mptcp_recvmsg_mskq() and mptcp_inq_hint() code and will also make possible the next patch.
---
v2 -> v3:
- do not use msk->first in release_cb to deal with MPTCP_SYNC_SEQ:
subflow->iasn access is (data) racy and msk->first can be null, instead recompute iasn from msk bytes_received and TFO skb len
- when updating copied_seq after remote key reception, add iasn to it instead of overwriting, to avoid deleting any partial progress.
v1 -> v2:
- deal correctly with peek, as usally "inspired" from the correspondent tcp code
- update mptcp_inq_hint(), too
Notes:
- this explicitly relays on "mptcp: do not drop partial packets" to avoid dropping partially consumed packets
- sashiko may confuse the 'offset' in mptcp_init_skb for an MPTCP-level one, but it refers to the TCP sequence space. Conclusion out of the that assumptions are wrong.
- the data race in mptcp_inq_hint() is real, but pre-existing and can impact only sockopt() output - the other call-sites are race free, as ack_seq updates are serialized by the RX path.
Fixing the race for good without sashiko tripping on other similar minor races would require another largish series. Postponed.
- sashiko may see a race vs copied_seq access in mptcp_recv_skb(), but that is not possible: copied_seq is maintained in sync with the TFO skb via __mptcp_sync_rcv_sequence
--- net/mptcp/fastopen.c | 15 +++++--- net/mptcp/protocol.c | 89 ++++++++++++++++++-------------------------- net/mptcp/protocol.h | 8 +++- net/mptcp/subflow.c | 7 +++- 4 files changed, 59 insertions(+), 60 deletions(-)
net/mptcp/fastopen.c +10 -5
@@ -9,6 +9,7 @@void mptcp_fastopen_subflow_synack_set_params (struct mptcp_subflow_context *subflow,struct request_sock *req) { +struct mptcp_sock *msk;struct sock *sk, *ssk;struct sk_buff *skb;struct tcp_sock *tp;
@@ -43,20 +44,24 @@ void mptcp_fastopen_subflow_synack_set_params(struct mptcp_subflow_context *subf subflow->ssn_offset += skb->len; has_rxtstamp =TCP_SKB_CB (skb)->has_rxtstamp; -/* Only the sequence delta is relevant */ -MPTCP_SKB_CB (skb)->map_seq = -skb->len; +/* The TFO segment data sits before the IASN; before receiving +* the remove key, IASN is assumed being 0. +*/ +MPTCP_SKB_CB (skb)->map_seq = -(u64 )skb->len;MPTCP_SKB_CB (skb)->end_seq =0 ; -MPTCP_SKB_CB (skb)->offset =0 ;MPTCP_SKB_CB (skb)->has_rxtstamp = has_rxtstamp;mptcp_data_lock (sk);DEBUG_NET_WARN_ON_ONCE (sock_owned_by_user_nocheck (sk)); -mptcp_sk (sk)->rcvd_dummy_seq = true; + msk =mptcp_sk (sk); + msk->rcvd_dummy_seq = true; + msk->copied_seq =MPTCP_SKB_CB (skb)->map_seq; + msk->tfo_skb_len = skb->len;mptcp_borrow_fwdmem (sk, skb);skb_set_owner_r (skb, sk);__skb_queue_tail (&sk->sk_receive_queue, skb); -mptcp_sk (sk)->bytes_received += skb->len; + msk->bytes_received += skb->len; sk->sk_data_ready (sk);
net/mptcp/protocol.c +37 -52
209 lines not on this page · raw message
net/mptcp/protocol.h +7 -1
@@ -129,7 +129,6 @@struct mptcp_skb_cb {u64 map_seq;u64 end_seq; -u32 offset;u8 has_rxtstamp; };
@@ -289,6 +288,7 @@ struct mptcp_sock {u64 bytes_sent;u64 snd_nxt;u64 bytes_received; +u64 copied_seq;u64 ack_seq;atomic64_t rcv_wnd_sent;u64 rcv_data_fin_seq;
@@ -308,6 +308,7 @@ struct mptcp_sock {u32 last_ack_recv;unsigned long timer_ival;u32 token; +u32 tfo_skb_len;unsigned long flags;unsigned long cb_flags;bool rcvd_dummy_seq;
@@ -859,6 +860,11 @@ struct sock *mptcp_subflow_get_retrans(struct mptcp_sock *msk);int mptcp_sched_get_send (struct mptcp_sock *msk);int mptcp_sched_get_retrans (struct mptcp_sock *msk); +static inline u64 mptcp_iasn (const struct mptcp_sock *msk) +{ +return msk->ack_seq - msk->bytes_received + msk->tfo_skb_len; +} +static inline u64 mptcp_data_avail (const struct mptcp_sock *msk) {return READ_ONCE (msk->bytes_received) -READ_ONCE (msk->bytes_consumed);
net/mptcp/subflow.c +5 -2
@@ -499,10 +499,13 @@ static void subflow_set_remote_key(struct mptcp_sock *msk,WRITE_ONCE (msk->can_ack, true);atomic64_set (&msk->rcv_wnd_sent, subflow->iasn); -if (!sock_owned_by_user (sk)) +if (!sock_owned_by_user (sk)) { +/* User space could have already read partially the TFO skb */ + msk->copied_seq += subflow->iasn;__mptcp_sync_rcv_sequence (sk); -else + }else {__set_bit (MPTCP_SYNC_SEQ , &msk->cb_flags); + } } static void mptcp_propagate_state(struct sock *sk, struct sock *ssk,
Signature · 1 line
2.54.0
Paolo Abeni
maintainerPATCH v3 5/10 mptcp: sync mptcp skb cb layout with tcp one
In reply to Paolo AbeniThe MPTCP protocol uses a significantly different CB layout WRT TCP, as it includes different information and use 64 bits for the sequence numbers.
As the msk-level rcvbuf buffer size is limited by the core socket code the INT_MAX; after validating the incoming skb vs the current receive window, we can safely use 32 bits for MPTCP-level sequence number. This allow updating the MPTCP CB layout so that fields with a corresponding TCP-level data use the same area inside the CB itself.
Add build time check to ensure the latter invariant.
---
v1 -> v2:
- use u64 for admission checks
rfc -> v1:
- keep ack_seq up2date
--- net/mptcp/fastopen.c | 6 ++-- net/mptcp/protocol.c | 78 ++++++++++++++++++++++++++++---------------- net/mptcp/protocol.h | 7 ++-- 3 files changed, 58 insertions(+), 33 deletions(-)
net/mptcp/fastopen.c +4 -2
@@ -47,8 +47,10 @@ void mptcp_fastopen_subflow_synack_set_params(struct mptcp_subflow_context *subf/* The TFO segment data sits before the IASN; before receiving * the remove key, IASN is assumed being 0. */ -MPTCP_SKB_CB (skb)->map_seq = -(u64 )skb->len; +MPTCP_SKB_CB (skb)->map_seq64 = -(u64 )skb->len; +MPTCP_SKB_CB (skb)->map_seq =MPTCP_SKB_CB (skb)->map_seq64;MPTCP_SKB_CB (skb)->end_seq =0 ; +MPTCP_SKB_CB (skb)->flags =0 ;MPTCP_SKB_CB (skb)->has_rxtstamp = has_rxtstamp;mptcp_data_lock (sk);
@@ -56,7 +58,7 @@ void mptcp_fastopen_subflow_synack_set_params(struct mptcp_subflow_context *subf msk =mptcp_sk (sk); msk->rcvd_dummy_seq = true; - msk->copied_seq =MPTCP_SKB_CB (skb)->map_seq; + msk->copied_seq =MPTCP_SKB_CB (skb)->map_seq64; msk->tfo_skb_len = skb->len;mptcp_borrow_fwdmem (sk, skb);skb_set_owner_r (skb, sk);
net/mptcp/protocol.c +49 -29
200 lines not on this page · raw message
net/mptcp/protocol.h +5 -2
@@ -127,9 +127,12 @@#define MPTCP_SYNC_SEQ 8struct mptcp_skb_cb { -u64 map_seq; -u64 end_seq; +u32 map_seq; +u32 end_seq; +u32 unused; +u16 flags;u8 has_rxtstamp; +u64 map_seq64; };#define MPTCP_SKB_CB (__skb) ((struct mptcp_skb_cb *)&((__skb)->cb[0]))
Signature · 1 line
2.54.0
Paolo Abeni
maintainerPATCH v3 7/10 mptcp: implemented OoO queue pruning
In reply to Paolo AbeniLeverage the hybrid helpers to implement the receive queue and OoO queue collapsing at ingress time when reaching memory bounds.
If the msk is owned by the user-space at incoming skb time, perform the pruning in the release_cb. The prune check is additionally performed when the skb reaches the msk-level queues.
---
v2 -> v3:
- deal with unsynced TFO skb at prune time - only possible when pruning in mptcp_over_limit()
v1 -> v2:
- collapse rcv queue, too
- deal with MPC map, too
- drop left-over sentence in the commit message
RFC -> v1:
- use data_seq only when available
- avoid ack_seq lockless access
- drop limit on fallback
- collapse rcvqueue, too
- drop only when pruning is not possible and over rcvbuf * 2
--- net/mptcp/mib.c | 3 ++ net/mptcp/mib.h | 3 ++ net/mptcp/options.c | 42 ++++++++++++++++++++++++--- net/mptcp/protocol.c | 69 ++++++++++++++++++++++++++++++++++++++++++++ net/mptcp/protocol.h | 2 ++ 5 files changed, 115 insertions(+), 4 deletions(-)
net/mptcp/mib.c +3 -0
@@ -85,6 +85,9 @@ static const struct snmp_mib mptcp_snmp_list[] = {SNMP_MIB_ITEM ("SimultConnectFallback" ,MPTCP_MIB_SIMULTCONNFALLBACK ),SNMP_MIB_ITEM ("FallbackFailed" ,MPTCP_MIB_FALLBACKFAILED ),SNMP_MIB_ITEM ("WinProbe" ,MPTCP_MIB_WINPROBE ), +SNMP_MIB_ITEM ("OfoPruned" ,MPTCP_MIB_OFO_PRUNED ), +SNMP_MIB_ITEM ("RcvPruned" ,MPTCP_MIB_RCVPRUNED ), + SNMP_MIB_ITEM("RcvCollapsed", MPTCP_MIB_RCVCOLLAPSED), };/* mptcp_mib_alloc - allocate percpu mib counters
net/mptcp/mib.h +3 -0
@@ -88,6 +88,9 @@ enum linux_mptcp_mib_field {MPTCP_MIB_SIMULTCONNFALLBACK ,/* Simultaneous connect */ MPTCP_MIB_FALLBACKFAILED ,/* Can't fallback due to msk status */ MPTCP_MIB_WINPROBE ,/* MPTCP-level zero window probe */ +MPTCP_MIB_OFO_PRUNED ,/* MPTCP-level OoO queue pruned */ +MPTCP_MIB_RCVPRUNED ,/* Dropped due to memory constrains */ +MPTCP_MIB_RCVCOLLAPSED ,/* Collapsed due to memory pressure */ __MPTCP_MIB_MAX };
net/mptcp/options.c +38 -4
67 lines not on this page · raw message
net/mptcp/protocol.c +69 -0
87 lines not on this page · raw message
net/mptcp/protocol.h +2 -0
@@ -125,6 +125,7 @@#define MPTCP_SYNC_STATE 6#define MPTCP_SYNC_SNDBUF 7#define MPTCP_SYNC_SEQ 8 +#define MPTCP_PRUNE 9struct mptcp_skb_cb {u32 map_seq;
@@ -832,6 +833,7 @@ bool __mptcp_close(struct sock *sk, long timeout);void mptcp_cancel_work (struct sock *sk);void __mptcp_unaccepted_force_close (struct sock *sk);void mptcp_set_state (struct sock *sk,int state); +bool __mptcp_check_prune (struct sock *sk,u32 seq);bool mptcp_addresses_equal (const struct mptcp_addr_info *a,const struct mptcp_addr_info *b,bool use_port);
Signature · 1 line
2.54.0
Paolo Abeni
maintainerPATCH v3 6/10 tcp: expose the tcp_collapse_ofo_queue() helper to mptcp usage, too
In reply to Paolo AbeniThe end goal is to avoid duplicating the quite untrivial strategy at MPTCP level.
After the previous patch, the mentioned helpers could process skbs standing in MPTCP-level queues without any CB-related adaptation.
The only additional adjustment needed is explicitly providing the OoO queue reference, to cope with different sk layout.
Additionally rename the helper to clearly document its hybrid nature and let it return the number of collapsed skbs, to allow proper accounting from the future MPTCP caller.
---
rfc -> v1:
- fix arg typo
Note:
- this will need a significant amount of testing at the TCP level and explicit approval from Eric, which I can't guess if we can hope.
--- include/net/tcp.h | 8 +++++++ net/ipv4/tcp_input.c | 55 ++++++++++++++++++++++++++++---------------- 2 files changed, 43 insertions(+), 20 deletions(-)
include/net/tcp.h +8 -0
@@ -1828,6 +1828,14 @@ extern void tcp_openreq_init_rwin(struct request_sock *req,void tcp_enter_memory_pressure (struct sock *sk);void tcp_leave_memory_pressure (struct sock *sk); +unsigned int xtcp_collapse (struct sock *sk,struct sk_buff_head *list, +struct rb_root *root,struct sk_buff *head, +struct sk_buff *tail,u32 start,u32 end, +u8 scaling_ratio); +unsigned int xtcp_collapse_ofo_queue (struct sock *sk, +struct rb_root *out_of_order_queue, +struct sk_buff **ooo_last_skb, +u8 scaling_ratio);static inline int keepalive_intvl_when (const struct tcp_sock *tp) {
net/ipv4/tcp_input.c +35 -20
130 lines not on this page · raw message
Signature · 1 line
2.54.0
Paolo Abeni
maintainerPATCH v3 8/10 mptcp: track prune recovery status
In reply to Paolo AbeniAfter dropping any data already acked at the TCP level, the MPTCP must avoid inducing TCP-level retransmission until the pruned data has been successfully acked at MPTCP level. Otherwise the subflows could keep retransmitting skbs carring OoO MPTCP data, preventing reinjections and stalling completely the data transfer.
Explicitly keep track of the highest pruned MPTCP-level seq number and stop dropping at TCP level until such sequence has been acked.
--- net/mptcp/options.c | 7 ++++++- net/mptcp/protocol.c | 14 +++++++++++++- net/mptcp/protocol.h | 3 +++ net/mptcp/subflow.c | 1 + 4 files changed, 23 insertions(+), 2 deletions(-)
net/mptcp/options.c +6 -1
@@ -1200,7 +1200,12 @@ static bool mptcp_over_limit(struct sock *sk, const struct sock *ssk,__set_bit (MPTCP_PRUNE , &msk->cb_flags); }mptcp_data_unlock (sk); -return ret; + +/* After pruning any packets ensure that MPTCP-driven drops do not +* cause TCP-level retransmission +*/ +return ret && + !before (READ_ONCE (msk->ack_seq),READ_ONCE (msk->pruned_seq)); }/* Return false when the caller must drop the packet, i.e. in case of error,
net/mptcp/protocol.c +13 -1
@@ -392,12 +392,14 @@ static void mptcp_prune_ofo_queue(struct sock *sk, u32 seq)struct mptcp_sock *msk =mptcp_sk (sk);struct rb_node *node, *prev;bool pruned = false; +u32 pruned_seq;if (RB_EMPTY_ROOT (&msk->out_of_order_queue))return ; node = &msk->ooo_last_skb->rbnode; + pruned_seq = msk->pruned_seq; do { struct sk_buff *skb = rb_to_skb(node);
@@ -408,16 +410,21 @@ static void mptcp_prune_ofo_queue(struct sock *sk, u32 seq) pruned = true; prev =rb_prev (node);rb_erase (node, &msk->out_of_order_queue); +if (after (MPTCP_SKB_CB (skb)->end_seq, pruned_seq)) + pruned_seq =MPTCP_SKB_CB (skb)->end_seq;mptcp_drop (sk, skb); msk->ooo_last_skb =rb_to_skb (prev); +if (atomic_read (&sk->sk_rmem_alloc) < sk->sk_rcvbuf)break ; node = prev; }while (node); -if (pruned) +if (pruned) { +WRITE_ONCE (msk->pruned_seq, pruned_seq);MPTCP_INC_STATS (sock_net (sk),MPTCP_MIB_OFO_PRUNED ); + } }bool __mptcp_check_prune (struct sock *sk,u32 seq)
@@ -462,6 +469,8 @@ static bool __mptcp_move_skb(struct sock *sk, struct sk_buff *skb)__mptcp_sync_rcv_sequence (sk);if (__mptcp_check_prune (sk,MPTCP_SKB_CB (skb)->map_seq)) { +if (after (MPTCP_SKB_CB (skb)->end_seq, msk->pruned_seq)) +WRITE_ONCE (msk->pruned_seq,MPTCP_SKB_CB (skb)->end_seq);MPTCP_INC_STATS (sock_net (sk),MPTCP_MIB_RCVPRUNED );mptcp_drop (sk, skb);return false;
@@ -896,6 +905,8 @@ static bool __mptcp_ofo_queue(struct mptcp_sock *msk)WRITE_ONCE (msk->ack_seq, msk->ack_seq + seq_delta); moved = true; } +if (after (msk->ack_seq, msk->pruned_seq)) +WRITE_ONCE (msk->pruned_seq, (u32 )msk->ack_seq);return moved; }
@@ -3539,6 +3550,7 @@ static int mptcp_disconnect(struct sock *sk, int flags)/* for fallback's sake */ WRITE_ONCE (msk->ack_seq,0 ); msk->copied_seq =0 ; +WRITE_ONCE (msk->pruned_seq,0 );WRITE_ONCE (sk->sk_shutdown,0 );sk_error_report (sk);
net/mptcp/protocol.h +3 -0
@@ -307,6 +307,9 @@ struct mptcp_sock {u64 bytes_acked;u64 snd_una;u64 wnd_end; +u32 pruned_seq;/* If above ack_seq, highest +* seq pruned. +*/ u32 last_data_sent;u32 last_data_recv;u32 last_ack_recv;
net/mptcp/subflow.c +1 -0
@@ -496,6 +496,7 @@ static void subflow_set_remote_key(struct mptcp_sock *msk,WRITE_ONCE (msk->remote_key, subflow->remote_key);WRITE_ONCE (msk->ack_seq, subflow->iasn); +WRITE_ONCE (msk->pruned_seq, subflow->iasn);WRITE_ONCE (msk->can_ack, true);atomic64_set (&msk->rcv_wnd_sent, subflow->iasn);
Signature · 1 line
2.54.0
Paolo Abeni
maintainerPATCH v3 9/10 mptcp: move the retrans loop to a separate helper
In reply to Paolo AbeniThis is a cleanup in order to make the next patch simpler. No functional change intended.
--- net/mptcp/protocol.c | 74 +++++++++++++++++++++++++------------------- 1 file changed, 43 insertions(+), 31 deletions(-)
net/mptcp/protocol.c +43 -31
@@ -2838,41 +2838,14 @@ static void mptcp_check_fastclose(struct mptcp_sock *msk)sk_error_report (sk); } -static void __mptcp_retrans (struct sock *sk) +/* Retransmit the specified data fragment on all the selected subflows. */ +static int __mptcp_push_retrans (struct sock *sk,struct mptcp_data_frag *dfrag) {struct mptcp_sendmsg_info info = { .data_lock_held = true, };struct mptcp_sock *msk =mptcp_sk (sk);struct mptcp_subflow_context *subflow; -struct mptcp_data_frag *dfrag;struct sock *ssk; -int ret, err; -u16 len =0 ; - -mptcp_clean_una_wakeup (sk); - -/* first check ssk: need to kick "stale" logic */ - err =mptcp_sched_get_retrans (msk); - dfrag =mptcp_rtx_head (sk); -if (!dfrag) { -if (mptcp_data_fin_enabled (msk)) { -struct inet_connection_sock *icsk =inet_csk (sk); - -WRITE_ONCE (icsk->icsk_retransmits, - icsk->icsk_retransmits +1 ); -mptcp_set_datafin_timeout (sk); -mptcp_send_ack (msk); - - goto reset_timer; - } - -if (!mptcp_send_head (sk)) - goto clear_scheduled; - - goto reset_timer; - } - -if (err) - goto reset_timer; +int ret, len =0 ;mptcp_for_each_subflow (msk, subflow) {if (READ_ONCE (subflow->scheduled)) {
@@ -2900,7 +2873,7 @@ static void __mptcp_retrans(struct sock *sk) !msk->allow_subflows) {spin_unlock_bh (&msk->fallback_lock);release_sock (ssk); - goto clear_scheduled; +return -1 ; }while (info.sent < info.limit) {
@@ -2923,6 +2896,45 @@ static void __mptcp_retrans(struct sock *sk)release_sock (ssk); } } +return len; +} + +static void __mptcp_retrans (struct sock *sk) +{ +struct mptcp_sock *msk =mptcp_sk (sk); +struct mptcp_subflow_context *subflow; +struct mptcp_data_frag *dfrag; +int err, len; + +mptcp_clean_una_wakeup (sk); + +/* first check ssk: need to kick "stale" logic */ + err =mptcp_sched_get_retrans (msk); + dfrag =mptcp_rtx_head (sk); +if (!dfrag) { +if (mptcp_data_fin_enabled (msk)) { +struct inet_connection_sock *icsk =inet_csk (sk); + +WRITE_ONCE (icsk->icsk_retransmits, + icsk->icsk_retransmits +1 ); +mptcp_set_datafin_timeout (sk); +mptcp_send_ack (msk); + + goto reset_timer; + } + +if (!mptcp_send_head (sk)) + goto clear_scheduled; + + goto reset_timer; + } + +if (err) + goto reset_timer; + + len =__mptcp_push_retrans (sk, dfrag); +if (len <0 ) + goto clear_scheduled; msk->bytes_retrans += len; dfrag->already_sent =max (dfrag->already_sent, len);
Signature · 1 line
2.54.0
Paolo Abeni
maintainerPATCH v3 10/10 mptcp: let the retrans scheduler do its job.
In reply to Paolo AbeniCurrently the MPTCP core enforces that when MPTCP-level retrans timer fires, at most a single dfrag is retransmitted. If some corner-cases it may be necessary retransmit multiple dfrags, and the MPTCP socket will need to wait multiple retrans timeout to accomplish that.
Remove the mentioned constraint, allowing to transmit multiple dfrags per retrans period, as long as the scheduler keeps selecting subflows for retransmissions and pending data is available in the rtx queue. The default scheduler will transmit a dfrag per available subflow.
---
v2 -> v3:
- fix infinite loop issue (should address tls tests failures)
v1 -> v2:
- fix retrans sequence update (sashiko)
--- net/mptcp/protocol.c | 82 ++++++++++++++++++++++++++------------------ 1 file changed, 48 insertions(+), 34 deletions(-)
net/mptcp/protocol.c +48 -34
@@ -1217,13 +1217,6 @@ static void __mptcp_clean_una_wakeup(struct sock *sk)mptcp_write_space (sk); } -static void mptcp_clean_una_wakeup (struct sock *sk) -{ -mptcp_data_lock (sk); -__mptcp_clean_una_wakeup (sk); -mptcp_data_unlock (sk); -} -static void mptcp_enter_memory_pressure (struct sock *sk) {struct mptcp_subflow_context *subflow;
@@ -2838,7 +2831,10 @@ static void mptcp_check_fastclose(struct mptcp_sock *msk)sk_error_report (sk); } -/* Retransmit the specified data fragment on all the selected subflows. */ +/* +* Retransmit the specified data fragment on all the selected subflows, +* starting from the specified sequence +*/ static int __mptcp_push_retrans (struct sock *sk,struct mptcp_data_frag *dfrag) {struct mptcp_sendmsg_info info = { .data_lock_held = true, };
@@ -2904,42 +2900,60 @@ static void __mptcp_retrans(struct sock *sk)struct mptcp_sock *msk =mptcp_sk (sk);struct mptcp_subflow_context *subflow;struct mptcp_data_frag *dfrag; +u64 retrans_seq;int err, len; -mptcp_clean_una_wakeup (sk); - -/* first check ssk: need to kick "stale" logic */ - err =mptcp_sched_get_retrans (msk); - dfrag =mptcp_rtx_head (sk); -if (!dfrag) { -if (mptcp_data_fin_enabled (msk)) { -struct inet_connection_sock *icsk =inet_csk (sk); +mptcp_data_lock (sk); +__mptcp_clean_una_wakeup (sk); + retrans_seq = msk->snd_una; +mptcp_data_unlock (sk); -WRITE_ONCE (icsk->icsk_retransmits, - icsk->icsk_retransmits +1 ); -mptcp_set_datafin_timeout (sk); -mptcp_send_ack (msk); +for (;;) { +/* first check ssk: need to kick "stale" logic */ + err =mptcp_sched_get_retrans (msk); + dfrag =mptcp_rtx_head (sk); +if (!dfrag) { +if (mptcp_data_fin_enabled (msk)) { +struct inet_connection_sock *icsk; + + icsk =inet_csk (sk); +WRITE_ONCE (icsk->icsk_retransmits, + icsk->icsk_retransmits +1 ); +mptcp_set_datafin_timeout (sk); +mptcp_send_ack (msk); +break ; + } - goto reset_timer; +if (!mptcp_send_head (sk)) + goto clear_scheduled; +break ; } -if (!mptcp_send_head (sk)) - goto clear_scheduled; - - goto reset_timer; - } +if (err) +break ; -if (err) - goto reset_timer; +/* Skip the data already retransmitted in this run */ +while (dfrag && !before64 (retrans_seq, dfrag->data_seq + + dfrag->data_len)) + dfrag =list_is_last (&dfrag->list, &msk->rtx_queue) ? +NULL :list_next_entry (dfrag, list); +if (!dfrag || !dfrag->already_sent) +break ; - len =__mptcp_push_retrans (sk, dfrag); -if (len <0 ) - goto clear_scheduled; + len =__mptcp_push_retrans (sk, dfrag); +if (len <0 ) + goto clear_scheduled; - msk->bytes_retrans += len; - dfrag->already_sent =max (dfrag->already_sent, len); + retrans_seq += len; + msk->bytes_retrans += len; + dfrag->already_sent =max (dfrag->already_sent, len); -reset_timer: +/* Attempt the next fragment only if the current one is +* completely retransmitted. +*/ +if (len < dfrag->data_len) +break ; + }mptcp_check_and_set_pending (sk);if (!mptcp_rtx_timer_pending (sk))
Signature · 1 line
2.54.0
Paolo Abeni
maintainerRe: [PATCH v3 mptcp-next 06/10] tcp: expose the tcp_collapse_ofo_queue() helper to mptcp usage, too
In reply to Paolo AbeniHi Eric,
I'd like to re-use some existing TCP helpers (tcp_collapse_ofo_queue() and tcp_collapse()) in the MPTCP code, to avoid duplicating the non trivial logic there.
The idea is that the 'struct sock *' in such helpers could be either a tcp_sock* or an mptcp_sock *, and the processed skbs could be sitting in TCP level queues or MPTCP ones.
That poses some hard constraints on the sk fields that the helpers can access. To make them clear I'm renaming such helpers using an 'xtcp_' prefix.
The full patch touching the TCP code is below. For completeness, the whole series is available here:
https://lore.kernel.org/mptcp/[email protected]/T/#me6eda13ec2c70fbb400b6223aa2d07a4fd3bf865
but I guess most of it is not interesting for you/from TCP perspective.
The MPTCP bits are not yet finalized, but I'd like to know it you would consider this approach doable or a complete no-go.
Thanks,
Paolo
---
Quoted 188 lines · Paolo Abeni
The end goal is to avoid duplicating the quite untrivial strategy at MPTCP level.
After the previous patch, the mentioned helpers could process skbs standing in MPTCP-level queues without any CB-related adaptation.
The only additional adjustment needed is explicitly providing the OoO queue reference, to cope with different sk layout.
Additionally rename the helper to clearly document its hybrid nature and let it return the number of collapsed skbs, to allow proper accounting from the future MPTCP caller.
Signed-off-by Paolo Abeni <[email protected]>---
rfc -> v1:
- fix arg typoNote:
- this will need a significant amount of testing at the TCP level and explicit approval from Eric, which I can't guess if we can hope.--- include/net/tcp.h | 8 +++++++ net/ipv4/tcp_input.c | 55 ++++++++++++++++++++++++++++---------------- 2 files changed, 43 insertions(+), 20 deletions(-)include/net/tcp.h +8
-0@@ -1828,6 +1828,14 @@ extern void tcp_openreq_init_rwin(struct request_sock *req,void tcp_enter_memory_pressure (struct sock *sk);void tcp_leave_memory_pressure (struct sock *sk); +unsigned int xtcp_collapse (struct sock *sk,struct sk_buff_head *list, +struct rb_root *root,struct sk_buff *head, +struct sk_buff *tail,u32 start,u32 end, +u8 scaling_ratio); +unsigned int xtcp_collapse_ofo_queue (struct sock *sk, +struct rb_root *out_of_order_queue, +struct sk_buff **ooo_last_skb, +u8 scaling_ratio);static inline int keepalive_intvl_when (const struct tcp_sock *tp) {net/ipv4/tcp_input.c +35
-20@@ -5725,16 +5725,22 @@ static struct sk_buff *tcp_collapse_one(struct sock *sk, struct sk_buff *skb,/* Collapse contiguous sequence of skbs head..tail with * sequence numbers start..end. * +* sk can be either a TCP or an MPTCP socket. +* * If tail is NULL, this means until the end of the queue. * * Segments with FIN/SYN are not collapsed (only because this * simplifies code) +* +* Returns the number of collapsed skbs. */ -static void -tcp_collapse (struct sock *sk,struct sk_buff_head *list,struct rb_root *root, -struct sk_buff *head,struct sk_buff *tail,u32 start,u32 end) +unsigned int +xtcp_collapse (struct sock *sk,struct sk_buff_head *list,struct rb_root *root, +struct sk_buff *head,struct sk_buff *tail,u32 start,u32 end, +u8 scaling_ratio) {struct sk_buff *skb = head, *n; +unsigned int collapsed =0 ;struct sk_buff_head tmp;bool end_of_skbs;@@ -5750,6 +5756,7 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,/* No new bits? It is possible on ofo queue. */ if (!before (start,TCP_SKB_CB (skb)->end_seq)) { + collapsed++; skb =tcp_collapse_one (sk, skb, list, root);if (!skb)break ;@@ -5762,7 +5769,7 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,* overlaps to the next one and mptcp allow collapsing. */ if (!(TCP_SKB_CB (skb)->tcp_flags & (TCPHDR_SYN |TCPHDR_FIN )) && - (tcp_win_from_space (sk, skb->truesize) > skb->len || + (__tcp_win_from_space (scaling_ratio, skb->truesize) > skb->len ||before (TCP_SKB_CB (skb)->seq, start))) { end_of_skbs = false;break ;@@ -5782,7 +5789,7 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,if (end_of_skbs || (TCP_SKB_CB (skb)->tcp_flags & (TCPHDR_SYN |TCPHDR_FIN )) || !skb_frags_readable (skb)) -return ; +return collapsed;__skb_queue_head_init (&tmp);@@ -5819,6 +5826,7 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, start += size; }if (!before (start,TCP_SKB_CB (skb)->end_seq)) { + collapsed++; skb =tcp_collapse_one (sk, skb, list, root); if (!skb || skb == tail ||@@ -5832,23 +5840,26 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, end: skb_queue_walk_safe(&tmp, skb, n)tcp_rbtree_insert (root, skb); +return collapsed; }/* Collapse ofo queue. Algorithm: select contiguous sequence of skbs -* and tcp_collapse() them until all the queue is collapsed. +* and xtcp_collapse() them until all the queue is collapsed. */ -static void tcp_collapse_ofo_queue (struct sock *sk) +unsigned int xtcp_collapse_ofo_queue (struct sock *sk, +struct rb_root *ooo_queue, +struct sk_buff **ooo_last_skb, +u8 scaling_ratio) { -struct tcp_sock *tp =tcp_sk (sk); -u32 range_truesize, sum_tiny =0 ; +u32 range_truesize, sum_tiny =0 , collapsed =0 ;struct sk_buff *skb, *head;u32 start, end; - skb =skb_rb_first (&tp->out_of_order_queue); + skb =skb_rb_first (ooo_queue); new_range:if (!skb) { - tp->ooo_last_skb =skb_rb_last (&tp->out_of_order_queue); -return ; + *ooo_last_skb =skb_rb_last (ooo_queue); +return collapsed; } start =TCP_SKB_CB (skb)->seq; end =TCP_SKB_CB (skb)->end_seq;@@ -5866,12 +5877,13 @@ static void tcp_collapse_ofo_queue(struct sock *sk)/* Do not attempt collapsing tiny skbs */ if (range_truesize != head->truesize || end - start >=SKB_WITH_OVERHEAD (PAGE_SIZE )) { -tcp_collapse (sk,NULL , &tp->out_of_order_queue, - head, skb, start, end); + collapsed +=xtcp_collapse (sk,NULL , ooo_queue, + head, skb, start, end, + scaling_ratio); }else { sum_tiny += range_truesize;if (sum_tiny > sk->sk_rcvbuf >>3 ) -return ; +return collapsed; } goto new_range; }@@ -5882,6 +5894,7 @@ static void tcp_collapse_ofo_queue(struct sock *sk)if (after (TCP_SKB_CB (skb)->end_seq, end)) end =TCP_SKB_CB (skb)->end_seq; } +return collapsed; }/* @@ -5969,12 +5982,14 @@ static int tcp_prune_queue(struct sock *sk, const struct sk_buff *in_skb)if (tcp_can_ingest (sk, in_skb))return 0 ; -tcp_collapse_ofo_queue (sk); +xtcp_collapse_ofo_queue (sk, &tp->out_of_order_queue, + &tp->ooo_last_skb, tp->scaling_ratio);if (!skb_queue_empty (&sk->sk_receive_queue)) -tcp_collapse (sk, &sk->sk_receive_queue,NULL , -skb_peek (&sk->sk_receive_queue), -NULL , - tp->copied_seq, tp->rcv_nxt); +xtcp_collapse (sk, &sk->sk_receive_queue,NULL , +skb_peek (&sk->sk_receive_queue), +NULL , + tp->copied_seq, tp->rcv_nxt, + tp->scaling_ratio);if (tcp_can_ingest (sk, in_skb))return 0 ;
Hi Paolo,
Thank you for your modifications, that's great!
Our CI did some validations and here is its report:
- KVM Validation: normal (except selftest_mptcp_join): Success! ✅
- KVM Validation: normal (only selftest_mptcp_join): Success! ✅
- KVM Validation: debug (except selftest_mptcp_join): Unstable: 1 failed test(s): packetdrill_sockopts ⚠️
- KVM Validation: debug (only selftest_mptcp_join): Success! ✅
- KVM Validation: btf-normal (only bpftest_all): Success! ✅
- KVM Validation: btf-debug (only bpftest_all): Success! ✅
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/25329194631
Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/4ee2213ed212
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=1089374
If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:
$ cd [kernel source code]
$ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
--pull always mptcp/mptcp-upstream-virtme-docker:latest \
auto-normalFor more details:
https://github.com/multipath-tcp/mptcp-upstream-virtme-docker
Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)
Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)
Re: [PATCH v3 mptcp-next 06/10] tcp: expose the tcp_collapse_ofo_queue() helper to mptcp usage, too
In reply to Paolo AbeniQuoted 18 lines · Paolo Abeni
Hi Eric,
I'd like to re-use some existing TCP helpers (
tcp_collapse_ofo_queue()andtcp_collapse()) in the MPTCP code, to avoid duplicating the non trivial logic there.The idea is that the 'struct sock *' in such helpers could be either a tcp_sock* or an
mptcp_sock*, and the processed skbs could be sitting in TCP level queues or MPTCP ones.That poses some hard constraints on the sk fields that the helpers can access. To make them clear I'm renaming such helpers using an '
xtcp_' prefix.The full patch touching the TCP code is below. For completeness, the whole series is available here:
Quoted 6 lines
https://lore.kernel.org/mptcp/[email protected]/T/#me6eda13ec2c70fbb400b6223aa2d07a4fd3bf865
but I guess most of it is not interesting for you/from TCP perspective.
The MPTCP bits are not yet finalized, but I'd like to know it you would consider this approach doable or a complete no-go.
No concerns from my side, if we except the fact that TCP collapses are a high risk, we might
remove them completely (or at least guard them under a sysctl)
Paolo Abeni
maintainerRe: [PATCH v3 mptcp-next 06/10] tcp: expose the tcp_collapse_ofo_queue() helper to mptcp usage, too
Quoted 18 lines · Eric Dumazet
Quoted 17 lines · Paolo Abeni
I'd like to re-use some existing TCP helpers (
tcp_collapse_ofo_queue()andtcp_collapse()) in the MPTCP code, to avoid duplicating the non trivial logic there.The idea is that the 'struct sock *' in such helpers could be either a tcp_sock* or an
mptcp_sock*, and the processed skbs could be sitting in TCP level queues or MPTCP ones.That poses some hard constraints on the sk fields that the helpers can access. To make them clear I'm renaming such helpers using an '
xtcp_' prefix.The full patch touching the TCP code is below. For completeness, the whole series is available here:
https://lore.kernel.org/mptcp/[email protected]/T/#me6eda13ec2c70fbb400b6223aa2d07a4fd3bf865
Quoted 8 lines
Quoted 4 lines
but I guess most of it is not interesting for you/from TCP perspective.
The MPTCP bits are not yet finalized, but I'd like to know it you would consider this approach doable or a complete no-go.
No concerns from my side, if we except the fact that TCP collapses are a high risk, we might
remove them completely (or at least guard them under a sysctl)
I'm sorry I can't follow the above suggestion, could you please rephrase?
The patch should not add bugs to TCP collapses for TCP sockets or MPTCP subflows (still tcp_sock). Collapses for MPTCP sockets could become buggy if the tcp_collapse_ofo_queue()/tcp_collapse() implementation will be modified in the future without care.
Thanks,
Paolo
Re: [PATCH v3 mptcp-next 06/10] tcp: expose the tcp_collapse_ofo_queue() helper to mptcp usage, too
Quoted 29 lines · Paolo Abeni
Quoted 18 lines · Eric Dumazet
Quoted 17 lines · Paolo Abeni
I'd like to re-use some existing TCP helpers (
tcp_collapse_ofo_queue()andtcp_collapse()) in the MPTCP code, to avoid duplicating the non trivial logic there.The idea is that the 'struct sock *' in such helpers could be either a tcp_sock* or an
mptcp_sock*, and the processed skbs could be sitting in TCP level queues or MPTCP ones.That poses some hard constraints on the sk fields that the helpers can access. To make them clear I'm renaming such helpers using an '
xtcp_' prefix.The full patch touching the TCP code is below. For completeness, the whole series is available here:
https://lore.kernel.org/mptcp/[email protected]/T/#me6eda13ec2c70fbb400b6223aa2d07a4fd3bf865
Quoted 8 lines
Quoted 4 lines
but I guess most of it is not interesting for you/from TCP perspective.
The MPTCP bits are not yet finalized, but I'd like to know it you would consider this approach doable or a complete no-go.
No concerns from my side, if we except the fact that TCP collapses are a high risk, we might
remove them completely (or at least guard them under a sysctl)
Quoted 6 lines
I'm sorry I can't follow the above suggestion, could you please rephrase?
The patch should not add bugs to TCP collapses for TCP sockets or MPTCP subflows (still
tcp_sock). Collapses for MPTCP sockets could become buggy if the tcp_collapse_ofo_queue()/tcp_collapse() implementation will be modified in the future without care.
I think TCP collapses should be avoided like the plague.
They were ok 20 years ago, when 'line rate' was 64Kbit :0)
Paolo Abeni
maintainerRe: [PATCH v3 mptcp-next 06/10] tcp: expose the tcp_collapse_ofo_queue() helper to mptcp usage, too
Quoted 32 lines · Eric Dumazet
Quoted 31 lines · Paolo Abeni
Quoted 18 lines · Eric Dumazet
Quoted 17 lines · Paolo Abeni
I'd like to re-use some existing TCP helpers (
tcp_collapse_ofo_queue()andtcp_collapse()) in the MPTCP code, to avoid duplicating the non trivial logic there.The idea is that the 'struct sock *' in such helpers could be either a tcp_sock* or an
mptcp_sock*, and the processed skbs could be sitting in TCP level queues or MPTCP ones.That poses some hard constraints on the sk fields that the helpers can access. To make them clear I'm renaming such helpers using an '
xtcp_' prefix.The full patch touching the TCP code is below. For completeness, the whole series is available here:
https://lore.kernel.org/mptcp/[email protected]/T/#me6eda13ec2c70fbb400b6223aa2d07a4fd3bf865
Quoted 8 lines
Quoted 4 lines
but I guess most of it is not interesting for you/from TCP perspective.
The MPTCP bits are not yet finalized, but I'd like to know it you would consider this approach doable or a complete no-go.
No concerns from my side, if we except the fact that TCP collapses are a high risk, we might
remove them completely (or at least guard them under a sysctl)I'm sorry I can't follow the above suggestion, could you please rephrase?
Quoted 8 lines
Quoted 4 lines
The patch should not add bugs to TCP collapses for TCP sockets or MPTCP subflows (still
tcp_sock). Collapses for MPTCP sockets could become buggy if the tcp_collapse_ofo_queue()/tcp_collapse() implementation will be modified in the future without care.I think TCP collapses should be avoided like the plague.
They were ok 20 years ago, when 'line rate' was 64Kbit :0)
Understood, thanks! It looks like the sysctl is a bit hortogonal to this patch.
FYI, this work sparked from some TLS-derived self-tests (sending data 1 byte segments with OoO) hitting the collapse condition with MPTCP. I think a similar test would call for collapses even with plain TCP - the main difference is that with MPTCP and multiple subflows OoO are physiological/frequent events.
Thanks,
Paolo
Matthieu Baerts
maintainerHi Geliang, Gang,
Quoted 12 lines · Paolo Abeni
This an attempt to fix the data transfer stall reported by Geliang and Gang more carefully enforcing memory constraints at the MPTCP level.
Patch 1/10 moves the bound check before entering the TCP socket. Patch 2, 3, 4 and 5 are cleanups/refactors finalized to safely re-using TCP helpers on MPTCP skbs.
Patch 6 makes TCP pruning related helpers available to MPTCP and patch 7 makes use of them. Patch 8 addresses an edge scenario that could still lead to transfer stall under memory pressure.
Finally patch 9 and 10 improve the MPTCP-level retransmission schema to make recovery from memory pressure significanly faster.
Quoted 6 lines
Note that the diffstat is biases by the quite large patch 4/9, which contains mechanical transformation of existing code; "real" changes are noticiable smaller.
Tested successfully vs the test cases proposed by Geliang and Gang and vs the selftests.
At the last meeting on Wednesday, Geliang mentioned he validated this series. Just to be sure, was it the v2 -- from last week -- or the v3 -- from this week, while you were in EU -- that you validated? Because Paolo couldn't reproduce the issue you mentioned on the v3 on his side.
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.
Quoted 22 lines · Matthieu Baerts
Hi Geliang, Gang,
Quoted 17 lines · Paolo Abeni
This an attempt to fix the data transfer stall reported by Geliang and
Gang more carefully enforcing memory constraints at the MPTCP level.Patch 1/10 moves the bound check before entering the TCP socket. Patch 2, 3, 4 and 5 are cleanups/refactors finalized to safely re-using TCP helpers on MPTCP skbs.
Patch 6 makes TCP pruning related helpers available to MPTCP and patch 7 makes use of them. Patch 8 addresses an edge scenario that could still lead to transfer stall under memory pressure.
Finally patch 9 and 10 improve the MPTCP-level retransmission schema to make recovery from memory pressure significanly faster.Note that the diffstat is biases by the quite large patch 4/9, which contains mechanical transformation of existing code; "real" changes are noticiable smaller.
Quoted 8 lines
Quoted 3 lines
Tested successfully vs the test cases proposed by Geliang and Gang and vs the selftests.
At the last meeting on Wednesday, Geliang mentioned he validated this series. Just to be sure, was it the v2 -- from last week -- or the v3 -- from this week, while you were in EU -- that you validated? Because Paolo couldn't reproduce the issue you mentioned on the v3 on his side.
Hi, Matt
The issue can also be reproduced on v3. I reproduced it using Docker's auto-debug mode with the mptcp_data.sh selftest:
‘’‘
Not running all tests but:
-------- 8< --------run_loop run_selftest_one mptcp_data.sh
-------- 8< --------
=== Attempt: 1 (Sat, 09 May 2026 06:55:44 +0000) ===
Selftest Test: ./mptcp_data.sh
TAP version 13
1..1
# add_addr_accepted 4 subflows 4 # id 1 flags signal 127.0.0.1 10001 # id 2 flags signal 127.0.0.1 10002 # id 3 flags signal 127.0.0.1 10003 # id 4 flags signal 127.0.0.1 10004 # TAP version 13 # 1..48 # # Starting 48 tests from 2 test cases. # # RUN global.mptcp_v6 ... # # OK global.mptcp_v6 # ok 1 global.mptcp_v6 # # RUN mptcp.shutdown_reuse ... # # OK mptcp.shutdown_reuse # ok 2 mptcp.shutdown_reuse ... # ok 48 mptcp.sendfile # # FAILED: 41 / 48 tests passed. # # Totals: pass:41 fail:7 xfail:0 xpass:0 skip:0 error:0 not ok 1 test: selftest_mptcp_data # FAIL # time=33 ’‘’
As Geliang mentioned in the weekly meeting, we will continue to debug and locate the problem based on Paolo's v3 patches.
We will post any updates to the mailing list once we make some progess.
Thanks
Gang
Quoted 5 lines
Cheers,
MattSignature · 1 line
Sponsored by the NGI0 Core fund.
Paolo Abeni
maintainerQuoted 74 lines · On 5/9/26 9:07 AM, [email protected] wrote:
Quoted 22 lines · Matthieu Baerts
Hi Geliang, Gang,
Quoted 17 lines · Paolo Abeni
This an attempt to fix the data transfer stall reported by Geliang and
Gang more carefully enforcing memory constraints at the MPTCP level.Patch 1/10 moves the bound check before entering the TCP socket. Patch 2, 3, 4 and 5 are cleanups/refactors finalized to safely re-using TCP helpers on MPTCP skbs.
Patch 6 makes TCP pruning related helpers available to MPTCP and patch 7 makes use of them. Patch 8 addresses an edge scenario that could still lead to transfer stall under memory pressure.
Finally patch 9 and 10 improve the MPTCP-level retransmission schema to make recovery from memory pressure significanly faster.Note that the diffstat is biases by the quite large patch 4/9, which contains mechanical transformation of existing code; "real" changes are noticiable smaller.
Quoted 8 lines
Quoted 3 lines
Tested successfully vs the test cases proposed by Geliang and Gang and vs the selftests.
At the last meeting on Wednesday, Geliang mentioned he validated this series. Just to be sure, was it the v2 -- from last week -- or the v3 -- from this week, while you were in EU -- that you validated? Because Paolo couldn't reproduce the issue you mentioned on the v3 on his side.
Hi, Matt
The issue can also be reproduced on v3. I reproduced it using Docker's auto-debug mode with the
mptcp_data.shselftest:‘’‘
Not running all tests but:-------- 8< --------
run_looprun_selftest_onemptcp_data.sh
-------- 8< --------=== Attempt: 1 (Sat, 09 May 2026 06:55:44 +0000) ===Selftest Test: ./mptcp_data.sh
TAP version 13
1..1# add_addr_accepted 4 subflows 4 # id 1 flags signal 127.0.0.1 10001 # id 2 flags signal 127.0.0.1 10002 # id 3 flags signal 127.0.0.1 10003 # id 4 flags signal 127.0.0.1 10004 # TAP version 13 # 1..48 # # Starting 48 tests from 2 test cases. # # RUN global.mptcp_v6 ... # # OK global.mptcp_v6 # ok 1 global.mptcp_v6 # # RUN mptcp.shutdown_reuse ... # # OK mptcp.shutdown_reuse # ok 2 mptcp.shutdown_reuse ... # ok 48 mptcp.sendfile # # FAILED: 41 / 48 tests passed. # # Totals: pass:41 fail:7 xfail:0 xpass:0 skip:0 error:0 not ok 1 test: selftest_mptcp_data # FAIL # time=33 ’‘’
Quoted 2 lines
As Geliang mentioned in the weekly meeting, we will continue to debug and locate the problem based on Paolo's v3 patches.
Thanks for testing. I went over a couple more revisions. I run more than 100 iterations successfully on a local build on top of v5 (no failures, I stop due to time constraints):
https://lore.kernel.org/mptcp/[email protected]/T/#t
please have a spin as such later version.
Thanks,
Paolo
Geliang Tang
reviewerHi Paolo,
Quoted 102 lines · Paolo Abeni
Quoted 92 lines · On 5/9/26 9:07 AM, [email protected] wrote:
May 8, 2026 at 6:49 PM, "Matthieu Baerts" <[email protected] mailto:
[email protected]?to=%22Matthieu%20Baerts%22%20%3Cmatttbe%40kernelQuoted 35 lines · .org%3E > wrote:
Hi Geliang, Gang,
Quoted 21 lines · Paolo Abeni
This an attempt to fix the data transfer stall reported by Geliang and
Gang more carefully enforcing memory constraints at the MPTCP level.
Patch 1/10 moves the bound check before entering the TCP socket.
Patch 2, 3, 4 and 5 are cleanups/refactors finalized to safely re-using
TCP helpers on MPTCP skbs.
Patch 6 makes TCP pruning related helpers available to MPTCP and patch 7
makes use of them. Patch 8 addresses an edge scenario that could still
lead to transfer stall under memory pressure. Finally patch 9 and 10 improve the MPTCP-level retransmission schema to
make recovery from memory pressure significanly faster.
Note that the diffstat is biases by the quite large patch 4/9,Quoted 9 lines
which
contains mechanical transformation of existing code; "real" changes are
noticiable smaller.
Tested successfully vs the test cases proposed by Geliang and Gang and
vs the selftests.Quoted 9 lines
At the last meeting on Wednesday, Geliang mentioned he validated this
series. Just to be sure, was it the v2 -- from last week -- or the v3 --
from this week, while you were in EU -- that you validated? Because
Paolo couldn't reproduce the issue you mentioned on the v3 on his side.Hi, Matt
The issue can also be reproduced on v3. I reproduced it using Docker's
auto-debug mode with themptcp_data.shselftest:‘’‘
Not running all tests but:-------- 8< --------
run_looprun_selftest_onemptcp_data.sh
-------- 8< --------=== Attempt: 1 (Sat, 09 May 2026 06:55:44 +0000) ===Selftest Test: ./mptcp_data.sh
TAP version 13
1..1# add_addr_accepted 4 subflows 4 # id 1 flags signal 127.0.0.1 10001 # id 2 flags signal 127.0.0.1 10002 # id 3 flags signal 127.0.0.1 10003 # id 4 flags signal 127.0.0.1 10004 # TAP version 13 # 1..48 # # Starting 48 tests from 2 test cases. # # RUN global.mptcp_v6 ... # # OK global.mptcp_v6 # ok 1 global.mptcp_v6 # # RUN mptcp.shutdown_reuse ... # # OK mptcp.shutdown_reuse # ok 2 mptcp.shutdown_reuse ... # ok 48 mptcp.sendfile # # FAILED: 41 / 48 tests passed. # # Totals: pass:41 fail:7 xfail:0 xpass:0 skip:0 error:0 not ok 1 test: selftest_mptcp_data # FAIL # time=33 ’‘’Quoted 2 lines
As Geliang mentioned in the weekly meeting, we will continue to debug and locate the problem based on Paolo's v3 patches.
Thanks for testing. I went over a couple more revisions. I run more than
100 iterations successfully on a local build on top of v5 (no failures,
I stop due to time constraints):
Quoted 3 lines
https://lore.kernel.org/mptcp/[email protected]/T/#t
please have a spin as such later version.
On v5, mptcp_data.sh works fine in normal mode with the virtme docker image, but in debug mode it's still unstable - failing after several loop iterations.
Thanks,
-Geliang
Quoted 5 lines
Thanks,
Paolo
Paolo Abeni
maintainerQuoted 103 lines · Geliang Tang
Quoted 102 lines · Paolo Abeni
Quoted 92 lines · On 5/9/26 9:07 AM, [email protected] wrote:
May 8, 2026 at 6:49 PM, "Matthieu Baerts" <[email protected] mailto:
[email protected]?to=%22Matthieu%20Baerts%22%20%3Cmatttbe%40kernelQuoted 35 lines · .org%3E > wrote:
Hi Geliang, Gang,
Quoted 21 lines · Paolo Abeni
This an attempt to fix the data transfer stall reported by Geliang and
Gang more carefully enforcing memory constraints at the MPTCP level.
Patch 1/10 moves the bound check before entering the TCP socket.
Patch 2, 3, 4 and 5 are cleanups/refactors finalized to safely re-using
TCP helpers on MPTCP skbs.
Patch 6 makes TCP pruning related helpers available to MPTCP and patch 7
makes use of them. Patch 8 addresses an edge scenario that could still
lead to transfer stall under memory pressure. Finally patch 9 and 10 improve the MPTCP-level retransmission schema to
make recovery from memory pressure significanly faster.
Note that the diffstat is biases by the quite large patch 4/9,Quoted 9 lines
which
contains mechanical transformation of existing code; "real" changes are
noticiable smaller.
Tested successfully vs the test cases proposed by Geliang and Gang and
vs the selftests.Quoted 9 lines
At the last meeting on Wednesday, Geliang mentioned he validated this
series. Just to be sure, was it the v2 -- from last week -- or the v3 --
from this week, while you were in EU -- that you validated? Because
Paolo couldn't reproduce the issue you mentioned on the v3 on his side.Hi, Matt
The issue can also be reproduced on v3. I reproduced it using Docker's
auto-debug mode with themptcp_data.shselftest:‘’‘
Not running all tests but:-------- 8< --------
run_looprun_selftest_onemptcp_data.sh
-------- 8< --------=== Attempt: 1 (Sat, 09 May 2026 06:55:44 +0000) ===Selftest Test: ./mptcp_data.sh
TAP version 13
1..1# add_addr_accepted 4 subflows 4 # id 1 flags signal 127.0.0.1 10001 # id 2 flags signal 127.0.0.1 10002 # id 3 flags signal 127.0.0.1 10003 # id 4 flags signal 127.0.0.1 10004 # TAP version 13 # 1..48 # # Starting 48 tests from 2 test cases. # # RUN global.mptcp_v6 ... # # OK global.mptcp_v6 # ok 1 global.mptcp_v6 # # RUN mptcp.shutdown_reuse ... # # OK mptcp.shutdown_reuse # ok 2 mptcp.shutdown_reuse ... # ok 48 mptcp.sendfile # # FAILED: 41 / 48 tests passed. # # Totals: pass:41 fail:7 xfail:0 xpass:0 skip:0 error:0 not ok 1 test: selftest_mptcp_data # FAIL # time=33 ’‘’Quoted 2 lines
As Geliang mentioned in the weekly meeting, we will continue to debug and locate the problem based on Paolo's v3 patches.
Thanks for testing. I went over a couple more revisions. I run more than
100 iterations successfully on a local build on top of v5 (no failures,
I stop due to time constraints):
Quoted 7 lines
Quoted 3 lines
https://lore.kernel.org/mptcp/[email protected]/T/#t
please have a spin as such later version.
On v5,
mptcp_data.shworks fine in normal mode with the virtme docker image, but in debug mode it's still unstable - failing after several loop iterations.
How many interactions? Can you please provide a sample of such failure? does the failure always happen in the same way/on the same test case?
Thanks!
Paolo
Paolo Abeni
maintainerQuoted 106 lines · Paolo Abeni
Quoted 105 lines · Geliang Tang
Quoted 104 lines · Paolo Abeni
Quoted 92 lines · On 5/9/26 9:07 AM, [email protected] wrote:
May 8, 2026 at 6:49 PM, "Matthieu Baerts" <[email protected] mailto:
[email protected]?to=%22Matthieu%20Baerts%22%20%3Cmatttbe%40kernelQuoted 35 lines · .org%3E > wrote:
Hi Geliang, Gang,
Quoted 21 lines · Paolo Abeni
This an attempt to fix the data transfer stall reported by Geliang and
Gang more carefully enforcing memory constraints at the MPTCP level.
Patch 1/10 moves the bound check before entering the TCP socket.
Patch 2, 3, 4 and 5 are cleanups/refactors finalized to safely re-using
TCP helpers on MPTCP skbs.
Patch 6 makes TCP pruning related helpers available to MPTCP and patch 7
makes use of them. Patch 8 addresses an edge scenario that could still
lead to transfer stall under memory pressure. Finally patch 9 and 10 improve the MPTCP-level retransmission schema to
make recovery from memory pressure significanly faster.
Note that the diffstat is biases by the quite large patch 4/9,Quoted 9 lines
which
contains mechanical transformation of existing code; "real" changes are
noticiable smaller.
Tested successfully vs the test cases proposed by Geliang and Gang and
vs the selftests.Quoted 9 lines
At the last meeting on Wednesday, Geliang mentioned he validated this
series. Just to be sure, was it the v2 -- from last week -- or the v3 --
from this week, while you were in EU -- that you validated? Because
Paolo couldn't reproduce the issue you mentioned on the v3 on his side.Hi, Matt
The issue can also be reproduced on v3. I reproduced it using Docker's
auto-debug mode with themptcp_data.shselftest:‘’‘
Not running all tests but:-------- 8< --------
run_looprun_selftest_onemptcp_data.sh
-------- 8< --------=== Attempt: 1 (Sat, 09 May 2026 06:55:44 +0000) ===Selftest Test: ./mptcp_data.sh
TAP version 13
1..1# add_addr_accepted 4 subflows 4 # id 1 flags signal 127.0.0.1 10001 # id 2 flags signal 127.0.0.1 10002 # id 3 flags signal 127.0.0.1 10003 # id 4 flags signal 127.0.0.1 10004 # TAP version 13 # 1..48 # # Starting 48 tests from 2 test cases. # # RUN global.mptcp_v6 ... # # OK global.mptcp_v6 # ok 1 global.mptcp_v6 # # RUN mptcp.shutdown_reuse ... # # OK mptcp.shutdown_reuse # ok 2 mptcp.shutdown_reuse ... # ok 48 mptcp.sendfile # # FAILED: 41 / 48 tests passed. # # Totals: pass:41 fail:7 xfail:0 xpass:0 skip:0 error:0 not ok 1 test: selftest_mptcp_data # FAIL # time=33 ’‘’Quoted 2 lines
As Geliang mentioned in the weekly meeting, we will continue to debug and locate the problem based on Paolo's v3 patches.
Thanks for testing. I went over a couple more revisions. I run more than
100 iterations successfully on a local build on top of v5 (no failures,
I stop due to time constraints):
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
please have a spin as such later version.
On v5,
mptcp_data.shworks fine in normal mode with the virtme docker image, but in debug mode it's still unstable - failing after several loop iterations.How many interactions? Can you please provide a sample of such failure? does the failure always happen in the same way/on the same test case?
I could reproduce some failures with several iteration in debug build. Interesting they pointed out to:
- a pre-existing issue (missing wake-up) that deserves a separate patch [1].
- an intrinsic problem with queue collapsing: it's (very) slow and can slow down the received a lot. I had to raise the timeout (replacing TEST_F with TEST_F_TIMEOUT) above 300 seconds to complete 600 mptcp_data runs without errors.
A working alternative to the latter change is avoid entirely the xtcp_*collapse() stuff. The trade off here is that such option will make ./mptcp_join.sh -R slower (as more mptcp-level retransmissions will be needed)
I'll try to share somewhat soonish the fix [1].
/P