mptcp: fallback to TCP on MP_FAIL with a single subflow
Subsystems (2)
v4 cover mptcp: fallback to TCP on MP_FAIL with a single subflow
From: Chenguang Zhao <[email protected]>
As suggested by Paolo, on MP_FAIL reception bound the MP_FAIL reply and the infinite mapping in the same ACK.
Patch1:
Add MPFailFallback MIB.
Patch2:
Makes a pure ACK able to carry an infinite mapping: when send_infinite_map is set, mptcp_established_options_dss() fills a DSS map with data_len=0 even without a data skb. After the ACK that carries both the MP_FAIL reply and that mapping is sent, mptcp_try_fallback() is called immediately so the connection leaves MPTCP mode without waiting for a later data transmit.
Chenguang Zhao (2):
mptcp: add MPFailFallback MIB
mptcp: send MP_FAIL and infinite mapping on the same ACK
net/mptcp/mib.c | 1 + net/mptcp/mib.h | 1 + net/mptcp/options.c | 33 ++++++++++++++++++++++++++++++++- net/mptcp/pm.c | 5 +++++ net/mptcp/protocol.c | 8 +------- 5 files changed, 40 insertions(+), 8 deletions(-)
--
v3:
https://lore.kernel.org/all/[email protected]/
v2:
https://lore.kernel.org/all/[email protected]/
v1:
https://lore.kernel.org/all/[email protected]/
Signature · 1 line
2.25.1
RFC PATCH v4 1/2 mptcp: add MPFailFallback MIB
From: Chenguang Zhao <[email protected]>
Add a counter for TCP fallback after accepting MP_FAIL.
--- net/mptcp/mib.c | 1 + net/mptcp/mib.h | 1 + 2 files changed, 2 insertions(+)
net/mptcp/mib.c +1 -0
@@ -88,6 +88,7 @@ static const struct snmp_mib mptcp_snmp_list[] = {SNMP_MIB_ITEM ("MD5SigFallback" ,MPTCP_MIB_MD5SIGFALLBACK ),SNMP_MIB_ITEM ("MD5SigReset" ,MPTCP_MIB_MD5SIGRESET ),SNMP_MIB_ITEM ("DssFallback" ,MPTCP_MIB_DSSFALLBACK ), +SNMP_MIB_ITEM ("MPFailFallback" ,MPTCP_MIB_MPFAILFALLBACK ),SNMP_MIB_ITEM ("DssReset" ,MPTCP_MIB_DSSRESET ),SNMP_MIB_ITEM ("SimultConnectFallback" ,MPTCP_MIB_SIMULTCONNFALLBACK ), SNMP_MIB_ITEM("FallbackFailed", MPTCP_MIB_FALLBACKFAILED),
net/mptcp/mib.h +1 -0
@@ -91,6 +91,7 @@ enum linux_mptcp_mib_field {MPTCP_MIB_MD5SIGFALLBACK ,/* Conflicting TCP option enabled */ MPTCP_MIB_MD5SIGRESET ,/* MP_RST: MD5SIG enabled on listener */ MPTCP_MIB_DSSFALLBACK ,/* Bad or missing DSS */ +MPTCP_MIB_MPFAILFALLBACK ,/* Received MP_FAIL, fallback */ MPTCP_MIB_DSSRESET ,/* MP_RST: bad data mapping */ MPTCP_MIB_SIMULTCONNFALLBACK ,/* Simultaneous connect */ MPTCP_MIB_FALLBACKFAILED, /* Can't fallback due to msk status */
Signature · 1 line
2.25.1
[email protected]
AI reviewRe: [PATCH mptcp-next RFC v4 1/2] mptcp: add MPFailFallback MIB
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The newly added MIB counter `MPTCP_MIB_MPFAILFALLBACK` is defined and exposed to userspace but…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The newly added MIB counter MPTCP_MIB_MPFAILFALLBACK is defined and exposed to userspace but never incremented anywhere in the codebase. --
--- Patch [1]: [PATCH mptcp-next RFC v4 1/2] mptcp: add MPFailFallback MIB --- commit da81a414aec944549813cb91e3dbd055d9e87a03
Author: Chenguang Zhao <[email protected]>
mptcp: add MPFailFallback MIB
This commit adds a new MIB counter for tracking TCP fallback after accepting MP_FAIL.
Quoted 4 lines
net/mptcp/mib.c +0
-0
[ ... ]
net/mptcp/mib.c static const struct snmp_mib mptcp_snmp_list[] = { · line 91
Quoted 5 lines
net/mptcp/mib.c +1
-0@@ -88,6 +88,7 @@ static const struct snmp_mib mptcp_snmp_list[] = {SNMP_MIB_ITEM ("MD5SigFallback" ,MPTCP_MIB_MD5SIGFALLBACK ),SNMP_MIB_ITEM ("MD5SigReset" ,MPTCP_MIB_MD5SIGRESET ),SNMP_MIB_ITEM ("DssFallback" ,MPTCP_MIB_DSSFALLBACK ), + SNMP_MIB_ITEM("MPFailFallback", MPTCP_MIB_MPFAILFALLBACK),
[Severity: Medium]
Is this counter actually incremented anywhere in the codebase?
It appears that MPTCP_MIB_MPFAILFALLBACK is defined and exposed here, but it is never passed to MPTCP_INC_STATS() or used in tracking functions like mptcp_try_fallback() within this patch.
Without being incremented when a fallback actually occurs, won't this counter just permanently read zero when queried by userspace?
Quoted 3 lines
net/mptcp/mib.c +0
-0SNMP_MIB_ITEM ("DssReset" ,MPTCP_MIB_DSSRESET ),SNMP_MIB_ITEM ("SimultConnectFallback" ,MPTCP_MIB_SIMULTCONNFALLBACK ), SNMP_MIB_ITEM("FallbackFailed", MPTCP_MIB_FALLBACKFAILED),
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
RFC PATCH v4 2/2 mptcp: send MP_FAIL and infinite mapping on the same ACK
In reply to Chenguang ZhaoFrom: Chenguang Zhao <[email protected]>
On a single subflow, accepting MP_FAIL must leave MPTCP mode at once (RFC8684 §3.7). Emit the infinite mapping on the MP_FAIL reply ACK, then call mptcp_try_fallback() immediately after.
--- net/mptcp/options.c | 33 ++++++++++++++++++++++++++++++++- net/mptcp/pm.c | 5 +++++ net/mptcp/protocol.c | 8 +------- 3 files changed, 38 insertions(+), 8 deletions(-)
net/mptcp/options.c +32 -1
@@ -638,7 +638,29 @@ static bool mptcp_established_options_dss(struct sock *sk, struct sk_buff *skb, opts->csum_reqd =READ_ONCE (msk->csum_enabled); mpext = skb ?mptcp_get_ext (skb) :NULL ; -if (!skb || (mpext && mpext->use_map) || snd_data_fin_enable) { +if (unlikely (subflow->send_infinite_map)) { +unsigned int map_size =TCPOLEN_MPTCP_DSS_BASE +TCPOLEN_MPTCP_DSS_MAP64 ; + +if (opts->csum_reqd) + map_size +=TCPOLEN_MPTCP_DSS_CHECKSUM ; + +if (mpext) { + opts->ext_copy = *mpext; + opts->ext_copy.infinite_map =1 ; + opts->ext_copy.data_len =0 ; + }else { + opts->ext_copy.use_map =1 ; + opts->ext_copy.dsn64 =1 ; + opts->ext_copy.infinite_map =1 ; + opts->ext_copy.data_len =0 ; + opts->ext_copy.data_seq =READ_ONCE (msk->snd_nxt); + opts->ext_copy.subflow_seq = subflow->rel_write_seq; + } + + dss_size = map_size; + opts->suboptions =OPTION_MPTCP_DSS ; + ret = true; + }else if (!skb || (mpext && mpext->use_map) || snd_data_fin_enable) {unsigned int map_size =TCPOLEN_MPTCP_DSS_BASE +TCPOLEN_MPTCP_DSS_MAP64 ;if (mpext) {
@@ -1566,6 +1588,15 @@ void mptcp_write_options(struct tcphdr *th, __be32 *ptr, struct tcp_sock *tp, TCPOPT_NOP << 8 | TCPOPT_NOP, ptr); } ptr +=1 ; + +if (unlikely (mpext->infinite_map)) { + subflow =mptcp_subflow_ctx (ssk); +if (subflow->send_infinite_map) { + subflow->send_infinite_map =0 ; +MPTCP_INC_STATS (sock_net (ssk), +MPTCP_MIB_INFINITEMAPTX ); + } + } }/* We might need to add MP_FAIL options in rare cases */
net/mptcp/pm.c +5 -0
@@ -895,6 +895,11 @@ void mptcp_pm_mp_fail_received(struct sock *sk, u64 fail_seq)pr_debug ("MP_FAIL response received\n" );WRITE_ONCE (subflow->fail_tout,0 ); } + +if (!mptcp_try_fallback (sk,MPTCP_MIB_MPFAILFALLBACK )) { +MPTCP_INC_STATS (sock_net (sk),MPTCP_MIB_FALLBACKFAILED ); +mptcp_subflow_reset (sk); + } }static int mptcp_add_addr_len (int family,bool echo,bool port)
net/mptcp/protocol.c +1 -7
@@ -1417,13 +1417,7 @@ static void mptcp_update_infinite_map(struct mptcp_sock *msk, mpext->infinite_map =1 ; mpext->data_len =0 ; - -if (!mptcp_try_fallback (ssk,MPTCP_MIB_INFINITEMAPTX )) { -MPTCP_INC_STATS (sock_net (ssk),MPTCP_MIB_FALLBACKFAILED ); -mptcp_subflow_reset (ssk); -return ; - } - +MPTCP_INC_STATS (sock_net (ssk),MPTCP_MIB_INFINITEMAPTX );mptcp_subflow_ctx (ssk)->send_infinite_map =0 ; }
Signature · 1 line
2.25.1
[email protected]
AI reviewRe: [PATCH mptcp-next RFC v4 2/2] mptcp: send MP_FAIL and infinite mapping on the same ACK
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] MP_FAIL is permanently dropped due to TCP options space exhaustion when packed alongside DSS and…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] MP_FAIL is permanently dropped due to TCP options space exhaustion when packed alongside DSS and Timestamps.
- [High] Dropped pure ACKs cause a fatal connection reset because immediate fallback disables MPTCP options on retransmissions.
- [High] The send_infinite_map flag is cleared when transmitting a pure ACK, omitting the map from the first data packet if the ACK drops. --
--- Patch [2]: [PATCH mptcp-next RFC v4 2/2] mptcp: send MP_FAIL and infinite mapping on the same ACK --- commit fa151aec0663a3e5b6abb8fb25a4fd411a972b0a
Author: Chenguang Zhao <[email protected]>
mptcp: send MP_FAIL and infinite mapping on the same ACK
This commit updates the handling of an MP_FAIL response on a single subflow. It emits the infinite mapping on the MP_FAIL reply ACK and then immediately transitions the socket into fallback mode.
Quoted 5 lines
mptcp: send
MP_FAILand infinite mapping on the same ACKOn a single subflow, accepting
MP_FAILmust leave MPTCP mode at once (RFC8684 §3.7). Emit the infinite mapping on theMP_FAILreply ACK, then callmptcp_try_fallback()immediately after.
[Severity: High]
If we pack both the MP_FAIL and the infinite mapping on the same ACK, is there enough TCP options space if timestamps are enabled?
With standard TCP Timestamps (12 bytes), the remaining option space is 28 bytes.
The infinite mapping requires at least 24 bytes (base + 64-bit map + ACK), leaving at most 4 bytes.
Since MP_FAIL requires 12 bytes, will it fail the space check in mptcp_established_options() and be silently dropped?
Quoted 4 lines
net/mptcp/options.c +0
-0
[ ... ]
net/mptcp/options.c void mptcp_write_options(struct tcphdr *th, __be32 *ptr, struct tcp_sock *tp, · line 1596
Quoted 9 lines
net/mptcp/options.c +5
-0@@ -1567,6 +1589,15 @@ void mptcp_write_options(struct tcphdr *th, __be32 *ptr, struct tcp_sock *tp, TCPOPT_NOP << 8 | TCPOPT_NOP, ptr); } ptr +=1 ; + +if (unlikely (mpext->infinite_map)) { + subflow =mptcp_subflow_ctx (ssk); +if (subflow->send_infinite_map) { + subflow->send_infinite_map =0 ;
[Severity: High]
Does clearing this flag on a pure ACK risk losing the infinite mapping if the network drops the packet?
Since TCP does not retransmit pure ACKs, the infinite mapping would need to be piggybacked on the next data packet.
If we clear the send_infinite_map flag here, will the next data packet fail to include the mapping and transmit as plain TCP instead?
Quoted 7 lines
net/mptcp/options.c +4
-0+MPTCP_INC_STATS (sock_net (ssk), +MPTCP_MIB_INFINITEMAPTX ); + } + } }/* We might need to add MP_FAIL options in rare cases */
[ ... ]
net/mptcp/pm.c void mptcp_pm_mp_fail_received(struct sock *sk, u64 fail_seq) · line 899
Quoted 10 lines
net/mptcp/pm.c +2
-0@@ -895,6 +895,11 @@ void mptcp_pm_mp_fail_received(struct sock *sk, u64 fail_seq)pr_debug ("MP_FAIL response received\n" );WRITE_ONCE (subflow->fail_tout,0 ); } + +if (!mptcp_try_fallback (sk,MPTCP_MIB_MPFAILFALLBACK )) {
[Severity: High]
Does entering fallback immediately after sending the pure ACK prevent retransmission of MPTCP options if the ACK is lost?
If the initial pure ACK drops, the remote peer will retransmit its MP_FAIL. When the local stack responds with another pure ACK, skb is NULL.
Because the socket is already in fallback mode, it looks like mptcp_established_options() will fail the fallback check:
if (unlikely(__mptcp_check_fallback(msk) && !mptcp_check_infinite_map(skb)))
return 0;Could this permanently prevent recovery by omitting the MP_FAIL option on subsequent ACKs?
Quoted 4 lines
net/mptcp/pm.c +3
-0+MPTCP_INC_STATS (sock_net (sk),MPTCP_MIB_FALLBACKFAILED ); +mptcp_subflow_reset (sk); + } }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Matthieu Baerts
maintainerRe: [PATCH mptcp-next RFC v4 2/2] mptcp: send MP_FAIL and infinite mapping on the same ACK
In reply to Chenguang ZhaoHi Chenguang,
Quoted 5 lines · Chenguang Zhao
From: Chenguang Zhao <[email protected]>
On a single subflow, accepting
MP_FAILmust leave MPTCP mode at once (RFC8684 §3.7). Emit the infinite mapping on theMP_FAILreply ACK, then callmptcp_try_fallback()immediately after.
I think Sashiko is right about the issue of lacking option space to have both the MP_FAIL and the infinite mapping sent together.
Please also note this from RFC 8684 §3.7.11 [1]:
Quoted 2 lines
This infinite mapping will be a DSS option (Section 3.3) on the first new packet, containing a Data Sequence Mapping that acts retroactively
The "on the first new packet" seems to suggest the infinite mapping shouldn't be attached to the MP_FAIL.
[1] https://datatracker.ietf.org/doc/html/rfc8684#section-3.7-11
Maybe a fallback for the single subflow without csum is enough? Or maybe nothing needs to be done? Or maybe this fallback is just an optimisation, and can be done in net-next (and both commits can be combined)? WDYT?
Quoted 24 lines
Fixes 1e39e5a32ad7 ("mptcp: infinite mapping sending")Signed-off-by Chenguang Zhao <[email protected]>--- net/mptcp/options.c | 33 ++++++++++++++++++++++++++++++++- net/mptcp/pm.c | 5 +++++ net/mptcp/protocol.c | 8 +------- 3 files changed, 38 insertions(+), 8 deletions(-)net/mptcp/options.c +7
-1@@ -638,7 +638,29 @@ static bool mptcp_established_options_dss(struct sock *sk, struct sk_buff *skb, opts->csum_reqd =READ_ONCE (msk->csum_enabled); mpext = skb ?mptcp_get_ext (skb) :NULL ; -if (!skb || (mpext && mpext->use_map) || snd_data_fin_enable) { +if (unlikely (subflow->send_infinite_map)) { +unsigned int map_size =TCPOLEN_MPTCP_DSS_BASE +TCPOLEN_MPTCP_DSS_MAP64 ; + +if (opts->csum_reqd) + map_size +=TCPOLEN_MPTCP_DSS_CHECKSUM ; + +if (mpext) {
net/mptcp/options.c static bool mptcp_established_options_dss(struct sock *sk, struct sk_buff *skb, · line 657
Quoted 10 lines
net/mptcp/options.c +10
-0+ opts->ext_copy = *mpext; + opts->ext_copy.infinite_map =1 ; + opts->ext_copy.data_len =0 ; + }else { + opts->ext_copy.use_map =1 ; + opts->ext_copy.dsn64 = 1;> + opts->ext_copy.infinite_map = 1; + opts->ext_copy.data_len =0 ; + opts->ext_copy.data_seq =READ_ONCE (msk->snd_nxt); + opts->ext_copy.subflow_seq = subflow->rel_write_seq; + }
You probably also need to reset "csum". Note that common fields (infinite_map and data_len) can also be set once here.
(...)
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.
Re: [PATCH mptcp-next RFC v4 2/2] mptcp: send MP_FAIL and infinite mapping on the same ACK
Quoted 10 lines · Matthieu Baerts
Hi Chenguang,
Quoted 5 lines · Chenguang Zhao
From: Chenguang Zhao <[email protected]>
On a single subflow, accepting
MP_FAILmust leave MPTCP mode at once (RFC8684 §3.7). Emit the infinite mapping on theMP_FAILreply ACK, then callmptcp_try_fallback()immediately after.I think Sashiko is right about the issue of lacking option space to have both the
MP_FAILand the infinite mapping sent together.
Yes, Sashiko is right, I did not consider all the aspects.
Quoted 7 lines
Please also note this from RFC 8684 §3.7.11 [1]:
Quoted 2 lines
This infinite mapping will be a DSS option (Section 3.3) on the first new packet, containing a Data Sequence Mapping that acts retroactively
The "on the first new packet" seems to suggest the infinite mapping shouldn't be attached to the
MP_FAIL.
Quoted 7 lines
[1] https://datatracker.ietf.org/doc/html/rfc8684#section-3.7-11
Maybe a fallback for the single subflow without csum is enough? Or maybe nothing needs to be done? Or maybe this fallback is just an optimisation, and can be done in net-next (and both commits can be combined)? WDYT?
Hi Matthieu
I am getting a bit confused, so let me summarize my current understanding to verify if I got this right. In the existing kernel code, fallback will be triggered upon receiving MP_FAIL. However, the FALLBACK_DONE flag is currently set inside mptcp_update_infinite_map(), which only gets invoked when transmitting a data packet. This creates a timing window between receiving MP_FAIL and sending the first subsequent packet, resulting in this minor bug. Our goal is to close this timing window: set the FALLBACK_DONE flag immediately after MP_FAIL is received. Besides, add a __mptcp_check_fallback() check within mptcp_can_accept_new_subflow() to reject MP_JOIN requests ahead of time.
Is my understanding correct? I will send out a v5 RFC based on the above ideas.
Quoted 41 lines
Quoted 24 lines
Fixes 1e39e5a32ad7 ("mptcp: infinite mapping sending")Signed-off-by Chenguang Zhao <[email protected]>--- net/mptcp/options.c | 33 ++++++++++++++++++++++++++++++++- net/mptcp/pm.c | 5 +++++ net/mptcp/protocol.c | 8 +------- 3 files changed, 38 insertions(+), 8 deletions(-)net/mptcp/options.c +7
-1@@ -638,7 +638,29 @@ static bool mptcp_established_options_dss(struct sock *sk, struct sk_buff *skb, opts->csum_reqd =READ_ONCE (msk->csum_enabled); mpext = skb ?mptcp_get_ext (skb) :NULL ; -if (!skb || (mpext && mpext->use_map) || snd_data_fin_enable) { +if (unlikely (subflow->send_infinite_map)) { +unsigned int map_size =TCPOLEN_MPTCP_DSS_BASE +TCPOLEN_MPTCP_DSS_MAP64 ; + +if (opts->csum_reqd) + map_size +=TCPOLEN_MPTCP_DSS_CHECKSUM ; + +if (mpext) {net/mptcp/options.c static bool mptcp_established_options_dss(struct sock *sk, struct sk_buff *skb, · line 657
Quoted 10 lines
net/mptcp/options.c +10
-0+ opts->ext_copy = *mpext; + opts->ext_copy.infinite_map =1 ; + opts->ext_copy.data_len =0 ; + }else { + opts->ext_copy.use_map =1 ; + opts->ext_copy.dsn64 = 1;> + opts->ext_copy.infinite_map = 1; + opts->ext_copy.data_len =0 ; + opts->ext_copy.data_seq =READ_ONCE (msk->snd_nxt); + opts->ext_copy.subflow_seq = subflow->rel_write_seq; + }You probably also need to reset "csum". Note that common fields (
infinite_mapanddata_len) can also be set once here.(...)
Cheers,
Matt
Matthieu Baerts
maintainerRe: [PATCH mptcp-next RFC v4 2/2] mptcp: send MP_FAIL and infinite mapping on the same ACK
Hi Chenguang,
Quoted 30 lines · Chenguang Zhao
Quoted 10 lines · Matthieu Baerts
Hi Chenguang,
Quoted 5 lines · Chenguang Zhao
From: Chenguang Zhao <[email protected]>
On a single subflow, accepting
MP_FAILmust leave MPTCP mode at once (RFC8684 §3.7). Emit the infinite mapping on theMP_FAILreply ACK, then callmptcp_try_fallback()immediately after.I think Sashiko is right about the issue of lacking option space to have both the
MP_FAILand the infinite mapping sent together.Yes, Sashiko is right, I did not consider all the aspects.
Quoted 7 lines
Please also note this from RFC 8684 §3.7.11 [1]:
Quoted 2 lines
This infinite mapping will be a DSS option (Section 3.3) on the first new packet, containing a Data Sequence Mapping that acts retroactively
The "on the first new packet" seems to suggest the infinite mapping shouldn't be attached to the
MP_FAIL.Quoted 7 lines
[1] https://datatracker.ietf.org/doc/html/rfc8684#section-3.7-11
Maybe a fallback for the single subflow without csum is enough? Or maybe nothing needs to be done? Or maybe this fallback is just an optimisation, and can be done in net-next (and both commits can be combined)? WDYT?
Hi Matthieu
Quoted 8 lines
I am getting a bit confused, so let me summarize my current understanding to verify if I got this right. In the existing kernel code, fallback will be triggered upon receiving
MP_FAIL. However, theFALLBACK_DONEflag is currently set insidemptcp_update_infinite_map(), which only gets invoked when transmitting a data packet. This creates a timing window between receivingMP_FAILand sending the first subsequent packet, resulting in this minor bug. Our goal is to close this timing window: set theFALLBACK_DONEflag immediately afterMP_FAILis received. Besides, add a__mptcp_check_fallback()check withinmptcp_can_accept_new_subflow()to rejectMP_JOINrequests ahead of time.Is my understanding correct? I will send out a v5 RFC based on the above ideas.
I'm not sure that it can work: if there is a fallback, no MPTCP options are supposed to be sent after that point. That's why the fallback should be done after having sent the infinite mapping.
I didn't check in the details: *maybe* the fully_established bit could be reset earlier in some cases, but it feels like it can only be done when a fallback can be done, so never had extra subflows or reinjection, and no checksum. But then again, in this case, we are not supposed to receive an MP_FAIL. Then why bother and not simply fallback, and that's it, no?
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.
Re: [PATCH mptcp-next RFC v4 2/2] mptcp: send MP_FAIL and infinite mapping on the same ACK
Quoted 50 lines · Matthieu Baerts
Hi Chenguang,
Quoted 28 lines · Chenguang Zhao
Quoted 10 lines · Matthieu Baerts
Hi Chenguang,
Quoted 5 lines · Chenguang Zhao
From: Chenguang Zhao <[email protected]>
On a single subflow, accepting
MP_FAILmust leave MPTCP mode at once (RFC8684 §3.7). Emit the infinite mapping on theMP_FAILreply ACK, then callmptcp_try_fallback()immediately after.I think Sashiko is right about the issue of lacking option space to have both the
MP_FAILand the infinite mapping sent together.Yes, Sashiko is right, I did not consider all the aspects.
Quoted 7 lines
Please also note this from RFC 8684 §3.7.11 [1]:
Quoted 2 lines
This infinite mapping will be a DSS option (Section 3.3) on the first new packet, containing a Data Sequence Mapping that acts retroactively
The "on the first new packet" seems to suggest the infinite mapping shouldn't be attached to the
MP_FAIL.Quoted 7 lines
[1] https://datatracker.ietf.org/doc/html/rfc8684#section-3.7-11
Maybe a fallback for the single subflow without csum is enough? Or maybe nothing needs to be done? Or maybe this fallback is just an optimisation, and can be done in net-next (and both commits can be combined)? WDYT?
Hi Matthieu
Quoted 8 lines
I am getting a bit confused, so let me summarize my current understanding to verify if I got this right. In the existing kernel code, fallback will be triggered upon receiving
MP_FAIL. However, theFALLBACK_DONEflag is currently set insidemptcp_update_infinite_map(), which only gets invoked when transmitting a data packet. This creates a timing window between receivingMP_FAILand sending the first subsequent packet, resulting in this minor bug. Our goal is to close this timing window: set theFALLBACK_DONEflag immediately afterMP_FAILis received. Besides, add a__mptcp_check_fallback()check withinmptcp_can_accept_new_subflow()to rejectMP_JOINrequests ahead of time.Is my understanding correct? I will send out a v5 RFC based on the above ideas.
I'm not sure that it can work: if there is a fallback, no MPTCP options are supposed to be sent after that point. That's why the fallback should be done after having sent the infinite mapping.
I didn't check in the details: *maybe* the
fully_establishedbit could be reset earlier in some cases, but it feels like it can only be done when a fallback can be done, so never had extra subflows or reinjection, and no checksum. But then again, in this case, we are not supposed to receive anMP_FAIL. Then why bother and not simply fallback, and that's it, no?
Quoted 2 lines
Cheers,
Matt
Hi Matt,
Thanks for your feedback. I'd like to share some thoughts on the
practical impact of the extra_subflow window.While looking into this,
I realized that the window where the leak can occur is actually quite
narrow in practice:
1. MP_FAIL itself is rare — it requires a checksum validation failure,
which typically only happens when a middlebox corrupts in-flight
data.
2. It only applies to single-subflow connections — for multi-subflow
connections, allow_infinite_fallback is cleared by
mptcp_subflow_joined() when the second subflow joins, so the
fallback path in mptcp_pm_mp_fail_received() is never reached.
3. The window is very short — it spans from the MP_FAIL ACK to the
first data packet carrying the infinite mapping, which is on the
order of microseconds under normal conditions.
4. Even deferring the fallback to the first data packet (as the original code does) does not appear to cause issues in practice, since the infinite mapping is still sent correctly and the connection eventually transitions to TCP.
Given the above, I'm inclined to put this issue on hold for now and
revisit it if a concrete need arises. I'd appreciate your thoughts on
whether that makes sense.
Best regards,
Chenguang
Matthieu Baerts
maintainerRe: [PATCH mptcp-next RFC v4 2/2] mptcp: send MP_FAIL and infinite mapping on the same ACK
Hi Chenguang,
Thank you for your reply!
Quoted 90 lines · Chenguang Zhao
Quoted 50 lines · Matthieu Baerts
Hi Chenguang,
Quoted 28 lines · Chenguang Zhao
Quoted 10 lines · Matthieu Baerts
Hi Chenguang,
Quoted 5 lines · Chenguang Zhao
From: Chenguang Zhao <[email protected]>
On a single subflow, accepting
MP_FAILmust leave MPTCP mode at once (RFC8684 §3.7). Emit the infinite mapping on theMP_FAILreply ACK, then callmptcp_try_fallback()immediately after.I think Sashiko is right about the issue of lacking option space to have both the
MP_FAILand the infinite mapping sent together.Yes, Sashiko is right, I did not consider all the aspects.
Quoted 7 lines
Please also note this from RFC 8684 §3.7.11 [1]:
Quoted 2 lines
This infinite mapping will be a DSS option (Section 3.3) on the first new packet, containing a Data Sequence Mapping that acts retroactively
The "on the first new packet" seems to suggest the infinite mapping shouldn't be attached to the
MP_FAIL.Quoted 7 lines
[1] https://datatracker.ietf.org/doc/html/rfc8684#section-3.7-11
Maybe a fallback for the single subflow without csum is enough? Or maybe nothing needs to be done? Or maybe this fallback is just an optimisation, and can be done in net-next (and both commits can be combined)? WDYT?
Hi Matthieu
Quoted 8 lines
I am getting a bit confused, so let me summarize my current understanding to verify if I got this right. In the existing kernel code, fallback will be triggered upon receiving
MP_FAIL. However, theFALLBACK_DONEflag is currently set insidemptcp_update_infinite_map(), which only gets invoked when transmitting a data packet. This creates a timing window between receivingMP_FAILand sending the first subsequent packet, resulting in this minor bug. Our goal is to close this timing window: set theFALLBACK_DONEflag immediately afterMP_FAILis received. Besides, add a__mptcp_check_fallback()check withinmptcp_can_accept_new_subflow()to rejectMP_JOINrequests ahead of time.Is my understanding correct? I will send out a v5 RFC based on the above ideas.
I'm not sure that it can work: if there is a fallback, no MPTCP options are supposed to be sent after that point. That's why the fallback should be done after having sent the infinite mapping.
I didn't check in the details: *maybe* the
fully_establishedbit could be reset earlier in some cases, but it feels like it can only be done when a fallback can be done, so never had extra subflows or reinjection, and no checksum. But then again, in this case, we are not supposed to receive anMP_FAIL. Then why bother and not simply fallback, and that's it, no?Quoted 2 lines
Cheers,
MattHi Matt,
Thanks for your feedback. I'd like to share some thoughts on the
practical impact of the
extra_subflowwindow.While looking into this,I realized that the window where the leak can occur is actually quite
narrow in practice:
1.
MP_FAILitself is rare — it requires a checksum validation failure,which typically only happens when a middlebox corrupts in-flight
data.
2. It only applies to single-subflow connections — for multi-subflow
connections,
allow_infinite_fallbackis cleared by
mptcp_subflow_joined()when the second subflow joins, so thefallback path in
mptcp_pm_mp_fail_received()is never reached.3. The window is very short — it spans from the
MP_FAILACK to thefirst data packet carrying the infinite mapping, which is on the
order of microseconds under normal conditions.
4. Even deferring the fallback to the first data packet (as the original code does) does not appear to cause issues in practice, since the infinite mapping is still sent correctly and the connection eventually transitions to TCP.
Quoted 5 lines
Given the above, I'm inclined to put this issue on hold for now and
revisit it if a concrete need arises. I'd appreciate your thoughts on
whether that makes sense.
Fine by me. For this specific case (single-subflow, no csum), either we:
- do nothing special because it shouldn't happen and what we are doing is not wrong
- or we do the minimum, i.e. a fallback without infinite mapping, e.g.
if (!READ_ONCE(msk->csum_enabled) && mptcp_try_fallback(...))
(...)Just one last thing while we are talking about MP_FAIL, I wonder if we do fully respect the RFC, e.g.:
- when multiple subflows are in used, an MP_FAIL should send an MP_FAIL back (on the same subflow! is it always the case?) + reset the current subflow (OK I think), but no infinite mapping to send in this case + new subflows are still allowed
https://datatracker.ietf.org/doc/html/rfc8684#section-3.7-8
- when an MP_FAIL is received on the single subflow case and it is not possible to send an infinite mapping, do we reset it?
https://datatracker.ietf.org/doc/html/rfc8684#section-3.7-12
If issues are confirmed, it would be good to create new tickets on GitHub, not to forget about them (except if you are already looking at that). Also, that's a shame we don't have any packetdrill tests involving MP_FAIL: that would be great to cover all these cases with packetdrill (hopefully MP_FAIL is fully supported there).
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.
Hi Chenguang,
Thank you for your modifications, that's great!
Our CI did some validations and here is its report:
- KVM Validation: normal (except selftest_mptcp_join): Success! ✅
- KVM Validation: normal (only selftest_mptcp_join): Success! ✅
- KVM Validation: debug (except selftest_mptcp_join): Success! ✅
- KVM Validation: debug (only selftest_mptcp_join): Success! ✅
- KVM Validation: btf-normal (only bpftest_all): Success! ✅
- KVM Validation: btf-debug (only bpftest_all): Success! ✅
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/32122351648
Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/02ed8cd94c7c
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=1147672
If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:
$ cd [kernel source code]
$ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
--pull always mptcp/mptcp-upstream-virtme-docker:latest \
auto-normalFor more details:
https://github.com/multipath-tcp/mptcp-upstream-virtme-docker
Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)
Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)