mptcp: cope better with mp_join storm
Subsystems (2)
Paolo Abeni
cover mptcp: cope better with mp_join storm
This is a rebase of the RFC, dropping old 5/5 and including a few somewhat related bugfixes - all targeting net-next because the fixed commit is only there or the buggy behavior can be triggered only enabling non backup subflows
Paolo Abeni (9):
subflow: always init 'rel_write_seq'
mptcp: avoid data corruption on reinsert
mptcp: mark as fallback even early ones
mptcp: zero token hash at creation time.
mptcp: make msk established at remote key reception time
mptcp: cleanup subflow_finish_connect()
subflow: explicitly check for plain tcp rsk
subflow: use rsk_ops->send_reset()
subflow: introduce and use mptcp_can_accept_new_subflow()
net/mptcp/options.c | 4 +- net/mptcp/protocol.c | 20 +++++++--- net/mptcp/protocol.h | 2 + net/mptcp/subflow.c | 87 +++++++++++++++++++++++++------------------- net/mptcp/token.c | 2 +- 5 files changed, 69 insertions(+), 46 deletions(-)
Signature · 1 line
2.26.2
Paolo Abeni
PATCH 1/9 subflow: always init 'rel_write_seq'
Currently we do not init the subflow write sequence for MP_JOIN subflows. This will cause bad mapping being generated as soon as we will use non backup subflow.
--- net/mptcp/protocol.c | 1 - net/mptcp/subflow.c | 1 + 2 files changed, 1 insertion(+), 1 deletion(-)
net/mptcp/protocol.c +0 -1
@@ -1819,7 +1819,6 @@ void mptcp_finish_connect(struct sock *ssk) ack_seq++; subflow->map_seq = ack_seq; subflow->map_subflow_seq =1 ; - subflow->rel_write_seq =1 ;/* the socket is not connected yet, no msk/subflow ops can access/race * accessing the field below
net/mptcp/subflow.c +1 -0
@@ -200,6 +200,7 @@ static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)if (subflow->conn_finished)return ; + subflow->rel_write_seq =1 ; subflow->conn_finished =1 ; subflow->ssn_offset =TCP_SKB_CB (skb)->seq;pr_debug ("subflow=%p synack seq=%x" , subflow, subflow->ssn_offset);
Signature · 1 line
2.26.2
Paolo Abeni
PATCH 2/9 mptcp: avoid data corruption on reinsert
In reply to Paolo AbeniWhen updating a partially acked data fragment, we actually corrupt it. This is irrelevant till we send data on a single subflow, as retransmitted data, if any are discarded by the peer as duplicate, but it will cause data corruption as soon as we will start creating non backup subflows.
--- net/mptcp/protocol.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
net/mptcp/protocol.c +6 -1
@@ -460,15 +460,20 @@ static void mptcp_clean_una(struct sock *sk) dfrag =mptcp_rtx_head (sk);if (dfrag &&after64 (snd_una, dfrag->data_seq)) { -u64 delta = dfrag->data_seq + dfrag->data_len - snd_una; +u64 delta = snd_una - dfrag->data_seq; + +if (WARN_ON_ONCE (delta > dfrag->data_len)) + goto out; dfrag->data_seq += delta; + dfrag->offset += delta; dfrag->data_len -= delta;dfrag_uncharge (sk, delta); cleaned = true; } +out:if (cleaned) {sk_mem_reclaim_partial (sk);
Signature · 1 line
2.26.2
Paolo Abeni
PATCH 3/9 mptcp: mark as fallback even early ones
In reply to Paolo AbeniIn the unlikely event of a failure at connect time, we currently clear the request_mptcp flag - so that the MPC handshake is not started at all, but the msk is not explicitly marked as fallback.
This would lead to later insertion of wrong DSS options in the xmitted packets, in violation of RFC specs and possibly fooling the peer.
--- net/mptcp/protocol.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
net/mptcp/protocol.c +9 -2
@@ -1949,6 +1949,13 @@ static int mptcp_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)return err; } +static void mptcp_subflow_early_fallback (struct mptcp_sock *msk, +struct mptcp_subflow_context *subflow) +{ + subflow->request_mptcp =0 ; +__mptcp_do_fallback (msk); +} +static int mptcp_stream_connect (struct socket *sock,struct sockaddr *uaddr,int addr_len,int flags) {
@@ -1980,10 +1987,10 @@ static int mptcp_stream_connect(struct socket *sock, struct sockaddr *uaddr,* TCP option space. */ if (rcu_access_pointer (tcp_sk (ssock->sk)->md5sig_info)) - subflow->request_mptcp =0 ; +mptcp_subflow_early_fallback (msk, subflow);#endif if (subflow->request_mptcp &&mptcp_token_new_connect (ssock->sk)) - subflow->request_mptcp =0 ; +mptcp_subflow_early_fallback (msk, subflow); do_connect: err = ssock->ops->connect (ssock, uaddr, addr_len, flags);
Signature · 1 line
2.26.2
Paolo Abeni
PATCH 4/9 mptcp: zero token hash at creation time.
In reply to Paolo AbeniOtherwise the 'chain_len' filed will carry random values, some token creation calls will fail due to excessive chain lenght, causing unexpected fallback to TCP.
--- net/mptcp/token.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
net/mptcp/token.c +1 -1
@@ -356,7 +356,7 @@ void __init mptcp_token_init(void)sizeof (struct token_bucket ),0 ,20 ,/* one slot per 1MB of memory */ -0 , +HASH_ZERO ,NULL , &token_mask, 0,
Signature · 1 line
2.26.2
Paolo Abeni
PATCH 5/9 mptcp: make msk established at remote key reception time
In reply to Paolo AbeniCurrently accepted msk sockets become established only after accept() returns the new sk to user-space.
As MP_JOIN request are refused as per RFC spec on non established socket, the above causes mp_join self-tests instabilities.
This change make the msk entering the established status as soon as it receives the peer's key. As that even may happen in different places, factor out an helper to handle remote key reception.
--- net/mptcp/options.c | 4 +--- net/mptcp/protocol.c | 1 - net/mptcp/protocol.h | 2 ++ net/mptcp/subflow.c | 19 +++++++++++++++---- 4 files changed, 18 insertions(+), 8 deletions(-)
net/mptcp/options.c +1 -3
@@ -724,9 +724,7 @@ static bool check_fully_established(struct mptcp_sock *msk, struct sock *sk,if (unlikely (!READ_ONCE (msk->pm.server_side)))pr_warn_once ("bogus mpc option on established client sk" ); - subflow->fully_established =1 ; - subflow->remote_key = mp_opt->sndr_key; - subflow->can_ack =1 ; +mptcp_subflow_fully_established (subflow, mp_opt); fully_established:if (likely (subflow->pm_notified))
net/mptcp/protocol.c +0 -1
@@ -1610,7 +1610,6 @@ static struct sock *mptcp_accept(struct sock *sk, int flags, int *err, newsk = new_mptcp_sock;mptcp_copy_inaddrs (newsk, ssk);list_add (&subflow->node, &msk->conn_list); -inet_sk_state_store (newsk,TCP_ESTABLISHED );mptcp_rcv_space_init (msk, ssk);bh_unlock_sock (new_mptcp_sock);
net/mptcp/protocol.h +2 -0
@@ -342,6 +342,8 @@ mptcp_subflow_get_mapped_dsn(const struct mptcp_subflow_context *subflow) }int mptcp_is_enabled (struct net *net); +void mptcp_subflow_fully_established (struct mptcp_subflow_context *subflow, +struct mptcp_options_received *mp_opt);bool mptcp_subflow_data_available (struct sock *sk);void __init mptcp_subflow_init (void );
net/mptcp/subflow.c +15 -4
@@ -387,6 +387,19 @@ static void subflow_drop_ctx(struct sock *ssk)kfree_rcu (ctx, rcu); } +void mptcp_subflow_fully_established (struct mptcp_subflow_context *subflow, +struct mptcp_options_received *mp_opt) +{ +struct mptcp_sock *msk =mptcp_sk (subflow->conn); + + subflow->remote_key = mp_opt->sndr_key; + subflow->fully_established =1 ; + subflow->can_ack =1 ; + +inet_sk_state_store (subflow->conn,TCP_ESTABLISHED ); +mptcp_pm_new_connection (msk,1 ); +} + static struct sock *subflow_syn_recv_sock(const struct sock *sk, struct sk_buff *skb, struct request_sock *req,
@@ -470,7 +483,6 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,* created mptcp socket */ new_msk->sk_destruct = mptcp_sock_destruct; -mptcp_pm_new_connection (mptcp_sk (new_msk),1 );mptcp_token_accept (subflow_req,mptcp_sk (new_msk)); ctx->conn = new_msk; new_msk =NULL ;
@@ -478,9 +490,8 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,/* with OoO packets we can reach here without ingress * mpc option */ - ctx->remote_key = mp_opt.sndr_key; - ctx->fully_established = mp_opt.mp_capable; - ctx->can_ack = mp_opt.mp_capable; +if (mp_opt.mp_capable) +mptcp_subflow_fully_established (ctx, &mp_opt); }else if (ctx->mp_join) {struct mptcp_sock *owner;
Signature · 1 line
2.26.2
Paolo Abeni
PATCH 6/9 mptcp: cleanup subflow_finish_connect()
In reply to Paolo AbeniThe mentioned function has several unneeded branches, handle each case - MP_CAPABLE, MP_JOIN, fallback - under a single conditional and drop quite a bit of duplicate code.
---
RFC -> v1:
- fix handling of syn-cookied mp_join
--- net/mptcp/subflow.c | 56 ++++++++++++++++++++------------------------- 1 file changed, 25 insertions(+), 31 deletions(-)
net/mptcp/subflow.c +25 -31
@@ -206,44 +206,34 @@ static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)pr_debug ("subflow=%p synack seq=%x" , subflow, subflow->ssn_offset);mptcp_get_options (skb, &mp_opt); -if (subflow->request_mptcp && mp_opt.mp_capable) { +if (subflow->request_mptcp) { +if (!mp_opt.mp_capable) { +MPTCP_INC_STATS (sock_net (sk), +MPTCP_MIB_MPCAPABLEACTIVEFALLBACK ); +mptcp_do_fallback (sk); +pr_fallback (mptcp_sk (subflow->conn)); + goto fallback; + } + subflow->mp_capable =1 ; subflow->can_ack =1 ; subflow->remote_key = mp_opt.sndr_key;pr_debug ("subflow=%p, remote_key=%llu" , subflow, subflow->remote_key); - }else if (subflow->request_join && mp_opt.mp_join) { - subflow->mp_join =1 ; +mptcp_finish_connect (sk); + }else if (subflow->request_join) { +u8 hmac[SHA256_DIGEST_SIZE ]; + +if (!mp_opt.mp_join) + goto do_reset; + subflow->thmac = mp_opt.thmac; subflow->remote_nonce = mp_opt.nonce;pr_debug ("subflow=%p, thmac=%llu, remote_nonce=%u" , subflow, subflow->thmac, subflow->remote_nonce); - }else { -if (subflow->request_mptcp) -MPTCP_INC_STATS (sock_net (sk), -MPTCP_MIB_MPCAPABLEACTIVEFALLBACK ); -mptcp_do_fallback (sk); -pr_fallback (mptcp_sk (subflow->conn)); - } -if (mptcp_check_fallback (sk)) { -mptcp_rcv_space_init (mptcp_sk (parent), sk); -return ; - } - -if (subflow->mp_capable) { -pr_debug ("subflow=%p, remote_key=%llu" ,mptcp_subflow_ctx (sk), - subflow->remote_key); -mptcp_finish_connect (sk); - }else if (subflow->mp_join) { -u8 hmac[SHA256_DIGEST_SIZE ]; - -pr_debug ("subflow=%p, thmac=%llu, remote_nonce=%u" , - subflow, subflow->thmac, - subflow->remote_nonce);if (!subflow_thmac_valid (subflow)) {MPTCP_INC_STATS (sock_net (sk),MPTCP_MIB_JOINACKMAC ); - subflow->mp_join =0 ; goto do_reset; }
@@ -251,18 +241,22 @@ static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb) subflow->local_nonce, subflow->remote_nonce, hmac); -memcpy (subflow->hmac, hmac,MPTCPOPT_HMAC_LEN );if (!mptcp_finish_join (sk)) goto do_reset; + subflow->mp_join =1 ;MPTCP_INC_STATS (sock_net (sk),MPTCP_MIB_JOINSYNACKRX ); - }else { -do_reset: -tcp_send_active_reset (sk,GFP_ATOMIC ); -tcp_done (sk); + }else if (mptcp_check_fallback (sk)) { +fallback: +mptcp_rcv_space_init (mptcp_sk (parent), sk); } +return ; + +do_reset: +tcp_send_active_reset (sk,GFP_ATOMIC ); +tcp_done (sk); }static struct request_sock_ops subflow_request_sock_ops;
Signature · 1 line
2.26.2
Paolo Abeni
PATCH 7/9 subflow: explicitly check for plain tcp rsk
In reply to Paolo AbeniWhen syncookie are in use, the TCP stack may feed into subflow_syn_recv_sock() plain TCP request sockets. We can't access mptcp_subflow_request_sock-specific fields on such sockets. Explicitly check the rsk ops to do safe accesses.
---
Note: this fixes an slab-out-of-bounds on syncookied MP_JOIN req, but handling of such req is still problematic:
we relay on the client closing the socket before the 3rd ack, otherwise the client socket will be created as plain TCP one - and again client will have to close it.
--- net/mptcp/subflow.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
net/mptcp/subflow.c +1 -1
@@ -417,7 +417,7 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,/* hopefully temporary handling for MP_JOIN+syncookie */ subflow_req =mptcp_subflow_rsk (req); - fallback_is_fatal = subflow_req->mp_join; + fallback_is_fatal =tcp_rsk (req)->is_mptcp && subflow_req->mp_join; fallback = !tcp_rsk (req)->is_mptcp;if (fallback) goto create_child;
Signature · 1 line
2.26.2
Paolo Abeni
PATCH 8/9 subflow: use rsk_ops->send_reset()
In reply to Paolo Abenitcp_send_active_reset() is more prone to transient errors (memory allocation or xmit queue full): in stress conditions the kernel may drop the egress packet, and the client will be stuck.
--- net/mptcp/subflow.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
net/mptcp/subflow.c +1 -1
@@ -520,9 +520,9 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk, dispose_child:subflow_drop_ctx (child);tcp_rsk (req)->drop_req = true; -tcp_send_active_reset (child,GFP_ATOMIC );inet_csk_prepare_for_destroy_sock (child);tcp_done (child); + req->rsk_ops->send_reset (sk, skb);/* The last child reference will be released by the caller */ return child;
Signature · 1 line
2.26.2
Paolo Abeni
PATCH 9/9 subflow: introduce and use mptcp_can_accept_new_subflow()
In reply to Paolo AbeniSo that we can easily perform some basic PM-related adimission checks before creating the child socket.
---
Note: I initially used mptcp_can_accept_new_subflow() in subflow_token_join_request(), too. The goal was add a follow-up patch sending reset at rsk creation time in case of !can_accept but I dismissed such patch because it feel "insecure"/potentially dos prone send this early reset: an attacker could force an mptcp server to send reset packets towards arbitrary IP/ports.
Additionally, after a quick look at mptcp.org, I could not find trace of such early reset - looks like resets for bad MP_JOINs are send only at syn_recv_sock() time.
--- net/mptcp/subflow.c | 7 +++++++ 1 file changed, 7 insertions(+)
net/mptcp/subflow.c +7 -0
@@ -53,6 +53,12 @@ static void subflow_generate_hmac(u64 key1, u64 key2, u32 nonce1, u32 nonce2,mptcp_crypto_hmac_sha (key1, key2, msg,8 , hmac); } +static bool mptcp_can_accept_new_subflow (const struct mptcp_sock *msk) +{ +return inet_sk_state_load ((struct sock *)msk) ==TCP_ESTABLISHED && +READ_ONCE (msk->pm.accept_subflow); +} +/* validate received token and create truncated hmac and nonce for SYN-ACK */ static struct mptcp_sock *subflow_token_join_request (struct request_sock *req,const struct sk_buff *skb)
@@ -445,6 +451,7 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk, }else if (subflow_req->mp_join) {mptcp_get_options (skb, &mp_opt);if (!mp_opt.mp_join || + !mptcp_can_accept_new_subflow (subflow_req->msk) || !subflow_hmac_valid (req, &mp_opt)) {SUBFLOW_REQ_INC_STATS (req,MPTCP_MIB_JOINACKMAC ); fallback = true;
Signature · 1 line
2.26.2
Mat Martineau
Quoted 15 lines · Paolo Abeni
This is a rebase of the RFC, dropping old 5/5 and including a few somewhat related bugfixes - all targeting net-next because the fixed commit is only there or the buggy behavior can be triggered only enabling non backup subflows
Paolo Abeni (9):
subflow: always init 'rel_write_seq'
mptcp: avoid data corruption on reinsert
mptcp: mark as fallback even early ones
mptcp: zero token hash at creation time.
mptcp: make msk established at remote key reception time
mptcp: cleanupsubflow_finish_connect()
subflow: explicitly check for plain tcp rsk
subflow: usersk_ops->send_reset()
subflow: introduce and usemptcp_can_accept_new_subflow()
Quoted 9 lines
net/mptcp/options.c | 4 +- net/mptcp/protocol.c | 20 +++++++--- net/mptcp/protocol.h | 2 + net/mptcp/subflow.c | 87 +++++++++++++++++++++++++------------------- net/mptcp/token.c | 2 +- 5 files changed, 69 insertions(+), 46 deletions(-)Signature · 1 line
2.26.2
Thanks Paolo, the changes look good. Should these go in mptcp_net-next for a testing pass before upstreaming?
--
Mat Martineau
Intel
Christoph Paasch
Quoted 25 lines · Mat Martineau
Quoted 22 lines · Paolo Abeni
This is a rebase of the RFC, dropping old 5/5 and including a few somewhat related bugfixes - all targeting net-next because the fixed commit is only there or the buggy behavior can be triggered only enabling non backup subflows
Paolo Abeni (9):
subflow: always init 'rel_write_seq'
mptcp: avoid data corruption on reinsert
mptcp: mark as fallback even early ones
mptcp: zero token hash at creation time.
mptcp: make msk established at remote key reception time
mptcp: cleanupsubflow_finish_connect()
subflow: explicitly check for plain tcp rsk
subflow: usersk_ops->send_reset()
subflow: introduce and usemptcp_can_accept_new_subflow()net/mptcp/options.c | 4 +- net/mptcp/protocol.c | 20 +++++++--- net/mptcp/protocol.h | 2 + net/mptcp/subflow.c | 87 +++++++++++++++++++++++++------------------- net/mptcp/token.c | 2 +- 5 files changed, 69 insertions(+), 46 deletions(-)
Quoted 5 lines
Quoted 2 lines
Signature · 1 line
2.26.2Thanks Paolo, the changes look good. Should these go in mptcp_net-next for a testing pass before upstreaming?
I picked these patches on top of export and am running syzkaller on top of it. I will try to do an mptcp.org-interop run later or Monday.
Christoph
Matthieu Baerts
Reviewed-by Mat Martineau Tested-by Christoph PaaschHi Paolo, Mat, Christoph,
Quoted 7 lines · Christoph Paasch
Quoted 6 lines · Mat Martineau
Quoted 3 lines · Paolo Abeni
This is a rebase of the RFC, dropping old 5/5 and including a few somewhat related bugfixes - all targeting net-next because the fixed commit is only there or the buggy behavior can be triggered only enabling non backup subflows
...
Quoted 5 lines
Quoted 2 lines
Thanks Paolo, the changes look good. Should these go in mptcp_net-next for a testing pass before upstreaming?
I picked these patches on top of export and am running syzkaller on top of it. I will try to do an mptcp.org-interop run later or Monday.
These patches are in the "export" branches with these tags:
Please tell me if you wish me to change this.
Cheers,
Matt
Signature · 2 lines
Tessares | Belgium | Hybrid Access Solutions www.tessares.net
Paolo Abeni
Hi,
Quoted 24 lines · Mat Martineau
Quoted 22 lines · Paolo Abeni
This is a rebase of the RFC, dropping old 5/5 and including a few somewhat related bugfixes - all targeting net-next because the fixed commit is only there or the buggy behavior can be triggered only enabling non backup subflows
Paolo Abeni (9):
subflow: always init 'rel_write_seq'
mptcp: avoid data corruption on reinsert
mptcp: mark as fallback even early ones
mptcp: zero token hash at creation time.
mptcp: make msk established at remote key reception time
mptcp: cleanupsubflow_finish_connect()
subflow: explicitly check for plain tcp rsk
subflow: usersk_ops->send_reset()
subflow: introduce and usemptcp_can_accept_new_subflow()net/mptcp/options.c | 4 +- net/mptcp/protocol.c | 20 +++++++--- net/mptcp/protocol.h | 2 + net/mptcp/subflow.c | 87 +++++++++++++++++++++++++------------------- net/mptcp/token.c | 2 +- 5 files changed, 69 insertions(+), 46 deletions(-)
Quoted 5 lines
Quoted 2 lines
Signature · 1 line
2.26.2Thanks Paolo, the changes look good. Should these go in mptcp_net-next for a testing pass before upstreaming?
Sorry for lagging behind. This whole week I'll be half off, so expect added latency.
Yes, the idea is staging this changes a bit in the export branch before pushing them upstream.
Additional note: the fixes commits included here have been found using a staging local self-test using multiple concurrent xmit subflows. I tried to spool them as I expect/fear to found some/many more before obtaining somewhat stable results in the mentioned scenario.
Cheers,
Paolo