Skip to contentSkip to conversation

mptcp: cope better with mp_join storm

RFC PATCH In review Paolo Abeni 2 participants 8 messages Jun 30, 2020 – Jul 10, 2020

Subsystems (2)
lore
Share & tools
lore

Paolo Abeni

author
#0 lore raw

cover mptcp: cope better with mp_join storm

This series it amied at fixing:

https://github.com/multipath-tcp/mptcp_net-next/issues/33

patch 3/5 do the main work, while patch 4 and 5 should reduce the server load avoid creating unneeded children.

Patch 1 and 2 are somewhat related cleanup.

Patch 2 && 5 are likely the most controversial. Any feedback more than welcome!

Paolo Abeni (5):
mptcp: cleanup subflow_finish_connect()
subflow: explicitly check for plain tcp rsk
subflow: use rsk_ops->send_reset()
subflow: introduce and use mptcp_can_accept_new_subflow()
subflow: do not create child subflow for fallback MP_JOIN

 net/mptcp/subflow.c | 76 +++++++++++++++++++++++++--------------------
 1 file changed, 42 insertions(+), 34 deletions(-)
Signature · 1 line
2.26.2
author
#1 lore raw

RFC PATCH 1/5 mptcp: cleanup subflow_finish_connect()

The mentioned function has several unneeded branches, handle each case - MP_CAPABLE, MP_JOIN, fallback - under a single conditional and drop quite a bit of duplicate code.

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
---
 net/mptcp/subflow.c | 46 +++++++++++++++++----------------------------
 1 file changed, 17 insertions(+), 29 deletions(-)
net/mptcp/subflow.c +17 -29
index 0f0fa1ba57a8..30e907690bc5 100644
@@ -211,38 +211,17 @@ static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
 		subflow->remote_key = mp_opt.sndr_key;
 		pr_debug("subflow=%p, remote_key=%llu", subflow,
 			 subflow->remote_key);
+		mptcp_finish_connect(sk);
 	} else if (subflow->request_join && mp_opt.mp_join) {
-		subflow->mp_join = 1;
+		u8 hmac[SHA256_DIGEST_SIZE];
+
 		subflow->thmac = mp_opt.thmac;
 		subflow->remote_nonce = mp_opt.nonce;
 		pr_debug("subflow=%p, thmac=%llu, remote_nonce=%u", subflow,
 			 subflow->thmac, subflow->remote_nonce);
-	} else {
-		if (subflow->request_mptcp)
-			MPTCP_INC_STATS(sock_net(sk),
-					MPTCP_MIB_MPCAPABLEACTIVEFALLBACK);
-		mptcp_do_fallback(sk);
-		pr_fallback(mptcp_sk(subflow->conn));
-	}
-
-	if (mptcp_check_fallback(sk)) {
-		mptcp_rcv_space_init(mptcp_sk(parent), sk);
-		return;
-	}
-
-	if (subflow->mp_capable) {
-		pr_debug("subflow=%p, remote_key=%llu", mptcp_subflow_ctx(sk),
-			 subflow->remote_key);
-		mptcp_finish_connect(sk);
-	} else if (subflow->mp_join) {
-		u8 hmac[SHA256_DIGEST_SIZE];
 
-		pr_debug("subflow=%p, thmac=%llu, remote_nonce=%u",
-			 subflow, subflow->thmac,
-			 subflow->remote_nonce);
 		if (!subflow_thmac_valid(subflow)) {
 			MPTCP_INC_STATS(sock_net(sk), MPTCP_MIB_JOINACKMAC);
-			subflow->mp_join = 0;
 			goto do_reset;
 		}
 
@@ -250,18 +229,27 @@ static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
 				      subflow->local_nonce,
 				      subflow->remote_nonce,
 				      hmac);
-
 		memcpy(subflow->hmac, hmac, MPTCPOPT_HMAC_LEN);
 
 		if (!mptcp_finish_join(sk))
 			goto do_reset;
 
+		subflow->mp_join = 1;
 		MPTCP_INC_STATS(sock_net(sk), MPTCP_MIB_JOINSYNACKRX);
-	} else {
-do_reset:
-		tcp_send_active_reset(sk, GFP_ATOMIC);
-		tcp_done(sk);
+	} else if (subflow->request_mptcp) {
+		MPTCP_INC_STATS(sock_net(sk),
+				MPTCP_MIB_MPCAPABLEACTIVEFALLBACK);
+		mptcp_do_fallback(sk);
+		pr_fallback(mptcp_sk(subflow->conn));
 	}
+
+	if (mptcp_check_fallback(sk))
+		mptcp_rcv_space_init(mptcp_sk(parent), sk);
+	return;
+
+do_reset:
+	tcp_send_active_reset(sk, GFP_ATOMIC);
+	tcp_done(sk);
 }
 
 static struct request_sock_ops subflow_request_sock_ops;
Signature · 1 line
2.26.2
author
#2 lore raw

RFC PATCH 2/5 subflow: explicitly check for plain tcp rsk

In reply to Paolo Abeni

When syncookie are in use, the TCP stack may feed into subflow_syn_recv_sock() plain TCP request sockets. We can't access mptcp_subflow_request_sock-specific fields on such sockets. Explicitly check the rsk ops to do safe accesses.

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>

---
Note: this fixes an slab-out-of-bounds on syncookied MP_JOIN req, but handling of such req is still problematic:
we relay on the client closing the socket before the 3rd ack, otherwise the client socket will be created as plain TCP one - and again client will have to close it.

---
 net/mptcp/subflow.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/mptcp/subflow.c +1 -1
index 30e907690bc5..862997d5d9f3 100644
@@ -397,7 +397,7 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 
 	/* hopefully temporary handling for MP_JOIN+syncookie */
 	subflow_req = mptcp_subflow_rsk(req);
-	fallback_is_fatal = subflow_req->mp_join;
+	fallback_is_fatal = tcp_rsk(req)->is_mptcp && subflow_req->mp_join;
 	fallback = !tcp_rsk(req)->is_mptcp;
 	if (fallback)
 		goto create_child;
Signature · 1 line
2.26.2
author
#3 lore raw

RFC PATCH 3/5 subflow: use rsk_ops->send_reset()

In reply to Paolo Abeni

tcp_send_active_reset() is more prone to transient errors (memory allocation or xmit queue full): in stress conditions the kernel may drop the egress packet, and the client will be stuck.

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
---
 net/mptcp/subflow.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/mptcp/subflow.c +1 -1
index 862997d5d9f3..f3517d4f70f6 100644
@@ -502,9 +502,9 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 dispose_child:
 	subflow_drop_ctx(child);
 	tcp_rsk(req)->drop_req = true;
-	tcp_send_active_reset(child, GFP_ATOMIC);
 	inet_csk_prepare_for_destroy_sock(child);
 	tcp_done(child);
+	req->rsk_ops->send_reset(sk, skb);
 
 	/* The last child reference will be released by the caller */
 	return child;
Signature · 1 line
2.26.2
author
#4 lore raw

RFC PATCH 4/5 subflow: introduce and use mptcp_can_accept_new_subflow()

In reply to Paolo Abeni

So that we can easily perform some basic PM-related adimission checks before creating the child socket.

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>

---
Note: I initially used mptcp_can_accept_new_subflow() in subflow_token_join_request(), too. The goal was add a follow-up patch sending reset at rsk creation time in case of !can_accept but I dismissed such patch because it feel "insecure"/potentially dos prone send this early reset: an attacker could force an mptcp server to send reset packets towards arbitrary IP/ports.

Additionally, after a quick look at mptcp.org, I could not find trace of such early reset - looks like resets for bad MP_JOINs are send only at syn_recv_sock() time.

---
 net/mptcp/subflow.c | 7 +++++++
 1 file changed, 7 insertions(+)
net/mptcp/subflow.c +7 -0
index f3517d4f70f6..3077e07375b5 100644
@@ -53,6 +53,12 @@ static void subflow_generate_hmac(u64 key1, u64 key2, u32 nonce1, u32 nonce2,
 	mptcp_crypto_hmac_sha(key1, key2, msg, 8, hmac);
 }
 
+static bool mptcp_can_accept_new_subflow(const struct mptcp_sock *msk)
+{
+	return inet_sk_state_load((struct sock *)msk) == TCP_ESTABLISHED &&
+	       READ_ONCE(msk->pm.accept_subflow);
+}
+
 /* validate received token and create truncated hmac and nonce for SYN-ACK */
 static struct mptcp_sock *subflow_token_join_request(struct request_sock *req,
 						     const struct sk_buff *skb)
@@ -425,6 +431,7 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 	} else if (subflow_req->mp_join) {
 		mptcp_get_options(skb, &mp_opt);
 		if (!mp_opt.mp_join ||
+		    !mptcp_can_accept_new_subflow(subflow_req->msk) ||
 		    !subflow_hmac_valid(req, &mp_opt)) {
 			SUBFLOW_REQ_INC_STATS(req, MPTCP_MIB_JOINACKMAC);
 			fallback = true;
Signature · 1 line
2.26.2
author
#5 lore raw

RFC PATCH 5/5 subflow: do not create child subflow for fallback MP_JOIN

In reply to Paolo Abeni

On MP_JOIN fallback, the child socket was used just to send-out the due reset packet. As we don't use the child sock anymore for such role, when can avoid creating the fallback MP_JOIN sock entirely.

This saves a busy server from quite a bit of useless work, if clients are storming with MP_JOIN reqs above the server limits

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>

---
Note: to avoid changing the hook into tcp_check_req(), I had to introduce a quite ugly const cast in subflow_syn_recv_sock(). I *think* it should be safe, but I'm 110% sure about unintended implications - alike cc optimizing the calling code assuming sk never changes, and generated code being bugged due to the above cast.

I think should be safe, as the struct sock is not accessed by tcp_check_req() when the relevant conditions held after syn_recv_sock(), still something require additional eyeballs.

---
 net/mptcp/subflow.c | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)
net/mptcp/subflow.c +17 -4
index 3077e07375b5..9f4f3e29587d 100644
@@ -405,8 +405,11 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 	subflow_req = mptcp_subflow_rsk(req);
 	fallback_is_fatal = tcp_rsk(req)->is_mptcp && subflow_req->mp_join;
 	fallback = !tcp_rsk(req)->is_mptcp;
-	if (fallback)
+	if (fallback) {
+		if (fallback_is_fatal)
+			goto dispose_req;
 		goto create_child;
+	}
 
 	/* if the sk is MP_CAPABLE, we try to fetch the client key */
 	if (subflow_req->mp_capable) {
@@ -434,7 +437,7 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 		    !mptcp_can_accept_new_subflow(subflow_req->msk) ||
 		    !subflow_hmac_valid(req, &mp_opt)) {
 			SUBFLOW_REQ_INC_STATS(req, MPTCP_MIB_JOINACKMAC);
-			fallback = true;
+			goto dispose_req;
 		}
 	}
 
@@ -506,15 +509,25 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 		      !mptcp_subflow_ctx(child)->conn));
 	return child;
 
+dispose_req:
+	/* The last req reference will be released by the caller */
+	child = NULL;
+	*own_req = false;
+	reqsk_queue_removed(&inet_csk(sk)->icsk_accept_queue, req);
+	inet_csk_reqsk_queue_drop((struct sock *)sk, req);
+	goto reset;
+
 dispose_child:
+	/* The last child reference will be released by the caller */
 	subflow_drop_ctx(child);
 	tcp_rsk(req)->drop_req = true;
 	inet_csk_prepare_for_destroy_sock(child);
 	tcp_done(child);
-	req->rsk_ops->send_reset(sk, skb);
 
-	/* The last child reference will be released by the caller */
+reset:
+	req->rsk_ops->send_reset(sk, skb);
 	return child;
+
 }
 
 static struct inet_connection_sock_af_ops subflow_specific;
Signature · 1 line
2.26.2
#6 lore raw

[MPTCP] Re: [RFC PATCH 5/5] subflow: do not create child subflow for fallback MP_JOIN

Quoted 51 lines · Paolo Abeni

On MP_JOIN fallback, the child socket was used just to send-out the due reset packet. As we don't use the child sock anymore for such role, when can avoid creating the fallback MP_JOIN sock entirely.

This saves a busy server from quite a bit of useless work, if clients are storming with MP_JOIN reqs above the server limits

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>

---
Note: to avoid changing the hook into tcp_check_req(), I had to introduce a quite ugly const cast in subflow_syn_recv_sock(). I *think* it should be safe, but I'm 110% sure about unintended implications - alike cc optimizing the calling code assuming sk never changes, and generated code being bugged due to the above cast.

I think should be safe, as the struct sock is not accessed by tcp_check_req() when the relevant conditions held after syn_recv_sock(), still something require additional eyeballs.
---
net/mptcp/subflow.c | 21 +++++++++++++++++----

1 file changed, 17 insertions(+), 4 deletions(-)
net/mptcp/subflow.c +5 -2
index 3077e07375b5..9f4f3e29587d 100644
@@ -405,8 +405,11 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 	subflow_req = mptcp_subflow_rsk(req);
 	fallback_is_fatal = tcp_rsk(req)->is_mptcp && subflow_req->mp_join;
 	fallback = !tcp_rsk(req)->is_mptcp;
-	if (fallback)
+	if (fallback) {
+		if (fallback_is_fatal)
+			goto dispose_req;
 		goto create_child;
+	}
 
 	/* if the sk is MP_CAPABLE, we try to fetch the client key */
 	if (subflow_req->mp_capable) {
@@ -434,7 +437,7 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 		    !mptcp_can_accept_new_subflow(subflow_req->msk) ||
 		    !subflow_hmac_valid(req, &mp_opt)) {
 			SUBFLOW_REQ_INC_STATS(req, MPTCP_MIB_JOINACKMAC);
-			fallback = true;
+			goto dispose_req;
 		}
 	}
 

net/mptcp/subflow.c static struct sock *subflow_syn_recv_sock(const struct sock *sk, · line 517

Quoted 9 lines
net/mptcp/subflow.c +6 -0
@@ -506,15 +509,25 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 		      !mptcp_subflow_ctx(child)->conn));
 	return child;
 
+dispose_req:
+	/* The last req reference will be released by the caller */
+	child = NULL;
+	*own_req = false;
+	reqsk_queue_removed(&inet_csk(sk)->icsk_accept_queue, req);
+	inet_csk_reqsk_queue_drop((struct sock *)sk, req);

I'm not sure about unintended consequences of this cast either. Even if it looks ok now, the calling code could be changed.

Eric constified this parameter in 0c27171e66d9, saying "We'll soon no longer hold listener socket lock, these functions do not modify the socket in any way". It looks like the only modification to the socket by calling inet_csk_reqsk_queue_drop() is an atomic_dec() or two, which should be an ok thing to do. Do you think that would justify reverting that change to the sk parameter?

Looking at the call from tcp_check_req(), it looks like ient_csk_reqsk_queue_drop() and req->rsk_ops->send_reset() can get called again. This function will return NULL on this code path, and if the tcp_abort_on_overflow sysctl is true, the drop/reset functions can get called again. Is that not possible for not-obvious-to-me reasons?

Quoted 14 lines
net/mptcp/subflow.c +2 -0
+	goto reset;
+

dispose_child:

net/mptcp/subflow.c +1 -0
+	/* The last child reference will be released by the caller */
	subflow_drop_ctx(child);
	tcp_rsk(req)->drop_req = true;
	inet_csk_prepare_for_destroy_sock(child);
	tcp_done(child);
net/mptcp/subflow.c +2 -2
-	req->rsk_ops->send_reset(sk, skb);
 
-	/* The last child reference will be released by the caller */
+reset:
+	req->rsk_ops->send_reset(sk, skb);
	return child;
Quoted 6 lines

+
}

static struct inet_connection_sock_af_ops subflow_specific;

Signature · 1 line
2.26.2

--
Mat Martineau
Intel

author
#7 lore raw

[MPTCP] Re: [RFC PATCH 5/5] subflow: do not create child subflow for fallback MP_JOIN

Quoted 66 lines · Mat Martineau
Quoted 51 lines · Paolo Abeni

On MP_JOIN fallback, the child socket was used just to send-out the due reset packet. As we don't use the child sock anymore for such role, when can avoid creating the fallback MP_JOIN sock entirely.

This saves a busy server from quite a bit of useless work, if clients are storming with MP_JOIN reqs above the server limits

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>

---
Note: to avoid changing the hook into tcp_check_req(), I had to introduce a quite ugly const cast in subflow_syn_recv_sock(). I *think* it should be safe, but I'm 110% sure about unintended implications - alike cc optimizing the calling code assuming sk never changes, and generated code being bugged due to the above cast.

I think should be safe, as the struct sock is not accessed by tcp_check_req() when the relevant conditions held after syn_recv_sock(), still something require additional eyeballs.
---
net/mptcp/subflow.c | 21 +++++++++++++++++----

1 file changed, 17 insertions(+), 4 deletions(-)
net/mptcp/subflow.c +5 -2
index 3077e07375b5..9f4f3e29587d 100644
@@ -405,8 +405,11 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 	subflow_req = mptcp_subflow_rsk(req);
 	fallback_is_fatal = tcp_rsk(req)->is_mptcp && subflow_req->mp_join;
 	fallback = !tcp_rsk(req)->is_mptcp;
-	if (fallback)
+	if (fallback) {
+		if (fallback_is_fatal)
+			goto dispose_req;
 		goto create_child;
+	}
 
 	/* if the sk is MP_CAPABLE, we try to fetch the client key */
 	if (subflow_req->mp_capable) {
@@ -434,7 +437,7 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 		    !mptcp_can_accept_new_subflow(subflow_req->msk) ||
 		    !subflow_hmac_valid(req, &mp_opt)) {
 			SUBFLOW_REQ_INC_STATS(req, MPTCP_MIB_JOINACKMAC);
-			fallback = true;
+			goto dispose_req;
 		}
 	}
 

net/mptcp/subflow.c static struct sock *subflow_syn_recv_sock(const struct sock *sk, · line 517

Quoted 9 lines
net/mptcp/subflow.c +6 -0
@@ -506,15 +509,25 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 		      !mptcp_subflow_ctx(child)->conn));
 	return child;
 
+dispose_req:
+	/* The last req reference will be released by the caller */
+	child = NULL;
+	*own_req = false;
+	reqsk_queue_removed(&inet_csk(sk)->icsk_accept_queue, req);
+	inet_csk_reqsk_queue_drop((struct sock *)sk, req);

I'm not sure about unintended consequences of this cast either. Even if it looks ok now, the calling code could be changed.

Quoted 6 lines

Eric constified this parameter in 0c27171e66d9, saying "We'll soon no longer hold listener socket lock, these functions do not modify the socket in any way". It looks like the only modification to the socket by calling inet_csk_reqsk_queue_drop() is an atomic_dec() or two, which should be an ok thing to do. Do you think that would justify reverting that change to the sk parameter?

Thank you for the feedback!

No, I think removing the 'const' from 'syn_recv_sock()' for this change is a no-go.

Quoted 5 lines

Looking at the call from tcp_check_req(), it looks like ient_csk_reqsk_queue_drop() and req->rsk_ops->send_reset() can get called again. This function will return NULL on this code path, and if the tcp_abort_on_overflow sysctl is true, the drop/reset functions can get called again. Is that not possible for not-obvious-to-me reasons?

No nothing prevent that code path from being executed - even if requires non-default sysctl. Anyhow a duplicated rst packet should not be too much problematic, I hope, and the 2nd execution of inet_csk_reqsk_queue_drop() will be a no-op, as the sk will not be hashed anymore.

Perhaps we can avoid the const cast changing a bit the hooking in tcp_check_req(), but I'm reluctant to do that, as I changed the code there recently.

@Florian: do you foresee to have to touch tcp_check_req() for syncookie handling? Perhaps we can unify the change there?

For the short term I'll send a v1 without 5/5 and including some other fixes I'm assembling while working on issues/17.

Thanks!

Paolo