Skip to contentSkip to conversation

mptcp: add reset reasons in skb in more cases

PATCH In review Jason Xing 3 participants 10 messages Apr 5, 2024

Subsystems (3)
lore
Share & tools
lore

Jason Xing

author
#0 lore raw

cover mptcp: add reset reasons in skb in more cases

From: Jason Xing <[email protected]>

The first patch only removes the check while the second adds reasons into some places.

Jason Xing (2):
mptcp: don't need to check SKB_EXT_MPTCP in mptcp_reset_option()
mptcp: add reset reason options in some places

 include/net/mptcp.h |  5 +----
 net/mptcp/subflow.c | 21 ++++++++++++++++++---
 2 files changed, 19 insertions(+), 7 deletions(-)
Signature · 1 line
2.37.3
author
#1 lore raw

PATCH 1/2 mptcp: don't need to check SKB_EXT_MPTCP in mptcp_reset_option()

From: Jason Xing <[email protected]>

Before this, what mptcp_reset_option() checks is totally the same as mptcp_get_ext() does, so we could skip it.

---
 include/net/mptcp.h | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
include/net/mptcp.h +1 -4
index fb996124b3d5..42d13ee26619 100644
@@ -215,10 +215,7 @@ __be32 mptcp_get_reset_option(const struct sk_buff *skb);
 
 static inline __be32 mptcp_reset_option(const struct sk_buff *skb)
 {
-	if (skb_ext_exist(skb, SKB_EXT_MPTCP))
-		return mptcp_get_reset_option(skb);
-
-	return htonl(0u);
+	return mptcp_get_reset_option(skb);
 }
 #else
 
Signature · 1 line
2.37.3
maintainer
#2 lore raw

Re: [PATCH net-next 1/2] mptcp: don't need to check SKB_EXT_MPTCP in mptcp_reset_option()

Quoted 4 lines · Jason Xing

From: Jason Xing <[email protected]>

Before this, what mptcp_reset_option() checks is totally the same as mptcp_get_ext() does, so we could skip it.

Note that the somewhat duplicate test is (a possibly not great) optimization to avoid jumping in the mptcp code (possible icache misses) for plain TCP sockets.

I guess we want to maintain it.

Cheers,

Paolo

author
#3 lore raw

Re: [PATCH net-next 1/2] mptcp: don't need to check SKB_EXT_MPTCP in mptcp_reset_option()

Hello Paolo,

Quoted 4 lines · Paolo Abeni
Quoted 2 lines · Jason Xing

From: Jason Xing <[email protected]>

Quoted 8 lines
Quoted 2 lines

Before this, what mptcp_reset_option() checks is totally the same as mptcp_get_ext() does, so we could skip it.

Note that the somewhat duplicate test is (a possibly not great) optimization to avoid jumping in the mptcp code (possible icache misses) for plain TCP sockets.

I guess we want to maintain it.

Okay, I just read code and found the duplication but may I ask why it has something to do with icache misses?

Thanks,
Jason

Quoted 5 lines

Cheers,

Paolo

maintainer
#4 lore raw

Re: [PATCH net-next 1/2] mptcp: don't need to check SKB_EXT_MPTCP in mptcp_reset_option()

Quoted 10 lines · Jason Xing

Hello Paolo,

Quoted 7 lines · Paolo Abeni
Quoted 4 lines · Jason Xing

From: Jason Xing <[email protected]>

Before this, what mptcp_reset_option() checks is totally the same as mptcp_get_ext() does, so we could skip it.

Quoted 8 lines
Quoted 5 lines

Note that the somewhat duplicate test is (a possibly not great) optimization to avoid jumping in the mptcp code (possible icache misses) for plain TCP sockets.

I guess we want to maintain it.

Okay, I just read code and found the duplication but may I ask why it has something to do with icache misses?

The first check/mptcp_get_ext() is in mptcp_reset_option() / tcp_v4_send_reset(). For plain TCP socket it will fail and the execution will continue inside the same compilation unit. The code locality should avoid icaches misses around there.

Removing such check, even when processing plain TCP packets, the code execution will have to call into mptcp_get_reset_option() in the mptcp code, decreasing the code locality and increasing the chance of icache misses.

I don't have actual profile data, so this is an early optimization (and thus root of all evil), but sounds reasonable to me (yep, I'm biased!)

Cheers,

Paolo

author
#5 lore raw

Re: [PATCH net-next 1/2] mptcp: don't need to check SKB_EXT_MPTCP in mptcp_reset_option()

Hello Paolo,

Quoted 26 lines · Paolo Abeni
Quoted 10 lines · Jason Xing

Hello Paolo,

Quoted 7 lines · Paolo Abeni
Quoted 4 lines · Jason Xing

From: Jason Xing <[email protected]>

Before this, what mptcp_reset_option() checks is totally the same as mptcp_get_ext() does, so we could skip it.

Quoted 8 lines
Quoted 5 lines

Note that the somewhat duplicate test is (a possibly not great) optimization to avoid jumping in the mptcp code (possible icache misses) for plain TCP sockets.

I guess we want to maintain it.

Okay, I just read code and found the duplication but may I ask why it has something to do with icache misses?

The first check/mptcp_get_ext() is in mptcp_reset_option() / tcp_v4_send_reset(). For plain TCP socket it will fail and the execution will continue inside the same compilation unit. The code locality should avoid icaches misses around there.

Quoted 4 lines

Removing such check, even when processing plain TCP packets, the code execution will have to call into mptcp_get_reset_option() in the mptcp code, decreasing the code locality and increasing the chance of icache misses.

Interesting. Thanks for the explanation:)

Quoted 3 lines

I don't have actual profile data, so this is an early optimization (and thus root of all evil), but sounds reasonable to me (yep, I'm biased!)

I'll drop this patch.

Thanks,
Jason

Quoted 5 lines

Cheers,

Paolo

author
#6 lore raw

PATCH 2/2 mptcp: add reset reason options in some places

In reply to Jason Xing

From: Jason Xing <[email protected]>

The reason codes are handled in two ways nowadays (quoting Mat Martineau):
1. Sending in the MPTCP option on RST packets when there is no subflow context available (these use subflow_add_reset_reason() and directly call a TCP-level send_reset function)
2. The "normal" way via subflow->reset_reason. This will propagate to both the outgoing reset packet and to a local path manager process via netlink in mptcp_event_sub_closed()

RFC 8684 defines the skb reset reason behaviour which is not required even though in some places:

A host sends a TCP RST in order to close a subflow or reject an attempt to open a subflow (MP_JOIN). In order to let the receiving host know why a subflow is being closed or rejected, the TCP RST packet MAY include the MP_TCPRST option (Figure 15). The host MAY use this information to decide, for example, whether it tries to re-establish the subflow immediately, later, or never.

Since the commit dc87efdb1a5cd ("mptcp: add mptcp reset option support") introduced this feature about three years ago, we can fully use it. There remains some places where we could insert reason into skb as we can see in this patch.

Many thanks to Mat for help:)

---
 net/mptcp/subflow.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
net/mptcp/subflow.c +18 -3
index 1626dd20c68f..49f746d91884 100644
@@ -301,8 +301,13 @@ static struct dst_entry *subflow_v4_route_req(const struct sock *sk,
 		return dst;
 
 	dst_release(dst);
-	if (!req->syncookie)
+	if (!req->syncookie) {
+		struct mptcp_ext *mpext = mptcp_get_ext(skb);
+
+		if (mpext)
+			subflow_add_reset_reason(skb, mpext->reset_reason);
 		tcp_request_sock_ops.send_reset(sk, skb);
+	}
 	return NULL;
 }
 
@@ -368,8 +373,13 @@ static struct dst_entry *subflow_v6_route_req(const struct sock *sk,
 		return dst;
 
 	dst_release(dst);
-	if (!req->syncookie)
+	if (!req->syncookie) {
+		struct mptcp_ext *mpext = mptcp_get_ext(skb);
+
+		if (mpext)
+			subflow_add_reset_reason(skb, mpext->reset_reason);
 		tcp6_request_sock_ops.send_reset(sk, skb);
+	}
 	return NULL;
 }
 #endif
@@ -873,13 +883,18 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 					 ntohs(inet_sk((struct sock *)owner)->inet_sport));
 				if (!mptcp_pm_sport_in_anno_list(owner, sk)) {
 					SUBFLOW_REQ_INC_STATS(req, MPTCP_MIB_MISMATCHPORTACKRX);
+					subflow_add_reset_reason(skb, MPTCP_RST_EPROHIBIT);
 					goto dispose_child;
 				}
 				SUBFLOW_REQ_INC_STATS(req, MPTCP_MIB_JOINPORTACKRX);
 			}
 
-			if (!mptcp_finish_join(child))
+			if (!mptcp_finish_join(child)) {
+				struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(child);
+
+				subflow_add_reset_reason(skb, subflow->reset_reason);
 				goto dispose_child;
+			}
 
 			SUBFLOW_REQ_INC_STATS(req, MPTCP_MIB_JOINACKRX);
 			tcp_rsk(req)->drop_req = true;
Signature · 1 line
2.37.3
maintainer
#7 lore raw

Re: [PATCH net-next 2/2] mptcp: add reset reason options in some places

Quoted 37 lines · Jason Xing

From: Jason Xing <[email protected]>

The reason codes are handled in two ways nowadays (quoting Mat Martineau):
1. Sending in the MPTCP option on RST packets when there is no subflow context available (these use subflow_add_reset_reason() and directly call a TCP-level send_reset function)
2. The "normal" way via subflow->reset_reason. This will propagate to both the outgoing reset packet and to a local path manager process via netlink in mptcp_event_sub_closed()

RFC 8684 defines the skb reset reason behaviour which is not required even though in some places:

A host sends a TCP RST in order to close a subflow or reject an attempt to open a subflow (MP_JOIN). In order to let the receiving host know why a subflow is being closed or rejected, the TCP RST packet MAY include the MP_TCPRST option (Figure 15). The host MAY use this information to decide, for example, whether it tries to re-establish the subflow immediately, later, or never.

Since the commit dc87efdb1a5cd ("mptcp: add mptcp reset option support") introduced this feature about three years ago, we can fully use it. There remains some places where we could insert reason into skb as we can see in this patch.

Many thanks to Mat for help:)

---
 net/mptcp/subflow.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
net/mptcp/subflow.c +0 -0
index 1626dd20c68f..49f746d91884 100644

net/mptcp/subflow.c static struct dst_entry *subflow_v4_route_req(const struct sock *sk, · line 308

Quoted 9 lines
net/mptcp/subflow.c +5 -1
@@ -301,8 +301,13 @@ static struct dst_entry *subflow_v4_route_req(const struct sock *sk,
 		return dst;
 
 	dst_release(dst);
-	if (!req->syncookie)
+	if (!req->syncookie) {
+		struct mptcp_ext *mpext = mptcp_get_ext(skb);
+
+		if (mpext)
+			subflow_add_reset_reason(skb, mpext->reset_reason);

uhm? subflow_add_reset_reason() will do:

	mptcp_ext_add(skb)->reset_reason = mpext->reset_reason

The above looks like a no-op.

Possibly we should instead ensure that subflow_check_req() calls subflow_add_reset_reason() with reasonable arguments on all the error paths?!?

Something alike the (completely untested) following

Cheers,

Paolo
---

net/mptcp/subflow.c +9 -4
index 6042a47da61b..298c6342a78c 100644
@@ -150,8 +150,10 @@ static int subflow_check_req(struct request_sock *req,
 	/* no MPTCP if MD5SIG is enabled on this socket or we may run out of
 	 * TCP option space.
 	 */
-	if (rcu_access_pointer(tcp_sk(sk_listener)->md5sig_info))
+	if (rcu_access_pointer(tcp_sk(sk_listener)->md5sig_info)) {
+		subflow_add_reset_reason(skb, MPTCP_RST_EMPTCP);
 		return -EINVAL;
+	}
 #endif
 
 	mptcp_get_options(skb, &mp_opt);
@@ -219,6 +221,7 @@ static int subflow_check_req(struct request_sock *req,
 				 ntohs(inet_sk((struct sock *)subflow_req->msk)->inet_sport));
 			if (!mptcp_pm_sport_in_anno_list(subflow_req->msk, sk_listener)) {
 				SUBFLOW_REQ_INC_STATS(req, MPTCP_MIB_MISMATCHPORTSYNRX);
+				subflow_add_reset_reason(skb, MPTCP_RST_EPROHIBIT);
 				return -EPERM;
 			}
 			SUBFLOW_REQ_INC_STATS(req, MPTCP_MIB_JOINPORTSYNRX);
@@ -227,10 +230,12 @@ static int subflow_check_req(struct request_sock *req,
 		subflow_req_create_thmac(subflow_req);
 
 		if (unlikely(req->syncookie)) {
-			if (mptcp_can_accept_new_subflow(subflow_req->msk))
-				subflow_init_req_cookie_join_save(subflow_req, skb);
-			else
+			if (!mptcp_can_accept_new_subflow(subflow_req->msk)) {
+				subflow_add_reset_reason(skb, MPTCP_RST_EPROHIBIT);
 				return -EPERM;
+			}
+
+			subflow_init_req_cookie_join_save(subflow_req, skb);
 		}
 
 		pr_debug("token=%u, remote_nonce=%u msk=%p", subflow_req->token,
author
#8 lore raw

Re: [PATCH net-next 2/2] mptcp: add reset reason options in some places

Quoted 51 lines · Paolo Abeni
Quoted 37 lines · Jason Xing

From: Jason Xing <[email protected]>

The reason codes are handled in two ways nowadays (quoting Mat Martineau):
1. Sending in the MPTCP option on RST packets when there is no subflow context available (these use subflow_add_reset_reason() and directly call a TCP-level send_reset function)
2. The "normal" way via subflow->reset_reason. This will propagate to both the outgoing reset packet and to a local path manager process via netlink in mptcp_event_sub_closed()

RFC 8684 defines the skb reset reason behaviour which is not required even though in some places:

A host sends a TCP RST in order to close a subflow or reject an attempt to open a subflow (MP_JOIN). In order to let the receiving host know why a subflow is being closed or rejected, the TCP RST packet MAY include the MP_TCPRST option (Figure 15). The host MAY use this information to decide, for example, whether it tries to re-establish the subflow immediately, later, or never.

Since the commit dc87efdb1a5cd ("mptcp: add mptcp reset option support") introduced this feature about three years ago, we can fully use it. There remains some places where we could insert reason into skb as we can see in this patch.

Many thanks to Mat for help:)

---
 net/mptcp/subflow.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
net/mptcp/subflow.c +0 -0
index 1626dd20c68f..49f746d91884 100644

net/mptcp/subflow.c static struct dst_entry *subflow_v4_route_req(const struct sock *sk, · line 308

Quoted 9 lines
net/mptcp/subflow.c +5 -1
@@ -301,8 +301,13 @@ static struct dst_entry *subflow_v4_route_req(const struct sock *sk,
              return dst;
 
      dst_release(dst);
-     if (!req->syncookie)
+     if (!req->syncookie) {
+             struct mptcp_ext *mpext = mptcp_get_ext(skb);
+
+             if (mpext)
+                     subflow_add_reset_reason(skb, mpext->reset_reason);

uhm? subflow_add_reset_reason() will do:

Quoted 3 lines
        mptcp_ext_add(skb)->reset_reason = mpext->reset_reason

The above looks like a no-op.

Ah, my bad. Actually I didn't add the mpext test statement in my original patch.

Yes, you're right. The 'if (mpext)' is totally unnecessary.

Quoted 4 lines

Possibly we should instead ensure that subflow_check_req() calls subflow_add_reset_reason() with reasonable arguments on all the error paths?!?

Absolutely yes, it would be great. The reason I didn't touch them is I'm still studying how to specify the error kind.

Quoted 38 lines

Something alike the (completely untested) following

Cheers,

Paolo
---

net/mptcp/subflow.c +4 -3
index 6042a47da61b..298c6342a78c 100644
@@ -150,8 +150,10 @@ static int subflow_check_req(struct request_sock *req,
        /* no MPTCP if MD5SIG is enabled on this socket or we may run out of
         * TCP option space.
         */
-       if (rcu_access_pointer(tcp_sk(sk_listener)->md5sig_info))
+       if (rcu_access_pointer(tcp_sk(sk_listener)->md5sig_info)) {
+               subflow_add_reset_reason(skb, MPTCP_RST_EMPTCP);
                return -EINVAL;
+       }
 #endif
 
        mptcp_get_options(skb, &mp_opt);
@@ -219,6 +221,7 @@ static int subflow_check_req(struct request_sock *req,
                                 ntohs(inet_sk((struct sock *)subflow_req->msk)->inet_sport));
                        if (!mptcp_pm_sport_in_anno_list(subflow_req->msk, sk_listener)) {
                                SUBFLOW_REQ_INC_STATS(req, MPTCP_MIB_MISMATCHPORTSYNRX);
+                               subflow_add_reset_reason(skb, MPTCP_RST_EPROHIBIT);
                                return -EPERM;
                        }
                        SUBFLOW_REQ_INC_STATS(req, MPTCP_MIB_JOINPORTSYNRX);
@@ -227,10 +230,12 @@ static int subflow_check_req(struct request_sock *req,
                subflow_req_create_thmac(subflow_req);
 
                if (unlikely(req->syncookie)) {
-                       if (mptcp_can_accept_new_subflow(subflow_req->msk))
-                               subflow_init_req_cookie_join_save(subflow_req, skb);

net/mptcp/subflow.c static int subflow_check_req(struct request_sock *req, · line 241

Quoted 10 lines
net/mptcp/subflow.c +5 -1
-                       else
+                       if (!mptcp_can_accept_new_subflow(subflow_req->msk)) {
+                               subflow_add_reset_reason(skb, MPTCP_RST_EPROHIBIT);
                                return -EPERM;
+                       }
+
+                       subflow_init_req_cookie_join_save(subflow_req, skb);
                }
 
                pr_debug("token=%u, remote_nonce=%u msk=%p", subflow_req->token,

Great! You complete it! Thanks for your instructions.

I'll test it and update it soon.

Thanks,
Jason

#9 lore raw
In reply to Jason Xing

Hi Jason,

Thank you for your modifications, that's great!

Our CI did some validations and here is its report:

- KVM Validation: normal: Success! ✅
- KVM Validation: debug: Success! ✅
- KVM Validation: btf (only bpftest_all): Success! ✅
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/8564316916

Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/f85bdbdcfadc
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=841648

If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:

    $ cd [kernel source code]
    $ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
        --pull always mptcp/mptcp-upstream-virtme-docker:latest \
        auto-normal

For more details:

https://github.com/multipath-tcp/mptcp-upstream-virtme-docker

Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)