Skip to contentSkip to conversation

mptcp: a couple of fixup

PATCH In review Paolo Abeni 3 participants 8 messages Apr 5, 2023 – Apr 6, 2023

Subsystems (2)
lore
Share & tools
lore

Paolo Abeni

author maintainer
#0 lore raw

cover mptcp: a couple of fixup

Hopefully addressing for good issues/371

There are actually several different problems that pop-up in random order with the repro from such issue.

@Christoph: could you please have a spin in your testbed?

Paolo Abeni (2):
Squash-to: "mptcp: fix accept vs worker race"
Squash-to: "mptcp: stops worker on unaccepted sockets at listener
close"

 net/mptcp/protocol.c | 13 +++++++++++--
 net/mptcp/subflow.c  |  4 ++--
 2 files changed, 13 insertions(+), 4 deletions(-)
Signature · 1 line
2.39.2
author maintainer
#1 lore raw

PATCH 1/2 Squash-to: "mptcp: fix accept vs worker race"

2 separate fixes needed for the mentioned patch

On some exceptional scenarios (e.g. listener socket disconnecting while the subflow is created) a newly created MPC is deleted by the TCP stack via inet_child_forget(). When that happen we leak the subflow context and the msk.

Address the issue explicitly detecting such scenario and forcing a full msk shutdown.

mptcp_subflow_drop_ctx() does not remove the subflow from the msk list, racing msk lookup done by the NL PM could fetch the msk just before deletion, and try to free again the subflow.

Just explicitly remove the subflow from said list.

---
 net/mptcp/protocol.c | 13 +++++++++++--
 net/mptcp/subflow.c  |  1 +
 2 files changed, 12 insertions(+), 2 deletions(-)
net/mptcp/protocol.c +11 -2
index 0e4961e87b48..76c1814c0b19 100644
@@ -2378,9 +2378,18 @@ static void __mptcp_close_ssk(struct sock *sk, struct sock *ssk,
 	 * to deliver the msk to user-space.
 	 * Do nothing at the moment and take action at accept and/or listener
 	 * shutdown.
+	 * If instead such subflow has been destroyed, e.g. by inet_child_forget
+	 * do the kill
 	 */
-	if (msk->in_accept_queue && msk->first == ssk)
-		return;
+	if (msk->in_accept_queue && msk->first == ssk) {
+		if (!sock_flag(ssk, SOCK_DEAD))
+			return;
+
+		/* ensure later check in mptcp_worker will dispose the msk */
+		sock_set_flag(sk, SOCK_DEAD);
+		inet_csk(sk)->icsk_mtup.probe_timestamp = tcp_jiffies32 -
+							  TCP_TIMEWAIT_LEN -1;
+	}
 
 	dispose_it = !msk->subflow || ssk != msk->subflow->sk;
 	if (dispose_it)
net/mptcp/subflow.c +1 -0
index 3d6ccf5067d7..32e54b7fdbbc 100644
@@ -715,6 +715,7 @@ void mptcp_subflow_drop_ctx(struct sock *ssk)
 	if (!ctx)
 		return;
 
+	list_del(&mptcp_subflow_ctx(ssk)->node);
 	subflow_ulp_fallback(ssk, ctx);
 	if (ctx->conn)
 		sock_put(ctx->conn);
Signature · 1 line
2.39.2
#2 lore raw

Re: [PATCH mptcp-net 1/2] Squash-to: "mptcp: fix accept vs worker race"

Hi Paolo,

Thank you for the patch!

It looks good to me but I just have some questions if you don't mind, just to be sure I understood the modifications.

Quoted 31 lines · Paolo Abeni

2 separate fixes needed for the mentioned patch

On some exceptional scenarios (e.g. listener socket disconnecting while the subflow is created) a newly created MPC is deleted by the TCP stack via inet_child_forget(). When that happen we leak the subflow context and the msk.

Address the issue explicitly detecting such scenario and forcing a full msk shutdown.

mptcp_subflow_drop_ctx() does not remove the subflow from the msk list, racing msk lookup done by the NL PM could fetch the msk just before deletion, and try to free again the subflow.

Just explicitly remove the subflow from said list.

---
 net/mptcp/protocol.c | 13 +++++++++++--
 net/mptcp/subflow.c  |  1 +
 2 files changed, 12 insertions(+), 2 deletions(-)
net/mptcp/protocol.c +1 -0
index 0e4961e87b48..76c1814c0b19 100644
@@ -2378,9 +2378,18 @@ static void __mptcp_close_ssk(struct sock *sk, struct sock *ssk,
 	 * to deliver the msk to user-space.
 	 * Do nothing at the moment and take action at accept and/or listener
 	 * shutdown.
+	 * If instead such subflow has been destroyed, e.g. by inet_child_forget

net/mptcp/protocol.c static void __mptcp_close_ssk(struct sock *sk, struct sock *ssk, · line 2389

Quoted 10 lines
net/mptcp/protocol.c +7 -2
+	 * do the kill
 	 */
-	if (msk->in_accept_queue && msk->first == ssk)
-		return;
+	if (msk->in_accept_queue && msk->first == ssk) {
+		if (!sock_flag(ssk, SOCK_DEAD))
+			return;
+
+		/* ensure later check in mptcp_worker will dispose the msk */
+		sock_set_flag(sk, SOCK_DEAD);

Is it OK to mark the msk as dead and continue below? I guess yes because there will not be any data to re-inject anyway?

net/mptcp/protocol.c static void __mptcp_close_ssk(struct sock *sk, struct sock *ssk, · line 2391

Quoted 2 lines
net/mptcp/protocol.c +2 -0
+		inet_csk(sk)->icsk_mtup.probe_timestamp = tcp_jiffies32 -
+							  TCP_TIMEWAIT_LEN -1;

Changing the timestamp looks like an ugly hack :)

But I guess it is needed not to wait for nothing and there is no clearer way? Or move this logic to the worker?

Cheers,
Matt

Signature · 2 lines
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net
author maintainer
#3 lore raw

Re: [PATCH mptcp-net 1/2] Squash-to: "mptcp: fix accept vs worker race"

Quoted 44 lines · Matthieu Baerts

Hi Paolo,

Thank you for the patch!

It looks good to me but I just have some questions if you don't mind, just to be sure I understood the modifications.

Quoted 36 lines · Paolo Abeni

2 separate fixes needed for the mentioned patch

On some exceptional scenarios (e.g. listener socket disconnecting while the subflow is created) a newly created MPC is deleted by the TCP stack via inet_child_forget(). When that happen we leak the subflow context and the msk.

Address the issue explicitly detecting such scenario and forcing a full msk shutdown.

mptcp_subflow_drop_ctx() does not remove the subflow from the msk list, racing msk lookup done by the NL PM could fetch the msk just before deletion, and try to free again the subflow.

Just explicitly remove the subflow from said list.

---
 net/mptcp/protocol.c | 13 +++++++++++--
 net/mptcp/subflow.c  |  1 +
 2 files changed, 12 insertions(+), 2 deletions(-)
net/mptcp/protocol.c +3 -2
index 0e4961e87b48..76c1814c0b19 100644
@@ -2378,9 +2378,18 @@ static void __mptcp_close_ssk(struct sock *sk, struct sock *ssk,
 	 * to deliver the msk to user-space.
 	 * Do nothing at the moment and take action at accept and/or listener
 	 * shutdown.
+	 * If instead such subflow has been destroyed, e.g. by inet_child_forget
+	 * do the kill
 	 */
-	if (msk->in_accept_queue && msk->first == ssk)
-		return;
+	if (msk->in_accept_queue && msk->first == ssk) {
Quoted 8 lines

net/mptcp/protocol.c static void __mptcp_close_ssk(struct sock *sk, struct sock *ssk, · line 2389

Quoted 5 lines
net/mptcp/protocol.c +5 -0
+		if (!sock_flag(ssk, SOCK_DEAD))
+			return;
+
+		/* ensure later check in mptcp_worker will dispose the msk */
+		sock_set_flag(sk, SOCK_DEAD);

Is it OK to mark the msk as dead and continue below? I guess yes because there will not be any data to re-inject anyway?

We don't process any data here (the msk is unaccepted at this point). Looks safe/sane to me. Overall this is the MPTCP equivalent of inet_child_forget()

Quoted 7 lines

net/mptcp/protocol.c static void __mptcp_close_ssk(struct sock *sk, struct sock *ssk, · line 2391

Quoted 2 lines
net/mptcp/protocol.c +2 -0
+		inet_csk(sk)->icsk_mtup.probe_timestamp = tcp_jiffies32 -
+							  TCP_TIMEWAIT_LEN -1;

Changing the timestamp looks like an ugly hack :)

But I guess it is needed not to wait for nothing and there is no clearer way? Or move this logic to the worker?

I can add another flag inside the msk. I did not do that here to keep the patch small, but no objection on my side to clean this path - the important thing is let syzkaller crunch this ;)

/P

#4 lore raw

Re: [PATCH mptcp-net 1/2] Squash-to: "mptcp: fix accept vs worker race"

Hi Paolo,

Quoted 54 lines · Paolo Abeni
Quoted 44 lines · Matthieu Baerts

Hi Paolo,

Thank you for the patch!

It looks good to me but I just have some questions if you don't mind, just to be sure I understood the modifications.

Quoted 36 lines · Paolo Abeni

2 separate fixes needed for the mentioned patch

On some exceptional scenarios (e.g. listener socket disconnecting while the subflow is created) a newly created MPC is deleted by the TCP stack via inet_child_forget(). When that happen we leak the subflow context and the msk.

Address the issue explicitly detecting such scenario and forcing a full msk shutdown.

mptcp_subflow_drop_ctx() does not remove the subflow from the msk list, racing msk lookup done by the NL PM could fetch the msk just before deletion, and try to free again the subflow.

Just explicitly remove the subflow from said list.

---
 net/mptcp/protocol.c | 13 +++++++++++--
 net/mptcp/subflow.c  |  1 +
 2 files changed, 12 insertions(+), 2 deletions(-)
net/mptcp/protocol.c +3 -2
index 0e4961e87b48..76c1814c0b19 100644
@@ -2378,9 +2378,18 @@ static void __mptcp_close_ssk(struct sock *sk, struct sock *ssk,
 	 * to deliver the msk to user-space.
 	 * Do nothing at the moment and take action at accept and/or listener
 	 * shutdown.
+	 * If instead such subflow has been destroyed, e.g. by inet_child_forget
+	 * do the kill
 	 */
-	if (msk->in_accept_queue && msk->first == ssk)
-		return;
+	if (msk->in_accept_queue && msk->first == ssk) {
Quoted 8 lines

net/mptcp/protocol.c static void __mptcp_close_ssk(struct sock *sk, struct sock *ssk, · line 2389

Quoted 5 lines
net/mptcp/protocol.c +5 -0
+		if (!sock_flag(ssk, SOCK_DEAD))
+			return;
+
+		/* ensure later check in mptcp_worker will dispose the msk */
+		sock_set_flag(sk, SOCK_DEAD);

Is it OK to mark the msk as dead and continue below? I guess yes because there will not be any data to re-inject anyway?

Quoted 3 lines

We don't process any data here (the msk is unaccepted at this point). Looks safe/sane to me. Overall this is the MPTCP equivalent of inet_child_forget()

Thank you for the confirmation!

Quoted 11 lines
Quoted 7 lines

net/mptcp/protocol.c static void __mptcp_close_ssk(struct sock *sk, struct sock *ssk, · line 2391

Quoted 2 lines
net/mptcp/protocol.c +2 -0
+		inet_csk(sk)->icsk_mtup.probe_timestamp = tcp_jiffies32 -
+							  TCP_TIMEWAIT_LEN -1;

Changing the timestamp looks like an ugly hack :)

But I guess it is needed not to wait for nothing and there is no clearer way? Or move this logic to the worker?

I can add another flag inside the msk. I did not do that here to keep the patch small, but no objection on my side to clean this path - the important thing is let syzkaller crunch this ;)

It might be cleaner yes. But this can be done later, better not to block the validation. I'm going to apply the two patches and if needed, I can modify the commit message/content later.

Cheers,
Matt

Signature · 2 lines
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net
author maintainer
#5 lore raw

PATCH 2/2 Squash-to: "mptcp: stops worker on unaccepted sockets at listener close"

In reply to Paolo Abeni

The mentioned patch carries a bad lockdep annotation. The msk socket lock is not nested, as that is the outermost one, held only by the user-space.

---
 net/mptcp/subflow.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
net/mptcp/subflow.c +1 -2
index 32e54b7fdbbc..c6ae5ddd3bb0 100644
@@ -1882,8 +1882,7 @@ void mptcp_subflow_queue_clean(struct sock *listener_sk, struct sock *listener_s
 		 */
 		mutex_release(&listener_sk->sk_lock.dep_map, _RET_IP_);
 		mptcp_cancel_work(sk);
-		mutex_acquire(&listener_sk->sk_lock.dep_map,
-			      SINGLE_DEPTH_NESTING, 0, _RET_IP_);
+		mutex_acquire(&listener_sk->sk_lock.dep_map, 0, 0, _RET_IP_);
 
 		sock_put(sk);
 	}
Signature · 1 line
2.39.2
#6 lore raw

Re: Squash-to: "mptcp: stops worker on unaccepted sockets at listener close": Tests Results

Hi Paolo,

Thank you for your modifications, that's great!

Our CI did some validations and here is its report:

- KVM Validation: normal (except selftest_mptcp_join):
- Success! ✅:
- Task: https://cirrus-ci.com/task/6146554927513600
- Summary: https://api.cirrus-ci.com/v1/artifact/task/6146554927513600/summary/summary.txt

- KVM Validation: normal (only selftest_mptcp_join):
- Unstable: 1 failed test(s): selftest_mptcp_join 🔴:
- Task: https://cirrus-ci.com/task/5583604974092288
- Summary: https://api.cirrus-ci.com/v1/artifact/task/5583604974092288/summary/summary.txt

- KVM Validation: debug (only selftest_mptcp_join):
- Success! ✅:
- Task: https://cirrus-ci.com/task/4563258183516160
- Summary: https://api.cirrus-ci.com/v1/artifact/task/4563258183516160/summary/summary.txt

- KVM Validation: debug (except selftest_mptcp_join):
- Success! ✅:
- Task: https://cirrus-ci.com/task/6709504880934912
- Summary: https://api.cirrus-ci.com/v1/artifact/task/6709504880934912/summary/summary.txt

Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/db821117c754

If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:

    $ cd [kernel source code]
    $ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
        --pull always mptcp/mptcp-upstream-virtme-docker:latest \
        auto-debug

For more details:

https://github.com/multipath-tcp/mptcp-upstream-virtme-docker

Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (Tessares)

#7 lore raw
In reply to Paolo Abeni

Hi Paolo,

Quoted 4 lines · Paolo Abeni

Hopefully addressing for good issues/371

There are actually several different problems that pop-up in random order with the repro from such issue.

Thank you for the fixes! They look good to me!

New patches for t/upstream-net and t/upstream:
- 3ac6e434b228: "squashed" patch 1/2 in "mptcp: fix accept vs worker race"
- 8e2c8f661424: "squashed" patch 2/2 in "mptcp: stops worker on unaccepted sockets at listener close"
- Results: 5b4478bd1afd..e8ce70ce4423 (export-net)
- Results: a73afdaa7080..2d58f6944092 (export)

Tests are now in progress:

https://cirrus-ci.com/github/multipath-tcp/mptcp_net-next/export-net/20230406T150700
https://cirrus-ci.com/github/multipath-tcp/mptcp_net-next/export/20230406T150700

Cheers,
Matt

Quoted 1 line

@Christoph: could you please have a spin in your testbed?

@Christoph: you can now use the export branch ;-)

Cheers,
Matt

Signature · 2 lines
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net