Skip to contentSkip to conversation

Squash to "Add mptcp_subflow bpf_iter support"

PATCH In review Geliang Tang 4 participants 13 messages Jan 24, 2025 – Feb 20, 2025

Subsystems (5)
lore
Share & tools
lore

Geliang Tang

author reviewer
#0 lore raw

cover Squash to "Add mptcp_subflow bpf_iter support"

From: Geliang Tang <[email protected]>

Address Martin's suggestions for v2.

Geliang Tang (3):
Squash to "bpf: Add mptcp_subflow bpf_iter"
Revert "bpf: Acquire and release mptcp socket" Squash to "selftests/bpf: Add mptcp_subflow bpf_iter subtest"

 net/mptcp/bpf.c                               | 34 +++++--------------
 .../testing/selftests/bpf/bpf_experimental.h  |  2 +-
 tools/testing/selftests/bpf/progs/mptcp_bpf.h |  3 --
 .../selftests/bpf/progs/mptcp_bpf_iters.c     |  6 +---
 4 files changed, 11 insertions(+), 34 deletions(-)
Signature · 1 line
2.43.0
author reviewer
#1 lore raw

PATCH 1/3 Squash to "bpf: Add mptcp_subflow bpf_iter"

From: Geliang Tang <[email protected]>

Drop the NULL check as Martin suggested.

Use the "struct sock *sk" instead of "struct mptcp-sock *msk" as the argument in the bpf_iter_mptcp_subflow_new as Martin suggested.

Use msk_owned_by_me().

---
 net/mptcp/bpf.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
net/mptcp/bpf.c +9 -6
index 923895322b2c..def4de34666d 100644
@@ -234,24 +234,27 @@ bpf_mptcp_subflow_ctx(const struct sock *sk)
 
 __bpf_kfunc static int
 bpf_iter_mptcp_subflow_new(struct bpf_iter_mptcp_subflow *it,
-			   struct mptcp_sock *msk)
+			   struct sock *sk)
 {
 	struct bpf_iter_mptcp_subflow_kern *kit = (void *)it;
-	struct sock *sk = (struct sock *)msk;
+	struct mptcp_sock *msk;
 
 	BUILD_BUG_ON(sizeof(struct bpf_iter_mptcp_subflow_kern) >
 		     sizeof(struct bpf_iter_mptcp_subflow));
 	BUILD_BUG_ON(__alignof__(struct bpf_iter_mptcp_subflow_kern) !=
 		     __alignof__(struct bpf_iter_mptcp_subflow));
 
-	kit->msk = msk;
-	if (!msk)
+	if (unlikely(!sk || !sk_fullsock(sk)))
 		return -EINVAL;
 
-	if (!sock_owned_by_user_nocheck(sk) &&
-	    !spin_is_locked(&sk->sk_lock.slock))
+	if (sk->sk_protocol != IPPROTO_MPTCP)
 		return -EINVAL;
 
+	msk = mptcp_sk(sk);
+	kit->msk = msk;
+
+	msk_owned_by_me(msk);
+
 	kit->pos = &msk->conn_list;
 	return 0;
 }
Signature · 1 line
2.43.0
maintainer
#2 lore raw

Re: [PATCH mptcp-next 1/3] Squash to "bpf: Add mptcp_subflow bpf_iter"

Hi Geliang,

(+cc Mat who reviewed the previous versions)

Quoted 8 lines · Geliang Tang

From: Geliang Tang <[email protected]>

Drop the NULL check as Martin suggested.

Use the "struct sock *sk" instead of "struct mptcp-sock *msk" as the argument in the bpf_iter_mptcp_subflow_new as Martin suggested.

Use msk_owned_by_me().

Regarding this, do you mind checking my question on the thread initially sent to the BPF mailing please?

Quoted 5 lines
Quoted 4 lines

This set is only showing the cg sockopt bpf prog but missing the major struct_ops piece. It is hard to comment. I assumed the lock situation is the same for the struct_ops where the lock will be held before calling the struct_ops prog?

Quoted 7 lines

Can we restrict the use of this helper only for some specific callbacks? Or for only some specific struct_ops?

Last time I looked, I thought that you could only restrict it like that:
"this helper can be used by (any) struct_ops". But then does it mean this helper could be called from a TCP BPF CC ops or another unrelated one for example? How can we avoid that?

Because with the suggested modification, there is no more checks regarding the lock situation: it is probably fine now with [gs]etsockopt(), but what about later with struct_ops? Can we restrict this kfunc to be used only in some contexts? e.g. a part or all the MPTCP struct_ops, but not the other ones?

Cheers,
Matt

Signature · 1 line
Sponsored by the NGI0 Core fund.
author reviewer
#3 lore raw

Re: [PATCH mptcp-next 1/3] Squash to "bpf: Add mptcp_subflow bpf_iter"

Hi Matt,

Thanks for your review.

Quoted 38 lines · Matthieu Baerts

Hi Geliang,

(+cc Mat who reviewed the previous versions)

Quoted 9 lines · Geliang Tang

From: Geliang Tang <[email protected]>

Drop the NULL check as Martin suggested.

Use the "struct sock *sk" instead of "struct mptcp-sock *msk" as the
argument in the bpf_iter_mptcp_subflow_new as Martin suggested.

Use msk_owned_by_me().

Regarding this, do you mind checking my question on the thread initially
sent to the BPF mailing please?

Quoted 12 lines
Quoted 7 lines

This set is only showing the cg sockopt bpf prog but missing the major
struct_ops piece. It is hard to comment. I assumed the lock situation is
the same for the struct_ops where the lock will be held before calling
the struct_ops prog?

Can we restrict the use of this helper only for some specific callbacks?
Or for only some specific struct_ops?

Quoted 7 lines

Last time I looked, I thought that you could only restrict it like that:
"this helper can be used by (any) struct_ops". But then does it mean
this helper could be called from a TCP BPF CC ops or another unrelated
one for example? How can we avoid that?

Quoted 6 lines

Because with the suggested modification, there is no more checks regarding the lock situation: it is probably fine now with [gs]etsockopt(), but what about later with struct_ops? Can we restrict
this kfunc to be used only in some contexts? e.g. a part or all the MPTCP struct_ops, but not the other ones?

We use register_btf_kfunc_id_set() to allow these kfuncs to be used in struct_ops:

register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,

                          &bpf_mptcp_common_kfunc_set);

And we also check if the protocol of the socket is MPTCP in bpf_iter_mptcp_subflow_new(). So this mptcp_subflow bpf_iter is restricted to be used for MPTCP. Other struct_ops likes a TCP BPF CC ops can't use it.

Do you think this restriction is enough?

Another way to restrict kfuncs is to set get_func_proto of struct bpf_verifier_ops. But this way has not been applied to other BPF iters.

Thanks,
-Geliang

Quoted 3 lines

Cheers,
Matt

maintainer
#4 lore raw

Re: [PATCH mptcp-next 1/3] Squash to "bpf: Add mptcp_subflow bpf_iter"

Hi Geliang,

Quoted 50 lines · Geliang Tang

Hi Matt,

Thanks for your review.

Quoted 38 lines · Matthieu Baerts

Hi Geliang,

(+cc Mat who reviewed the previous versions)

Quoted 9 lines · Geliang Tang

From: Geliang Tang <[email protected]>

Drop the NULL check as Martin suggested.

Use the "struct sock *sk" instead of "struct mptcp-sock *msk" as the
argument in the bpf_iter_mptcp_subflow_new as Martin suggested.

Use msk_owned_by_me().

Regarding this, do you mind checking my question on the thread initially
sent to the BPF mailing please?

Quoted 12 lines
Quoted 7 lines

This set is only showing the cg sockopt bpf prog but missing the major
struct_ops piece. It is hard to comment. I assumed the lock situation is
the same for the struct_ops where the lock will be held before calling
the struct_ops prog?

Can we restrict the use of this helper only for some specific callbacks?
Or for only some specific struct_ops?

Quoted 7 lines

Last time I looked, I thought that you could only restrict it like that:
"this helper can be used by (any) struct_ops". But then does it mean
this helper could be called from a TCP BPF CC ops or another unrelated
one for example? How can we avoid that?

Quoted 6 lines

Because with the suggested modification, there is no more checks regarding the lock situation: it is probably fine now with [gs]etsockopt(), but what about later with struct_ops? Can we restrict
this kfunc to be used only in some contexts? e.g. a part or all the MPTCP struct_ops, but not the other ones?

Quoted 5 lines

We use register_btf_kfunc_id_set() to allow these kfuncs to be used in struct_ops:

register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,

                          &bpf_mptcp_common_kfunc_set);

OK so if I understand correctly, it means that the kfunc that are part of this 'mptcp_common' set can be used from any struct ops, right? Or in other words, the new MPTCP kfunc could be used from the TCP BPF CC struct ops for example, right?

If yes, it might be clearer to ask BPF maintainers how we should design the new kfunc that are supposed to be used only in the context of the new BPF packet scheduler, e.g. should we check who own the msk socket.

I can ask this question if you prefer.

Quoted 4 lines

And we also check if the protocol of the socket is MPTCP in bpf_iter_mptcp_subflow_new(). So this mptcp_subflow bpf_iter is restricted to be used for MPTCP. Other struct_ops likes a TCP BPF CC ops can't use it.

I see. For the bpf_iter case, these restrictions should be fine. Maybe not for the kfunc needed for the scheduler part.

Quoted 4 lines

Do you think this restriction is enough?

Another way to restrict kfuncs is to set get_func_proto of struct bpf_verifier_ops. But this way has not been applied to other BPF iters.

No, I guess for bpf_iter, we don't need that. Probably best to see them as a "generic thing" and have the previous checks we already have.

Cheers,
Matt

Signature · 1 line
Sponsored by the NGI0 Core fund.
maintainer
#5 lore raw

Re: [PATCH mptcp-next 1/3] Squash to "bpf: Add mptcp_subflow bpf_iter"

In reply to Geliang Tang
Quoted 44 lines · Geliang Tang

From: Geliang Tang <[email protected]>

Drop the NULL check as Martin suggested.

Use the "struct sock *sk" instead of "struct mptcp-sock *msk" as the argument in the bpf_iter_mptcp_subflow_new as Martin suggested.

Use msk_owned_by_me().

---
net/mptcp/bpf.c | 15 +++++++++------

1 file changed, 9 insertions(+), 6 deletions(-)
net/mptcp/bpf.c +0 -0
index 923895322b2c..def4de34666d 100644
@@ -234,24 +234,27 @@ bpf_mptcp_subflow_ctx(const struct sock *sk)
 

__bpf_kfunc static int bpf_iter_mptcp_subflow_new(struct bpf_iter_mptcp_subflow *it,

net/mptcp/bpf.c +1 -1
-			   struct mptcp_sock *msk)
+			   struct sock *sk)
{
	struct bpf_iter_mptcp_subflow_kern *kit = (void *)it;
net/mptcp/bpf.c +1 -1
-	struct sock *sk = (struct sock *)msk;
+	struct mptcp_sock *msk;
	BUILD_BUG_ON(sizeof(struct bpf_iter_mptcp_subflow_kern) >
		     sizeof(struct bpf_iter_mptcp_subflow));
	BUILD_BUG_ON(__alignof__(struct bpf_iter_mptcp_subflow_kern) !=
		     __alignof__(struct bpf_iter_mptcp_subflow));
net/mptcp/bpf.c +1 -2
-	kit->msk = msk;
-	if (!msk)
+	if (unlikely(!sk || !sk_fullsock(sk)))
		return -EINVAL;
net/mptcp/bpf.c +1 -2
-	if (!sock_owned_by_user_nocheck(sk) &&
-	    !spin_is_locked(&sk->sk_lock.slock))
+	if (sk->sk_protocol != IPPROTO_MPTCP)
		return -EINVAL;

net/mptcp/bpf.c bpf_mptcp_subflow_ctx(const struct sock *sk) · line 256

Quoted 4 lines
net/mptcp/bpf.c +4 -0
+	msk = mptcp_sk(sk);
+	kit->msk = msk;
+
+	msk_owned_by_me(msk);

Hi Geliang -

The convention is to check the owner before assigning the msk.

I also noticed in the reply to patch 5 that Martin mentioned adding a "bpf_mptcp_sock_from_sock check" to bpf_iter_mptcp_subflow_new (https://lore.kernel.org/netdev/[email protected]/)

- Mat

Quoted 9 lines

+

	kit->pos = &msk->conn_list;
	return 0;
}
Signature · 1 line
2.43.0
author reviewer
#6 lore raw

Re: [PATCH mptcp-next 1/3] Squash to "bpf: Add mptcp_subflow bpf_iter"

Hi Mat,

Thanks for your review.

Quoted 48 lines · Mat Martineau
Quoted 46 lines · Geliang Tang

From: Geliang Tang <[email protected]>

Drop the NULL check as Martin suggested.

Use the "struct sock *sk" instead of "struct mptcp-sock *msk" as the
argument in the bpf_iter_mptcp_subflow_new as Martin suggested.

Use msk_owned_by_me().

---
net/mptcp/bpf.c | 15 +++++++++------

1 file changed, 9 insertions(+), 6 deletions(-)
net/mptcp/bpf.c +0 -0
index 923895322b2c..def4de34666d 100644
@@ -234,24 +234,27 @@ bpf_mptcp_subflow_ctx(const struct sock *sk)
 

__bpf_kfunc static int bpf_iter_mptcp_subflow_new(struct bpf_iter_mptcp_subflow *it,

net/mptcp/bpf.c +1 -1
-			   struct mptcp_sock *msk)
+			   struct sock *sk)
{
	struct bpf_iter_mptcp_subflow_kern *kit = (void *)it;
net/mptcp/bpf.c +1 -1
-	struct sock *sk = (struct sock *)msk;
+	struct mptcp_sock *msk;
	BUILD_BUG_ON(sizeof(struct bpf_iter_mptcp_subflow_kern) >
		     sizeof(struct bpf_iter_mptcp_subflow));
	BUILD_BUG_ON(__alignof__(struct
bpf_iter_mptcp_subflow_kern) !=
		     __alignof__(struct bpf_iter_mptcp_subflow));
net/mptcp/bpf.c +1 -2
-	kit->msk = msk;
-	if (!msk)
+	if (unlikely(!sk || !sk_fullsock(sk)))
		return -EINVAL;
net/mptcp/bpf.c +1 -2
-	if (!sock_owned_by_user_nocheck(sk) &&
-	    !spin_is_locked(&sk->sk_lock.slock))
+	if (sk->sk_protocol != IPPROTO_MPTCP)
		return -EINVAL;
Quoted 8 lines

net/mptcp/bpf.c bpf_mptcp_subflow_ctx(const struct sock *sk) · line 256

Quoted 4 lines
net/mptcp/bpf.c +4 -0
+	msk = mptcp_sk(sk);
+	kit->msk = msk;
+
+	msk_owned_by_me(msk);

Hi Geliang -

The convention is to check the owner before assigning the msk.

Yes, indeed. Updated in v2.

Quoted 3 lines

I also noticed in the reply to patch 5 that Martin mentioned adding a "bpf_mptcp_sock_from_sock check" to bpf_iter_mptcp_subflow_new

bpf_mptcp_sock_from_sock should be dropped, so open-code it in bpf_iter_mptcp_subflow_new.

Thanks,
-Geliang

Quoted 16 lines

(
https://lore.kernel.org/netdev/9b373a23-c093-42d8-b4ae-99f2e62e7681@li
nux.dev/)

- Mat

Quoted 9 lines

+

	kit->pos = &msk->conn_list;
	return 0;
}
Signature · 1 line
2.43.0
author reviewer
#7 lore raw

PATCH 2/3 Revert "bpf: Acquire and release mptcp socket"

In reply to Geliang Tang

From: Geliang Tang <[email protected]>

Drop this patch as Martin suggested.

---
 net/mptcp/bpf.c | 19 -------------------
 1 file changed, 19 deletions(-)
net/mptcp/bpf.c +0 -19
index def4de34666d..27a8c2e9ca2b 100644
@@ -276,23 +276,6 @@ bpf_iter_mptcp_subflow_destroy(struct bpf_iter_mptcp_subflow *it)
 {
 }
 
-__bpf_kfunc static struct
-mptcp_sock *bpf_mptcp_sock_acquire(struct mptcp_sock *msk)
-{
-	struct sock *sk = (struct sock *)msk;
-
-	if (sk && refcount_inc_not_zero(&sk->sk_refcnt))
-		return msk;
-	return NULL;
-}
-
-__bpf_kfunc static void bpf_mptcp_sock_release(struct mptcp_sock *msk)
-{
-	struct sock *sk = (struct sock *)msk;
-
-	WARN_ON_ONCE(!sk || !refcount_dec_not_one(&sk->sk_refcnt));
-}
-
 __bpf_kfunc struct mptcp_subflow_context *
 bpf_mptcp_subflow_ctx_by_pos(const struct mptcp_sched_data *data, unsigned int pos)
 {
@@ -313,8 +296,6 @@ BTF_ID_FLAGS(func, bpf_mptcp_subflow_ctx, KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_iter_mptcp_subflow_new, KF_ITER_NEW | KF_TRUSTED_ARGS)
 BTF_ID_FLAGS(func, bpf_iter_mptcp_subflow_next, KF_ITER_NEXT | KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_iter_mptcp_subflow_destroy, KF_ITER_DESTROY)
-BTF_ID_FLAGS(func, bpf_mptcp_sock_acquire, KF_ACQUIRE | KF_RET_NULL)
-BTF_ID_FLAGS(func, bpf_mptcp_sock_release, KF_RELEASE)
 BTF_KFUNCS_END(bpf_mptcp_common_kfunc_ids)
 
 static const struct btf_kfunc_id_set bpf_mptcp_common_kfunc_set = {
Signature · 1 line
2.43.0
author reviewer
#8 lore raw

PATCH 3/3 Squash to "selftests/bpf: Add mptcp_subflow bpf_iter subtest"

In reply to Geliang Tang

From: Geliang Tang <[email protected]>

Drop bpf_mptcp_sock_acquire/release.

---
 tools/testing/selftests/bpf/bpf_experimental.h      | 2 +-
 tools/testing/selftests/bpf/progs/mptcp_bpf.h       | 3 ---
 tools/testing/selftests/bpf/progs/mptcp_bpf_iters.c | 6 +-----
 3 files changed, 2 insertions(+), 9 deletions(-)
tools/testing/selftests/bpf/bpf_experimental.h +1 -1
index 2ab3f0063c0f..6a96c56f0725 100644
@@ -577,7 +577,7 @@ extern void bpf_iter_css_destroy(struct bpf_iter_css *it) __weak __ksym;
 
 struct bpf_iter_mptcp_subflow;
 extern int bpf_iter_mptcp_subflow_new(struct bpf_iter_mptcp_subflow *it,
-				      struct mptcp_sock *msk) __weak __ksym;
+				      struct sock *sk) __weak __ksym;
 extern struct mptcp_subflow_context *
 bpf_iter_mptcp_subflow_next(struct bpf_iter_mptcp_subflow *it) __weak __ksym;
 extern void
tools/testing/selftests/bpf/progs/mptcp_bpf.h +0 -3
index b1f6e1fb467e..ede9111ee597 100644
@@ -43,9 +43,6 @@ mptcp_subflow_tcp_sock(const struct mptcp_subflow_context *subflow)
 }
 
 /* ksym */
-extern struct mptcp_sock *bpf_mptcp_sock_acquire(struct mptcp_sock *msk) __ksym;
-extern void bpf_mptcp_sock_release(struct mptcp_sock *msk) __ksym;
-
 extern struct mptcp_subflow_context *
 bpf_mptcp_subflow_ctx(const struct sock *sk) __ksym;
 extern struct sock *
tools/testing/selftests/bpf/progs/mptcp_bpf_iters.c +1 -5
index fd5691a4073b..6124c484efe3 100644
@@ -28,10 +28,7 @@ int iters_subflow(struct bpf_sockopt *ctx)
 	if (!msk || msk->pm.server_side || !msk->pm.subflows)
 		return 1;
 
-	msk = bpf_mptcp_sock_acquire(msk);
-	if (!msk)
-		return 1;
-	bpf_for_each(mptcp_subflow, subflow, msk) {
+	bpf_for_each(mptcp_subflow, subflow, (struct sock *)sk) {
 		/* Here MPTCP-specific packet scheduler kfunc can be called:
 		 * this test is not doing anything really useful, only to
 		 * verify the iteration works.
@@ -58,6 +55,5 @@ int iters_subflow(struct bpf_sockopt *ctx)
 	ids = local_ids;
 
 out:
-	bpf_mptcp_sock_release(msk);
 	return 1;
 }
Signature · 1 line
2.43.0
maintainer
#9 lore raw

Re: [PATCH mptcp-next 3/3] Squash to "selftests/bpf: Add mptcp_subflow bpf_iter subtest"

Quoted 3 lines · Geliang Tang

From: Geliang Tang <[email protected]>

Drop bpf_mptcp_sock_acquire/release.

It might be good to share a link to the WIP commits/patches to Martin (and us) to better understand how it is going to be used.

Maybe a link to a git repo containing commits from the series "use bpf_iter in bpf schedulers" where the squash-to commits have been squashed?

Cheers,
Matt

Signature · 1 line
Sponsored by the NGI0 Core fund.
author reviewer
#10 lore raw

Re: [PATCH mptcp-next 3/3] Squash to "selftests/bpf: Add mptcp_subflow bpf_iter subtest"

Quoted 11 lines · Matthieu Baerts
Quoted 3 lines · Geliang Tang

From: Geliang Tang <[email protected]>

Drop bpf_mptcp_sock_acquire/release.

It might be good to share a link to the WIP commits/patches to Martin (and us) to better understand how it is going to be used.

Maybe a link to a git repo containing commits from the series "use bpf_iter in bpf schedulers" where the squash-to commits have been squashed?

Sure, I'll do this.

I think it would be better to directly send these patches to BPF maintainers as RFC ones for more suggestions. Since the last time I sent the link to Martin, he didn't have time to read these patches.

Thanks,
-Geliang

Quoted 3 lines

Cheers,
Matt

maintainer
#11 lore raw

Re: [PATCH mptcp-next 3/3] Squash to "selftests/bpf: Add mptcp_subflow bpf_iter subtest"

Hi Geliang,

(sorry, it looks like I missed this reply)

Quoted 13 lines · Geliang Tang
Quoted 11 lines · Matthieu Baerts
Quoted 3 lines · Geliang Tang

From: Geliang Tang <[email protected]>

Drop bpf_mptcp_sock_acquire/release.

It might be good to share a link to the WIP commits/patches to Martin (and us) to better understand how it is going to be used.

Maybe a link to a git repo containing commits from the series "use bpf_iter in bpf schedulers" where the squash-to commits have been squashed?

Quoted 5 lines

Sure, I'll do this.

I think it would be better to directly send these patches to BPF maintainers as RFC ones for more suggestions. Since the last time I sent the link to Martin, he didn't have time to read these patches.

I think this time, it is OK with just a link, because Martin asked for it. Probably best to wait before sending these patches not to add more pressure to rework the scheduler API.

Cheers,
Matt

Signature · 1 line
Sponsored by the NGI0 Core fund.
#12 lore raw
In reply to Geliang Tang

Hi Geliang,

Thank you for your modifications, that's great!

Our CI did some validations and here is its report:

- KVM Validation: normal: Success! ✅
- KVM Validation: debug: Success! ✅
- KVM Validation: btf-normal (only bpftest_all): Success! ✅
- KVM Validation: btf-debug (only bpftest_all): Success! ✅
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/12948765422

Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/927f2c1cee5c
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=928101

If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:

    $ cd [kernel source code]
    $ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
        --pull always mptcp/mptcp-upstream-virtme-docker:latest \
        auto-normal

For more details:

https://github.com/multipath-tcp/mptcp-upstream-virtme-docker

Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)