Skip to contentSkip to conversation

BPF packet scheduler updates part 3

PATCH In review Geliang Tang 4 participants 15 messages Aug 3, 2023 – Aug 18, 2023

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

v3 cover BPF packet scheduler updates part 3

v3:
- define tcp_rtx_and_write_queues_empty() and sk_stream_memory_free() in
BPF context.

v2:
- use __always_inline instead of inline.

Some cleanups.

Geliang Tang (6):
Squash to "mptcp: add sched_data helpers" Squash to "bpf: Add bpf_mptcp_sched_kfunc_set" Squash to "selftests/bpf: add two mptcp netns helpers" Squash to "selftests/bpf: Add bpf_rr scheduler" Squash to "bpf: Export more bpf_burst related functions" Squash to "selftests/bpf: Add bpf_burst scheduler"

 net/mptcp/bpf.c                               | 19 ++++----
 net/mptcp/protocol.h                          |  6 ---
 net/mptcp/sched.c                             |  6 +++
 tools/testing/selftests/bpf/bpf_tcp_helpers.h |  3 +-
 .../testing/selftests/bpf/prog_tests/mptcp.c  |  1 -
 .../selftests/bpf/progs/mptcp_bpf_burst.c     | 43 +++++++++++++++----
 6 files changed, 52 insertions(+), 26 deletions(-)
Signature · 1 line
2.35.3
#1 lore raw

PATCH v3 1/6 Squash to "mptcp: add sched_data helpers"

Add "ignore -Wmissing-prototypes" for mptcp_sched_data_set_contexts() and mptcp_subflow_ctx_by_pos(). Then the declarations of these functions in protocol.h can be dropped.

---
 net/mptcp/protocol.h | 4 ----
 net/mptcp/sched.c    | 6 ++++++
 2 files changed, 6 insertions(+), 4 deletions(-)
net/mptcp/protocol.h +0 -4
index eb0dc1027bab..ba6a63e6d39c 100644
@@ -671,10 +671,6 @@ int mptcp_init_sched(struct mptcp_sock *msk,
 void mptcp_release_sched(struct mptcp_sock *msk);
 void mptcp_subflow_set_scheduled(struct mptcp_subflow_context *subflow,
 				 bool scheduled);
-void mptcp_sched_data_set_contexts(const struct mptcp_sock *msk,
-				   struct mptcp_sched_data *data);
-struct mptcp_subflow_context *
-mptcp_subflow_ctx_by_pos(const struct mptcp_sched_data *data, unsigned int pos);
 struct sock *mptcp_subflow_get_send(struct mptcp_sock *msk);
 struct sock *mptcp_subflow_get_retrans(struct mptcp_sock *msk);
 int mptcp_sched_get_send(struct mptcp_sock *msk);
net/mptcp/sched.c +6 -0
index a80cf0481edf..5d78efc9c96c 100644
@@ -127,6 +127,10 @@ void mptcp_subflow_set_scheduled(struct mptcp_subflow_context *subflow,
 	WRITE_ONCE(subflow->scheduled, scheduled);
 }
 
+__diag_push();
+__diag_ignore_all("-Wmissing-prototypes",
+		  "kfuncs which will be used in BPF programs");
+
 void mptcp_sched_data_set_contexts(const struct mptcp_sock *msk,
 				   struct mptcp_sched_data *data)
 {
@@ -155,6 +159,8 @@ mptcp_subflow_ctx_by_pos(const struct mptcp_sched_data *data, unsigned int pos)
 	return data->contexts[pos];
 }
 
+__diag_pop();
+
 int mptcp_sched_get_send(struct mptcp_sock *msk)
 {
 	struct mptcp_subflow_context *subflow;
Signature · 1 line
2.35.3
#2 lore raw

Re: [PATCH mptcp-next v3 1/6] Squash to "mptcp: add sched_data helpers"

Hi Geliang,

Quoted 3 lines · Geliang Tang

Add "ignore -Wmissing-prototypes" for mptcp_sched_data_set_contexts() and mptcp_subflow_ctx_by_pos(). Then the declarations of these functions in protocol.h can be dropped.

I just had to revert this patch because it was causing issues reported by the CI when using 'sparce':

Quoted 3 lines
$ touch net/mptcp/sched.c && make C=1 net/mptcp/sched.o
net/mptcp/sched.c:203:6: warning: symbol 'mptcp_sched_data_set_contexts' was not declared. Should it be static?
net/mptcp/sched.c:223:30: warning: symbol 'mptcp_subflow_ctx_by_pos' was not declared. Should it be static?

Would it not be better to move these two functions (mptcp_sched_data_set_contexts - mptcp_subflow_ctx_by_pos) to net/mptcp/bpf.c where they are "used" with "BTF_ID_FLAGS(func, <name>)"?

Also, when I look at other kfunc in the kernel (e.g. bpf_skb_set_fou_encap()), I see that they are also always declared where they are being "used" (BTF_ID_FLAGS(func, <name>)) but also they are declared with "__bpf_kfunc" and the function name is prefixed with "bpf_" (which makes sense if they are BPF specific). Should we not do the same here with these 2 functions + bpf_mptcp_subflow_queues_empty() (only "__bpf_kfunc" is missing for this one, it already has the prefix and is declared where it is "used")?

About the revert:

New patches for t/upstream:
- 1701483e1326: Revert "Squash to "mptcp: add sched_data helpers""
- Results: ac006cd93f91..d30c287ad522 (export)

Tests are now in progress:

https://cirrus-ci.com/github/multipath-tcp/mptcp_net-next/export/20230816T103753

Cheers,
Matt

Signature · 2 lines
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net
#3 lore raw

PATCH v3 2/6 Squash to "bpf: Add bpf_mptcp_sched_kfunc_set"

In reply to Geliang Tang

Rename bpf_mptcp_sched_kfunc_init to bpf_mptcp_kfunc_init. Since some kfuncs related "mptcpify" will init in this function too.

---
 net/mptcp/bpf.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
net/mptcp/bpf.c +2 -2
index f6f9baa822ef..2e1a228a2ab4 100644
@@ -172,12 +172,12 @@ static const struct btf_kfunc_id_set bpf_mptcp_sched_kfunc_set = {
 	.set	= &bpf_mptcp_sched_kfunc_ids,
 };
 
-static int __init bpf_mptcp_sched_kfunc_init(void)
+static int __init bpf_mptcp_kfunc_init(void)
 {
 	return register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
 					 &bpf_mptcp_sched_kfunc_set);
 }
-late_initcall(bpf_mptcp_sched_kfunc_init);
+late_initcall(bpf_mptcp_kfunc_init);
 #endif /* CONFIG_BPF_JIT */
 
 struct mptcp_sock *bpf_mptcp_sock_from_subflow(struct sock *sk)
Signature · 1 line
2.35.3
#4 lore raw

PATCH v3 3/6 Squash to "selftests/bpf: add two mptcp netns helpers"

In reply to Geliang Tang

Just drop a black line.

---
 tools/testing/selftests/bpf/prog_tests/mptcp.c | 1 -
 1 file changed, 1 deletion(-)
tools/testing/selftests/bpf/prog_tests/mptcp.c +0 -1
index ad6b5e889d9d..09770aa31f4a 100644
@@ -198,7 +198,6 @@ static void test_base(void)
 
 fail:
 	cleanup_netns(nstoken);
-
 	close(cgroup_fd);
 }
 
Signature · 1 line
2.35.3
#5 lore raw

PATCH v3 4/6 Squash to "selftests/bpf: Add bpf_rr scheduler"

In reply to Geliang Tang

Use __always_inline instead of inline.

---
 tools/testing/selftests/bpf/bpf_tcp_helpers.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/bpf/bpf_tcp_helpers.h +1 -1
index 5763bd27f56d..ff4dd13fd8c6 100644
@@ -274,7 +274,7 @@ extern void mptcp_sched_data_set_contexts(const struct mptcp_sock *msk,
 					  struct mptcp_sched_data *data) __ksym;
 extern struct mptcp_subflow_context *
 mptcp_subflow_ctx_by_pos(const struct mptcp_sched_data *data, unsigned int pos) __ksym;
-static inline struct sock *
+static __always_inline struct sock *
 mptcp_subflow_tcp_sock(const struct mptcp_subflow_context *subflow)
 {
 	return subflow->tcp_sock;
Signature · 1 line
2.35.3
#6 lore raw

PATCH v3 5/6 Squash to "bpf: Export more bpf_burst related functions"

In reply to Geliang Tang

Add "ignore -Wmissing-prototypes" for bpf_mptcp_subflow_queues_empty(), drop the declarations in protocol.h.

Drop bpf_mptcp_subflow_memory_free(), it will be defined in BPF context.

Update bpf_mptcp_subflow_queues_empty().

---
 net/mptcp/bpf.c      | 15 ++++++++-------
 net/mptcp/protocol.h |  2 --
 2 files changed, 8 insertions(+), 9 deletions(-)
net/mptcp/bpf.c +8 -7
index 2e1a228a2ab4..a48ca9ea55cd 100644
@@ -145,16 +145,17 @@ struct bpf_struct_ops bpf_mptcp_sched_ops = {
 	.name		= "mptcp_sched_ops",
 };
 
-bool bpf_mptcp_subflow_memory_free(const struct sock *sk)
-{
-	return sk_stream_memory_free(sk);
-}
+__diag_push();
+__diag_ignore_all("-Wmissing-prototypes",
+		  "kfuncs which will be used in BPF programs");
 
-bool bpf_mptcp_subflow_queues_empty(const struct sock *sk)
+bool bpf_mptcp_subflow_queues_empty(struct sock *sk)
 {
-	return tcp_rtx_and_write_queues_empty(sk);
+	return tcp_rtx_queue_empty(sk);
 }
 
+__diag_pop();
+
 BTF_SET8_START(bpf_mptcp_sched_kfunc_ids)
 BTF_ID_FLAGS(func, mptcp_subflow_set_scheduled)
 BTF_ID_FLAGS(func, mptcp_sched_data_set_contexts)
@@ -162,7 +163,7 @@ BTF_ID_FLAGS(func, mptcp_subflow_ctx_by_pos)
 BTF_ID_FLAGS(func, mptcp_subflow_active)
 BTF_ID_FLAGS(func, mptcp_set_timeout)
 BTF_ID_FLAGS(func, mptcp_wnd_end)
-BTF_ID_FLAGS(func, bpf_mptcp_subflow_memory_free)
+BTF_ID_FLAGS(func, tcp_stream_memory_free)
 BTF_ID_FLAGS(func, bpf_mptcp_subflow_queues_empty)
 BTF_ID_FLAGS(func, mptcp_pm_subflow_chk_stale)
 BTF_SET8_END(bpf_mptcp_sched_kfunc_ids)
net/mptcp/protocol.h +0 -2
index ba6a63e6d39c..4572f01b978d 100644
@@ -642,8 +642,6 @@ void mptcp_subflow_queue_clean(struct sock *sk, struct sock *ssk);
 void mptcp_sock_graft(struct sock *sk, struct socket *parent);
 u64 mptcp_wnd_end(const struct mptcp_sock *msk);
 void mptcp_set_timeout(struct sock *sk);
-bool bpf_mptcp_subflow_memory_free(const struct sock *sk);
-bool bpf_mptcp_subflow_queues_empty(const struct sock *sk);
 struct sock *__mptcp_nmpc_sk(struct mptcp_sock *msk);
 bool __mptcp_close(struct sock *sk, long timeout);
 void mptcp_cancel_work(struct sock *sk);
Signature · 1 line
2.35.3
#7 lore raw

Re: [PATCH mptcp-next v3 5/6] Squash to "bpf: Export more bpf_burst related functions"

Hi Geliang,

Quoted 6 lines · Geliang Tang

Add "ignore -Wmissing-prototypes" for bpf_mptcp_subflow_queues_empty(), drop the declarations in protocol.h.

Drop bpf_mptcp_subflow_memory_free(), it will be defined in BPF context.

Update bpf_mptcp_subflow_queues_empty().

I also had to partially revert this commit.

Maybe "__bpf_kfunc" would avoid such issues reported by Sparse?

  #define __bpf_kfunc __used noinline

Do you mind looking at that please?

New patches for t/upstream:
- 11ace9718934: mptcp: partially revert "Squash to "bpf: Export more bpf_burst related functions""
- Results: d30c287ad522..dd7fb06cb618 (export)

Tests are now in progress:

https://cirrus-ci.com/github/multipath-tcp/mptcp_net-next/export/20230816T115317

Cheers,
Matt

Signature · 2 lines
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net
#8 lore raw

Re: [PATCH mptcp-next v3 5/6] Squash to "bpf: Export more bpf_burst related functions"

Quoted 10 lines · Matthieu Baerts

Hi Geliang,

Quoted 6 lines · Geliang Tang

Add "ignore -Wmissing-prototypes" for bpf_mptcp_subflow_queues_empty(), drop the declarations in protocol.h.

Drop bpf_mptcp_subflow_memory_free(), it will be defined in BPF context.

Update bpf_mptcp_subflow_queues_empty().

Quoted 7 lines

I also had to partially revert this commit.

Maybe "__bpf_kfunc" would avoid such issues reported by Sparse?

  #define __bpf_kfunc __used noinline

Do you mind looking at that please?

Hi Matt,

"__bpf_kfunc" doesn't work. I have to ask BPF maintainers what to do for this and hope to receive their response.

Thanks,
-Geliang

Quoted 15 lines

New patches for t/upstream:
- 11ace9718934: mptcp: partially revert "Squash to "bpf: Export more bpf_burst related functions""
- Results: d30c287ad522..dd7fb06cb618 (export)

Tests are now in progress:

https://cirrus-ci.com/github/multipath-tcp/mptcp_net-next/export/20230816T115317

Cheers,
Matt

Signature · 2 lines
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net
#9 lore raw

Re: [PATCH mptcp-next v3 5/6] Squash to "bpf: Export more bpf_burst related functions"

Hi Geliang,

Quoted 15 lines · Geliang Tang
Quoted 14 lines · Matthieu Baerts

Hi Geliang,

Quoted 6 lines · Geliang Tang

Add "ignore -Wmissing-prototypes" for bpf_mptcp_subflow_queues_empty(), drop the declarations in protocol.h.

Drop bpf_mptcp_subflow_memory_free(), it will be defined in BPF context.

Update bpf_mptcp_subflow_queues_empty().

I also had to partially revert this commit.

Maybe "__bpf_kfunc" would avoid such issues reported by Sparse?

Quoted 8 lines
Quoted 3 lines
  #define __bpf_kfunc __used noinline

Do you mind looking at that please?

Hi Matt,

"__bpf_kfunc" doesn't work. I have to ask BPF maintainers what to do for this and hope to receive their response.

Thank you for having asked!

Good idea to have asked on the previous patch, I didn't see that netdev's CI complained about that on this one:

https://patchwork.kernel.org/project/netdevbpf/patch/ac84be00f97072a46f8a72b4e2be46cbb7fa5053.1692147782.git.geliang.tang@suse.com/

We can see a similar warning for update_socket_protocol() in netdev/build_allmodconfig_warn and netdev/build_32bit results.

Cheers,
Matt

Signature · 2 lines
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net
#10 lore raw

PATCH v3 6/6 Squash to "selftests/bpf: Add bpf_burst scheduler"

In reply to Geliang Tang

Use __always_inline instead of inline.

Define tcp_rtx_and_write_queues_empty() and sk_stream_memory_free() in BPF context.

---
 tools/testing/selftests/bpf/bpf_tcp_helpers.h |  1 +
 .../selftests/bpf/progs/mptcp_bpf_burst.c     | 43 +++++++++++++++----
 2 files changed, 35 insertions(+), 9 deletions(-)
tools/testing/selftests/bpf/bpf_tcp_helpers.h +1 -0
index ff4dd13fd8c6..776c54948a4a 100644
@@ -36,6 +36,7 @@ enum sk_pacing {
 struct sock {
 	struct sock_common	__sk_common;
 #define sk_state		__sk_common.skc_state
+	int			sk_sndbuf;
 	int			sk_wmem_queued;
 	unsigned long		sk_pacing_rate;
 	__u32			sk_pacing_status; /* see enum sk_pacing */
tools/testing/selftests/bpf/progs/mptcp_bpf_burst.c +34 -9

70 lines not on this page · raw message

Signature · 1 line
2.35.3
#11 lore raw

Re: Squash to "selftests/bpf: Add bpf_burst scheduler": Build Failure

Hi Geliang,

Thank you for your modifications, that's great!

But sadly, our CI spotted some issues with it when trying to build it.

You can find more details there:

https://patchwork.kernel.org/project/mptcp/patch/4f65c2b44b82e8682abcf5c5b91dceced0f07480.1691036765.git.geliang.tang@suse.com/
https://github.com/multipath-tcp/mptcp_net-next/actions/runs/5746700485

Status: failure
Initiator: MPTCPimporter
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/1f357c615e99

Feel free to reply to this email if you cannot access logs, if you need some support to fix the error, if this doesn't seem to be caused by your modifications or if the error is a false positive one.

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (Tessares)

#12 lore raw

Re: Squash to "selftests/bpf: Add bpf_burst scheduler": Tests Results

In reply to Geliang Tang

Hi Geliang,

Thank you for your modifications, that's great!

Our CI did some validations and here is its report:

- KVM Validation: normal (except selftest_mptcp_join):
- Success! ✅:
- Task: https://cirrus-ci.com/task/5651357470818304
- Summary: https://api.cirrus-ci.com/v1/artifact/task/5651357470818304/summary/summary.txt

- KVM Validation: debug (except selftest_mptcp_join):
- Success! ✅:
- Task: https://cirrus-ci.com/task/6214307424239616
- Summary: https://api.cirrus-ci.com/v1/artifact/task/6214307424239616/summary/summary.txt

- KVM Validation: normal (only selftest_mptcp_join):
- Success! ✅:
- Task: https://cirrus-ci.com/task/5088407517396992
- Summary: https://api.cirrus-ci.com/v1/artifact/task/5088407517396992/summary/summary.txt

- KVM Validation: debug (only selftest_mptcp_join):
- Success! ✅:
- Task: https://cirrus-ci.com/task/4806932540686336
- Summary: https://api.cirrus-ci.com/v1/artifact/task/4806932540686336/summary/summary.txt

Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/1f357c615e99

If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:

    $ cd [kernel source code]
    $ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
        --pull always mptcp/mptcp-upstream-virtme-docker:latest \
        auto-debug

For more details:

https://github.com/multipath-tcp/mptcp-upstream-virtme-docker

Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (Tessares)

maintainer
#13 lore raw
In reply to Geliang Tang
Quoted 3 lines · Geliang Tang

v3:
- define tcp_rtx_and_write_queues_empty() and sk_stream_memory_free() in BPF context.

v3 looks good to squash, thanks Geliang.

- Mat

Quoted 27 lines

v2:
- use __always_inline instead of inline.

Some cleanups.

Geliang Tang (6):
Squash to "mptcp: add sched_data helpers" Squash to "bpf: Add bpf_mptcp_sched_kfunc_set" Squash to "selftests/bpf: add two mptcp netns helpers" Squash to "selftests/bpf: Add bpf_rr scheduler" Squash to "bpf: Export more bpf_burst related functions" Squash to "selftests/bpf: Add bpf_burst scheduler"

net/mptcp/bpf.c                               | 19 ++++----
net/mptcp/protocol.h                          |  6 ---
net/mptcp/sched.c                             |  6 +++
tools/testing/selftests/bpf/bpf_tcp_helpers.h |  3 +-
.../testing/selftests/bpf/prog_tests/mptcp.c  |  1 -
.../selftests/bpf/progs/mptcp_bpf_burst.c     | 43 +++++++++++++++----
6 files changed, 52 insertions(+), 26 deletions(-)
Signature · 1 line
2.35.3
#14 lore raw
In reply to Geliang Tang

Hi Geliang, Mat,

Quoted 7 lines · Geliang Tang

Geliang Tang (6):
Squash to "mptcp: add sched_data helpers" Squash to "bpf: Add bpf_mptcp_sched_kfunc_set" Squash to "selftests/bpf: add two mptcp netns helpers" Squash to "selftests/bpf: Add bpf_rr scheduler" Squash to "bpf: Export more bpf_burst related functions" Squash to "selftests/bpf: Add bpf_burst scheduler"

Thank you for the patches and the reviews!

Now in our tree:

New patches for t/upstream:
- a88f3f498b2f: "squashed" patch 1/6 (with conflicts) in "mptcp: add sched_data helpers"
- 5d8795774463: "squashed" patch 2/6 in "bpf: Add bpf_mptcp_sched_kfunc_set"
- be959ff059e4: "squashed" patch 3/6 in "selftests/bpf: add two mptcp netns helpers"
- 1736fcc031f4: "squashed" patch 4/6 in "selftests/bpf: Add bpf_rr scheduler"
- 574847a8b558: "squashed" patch 5/6 in "bpf: Export more bpf_burst related functions"
- 206993ff5b4f: "squashed" patch 6/6 in "selftests/bpf: Add bpf_burst scheduler"
- Results: 2d42b08369ef..03fd7c30320a (export)

Tests are now in progress:

https://cirrus-ci.com/github/multipath-tcp/mptcp_net-next/export/20230811T205838

Cheers,
Matt

Signature · 2 lines
Tessares | Belgium | Hybrid Access Solutions
www.tessares.net