BPF redundant scheduler
Subsystems (5)
v12 cover BPF redundant scheduler
v12:
- fix WARN_ON_ONCE(reuse_skb) and WARN_ON_ONCE(!msk->recovery) errors in kernel logs.
v11:
- address to Mat's comments in v10.
- rebase to export/20220908T063452
v10:
- send multiple dfrags in __mptcp_push_pending().
v9:
- drop the extra *err paramenter of mptcp_sched_get_send() as Florian suggested.
v8:
- update __mptcp_push_pending(), send the same data on each subflow.
- update __mptcp_retrans, track the max sent data.
= add a new patch.
v7:
- drop redundant flag in v6
- drop __mptcp_subflows_push_pending in v6
- update redundant subflows support in __mptcp_push_pending
- update redundant subflows support in __mptcp_retrans
v6:
- Add redundant flag for struct mptcp_sched_ops.
- add a dedicated function __mptcp_subflows_push_pending() to deal with redundat subflows push pending.
v5:
- address to Paolo's comment, keep the optimization tomptcp_subflow_get_send() for the non eBPF case.
- merge mptcp_sched_get_send() and __mptcp_sched_get_send() in v4 into one.
- depends on "cleanups for bpf sched selftests".
v4:
- small cleanups in patch 1, 2.
- add TODO in patch 3.
- rebase patch 5 on 'cleanups for bpf sched selftests'.
v3:
- use new API.
- fix the link failure tests issue mentioned in ("https://patchwork.kernel.org/project/mptcp/cover/[email protected]/").
v2:
- add MPTCP_SUBFLOWS_MAX limit to avoid infinite loops when the scheduler always sets call_again to true.
- track the largest copied amount.
- deal with __mptcp_subflow_push_pending() and the retransmit loop.
- depends on "BPF round-robin scheduler" v14.
v1:
Implements the redundant BPF MPTCP scheduler, which sends all packets redundantly on all available subflows.
Geliang Tang (5):
Squash to "mptcp: add get_subflow wrappers"
mptcp: redundant subflows push pending
mptcp: redundant subflows retrans support
selftests/bpf: Add bpf_red scheduler
selftests/bpf: Add bpf_red test
net/mptcp/protocol.c | 190 +++++++++++------- net/mptcp/protocol.h | 13 +- net/mptcp/sched.c | 59 +++--- .../testing/selftests/bpf/prog_tests/mptcp.c | 34 ++++ .../selftests/bpf/progs/mptcp_bpf_red.c | 36 ++++ 5 files changed, 231 insertions(+), 101 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/mptcp_bpf_red.c
Signature · 1 line
2.35.3
PATCH v12 1/5 Squash to "mptcp: add get_subflow wrappers"
Please update the commit log:
'''
This patch defines two new wrappers mptcp_sched_get_send() and mptcp_sched_get_retrans(), invoke get_subflow() of msk->sched in them. Use them instead of using mptcp_subflow_get_send() or mptcp_subflow_get_retrans() directly.
Set the subflow pointers array in struct mptcp_sched_data before invoking get_subflow(), then it can be used in get_subflow() in the BPF contexts.
Check the subflow scheduled flags to test which subflow or subflows are picked by the scheduler.
Move sock_owned_by_me() and the fallback check code from mptcp_subflow_get_send/retrans() into the wrappers.
Redundant subflows are not supported in __mptcp_subflow_push_pending() yet. This patch adds a placeholder in mptcp_sched_get_send() to pick the first subflow for the redundant subflows case.
'''
--- net/mptcp/protocol.c | 4 +-- net/mptcp/protocol.h | 2 +- net/mptcp/sched.c | 59 +++++++++++++++++++++++++------------------- 3 files changed, 37 insertions(+), 28 deletions(-)
net/mptcp/protocol.c +2 -2
@@ -1547,7 +1547,7 @@ void __mptcp_push_pending(struct sock *sk, unsigned int flags)int ret =0 ; prev_ssk = ssk; - ssk =mptcp_sched_get_send (msk); + ssk =mptcp_subflow_get_send (msk);/* First check. If the ssk has changed since * the last round, release prev_ssk
@@ -2478,7 +2478,7 @@ static void __mptcp_retrans(struct sock *sk)mptcp_clean_una_wakeup (sk);/* first check ssk: need to kick "stale" logic */ - ssk =mptcp_sched_get_retrans (msk); + ssk =mptcp_subflow_get_retrans (msk); dfrag =mptcp_rtx_head (sk);if (!dfrag) {if (mptcp_data_fin_enabled (msk)) {
net/mptcp/protocol.h +1 -1
@@ -641,7 +641,7 @@ void mptcp_subflow_set_scheduled(struct mptcp_subflow_context *subflow,struct sock *mptcp_subflow_get_send (struct mptcp_sock *msk);struct sock *mptcp_subflow_get_retrans (struct mptcp_sock *msk);struct sock *mptcp_sched_get_send (struct mptcp_sock *msk); -struct sock *mptcp_sched_get_retrans (struct mptcp_sock *msk); +int mptcp_sched_get_retrans (struct mptcp_sock *msk);static inline bool __tcp_can_send (const struct sock *ssk) {
net/mptcp/sched.c +34 -25
100 lines not on this page · raw message
Signature · 1 line
2.35.3
PATCH v12 2/5 mptcp: redundant subflows push pending
In reply to Geliang TangThis patch adds the redundant subflows support for __mptcp_push_pending(). Use mptcp_sched_get_send() wrapper instead of mptcp_subflow_get_send() in it.
Check the subflow scheduled flags to test which subflow or subflows are picked by the scheduler, use them to send data.
--- net/mptcp/protocol.c | 128 +++++++++++++++++++++++++------------------ net/mptcp/protocol.h | 11 ++++ 2 files changed, 87 insertions(+), 52 deletions(-)
net/mptcp/protocol.c +76 -52
@@ -1005,6 +1005,9 @@ static void __mptcp_clean_una(struct sock *sk)if (after64 (dfrag->data_seq + dfrag->data_len, snd_una))break ; +if (msk->first_pending) +break ; +if (unlikely (dfrag == msk->first_pending)) {/* in recovery mode can see ack after the current snd head */ if (WARN_ON_ONCE (!msk->recovery))
@@ -1020,6 +1023,9 @@ static void __mptcp_clean_una(struct sock *sk)if (dfrag &&after64 (snd_una, dfrag->data_seq)) {u64 delta = snd_una - dfrag->data_seq; +if (msk->first_pending) + goto out; +/* prevent wrap around in recovery mode */ if (unlikely (delta > dfrag->already_sent)) {if (WARN_ON_ONCE (!msk->recovery))
@@ -1497,18 +1503,14 @@ static void mptcp_push_release(struct sock *ssk, struct mptcp_sendmsg_info *inforelease_sock (ssk); } -static void mptcp_update_post_push (struct mptcp_sock *msk, -struct mptcp_data_frag *dfrag, -u32 sent) +static void __mptcp_update_post_push (struct mptcp_sock *msk, +const struct mptcp_data_frag *dfrag, +u32 sent) { -u64 snd_nxt_new = dfrag->data_seq; - - dfrag->already_sent += sent; +u64 snd_nxt_new = dfrag->data_seq + dfrag->already_sent + sent; msk->snd_burst -= sent; - snd_nxt_new += dfrag->already_sent; -/* snd_nxt_new can be smaller than snd_nxt in case mptcp * is recovering after a failover. In that event, this re-sends * old segments.
@@ -1522,6 +1524,26 @@ static void mptcp_update_post_push(struct mptcp_sock *msk, msk->snd_nxt = snd_nxt_new; } +static void mptcp_update_post_push (struct mptcp_sock *msk, +struct mptcp_data_frag *dfrag, +u32 sent) +{ +__mptcp_update_post_push (msk, dfrag, sent); + dfrag->already_sent += sent; +} + +static void mptcp_update_dfrags (struct sock *sk) +{ +struct mptcp_sock *msk =mptcp_sk (sk); +struct mptcp_data_frag *dfrag; + +while ((dfrag =mptcp_send_head (sk))) { +if (dfrag->info_sent > dfrag->already_sent) + dfrag->already_sent = dfrag->info_sent; +WRITE_ONCE (msk->first_pending,mptcp_send_next (sk)); + } +} +void mptcp_check_and_set_pending (struct sock *sk) {if (mptcp_send_head (sk))
@@ -1530,60 +1552,62 @@ void mptcp_check_and_set_pending(struct sock *sk)void __mptcp_push_pending (struct sock *sk,unsigned int flags) { -struct sock *prev_ssk =NULL , *ssk =NULL ;struct mptcp_sock *msk =mptcp_sk (sk); -struct mptcp_sendmsg_info info = { - .flags = flags, - }; +struct mptcp_subflow_context *subflow;bool do_check_data_fin = false; -struct mptcp_data_frag *dfrag;int len; -while ((dfrag =mptcp_send_head (sk))) { - info.sent = dfrag->already_sent; - info.limit = dfrag->data_len; - len = dfrag->data_len - dfrag->already_sent; -while (len >0 ) { -int ret =0 ; - - prev_ssk = ssk; - ssk =mptcp_subflow_get_send (msk); - -/* First check. If the ssk has changed since -* the last round, release prev_ssk -*/ -if (ssk != prev_ssk && prev_ssk) -mptcp_push_release (prev_ssk, &info); -if (!ssk) - goto out; +if (!mptcp_sched_get_send (msk)) + goto out; -/* Need to lock the new subflow only if different -* from the previous one, otherwise we are still -* helding the relevant lock -*/ -if (ssk != prev_ssk) -lock_sock (ssk); +mptcp_for_each_subflow (msk, subflow) { +if (READ_ONCE (subflow->scheduled)) { +struct sock *ssk =mptcp_subflow_tcp_sock (subflow); +struct mptcp_sendmsg_info info = { + .flags = flags, + }; +struct mptcp_data_frag *dfrag; - ret =mptcp_sendmsg_frag (sk, ssk, dfrag, &info); -if (ret <=0 ) { -if (ret == -EAGAIN ) -continue ; -mptcp_push_release (ssk, &info); + dfrag =mptcp_send_head (sk); +if (!dfrag) goto out; - } - do_check_data_fin = true; - info.sent += ret; - len -= ret; - -mptcp_update_post_push (msk, dfrag, ret); +lock_sock (ssk); + +do { + info.sent = dfrag->already_sent; + info.limit = dfrag->data_len; + len = dfrag->data_len - dfrag->already_sent; +while (len >0 ) { +int ret =0 ; + + ret =mptcp_sendmsg_frag (sk, ssk, dfrag, &info); +if (ret <=0 ) { +if (ret == -EAGAIN ) +continue ; +if (info.sent > dfrag->already_sent) { + dfrag->info_sent = info.sent; + dfrag->already_sent = dfrag->info_sent; + } +mptcp_push_release (ssk, &info); + goto out; + } + + do_check_data_fin = true; + info.sent += ret; + len -= ret; + +__mptcp_update_post_push (msk, dfrag, ret); + } + dfrag->info_sent = info.sent; + }while ((dfrag =mptcp_next_frag (sk, dfrag))); + +mptcp_push_release (ssk, &info); + msk->last_snd = ssk; +mptcp_subflow_set_scheduled (subflow, false); } -WRITE_ONCE (msk->first_pending,mptcp_send_next (sk)); } - -/* at this point we held the socket lock for the last subflow we used */ -if (ssk) -mptcp_push_release (ssk, &info); +mptcp_update_dfrags (sk); out:/* ensure the rtx timer is running */
net/mptcp/protocol.h +11 -0
@@ -243,6 +243,7 @@ struct mptcp_data_frag {u16 offset;u16 overhead;u16 already_sent; +u16 info_sent;struct page *page; };
@@ -358,6 +359,16 @@ static inline struct mptcp_data_frag *mptcp_send_next(struct sock *sk)list_next_entry (cur, list); } +static inline struct mptcp_data_frag *mptcp_next_frag (struct sock *sk, +struct mptcp_data_frag *cur) +{ +if (!cur) +return NULL ; + +return list_is_last (&cur->list, &mptcp_sk (sk)->rtx_queue) ?NULL : +list_next_entry (cur, list); +} +static inline struct mptcp_data_frag *mptcp_pending_tail (const struct sock *sk) {struct mptcp_sock *msk =mptcp_sk (sk);
Signature · 1 line
2.35.3
PATCH v12 3/5 mptcp: redundant subflows retrans support
In reply to Geliang TangThis patch adds the redundant subflows support for __mptcp_retrans(). In it, use sched_get_retrans() wrapper instead of mptcp_subflow_get_retrans().
Iterate each subflow of msk, check the scheduled flag to test if it is picked by the scheduler. If so, use it to send data.
--- net/mptcp/protocol.c | 62 ++++++++++++++++++++++++++++---------------- 1 file changed, 39 insertions(+), 23 deletions(-)
net/mptcp/protocol.c +39 -23
@@ -2493,16 +2493,17 @@ static void mptcp_check_fastclose(struct mptcp_sock *msk)static void __mptcp_retrans (struct sock *sk) {struct mptcp_sock *msk =mptcp_sk (sk); +struct mptcp_subflow_context *subflow;struct mptcp_sendmsg_info info = {};struct mptcp_data_frag *dfrag; -size_t copied =0 ;struct sock *ssk; -int ret; +int ret, err; +u16 len =0 ;mptcp_clean_una_wakeup (sk);/* first check ssk: need to kick "stale" logic */ - ssk =mptcp_subflow_get_retrans (msk); + err =mptcp_sched_get_retrans (msk); dfrag =mptcp_rtx_head (sk);if (!dfrag) {if (mptcp_data_fin_enabled (msk)) {
@@ -2521,31 +2522,46 @@ static void __mptcp_retrans(struct sock *sk) goto reset_timer; } -if (!ssk) +if (err) goto reset_timer; -lock_sock (ssk); +mptcp_for_each_subflow (msk, subflow) { +if (READ_ONCE (subflow->scheduled)) { +u16 copied =0 ; -/* limit retransmission to the bytes already sent on some subflows */ - info.sent =0 ; - info.limit =READ_ONCE (msk->csum_enabled) ? dfrag->data_len : dfrag->already_sent; -while (info.sent < info.limit) { - ret =mptcp_sendmsg_frag (sk, ssk, dfrag, &info); -if (ret <=0 ) -break ; + ssk =mptcp_subflow_tcp_sock (subflow); +if (!ssk) + goto reset_timer; -MPTCP_INC_STATS (sock_net (sk),MPTCP_MIB_RETRANSSEGS ); - copied += ret; - info.sent += ret; - } -if (copied) { - dfrag->already_sent =max (dfrag->already_sent, info.sent); -tcp_push (ssk,0 , info.mss_now,tcp_sk (ssk)->nonagle, - info.size_goal); -WRITE_ONCE (msk->allow_infinite_fallback, false); - } +lock_sock (ssk); -release_sock (ssk); +/* limit retransmission to the bytes already sent on some subflows */ + info.sent =0 ; + info.limit =READ_ONCE (msk->csum_enabled) ? dfrag->data_len : + dfrag->already_sent; +while (info.sent < info.limit) { + ret =mptcp_sendmsg_frag (sk, ssk, dfrag, &info); +if (ret <=0 ) +break ; + +MPTCP_INC_STATS (sock_net (sk),MPTCP_MIB_RETRANSSEGS ); + copied += ret; + info.sent += ret; + } +if (copied) { + len =max (copied, len); +tcp_push (ssk,0 , info.mss_now,tcp_sk (ssk)->nonagle, + info.size_goal); +WRITE_ONCE (msk->allow_infinite_fallback, false); + } + +release_sock (ssk); + + msk->last_snd = ssk; +mptcp_subflow_set_scheduled (subflow, false); + } + } + dfrag->already_sent =max (dfrag->already_sent, len); reset_timer:mptcp_check_and_set_pending (sk);
Signature · 1 line
2.35.3
PATCH v12 4/5 selftests/bpf: Add bpf_red scheduler
In reply to Geliang TangThis patch implements the redundant BPF MPTCP scheduler, named bpf_red, which sends all packets redundantly on all available subflows.
--- .../selftests/bpf/progs/mptcp_bpf_red.c | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/mptcp_bpf_red.c
tools/testing/selftests/bpf/progs/mptcp_bpf_red.c new file +36 -0
@@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2022, SUSE. */ + +#include <linux/bpf.h> +#include "bpf_tcp_helpers.h" + +char _license[] SEC("license") = "GPL"; + +SEC("struct_ops/mptcp_sched_red_init") +void BPF_PROG (mptcp_sched_red_init ,const struct mptcp_sock *msk) +{ +} + +SEC("struct_ops/mptcp_sched_red_release") +void BPF_PROG (mptcp_sched_red_release ,const struct mptcp_sock *msk) +{ +} + +void BPF_STRUCT_OPS (bpf_red_get_subflow ,const struct mptcp_sock *msk, +struct mptcp_sched_data *data) +{ +for (int i =0 ; i <MPTCP_SUBFLOWS_MAX ; i++) { +if (!data->contexts[i]) +break ; + +mptcp_subflow_set_scheduled (data->contexts[i], true); + } +} + +SEC(".struct_ops") +struct mptcp_sched_ops red = { + .init = (void *)mptcp_sched_red_init, + .release = (void *)mptcp_sched_red_release, + .get_subflow = (void *)bpf_red_get_subflow, + .name ="bpf_red" , +};
Signature · 1 line
2.35.3
PATCH v12 5/5 selftests/bpf: Add bpf_red test
In reply to Geliang TangThis patch adds the redundant BPF MPTCP scheduler test: test_red(). Use sysctl to set net.mptcp.scheduler to use this sched. Add two veth net devices to simulate the multiple addresses case. Use 'ip mptcp endpoint' command to add the new endpoint ADDR_2 to PM netlink. Send data and check bytes_sent of 'ss' output after it to make sure the data has been redundantly sent on both net devices.
--- .../testing/selftests/bpf/prog_tests/mptcp.c | 34 +++++++++++++++++++ 1 file changed, 34 insertions(+)
tools/testing/selftests/bpf/prog_tests/mptcp.c +34 -0
@@ -9,6 +9,7 @@#include "mptcp_bpf_first.skel.h" #include "mptcp_bpf_bkup.skel.h" #include "mptcp_bpf_rr.skel.h" +#include "mptcp_bpf_red.skel.h" #ifndef TCP_CA_NAME_MAX #define TCP_CA_NAME_MAX 16
@@ -381,6 +382,37 @@ static void test_rr(void)mptcp_bpf_rr__destroy (rr_skel); } +static void test_red (void ) +{ +struct mptcp_bpf_red *red_skel; +int server_fd, client_fd; +struct bpf_link *link; + + red_skel =mptcp_bpf_red__open_and_load (); +if (!ASSERT_OK_PTR (red_skel,"bpf_red__open_and_load" )) +return ; + + link =bpf_map__attach_struct_ops (red_skel->maps.red); +if (!ASSERT_OK_PTR (link,"bpf_map__attach_struct_ops" )) { +mptcp_bpf_red__destroy (red_skel); +return ; + } + +sched_init ("subflow" ,"bpf_red" ); + server_fd =start_mptcp_server (AF_INET ,ADDR_1 ,0 ,0 ); + client_fd =connect_to_fd (server_fd,0 ); + +send_data (server_fd, client_fd); +ASSERT_OK (has_bytes_sent (ADDR_1 ),"has_bytes_sent addr 1" ); +ASSERT_OK (has_bytes_sent (ADDR_2 ),"has_bytes_sent addr 2" ); + +close (client_fd); +close (server_fd); +sched_cleanup (); +bpf_link__destroy (link); +mptcp_bpf_red__destroy (red_skel); +} +void test_mptcp (void ) {if (test__start_subtest ("base" ))
@@ -391,4 +423,6 @@ void test_mptcp(void)test_bkup ();if (test__start_subtest ("rr" ))test_rr (); +if (test__start_subtest ("red" )) +test_red (); }
Signature · 1 line
2.35.3
Re: selftests/bpf: Add bpf_red test: Tests Results
Hi Geliang,
Thank you for your modifications, that's great!
Our CI did some validations and here is its report:
- KVM Validation: normal:
- Critical: Global Timeout ❌:
- Task: https://cirrus-ci.com/task/5154530944876544
- Summary: https://api.cirrus-ci.com/v1/artifact/task/5154530944876544/summary/summary.txt
- KVM Validation: debug:
- Critical: 104 Call Trace(s) - Critical: Global Timeout ❌:
- Task: https://cirrus-ci.com/task/6280430851719168
- Summary: https://api.cirrus-ci.com/v1/artifact/task/6280430851719168/summary/summary.txt
Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/04ba877e852b
If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:
$ cd [kernel source code]
$ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
--pull always mptcp/mptcp-upstream-virtme-docker:latest \
auto-debugFor more details:
https://github.com/multipath-tcp/mptcp-upstream-virtme-docker
Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)
Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (Tessares)
Quoted 4 lines · Geliang Tang
v12:
- fixWARN_ON_ONCE(reuse_skb)andWARN_ON_ONCE(!msk->recovery)errors in kernel logs.
Hi Geliang -
Sorry I didn't see your v11 comment last week.
I was partway through a review of the scheduler loops in this series yesterday, but it made sense to focus on the "refactor push pending" patches today. I'm going to mark this v12 series as "superseded" in patchwork while we continue discussing "refactor push pending".
Thanks,
Mat
Quoted 61 lines
v11:
- address to Mat's comments in v10.
- rebase to export/20220908T063452v10:
- send multiple dfrags in__mptcp_push_pending().v9:
- drop the extra *err paramenter ofmptcp_sched_get_send()as Florian suggested.v8:
- update__mptcp_push_pending(), send the same data on each subflow.
- update__mptcp_retrans, track the max sent data.
= add a new patch.v7:
- drop redundant flag in v6
- drop__mptcp_subflows_push_pendingin v6
- update redundant subflows support in__mptcp_push_pending
- update redundant subflows support in__mptcp_retransv6:
- Add redundant flag for structmptcp_sched_ops.
- add a dedicated function__mptcp_subflows_push_pending()to deal with redundat subflows push pending.v5:
- address to Paolo's comment, keep the optimization tomptcp_subflow_get_send()for the non eBPF case.
- mergemptcp_sched_get_send()and__mptcp_sched_get_send()in v4 into one.
- depends on "cleanups for bpf sched selftests".v4:
- small cleanups in patch 1, 2.
- add TODO in patch 3.
- rebase patch 5 on 'cleanups for bpf sched selftests'.v3:
- use new API.
- fix the link failure tests issue mentioned in ("https://patchwork.kernel.org/project/mptcp/cover/[email protected]/").v2:
- addMPTCP_SUBFLOWS_MAXlimit to avoid infinite loops when the scheduler always setscall_againto true.
- track the largest copied amount.
- deal with__mptcp_subflow_push_pending()and the retransmit loop.
- depends on "BPF round-robin scheduler" v14.v1:
Implements the redundant BPF MPTCP scheduler, which sends all packets redundantly on all available subflows.
Geliang Tang (5):
Squash to "mptcp: addget_subflowwrappers"
mptcp: redundant subflows push pending
mptcp: redundant subflows retrans support
selftests/bpf: Addbpf_redscheduler
selftests/bpf: Addbpf_redtest
Quoted 13 lines
net/mptcp/protocol.c | 190 +++++++++++------- net/mptcp/protocol.h | 13 +- net/mptcp/sched.c | 59 +++--- .../testing/selftests/bpf/prog_tests/mptcp.c | 34 ++++ .../selftests/bpf/progs/mptcp_bpf_red.c | 36 ++++ 5 files changed, 231 insertions(+), 101 deletions(-)create mode 100644
tools/testing/selftests/bpf/progs/mptcp_bpf_red.cSignature · 1 line
2.35.3
--
Mat Martineau
Intel