BPF redundant scheduler, part 1
Subsystems (5)
v19 cover BPF redundant scheduler, part 1
v19:
- update patch 1.
- split "BPF redundant scheduler" into two parts.
- rebased on "export/20221111T055536".
Geliang Tang (9):
mptcp: refactor push_pending logic
mptcp: drop last_snd and MPTCP_RESET_SCHEDULER
mptcp: add sched_data_set_contexts helper Squash to "mptcp: add struct mptcp_sched_ops" Squash to "bpf: Add bpf_mptcp_sched_ops" Squash to "bpf: Add bpf_mptcp_sched_kfunc_set" Squash to "selftests/bpf: Add bpf_first scheduler" Squash to "selftests/bpf: Add bpf_bkup scheduler" Squash to "selftests/bpf: Add bpf_rr scheduler"
include/net/mptcp.h | 6 +- net/mptcp/bpf.c | 1 + net/mptcp/pm.c | 9 +- net/mptcp/pm_netlink.c | 3 - net/mptcp/protocol.c | 161 +++++++++--------- net/mptcp/protocol.h | 1 - net/mptcp/sched.c | 21 ++- tools/testing/selftests/bpf/bpf_tcp_helpers.h | 8 +- .../selftests/bpf/progs/mptcp_bpf_bkup.c | 10 +- .../selftests/bpf/progs/mptcp_bpf_first.c | 10 +- .../selftests/bpf/progs/mptcp_bpf_rr.c | 10 +- 11 files changed, 143 insertions(+), 97 deletions(-)
Signature · 1 line
2.35.3
PATCH v19 1/9 mptcp: refactor push_pending logic
To support redundant package schedulers more easily, this patch refactors __mptcp_push_pending() logic from:
For each dfrag:
While sends succeed:
Call the scheduler (selects subflow and msk->snd_burst) Update subflow locks (push/release/acquire as needed) Send the dfrag data with mptcp_sendmsg_frag() Update already_sent, snd_nxt, snd_burst
Update msk->first_pending
Push/release on final subflow
->
While first_pending isn't empty:
Call the scheduler (selects subflow and msk->snd_burst) Update subflow locks (push/release/acquire as needed) For each pending dfrag:
While sends succeed:
Send the dfrag data with mptcp_sendmsg_frag() Update already_sent, snd_nxt, snd_burst
Update msk->first_pending
Break if required by msk->snd_burst / etc
Push/release on final subflow
Refactors __mptcp_subflow_push_pending logic from:
For each dfrag:
While sends succeed: Call the scheduler (selects subflow and msk->snd_burst) Send the dfrag data with mptcp_subflow_delegate(), break Send the dfrag data with mptcp_sendmsg_frag() Update dfrag->already_sent, msk->snd_nxt, msk->snd_burst Update msk->first_pending
->
While first_pending isn't empty:
Call the scheduler (selects subflow and msk->snd_burst) Send the dfrag data with mptcp_subflow_delegate(), break Send the dfrag data with mptcp_sendmsg_frag()
For each pending dfrag:
While sends succeed:
Send the dfrag data with mptcp_sendmsg_frag() Update already_sent, snd_nxt, snd_burst
Update msk->first_pending
Break if required by msk->snd_burst / etc
Move the duplicate code from __mptcp_push_pending() and __mptcp_subflow_push_pending() into a new helper function, named __subflow_push_pending(). Simplify __mptcp_push_pending() and __mptcp_subflow_push_pending() by invoking this helper.
Also move the burst check conditions out of the function mptcp_subflow_get_send(), check them in __subflow_push_pending() in the inner "for each pending dfrag" loop.
--- net/mptcp/protocol.c | 154 +++++++++++++++++++++++-------------------- 1 file changed, 83 insertions(+), 71 deletions(-)
net/mptcp/protocol.c +83 -71
@@ -1417,14 +1417,6 @@ static struct sock *mptcp_subflow_get_send(struct mptcp_sock *msk)sk_stream_memory_free (msk->first) ? msk->first :NULL ; } -/* re-use last subflow, if the burst allow that */ -if (msk->last_snd && msk->snd_burst >0 && -sk_stream_memory_free (msk->last_snd) && -mptcp_subflow_active (mptcp_subflow_ctx (msk->last_snd))) { -mptcp_set_timeout (sk); -return msk->last_snd; - } -/* pick the subflow with the lower wmem/wspace ratio */ for (i =0 ; i <SSK_MODE_MAX ; ++i) { send_info[i].ssk =NULL ;
@@ -1528,57 +1520,86 @@ void mptcp_check_and_set_pending(struct sock *sk)mptcp_sk (sk)->push_pending |=BIT (MPTCP_PUSH_PENDING ); } -void __mptcp_push_pending (struct sock *sk,unsigned int flags) +static int __subflow_push_pending (struct sock *sk,struct sock *ssk, +struct mptcp_sendmsg_info *info) { -struct sock *prev_ssk =NULL , *ssk =NULL ;struct mptcp_sock *msk =mptcp_sk (sk); -struct mptcp_sendmsg_info info = { - .flags = flags, - }; -bool do_check_data_fin = false;struct mptcp_data_frag *dfrag; -int len; +int len, copied =0 , err =0 ;while ((dfrag =mptcp_send_head (sk))) { - info.sent = dfrag->already_sent; - info.limit = dfrag->data_len; + info->sent = dfrag->already_sent; + info->limit = dfrag->data_len; len = dfrag->data_len - dfrag->already_sent;while (len >0 ) {int ret =0 ; - prev_ssk = ssk; - ssk =mptcp_subflow_get_send (msk); - -/* First check. If the ssk has changed since -* the last round, release prev_ssk -*/ -if (ssk != prev_ssk && prev_ssk) -mptcp_push_release (prev_ssk, &info); -if (!ssk) - goto out; - -/* Need to lock the new subflow only if different -* from the previous one, otherwise we are still -* helding the relevant lock -*/ -if (ssk != prev_ssk) -lock_sock (ssk); - - ret =mptcp_sendmsg_frag (sk, ssk, dfrag, &info); + ret =mptcp_sendmsg_frag (sk, ssk, dfrag, info);if (ret <=0 ) { -if (ret == -EAGAIN ) -continue ; -mptcp_push_release (ssk, &info); + err = copied ? : ret; goto out; } - do_check_data_fin = true; - info.sent += ret; + info->sent += ret; + copied += ret; len -= ret;mptcp_update_post_push (msk, dfrag, ret); }WRITE_ONCE (msk->first_pending,mptcp_send_next (sk)); + +if (msk->snd_burst <=0 || + !sk_stream_memory_free (ssk) || + !mptcp_subflow_active (mptcp_subflow_ctx (ssk))) { + err = copied ? : -EAGAIN ; + goto out; + } +mptcp_set_timeout (sk); + } + err = copied; + +out: +return err; +} + +void __mptcp_push_pending (struct sock *sk,unsigned int flags) +{ +struct sock *prev_ssk =NULL , *ssk =NULL ; +struct mptcp_sock *msk =mptcp_sk (sk); +struct mptcp_sendmsg_info info = { + .flags = flags, + }; +bool do_check_data_fin = false; + +while (mptcp_send_head (sk)) { +int ret =0 ; + + prev_ssk = ssk; + ssk =mptcp_subflow_get_send (msk); + +/* First check. If the ssk has changed since +* the last round, release prev_ssk +*/ +if (ssk != prev_ssk && prev_ssk) +mptcp_push_release (prev_ssk, &info); +if (!ssk) + goto out; + +/* Need to lock the new subflow only if different +* from the previous one, otherwise we are still +* helding the relevant lock +*/ +if (ssk != prev_ssk) +lock_sock (ssk); + + ret =__subflow_push_pending (sk, ssk, &info); +if (ret <=0 ) { +if (ret == -EAGAIN ) +continue ; +mptcp_push_release (ssk, &info); + goto out; + } + do_check_data_fin = true; }/* at this point we held the socket lock for the last subflow we used */
@@ -1599,42 +1620,33 @@ static void __mptcp_subflow_push_pending(struct sock *sk, struct sock *ssk, boolstruct mptcp_sendmsg_info info = { .data_lock_held = true, }; -struct mptcp_data_frag *dfrag;struct sock *xmit_ssk; -int len, copied =0 ; +int copied =0 ; info.flags =0 ; -while ((dfrag =mptcp_send_head (sk))) { - info.sent = dfrag->already_sent; - info.limit = dfrag->data_len; - len = dfrag->data_len - dfrag->already_sent; -while (len >0 ) { -int ret =0 ; +while (mptcp_send_head (sk)) { +int ret =0 ; -/* check for a different subflow usage only after -* spooling the first chunk of data -*/ - xmit_ssk = first ? ssk :mptcp_subflow_get_send (msk); -if (!xmit_ssk) - goto out; -if (xmit_ssk != ssk) { -mptcp_subflow_delegate (mptcp_subflow_ctx (xmit_ssk), -MPTCP_DELEGATE_SEND ); - goto out; - } - - ret =mptcp_sendmsg_frag (sk, ssk, dfrag, &info); -if (ret <=0 ) - goto out; - - info.sent += ret; - copied += ret; - len -= ret; - first = false; +/* check for a different subflow usage only after +* spooling the first chunk of data +*/ + xmit_ssk = first ? ssk :mptcp_subflow_get_send (msk); +if (!xmit_ssk) + goto out; +if (xmit_ssk != ssk) { +mptcp_subflow_delegate (mptcp_subflow_ctx (xmit_ssk), +MPTCP_DELEGATE_SEND ); + goto out; + } -mptcp_update_post_push (msk, dfrag, ret); + ret =__subflow_push_pending (sk, ssk, &info); + first = false; +if (ret <=0 ) { +if (ret == -EAGAIN ) +continue ; +break ; } -WRITE_ONCE (msk->first_pending,mptcp_send_next (sk)); + copied += ret; } out:
Signature · 1 line
2.35.3
PATCH v19 2/9 mptcp: drop last_snd and MPTCP_RESET_SCHEDULER
In reply to Geliang TangSince the burst check conditions have moved out of the function mptcp_subflow_get_send(), it makes some msk->last_snd useless. This patch drops them as well as the macro MPTCP_RESET_SCHEDULER.
--- net/mptcp/pm.c | 9 +-------- net/mptcp/pm_netlink.c | 3 --- net/mptcp/protocol.c | 7 +------ net/mptcp/protocol.h | 1 - 4 files changed, 2 insertions(+), 18 deletions(-)
net/mptcp/pm.c +1 -8
@@ -282,15 +282,8 @@ void mptcp_pm_mp_prio_received(struct sock *ssk, u8 bkup)pr_debug ("subflow->backup=%d, bkup=%d\n" , subflow->backup, bkup); msk =mptcp_sk (sk); -if (subflow->backup != bkup) { +if (subflow->backup != bkup) subflow->backup = bkup; -mptcp_data_lock (sk); -if (!sock_owned_by_user (sk)) - msk->last_snd =NULL ; -else -__set_bit (MPTCP_RESET_SCHEDULER , &msk->cb_flags); -mptcp_data_unlock (sk); - }mptcp_event (MPTCP_EVENT_SUB_PRIORITY , msk, ssk,GFP_ATOMIC ); }
net/mptcp/pm_netlink.c +0 -3
@@ -475,9 +475,6 @@ static void __mptcp_pm_send_ack(struct mptcp_sock *msk, struct mptcp_subflow_con slow =lock_sock_fast (ssk);if (prio) { -if (subflow->backup != backup) - msk->last_snd =NULL ; - subflow->send_mp_prio =1 ; subflow->backup = backup; subflow->request_bkup = backup;
net/mptcp/protocol.c +1 -6
@@ -1469,16 +1469,13 @@ static struct sock *mptcp_subflow_get_send(struct mptcp_sock *msk) burst =min_t (int,MPTCP_SEND_BURST_SIZE ,mptcp_wnd_end (msk) - msk->snd_nxt); wmem =READ_ONCE (ssk->sk_wmem_queued); -if (!burst) { - msk->last_snd =NULL ; +if (!burst)return ssk; - } subflow =mptcp_subflow_ctx (ssk); subflow->avg_pacing_rate =div_u64 ((u64 )subflow->avg_pacing_rate * wmem +READ_ONCE (ssk->sk_pacing_rate) * burst, burst + wmem); - msk->last_snd = ssk; msk->snd_burst = burst;return ssk; }
@@ -3284,8 +3281,6 @@ static void mptcp_release_cb(struct sock *sk)__mptcp_set_connected (sk);if (__test_and_clear_bit (MPTCP_ERROR_REPORT , &msk->cb_flags))__mptcp_error_report (sk); -if (__test_and_clear_bit (MPTCP_RESET_SCHEDULER , &msk->cb_flags)) - msk->last_snd =NULL ; }__mptcp_update_rmem (sk);
net/mptcp/protocol.h +0 -1
@@ -124,7 +124,6 @@#define MPTCP_RETRANSMIT 4#define MPTCP_FLUSH_JOIN_LIST 5#define MPTCP_CONNECTED 6 -#define MPTCP_RESET_SCHEDULER 7struct mptcp_skb_cb {u64 map_seq;
Signature · 1 line
2.35.3
PATCH v19 3/9 mptcp: add sched_data_set_contexts helper
In reply to Geliang TangAdd a new helper mptcp_sched_data_set_contexts() to set the subflow pointers array in struct mptcp_sched_data. It will be invoked by the BPF schedulers to export the subflow pointers to the BPF contexts.
--- net/mptcp/sched.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
net/mptcp/sched.c +19 -0
@@ -93,3 +93,22 @@ void mptcp_subflow_set_scheduled(struct mptcp_subflow_context *subflow, {WRITE_ONCE (subflow->scheduled, scheduled); } + +void mptcp_sched_data_set_contexts (const struct mptcp_sock *msk, +struct mptcp_sched_data *data) +{ +struct mptcp_subflow_context *subflow; +int i =0 ; + +mptcp_for_each_subflow (msk, subflow) { +if (i ==MPTCP_SUBFLOWS_MAX ) { +pr_warn_once ("too many subflows" ); +break ; + } +mptcp_subflow_set_scheduled (subflow, false); + data->contexts[i++] = subflow; + } + +for (; i <MPTCP_SUBFLOWS_MAX ; i++) + data->contexts[i] =NULL ; +}
Signature · 1 line
2.35.3
PATCH v19 4/9 Squash to "mptcp: add struct mptcp_sched_ops"
In reply to Geliang TangNew api:
- add new data_init
- add an int return value for get_subflow
--- include/net/mptcp.h | 6 ++++-- net/mptcp/sched.c | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-)
include/net/mptcp.h +4 -2
@@ -105,8 +105,10 @@ struct mptcp_sched_data { };struct mptcp_sched_ops { -void (*get_subflow)(const struct mptcp_sock *msk, -struct mptcp_sched_data *data); +void (*data_init)(const struct mptcp_sock *msk, +struct mptcp_sched_data *data); +int (*get_subflow)(const struct mptcp_sock *msk, +struct mptcp_sched_data *data);char name[MPTCP_SCHED_NAME_MAX ];struct module *owner;
net/mptcp/sched.c +1 -1
@@ -33,7 +33,7 @@ struct mptcp_sched_ops *mptcp_sched_find(const char *name)int mptcp_register_scheduler (struct mptcp_sched_ops *sched) { -if (!sched->get_subflow) +if (!sched->data_init || !sched->get_subflow)return -EINVAL ;spin_lock (&mptcp_sched_list_lock);
Signature · 1 line
2.35.3
PATCH v19 5/9 Squash to "bpf: Add bpf_mptcp_sched_ops"
In reply to Geliang TangUse new API.
--- tools/testing/selftests/bpf/bpf_tcp_helpers.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/bpf_tcp_helpers.h +6 -2
@@ -249,8 +249,10 @@ struct mptcp_sched_ops {void (*init)(const struct mptcp_sock *msk);void (*release)(const struct mptcp_sock *msk); -void (*get_subflow)(const struct mptcp_sock *msk, -struct mptcp_sched_data *data); +void (*data_init)(const struct mptcp_sock *msk, +struct mptcp_sched_data *data); +int (*get_subflow)(const struct mptcp_sock *msk, +struct mptcp_sched_data *data);void *owner; };
@@ -265,5 +267,7 @@ struct mptcp_sock { extern void mptcp_subflow_set_scheduled(struct mptcp_subflow_context *subflow, bool scheduled) __ksym; +extern void mptcp_sched_data_set_contexts(const struct mptcp_sock *msk, + struct mptcp_sched_data *data) __ksym;#endif
Signature · 1 line
2.35.3
PATCH v19 6/9 Squash to "bpf: Add bpf_mptcp_sched_kfunc_set"
In reply to Geliang TangAdd mptcp_sched_data_set_contexts in kfunc_set.
--- net/mptcp/bpf.c | 1 + 1 file changed, 1 insertion(+)
net/mptcp/bpf.c +1 -0
@@ -164,6 +164,7 @@ struct bpf_struct_ops bpf_mptcp_sched_ops = {BTF_SET8_START (bpf_mptcp_sched_kfunc_ids)BTF_ID_FLAGS (func ,mptcp_subflow_set_scheduled ) +BTF_ID_FLAGS (func, mptcp_sched_data_set_contexts)BTF_SET8_END (bpf_mptcp_sched_kfunc_ids) static const struct btf_kfunc_id_set bpf_mptcp_sched_kfunc_set = {
Signature · 1 line
2.35.3
PATCH v19 7/9 Squash to "selftests/bpf: Add bpf_first scheduler"
In reply to Geliang TangUse new API.
--- tools/testing/selftests/bpf/progs/mptcp_bpf_first.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/progs/mptcp_bpf_first.c +9 -1
@@ -16,16 +16,24 @@ void BPF_PROG(mptcp_sched_first_release, const struct mptcp_sock *msk) { } -void BPF_STRUCT_OPS (bpf_first_get_subflow ,const struct mptcp_sock *msk, +void BPF_STRUCT_OPS (bpf_first_data_init ,const struct mptcp_sock *msk,struct mptcp_sched_data *data) +{ +mptcp_sched_data_set_contexts (msk, data); +} + +int BPF_STRUCT_OPS (bpf_first_get_subflow ,const struct mptcp_sock *msk, +struct mptcp_sched_data *data) {mptcp_subflow_set_scheduled (data->contexts[0 ], true); +return 0 ; } SEC(".struct_ops")struct mptcp_sched_ops first = { .init = (void *)mptcp_sched_first_init, .release = (void *)mptcp_sched_first_release, + .data_init = (void *)bpf_first_data_init, .get_subflow = (void *)bpf_first_get_subflow, .name ="bpf_first" , };
Signature · 1 line
2.35.3
PATCH v19 8/9 Squash to "selftests/bpf: Add bpf_bkup scheduler"
In reply to Geliang TangUse new API.
--- tools/testing/selftests/bpf/progs/mptcp_bpf_bkup.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/progs/mptcp_bpf_bkup.c +9 -1
@@ -16,8 +16,14 @@ void BPF_PROG(mptcp_sched_bkup_release, const struct mptcp_sock *msk) { } -void BPF_STRUCT_OPS (bpf_bkup_get_subflow ,const struct mptcp_sock *msk, +void BPF_STRUCT_OPS (bpf_bkup_data_init ,const struct mptcp_sock *msk,struct mptcp_sched_data *data) +{ +mptcp_sched_data_set_contexts (msk, data); +} + +int BPF_STRUCT_OPS (bpf_bkup_get_subflow ,const struct mptcp_sock *msk, +struct mptcp_sched_data *data) {int nr =0 ;
@@ -32,12 +38,14 @@ void BPF_STRUCT_OPS(bpf_bkup_get_subflow, const struct mptcp_sock *msk, }mptcp_subflow_set_scheduled (data->contexts[nr], true); +return 0 ; } SEC(".struct_ops")struct mptcp_sched_ops bkup = { .init = (void *)mptcp_sched_bkup_init, .release = (void *)mptcp_sched_bkup_release, + .data_init = (void *)bpf_bkup_data_init, .get_subflow = (void *)bpf_bkup_get_subflow, .name ="bpf_bkup" , };
Signature · 1 line
2.35.3
PATCH v19 9/9 Squash to "selftests/bpf: Add bpf_rr scheduler"
In reply to Geliang TangUse new API.
--- tools/testing/selftests/bpf/progs/mptcp_bpf_rr.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/progs/mptcp_bpf_rr.c +9 -1
@@ -16,8 +16,14 @@ void BPF_PROG(mptcp_sched_rr_release, const struct mptcp_sock *msk) { } -void BPF_STRUCT_OPS (bpf_rr_get_subflow ,const struct mptcp_sock *msk, +void BPF_STRUCT_OPS (bpf_rr_data_init ,const struct mptcp_sock *msk,struct mptcp_sched_data *data) +{ +mptcp_sched_data_set_contexts (msk, data); +} + +int BPF_STRUCT_OPS (bpf_rr_get_subflow ,const struct mptcp_sock *msk, +struct mptcp_sched_data *data) {int nr =0 ;
@@ -35,12 +41,14 @@ void BPF_STRUCT_OPS(bpf_rr_get_subflow, const struct mptcp_sock *msk, }mptcp_subflow_set_scheduled (data->contexts[nr], true); +return 0 ; } SEC(".struct_ops")struct mptcp_sched_ops rr = { .init = (void *)mptcp_sched_rr_init, .release = (void *)mptcp_sched_rr_release, + .data_init = (void *)bpf_rr_data_init, .get_subflow = (void *)bpf_rr_get_subflow, .name ="bpf_rr" , };
Signature · 1 line
2.35.3
Re: Squash to "selftests/bpf: Add bpf_rr scheduler": Build Failure
Hi Geliang,
Thank you for your modifications, that's great!
But sadly, our CI spotted some issues with it when trying to build it.
You can find more details there:
https://patchwork.kernel.org/project/mptcp/patch/d1d8d0acb444c6ac39eb6ae945bf439d633e74b3.1668151131.git.geliang.tang@suse.com/
https://github.com/multipath-tcp/mptcp_net-next/actions/runs/3443397477
Status: failure
Initiator: MPTCPimporter
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/b52f03a91914
Feel free to reply to this email if you cannot access logs, if you need some support to fix the error, if this doesn't seem to be caused by your modifications or if the error is a false positive one.
Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (Tessares)
Re: Squash to "selftests/bpf: Add bpf_rr scheduler": Tests Results
In reply to Geliang TangHi Geliang,
Thank you for your modifications, that's great!
Our CI did some validations and here is its report:
- KVM Validation: normal:
- Unstable: 1 failed test(s): mptcp_connect_mmap 🔴:
- Task: https://cirrus-ci.com/task/5566491222343680
- Summary: https://api.cirrus-ci.com/v1/artifact/task/5566491222343680/summary/summary.txt
- KVM Validation: debug:
- Unstable: 2 failed test(s): mptcp_connect_mmap packetdrill_add_addr - Critical: Global Timeout ❌:
- Task: https://cirrus-ci.com/task/6692391129186304
- Summary: https://api.cirrus-ci.com/v1/artifact/task/6692391129186304/summary/summary.txt
Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/b52f03a91914
If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:
$ cd [kernel source code]
$ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
--pull always mptcp/mptcp-upstream-virtme-docker:latest \
auto-debugFor more details:
https://github.com/multipath-tcp/mptcp-upstream-virtme-docker
Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)
Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (Tessares)
Quoted 5 lines · Geliang Tang
v19:
- update patch 1.
- split "BPF redundant scheduler" into two parts.
- rebased on "export/20221111T055536".
Thanks Geliang.
I think v19 (part 1) looks ok for the export branch. simult_flows.sh was slightly faster (small sample size :) ) on my VM, for what it's worth.
Looks like the first three patches go at the beginning of the "features net-next-next" section, correct?
Quoted 11 lines
Geliang Tang (9):
mptcp: refactorpush_pendinglogic
mptcp: droplast_sndandMPTCP_RESET_SCHEDULER
mptcp: addsched_data_set_contextshelper Squash to "mptcp: add structmptcp_sched_ops" Squash to "bpf: Addbpf_mptcp_sched_ops" Squash to "bpf: Addbpf_mptcp_sched_kfunc_set" Squash to "selftests/bpf: Addbpf_firstscheduler" Squash to "selftests/bpf: Addbpf_bkupscheduler" Squash to "selftests/bpf: Addbpf_rrscheduler"
Quoted 18 lines
include/net/mptcp.h | 6 +- net/mptcp/bpf.c | 1 + net/mptcp/pm.c | 9 +- net/mptcp/pm_netlink.c | 3 - net/mptcp/protocol.c | 161 +++++++++--------- net/mptcp/protocol.h | 1 - net/mptcp/sched.c | 21 ++- tools/testing/selftests/bpf/bpf_tcp_helpers.h | 8 +- .../selftests/bpf/progs/mptcp_bpf_bkup.c | 10 +- .../selftests/bpf/progs/mptcp_bpf_first.c | 10 +- .../selftests/bpf/progs/mptcp_bpf_rr.c | 10 +- 11 files changed, 143 insertions(+), 97 deletions(-)Signature · 1 line
2.35.3
--
Mat Martineau
Intel
Re: Squash to "selftests/bpf: Add bpf_rr scheduler": Build Failure
In reply to Geliang TangHi Geliang,
Thank you for your modifications, that's great!
But sadly, our CI spotted some issues with it when trying to build it.
You can find more details there:
https://patchwork.kernel.org/project/mptcp/patch/d1d8d0acb444c6ac39eb6ae945bf439d633e74b3.1668151131.git.geliang.tang@suse.com/
https://github.com/multipath-tcp/mptcp_net-next/actions/runs/3475381466
Status: failure
Initiator: MPTCPimporter
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/4174181fc84f
Feel free to reply to this email if you cannot access logs, if you need some support to fix the error, if this doesn't seem to be caused by your modifications or if the error is a false positive one.
Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (Tessares)
Quoted 8 lines · Mat Martineau
Quoted 5 lines · Geliang Tang
v19:
- update patch 1.
- split "BPF redundant scheduler" into two parts.
- rebased on "export/20221111T055536".
Quoted 7 lines
Thanks Geliang.
I think v19 (part 1) looks ok for the export branch.
simult_flows.shwas slightly faster (small sample size :) ) on my VM, for what it's worth.Looks like the first three patches go at the beginning of the "features net-next-next" section, correct?
Yes, the first two patches ("mptcp: refactor push_pending logic" and "mptcp: drop last_snd and MPTCP_RESET_SCHEDULER") go at the beginning of the "features net-next-next" section. But the third one "mptcp: add sched_data_set_contexts helper" should be inserted between the commits "mptcp: add scheduled in mptcp_subflow_context" and "bpf: Add bpf_mptcp_sched_ops".
Thanks,
-Geliang
Quoted 36 lines
Reviewed-by Mat Martineau <[email protected]>Quoted 11 lines
Geliang Tang (9):
mptcp: refactorpush_pendinglogic
mptcp: droplast_sndandMPTCP_RESET_SCHEDULER
mptcp: addsched_data_set_contextshelper Squash to "mptcp: add structmptcp_sched_ops" Squash to "bpf: Addbpf_mptcp_sched_ops" Squash to "bpf: Addbpf_mptcp_sched_kfunc_set" Squash to "selftests/bpf: Addbpf_firstscheduler" Squash to "selftests/bpf: Addbpf_bkupscheduler" Squash to "selftests/bpf: Addbpf_rrscheduler"Quoted 18 lines
include/net/mptcp.h | 6 +- net/mptcp/bpf.c | 1 + net/mptcp/pm.c | 9 +- net/mptcp/pm_netlink.c | 3 - net/mptcp/protocol.c | 161 +++++++++--------- net/mptcp/protocol.h | 1 - net/mptcp/sched.c | 21 ++- tools/testing/selftests/bpf/bpf_tcp_helpers.h | 8 +- .../selftests/bpf/progs/mptcp_bpf_bkup.c | 10 +- .../selftests/bpf/progs/mptcp_bpf_first.c | 10 +- .../selftests/bpf/progs/mptcp_bpf_rr.c | 10 +- 11 files changed, 143 insertions(+), 97 deletions(-)Signature · 1 line
2.35.3--
Mat Martineau
Intel
Re: Squash to "selftests/bpf: Add bpf_rr scheduler": Tests Results
In reply to Geliang TangHi Geliang,
Thank you for your modifications, that's great!
Our CI did some validations and here is its report:
- KVM Validation: normal:
- Success! ✅:
- Task: https://cirrus-ci.com/task/5459335883522048
- Summary: https://api.cirrus-ci.com/v1/artifact/task/5459335883522048/summary/summary.txt
- KVM Validation: debug:
- Success! ✅:
- Task: https://cirrus-ci.com/task/6585235790364672
- Summary: https://api.cirrus-ci.com/v1/artifact/task/6585235790364672/summary/summary.txt
Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/4174181fc84f
If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:
$ cd [kernel source code]
$ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
--pull always mptcp/mptcp-upstream-virtme-docker:latest \
auto-debugFor more details:
https://github.com/multipath-tcp/mptcp-upstream-virtme-docker
Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)
Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (Tessares)
Hi Geliang, Mat,
Quoted 5 lines · Geliang Tang
v19:
- update patch 1.
- split "BPF redundant scheduler" into two parts.
- rebased on "export/20221111T055536".
Quoted 10 lines
Geliang Tang (9):
mptcp: refactorpush_pendinglogic
mptcp: droplast_sndandMPTCP_RESET_SCHEDULER
mptcp: addsched_data_set_contextshelper Squash to "mptcp: add structmptcp_sched_ops" Squash to "bpf: Addbpf_mptcp_sched_ops" Squash to "bpf: Addbpf_mptcp_sched_kfunc_set" Squash to "selftests/bpf: Addbpf_firstscheduler" Squash to "selftests/bpf: Addbpf_bkupscheduler" Squash to "selftests/bpf: Addbpf_rrscheduler"
Thank you for the patches and the reviews!
I just applied them (+ an additional squash-to patch) in our tree (feat. for other trees (bpf)) with Mat's RvB tag.
Patches 1/9 and 2/9 have been added at the beginning. Patch 3/9 depends on "mptcp: add struct mptcp_sched_ops" and it has been applied just before "bpf: Add bpf_mptcp_sched_ops" as mentioned by Geliang.
Please also note that patch 5/9 has been squashed in "selftests/bpf: Add mptcp sched structs" instead of "bpf: Add bpf_mptcp_sched_ops. Do not hesitate to tell me if something is not OK!
New patches for t/upstream:
- 602fba0955d1: mptcp: refactor push_pending logic
- 343f8eed9abb: mptcp: drop last_snd and MPTCP_RESET_SCHEDULER
- Results: 69720c6cd168..ac857086fdb8 (export)
- 65e9504361f6: mptcp: add sched_data_set_contexts helper
- Results: ac857086fdb8..ae9c4d4cf58f (export)
- 82721c73112e: "squashed" patch 4/9 in "mptcp: add struct mptcp_sched_ops"
- a7c2367dd1d2: "squashed" patch 5/9 in "selftests/bpf: Add mptcp sched structs"
- ee74ff707e4b: "squashed" patch 6/9 in "bpf: Add bpf_mptcp_sched_kfunc_set"
- 813537758eb5: "squashed" patch 7/9 in "selftests/bpf: Add bpf_first scheduler"
- 762c5b194975: "squashed" patch 8/9 in "selftests/bpf: Add bpf_bkup scheduler"
- 5db53c2d3ef6: "squashed" patch 9/9 in "selftests/bpf: Add bpf_rr scheduler"
- Results: ae9c4d4cf58f..0814e9ff0a94 (export)
- 7b3058027867: "squashed" in "mptcp: refactor push_pending logic"
- Results: 0814e9ff0a94..d98411bcedd1 (export)
Tests are now in progress:
https://cirrus-ci.com/github/multipath-tcp/mptcp_net-next/export/20221118T145003
Cheers,
Matt
Signature · 2 lines
Tessares | Belgium | Hybrid Access Solutions www.tessares.net
Quoted 24 lines · Matthieu Baerts
Hi Geliang, Mat,
Quoted 5 lines · Geliang Tang
v19:
- update patch 1.
- split "BPF redundant scheduler" into two parts.
- rebased on "export/20221111T055536".Quoted 10 lines
Geliang Tang (9):
mptcp: refactorpush_pendinglogic
mptcp: droplast_sndandMPTCP_RESET_SCHEDULER
mptcp: addsched_data_set_contextshelper Squash to "mptcp: add structmptcp_sched_ops" Squash to "bpf: Addbpf_mptcp_sched_ops" Squash to "bpf: Addbpf_mptcp_sched_kfunc_set" Squash to "selftests/bpf: Addbpf_firstscheduler" Squash to "selftests/bpf: Addbpf_bkupscheduler" Squash to "selftests/bpf: Addbpf_rrscheduler"Thank you for the patches and the reviews!
I just applied them (+ an additional squash-to patch) in our tree (feat. for other trees (bpf)) with Mat's RvB tag.
Quoted 7 lines
Patches 1/9 and 2/9 have been added at the beginning. Patch 3/9 depends on "mptcp: add struct
mptcp_sched_ops" and it has been applied just before "bpf: Addbpf_mptcp_sched_ops" as mentioned by Geliang.Please also note that patch 5/9 has been squashed in "selftests/bpf: Add mptcp sched structs" instead of "bpf: Add
bpf_mptcp_sched_ops. Do not hesitate to tell me if something is not OK!
Yes, it should be squashed in "selftests/bpf: Add mptcp sched structs". Thanks for fix this.
-Geliang
Quoted 33 lines
New patches for t/upstream:
-602fba0955d1: mptcp: refactorpush_pendinglogic
-343f8eed9abb: mptcp: droplast_sndandMPTCP_RESET_SCHEDULER
- Results:69720c6cd168..ac857086fdb8(export)-
65e9504361f6: mptcp: addsched_data_set_contextshelper
- Results:ac857086fdb8..ae9c4d4cf58f(export)-
82721c73112e: "squashed" patch 4/9 in "mptcp: add structmptcp_sched_ops"
-a7c2367dd1d2: "squashed" patch 5/9 in "selftests/bpf: Add mptcp sched structs"
-ee74ff707e4b: "squashed" patch 6/9 in "bpf: Addbpf_mptcp_sched_kfunc_set"
-813537758eb5: "squashed" patch 7/9 in "selftests/bpf: Addbpf_firstscheduler"
-762c5b194975: "squashed" patch 8/9 in "selftests/bpf: Addbpf_bkupscheduler"
-5db53c2d3ef6: "squashed" patch 9/9 in "selftests/bpf: Addbpf_rrscheduler"
- Results:ae9c4d4cf58f..0814e9ff0a94(export)-
7b3058027867: "squashed" in "mptcp: refactorpush_pendinglogic"
- Results:0814e9ff0a94..d98411bcedd1(export)Tests are now in progress:
https://cirrus-ci.com/github/multipath-tcp/mptcp_net-next/export/20221118T145003
Cheers,
MattSignature · 2 lines
Tessares | Belgium | Hybrid Access Solutions www.tessares.net
Re: [PATCH mptcp-next v19 3/9] mptcp: add sched_data_set_contexts helper
In reply to Geliang TangHi Geliang, Mat,
Quoted 3 lines · Geliang Tang
Add a new helper
mptcp_sched_data_set_contexts()to set the subflow pointers array in structmptcp_sched_data. It will be invoked by the BPF schedulers to export the subflow pointers to the BPF contexts.
I have an issue with the CI and this commit: it adds a new helper which is not used before before "selftests/bpf: Add mptcp sched structs" commit.
We then have this warning when compiling with W=1:
$ make W=1 net/mptcp/sched.o net/mptcp/sched.c:97:6: warning: symbol 'mptcp_sched_data_set_contexts' was not declared. Should it be static?
I'm not sure what to do:
- modify the script to parse error and ignore it (I would prefer to avoid case per case)
- move the commit after "selftests/bpf: Add mptcp sched structs" and directly use it (extern + kfunc)
- squash it elsewhere?
- add a patch like that:
------------------- 8< -------------------
net/mptcp/sched.c +0 -0
@@ -94,6 +94,8 @@ void mptcp_subflow_set_scheduled(struct
mptcp_subflow_context *subflow,
WRITE_ONCE(subflow->scheduled, scheduled);
}
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Wmissing-prototypes"
void mptcp_sched_data_set_contexts(const struct mptcp_sock *msk,
struct mptcp_sched_data *data)
{
@@ -112,3 +114,4 @@ void mptcp_sched_data_set_contexts(const struct mptcp_sock *msk,
for (; i < MPTCP_SUBFLOWS_MAX; i++)
data->contexts[i] = NULL; }
+#pragma GCC diagnostic pop
------------------- 8< -------------------
But well... :-)
I suggest to apply this patch for the moment to avoid errors with all new patches and check later which solution to use.
Cheers,
Matt
Signature · 2 lines
Tessares | Belgium | Hybrid Access Solutions www.tessares.net
Re: [PATCH mptcp-next v19 3/9] mptcp: add sched_data_set_contexts helper
Quoted 43 lines · Matthieu Baerts
Hi Geliang, Mat,
Quoted 3 lines · Geliang Tang
Add a new helper
mptcp_sched_data_set_contexts()to set the subflow pointers array in structmptcp_sched_data. It will be invoked by the BPF schedulers to export the subflow pointers to the BPF contexts.I have an issue with the CI and this commit: it adds a new helper which is not used before before "selftests/bpf: Add mptcp sched structs" commit.
We then have this warning when compiling with W=1:
$ make W=1 net/mptcp/sched.o net/mptcp/sched.c:97:6: warning: symbol 'mptcp_sched_data_set_contexts' was not declared. Should it be static?I'm not sure what to do:
- modify the script to parse error and ignore it (I would prefer to avoid case per case)
- move the commit after "selftests/bpf: Add mptcp sched structs" and directly use it (extern + kfunc)
- squash it elsewhere?
- add a patch like that:------------------- 8< -------------------
net/mptcp/sched.c +2
-0@@ -94,6 +94,8 @@ void mptcp_subflow_set_scheduled(struct mptcp_subflow_context *subflow,WRITE_ONCE (subflow->scheduled, scheduled); } +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wmissing-prototypes"void mptcp_sched_data_set_contexts (const struct mptcp_sock *msk,struct mptcp_sched_data *data){
net/mptcp/sched.c +0
-0@@ -112,3 +114,4 @@ void mptcp_sched_data_set_contexts(const structmptcp_sock *msk, for (; i < MPTCP_SUBFLOWS_MAX; i++) data->contexts[i] =NULL ;
Quoted 3 lines
}
+#pragma GCC diagnostic pop
------------------- 8< -------------------
One more option:
Add a declaration in net/mptcp/protocol.h
Quoted 5 lines
But well... :-)
I suggest to apply this patch for the moment to avoid errors with all new patches and check later which solution to use.
This is fine with me, if you'd rather wait for a squash-to patch.
--
Mat Martineau
Intel
Re: [PATCH mptcp-next v19 3/9] mptcp: add sched_data_set_contexts helper
Quoted 55 lines · Mat Martineau
Quoted 43 lines · Matthieu Baerts
Hi Geliang, Mat,
Quoted 3 lines · Geliang Tang
Add a new helper
mptcp_sched_data_set_contexts()to set the subflow pointers array in structmptcp_sched_data. It will be invoked by the BPF schedulers to export the subflow pointers to the BPF contexts.I have an issue with the CI and this commit: it adds a new helper which is not used before before "selftests/bpf: Add mptcp sched structs" commit.
We then have this warning when compiling with W=1:
$ make W=1 net/mptcp/sched.o net/mptcp/sched.c:97:6: warning: symbol 'mptcp_sched_data_set_contexts' was not declared. Should it be static?I'm not sure what to do:
- modify the script to parse error and ignore it (I would prefer to avoid case per case)
- move the commit after "selftests/bpf: Add mptcp sched structs" and directly use it (extern + kfunc)
- squash it elsewhere?
- add a patch like that:------------------- 8< -------------------
net/mptcp/sched.c +2
-0@@ -94,6 +94,8 @@ void mptcp_subflow_set_scheduled(struct mptcp_subflow_context *subflow,WRITE_ONCE (subflow->scheduled, scheduled); } +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wmissing-prototypes"void mptcp_sched_data_set_contexts (const struct mptcp_sock *msk,struct mptcp_sched_data *data){
net/mptcp/sched.c +0
-0@@ -112,3 +114,4 @@ void mptcp_sched_data_set_contexts(const structmptcp_sock *msk, for (; i < MPTCP_SUBFLOWS_MAX; i++) data->contexts[i] =NULL ;Quoted 3 lines
}
+#pragma GCC diagnostic pop
------------------- 8< -------------------One more option:
Add a declaration in
net/mptcp/protocol.hQuoted 1 line
Quoted 6 lines
Quoted 4 lines
But well... :-)
I suggest to apply this patch for the moment to avoid errors with all new patches and check later which solution to use.
This is fine with me, if you'd rather wait for a squash-to patch.
Hi Mat, Matt,
I just sent a squash-to patch to fix this.
Thanks,
-Geliang
Quoted 4 lines
--
Mat Martineau
Intel
Quoted 15 lines · Matthieu Baerts
Hi Geliang, Mat,
Quoted 12 lines · Geliang Tang
v19:
- update patch 1.
- split "BPF redundant scheduler" into two parts.
- rebased on "export/20221111T055536".Geliang Tang (9):
mptcp: refactorpush_pendinglogic
mptcp: droplast_sndandMPTCP_RESET_SCHEDULER
mptcp: addsched_data_set_contextshelper
Squash to "mptcp: add structmptcp_sched_ops"
Squash to "bpf: Addbpf_mptcp_sched_ops"
Squash to "bpf: Addbpf_mptcp_sched_kfunc_set"
Hi Matt,
The commit logs in these three patches become mismatched since the squashed code, please update them for me:
'''
mptcp: add struct mptcp_sched_ops
This patch defines struct mptcp_sched_ops, which has three struct members, name, owner and list, and four function pointers: init(), release(), data_init() and get_subflow().
The scheduler functions data_init() and get_subflow() have a struct mptcp_sched_data parameter, which contains a reinject flag and a mptcp_subflow_context array.
Add the scheduler registering, unregistering and finding functions to add, delete and find a packet scheduler on the global list mptcp_sched_list.
bpf: Add bpf_mptcp_sched_kfunc_set
This patch adds a new struct btf_kfunc_id_set for MPTCP scheduler. Add mptcp_subflow_set_scheduled() and mptcp_sched_data_set_contexts() helpers into this id_set, and register it in bpf_mptcp_sched_kfunc_init() to make sure these helpers can be accessed from the BPF context.
selftests/bpf: Add mptcp sched structs
This patch adds three MPTCP scheduler structures: struct mptcp_sched_ops, struct mptcp_sched_data and struct mptcp_subflow_context; and exports mptcp_subflow_set_scheduled() and mptcp_sched_data_set_contexts() helpers for bpf selftests.
'''
Thanks,
-Geliang
Quoted 49 lines
Quoted 3 lines
Squash to "selftests/bpf: Add
bpf_firstscheduler" Squash to "selftests/bpf: Addbpf_bkupscheduler" Squash to "selftests/bpf: Addbpf_rrscheduler"Thank you for the patches and the reviews!
I just applied them (+ an additional squash-to patch) in our tree (feat. for other trees (bpf)) with Mat's RvB tag.
Patches 1/9 and 2/9 have been added at the beginning. Patch 3/9 depends on "mptcp: add struct
mptcp_sched_ops" and it has been applied just before "bpf: Addbpf_mptcp_sched_ops" as mentioned by Geliang.Please also note that patch 5/9 has been squashed in "selftests/bpf: Add mptcp sched structs" instead of "bpf: Add
bpf_mptcp_sched_ops. Do not hesitate to tell me if something is not OK!New patches for t/upstream:
-602fba0955d1: mptcp: refactorpush_pendinglogic
-343f8eed9abb: mptcp: droplast_sndandMPTCP_RESET_SCHEDULER
- Results:69720c6cd168..ac857086fdb8(export)-
65e9504361f6: mptcp: addsched_data_set_contextshelper
- Results:ac857086fdb8..ae9c4d4cf58f(export)-
82721c73112e: "squashed" patch 4/9 in "mptcp: add structmptcp_sched_ops"
-a7c2367dd1d2: "squashed" patch 5/9 in "selftests/bpf: Add mptcp sched structs"
-ee74ff707e4b: "squashed" patch 6/9 in "bpf: Addbpf_mptcp_sched_kfunc_set"
-813537758eb5: "squashed" patch 7/9 in "selftests/bpf: Addbpf_firstscheduler"
-762c5b194975: "squashed" patch 8/9 in "selftests/bpf: Addbpf_bkupscheduler"
-5db53c2d3ef6: "squashed" patch 9/9 in "selftests/bpf: Addbpf_rrscheduler"
- Results:ae9c4d4cf58f..0814e9ff0a94(export)-
7b3058027867: "squashed" in "mptcp: refactorpush_pendinglogic"
- Results:0814e9ff0a94..d98411bcedd1(export)Tests are now in progress:
https://cirrus-ci.com/github/multipath-tcp/mptcp_net-next/export/20221118T145003
Cheers,
MattSignature · 2 lines
Tessares | Belgium | Hybrid Access Solutions www.tessares.net
Hi Geliang,
Quoted 17 lines · Geliang Tang
Quoted 15 lines · Matthieu Baerts
Hi Geliang, Mat,
Quoted 12 lines · Geliang Tang
v19:
- update patch 1.
- split "BPF redundant scheduler" into two parts.
- rebased on "export/20221111T055536".Geliang Tang (9):
mptcp: refactorpush_pendinglogic
mptcp: droplast_sndandMPTCP_RESET_SCHEDULER
mptcp: addsched_data_set_contextshelper
Squash to "mptcp: add structmptcp_sched_ops"
Squash to "bpf: Addbpf_mptcp_sched_ops"
Squash to "bpf: Addbpf_mptcp_sched_kfunc_set"
Quoted 4 lines
Hi Matt,
The commit logs in these three patches become mismatched since the squashed code, please update them for me:
Good catch!
Sure, just did:
New patches for t/upstream:
- 8685053d6d06: tg:msg: update as requested by Geliang
- b81ee295d042: tg:msg: update as requested by Geliang
- e399a35fdc24: tg:msg: update as requested by Geliang
- Results: 6d7665253eef..8ef79d8e4dfb (export)
Tests are now in progress:
https://cirrus-ci.com/github/multipath-tcp/mptcp_net-next/export/20221214T173325
Cheers,
Matt
Signature · 2 lines
Tessares | Belgium | Hybrid Access Solutions www.tessares.net