bpf qdisc
Subsystems (8)
v5 cover bpf qdisc
Hi all,
This patchset aims to support implementing qdisc using bpf struct_ops. This version takes a step back and only implements the minimum support for bpf qdisc. 1) support of adding skb to bpf_list and bpf_rbtree directly and 2) classful qdisc are deferred to future patchsets. In addition, we only allow attaching bpf qdisc to root or mq for now. This is to prevent accidentally breaking exisiting classful qdiscs that rely on data in a child qdisc. This limit may be lifted in the future after careful inspection.
* Overview *
This series supports implementing qdisc using bpf struct_ops. bpf qdisc aims to be a flexible and easy-to-use infrastructure that allows users to quickly experiment with different scheduling algorithms/policies. It only requires users to implement core qdisc logic using bpf and implements the mundane part for them. In addition, the ability to easily communicate between qdisc and other components will also bring new opportunities for new applications and optimizations.
* struct_ops changes *
To make struct_ops works better with bpf qdisc, two new changes are introduced to bpf specifically for struct_ops programs. Frist, we introduce "__ref" postfix for arguments in stub functions in patch 1-2. It allows Qdisc_ops->enqueue to acquire an unique referenced kptr to the skb argument. Through the reference object tracking mechanism in the verifier, we can make sure that the acquired skb will be either enqueued or dropped. Besides, no duplicate references can be acquired. Then, we allow a referenced kptr to be returned from struct_ops programs so that we can return an skb naturally. This is done and tested in patch 3 and 4.
* Performance of bpf qdisc *
This patchset includes two qdisc examples, bpf_fifo and bpf_fq, for __testing__ purposes. For performance test, we compare selftests and their kernel counterparts to give you a sense of the performance of qdisc implemented in bpf.
The implementation of bpf_fq is fairly complex and slightly different from fq so later we only compare the two fifo qdiscs. bpf_fq implements a scheduling algorithm similar to fq before commit 29f834aa326e ("net_sched:sch_fq: add 3 bands and WRR scheduling") was introduced. bpf_fifo uses a single bpf_list as a queue instead of three queues for different priorities in pfifo_fast. The time complexity of fifo however should be similar since the queue selection time is negligible.
Test setup:
client -> qdisc -------------> server
~~~~~~~~~~~~~~~ ~~~~~~
nested VM1 @ DC1 VM2 @ DC2Throghput: iperf3 -t 600, 5 times
Qdisc Average (GBits/sec)
---------- -------------------
pfifo_fast 12.52 ± 0.26
bpf_fifo 11.72 ± 0.32
fq 10.24 ± 0.13
bpf_fq 11.92 ± 0.64 Latency: sockperf pp --tcp -t 600, 5 times
Qdisc Average (usec)
---------- --------------
pfifo_fast 244.58 ± 7.93
bpf_fifo 244.92 ± 15.22
fq 234.30 ± 19.25
bpf_fq 221.34 ± 10.76Looking at the two fifo qdiscs, the 6.4% drop in throughput in the bpf implementatioin is consistent with previous observation (v8 throughput test on a loopback device). This should be able to be mitigated by supporting adding skb to bpf_list or bpf_rbtree directly in the future.
* Clean up skb in bpf qdisc during reset *
The current implementation relies on bpf qdisc implementors to correctly release skbs in queues (bpf graphs or maps) in .reset, which might not be a safe thing to do. The solution as Martin has suggested would be supporting private data in struct_ops. This can also help simplifying implementation of qdisc that works with mq. For examples, qdiscs in the selftest mostly use global data. Therefore, even if user add multiple qdisc instances under mq, they would still share the same queue.
---
v5:
* Rebase to bpf-next/master
* Remove prerequisite bpf patches that has landed
* Add Acked-by from Cong
v4:
* Rebase to bpf-next/master
* Move the root/mq attaching check to bpf_qdisc.c
(now patch 15)
* Replace netdevsim with veth for attaching mq
v3:
* Rebase to bpf-next/master
* Remove the requirement in the verifier that "__ref arguments must acquire ref_obj_id first" by making each prog keeping a copy of arg_ctx_info and saving ref_obj_id in it.
* Generalize prog_ops_moff() to work with any struct_op (now called
bpf_struct_ops_prog_moff())
* Use bpf_struct_ops_prog_moff(prog) instead of
prog->aux->attach_func_name to infer the ops of a program * Limit attach to root and mq for now and add corresponding selftests
* Simplify qdisc selftests with network_helper
* Fix fq_remove_flow() not deleting the stashed flow
v2: Rebase to bpf-next/master
Patch 1-4
Remove the use of ctx_arg_info->ref_obj_id when acquiring referenced kptr from struct_ops arg Improve type comparison when checking kptr return from struct_ops
Simplify selftests with test_loader and nomerge attribute
Patch 5
Remove redundant checks in qdisc_init
Disallow tail_call
Patch 6
Improve kfunc ops availabilty filter by
i) Checking struct_ops->type
ii) Defining op-specific kfunc set
Patch 7
Search and add bpf_kfunc_desc after gen_prologue/epilogue
Patch 8
Use gen_prologue/epilogue to init/cancel watchdog timer
Patch 12
Mark read-only func arg and struct member const in libbpf
v1:
Fix struct_ops referenced kptr acquire/return mechanisms Allow creating dynptr from skb
Add bpf qdisc kfunc filter
Support updating bstats and qstats
Update qdiscs in selftest to update stats
Add gc, handle hash collision and fix bugs in fq_bpf
past RFCs
v9: Drop classful qdisc operations and kfuncs
Drop support of enqueuing skb directly to bpf_rbtree/list
v8: Implement support of bpf qdisc using struct_ops
Allow struct_ops to acquire referenced kptr via argument Allow struct_ops to release and return referenced kptr Support enqueuing sk_buff to bpf_rbtree/list Move examples from samples to selftests
Add a classful qdisc selftest
v7: Reference skb using kptr to sk_buff instead of __sk_buff
Use the new bpf rbtree/link to for skb queues Add reset and init programs
Add a bpf fq qdisc sample
Add a bpf netem qdisc sample
v6: switch to kptr based approach
v5: mv kernel/bpf/skb_map.c net/core/skb_map.c
implement flow map as map-in-map
rename bpf_skb_tc_classify() and move it to net/sched/cls_api.c clean up eBPF qdisc program context
v4: get rid of PIFO, use rbtree directly
v3: move priority queue from sch_bpf to skb map
introduce skb map and its helpers
introduce bpf_skb_classify()
use netdevice notifier to reset skb's
Rebase on latest bpf-next
v2: Rebase on latest net-next
Make the code more complete (but still incomplete)
Amery Hung (13):
bpf: Prepare to reuse get_ctx_arg_idx
bpf: Generalize finding member offset of struct_ops prog
bpf: net_sched: Support implementation of Qdisc_ops in bpf
bpf: net_sched: Add basic bpf qdisc kfuncs
bpf: net_sched: Add a qdisc watchdog timer
bpf: net_sched: Support updating bstats
bpf: net_sched: Support updating qstats
bpf: net_sched: Allow writing to more Qdisc members
bpf: net_sched: Disable attaching bpf qdisc to non root
libbpf: Support creating and destroying qdisc selftests/bpf: Add a basic fifo qdisc test selftests/bpf: Add a bpf fq qdisc to selftest selftests/bpf: Test attaching bpf qdisc to mq and non root
include/linux/bpf.h | 2 + include/linux/btf.h | 1 + kernel/bpf/bpf_struct_ops.c | 15 + kernel/bpf/btf.c | 6 +- net/ipv4/bpf_tcp_ca.c | 23 +- net/sched/Kconfig | 12 + net/sched/Makefile | 1 + net/sched/bpf_qdisc.c | 465 ++++++++++++ net/sched/sch_api.c | 7 +- net/sched/sch_generic.c | 3 +- tools/lib/bpf/libbpf.h | 5 +- tools/lib/bpf/netlink.c | 20 +- tools/testing/selftests/bpf/config | 2 + .../selftests/bpf/prog_tests/bpf_qdisc.c | 178 +++++ .../selftests/bpf/progs/bpf_qdisc_common.h | 27 + .../selftests/bpf/progs/bpf_qdisc_fifo.c | 117 +++ .../selftests/bpf/progs/bpf_qdisc_fq.c | 718 ++++++++++++++++++ 17 files changed, 1570 insertions(+), 32 deletions(-) create mode 100644 net/sched/bpf_qdisc.c create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_qdisc_common.h create mode 100644 tools/testing/selftests/bpf/progs/bpf_qdisc_fifo.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_qdisc_fq.c
Signature · 1 line
2.47.1
PATCH v5 1/13 bpf: Prepare to reuse get_ctx_arg_idx
Rename get_ctx_arg_idx to bpf_ctx_arg_idx, and allow others to call it. No functional change.
--- include/linux/btf.h | 1 + kernel/bpf/btf.c | 6 +++--- 2 files changed, 4 insertions(+), 3 deletions(-)
include/linux/btf.h +1 -0
@@ -522,6 +522,7 @@ bool btf_param_match_suffix(const struct btf *btf, const char *suffix);int btf_ctx_arg_offset (const struct btf *btf,const struct btf_type *func_proto,u32 arg_no); +u32 btf_ctx_arg_idx (struct btf *btf,const struct btf_type *func_proto,int off);struct bpf_verifier_log ;
kernel/bpf/btf.c +3 -3
@@ -6369,8 +6369,8 @@ static bool is_int_ptr(struct btf *btf, const struct btf_type *t)return btf_type_is_int (t); } -static u32 get_ctx_arg_idx (struct btf *btf,const struct btf_type *func_proto, -int off) +u32 btf_ctx_arg_idx (struct btf *btf,const struct btf_type *func_proto, +int off) {const struct btf_param *args;const struct btf_type *t;
@@ -6649,7 +6649,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, tname, off);return false; } - arg =get_ctx_arg_idx (btf, t, off); + arg =btf_ctx_arg_idx (btf, t, off); args = (const struct btf_param *)(t +1 );/* if (t == NULL) Fall back to default BPF prog with * MAX_BPF_FUNC_REG_ARGS u64 arguments.
Signature · 1 line
2.47.1
PATCH v5 2/13 bpf: Generalize finding member offset of struct_ops prog
In reply to Amery HungGeneralize prog_ops_moff() so that we can use it to retrieve a struct_ops program's offset for different ops.
--- include/linux/bpf.h | 1 + kernel/bpf/bpf_struct_ops.c | 13 +++++++++++++ net/ipv4/bpf_tcp_ca.c | 23 ++--------------------- 3 files changed, 16 insertions(+), 21 deletions(-)
include/linux/bpf.h +1 -0
@@ -1908,6 +1908,7 @@ static inline void bpf_module_put(const void *data, struct module *owner)module_put (owner); }int bpf_struct_ops_link_create (union bpf_attr *attr); +u32 bpf_struct_ops_prog_moff (const struct bpf_prog *prog);#ifdef CONFIG_NET /* Define it here to avoid the use of forward declaration */
kernel/bpf/bpf_struct_ops.c +13 -0
@@ -1387,3 +1387,16 @@ void bpf_map_struct_ops_info_fill(struct bpf_map_info *info, struct bpf_map *map info->btf_vmlinux_id =btf_obj_id (st_map->btf); } + +u32 bpf_struct_ops_prog_moff (const struct bpf_prog *prog) +{ +const struct btf_member *m; +const struct btf_type *t; +u32 midx; + + t =btf_type_by_id (prog->aux->attach_btf, prog->aux->attach_btf_id); + midx = prog->expected_attach_type; + m = &btf_type_member (t)[midx]; + +return __btf_member_bit_offset (t, m) /8 ; +}
net/ipv4/bpf_tcp_ca.c +2 -21
@@ -16,7 +16,6 @@ static struct bpf_struct_ops bpf_tcp_congestion_ops;static const struct btf_type *tcp_sock_type;static u32 tcp_sock_id, sock_id; -static const struct btf_type *tcp_congestion_ops_type;static int bpf_tcp_ca_init (struct btf *btf) {
@@ -33,11 +32,6 @@ static int bpf_tcp_ca_init(struct btf *btf) tcp_sock_id = type_id; tcp_sock_type =btf_type_by_id (btf, tcp_sock_id); - type_id =btf_find_by_name_kind (btf,"tcp_congestion_ops" ,BTF_KIND_STRUCT ); -if (type_id <0 ) -return -EINVAL ; - tcp_congestion_ops_type =btf_type_by_id (btf, type_id); -return 0 ; }
@@ -135,19 +129,6 @@ static const struct bpf_func_proto bpf_tcp_send_ack_proto = { .arg2_type = ARG_ANYTHING, }; -static u32 prog_ops_moff (const struct bpf_prog *prog) -{ -const struct btf_member *m; -const struct btf_type *t; -u32 midx; - - midx = prog->expected_attach_type; - t = tcp_congestion_ops_type; - m = &btf_type_member (t)[midx]; - -return __btf_member_bit_offset (t, m) /8 ; -} -static const struct bpf_func_proto *bpf_tcp_ca_get_func_proto (enum bpf_func_id func_id,const struct bpf_prog *prog)
@@ -166,7 +147,7 @@ bpf_tcp_ca_get_func_proto(enum bpf_func_id func_id,* setsockopt() to make further changes which * may potentially allocate new resources. */ -if (prog_ops_moff (prog) != +if (bpf_struct_ops_prog_moff (prog) != offsetof(struct tcp_congestion_ops , release))return &bpf_sk_setsockopt_proto;return NULL ;
@@ -177,7 +158,7 @@ bpf_tcp_ca_get_func_proto(enum bpf_func_id func_id,* The bpf-tcp-cc already has a more powerful way * to read tcp_sock from the PTR_TO_BTF_ID. */ -if (prog_ops_moff (prog) != +if (bpf_struct_ops_prog_moff (prog) != offsetof(struct tcp_congestion_ops , release))return &bpf_sk_getsockopt_proto;return NULL ;
Signature · 1 line
2.47.1
PATCH v5 3/13 bpf: net_sched: Support implementation of Qdisc_ops in bpf
In reply to Amery HungFrom: Amery Hung <[email protected]>
Enable users to implement a classless qdisc using bpf. The last few patches in this series has prepared struct_ops to support core operators in Qdisc_ops. The recent advancement in bpf such as allocated objects, bpf list and bpf rbtree has also provided powerful and flexible building blocks to realize sophisticated scheduling algorithms. Therefore, in this patch, we start allowing qdisc to be implemented using bpf struct_ops. Users can implement Qdisc_ops.{enqueue, dequeue, init, reset, and .destroy in Qdisc_ops in bpf and register the qdisc dynamically into the kernel.
--- net/sched/Kconfig | 12 +++ net/sched/Makefile | 1 + net/sched/bpf_qdisc.c | 210 ++++++++++++++++++++++++++++++++++++++++ net/sched/sch_api.c | 7 +- net/sched/sch_generic.c | 3 +- 5 files changed, 229 insertions(+), 4 deletions(-) create mode 100644 net/sched/bpf_qdisc.c
net/sched/Kconfig +12 -0
@@ -403,6 +403,18 @@ config NET_SCH_ETS If unsure, say N. +config NET_SCH_BPF + bool "BPF-based Qdisc" + depends on BPF_SYSCALL && BPF_JIT && DEBUG_INFO_BTF + help + This option allows BPF-based queueing disiplines. With BPF struct_ops, + users can implement supported operators in Qdisc_ops using BPF programs. + The queue holding skb can be built with BPF maps or graphs. + + Say Y here if you want to use BPF-based Qdisc. + + If unsure, say N. + menuconfig NET_SCH_DEFAULT bool "Allow override default queue discipline" help
net/sched/Makefile +1 -0
@@ -62,6 +62,7 @@ obj-$(CONFIG_NET_SCH_FQ_PIE) += sch_fq_pie.o obj-$(CONFIG_NET_SCH_CBS) += sch_cbs.o obj-$(CONFIG_NET_SCH_ETF) += sch_etf.o obj-$(CONFIG_NET_SCH_TAPRIO) += sch_taprio.o +obj-$(CONFIG_NET_SCH_BPF) += bpf_qdisc.o obj-$(CONFIG_NET_CLS_U32) += cls_u32.o obj-$(CONFIG_NET_CLS_ROUTE4) += cls_route.o
net/sched/bpf_qdisc.c new file +210 -0
210 lines not on this page · raw message
net/sched/sch_api.c +4 -3
@@ -25,6 +25,7 @@#include <linux/hrtimer.h> #include <linux/slab.h> #include <linux/hashtable.h> +#include <linux/bpf.h> #include <net/net_namespace.h> #include <net/sock.h>
@@ -358,7 +359,7 @@ static struct Qdisc_ops *qdisc_lookup_ops(struct nlattr *kind)read_lock (&qdisc_mod_lock);for (q = qdisc_base; q; q = q->next) {if (nla_strcmp (kind, q->id) ==0 ) { -if (!try_module_get (q->owner)) +if (!bpf_try_module_get (q, q->owner)) q =NULL ;break ; }
@@ -1287,7 +1288,7 @@ static struct Qdisc *qdisc_create(struct net_device *dev,/* We will try again qdisc_lookup_ops, * so don't keep a reference. */ -module_put (ops->owner); +bpf_module_put (ops, ops->owner); err = -EAGAIN ; goto err_out; }
@@ -1398,7 +1399,7 @@ static struct Qdisc *qdisc_create(struct net_device *dev,netdev_put (dev, &sch->dev_tracker);qdisc_free (sch); err_out2: -module_put (ops->owner); +bpf_module_put (ops, ops->owner); err_out: *errp = err;return NULL ;
net/sched/sch_generic.c +2 -1
@@ -24,6 +24,7 @@#include <linux/if_vlan.h> #include <linux/skb_array.h> #include <linux/if_macvlan.h> +#include <linux/bpf.h> #include <net/sch_generic.h> #include <net/pkt_sched.h> #include <net/dst.h>
@@ -1078,7 +1079,7 @@ static void __qdisc_destroy(struct Qdisc *qdisc) ops->destroy (qdisc);lockdep_unregister_key (&qdisc->root_lock_key); -module_put (ops->owner); +bpf_module_put (ops, ops->owner);netdev_put (dev, &qdisc->dev_tracker);trace_qdisc_destroy (qdisc);
Signature · 1 line
2.47.1
PATCH v5 4/13 bpf: net_sched: Add basic bpf qdisc kfuncs
In reply to Amery HungFrom: Amery Hung <[email protected]>
Add basic kfuncs for working on skb in qdisc.
Both bpf_qdisc_skb_drop() and bpf_kfree_skb() can be used to release a reference to an skb. However, bpf_qdisc_skb_drop() can only be called in .enqueue where a to_free skb list is available from kernel to defer the release. bpf_kfree_skb() should be used elsewhere. It is also used in bpf_obj_free_fields() when cleaning up skb in maps and collections.
bpf_skb_get_hash() returns the flow hash of an skb, which can be used to build flow-based queueing algorithms.
Finally, allow users to create read-only dynptr via bpf_dynptr_from_skb().
--- include/linux/bpf.h | 1 + kernel/bpf/bpf_struct_ops.c | 2 + net/sched/bpf_qdisc.c | 93 ++++++++++++++++++++++++++++++++++++- 3 files changed, 95 insertions(+), 1 deletion(-)
include/linux/bpf.h +1 -0
@@ -1826,6 +1826,7 @@ struct bpf_struct_ops {void *cfi_stubs;struct module *owner;const char *name; +const struct btf_type *type;struct btf_func_model func_models[BPF_STRUCT_OPS_MAX_NR_MEMBERS ]; };
kernel/bpf/bpf_struct_ops.c +2 -0
@@ -461,6 +461,8 @@ int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc, goto errout; } + st_ops->type = t; +return 0 ; errout:
net/sched/bpf_qdisc.c +92 -1
106 lines not on this page · raw message
Signature · 1 line
2.47.1
Re: [PATCH bpf-next v5 04/13] bpf: net_sched: Add basic bpf qdisc kfuncs
Quoted 28 lines · Amery Hung
From: Amery Hung <[email protected]>
Add basic kfuncs for working on skb in qdisc.
Both
bpf_qdisc_skb_drop()andbpf_kfree_skb()can be used to release a reference to an skb. However,bpf_qdisc_skb_drop()can only be called in .enqueue where ato_freeskb list is available from kernel to defer the release.bpf_kfree_skb()should be used elsewhere. It is also used inbpf_obj_free_fields()when cleaning up skb in maps and collections.
bpf_skb_get_hash()returns the flow hash of an skb, which can be used to build flow-based queueing algorithms.Finally, allow users to create read-only dynptr via
bpf_dynptr_from_skb().Signed-off-by Amery Hung <[email protected]>--- include/linux/bpf.h | 1 + kernel/bpf/bpf_struct_ops.c | 2 + net/sched/bpf_qdisc.c | 93 ++++++++++++++++++++++++++++++++++++- 3 files changed, 95 insertions(+), 1 deletion(-)include/linux/bpf.h +0
-0
include/linux/bpf.h struct bpf_struct_ops { · line 1831
Quoted 6 lines
include/linux/bpf.h +1
-0@@ -1826,6 +1826,7 @@ struct bpf_struct_ops {void *cfi_stubs;struct module *owner;const char *name; +const struct btf_type *type;struct btf_func_model func_models[BPF_STRUCT_OPS_MAX_NR_MEMBERS ]; };
there is an alternative to this...
Quoted 7 lines
include/linux/bpf.h +7
-0+static int bpf_qdisc_kfunc_filter (const struct bpf_prog *prog,u32 kfunc_id) +{ +if (bpf_Qdisc_ops.type !=btf_type_by_id (prog->aux->attach_btf, + prog->aux->attach_btf_id)) +return 0 ; + +/* Skip the check when prog->attach_func_name is not yet available
include/linux/bpf.h struct bpf_struct_ops {
Quoted 10 lines
include/linux/bpf.h +10
-0+* during check_cfg(). +*/ +if (!btf_id_set8_contains (&qdisc_kfunc_ids, kfunc_id) || + !prog->aux->attach_func_name) +return 0 ; + +if (bpf_struct_ops_prog_moff (prog) == offsetof(struct Qdisc_ops , enqueue)) { +if (btf_id_set_contains (&qdisc_enqueue_kfunc_set, kfunc_id)) +return 0 ; + }
Instead of logic in this patch and patch 2,
I think it's cleaner to do:
https://lore.kernel.org/all/AM6PR03MB50804BE76B752350307B6B4C99C22@AM6PR03MB5080.eurprd03.prod.outlook.com/
then in this patch it will be
if (prog->aux->st_ops != &bpf_Qdisc_ops)
and instead of unchecked array accesses in bpf_struct_ops_prog_moff() it will be prog->aux->attach_st_ops_member_off
Also see flag based approach in Juntong's patch 3+4.
imo it looks cleaner (more extensible with more checks per st_ops hook) than offsetof() approach above.
Re: [PATCH bpf-next v5 04/13] bpf: net_sched: Add basic bpf qdisc kfuncs
Quoted 63 lines · Alexei Starovoitov
Quoted 28 lines · Amery Hung
From: Amery Hung <[email protected]>
Add basic kfuncs for working on skb in qdisc.
Both
bpf_qdisc_skb_drop()andbpf_kfree_skb()can be used to release a reference to an skb. However,bpf_qdisc_skb_drop()can only be called in .enqueue where ato_freeskb list is available from kernel to defer the release.bpf_kfree_skb()should be used elsewhere. It is also used inbpf_obj_free_fields()when cleaning up skb in maps and collections.
bpf_skb_get_hash()returns the flow hash of an skb, which can be used to build flow-based queueing algorithms.Finally, allow users to create read-only dynptr via
bpf_dynptr_from_skb().Signed-off-by Amery Hung <[email protected]>--- include/linux/bpf.h | 1 + kernel/bpf/bpf_struct_ops.c | 2 + net/sched/bpf_qdisc.c | 93 ++++++++++++++++++++++++++++++++++++- 3 files changed, 95 insertions(+), 1 deletion(-)include/linux/bpf.h +0
-0include/linux/bpf.h struct bpf_struct_ops { · line 1831
Quoted 6 lines
include/linux/bpf.h +1
-0@@ -1826,6 +1826,7 @@ struct bpf_struct_ops {void *cfi_stubs;struct module *owner;const char *name; +const struct btf_type *type;struct btf_func_model func_models[BPF_STRUCT_OPS_MAX_NR_MEMBERS ]; };there is an alternative to this...
Quoted 7 lines
include/linux/bpf.h +7
-0+static int bpf_qdisc_kfunc_filter (const struct bpf_prog *prog,u32 kfunc_id) +{ +if (bpf_Qdisc_ops.type !=btf_type_by_id (prog->aux->attach_btf, + prog->aux->attach_btf_id)) +return 0 ; + +/* Skip the check when prog->attach_func_name is not yet available include/linux/bpf.h struct bpf_struct_ops {
Quoted 10 lines
include/linux/bpf.h +10
-0+* during check_cfg(). +*/ +if (!btf_id_set8_contains (&qdisc_kfunc_ids, kfunc_id) || + !prog->aux->attach_func_name) +return 0 ; + +if (bpf_struct_ops_prog_moff (prog) == offsetof(struct Qdisc_ops , enqueue)) { +if (btf_id_set_contains (&qdisc_enqueue_kfunc_set, kfunc_id)) +return 0 ; + }Instead of logic in this patch and patch 2,
I think it's cleaner to do:
https://lore.kernel.org/all/AM6PR03MB50804BE76B752350307B6B4C99C22@AM6PR03MB5080.eurprd03.prod.outlook.com/then in this patch it will be
Quoted 8 lines
if (
prog->aux->st_ops!=&bpf_Qdisc_ops)and instead of unchecked array accesses in
bpf_struct_ops_prog_moff()it will beprog->aux->attach_st_ops_member_offAlso see flag based approach in Juntong's patch 3+4.
imo it looks cleaner (more extensible with more checks perst_opshook) thanoffsetof()approach above.
Thanks for the pointer! I will drop patch 2 and adopt the flag-based kfunc filter.
PATCH v5 5/13 bpf: net_sched: Add a qdisc watchdog timer
In reply to Amery HungFrom: Amery Hung <[email protected]>
Add a watchdog timer to bpf qdisc. The watchdog can be used to schedule the execution of qdisc through kfunc, bpf_qdisc_schedule(). It can be useful for building traffic shaping scheduling algorithm, where the time the next packet will be dequeued is known.
--- net/sched/bpf_qdisc.c | 92 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+)
net/sched/bpf_qdisc.c +92 -0
@@ -8,6 +8,10 @@static struct bpf_struct_ops bpf_Qdisc_ops; +struct bpf_sched_data { +struct qdisc_watchdog watchdog; +}; +struct bpf_sk_buff_ptr {struct sk_buff *skb; };
@@ -111,6 +115,46 @@ static int bpf_qdisc_btf_struct_access(struct bpf_verifier_log *log,return 0 ; } +BTF_ID_LIST (bpf_qdisc_init_prologue_ids) +BTF_ID (func, bpf_qdisc_init_prologue) + +static int bpf_qdisc_gen_prologue (struct bpf_insn *insn_buf,bool direct_write, +const struct bpf_prog *prog) +{ + struct bpf_insn *insn = insn_buf; + + if (bpf_struct_ops_prog_moff(prog) != offsetof(struct Qdisc_ops, init)) + return 0; + + *insn++ = BPF_MOV64_REG(BPF_REG_6, BPF_REG_1); + *insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, 0); + *insn++ = BPF_CALL_KFUNC(0, bpf_qdisc_init_prologue_ids[0]); + *insn++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_6); + *insn++ = prog->insnsi[0]; + + return insn - insn_buf; +} + +BTF_ID_LIST(bpf_qdisc_reset_destroy_epilogue_ids) +BTF_ID(func, bpf_qdisc_reset_destroy_epilogue) + +static int bpf_qdisc_gen_epilogue(struct bpf_insn *insn_buf, const struct bpf_prog *prog, + s16 ctx_stack_off) +{ +struct bpf_insn *insn = insn_buf; + +if (bpf_struct_ops_prog_moff (prog) != offsetof(struct Qdisc_ops , reset) && +bpf_struct_ops_prog_moff (prog) != offsetof(struct Qdisc_ops , destroy)) +return 0 ; + + *insn++ =BPF_LDX_MEM (BPF_DW ,BPF_REG_1 ,BPF_REG_FP , ctx_stack_off); + *insn++ =BPF_LDX_MEM (BPF_DW ,BPF_REG_1 ,BPF_REG_1 ,0 ); + *insn++ =BPF_CALL_KFUNC (0 , bpf_qdisc_reset_destroy_epilogue_ids[0 ]); + *insn++ =BPF_EXIT_INSN (); + +return insn - insn_buf; +} +__bpf_kfunc_start_defs ();/* bpf_skb_get_hash - Get the flow hash of an skb.
@@ -139,6 +183,36 @@ __bpf_kfunc void bpf_qdisc_skb_drop(struct sk_buff *skb,__qdisc_drop (skb, (struct sk_buff **)to_free_list); } +/* bpf_qdisc_watchdog_schedule - Schedule a qdisc to a later time using a timer. +* @sch: The qdisc to be scheduled. +* @expire: The expiry time of the timer. +* @delta_ns: The slack range of the timer. +*/ +__bpf_kfunc void bpf_qdisc_watchdog_schedule(struct Qdisc *sch, u64 expire, u64 delta_ns) +{ +struct bpf_sched_data *q =qdisc_priv (sch); + +qdisc_watchdog_schedule_range_ns (&q->watchdog, expire, delta_ns); +} + +/* bpf_qdisc_init_prologue - Hidden kfunc called in prologue of .init. */ +__bpf_kfunc void bpf_qdisc_init_prologue(struct Qdisc *sch) +{ +struct bpf_sched_data *q =qdisc_priv (sch); + +qdisc_watchdog_init (&q->watchdog, sch); +} + +/* bpf_qdisc_reset_destroy_epilogue - Hidden kfunc called in epilogue of .reset +* and .destroy +*/ +__bpf_kfunc void bpf_qdisc_reset_destroy_epilogue(struct Qdisc *sch) +{ +struct bpf_sched_data *q =qdisc_priv (sch); + +qdisc_watchdog_cancel (&q->watchdog); +} +__bpf_kfunc_end_defs ();BTF_KFUNCS_START (qdisc_kfunc_ids)
@@ -146,6 +220,9 @@ BTF_ID_FLAGS(func, bpf_skb_get_hash, KF_TRUSTED_ARGS) BTF_ID_FLAGS(func, bpf_kfree_skb, KF_RELEASE) BTF_ID_FLAGS(func, bpf_qdisc_skb_drop, KF_RELEASE) BTF_ID_FLAGS(func, bpf_dynptr_from_skb, KF_TRUSTED_ARGS) +BTF_ID_FLAGS(func, bpf_qdisc_watchdog_schedule, KF_TRUSTED_ARGS) +BTF_ID_FLAGS(func, bpf_qdisc_init_prologue, KF_TRUSTED_ARGS) +BTF_ID_FLAGS(func, bpf_qdisc_reset_destroy_epilogue, KF_TRUSTED_ARGS)BTF_KFUNCS_END (qdisc_kfunc_ids)BTF_SET_START (qdisc_common_kfunc_set)
@@ -156,8 +233,13 @@ BTF_SET_END(qdisc_common_kfunc_set) BTF_SET_START(qdisc_enqueue_kfunc_set) BTF_ID(func, bpf_qdisc_skb_drop) +BTF_ID(func, bpf_qdisc_watchdog_schedule) BTF_SET_END(qdisc_enqueue_kfunc_set) +BTF_SET_START(qdisc_dequeue_kfunc_set) +BTF_ID(func, bpf_qdisc_watchdog_schedule) +BTF_SET_END(qdisc_dequeue_kfunc_set) + static int bpf_qdisc_kfunc_filter(const struct bpf_prog *prog, u32 kfunc_id) { if (bpf_Qdisc_ops.type != btf_type_by_id(prog->aux->attach_btf,
@@ -174,6 +256,9 @@ static int bpf_qdisc_kfunc_filter(const struct bpf_prog *prog, u32 kfunc_id)if (bpf_struct_ops_prog_moff (prog) == offsetof(struct Qdisc_ops , enqueue)) {if (btf_id_set_contains (&qdisc_enqueue_kfunc_set, kfunc_id))return 0 ; + }else if (bpf_struct_ops_prog_moff (prog) == offsetof(struct Qdisc_ops , dequeue)) { +if (btf_id_set_contains (&qdisc_dequeue_kfunc_set, kfunc_id)) +return 0 ; }return btf_id_set_contains (&qdisc_common_kfunc_set, kfunc_id) ?0 : -EACCES ;
@@ -189,6 +274,8 @@ static const struct bpf_verifier_ops bpf_qdisc_verifier_ops = { .get_func_proto = bpf_qdisc_get_func_proto, .is_valid_access = bpf_qdisc_is_valid_access, .btf_struct_access = bpf_qdisc_btf_struct_access, + .gen_prologue = bpf_qdisc_gen_prologue, + .gen_epilogue = bpf_qdisc_gen_epilogue, }; static int bpf_qdisc_init_member(const struct btf_type *t,
@@ -204,6 +291,11 @@ static int bpf_qdisc_init_member(const struct btf_type *t, moff =__btf_member_bit_offset (t, member) /8 ;switch (moff) { +case offsetof(struct Qdisc_ops , priv_size): +if (uqdisc_ops->priv_size) +return -EINVAL ; + qdisc_ops->priv_size =sizeof (struct bpf_sched_data ); +return 1 ;case offsetof(struct Qdisc_ops , peek): qdisc_ops->peek = qdisc_peek_dequeued;return 0 ;
Signature · 1 line
2.47.1
PATCH v5 6/13 bpf: net_sched: Support updating bstats
In reply to Amery HungFrom: Amery Hung <[email protected]>
Add a kfunc to update Qdisc bstats when an skb is dequeued. The kfunc is only available in .dequeue programs.
--- net/sched/bpf_qdisc.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
net/sched/bpf_qdisc.c +11 -0
@@ -213,6 +213,15 @@ __bpf_kfunc void bpf_qdisc_reset_destroy_epilogue(struct Qdisc *sch)qdisc_watchdog_cancel (&q->watchdog); } +/* bpf_qdisc_bstats_update - Update Qdisc basic statistics +* @sch: The qdisc from which an skb is dequeued. +* @skb: The skb to be dequeued. +*/ +__bpf_kfunc void bpf_qdisc_bstats_update(struct Qdisc *sch, const struct sk_buff *skb) +{ +bstats_update (&sch->bstats, skb); +} +__bpf_kfunc_end_defs ();BTF_KFUNCS_START (qdisc_kfunc_ids)
@@ -223,6 +232,7 @@ BTF_ID_FLAGS(func, bpf_dynptr_from_skb, KF_TRUSTED_ARGS) BTF_ID_FLAGS(func, bpf_qdisc_watchdog_schedule, KF_TRUSTED_ARGS) BTF_ID_FLAGS(func, bpf_qdisc_init_prologue, KF_TRUSTED_ARGS) BTF_ID_FLAGS(func, bpf_qdisc_reset_destroy_epilogue, KF_TRUSTED_ARGS) +BTF_ID_FLAGS(func, bpf_qdisc_bstats_update, KF_TRUSTED_ARGS)BTF_KFUNCS_END (qdisc_kfunc_ids)BTF_SET_START (qdisc_common_kfunc_set)
@@ -238,6 +248,7 @@ BTF_SET_END(qdisc_enqueue_kfunc_set) BTF_SET_START(qdisc_dequeue_kfunc_set) BTF_ID(func, bpf_qdisc_watchdog_schedule) +BTF_ID(func, bpf_qdisc_bstats_update) BTF_SET_END(qdisc_dequeue_kfunc_set) static int bpf_qdisc_kfunc_filter(const struct bpf_prog *prog, u32 kfunc_id)
Signature · 1 line
2.47.1
PATCH v5 7/13 bpf: net_sched: Support updating qstats
In reply to Amery HungFrom: Amery Hung <[email protected]>
Allow bpf qdisc programs to update Qdisc qstats directly with btf struct access.
--- net/sched/bpf_qdisc.c | 53 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 45 insertions(+), 8 deletions(-)
net/sched/bpf_qdisc.c +45 -8
@@ -36,6 +36,7 @@ bpf_qdisc_get_func_proto(enum bpf_func_id func_id, } } +BTF_ID_LIST_SINGLE(bpf_qdisc_ids, struct, Qdisc) BTF_ID_LIST_SINGLE(bpf_sk_buff_ids, struct, sk_buff)BTF_ID_LIST_SINGLE (bpf_sk_buff_ptr_ids, struct, bpf_sk_buff_ptr)
@@ -60,20 +61,37 @@ static bool bpf_qdisc_is_valid_access(int off, int size,return bpf_tracing_btf_ctx_access (off, size, type, prog, info); } -static int bpf_qdisc_btf_struct_access (struct bpf_verifier_log *log, -const struct bpf_reg_state *reg, -int off,int size) +static int bpf_qdisc_qdisc_access (struct bpf_verifier_log *log, +const struct bpf_reg_state *reg, +int off,int size) { -const struct btf_type *t, *skbt;size_t end; - skbt =btf_type_by_id (reg->btf, bpf_sk_buff_ids[0 ]); - t =btf_type_by_id (reg->btf, reg->btf_id); -if (t != skbt) { -bpf_log (log,"only read is supported\n" ); +switch (off) { + case offsetof(struct Qdisc, qstats) ... offsetofend(struct Qdisc, qstats) - 1: + end = offsetofend(struct Qdisc, qstats); +break ; +default : +bpf_log (log,"no write support to Qdisc at off %d\n" , off); +return -EACCES ; + } + +if (off + size > end) { +bpf_log (log, +"write access at off %d with size %d beyond the member of Qdisc ended at %zu\n" , + off, size, end);return -EACCES ; } +return 0 ; +} + +static int bpf_qdisc_sk_buff_access (struct bpf_verifier_log *log, +const struct bpf_reg_state *reg, +int off,int size) +{ +size_t end; +switch (off) {case offsetof(struct sk_buff , tstamp): end = offsetofend(struct sk_buff, tstamp);
@@ -115,6 +133,25 @@ static int bpf_qdisc_btf_struct_access(struct bpf_verifier_log *log,return 0 ; } +static int bpf_qdisc_btf_struct_access (struct bpf_verifier_log *log, +const struct bpf_reg_state *reg, +int off,int size) +{ +const struct btf_type *t, *skbt, *qdisct; + + skbt =btf_type_by_id (reg->btf, bpf_sk_buff_ids[0 ]); + qdisct =btf_type_by_id (reg->btf, bpf_qdisc_ids[0 ]); + t =btf_type_by_id (reg->btf, reg->btf_id); + +if (t == skbt) +return bpf_qdisc_sk_buff_access (log, reg, off, size); +else if (t == qdisct) +return bpf_qdisc_qdisc_access (log, reg, off, size); + +bpf_log (log,"only read is supported\n" ); +return -EACCES ; +} +BTF_ID_LIST (bpf_qdisc_init_prologue_ids)BTF_ID (func, bpf_qdisc_init_prologue)
Signature · 1 line
2.47.1
Re: [PATCH bpf-next v5 07/13] bpf: net_sched: Support updating qstats
Quoted 24 lines · Amery Hung
From: Amery Hung <[email protected]>
Allow bpf qdisc programs to update Qdisc qstats directly with btf struct access.
Signed-off-by Amery Hung <[email protected]>--- net/sched/bpf_qdisc.c | 53 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 45 insertions(+), 8 deletions(-)net/sched/bpf_qdisc.c +1
-0@@ -36,6 +36,7 @@ bpf_qdisc_get_func_proto(enum bpf_func_id func_id, } } +BTF_ID_LIST_SINGLE(bpf_qdisc_ids, struct, Qdisc) BTF_ID_LIST_SINGLE(bpf_sk_buff_ids, struct, sk_buff)BTF_ID_LIST_SINGLE (bpf_sk_buff_ptr_ids, struct, bpf_sk_buff_ptr)
net/sched/bpf_qdisc.c static bool bpf_qdisc_is_valid_access(int off, int size, · line 66
Quoted 9 lines
net/sched/bpf_qdisc.c +3
-3@@ -60,20 +61,37 @@ static bool bpf_qdisc_is_valid_access(int off, int size,return bpf_tracing_btf_ctx_access (off, size, type, prog, info); } -static int bpf_qdisc_btf_struct_access (struct bpf_verifier_log *log, -const struct bpf_reg_state *reg, -int off,int size) +static int bpf_qdisc_qdisc_access (struct bpf_verifier_log *log, +const struct bpf_reg_state *reg, +int off,int size)
Introducing this func in patch 3 and refactoring in patch 7 ? pls avoid the churn.
squash it ?
if (off + size > end) check wouldn't need to be duplicated. Can get the name of struct from btf for bpf_log() purpose.
Quoted 65 lines
net/sched/bpf_qdisc.c +41
-6{ -const struct btf_type *t, *skbt;size_t end; - skbt =btf_type_by_id (reg->btf, bpf_sk_buff_ids[0 ]); - t =btf_type_by_id (reg->btf, reg->btf_id); -if (t != skbt) { -bpf_log (log,"only read is supported\n" ); +switch (off) { + case offsetof(struct Qdisc, qstats) ... offsetofend(struct Qdisc, qstats) - 1: + end = offsetofend(struct Qdisc, qstats); +break ; +default : +bpf_log (log,"no write support to Qdisc at off %d\n" , off); +return -EACCES ; + } + +if (off + size > end) { +bpf_log (log, +"write access at off %d with size %d beyond the member of Qdisc ended at %zu\n" , + off, size, end);return -EACCES ; } +return 0 ; +} + +static int bpf_qdisc_sk_buff_access (struct bpf_verifier_log *log, +const struct bpf_reg_state *reg, +int off,int size) +{ +size_t end; +switch (off) {case offsetof(struct sk_buff , tstamp): end = offsetofend(struct sk_buff, tstamp);@@ -115,6 +133,25 @@ static int bpf_qdisc_btf_struct_access(struct bpf_verifier_log *log,return 0 ; } +static int bpf_qdisc_btf_struct_access (struct bpf_verifier_log *log, +const struct bpf_reg_state *reg, +int off,int size) +{ +const struct btf_type *t, *skbt, *qdisct; + + skbt =btf_type_by_id (reg->btf, bpf_sk_buff_ids[0 ]); + qdisct =btf_type_by_id (reg->btf, bpf_qdisc_ids[0 ]); + t =btf_type_by_id (reg->btf, reg->btf_id); + +if (t == skbt) +return bpf_qdisc_sk_buff_access (log, reg, off, size); +else if (t == qdisct) +return bpf_qdisc_qdisc_access (log, reg, off, size); + +bpf_log (log,"only read is supported\n" ); +return -EACCES ; +} +BTF_ID_LIST (bpf_qdisc_init_prologue_ids)BTF_ID (func, bpf_qdisc_init_prologue) --2.47.1
Re: [PATCH bpf-next v5 07/13] bpf: net_sched: Support updating qstats
Quoted 40 lines · Alexei Starovoitov
Quoted 24 lines · Amery Hung
From: Amery Hung <[email protected]>
Allow bpf qdisc programs to update Qdisc qstats directly with btf struct access.
Signed-off-by Amery Hung <[email protected]>--- net/sched/bpf_qdisc.c | 53 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 45 insertions(+), 8 deletions(-)net/sched/bpf_qdisc.c +1
-0@@ -36,6 +36,7 @@ bpf_qdisc_get_func_proto(enum bpf_func_id func_id, } } +BTF_ID_LIST_SINGLE(bpf_qdisc_ids, struct, Qdisc) BTF_ID_LIST_SINGLE(bpf_sk_buff_ids, struct, sk_buff)BTF_ID_LIST_SINGLE (bpf_sk_buff_ptr_ids, struct, bpf_sk_buff_ptr)net/sched/bpf_qdisc.c static bool bpf_qdisc_is_valid_access(int off, int size, · line 66
Quoted 9 lines
net/sched/bpf_qdisc.c +3
-3@@ -60,20 +61,37 @@ static bool bpf_qdisc_is_valid_access(int off, int size,return bpf_tracing_btf_ctx_access (off, size, type, prog, info); } -static int bpf_qdisc_btf_struct_access (struct bpf_verifier_log *log, -const struct bpf_reg_state *reg, -int off,int size) +static int bpf_qdisc_qdisc_access (struct bpf_verifier_log *log, +const struct bpf_reg_state *reg, +int off,int size)Introducing this func in patch 3 and refactoring in patch 7 ? pls avoid the churn.
squash it ?
Quoted 3 lines
if (off + size > end) check wouldn't need to be duplicated. Can get the name of struct from btf for
bpf_log()purpose.
I will squash this patch to patch 3 and share the check in bpf_qdisc_btf_struct_access() to avoid duplication.
Thanks,
Amery
Quoted 65 lines
Quoted 65 lines
net/sched/bpf_qdisc.c +41
-6{ -const struct btf_type *t, *skbt;size_t end; - skbt =btf_type_by_id (reg->btf, bpf_sk_buff_ids[0 ]); - t =btf_type_by_id (reg->btf, reg->btf_id); -if (t != skbt) { -bpf_log (log,"only read is supported\n" ); +switch (off) { + case offsetof(struct Qdisc, qstats) ... offsetofend(struct Qdisc, qstats) - 1: + end = offsetofend(struct Qdisc, qstats); +break ; +default : +bpf_log (log,"no write support to Qdisc at off %d\n" , off); +return -EACCES ; + } + +if (off + size > end) { +bpf_log (log, +"write access at off %d with size %d beyond the member of Qdisc ended at %zu\n" , + off, size, end);return -EACCES ; } +return 0 ; +} + +static int bpf_qdisc_sk_buff_access (struct bpf_verifier_log *log, +const struct bpf_reg_state *reg, +int off,int size) +{ +size_t end; +switch (off) {case offsetof(struct sk_buff , tstamp): end = offsetofend(struct sk_buff, tstamp);@@ -115,6 +133,25 @@ static int bpf_qdisc_btf_struct_access(struct bpf_verifier_log *log,return 0 ; } +static int bpf_qdisc_btf_struct_access (struct bpf_verifier_log *log, +const struct bpf_reg_state *reg, +int off,int size) +{ +const struct btf_type *t, *skbt, *qdisct; + + skbt =btf_type_by_id (reg->btf, bpf_sk_buff_ids[0 ]); + qdisct =btf_type_by_id (reg->btf, bpf_qdisc_ids[0 ]); + t =btf_type_by_id (reg->btf, reg->btf_id); + +if (t == skbt) +return bpf_qdisc_sk_buff_access (log, reg, off, size); +else if (t == qdisct) +return bpf_qdisc_qdisc_access (log, reg, off, size); + +bpf_log (log,"only read is supported\n" ); +return -EACCES ; +} +BTF_ID_LIST (bpf_qdisc_init_prologue_ids)BTF_ID (func, bpf_qdisc_init_prologue) --2.47.1
PATCH v5 8/13 bpf: net_sched: Allow writing to more Qdisc members
In reply to Amery HungFrom: Amery Hung <[email protected]>
Allow bpf qdisc to write to Qdisc->limit and Qdisc->q.qlen.
--- net/sched/bpf_qdisc.c | 6 ++++++ 1 file changed, 6 insertions(+)
net/sched/bpf_qdisc.c +6 -0
@@ -68,6 +68,12 @@ static int bpf_qdisc_qdisc_access(struct bpf_verifier_log *log,size_t end;switch (off) { +case offsetof(struct Qdisc , limit): + end = offsetofend(struct Qdisc, limit); +break ; +case offsetof(struct Qdisc , q) + offsetof(struct qdisc_skb_head , qlen): + end = offsetof(struct Qdisc, q) + offsetofend(struct qdisc_skb_head, qlen); +break ; case offsetof(struct Qdisc, qstats) ... offsetofend(struct Qdisc, qstats) - 1: end = offsetofend(struct Qdisc, qstats);break ;
Signature · 1 line
2.47.1
PATCH v5 9/13 bpf: net_sched: Disable attaching bpf qdisc to non root
In reply to Amery HungDo not allow users to attach bpf qdiscs to classful qdiscs. This is to prevent accidentally breaking existings classful qdiscs if they rely on some data in the child qdisc. This restriction can potentially be lifted in the future. Note that, we still allow bpf qdisc to be attached to mq.
--- net/sched/bpf_qdisc.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)
net/sched/bpf_qdisc.c +19 -1
@@ -170,8 +170,11 @@ static int bpf_qdisc_gen_prologue(struct bpf_insn *insn_buf, bool direct_write,return 0 ; *insn++ =BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ); + *insn++ =BPF_LDX_MEM (BPF_DW ,BPF_REG_2 ,BPF_REG_1 ,16 ); *insn++ =BPF_LDX_MEM (BPF_DW ,BPF_REG_1 ,BPF_REG_1 ,0 ); *insn++ =BPF_CALL_KFUNC (0 , bpf_qdisc_init_prologue_ids[0 ]); + *insn++ =BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,1 ); + *insn++ =BPF_EXIT_INSN (); *insn++ =BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ); *insn++ = prog->insnsi[0 ];
@@ -239,11 +242,26 @@ __bpf_kfunc void bpf_qdisc_watchdog_schedule(struct Qdisc *sch, u64 expire, u64 }/* bpf_qdisc_init_prologue - Hidden kfunc called in prologue of .init. */ -__bpf_kfunc void bpf_qdisc_init_prologue(struct Qdisc *sch) +__bpf_kfunc int bpf_qdisc_init_prologue(struct Qdisc *sch, +struct netlink_ext_ack *extack) {struct bpf_sched_data *q =qdisc_priv (sch); +struct net_device *dev =qdisc_dev (sch); +struct Qdisc *p; + +if (sch->parent !=TC_H_ROOT ) { + p =qdisc_lookup (dev,TC_H_MAJ (sch->parent)); +if (!p) +return -ENOENT ; + +if (!(p->flags &TCQ_F_MQROOT )) { +NL_SET_ERR_MSG (extack,"BPF qdisc only supported on root or mq" ); +return -EINVAL ; + } + }qdisc_watchdog_init (&q->watchdog, sch); +return 0 ; }/* bpf_qdisc_reset_destroy_epilogue - Hidden kfunc called in epilogue of .reset
Signature · 1 line
2.47.1
Re: [PATCH bpf-next v5 09/13] bpf: net_sched: Disable attaching bpf qdisc to non root
Quoted 16 lines · Amery Hung
Do not allow users to attach bpf qdiscs to classful qdiscs. This is to prevent accidentally breaking existings classful qdiscs if they rely on some data in the child qdisc. This restriction can potentially be lifted in the future. Note that, we still allow bpf qdisc to be attached to mq.
Signed-off-by Amery Hung <[email protected]>--- net/sched/bpf_qdisc.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)net/sched/bpf_qdisc.c +0
-0
net/sched/bpf_qdisc.c static int bpf_qdisc_gen_prologue(struct bpf_insn *insn_buf, bool direct_write, · line 174
Quoted 5 lines
net/sched/bpf_qdisc.c +1
-0@@ -170,8 +170,11 @@ static int bpf_qdisc_gen_prologue(struct bpf_insn *insn_buf, bool direct_write,return 0 ; *insn++ =BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ); + *insn++ =BPF_LDX_MEM (BPF_DW ,BPF_REG_2 ,BPF_REG_1 ,16 ); *insn++ =BPF_LDX_MEM (BPF_DW ,BPF_REG_1 ,BPF_REG_1 ,0 );
Both loads need a comment.
It's st_ops callback specific and not obvious what ends up in r1 and r2.
Quoted 37 lines
net/sched/bpf_qdisc.c +18
-2*insn++ =BPF_CALL_KFUNC (0 , bpf_qdisc_init_prologue_ids[0 ]); + *insn++ =BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,1 ); + *insn++ =BPF_EXIT_INSN (); *insn++ =BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ); *insn++ = prog->insnsi[0 ];@@ -239,11 +242,26 @@ __bpf_kfunc void bpf_qdisc_watchdog_schedule(struct Qdisc *sch, u64 expire, u64 }/* bpf_qdisc_init_prologue - Hidden kfunc called in prologue of .init. */ -__bpf_kfunc void bpf_qdisc_init_prologue(struct Qdisc *sch) +__bpf_kfunc int bpf_qdisc_init_prologue(struct Qdisc *sch, +struct netlink_ext_ack *extack) {struct bpf_sched_data *q =qdisc_priv (sch); +struct net_device *dev =qdisc_dev (sch); +struct Qdisc *p; + +if (sch->parent !=TC_H_ROOT ) { + p =qdisc_lookup (dev,TC_H_MAJ (sch->parent)); +if (!p) +return -ENOENT ; + +if (!(p->flags &TCQ_F_MQROOT )) { +NL_SET_ERR_MSG (extack,"BPF qdisc only supported on root or mq" ); +return -EINVAL ; + } + }qdisc_watchdog_init (&q->watchdog, sch); +return 0 ; }/* bpf_qdisc_reset_destroy_epilogue - Hidden kfunc called in epilogue of .reset -- 2.47.1
Re: [PATCH bpf-next v5 09/13] bpf: net_sched: Disable attaching bpf qdisc to non root
Quoted 20 lines · Alexei Starovoitov
Quoted 18 lines · Amery Hung
Do not allow users to attach bpf qdiscs to classful qdiscs. This is to prevent accidentally breaking existings classful qdiscs if they rely on some data in the child qdisc. This restriction can potentially be lifted in the future. Note that, we still allow bpf qdisc to be attached to mq.
Signed-off-by Amery Hung <[email protected]>--- net/sched/bpf_qdisc.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)net/sched/bpf_qdisc.c +0
-0@@ -170,8 +170,11 @@ static int bpf_qdisc_gen_prologue(struct bpf_insn *insn_buf, bool direct_write,return 0 ;
Quoted 7 lines
net/sched/bpf_qdisc.c static int bpf_qdisc_gen_prologue(struct bpf_insn *insn_buf, bool direct_write, · line 174
Quoted 3 lines
net/sched/bpf_qdisc.c +1
-0*insn++ =BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_1 ); + *insn++ =BPF_LDX_MEM (BPF_DW ,BPF_REG_2 ,BPF_REG_1 ,16 ); *insn++ =BPF_LDX_MEM (BPF_DW ,BPF_REG_1 ,BPF_REG_1 ,0 );Both loads need a comment.
It'sst_opscallback specific and not obvious what ends up in r1 and r2.
Got it. I will clarify this in the comment.
Quoted 37 lines
Quoted 37 lines
net/sched/bpf_qdisc.c +18
-2*insn++ =BPF_CALL_KFUNC (0 , bpf_qdisc_init_prologue_ids[0 ]); + *insn++ =BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,1 ); + *insn++ =BPF_EXIT_INSN (); *insn++ =BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ); *insn++ = prog->insnsi[0 ];@@ -239,11 +242,26 @@ __bpf_kfunc void bpf_qdisc_watchdog_schedule(struct Qdisc *sch, u64 expire, u64 }/* bpf_qdisc_init_prologue - Hidden kfunc called in prologue of .init. */ -__bpf_kfunc void bpf_qdisc_init_prologue(struct Qdisc *sch) +__bpf_kfunc int bpf_qdisc_init_prologue(struct Qdisc *sch, +struct netlink_ext_ack *extack) {struct bpf_sched_data *q =qdisc_priv (sch); +struct net_device *dev =qdisc_dev (sch); +struct Qdisc *p; + +if (sch->parent !=TC_H_ROOT ) { + p =qdisc_lookup (dev,TC_H_MAJ (sch->parent)); +if (!p) +return -ENOENT ; + +if (!(p->flags &TCQ_F_MQROOT )) { +NL_SET_ERR_MSG (extack,"BPF qdisc only supported on root or mq" ); +return -EINVAL ; + } + }qdisc_watchdog_init (&q->watchdog, sch); +return 0 ; }/* bpf_qdisc_reset_destroy_epilogue - Hidden kfunc called in epilogue of .reset -- 2.47.1
PATCH v5 10/13 libbpf: Support creating and destroying qdisc
In reply to Amery HungFrom: Amery Hung <[email protected]>
Extend struct bpf_tc_hook with handle, qdisc name and a new attach type, BPF_TC_QDISC, to allow users to add or remove any qdisc specified in addition to clsact.
--- tools/lib/bpf/libbpf.h | 5 ++++- tools/lib/bpf/netlink.c | 20 +++++++++++++++++--- 2 files changed, 21 insertions(+), 4 deletions(-)
tools/lib/bpf/libbpf.h +4 -1
@@ -1283,6 +1283,7 @@ enum bpf_tc_attach_point {BPF_TC_INGRESS =1 <<0 ,BPF_TC_EGRESS =1 <<1 ,BPF_TC_CUSTOM =1 <<2 , + BPF_TC_QDISC = 1 << 3, };#define BPF_TC_PARENT (a, b) \
@@ -1297,9 +1298,11 @@ struct bpf_tc_hook {int ifindex;enum bpf_tc_attach_point attach_point;__u32 parent; +__u32 handle; +const char *qdisc; size_t :0; }; -#define bpf_tc_hook__last_field parent +#define bpf_tc_hook__last_field qdiscstruct bpf_tc_opts {size_t sz;
tools/lib/bpf/netlink.c +17 -3
@@ -529,9 +529,9 @@ int bpf_xdp_query_id(int ifindex, int flags, __u32 *prog_id) } -typedef int (*qdisc_config_t )(struct libbpf_nla_req *req); +typedef int (*qdisc_config_t )(struct libbpf_nla_req *req,const struct bpf_tc_hook *hook); -static int clsact_config (struct libbpf_nla_req *req) +static int clsact_config (struct libbpf_nla_req *req,const struct bpf_tc_hook *hook) { req->tc.tcm_parent =TC_H_CLSACT ; req->tc.tcm_handle =TC_H_MAKE (TC_H_CLSACT ,0 );
@@ -539,6 +539,16 @@ static int clsact_config(struct libbpf_nla_req *req)return nlattr_add (req,TCA_KIND ,"clsact" ,sizeof ("clsact" )); } +static int qdisc_config (struct libbpf_nla_req *req,const struct bpf_tc_hook *hook) +{ +const char *qdisc =OPTS_GET (hook, qdisc,NULL ); + + req->tc.tcm_parent =OPTS_GET (hook, parent,TC_H_ROOT ); + req->tc.tcm_handle =OPTS_GET (hook, handle,0 ); + +return nlattr_add (req,TCA_KIND , qdisc,strlen (qdisc) +1 ); +} +static int attach_point_to_config (struct bpf_tc_hook *hook,qdisc_config_t *config) {
@@ -552,6 +562,9 @@ static int attach_point_to_config(struct bpf_tc_hook *hook,return 0 ;case BPF_TC_CUSTOM :return -EOPNOTSUPP ; +case BPF_TC_QDISC : + *config = &qdisc_config; +return 0 ;default :return -EINVAL ; }
@@ -596,7 +609,7 @@ static int tc_qdisc_modify(struct bpf_tc_hook *hook, int cmd, int flags) req.tc.tcm_family =AF_UNSPEC ; req.tc.tcm_ifindex =OPTS_GET (hook, ifindex,0 ); - ret =config (&req); + ret =config (&req, hook);if (ret <0 )return ret;
@@ -639,6 +652,7 @@ int bpf_tc_hook_destroy(struct bpf_tc_hook *hook)case BPF_TC_INGRESS :case BPF_TC_EGRESS :return libbpf_err (__bpf_tc_detach (hook,NULL , true)); +case BPF_TC_QDISC :case BPF_TC_INGRESS |BPF_TC_EGRESS :return libbpf_err (tc_qdisc_delete (hook));case BPF_TC_CUSTOM :
Signature · 1 line
2.47.1
PATCH v5 11/13 selftests/bpf: Add a basic fifo qdisc test
In reply to Amery HungFrom: Amery Hung <[email protected]>
This selftest includes a bare minimum fifo qdisc, which simply enqueues sk_buffs into the back of a bpf list and dequeues from the front of the list.
--- tools/testing/selftests/bpf/config | 1 + .../selftests/bpf/prog_tests/bpf_qdisc.c | 79 ++++++++++++ .../selftests/bpf/progs/bpf_qdisc_common.h | 27 ++++ .../selftests/bpf/progs/bpf_qdisc_fifo.c | 117 ++++++++++++++++++ 4 files changed, 224 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_qdisc_common.h create mode 100644 tools/testing/selftests/bpf/progs/bpf_qdisc_fifo.c
tools/testing/selftests/bpf/config +1 -0
@@ -71,6 +71,7 @@ CONFIG_NET_IPGRE=y CONFIG_NET_IPGRE_DEMUX=y CONFIG_NET_IPIP=y CONFIG_NET_MPLS_GSO=y +CONFIG_NET_SCH_BPF=y CONFIG_NET_SCH_FQ=y CONFIG_NET_SCH_INGRESS=y CONFIG_NET_SCHED=y
tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c new file +79 -0
79 lines not on this page · raw message
tools/testing/selftests/bpf/progs/bpf_qdisc_common.h new file +27 -0
@@ -0,0 +1,27 @@ +#ifndef _BPF_QDISC_COMMON_H +#define _BPF_QDISC_COMMON_H + +#define NET_XMIT_SUCCESS 0x00 +#define NET_XMIT_DROP 0x01/* skb dropped */ +#define NET_XMIT_CN 0x02/* congestion notification */ + +#define TC_PRIO_CONTROL 7 +#define TC_PRIO_MAX 15 + +u32 bpf_skb_get_hash(struct sk_buff *p) __ksym; +void bpf_kfree_skb(struct sk_buff *p) __ksym; +void bpf_qdisc_skb_drop(struct sk_buff *p, struct bpf_sk_buff_ptr *to_free) __ksym; +void bpf_qdisc_watchdog_schedule(struct Qdisc *sch, u64 expire, u64 delta_ns) __ksym; +void bpf_qdisc_bstats_update(struct Qdisc *sch, const struct sk_buff *skb) __ksym; + +static struct qdisc_skb_cb *qdisc_skb_cb (const struct sk_buff *skb) +{ +return (struct qdisc_skb_cb *)skb->cb; +} + +static inline unsigned int qdisc_pkt_len (const struct sk_buff *skb) +{ +return qdisc_skb_cb (skb)->pkt_len; +} + +#endif
tools/testing/selftests/bpf/progs/bpf_qdisc_fifo.c new file +117 -0
117 lines not on this page · raw message
Signature · 1 line
2.47.1
PATCH v5 12/13 selftests/bpf: Add a bpf fq qdisc to selftest
In reply to Amery HungFrom: Amery Hung <[email protected]>
This test implements a more sophisticated qdisc using bpf. The bpf fair- queueing (fq) qdisc gives each flow an equal chance to transmit data. It also respects the timestamp of skb for rate limiting.
--- .../selftests/bpf/prog_tests/bpf_qdisc.c | 24 + .../selftests/bpf/progs/bpf_qdisc_fq.c | 718 ++++++++++++++++++ 2 files changed, 742 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/bpf_qdisc_fq.c
tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c +24 -0
@@ -4,6 +4,7 @@#include "network_helpers.h" #include "bpf_qdisc_fifo.skel.h" +#include "bpf_qdisc_fq.skel.h" #define LO_IFINDEX 1
@@ -64,6 +65,27 @@ static void test_fifo(void)bpf_qdisc_fifo__destroy (fifo_skel); } +static void test_fq (void ) +{ +struct bpf_qdisc_fq *fq_skel; +struct bpf_link *link; + + fq_skel =bpf_qdisc_fq__open_and_load (); +if (!ASSERT_OK_PTR (fq_skel,"bpf_qdisc_fq__open_and_load" )) +return ; + + link =bpf_map__attach_struct_ops (fq_skel->maps.fq); +if (!ASSERT_OK_PTR (link,"bpf_map__attach_struct_ops" )) { +bpf_qdisc_fq__destroy (fq_skel); +return ; + } + +do_test ("bpf_fq" ); + +bpf_link__destroy (link); +bpf_qdisc_fq__destroy (fq_skel); +} +void test_bpf_qdisc (void ) {struct netns_obj *netns;
@@ -74,6 +96,8 @@ void test_bpf_qdisc(void)if (test__start_subtest ("fifo" ))test_fifo (); +if (test__start_subtest ("fq" )) +test_fq ();netns_free (netns); }
tools/testing/selftests/bpf/progs/bpf_qdisc_fq.c new file +718 -0
718 lines not on this page · raw message
Signature · 1 line
2.47.1
Re: [PATCH bpf-next v5 12/13] selftests/bpf: Add a bpf fq qdisc to selftest
Quoted 11 lines · Amery Hung
From: Amery Hung <[email protected]>
This test implements a more sophisticated qdisc using bpf. The bpf fair- queueing (fq) qdisc gives each flow an equal chance to transmit data. It also respects the timestamp of skb for rate limiting.
Signed-off-by Amery Hung <[email protected]>--- .../selftests/bpf/prog_tests/bpf_qdisc.c | 24 + .../selftests/bpf/progs/bpf_qdisc_fq.c | 718 ++++++++++++++++++
On the look of it, it's a pretty functional qdisc. Since bpftool supports loading st_ops,
please list commands bpftool and tc the one can enter to use this qdisc without running selftests.
Probably at the comment section at the top of bpf_qdisc_fq.c
It also needs SPDX and copyright.
pw-bot: cr
Re: [PATCH bpf-next v5 12/13] selftests/bpf: Add a bpf fq qdisc to selftest
Quoted 14 lines · Alexei Starovoitov
Quoted 11 lines · Amery Hung
From: Amery Hung <[email protected]>
This test implements a more sophisticated qdisc using bpf. The bpf fair- queueing (fq) qdisc gives each flow an equal chance to transmit data. It also respects the timestamp of skb for rate limiting.
Signed-off-by Amery Hung <[email protected]>--- .../selftests/bpf/prog_tests/bpf_qdisc.c | 24 + .../selftests/bpf/progs/bpf_qdisc_fq.c | 718 ++++++++++++++++++
Quoted 5 lines
On the look of it, it's a pretty functional qdisc. Since bpftool supports loading
st_ops,
please list commands bpftool and tc the one can enter to use this qdisc without running selftests.
Thanks for the suggestion. That should be very helpful. I will add the commands.
Quoted 3 lines
Probably at the comment section at the top of
bpf_qdisc_fq.cIt also needs SPDX and copyright.
I will add SPDX and copyright as well.
Thanks,
Amery
Quoted 2 lines
pw-bot: cr
PATCH v5 13/13 selftests/bpf: Test attaching bpf qdisc to mq and non root
In reply to Amery HungUntil we are certain that existing classful qdiscs work with bpf qdisc, make sure we don't allow attaching a bpf qdisc to non root. Meanwhile, attaching to mq is allowed.
--- tools/testing/selftests/bpf/config | 1 + .../selftests/bpf/prog_tests/bpf_qdisc.c | 75 +++++++++++++++++++ 2 files changed, 76 insertions(+)
tools/testing/selftests/bpf/config +1 -0
@@ -74,6 +74,7 @@ CONFIG_NET_MPLS_GSO=y CONFIG_NET_SCH_BPF=y CONFIG_NET_SCH_FQ=y CONFIG_NET_SCH_INGRESS=y +CONFIG_NET_SCH_HTB=y CONFIG_NET_SCHED=y CONFIG_NETDEVSIM=y CONFIG_NETFILTER=y
tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c +75 -0
87 lines not on this page · raw message
Signature · 1 line
2.47.1
Quoted 80 lines · Amery Hung
Hi all,
This patchset aims to support implementing qdisc using bpf
struct_ops. This version takes a step back and only implements the minimum support for bpf qdisc. 1) support of adding skb tobpf_listandbpf_rbtreedirectly and 2) classful qdisc are deferred to future patchsets. In addition, we only allow attaching bpf qdisc to root or mq for now. This is to prevent accidentally breaking exisiting classful qdiscs that rely on data in a child qdisc. This limit may be lifted in the future after careful inspection.* Overview *
This series supports implementing qdisc using bpf
struct_ops. bpf qdisc aims to be a flexible and easy-to-use infrastructure that allows users to quickly experiment with different scheduling algorithms/policies. It only requires users to implement core qdisc logic using bpf and implements the mundane part for them. In addition, the ability to easily communicate between qdisc and other components will also bring new opportunities for new applications and optimizations.*
struct_opschanges *To make
struct_opsworks better with bpf qdisc, two new changes are introduced to bpf specifically forstruct_opsprograms. Frist, we introduce "__ref" postfix for arguments in stub functions in patch 1-2. It allowsQdisc_ops->enqueueto acquire an unique referenced kptr to the skb argument. Through the reference object tracking mechanism in the verifier, we can make sure that the acquired skb will be either enqueued or dropped. Besides, no duplicate references can be acquired. Then, we allow a referenced kptr to be returned fromstruct_opsprograms so that we can return an skb naturally. This is done and tested in patch 3 and 4.* Performance of bpf qdisc *
This patchset includes two qdisc examples,
bpf_fifoandbpf_fq, for__testing__purposes. For performance test, we compare selftests and their kernel counterparts to give you a sense of the performance of qdisc implemented in bpf.The implementation of
bpf_fqis fairly complex and slightly different from fq so later we only compare the two fifo qdiscs.bpf_fqimplements a scheduling algorithm similar to fq before commit29f834aa326e("net_sched:sch_fq: add 3 bands and WRR scheduling") was introduced.bpf_fifouses a singlebpf_listas a queue instead of three queues for different priorities inpfifo_fast. The time complexity of fifo however should be similar since the queue selection time is negligible.Test setup:
client -> qdisc -------------> server ~~~~~~~~~~~~~~~ ~~~~~~ nested VM1 @ DC1 VM2 @ DC2Throghput: iperf3 -t 600, 5 times
Qdisc Average (GBits/sec) ---------- ------------------- pfifo_fast 12.52 ± 0.26 bpf_fifo 11.72 ± 0.32 fq 10.24 ± 0.13 bpf_fq 11.92 ± 0.64Latency: sockperf pp --tcp -t 600, 5 times
Qdisc Average (usec) ---------- -------------- pfifo_fast 244.58 ± 7.93 bpf_fifo 244.92 ± 15.22 fq 234.30 ± 19.25 bpf_fq 221.34 ± 10.76Looking at the two fifo qdiscs, the 6.4% drop in throughput in the bpf implementatioin is consistent with previous observation (v8 throughput test on a loopback device). This should be able to be mitigated by supporting adding skb to
bpf_listorbpf_rbtreedirectly in the future.* Clean up skb in bpf qdisc during reset *
Quoted 7 lines
The current implementation relies on bpf qdisc implementors to correctly release skbs in queues (bpf graphs or maps) in .reset, which might not be a safe thing to do. The solution as Martin has suggested would be supporting private data in
struct_ops. This can also help simplifying implementation of qdisc that works with mq. For examples, qdiscs in the selftest mostly use global data. Therefore, even if user add multiple qdisc instances under mq, they would still share the same queue.
Very cool to see this progress!
Are you aware that the series has a mix of commit author email addresses (mixing your bytedance.com and gmail addresses)?
Otherwise, for the series:
Quoted 14 lines · Toke Høiland-Jørgensen
Quoted 10 lines · Amery Hung
Hi all,
This patchset aims to support implementing qdisc using bpf
struct_ops. This version takes a step back and only implements the minimum support for bpf qdisc. 1) support of adding skb tobpf_listandbpf_rbtreedirectly and 2) classful qdisc are deferred to future patchsets. In addition, we only allow attaching bpf qdisc to root or mq for now. This is to prevent accidentally breaking exisiting classful qdiscs that rely on data in a child qdisc. This limit may be lifted in the future after careful inspection.
Quoted 5 lines
Very cool to see this progress!
Are you aware that the series has a mix of commit author email addresses (mixing your bytedance.com and gmail addresses)?
Yes. I have changed my affiliation so some recently added patches use another email.
Thanks for reviewing!
Amery
Quoted 4 lines
Otherwise, for the series: