bpf: Add __aux tag to pass in prog->aux
Kumar Kartikeya Dwivedi
maintainerPATCH 1/1 bpf: Add __aux tag to pass in prog->aux
Instead of hardcoding the list of kfuncs that need prog->aux passed to them with a combination of fixup_kfunc_call adjustment + __ign suffix, combine both in __aux suffix, which ignores the argument passed in, and fixes it up to the prog->aux. This allows kfuncs to have the prog->aux passed into them without having to touch the verifier.
--- include/linux/bpf_verifier.h | 1 + kernel/bpf/helpers.c | 4 ++-- kernel/bpf/verifier.c | 33 +++++++++++++++++++++++++++------ 3 files changed, 30 insertions(+), 8 deletions(-)
include/linux/bpf_verifier.h +1 -0
@@ -606,6 +606,7 @@ struct bpf_insn_aux_data {bool calls_callback;/* registers alive before this instruction. */ u16 live_regs_before; +u16 arg_prog_aux; };#define MAX_USED_MAPS 64/* max number of maps accessed by one eBPF program */
kernel/bpf/helpers.c +2 -2
@@ -3012,9 +3012,9 @@ __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) __bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq,int (callback_fn )(void *map,int *key,void *value),unsigned int flags, -void *aux__ign) +void *aux__aux) { -struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__ign; +struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__aux;struct bpf_async_kern *async = (struct bpf_async_kern *)wq;if (flags)
kernel/bpf/verifier.c +27 -6
@@ -343,6 +343,7 @@ struct bpf_kfunc_call_arg_meta {int uid; } map;u64 mem_size; +u32 arg_prog_aux; };struct btf *btf_vmlinux;
@@ -11897,6 +11898,11 @@ static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_paramreturn btf_param_match_suffix (btf, arg,"__irq_flag" ); } +static bool is_kfunc_arg_prog_aux (const struct btf *btf,const struct btf_param *arg) +{ +return btf_param_match_suffix (btf, arg,"__aux" ); +} +static bool is_kfunc_arg_scalar_with_name (const struct btf *btf,const struct btf_param *arg,const char *name)
@@ -12938,6 +12944,17 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_if (is_kfunc_arg_ignore (btf, &args[i]))continue ; +if (is_kfunc_arg_prog_aux (btf, &args[i])) { +/* Used to reject repeated use of __aux. */ +if (meta->arg_prog_aux) { +verbose (env,"Only 1 prog->aux argument supported per-kfunc\n" ); +return -EFAULT ; + } + meta->arg_prog_aux = regno; +cur_aux (env)->arg_prog_aux = regno; +continue ; + } +if (btf_type_is_scalar (t)) {if (reg->type !=SCALAR_VALUE ) {verbose (env,"R%d is not a scalar\n" , regno);
@@ -21517,13 +21534,17 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) { insn_buf[0 ] =BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_1 ); *cnt =1 ; - } else if (is_bpf_wq_set_callback_impl_kfunc(desc->func_id)) { -struct bpf_insn ld_addrs[2 ] = {BPF_LD_IMM64 (BPF_REG_4 , (long )env->prog->aux) }; + } - insn_buf[0 ] = ld_addrs[0 ]; - insn_buf[1 ] = ld_addrs[1 ]; - insn_buf[2 ] = *insn; - *cnt =3 ; +if (env->insn_aux_data[insn_idx].arg_prog_aux) { +u32 regno = env->insn_aux_data[insn_idx].arg_prog_aux; +struct bpf_insn ld_addrs[2 ] = {BPF_LD_IMM64 (regno, (long )env->prog->aux) }; +int idx = *cnt; + + insn_buf[idx++] = ld_addrs[0 ]; + insn_buf[idx++] = ld_addrs[1 ]; + insn_buf[idx++] = *insn; + *cnt = idx; }return 0 ; }
--
2.47.1
Quoted 6 lines · Kumar Kartikeya Dwivedi
Instead of hardcoding the list of kfuncs that need
prog->auxpassed to them with a combination offixup_kfunc_calladjustment +__ignsuffix, combine both in__auxsuffix, which ignores the argument passed in, and fixes it up to theprog->aux. This allows kfuncs to have theprog->auxpassed into them without having to touch the verifier.
I have this wet dream that one day we'll make sure that all BPF programs have bpf_run_ctx set up for them, across all program types, and we'll just use that for these "BPF environment"-like things like getting currently-running bpf_prog/bpf_prog_aux pointer.
Do you think it makes sense? One of the concerns for wiring bpf_run_ctx for, say, XDP programs was perceived potential tiny perf regression (but it's just current->bpf_ctx swap, so shouldn't be a big deal at all, IMO) and just generally no immediate use case. So maybe this bpf_prog_aux access is a good enough reason now, WDYT?
Quoted 9 lines
Cc Tejun Heo <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf_verifier.h | 1 + kernel/bpf/helpers.c | 4 ++-- kernel/bpf/verifier.c | 33 +++++++++++++++++++++++++++------ 3 files changed, 30 insertions(+), 8 deletions(-)
[...]
Kumar Kartikeya Dwivedi
maintainerQuoted 10 lines · Andrii Nakryiko
Quoted 6 lines · Kumar Kartikeya Dwivedi
Instead of hardcoding the list of kfuncs that need
prog->auxpassed to them with a combination offixup_kfunc_calladjustment +__ignsuffix, combine both in__auxsuffix, which ignores the argument passed in, and fixes it up to theprog->aux. This allows kfuncs to have theprog->auxpassed into them without having to touch the verifier.
Quoted 5 lines
I have this wet dream that one day we'll make sure that all BPF programs have
bpf_run_ctxset up for them, across all program types, and we'll just use that for these "BPF environment"-like things like getting currently-running bpf_prog/bpf_prog_aux pointer.
For this particular patch, it's mostly refactoring. But I agree in general having bpf_run_ctx everywhere would be useful. Currently being able to rely on it in some cases and not in others is a bit annoying.
Quoted 4 lines
Do you think it makes sense? One of the concerns for wiring
bpf_run_ctxfor, say, XDP programs was perceived potential tiny perf regression (but it's justcurrent->bpf_ctxswap, so shouldn't be a big deal at all, IMO) and just generally no immediate use case.
I think the performance concern may be real. I can do some benchmarking with xdp-bench, but it's probably not the raw swaps that cost too much but potential cache miss on touching current. I am not sure how problematic that would be in terms of pps regression for small programs, but even if amortized it may end up regressing when done unconditionally.
XDP is sort of an edge case though. I am not sure the concern on ns-scale effects holds for any other program type.
Quoted 14 lines
So maybe
thisbpf_prog_auxaccess is a good enough reason now, WDYT?Quoted 9 lines
Cc Tejun Heo <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf_verifier.h | 1 + kernel/bpf/helpers.c | 4 ++-- kernel/bpf/verifier.c | 33 +++++++++++++++++++++++++++------ 3 files changed, 30 insertions(+), 8 deletions(-)[...]
Quoted 18 lines · Kumar Kartikeya Dwivedi
Quoted 10 lines · Andrii Nakryiko
Quoted 6 lines · Kumar Kartikeya Dwivedi
Instead of hardcoding the list of kfuncs that need
prog->auxpassed to them with a combination offixup_kfunc_calladjustment +__ignsuffix, combine both in__auxsuffix, which ignores the argument passed in, and fixes it up to theprog->aux. This allows kfuncs to have theprog->auxpassed into them without having to touch the verifier.Quoted 5 lines
I have this wet dream that one day we'll make sure that all BPF programs have
bpf_run_ctxset up for them, across all program types, and we'll just use that for these "BPF environment"-like things like getting currently-running bpf_prog/bpf_prog_aux pointer.
Quoted 4 lines
For this particular patch, it's mostly refactoring. But I agree in general having
bpf_run_ctxeverywhere would be useful. Currently being able to rely on it in some cases and not in others is a bit annoying.
Yep, agreed.
Quoted 6 lines
Quoted 5 lines
Do you think it makes sense? One of the concerns for wiring
bpf_run_ctxfor, say, XDP programs was perceived potential tiny perf regression (but it's justcurrent->bpf_ctxswap, so shouldn't be a big deal at all, IMO) and just generally no immediate use case.
Quoted 6 lines
I think the performance concern may be real. I can do some benchmarking with xdp-bench, but it's probably not the raw swaps that cost too much but potential cache miss on touching current. I am not sure how problematic that would be in terms of pps regression for small programs, but even if amortized it may end up regressing when done unconditionally.
Yep, cache line access is the expensive thing, not the swap itself, of course. My hope is that current is frequently accessed enough and we might get lucky with that cache line already being present. Or if XDP calls are frequent enough, we'll just amortize the cost across multiple packets, as you mentioned.
But yes, maybe a bit of benchmarking would give us a bit better idea, thanks!
Quoted 3 lines
XDP is sort of an edge case though. I am not sure the concern on ns-scale effects holds for any other program type.
Sure, XDP is an extreme case, but if bpf_run_ctx is everywhere *but* XDP, that still makes it a non-universal mechanism, and we'll have to always remember that some run_ctx-related functionality doesn't work on XDP. Maybe that's fine, I don't know, but it certainly would be better to not have these exceptions.
Quoted 15 lines
Quoted 15 lines
So maybe
thisbpf_prog_auxaccess is a good enough reason now, WDYT?Quoted 9 lines
Cc Tejun Heo <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf_verifier.h | 1 + kernel/bpf/helpers.c | 4 ++-- kernel/bpf/verifier.c | 33 +++++++++++++++++++++++++++------ 3 files changed, 30 insertions(+), 8 deletions(-)[...]
Quoted 32 lines · Kumar Kartikeya Dwivedi
Instead of hardcoding the list of kfuncs that need
prog->auxpassed to them with a combination offixup_kfunc_calladjustment +__ignsuffix, combine both in__auxsuffix, which ignores the argument passed in, and fixes it up to theprog->aux. This allows kfuncs to have theprog->auxpassed into them without having to touch the verifier.Cc Tejun Heo <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf_verifier.h | 1 + kernel/bpf/helpers.c | 4 ++-- kernel/bpf/verifier.c | 33 +++++++++++++++++++++++++++------ 3 files changed, 30 insertions(+), 8 deletions(-)include/linux/bpf_verifier.h +1
-0@@ -606,6 +606,7 @@ struct bpf_insn_aux_data {bool calls_callback;/* registers alive before this instruction. */ u16 live_regs_before; +u16 arg_prog_aux; };#define MAX_USED_MAPS 64/* max number of maps accessed by one eBPF program */ kernel/bpf/helpers.c +0
-0
kernel/bpf/helpers.c __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) · line 3015
Quoted 5 lines
kernel/bpf/helpers.c +1
-1@@ -3012,9 +3012,9 @@ __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) __bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq,int (callback_fn )(void *map,int *key,void *value),unsigned int flags, -void *aux__ign) +void *aux__aux)
aux__aux is an odd name.
"__aux" as a suffix also looks strange.
How about "__prog" suffix ?
It will be similar to the existing "__map" suffix.
We can also standardize the argument name as __bpf_kfunc int bpf_wq_set_callback_impl(.. , void *aux__prog)
then the name is more or less explanatory.
kernel/bpf/helpers.c __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) · line 3017
Quoted 3 lines
kernel/bpf/helpers.c +1
-1{ -struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__ign; +struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__aux;
and here it will be:
+ struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__prog;
which looks ok to me.
pw-bot: cr
Kumar Kartikeya Dwivedi
maintainerQuoted 29 lines · Alexei Starovoitov
Quoted 26 lines · Kumar Kartikeya Dwivedi
Instead of hardcoding the list of kfuncs that need
prog->auxpassed to them with a combination offixup_kfunc_calladjustment +__ignsuffix, combine both in__auxsuffix, which ignores the argument passed in, and fixes it up to theprog->aux. This allows kfuncs to have theprog->auxpassed into them without having to touch the verifier.Cc Tejun Heo <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf_verifier.h | 1 + kernel/bpf/helpers.c | 4 ++-- kernel/bpf/verifier.c | 33 +++++++++++++++++++++++++++------ 3 files changed, 30 insertions(+), 8 deletions(-)include/linux/bpf_verifier.h +1
-0@@ -606,6 +606,7 @@ struct bpf_insn_aux_data {bool calls_callback;/* registers alive before this instruction. */ u16 live_regs_before; +u16 arg_prog_aux; };
Quoted 15 lines
kernel/bpf/helpers.c __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) · line 3015
Quoted 11 lines
include/linux/bpf_verifier.h +0
-0#define MAX_USED_MAPS 64/* max number of maps accessed by one eBPF program */ kernel/bpf/helpers.c +1
-1@@ -3012,9 +3012,9 @@ __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) __bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq,int (callback_fn )(void *map,int *key,void *value),unsigned int flags, -void *aux__ign) +void *aux__aux)
aux__auxis an odd name.
"__aux" as a suffix also looks strange.
We can call it prog__aux.
Quoted 2 lines
How about "
__prog" suffix ?
It will be similar to the existing "__map" suffix.
But it's a bit misleading, it's not the prog, it's prog->aux.
Quoted 5 lines
We can also standardize the argument name as
__bpf_kfuncint bpf_wq_set_callback_impl(.. , void*aux__prog)then the name is more or less explanatory.
You can call it foo__prog, the part before __ is arbitrary.
Quoted 10 lines
kernel/bpf/helpers.c __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) · line 3017
Quoted 4 lines
kernel/bpf/helpers.c +1
-1{ -struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__ign; +struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__aux;and here it will be:
+ struct
bpf_prog_aux*aux = (structbpf_prog_aux*)aux__prog;which looks ok to me.
How about I rename the parameter to prog__aux?
Or __prog_aux as the tag name?
Quoted 2 lines
pw-bot: cr
Quoted 48 lines · Kumar Kartikeya Dwivedi
Quoted 29 lines · Alexei Starovoitov
Quoted 26 lines · Kumar Kartikeya Dwivedi
Instead of hardcoding the list of kfuncs that need
prog->auxpassed to them with a combination offixup_kfunc_calladjustment +__ignsuffix, combine both in__auxsuffix, which ignores the argument passed in, and fixes it up to theprog->aux. This allows kfuncs to have theprog->auxpassed into them without having to touch the verifier.Cc Tejun Heo <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf_verifier.h | 1 + kernel/bpf/helpers.c | 4 ++-- kernel/bpf/verifier.c | 33 +++++++++++++++++++++++++++------ 3 files changed, 30 insertions(+), 8 deletions(-)include/linux/bpf_verifier.h +1
-0@@ -606,6 +606,7 @@ struct bpf_insn_aux_data {bool calls_callback;/* registers alive before this instruction. */ u16 live_regs_before; +u16 arg_prog_aux; };Quoted 15 lines
kernel/bpf/helpers.c __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) · line 3015
Quoted 11 lines
include/linux/bpf_verifier.h +0
-0#define MAX_USED_MAPS 64/* max number of maps accessed by one eBPF program */ kernel/bpf/helpers.c +1
-1@@ -3012,9 +3012,9 @@ __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) __bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq,int (callback_fn )(void *map,int *key,void *value),unsigned int flags, -void *aux__ign) +void *aux__aux)
aux__auxis an odd name.
"__aux" as a suffix also looks strange.
Quoted 6 lines
We can call it
prog__aux.Quoted 2 lines
How about "
__prog" suffix ?
It will be similar to the existing "__map" suffix.But it's a bit misleading, it's not the prog, it's
prog->aux.
prog or prog->aux is the same thing. It's a pointer to prog. bpf_prog is read only part of it.bpf_prog_uax is writeable.
Quoted 14 lines
Quoted 6 lines
We can also standardize the argument name as
__bpf_kfuncint bpf_wq_set_callback_impl(.. , void*aux__prog)then the name is more or less explanatory.
You can call it
foo__prog, the part before __ is arbitrary.Quoted 5 lines
Quoted 4 lines
kernel/bpf/helpers.c +1
-1{ -struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__ign; +struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__aux;
Quoted 8 lines
Quoted 5 lines
and here it will be:
+ struct
bpf_prog_aux*aux = (structbpf_prog_aux*)aux__prog;which looks ok to me.
How about I rename the parameter to
prog__aux?
Or__prog_auxas the tag name?
I still prefer aux__prog.