Skip to contentSkip to conversation

bpf: Add __aux tag to pass in prog->aux

PATCH Superseded Kumar Kartikeya Dwivedi 3 participants 7 messages May 12, 2025 – May 13, 2025

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Kumar Kartikeya Dwivedi

author maintainer
#0 lore raw

PATCH 1/1 bpf: Add __aux tag to pass in prog->aux

Instead of hardcoding the list of kfuncs that need prog->aux passed to them with a combination of fixup_kfunc_call adjustment + __ign suffix, combine both in __aux suffix, which ignores the argument passed in, and fixes it up to the prog->aux. This allows kfuncs to have the prog->aux passed into them without having to touch the verifier.

---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/helpers.c         |  4 ++--
 kernel/bpf/verifier.c        | 33 +++++++++++++++++++++++++++------
 3 files changed, 30 insertions(+), 8 deletions(-)
include/linux/bpf_verifier.h +1 -0
index 9734544b6957..1d90e44a1d04 100644
@@ -606,6 +606,7 @@ struct bpf_insn_aux_data {
 	bool calls_callback;
 	/* registers alive before this instruction. */
 	u16 live_regs_before;
+	u16 arg_prog_aux;
 };
 
 #define MAX_USED_MAPS 64 /* max number of maps accessed by one eBPF program */
kernel/bpf/helpers.c +2 -2
index fed53da75025..2b6bac4bf6e3 100644
@@ -3012,9 +3012,9 @@ __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags)
 __bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq,
 					 int (callback_fn)(void *map, int *key, void *value),
 					 unsigned int flags,
-					 void *aux__ign)
+					 void *aux__aux)
 {
-	struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__ign;
+	struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__aux;
 	struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
 
 	if (flags)
kernel/bpf/verifier.c +27 -6
index 28f5a7899bd6..151bd18c086a 100644
@@ -343,6 +343,7 @@ struct bpf_kfunc_call_arg_meta {
 		int uid;
 	} map;
 	u64 mem_size;
+	u32 arg_prog_aux;
 };
 
 struct btf *btf_vmlinux;
@@ -11897,6 +11898,11 @@ static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_param
 	return btf_param_match_suffix(btf, arg, "__irq_flag");
 }
 
+static bool is_kfunc_arg_prog_aux(const struct btf *btf, const struct btf_param *arg)
+{
+	return btf_param_match_suffix(btf, arg, "__aux");
+}
+
 static bool is_kfunc_arg_scalar_with_name(const struct btf *btf,
 					  const struct btf_param *arg,
 					  const char *name)
@@ -12938,6 +12944,17 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 		if (is_kfunc_arg_ignore(btf, &args[i]))
 			continue;
 
+		if (is_kfunc_arg_prog_aux(btf, &args[i])) {
+			/* Used to reject repeated use of __aux. */
+			if (meta->arg_prog_aux) {
+				verbose(env, "Only 1 prog->aux argument supported per-kfunc\n");
+				return -EFAULT;
+			}
+			meta->arg_prog_aux = regno;
+			cur_aux(env)->arg_prog_aux = regno;
+			continue;
+		}
+
 		if (btf_type_is_scalar(t)) {
 			if (reg->type != SCALAR_VALUE) {
 				verbose(env, "R%d is not a scalar\n", regno);
@@ -21517,13 +21534,17 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		   desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) {
 		insn_buf[0] = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1);
 		*cnt = 1;
-	} else if (is_bpf_wq_set_callback_impl_kfunc(desc->func_id)) {
-		struct bpf_insn ld_addrs[2] = { BPF_LD_IMM64(BPF_REG_4, (long)env->prog->aux) };
+	}
 
-		insn_buf[0] = ld_addrs[0];
-		insn_buf[1] = ld_addrs[1];
-		insn_buf[2] = *insn;
-		*cnt = 3;
+	if (env->insn_aux_data[insn_idx].arg_prog_aux) {
+		u32 regno = env->insn_aux_data[insn_idx].arg_prog_aux;
+		struct bpf_insn ld_addrs[2] = { BPF_LD_IMM64(regno, (long)env->prog->aux) };
+		int idx = *cnt;
+
+		insn_buf[idx++] = ld_addrs[0];
+		insn_buf[idx++] = ld_addrs[1];
+		insn_buf[idx++] = *insn;
+		*cnt = idx;
 	}
 	return 0;
 }

--
2.47.1

#1 lore raw
Quoted 6 lines · Kumar Kartikeya Dwivedi

Instead of hardcoding the list of kfuncs that need prog->aux passed to them with a combination of fixup_kfunc_call adjustment + __ign suffix, combine both in __aux suffix, which ignores the argument passed in, and fixes it up to the prog->aux. This allows kfuncs to have the prog->aux passed into them without having to touch the verifier.

I have this wet dream that one day we'll make sure that all BPF programs have bpf_run_ctx set up for them, across all program types, and we'll just use that for these "BPF environment"-like things like getting currently-running bpf_prog/bpf_prog_aux pointer.

Do you think it makes sense? One of the concerns for wiring bpf_run_ctx for, say, XDP programs was perceived potential tiny perf regression (but it's just current->bpf_ctx swap, so shouldn't be a big deal at all, IMO) and just generally no immediate use case. So maybe this bpf_prog_aux access is a good enough reason now, WDYT?

Quoted 9 lines
---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/helpers.c         |  4 ++--
 kernel/bpf/verifier.c        | 33 +++++++++++++++++++++++++++------
 3 files changed, 30 insertions(+), 8 deletions(-)

[...]

author maintainer
#2 lore raw
Quoted 10 lines · Andrii Nakryiko
Quoted 6 lines · Kumar Kartikeya Dwivedi

Instead of hardcoding the list of kfuncs that need prog->aux passed to them with a combination of fixup_kfunc_call adjustment + __ign suffix, combine both in __aux suffix, which ignores the argument passed in, and fixes it up to the prog->aux. This allows kfuncs to have the prog->aux passed into them without having to touch the verifier.

Quoted 5 lines

I have this wet dream that one day we'll make sure that all BPF programs have bpf_run_ctx set up for them, across all program types, and we'll just use that for these "BPF environment"-like things like getting currently-running bpf_prog/bpf_prog_aux pointer.

For this particular patch, it's mostly refactoring. But I agree in general having bpf_run_ctx everywhere would be useful. Currently being able to rely on it in some cases and not in others is a bit annoying.

Quoted 4 lines

Do you think it makes sense? One of the concerns for wiring bpf_run_ctx for, say, XDP programs was perceived potential tiny perf regression (but it's just current->bpf_ctx swap, so shouldn't be a big deal at all, IMO) and just generally no immediate use case.

I think the performance concern may be real. I can do some benchmarking with xdp-bench, but it's probably not the raw swaps that cost too much but potential cache miss on touching current. I am not sure how problematic that would be in terms of pps regression for small programs, but even if amortized it may end up regressing when done unconditionally.

XDP is sort of an edge case though. I am not sure the concern on ns-scale effects holds for any other program type.

Quoted 14 lines

So maybe
this bpf_prog_aux access is a good enough reason now, WDYT?

Quoted 9 lines
---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/helpers.c         |  4 ++--
 kernel/bpf/verifier.c        | 33 +++++++++++++++++++++++++++------
 3 files changed, 30 insertions(+), 8 deletions(-)

[...]

#3 lore raw
Quoted 18 lines · Kumar Kartikeya Dwivedi
Quoted 10 lines · Andrii Nakryiko
Quoted 6 lines · Kumar Kartikeya Dwivedi

Instead of hardcoding the list of kfuncs that need prog->aux passed to them with a combination of fixup_kfunc_call adjustment + __ign suffix, combine both in __aux suffix, which ignores the argument passed in, and fixes it up to the prog->aux. This allows kfuncs to have the prog->aux passed into them without having to touch the verifier.

Quoted 5 lines

I have this wet dream that one day we'll make sure that all BPF programs have bpf_run_ctx set up for them, across all program types, and we'll just use that for these "BPF environment"-like things like getting currently-running bpf_prog/bpf_prog_aux pointer.

Quoted 4 lines

For this particular patch, it's mostly refactoring. But I agree in general having bpf_run_ctx everywhere would be useful. Currently being able to rely on it in some cases and not in others is a bit annoying.

Yep, agreed.

Quoted 6 lines
Quoted 5 lines

Do you think it makes sense? One of the concerns for wiring bpf_run_ctx for, say, XDP programs was perceived potential tiny perf regression (but it's just current->bpf_ctx swap, so shouldn't be a big deal at all, IMO) and just generally no immediate use case.

Quoted 6 lines

I think the performance concern may be real. I can do some benchmarking with xdp-bench, but it's probably not the raw swaps that cost too much but potential cache miss on touching current. I am not sure how problematic that would be in terms of pps regression for small programs, but even if amortized it may end up regressing when done unconditionally.

Yep, cache line access is the expensive thing, not the swap itself, of course. My hope is that current is frequently accessed enough and we might get lucky with that cache line already being present. Or if XDP calls are frequent enough, we'll just amortize the cost across multiple packets, as you mentioned.

But yes, maybe a bit of benchmarking would give us a bit better idea, thanks!

Quoted 3 lines

XDP is sort of an edge case though. I am not sure the concern on ns-scale effects holds for any other program type.

Sure, XDP is an extreme case, but if bpf_run_ctx is everywhere *but* XDP, that still makes it a non-universal mechanism, and we'll have to always remember that some run_ctx-related functionality doesn't work on XDP. Maybe that's fine, I don't know, but it certainly would be better to not have these exceptions.

Quoted 15 lines
Quoted 15 lines

So maybe
this bpf_prog_aux access is a good enough reason now, WDYT?

Quoted 9 lines
---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/helpers.c         |  4 ++--
 kernel/bpf/verifier.c        | 33 +++++++++++++++++++++++++++------
 3 files changed, 30 insertions(+), 8 deletions(-)

[...]

#4 lore raw
In reply to Kumar Kartikeya Dwivedi
Quoted 32 lines · Kumar Kartikeya Dwivedi

Instead of hardcoding the list of kfuncs that need prog->aux passed to them with a combination of fixup_kfunc_call adjustment + __ign suffix, combine both in __aux suffix, which ignores the argument passed in, and fixes it up to the prog->aux. This allows kfuncs to have the prog->aux passed into them without having to touch the verifier.

---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/helpers.c         |  4 ++--
 kernel/bpf/verifier.c        | 33 +++++++++++++++++++++++++++------
 3 files changed, 30 insertions(+), 8 deletions(-)
include/linux/bpf_verifier.h +1 -0
index 9734544b6957..1d90e44a1d04 100644
@@ -606,6 +606,7 @@ struct bpf_insn_aux_data {
        bool calls_callback;
        /* registers alive before this instruction. */
        u16 live_regs_before;
+       u16 arg_prog_aux;
 };
 
 #define MAX_USED_MAPS 64 /* max number of maps accessed by one eBPF program */
kernel/bpf/helpers.c +0 -0
index fed53da75025..2b6bac4bf6e3 100644

kernel/bpf/helpers.c __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) · line 3015

Quoted 5 lines
kernel/bpf/helpers.c +1 -1
@@ -3012,9 +3012,9 @@ __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags)
 __bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq,
                                         int (callback_fn)(void *map, int *key, void *value),
                                         unsigned int flags,
-                                        void *aux__ign)
+                                        void *aux__aux)

aux__aux is an odd name.
"__aux" as a suffix also looks strange.

How about "__prog" suffix ?
It will be similar to the existing "__map" suffix.

We can also standardize the argument name as __bpf_kfunc int bpf_wq_set_callback_impl(.. , void *aux__prog)

then the name is more or less explanatory.

kernel/bpf/helpers.c __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) · line 3017

Quoted 3 lines
kernel/bpf/helpers.c +1 -1
 {
-       struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__ign;
+       struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__aux;

and here it will be:

+ struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__prog;

which looks ok to me.

pw-bot: cr

author maintainer
#5 lore raw
Quoted 29 lines · Alexei Starovoitov
Quoted 26 lines · Kumar Kartikeya Dwivedi

Instead of hardcoding the list of kfuncs that need prog->aux passed to them with a combination of fixup_kfunc_call adjustment + __ign suffix, combine both in __aux suffix, which ignores the argument passed in, and fixes it up to the prog->aux. This allows kfuncs to have the prog->aux passed into them without having to touch the verifier.

---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/helpers.c         |  4 ++--
 kernel/bpf/verifier.c        | 33 +++++++++++++++++++++++++++------
 3 files changed, 30 insertions(+), 8 deletions(-)
include/linux/bpf_verifier.h +1 -0
index 9734544b6957..1d90e44a1d04 100644
@@ -606,6 +606,7 @@ struct bpf_insn_aux_data {
        bool calls_callback;
        /* registers alive before this instruction. */
        u16 live_regs_before;
+       u16 arg_prog_aux;
 };
 
Quoted 15 lines

kernel/bpf/helpers.c __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) · line 3015

Quoted 11 lines
include/linux/bpf_verifier.h +0 -0
 #define MAX_USED_MAPS 64 /* max number of maps accessed by one eBPF program */
kernel/bpf/helpers.c +1 -1
index fed53da75025..2b6bac4bf6e3 100644
@@ -3012,9 +3012,9 @@ __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags)
 __bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq,
                                         int (callback_fn)(void *map, int *key, void *value),
                                         unsigned int flags,
-                                        void *aux__ign)
+                                        void *aux__aux)

aux__aux is an odd name.
"__aux" as a suffix also looks strange.

We can call it prog__aux.

Quoted 2 lines

How about "__prog" suffix ?
It will be similar to the existing "__map" suffix.

But it's a bit misleading, it's not the prog, it's prog->aux.

Quoted 5 lines

We can also standardize the argument name as __bpf_kfunc int bpf_wq_set_callback_impl(.. , void *aux__prog)

then the name is more or less explanatory.

You can call it foo__prog, the part before __ is arbitrary.

Quoted 10 lines

kernel/bpf/helpers.c __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) · line 3017

Quoted 4 lines
kernel/bpf/helpers.c +1 -1
 {
-       struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__ign;
+       struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__aux;

and here it will be:

+ struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__prog;

which looks ok to me.

How about I rename the parameter to prog__aux?
Or __prog_aux as the tag name?

Quoted 2 lines

pw-bot: cr

#6 lore raw
Quoted 48 lines · Kumar Kartikeya Dwivedi
Quoted 29 lines · Alexei Starovoitov
Quoted 26 lines · Kumar Kartikeya Dwivedi

Instead of hardcoding the list of kfuncs that need prog->aux passed to them with a combination of fixup_kfunc_call adjustment + __ign suffix, combine both in __aux suffix, which ignores the argument passed in, and fixes it up to the prog->aux. This allows kfuncs to have the prog->aux passed into them without having to touch the verifier.

---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/helpers.c         |  4 ++--
 kernel/bpf/verifier.c        | 33 +++++++++++++++++++++++++++------
 3 files changed, 30 insertions(+), 8 deletions(-)
include/linux/bpf_verifier.h +1 -0
index 9734544b6957..1d90e44a1d04 100644
@@ -606,6 +606,7 @@ struct bpf_insn_aux_data {
        bool calls_callback;
        /* registers alive before this instruction. */
        u16 live_regs_before;
+       u16 arg_prog_aux;
 };
 
Quoted 15 lines

kernel/bpf/helpers.c __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) · line 3015

Quoted 11 lines
include/linux/bpf_verifier.h +0 -0
 #define MAX_USED_MAPS 64 /* max number of maps accessed by one eBPF program */
kernel/bpf/helpers.c +1 -1
index fed53da75025..2b6bac4bf6e3 100644
@@ -3012,9 +3012,9 @@ __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags)
 __bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq,
                                         int (callback_fn)(void *map, int *key, void *value),
                                         unsigned int flags,
-                                        void *aux__ign)
+                                        void *aux__aux)

aux__aux is an odd name.
"__aux" as a suffix also looks strange.

Quoted 6 lines

We can call it prog__aux.

Quoted 2 lines

How about "__prog" suffix ?
It will be similar to the existing "__map" suffix.

But it's a bit misleading, it's not the prog, it's prog->aux.

prog or prog->aux is the same thing. It's a pointer to prog. bpf_prog is read only part of it.
bpf_prog_uax is writeable.

Quoted 14 lines
Quoted 6 lines

We can also standardize the argument name as __bpf_kfunc int bpf_wq_set_callback_impl(.. , void *aux__prog)

then the name is more or less explanatory.

You can call it foo__prog, the part before __ is arbitrary.

Quoted 5 lines
Quoted 4 lines
kernel/bpf/helpers.c +1 -1
 {
-       struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__ign;
+       struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__aux;
Quoted 8 lines
Quoted 5 lines

and here it will be:

+ struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__prog;

which looks ok to me.

How about I rename the parameter to prog__aux?
Or __prog_aux as the tag name?

I still prefer aux__prog.