bpf: tail calls in sleepable programs
Subsystems (4)
Jiri Olsa
reviewercover bpf: tail calls in sleepable programs
hi,
this patchset allows sleepable programs to use tail calls.
At the moment we need to have separate sleepable uprobe program to retrieve user space data and pass it to complex program with tail calls. It'd be great if the program with tail calls could be sleepable and do the data retrieval directly.
thanks,
jirka
---
Jiri Olsa (2):
bpf: Allow sleepable programs to use tail calls selftests/bpf: Add test for sleepable program tailcalls
include/linux/bpf.h | 1 + kernel/bpf/core.c | 4 +++- kernel/bpf/verifier.c | 1 + tools/testing/selftests/bpf/prog_tests/tailcalls.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/testing/selftests/bpf/progs/tailcall_sleepable.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 123 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/tailcall_sleepable.c
Jiri Olsa
reviewerPATCH 1/2 bpf: Allow sleepable programs to use tail calls
Allowing sleepable programs to use tail calls.
Making sure we can't mix sleepable and non-sleepable bpf programs in tail call map (BPF_MAP_TYPE_PROG_ARRAY) and allowing it to be used in sleepable programs.
Sleepable programs can be preempted and sleep which might bring new source of race conditions, but both direct and indirect tail calls should not be affected.
Direct tail calls work by patching direct jump to callee into bpf caller program, so no problem there. We atomically switch from nop to jump instruction.
Indirect tail call reads the callee from the map and then jumps to it. The callee bpf program can't disappear (be released) from the caller, because it is executed under rcu lock (rcu_read_lock_trace).
--- include/linux/bpf.h | 1 + kernel/bpf/core.c | 4 +++- kernel/bpf/verifier.c | 1 + 3 files changed, 5 insertions(+), 1 deletion(-)
include/linux/bpf.h +1 -0
@@ -287,6 +287,7 @@ struct bpf_map_owner {enum bpf_prog_type type;bool jited;bool xdp_has_frags; +bool sleepable;u64 storage_cookie[MAX_BPF_CGROUP_STORAGE_TYPE ];const struct btf_type *attach_func_proto;enum bpf_attach_type expected_attach_type;
kernel/bpf/core.c +3 -1
@@ -2401,6 +2401,7 @@ static bool __bpf_prog_map_compatible(struct bpf_map *map, map->owner->type = prog_type; map->owner->jited = fp->jited; map->owner->xdp_has_frags = aux->xdp_has_frags; + map->owner->sleepable = fp->sleepable; map->owner->expected_attach_type = fp->expected_attach_type; map->owner->attach_func_proto = aux->attach_func_proto;for_each_cgroup_storage_type (i) {
@@ -2412,7 +2413,8 @@ static bool __bpf_prog_map_compatible(struct bpf_map *map, }else { ret = map->owner->type == prog_type && map->owner->jited == fp->jited && - map->owner->xdp_has_frags == aux->xdp_has_frags; + map->owner->xdp_has_frags == aux->xdp_has_frags && + map->owner->sleepable == fp->sleepable;if (ret && map->map_type ==BPF_MAP_TYPE_PROG_ARRAY && map->owner->expected_attach_type != fp->expected_attach_type)
kernel/bpf/verifier.c +1 -0
@@ -21386,6 +21386,7 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env,case BPF_MAP_TYPE_STACK :case BPF_MAP_TYPE_ARENA :case BPF_MAP_TYPE_INSN_ARRAY : +case BPF_MAP_TYPE_PROG_ARRAY :break ;default : verbose(env,
Signature · 1 line
2.52.0
Re: [PATCH bpf-next 1/2] bpf: Allow sleepable programs to use tail calls
Quoted 64 lines · Jiri Olsa
Allowing sleepable programs to use tail calls.
Making sure we can't mix sleepable and non-sleepable bpf programs in tail call map (
BPF_MAP_TYPE_PROG_ARRAY) and allowing it to be used in sleepable programs.Sleepable programs can be preempted and sleep which might bring new source of race conditions, but both direct and indirect tail calls should not be affected.
Direct tail calls work by patching direct jump to callee into bpf caller program, so no problem there. We atomically switch from nop to jump instruction.
Indirect tail call reads the callee from the map and then jumps to it. The callee bpf program can't disappear (be released) from the caller, because it is executed under rcu lock (
rcu_read_lock_trace).Signed-off-by Jiri Olsa <[email protected]>--- include/linux/bpf.h | 1 + kernel/bpf/core.c | 4 +++- kernel/bpf/verifier.c | 1 + 3 files changed, 5 insertions(+), 1 deletion(-)include/linux/bpf.h +1
-0@@ -287,6 +287,7 @@ struct bpf_map_owner {enum bpf_prog_type type;bool jited;bool xdp_has_frags; +bool sleepable;u64 storage_cookie[MAX_BPF_CGROUP_STORAGE_TYPE ];const struct btf_type *attach_func_proto;enum bpf_attach_type expected_attach_type;kernel/bpf/core.c +3
-1@@ -2401,6 +2401,7 @@ static bool __bpf_prog_map_compatible(struct bpf_map *map, map->owner->type = prog_type; map->owner->jited = fp->jited; map->owner->xdp_has_frags = aux->xdp_has_frags; + map->owner->sleepable = fp->sleepable; map->owner->expected_attach_type = fp->expected_attach_type; map->owner->attach_func_proto = aux->attach_func_proto;for_each_cgroup_storage_type (i) {@@ -2412,7 +2413,8 @@ static bool __bpf_prog_map_compatible(struct bpf_map *map, }else { ret = map->owner->type == prog_type && map->owner->jited == fp->jited && - map->owner->xdp_has_frags == aux->xdp_has_frags; + map->owner->xdp_has_frags == aux->xdp_has_frags && + map->owner->sleepable == fp->sleepable;if (ret && map->map_type ==BPF_MAP_TYPE_PROG_ARRAY && map->owner->expected_attach_type != fp->expected_attach_type)kernel/bpf/verifier.c +0
-0
kernel/bpf/verifier.c static int check_map_prog_compatibility(struct bpf_verifier_env *env, · line 21392
Quoted 7 lines
kernel/bpf/verifier.c +1
-0@@ -21386,6 +21386,7 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env,case BPF_MAP_TYPE_STACK :case BPF_MAP_TYPE_ARENA :case BPF_MAP_TYPE_INSN_ARRAY : +case BPF_MAP_TYPE_PROG_ARRAY :break ;default : verbose(env,
This change also prevents a bad use case: a sleepable program FD could be inserted into a prog_array map whose owner is a non-sleepable program.
Looking at __bpf_prog_map_compatible(), are there any other program properties we should consider checking to ensure compatibility when updating a prog_array map?
From the tail call compatibility point of view, the restriction on mixing sleepable and non-sleepable programs looks correct to me.
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next 1/2] bpf: Allow sleepable programs to use tail calls
In reply to Jiri OlsaQuoted 51 lines · Jiri Olsa
Allowing sleepable programs to use tail calls.
Making sure we can't mix sleepable and non-sleepable bpf programs in tail call map (
BPF_MAP_TYPE_PROG_ARRAY) and allowing it to be used in sleepable programs.Sleepable programs can be preempted and sleep which might bring new source of race conditions, but both direct and indirect tail calls should not be affected.
Direct tail calls work by patching direct jump to callee into bpf caller program, so no problem there. We atomically switch from nop to jump instruction.
Indirect tail call reads the callee from the map and then jumps to it. The callee bpf program can't disappear (be released) from the caller, because it is executed under rcu lock (
rcu_read_lock_trace).Signed-off-by Jiri Olsa <[email protected]>--- include/linux/bpf.h | 1 + kernel/bpf/core.c | 4 +++- kernel/bpf/verifier.c | 1 + 3 files changed, 5 insertions(+), 1 deletion(-)include/linux/bpf.h +1
-0@@ -287,6 +287,7 @@ struct bpf_map_owner {enum bpf_prog_type type;bool jited;bool xdp_has_frags; +bool sleepable;u64 storage_cookie[MAX_BPF_CGROUP_STORAGE_TYPE ];const struct btf_type *attach_func_proto;enum bpf_attach_type expected_attach_type;kernel/bpf/core.c +1
-0@@ -2401,6 +2401,7 @@ static bool __bpf_prog_map_compatible(struct bpf_map *map, map->owner->type = prog_type; map->owner->jited = fp->jited; map->owner->xdp_has_frags = aux->xdp_has_frags; + map->owner->sleepable = fp->sleepable; map->owner->expected_attach_type = fp->expected_attach_type; map->owner->attach_func_proto = aux->attach_func_proto;for_each_cgroup_storage_type (i) {
kernel/bpf/core.c static bool __bpf_prog_map_compatible(struct bpf_map *map, · line 2417
Quoted 6 lines
kernel/bpf/core.c +2
-1@@ -2412,7 +2413,8 @@ static bool __bpf_prog_map_compatible(struct bpf_map *map, }else { ret = map->owner->type == prog_type && map->owner->jited == fp->jited && - map->owner->xdp_has_frags == aux->xdp_has_frags; + map->owner->xdp_has_frags == aux->xdp_has_frags && + map->owner->sleepable == fp->sleepable;
Sounds reasonable, but why not permit tail call of non-sleepable prog from sleepable prog?
The problem is only if it's the other way around.
Quoted 19 lines
kernel/bpf/core.c +0
-0if (ret && map->map_type ==BPF_MAP_TYPE_PROG_ARRAY && map->owner->expected_attach_type != fp->expected_attach_type)kernel/bpf/verifier.c +1
-1@@ -21386,6 +21386,7 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env,case BPF_MAP_TYPE_STACK :case BPF_MAP_TYPE_ARENA :case BPF_MAP_TYPE_INSN_ARRAY : +case BPF_MAP_TYPE_PROG_ARRAY :break ;default : verbose(env, --2.52.0
Re: [PATCH bpf-next 1/2] bpf: Allow sleepable programs to use tail calls
Quoted 55 lines · Kumar Kartikeya Dwivedi
Quoted 53 lines · Jiri Olsa
Allowing sleepable programs to use tail calls.
Making sure we can't mix sleepable and non-sleepable bpf programs in tail call map (
BPF_MAP_TYPE_PROG_ARRAY) and allowing it to be used in sleepable programs.Sleepable programs can be preempted and sleep which might bring new source of race conditions, but both direct and indirect tail calls should not be affected.
Direct tail calls work by patching direct jump to callee into bpf caller program, so no problem there. We atomically switch from nop to jump instruction.
Indirect tail call reads the callee from the map and then jumps to it. The callee bpf program can't disappear (be released) from the caller, because it is executed under rcu lock (
rcu_read_lock_trace).Signed-off-by Jiri Olsa <[email protected]>--- include/linux/bpf.h | 1 + kernel/bpf/core.c | 4 +++- kernel/bpf/verifier.c | 1 + 3 files changed, 5 insertions(+), 1 deletion(-)include/linux/bpf.h +1
-0@@ -287,6 +287,7 @@ struct bpf_map_owner {enum bpf_prog_type type;bool jited;bool xdp_has_frags; +bool sleepable;u64 storage_cookie[MAX_BPF_CGROUP_STORAGE_TYPE ];const struct btf_type *attach_func_proto;enum bpf_attach_type expected_attach_type;kernel/bpf/core.c +1
-0@@ -2401,6 +2401,7 @@ static bool __bpf_prog_map_compatible(struct bpf_map *map, map->owner->type = prog_type; map->owner->jited = fp->jited; map->owner->xdp_has_frags = aux->xdp_has_frags; + map->owner->sleepable = fp->sleepable; map->owner->expected_attach_type = fp->expected_attach_type; map->owner->attach_func_proto = aux->attach_func_proto;for_each_cgroup_storage_type (i) {@@ -2412,7 +2413,8 @@ static bool __bpf_prog_map_compatible(struct bpf_map *map, }else { ret = map->owner->type == prog_type &&
Quoted 8 lines
kernel/bpf/core.c static bool __bpf_prog_map_compatible(struct bpf_map *map, · line 2417
Quoted 4 lines
kernel/bpf/core.c +2
-1map->owner->jited == fp->jited && - map->owner->xdp_has_frags == aux->xdp_has_frags; + map->owner->xdp_has_frags == aux->xdp_has_frags && + map->owner->sleepable == fp->sleepable;Sounds reasonable, but why not permit tail call of non-sleepable prog from sleepable prog?
The problem is only if it's the other way around.
I think that needs deeper discussion.
sleepable only does rcu_task_trace while non-sleepable relies on implicit rcu.rcu_trace_implies_rcu_gp() is always true and this helper will be removed soon,
but I would still consult Paul and think through the implications.
So I'm going to apply it as-is.
Jiri Olsa
reviewerPATCH 2/2 selftests/bpf: Add test for sleepable program tailcalls
In reply to Jiri OlsaAdding test that makes sure we can't mix sleepable and non-sleepable bpf programs in the BPF_MAP_TYPE_PROG_ARRAY map and that we can do tail call in the sleepable program.
--- .../selftests/bpf/prog_tests/tailcalls.c | 74 +++++++++++++++++++ .../selftests/bpf/progs/tailcall_sleepable.c | 43 +++++++++++ 2 files changed, 117 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/tailcall_sleepable.c
tools/testing/selftests/bpf/prog_tests/tailcalls.c +74 -0
@@ -8,6 +8,7 @@#include "tailcall_freplace.skel.h" #include "tc_bpf2bpf.skel.h" #include "tailcall_fail.skel.h" +#include "tailcall_sleepable.skel.h" /* test_tailcall_1 checks basic functionality by patching multiple locations * in a single program for a single tail call slot with nop->jmp, jmp->nop
@@ -1653,6 +1654,77 @@ static void test_tailcall_failure()RUN_TESTS (tailcall_fail); } +noinline void uprobe_sleepable_trigger(void) +{ + asmvolatile ("" ); +} + +static void test_tailcall_sleepable (void ) +{ +LIBBPF_OPTS (bpf_uprobe_opts, opts); +struct tailcall_sleepable *skel; +int prog_fd, map_fd; +int err, key; + + skel =tailcall_sleepable__open (); +if (!ASSERT_OK_PTR (skel,"tailcall_sleepable__open" )) +return ; + +/* +* Test that we can't load uprobe_normal and uprobe_sleepable_1, +* because they share tailcall map. +*/ +bpf_program__set_autoload (skel->progs.uprobe_normal, true); +bpf_program__set_autoload (skel->progs.uprobe_sleepable_1, true); + + err =tailcall_sleepable__load (skel); +if (!ASSERT_ERR (err,"tailcall_sleepable__load" )) + goto out; + +tailcall_sleepable__destroy (skel); + +/* +* Test that we can tail call from sleepable to sleepable program. +*/ + skel =tailcall_sleepable__open (); +if (!ASSERT_OK_PTR (skel,"tailcall_sleepable__open" )) +return ; + +bpf_program__set_autoload (skel->progs.uprobe_sleepable_1, true); +bpf_program__set_autoload (skel->progs.uprobe_sleepable_2, true); + + err =tailcall_sleepable__load (skel); +if (!ASSERT_OK (err,"tailcall_sleepable__load" )) + goto out; + +/* Add sleepable uprobe_sleepable_2 to jmp_table[0]. */ + key =0 ; + prog_fd =bpf_program__fd (skel->progs.uprobe_sleepable_2); + map_fd =bpf_map__fd (skel->maps.jmp_table); + err =bpf_map_update_elem (map_fd, &key, &prog_fd,BPF_ANY ); +if (!ASSERT_OK (err,"update jmp_table" )) + goto out; + + skel->bss->my_pid =getpid (); + +/* Attach uprobe_sleepable_1 to uprobe_sleepable_trigger and hit it. */ + opts.func_name ="uprobe_sleepable_trigger" ; + skel->links.uprobe_sleepable_1 =bpf_program__attach_uprobe_opts ( + skel->progs.uprobe_sleepable_1, +-1 , +"/proc/self/exe" , +0 /* offset */ , + &opts); +if (!ASSERT_OK_PTR (skel->links.uprobe_sleepable_1,"bpf_program__attach_uprobe_opts" )) + goto out; + +uprobe_sleepable_trigger (); +ASSERT_EQ (skel->bss->executed,1 ,"executed" ); + +out: +tailcall_sleepable__destroy (skel); +} +void test_tailcalls (void ) {if (test__start_subtest ("tailcall_1" ))
@@ -1707,4 +1779,6 @@ void test_tailcalls(void)test_tailcall_bpf2bpf_freplace ();if (test__start_subtest ("tailcall_failure" ))test_tailcall_failure (); +if (test__start_subtest ("tailcall_sleepable" )) +test_tailcall_sleepable (); }
tools/testing/selftests/bpf/progs/tailcall_sleepable.c new file +43 -0
43 lines not on this page · raw message
Signature · 1 line
2.52.0
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 9 lines · On Fri, 30 Jan 2026 09:12:06 +0100 you wrote:
hi,
this patchset allows sleepable programs to use tail calls.At the moment we need to have separate sleepable uprobe program to retrieve user space data and pass it to complex program with tail calls. It'd be great if the program with tail calls could be sleepable and do the data retrieval directly.
[...]
Here is the summary with links:
- [bpf-next,1/2] bpf: Allow sleepable programs to use tail calls
https://git.kernel.org/bpf/bpf-next/c/0f0c332992b8 - [bpf-next,2/2] selftests/bpf: Add test for sleepable program tailcalls https://git.kernel.org/bpf/bpf-next/c/15ac1adf0f84
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html