bpf: Fix tailcall infinite loop caused by freplace
Subsystems (4)
v8 cover bpf: Fix tailcall infinite loop caused by freplace
Previously, I addressed a tailcall infinite loop issue related to trampolines[0].
In this patchset, I resolve a similar issue where a tailcall infinite loop can occur due to the combination of tailcalls and freplace programs. The fix prevents adding extended programs to the prog_array map and blocks the extension of a tail callee program with freplace.
Key changes:
1. If a program or its subprogram has been extended by an freplace program, it can no longer be updated to a prog_array map.
2. If a program has been added to a prog_array map, neither it nor its subprograms can be extended by an freplace program.
Additionally, an extension program should not be tailcalled. As a result, return -EINVAL if the program has a type of BPF_PROG_TYPE_EXT when adding it to a prog_array map.
Changes:
v7 -> v8:
* Address comment from Alexei:
* guard(mutex) should not hold range all the way through bpf_arch_text_poke().
* Address suggestion from Xu Kuohai:
* Extension prog should not be tailcalled independently.
v6 -> v7:
* Address comments from Alexei:
* Rewrite commit message more imperative and consice with AI.
* Extend bpf_trampoline_link_prog() and bpf_trampoline_unlink_prog() to link and unlink target prog for freplace prog.
* Use guard(mutex)(&tgt_prog->aux->ext_mutex) instead of mutex_lock()&mutex_unlock() pair.
* Address comment from Eduard:
* Remove misplaced "Reported-by" and "Closes" tags.
v5 -> v6:
* Fix a build warning reported by kernel test robot.
v4 -> v5:
* Move code of linking/unlinking target prog of freplace to trampoline.c.
* Address comments from Alexei:
* Change type of prog_array_member_cnt to u64.
* Combine two patches to one.
v3 -> v4:
* Address comments from Eduard:
* Rename 'tail_callee_cnt' to 'prog_array_member_cnt'.
* Add comment to 'prog_array_member_cnt'.
* Use a mutex to protect 'is_extended' and 'prog_array_member_cnt'.
v2 -> v3:
* Address comments from Alexei:
* Stop hacking JIT.
* Prevent the specific use case at attach/update time.
v1 -> v2:
* Address comment from Eduard:
* Explain why nop5 and xor/nop3 are swapped at prologue.
* Address comment from Alexei:
* Disallow attaching tail_call_reachable freplace prog to not-tail_call_reachable target in verifier.
* Update "bpf, arm64: Fix tailcall infinite loop caused by freplace" with latest arm64 JIT code.
Links:
[0] https://lore.kernel.org/bpf/[email protected]/
Leon Hwang (2):
bpf: Prevent tailcall infinite loop caused by freplace selftests/bpf: Add test to verify tailcall and freplace restrictions
include/linux/bpf.h | 17 ++- kernel/bpf/arraymap.c | 26 +++- kernel/bpf/core.c | 1 + kernel/bpf/syscall.c | 7 +- kernel/bpf/trampoline.c | 47 +++++-- .../selftests/bpf/prog_tests/tailcalls.c | 120 ++++++++++++++++-- .../testing/selftests/bpf/progs/tc_bpf2bpf.c | 5 +- 7 files changed, 190 insertions(+), 33 deletions(-)
Signature · 1 line
2.44.0
PATCH v8 1/2 bpf: Prevent tailcall infinite loop caused by freplace
There is a potential infinite loop issue that can occur when using a combination of tail calls and freplace.
In an upcoming selftest, the attach target for entry_freplace of tailcall_freplace.c is subprog_tc of tc_bpf2bpf.c, while the tail call in entry_freplace leads to entry_tc. This results in an infinite loop:
entry_tc -> subprog_tc -> entry_freplace --tailcall-> entry_tc.
The problem arises because the tail_call_cnt in entry_freplace resets to zero each time entry_freplace is executed, causing the tail call mechanism to never terminate, eventually leading to a kernel panic.
To fix this issue, the solution is twofold:
1. Prevent updating a program extended by an freplace program to a prog_array map.
2. Prevent extending a program that is already part of a prog_array map with an freplace program.
This ensures that:
* If a program or its subprogram has been extended by an freplace program, it can no longer be updated to a prog_array map.
* If a program has been added to a prog_array map, neither it nor its subprograms can be extended by an freplace program.
Moreover, an extension program should not be tailcalled. As such, return -EINVAL if the program has a type of BPF_PROG_TYPE_EXT when adding it to a prog_array map.
Additionally, fix a minor code style issue by replacing eight spaces with a tab for proper formatting.
--- include/linux/bpf.h | 17 +++++++++++---- kernel/bpf/arraymap.c | 26 +++++++++++++++++++++-- kernel/bpf/core.c | 1 + kernel/bpf/syscall.c | 7 +++--- kernel/bpf/trampoline.c | 47 ++++++++++++++++++++++++++++++++++------- 5 files changed, 81 insertions(+), 17 deletions(-)
include/linux/bpf.h +13 -4
@@ -1292,8 +1292,12 @@ void *__bpf_dynptr_data_rw(const struct bpf_dynptr_kern *ptr, u32 len);bool __bpf_dynptr_is_rdonly (const struct bpf_dynptr_kern *ptr);#ifdef CONFIG_BPF_JIT -int bpf_trampoline_link_prog (struct bpf_tramp_link *link,struct bpf_trampoline *tr); -int bpf_trampoline_unlink_prog (struct bpf_tramp_link *link,struct bpf_trampoline *tr); +int bpf_trampoline_link_prog (struct bpf_tramp_link *link, +struct bpf_trampoline *tr, +struct bpf_prog *tgt_prog); +int bpf_trampoline_unlink_prog (struct bpf_tramp_link *link, +struct bpf_trampoline *tr, +struct bpf_prog *tgt_prog);struct bpf_trampoline *bpf_trampoline_get (u64 key,struct bpf_attach_target_info *tgt_info);void bpf_trampoline_put (struct bpf_trampoline *tr);
@@ -1374,12 +1378,14 @@ void bpf_jit_uncharge_modmem(u32 size);bool bpf_prog_has_trampoline (const struct bpf_prog *prog);#else static inline int bpf_trampoline_link_prog (struct bpf_tramp_link *link, -struct bpf_trampoline *tr) +struct bpf_trampoline *tr, +struct bpf_prog *tgt_prog) {return -ENOTSUPP ; }static inline int bpf_trampoline_unlink_prog (struct bpf_tramp_link *link, -struct bpf_trampoline *tr) +struct bpf_trampoline *tr, +struct bpf_prog *tgt_prog) {return -ENOTSUPP ; }
@@ -1483,6 +1489,9 @@ struct bpf_prog_aux {bool xdp_has_frags;bool exception_cb;bool exception_boundary; +bool is_extended;/* true if extended by freplace program */ +u64 prog_array_member_cnt;/* counts how many times as member of prog_array */ +struct mutex ext_mutex;/* mutex for is_extended and prog_array_member_cnt */ struct bpf_arena *arena;/* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto;
kernel/bpf/arraymap.c +24 -2
@@ -947,22 +947,44 @@ static void *prog_fd_array_get_ptr(struct bpf_map *map, struct file *map_file, int fd) {struct bpf_prog *prog =bpf_prog_get (fd); +bool is_extended;if (IS_ERR (prog))return prog; -if (!bpf_prog_map_compatible (map, prog)) { +if (prog->type ==BPF_PROG_TYPE_EXT || + !bpf_prog_map_compatible (map, prog)) {bpf_prog_put (prog);return ERR_PTR (-EINVAL ); } +mutex_lock (&prog->aux->ext_mutex); + is_extended = prog->aux->is_extended; +if (!is_extended) + prog->aux->prog_array_member_cnt++; +mutex_unlock (&prog->aux->ext_mutex); +if (is_extended) { +/* Extended prog can not be tail callee. It's to prevent a +* potential infinite loop like: +* tail callee prog entry -> tail callee prog subprog -> +* freplace prog entry --tailcall-> tail callee prog entry. +*/ +bpf_prog_put (prog); +return ERR_PTR (-EBUSY ); + } +return prog; }static void prog_fd_array_put_ptr (struct bpf_map *map,void *ptr,bool need_defer) { +struct bpf_prog *prog = ptr; + +mutex_lock (&prog->aux->ext_mutex); + prog->aux->prog_array_member_cnt--; +mutex_unlock (&prog->aux->ext_mutex);/* bpf_prog is freed after one RCU or tasks trace grace period */ -bpf_prog_put (ptr); +bpf_prog_put (prog); }static u32 prog_fd_array_sys_lookup_elem (void *ptr)
kernel/bpf/core.c +1 -0
@@ -131,6 +131,7 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flagINIT_LIST_HEAD_RCU (&fp->aux->ksym_prefix.lnode);#endif mutex_init (&fp->aux->used_maps_mutex); +mutex_init (&fp->aux->ext_mutex);mutex_init (&fp->aux->dst_mutex);return fp;
kernel/bpf/syscall.c +4 -3
@@ -3214,7 +3214,8 @@ static void bpf_tracing_link_release(struct bpf_link *link) container_of(link, struct bpf_tracing_link, link.link);WARN_ON_ONCE (bpf_trampoline_unlink_prog (&tr_link->link, - tr_link->trampoline)); + tr_link->trampoline, + tr_link->tgt_prog));bpf_trampoline_put (tr_link->trampoline);
@@ -3354,7 +3355,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,* in prog->aux * * - if prog->aux->dst_trampoline is NULL, the program has already been -* attached to a target and its initial target was cleared (below) +* attached to a target and its initial target was cleared (below) * * - if tgt_prog != NULL, the caller specified tgt_prog_fd + * target_btf_id using the link_create API.
@@ -3429,7 +3430,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,if (err) goto out_unlock; - err =bpf_trampoline_link_prog (&link->link, tr); + err =bpf_trampoline_link_prog (&link->link, tr, tgt_prog);if (err) {bpf_link_cleanup (&link_primer); link =NULL ;
kernel/bpf/trampoline.c +39 -8
101 lines not on this page · raw message
Signature · 1 line
2.44.0
PATCH v8 2/2 selftests/bpf: Add test to verify tailcall and freplace restrictions
In reply to Leon HwangAdd a test case to ensure that attaching a tail callee program with an freplace program fails, and that updating an extended program to a prog_array map is also prohibited.
This test is designed to prevent the potential infinite loop issue caused by the combination of tail calls and freplace, ensuring the correct behavior and stability of the system.
Additionally, fix the broken tailcalls/tailcall_freplace selftest because an extension prog should not be tailcalled.
cd tools/testing/selftests/bpf; ./test_progs -t tailcalls
337/25 tailcalls/tailcall_freplace:OK 337/26 tailcalls/tailcall_bpf2bpf_freplace:OK 337 tailcalls:OK
Summary: 1/26 PASSED, 0 SKIPPED, 0 FAILED
--- .../selftests/bpf/prog_tests/tailcalls.c | 120 ++++++++++++++++-- .../testing/selftests/bpf/progs/tc_bpf2bpf.c | 5 +- 2 files changed, 109 insertions(+), 16 deletions(-)
tools/testing/selftests/bpf/prog_tests/tailcalls.c +106 -14
@@ -1496,8 +1496,8 @@ static void test_tailcall_bpf2bpf_hierarchy_3(void)RUN_TESTS (tailcall_bpf2bpf_hierarchy3); } -/* test_tailcall_freplace checks that the attached freplace prog is OK to -* update the prog_array map. +/* test_tailcall_freplace checks that the freplace prog fails to update the +* prog_array map, no matter whether the freplace prog attaches to its target. */ static void test_tailcall_freplace (void ) {
@@ -1505,7 +1505,7 @@ static void test_tailcall_freplace(void)struct bpf_link *freplace_link =NULL ;struct bpf_program *freplace_prog;struct tc_bpf2bpf *tc_skel =NULL ; -int prog_fd, map_fd; +int prog_fd, tc_prog_fd, map_fd;char buff[128 ] = {};int err, key;
@@ -1523,9 +1523,10 @@ static void test_tailcall_freplace(void)if (!ASSERT_OK_PTR (tc_skel,"tc_bpf2bpf__open_and_load" )) goto out; - prog_fd =bpf_program__fd (tc_skel->progs.entry_tc); + tc_prog_fd =bpf_program__fd (tc_skel->progs.entry_tc); freplace_prog = freplace_skel->progs.entry_freplace; - err =bpf_program__set_attach_target (freplace_prog, prog_fd,"subprog" ); + err =bpf_program__set_attach_target (freplace_prog, tc_prog_fd, +"subprog_tc" );if (!ASSERT_OK (err,"set_attach_target" )) goto out;
@@ -1533,27 +1534,116 @@ static void test_tailcall_freplace(void)if (!ASSERT_OK (err,"tailcall_freplace__load" )) goto out; - freplace_link =bpf_program__attach_freplace (freplace_prog, prog_fd, -"subprog" ); + map_fd =bpf_map__fd (freplace_skel->maps.jmp_table); + prog_fd =bpf_program__fd (freplace_prog); + key =0 ; + err =bpf_map_update_elem (map_fd, &key, &prog_fd,BPF_ANY ); +ASSERT_ERR (err,"update jmp_table failure" ); + + freplace_link =bpf_program__attach_freplace (freplace_prog, tc_prog_fd, +"subprog_tc" );if (!ASSERT_OK_PTR (freplace_link,"attach_freplace" )) goto out; - map_fd =bpf_map__fd (freplace_skel->maps.jmp_table); - prog_fd =bpf_program__fd (freplace_prog); + err =bpf_map_update_elem (map_fd, &key, &prog_fd,BPF_ANY ); +ASSERT_ERR (err,"update jmp_table failure" ); + +out: +bpf_link__destroy (freplace_link); +tailcall_freplace__destroy (freplace_skel); +tc_bpf2bpf__destroy (tc_skel); +} + +/* test_tailcall_bpf2bpf_freplace checks the failure that fails to attach a tail +* callee prog with freplace prog or fails to update an extended prog to +* prog_array map. +*/ +static void test_tailcall_bpf2bpf_freplace (void ) +{ +struct tailcall_freplace *freplace_skel =NULL ; +struct bpf_link *freplace_link =NULL ; +struct tc_bpf2bpf *tc_skel =NULL ; +char buff[128 ] = {}; +int prog_fd, map_fd; +int err, key; + +LIBBPF_OPTS (bpf_test_run_opts, topts, + .data_in = buff, + .data_size_in = sizeof(buff), + .repeat = 1, + ); + + tc_skel =tc_bpf2bpf__open_and_load (); +if (!ASSERT_OK_PTR (tc_skel,"tc_bpf2bpf__open_and_load" )) + goto out; + + prog_fd =bpf_program__fd (tc_skel->progs.entry_tc); + freplace_skel =tailcall_freplace__open (); +if (!ASSERT_OK_PTR (freplace_skel,"tailcall_freplace__open" )) + goto out; + + err =bpf_program__set_attach_target (freplace_skel->progs.entry_freplace, + prog_fd,"subprog_tc" ); +if (!ASSERT_OK (err,"set_attach_target" )) + goto out; + + err =tailcall_freplace__load (freplace_skel); +if (!ASSERT_OK (err,"tailcall_freplace__load" )) + goto out; + +/* OK to attach then detach freplace prog. */ + + freplace_link =bpf_program__attach_freplace (freplace_skel->progs.entry_freplace, + prog_fd,"subprog_tc" ); +if (!ASSERT_OK_PTR (freplace_link,"attach_freplace" )) + goto out; + + err =bpf_link__destroy (freplace_link); +if (!ASSERT_OK (err,"destroy link" )) + goto out; + +/* OK to update prog_array map then delete element from the map. */ + key =0 ; + map_fd =bpf_map__fd (freplace_skel->maps.jmp_table); err =bpf_map_update_elem (map_fd, &key, &prog_fd,BPF_ANY );if (!ASSERT_OK (err,"update jmp_table" )) goto out; - prog_fd =bpf_program__fd (tc_skel->progs.entry_tc); - err =bpf_prog_test_run_opts (prog_fd, &topts); -ASSERT_OK (err,"test_run" ); -ASSERT_EQ (topts.retval,34 ,"test_run retval" ); + err =bpf_map_delete_elem (map_fd, &key); +if (!ASSERT_OK (err,"delete_elem from jmp_table" )) + goto out; + +/* Fail to attach a tail callee prog with freplace prog. */ + + err =bpf_map_update_elem (map_fd, &key, &prog_fd,BPF_ANY ); +if (!ASSERT_OK (err,"update jmp_table" )) + goto out; + + freplace_link =bpf_program__attach_freplace (freplace_skel->progs.entry_freplace, + prog_fd,"subprog_tc" ); +if (!ASSERT_ERR_PTR (freplace_link,"attach_freplace failure" )) + goto out; + + err =bpf_map_delete_elem (map_fd, &key); +if (!ASSERT_OK (err,"delete_elem from jmp_table" )) + goto out; + +/* Fail to update an extended prog to prog_array map. */ + + freplace_link =bpf_program__attach_freplace (freplace_skel->progs.entry_freplace, + prog_fd,"subprog_tc" ); +if (!ASSERT_OK_PTR (freplace_link,"attach_freplace" )) + goto out; + + err =bpf_map_update_elem (map_fd, &key, &prog_fd,BPF_ANY ); +if (!ASSERT_ERR (err,"update jmp_table failure" )) + goto out; out:bpf_link__destroy (freplace_link); -tc_bpf2bpf__destroy (tc_skel);tailcall_freplace__destroy (freplace_skel); +tc_bpf2bpf__destroy (tc_skel); }void test_tailcalls (void )
@@ -1606,4 +1696,6 @@ void test_tailcalls(void)test_tailcall_bpf2bpf_hierarchy_3 ();if (test__start_subtest ("tailcall_freplace" ))test_tailcall_freplace (); +if (test__start_subtest ("tailcall_bpf2bpf_freplace" )) +test_tailcall_bpf2bpf_freplace (); }
tools/testing/selftests/bpf/progs/tc_bpf2bpf.c +3 -2
@@ -5,10 +5,11 @@#include "bpf_misc.h" __noinline -int subprog(struct __sk_buff *skb) +int subprog_tc(struct __sk_buff *skb) {int ret =1 ; +__sink (skb);__sink (ret);return ret; }
@@ -16,7 +17,7 @@ int subprog(struct __sk_buff *skb) SEC("tc")int entry_tc (struct __sk_buff *skb) { -return subprog (skb); +return subprog_tc (skb); } char __license[] SEC("license") = "GPL";
Signature · 1 line
2.44.0
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 9 lines · On Tue, 15 Oct 2024 23:02:05 +0800 you wrote:
Previously, I addressed a tailcall infinite loop issue related to trampolines[0].
In this patchset, I resolve a similar issue where a tailcall infinite loop can occur due to the combination of tailcalls and freplace programs. The fix prevents adding extended programs to the
prog_arraymap and blocks the extension of a tail callee program with freplace.[...]
Here is the summary with links:
- [bpf-next,v8,1/2] bpf: Prevent tailcall infinite loop caused by freplace
https://git.kernel.org/bpf/bpf-next/c/d24c6a9d3a9f - [bpf-next,v8,2/2] selftests/bpf: Add test to verify tailcall and freplace restrictions https://git.kernel.org/bpf/bpf-next/c/5a2ac3844d47
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html