bpf: Fix map poke update
Subsystems (2)
Jiri Olsa
reviewercover bpf: Fix map poke update
hi,
sending the 5.15 backport of map poke update fix [1].
It's not straight backport, the details are in changelog. I also backported the bpf selftest [2] to reproduce the issue and verify the fix, but it's more deviated from the upstream so I decided not to post it. Also I had to fix bpf selftests on other place to be able to compile them.
thanks,
jirka
[1] 4b7de801606e ("bpf: Fix prog_array_map_poke_run map poke update")
[2] ffed24eff9e0 ("selftests/bpf: Add test for early update in prog_array_map_poke_run")
---
Jiri Olsa (1):
bpf: Fix prog_array_map_poke_run map poke update
arch/x86/net/bpf_jit_comp.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ include/linux/bpf.h | 3 +++ kernel/bpf/arraymap.c | 58 ++++++++++------------------------------------------------ 3 files changed, 59 insertions(+), 48 deletions(-)
Jiri Olsa
reviewerPATCH 1/1 bpf: Fix prog_array_map_poke_run map poke update
commit 4b7de801606e504e69689df71475d27e35336fb3 upstream.
[ The patch differs from upstream by calling __bpf_arch_text_poke with
text_live=true, because 5.15 does not have commit 1022a5498f6f ]
Lee pointed out issue found by syscaller [0] hitting BUG in prog array map poke update in prog_array_map_poke_run function due to error value returned from bpf_arch_text_poke function.
There's race window where bpf_arch_text_poke can fail due to missing bpf program kallsym symbols, which is accounted for with check for -EINVAL in that BUG_ON call.
The problem is that in such case we won't update the tail call jump and cause imbalance for the next tail call update check which will fail with -EBUSY in bpf_arch_text_poke.
I'm hitting following race during the program load:
CPU 0 CPU 1
bpf_prog_load
bpf_check
do_misc_fixups
prog_array_map_poke_track
map_update_elem
bpf_fd_array_map_update_elem
prog_array_map_poke_run
bpf_arch_text_poke returns -EINVAL
bpf_prog_kallsyms_add
After bpf_arch_text_poke (CPU 1) fails to update the tail call jump, the next poke update fails on expected jump instruction check in bpf_arch_text_poke with -EBUSY and triggers the BUG_ON in prog_array_map_poke_run.
Similar race exists on the program unload.
Fixing this by moving the update to bpf_arch_poke_desc_update function which makes sure we call __bpf_arch_text_poke that skips the bpf address check.
Each architecture has slightly different approach wrt looking up bpf address in bpf_arch_text_poke, so instead of splitting the function or adding new 'checkip' argument in previous version, it seems best to move the whole map_poke_run update as arch specific code.
[0] https://syzkaller.appspot.com/bug?extid=97a4fe20470e9bc30810
--- arch/x86/net/bpf_jit_comp.c | 46 +++++++++++++++++++++++++++++ include/linux/bpf.h | 3 ++ kernel/bpf/arraymap.c | 58 +++++++------------------------------ 3 files changed, 59 insertions(+), 48 deletions(-)
arch/x86/net/bpf_jit_comp.c +46 -0
@@ -2403,3 +2403,49 @@ bool bpf_jit_supports_kfunc_call(void) {return true; } + +void bpf_arch_poke_desc_update (struct bpf_jit_poke_descriptor *poke, +struct bpf_prog *new,struct bpf_prog *old) +{ +u8 *old_addr, *new_addr, *old_bypass_addr; +int ret; + + old_bypass_addr = old ?NULL : poke->bypass_addr; + old_addr = old ? (u8 *)old->bpf_func + poke->adj_off :NULL ; + new_addr = new ? (u8 *)new->bpf_func + poke->adj_off :NULL ; + +/* +* On program loading or teardown, the program's kallsym entry +* might not be in place, so we use __bpf_arch_text_poke to skip +* the kallsyms check. +*/ +if (new) { + ret =__bpf_arch_text_poke (poke->tailcall_target, +BPF_MOD_JUMP , + old_addr, new_addr, true); +BUG_ON (ret <0 ); +if (!old) { + ret =__bpf_arch_text_poke (poke->tailcall_bypass, +BPF_MOD_JUMP , + poke->bypass_addr, +NULL , true); +BUG_ON (ret <0 ); + } + }else { + ret =__bpf_arch_text_poke (poke->tailcall_bypass, +BPF_MOD_JUMP , + old_bypass_addr, + poke->bypass_addr, true); +BUG_ON (ret <0 ); +/* let other CPUs finish the execution of program +* so that it will not possible to expose them +* to invalid nop, stack unwind, nop state +*/ +if (!ret) +synchronize_rcu (); + ret =__bpf_arch_text_poke (poke->tailcall_target, +BPF_MOD_JUMP , + old_addr,NULL , true); +BUG_ON (ret <0 ); + } +}
include/linux/bpf.h +3 -0
@@ -2280,6 +2280,9 @@ enum bpf_text_poke_type {int bpf_arch_text_poke (void *ip,enum bpf_text_poke_type t,void *addr1,void *addr2); +void bpf_arch_poke_desc_update (struct bpf_jit_poke_descriptor *poke, +struct bpf_prog *new,struct bpf_prog *old); +struct btf_id_set ;bool btf_id_set_contains (const struct btf_id_set *set,u32 id);
kernel/bpf/arraymap.c +10 -48
88 lines not on this page · raw message
Signature · 1 line
2.43.0
Quoted 7 lines · Jiri Olsa
hi,
sending the 5.15 backport of map poke update fix [1].It's not straight backport, the details are in changelog. I also backported the bpf selftest [2] to reproduce the issue and verify the fix, but it's more deviated from the upstream so I decided not to post it. Also I had to fix bpf selftests on other place to be able to compile them.
Thanks, now queue dup.
greg k-h