Skip to contentSkip to conversation

bpf: Fix map poke update

PATCH In review Jiri Olsa 2 participants 3 messages Jan 3, 2024

Subsystems (2)
lore
Share & tools
lore

Jiri Olsa

author reviewer
#0 lore raw

cover bpf: Fix map poke update

hi,
sending the 5.15 backport of map poke update fix [1].

It's not straight backport, the details are in changelog. I also backported the bpf selftest [2] to reproduce the issue and verify the fix, but it's more deviated from the upstream so I decided not to post it. Also I had to fix bpf selftests on other place to be able to compile them.

thanks,
jirka

[1] 4b7de801606e ("bpf: Fix prog_array_map_poke_run map poke update")
[2] ffed24eff9e0 ("selftests/bpf: Add test for early update in prog_array_map_poke_run")
---
Jiri Olsa (1):
bpf: Fix prog_array_map_poke_run map poke update

 arch/x86/net/bpf_jit_comp.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/bpf.h         |  3 +++
 kernel/bpf/arraymap.c       | 58 ++++++++++------------------------------------------------
 3 files changed, 59 insertions(+), 48 deletions(-)
author reviewer
#1 lore raw

PATCH 1/1 bpf: Fix prog_array_map_poke_run map poke update

commit 4b7de801606e504e69689df71475d27e35336fb3 upstream.

[ The patch differs from upstream by calling __bpf_arch_text_poke with
text_live=true, because 5.15 does not have commit 1022a5498f6f ]

Lee pointed out issue found by syscaller [0] hitting BUG in prog array map poke update in prog_array_map_poke_run function due to error value returned from bpf_arch_text_poke function.

There's race window where bpf_arch_text_poke can fail due to missing bpf program kallsym symbols, which is accounted for with check for -EINVAL in that BUG_ON call.

The problem is that in such case we won't update the tail call jump and cause imbalance for the next tail call update check which will fail with -EBUSY in bpf_arch_text_poke.

I'm hitting following race during the program load:

CPU 0 CPU 1

bpf_prog_load
bpf_check
do_misc_fixups
prog_array_map_poke_track

map_update_elem
bpf_fd_array_map_update_elem
prog_array_map_poke_run

bpf_arch_text_poke returns -EINVAL

bpf_prog_kallsyms_add

After bpf_arch_text_poke (CPU 1) fails to update the tail call jump, the next poke update fails on expected jump instruction check in bpf_arch_text_poke with -EBUSY and triggers the BUG_ON in prog_array_map_poke_run.

Similar race exists on the program unload.

Fixing this by moving the update to bpf_arch_poke_desc_update function which makes sure we call __bpf_arch_text_poke that skips the bpf address check.

Each architecture has slightly different approach wrt looking up bpf address in bpf_arch_text_poke, so instead of splitting the function or adding new 'checkip' argument in previous version, it seems best to move the whole map_poke_run update as arch specific code.

[0] https://syzkaller.appspot.com/bug?extid=97a4fe20470e9bc30810

Fixes ebf7d1f508a7 ("bpf, x64: rework pro/epilogue and tailcall handling in JIT")
Reported-by [email protected]
Cc Lee Jones <[email protected]>
Cc Maciej Fijalkowski <[email protected]>
---
 arch/x86/net/bpf_jit_comp.c | 46 +++++++++++++++++++++++++++++
 include/linux/bpf.h         |  3 ++
 kernel/bpf/arraymap.c       | 58 +++++++------------------------------
 3 files changed, 59 insertions(+), 48 deletions(-)
arch/x86/net/bpf_jit_comp.c +46 -0
index dccaab2113f9..ee6e3cda1738 100644
@@ -2403,3 +2403,49 @@ bool bpf_jit_supports_kfunc_call(void)
 {
 	return true;
 }
+
+void bpf_arch_poke_desc_update(struct bpf_jit_poke_descriptor *poke,
+			       struct bpf_prog *new, struct bpf_prog *old)
+{
+	u8 *old_addr, *new_addr, *old_bypass_addr;
+	int ret;
+
+	old_bypass_addr = old ? NULL : poke->bypass_addr;
+	old_addr = old ? (u8 *)old->bpf_func + poke->adj_off : NULL;
+	new_addr = new ? (u8 *)new->bpf_func + poke->adj_off : NULL;
+
+	/*
+	 * On program loading or teardown, the program's kallsym entry
+	 * might not be in place, so we use __bpf_arch_text_poke to skip
+	 * the kallsyms check.
+	 */
+	if (new) {
+		ret = __bpf_arch_text_poke(poke->tailcall_target,
+					   BPF_MOD_JUMP,
+					   old_addr, new_addr, true);
+		BUG_ON(ret < 0);
+		if (!old) {
+			ret = __bpf_arch_text_poke(poke->tailcall_bypass,
+						   BPF_MOD_JUMP,
+						   poke->bypass_addr,
+						   NULL, true);
+			BUG_ON(ret < 0);
+		}
+	} else {
+		ret = __bpf_arch_text_poke(poke->tailcall_bypass,
+					   BPF_MOD_JUMP,
+					   old_bypass_addr,
+					   poke->bypass_addr, true);
+		BUG_ON(ret < 0);
+		/* let other CPUs finish the execution of program
+		 * so that it will not possible to expose them
+		 * to invalid nop, stack unwind, nop state
+		 */
+		if (!ret)
+			synchronize_rcu();
+		ret = __bpf_arch_text_poke(poke->tailcall_target,
+					   BPF_MOD_JUMP,
+					   old_addr, NULL, true);
+		BUG_ON(ret < 0);
+	}
+}
include/linux/bpf.h +3 -0
index 9ab087d73ab3..00c615fc8ec3 100644
@@ -2280,6 +2280,9 @@ enum bpf_text_poke_type {
 int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t,
 		       void *addr1, void *addr2);
 
+void bpf_arch_poke_desc_update(struct bpf_jit_poke_descriptor *poke,
+			       struct bpf_prog *new, struct bpf_prog *old);
+
 struct btf_id_set;
 bool btf_id_set_contains(const struct btf_id_set *set, u32 id);
 
kernel/bpf/arraymap.c +10 -48

88 lines not on this page · raw message

Signature · 1 line
2.43.0
#2 lore raw
In reply to Jiri Olsa
Quoted 7 lines · Jiri Olsa

hi,
sending the 5.15 backport of map poke update fix [1].

It's not straight backport, the details are in changelog. I also backported the bpf selftest [2] to reproduce the issue and verify the fix, but it's more deviated from the upstream so I decided not to post it. Also I had to fix bpf selftests on other place to be able to compile them.

Thanks, now queue dup.

greg k-h