Skip to contentSkip to conversation

Zero overhead PROBE_MEM

PATCH Superseded Kumar Kartikeya Dwivedi 3 participants 5 messages Jun 19, 2024 – Jun 25, 2024

Subsystems (4)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Kumar Kartikeya Dwivedi

author maintainer
#0 lore raw

v2 cover Zero overhead PROBE_MEM

BPF programs that are loaded by privileged users (with CAP_BPF and CAP_PERFMON) are allowed to be non-confidential. This means that they can read arbitrary kernel memory, and also communicate kernel pointers through maps and other channels of communication from BPF programs to applications running in userspace.

This is a critical use case for applications that implement kernel tracing, and observability functionality using BPF programs, and provides users with much needed visibility and context into a running kernel.

There are two supported methods of such kernel memory "probing", using bpf_probe_read_kernel (and related) helpers, or using direct load instructions of untrusted kernel memory (e.g. arguments to tracepoint programs, through bpf_core_cast casting, etc.).

For direct load instructions on untrusted kernel pointers, the verifier converts these to PROBE_MEM loads, and the JIT handles these loads by adding a bounds check and handling exceptions on page faults (when reading invalid kernel memory).

So far, the implementation of PROBE_MEM (particularly on x86) has relied on bounds check because it needs to protect the BPF program from reading user addresses. Loads for such addresses will lead to a kernel panic due to panic in do_user_addr_fault, because the page fault on accessing userspace address in kernel mode will be unhandled.

This patch instead proposes to do exception handling in do_user_addr_fault when user addresses are accessed by a BPF program, and when SMAP is enabled on x86. This would obviate the need for the BPF JIT to emit bounds checking for PROBE_MEM load instructions, and any invalid memory accesses (either for user addresses or unmapped kernel addresses) will be handled by the page fault handler.

This set does not grant programs any additional privileges than those they already had. Instead, it optimizes the common case of doing loads on valid kernel memory, while shifting the cost to cases where invalid kernel memory is accessed without sanitization by a program.

Changelog:
----------
v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]

* Rebase on bpf-next

Kumar Kartikeya Dwivedi (2):
x86: Perform BPF exception fixup in do_user_addr_fault bpf, x86: Skip bounds checking for PROBE_MEM with SMAP

 arch/x86/mm/fault.c         | 11 +++++++++++
 arch/x86/net/bpf_jit_comp.c | 11 +++++++++--
 2 files changed, 20 insertions(+), 2 deletions(-)

base-commit: f6afdaf72af7583d251bd569ded8d7d1eeb849c2

Signature · 1 line
2.43.0
author maintainer
#1 lore raw

PATCH v2 1/2 x86: Perform BPF exception fixup in do_user_addr_fault

Currently, on x86, when SMAP is enabled, and a page fault occurs in kernel mode for accessing a user address, the kernel will rightly panic as no valid kernel code can cause such a page fault (unless buggy). There is no valid correct kernel code that can generate such a fault, therefore this behavior would be correct.

BPF programs that currently encounter user addresses when doing PROBE_MEM loads (load instructions which are allowed to read any kernel address, only available for root users) avoid a page fault by performing bounds checking on the address. This requires the JIT to emit a jump over each PROBE_MEM load instruction to avoid hitting page faults.

We would prefer avoiding these jump instructions to improve performance of programs which use PROBE_MEM loads pervasively. For correct behavior, programs already rely on the kernel addresses being valid when they are executing, but BPF's safety properties must still ensure kernel safety in presence of invalid addresses. Therefore, for correct programs, the bounds checking is an added cost meant to ensure kernel safety. If the do_user_addr_fault handler could perform fixups for the BPF program in such a case, the bounds checking could be eliminated, the load instruction could be emitted directly without any checking.

Thus, in case SMAP is enabled (which would mean the kernel traps on accessing a user address), and the instruction pointer belongs to a BPF program, perform fixup for the access by searching exception tables. All BPF programs already execute with SMAP protection. When SMAP is not enabled, the BPF JIT will continue to emit bounds checking instructions.

---
 arch/x86/mm/fault.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
arch/x86/mm/fault.c +11 -0
index e6c469b323cc..189e93d88bd4 100644
@@ -21,6 +21,7 @@
 #include <linux/mm_types.h>
 #include <linux/mm.h>			/* find_and_lock_vma() */
 #include <linux/vmalloc.h>
+#include <linux/filter.h>		/* is_bpf_text_address()	*/
 
 #include <asm/cpufeature.h>		/* boot_cpu_has, ...		*/
 #include <asm/traps.h>			/* dotraplinkage, ...		*/
@@ -1257,6 +1258,16 @@ void do_user_addr_fault(struct pt_regs *regs,
 	if (unlikely(cpu_feature_enabled(X86_FEATURE_SMAP) &&
 		     !(error_code & X86_PF_USER) &&
 		     !(regs->flags & X86_EFLAGS_AC))) {
+		/*
+		 * If the kernel access happened to an invalid user pointer
+		 * under SMAP by a BPF program, we will have an extable entry
+		 * here, and need to perform the fixup.
+		 */
+		if (is_bpf_text_address(regs->ip)) {
+			kernelmode_fixup_or_oops(regs, error_code, address,
+						 0, 0, ARCH_DEFAULT_PKEY);
+			return;
+		}
 		/*
 		 * No extable entry here.  This was a kernel access to an
 		 * invalid pointer.  get_kernel_nofault() will not get here.
Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH bpf-next v2 1/2] x86: Perform BPF exception fixup in do_user_addr_fault

Quoted 50 lines · Kumar Kartikeya Dwivedi

Currently, on x86, when SMAP is enabled, and a page fault occurs in kernel mode for accessing a user address, the kernel will rightly panic as no valid kernel code can cause such a page fault (unless buggy). There is no valid correct kernel code that can generate such a fault, therefore this behavior would be correct.

BPF programs that currently encounter user addresses when doing PROBE_MEM loads (load instructions which are allowed to read any kernel address, only available for root users) avoid a page fault by performing bounds checking on the address. This requires the JIT to emit a jump over each PROBE_MEM load instruction to avoid hitting page faults.

We would prefer avoiding these jump instructions to improve performance of programs which use PROBE_MEM loads pervasively. For correct behavior, programs already rely on the kernel addresses being valid when they are executing, but BPF's safety properties must still ensure kernel safety in presence of invalid addresses. Therefore, for correct programs, the bounds checking is an added cost meant to ensure kernel safety. If the do_user_addr_fault handler could perform fixups for the BPF program in such a case, the bounds checking could be eliminated, the load instruction could be emitted directly without any checking.

Thus, in case SMAP is enabled (which would mean the kernel traps on accessing a user address), and the instruction pointer belongs to a BPF program, perform fixup for the access by searching exception tables. All BPF programs already execute with SMAP protection. When SMAP is not enabled, the BPF JIT will continue to emit bounds checking instructions.

---
 arch/x86/mm/fault.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
arch/x86/mm/fault.c +1 -0
index e6c469b323cc..189e93d88bd4 100644
@@ -21,6 +21,7 @@
 #include <linux/mm_types.h>
 #include <linux/mm.h>                  /* find_and_lock_vma() */
 #include <linux/vmalloc.h>
+#include <linux/filter.h>              /* is_bpf_text_address()        */
 
 #include <asm/cpufeature.h>            /* boot_cpu_has, ...            */
 #include <asm/traps.h>                 /* dotraplinkage, ...           */
@@ -1257,6 +1258,16 @@ void do_user_addr_fault(struct pt_regs *regs,
        if (unlikely(cpu_feature_enabled(X86_FEATURE_SMAP) &&
                     !(error_code & X86_PF_USER) &&
                     !(regs->flags & X86_EFLAGS_AC))) {

arch/x86/mm/fault.c void do_user_addr_fault(struct pt_regs *regs, · line 1270

Quoted 10 lines
arch/x86/mm/fault.c +10 -0
+               /*
+                * If the kernel access happened to an invalid user pointer
+                * under SMAP by a BPF program, we will have an extable entry
+                * here, and need to perform the fixup.
+                */
+               if (is_bpf_text_address(regs->ip)) {
+                       kernelmode_fixup_or_oops(regs, error_code, address,
+                                                0, 0, ARCH_DEFAULT_PKEY);
+                       return;
+               }

I see no harm doing this check here. Looks correct.

Andy,
we've talked about this idea in the past.
Any objections to this optimization?

author maintainer
#3 lore raw

PATCH v2 2/2 bpf, x86: Skip bounds checking for PROBE_MEM with SMAP

In reply to Kumar Kartikeya Dwivedi

The previous patch changed the do_user_addr_fault page fault handler to invoke BPF's fixup routines (by searching exception tables and calling ex_handler_bpf). This would only occur when SMAP is enabled, such that any user address access from BPF programs running in kernel mode would reach this path and invoke the fixup routines.

Relying on this behavior, disable any bounds checking instrumentation in the BPF JIT for x86 when X86_FEATURE_SMAP is available. All BPF programs execute with SMAP enabled, therefore when this feature is available, we can assume that SMAP will be enabled during program execution at runtime.

This optimizes PROBE_MEM loads down to a normal unchecked load instruction. Any page faults for user or kernel addresses will be handled using the fixup routines, and the generation exception table entries for such load instructions.

All in all, this ensures that PROBE_MEM loads will now incur no runtime overhead, and become practically free.

---
 arch/x86/net/bpf_jit_comp.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
arch/x86/net/bpf_jit_comp.c +9 -2
index 5159c7a22922..f8a39189cddc 100644
@@ -1864,8 +1864,8 @@ st:			if (is_imm8(insn->off))
 		case BPF_LDX | BPF_PROBE_MEMSX | BPF_W:
 			insn_off = insn->off;
 
-			if (BPF_MODE(insn->code) == BPF_PROBE_MEM ||
-			    BPF_MODE(insn->code) == BPF_PROBE_MEMSX) {
+			if ((BPF_MODE(insn->code) == BPF_PROBE_MEM ||
+			     BPF_MODE(insn->code) == BPF_PROBE_MEMSX) && !cpu_feature_enabled(X86_FEATURE_SMAP)) {
 				/* Conservatively check that src_reg + insn->off is a kernel address:
 				 *   src_reg + insn->off > TASK_SIZE_MAX + PAGE_SIZE
 				 *   and
@@ -1912,6 +1912,9 @@ st:			if (is_imm8(insn->off))
 				/* populate jmp_offset for JAE above to jump to start_of_ldx */
 				start_of_ldx = prog;
 				end_of_jmp[-1] = start_of_ldx - end_of_jmp;
+			} else if ((BPF_MODE(insn->code) == BPF_PROBE_MEM ||
+				    BPF_MODE(insn->code) == BPF_PROBE_MEMSX)) {
+				start_of_ldx = prog;
 			}
 			if (BPF_MODE(insn->code) == BPF_PROBE_MEMSX ||
 			    BPF_MODE(insn->code) == BPF_MEMSX)
@@ -1924,9 +1927,13 @@ st:			if (is_imm8(insn->off))
 				u8 *_insn = image + proglen + (start_of_ldx - temp);
 				s64 delta;
 
+				if (cpu_feature_enabled(X86_FEATURE_SMAP))
+					goto extable_fixup;
+
 				/* populate jmp_offset for JMP above */
 				start_of_ldx[-1] = prog - start_of_ldx;
 
+			extable_fixup:
 				if (!bpf_prog->aux->extable)
 					break;
 
Signature · 1 line
2.43.0
Acked-by on the v2 series
#4 lore raw
In reply to Kumar Kartikeya Dwivedi
Quoted 50 lines · Kumar Kartikeya Dwivedi

BPF programs that are loaded by privileged users (with CAP_BPF and CAP_PERFMON) are allowed to be non-confidential. This means that they can read arbitrary kernel memory, and also communicate kernel pointers through maps and other channels of communication from BPF programs to applications running in userspace.

This is a critical use case for applications that implement kernel tracing, and observability functionality using BPF programs, and provides users with much needed visibility and context into a running kernel.

There are two supported methods of such kernel memory "probing", using bpf_probe_read_kernel (and related) helpers, or using direct load instructions of untrusted kernel memory (e.g. arguments to tracepoint programs, through bpf_core_cast casting, etc.).

For direct load instructions on untrusted kernel pointers, the verifier converts these to PROBE_MEM loads, and the JIT handles these loads by adding a bounds check and handling exceptions on page faults (when reading invalid kernel memory).

So far, the implementation of PROBE_MEM (particularly on x86) has relied on bounds check because it needs to protect the BPF program from reading user addresses. Loads for such addresses will lead to a kernel panic due to panic in do_user_addr_fault, because the page fault on accessing userspace address in kernel mode will be unhandled.

This patch instead proposes to do exception handling in do_user_addr_fault when user addresses are accessed by a BPF program, and when SMAP is enabled on x86. This would obviate the need for the BPF JIT to emit bounds checking for PROBE_MEM load instructions, and any invalid memory accesses (either for user addresses or unmapped kernel addresses) will be handled by the page fault handler.

This set does not grant programs any additional privileges than those they already had. Instead, it optimizes the common case of doing loads on valid kernel memory, while shifting the cost to cases where invalid kernel memory is accessed without sanitization by a program.

Changelog:
----------
v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]

* Rebase on bpf-next

Kumar Kartikeya Dwivedi (2):
x86: Perform BPF exception fixup in do_user_addr_fault bpf, x86: Skip bounds checking for PROBE_MEM with SMAP

Quoted 8 lines
 arch/x86/mm/fault.c         | 11 +++++++++++
 arch/x86/net/bpf_jit_comp.c | 11 +++++++++--
 2 files changed, 20 insertions(+), 2 deletions(-)

base-commit: f6afdaf72af7583d251bd569ded8d7d1eeb849c2

Signature · 1 line
2.43.0

We can also do something like this for ARM64 when PAN(Privileged Access Never) is available. And if we are doing it then for RISC-V we can remove this bounds checking completely because RISC-V always traps when kernel accesses userspace addresses outside of uaccess routines.

But I am curious to know what other developers think about this.

Thanks,
Puranjay