Support kCFI + BPF on arm64
Sami Tolvanen
maintainerv10 cover Support kCFI + BPF on arm64
Hi folks,
These patches add KCFI types to arm64 BPF JIT output. Puranjay and Maxwell have been working on this for some time now, but I haven't seen any progress since June 2024, so I decided to pick up the latest version[1] posted by Maxwell and fix the few remaining issues I noticed. I confirmed that with these patches applied, I no longer see CFI failures in jitted code when running BPF self-tests on arm64.
Note that in order to enable CFI for jitted code, we need to define __bpfcall in a file included by include/linux/bpf.h. In v10, I'm still adding an include/asm/cfi.h header file for consistency with other architectures, even though the file no longer contains anything else. If you'd prefer to move this to another header file, I'm certainly open to suggestions.
Sami
---
v10:
- Rebased to bpf-next/master again.
- Added a patch to moved duplicate type hash variables and helper functions to common CFI code.
v9: https://lore.kernel.org/bpf/[email protected]/
- Rebased to bpf-next/master to fix merge x86 merge conflicts.
- Fixed checkpatch warnings about Co-developed-by tags and including <asm/cfi.h>.
- Picked up Tested-by tags.
v8: https://lore.kernel.org/bpf/[email protected]/
- Changed DEFINE_CFI_TYPE to use .4byte to match __CFI_TYPE.
- Changed cfi_get_func_hash() to again use get_kernel_nofault().
- Fixed a panic in bpf_jit_free() by resetting prog->bpf_func before calling bpf_jit_binary_pack_hdr().
---
Mark Rutland (1):
cfi: add C CFI type macro
Puranjay Mohan (1):
arm64/cfi,bpf: Support kCFI + BPF on arm64
Sami Tolvanen (1):
cfi: Move BPF CFI types and helpers to generic code
arch/arm64/include/asm/cfi.h | 7 +++++ arch/arm64/net/bpf_jit_comp.c | 22 +++++++++++++-- arch/riscv/include/asm/cfi.h | 16 ----------- arch/riscv/kernel/cfi.c | 53 ----------------------------------- arch/x86/include/asm/cfi.h | 9 ------ arch/x86/kernel/alternative.c | 37 ------------------------ include/linux/cfi.h | 37 ++++++++++++++++++------ include/linux/cfi_types.h | 23 +++++++++++++++ kernel/cfi.c | 25 +++++++++++++++++ 9 files changed, 103 insertions(+), 126 deletions(-) create mode 100644 arch/arm64/include/asm/cfi.h
base-commit: e860a98c8aebd8de82c0ee901acf5a759acd4570
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
Sami Tolvanen
maintainerPATCH v10 1/3 cfi: add C CFI type macro
From: Mark Rutland <[email protected]>
Currently x86 and riscv open-code 4 instances of the same logic to define a u32 variable with the KCFI typeid of a given function.
Replace the duplicate logic with a common macro.
--- arch/riscv/kernel/cfi.c | 35 +++-------------------------------- arch/x86/kernel/alternative.c | 31 +++---------------------------- include/linux/cfi_types.h | 23 +++++++++++++++++++++++ 3 files changed, 29 insertions(+), 60 deletions(-)
arch/riscv/kernel/cfi.c +3 -32
@@ -4,6 +4,7 @@* * Copyright (C) 2023 Google LLC */ +#include <linux/cfi_types.h> #include <linux/cfi.h> #include <asm/insn.h>
@@ -82,41 +83,11 @@ struct bpf_insn;/* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */ extern unsigned int __bpf_prog_runX (const void *ctx,const struct bpf_insn *insn); - -/* -* Force a reference to the external symbol so the compiler generates -* __kcfi_typid. -*/ -__ADDRESSABLE (__bpf_prog_runX); - -/* u32 __ro_after_init cfi_bpf_hash = __kcfi_typeid___bpf_prog_runX; */ -asm ( -" .pushsection .data..ro_after_init,\"aw\",@progbits \n" -" .type cfi_bpf_hash,@object \n" -" .globl cfi_bpf_hash \n" -" .p2align 2, 0x0 \n" -"cfi_bpf_hash: \n" -" .word __kcfi_typeid___bpf_prog_runX \n" -" .size cfi_bpf_hash, 4 \n" -" .popsection \n" -); +DEFINE_CFI_TYPE (cfi_bpf_hash, __bpf_prog_runX);/* Must match bpf_callback_t */ extern u64 __bpf_callback_fn (u64 ,u64 ,u64 ,u64 ,u64 ); - -__ADDRESSABLE (__bpf_callback_fn); - -/* u32 __ro_after_init cfi_bpf_subprog_hash = __kcfi_typeid___bpf_callback_fn; */ -asm ( -" .pushsection .data..ro_after_init,\"aw\",@progbits \n" -" .type cfi_bpf_subprog_hash,@object \n" -" .globl cfi_bpf_subprog_hash \n" -" .p2align 2, 0x0 \n" -"cfi_bpf_subprog_hash: \n" -" .word __kcfi_typeid___bpf_callback_fn \n" -" .size cfi_bpf_subprog_hash, 4 \n" -" .popsection \n" -); +DEFINE_CFI_TYPE (cfi_bpf_subprog_hash, __bpf_callback_fn);u32 cfi_get_func_hash (void *func) {
arch/x86/kernel/alternative.c +3 -28
@@ -2,6 +2,7 @@#define pr_fmt (fmt) "SMP alternatives: " fmt#include <linux/mmu_context.h> +#include <linux/cfi_types.h> #include <linux/perf_event.h> #include <linux/vmalloc.h> #include <linux/memory.h>
@@ -1189,37 +1190,11 @@ struct bpf_insn;/* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */ extern unsigned int __bpf_prog_runX (const void *ctx,const struct bpf_insn *insn); - -KCFI_REFERENCE (__bpf_prog_runX); - -/* u32 __ro_after_init cfi_bpf_hash = __kcfi_typeid___bpf_prog_runX; */ -asm ( -" .pushsection .data..ro_after_init,\"aw\",@progbits \n" -" .type cfi_bpf_hash,@object \n" -" .globl cfi_bpf_hash \n" -" .p2align 2, 0x0 \n" -"cfi_bpf_hash: \n" -" .long __kcfi_typeid___bpf_prog_runX \n" -" .size cfi_bpf_hash, 4 \n" -" .popsection \n" -); +DEFINE_CFI_TYPE (cfi_bpf_hash, __bpf_prog_runX);/* Must match bpf_callback_t */ extern u64 __bpf_callback_fn (u64 ,u64 ,u64 ,u64 ,u64 ); - -KCFI_REFERENCE (__bpf_callback_fn); - -/* u32 __ro_after_init cfi_bpf_subprog_hash = __kcfi_typeid___bpf_callback_fn; */ -asm ( -" .pushsection .data..ro_after_init,\"aw\",@progbits \n" -" .type cfi_bpf_subprog_hash,@object \n" -" .globl cfi_bpf_subprog_hash \n" -" .p2align 2, 0x0 \n" -"cfi_bpf_subprog_hash: \n" -" .long __kcfi_typeid___bpf_callback_fn \n" -" .size cfi_bpf_subprog_hash, 4 \n" -" .popsection \n" -); +DEFINE_CFI_TYPE (cfi_bpf_subprog_hash, __bpf_callback_fn);u32 cfi_get_func_hash (void *func) {
include/linux/cfi_types.h +23 -0
@@ -41,5 +41,28 @@ SYM_TYPED_START(name, SYM_L_GLOBAL, SYM_A_ALIGN)#endif +#else /* __ASSEMBLY__ */ + +#ifdef CONFIG_CFI_CLANG +#define DEFINE_CFI_TYPE (name, func) \ +/* \ +* Force a reference to the function so the compiler generates \ +* __kcfi_typeid_<func>. \ +*/ \ + __ADDRESSABLE(func); \ +/* u32 name __ro_after_init = __kcfi_typeid_<func> */ \ +extern u32 name; \ + asm ( \ + " .pushsection .data..ro_after_init,\"aw\",@progbits \n" \ + " .type " #name ",@object \n" \ + " .globl " #name " \n" \ + " .p2align 2, 0x0 \n" \ + #name ": \n" \ + " .4byte __kcfi_typeid_" #func " \n" \ + " .size " #name ", 4 \n" \ + " .popsection \n" \ + ); +#endif +#endif /* __ASSEMBLY__ */ #endif /* _LINUX_CFI_TYPES_H */
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
Re: [PATCH bpf-next v10 1/3] cfi: add C CFI type macro
Quoted 18 lines · Sami Tolvanen
From: Mark Rutland <[email protected]>
Currently x86 and riscv open-code 4 instances of the same logic to define a u32 variable with the KCFI typeid of a given function.
Replace the duplicate logic with a common macro.
Signed-off-by Mark Rutland <[email protected]>Co-developed-by Maxwell Bland <[email protected]>Signed-off-by Maxwell Bland <[email protected]>Co-developed-by Sami Tolvanen <[email protected]>Signed-off-by Sami Tolvanen <[email protected]>Tested-by Dao Huang <[email protected]>--- arch/riscv/kernel/cfi.c | 35 +++-------------------------------- arch/x86/kernel/alternative.c | 31 +++---------------------------- include/linux/cfi_types.h | 23 +++++++++++++++++++++++ 3 files changed, 29 insertions(+), 60 deletions(-)
[...]
Quoted 20 lines
include/linux/cfi_types.h +12
-0@@ -41,5 +41,28 @@ SYM_TYPED_START(name, SYM_L_GLOBAL, SYM_A_ALIGN)#endif +#else /* __ASSEMBLY__ */ + +#ifdef CONFIG_CFI_CLANG +#define DEFINE_CFI_TYPE (name, func) \ +/* \ +* Force a reference to the function so the compiler generates \ +* __kcfi_typeid_<func>. \ +*/ \ + __ADDRESSABLE(func); \ +/* u32 name __ro_after_init = __kcfi_typeid_<func> */ \ +extern u32 name; \ + asm ( \
include/linux/cfi_types.h line 65
Quoted 10 lines
include/linux/cfi_types.h +10
-0+" .pushsection .data..ro_after_init,\"aw\",@progbits \n" \ + " .type " #name ",@object \n" \ + " .globl " #name " \n" \ + " .p2align 2, 0x0 \n" \ + #name ": \n" \ + " .4byte __kcfi_typeid_" #func " \n" \ + " .size " #name ", 4 \n" \ + " .popsection \n" \ + ); +#endif
This looks good to me. I was initially a bit worried about the portability of the '.4byte' directive, but it seems that cfi_types.h is already using that for the __CFI_TYPE() macro so I'm assuming it's not an issue.
In which case:
Thanks for cleaning it up.
Will
Sami Tolvanen
maintainerPATCH v10 2/3 cfi: Move BPF CFI types and helpers to generic code
In reply to Sami TolvanenInstead of duplicating the same code for each architecture, move the CFI type hash variables for BPF function types and related helper functions to generic CFI code, and allow architectures to override the function definitions if needed.
--- arch/riscv/include/asm/cfi.h | 16 --------------- arch/riscv/kernel/cfi.c | 24 ----------------------- arch/x86/include/asm/cfi.h | 9 --------- arch/x86/kernel/alternative.c | 12 ------------ include/linux/cfi.h | 37 +++++++++++++++++++++++++++-------- kernel/cfi.c | 25 +++++++++++++++++++++++ 6 files changed, 54 insertions(+), 69 deletions(-)
arch/riscv/include/asm/cfi.h +0 -16
@@ -14,27 +14,11 @@ struct pt_regs;#ifdef CONFIG_CFI_CLANG enum bug_trap_type handle_cfi_failure (struct pt_regs *regs);#define __bpfcall -static inline int cfi_get_offset (void ) -{ -return 4 ; -} - -#define cfi_get_offset cfi_get_offset -extern u32 cfi_bpf_hash; -extern u32 cfi_bpf_subprog_hash; -extern u32 cfi_get_func_hash (void *func);#else static inline enum bug_trap_type handle_cfi_failure (struct pt_regs *regs) {return BUG_TRAP_TYPE_NONE ; } - -#define cfi_bpf_hash 0U -#define cfi_bpf_subprog_hash 0U -static inline u32 cfi_get_func_hash (void *func) -{ -return 0 ; -}#endif /* CONFIG_CFI_CLANG */ #endif /* _ASM_RISCV_CFI_H */
arch/riscv/kernel/cfi.c +0 -24
@@ -4,7 +4,6 @@* * Copyright (C) 2023 Google LLC */ -#include <linux/cfi_types.h> #include <linux/cfi.h> #include <asm/insn.h>
@@ -76,26 +75,3 @@ enum bug_trap_type handle_cfi_failure(struct pt_regs *regs)return report_cfi_failure (regs, regs->epc, &target, type); } - -#ifdef CONFIG_CFI_CLANG -struct bpf_insn ; - -/* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */ -extern unsigned int __bpf_prog_runX (const void *ctx, -const struct bpf_insn *insn); -DEFINE_CFI_TYPE (cfi_bpf_hash, __bpf_prog_runX); - -/* Must match bpf_callback_t */ -extern u64 __bpf_callback_fn (u64 ,u64 ,u64 ,u64 ,u64 ); -DEFINE_CFI_TYPE (cfi_bpf_subprog_hash, __bpf_callback_fn); - -u32 cfi_get_func_hash (void *func) -{ -u32 hash; - -if (get_kernel_nofault (hash, func -cfi_get_offset ())) -return 0 ; - -return hash; -} -#endif
arch/x86/include/asm/cfi.h +0 -9
@@ -116,8 +116,6 @@ struct pt_regs;#ifdef CONFIG_CFI_CLANG enum bug_trap_type handle_cfi_failure (struct pt_regs *regs);#define __bpfcall -extern u32 cfi_bpf_hash; -extern u32 cfi_bpf_subprog_hash;static inline int cfi_get_offset (void ) {
@@ -134,7 +132,6 @@ static inline int cfi_get_offset(void) }#define cfi_get_offset cfi_get_offset -extern u32 cfi_get_func_hash (void *func);extern int cfi_get_func_arity (void *func);#ifdef CONFIG_FINEIBT
@@ -153,12 +150,6 @@ static inline enum bug_trap_type handle_cfi_failure(struct pt_regs *regs) {return BUG_TRAP_TYPE_NONE ; } -#define cfi_bpf_hash 0U -#define cfi_bpf_subprog_hash 0U -static inline u32 cfi_get_func_hash (void *func) -{ -return 0 ; -}static inline int cfi_get_func_arity (void *func) {return 0 ;
arch/x86/kernel/alternative.c +0 -12
@@ -2,7 +2,6 @@#define pr_fmt (fmt) "SMP alternatives: " fmt#include <linux/mmu_context.h> -#include <linux/cfi_types.h> #include <linux/perf_event.h> #include <linux/vmalloc.h> #include <linux/memory.h>
@@ -1185,17 +1184,6 @@ bool cfi_bhi __ro_after_init = false;#endif #ifdef CONFIG_CFI_CLANG -struct bpf_insn ; - -/* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */ -extern unsigned int __bpf_prog_runX (const void *ctx, -const struct bpf_insn *insn); -DEFINE_CFI_TYPE (cfi_bpf_hash, __bpf_prog_runX); - -/* Must match bpf_callback_t */ -extern u64 __bpf_callback_fn (u64 ,u64 ,u64 ,u64 ,u64 ); -DEFINE_CFI_TYPE (cfi_bpf_subprog_hash, __bpf_callback_fn); -u32 cfi_get_func_hash (void *func) {u32 hash;
include/linux/cfi.h +29 -8
@@ -11,16 +11,9 @@#include <linux/module.h> #include <asm/cfi.h> +#ifdef CONFIG_CFI_CLANG extern bool cfi_warn; -#ifndef cfi_get_offset -static inline int cfi_get_offset (void ) -{ -return 0 ; -} -#endif - -#ifdef CONFIG_CFI_CLANG enum bug_trap_type report_cfi_failure (struct pt_regs *regs,unsigned long addr,unsigned long *target,u32 type);
@@ -29,6 +22,34 @@ static inline enum bug_trap_type report_cfi_failure_noaddr(struct pt_regs *regs, {return report_cfi_failure (regs, addr,NULL ,0 ); } + +#ifndef cfi_get_offset +/* +* Returns the CFI prefix offset. By default, the compiler emits only +* a 4-byte CFI type hash before the function. If an architecture +* uses -fpatchable-function-entry=N,M where M>0 to change the prefix +* offset, they must override this function. +*/ +static inline int cfi_get_offset (void ) +{ +return 4 ; +} +#endif + +u32 cfi_get_func_hash (void *func); + +/* CFI type hashes for BPF function types */ +extern u32 cfi_bpf_hash; +extern u32 cfi_bpf_subprog_hash; + +#else /* CONFIG_CFI_CLANG */ + +static inline int cfi_get_offset (void ) {return 0 ; } +static inline u32 cfi_get_func_hash (void *func) {return 0 ; } + +#define cfi_bpf_hash 0U +#define cfi_bpf_subprog_hash 0U +#endif /* CONFIG_CFI_CLANG */ #ifdef CONFIG_ARCH_USES_CFI_TRAPS
kernel/cfi.c +25 -0
@@ -5,6 +5,8 @@* Copyright (C) 2022 Google LLC */ +#include <linux/bpf.h> +#include <linux/cfi_types.h> #include <linux/cfi.h> bool cfi_warn__ro_after_init =IS_ENABLED (CONFIG_CFI_PERMISSIVE );
@@ -27,6 +29,29 @@ enum bug_trap_type report_cfi_failure(struct pt_regs *regs, unsigned long addr,return BUG_TRAP_TYPE_BUG ; } +u32 __weak cfi_get_func_hash (void *func) +{ +u32 hash; + +if (get_kernel_nofault (hash, func -cfi_get_offset ())) +return 0 ; + +return hash; +} + +/* +* Declare two non-existent functions with types that match bpf_func_t and +* bpf_callback_t pointers, and use DEFINE_CFI_TYPE to define type hash +* variables for each function type. The cfi_bpf_* variables are used by +* arch-specific BPF JIT implementations to ensure indirectly callable JIT +* code has matching CFI type hashes. +*/ +typeof(*(bpf_func_t)0) __bpf_prog_runX; +DEFINE_CFI_TYPE (cfi_bpf_hash, __bpf_prog_runX); + +typeof(*(bpf_callback_t)0) __bpf_callback_fn; +DEFINE_CFI_TYPE (cfi_bpf_subprog_hash, __bpf_callback_fn); +#ifdef CONFIG_ARCH_USES_CFI_TRAPS static inline unsigned long trap_address (s32 *p) {
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
Will Deacon
maintainerRe: [PATCH bpf-next v10 2/3] cfi: Move BPF CFI types and helpers to generic code
Quoted 14 lines · Sami Tolvanen
Instead of duplicating the same code for each architecture, move the CFI type hash variables for BPF function types and related helper functions to generic CFI code, and allow architectures to override the function definitions if needed.
Signed-off-by Sami Tolvanen <[email protected]>--- arch/riscv/include/asm/cfi.h | 16 --------------- arch/riscv/kernel/cfi.c | 24 ----------------------- arch/x86/include/asm/cfi.h | 9 --------- arch/x86/kernel/alternative.c | 12 ------------ include/linux/cfi.h | 37 +++++++++++++++++++++++++++-------- kernel/cfi.c | 25 +++++++++++++++++++++++ 6 files changed, 54 insertions(+), 69 deletions(-)
[...]
kernel/cfi.c enum bug_trap_type report_cfi_failure(struct pt_regs *regs, unsigned long addr, · line 40
Quoted 13 lines
kernel/cfi.c +9
-0@@ -27,6 +29,29 @@ enum bug_trap_type report_cfi_failure(struct pt_regs *regs, unsigned long addr,return BUG_TRAP_TYPE_BUG ; } +u32 __weak cfi_get_func_hash (void *func) +{ +u32 hash; + +if (get_kernel_nofault (hash, func -cfi_get_offset ())) +return 0 ; + +return hash; +}
Do you really need the '__weak' function definition here, or could you use an '#ifndef cfi_get_func_hash' guard, a bit like you're doing for cfi_get_offset()?
Will
Sami Tolvanen
maintainerRe: [PATCH bpf-next v10 2/3] cfi: Move BPF CFI types and helpers to generic code
Quoted 16 lines · Will Deacon
kernel/cfi.c enum bug_trap_type report_cfi_failure(struct pt_regs *regs, unsigned long addr, · line 40
Quoted 10 lines · Sami Tolvanen
kernel/cfi.c +9
-0+u32 __weak cfi_get_func_hash (void *func) +{ +u32 hash; + +if (get_kernel_nofault (hash, func -cfi_get_offset ())) +return 0 ; + +return hash; +}Do you really need the '
__weak' function definition here, or could you use an '#ifndefcfi_get_func_hash' guard, a bit like you're doing forcfi_get_offset()?
Good point, it's not really necessary. I'll move this behind a guard instead.
Sami
Sami Tolvanen
maintainerPATCH v10 3/3 arm64/cfi,bpf: Support kCFI + BPF on arm64
In reply to Sami TolvanenFrom: Puranjay Mohan <[email protected]>
Currently, bpf_dispatcher_*_func() is marked with __nocfi therefore calling BPF programs from this interface doesn't cause CFI warnings.
When BPF programs are called directly from C: from BPF helpers or struct_ops, CFI warnings are generated.
Implement proper CFI prologues for the BPF programs and callbacks and drop __nocfi for arm64. Fix the trampoline generation code to emit kCFI prologue when a struct_ops trampoline is being prepared.
--- arch/arm64/include/asm/cfi.h | 7 +++++++ arch/arm64/net/bpf_jit_comp.c | 22 +++++++++++++++++++--- 2 files changed, 26 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/include/asm/cfi.h
arch/arm64/include/asm/cfi.h new file +7 -0
@@ -0,0 +1,7 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_ARM64_CFI_H +#define _ASM_ARM64_CFI_H + +#define __bpfcall + +#endif /* _ASM_ARM64_CFI_H */
arch/arm64/net/bpf_jit_comp.c +19 -3
@@ -10,6 +10,7 @@#include <linux/arm-smccc.h> #include <linux/bitfield.h> #include <linux/bpf.h> +#include <linux/cfi.h> #include <linux/filter.h> #include <linux/memory.h> #include <linux/printk.h>
@@ -166,6 +167,12 @@ static inline void emit_bti(u32 insn, struct jit_ctx *ctx)emit (insn, ctx); } +static inline void emit_kcfi (u32 hash,struct jit_ctx *ctx) +{ +if (IS_ENABLED (CONFIG_CFI_CLANG )) +emit (hash, ctx); +} +/* * Kernel addresses in the vmalloc space use at most 48 bits, and the * remaining bits are guaranteed to be 0x1. So we can compose the address
@@ -476,7 +483,6 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)const bool is_main_prog = !bpf_is_subprog (prog);const u8 fp = bpf2a64[BPF_REG_FP ];const u8 arena_vm_base = bpf2a64[ARENA_VM_START ]; -const int idx0 = ctx->idx;int cur_offset;/*
@@ -502,6 +508,9 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)* */ +emit_kcfi (is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx); +const int idx0 = ctx->idx; +/* bpf function may be invoked by 3 instruction types: * 1. bl, attached via freplace to bpf prog via short jump * 2. br, attached via freplace to bpf prog via long jump
@@ -2055,9 +2064,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) jit_data->ro_header = ro_header; } - prog->bpf_func = (void *)ctx.ro_image; + prog->bpf_func = (void *)ctx.ro_image +cfi_get_offset (); prog->jited =1 ; - prog->jited_len = prog_size; + prog->jited_len = prog_size -cfi_get_offset ();if (!prog->is_func || extra_pass) {int i;
@@ -2431,6 +2440,12 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,/* return address locates above FP */ retaddr_off = stack_size +8 ; +if (flags &BPF_TRAMP_F_INDIRECT ) { +/* +* Indirect call for bpf_struct_ops +*/ +emit_kcfi (cfi_get_func_hash (func_addr), ctx); + }/* bpf trampoline may be invoked by 3 instruction types: * 1. bl, attached to bpf prog or kernel function via short jump * 2. br, attached to bpf prog or kernel function via long jump
@@ -2947,6 +2962,7 @@ void bpf_jit_free(struct bpf_prog *prog) sizeof(jit_data->header->size));kfree (jit_data); } + prog->bpf_func -=cfi_get_offset (); hdr =bpf_jit_binary_pack_hdr (prog);bpf_jit_binary_pack_free (hdr,NULL );WARN_ON_ONCE (!bpf_prog_kallsyms_verify_off (prog));
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
Re: [PATCH bpf-next v10 3/3] arm64/cfi,bpf: Support kCFI + BPF on arm64
Quoted 30 lines · Sami Tolvanen
From: Puranjay Mohan <[email protected]>
Currently, bpf_dispatcher_*_func() is marked with
__nocfitherefore calling BPF programs from this interface doesn't cause CFI warnings.When BPF programs are called directly from C: from BPF helpers or
struct_ops, CFI warnings are generated.Implement proper CFI prologues for the BPF programs and callbacks and drop
__nocfifor arm64. Fix the trampoline generation code to emit kCFI prologue when astruct_opstrampoline is being prepared.Signed-off-by Puranjay Mohan <[email protected]>Co-developed-by Maxwell Bland <[email protected]>Signed-off-by Maxwell Bland <[email protected]>Co-developed-by Sami Tolvanen <[email protected]>Signed-off-by Sami Tolvanen <[email protected]>Tested-by Dao Huang <[email protected]>--- arch/arm64/include/asm/cfi.h | 7 +++++++ arch/arm64/net/bpf_jit_comp.c | 22 +++++++++++++++++++--- 2 files changed, 26 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/include/asm/cfi.harch/arm64/include/asm/cfi.h new file +0
-0
arch/arm64/include/asm/cfi.h line 7
Quoted 7 lines
arch/arm64/include/asm/cfi.h new file +7
-0@@ -0,0 +1,7 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_ARM64_CFI_H +#define _ASM_ARM64_CFI_H + +#define __bpfcall + +#endif /* _ASM_ARM64_CFI_H */
So close to not needing the arch header at all!
Will