x86/cfi,bpf: Fix CFI vs eBPF
Subsystems (9)
v3 cover x86/cfi,bpf: Fix CFI vs eBPF
Hi!
What started with the simple observation that bpf_dispatcher_*_func() was broken for calling CFI functions with a __nocfi calling context for FineIBT ended up with a complete BPF wide CFI fixup.
With these changes on the BPF selftest suite passes without crashing -- there's still a few failures, but Alexei has graciously offered to look into those.
(Alexei, I have presumed your SoB on the very last patch, please update as you see fit)
Changes since v2 are numerous but include:
- cfi_get_offset() -- as a means to communicate the offset (ast)
- 5 new patches fixing various BPF internals to be CFI clean
Note: it *might* be possible to merge the bpf_bpf_tcp_ca.c:unsupported_ops[] thing into the CFI stubs, as is get_info will have a NULL stub, unlike the others.
--- arch/riscv/include/asm/cfi.h | 3 +- arch/riscv/kernel/cfi.c | 2 +- arch/x86/include/asm/cfi.h | 126 +++++++++++++++++++++++++++++++++++++- arch/x86/kernel/alternative.c | 87 +++++++++++++++++++++++--- arch/x86/kernel/cfi.c | 4 +- arch/x86/net/bpf_jit_comp.c | 134 +++++++++++++++++++++++++++++++++++------ include/asm-generic/Kbuild | 1 + include/linux/bpf.h | 27 ++++++++- include/linux/cfi.h | 12 ++++ kernel/bpf/bpf_struct_ops.c | 16 ++--- kernel/bpf/core.c | 25 ++++++++ kernel/bpf/cpumask.c | 8 ++- kernel/bpf/helpers.c | 18 +++++- net/bpf/bpf_dummy_struct_ops.c | 31 +++++++++- net/bpf/test_run.c | 15 ++++- net/ipv4/bpf_tcp_ca.c | 69 +++++++++++++++++++++ 16 files changed, 528 insertions(+), 50 deletions(-)
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH v3 1/7 cfi: Flip headers
Normal include order is that linux/foo.h should include asm/foo.h, CFI has it the wrong way around.
--- arch/riscv/include/asm/cfi.h | 3 ++- arch/riscv/kernel/cfi.c | 2 +- arch/x86/include/asm/cfi.h | 3 ++- arch/x86/kernel/cfi.c | 4 ++-- include/asm-generic/Kbuild | 1 + include/asm-generic/cfi.h | 5 +++++ include/linux/cfi.h | 1 + 7 files changed, 14 insertions(+), 5 deletions(-)
arch/riscv/include/asm/cfi.h +2 -1
@@ -7,8 +7,9 @@* * Copyright (C) 2023 Google LLC */ +#include <linux/bug.h> -#include <linux/cfi.h> +struct pt_regs ;#ifdef CONFIG_CFI_CLANG enum bug_trap_type handle_cfi_failure (struct pt_regs *regs);
arch/riscv/kernel/cfi.c +1 -1
@@ -4,7 +4,7 @@* * Copyright (C) 2023 Google LLC */ -#include <asm/cfi.h> +#include <linux/cfi.h> #include <asm/insn.h> /*
arch/x86/include/asm/cfi.h +2 -1
@@ -7,8 +7,9 @@* * Copyright (C) 2022 Google LLC */ +#include <linux/bug.h> -#include <linux/cfi.h> +struct pt_regs ;#ifdef CONFIG_CFI_CLANG enum bug_trap_type handle_cfi_failure (struct pt_regs *regs);
arch/x86/kernel/cfi.c +2 -2
@@ -4,10 +4,10 @@* * Copyright (C) 2022 Google LLC */ -#include <asm/cfi.h> +#include <linux/string.h> +#include <linux/cfi.h> #include <asm/insn.h> #include <asm/insn-eval.h> -#include <linux/string.h> /* * Returns the target address and the expected type when regs->ip points
include/asm-generic/Kbuild +1 -0
@@ -11,6 +11,7 @@ mandatory-y += bitops.h mandatory-y += bug.h mandatory-y += bugs.h mandatory-y += cacheflush.h +mandatory-y += cfi.h mandatory-y += checksum.h mandatory-y += compat.h mandatory-y += current.h
include/asm-generic/cfi.h new file +5 -0
@@ -0,0 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_GENERIC_CFI_H +#define __ASM_GENERIC_CFI_H + +#endif /* __ASM_GENERIC_CFI_H */
include/linux/cfi.h +1 -0
@@ -9,6 +9,7 @@#include <linux/bug.h> #include <linux/module.h> +#include <asm/cfi.h> #ifdef CONFIG_CFI_CLANG enum bug_trap_type report_cfi_failure(struct pt_regs *regs, unsigned long addr,
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH v3 2/7 x86/cfi,bpf: Fix BPF JIT call
In reply to Peter ZijlstraThe current BPF call convention is __nocfi, except when it calls !JIT things, then it calls regular C functions.
It so happens that with FineIBT the __nocfi and C calling conventions are incompatible. Specifically __nocfi will call at func+0, while FineIBT will have endbr-poison there, which is not a valid indirect target. Causing #CP.
Notably this only triggers on IBT enabled hardware, which is probably why this hasn't been reported (also, most people will have JIT on anyway).
Implement proper CFI prologues for the BPF JIT codegen and drop __nocfi for x86.
--- arch/x86/include/asm/cfi.h | 110 ++++++++++++++++++++++++++++++++++++++++++ arch/x86/kernel/alternative.c | 47 ++++++++++++++--- arch/x86/net/bpf_jit_comp.c | 82 +++++++++++++++++++++++++++++-- include/linux/bpf.h | 12 +++- include/linux/cfi.h | 7 ++ kernel/bpf/core.c | 25 +++++++++ 6 files changed, 269 insertions(+), 14 deletions(-)
arch/x86/include/asm/cfi.h +110 -0
@@ -9,15 +9,125 @@*/ #include <linux/bug.h> +/* +* An overview of the various calling conventions... +* +* Traditional: +* +* foo: +* ... code here ... +* ret +* +* direct caller: +* call foo +* +* indirect caller: +* lea foo(%rip), %r11 +* ... +* call *%r11 +* +* +* IBT: +* +* foo: +* endbr64 +* ... code here ... +* ret +* +* direct caller: +* call foo / call foo+4 +* +* indirect caller: +* lea foo(%rip), %r11 +* ... +* call *%r11 +* +* +* kCFI: +* +* __cfi_foo: +* movl $0x12345678, %eax +* # 11 nops when CONFIG_CALL_PADDING +* foo: +* endbr64 # when IBT +* ... code here ... +* ret +* +* direct call: +* call foo # / call foo+4 when IBT +* +* indirect call: +* lea foo(%rip), %r11 +* ... +* movl $(-0x12345678), %r10d +* addl -4(%r11), %r10d # -15 when CONFIG_CALL_PADDING +* jz 1f +* ud2 +* 1:call *%r11 +* +* +* FineIBT (builds as kCFI + CALL_PADDING + IBT + RETPOLINE and runtime patches into): +* +* __cfi_foo: +* endbr64 +* subl 0x12345678, %r10d +* jz foo +* ud2 +* nop +* foo: +* osp nop3 # was endbr64 +* ... code here ... +* ret +* +* direct caller: +* call foo / call foo+4 +* +* indirect caller: +* lea foo(%rip), %r11 +* ... +* movl $0x12345678, %r10d +* subl $16, %r11 +* nop4 +* call *%r11 +* +*/ +enum cfi_mode { +CFI_DEFAULT ,/* FineIBT if hardware has IBT, otherwise kCFI */ +CFI_OFF ,/* Taditional / IBT depending on .config */ +CFI_KCFI ,/* Optionally CALL_PADDING, IBT, RETPOLINE */ +CFI_FINEIBT ,/* see arch/x86/kernel/alternative.c */ +}; + +extern enum cfi_mode cfi_mode; +struct pt_regs ;#ifdef CONFIG_CFI_CLANG enum bug_trap_type handle_cfi_failure (struct pt_regs *regs); +#define __bpfcall +extern u32 cfi_bpf_hash; + +static inline int cfi_get_offset (void ) +{ +switch (cfi_mode) { +case CFI_FINEIBT : +return 16 ; +case CFI_KCFI : +if (IS_ENABLED (CONFIG_CALL_PADDING )) +return 16 ; +return 5 ; +default : +return 0 ; + } +} +#define cfi_get_offset cfi_get_offset +#else static inline enum bug_trap_type handle_cfi_failure (struct pt_regs *regs) {return BUG_TRAP_TYPE_NONE ; } +#define cfi_bpf_hash 0U#endif /* CONFIG_CFI_CLANG */ #endif /* _ASM_X86_CFI_H */
arch/x86/kernel/alternative.c +39 -8
68 lines not on this page · raw message
arch/x86/net/bpf_jit_comp.c +78 -4
124 lines not on this page · raw message
include/linux/bpf.h +10 -2
@@ -29,6 +29,7 @@#include <linux/rcupdate_trace.h> #include <linux/static_call.h> #include <linux/memcontrol.h> +#include <linux/cfi.h> struct bpf_verifier_env ;struct bpf_verifier_log ;
@@ -1211,7 +1212,11 @@ struct bpf_dispatcher {#endif }; -static __always_inline __nocfi unsigned int bpf_dispatcher_nop_func( +#ifndef __bpfcall +#define __bpfcall __nocfi +#endif + +static __always_inline __bpfcall unsigned int bpf_dispatcher_nop_func(const void *ctx,const struct bpf_insn *insnsi,bpf_func_t bpf_func)
@@ -1303,7 +1308,7 @@ int arch_prepare_bpf_dispatcher(void *im#define DEFINE_BPF_DISPATCHER (name) \ __BPF_DISPATCHER_SC(name); \ - noinline __nocfi unsigned int bpf_dispatcher_##name##_func( \ + noinline __bpfcall unsigned int bpf_dispatcher_##name##_func( \ const void *ctx, \ const struct bpf_insn *insnsi, \ bpf_func_t bpf_func) \
@@ -1453,6 +1458,9 @@ struct bpf_prog_aux {struct bpf_kfunc_desc_tab *kfunc_tab;struct bpf_kfunc_btf_tab *kfunc_btf_tab;u32 size_poke_tab; +#ifdef CONFIG_FINEIBT +struct bpf_ksym ksym_prefix; +#endif struct bpf_ksym ksym;const struct bpf_prog_ops *ops;struct bpf_map **used_maps;
include/linux/cfi.h +7 -0
@@ -11,6 +11,13 @@#include <linux/module.h> #include <asm/cfi.h> +#ifndef cfi_get_offset +static inline int cfi_get_offset (void ) +{ +return 0 ; +} +#endif +#ifdef CONFIG_CFI_CLANG enum bug_trap_type report_cfi_failure (struct pt_regs *regs,unsigned long addr,unsigned long *target,u32 type);
kernel/bpf/core.c +25 -0
@@ -121,6 +121,9 @@ struct bpf_prog *bpf_prog_alloc_no_stats#endif INIT_LIST_HEAD_RCU (&fp->aux->ksym.lnode); +#ifdef CONFIG_FINEIBT +INIT_LIST_HEAD_RCU (&fp->aux->ksym_prefix.lnode); +#endif mutex_init (&fp->aux->used_maps_mutex);mutex_init (&fp->aux->dst_mutex);
@@ -683,6 +686,23 @@ void bpf_prog_kallsyms_add(struct bpf_pr fp->aux->ksym.prog = true;bpf_ksym_add (&fp->aux->ksym); + +#ifdef CONFIG_FINEIBT +/* +* When FineIBT, code in the __cfi_foo() symbols can get executed +* and hence unwinder needs help. +*/ +if (cfi_mode !=CFI_FINEIBT ) +return ; + +snprintf (fp->aux->ksym_prefix.name,KSYM_NAME_LEN , +"__cfi_%s" , fp->aux->ksym.name); + + fp->aux->ksym_prefix.start = (unsigned long ) fp->bpf_func -16 ; + fp->aux->ksym_prefix.end = (unsigned long ) fp->bpf_func; + +bpf_ksym_add (&fp->aux->ksym_prefix); +#endif }void bpf_prog_kallsyms_del (struct bpf_prog *fp)
@@ -691,6 +711,11 @@ void bpf_prog_kallsyms_del(struct bpf_prreturn ;bpf_ksym_del (&fp->aux->ksym); +#ifdef CONFIG_FINEIBT +if (cfi_mode !=CFI_FINEIBT ) +return ; +bpf_ksym_del (&fp->aux->ksym_prefix); +#endif }static struct bpf_ksym *bpf_ksym_find (unsigned long addr)
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH v3 3/7 x86/cfi,bpf: Fix bpf_callback_t CFI
In reply to Peter ZijlstraWhere the main BPF program is expected to match bpf_func_t, sub-programs are expected to match bpf_callback_t.
This fixes things like:
tools/testing/selftests/bpf/progs/bloom_filter_bench.c:
bpf_for_each_map_elem(&array_map, bloom_callback, &data, 0);
--- arch/x86/include/asm/cfi.h | 2 ++ arch/x86/kernel/alternative.c | 18 ++++++++++++++++++ arch/x86/net/bpf_jit_comp.c | 18 ++++++++++-------- 3 files changed, 30 insertions(+), 8 deletions(-)
arch/x86/include/asm/cfi.h +2 -0
@@ -106,6 +106,7 @@ struct pt_regs;enum bug_trap_type handle_cfi_failure (struct pt_regs *regs);#define __bpfcallextern u32 cfi_bpf_hash; +extern u32 cfi_bpf_subprog_hash;static inline int cfi_get_offset (void ) {
@@ -128,6 +129,7 @@ static inline enum bug_trap_type handle_return BUG_TRAP_TYPE_NONE ; }#define cfi_bpf_hash 0U +#define cfi_bpf_subprog_hash 0U#endif /* CONFIG_CFI_CLANG */ #endif /* _ASM_X86_CFI_H */
arch/x86/kernel/alternative.c +18 -0
@@ -866,6 +866,23 @@ asm (" .size cfi_bpf_hash, 4 \n" " .popsection \n" ); + +/* Must match bpf_callback_t */ +extern u64 __bpf_callback_fn (u64 ,u64 ,u64 ,u64 ,u64 ); + +__ADDRESSABLE (__bpf_callback_fn); + +/* u32 __ro_after_init cfi_bpf_subprog_hash = __kcfi_typeid___bpf_callback_fn; */ +asm ( +" .pushsection .data..ro_after_init,\"aw\",@progbits \n" +" .type cfi_bpf_subprog_hash,@object \n" +" .globl cfi_bpf_subprog_hash \n" +" .p2align 2, 0x0 \n" +"cfi_bpf_subprog_hash: \n" +" .long __kcfi_typeid___bpf_callback_fn \n" +" .size cfi_bpf_subprog_hash, 4 \n" +" .popsection \n" +);#endif #ifdef CONFIG_FINEIBT
@@ -1181,6 +1198,7 @@ static void __apply_fineibt(s32 *start_rif (builtin) { cfi_seed =get_random_u32 (); cfi_bpf_hash =cfi_rehash (cfi_bpf_hash); + cfi_bpf_subprog_hash =cfi_rehash (cfi_bpf_subprog_hash); } ret =cfi_rand_preamble (start_cfi, end_cfi);
arch/x86/net/bpf_jit_comp.c +10 -8
@@ -312,12 +312,13 @@ static void pop_callee_regs(u8 **pprog,* in arch/x86/kernel/alternative.c */ -static void emit_fineibt (u8 **pprog) +static void emit_fineibt (u8 **pprog,bool is_subprog) { +u32 hash = is_subprog ? cfi_bpf_subprog_hash : cfi_bpf_hash;u8 *prog = *pprog;EMIT_ENDBR (); -EMIT3_off32 (0x41 ,0x81 ,0xea , cfi_bpf_hash);/* subl $hash, %r10d */ +EMIT3_off32 (0x41 ,0x81 ,0xea , hash);/* subl $hash, %r10d */ EMIT2 (0x74 ,0x07 );/* jz.d8 +7 */ EMIT2 (0x0f ,0x0b );/* ud2 */ EMIT1 (0x90 );/* nop */
@@ -326,11 +327,12 @@ static void emit_fineibt(u8 **pprog) *pprog = prog; } -static void emit_kcfi (u8 **pprog) +static void emit_kcfi (u8 **pprog,bool is_subprog) { +u32 hash = is_subprog ? cfi_bpf_subprog_hash : cfi_bpf_hash;u8 *prog = *pprog; -EMIT1_off32 (0xb8 , cfi_bpf_hash);/* movl $hash, %eax */ +EMIT1_off32 (0xb8 , hash);/* movl $hash, %eax */ #ifdef CONFIG_CALL_PADDING EMIT1 (0x90 );EMIT1 (0x90 );
@@ -349,17 +351,17 @@ static void emit_kcfi(u8 **pprog) *pprog = prog; } -static void emit_cfi (u8 **pprog) +static void emit_cfi (u8 **pprog,bool is_subprog) {u8 *prog = *pprog;switch (cfi_mode) {case CFI_FINEIBT : -emit_fineibt (&prog); +emit_fineibt (&prog, is_subprog);break ;case CFI_KCFI : -emit_kcfi (&prog); +emit_kcfi (&prog, is_subprog);break ;default :
@@ -381,7 +383,7 @@ static void emit_prologue(u8 **pprog, u3 {u8 *prog = *pprog; -emit_cfi (&prog); +emit_cfi (&prog, is_subprog);/* BPF trampoline can be made to work without these nops, * but let's waste 5 bytes for now and optimize later */
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH v3 4/7 x86/cfi,bpf: Fix bpf_struct_ops CFI
In reply to Peter ZijlstraBPF struct_ops uses __arch_prepare_bpf_trampoline() to write trampolines for indirect function calls. These tramplines much have matching CFI.
In order to obtain the correct CFI hash for the various methods, add a matching structure that contains stub functions, the compiler will generate correct CFI which we can pilfer for the trampolines.
--- arch/x86/include/asm/cfi.h | 6 +++ arch/x86/kernel/alternative.c | 22 +++++++++++++ arch/x86/net/bpf_jit_comp.c | 66 +++++++++++++++++++++++++-------------- include/linux/bpf.h | 13 +++++++ kernel/bpf/bpf_struct_ops.c | 16 ++++----- net/bpf/bpf_dummy_struct_ops.c | 31 +++++++++++++++++- net/ipv4/bpf_tcp_ca.c | 69 +++++++++++++++++++++++++++++++++++++++++ 7 files changed, 191 insertions(+), 32 deletions(-)
arch/x86/include/asm/cfi.h +6 -0
@@ -123,6 +123,8 @@ static inline int cfi_get_offset(void) }#define cfi_get_offset cfi_get_offset +extern u32 cfi_get_func_hash (void *func); +#else static inline enum bug_trap_type handle_cfi_failure (struct pt_regs *regs) {
@@ -130,6 +132,10 @@ static inline enum bug_trap_type handle_ }#define cfi_bpf_hash 0U#define cfi_bpf_subprog_hash 0U +static inline u32 cfi_get_func_hash (void *func) +{ +return 0 ; +}#endif /* CONFIG_CFI_CLANG */ #endif /* _ASM_X86_CFI_H */
arch/x86/kernel/alternative.c +22 -0
@@ -883,6 +883,28 @@ asm (" .size cfi_bpf_subprog_hash, 4 \n" " .popsection \n" ); + +u32 cfi_get_func_hash (void *func) +{ +u32 hash; + + func -=cfi_get_offset (); +switch (cfi_mode) { +case CFI_FINEIBT : + func +=7 ; +break ; +case CFI_KCFI : + func +=1 ; +break ; +default : +return 0 ; + } + +if (get_kernel_nofault (hash, func)) +return 0 ; + +return hash; +}#endif #ifdef CONFIG_FINEIBT
arch/x86/net/bpf_jit_comp.c +43 -23
173 lines not on this page · raw message
include/linux/bpf.h +13 -0
@@ -1060,6 +1060,17 @@ struct btf_func_model {*/ #define BPF_TRAMP_F_TAIL_CALL_CTX BIT(7) +/* +* Indicate the trampoline should be suitable to receive indirect calls; +* without this indirectly calling the generated code can result in #UD/#CP, +* depending on the CFI options. +* +* Used by bpf_struct_ops. +* +* Incompatible with FENTRY usage, overloads @func_addr argument. +*/ +#define BPF_TRAMP_F_INDIRECT BIT(8) +/* Each call __bpf_prog_enter + call bpf_func + call __bpf_prog_exit is ~50 * bytes on x86. */
@@ -1695,6 +1706,7 @@ struct bpf_struct_ops {struct btf_func_model func_models[BPF_STRUCT_OPS_MAX_NR_MEMBERS ];u32 type_id;u32 value_id; +void *cfi_stubs; };#if defined(CONFIG_BPF_JIT ) && defined(CONFIG_BPF_SYSCALL )
@@ -1708,6 +1720,7 @@ int bpf_struct_ops_map_sys_lookup_elem(sint bpf_struct_ops_prepare_trampoline (struct bpf_tramp_links *tlinks,struct bpf_tramp_link *link,const struct btf_func_model *model, +void *stub_func,void *image,void *image_end);static inline bool bpf_try_module_get (const void *data,struct module *owner) {
kernel/bpf/bpf_struct_ops.c +8 -8
@@ -352,17 +352,16 @@ const struct bpf_link_ops bpf_struct_opsint bpf_struct_ops_prepare_trampoline (struct bpf_tramp_links *tlinks,struct bpf_tramp_link *link,const struct btf_func_model *model, -void *image,void *image_end) +void *stub_func,void *image,void *image_end) { -u32 flags; +u32 flags =BPF_TRAMP_F_INDIRECT ;int size; tlinks[BPF_TRAMP_FENTRY ].links[0 ] = link; tlinks[BPF_TRAMP_FENTRY ].nr_links =1 ; -/* BPF_TRAMP_F_RET_FENTRY_RET is only used by bpf_struct_ops, -* and it must be used alone. -*/ - flags = model->ret_size >0 ?BPF_TRAMP_F_RET_FENTRY_RET :0 ; + +if (model->ret_size >0 ) + flags |=BPF_TRAMP_F_RET_FENTRY_RET ; size =arch_bpf_trampoline_size (model, flags, tlinks,NULL );if (size <0 )
@@ -370,7 +369,7 @@ int bpf_struct_ops_prepare_trampoline(stif (size > (unsigned long )image_end - (unsigned long )image)return -E2BIG ;return arch_prepare_bpf_trampoline (NULL , image, image_end, - model, flags, tlinks,NULL ); + model, flags, tlinks, stub_func); } static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,
@@ -504,11 +503,12 @@ static long bpf_struct_ops_map_update_el err =bpf_struct_ops_prepare_trampoline (tlinks, link, &st_ops->func_models[i], + *(void **)(st_ops->cfi_stubs + moff), image, image_end);if (err <0 ) goto reset_unlock; - *(void **)(kdata + moff) = image; + *(void **)(kdata + moff) = image +cfi_get_offset (); image += err;/* put prog_id to udata */
net/bpf/bpf_dummy_struct_ops.c +30 -1
@@ -12,6 +12,11 @@ extern struct bpf_struct_ops bpf_bpf_dum/* A common type for test_N with return value in bpf_dummy_ops */ typedef int (*dummy_ops_test_ret_fn )(struct bpf_dummy_ops_state *state, ...); +static int dummy_ops_test_ret_function (struct bpf_dummy_ops_state *state, ...) +{ +return 0 ; +} +struct bpf_dummy_ops_test_args {u64 args[MAX_BPF_FUNC_ARGS ];struct bpf_dummy_ops_state state;
@@ -62,7 +67,7 @@ static int dummy_ops_copy_args(struct bpstatic int dummy_ops_call_op (void *image,struct bpf_dummy_ops_test_args *args) { -dummy_ops_test_ret_fn test = (void *)image; +dummy_ops_test_ret_fn test = (void *)image +cfi_get_offset ();struct bpf_dummy_ops_state *state =NULL ;/* state needs to be NULL if args[0] is 0 */
@@ -119,6 +124,7 @@ int bpf_struct_ops_test_run(struct bpf_p op_idx = prog->expected_attach_type; err =bpf_struct_ops_prepare_trampoline (tlinks, link, &st_ops->func_models[op_idx], + &dummy_ops_test_ret_function, image, image +PAGE_SIZE );if (err <0 ) goto out;
@@ -219,6 +225,28 @@ static void bpf_dummy_unreg(void *kdata) { } +static int bpf_dummy_test_1 (struct bpf_dummy_ops_state *cb) +{ +return 0 ; +} + +static int bpf_dummy_test_2 (struct bpf_dummy_ops_state *cb,int a1,unsigned short a2, +char a3,unsigned long a4) +{ +return 0 ; +} + +static int bpf_dummy_test_sleepable (struct bpf_dummy_ops_state *cb) +{ +return 0 ; +} + +static struct bpf_dummy_ops __bpf_bpf_dummy_ops = { + .test_1 = bpf_dummy_test_1, + .test_2 = bpf_dummy_test_2, + .test_sleepable = bpf_dummy_test_sleepable, +}; +struct bpf_struct_ops bpf_bpf_dummy_ops = { .verifier_ops = &bpf_dummy_verifier_ops, .init = bpf_dummy_init,
@@ -227,4 +255,5 @@ struct bpf_struct_ops bpf_bpf_dummy_ops .reg = bpf_dummy_reg, .unreg = bpf_dummy_unreg, .name = "bpf_dummy_ops", + .cfi_stubs = &__bpf_bpf_dummy_ops, };
net/ipv4/bpf_tcp_ca.c +69 -0
81 lines not on this page · raw message
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH v3 5/7 cfi: Add CFI_NOSEAL()
In reply to Peter ZijlstraAdd a CFI_NOSEAL() helper to mark functions that need to retain their CFI information, despite not otherwise leaking their address.
--- arch/x86/include/asm/cfi.h | 5 +++++ include/linux/cfi.h | 4 ++++ 2 files changed, 9 insertions(+)
arch/x86/include/asm/cfi.h +5 -0
@@ -8,6 +8,7 @@* Copyright (C) 2022 Google LLC */ #include <linux/bug.h> +#include <asm/ibt.h> /* * An overview of the various calling conventions...
@@ -138,4 +139,8 @@ static inline u32 cfi_get_func_hash(void }#endif /* CONFIG_CFI_CLANG */ +#if HAS_KERNEL_IBT ==1 +#define CFI_NOSEAL (x) asm(IBT_NOSEAL(__stringify(x))) +#endif +#endif /* _ASM_X86_CFI_H */
include/linux/cfi.h +4 -0
@@ -46,4 +46,8 @@ static inline void module_cfi_finalize(c#endif /* CONFIG_ARCH_USES_CFI_TRAPS */ #endif /* CONFIG_MODULES */ +#ifndef CFI_NOSEAL +#define CFI_NOSEAL (x) +#endif +#endif /* _LINUX_CFI_H */
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH v3 6/7 bpf: Fix dtor CFI
In reply to Peter ZijlstraEnsure the various dtor functions match their prototype and retain their CFI signatures, since they don't have their address taken, they are prone to not getting CFI, making them impossible to call indirectly.
--- kernel/bpf/cpumask.c | 8 +++++++- kernel/bpf/helpers.c | 16 ++++++++++++++-- net/bpf/test_run.c | 15 +++++++++++++-- 3 files changed, 34 insertions(+), 5 deletions(-)
kernel/bpf/cpumask.c +7 -1
@@ -96,6 +96,12 @@ __bpf_kfunc void bpf_cpumask_release(strmigrate_enable (); } +__bpf_kfunc void bpf_cpumask_release_dtor(void *cpumask) +{ +bpf_cpumask_release (cpumask); +} +CFI_NOSEAL (bpf_cpumask_release_dtor); +/** * bpf_cpumask_first() - Get the index of the first nonzero bit in the cpumask. * @cpumask: The cpumask being queried.
@@ -441,7 +447,7 @@ static const struct btf_kfunc_id_set cpuBTF_ID_LIST (cpumask_dtor_ids) BTF_ID(struct, bpf_cpumask) -BTF_ID (func, bpf_cpumask_release) +BTF_ID (func, bpf_cpumask_release_dtor) static int__init cpumask_kfunc_init (void) {
kernel/bpf/helpers.c +14 -2
@@ -2150,6 +2150,12 @@ __bpf_kfunc void bpf_task_release(structput_task_struct_rcu_user (p); } +__bpf_kfunc void bpf_task_release_dtor(void *p) +{ +put_task_struct_rcu_user (p); +} +CFI_NOSEAL (bpf_task_release_dtor); +#ifdef CONFIG_CGROUPS /** * bpf_cgroup_acquire - Acquire a reference to a cgroup. A cgroup acquired by
@@ -2174,6 +2180,12 @@ __bpf_kfunc void bpf_cgroup_release(strucgroup_put (cgrp); } +__bpf_kfunc void bpf_cgroup_release_dtor(void *cgrp) +{ +cgroup_put (cgrp); +} +CFI_NOSEAL (bpf_cgroup_release_dtor); +/** * bpf_cgroup_ancestor - Perform a lookup on an entry in a cgroup's ancestor * array. A cgroup returned by this kfunc which is not subsequently stored in a
@@ -2570,10 +2582,10 @@ static const struct btf_kfunc_id_set gen BTF_ID_LIST(generic_dtor_ids) BTF_ID(struct, task_struct) -BTF_ID(func, bpf_task_release) +BTF_ID(func, bpf_task_release_dtor) #ifdef CONFIG_CGROUPS BTF_ID(struct, cgroup) -BTF_ID(func, bpf_cgroup_release) +BTF_ID(func, bpf_cgroup_release_dtor) #endif BTF_SET8_START(common_btf_ids)
net/bpf/test_run.c +13 -2
@@ -600,10 +600,21 @@ __bpf_kfunc void bpf_kfunc_call_test_relrefcount_dec (&p->cnt); } +__bpf_kfunc void bpf_kfunc_call_test_release_dtor(void *p) +{ +bpf_kfunc_call_test_release (p); +} +CFI_NOSEAL (bpf_kfunc_call_test_release_dtor); + __bpf_kfunc void bpf_kfunc_call_memb_release(struct prog_test_member *p) { } +__bpf_kfunc void bpf_kfunc_call_memb_release_dtor(void *p) +{ +} +CFI_NOSEAL (bpf_kfunc_call_memb_release_dtor); +__bpf_kfunc_end_defs ();BTF_SET8_START (bpf_test_modify_return_ids)
@@ -1671,9 +1682,9 @@ static const struct btf_kfunc_id_set bpfBTF_ID_LIST (bpf_prog_test_dtor_kfunc_ids) BTF_ID(struct, prog_test_ref_kfunc) -BTF_ID (func, bpf_kfunc_call_test_release) +BTF_ID (func, bpf_kfunc_call_test_release_dtor)BTF_ID (struct, prog_test_member) -BTF_ID (func, bpf_kfunc_call_memb_release) +BTF_ID (func, bpf_kfunc_call_memb_release_dtor) static int__init bpf_prog_test_run_init (void) {
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH v3 7/7 x86/cfi,bpf: Fix bpf_exception_cb() signature
In reply to Peter ZijlstraFrom: Alexei Starovoitov <[email protected]>
As per the earlier patches, BPF sub-programs have bpf_callback_t signature and CFI expects callers to have matching signature. This is violated by bpf_prog_aux::bpf_exception_cb().
--- include/linux/bpf.h | 2 +- kernel/bpf/helpers.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
include/linux/bpf.h +1 -1
@@ -1484,7 +1484,7 @@ struct bpf_prog_aux {int cgroup_atype;/* enum cgroup_bpf_attach_type */ struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE ];char name[BPF_OBJ_NAME_LEN ]; -unsigned int (*bpf_exception_cb)(u64 cookie,u64 sp,u64 bp); +u64 (*bpf_exception_cb)(u64 cookie,u64 sp,u64 bp,u64 ,u64 );#ifdef CONFIG_SECURITY void *security;#endif
kernel/bpf/helpers.c +1 -1
@@ -2537,7 +2537,7 @@ __bpf_kfunc void bpf_throw(u64 cookie)* which skips compiler generated instrumentation to do the same. */ kasan_unpoison_task_stack_below ((void *)(long )ctx.sp); - ctx.aux->bpf_exception_cb (cookie, ctx.sp, ctx.bp); + ctx.aux->bpf_exception_cb (cookie, ctx.sp, ctx.bp,0 ,0 );WARN (1 ,"A call to BPF exception callback should never return\n" ); }
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 10 lines · On Fri, 15 Dec 2023 10:12:16 +0100 you wrote:
Hi!
What started with the simple observation that bpf_dispatcher_*_func() was broken for calling CFI functions with a
__nocficalling context for FineIBT ended up with a complete BPF wide CFI fixup.With these changes on the BPF selftest suite passes without crashing -- there's still a few failures, but Alexei has graciously offered to look into those.
[...]
Here is the summary with links:
- [v3,1/7] cfi: Flip headers
https://git.kernel.org/bpf/bpf-next/c/4382159696c9 - [v3,2/7] x86/cfi,bpf: Fix BPF JIT call https://git.kernel.org/bpf/bpf-next/c/4f9087f16651 - [v3,3/7] x86/cfi,bpf: Fix bpf_callback_t CFI https://git.kernel.org/bpf/bpf-next/c/e72d88d18df4 - [v3,4/7] x86/cfi,bpf: Fix bpf_struct_ops CFI https://git.kernel.org/bpf/bpf-next/c/2cd3e3772e41 - [v3,5/7] cfi: Add CFI_NOSEAL() https://git.kernel.org/bpf/bpf-next/c/e9d13b9d2f99 - [v3,6/7] bpf: Fix dtor CFI https://git.kernel.org/bpf/bpf-next/c/e4c00339891c - [v3,7/7] x86/cfi,bpf: Fix bpf_exception_cb() signature https://git.kernel.org/bpf/bpf-next/c/852486b35f34
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Quoted 10 lines · Peter Zijlstra
Hi!
What started with the simple observation that bpf_dispatcher_*_func() was broken for calling CFI functions with a
__nocficalling context for FineIBT ended up with a complete BPF wide CFI fixup.With these changes on the BPF selftest suite passes without crashing -- there's still a few failures, but Alexei has graciously offered to look into those.
Quoted 6 lines
(Alexei, I have presumed your SoB on the very last patch, please update as you see fit)
Changes since v2 are numerous but include:
-cfi_get_offset()-- as a means to communicate the offset (ast)
- 5 new patches fixing various BPF internals to be CFI clean
Looks great to me. Pushed to bpf-next.
There is a failure on s390 that I temporarily denylisted with an extra patch.
And sent a proposed fix:
https://lore.kernel.org/bpf/[email protected]/
Ilya,
please take a look.
Quoted 3 lines
Note: it *might* be possible to merge the bpf_bpf_tcp_ca.c:unsupported_ops[] thing into the CFI stubs, as is
get_infowill have a NULL stub, unlike the others.
That's a good idea. Will clean up unsupported_ops. Either myself or Martin will follow up.