Support kCFI + BPF on arm64
v9 cover Support kCFI + BPF on arm64
Hi folks,
These patches add KCFI types to arm64 BPF JIT output. Puranjay and Maxwell have been working on this for some time now, but I haven't seen any progress since June 2024, so I decided to pick up the latest version[1] posted by Maxwell and fix the few remaining issues I noticed. I confirmed that with these patches applied, I no longer see CFI failures when running BPF self-tests on arm64.
Sami
---
v9:
- Rebased to bpf-next/master to fix x86 merge conflicts.
- Fixed checkpatch warnings about Co-developed-by tags and including <asm/cfi.h>.
- Picked up Tested-by tags.
v8: https://lore.kernel.org/bpf/[email protected]/
- Changed DEFINE_CFI_TYPE to use .4byte to match __CFI_TYPE.
- Changed cfi_get_func_hash() to again use get_kernel_nofault().
- Fixed a panic in bpf_jit_free() by resetting prog->bpf_func before calling bpf_jit_binary_pack_hdr().
---
Mark Rutland (1):
cfi: add C CFI type macro
Puranjay Mohan (1):
arm64/cfi,bpf: Support kCFI + BPF on arm64
arch/arm64/include/asm/cfi.h | 23 ++++++++++++++++++++++ arch/arm64/kernel/alternative.c | 25 +++++++++++++++++++++++ arch/arm64/net/bpf_jit_comp.c | 22 ++++++++++++++++++--- arch/riscv/kernel/cfi.c | 35 +++------------------------------ arch/x86/kernel/alternative.c | 31 +++-------------------------- include/linux/cfi_types.h | 23 ++++++++++++++++++++++ 6 files changed, 96 insertions(+), 63 deletions(-) create mode 100644 arch/arm64/include/asm/cfi.h
base-commit: f263336a41da287c5aebd35be8f1e0422e49bc5c
Signature · 1 line
2.49.0.967.g6a0df3ecc3-goog
PATCH v9 1/2 cfi: add C CFI type macro
From: Mark Rutland <[email protected]>
Currently x86 and riscv open-code 4 instances of the same logic to define a u32 variable with the KCFI typeid of a given function.
Replace the duplicate logic with a common macro.
--- arch/riscv/kernel/cfi.c | 35 +++-------------------------------- arch/x86/kernel/alternative.c | 31 +++---------------------------- include/linux/cfi_types.h | 23 +++++++++++++++++++++++ 3 files changed, 29 insertions(+), 60 deletions(-)
arch/riscv/kernel/cfi.c +3 -32
@@ -4,6 +4,7 @@* * Copyright (C) 2023 Google LLC */ +#include <linux/cfi_types.h> #include <linux/cfi.h> #include <asm/insn.h>
@@ -82,41 +83,11 @@ struct bpf_insn;/* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */ extern unsigned int __bpf_prog_runX (const void *ctx,const struct bpf_insn *insn); - -/* -* Force a reference to the external symbol so the compiler generates -* __kcfi_typid. -*/ -__ADDRESSABLE (__bpf_prog_runX); - -/* u32 __ro_after_init cfi_bpf_hash = __kcfi_typeid___bpf_prog_runX; */ -asm ( -" .pushsection .data..ro_after_init,\"aw\",@progbits \n" -" .type cfi_bpf_hash,@object \n" -" .globl cfi_bpf_hash \n" -" .p2align 2, 0x0 \n" -"cfi_bpf_hash: \n" -" .word __kcfi_typeid___bpf_prog_runX \n" -" .size cfi_bpf_hash, 4 \n" -" .popsection \n" -); +DEFINE_CFI_TYPE (cfi_bpf_hash, __bpf_prog_runX);/* Must match bpf_callback_t */ extern u64 __bpf_callback_fn (u64 ,u64 ,u64 ,u64 ,u64 ); - -__ADDRESSABLE (__bpf_callback_fn); - -/* u32 __ro_after_init cfi_bpf_subprog_hash = __kcfi_typeid___bpf_callback_fn; */ -asm ( -" .pushsection .data..ro_after_init,\"aw\",@progbits \n" -" .type cfi_bpf_subprog_hash,@object \n" -" .globl cfi_bpf_subprog_hash \n" -" .p2align 2, 0x0 \n" -"cfi_bpf_subprog_hash: \n" -" .word __kcfi_typeid___bpf_callback_fn \n" -" .size cfi_bpf_subprog_hash, 4 \n" -" .popsection \n" -); +DEFINE_CFI_TYPE (cfi_bpf_subprog_hash, __bpf_callback_fn);u32 cfi_get_func_hash (void *func) {
arch/x86/kernel/alternative.c +3 -28
@@ -1,6 +1,7 @@// SPDX-License-Identifier: GPL-2.0-only #define pr_fmt (fmt) "SMP alternatives: " fmt +#include <linux/cfi_types.h> #include <linux/module.h> #include <linux/sched.h> #include <linux/perf_event.h>
@@ -947,37 +948,11 @@ struct bpf_insn;/* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */ extern unsigned int __bpf_prog_runX (const void *ctx,const struct bpf_insn *insn); - -KCFI_REFERENCE (__bpf_prog_runX); - -/* u32 __ro_after_init cfi_bpf_hash = __kcfi_typeid___bpf_prog_runX; */ -asm ( -" .pushsection .data..ro_after_init,\"aw\",@progbits \n" -" .type cfi_bpf_hash,@object \n" -" .globl cfi_bpf_hash \n" -" .p2align 2, 0x0 \n" -"cfi_bpf_hash: \n" -" .long __kcfi_typeid___bpf_prog_runX \n" -" .size cfi_bpf_hash, 4 \n" -" .popsection \n" -); +DEFINE_CFI_TYPE (cfi_bpf_hash, __bpf_prog_runX);/* Must match bpf_callback_t */ extern u64 __bpf_callback_fn (u64 ,u64 ,u64 ,u64 ,u64 ); - -KCFI_REFERENCE (__bpf_callback_fn); - -/* u32 __ro_after_init cfi_bpf_subprog_hash = __kcfi_typeid___bpf_callback_fn; */ -asm ( -" .pushsection .data..ro_after_init,\"aw\",@progbits \n" -" .type cfi_bpf_subprog_hash,@object \n" -" .globl cfi_bpf_subprog_hash \n" -" .p2align 2, 0x0 \n" -"cfi_bpf_subprog_hash: \n" -" .long __kcfi_typeid___bpf_callback_fn \n" -" .size cfi_bpf_subprog_hash, 4 \n" -" .popsection \n" -); +DEFINE_CFI_TYPE (cfi_bpf_subprog_hash, __bpf_callback_fn);u32 cfi_get_func_hash (void *func) {
include/linux/cfi_types.h +23 -0
@@ -41,5 +41,28 @@ SYM_TYPED_START(name, SYM_L_GLOBAL, SYM_A_ALIGN)#endif +#else /* __ASSEMBLY__ */ + +#ifdef CONFIG_CFI_CLANG +#define DEFINE_CFI_TYPE (name, func) \ +/* \ +* Force a reference to the function so the compiler generates \ +* __kcfi_typeid_<func>. \ +*/ \ + __ADDRESSABLE(func); \ +/* u32 name = __kcfi_typeid_<func> */ \ +extern u32 name; \ + asm ( \ + " .pushsection .data..ro_after_init,\"aw\",@progbits \n" \ + " .type " #name ",@object \n" \ + " .globl " #name " \n" \ + " .p2align 2, 0x0 \n" \ + #name ": \n" \ + " .4byte __kcfi_typeid_" #func " \n" \ + " .size " #name ", 4 \n" \ + " .popsection \n" \ + ); +#endif +#endif /* __ASSEMBLY__ */ #endif /* _LINUX_CFI_TYPES_H */
Signature · 1 line
2.49.0.967.g6a0df3ecc3-goog
PATCH v9 2/2 arm64/cfi,bpf: Support kCFI + BPF on arm64
In reply to Sami TolvanenFrom: Puranjay Mohan <[email protected]>
Currently, bpf_dispatcher_*_func() is marked with __nocfi therefore calling BPF programs from this interface doesn't cause CFI warnings.
When BPF programs are called directly from C: from BPF helpers or struct_ops, CFI warnings are generated.
Implement proper CFI prologues for the BPF programs and callbacks and drop __nocfi for arm64. Fix the trampoline generation code to emit kCFI prologue when a struct_ops trampoline is being prepared.
--- arch/arm64/include/asm/cfi.h | 23 +++++++++++++++++++++++ arch/arm64/kernel/alternative.c | 25 +++++++++++++++++++++++++ arch/arm64/net/bpf_jit_comp.c | 22 +++++++++++++++++++--- 3 files changed, 67 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/include/asm/cfi.h
arch/arm64/include/asm/cfi.h new file +23 -0
@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_ARM64_CFI_H +#define _ASM_ARM64_CFI_H + +#ifdef CONFIG_CFI_CLANG +#define __bpfcall +static inline int cfi_get_offset (void ) +{ +return 4 ; +} +#define cfi_get_offset cfi_get_offset +extern u32 cfi_bpf_hash; +extern u32 cfi_bpf_subprog_hash; +extern u32 cfi_get_func_hash (void *func); +#else +#define cfi_bpf_hash 0U +#define cfi_bpf_subprog_hash 0U +static inline u32 cfi_get_func_hash (void *func) +{ +return 0 ; +} +#endif /* CONFIG_CFI_CLANG */ +#endif /* _ASM_ARM64_CFI_H */
arch/arm64/kernel/alternative.c +25 -0
@@ -8,11 +8,13 @@#define pr_fmt (fmt) "alternatives: " fmt +#include <linux/cfi_types.h> #include <linux/init.h> #include <linux/cpu.h> #include <linux/elf.h> #include <asm/cacheflush.h> #include <asm/alternative.h> +#include <asm/cfi.h> #include <asm/cpufeature.h> #include <asm/insn.h> #include <asm/module.h>
@@ -298,3 +300,26 @@ noinstr void alt_cb_patch_nops(struct alt_instr *alt, __le32 *origptr, updptr[i] =cpu_to_le32 (aarch64_insn_gen_nop ()); }EXPORT_SYMBOL (alt_cb_patch_nops); + +#ifdef CONFIG_CFI_CLANG +struct bpf_insn ; + +/* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */ +extern unsigned int __bpf_prog_runX (const void *ctx, +const struct bpf_insn *insn); +DEFINE_CFI_TYPE (cfi_bpf_hash, __bpf_prog_runX); + +/* Must match bpf_callback_t */ +extern u64 __bpf_callback_fn (u64 ,u64 ,u64 ,u64 ,u64 ); +DEFINE_CFI_TYPE (cfi_bpf_subprog_hash, __bpf_callback_fn); + +u32 cfi_get_func_hash (void *func) +{ +u32 hash; + +if (get_kernel_nofault (hash, func -cfi_get_offset ())) +return 0 ; + +return hash; +} +#endif /* CONFIG_CFI_CLANG */
arch/arm64/net/bpf_jit_comp.c +19 -3
@@ -9,6 +9,7 @@#include <linux/bitfield.h> #include <linux/bpf.h> +#include <linux/cfi.h> #include <linux/filter.h> #include <linux/memory.h> #include <linux/printk.h>
@@ -164,6 +165,12 @@ static inline void emit_bti(u32 insn, struct jit_ctx *ctx)emit (insn, ctx); } +static inline void emit_kcfi (u32 hash,struct jit_ctx *ctx) +{ +if (IS_ENABLED (CONFIG_CFI_CLANG )) +emit (hash, ctx); +} +/* * Kernel addresses in the vmalloc space use at most 48 bits, and the * remaining bits are guaranteed to be 0x1. So we can compose the address
@@ -474,7 +481,6 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)const bool is_main_prog = !bpf_is_subprog (prog);const u8 fp = bpf2a64[BPF_REG_FP ];const u8 arena_vm_base = bpf2a64[ARENA_VM_START ]; -const int idx0 = ctx->idx;int cur_offset;/*
@@ -500,6 +506,9 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)* */ +emit_kcfi (is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx); +const int idx0 = ctx->idx; +/* bpf function may be invoked by 3 instruction types: * 1. bl, attached via freplace to bpf prog via short jump * 2. br, attached via freplace to bpf prog via long jump
@@ -2009,9 +2018,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) jit_data->ro_header = ro_header; } - prog->bpf_func = (void *)ctx.ro_image; + prog->bpf_func = (void *)ctx.ro_image +cfi_get_offset (); prog->jited =1 ; - prog->jited_len = prog_size; + prog->jited_len = prog_size -cfi_get_offset ();if (!prog->is_func || extra_pass) {int i;
@@ -2271,6 +2280,12 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,/* return address locates above FP */ retaddr_off = stack_size +8 ; +if (flags &BPF_TRAMP_F_INDIRECT ) { +/* +* Indirect call for bpf_struct_ops +*/ +emit_kcfi (cfi_get_func_hash (func_addr), ctx); + }/* bpf trampoline may be invoked by 3 instruction types: * 1. bl, attached to bpf prog or kernel function via short jump * 2. br, attached to bpf prog or kernel function via long jump
@@ -2790,6 +2805,7 @@ void bpf_jit_free(struct bpf_prog *prog) sizeof(jit_data->header->size));kfree (jit_data); } + prog->bpf_func -=cfi_get_offset (); hdr =bpf_jit_binary_pack_hdr (prog);bpf_jit_binary_pack_free (hdr,NULL );WARN_ON_ONCE (!bpf_prog_kallsyms_verify_off (prog));
Signature · 1 line
2.49.0.967.g6a0df3ecc3-goog
Will Deacon
maintainerRe: [PATCH bpf-next v9 2/2] arm64/cfi,bpf: Support kCFI + BPF on arm64
Quoted 31 lines · Sami Tolvanen
From: Puranjay Mohan <[email protected]>
Currently, bpf_dispatcher_*_func() is marked with
__nocfitherefore calling BPF programs from this interface doesn't cause CFI warnings.When BPF programs are called directly from C: from BPF helpers or
struct_ops, CFI warnings are generated.Implement proper CFI prologues for the BPF programs and callbacks and drop
__nocfifor arm64. Fix the trampoline generation code to emit kCFI prologue when astruct_opstrampoline is being prepared.Signed-off-by Puranjay Mohan <[email protected]>Co-developed-by Maxwell Bland <[email protected]>Signed-off-by Maxwell Bland <[email protected]>Co-developed-by Sami Tolvanen <[email protected]>Signed-off-by Sami Tolvanen <[email protected]>Tested-by Dao Huang <[email protected]>--- arch/arm64/include/asm/cfi.h | 23 +++++++++++++++++++++++ arch/arm64/kernel/alternative.c | 25 +++++++++++++++++++++++++ arch/arm64/net/bpf_jit_comp.c | 22 +++++++++++++++++++--- 3 files changed, 67 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/include/asm/cfi.harch/arm64/include/asm/cfi.h new file +0
-0
arch/arm64/include/asm/cfi.h line 9
Quoted 9 lines
arch/arm64/include/asm/cfi.h new file +9
-0@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_ARM64_CFI_H +#define _ASM_ARM64_CFI_H + +#ifdef CONFIG_CFI_CLANG +#define __bpfcall +static inline int cfi_get_offset(void) +{ + return 4;
Needs a comment.
Quoted 4 lines
arch/arm64/include/asm/cfi.h new file +4
-0+} +#define cfi_get_offset cfi_get_offset +extern u32 cfi_bpf_hash; +extern u32 cfi_bpf_subprog_hash;
arch/arm64/include/asm/cfi.h line 23
Quoted 10 lines
arch/arm64/include/asm/cfi.h new file +10
-0+extern u32 cfi_get_func_hash (void *func); +#else +#define cfi_bpf_hash 0U +#define cfi_bpf_subprog_hash 0U +static inline u32 cfi_get_func_hash (void *func) +{ +return 0 ; +} +#endif /* CONFIG_CFI_CLANG */ +#endif /* _ASM_ARM64_CFI_H */
This looks like an awful lot of boiler plate to me. The only thing you seem to need is the CFI offset -- why isn't that just a constant that we can define (or a Kconfig symbol?).
Quoted 35 lines
arch/arm64/kernel/alternative.c +15
-0@@ -8,11 +8,13 @@#define pr_fmt (fmt) "alternatives: " fmt +#include <linux/cfi_types.h> #include <linux/init.h> #include <linux/cpu.h> #include <linux/elf.h> #include <asm/cacheflush.h> #include <asm/alternative.h> +#include <asm/cfi.h> #include <asm/cpufeature.h> #include <asm/insn.h> #include <asm/module.h> @@ -298,3 +300,26 @@ noinstr void alt_cb_patch_nops(struct alt_instr *alt, __le32 *origptr, updptr[i] =cpu_to_le32 (aarch64_insn_gen_nop ()); }EXPORT_SYMBOL (alt_cb_patch_nops); + +#ifdef CONFIG_CFI_CLANG +struct bpf_insn ; + +/* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */ +extern unsigned int __bpf_prog_runX (const void *ctx, +const struct bpf_insn *insn); +DEFINE_CFI_TYPE (cfi_bpf_hash, __bpf_prog_runX); + +/* Must match bpf_callback_t */ +extern u64 __bpf_callback_fn (u64 ,u64 ,u64 ,u64 ,u64 ); +DEFINE_CFI_TYPE (cfi_bpf_subprog_hash, __bpf_callback_fn); +
arch/arm64/kernel/alternative.c noinstr void alt_cb_patch_nops(struct alt_instr *alt, __le32 *origptr, · line 325
Quoted 10 lines
arch/arm64/kernel/alternative.c +10
-0+u32 cfi_get_func_hash (void *func) +{ +u32 hash; + +if (get_kernel_nofault (hash, func -cfi_get_offset ())) +return 0 ; + +return hash; +} +#endif /* CONFIG_CFI_CLANG */
I don't think this should be in alternative.c. It's probably better off either as a 'static inline' in the new cfi.h header.
Quoted 44 lines
arch/arm64/net/bpf_jit_comp.c +10
-1@@ -9,6 +9,7 @@#include <linux/bitfield.h> #include <linux/bpf.h> +#include <linux/cfi.h> #include <linux/filter.h> #include <linux/memory.h> #include <linux/printk.h> @@ -164,6 +165,12 @@ static inline void emit_bti(u32 insn, struct jit_ctx *ctx)emit (insn, ctx); } +static inline void emit_kcfi (u32 hash,struct jit_ctx *ctx) +{ +if (IS_ENABLED (CONFIG_CFI_CLANG )) +emit (hash, ctx); +} +/* * Kernel addresses in the vmalloc space use at most 48 bits, and the * remaining bits are guaranteed to be 0x1. So we can compose the address @@ -474,7 +481,6 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)const bool is_main_prog = !bpf_is_subprog (prog);const u8 fp = bpf2a64[BPF_REG_FP ];const u8 arena_vm_base = bpf2a64[ARENA_VM_START ]; -const int idx0 = ctx->idx;int cur_offset;/* @@ -500,6 +506,9 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)* */ +emit_kcfi (is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx); +const int idx0 = ctx->idx; +/* bpf function may be invoked by 3 instruction types: * 1. bl, attached via freplace to bpf prog via short jump * 2. br, attached via freplace to bpf prog via long jump
arch/arm64/net/bpf_jit_comp.c struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) · line 2023
Quoted 8 lines
arch/arm64/net/bpf_jit_comp.c +2
-2@@ -2009,9 +2018,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) jit_data->ro_header = ro_header; } - prog->bpf_func = (void *)ctx.ro_image; + prog->bpf_func = (void *)ctx.ro_image +cfi_get_offset (); prog->jited =1 ; - prog->jited_len = prog_size; + prog->jited_len = prog_size -cfi_get_offset ();
Why do we add the offset even when CONFIG_CFI_CLANG is not enabled?
Will
Re: [PATCH bpf-next v9 2/2] arm64/cfi,bpf: Support kCFI + BPF on arm64
Hi Will,
Quoted 36 lines · Will Deacon
Quoted 34 lines · Sami Tolvanen
From: Puranjay Mohan <[email protected]>
Currently, bpf_dispatcher_*_func() is marked with
__nocfitherefore calling BPF programs from this interface doesn't cause CFI warnings.When BPF programs are called directly from C: from BPF helpers or
struct_ops, CFI warnings are generated.Implement proper CFI prologues for the BPF programs and callbacks and drop
__nocfifor arm64. Fix the trampoline generation code to emit kCFI prologue when astruct_opstrampoline is being prepared.Signed-off-by Puranjay Mohan <[email protected]>Co-developed-by Maxwell Bland <[email protected]>Signed-off-by Maxwell Bland <[email protected]>Co-developed-by Sami Tolvanen <[email protected]>Signed-off-by Sami Tolvanen <[email protected]>Tested-by Dao Huang <[email protected]>--- arch/arm64/include/asm/cfi.h | 23 +++++++++++++++++++++++ arch/arm64/kernel/alternative.c | 25 +++++++++++++++++++++++++ arch/arm64/net/bpf_jit_comp.c | 22 +++++++++++++++++++--- 3 files changed, 67 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/include/asm/cfi.harch/arm64/include/asm/cfi.h new file +3
-0@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_ARM64_CFI_H +#define _ASM_ARM64_CFI_H
Quoted 8 lines
arch/arm64/include/asm/cfi.h line 9
Quoted 6 lines
arch/arm64/include/asm/cfi.h new file +6
-0+ +#ifdef CONFIG_CFI_CLANG +#define __bpfcall +static inline int cfi_get_offset (void ) +{ +return 4 ;Needs a comment.
Ack.
Quoted 10 lines
Quoted 10 lines
arch/arm64/include/asm/cfi.h new file +10
-0+} +#define cfi_get_offset cfi_get_offset +extern u32 cfi_bpf_hash; +extern u32 cfi_bpf_subprog_hash; +extern u32 cfi_get_func_hash (void *func); +#else +#define cfi_bpf_hash 0U +#define cfi_bpf_subprog_hash 0U +static inline u32 cfi_get_func_hash (void *func) +{
Quoted 8 lines
arch/arm64/include/asm/cfi.h line 23
Quoted 4 lines
arch/arm64/include/asm/cfi.h new file +4
-0+return 0 ; +} +#endif /* CONFIG_CFI_CLANG */ +#endif /* _ASM_ARM64_CFI_H */ This looks like an awful lot of boiler plate to me. The only thing you seem to need is the CFI offset -- why isn't that just a constant that we can define (or a
Kconfigsymbol?).
The cfi_get_offset function was originally added in commit 4f9087f16651 ("x86/cfi,bpf: Fix BPF JIT call") because the offset can change on x86 depending on which CFI scheme is enabled at runtime. Starting with commit 2cd3e3772e41 ("x86/cfi,bpf: Fix bpf_struct_ops CFI") the function is also called by the generic BPF code, so we can't trivially replace it with a constant. However, since this defaults to 4 unless the architecture adds pre-function NOPs, I think we could simply move the default implementation to include/linux/cfi.h (and also drop the RISC-V version). Come to think of it, we could probably move most of this boilerplate to generic code. I'll refactor this and send a new version.
Quoted 40 lines
Quoted 40 lines
arch/arm64/kernel/alternative.c +20
-0@@ -8,11 +8,13 @@#define pr_fmt (fmt) "alternatives: " fmt +#include <linux/cfi_types.h> #include <linux/init.h> #include <linux/cpu.h> #include <linux/elf.h> #include <asm/cacheflush.h> #include <asm/alternative.h> +#include <asm/cfi.h> #include <asm/cpufeature.h> #include <asm/insn.h> #include <asm/module.h> @@ -298,3 +300,26 @@ noinstr void alt_cb_patch_nops(struct alt_instr *alt, __le32 *origptr, updptr[i] =cpu_to_le32 (aarch64_insn_gen_nop ()); }EXPORT_SYMBOL (alt_cb_patch_nops); + +#ifdef CONFIG_CFI_CLANG +struct bpf_insn ; + +/* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */ +extern unsigned int __bpf_prog_runX (const void *ctx, +const struct bpf_insn *insn); +DEFINE_CFI_TYPE (cfi_bpf_hash, __bpf_prog_runX); + +/* Must match bpf_callback_t */ +extern u64 __bpf_callback_fn (u64 ,u64 ,u64 ,u64 ,u64 ); +DEFINE_CFI_TYPE (cfi_bpf_subprog_hash, __bpf_callback_fn); + +u32 cfi_get_func_hash (void *func) +{ +u32 hash; + +if (get_kernel_nofault (hash, func -cfi_get_offset ()))
Quoted 8 lines
arch/arm64/kernel/alternative.c noinstr void alt_cb_patch_nops(struct alt_instr *alt, __le32 *origptr, · line 325
Quoted 5 lines
arch/arm64/kernel/alternative.c +5
-0+return 0 ; + + return hash; +} +#endif /* CONFIG_CFI_CLANG */ I don't think this should be in
alternative.c. It's probably better off either as a 'static inline' in the newcfi.hheader.
Sure, I'll find a better place for this. RISC-V again seems to have the exact same function, so I think a __weak implementation in kernel/cfi.c would work here, allowing x86 to still conveniently override the function.
Quoted 47 lines
Quoted 47 lines
arch/arm64/net/bpf_jit_comp.c +10
-1@@ -9,6 +9,7 @@#include <linux/bitfield.h> #include <linux/bpf.h> +#include <linux/cfi.h> #include <linux/filter.h> #include <linux/memory.h> #include <linux/printk.h> @@ -164,6 +165,12 @@ static inline void emit_bti(u32 insn, struct jit_ctx *ctx)emit (insn, ctx); } +static inline void emit_kcfi (u32 hash,struct jit_ctx *ctx) +{ +if (IS_ENABLED (CONFIG_CFI_CLANG )) +emit (hash, ctx); +} +/* * Kernel addresses in the vmalloc space use at most 48 bits, and the * remaining bits are guaranteed to be 0x1. So we can compose the address @@ -474,7 +481,6 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)const bool is_main_prog = !bpf_is_subprog (prog);const u8 fp = bpf2a64[BPF_REG_FP ];const u8 arena_vm_base = bpf2a64[ARENA_VM_START ]; -const int idx0 = ctx->idx;int cur_offset;/* @@ -500,6 +506,9 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)* */ +emit_kcfi (is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx); +const int idx0 = ctx->idx; +/* bpf function may be invoked by 3 instruction types: * 1. bl, attached via freplace to bpf prog via short jump * 2. br, attached via freplace to bpf prog via long jump @@ -2009,9 +2018,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) jit_data->ro_header = ro_header; }
Quoted 7 lines
arch/arm64/net/bpf_jit_comp.c struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) · line 2023
Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +2
-2- prog->bpf_func = (void *)ctx.ro_image; + prog->bpf_func = (void *)ctx.ro_image +cfi_get_offset (); prog->jited =1 ; - prog->jited_len = prog_size; + prog->jited_len = prog_size -cfi_get_offset ();Why do we add the offset even when
CONFIG_CFI_CLANGis not enabled?
The function returns zero if CFI is not enabled, so I believe it's just to avoid extra if (IS_ENABLED(CONFIG_CFI_CLANG)) statements in the code. IMO this is cleaner, but I can certainly change this if you prefer.
Thanks for taking a look!
Sami
Will Deacon
maintainerRe: [PATCH bpf-next v9 2/2] arm64/cfi,bpf: Support kCFI + BPF on arm64
Hey Sami,
Quoted 18 lines · Sami Tolvanen
Quoted 9 lines
Quoted 9 lines
arch/arm64/include/asm/cfi.h new file +9
-0+#define cfi_get_offset cfi_get_offset +extern u32 cfi_bpf_hash; +extern u32 cfi_bpf_subprog_hash; +extern u32 cfi_get_func_hash (void *func); +#else +#define cfi_bpf_hash 0U +#define cfi_bpf_subprog_hash 0U +static inline u32 cfi_get_func_hash (void *func) +{Quoted 8 lines
arch/arm64/include/asm/cfi.h line 23
Quoted 4 lines
arch/arm64/include/asm/cfi.h new file +4
-0+return 0 ; +} +#endif /* CONFIG_CFI_CLANG */ +#endif /* _ASM_ARM64_CFI_H */ This looks like an awful lot of boiler plate to me. The only thing you seem to need is the CFI offset -- why isn't that just a constant that we can define (or a
Kconfigsymbol?).
Quoted 11 lines
The
cfi_get_offsetfunction was originally added in commit4f9087f16651("x86/cfi,bpf: Fix BPF JIT call") because the offset can change on x86 depending on which CFI scheme is enabled at runtime. Starting with commit2cd3e3772e41("x86/cfi,bpf: Fixbpf_struct_opsCFI") the function is also called by the generic BPF code, so we can't trivially replace it with a constant. However, since this defaults to4unless the architecture adds pre-function NOPs, I think we could simply move the default implementation toinclude/linux/cfi.h(and also drop the RISC-V version). Come to think of it, we could probably move most of this boilerplate to generic code. I'll refactor this and send a new version.
Excellent, thanks.
Quoted 12 lines
Quoted 11 lines
arch/arm64/net/bpf_jit_comp.c struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) · line 2023
Quoted 9 lines
arch/arm64/net/bpf_jit_comp.c +2
-2@@ -2009,9 +2018,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) jit_data->ro_header = ro_header; } - prog->bpf_func = (void *)ctx.ro_image; + prog->bpf_func = (void *)ctx.ro_image +cfi_get_offset (); prog->jited =1 ; - prog->jited_len = prog_size; + prog->jited_len = prog_size -cfi_get_offset ();Why do we add the offset even when
CONFIG_CFI_CLANGis not enabled?
Quoted 4 lines
The function returns zero if CFI is not enabled, so I believe it's just to avoid extra if (
IS_ENABLED(CONFIG_CFI_CLANG)) statements in the code. IMO this is cleaner, but I can certainly change this if you prefer.
Ah, that caught me out because the !CONFIG_CFI_CLANG stub is in the core code (and I'm extra susceptible to being caught out on a warm Friday evening!).
Hopefully if you're able to trim down the boilerplate then this will become more obvious too.
Cheers,
Will
Quoted 8 lines · Sami Tolvanen
Hi folks,
These patches add KCFI types to arm64 BPF JIT output. Puranjay and Maxwell have been working on this for some time now, but I haven't seen any progress since June 2024, so I decided to pick up the latest version[1] posted by Maxwell and fix the few remaining issues I noticed. I confirmed that with these patches applied, I no longer see CFI failures when running BPF self-tests on arm64.
Bump! Thank you Sami for following up on this, hopefully the maintainers will have time to take a look!
Regards,
Maxwell