Skip to contentSkip to conversation

Support kCFI + BPF on arm64

PATCH Superseded Sami Tolvanen 3 participants 7 messages May 5, 2025 – Jul 13, 2025

Subsystems (5)
x86 BPF RISC-V arm64 +1 more subsystems

v9 of 14 Superseded Latest is v14 →

lore
Share & tools
lore
#0 lore raw

v9 cover Support kCFI + BPF on arm64

Hi folks,

These patches add KCFI types to arm64 BPF JIT output. Puranjay and Maxwell have been working on this for some time now, but I haven't seen any progress since June 2024, so I decided to pick up the latest version[1] posted by Maxwell and fix the few remaining issues I noticed. I confirmed that with these patches applied, I no longer see CFI failures when running BPF self-tests on arm64.

[1] https://lore.kernel.org/linux-arm-kernel/ptrugmna4xb5o5lo4xislf4rlz7avdmd4pfho5fjwtjj7v422u@iqrwfrbwuxrq/

Sami

---

v9:
- Rebased to bpf-next/master to fix x86 merge conflicts.
- Fixed checkpatch warnings about Co-developed-by tags and including <asm/cfi.h>.
- Picked up Tested-by tags.

v8: https://lore.kernel.org/bpf/[email protected]/
- Changed DEFINE_CFI_TYPE to use .4byte to match __CFI_TYPE.
- Changed cfi_get_func_hash() to again use get_kernel_nofault().
- Fixed a panic in bpf_jit_free() by resetting prog->bpf_func before calling bpf_jit_binary_pack_hdr().

---

Mark Rutland (1):
cfi: add C CFI type macro

Puranjay Mohan (1):
arm64/cfi,bpf: Support kCFI + BPF on arm64

 arch/arm64/include/asm/cfi.h    | 23 ++++++++++++++++++++++
 arch/arm64/kernel/alternative.c | 25 +++++++++++++++++++++++
 arch/arm64/net/bpf_jit_comp.c   | 22 ++++++++++++++++++---
 arch/riscv/kernel/cfi.c         | 35 +++------------------------------
 arch/x86/kernel/alternative.c   | 31 +++--------------------------
 include/linux/cfi_types.h       | 23 ++++++++++++++++++++++
 6 files changed, 96 insertions(+), 63 deletions(-)
 create mode 100644 arch/arm64/include/asm/cfi.h

base-commit: f263336a41da287c5aebd35be8f1e0422e49bc5c

Signature · 1 line
2.49.0.967.g6a0df3ecc3-goog
#1 lore raw

PATCH v9 1/2 cfi: add C CFI type macro

From: Mark Rutland <[email protected]>

Currently x86 and riscv open-code 4 instances of the same logic to define a u32 variable with the KCFI typeid of a given function.

Replace the duplicate logic with a common macro.

---
 arch/riscv/kernel/cfi.c       | 35 +++--------------------------------
 arch/x86/kernel/alternative.c | 31 +++----------------------------
 include/linux/cfi_types.h     | 23 +++++++++++++++++++++++
 3 files changed, 29 insertions(+), 60 deletions(-)
arch/riscv/kernel/cfi.c +3 -32
index 64bdd3e1ab8c..e7aec5f36dd5 100644
@@ -4,6 +4,7 @@
  *
  * Copyright (C) 2023 Google LLC
  */
+#include <linux/cfi_types.h>
 #include <linux/cfi.h>
 #include <asm/insn.h>
 
@@ -82,41 +83,11 @@ struct bpf_insn;
 /* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */
 extern unsigned int __bpf_prog_runX(const void *ctx,
 				    const struct bpf_insn *insn);
-
-/*
- * Force a reference to the external symbol so the compiler generates
- * __kcfi_typid.
- */
-__ADDRESSABLE(__bpf_prog_runX);
-
-/* u32 __ro_after_init cfi_bpf_hash = __kcfi_typeid___bpf_prog_runX; */
-asm (
-"	.pushsection	.data..ro_after_init,\"aw\",@progbits	\n"
-"	.type	cfi_bpf_hash,@object				\n"
-"	.globl	cfi_bpf_hash					\n"
-"	.p2align	2, 0x0					\n"
-"cfi_bpf_hash:							\n"
-"	.word	__kcfi_typeid___bpf_prog_runX			\n"
-"	.size	cfi_bpf_hash, 4					\n"
-"	.popsection						\n"
-);
+DEFINE_CFI_TYPE(cfi_bpf_hash, __bpf_prog_runX);
 
 /* Must match bpf_callback_t */
 extern u64 __bpf_callback_fn(u64, u64, u64, u64, u64);
-
-__ADDRESSABLE(__bpf_callback_fn);
-
-/* u32 __ro_after_init cfi_bpf_subprog_hash = __kcfi_typeid___bpf_callback_fn; */
-asm (
-"	.pushsection	.data..ro_after_init,\"aw\",@progbits	\n"
-"	.type	cfi_bpf_subprog_hash,@object			\n"
-"	.globl	cfi_bpf_subprog_hash				\n"
-"	.p2align	2, 0x0					\n"
-"cfi_bpf_subprog_hash:						\n"
-"	.word	__kcfi_typeid___bpf_callback_fn			\n"
-"	.size	cfi_bpf_subprog_hash, 4				\n"
-"	.popsection						\n"
-);
+DEFINE_CFI_TYPE(cfi_bpf_subprog_hash, __bpf_callback_fn);
 
 u32 cfi_get_func_hash(void *func)
 {
arch/x86/kernel/alternative.c +3 -28
index bf82c6f7d690..a5147fcd8397 100644
@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0-only
 #define pr_fmt(fmt) "SMP alternatives: " fmt
 
+#include <linux/cfi_types.h>
 #include <linux/module.h>
 #include <linux/sched.h>
 #include <linux/perf_event.h>
@@ -947,37 +948,11 @@ struct bpf_insn;
 /* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */
 extern unsigned int __bpf_prog_runX(const void *ctx,
 				    const struct bpf_insn *insn);
-
-KCFI_REFERENCE(__bpf_prog_runX);
-
-/* u32 __ro_after_init cfi_bpf_hash = __kcfi_typeid___bpf_prog_runX; */
-asm (
-"	.pushsection	.data..ro_after_init,\"aw\",@progbits	\n"
-"	.type	cfi_bpf_hash,@object				\n"
-"	.globl	cfi_bpf_hash					\n"
-"	.p2align	2, 0x0					\n"
-"cfi_bpf_hash:							\n"
-"	.long	__kcfi_typeid___bpf_prog_runX			\n"
-"	.size	cfi_bpf_hash, 4					\n"
-"	.popsection						\n"
-);
+DEFINE_CFI_TYPE(cfi_bpf_hash, __bpf_prog_runX);
 
 /* Must match bpf_callback_t */
 extern u64 __bpf_callback_fn(u64, u64, u64, u64, u64);
-
-KCFI_REFERENCE(__bpf_callback_fn);
-
-/* u32 __ro_after_init cfi_bpf_subprog_hash = __kcfi_typeid___bpf_callback_fn; */
-asm (
-"	.pushsection	.data..ro_after_init,\"aw\",@progbits	\n"
-"	.type	cfi_bpf_subprog_hash,@object			\n"
-"	.globl	cfi_bpf_subprog_hash				\n"
-"	.p2align	2, 0x0					\n"
-"cfi_bpf_subprog_hash:						\n"
-"	.long	__kcfi_typeid___bpf_callback_fn			\n"
-"	.size	cfi_bpf_subprog_hash, 4				\n"
-"	.popsection						\n"
-);
+DEFINE_CFI_TYPE(cfi_bpf_subprog_hash, __bpf_callback_fn);
 
 u32 cfi_get_func_hash(void *func)
 {
include/linux/cfi_types.h +23 -0
index 6b8713675765..209c8a16ac4e 100644
@@ -41,5 +41,28 @@
 	SYM_TYPED_START(name, SYM_L_GLOBAL, SYM_A_ALIGN)
 #endif
 
+#else /* __ASSEMBLY__ */
+
+#ifdef CONFIG_CFI_CLANG
+#define DEFINE_CFI_TYPE(name, func)						\
+	/*									\
+	 * Force a reference to the function so the compiler generates		\
+	 * __kcfi_typeid_<func>.						\
+	 */									\
+	__ADDRESSABLE(func);							\
+	/* u32 name = __kcfi_typeid_<func> */					\
+	extern u32 name;							\
+	asm (									\
+	"	.pushsection	.data..ro_after_init,\"aw\",@progbits	\n"	\
+	"	.type	" #name ",@object				\n"	\
+	"	.globl	" #name "					\n"	\
+	"	.p2align	2, 0x0					\n"	\
+	#name ":							\n"	\
+	"	.4byte	__kcfi_typeid_" #func "				\n"	\
+	"	.size	" #name ", 4					\n"	\
+	"	.popsection						\n"	\
+	);
+#endif
+
 #endif /* __ASSEMBLY__ */
 #endif /* _LINUX_CFI_TYPES_H */
Signature · 1 line
2.49.0.967.g6a0df3ecc3-goog
#2 lore raw

PATCH v9 2/2 arm64/cfi,bpf: Support kCFI + BPF on arm64

In reply to Sami Tolvanen

From: Puranjay Mohan <[email protected]>

Currently, bpf_dispatcher_*_func() is marked with __nocfi therefore calling BPF programs from this interface doesn't cause CFI warnings.

When BPF programs are called directly from C: from BPF helpers or struct_ops, CFI warnings are generated.

Implement proper CFI prologues for the BPF programs and callbacks and drop __nocfi for arm64. Fix the trampoline generation code to emit kCFI prologue when a struct_ops trampoline is being prepared.

---
 arch/arm64/include/asm/cfi.h    | 23 +++++++++++++++++++++++
 arch/arm64/kernel/alternative.c | 25 +++++++++++++++++++++++++
 arch/arm64/net/bpf_jit_comp.c   | 22 +++++++++++++++++++---
 3 files changed, 67 insertions(+), 3 deletions(-)
 create mode 100644 arch/arm64/include/asm/cfi.h
arch/arm64/include/asm/cfi.h new file +23 -0
new file mode 100644
index 000000000000..670e191f8628
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_ARM64_CFI_H
+#define _ASM_ARM64_CFI_H
+
+#ifdef CONFIG_CFI_CLANG
+#define __bpfcall
+static inline int cfi_get_offset(void)
+{
+	return 4;
+}
+#define cfi_get_offset cfi_get_offset
+extern u32 cfi_bpf_hash;
+extern u32 cfi_bpf_subprog_hash;
+extern u32 cfi_get_func_hash(void *func);
+#else
+#define cfi_bpf_hash 0U
+#define cfi_bpf_subprog_hash 0U
+static inline u32 cfi_get_func_hash(void *func)
+{
+	return 0;
+}
+#endif /* CONFIG_CFI_CLANG */
+#endif /* _ASM_ARM64_CFI_H */
arch/arm64/kernel/alternative.c +25 -0
index 8ff6610af496..71c153488dad 100644
@@ -8,11 +8,13 @@
 
 #define pr_fmt(fmt) "alternatives: " fmt
 
+#include <linux/cfi_types.h>
 #include <linux/init.h>
 #include <linux/cpu.h>
 #include <linux/elf.h>
 #include <asm/cacheflush.h>
 #include <asm/alternative.h>
+#include <asm/cfi.h>
 #include <asm/cpufeature.h>
 #include <asm/insn.h>
 #include <asm/module.h>
@@ -298,3 +300,26 @@ noinstr void alt_cb_patch_nops(struct alt_instr *alt, __le32 *origptr,
 		updptr[i] = cpu_to_le32(aarch64_insn_gen_nop());
 }
 EXPORT_SYMBOL(alt_cb_patch_nops);
+
+#ifdef CONFIG_CFI_CLANG
+struct bpf_insn;
+
+/* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */
+extern unsigned int __bpf_prog_runX(const void *ctx,
+				    const struct bpf_insn *insn);
+DEFINE_CFI_TYPE(cfi_bpf_hash, __bpf_prog_runX);
+
+/* Must match bpf_callback_t */
+extern u64 __bpf_callback_fn(u64, u64, u64, u64, u64);
+DEFINE_CFI_TYPE(cfi_bpf_subprog_hash, __bpf_callback_fn);
+
+u32 cfi_get_func_hash(void *func)
+{
+	u32 hash;
+
+	if (get_kernel_nofault(hash, func - cfi_get_offset()))
+		return 0;
+
+	return hash;
+}
+#endif /* CONFIG_CFI_CLANG */
arch/arm64/net/bpf_jit_comp.c +19 -3
index 70d7c89d3ac9..3b3691e88dd5 100644
@@ -9,6 +9,7 @@
 
 #include <linux/bitfield.h>
 #include <linux/bpf.h>
+#include <linux/cfi.h>
 #include <linux/filter.h>
 #include <linux/memory.h>
 #include <linux/printk.h>
@@ -164,6 +165,12 @@ static inline void emit_bti(u32 insn, struct jit_ctx *ctx)
 		emit(insn, ctx);
 }
 
+static inline void emit_kcfi(u32 hash, struct jit_ctx *ctx)
+{
+	if (IS_ENABLED(CONFIG_CFI_CLANG))
+		emit(hash, ctx);
+}
+
 /*
  * Kernel addresses in the vmalloc space use at most 48 bits, and the
  * remaining bits are guaranteed to be 0x1. So we can compose the address
@@ -474,7 +481,6 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
 	const bool is_main_prog = !bpf_is_subprog(prog);
 	const u8 fp = bpf2a64[BPF_REG_FP];
 	const u8 arena_vm_base = bpf2a64[ARENA_VM_START];
-	const int idx0 = ctx->idx;
 	int cur_offset;
 
 	/*
@@ -500,6 +506,9 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
 	 *
 	 */
 
+	emit_kcfi(is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx);
+	const int idx0 = ctx->idx;
+
 	/* bpf function may be invoked by 3 instruction types:
 	 * 1. bl, attached via freplace to bpf prog via short jump
 	 * 2. br, attached via freplace to bpf prog via long jump
@@ -2009,9 +2018,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		jit_data->ro_header = ro_header;
 	}
 
-	prog->bpf_func = (void *)ctx.ro_image;
+	prog->bpf_func = (void *)ctx.ro_image + cfi_get_offset();
 	prog->jited = 1;
-	prog->jited_len = prog_size;
+	prog->jited_len = prog_size - cfi_get_offset();
 
 	if (!prog->is_func || extra_pass) {
 		int i;
@@ -2271,6 +2280,12 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
 	/* return address locates above FP */
 	retaddr_off = stack_size + 8;
 
+	if (flags & BPF_TRAMP_F_INDIRECT) {
+		/*
+		 * Indirect call for bpf_struct_ops
+		 */
+		emit_kcfi(cfi_get_func_hash(func_addr), ctx);
+	}
 	/* bpf trampoline may be invoked by 3 instruction types:
 	 * 1. bl, attached to bpf prog or kernel function via short jump
 	 * 2. br, attached to bpf prog or kernel function via long jump
@@ -2790,6 +2805,7 @@ void bpf_jit_free(struct bpf_prog *prog)
 					   sizeof(jit_data->header->size));
 			kfree(jit_data);
 		}
+		prog->bpf_func -= cfi_get_offset();
 		hdr = bpf_jit_binary_pack_hdr(prog);
 		bpf_jit_binary_pack_free(hdr, NULL);
 		WARN_ON_ONCE(!bpf_prog_kallsyms_verify_off(prog));
Signature · 1 line
2.49.0.967.g6a0df3ecc3-goog
maintainer
#3 lore raw

Re: [PATCH bpf-next v9 2/2] arm64/cfi,bpf: Support kCFI + BPF on arm64

Quoted 31 lines · Sami Tolvanen

From: Puranjay Mohan <[email protected]>

Currently, bpf_dispatcher_*_func() is marked with __nocfi therefore calling BPF programs from this interface doesn't cause CFI warnings.

When BPF programs are called directly from C: from BPF helpers or struct_ops, CFI warnings are generated.

Implement proper CFI prologues for the BPF programs and callbacks and drop __nocfi for arm64. Fix the trampoline generation code to emit kCFI prologue when a struct_ops trampoline is being prepared.

---
 arch/arm64/include/asm/cfi.h    | 23 +++++++++++++++++++++++
 arch/arm64/kernel/alternative.c | 25 +++++++++++++++++++++++++
 arch/arm64/net/bpf_jit_comp.c   | 22 +++++++++++++++++++---
 3 files changed, 67 insertions(+), 3 deletions(-)
 create mode 100644 arch/arm64/include/asm/cfi.h
arch/arm64/include/asm/cfi.h new file +0 -0
new file mode 100644
index 000000000000..670e191f8628

arch/arm64/include/asm/cfi.h line 9

Quoted 9 lines
arch/arm64/include/asm/cfi.h new file +9 -0
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_ARM64_CFI_H
+#define _ASM_ARM64_CFI_H
+
+#ifdef CONFIG_CFI_CLANG
+#define __bpfcall
+static inline int cfi_get_offset(void)
+{
+	return 4;

Needs a comment.

Quoted 4 lines
arch/arm64/include/asm/cfi.h new file +4 -0
+}
+#define cfi_get_offset cfi_get_offset
+extern u32 cfi_bpf_hash;
+extern u32 cfi_bpf_subprog_hash;

arch/arm64/include/asm/cfi.h line 23

Quoted 10 lines
arch/arm64/include/asm/cfi.h new file +10 -0
+extern u32 cfi_get_func_hash(void *func);
+#else
+#define cfi_bpf_hash 0U
+#define cfi_bpf_subprog_hash 0U
+static inline u32 cfi_get_func_hash(void *func)
+{
+	return 0;
+}
+#endif /* CONFIG_CFI_CLANG */
+#endif /* _ASM_ARM64_CFI_H */

This looks like an awful lot of boiler plate to me. The only thing you seem to need is the CFI offset -- why isn't that just a constant that we can define (or a Kconfig symbol?).

Quoted 35 lines
arch/arm64/kernel/alternative.c +15 -0
index 8ff6610af496..71c153488dad 100644
@@ -8,11 +8,13 @@
 
 #define pr_fmt(fmt) "alternatives: " fmt
 
+#include <linux/cfi_types.h>
 #include <linux/init.h>
 #include <linux/cpu.h>
 #include <linux/elf.h>
 #include <asm/cacheflush.h>
 #include <asm/alternative.h>
+#include <asm/cfi.h>
 #include <asm/cpufeature.h>
 #include <asm/insn.h>
 #include <asm/module.h>
@@ -298,3 +300,26 @@ noinstr void alt_cb_patch_nops(struct alt_instr *alt, __le32 *origptr,
 		updptr[i] = cpu_to_le32(aarch64_insn_gen_nop());
 }
 EXPORT_SYMBOL(alt_cb_patch_nops);
+
+#ifdef CONFIG_CFI_CLANG
+struct bpf_insn;
+
+/* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */
+extern unsigned int __bpf_prog_runX(const void *ctx,
+				    const struct bpf_insn *insn);
+DEFINE_CFI_TYPE(cfi_bpf_hash, __bpf_prog_runX);
+
+/* Must match bpf_callback_t */
+extern u64 __bpf_callback_fn(u64, u64, u64, u64, u64);
+DEFINE_CFI_TYPE(cfi_bpf_subprog_hash, __bpf_callback_fn);
+

arch/arm64/kernel/alternative.c noinstr void alt_cb_patch_nops(struct alt_instr *alt, __le32 *origptr, · line 325

Quoted 10 lines
arch/arm64/kernel/alternative.c +10 -0
+u32 cfi_get_func_hash(void *func)
+{
+	u32 hash;
+
+	if (get_kernel_nofault(hash, func - cfi_get_offset()))
+		return 0;
+
+	return hash;
+}
+#endif /* CONFIG_CFI_CLANG */

I don't think this should be in alternative.c. It's probably better off either as a 'static inline' in the new cfi.h header.

Quoted 44 lines
arch/arm64/net/bpf_jit_comp.c +10 -1
index 70d7c89d3ac9..3b3691e88dd5 100644
@@ -9,6 +9,7 @@
 
 #include <linux/bitfield.h>
 #include <linux/bpf.h>
+#include <linux/cfi.h>
 #include <linux/filter.h>
 #include <linux/memory.h>
 #include <linux/printk.h>
@@ -164,6 +165,12 @@ static inline void emit_bti(u32 insn, struct jit_ctx *ctx)
 		emit(insn, ctx);
 }
 
+static inline void emit_kcfi(u32 hash, struct jit_ctx *ctx)
+{
+	if (IS_ENABLED(CONFIG_CFI_CLANG))
+		emit(hash, ctx);
+}
+
 /*
  * Kernel addresses in the vmalloc space use at most 48 bits, and the
  * remaining bits are guaranteed to be 0x1. So we can compose the address
@@ -474,7 +481,6 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
 	const bool is_main_prog = !bpf_is_subprog(prog);
 	const u8 fp = bpf2a64[BPF_REG_FP];
 	const u8 arena_vm_base = bpf2a64[ARENA_VM_START];
-	const int idx0 = ctx->idx;
 	int cur_offset;
 
 	/*
@@ -500,6 +506,9 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
 	 *
 	 */
 
+	emit_kcfi(is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx);
+	const int idx0 = ctx->idx;
+
 	/* bpf function may be invoked by 3 instruction types:
 	 * 1. bl, attached via freplace to bpf prog via short jump
 	 * 2. br, attached via freplace to bpf prog via long jump

arch/arm64/net/bpf_jit_comp.c struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) · line 2023

Quoted 8 lines
arch/arm64/net/bpf_jit_comp.c +2 -2
@@ -2009,9 +2018,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		jit_data->ro_header = ro_header;
 	}
 
-	prog->bpf_func = (void *)ctx.ro_image;
+	prog->bpf_func = (void *)ctx.ro_image + cfi_get_offset();
 	prog->jited = 1;
-	prog->jited_len = prog_size;
+	prog->jited_len = prog_size - cfi_get_offset();

Why do we add the offset even when CONFIG_CFI_CLANG is not enabled?

Will

#4 lore raw

Re: [PATCH bpf-next v9 2/2] arm64/cfi,bpf: Support kCFI + BPF on arm64

Hi Will,

Quoted 36 lines · Will Deacon
Quoted 34 lines · Sami Tolvanen

From: Puranjay Mohan <[email protected]>

Currently, bpf_dispatcher_*_func() is marked with __nocfi therefore calling BPF programs from this interface doesn't cause CFI warnings.

When BPF programs are called directly from C: from BPF helpers or struct_ops, CFI warnings are generated.

Implement proper CFI prologues for the BPF programs and callbacks and drop __nocfi for arm64. Fix the trampoline generation code to emit kCFI prologue when a struct_ops trampoline is being prepared.

---
 arch/arm64/include/asm/cfi.h    | 23 +++++++++++++++++++++++
 arch/arm64/kernel/alternative.c | 25 +++++++++++++++++++++++++
 arch/arm64/net/bpf_jit_comp.c   | 22 +++++++++++++++++++---
 3 files changed, 67 insertions(+), 3 deletions(-)
 create mode 100644 arch/arm64/include/asm/cfi.h
arch/arm64/include/asm/cfi.h new file +3 -0
new file mode 100644
index 000000000000..670e191f8628
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_ARM64_CFI_H
+#define _ASM_ARM64_CFI_H
Quoted 8 lines

arch/arm64/include/asm/cfi.h line 9

Quoted 6 lines
arch/arm64/include/asm/cfi.h new file +6 -0
+
+#ifdef CONFIG_CFI_CLANG
+#define __bpfcall
+static inline int cfi_get_offset(void)
+{
+     return 4;

Needs a comment.

Ack.

Quoted 10 lines
Quoted 10 lines
arch/arm64/include/asm/cfi.h new file +10 -0
+}
+#define cfi_get_offset cfi_get_offset
+extern u32 cfi_bpf_hash;
+extern u32 cfi_bpf_subprog_hash;
+extern u32 cfi_get_func_hash(void *func);
+#else
+#define cfi_bpf_hash 0U
+#define cfi_bpf_subprog_hash 0U
+static inline u32 cfi_get_func_hash(void *func)
+{
Quoted 8 lines

arch/arm64/include/asm/cfi.h line 23

Quoted 4 lines
arch/arm64/include/asm/cfi.h new file +4 -0
+     return 0;
+}
+#endif /* CONFIG_CFI_CLANG */
+#endif /* _ASM_ARM64_CFI_H */

This looks like an awful lot of boiler plate to me. The only thing you seem to need is the CFI offset -- why isn't that just a constant that we can define (or a Kconfig symbol?).

The cfi_get_offset function was originally added in commit 4f9087f16651 ("x86/cfi,bpf: Fix BPF JIT call") because the offset can change on x86 depending on which CFI scheme is enabled at runtime. Starting with commit 2cd3e3772e41 ("x86/cfi,bpf: Fix bpf_struct_ops CFI") the function is also called by the generic BPF code, so we can't trivially replace it with a constant. However, since this defaults to 4 unless the architecture adds pre-function NOPs, I think we could simply move the default implementation to include/linux/cfi.h (and also drop the RISC-V version). Come to think of it, we could probably move most of this boilerplate to generic code. I'll refactor this and send a new version.

Quoted 40 lines
Quoted 40 lines
arch/arm64/kernel/alternative.c +20 -0
index 8ff6610af496..71c153488dad 100644
@@ -8,11 +8,13 @@
 
 #define pr_fmt(fmt) "alternatives: " fmt
 
+#include <linux/cfi_types.h>
 #include <linux/init.h>
 #include <linux/cpu.h>
 #include <linux/elf.h>
 #include <asm/cacheflush.h>
 #include <asm/alternative.h>
+#include <asm/cfi.h>
 #include <asm/cpufeature.h>
 #include <asm/insn.h>
 #include <asm/module.h>
@@ -298,3 +300,26 @@ noinstr void alt_cb_patch_nops(struct alt_instr *alt, __le32 *origptr,
              updptr[i] = cpu_to_le32(aarch64_insn_gen_nop());
 }
 EXPORT_SYMBOL(alt_cb_patch_nops);
+
+#ifdef CONFIG_CFI_CLANG
+struct bpf_insn;
+
+/* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */
+extern unsigned int __bpf_prog_runX(const void *ctx,
+                                 const struct bpf_insn *insn);
+DEFINE_CFI_TYPE(cfi_bpf_hash, __bpf_prog_runX);
+
+/* Must match bpf_callback_t */
+extern u64 __bpf_callback_fn(u64, u64, u64, u64, u64);
+DEFINE_CFI_TYPE(cfi_bpf_subprog_hash, __bpf_callback_fn);
+
+u32 cfi_get_func_hash(void *func)
+{
+     u32 hash;
+
+     if (get_kernel_nofault(hash, func - cfi_get_offset()))
Quoted 8 lines

arch/arm64/kernel/alternative.c noinstr void alt_cb_patch_nops(struct alt_instr *alt, __le32 *origptr, · line 325

Quoted 5 lines
arch/arm64/kernel/alternative.c +5 -0
+             return 0;
+
+     return hash;
+}
+#endif /* CONFIG_CFI_CLANG */

I don't think this should be in alternative.c. It's probably better off either as a 'static inline' in the new cfi.h header.

Sure, I'll find a better place for this. RISC-V again seems to have the exact same function, so I think a __weak implementation in kernel/cfi.c would work here, allowing x86 to still conveniently override the function.

Quoted 47 lines
Quoted 47 lines
arch/arm64/net/bpf_jit_comp.c +10 -1
index 70d7c89d3ac9..3b3691e88dd5 100644
@@ -9,6 +9,7 @@
 
 #include <linux/bitfield.h>
 #include <linux/bpf.h>
+#include <linux/cfi.h>
 #include <linux/filter.h>
 #include <linux/memory.h>
 #include <linux/printk.h>
@@ -164,6 +165,12 @@ static inline void emit_bti(u32 insn, struct jit_ctx *ctx)
              emit(insn, ctx);
 }
 
+static inline void emit_kcfi(u32 hash, struct jit_ctx *ctx)
+{
+     if (IS_ENABLED(CONFIG_CFI_CLANG))
+             emit(hash, ctx);
+}
+
 /*
  * Kernel addresses in the vmalloc space use at most 48 bits, and the
  * remaining bits are guaranteed to be 0x1. So we can compose the address
@@ -474,7 +481,6 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
      const bool is_main_prog = !bpf_is_subprog(prog);
      const u8 fp = bpf2a64[BPF_REG_FP];
      const u8 arena_vm_base = bpf2a64[ARENA_VM_START];
-     const int idx0 = ctx->idx;
      int cur_offset;
 
      /*
@@ -500,6 +506,9 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
       *
       */
 
+     emit_kcfi(is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx);
+     const int idx0 = ctx->idx;
+
      /* bpf function may be invoked by 3 instruction types:
       * 1. bl, attached via freplace to bpf prog via short jump
       * 2. br, attached via freplace to bpf prog via long jump
@@ -2009,9 +2018,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
              jit_data->ro_header = ro_header;
      }
 
Quoted 7 lines

arch/arm64/net/bpf_jit_comp.c struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) · line 2023

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +2 -2
-     prog->bpf_func = (void *)ctx.ro_image;
+     prog->bpf_func = (void *)ctx.ro_image + cfi_get_offset();
      prog->jited = 1;
-     prog->jited_len = prog_size;
+     prog->jited_len = prog_size - cfi_get_offset();

Why do we add the offset even when CONFIG_CFI_CLANG is not enabled?

The function returns zero if CFI is not enabled, so I believe it's just to avoid extra if (IS_ENABLED(CONFIG_CFI_CLANG)) statements in the code. IMO this is cleaner, but I can certainly change this if you prefer.

Thanks for taking a look!

Sami

maintainer
#5 lore raw

Re: [PATCH bpf-next v9 2/2] arm64/cfi,bpf: Support kCFI + BPF on arm64

Hey Sami,

Quoted 18 lines · Sami Tolvanen
Quoted 9 lines
Quoted 9 lines
arch/arm64/include/asm/cfi.h new file +9 -0
+#define cfi_get_offset cfi_get_offset
+extern u32 cfi_bpf_hash;
+extern u32 cfi_bpf_subprog_hash;
+extern u32 cfi_get_func_hash(void *func);
+#else
+#define cfi_bpf_hash 0U
+#define cfi_bpf_subprog_hash 0U
+static inline u32 cfi_get_func_hash(void *func)
+{
Quoted 8 lines

arch/arm64/include/asm/cfi.h line 23

Quoted 4 lines
arch/arm64/include/asm/cfi.h new file +4 -0
+     return 0;
+}
+#endif /* CONFIG_CFI_CLANG */
+#endif /* _ASM_ARM64_CFI_H */

This looks like an awful lot of boiler plate to me. The only thing you seem to need is the CFI offset -- why isn't that just a constant that we can define (or a Kconfig symbol?).

Quoted 11 lines

The cfi_get_offset function was originally added in commit 4f9087f16651 ("x86/cfi,bpf: Fix BPF JIT call") because the offset can change on x86 depending on which CFI scheme is enabled at runtime. Starting with commit 2cd3e3772e41 ("x86/cfi,bpf: Fix bpf_struct_ops CFI") the function is also called by the generic BPF code, so we can't trivially replace it with a constant. However, since this defaults to 4 unless the architecture adds pre-function NOPs, I think we could simply move the default implementation to include/linux/cfi.h (and also drop the RISC-V version). Come to think of it, we could probably move most of this boilerplate to generic code. I'll refactor this and send a new version.

Excellent, thanks.

Quoted 12 lines
Quoted 11 lines

arch/arm64/net/bpf_jit_comp.c struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) · line 2023

Quoted 9 lines
arch/arm64/net/bpf_jit_comp.c +2 -2
@@ -2009,9 +2018,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
              jit_data->ro_header = ro_header;
      }
 
-     prog->bpf_func = (void *)ctx.ro_image;
+     prog->bpf_func = (void *)ctx.ro_image + cfi_get_offset();
      prog->jited = 1;
-     prog->jited_len = prog_size;
+     prog->jited_len = prog_size - cfi_get_offset();

Why do we add the offset even when CONFIG_CFI_CLANG is not enabled?

Quoted 4 lines

The function returns zero if CFI is not enabled, so I believe it's just to avoid extra if (IS_ENABLED(CONFIG_CFI_CLANG)) statements in the code. IMO this is cleaner, but I can certainly change this if you prefer.

Ah, that caught me out because the !CONFIG_CFI_CLANG stub is in the core code (and I'm extra susceptible to being caught out on a warm Friday evening!).

Hopefully if you're able to trim down the boilerplate then this will become more obvious too.

Cheers,

Will

#6 lore raw
In reply to Sami Tolvanen
Quoted 8 lines · Sami Tolvanen

Hi folks,

These patches add KCFI types to arm64 BPF JIT output. Puranjay and Maxwell have been working on this for some time now, but I haven't seen any progress since June 2024, so I decided to pick up the latest version[1] posted by Maxwell and fix the few remaining issues I noticed. I confirmed that with these patches applied, I no longer see CFI failures when running BPF self-tests on arm64.

Bump! Thank you Sami for following up on this, hopefully the maintainers will have time to take a look!

Regards,
Maxwell