bpf: Inline kfuncs that have a BPF body
Subsystems (8)
cover bpf: Inline kfuncs that have a BPF body
The BPF JIT translates one BPF instruction at a time. Operations that the CPU does in one instruction, such as a rotate, a conditional select or a big-endian load, take several BPF instructions, and they stay several after the JIT. On 27 small compute benchmarks, each compiled from the same C source to BPF and to native code, BPF ran 1.71x (x86-64) and 2.05x (arm64) slower than native code, while compiling the same BPF bytecode with LLVM came within 7% of native code [1]. The loss is in the JIT, not in the bytecode. Our talk in the eBPF track at LPC 2026 [5] covers this gap and the approach of this series.
A kfunc could do such an operation, but the call costs about as much as the operation, and the verifier knows nothing about the result. After off = bpf_select64(c, 4, 8), it does not know that off is 4 or 8, so a packet load at data + off needs another bounds check.
This series lets a kfunc have a body: a short sequence of BPF instructions that computes the same result, and optionally a callback that emits native code for a call. For example:
/* x << n | x >> (-n & 63) */
static const struct bpf_insn rol64_body[] = { ... };
static const struct bpf_kfunc_body bodies[] = {
{ &body_ids[0], rol64_body, ARRAY_SIZE(rol64_body), rol64_emit },
};Before its analysis, the verifier replaces each call of such a kfunc with its body, so it analyzes the result like any other BPF code. After the analysis, the call is restored, and the JIT puts native code in its place. The native code uses the registers that already hold the operands, so the argument and result moves around the call go away. If the JIT has no native code for the call, as on other architectures or with constant blinding, the verified body stays in the program instead of a call.
Programs call these kfuncs like any other kfunc; arguments named __k must be known constants. There is no new instruction encoding and no UAPI change. Because the verifier analyzes the body at every call, a program that uses these kfuncs costs as much to verify as the same operations written in BPF, which is more than an opaque call. JITs already inline some helper calls, and the verifier already inlines the numeric iterator kfuncs as BPF. This series combines the two, and in every case the body is what tells the verifier the result. Instructions that BPF lacks are the first use, but any small kfunc can have a body, including kfuncs in modules.
The native code comes with the kfunc, not from the JIT. By default the x86-64 JIT copies the compiled kfunc and renames its registers, as was suggested for the bitops kfuncs [2]. A copy keeps constant arguments in registers, cannot write the result into the register of an argument, and has only the baseline x86-64 instructions that the kernel is built for, so no movbe or bextr. A kfunc can therefore have an emit callback instead, which gets the registers that hold the operands and the values of the constant arguments. Five of the kfuncs in this series have one. The kfuncs and their x86-64 code are in kernel/bpf/insn_kfuncs/, which can be built as a module.
In a loop of dependent operations on x86-64 (Core Ultra 9 285K, KVM; numbers vary by about 5% between boots), plain is the body as the JIT translates it today, emit the code from the callback and copy the copied kfunc:
x86-64 code ns per operation operation emitted copied plain emit copy rotate by 13 rol $0xd,%rbx 5 insns 0.38 0.19 0.20 bit extract mov $0x110d,%eax; bextr 10 insns 0.57 0.57 0.57 big-endian load movbe (%rdi),%rax 4 insns 0.57 0.57 1.53 lea lea 0x10(%rbx,%rsi,8),%rbx 8 insns 0.41 0.37 0.41 select test; mov; cmovne same 3.29 1.22 1.21
The emitted code is much shorter, but a copy is about as fast, except for the big-endian load, which is 2.7x slower as a copy, and lea, which is about 10% slower. The select row uses an unpredictable condition; with a predictable one, the branch in plain BPF is 3x faster than cmov.
The paper's prototype also had the verifier check a BPF version of each operation and ran native code in its place [1]. It made the 27 benchmarks 1.24x (x86-64) and 1.22x (arm64) faster, and raised the throughput of the Cilium datapath on x86-64 by 7.4%. Where the native code pays off depends on the CPU and the workload: for Katran on arm64, 21 chosen call sites gave 7.3% more throughput, but all 62 gave 0.5% less.
- Patch 1 lets a kfunc set give its kfuncs a body and native code.
- Patch 2 is the verifier side, in kernel/bpf/kfunc_inline.c.
- Patch 3 makes the x86-64 JIT inline the native code of a kfunc, or else a copy of the compiled kfunc.
- Patch 4 adds kfuncs with bodies for rotate, select, bit field extract, big-endian load, prefetch, 16-byte copy and lea, under CONFIG_BPF_INSN_KFUNCS.
- Patch 5 adds x86-64 code for five of them.
- Patches 6 and 7 add selftests and documentation.
Testing on x86-64: verifier_kfunc_inline (24 subtests) and kfunc_body_registration pass with CONFIG_BPF_INSN_KFUNCS=y. With bpf_jit_harden=2 the bodies stay in the program, so the subtests that check the JIT output fail as expected, and all results are correct.
Questions:
1. Is it acceptable that the verifier replaces calls with their bodies before the analysis and restores them afterwards?
2. Should kfuncs bring native code through an emit callback, or should the JIT only copy compiled kfuncs? The table shows what copying alone costs.
3. Native code is trusted like the rest of the JIT, and the selftests compare it with the bodies on random inputs. Is that enough?
4. The bodies are written by hand. Should they rather be compiled from the kfunc's C source, as in the inlinable kfuncs RFC [3]?
Not done yet: arm64 has no native code and runs the bodies. Copying compiled kfuncs would be the first step there.
Earlier work:
- Leon Hwang's bitops kfuncs [2] emitted native code for kfuncs in the JIT. The review asked to keep the C version as a fallback and suggested copying the compiled kfunc. Here the fallback is the body, and the native code is kept with the kfuncs instead of in the JIT.
- Eduard Zingerman's inlinable kfuncs [3] and Puranjay Mohan's iterator inlining [4] inline kfuncs as BPF, so that the verifier sees what they compute. This series does the same and then lets the JIT put native code in place of the call.
[1] https://arxiv.org/abs/2606.24213
[2] https://lore.kernel.org/bpf/[email protected]/
[3] https://lore.kernel.org/bpf/[email protected]/
[4] https://lore.kernel.org/bpf/[email protected]/
[5] https://lpc.events/event/20/contributions/2445/
Yusheng Zheng (7):
bpf: Let kfunc sets give kfuncs a BPF body
bpf: Verify calls of kfuncs with a body through the body bpf, x86: Inline native code for kfuncs that have a body
bpf: Add kfuncs with bodies for common operations bpf, x86: Add native code for some inline kfuncs selftests/bpf: Test inline kfuncs Documentation/bpf: Describe inline kfuncs
Documentation/bpf/kfuncs.rst | 53 ++ arch/x86/net/bpf_jit_comp.c | 181 ++++++ include/linux/bpf_verifier.h | 34 ++ include/linux/btf.h | 27 + include/linux/filter.h | 2 + kernel/bpf/Kconfig | 1 + kernel/bpf/Makefile | 3 +- kernel/bpf/btf.c | 126 ++++ kernel/bpf/const_fold.c | 4 + kernel/bpf/core.c | 9 + kernel/bpf/insn_kfuncs/Kconfig | 16 + kernel/bpf/insn_kfuncs/Makefile | 5 + kernel/bpf/insn_kfuncs/bpf_insn_kfuncs.c | 184 ++++++ kernel/bpf/insn_kfuncs/x86/insn_kfuncs.h | 155 +++++ kernel/bpf/kfunc_inline.c | 273 +++++++++ kernel/bpf/verifier.c | 53 +- tools/testing/selftests/bpf/Makefile | 2 +- tools/testing/selftests/bpf/config | 1 + .../bpf/prog_tests/kfunc_body_registration.c | 18 + .../selftests/bpf/prog_tests/verifier.c | 2 + .../bpf/progs/verifier_kfunc_inline.c | 543 ++++++++++++++++++ .../testing/selftests/bpf/test_kmods/Makefile | 2 +- .../bpf/test_kmods/bpf_test_kfunc_body.c | 121 ++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 77 +++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 4 + 25 files changed, 1877 insertions(+), 19 deletions(-) create mode 100644 kernel/bpf/insn_kfuncs/Kconfig create mode 100644 kernel/bpf/insn_kfuncs/Makefile create mode 100644 kernel/bpf/insn_kfuncs/bpf_insn_kfuncs.c create mode 100644 kernel/bpf/insn_kfuncs/x86/insn_kfuncs.h create mode 100644 kernel/bpf/kfunc_inline.c create mode 100644 tools/testing/selftests/bpf/prog_tests/kfunc_body_registration.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_kfunc_inline.c create mode 100644 tools/testing/selftests/bpf/test_kmods/bpf_test_kfunc_body.c
base-commit: 99dc1ba542420db6b8df209744f55cc52466ad91
Signature · 1 line
2.51.1
RFC PATCH 1/7 bpf: Let kfunc sets give kfuncs a BPF body
Let a kfunc set give some of its kfuncs a body: a few BPF instructions that compute the kfunc from its arguments in R1-R5 into R0. The following patches make the verifier check each call of such a kfunc as its body and the JIT inline native code for the call, so that small kfuncs such as a rotate can be used like instructions while the verifier knows what they compute.
A kfunc set lists the bodies in .bodies and .body_cnt, next to its BTF ID set. A body may also have an emit callback that writes native code for a call, so that the native code of a kfunc comes with the kfunc and not from the JIT; a following patch makes the x86-64 JIT use it. Registration checks that each kfunc with a body is in the set and has no kfunc flags, that it takes each argument and returns its value in one register, with constant (__k) arguments in 32 bits, and that the body uses only R0-R5, ALU instructions, loads, stores and forward jumps that land within it, leaving out the cpu v4 instructions that not every JIT has. btf_find_kfunc_body() finds the body of a kfunc.
--- include/linux/btf.h | 27 ++++++++++ kernel/bpf/btf.c | 126 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 153 insertions(+)
include/linux/btf.h +27 -0
@@ -120,10 +120,36 @@ struct bpf_prog;typedef int (*btf_kfunc_filter_t )(const struct bpf_prog *prog,u32 kfunc_id); +#define BPF_KFUNC_BODY_MAX_INSNS 32 +#define BPF_KFUNC_INLINE_MAX 128 + +/* +* The body of a kfunc: len BPF instructions that compute the kfunc from its +* arguments in R1-R5 into R0. The verifier analyzes each call of the kfunc as +* the body, and the body runs in place of the call unless the JIT has native +* code for it, see kernel/bpf/kfunc_inline.c. +* +* emit, if set, writes native code for a call to buf, at most +* BPF_KFUNC_INLINE_MAX bytes, and returns its length, or an error if it has +* no code, for example because the CPU lacks a feature; the JIT then copies +* the compiled kfunc. reg[i] is the native register that the verifier bound +* Ri to, for R0-R5, and those of R1-R5 that are not arguments are free to +* use. imm[i] is the value of Ri if it is a constant (__k) argument. Like the +* rest of the JIT, native code is trusted to compute what the body computes. +*/ +struct bpf_kfunc_body { +const u32 *id; +const struct bpf_insn *insns; +u32 len; +int (*emit)(const u8 *reg,const s32 *imm,u8 *buf); +}; +struct btf_kfunc_id_set {struct module *owner;struct btf_id_set8 *set;btf_kfunc_filter_t filter; +const struct bpf_kfunc_body *bodies; +u32 body_cnt; };struct btf_id_dtor_kfunc {
@@ -604,6 +630,7 @@ const char *btf_str_by_offset(const struct btf *btf, u32 offset);struct btf *btf_parse_vmlinux (void );struct btf *bpf_prog_get_target_btf (const struct bpf_prog *prog);u32 *btf_kfunc_flags (const struct btf *btf,u32 kfunc_btf_id,const struct bpf_prog *prog); +const struct bpf_kfunc_body *btf_find_kfunc_body (const struct btf *btf,u32 kfunc_btf_id);int btf_kfunc_check_flag (const struct btf *btf,u32 kfunc_btf_id,u32 flag);bool btf_kfunc_is_allowed (const struct btf *btf,u32 kfunc_btf_id,const struct bpf_prog *prog); u32 *btf_kfunc_is_modify_return(const struct btf *btf, u32 kfunc_btf_id,
kernel/bpf/btf.c +126 -0
156 lines not on this page · raw message
Signature · 1 line
2.51.1
[email protected]
AI reviewRe: [RFC PATCH bpf-next 1/7] bpf: Let kfunc sets give kfuncs a BPF body
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] bpf: integer conversion bypasses kfunc body length check - [Medium] bpf: type confusion when…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] bpf: integer conversion bypasses kfunc body length check
- [Medium] bpf: type confusion when checking size of pointer arguments
--
commit ab41a09d317c7611ac2e010f734c868245e66554
Author: Yusheng Zheng <[email protected]>
bpf: Let kfunc sets give kfuncs a BPF body
This commit allows a kfunc set to provide a BPF instruction body for its kfuncs, enabling the verifier to analyze the instructions inline. These small kfuncs can then be used like instructions while the verifier knows exactly what they compute.
Quoted 3 lines
kernel/bpf/btf.c +0
-0
[ ... ]
kernel/bpf/btf.c u32 *btf_kfunc_is_modify_return(const struct btf *btf, u32 kfunc_btf_id, · line 9746
Quoted 12 lines
kernel/bpf/btf.c +12
-0+static bool btf_check_kfunc_body (const struct btf *btf,const struct btf_type *func, +const struct bpf_kfunc_body *b) +{ +const struct btf_type *proto =btf_type_by_id (btf, func->type); +const struct btf_param *args =btf_params (proto); +int i, n =btf_type_vlen (proto), len = b->len; +const struct bpf_insn *insn; +u8 op; + +if (n >MAX_BPF_FUNC_REG_ARGS || !b->insns || !len || len >BPF_KFUNC_BODY_MAX_INSNS || + (proto->type && !btf_kfunc_reg_type (btf, proto->type))) +return false;
[Severity: Medium]
Does this code inadvertently bypass the bounds check and instruction validation loop if b->len is very large?
b->len is an unsigned 32-bit integer. Assigning it to int len implicitly casts it to a signed integer. If a module registers a kfunc with b->len >= 0x80000000, len becomes negative. The check
len > BPF_KFUNC_BODY_MAX_INSNS evaluates as false, and the subsequent for (i = 0; i < len; i++) loop is skipped entirely because
0 < negative_value is false. This could allow invalid or excessively large BPF instruction sequences to be registered.
kernel/bpf/btf.c u32 *btf_kfunc_is_modify_return(const struct btf *btf, u32 kfunc_btf_id, · line 9751
Quoted 5 lines
kernel/bpf/btf.c +5
-0+for (i =0 ; i < n; i++) +if (!btf_kfunc_reg_type (btf, args[i].type) || + (btf_param_match_suffix (btf, &args[i],"__k" ) && +btf_type_skip_modifiers (btf, args[i].type,NULL )->size >sizeof (s32))) +return false;
[Severity: Medium]
Does this code check the correct union field for pointer arguments?
When evaluating pointer arguments ending in "__k", this accesses ->size on the returned btf_type. However, for BTF_KIND_PTR, the size field shares a union with type (the pointee BTF ID). This evaluates the arbitrary BTF ID of the pointee type instead of its byte size, causing unpredictable validation results or erroneously rejecting arguments based on their pointee BTF type ID.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
[email protected]
AI reviewRe: [RFC PATCH bpf-next 1/7] bpf: Let kfunc sets give kfuncs a BPF body
In reply to Yusheng ZhengTL;DR: Kfunc bodies are added before btf_populate_kfunc_set(), so if populate fails the set's bodies may stay registered. A later registration may then be skipped and the verifier may use the stale…
Quoted 1 line
bpf: Let kfunc sets give kfuncs a BPF body
TL;DR: Kfunc bodies are added before btf_populate_kfunc_set(), so if populate fails the set's bodies may stay registered. A later registration may then be skipped and the verifier may use the stale body. No memory-safety impact seen.
Quoted 4 lines
kernel/bpf/btf.c +0
-0
[ ... ]
Quoted 1 line
kernel/bpf/btf.c +0
-0@@ -9695,6 +9705,118 @@ u32 *btf_kfunc_is_modify_return(const struct btf *btf, u32 kfunc_btf_id,
[ ... ]
Quoted 2 lines
kernel/bpf/btf.c +2
-0+static int btf_add_kfunc_bodies (struct btf *btf,const struct btf_kfunc_id_set *kset) +{
[ ... ]
Quoted 20 lines
kernel/bpf/btf.c +16
-0+/* the body stands for the call, so no kfunc flags apply */ +if (!pair || pair[1 ] || !t || !btf_type_is_func (t) || + !btf_check_kfunc_body (btf, t, b)) { +/* a set that fails to register leaves no bodies */ + tab->cnt = cnt; +return -EINVAL ; + } +/* a set registered for several hooks adds its bodies once */ +if (!btf_find_kfunc_body (btf, id)) { + tab->bodies[tab->cnt].id = id; + tab->bodies[tab->cnt++].body = b; + } + } +return 0 ; +} +static int __register_btf_kfunc_id_set (enum btf_kfunc_hook hook,const struct btf_kfunc_id_set *kset) {
kernel/bpf/btf.c static int __register_btf_kfunc_id_set(enum btf_kfunc_hook hook, · line 9845
Quoted 10 lines
kernel/bpf/btf.c +4
-0@@ -9714,6 +9836,10 @@ static int __register_btf_kfunc_id_set(enum btf_kfunc_hook hook, goto err_out; } + ret =btf_add_kfunc_bodies (btf, kset); +if (ret) + goto err_out; + ret =btf_populate_kfunc_set (btf, hook, kset); err_out:
btf_add_kfunc_bodies() rolls back its own additions when a body fails validation ("a set that fails to register leaves no bodies"), but __register_btf_kfunc_id_set() calls it before btf_populate_kfunc_set(), and the populate step can still fail after the bodies have been committed.
For example, btf_populate_kfunc_set() can return an error for a hook that is not below BTF_KFUNC_HOOK_MAX (register_btf_kfunc_id_set() with a prog type that bpf_prog_type_to_kfunc_hook() does not map), for -E2BIG from the filter or BTF_KFUNC_SET_MAX_CNT limits, for -ENOMEM from krealloc, or from the WARN_ON_ONCE for a module set registered twice for one hook.
On those paths err_out only does btf_put(), so btf->kfunc_body_tab->cnt is not restored and the set fails to register while its bodies stay in the table. The populate error path also frees the whole kfunc_set_tab via btf_free_kfunc_set_tab(), but kfunc_body_tab is left in place, so the two tables disagree.
Can this leave a stale body behind? If the same kfunc id is registered again later (another set, or a module init that ignores the error and keeps registering), the dedup check
if (!btf_find_kfunc_body(btf, id)) {finds the stale entry from the failed set and silently skips the new set's body. bpf_add_kfunc_call() in 47ef3ddba448 ("bpf: Verify calls of kfuncs with a body through the body") then uses the stale body.
There is no memory-safety impact that I can see, since for module BTF the stale body pointers can only be reached through btf_try_get_module(), which fails once the module is gone. But the behaviour contradicts the comment.
Would it make sense to save the old count and restore it when btf_populate_kfunc_set() fails, or to add the bodies only after populate succeeds? No later commit in the series touches kernel/bpf/btf.c.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/37326585321
RFC PATCH 2/7 bpf: Verify calls of kfuncs with a body through the body
In reply to Yusheng ZhengBefore the CFG check, replace each call of a kfunc that has a body with the body, using bpf_patch_insn_data() like the other inlining in the verifier, so that the verifier analyzes the result in the caller's context. The entry and exit of the body have the register effects of the call: only the arguments are readable at the entry, and R1-R5 are not readable after the exit. Constant folding takes R0-R5 as unknown after each instruction of the body. Constant (__k) arguments must be known at the entry; they are marked precise, and their values are kept for native code. A kfunc that the program may not call keeps its call, which check_kfunc_call() then rejects as before.
After verification, restore a call if the JIT has native code for it, from bpf_jit_inline_kfunc(), and the verifier did not rewrite the body
later: no constant blinding, speculation barriers, sanitation or arena conversion, and memory accesses only to the stack, map values, memory and packets, and only to the stack when the JIT adds KASAN checks. When liveness allows, the moves of arguments from R6-R9 right before the call and the move of the result to R6-R9 right after it are removed, and the native code uses those registers directly. The removed instructions become nops for bpf_opt_remove_nops(). Otherwise the body stays, so the program runs on every JIT.
The new code is in kernel/bpf/kfunc_inline.c; verifier.c only calls it.
--- include/linux/bpf_verifier.h | 34 +++++ include/linux/filter.h | 2 + kernel/bpf/Makefile | 2 +- kernel/bpf/const_fold.c | 4 + kernel/bpf/core.c | 9 ++ kernel/bpf/kfunc_inline.c | 273 +++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 53 +++++-- 7 files changed, 360 insertions(+), 17 deletions(-) create mode 100644 kernel/bpf/kfunc_inline.c
include/linux/bpf_verifier.h +34 -0
@@ -634,6 +634,7 @@ struct bpf_insn_aux_data {enum bpf_reg_type ptr_type;/* pointer type for load/store insns */ struct bpf_map_ptr_state map_ptr_state;s32 call_imm;/* saved imm field of call insn */ +u32 kfunc_inline;/* 1 + index into env->kfunc_inlines, at a call */ u32 alu_limit;/* limit for add/sub register with pointer */ struct {u32 map_index;/* index into used_maps[] */
@@ -683,6 +684,9 @@ struct bpf_insn_aux_data {*/ u8 fastcall_spills_num:3; u8 arg_prog:4; +/* insn belongs to the body of a kfunc call, see bpf_inline_kfunc_bodies() */ + u8 kfunc_body:1; + u8 kfunc_body_entry:1;/* below fields are initialized once */ unsigned int orig_idx;/* original instruction index */
@@ -1083,6 +1087,8 @@ struct bpf_verifier_env {u32 scc_cnt;struct bpf_iarray *succ;struct bpf_iarray *gotox_tmp_buf; +struct bpf_kfunc_inline *kfunc_inlines; +u32 kfunc_inline_cnt; };static inline struct bpf_func_info_aux *subprog_aux (struct bpf_verifier_env *env,int subprog)
@@ -1794,14 +1800,42 @@ enum bpf_reg_arg_type {#define MAX_KFUNC_CALL_DESCS (MAX_KFUNC_DESCS * 2)static_assert (MAX_KFUNC_CALL_DESCS <=S16_MAX +1 ); +/* A call of a kfunc with a body, which the verifier replaced by the body */ +struct bpf_kfunc_inline { +struct bpf_insn call; +const struct bpf_kfunc_body *body; +unsigned long addr;/* of the compiled kfunc */ +u8 *image;/* native code for the JIT */ +u32 start; +/* the BPF registers that R0-R5 are bound to, and the constant arguments */ +u8 reg[MAX_BPF_FUNC_REG_ARGS +1 ]; +s32 imm[MAX_BPF_FUNC_REG_ARGS +1 ]; +u8 image_len; +u8 nargs; +u8 imm_mask;/* R1-R5 that hold constant (__k) arguments */ +bool entered;/* the verifier reached the body */ +bool ret;/* the kfunc returns a value */ +bool copy;/* the native code is a copy of the compiled kfunc */ +}; +struct bpf_kfunc_desc {struct btf_func_model func_model;struct bpf_func_proto proto; +const struct bpf_kfunc_body *body;u32 func_id;u16 offset; +u8 body_imm;/* R1-R5 that are constant (__k) arguments of the body */ unsigned long addr; }; +struct bpf_kfunc_desc *bpf_find_kfunc_desc (const struct bpf_prog *prog,u32 func_id,u16 offset); +int bpf_inline_kfunc_bodies (struct bpf_verifier_env *env); +int bpf_mark_kfunc_body_regs (struct bpf_verifier_env *env,int prev_insn_idx, +const struct bpf_insn_aux_data *aux); +void bpf_restore_kfunc_calls (struct bpf_verifier_env *env); +void bpf_free_kfunc_inlines (struct bpf_verifier_env *env); +const struct bpf_kfunc_inline *bpf_kfunc_native (const struct bpf_verifier_env *env,int idx); +struct bpf_kfunc_desc_tab {u32 nr_descs;u32 nr_base_descs;
include/linux/filter.h +2 -0
@@ -1239,6 +1239,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prvoid bpf_jit_compile (struct bpf_prog *prog);bool bpf_jit_needs_zext (void );bool bpf_jit_inlines_helper_call (s32 imm); +struct bpf_kfunc_inline ; +int bpf_jit_inline_kfunc (const struct bpf_kfunc_inline *in,u8 *buf);bool bpf_jit_supports_subprog_tailcalls (void );bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void );
kernel/bpf/Makefile +1 -1
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_ obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o -obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o +obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o kfunc_inline.o obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o obj-$(CONFIG_BPF_JIT) += trampoline.o
kernel/bpf/const_fold.c +4 -0
@@ -268,6 +268,10 @@ int bpf_compute_const_regs(struct bpf_verifier_env *env)memcpy (ci_out, ci,sizeof (ci_out));const_reg_xfer (env, ci_out, insn, insns, idx); +/* the body of a kfunc call leaves R0-R5 unknown, like the call */ +if (insn_aux[idx].kfunc_body) +for (r =BPF_REG_0 ; r <=BPF_REG_5 ; r++) + ci_out[r] = unknown; succ =bpf_insn_successors (env, idx);for (int s =0 ; s < succ->cnt; s++)
kernel/bpf/core.c +9 -0
@@ -3299,6 +3299,15 @@ bool __weak bpf_jit_inlines_helper_call(s32 imm)return false; } +/* Write native code for an inlined kfunc call, see struct bpf_kfunc_inline, +* to @buf for the JIT to copy. Return its length, or an error to keep the +* body of the kfunc. +*/ +int __weak bpf_jit_inline_kfunc (const struct bpf_kfunc_inline *in,u8 *buf) +{ +return -EOPNOTSUPP ; +} +/* Return TRUE if the JIT backend supports mixing bpf2bpf and tailcalls. */ bool __weak bpf_jit_supports_subprog_tailcalls (void ) {
kernel/bpf/kfunc_inline.c new file +273 -0
273 lines not on this page · raw message
kernel/bpf/verifier.c +37 -16
128 lines not on this page · raw message
Signature · 1 line
2.51.1
[email protected]
AI reviewRe: [RFC PATCH bpf-next 2/7] bpf: Verify calls of kfuncs with a body through the body
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Critical] bpf: uninitialized data leak and state mismatch for void kfuncs - [Critical] bpf: argument…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] bpf: uninitialized data leak and state mismatch for void kfuncs
- [Critical] bpf: argument checking bypass for nested kfunc calls
- [High] bpf: value truncation of 64-bit constant arguments in kfunc_bind --
commit 0eafd0ff2fd1aa892e8a507b878aae0d4eedd33b
Author: Yusheng Zheng <[email protected]>
bpf: Verify calls of kfuncs with a body through the body
This commit allows the BPF verifier to inline the bodies of certain kfuncs for context-aware analysis. It updates the CFG to replace kfunc calls with their respective bodies before verification, tracks constant arguments, and attempts to natively execute these inlined kfuncs if supported by the JIT.
Quoted 3 lines
include/linux/bpf_verifier.h +0
-0
[ ... ]
include/linux/bpf_verifier.h enum bpf_reg_arg_type { · line 1812
Quoted 10 lines
include/linux/bpf_verifier.h +10
-0+/* A call of a kfunc with a body, which the verifier replaced by the body */ +struct bpf_kfunc_inline { +struct bpf_insn call; +const struct bpf_kfunc_body *body; +unsigned long addr;/* of the compiled kfunc */ +u8 *image;/* native code for the JIT */ +u32 start; +/* the BPF registers that R0-R5 are bound to, and the constant arguments */ +u8 reg[MAX_BPF_FUNC_REG_ARGS +1 ]; +s32 imm[MAX_BPF_FUNC_REG_ARGS +1 ];
[Severity: High]
Is it intentional to use s32 for the constant arguments here? Since BPF registers hold 64-bit values, could this lead to truncating 64-bit constants if a kfunc expects a u64 or s64?
[ ... ]
Quoted 3 lines
kernel/bpf/kfunc_inline.c +0
-0
[ ... ]
Quoted 2 lines
kernel/bpf/kfunc_inline.c +2
-0+int bpf_inline_kfunc_bodies (struct bpf_verifier_env *env) +{
[ ... ]
kernel/bpf/kfunc_inline.c line 62
Quoted 8 lines
kernel/bpf/kfunc_inline.c +8
-0+ prog =bpf_patch_insn_data (env, i, r->body->insns, r->body->len); +if (!prog) +return -ENOMEM ; + env->prog = prog; +for (j = i; j < i + r->body->len; j++) + env->insn_aux_data[j].kfunc_body =1 ; + env->insn_aux_data[i].kfunc_body_entry =1 ; + i += r->body->len -1 ;
[Severity: Critical]
Since bpf_inline_kfunc_bodies() skips over the newly inlined instructions, it appears nested kfunc calls inside the inlined body are not inlined themselves and remain as standard BPF call instructions.
Combined with the skipped prototype initialization in bpf_add_kfunc_call(), could this allow a user to pass arbitrary unverified arguments to the nested kfunc if do_check_main() later evaluates it using a zeroed descriptor?
Quoted 11 lines
kernel/bpf/kfunc_inline.c +11
-0+ } +return 0 ; +} + +/* +* The body of a kfunc gets only the arguments of the call and leaves R1-R5 +* like it. The constant (__k) arguments must be known, and native code may +* use their values. +*/ +int bpf_mark_kfunc_body_regs (struct bpf_verifier_env *env,int prev_insn_idx, +const struct bpf_insn_aux_data *aux)
kernel/bpf/kfunc_inline.c line 83
Quoted 10 lines
kernel/bpf/kfunc_inline.c +10
-0+{ +struct bpf_kfunc_inline *r = env->kfunc_inlines; +struct bpf_reg_state *regs =cur_regs (env); +u32 clobber =0 ; +int i, err; + +/* leaving the body */ +if (prev_insn_idx >=0 && env->insn_aux_data[prev_insn_idx].kfunc_body && + (!aux->kfunc_body || aux->kfunc_body_entry)) + clobber |=GENMASK (BPF_REG_5 ,BPF_REG_1 );
[Severity: Critical]
Should this also clobber R0 when leaving a void kfunc?
If the kfunc does not return a value, R0 appears to be left uninitialized. If the JIT falls back to the compiled C function, it might leave R0 with uninitialized garbage or a user-controlled value. Could the BPF program then dereference R0, leading to arbitrary memory reads or privilege escalation?
Quoted 6 lines
kernel/bpf/kfunc_inline.c +6
-0+if (aux->kfunc_body_entry) { +while (r->start != env->insn_idx) + r++; +for (i =BPF_REG_1 ; i <=BPF_REG_5 && !env->cur_state->speculative; i++) { +if (!(r->imm_mask &BIT (i))) +continue ;
[ ... ]
kernel/bpf/kfunc_inline.c line 100
Quoted 7 lines
kernel/bpf/kfunc_inline.c +7
-0+ err =mark_chain_precision (env, i); +if (err) +return err; +/* emitted code needs the same constants on every path */ +if (r->entered && r->imm[i] != (s32 )regs[i].var_off.value) + r->copy = true; + r->imm[i] = regs[i].var_off.value;
[Severity: High]
Will this safely handle 64-bit constants?
If multiple paths call the same kfunc with 64-bit constants that share the same lower 32 bits, the verifier might incorrectly assume they are identical due to the s32 cast. Could this cause the JIT to emit native code with the wrong constant?
Quoted 5 lines
kernel/bpf/kfunc_inline.c +5
-0+ } + r->entered = true; + clobber |=BIT (BPF_REG_0 ) | (GENMASK (BPF_REG_5 ,BPF_REG_0 ) & + ~GENMASK (r->nargs,BPF_REG_0 )); + }
[ ... ]
Quoted 3 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
kernel/bpf/verifier.c int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset) · line 2997
Quoted 9 lines
kernel/bpf/verifier.c +4
-1@@ -2990,7 +2991,10 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset) desc = &tab->descs[tab->nr_descs];memset (desc,0 ,sizeof (*desc)); - err =gen_kfunc_arg_proto (env, &meta, &func_model, &desc->proto); +/* the body of a kfunc that the program may call checks its arguments */ + body = kfunc.flags &&btf_kfunc_is_allowed (kfunc.btf, func_id, env->prog) ? +btf_find_kfunc_body (kfunc.btf, func_id) :NULL ; + err = body ?0 :gen_kfunc_arg_proto (env, &meta, &func_model, &desc->proto);
[Severity: Critical]
If gen_kfunc_arg_proto() is skipped for kfuncs with bodies, desc->proto remains zero-initialized. Could this bypass argument validation for nested kfunc calls that do not get inlined?
Quoted 3 lines
kernel/bpf/verifier.c +0
-0if (err)return err;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
RFC PATCH 3/7 bpf, x86: Inline native code for kfuncs that have a body
In reply to Yusheng ZhengImplement bpf_jit_inline_kfunc() for x86-64. The native code for a call comes from the emit callback of the kfunc's body, which gets the x86 registers that the operands are bound to and the values of the constant arguments. Without a callback, when it returns an error, or when the constant arguments differ between paths, the JIT copies the code that the compiler produced for the kfunc, as was suggested for the bitops kfuncs [1], with the registers of R0-R5 renamed. Only straight-line moves, ALU instructions, cmovcc, bswap, prefetch and lea up to the return are copied: no control flow, rip-relative addressing or division, no registers but those of R0-R5, r10 and r11, and no implicit operand, such as %rax in "and $imm, %eax", in a renamed register. Otherwise the body stays. Copying needs the instruction decoder.
The JIT has no code for any particular kfunc.
[1] https://lore.kernel.org/bpf/CAADnVQLNmQGKf5S5ZNwHYzScYBhnWFmnzLg=5Xxy4SgYKE3EfQ@mail.gmail.com/
--- arch/x86/net/bpf_jit_comp.c | 181 ++++++++++++++++++++++++++++++++++++ 1 file changed, 181 insertions(+)
arch/x86/net/bpf_jit_comp.c +181 -0
@@ -15,8 +15,12 @@#include <linux/memory.h> #include <linux/sort.h> #include <linux/execmem.h> +#include <linux/kallsyms.h> +#include <linux/uaccess.h> #include <asm/extable.h> #include <asm/ftrace.h> +#include <asm/insn.h> +#include <asm/insn-eval.h> #include <asm/set_memory.h> #include <asm/nospec-branch.h> #include <asm/text-patching.h>
@@ -2000,6 +2004,174 @@ static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,return prog - start; } +/* +* Registers that copied kfunc code may use: those of R0-R5, which hold the +* arguments and the result as for a call, and the scratch registers r10 and +* r11. r9 can hold the private frame pointer. +*/ +#define KFUNC_COPY_REGS (BIT(0) | BIT(1) | BIT(2) | BIT(6) | BIT(7) | BIT(8) | \ + BIT(10) | BIT(11)) + +/* +* Write an instruction of a compiled kfunc with the registers of R0-R5 +* renamed by @map to those that the verifier bound them to. It must be a +* move, ALU or address computation without control flow, prefixes other than +* operand size, rip-relative addressing or registers other than +* KFUNC_COPY_REGS, and no division, which can trap. +*/ +static u8 *kfunc_copy_insn (u8 *p,const struct insn *insn,const u8 *c,const u8 *map) +{ +u8 op = insn->opcode.bytes[insn->opcode.nbytes -1 ], rex = insn->rex_prefix.bytes[0 ]; +u8 modrm = insn->modrm.value, sib = insn->sib.value, ext =X86_MODRM_REG (modrm); +u8 mod =X86_MODRM_MOD (modrm), reg = ext, rm =0 , idx =4 ; +bool two = insn->opcode.nbytes ==2 , group, opreg, disp8, implicit; +u16 regs =0 ; +int head; + +if (insn->vex_prefix.nbytes || insn->opcode.nbytes >2 || insn->prefixes.nbytes >1 || + (insn->prefixes.nbytes && insn->prefixes.bytes[0 ] !=0x66 )) +return NULL ; +if (two) { +/* cmovcc, imul, movzx/movsx of words, bswap, prefetch */ + group = op ==0x18 ; +if ((op &0xf0 ) !=0x40 && op !=0xaf && op !=0xb7 && op !=0xbf && + (op &0xf8 ) !=0xc8 && !(group && ext <4 )) +return NULL ; + }else { +/* add, or, and, sub, xor, cmp, mov, lea, test, imul, shifts, not, neg, mul */ + group = op ==0x81 || op ==0x83 || op ==0xc1 || op ==0xc7 || + op ==0xd1 || op ==0xd3 || op ==0xf7 ; +if (!(op <0x40 && (op &0xf0 ) !=0x10 && (op &7 ) %2 && (op &7 ) <6 ) && + op !=0x63 && op !=0x69 && op !=0x6b && op !=0x85 && op !=0x89 && + op !=0x8b && op !=0x8d && op !=0x98 && op !=0x99 && op !=0xa9 && + (op &0xf8 ) !=0xb8 && + !(group && op !=0xc7 && op !=0xf7 && ext !=2 && ext !=3 ) && + !(op ==0xc7 && !ext) && !(op ==0xf7 && ext !=1 && ext <6 )) +return NULL ; + } + +/* uses %rax, %rcx or %rdx without naming it, as in "and $imm, %eax" */ + implicit = !two && ((op <0x40 && (op &7 ) ==5 ) || op ==0x98 || op ==0x99 || + op ==0xa9 || op ==0xd3 || (op ==0xf7 && (ext ==4 || ext ==5 ))); + + opreg = (op &0xf8 ) == (two ?0xc8 :0xb8 ); +if (opreg) + rm = (op &7 ) + (X86_REX_B (rex) ?8 :0 ); +if (insn->modrm.nbytes) { + reg = group ? ext : ext + (X86_REX_R (rex) ?8 :0 ); + rm = (insn->sib.nbytes ?X86_SIB_BASE (sib) :X86_MODRM_RM (modrm)) + + (X86_REX_B (rex) ?8 :0 ); +if (insn->sib.nbytes) + idx =X86_SIB_INDEX (sib) + (X86_REX_X (rex) ?8 :0 ); +/* rip-relative or absolute */ +if (!mod && (rm &7 ) ==5 ) +return NULL ; + regs = (group ?0 :BIT (reg)) | (idx !=4 ?BIT (idx) :0 ); + } +if ((regs | (opreg || insn->modrm.nbytes ?BIT (rm) :0 )) & ~KFUNC_COPY_REGS ) +return NULL ; +/* which cannot be renamed, so they must hold their operands as for a call */ +if (implicit && (map[0 ] !=0 || map[1 ] !=1 || map[2 ] !=2 )) +return NULL ; + +/* the instruction again, with the registers renamed */ +if (!group) + reg = map[reg]; + rm = map[rm]; + idx = idx !=4 ? map[idx] :4 ; +if (insn->prefixes.nbytes) + *p++ =0x66 ; + rex =0x40 |X86_REX_W (rex) | (reg &8 ?4 :0 ) | (idx &8 ?2 :0 ) | (rm &8 ?1 :0 ); +if (rex !=0x40 ) + *p++ = rex; +if (two) + *p++ =0x0f ; + *p++ = opreg ? (op &0xf8 ) | (rm &7 ) : op; +if (insn->modrm.nbytes) { +/* a base of r13 needs a displacement */ + disp8 = !mod && (rm &7 ) ==5 ; + *p++ = (disp8 ?1 : mod) <<6 | (reg &7 ) <<3 | (insn->sib.nbytes ?4 : rm &7 ); +if (insn->sib.nbytes) + *p++ = (sib &0xc0 ) | (idx &7 ) <<3 | (rm &7 ); +if (disp8) + *p++ =0 ; + } + head = insn->prefixes.nbytes + insn->rex_prefix.nbytes + insn->opcode.nbytes + + insn->modrm.nbytes + insn->sib.nbytes; +memcpy (p, c + head, insn->length - head); +return p + insn->length - head; +} + +/* +* Copy the compiled kfunc of an inlined call up to its return, without the +* ENDBR and NOPs at its entry, with the operands renamed by @map. +*/ +static int kfunc_copy (const struct bpf_kfunc_inline *in,const u8 *map,u8 *buf) +{ +u8 code[BPF_KFUNC_INLINE_MAX +MAX_INSN_SIZE ], *c, *p = buf; +unsigned long size, off, ret; +struct insn insn; +int pos; + +if (!kallsyms_lookup_size_offset (in->addr, &size, &off) || off) +return -EINVAL ; + size =min (size,sizeof (code)); +if (copy_from_kernel_nofault (code, (void *)in->addr, size)) +return -EFAULT ; +for (pos =0 ; pos < size; pos += insn.length) { +if (insn_decode (&insn, code + pos, size - pos,INSN_MODE_64 )) +return -EINVAL ; + c = code + pos; +/* ret, or a jump to the return thunk */ + ret = in->addr + pos + insn.length + insn.immediate.value; +if ((insn.length ==1 && c[0 ] ==0xc3 ) || + (c[0 ] ==0xe9 && (ret == (unsigned long )x86_return_thunk || + ret == (unsigned long )__x86_return_thunk))) +return p > buf ? p - buf : -EINVAL ; +if ((insn.length ==4 &&is_endbr ((u32 *)c)) ||insn_is_nop (&insn)) +continue ; +if (insn_is_rex2 (&insn)) +return -EINVAL ; +/* renaming adds at most a REX prefix and a displacement */ +if (p - buf + insn.length +2 >BPF_KFUNC_INLINE_MAX ) +return -E2BIG ; + p =kfunc_copy_insn (p, &insn, c, map); +if (!p) +return -EINVAL ; + } +return -EINVAL ; +} + +static u8 x86_reg (u32 reg) +{ +return reg2hex[reg] + (is_ereg (reg) ?8 :0 ); +} + +/* +* Get native code for an inlined kfunc call, with the operands in the x86 +* registers that the verifier bound them to: the code from the emit callback +* of the kfunc's body, or else a copy of the compiled kfunc. +*/ +int bpf_jit_inline_kfunc (const struct bpf_kfunc_inline *in,u8 *buf) +{ +u8 reg[MAX_BPF_FUNC_REG_ARGS +1 ], map[16 ]; +int i, len; + +for (i =0 ; i <ARRAY_SIZE (map); i++) + map[i] = i; +for (i =BPF_REG_0 ; i <=BPF_REG_5 ; i++) { + reg[i] =x86_reg (in->reg[i]); + map[x86_reg (i)] = reg[i]; + } +/* copying needs the instruction decoder */ +if (in->copy &&IS_ENABLED (CONFIG_INSTRUCTION_DECODER )) +return kfunc_copy (in, map, buf); +if (in->copy || !in->body->emit) +return -EOPNOTSUPP ; + len = in->body->emit (reg, in->imm, buf); +return len >0 && len <=BPF_KFUNC_INLINE_MAX ? len : -EINVAL ; +} +static int do_jit (struct bpf_verifier_env *env,struct bpf_prog *bpf_prog,int *addrs,u8 *image,u8 *rw_image,int oldproglen,struct jit_context *ctx,bool jmp_padding) {
@@ -2952,6 +3124,15 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *if (!imm32)return -EINVAL ;if (src_reg ==BPF_PSEUDO_KFUNC_CALL ) { +const struct bpf_kfunc_inline *in; + +/* an inlined kfunc call gets its native code */ + in =bpf_kfunc_native (env, insn_idx); +if (in) { +memcpy (prog, in->image, in->image_len); + prog += in->image_len; +break ; + } fm =bpf_jit_find_kfunc_model (bpf_prog, insn);if (!fm)return -EINVAL ;
Signature · 1 line
2.51.1
RFC PATCH 4/7 bpf: Add kfuncs with bodies for common operations
In reply to Yusheng ZhengAdd kfuncs with bodies for the operation families evaluated in [1]:bpf_rol64() (rotate), bpf_select64() (conditional select), bpf_extract64() (bit field extract), bpf_load_be64() (big-endian load), bpf_prefetch(), bpf_copy16() (16-byte copy) and bpf_lea64() (address computation). They are in kernel/bpf/insn_kfuncs/, apart from the verifier and the JITs, under the new CONFIG_BPF_INSN_KFUNCS, which can be built in or as a module. The x86-64 JIT inlines copies of their compiled code.
A prefetch is a load whose value is not used, so the verifier checks the address like that of any load. bpf_copy16() loads both halves before it stores them, like its body, so that a copy of it and the body agree when the buffers overlap.
[1] https://arxiv.org/abs/2606.24213
--- kernel/bpf/Kconfig | 1 + kernel/bpf/Makefile | 1 + kernel/bpf/insn_kfuncs/Kconfig | 11 ++ kernel/bpf/insn_kfuncs/Makefile | 2 + kernel/bpf/insn_kfuncs/bpf_insn_kfuncs.c | 173 +++++++++++++++++++++++ 5 files changed, 188 insertions(+) create mode 100644 kernel/bpf/insn_kfuncs/Kconfig create mode 100644 kernel/bpf/insn_kfuncs/Makefile create mode 100644 kernel/bpf/insn_kfuncs/bpf_insn_kfuncs.c
kernel/bpf/Kconfig +1 -0
@@ -101,6 +101,7 @@ config BPF_CRYPTO data. The supported algorithms are AES-CBC and AES-ECB. source "kernel/bpf/preload/Kconfig" +source "kernel/bpf/insn_kfuncs/Kconfig" config BPF_LSM bool "Enable BPF LSM Instrumentation"
kernel/bpf/Makefile +1 -0
@@ -60,6 +60,7 @@ obj-${CONFIG_BPF_LSM} += bpf_lsm_proto.o bpf_lsm.o endif obj-$(CONFIG_BPF_CRYPTO) += crypto.o obj-$(CONFIG_BPF_PRELOAD) += preload/ +obj-$(CONFIG_BPF_INSN_KFUNCS) += insn_kfuncs/ obj-$(CONFIG_BPF_SYSCALL) += relo_core.o obj-$(CONFIG_BPF_SYSCALL) += btf_iter.o
kernel/bpf/insn_kfuncs/Kconfig new file +11 -0
@@ -0,0 +1,11 @@ +# SPDX-License-Identifier: GPL-2.0-only +config BPF_INSN_KFUNCS + tristate "Kfuncs for CPU instructions that BPF lacks" + depends on BPF_SYSCALL && BPF_JIT && DEBUG_INFO_BTF + help + Provide kfuncs for rotate, conditional select, bit field extract, + big-endian load, prefetch, 16-byte copy and address computation. + The verifier checks each call as the kfunc's BPF body, and the JIT + puts native code in its place where it can. + + If unsure, say N.
kernel/bpf/insn_kfuncs/Makefile new file +2 -0
@@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0 +obj-$(CONFIG_BPF_INSN_KFUNCS) += bpf_insn_kfuncs.o
kernel/bpf/insn_kfuncs/bpf_insn_kfuncs.c new file +173 -0
173 lines not on this page · raw message
Signature · 1 line
2.51.1
[email protected]
AI reviewRe: [RFC PATCH bpf-next 4/7] bpf: Add kfuncs with bodies for common operations
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] bpf: bpf_extract64 lacks bounds checking, leading to UB and incorrect results --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] bpf: bpf_extract64 lacks bounds checking, leading to UB and incorrect results --
commit 585ec9b342241a745cc98fb7ad012ebceacf697c
Author: Yusheng Zheng <[email protected]>
bpf: Add kfuncs with bodies for common operations
This commit adds kfuncs with bodies for common operation families, including bpf_extract64 for bit field extraction. They are placed in kernel/bpf/insn_kfuncs/ and can be inlined by the x86-64 JIT.
[1] https://arxiv.org/abs/2606.24213
Quoted 4 lines
kernel/bpf/insn_kfuncs/bpf_insn_kfuncs.c new file +0
-0
[ ... ]
kernel/bpf/insn_kfuncs/bpf_insn_kfuncs.c line 28
Quoted 9 lines
kernel/bpf/insn_kfuncs/bpf_insn_kfuncs.c new file +9
-0+__bpf_kfunc u64 bpf_select64(u64 cond, u64 a, u64 b) +{ +return cond ? a : b; +} + +__bpf_kfunc u64 bpf_extract64(u64 x, u32 start__k, u32 len__k) +{ +return x << (64 - start__k - len__k) >> (64 - len__k); +}
[Severity: Medium]
Will this shift operation cause undefined behavior when len__k is 0 or when start__k + len__k is greater than 64?
In C, shifting a 64-bit value by 64 or more is undefined behavior. For example, when len__k is 0, 64 - len__k evaluates to 64. Even when this is masked by the architecture or BPF semantics, a zero-length extract might incorrectly return the unmasked value instead of 0, and out-of-bounds extracts could return garbage.
Quoted 3 lines
kernel/bpf/insn_kfuncs/bpf_insn_kfuncs.c new file +3
-0+ +__bpf_kfunc u64 bpf_load_be64(const void *p, s32 off__k) +{
[ ... ]
Quoted 8 lines
kernel/bpf/insn_kfuncs/bpf_insn_kfuncs.c new file +8
-0+/* the jump lands within the body, here on its last instruction */ +static const struct bpf_insn select64_body[] = { +BPF_JMP_IMM (BPF_JNE ,BPF_REG_1 ,0 ,1 ), +BPF_MOV64_REG (BPF_REG_2 ,BPF_REG_3 ), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_2 ), +}; + +/* x << (64 - start - len) >> (64 - len) */
kernel/bpf/insn_kfuncs/bpf_insn_kfuncs.c line 86
Quoted 10 lines
kernel/bpf/insn_kfuncs/bpf_insn_kfuncs.c new file +10
-0+static const struct bpf_insn extract64_body[] = { +BPF_MOV32_IMM (BPF_REG_4 ,64 ), +BPF_ALU32_REG (BPF_SUB ,BPF_REG_4 ,BPF_REG_2 ), +BPF_ALU32_REG (BPF_SUB ,BPF_REG_4 ,BPF_REG_3 ), +BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_1 ), +BPF_ALU64_REG (BPF_LSH ,BPF_REG_0 ,BPF_REG_4 ), +BPF_MOV32_IMM (BPF_REG_4 ,64 ), +BPF_ALU32_REG (BPF_SUB ,BPF_REG_4 ,BPF_REG_3 ), +BPF_ALU64_REG (BPF_RSH ,BPF_REG_0 ,BPF_REG_4 ), +};
[Severity: Medium]
Does this BPF body also lack bounds checking for the extraction lengths?
If a BPF program calls bpf_extract64 with invalid arguments and falls back to these BPF instructions, this logic mimics the C code above. This appears to mask a shift of 64 down to 0, which would return an incorrect non-zero value for 0-length extracts instead of 0.
Quoted 3 lines
kernel/bpf/insn_kfuncs/bpf_insn_kfuncs.c new file +3
-0+ +/* the offset is an s32 */ +static const struct bpf_insn load_be64_body[] = {
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
RFC PATCH 5/7 bpf, x86: Add native code for some inline kfuncs
In reply to Yusheng ZhengA copy of a compiled kfunc keeps constant arguments in registers and cannot put the result in the register of an argument. Give five of the kfuncs in insn_kfuncs an emit callback that writes x86-64 code: rol by an immediate, or rorx with BMI2 into another register; test and cmovcc, with the inverse condition when the result is in the register of the first value; mov of the control word and bextr with BMI1; movbe; and lea. Rotating a register in place by 13 is then "rol $13, %rbx" where a copy takes five instructions. Without BMI1 or MOVBE, or with constants that the instruction cannot take, the callback returns an error and the JIT copies the kfunc.
The code is in insn_kfuncs/x86/insn_kfuncs.h, which bpf_insn_kfuncs.c includes under CONFIG_BPF_INSN_KFUNCS_ARCH, as lib/crc includes the code for each architecture. The prefetch and the 16-byte copy compile to the instructions themselves, so they have no callback.
--- kernel/bpf/insn_kfuncs/Kconfig | 5 + kernel/bpf/insn_kfuncs/Makefile | 3 + kernel/bpf/insn_kfuncs/bpf_insn_kfuncs.c | 21 ++- kernel/bpf/insn_kfuncs/x86/insn_kfuncs.h | 155 +++++++++++++++++++++++ 4 files changed, 179 insertions(+), 5 deletions(-) create mode 100644 kernel/bpf/insn_kfuncs/x86/insn_kfuncs.h
kernel/bpf/insn_kfuncs/Kconfig +5 -0
@@ -9,3 +9,8 @@ config BPF_INSN_KFUNCS puts native code in its place where it can. If unsure, say N. + +config BPF_INSN_KFUNCS_ARCH + bool + depends on BPF_INSN_KFUNCS + default y if X86_64
kernel/bpf/insn_kfuncs/Makefile +3 -0
@@ -1,2 +1,5 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_BPF_INSN_KFUNCS) += bpf_insn_kfuncs.o +ifeq ($(CONFIG_BPF_INSN_KFUNCS_ARCH),y) +CFLAGS_bpf_insn_kfuncs.o += -I$(src)/$(SRCARCH) +endif
kernel/bpf/insn_kfuncs/bpf_insn_kfuncs.c +16 -5
@@ -118,6 +118,16 @@ static const struct bpf_insn lea64_body[] = { BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_4), }; +#ifdef CONFIG_BPF_INSN_KFUNCS_ARCH +#include "insn_kfuncs.h" /* $(SRCARCH)/insn_kfuncs.h */ +#else +#define rol64_emit NULL +#define select64_emit NULL +#define extract64_emit NULL +#define load_be64_emit NULL +#define lea64_emit NULL +#endif + BTF_KFUNCS_START(insn_kfunc_ids) BTF_ID_FLAGS(func, bpf_rol64) BTF_ID_FLAGS(func, bpf_select64)
@@ -139,14 +149,15 @@ BTF_ID(func, bpf_lea64)#define BODY (i, op, emit) { &body_ids[i], op##_body, ARRAY_SIZE(op##_body), emit } +/* without emit, the JIT copies the kfunc, whose code is the instruction itself */ static const struct bpf_kfunc_body bodies[] = { -BODY (0 , rol64,NULL ), -BODY (1 , select64,NULL ), -BODY (2 , extract64,NULL ), -BODY (3 , load_be64,NULL ), +BODY (0 , rol64, rol64_emit), +BODY (1 , select64, select64_emit), +BODY (2 , extract64, extract64_emit), +BODY (3 , load_be64, load_be64_emit),BODY (4 , prefetch,NULL ),BODY (5 , copy16,NULL ), -BODY (6 , lea64,NULL ), +BODY (6 , lea64, lea64_emit), };static const struct btf_kfunc_id_set insn_kfunc_set = {
kernel/bpf/insn_kfuncs/x86/insn_kfuncs.h new file +155 -0
155 lines not on this page · raw message
Signature · 1 line
2.51.1
RFC PATCH 6/7 selftests/bpf: Test inline kfuncs
In reply to Yusheng Zhengverifier_kfunc_inline checks:
- the x86-64 native code of each kfunc with a body, emitted or
copied, and the binding of operands;
- the precision of results, the register effects at the entry and exit of the body, constant arguments and a prefetch through a scalar;
- kfuncs with a body from a module: with native code from the module, copied, kept for a division, and rejected in a program type that may not call them;
- that the kfuncs agree with plain BPF on random inputs.
bpf_test_kfunc_body.ko checks that registration rejects bad bodies. The config enables CONFIG_BPF_INSN_KFUNCS.
--- tools/testing/selftests/bpf/Makefile | 2 +- tools/testing/selftests/bpf/config | 1 + .../bpf/prog_tests/kfunc_body_registration.c | 18 + .../selftests/bpf/prog_tests/verifier.c | 2 + .../bpf/progs/verifier_kfunc_inline.c | 543 ++++++++++++++++++ .../testing/selftests/bpf/test_kmods/Makefile | 2 +- .../bpf/test_kmods/bpf_test_kfunc_body.c | 121 ++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 77 +++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 4 + 9 files changed, 768 insertions(+), 2 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/kfunc_body_registration.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_kfunc_inline.c create mode 100644 tools/testing/selftests/bpf/test_kmods/bpf_test_kfunc_body.c
tools/testing/selftests/bpf/Makefile +1 -1
@@ -48,7 +48,7 @@ TEST_PROGS_EXTENDED := \ ima_setup.sh verify_sig_setup.sh TEST_KMODS := bpf_testmod.ko bpf_test_no_cfi.ko bpf_test_modorder_x.ko \ - bpf_test_modorder_y.ko bpf_test_rqspinlock.ko + bpf_test_modorder_y.ko bpf_test_rqspinlock.ko bpf_test_kfunc_body.ko TEST_KMOD_TARGETS = $(addprefix $(OUTPUT)/,$(TEST_KMODS)) # Compile but not part of 'make run_tests'
tools/testing/selftests/bpf/config +1 -0
@@ -3,6 +3,7 @@ CONFIG_BOOTPARAM_HARDLOCKUP_PANIC=y CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=1 CONFIG_BPF=y CONFIG_BPF_EVENTS=y +CONFIG_BPF_INSN_KFUNCS=y CONFIG_BPF_JIT=y CONFIG_BPF_KPROBE_OVERRIDE=y CONFIG_BPF_LIRC_MODE2=y
tools/testing/selftests/bpf/prog_tests/kfunc_body_registration.c new file +18 -0
@@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <test_progs.h> +#include <testing_helpers.h> + +/* bpf_test_kfunc_body.ko fails to load unless bad kfunc bodies are rejected */ +void test_kfunc_body_registration (void ) +{ +int fd, err; + + fd =open ("bpf_test_kfunc_body.ko" ,O_RDONLY ); +if (!ASSERT_GE (fd,0 ,"open" )) +return ; + err =finit_module (fd,"" ,0 ); +close (fd); +if (!ASSERT_OK (err,"finit_module" )) +return ; +ASSERT_OK (delete_module ("bpf_test_kfunc_body" ,0 ),"delete_module" ); +}
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -60,6 +60,7 @@#include "verifier_iterating_callbacks.skel.h" #include "verifier_jeq_infer_not_null.skel.h" #include "verifier_jit_convergence.skel.h" +#include "verifier_kfunc_inline.skel.h" #include "verifier_kfunc_packet_access.skel.h" #include "verifier_kfunc_uninit.skel.h" #include "verifier_kfunc_uninit_multi.skel.h"
@@ -245,6 +246,7 @@ void test_verifier_int_ptr(void) { RUN(verifier_int_ptr); }void test_verifier_iterating_callbacks (void ) {RUN (verifier_iterating_callbacks); }void test_verifier_jeq_infer_not_null (void ) {RUN (verifier_jeq_infer_not_null); }void test_verifier_jit_convergence (void ) {RUN (verifier_jit_convergence); } +void test_verifier_kfunc_inline (void ) {RUN_TESTS (verifier_kfunc_inline); }void test_verifier_kfunc_packet_access (void ) {RUN_TESTS (verifier_kfunc_packet_access); }void test_verifier_kfunc_uninit (void ) {RUN_TESTS (verifier_kfunc_uninit); }void test_verifier_kfunc_uninit_multi (void ) {RUN_TESTS (verifier_kfunc_uninit_multi); }
tools/testing/selftests/bpf/progs/verifier_kfunc_inline.c new file +543 -0
543 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/Makefile +1 -1
@@ -19,7 +19,7 @@ Q = @ endif MODULES = bpf_testmod.ko bpf_test_no_cfi.ko bpf_test_modorder_x.ko \ - bpf_test_modorder_y.ko bpf_test_rqspinlock.ko + bpf_test_modorder_y.ko bpf_test_rqspinlock.ko bpf_test_kfunc_body.ko $(foreach m,$(MODULES),$(eval obj-m += $(m:.ko=.o)))
tools/testing/selftests/bpf/test_kmods/bpf_test_kfunc_body.c new file +121 -0
121 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +77 -0
95 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +4 -0
@@ -357,4 +357,8 @@ void bpf_testmod_test_hardirq_fn(void);void bpf_testmod_test_softirq_fn (void ); void bpf_kfunc_trigger_ctx_check(void) __ksym; +u64 bpf_testmod_inline_mov(u64 x) __ksym; +u64 bpf_testmod_inline_xor(u64 a, u64 b) __ksym; +u64 bpf_testmod_inline_div(u64 a, u64 b) __ksym; +#endif /* _BPF_TESTMOD_KFUNC_H */
Signature · 1 line
2.51.1
RFC PATCH 7/7 Documentation/bpf: Describe inline kfuncs
In reply to Yusheng ZhengAdd a section on kfuncs with a body to kfuncs.rst: how the verifier handles their calls, where their native code comes from, and how a kfunc set gives a kfunc a body.
--- Documentation/bpf/kfuncs.rst | 53 ++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+)
Documentation/bpf/kfuncs.rst +53 -0
@@ -695,6 +695,59 @@ verified inline, so an unassigned R2 is simply passed back to the caller as uninitialized and only a caller that reads it fails. A stack pointer left in R2 is still rejected there, just as one in R0 is. +2.10 Inline kfuncs +------------------ + +A kfunc can come with a body: a few BPF instructions that compute it from its +arguments in R1-R5 into R0. The verifier then checks each call of the kfunc as +its body, so it knows as much about the result as if the program had computed +it in BPF, and the JIT puts native code in place of the call. Where the JIT has +no native code, the body runs in place of the call. BPF programs call such +kfuncs like any other, and arguments whose names end in ``__k`` must be known +constants (see section 2.3.2). ``CONFIG_BPF_INSN_KFUNCS`` provides +``bpf_rol64()``, ``bpf_select64()``, ``bpf_extract64()``, ``bpf_load_be64()``, +``bpf_prefetch()``, ``bpf_copy16()`` and ``bpf_lea64()``. + +The verifier replaces each call with the body before it analyzes the program. +Only the arguments are readable at the entry of the body, and R1-R5 are not +readable after it, as after a call. After the analysis, a call goes back into +the program if the JIT has native code for it, with its operands bound to the +registers that the moves around the call copied them from or to. The body stays +when the verifier rewrites it later, for example with speculation barriers, +when it accesses memory other than the stack, map values, memory and packets, +and when constant blinding is on. + +The native code comes from the kfunc set: an ``emit`` callback writes it for +the architecture, such as ``rol $13`` or ``movbe 8(%rdi)`` on x86-64. Without +``emit``, or when it has no code for the CPU, the x86-64 JIT copies the +compiled kfunc with its registers renamed. It copies only straight-line moves, +ALU instructions and address computations on the registers of a call, without +division or rip-relative addressing. Native code is trusted like the rest of +the JIT: it has to compute what the body computes, with the same memory +accesses. + +A kfunc set gives bodies to some of its kfuncs:: + + static const struct bpf_insn rol64_body[] = { ... }; + + static const struct bpf_kfunc_body bodies[] = { + { &body_ids[0], rol64_body, ARRAY_SIZE(rol64_body), rol64_emit }, + }; + + static const struct btf_kfunc_id_set kfunc_set = { + .set = &kfunc_ids, + .bodies = bodies, + .body_cnt = ARRAY_SIZE(bodies), + }; + +Registration checks each body: it may use R0-R5, ALU instructions, loads and +stores, and forward jumps that land within it, so that it ends by falling +through its last instruction, but not the sign extension, signed division and +byte swap of cpu v4, which not every JIT has. Each argument and the result must +fit in one register, and the kfunc may have no kfunc flags. ``emit`` writes at +most ``BPF_KFUNC_INLINE_MAX`` bytes. Modules give their kfuncs bodies and +native code in the same way. + .. _BPF_kfunc_lifecycle_expectations: 3. kfunc lifecycle expectations
Signature · 1 line
2.51.1
[email protected]
AI reviewRe: [RFC PATCH bpf-next 7/7] Documentation/bpf: Describe inline kfuncs
TL;DR: Documentation only: the new inline kfuncs section may be incomplete. It omits the CONFIG_BPF_JIT_KASAN case that blocks native code, and the 32-instruction and 32-bit __k limits that make…
Quoted 1 line
Documentation/bpf: Describe inline kfuncs
TL;DR: Documentation only: the new inline kfuncs section may be incomplete. It omits the CONFIG_BPF_JIT_KASAN case that blocks native code, and the 32-instruction and 32-bit __k limits that make registration fail.
Quoted 4 lines
Documentation/bpf/kfuncs.rst +0
-0
[ ... ]
Quoted 6 lines
Documentation/bpf/kfuncs.rst +2
-0@@ -695,6 +695,59 @@ verified inline, so an unassigned R2 is simply passed back to the caller as uninitialized and only a caller that reads it fails. A stack pointer left in R2 is still rejected there, just as one in R0 is. +2.10 Inline kfuncs +------------------
[ ... ]
Documentation/bpf/kfuncs.rst verified inline, so an unassigned R2 is simply passed back to the caller as · line 718
Quoted 8 lines
Documentation/bpf/kfuncs.rst +8
-0+The verifier replaces each call with the body before it analyzes the program. +Only the arguments are readable at the entry of the body, and R1-R5 are not +readable after it, as after a call. After the analysis, a call goes back into +the program if the JIT has native code for it, with its operands bound to the +registers that the moves around the call copied them from or to. The body stays +when the verifier rewrites it later, for example with speculation barriers, +when it accesses memory other than the stack, map values, memory and packets, +and when constant blinding is on.
Is this list of cases complete? In kfunc_native_ok() in kernel/bpf/kfunc_inline.c, the case for PTR_TO_MAP_VALUE, PTR_TO_MEM, PTR_TO_PACKET and PTR_TO_PACKET_META ends with:
/* the JIT checks these accesses with KASAN, native code does not */
if (IS_ENABLED(CONFIG_BPF_JIT_KASAN))
return false;When CONFIG_BPF_JIT_KASAN=y (HAVE_EBPF_JIT_KASAN is selected on x86-64), a body that loads or stores through a map value, memory or packet pointer keeps its body and gets no native code.
Of the in-tree kfuncs that covers bpf_load_be64(), bpf_prefetch() and bpf_copy16(). The text names movbe for bpf_load_be64() as the x86-64 example, so a reader who finds that kfunc not inlined has nothing here to explain it. Only accesses to the stack remain inlined when CONFIG_BPF_JIT_KASAN is set.
Should CONFIG_BPF_JIT_KASAN be added to the list, for example:
... when it accesses memory other than the stack, map values, memory and packets (or any memory but the stack with CONFIG_BPF_JIT_KASAN), and when constant blinding is on.
[ ... ]
Documentation/bpf/kfuncs.rst verified inline, so an unassigned R2 is simply passed back to the caller as · line 749
Quoted 7 lines
Documentation/bpf/kfuncs.rst +7
-0+Registration checks each body: it may use R0-R5, ALU instructions, loads and +stores, and forward jumps that land within it, so that it ends by falling +through its last instruction, but not the sign extension, signed division and +byte swap of cpu v4, which not every JIT has. Each argument and the result must +fit in one register, and the kfunc may have no kfunc flags. ``emit`` writes at +most ``BPF_KFUNC_INLINE_MAX`` bytes. Modules give their kfuncs bodies and +native code in the same way.
This paragraph reads as the list of what registration checks, but btf_check_kfunc_body() in kernel/bpf/btf.c rejects a body in two more cases that are not listed.
First, a body may have at most BPF_KFUNC_BODY_MAX_INSNS instructions (32, from include/linux/btf.h):
!len || len > BPF_KFUNC_BODY_MAX_INSNS
Second, a __k argument must be at most 32 bits wide, even though a u64 fits in one register:
btf_param_match_suffix(btf, &args[i], "__k") &&
btf_type_skip_modifiers(btf, args[i].type, NULL)->size > sizeof(s32)A kfunc with a "u64 n__k" argument and a two-instruction body satisfies every rule in the text, yet register_btf_kfunc_id_set() still fails with -EINVAL. A rejected body makes btf_add_kfunc_bodies() fail, so the whole kfunc set is rejected, with no message pointing at the cause.
The text also doesn't say that atomics, ld_imm64, call, exit, gotol and may_goto are not allowed in a body, although the positive list implies it.
Could the 32 instruction limit and the 32-bit limit on __k arguments be stated next to the "Each argument and the result must fit in one register" rule?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/37326585321
Quoted 2 lines · Yusheng Zheng
1. Is it acceptable that the verifier replaces calls with their bodies before the analysis and restores them afterwards?
yes.
Quoted 3 lines
2. Should kfuncs bring native code through an emit callback, or should the JIT only copy compiled kfuncs? The table shows what copying alone costs.
Copy only.
Quoted 4 lines
3. Native code is trusted like the rest of the JIT, and the selftests compare it with the bodies on random inputs. Is that enough?
4. The bodies are written by hand. Should they rather be compiled from the kfunc's C source, as in the inlinable kfuncs RFC [3]?
Both questions go away when C is the only implementation.
As I suggested to Leon: keep C as the fallback, let KF_FASTCALL be the signal, check the compiled kfunc with the insn decoder and copy it without 'ret'.
llvm keeps r1-r5 that are not arguments live across bpf_fastcall. When JIT can inline, the verifier removes spill/fill. When it cannot, they stay and C is called. That works on all archs from day one. The arguments are in rdi, rsi, rdx already, so memcpy is enough. No need to re-encode the insns.
clang compiles bpf_rol64() into
mov %esi,%ecx; mov %rdi,%rax; rol %cl,%rax
rcx is R4 and it's not an argument, hence push/pop around it. Then no need for bodies, emit, capture of __k constants, kfunc_bind(), register renaming, body table in btf, hooks in do_check() and const_fold, insn_kfuncs/ with two Kconfigs, etc.
What is missing: arm64, byte ops like setcc, selftests,
no push/pop when the register is dead.
Something like this (compile tested only):
arch/x86/net/bpf_jit_comp.c +157 -0
@@ -15,8 +15,11 @@#include <linux/memory.h> #include <linux/sort.h> #include <linux/execmem.h> +#include <linux/ftrace.h> +#include <linux/kallsyms.h> #include <asm/extable.h> #include <asm/ftrace.h> +#include <asm/insn.h> #include <asm/set_memory.h> #include <asm/nospec-branch.h> #include <asm/text-patching.h>
@@ -2501,6 +2504,154 @@ static void adjust_cmp_imm(u32 *op, s32 *imm) } } +/* The longest kfunc to inline, in bytes */ +#define KFUNC_INLINE_MAX 64 + +static u32 x86_reg_bit (u32 bpf_reg) +{ +return BIT (reg2hex[bpf_reg] |is_ereg (bpf_reg) <<3 ); +} + +/* +* Return the mask of x86 registers named by an insn of a kfunc or -EINVAL if +* the insn cannot be copied. Only 32 and 64-bit moves and ALU ops can be: +* no control flow, prefixes, rip-relative addressing or implicit destination. +*/ +static int kfunc_insn_regs (struct insn *insn) +{ +u8 rex = insn->rex_prefix.bytes[0 ], modrm = insn->modrm.bytes[0 ], sib = insn->sib.bytes[0 ]; +u8 ext =X86_MODRM_REG (modrm), idx =X86_SIB_INDEX (sib) |X86_REX_X (rex) <<2 ; +u32 op = insn->opcode.bytes[0 ]; +int regs =0 ; + +if (insn->prefixes.nbytes || insn->rex_prefix.nbytes >1 || insn->opcode.nbytes >2 || +insn_is_avx_or_xop (insn) ||insn_rip_relative (insn)) +return -EINVAL ; +if (insn->opcode.nbytes ==2 ) + op = op <<8 | insn->opcode.bytes[1 ]; + +switch (op) { +case 0x01 :case 0x03 :/* add */ +case 0x09 :case 0x0b :/* or */ +case 0x21 :case 0x23 :/* and */ +case 0x29 :case 0x2b :/* sub */ +case 0x31 :case 0x33 :/* xor */ +case 0x39 :case 0x3b :/* cmp */ +case 0x63 :/* movsxd */ +case 0x69 :case 0x6b :case 0x0faf :/* imul */ +case 0x85 :/* test */ +case 0x89 :case 0x8b :/* mov */ +case 0x8d :/* lea */ + case 0x0f40 ... 0x0f4f: /* cmovcc */ +case 0x0fb7 :case 0x0fbf :/* movzx, movsx */ + regs =BIT (ext |X86_REX_R (rex) <<1 ); +break ; +case 0x81 :case 0x83 :/* ALU ops with imm */ +case 0xc1 :case 0xd1 :case 0xd3 :/* shifts and rotates */ +break ; +case 0xc7 :/* mov imm */ +if (ext) +return -EINVAL ; +break ; +case 0xf7 :/* test, not, neg */ +if (ext >3 ) +return -EINVAL ; +break ; + case 0xb8 ... 0xbf: /* mov imm */ + case 0x0fc8 ... 0x0fcf: /* bswap */ +return BIT ((op &7 ) |X86_REX_B (rex) <<3 ); +default : +return -EINVAL ; + } +if (!insn->sib.nbytes) +return regs |BIT (X86_MODRM_RM (modrm) |X86_REX_B (rex) <<3 ); +if (idx !=4 ) + regs |=BIT (idx); +return regs |BIT (X86_SIB_BASE (sib) |X86_REX_B (rex) <<3 ); +} + +/* @op is 0x50 for push and 0x58 for pop of x86 register @reg */ +static void emit_push_pop (u8 **pprog,u8 op,int reg) +{ +u8 *prog = *pprog; + +if (reg >7 ) +EMIT1 (0x41 ); +EMIT1 (op | (reg &7 )); + *pprog = prog; +} + +/* +* Inline a KF_FASTCALL kfunc: copy what the compiler generated for it without +* ENDBR, fentry and 'ret'. bpf_fastcall lets it clobber only R0 and the +* arguments. Push/pop the other caller saved registers that it uses. +*/ +static int emit_kfunc_inline (u8 **pprog,unsigned long addr,const struct btf_func_model *m) +{ +/* a call clobbers r10 and r11, hence nothing is live there */ +u32 scratch =x86_reg_bit (BPF_REG_AX ) |x86_reg_bit (AUX_REG ); +u32 save =x86_reg_bit (X86_REG_R9 ), used =0 ; +u8 *p = (u8 *)addr, *start, *prog = *pprog; +unsigned long size, off; +struct insn insn; +int i, regs; + +if (m->nr_args >MAX_BPF_FUNC_REG_ARGS ) +return -EINVAL ; +for (i =0 ; i < m->nr_args; i++) +if (m->arg_size[i] >8 || (m->arg_flags[i] &BTF_FMODEL_ARENA_ARG )) +return -EINVAL ; +for (i =BPF_REG_0 ; i <=BPF_REG_5 ; i++) { +if (i ? i <= m->nr_args : m->ret_size) + scratch |=x86_reg_bit (i); +else + save |=x86_reg_bit (i); + } + +if (!kallsyms_lookup_size_offset (addr, &size, &off) || off) +return -EINVAL ; +if (is_endbr ((u32 *)p)) + p +=ENDBR_INSN_SIZE ; +if (ftrace_location (addr) == (unsigned long )p) + p +=MCOUNT_INSN_SIZE ; + +for (start = p;; p += insn.length) { +if (p - start >KFUNC_INLINE_MAX || p >= (u8 *)addr + size || +insn_decode_kernel (&insn, p)) +return -EINVAL ; +if (insn.opcode.bytes[0 ] ==RET_INSN_OPCODE || + (insn.opcode.bytes[0 ] ==JMP32_INSN_OPCODE && + p + insn.length + insn.immediate.value == (u8 *)x86_return_thunk)) +break ; + regs =kfunc_insn_regs (&insn); +if (regs <0 ) +return regs; + used |= regs; + } +if (used & ~(scratch | save)) +return -EINVAL ; + save &= used; + +for (i =0 ; i <16 ; i++) +if (save &BIT (i)) +emit_push_pop (&prog,0x50 , i); +memcpy (prog, start, p - start); + prog += p - start; +for (i =15 ; i >=0 ; i--) +if (save &BIT (i)) +emit_push_pop (&prog,0x58 , i); + + *pprog = prog; +return 0 ; +} + +bool bpf_jit_inlines_kfunc_call (unsigned long addr,const struct btf_func_model *m) +{ +u8 buf[BPF_MAX_INSN_SIZE ], *prog = buf; + +return !emit_kfunc_inline (&prog, addr, m); +} +static int do_jit (struct bpf_verifier_env *env,struct bpf_prog *bpf_prog,int *addrs,u8 *image,u8 *rw_image,int oldproglen,struct jit_context *ctx,bool jmp_padding) {
@@ -3424,6 +3575,12 @@ populate_extable: fm =bpf_jit_find_kfunc_model (bpf_prog, insn);if (!fm)return -EINVAL ; +if (fm->ret_flags &BTF_FMODEL_JIT_INLINE ) { + err =emit_kfunc_inline (&prog, (unsigned long )func, fm); +if (err) +return err; +break ; + } err =emit_kfunc_arena_args (bpf_prog, fm, &prog);if (err <0 )return err;
include/linux/bpf.h +3 -0
@@ -1256,6 +1256,9 @@ struct bpf_prog_offload {/* The argument is nullable. */ #define BTF_FMODEL_NULLABLE_ARG BIT(3) +/* In ret_flags: the JIT inlines calls of this kfunc. */ +#define BTF_FMODEL_JIT_INLINE BIT(4) +struct btf_func_model {u8 ret_size;u8 ret_flags;
include/linux/filter.h +1 -0
@@ -1239,6 +1239,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prvoid bpf_jit_compile (struct bpf_prog *prog);bool bpf_jit_needs_zext (void );bool bpf_jit_inlines_helper_call (s32 imm); +bool bpf_jit_inlines_kfunc_call (unsigned long addr,const struct btf_func_model *m);bool bpf_jit_supports_subprog_tailcalls (void );bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void );
kernel/bpf/core.c +9 -0
@@ -3295,6 +3295,15 @@ bool __weak bpf_jit_inlines_helper_call(s32 imm)return false; } +/* Return TRUE if the JIT backend inlines the KF_FASTCALL kfunc at @addr +* without clobbering registers that bpf_fastcall contract doesn't allow. +* Otherwise the kfunc is called and spills/fills around the call stay. +*/ +bool __weak bpf_jit_inlines_kfunc_call (unsigned long addr,const struct btf_func_model *m) +{ +return false; +} +/* Return TRUE if the JIT backend supports mixing bpf2bpf and tailcalls. */ bool __weak bpf_jit_supports_subprog_tailcalls (void ) {
kernel/bpf/helpers.c +12 -0
@@ -3383,6 +3383,16 @@ __bpf_kfunc void *bpf_rdonly_cast(const void *obj__ign, u32 btf_id__k)return (void *)obj__ign; } +__bpf_kfunc u64 bpf_rol64(u64 x, u32 n) +{ +return rol64 (x, n); +} + +__bpf_kfunc u64 bpf_select64(u64 cond, u64 a, u64 b) +{ +return cond ? a : b; +} + __bpf_kfunc void bpf_rcu_read_lock(void) {rcu_read_lock ();
@@ -5147,6 +5157,8 @@ BTF_ID(func, bpf_cgroup_release_dtor) BTF_KFUNCS_START(common_btf_ids) BTF_ID_FLAGS(func, bpf_cast_to_kern_ctx, KF_FASTCALL) BTF_ID_FLAGS(func, bpf_rdonly_cast, KF_FASTCALL | KF_PERFMON) +BTF_ID_FLAGS(func, bpf_rol64, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_select64, KF_FASTCALL) BTF_ID_FLAGS(func, bpf_rcu_read_lock) BTF_ID_FLAGS(func, bpf_rcu_read_unlock) BTF_ID_FLAGS(func, bpf_dynptr_slice, KF_RET_NULL)
kernel/bpf/verifier.c +15 -3
@@ -3151,6 +3151,8 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset) meta.func_proto = kfunc.proto; meta.func_name = kfunc.name; meta.kfunc_flags = kfunc.flags ? *kfunc.flags :0 ; +if ((meta.kfunc_flags &KF_FASTCALL ) &&bpf_jit_inlines_kfunc_call (addr, &func_model)) + func_model.ret_flags |=BTF_FMODEL_JIT_INLINE ; tab =krealloc (tab,struct_size (tab, descs, tab->nr_descs +1 ),GFP_KERNEL_ACCOUNT );if (!tab)
@@ -19257,6 +19259,13 @@ bool bpf_verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm) } } +/* True if bpf_fixup_kfunc_call() replaces calls to the kfunc */ +static bool verifier_inlines_kfunc_call (u32 func_id) +{ +return func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || + func_id == special_kfunc_list[KF_bpf_rdonly_cast]; +} +/* If @call is a kfunc or helper call, fills @cs and returns true, * otherwise returns false. */
@@ -19285,14 +19294,18 @@ bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call, }if (bpf_pseudo_kfunc_call (call)) { +struct bpf_kfunc_desc *desc;int err; err =bpf_fetch_kfunc_arg_meta (env, call->imm, call->off, &meta);if (err <0 )/* error would be reported later */ return false; + desc =find_kfunc_desc (env->prog, call->imm, call->off); cs->arg_slot_cnt =btf_proto_slots (meta.btf, meta.func_proto); - cs->fastcall = meta.kfunc_flags &KF_FASTCALL ; + cs->fastcall = (meta.kfunc_flags &KF_FASTCALL ) && + (verifier_inlines_kfunc_call (meta.func_id) || + (desc && (desc->func_model.ret_flags &BTF_FMODEL_JIT_INLINE ))); cs->is_void =btf_type_is_void (btf_type_by_id (meta.btf, meta.func_proto->type));return true; }
@@ -22945,8 +22958,7 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,__fixup_collection_insert_kfunc (&env->insn_aux_data[insn_idx], struct_meta_reg, node_offset_reg, insn, insn_buf, cnt); - }else if (desc->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || - desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) { + }else if (verifier_inlines_kfunc_call (desc->func_id)) { insn_buf[0 ] =BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_1 ); *cnt =1 ; } else if (desc->func_id == special_kfunc_list[KF_bpf_session_is_return] &&