bpf, x86: Support fetching AND/OR/XOR atomics in arena
Subsystems (6)
v5 cover bpf, x86: Support fetching AND/OR/XOR atomics in arena
Changelog:
v1: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
Changes in v5:
- Patch 1: support these ops for accesses through a number too, instead of refusing them, per Alexei. BPF_F_ARENA_SCALAR accesses keep their address in BPF_REG_AX, where the loop saves R0, so for those R0 is saved on the stack with push and pop, as the div and mod lowering does. An R0 operand stays in that stack slot: the new value is built from it, RAX ends up holding the old value that R0 is to return, and the slot is dropped instead of popped. Both exception table entries resume before the pop, so a fault leaves the stack balanced. Accesses through an arena pointer are emitted exactly as in v4. The refusal in v4 also hid the reason: programs passed the verifier and then failed to load with ENOTSUPP and an empty log, where the verifier used to say the atomic is not allowed.
- Patch 2: the fetching and/or/xor tests in verifier_arena_scalar now have to load and pass on x86-64 as well as arm64, at both widths and with the operand in a register and in R0. Two more run a fetching or against a freed page, with the operand in a register and in R0, so that the access faults, and check that the fetched register comes back 0 and that R0 is left alone. The page holds a nonzero value first, so that an access which did not fault would show, and R0 is compared with a value derived from the page address, as a constant comparison would be decided by the verifier and dropped. The 32-bit operands have bits set above 32.
- Patch 2: read the address back after the faulting access, in both the freed-page tests and uaf. A fault that missed the JIT's exception table entry would be recovered by the arena mapping its zeroed scratch page and the access would succeed, which the checks on the registers alone could not tell apart. uaf now faults on the page after the freed one, as the freed page's 32-bit address is 0 and the address check was empty.
- Patch 2: skip the uaf register checks on powerpc64, whose JIT does not clear the fetched register of a faulting atomic yet.
- Rebased.
Changes in v4:
- Patch 1: go back to the v1 lowering, keeping the load of the old value and giving it a second exception table entry, per Tejun's review. v2 and v3 dropped the load so that the CMPXCHG was the only faulting insn, but the first CMPXCHG then almost always loses, so every operation paid for an extra locked CMPXCHG and the fetching insn ended up more expensive on x86 than the load and CMPXCHG loop it is meant to replace. The extra entry is accounted for in bpf_int_jit_compile() again, as in v1. The cleanups from v2 stay.
- Patch 1: refuse these ops when dst_reg is BPF_REG_AX (replaced in v5).
- Patch 1: correct the commit message, which claimed to add the reg2pt_regs[BPF_REG_AX] entry that BPF_F_ARENA_SCALAR already added, and update the extable sizing comment.
- Patch 2: keep each *_noreturn_value in its operation's block and the or block's u32s first. v3 grouped them, leaving two 4-byte holes in .addr_space.1 that clang-17 fails to build.
- Patch 2: check the register state after the r0 forms in uaf fault: the fetched register is cleared and the address register is untouched. Use values with high bits set in fetch_r0. Correct the comment about which spelling selects the non-fetching insn.
- Patch 3: say in the commit message that x86-64 schedulers built from this header need a kernel whose JIT accepts these ops, and that already-set bits now cost a locked write. Describe the comment that is actually being removed.
Changes in v3:
- Patch 2: keep the 32-bit non-fetching and/or/xor as well.
- Patch 2: guard the new uaf asm on __BPF_FEATURE_ADDR_SPACE_CAST, since textual addr_space_cast needs LLVM 19.
- Patch 3: use __sync_fetch_and_or/and in cid.bpf.h and delete the cmpxchg loop, rather than only rewording the comment that justified it.
Changes in v2:
- Patch 1: drop the initial load in the arena lowering instead of adding a second exception table entry (reverted in v4).
- Patch 1: note the dropped bpf_jit_supports_insn() rejection in the changelog, rename the shared "mem32 extable bug" message, let the non-arena caller pass NULL for the out-params, correct the INSN_LEN comment, and drop an unrelated whitespace hunk.
- Patch 2: keep non-fetching coverage, add the R0-aliased forms to uaf, and correct the comment about why arm64 stays out of uaf.
- Patch 3: new.
A fetching AND/OR/XOR against arena memory is rejected on x86-64:
BPF_ATOMIC stores into R1 arena is not allowed
x86-64 has no single instruction for these, so the JIT lowers them to a CMPXCHG loop. The loop performs two memory accesses, the load of the old value and the CMPXCHG itself, and either can fault when the arena page goes away. The verifier reserves one exception table entry per instruction, so there was nowhere to record the second one and bpf_jit_supports_insn() refused the three opcodes instead.
x86-64 is the only architecture that needs this. riscv64 has native AMOAND/AMOOR/AMOXOR with fetch, s390 has LAN/LAO/LAX, and arm64 with LSE has LDCLRAL/LDSETAL/LDEORAL, so these already work in an arena, as does powerpc64. arm64 without LSE and loongarch reject every arena RMW atomic, CMPXCHG included, so nothing can be lowered there either way.
Patch 1 emits the loop with R12-indexed addressing and gives the load and the CMPXCHG an exception table entry each. Both resume past the whole loop rather than past the faulting instruction, with the fetch destination cleared, so a fault cannot re-enter the loop. The JIT accounts for the extra entry by rescanning the instruction stream in bpf_int_jit_compile() before the extable is sized; s390 does the same in bpf_jit_alloc() for its BPF_XCHG lowering. The resulting code is the same load and CMPXCHG loop as the non-arena lowering, which is what programs hand-roll today, minus the early exit a hand-rolled loop can take when the value is already as wanted. The loop needs RAX, so R0 is saved in BPF_REG_AX. An access through a number keeps its address there, and every arena access in Rust is one, so those save R0 on the stack instead, as div and mod do.
Patch 2 makes the selftests cover this. The existing arena and/or/xor tests discarded the returned value, so clang emitted the non-fetching instruction and the fetching one was never exercised. That was deliberate:
commit 2897b1e2a2f4 ("selftests/bpf: Fix arena_atomics failure due to llvm change") switched them to __c11_atomic_fetch_*() with memory_order_relaxed to dodge the limitation patch 1 removes. They now check the old value via __sync_fetch_and_*(), while keeping a __c11_atomic_fetch_*() of each width per operation with the result discarded so the non-fetching lowering keeps its coverage. A new fetch_r0 test pins the two register assignments the JIT special-cases, and both also go into the uaf test, since the BPF_REG_AX substitution only reaches the exception table when such an access faults. x86 is dropped from the uaf exclusion list. Accesses through a number are covered in verifier_arena_scalar.
Patch 3 drops the hand-rolled cmpxchg loop in sched_ext's cid.bpf.h, which only existed because of the x86 restriction patch 1 removes, and uses the fetching atomics directly. The loop was not a portability fallback either:
arm64 without LSE rejects every arena read-modify-write including CMPXCHG, and riscv without Zacas rejects only CMPXCHG, so the loop is strictly worse than a fetching OR there.
Testing
=======
test_progs arena_atomics passes on x86-64, arm64 and s390x, and verifier_arena_scalar on x86-64 and arm64, the architectures that take BPF_F_ARENA_SCALAR.
tools/sched_ext builds, and scx_qmap - the only in-tree user of the cmask helpers - lowers them to 17 atomic_fetch_or and 4 atomic_fetch_and with no CMPXCHG left in the cmask helpers. Nothing in test_progs exercises cmask_*. The uaf test faults on the load; the CMPXCHG entry is only reached when the page goes away between the two accesses, which no test can arrange.
Puranjay Mohan (3):
bpf, x86: Support fetching AND/OR/XOR atomics in arena selftests/bpf: Test fetching AND/OR/XOR atomics in arena
sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop
arch/x86/net/bpf_jit_comp.c | 295 +++++++++++++----- tools/sched_ext/include/scx/cid.bpf.h | 79 +---- .../selftests/bpf/prog_tests/arena_atomics.c | 53 ++++ .../selftests/bpf/progs/arena_atomics.c | 211 ++++++++++--- .../bpf/progs/verifier_arena_scalar.c | 256 +++++++++++++++ 5 files changed, 707 insertions(+), 187 deletions(-)
base-commit: 242d8416f60cef4864d1a9de0943bd3f299e960c
Signature · 1 line
2.53.0-Meta
PATCH v5 1/3 bpf, x86: Support fetching AND/OR/XOR atomics in arena
x86-64 has no single instruction for a fetching AND/OR/XOR, so the JIT lowers them to a CMPXCHG loop. That loop could not be used against arena
memory: it contains two memory accesses, the load of the old value and the CMPXCHG itself, either of which can fault when the arena page goes away, while the verifier reserves only one exception table entry per instruction. bpf_jit_supports_insn() therefore rejected the three opcodes and such programs failed to load with
BPF_ATOMIC stores into R1 arena is not allowed
Emit the same loop with R12-indexed addressing and give each of the two accesses its own exception table entry. The verifier still reserves one per instruction, so the JIT accounts for the extra entry itself by rescanning the instruction stream in bpf_int_jit_compile() before the extable is sized, as s390 does in bpf_jit_alloc() for its BPF_XCHG lowering. The rescan sits after the extra-pass early exit, so a program is counted exactly once. Drop the bpf_jit_supports_insn() rejection.
A loop without the load would need only one entry, since a CMPXCHG that loses the comparison reloads RAX, but its first CMPXCHG would then almost always lose. Every operation would pay for a second locked CMPXCHG, which makes the fetching insn more expensive than the load and CMPXCHG loop programs write by hand today. The second entry is paid for once, in the JIT.
Both entries resume past the whole loop rather than past the faulting instruction alone, with the fetch destination cleared, so a fault cannot re-enter the loop, and both are reported as writes since the BPF insn is a read-modify-write. This repurposes the INSN_LEN field of the fixup as a resume distance rather than an instruction length; a sequence long enough to overflow its 8 bits would first have to exceed BPF_MAX_INSN_SIZE, which do_jit() rejects before the image is used.
The loop needs RAX for CMPXCHG and saves R0 in BPF_REG_AX, which it also substitutes for R0 when either operand is R0. ex_handler_bpf() may then have to name r10 as the register holding the arena address or the one to clear on fault; reg2pt_regs[] already has that entry.
An access through a number, made by a program loaded with BPF_F_ARENA_SCALAR, keeps its address in BPF_REG_AX itself (see arena_scalar_access()), which is where the loop would save R0. Save R0 on the stack in that case, as the div and mod lowering does. An R0 operand stays in that stack slot: the new value is built from it, RAX ends up holding the old value that R0 is to return, and the slot is dropped instead of popped. Both entries resume before that pop or drop, so a fault leaves the stack balanced.
--- arch/x86/net/bpf_jit_comp.c | 295 ++++++++++++++++++++++++++---------- 1 file changed, 217 insertions(+), 78 deletions(-)
arch/x86/net/bpf_jit_comp.c +217 -78
@@ -239,6 +239,13 @@ static const int reg2pt_regs[] = { [BPF_REG_AX] = offsetof(struct pt_regs, r10), }; +static bool is_atomic_fetch_op (const struct bpf_insn *insn) +{ +return insn->imm == (BPF_AND |BPF_FETCH ) || + insn->imm == (BPF_OR |BPF_FETCH ) || + insn->imm == (BPF_XOR |BPF_FETCH ); +} +/* * is_ereg() == true if BPF register 'reg' maps to x86-64 r8..r15 * which need extra byte of encoding.
@@ -1784,6 +1791,93 @@ static int emit_atomic_ld_st_index(u8 **pprog, u32 atomic_op, u32 size,return 0 ; } +/* +* A fetching AND/OR/XOR can't be implemented with a single x86 insn, so do a +* CMPXCHG loop. @index_reg is X86_REG_R12 for an arena access or -1 otherwise. +* Unless @r0_on_stack, the caller has substituted BPF_REG_AX for R0 in +* @dst_reg and @src_reg. +* +* Both the load and the CMPXCHG can fault when accessing the arena, so their +* addresses are handed back in @fault[0] and @fault[1]. A fault has to resume +* at @resume, which is past the loop and past the move that delivers the old +* value, but before R0 is restored. Both are NULL for the non-arena case, +* which needs no fixup. +* +* R0 is expected to be saved in BPF_REG_AX by the caller, or pushed on the +* stack if @r0_on_stack. In that case an R0 operand stays in the stack slot, +* RAX ends up holding the old value that R0 is to return, and the slot is +* dropped rather than popped. +*/ +static int emit_atomic_fetch_rmw (u8 **pprog,struct bpf_insn *insn,u32 dst_reg, +u32 src_reg,int index_reg,bool r0_on_stack, +u8 **fault,u8 **resume) +{ +bool is64 =BPF_SIZE (insn->code) ==BPF_DW ; +bool operand_on_stack = r0_on_stack && src_reg ==BPF_REG_0 ; +u8 *branch_target, *prog = *pprog; +int err; + + branch_target = prog; + +/* Load old value */ +if (fault) + fault[0 ] = prog; +if (index_reg <0 ) +emit_ldx (&prog,BPF_SIZE (insn->code),BPF_REG_0 , dst_reg, insn->off); +else +emit_ldx_index (&prog,BPF_SIZE (insn->code),BPF_REG_0 , dst_reg, + index_reg, insn->off); + +/* +* Perform the (commutative) operation locally, put the result in +* the AUX_REG. +*/ +if (operand_on_stack) { +/* mov r11, [rsp]: the operand is the R0 saved there, RAX is the old value */ +if (is64) +EMIT4 (0x4C ,0x8B ,0x1C ,0x24 ); +else +EMIT4 (0x44 ,0x8B ,0x1C ,0x24 ); + }else { +emit_mov_reg (&prog, is64,AUX_REG ,BPF_REG_0 ); + } +maybe_emit_mod (&prog,AUX_REG , src_reg, is64); +EMIT2 (simple_alu_opcodes[BPF_OP (insn->imm)], +add_2reg (0xC0 ,AUX_REG , src_reg)); + +/* Attempt to swap in new value */ +if (fault) + fault[1 ] = prog; +if (index_reg <0 ) + err =emit_atomic_rmw (&prog,BPF_CMPXCHG , dst_reg,AUX_REG , + insn->off,BPF_SIZE (insn->code)); +else + err =emit_atomic_rmw_index (&prog,BPF_CMPXCHG ,BPF_SIZE (insn->code), + dst_reg,AUX_REG , index_reg, insn->off); +if (WARN_ON (err)) +return err; + +/* ZF tells us whether we won the race. If it's cleared we need to try again. */ +EMIT2 (X86_JNE , -(prog - branch_target) -2 ); +/* Return the pre-modification value, which RAX already is for R0 */ +if (!operand_on_stack) +emit_mov_reg (&prog, is64, src_reg,BPF_REG_0 ); + +if (resume) + *resume = prog; + +/* Restore R0 after clobbering RAX */ +if (!r0_on_stack) +emit_mov_reg (&prog, true,BPF_REG_0 ,BPF_REG_AX ); +else if (operand_on_stack) +EMIT2 (0x41 ,0x5B );/* pop r11, dropping the operand */ +else +EMIT1 (0x58 );/* pop rax */ + + *pprog = prog; +return 0 ; +} +/* * Metadata encoding for exception handling in JITed code. *
@@ -1797,7 +1891,12 @@ static int emit_atomic_ld_st_index(u8 **pprog, u32 atomic_op, u32 size,* | ARENA_ACC | ARENA_WRITE | Unused | ARENA_REG | DST_REG | INSN_LEN | * +-----------+-------------+--------+-----------+---------+----------+ * -* - INSN_LEN (8 bits): Length of faulting insn (max x86 insn = 15 bytes (fits in 8 bits)). +* - INSN_LEN (8 bits): How far past the faulting insn to resume. That is its own length +* for a single-insn access, but the distance to the end of the whole +* sequence where one BPF insn became several, as for the CMPXCHG loop +* of a fetching AND/OR/XOR. A sequence long enough to overflow this +* field would first have to exceed BPF_MAX_INSN_SIZE, which do_jit() +* rejects with -EFAULT before the image is used. * - DST_REG (8 bits): Offset of dst_reg from reg2pt_regs[] (max offset = 112 (fits in 8 bits)). * This is set to DONT_CLEAR if the insn does not read into a register. * - ARENA_REG (8 bits): Offset of the register that is used to calculate the
@@ -1844,7 +1943,7 @@ bool ex_handler_bpf(const struct exception_table_entry *x, struct pt_regs *regs)bpf_prog_report_arena_violation (is_write, addr, regs->ip); } -/* jump over faulting load and clear dest register */ +/* jump over the faulting access, or the sequence it is part of, and clear dest register */ if (reg !=DONT_CLEAR ) *(unsigned long *)((void *)regs + reg) =0 ; regs->ip += insn_len;
@@ -1852,6 +1951,44 @@ bool ex_handler_bpf(const struct exception_table_entry *x, struct pt_regs *regs)return true; } +/* +* Record an arena access that may fault. @fault_ip is the address of the +* faulting insn in the RO image, @resume_off how far past it execution has to +* resume: for a multi-insn lowering that is the end of the whole sequence, not +* the end of the one insn. +*/ +static int emit_arena_exentry (struct bpf_prog *bpf_prog,u8 *image,u8 *rw_image, +int *excnt,u8 *fault_ip,u32 resume_off, +u32 fixup_reg,u32 arena_reg,bool is_write,s16 off) +{ +struct exception_table_entry *ex; +s64 delta; + +if (!bpf_prog->aux->extable) +return 0 ; + +if (*excnt >= bpf_prog->aux->num_exentries) { +pr_err ("arena extable bug\n" ); +return -EFAULT ; + } + ex = &bpf_prog->aux->extable[(*excnt)++]; + + delta = fault_ip - (u8 *)&ex->insn; +/* switch ex to rw buffer for writes */ + ex = (void *)rw_image + ((void *)ex - (void *)image); + + ex->insn = delta; + ex->data =EX_TYPE_BPF |FIELD_PREP (DATA_ARENA_OFFSET_MASK , off); + ex->fixup =FIELD_PREP (FIXUP_INSN_LEN_MASK , resume_off) | +FIELD_PREP (FIXUP_ARENA_REG_MASK , arena_reg) | +FIELD_PREP (FIXUP_REG_MASK , fixup_reg) | +FIXUP_ARENA_ACCESS ; +if (is_write) + ex->fixup |=FIXUP_ARENA_WRITE ; + +return 0 ; +} +static void detect_reg_usage (struct bpf_insn *insn,int insn_cnt,bool *regs_used) {
@@ -2967,28 +3104,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * } populate_extable: { -struct exception_table_entry *ex; -u8 *_insn = image + proglen + (start_of_ldx - temp);u32 arena_reg, fixup_reg;bool is_write; -s64 delta; - -if (!bpf_prog->aux->extable) -break ; - -if (excnt >= bpf_prog->aux->num_exentries) { -pr_err ("mem32 extable bug\n" ); -return -EFAULT ; - } - ex = &bpf_prog->aux->extable[excnt++]; - - delta = _insn - (u8 *)&ex->insn; -/* switch ex to rw buffer for writes */ - ex = (void *)rw_image + ((void *)ex - (void *)image); - - ex->insn = delta; - - ex->data =EX_TYPE_BPF ;/* * src_reg/dst_reg holds the address in the arena region with upper
@@ -3024,14 +3141,12 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * is_write = true; } - ex->fixup =FIELD_PREP (FIXUP_INSN_LEN_MASK , prog - start_of_ldx) | -FIELD_PREP (FIXUP_ARENA_REG_MASK , arena_reg) | -FIELD_PREP (FIXUP_REG_MASK , fixup_reg); - ex->fixup |=FIXUP_ARENA_ACCESS ; -if (is_write) - ex->fixup |=FIXUP_ARENA_WRITE ; - - ex->data |=FIELD_PREP (DATA_ARENA_OFFSET_MASK , insn->off); + err =emit_arena_exentry (bpf_prog, image, rw_image, &excnt, + image + proglen + (start_of_ldx - temp), + prog - start_of_ldx, fixup_reg, + arena_reg, is_write, insn->off); +if (err) +return err; }break ;
@@ -3177,20 +3292,12 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * fallthrough;case BPF_STX |BPF_ATOMIC |BPF_W :case BPF_STX |BPF_ATOMIC |BPF_DW : { -bool is64 =BPF_SIZE (insn->code) ==BPF_DW ;u32 real_src_reg = src_reg;u32 real_dst_reg = dst_reg; +bool is_atomic_fetch =is_atomic_fetch_op (insn);u8 *old_prog; -bool is_atomic_fetch = - (insn->imm == (BPF_AND |BPF_FETCH ) || - insn->imm == (BPF_OR |BPF_FETCH ) || - insn->imm == (BPF_XOR |BPF_FETCH )); -if (is_atomic_fetch) { -/* -* Can't be implemented with a single x86 insn. -* Need to do a CMPXCHG loop. -*/ +if (is_atomic_fetch) {/* Will need RAX as a CMPXCHG operand so save R0 */ old_prog = prog;emit_mov_reg (&prog, true,BPF_REG_AX ,BPF_REG_0 );
@@ -3211,34 +3318,11 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * } }if (is_atomic_fetch) { -u8 *branch_target = prog; -/* Load old value */ -emit_ldx (&prog,BPF_SIZE (insn->code), -BPF_REG_0 , real_dst_reg, insn->off); -/* -* Perform the (commutative) operation locally, -* put the result in the AUX_REG. -*/ -emit_mov_reg (&prog, is64,AUX_REG ,BPF_REG_0 ); -maybe_emit_mod (&prog,AUX_REG , real_src_reg, is64); -EMIT2 (simple_alu_opcodes[BPF_OP (insn->imm)], -add_2reg (0xC0 ,AUX_REG , real_src_reg)); -/* Attempt to swap in new value */ - err =emit_atomic_rmw (&prog,BPF_CMPXCHG , - real_dst_reg,AUX_REG , - insn->off, -BPF_SIZE (insn->code)); -if (WARN_ON (err)) + err =emit_atomic_fetch_rmw (&prog, insn, real_dst_reg, + real_src_reg,-1 , false, +NULL ,NULL ); +if (err)return err; -/* -* ZF tells us whether we won the race. If it's -* cleared we need to try again. -*/ -EMIT2 (X86_JNE , -(prog - branch_target) -2 ); -/* Return the pre-modification value */ -emit_mov_reg (&prog, is64, real_src_reg,BPF_REG_0 ); -/* Restore R0 after clobbering RAX */ -emit_mov_reg (&prog, true,BPF_REG_0 ,BPF_REG_AX );break ; }
@@ -3262,6 +3346,53 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * fallthrough;case BPF_STX |BPF_PROBE_ATOMIC |BPF_W :case BPF_STX |BPF_PROBE_ATOMIC |BPF_DW : +if (is_atomic_fetch_op (insn)) { +u32 real_src_reg = src_reg, real_dst_reg = dst_reg; +/* +* An access through a number has its address in +* BPF_REG_AX (see arena_scalar_access()), so R0 is +* saved on the stack instead, as div and mod do. +*/ +bool r0_on_stack = dst_reg ==BPF_REG_AX ; +u8 *fault[2 ], *resume; +int j; + +/* Will need RAX as a CMPXCHG operand so save R0 */ +if (r0_on_stack) { +EMIT1 (0x50 );/* push rax */ + }else { +emit_mov_reg (&prog, true,BPF_REG_AX ,BPF_REG_0 ); +if (src_reg ==BPF_REG_0 ) + real_src_reg =BPF_REG_AX ; +if (dst_reg ==BPF_REG_0 ) + real_dst_reg =BPF_REG_AX ; + } + + err =emit_atomic_fetch_rmw (&prog, insn, real_dst_reg, + real_src_reg,X86_REG_R12 , + r0_on_stack, fault, &resume); +if (err) +return err; + +/* +* The page may be unmapped between the load and the +* CMPXCHG, so both need an entry. Both report a write, +* since the BPF insn is a read-modify-write. +*/ +for (j =0 ; j <2 ; j++) { + err =emit_arena_exentry (bpf_prog, image, rw_image, + &excnt, + image + proglen + + (fault[j] - temp), + resume - fault[j], + reg2pt_regs[real_src_reg], + reg2pt_regs[real_dst_reg], + true, insn->off); +if (err) +return err; + } +break ; + } start_of_ldx = prog;if (bpf_atomic_is_load_store (insn))
@@ -4707,6 +4838,22 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr padding = true; goto skip_init_addrs; } + +/* +* An arena fetching AND/OR/XOR is lowered to a CMPXCHG loop whose load +* and CMPXCHG can both fault, one more entry than the verifier reserved. +* The extra pass of a program with subprograms returned above, so the +* count is adjusted once. +*/ +for (i =0 ; prog->aux->arena && i < prog->len; i++) { +const struct bpf_insn *insn = &prog->insnsi[i]; + +if (BPF_CLASS (insn->code) ==BPF_STX && +BPF_MODE (insn->code) ==BPF_PROBE_ATOMIC && +is_atomic_fetch_op (insn)) + prog->aux->num_exentries++; + } + addrs =kvmalloc_objs (*addrs, prog->len +1 );if (!addrs) goto out_addrs;
@@ -4761,7 +4908,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr* The number of entries in extable is the number of BPF_LDX * insns that access kernel memory via "pointer to BTF type". * The verifier changed their opcode from LDX|MEM|size -* to LDX|PROBE_MEM|size to make JITing easier. +* to LDX|PROBE_MEM|size to make JITing easier. Arena +* accesses add one each, and an arena fetching AND/OR/XOR +* one more, see the adjustment of num_exentries above. */ u32 align = __alignof__(struct exception_table_entry ); u32 extable_size = prog->aux->num_exentries *
@@ -5035,16 +5184,6 @@ bool bpf_jit_supports_arena_scalar(void)bool bpf_jit_supports_insn (struct bpf_insn *insn,bool in_arena) { -if (!in_arena) -return true; -switch (insn->code) { -case BPF_STX |BPF_ATOMIC |BPF_W : -case BPF_STX |BPF_ATOMIC |BPF_DW : -if (insn->imm == (BPF_AND |BPF_FETCH ) || - insn->imm == (BPF_OR |BPF_FETCH ) || - insn->imm == (BPF_XOR |BPF_FETCH )) -return false; - }return true; }
Signature · 1 line
2.53.0-Meta
Quoted 47 lines · Puranjay Mohan
x86-64 has no single instruction for a fetching AND/OR/XOR, so the JIT lowers them to a CMPXCHG loop. That loop could not be used against arena
memory: it contains two memory accesses, the load of the old value and the CMPXCHG itself, either of which can fault when the arena page goes away, while the verifier reserves only one exception table entry per instruction.bpf_jit_supports_insn()therefore rejected the three opcodes and such programs failed to load with
BPF_ATOMICstores into R1 arena is not allowedEmit the same loop with R12-indexed addressing and give each of the two accesses its own exception table entry. The verifier still reserves one per instruction, so the JIT accounts for the extra entry itself by rescanning the instruction stream in
bpf_int_jit_compile()before the extable is sized, as s390 does inbpf_jit_alloc()for itsBPF_XCHGlowering. The rescan sits after the extra-pass early exit, so a program is counted exactly once. Drop thebpf_jit_supports_insn()rejection.A loop without the load would need only one entry, since a CMPXCHG that loses the comparison reloads RAX, but its first CMPXCHG would then almost always lose. Every operation would pay for a second locked CMPXCHG, which makes the fetching insn more expensive than the load and CMPXCHG loop programs write by hand today. The second entry is paid for once, in the JIT.
Both entries resume past the whole loop rather than past the faulting instruction alone, with the fetch destination cleared, so a fault cannot re-enter the loop, and both are reported as writes since the BPF insn is a read-modify-write. This repurposes the
INSN_LENfield of the fixup as a resume distance rather than an instruction length; a sequence long enough to overflow its 8 bits would first have to exceedBPF_MAX_INSN_SIZE, whichdo_jit()rejects before the image is used.The loop needs RAX for CMPXCHG and saves R0 in
BPF_REG_AX, which it also substitutes for R0 when either operand is R0.ex_handler_bpf()may then have to name r10 as the register holding the arena address or the one to clear on fault; reg2pt_regs[] already has that entry.An access through a number, made by a program loaded with
BPF_F_ARENA_SCALAR, keeps its address inBPF_REG_AXitself (seearena_scalar_access()), which is where the loop would save R0. Save R0 on the stack in that case, as the div and mod lowering does. An R0 operand stays in that stack slot: the new value is built from it, RAX ends up holding the old value that R0 is to return, and the slot is dropped instead of popped. Both entries resume before that pop or drop, so a fault leaves the stack balanced.
Quoted 1 line
Signed-off-by Puranjay Mohan <[email protected]>
Thanks.
Signature · 1 line
tejun
PATCH v5 2/3 selftests/bpf: Test fetching AND/OR/XOR atomics in arena
In reply to Puranjay MohanThe arena and/or/xor tests never exercised the fetching form of the instruction. Commit 2897b1e2a2f4 ("selftests/bpf: Fix arena_atomics failure due to llvm change") deliberately switched them to __c11_atomic_fetch_*() with memory_order_relaxed, which clang lowers to a non-fetching locked instruction when the result is unused, because x86-64 could not JIT the fetching one against an arena.
It can now, so check the returned old value with __sync_fetch_and_*(). Keep a __c11_atomic_fetch_*() of each width per operation with the result
discarded: that spelling is what selects the non-fetching insn, which would otherwise lose its only coverage, on every architecture rather than just x86. Compiling the prog with clang 22 before and after confirms the
split: 6 non-fetching and no fetching arena and/or/xor before, 6 and 22 after.
The new globals go in their operation's block, and the or block keeps its u32s first, so .addr_space.1 stays free of the padding holes that clang before 18 cannot fill when it is built without address space casts.
Add a fetch_r0 test for the two register assignments the x86 JIT has to special-case, where the operand and where the arena pointer is R0 and BPF_REG_AX is substituted for it. Clang picks its own registers and will not reliably produce either, so spell the instructions out. The values have bits set above 32 so that a truncating move on that path shows up. Add both forms to the uaf test as well, since the substitution only reaches the exception table when such an access faults, and check that the fault cleared the fetched register and left the address alone. They fault on the page after the freed one: the freed page is the first of the arena, whose 32-bit address is 0, which would make the address check empty. The address is read back afterwards, which gives 0 if the JIT's exception table entry recovered the fault, but not if the arena had mapped its zeroed scratch page and let the access succeed. The powerpc64 JIT does not clear the register of a faulting atomic yet, so those checks are skipped there.
Both spell addr_space_cast textually, which needs LLVM 19, so uaf gains the __BPF_FEATURE_ADDR_SPACE_CAST guard that every other textual use in the file already has. It was plain C before and built on older clang.
Drop x86 from the exclusion list of the uaf test, which covers the fault path. Its page is gone before the first access, so it faults on the load; the CMPXCHG entry is only reached when the page goes away between the two accesses, which a test cannot arrange.
Finally cover fetching AND/OR/XOR through a number in verifier_arena_scalar, on x86-64 and arm64. Both widths are run with the operand in a register and in R0, which is where the x86-64 JIT keeps the operand on the stack because BPF_REG_AX holds the address. The 32-bit operands have bits set above 32 so that a missing zero-extension shows. The page is also freed first, after storing a nonzero value in it so that an access which did not fault would show, and the test checks that the fetched register comes back 0 and R0 is left alone, and that reading the address back gives 0. R0 is compared with a value derived from the page
address: the verifier would decide a comparison with a constant at load time and drop it.
--- .../selftests/bpf/prog_tests/arena_atomics.c | 53 ++++ .../selftests/bpf/progs/arena_atomics.c | 211 ++++++++++++--- .../bpf/progs/verifier_arena_scalar.c | 256 ++++++++++++++++++ 3 files changed, 482 insertions(+), 38 deletions(-)
tools/testing/selftests/bpf/prog_tests/arena_atomics.c +53 -0
@@ -68,6 +68,12 @@ static void test_and(struct arena_atomics *skel)ASSERT_EQ (skel->arena->and64_value,0x010ull <<32 ,"and64_value" );ASSERT_EQ (skel->arena->and32_value,0x010 ,"and32_value" ); + +ASSERT_EQ (skel->arena->and64_result,0x110ull <<32 ,"and64_result" ); +ASSERT_EQ (skel->arena->and32_result,0x110 ,"and32_result" ); + +ASSERT_EQ (skel->arena->and64_noreturn_value,0x010ull <<32 ,"and64_noreturn_value" ); +ASSERT_EQ (skel->arena->and32_noreturn_value,0x010 ,"and32_noreturn_value" ); }static void test_or (struct arena_atomics *skel)
@@ -85,6 +91,12 @@ static void test_or(struct arena_atomics *skel)ASSERT_EQ (skel->arena->or64_value,0x111ull <<32 ,"or64_value" );ASSERT_EQ (skel->arena->or32_value,0x111 ,"or32_value" ); + +ASSERT_EQ (skel->arena->or64_result,0x110ull <<32 ,"or64_result" ); +ASSERT_EQ (skel->arena->or32_result,0x110 ,"or32_result" ); + +ASSERT_EQ (skel->arena->or64_noreturn_value,0x111ull <<32 ,"or64_noreturn_value" ); +ASSERT_EQ (skel->arena->or32_noreturn_value,0x111 ,"or32_noreturn_value" ); }static void test_xor (struct arena_atomics *skel)
@@ -102,6 +114,12 @@ static void test_xor(struct arena_atomics *skel)ASSERT_EQ (skel->arena->xor64_value,0x101ull <<32 ,"xor64_value" );ASSERT_EQ (skel->arena->xor32_value,0x101 ,"xor32_value" ); + +ASSERT_EQ (skel->arena->xor64_result,0x110ull <<32 ,"xor64_result" ); +ASSERT_EQ (skel->arena->xor32_result,0x110 ,"xor32_result" ); + +ASSERT_EQ (skel->arena->xor64_noreturn_value,0x101ull <<32 ,"xor64_noreturn_value" ); +ASSERT_EQ (skel->arena->xor32_noreturn_value,0x101 ,"xor32_noreturn_value" ); }static void test_cmpxchg (struct arena_atomics *skel)
@@ -146,6 +164,27 @@ static void test_xchg(struct arena_atomics *skel)ASSERT_EQ (skel->arena->xchg32_result,1 ,"xchg32_result" ); } +static void test_fetch_r0 (struct arena_atomics *skel) +{ +LIBBPF_OPTS (bpf_test_run_opts, topts); +int err, prog_fd; + +/* No need to attach it, just run it directly */ + prog_fd =bpf_program__fd (skel->progs.fetch_r0); + err =bpf_prog_test_run_opts (prog_fd, &topts); +if (!ASSERT_OK (err,"test_run_opts err" )) +return ; +if (!ASSERT_OK (topts.retval,"test_run_opts retval" )) +return ; + +ASSERT_EQ (skel->arena->fetch_src_r0_value,0x11100000111ull ,"fetch_src_r0_value" ); +ASSERT_EQ (skel->arena->fetch_src_r0_result,0x11000000110ull ,"fetch_src_r0_result" ); + +ASSERT_EQ (skel->arena->fetch_dst_r0_value,0x11100000111ull ,"fetch_dst_r0_value" ); +ASSERT_EQ (skel->arena->fetch_dst_r0_result,0x11000000110ull ,"fetch_dst_r0_result" ); +ASSERT_EQ (skel->arena->fetch_dst_r0_readback,0x11100000111ull ,"fetch_dst_r0_readback" ); +} +static void test_uaf (struct arena_atomics *skel) {LIBBPF_OPTS (bpf_test_run_opts, topts);
@@ -160,6 +199,18 @@ static void test_uaf(struct arena_atomics *skel)return ;ASSERT_EQ (skel->arena->uaf_recovery_fails,0 ,"uaf_recovery_fails" ); + +#ifndef __powerpc64__ +/* +* A fault clears the fetched register and leaves the address alone. The +* powerpc64 JIT does not clear the register of a faulting atomic yet. +*/ +ASSERT_EQ (skel->arena->uaf_r1_after_dst_r0,0 ,"uaf_r1_after_dst_r0" ); +ASSERT_EQ (skel->arena->uaf_r0_after_dst_r0, skel->arena->uaf_r0_before_dst_r0, +"uaf_r0_after_dst_r0" ); +ASSERT_EQ (skel->arena->uaf_r0_after_src_r0,0 ,"uaf_r0_after_src_r0" ); +ASSERT_EQ (skel->arena->uaf_reread,0 ,"uaf_reread" ); +#endif }static void test_load_acquire (struct arena_atomics *skel)
@@ -256,6 +307,8 @@ void serial_test_arena_atomics(void)test_cmpxchg (skel);if (test__start_subtest ("xchg" ))test_xchg (skel); +if (test__start_subtest ("fetch_r0" )) +test_fetch_r0 (skel);if (test__start_subtest ("uaf" ))test_uaf (skel);if (test__start_subtest ("load_acquire" ))
tools/testing/selftests/bpf/progs/arena_atomics.c +173 -38
278 lines not on this page · raw message
tools/testing/selftests/bpf/progs/verifier_arena_scalar.c +256 -0
274 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v5 2/3] selftests/bpf: Test fetching AND/OR/XOR atomics in arena
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] selftests/bpf: uaf test silently passes on ARM64 instead of skipping --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] selftests/bpf: uaf test silently passes on ARM64 instead of skipping --
commit a4f52eddc0e670a7a41a5b1939916ca84274f849
Author: Puranjay Mohan <[email protected]>
selftests/bpf: Test fetching AND/OR/XOR atomics in arena
The arena and/or/xor tests never exercised the fetching form of the instruction. Commit 2897b1e2a2f4 ("selftests/bpf: Fix arena_atomics failure due to llvm change") deliberately switched them to __c11_atomic_fetch_*() with memory_order_relaxed, which clang lowers to a non-fetching locked instruction when the result is unused, because x86-64 could not JIT the fetching one against an arena.
Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/arena_atomics.c +0
-0
[ ... ]
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/arena_atomics.c +3
-0@@ -160,6 +199,18 @@ static void test_uaf(struct arena_atomics *skel)return ;ASSERT_EQ (skel->arena->uaf_recovery_fails,0 ,"uaf_recovery_fails" ); + +#ifndef __powerpc64__ +/*
tools/testing/selftests/bpf/prog_tests/arena_atomics.c static void test_uaf(struct arena_atomics *skel) · line 214
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/arena_atomics.c +9
-0+* A fault clears the fetched register and leaves the address alone. The +* powerpc64 JIT does not clear the register of a faulting atomic yet. +*/ +ASSERT_EQ (skel->arena->uaf_r1_after_dst_r0,0 ,"uaf_r1_after_dst_r0" ); +ASSERT_EQ (skel->arena->uaf_r0_after_dst_r0, skel->arena->uaf_r0_before_dst_r0, +"uaf_r0_after_dst_r0" ); +ASSERT_EQ (skel->arena->uaf_r0_after_src_r0,0 ,"uaf_r0_after_src_r0" ); +ASSERT_EQ (skel->arena->uaf_reread,0 ,"uaf_reread" ); +#endif }
[Severity: Low]
Does this test silently pass on ARM64?
In the uaf program in arena_atomics.c, the fault testing logic is compiled out when __TARGET_ARCH_arm64 is defined:
#if defined(ENABLE_ATOMICS_TESTS) && !defined(__TARGET_ARCH_arm64) && \
defined(__BPF_FEATURE_ADDR_SPACE_CAST)
Because the BPF program does nothing on ARM64, the global variables (uaf_r1_after_dst_r0, uaf_reread, etc.) remain untouched and hold their zero-initialized default values. The host runner here unconditionally asserts that these variables equal 0, which makes the assertions tautologically succeed on ARM64.
Could the BPF program export a flag so the userspace runner can explicitly call test__skip() when the test is unsupported, instead of silently passing?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Re: [PATCH bpf-next v5 2/3] selftests/bpf: Test fetching AND/OR/XOR atomics in arena
Quoted 48 lines · On Fri, Oct 16, 2026 at 6:26 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] selftests/bpf: uaf test silently passes on ARM64 instead of skipping --commit
a4f52eddc0e670a7a41a5b1939916ca84274f849
Author: Puranjay Mohan <[email protected]>selftests/bpf: Test fetching AND/OR/XOR atomics in arena
The arena and/or/xor tests never exercised the fetching form of the instruction. Commit
2897b1e2a2f4("selftests/bpf: Fixarena_atomicsfailure due to llvm change") deliberately switched them to __c11_atomic_fetch_*() withmemory_order_relaxed, which clang lowers to a non-fetching locked instruction when the result is unused, because x86-64 could not JIT the fetching one against an arena.Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/arena_atomics.c +0
-0[ ... ]
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/arena_atomics.c +3
-0@@ -160,6 +199,18 @@ static void test_uaf(struct arena_atomics *skel)return ;ASSERT_EQ (skel->arena->uaf_recovery_fails,0 ,"uaf_recovery_fails" ); + +#ifndef __powerpc64__ +/* tools/testing/selftests/bpf/prog_tests/arena_atomics.c static void test_uaf(struct arena_atomics *skel) · line 214
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/arena_atomics.c +9
-0+* A fault clears the fetched register and leaves the address alone. The +* powerpc64 JIT does not clear the register of a faulting atomic yet. +*/ +ASSERT_EQ (skel->arena->uaf_r1_after_dst_r0,0 ,"uaf_r1_after_dst_r0" ); +ASSERT_EQ (skel->arena->uaf_r0_after_dst_r0, skel->arena->uaf_r0_before_dst_r0, +"uaf_r0_after_dst_r0" ); +ASSERT_EQ (skel->arena->uaf_r0_after_src_r0,0 ,"uaf_r0_after_src_r0" ); +ASSERT_EQ (skel->arena->uaf_reread,0 ,"uaf_reread" ); +#endif }[Severity: Low]
Does this test silently pass on ARM64?In the uaf program in
arena_atomics.c, the fault testing logic is compiled out when__TARGET_ARCH_arm64is defined:#if
defined(ENABLE_ATOMICS_TESTS)&& !defined(__TARGET_ARCH_arm64) && \
defined(__BPF_FEATURE_ADDR_SPACE_CAST)
Quoted 8 lines
Because the BPF program does nothing on ARM64, the global variables (
uaf_r1_after_dst_r0,uaf_reread, etc.) remain untouched and hold their zero-initialized default values. The host runner here unconditionally asserts that these variables equal 0, which makes the assertions tautologically succeed on ARM64.Could the BPF program export a flag so the userspace runner can explicitly call
test__skip()when the test is unsupported, instead of silently passing?
The suggested work is correct but the underlying issue is not introduced by this patch, so should be done separately.
Re: [PATCH bpf-next v5 2/3] selftests/bpf: Test fetching AND/OR/XOR atomics in arena
In reply to Puranjay MohanQuoted 56 lines · Puranjay Mohan
The arena and/or/xor tests never exercised the fetching form of the instruction. Commit
2897b1e2a2f4("selftests/bpf: Fixarena_atomicsfailure due to llvm change") deliberately switched them to __c11_atomic_fetch_*() withmemory_order_relaxed, which clang lowers to a non-fetching locked instruction when the result is unused, because x86-64 could not JIT the fetching one against an arena.It can now, so check the returned old value with __sync_fetch_and_*(). Keep a __c11_atomic_fetch_*() of each width per operation with the result
discarded: that spelling is what selects the non-fetching insn, which would otherwise lose its only coverage, on every architecture rather than just x86. Compiling the prog with clang 22 before and after confirms the
split: 6 non-fetching and no fetching arena and/or/xor before, 6 and 22 after.The new globals go in their operation's block, and the or block keeps its u32s first, so .addr_space.1 stays free of the padding holes that clang before 18 cannot fill when it is built without address space casts.
Add a
fetch_r0test for the two register assignments the x86 JIT has to special-case, where the operand and where the arena pointer is R0 andBPF_REG_AXis substituted for it. Clang picks its own registers and will not reliably produce either, so spell the instructions out. The values have bits set above 32 so that a truncating move on that path shows up. Add both forms to the uaf test as well, since the substitution only reaches the exception table when such an access faults, and check that the fault cleared the fetched register and left the address alone. They fault on the page after the freed one: the freed page is the first of the arena, whose 32-bit address is 0, which would make the address check empty. The address is read back afterwards, which gives 0 if the JIT's exception table entry recovered the fault, but not if the arena had mapped its zeroed scratch page and let the access succeed. The powerpc64 JIT does not clear the register of a faulting atomic yet, so those checks are skipped there.Both spell
addr_space_casttextually, which needs LLVM 19, so uaf gains the__BPF_FEATURE_ADDR_SPACE_CASTguard that every other textual use in the file already has. It was plain C before and built on older clang.Drop x86 from the exclusion list of the uaf test, which covers the fault path. Its page is gone before the first access, so it faults on the load; the CMPXCHG entry is only reached when the page goes away between the two accesses, which a test cannot arrange.
Finally cover fetching AND/OR/XOR through a number in
verifier_arena_scalar, on x86-64 and arm64. Both widths are run with the operand in a register and in R0, which is where the x86-64 JIT keeps the operand on the stack becauseBPF_REG_AXholds the address. The 32-bit operands have bits set above 32 so that a missing zero-extension shows. The page is also freed first, after storing a nonzero value in it so that an access which did not fault would show, and the test checks that the fetched register comes back 0 and R0 is left alone, and that reading the address back gives 0. R0 is compared with a value derived from the page
address: the verifier would decide a comparison with a constant at load time and drop it.
Quoted 1 line
Signed-off-by Puranjay Mohan <[email protected]>
Thanks.
Signature · 1 line
tejun
PATCH v5 3/3 sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop
In reply to Puranjay Mohancid.bpf.h implements cmask_set(), cmask_clear() and the test_and_ variants with a bounded cmpxchg loop, because the x86 BPF JIT used to reject BPF_OR | BPF_FETCH and BPF_AND | BPF_FETCH on arena pointers.
The x86 JIT accepts them now, and the loop was only ever an x86 workaround, not a fallback for the JITs that still reject these. arm64 without LSE rejects every arena read-modify-write, CMPXCHG included, and riscv without Zacas rejects only CMPXCHG, where a fetching OR or AND is accepted and the loop is not.
Use the fetching atomics directly, and drop CMASK_CAS_TRIES along with the error paths that could only be reached from the loop.
Clang emits the fetching form even when the result is unused, so all four helpers now need an x86-64 kernel whose JIT accepts fetching AND/OR/XOR in an arena; schedulers built from this header no longer load on older x86-64 kernels. When the bit is already as wanted, the helpers now also do a locked write of the unchanged value where the loop returned after a plain load.
--- tools/sched_ext/include/scx/cid.bpf.h | 79 +++------------------------ 1 file changed, 8 insertions(+), 71 deletions(-)
tools/sched_ext/include/scx/cid.bpf.h +8 -71
@@ -3,8 +3,8 @@* BPF-side helpers for cids and cmasks. See kernel/sched/ext/cid.h for the * authoritative layout and semantics. The BPF-side helpers use the cmask_* * naming (no scx_ prefix); cmask is the SCX bitmap type so the prefix is -* redundant in BPF code. Atomics use __sync_val_compare_and_swap and every -* helper is inline (no .c counterpart). +* redundant in BPF code. Atomics use the __sync_* builtins and every helper +* is inline (no .c counterpart). * * Included by scx/common.bpf.h; don't include directly. *
@@ -129,101 +129,38 @@ static __always_inline bool cmask_test(u32 cid, const struct scx_cmask __arena *return *__cmask_word (cid, m) &BIT_U64 (cid &63 ); } -/* -* x86 BPF JIT rejects BPF_OR | BPF_FETCH and BPF_AND | BPF_FETCH on arena -* pointers (see bpf_jit_supports_insn() in arch/x86/net/bpf_jit_comp.c). Only -* BPF_CMPXCHG / BPF_XCHG / BPF_ADD with FETCH are allowed. Implement -* test_and_{set,clear} and the atomic set/clear via a cmpxchg loop. -* -* CMASK_CAS_TRIES is sized so exhausting it means seconds of real spinning -* on one word - past any plausible contention. Abort hard. -*/ -#define CMASK_CAS_TRIES (1U << 23) - static __always_inline void cmask_set(u32 cid, struct scx_cmask __arena *m) { - u64 __arena *w; -u64 bit, old, new; -u32 i; -if (!__cmask_contains (cid, m))return ; - w =__cmask_word (cid, m); - bit =BIT_U64 (cid &63 ); - bpf_for(i, 0, CMASK_CAS_TRIES) { - old = *w; -if (old & bit) -return ; - new = old | bit; -if (__sync_val_compare_and_swap (w, old, new) == old) -return ; - } -scx_bpf_error ("cmask_set CAS exhausted at cid %u" , cid); +__sync_fetch_and_or (__cmask_word (cid, m),BIT_U64 (cid &63 )); } static __always_inline void cmask_clear(u32 cid, struct scx_cmask __arena *m) { - u64 __arena *w; -u64 bit, old, new; -u32 i; -if (!__cmask_contains (cid, m))return ; - w =__cmask_word (cid, m); - bit =BIT_U64 (cid &63 ); - bpf_for(i, 0, CMASK_CAS_TRIES) { - old = *w; -if (!(old & bit)) -return ; - new = old & ~bit; -if (__sync_val_compare_and_swap (w, old, new) == old) -return ; - } -scx_bpf_error ("cmask_clear CAS exhausted at cid %u" , cid); +__sync_fetch_and_and (__cmask_word (cid, m), ~BIT_U64 (cid &63 )); } static __always_inline bool cmask_test_and_set(u32 cid, struct scx_cmask __arena *m) { - u64 __arena *w; -u64 bit, old, new; -u32 i; +u64 bit;if (!__cmask_contains (cid, m))return false; - w =__cmask_word (cid, m); bit =BIT_U64 (cid &63 ); - bpf_for(i, 0, CMASK_CAS_TRIES) { - old = *w; -if (old & bit) -return true; - new = old | bit; -if (__sync_val_compare_and_swap (w, old, new) == old) -return false; - } -scx_bpf_error ("cmask_test_and_set CAS exhausted at cid %u" , cid); -return false; +return __sync_fetch_and_or (__cmask_word (cid, m), bit) & bit; } static __always_inline bool cmask_test_and_clear(u32 cid, struct scx_cmask __arena *m) { - u64 __arena *w; -u64 bit, old, new; -u32 i; +u64 bit;if (!__cmask_contains (cid, m))return false; - w =__cmask_word (cid, m); bit =BIT_U64 (cid &63 ); - bpf_for(i, 0, CMASK_CAS_TRIES) { - old = *w; -if (!(old & bit)) -return false; - new = old & ~bit; -if (__sync_val_compare_and_swap (w, old, new) == old) -return true; - } -scx_bpf_error ("cmask_test_and_clear CAS exhausted at cid %u" , cid); -return false; +return __sync_fetch_and_and (__cmask_word (cid, m), ~bit) & bit; } static __always_inline void __cmask_set(u32 cid, struct scx_cmask __arena *m)
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v5 3/3] sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop
TL;DR: Replacing the cmpxchg loops in cid.bpf.h with fetching atomics may make schedulers (e.g. scx_qmap) fail verification on x86-64 v7.2/v7.3 kernels that already ship cmask; should it be…
Quoted 1 line
sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop
TL;DR: Replacing the cmpxchg loops in cid.bpf.h with fetching atomics may make schedulers (e.g. scx_qmap) fail verification on x86-64 v7.2/v7.3 kernels that already ship cmask; should it be version-gated with a fallback?
Quoted 4 lines
tools/sched_ext/include/scx/cid.bpf.h +0
-0
[ ... ]
Quoted 36 lines
tools/sched_ext/include/scx/cid.bpf.h +1
-26@@ -129,101 +129,38 @@ static __always_inline bool cmask_test(u32 cid, const struct scx_cmask __arena *return *__cmask_word (cid, m) &BIT_U64 (cid &63 ); } -/* -* x86 BPF JIT rejects BPF_OR | BPF_FETCH and BPF_AND | BPF_FETCH on arena -* pointers (see bpf_jit_supports_insn() in arch/x86/net/bpf_jit_comp.c). Only -* BPF_CMPXCHG / BPF_XCHG / BPF_ADD with FETCH are allowed. Implement -* test_and_{set,clear} and the atomic set/clear via a cmpxchg loop. -* -* CMASK_CAS_TRIES is sized so exhausting it means seconds of real spinning -* on one word - past any plausible contention. Abort hard. -*/ -#define CMASK_CAS_TRIES (1U << 23) - static __always_inline void cmask_set(u32 cid, struct scx_cmask __arena *m) { - u64 __arena *w; -u64 bit, old, new; -u32 i; -if (!__cmask_contains (cid, m))return ; - w =__cmask_word (cid, m); - bit =BIT_U64 (cid &63 ); - bpf_for(i, 0, CMASK_CAS_TRIES) { - old = *w; -if (old & bit) -return ; - new = old | bit; -if (__sync_val_compare_and_swap (w, old, new) == old) -return ; - } -scx_bpf_error ("cmask_set CAS exhausted at cid %u" , cid); +__sync_fetch_and_or (__cmask_word (cid, m),BIT_U64 (cid &63 )); }
[ ... ]
Quoted 13 lines
static
__always_inlineboolcmask_test_and_clear(u32 cid, struct scx_cmask __arena *m){tools/sched_ext/include/scx/cid.bpf.h +1
-6- u64 __arena *w; -u64 bit, old, new; -u32 i; +u64 bit;if (!__cmask_contains (cid, m))return false; - w =__cmask_word (cid, m); bit =BIT_U64 (cid &63 ); - bpf_for(i, 0, CMASK_CAS_TRIES) { - old = *w;
tools/sched_ext/include/scx/cid.bpf.h static __always_inline bool cmask_test(u32 cid, const struct scx_cmask __arena * · line 164
Quoted 10 lines
tools/sched_ext/include/scx/cid.bpf.h +1
-8-if (!(old & bit)) -return false; - new = old & ~bit; -if (__sync_val_compare_and_swap (w, old, new) == old) -return true; - } -scx_bpf_error ("cmask_test_and_clear CAS exhausted at cid %u" , cid); -return false; +return __sync_fetch_and_and (__cmask_word (cid, m), ~bit) & bit; }
[ ... ]
Will schedulers built from this header still load on the x86-64 kernels that already ship cmask?
cmask and cid.bpf.h were added in a58e6b79b432 ("sched_ext: Add cmask, a base-windowed bitmap over cid space"), which is in v7.2 and in v7.3-rc. In both of those trees, x86 bpf_jit_supports_insn() still rejects the fetching forms in an arena:
case BPF_STX | BPF_ATOMIC | BPF_DW:
if (insn->imm == (BPF_AND | BPF_FETCH) ||
insn->imm == (BPF_OR | BPF_FETCH) ||
insn->imm == (BPF_XOR | BPF_FETCH))
return false;Only 4432f58d8449 earlier in this series lifts that restriction.
As an in-tree example, scx_qmap's pick_direct_dispatch_cid() calls cmask_test_and_clear(prev_cid, &qa.idle_cids.mask). Built at this commit, it would fail verification on x86-64 v7.2/v7.3 with "BPF_ATOMIC stores into R1 arena is not allowed". Before this change it loaded there, because the cmpxchg loop existed for exactly that reason. With LLVM >= 20, cmask_set() and cmask_clear() hit the same problem.
The commit message does say that "schedulers built from this header no longer load on older x86-64 kernels". However, tools/sched_ext/include/ scx/compat.h says the compat mechanisms "keep BPF schedulers built against newer headers running on older kernels", and compat.bpf.h already gates on kernel version, for example:
if (bpf_ksym_exists(scx_bpf_dsq_peek) &&
LINUX_KERNEL_VERSION >= KERNEL_VERSION(7, 1, 0))The verifier prunes the branch that is not taken, so the same pattern could keep the cmpxchg loop as a fallback on kernels whose x86 JIT lacks the new support. This change instead makes x86-64 v7.2 and v7.3, the platform where these helpers used to work, unable to load any scheduler that uses them.
Should the fetching atomics be gated, for example on LINUX_KERNEL_VERSION or on a CO-RE check, with the old loop kept as the fallback, rather than dropping support for every released kernel that has cmask?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/38939225700
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 38 lines · On Fri, 16 Oct 2026 10:13:11 -0700 you wrote:
Changelog:
v1: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/Changes in v5:
- Patch 1: support these ops for accesses through a number too, instead of refusing them, per Alexei.BPF_F_ARENA_SCALARaccesses keep their address inBPF_REG_AX, where the loop saves R0, so for those R0 is saved on the stack with push and pop, as the div and mod lowering does. An R0 operand stays in that stack slot: the new value is built from it, RAX ends up holding the old value that R0 is to return, and the slot is dropped instead of popped. Both exception table entries resume before the pop, so a fault leaves the stack balanced. Accesses through an arena pointer are emitted exactly as in v4. The refusal in v4 also hid the reason: programs passed the verifier and then failed to load with ENOTSUPP and an empty log, where the verifier used to say the atomic is not allowed.
- Patch 2: the fetching and/or/xor tests inverifier_arena_scalarnow have to load and pass on x86-64 as well as arm64, at both widths and with the operand in a register and in R0. Two more run a fetching or against a freed page, with the operand in a register and in R0, so that the access faults, and check that the fetched register comes back 0 and that R0 is left alone. The page holds a nonzero value first, so that an access which did not fault would show, and R0 is compared with a value derived from the page address, as a constant comparison would be decided by the verifier and dropped. The 32-bit operands have bits set above 32.
- Patch 2: read the address back after the faulting access, in both the freed-page tests and uaf. A fault that missed the JIT's exception table entry would be recovered by the arena mapping its zeroed scratch page and the access would succeed, which the checks on the registers alone could not tell apart. uaf now faults on the page after the freed one, as the freed page's 32-bit address is 0 and the address check was empty.
- Patch 2: skip the uaf register checks on powerpc64, whose JIT does not clear the fetched register of a faulting atomic yet.
- Rebased.
Quoted 1 line
[...]
Here is the summary with links:
- [bpf-next,v5,1/3] bpf, x86: Support fetching AND/OR/XOR atomics in arena
https://git.kernel.org/bpf/bpf-next/c/d9efbaed7e61 - [bpf-next,v5,2/3] selftests/bpf: Test fetching AND/OR/XOR atomics in arena https://git.kernel.org/bpf/bpf-next/c/9e35f216e5c5
- [bpf-next,v5,3/3] sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop (no matching commit)
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html