LoongArch bpf kptr xchg inline support
Subsystems (5)
v4 cover LoongArch bpf kptr xchg inline support
From: Chenguang Zhao <[email protected]>
This series enables the BPF verifier to inline bpf_kptr_xchg() into BPF_XCHG on LoongArch64, and fixes the underlying JIT atomic ordering that makes such inlining safe.
The BPF verifier can lower bpf_kptr_xchg() to a single BPF_XCHG atomic when the JIT advertises ptr xchg support via bpf_jit_supports_ptr_xchg(). This removes helper-call overhead from the kptr exchange fast path. Inlining is only correct when the JITed exchange provides the same sequentially consistent ordering as the bpf_kptr_xchg() helper.
Patch 1 fixes memory ordering for all value-returning BPF atomic RMW operations emitted by the LoongArch bpf JIT. Per LKMM, value-returning atomic RMW must provide sequentially consistent ordering. Plain AMO instructions and bare ll/sc loops on LoongArch do not satisfy this by themselves:
- BPF_FETCH (ADD/AND/OR/XOR): switch to am*_db.{b,h,w,d}
- BPF_XCHG: switch to amswap_db.{b,h,w,d}
- BPF_CMPXCHG: emit dbar 0x700 after the ll/sc loop, matching __WEAK_LLSC_MB in cmpxchg.h
Non-value-returning RMW ops (plain BPF_ADD, BPF_AND, etc.) remain weakly ordered, consistent with LKMM. This fix is independent of kptr inlining and benefits all BPF programs using value-returning atomics on LoongArch.
Patch 2 implements bpf_jit_supports_ptr_xchg() so the verifier may inline bpf_kptr_xchg() on LoongArch64.
Patches 3 and 4 extend bpf selftests: functional coverage via kptr_xchg_inline, and an optional kptr-xchg benchmark to compare helper vs inlined paths.
Chenguang Zhao (4):
LoongArch: bpf: Fix memory ordering for value-returning atomics
LoongArch: bpf: Advertise JIT support for kptr xchg inline selftests/bpf: Enable kptr_xchg_inline test on LoongArch selftests/bpf: Add kptr-xchg benchmark
arch/loongarch/include/asm/inst.h | 18 ++++ arch/loongarch/net/bpf_jit.c | 37 ++++--- tools/testing/selftests/bpf/Makefile | 2 + tools/testing/selftests/bpf/bench.c | 2 + .../selftests/bpf/benchs/bench_kptr_xchg.c | 96 +++++++++++++++++++ .../bpf/prog_tests/kptr_xchg_inline.c | 3 +- .../selftests/bpf/progs/kptr_xchg_bench.c | 49 ++++++++++ 7 files changed, 192 insertions(+), 15 deletions(-) create mode 100644 tools/testing/selftests/bpf/benchs/bench_kptr_xchg.c create mode 100644 tools/testing/selftests/bpf/progs/kptr_xchg_bench.c
---
v4:
- Resend patches against new kernel.
v3:
- https://lore.kernel.org/all/[email protected]/
v2:
- https://lore.kernel.org/all/[email protected]/
v1:
- https://lore.kernel.org/all/[email protected]/
Signature · 1 line
2.25.1
PATCH v4 1/4 LoongArch: bpf: Fix memory ordering for value-returning atomics
From: Chenguang Zhao <[email protected]>
Per the Linux Kernel Memory Model, value-returning atomic RMW operations must provide sequentially consistent ordering (a full memory barrier). On LoongArch, plain AMO instructions and bare ll/sc loops do not satisfy this requirement by themselves.
Update emit_atomic_rmw() to emit barrier-carrying instructions for all value-returning BPF atomics:
- BPF_FETCH (ADD/AND/OR/XOR): use am*_db.{b,h,w,d}
- BPF_XCHG: use amswap_db.{b,h,w,d}
- BPF_CMPXCHG: emit dbar 0x700 after the ll/sc loop, matching __WEAK_LLSC_MB in cmpxchg.h
Add the corresponding instruction encodings and emit helpers to inst.h. Non-value-returning RMW ops (plain BPF_ADD, BPF_AND, etc.) are left as weakly ordered, consistent with LKMM.
--- arch/loongarch/include/asm/inst.h | 18 +++++++++++++++++ arch/loongarch/net/bpf_jit.c | 32 +++++++++++++++++-------------- 2 files changed, 36 insertions(+), 14 deletions(-)
arch/loongarch/include/asm/inst.h +18 -0
@@ -199,6 +199,10 @@ enum reg3_op { amswaph_op =0x70b9 , amaddb_op =0x70ba , amaddh_op =0x70bb , + amswapdbb_op =0x70bc , + amswapdbh_op =0x70bd , + amadddbb_op =0x70be , + amadddbh_op =0x70bf , amswapw_op =0x70c0 , amswapd_op =0x70c1 , amaddw_op = 0x70c2,
@@ -783,6 +787,20 @@ DEF_EMIT_REG3_FORMAT(amswapb, amswapb_op) DEF_EMIT_REG3_FORMAT(amswaph, amswaph_op) DEF_EMIT_REG3_FORMAT(amswapw, amswapw_op) DEF_EMIT_REG3_FORMAT(amswapd, amswapd_op) +DEF_EMIT_REG3_FORMAT(amswapdbb, amswapdbb_op) +DEF_EMIT_REG3_FORMAT(amswapdbh, amswapdbh_op) +DEF_EMIT_REG3_FORMAT(amadddbb, amadddbb_op) +DEF_EMIT_REG3_FORMAT(amadddbh, amadddbh_op) +DEF_EMIT_REG3_FORMAT(amadddbw, amadddbw_op) +DEF_EMIT_REG3_FORMAT(amadddbd, amadddbd_op) +DEF_EMIT_REG3_FORMAT(amanddbw, amanddbw_op) +DEF_EMIT_REG3_FORMAT(amanddbd, amanddbd_op) +DEF_EMIT_REG3_FORMAT(amordbw, amordbw_op) +DEF_EMIT_REG3_FORMAT(amordbd, amordbd_op) +DEF_EMIT_REG3_FORMAT(amxordbw, amxordbw_op) +DEF_EMIT_REG3_FORMAT(amxordbd, amxordbd_op) +DEF_EMIT_REG3_FORMAT(amswapdbw, amswapdbw_op) +DEF_EMIT_REG3_FORMAT(amswapdbd, amswapdbd_op)#define DEF_EMIT_REG3SA2_FORMAT (NAME ,OP ) \ static inline void emit_##NAME(union loongarch_instruction *insn, \
arch/loongarch/net/bpf_jit.c +18 -14
@@ -8,6 +8,9 @@#include <asm/asm-offsets.h> #include "bpf_jit.h" +/* dbar hint for ll/sc completion ordering, see __WEAK_LLSC_MB */ +#define DBAR_LLSC_MB 0x700 +#define LOONGARCH_MAX_REG_ARGS 8#define LOONGARCH_SAVE_RA_NINSNS 1
@@ -433,7 +436,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)pr_err_once ("bpf-jit: amadd.b instruction is not supported\n" );return -EINVAL ; } -emit_insn (ctx, amaddb, src, t1, t3); +emit_insn (ctx, amadddbb, src, t1, t3);emit_zext_32 (ctx, src, true);break ;case BPF_H :
@@ -441,39 +444,39 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)pr_err_once ("bpf-jit: amadd.h instruction is not supported\n" );return -EINVAL ; } -emit_insn (ctx, amaddh, src, t1, t3); +emit_insn (ctx, amadddbh, src, t1, t3);emit_zext_32 (ctx, src, true);break ;case BPF_W : -emit_insn (ctx, amaddw, src, t1, t3); +emit_insn (ctx, amadddbw, src, t1, t3);emit_zext_32 (ctx, src, true);break ;case BPF_DW : -emit_insn (ctx, amaddd, src, t1, t3); +emit_insn (ctx, amadddbd, src, t1, t3);break ; }break ;case BPF_AND |BPF_FETCH :if (isdw) { -emit_insn (ctx, amandd, src, t1, t3); +emit_insn (ctx, amanddbd, src, t1, t3); }else { -emit_insn (ctx, amandw, src, t1, t3); +emit_insn (ctx, amanddbw, src, t1, t3);emit_zext_32 (ctx, src, true); }break ;case BPF_OR |BPF_FETCH :if (isdw) { -emit_insn (ctx, amord, src, t1, t3); +emit_insn (ctx, amordbd, src, t1, t3); }else { -emit_insn (ctx, amorw, src, t1, t3); +emit_insn (ctx, amordbw, src, t1, t3);emit_zext_32 (ctx, src, true); }break ;case BPF_XOR |BPF_FETCH :if (isdw) { -emit_insn (ctx, amxord, src, t1, t3); +emit_insn (ctx, amxordbd, src, t1, t3); }else { -emit_insn (ctx, amxorw, src, t1, t3); +emit_insn (ctx, amxordbw, src, t1, t3);emit_zext_32 (ctx, src, true); }break ;
@@ -485,7 +488,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)pr_err_once ("bpf-jit: amswap.b instruction is not supported\n" );return -EINVAL ; } -emit_insn (ctx, amswapb, src, t1, t3); +emit_insn (ctx, amswapdbb, src, t1, t3);emit_zext_32 (ctx, src, true);break ;case BPF_H :
@@ -493,15 +496,15 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)pr_err_once ("bpf-jit: amswap.h instruction is not supported\n" );return -EINVAL ; } -emit_insn (ctx, amswaph, src, t1, t3); +emit_insn (ctx, amswapdbh, src, t1, t3);emit_zext_32 (ctx, src, true);break ;case BPF_W : -emit_insn (ctx, amswapw, src, t1, t3); +emit_insn (ctx, amswapdbw, src, t1, t3);emit_zext_32 (ctx, src, true);break ;case BPF_DW : -emit_insn (ctx, amswapd, src, t1, t3); +emit_insn (ctx, amswapdbd, src, t1, t3);break ; }break ;
@@ -524,6 +527,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)emit_insn (ctx, beq, t3,LOONGARCH_GPR_ZERO ,-6 );emit_zext_32 (ctx, r0, true); } +emit_insn (ctx, dbar,DBAR_LLSC_MB );break ;default :pr_err_once ("bpf-jit: invalid atomic read-modify-write opcode %02x\n" , imm);
Signature · 1 line
2.25.1
Re: [PATCH v4 1/4] LoongArch: bpf: Fix memory ordering for value-returning atomics
Quoted 8 lines · Chenguang Zhao
From: Chenguang Zhao <[email protected]>
Per the Linux Kernel Memory Model, value-returning atomic RMW operations must provide sequentially consistent ordering (a full memory barrier). On LoongArch, plain AMO instructions and bare ll/sc loops do not satisfy this requirement by themselves.
The subject line could be:
LoongArch: bpf: Align value-returning atomics with LKMM ordering
I think it's not a fix and we don't need a Fixes tag.
Other than that,
Quoted 165 lines
Update
emit_atomic_rmw()to emit barrier-carrying instructions for all value-returning BPF atomics:-
BPF_FETCH(ADD/AND/OR/XOR): use am*_db.{b,h,w,d}
-BPF_XCHG: use amswap_db.{b,h,w,d}
-BPF_CMPXCHG: emit dbar0x700after the ll/sc loop, matching__WEAK_LLSC_MBincmpxchg.hAdd the corresponding instruction encodings and emit helpers to
inst.h. Non-value-returning RMW ops (plainBPF_ADD,BPF_AND, etc.) are left as weakly ordered, consistent with LKMM.Signed-off-by Chenguang Zhao <[email protected]>--- arch/loongarch/include/asm/inst.h | 18 +++++++++++++++++ arch/loongarch/net/bpf_jit.c | 32 +++++++++++++++++-------------- 2 files changed, 36 insertions(+), 14 deletions(-)arch/loongarch/include/asm/inst.h +18
-0@@ -199,6 +199,10 @@ enum reg3_op { amswaph_op =0x70b9 , amaddb_op =0x70ba , amaddh_op =0x70bb , + amswapdbb_op =0x70bc , + amswapdbh_op =0x70bd , + amadddbb_op =0x70be , + amadddbh_op =0x70bf , amswapw_op =0x70c0 , amswapd_op =0x70c1 , amaddw_op = 0x70c2,@@ -783,6 +787,20 @@ DEF_EMIT_REG3_FORMAT(amswapb, amswapb_op) DEF_EMIT_REG3_FORMAT(amswaph, amswaph_op) DEF_EMIT_REG3_FORMAT(amswapw, amswapw_op) DEF_EMIT_REG3_FORMAT(amswapd, amswapd_op) +DEF_EMIT_REG3_FORMAT(amswapdbb, amswapdbb_op) +DEF_EMIT_REG3_FORMAT(amswapdbh, amswapdbh_op) +DEF_EMIT_REG3_FORMAT(amadddbb, amadddbb_op) +DEF_EMIT_REG3_FORMAT(amadddbh, amadddbh_op) +DEF_EMIT_REG3_FORMAT(amadddbw, amadddbw_op) +DEF_EMIT_REG3_FORMAT(amadddbd, amadddbd_op) +DEF_EMIT_REG3_FORMAT(amanddbw, amanddbw_op) +DEF_EMIT_REG3_FORMAT(amanddbd, amanddbd_op) +DEF_EMIT_REG3_FORMAT(amordbw, amordbw_op) +DEF_EMIT_REG3_FORMAT(amordbd, amordbd_op) +DEF_EMIT_REG3_FORMAT(amxordbw, amxordbw_op) +DEF_EMIT_REG3_FORMAT(amxordbd, amxordbd_op) +DEF_EMIT_REG3_FORMAT(amswapdbw, amswapdbw_op) +DEF_EMIT_REG3_FORMAT(amswapdbd, amswapdbd_op)#define DEF_EMIT_REG3SA2_FORMAT (NAME ,OP ) \ static inline void emit_##NAME(union loongarch_instruction *insn, \arch/loongarch/net/bpf_jit.c +18
-15@@ -8,6 +8,9 @@#include <asm/asm-offsets.h> #include "bpf_jit.h" +/* dbar hint for ll/sc completion ordering, see __WEAK_LLSC_MB */ +#define DBAR_LLSC_MB 0x700 +#define LOONGARCH_MAX_REG_ARGS 8#define LOONGARCH_SAVE_RA_NINSNS 1@@ -433,7 +436,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)pr_err_once ("bpf-jit: amadd.b instruction is not supported\n" );return -EINVAL ; } -emit_insn (ctx, amaddb, src, t1, t3); +emit_insn (ctx, amadddbb, src, t1, t3);emit_zext_32 (ctx, src, true);break ;case BPF_H :@@ -441,39 +444,39 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)pr_err_once ("bpf-jit: amadd.h instruction is not supported\n" );return -EINVAL ; } -emit_insn (ctx, amaddh, src, t1, t3); +emit_insn (ctx, amadddbh, src, t1, t3);emit_zext_32 (ctx, src, true);break ;case BPF_W : -emit_insn (ctx, amaddw, src, t1, t3); +emit_insn (ctx, amadddbw, src, t1, t3);emit_zext_32 (ctx, src, true);break ;case BPF_DW : -emit_insn (ctx, amaddd, src, t1, t3); +emit_insn (ctx, amadddbd, src, t1, t3);break ; }break ;case BPF_AND |BPF_FETCH :if (isdw) { -emit_insn (ctx, amandd, src, t1, t3); +emit_insn (ctx, amanddbd, src, t1, t3); }else { -emit_insn (ctx, amandw, src, t1, t3); +emit_insn (ctx, amanddbw, src, t1, t3);emit_zext_32 (ctx, src, true); }break ;case BPF_OR |BPF_FETCH :if (isdw) { -emit_insn (ctx, amord, src, t1, t3); +emit_insn (ctx, amordbd, src, t1, t3); }else { -emit_insn (ctx, amorw, src, t1, t3); +emit_insn (ctx, amordbw, src, t1, t3);emit_zext_32 (ctx, src, true); }break ;case BPF_XOR |BPF_FETCH :if (isdw) { -emit_insn (ctx, amxord, src, t1, t3); +emit_insn (ctx, amxordbd, src, t1, t3); }else { -emit_insn (ctx, amxorw, src, t1, t3); +emit_insn (ctx, amxordbw, src, t1, t3);emit_zext_32 (ctx, src, true); }break ;@@ -485,7 +488,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)pr_err_once ("bpf-jit: amswap.b instruction is not supported\n" );return -EINVAL ; } -emit_insn (ctx, amswapb, src, t1, t3); +emit_insn (ctx, amswapdbb, src, t1, t3);emit_zext_32 (ctx, src, true);break ;case BPF_H :@@ -493,15 +496,15 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)pr_err_once ("bpf-jit: amswap.h instruction is not supported\n" );return -EINVAL ; } -emit_insn (ctx, amswaph, src, t1, t3); +emit_insn (ctx, amswapdbh, src, t1, t3);emit_zext_32 (ctx, src, true);break ;case BPF_W : -emit_insn (ctx, amswapw, src, t1, t3); +emit_insn (ctx, amswapdbw, src, t1, t3);emit_zext_32 (ctx, src, true);break ;case BPF_DW : -emit_insn (ctx, amswapd, src, t1, t3); +emit_insn (ctx, amswapdbd, src, t1, t3);break ; }break ;@@ -524,6 +527,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)emit_insn (ctx, beq, t3,LOONGARCH_GPR_ZERO ,-6 );emit_zext_32 (ctx, r0, true); } +emit_insn (ctx, dbar,DBAR_LLSC_MB );break ;default :pr_err_once ("bpf-jit: invalid atomic read-modify-write opcode %02x\n" , imm); --2.25.1
PATCH v4 2/4 LoongArch: bpf: Advertise JIT support for kptr xchg inline
In reply to Chenguang ZhaoFrom: Chenguang Zhao <[email protected]>
The BPF verifier can lower bpf_kptr_xchg() to BPF_XCHG when the JIT advertises ptr xchg support. With ordered amswap_db.* emission from the previous patch, declare that LoongArch bpf JIT supports this inlining.
--- arch/loongarch/net/bpf_jit.c | 5 +++++ 1 file changed, 5 insertions(+)
arch/loongarch/net/bpf_jit.c +5 -0
@@ -2396,6 +2396,11 @@ bool bpf_jit_supports_fsession(void)return true; } +bool bpf_jit_supports_ptr_xchg (void ) +{ +return true; +} +/* Indicate the JIT backend supports mixing bpf2bpf and tailcalls. */ bool bpf_jit_supports_subprog_tailcalls (void ) {
Signature · 1 line
2.25.1
Re: [PATCH v4 2/4] LoongArch: bpf: Advertise JIT support for kptr xchg inline
Quoted 7 lines · Chenguang Zhao
From: Chenguang Zhao <[email protected]>
The BPF verifier can lower
bpf_kptr_xchg()toBPF_XCHGwhen the JIT advertises ptr xchg support. With ordered amswap_db.* emission from the previous patch, declare that LoongArch bpf JIT supports this inlining.
Quoted 24 lines
Signed-off-by Chenguang Zhao <[email protected]>--- arch/loongarch/net/bpf_jit.c | 5 +++++ 1 file changed, 5 insertions(+)arch/loongarch/net/bpf_jit.c +5
-1@@ -2396,6 +2396,11 @@ bool bpf_jit_supports_fsession(void)return true; } +bool bpf_jit_supports_ptr_xchg (void ) +{ +return true; +} +/* Indicate the JIT backend supports mixing bpf2bpf and tailcalls. */ bool bpf_jit_supports_subprog_tailcalls (void ) { --2.25.1
PATCH v4 3/4 selftests/bpf: Enable kptr_xchg_inline test on LoongArch
In reply to Chenguang ZhaoFrom: Chenguang Zhao <[email protected]>
Run the kptr_xchg_inline functional test on LoongArch64 now that the bpf JIT can inline bpf_kptr_xchg() with correct memory ordering.
--- tools/testing/selftests/bpf/prog_tests/kptr_xchg_inline.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/prog_tests/kptr_xchg_inline.c +2 -1
@@ -14,7 +14,8 @@ void test_kptr_xchg_inline(void)int err;#if !(defined(__x86_64__) || defined(__aarch64__) || \ - (defined(__riscv) && __riscv_xlen ==64 )) + (defined(__riscv) && __riscv_xlen ==64 ) || \ + (defined(__loongarch__) && __loongarch_grlen ==64 ))test__skip ();return ;#endif
Signature · 1 line
2.25.1
Re: [PATCH v4 3/4] selftests/bpf: Enable kptr_xchg_inline test on LoongArch
Quoted 6 lines · Chenguang Zhao
From: Chenguang Zhao <[email protected]>
Run the
kptr_xchg_inlinefunctional test on LoongArch64 now that the bpf JIT can inlinebpf_kptr_xchg()with correct memory ordering.
Quoted 22 lines
Signed-off-by Chenguang Zhao <[email protected]>--- tools/testing/selftests/bpf/prog_tests/kptr_xchg_inline.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)tools/testing/selftests/bpf/prog_tests/kptr_xchg_inline.c +2
-2@@ -14,7 +14,8 @@ void test_kptr_xchg_inline(void)int err;#if !(defined(__x86_64__) || defined(__aarch64__) || \ - (defined(__riscv) && __riscv_xlen ==64 )) + (defined(__riscv) && __riscv_xlen ==64 ) || \ + (defined(__loongarch__) && __loongarch_grlen ==64 ))test__skip ();return ;#endif --2.25.1
PATCH v4 4/4 selftests/bpf: Add kptr-xchg benchmark
In reply to Chenguang ZhaoFrom: Chenguang Zhao <[email protected]>
Add a bpf selftest benchmark that exercises bpf_kptr_xchg() in a tight loop so helper vs inlined JIT paths can be compared on supported architectures.
--- tools/testing/selftests/bpf/Makefile | 2 + tools/testing/selftests/bpf/bench.c | 2 + .../selftests/bpf/benchs/bench_kptr_xchg.c | 96 +++++++++++++++++++ .../selftests/bpf/progs/kptr_xchg_bench.c | 49 ++++++++++ 4 files changed, 149 insertions(+) create mode 100644 tools/testing/selftests/bpf/benchs/bench_kptr_xchg.c create mode 100644 tools/testing/selftests/bpf/progs/kptr_xchg_bench.c
tools/testing/selftests/bpf/Makefile +2 -0
@@ -980,6 +980,7 @@ $(OUTPUT)/bench_lpm_trie_map.o: $(OUTPUT)/lpm_trie_bench.skel.h $(OUTPUT)/lpm_tr $(OUTPUT)/bench_bpf_nop.o: $(OUTPUT)/bpf_nop_bench.skel.h bench_bpf_timing.h $(OUTPUT)/bench_xdp_lb.o: $(OUTPUT)/xdp_lb_bench.skel.h bench_bpf_timing.h $(OUTPUT)/bench_bpf_timing.o: bench_bpf_timing.h +$(OUTPUT)/bench_kptr_xchg.o: $(OUTPUT)/kptr_xchg_bench.skel.h $(OUTPUT)/bench.o: bench.h testing_helpers.h $(BPFOBJ) $(OUTPUT)/bench: LDLIBS += -lm $(OUTPUT)/bench: $(OUTPUT)/bench.o \
@@ -1005,6 +1006,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \ $(OUTPUT)/bench_bpf_timing.o \ $(OUTPUT)/bench_bpf_nop.o \ $(OUTPUT)/bench_xdp_lb.o \ + $(OUTPUT)/bench_kptr_xchg.o \ $(OUTPUT)/usdt_1.o \ $(OUTPUT)/usdt_2.o \ #
tools/testing/selftests/bpf/bench.c +2 -0
@@ -582,6 +582,7 @@ extern const struct bench bench_lpm_trie_delete;extern const struct bench bench_lpm_trie_free;extern const struct bench bench_bpf_nop;extern const struct bench bench_xdp_lb; +extern const struct bench bench_kptr_xchg;static const struct bench *benchs[] = { &bench_count_global,
@@ -665,6 +666,7 @@ static const struct bench *benchs[] = { &bench_lpm_trie_free, &bench_bpf_nop, &bench_xdp_lb, + &bench_kptr_xchg, };static void find_benchmark (void )
tools/testing/selftests/bpf/benchs/bench_kptr_xchg.c new file +96 -0
96 lines not on this page · raw message
tools/testing/selftests/bpf/progs/kptr_xchg_bench.c new file +49 -0
49 lines not on this page · raw message
Signature · 1 line
2.25.1
Tiezhu Yang
maintainerRe: [PATCH v4 4/4] selftests/bpf: Add kptr-xchg benchmark
Quoted 6 lines · Chenguang Zhao
From: Chenguang Zhao <[email protected]>
Add a bpf selftest benchmark that exercises
bpf_kptr_xchg()in a tight loop so helper vs inlined JIT paths can be compared on supported architectures.Signed-off-by Chenguang Zhao <[email protected]>
Quoted 8 lines
--- tools/testing/selftests/bpf/Makefile | 2 + tools/testing/selftests/bpf/bench.c | 2 + .../selftests/bpf/benchs/bench_kptr_xchg.c | 96 +++++++++++++++++++ .../selftests/bpf/progs/kptr_xchg_bench.c | 49 ++++++++++ 4 files changed, 149 insertions(+)create mode 100644
tools/testing/selftests/bpf/benchs/bench_kptr_xchg.ccreate mode 100644tools/testing/selftests/bpf/progs/kptr_xchg_bench.c
Hi Chenguang,
Please send this generic selftest patch to the BPF mailing list. It should be routed through the bpf-next tree.
Also, please use the '[PATCH bpf-next]' subject prefix to specify that this is destined for the bpf-next tree.
For more info, please see:
https://www.kernel.org/doc/html/latest/bpf/bpf_devel_QA.html
Thanks,
Tiezhu
Huacai Chen
maintainerQueued the first 3 patches, since the last one is not LoongArch-specific.
Huacai
Quoted 69 lines · Chenguang Zhao
From: Chenguang Zhao <[email protected]>
This series enables the BPF verifier to inline
bpf_kptr_xchg()intoBPF_XCHGon LoongArch64, and fixes the underlying JIT atomic ordering that makes such inlining safe.The BPF verifier can lower
bpf_kptr_xchg()to a singleBPF_XCHGatomic when the JIT advertises ptr xchg support viabpf_jit_supports_ptr_xchg(). This removes helper-call overhead from the kptr exchange fast path. Inlining is only correct when the JITed exchange provides the same sequentially consistent ordering as thebpf_kptr_xchg()helper.Patch 1 fixes memory ordering for all value-returning BPF atomic RMW operations emitted by the LoongArch bpf JIT. Per LKMM, value-returning atomic RMW must provide sequentially consistent ordering. Plain AMO instructions and bare ll/sc loops on LoongArch do not satisfy this by themselves:
-BPF_FETCH(ADD/AND/OR/XOR): switch to am*_db.{b,h,w,d}
-BPF_XCHG: switch to amswap_db.{b,h,w,d}
-BPF_CMPXCHG: emit dbar0x700after the ll/sc loop, matching__WEAK_LLSC_MBincmpxchg.h
Non-value-returning RMW ops (plainBPF_ADD,BPF_AND, etc.) remain weakly ordered, consistent with LKMM. This fix is independent of kptr inlining and benefits all BPF programs using value-returning atomics on LoongArch.Patch 2 implements
bpf_jit_supports_ptr_xchg()so the verifier may inlinebpf_kptr_xchg()on LoongArch64.Patches 3 and 4 extend bpf selftests: functional coverage via
kptr_xchg_inline, and an optional kptr-xchg benchmark to compare helper vs inlined paths.Chenguang Zhao (4):
LoongArch: bpf: Fix memory ordering for value-returning atomics
LoongArch: bpf: Advertise JIT support for kptr xchg inline selftests/bpf: Enablekptr_xchg_inlinetest on LoongArch selftests/bpf: Add kptr-xchg benchmarkarch/loongarch/include/asm/inst.h | 18 ++++ arch/loongarch/net/bpf_jit.c | 37 ++++--- tools/testing/selftests/bpf/Makefile | 2 + tools/testing/selftests/bpf/bench.c | 2 + .../selftests/bpf/benchs/bench_kptr_xchg.c | 96 +++++++++++++++++++ .../bpf/prog_tests/kptr_xchg_inline.c | 3 +- .../selftests/bpf/progs/kptr_xchg_bench.c | 49 ++++++++++ 7 files changed, 192 insertions(+), 15 deletions(-) create mode 100644 tools/testing/selftests/bpf/benchs/bench_kptr_xchg.c create mode 100644 tools/testing/selftests/bpf/progs/kptr_xchg_bench.c---
v4:
- Resend patches against new kernel.v3:
- https://lore.kernel.org/all/[email protected]/v2:
- https://lore.kernel.org/all/[email protected]/v1:
- https://lore.kernel.org/all/[email protected]/--
2.25.1
Quoted 5 lines · Chenguang Zhao
From: Chenguang Zhao <[email protected]>
This series enables the BPF verifier to inline
bpf_kptr_xchg()intoBPF_XCHGon LoongArch64, and fixes the underlying JIT atomic ordering that makes such inlining safe.
The first three patches apply cleanly on top of the loongarch-next branch of linux-loongson.git.
The following selftest passed on LoongArch:
# ./test_progs -t kptr_xchg_inline
Thanks,
Tiezhu