Skip to contentSkip to conversation

LoongArch bpf kptr xchg inline support

PATCH In review Chenguang Zhao 4 participants 11 messages Jul 29 – Aug 6

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

v4 cover LoongArch bpf kptr xchg inline support

From: Chenguang Zhao <[email protected]>

This series enables the BPF verifier to inline bpf_kptr_xchg() into BPF_XCHG on LoongArch64, and fixes the underlying JIT atomic ordering that makes such inlining safe.

The BPF verifier can lower bpf_kptr_xchg() to a single BPF_XCHG atomic when the JIT advertises ptr xchg support via bpf_jit_supports_ptr_xchg(). This removes helper-call overhead from the kptr exchange fast path. Inlining is only correct when the JITed exchange provides the same sequentially consistent ordering as the bpf_kptr_xchg() helper.

Patch 1 fixes memory ordering for all value-returning BPF atomic RMW operations emitted by the LoongArch bpf JIT. Per LKMM, value-returning atomic RMW must provide sequentially consistent ordering. Plain AMO instructions and bare ll/sc loops on LoongArch do not satisfy this by themselves:
- BPF_FETCH (ADD/AND/OR/XOR): switch to am*_db.{b,h,w,d}
- BPF_XCHG: switch to amswap_db.{b,h,w,d}
- BPF_CMPXCHG: emit dbar 0x700 after the ll/sc loop, matching __WEAK_LLSC_MB in cmpxchg.h
Non-value-returning RMW ops (plain BPF_ADD, BPF_AND, etc.) remain weakly ordered, consistent with LKMM. This fix is independent of kptr inlining and benefits all BPF programs using value-returning atomics on LoongArch.

Patch 2 implements bpf_jit_supports_ptr_xchg() so the verifier may inline bpf_kptr_xchg() on LoongArch64.

Patches 3 and 4 extend bpf selftests: functional coverage via kptr_xchg_inline, and an optional kptr-xchg benchmark to compare helper vs inlined paths.

Chenguang Zhao (4):
LoongArch: bpf: Fix memory ordering for value-returning atomics
LoongArch: bpf: Advertise JIT support for kptr xchg inline selftests/bpf: Enable kptr_xchg_inline test on LoongArch selftests/bpf: Add kptr-xchg benchmark

 arch/loongarch/include/asm/inst.h             | 18 ++++
 arch/loongarch/net/bpf_jit.c                  | 37 ++++---
 tools/testing/selftests/bpf/Makefile          |  2 +
 tools/testing/selftests/bpf/bench.c           |  2 +
 .../selftests/bpf/benchs/bench_kptr_xchg.c    | 96 +++++++++++++++++++
 .../bpf/prog_tests/kptr_xchg_inline.c         |  3 +-
 .../selftests/bpf/progs/kptr_xchg_bench.c     | 49 ++++++++++
 7 files changed, 192 insertions(+), 15 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_kptr_xchg.c
 create mode 100644 tools/testing/selftests/bpf/progs/kptr_xchg_bench.c

---
v4:
- Resend patches against new kernel.

v3:
- https://lore.kernel.org/all/[email protected]/

v2:
- https://lore.kernel.org/all/[email protected]/

v1:
- https://lore.kernel.org/all/[email protected]/

Signature · 1 line
2.25.1
#1 lore raw

PATCH v4 1/4 LoongArch: bpf: Fix memory ordering for value-returning atomics

From: Chenguang Zhao <[email protected]>

Per the Linux Kernel Memory Model, value-returning atomic RMW operations must provide sequentially consistent ordering (a full memory barrier). On LoongArch, plain AMO instructions and bare ll/sc loops do not satisfy this requirement by themselves.

Update emit_atomic_rmw() to emit barrier-carrying instructions for all value-returning BPF atomics:

- BPF_FETCH (ADD/AND/OR/XOR): use am*_db.{b,h,w,d}
- BPF_XCHG: use amswap_db.{b,h,w,d}
- BPF_CMPXCHG: emit dbar 0x700 after the ll/sc loop, matching __WEAK_LLSC_MB in cmpxchg.h

Add the corresponding instruction encodings and emit helpers to inst.h. Non-value-returning RMW ops (plain BPF_ADD, BPF_AND, etc.) are left as weakly ordered, consistent with LKMM.

---
 arch/loongarch/include/asm/inst.h | 18 +++++++++++++++++
 arch/loongarch/net/bpf_jit.c      | 32 +++++++++++++++++--------------
 2 files changed, 36 insertions(+), 14 deletions(-)
arch/loongarch/include/asm/inst.h +18 -0
index 76b723590023..bdbc17d07110 100644
@@ -199,6 +199,10 @@ enum reg3_op {
 	amswaph_op	= 0x70b9,
 	amaddb_op	= 0x70ba,
 	amaddh_op	= 0x70bb,
+	amswapdbb_op	= 0x70bc,
+	amswapdbh_op	= 0x70bd,
+	amadddbb_op	= 0x70be,
+	amadddbh_op	= 0x70bf,
 	amswapw_op	= 0x70c0,
 	amswapd_op	= 0x70c1,
 	amaddw_op	= 0x70c2,
@@ -783,6 +787,20 @@ DEF_EMIT_REG3_FORMAT(amswapb, amswapb_op)
 DEF_EMIT_REG3_FORMAT(amswaph, amswaph_op)
 DEF_EMIT_REG3_FORMAT(amswapw, amswapw_op)
 DEF_EMIT_REG3_FORMAT(amswapd, amswapd_op)
+DEF_EMIT_REG3_FORMAT(amswapdbb, amswapdbb_op)
+DEF_EMIT_REG3_FORMAT(amswapdbh, amswapdbh_op)
+DEF_EMIT_REG3_FORMAT(amadddbb, amadddbb_op)
+DEF_EMIT_REG3_FORMAT(amadddbh, amadddbh_op)
+DEF_EMIT_REG3_FORMAT(amadddbw, amadddbw_op)
+DEF_EMIT_REG3_FORMAT(amadddbd, amadddbd_op)
+DEF_EMIT_REG3_FORMAT(amanddbw, amanddbw_op)
+DEF_EMIT_REG3_FORMAT(amanddbd, amanddbd_op)
+DEF_EMIT_REG3_FORMAT(amordbw, amordbw_op)
+DEF_EMIT_REG3_FORMAT(amordbd, amordbd_op)
+DEF_EMIT_REG3_FORMAT(amxordbw, amxordbw_op)
+DEF_EMIT_REG3_FORMAT(amxordbd, amxordbd_op)
+DEF_EMIT_REG3_FORMAT(amswapdbw, amswapdbw_op)
+DEF_EMIT_REG3_FORMAT(amswapdbd, amswapdbd_op)
 
 #define DEF_EMIT_REG3SA2_FORMAT(NAME, OP)				\
 static inline void emit_##NAME(union loongarch_instruction *insn,	\
arch/loongarch/net/bpf_jit.c +18 -14
index 29c281bef28e..5417a5624327 100644
@@ -8,6 +8,9 @@
 #include <asm/asm-offsets.h>
 #include "bpf_jit.h"
 
+/* dbar hint for ll/sc completion ordering, see __WEAK_LLSC_MB */
+#define DBAR_LLSC_MB	0x700
+
 #define LOONGARCH_MAX_REG_ARGS 8
 
 #define LOONGARCH_SAVE_RA_NINSNS   1
@@ -433,7 +436,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
 				pr_err_once("bpf-jit: amadd.b instruction is not supported\n");
 				return -EINVAL;
 			}
-			emit_insn(ctx, amaddb, src, t1, t3);
+			emit_insn(ctx, amadddbb, src, t1, t3);
 			emit_zext_32(ctx, src, true);
 			break;
 		case BPF_H:
@@ -441,39 +444,39 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
 				pr_err_once("bpf-jit: amadd.h instruction is not supported\n");
 				return -EINVAL;
 			}
-			emit_insn(ctx, amaddh, src, t1, t3);
+			emit_insn(ctx, amadddbh, src, t1, t3);
 			emit_zext_32(ctx, src, true);
 			break;
 		case BPF_W:
-			emit_insn(ctx, amaddw, src, t1, t3);
+			emit_insn(ctx, amadddbw, src, t1, t3);
 			emit_zext_32(ctx, src, true);
 			break;
 		case BPF_DW:
-			emit_insn(ctx, amaddd, src, t1, t3);
+			emit_insn(ctx, amadddbd, src, t1, t3);
 			break;
 		}
 		break;
 	case BPF_AND | BPF_FETCH:
 		if (isdw) {
-			emit_insn(ctx, amandd, src, t1, t3);
+			emit_insn(ctx, amanddbd, src, t1, t3);
 		} else {
-			emit_insn(ctx, amandw, src, t1, t3);
+			emit_insn(ctx, amanddbw, src, t1, t3);
 			emit_zext_32(ctx, src, true);
 		}
 		break;
 	case BPF_OR | BPF_FETCH:
 		if (isdw) {
-			emit_insn(ctx, amord, src, t1, t3);
+			emit_insn(ctx, amordbd, src, t1, t3);
 		} else {
-			emit_insn(ctx, amorw, src, t1, t3);
+			emit_insn(ctx, amordbw, src, t1, t3);
 			emit_zext_32(ctx, src, true);
 		}
 		break;
 	case BPF_XOR | BPF_FETCH:
 		if (isdw) {
-			emit_insn(ctx, amxord, src, t1, t3);
+			emit_insn(ctx, amxordbd, src, t1, t3);
 		} else {
-			emit_insn(ctx, amxorw, src, t1, t3);
+			emit_insn(ctx, amxordbw, src, t1, t3);
 			emit_zext_32(ctx, src, true);
 		}
 		break;
@@ -485,7 +488,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
 				pr_err_once("bpf-jit: amswap.b instruction is not supported\n");
 				return -EINVAL;
 			}
-			emit_insn(ctx, amswapb, src, t1, t3);
+			emit_insn(ctx, amswapdbb, src, t1, t3);
 			emit_zext_32(ctx, src, true);
 			break;
 		case BPF_H:
@@ -493,15 +496,15 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
 				pr_err_once("bpf-jit: amswap.h instruction is not supported\n");
 				return -EINVAL;
 			}
-			emit_insn(ctx, amswaph, src, t1, t3);
+			emit_insn(ctx, amswapdbh, src, t1, t3);
 			emit_zext_32(ctx, src, true);
 			break;
 		case BPF_W:
-			emit_insn(ctx, amswapw, src, t1, t3);
+			emit_insn(ctx, amswapdbw, src, t1, t3);
 			emit_zext_32(ctx, src, true);
 			break;
 		case BPF_DW:
-			emit_insn(ctx, amswapd, src, t1, t3);
+			emit_insn(ctx, amswapdbd, src, t1, t3);
 			break;
 		}
 		break;
@@ -524,6 +527,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
 			emit_insn(ctx, beq, t3, LOONGARCH_GPR_ZERO, -6);
 			emit_zext_32(ctx, r0, true);
 		}
+		emit_insn(ctx, dbar, DBAR_LLSC_MB);
 		break;
 	default:
 		pr_err_once("bpf-jit: invalid atomic read-modify-write opcode %02x\n", imm);
Signature · 1 line
2.25.1
reviewer Acked-by on v4 1/4
#2 lore raw

Re: [PATCH v4 1/4] LoongArch: bpf: Fix memory ordering for value-returning atomics

Quoted 8 lines · Chenguang Zhao

From: Chenguang Zhao <[email protected]>

Per the Linux Kernel Memory Model, value-returning atomic RMW operations must provide sequentially consistent ordering (a full memory barrier). On LoongArch, plain AMO instructions and bare ll/sc loops do not satisfy this requirement by themselves.

The subject line could be:
LoongArch: bpf: Align value-returning atomics with LKMM ordering

I think it's not a fix and we don't need a Fixes tag.

Other than that,

Quoted 165 lines

Update emit_atomic_rmw() to emit barrier-carrying instructions for all value-returning BPF atomics:

- BPF_FETCH (ADD/AND/OR/XOR): use am*_db.{b,h,w,d}
- BPF_XCHG: use amswap_db.{b,h,w,d}
- BPF_CMPXCHG: emit dbar 0x700 after the ll/sc loop, matching __WEAK_LLSC_MB in cmpxchg.h

Add the corresponding instruction encodings and emit helpers to inst.h. Non-value-returning RMW ops (plain BPF_ADD, BPF_AND, etc.) are left as weakly ordered, consistent with LKMM.

---
 arch/loongarch/include/asm/inst.h | 18 +++++++++++++++++
 arch/loongarch/net/bpf_jit.c      | 32 +++++++++++++++++--------------
 2 files changed, 36 insertions(+), 14 deletions(-)
arch/loongarch/include/asm/inst.h +18 -0
index 76b723590023..bdbc17d07110 100644
@@ -199,6 +199,10 @@ enum reg3_op {
        amswaph_op      = 0x70b9,
        amaddb_op       = 0x70ba,
        amaddh_op       = 0x70bb,
+       amswapdbb_op    = 0x70bc,
+       amswapdbh_op    = 0x70bd,
+       amadddbb_op     = 0x70be,
+       amadddbh_op     = 0x70bf,
        amswapw_op      = 0x70c0,
        amswapd_op      = 0x70c1,
        amaddw_op       = 0x70c2,
@@ -783,6 +787,20 @@ DEF_EMIT_REG3_FORMAT(amswapb, amswapb_op)
 DEF_EMIT_REG3_FORMAT(amswaph, amswaph_op)
 DEF_EMIT_REG3_FORMAT(amswapw, amswapw_op)
 DEF_EMIT_REG3_FORMAT(amswapd, amswapd_op)
+DEF_EMIT_REG3_FORMAT(amswapdbb, amswapdbb_op)
+DEF_EMIT_REG3_FORMAT(amswapdbh, amswapdbh_op)
+DEF_EMIT_REG3_FORMAT(amadddbb, amadddbb_op)
+DEF_EMIT_REG3_FORMAT(amadddbh, amadddbh_op)
+DEF_EMIT_REG3_FORMAT(amadddbw, amadddbw_op)
+DEF_EMIT_REG3_FORMAT(amadddbd, amadddbd_op)
+DEF_EMIT_REG3_FORMAT(amanddbw, amanddbw_op)
+DEF_EMIT_REG3_FORMAT(amanddbd, amanddbd_op)
+DEF_EMIT_REG3_FORMAT(amordbw, amordbw_op)
+DEF_EMIT_REG3_FORMAT(amordbd, amordbd_op)
+DEF_EMIT_REG3_FORMAT(amxordbw, amxordbw_op)
+DEF_EMIT_REG3_FORMAT(amxordbd, amxordbd_op)
+DEF_EMIT_REG3_FORMAT(amswapdbw, amswapdbw_op)
+DEF_EMIT_REG3_FORMAT(amswapdbd, amswapdbd_op)
 
 #define DEF_EMIT_REG3SA2_FORMAT(NAME, OP)                              \
 static inline void emit_##NAME(union loongarch_instruction *insn,      \
arch/loongarch/net/bpf_jit.c +18 -15
index 29c281bef28e..5417a5624327 100644
@@ -8,6 +8,9 @@
 #include <asm/asm-offsets.h>
 #include "bpf_jit.h"
 
+/* dbar hint for ll/sc completion ordering, see __WEAK_LLSC_MB */
+#define DBAR_LLSC_MB   0x700
+
 #define LOONGARCH_MAX_REG_ARGS 8
 
 #define LOONGARCH_SAVE_RA_NINSNS   1
@@ -433,7 +436,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
                                pr_err_once("bpf-jit: amadd.b instruction is not supported\n");
                                return -EINVAL;
                        }
-                       emit_insn(ctx, amaddb, src, t1, t3);
+                       emit_insn(ctx, amadddbb, src, t1, t3);
                        emit_zext_32(ctx, src, true);
                        break;
                case BPF_H:
@@ -441,39 +444,39 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
                                pr_err_once("bpf-jit: amadd.h instruction is not supported\n");
                                return -EINVAL;
                        }
-                       emit_insn(ctx, amaddh, src, t1, t3);
+                       emit_insn(ctx, amadddbh, src, t1, t3);
                        emit_zext_32(ctx, src, true);
                        break;
                case BPF_W:
-                       emit_insn(ctx, amaddw, src, t1, t3);
+                       emit_insn(ctx, amadddbw, src, t1, t3);
                        emit_zext_32(ctx, src, true);
                        break;
                case BPF_DW:
-                       emit_insn(ctx, amaddd, src, t1, t3);
+                       emit_insn(ctx, amadddbd, src, t1, t3);
                        break;
                }
                break;
        case BPF_AND | BPF_FETCH:
                if (isdw) {
-                       emit_insn(ctx, amandd, src, t1, t3);
+                       emit_insn(ctx, amanddbd, src, t1, t3);
                } else {
-                       emit_insn(ctx, amandw, src, t1, t3);
+                       emit_insn(ctx, amanddbw, src, t1, t3);
                        emit_zext_32(ctx, src, true);
                }
                break;
        case BPF_OR | BPF_FETCH:
                if (isdw) {
-                       emit_insn(ctx, amord, src, t1, t3);
+                       emit_insn(ctx, amordbd, src, t1, t3);
                } else {
-                       emit_insn(ctx, amorw, src, t1, t3);
+                       emit_insn(ctx, amordbw, src, t1, t3);
                        emit_zext_32(ctx, src, true);
                }
                break;
        case BPF_XOR | BPF_FETCH:
                if (isdw) {
-                       emit_insn(ctx, amxord, src, t1, t3);
+                       emit_insn(ctx, amxordbd, src, t1, t3);
                } else {
-                       emit_insn(ctx, amxorw, src, t1, t3);
+                       emit_insn(ctx, amxordbw, src, t1, t3);
                        emit_zext_32(ctx, src, true);
                }
                break;
@@ -485,7 +488,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
                                pr_err_once("bpf-jit: amswap.b instruction is not supported\n");
                                return -EINVAL;
                        }
-                       emit_insn(ctx, amswapb, src, t1, t3);
+                       emit_insn(ctx, amswapdbb, src, t1, t3);
                        emit_zext_32(ctx, src, true);
                        break;
                case BPF_H:
@@ -493,15 +496,15 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
                                pr_err_once("bpf-jit: amswap.h instruction is not supported\n");
                                return -EINVAL;
                        }
-                       emit_insn(ctx, amswaph, src, t1, t3);
+                       emit_insn(ctx, amswapdbh, src, t1, t3);
                        emit_zext_32(ctx, src, true);
                        break;
                case BPF_W:
-                       emit_insn(ctx, amswapw, src, t1, t3);
+                       emit_insn(ctx, amswapdbw, src, t1, t3);
                        emit_zext_32(ctx, src, true);
                        break;
                case BPF_DW:
-                       emit_insn(ctx, amswapd, src, t1, t3);
+                       emit_insn(ctx, amswapdbd, src, t1, t3);
                        break;
                }
                break;
@@ -524,6 +527,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
                        emit_insn(ctx, beq, t3, LOONGARCH_GPR_ZERO, -6);
                        emit_zext_32(ctx, r0, true);
                }
+               emit_insn(ctx, dbar, DBAR_LLSC_MB);
                break;
        default:
                pr_err_once("bpf-jit: invalid atomic read-modify-write opcode %02x\n", imm);
--

2.25.1

#3 lore raw

PATCH v4 2/4 LoongArch: bpf: Advertise JIT support for kptr xchg inline

In reply to Chenguang Zhao

From: Chenguang Zhao <[email protected]>

The BPF verifier can lower bpf_kptr_xchg() to BPF_XCHG when the JIT advertises ptr xchg support. With ordered amswap_db.* emission from the previous patch, declare that LoongArch bpf JIT supports this inlining.

---
 arch/loongarch/net/bpf_jit.c | 5 +++++
 1 file changed, 5 insertions(+)
arch/loongarch/net/bpf_jit.c +5 -0
index 5417a5624327..686d333146ce 100644
@@ -2396,6 +2396,11 @@ bool bpf_jit_supports_fsession(void)
 	return true;
 }
 
+bool bpf_jit_supports_ptr_xchg(void)
+{
+	return true;
+}
+
 /* Indicate the JIT backend supports mixing bpf2bpf and tailcalls. */
 bool bpf_jit_supports_subprog_tailcalls(void)
 {
Signature · 1 line
2.25.1
reviewer Acked-by on v4 2/4
#4 lore raw

Re: [PATCH v4 2/4] LoongArch: bpf: Advertise JIT support for kptr xchg inline

Quoted 7 lines · Chenguang Zhao

From: Chenguang Zhao <[email protected]>

The BPF verifier can lower bpf_kptr_xchg() to BPF_XCHG when the JIT advertises ptr xchg support. With ordered amswap_db.* emission from the previous patch, declare that LoongArch bpf JIT supports this inlining.

Quoted 24 lines
---
 arch/loongarch/net/bpf_jit.c | 5 +++++
 1 file changed, 5 insertions(+)
arch/loongarch/net/bpf_jit.c +5 -1
index 5417a5624327..686d333146ce 100644
@@ -2396,6 +2396,11 @@ bool bpf_jit_supports_fsession(void)
        return true;
 }
 
+bool bpf_jit_supports_ptr_xchg(void)
+{
+       return true;
+}
+
 /* Indicate the JIT backend supports mixing bpf2bpf and tailcalls. */
 bool bpf_jit_supports_subprog_tailcalls(void)
 {
--

2.25.1

#5 lore raw

PATCH v4 3/4 selftests/bpf: Enable kptr_xchg_inline test on LoongArch

In reply to Chenguang Zhao

From: Chenguang Zhao <[email protected]>

Run the kptr_xchg_inline functional test on LoongArch64 now that the bpf JIT can inline bpf_kptr_xchg() with correct memory ordering.

---
 tools/testing/selftests/bpf/prog_tests/kptr_xchg_inline.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/prog_tests/kptr_xchg_inline.c +2 -1
index 7def158da9eb..1215d6edd590 100644
@@ -14,7 +14,8 @@ void test_kptr_xchg_inline(void)
 	int err;
 
 #if !(defined(__x86_64__) || defined(__aarch64__) || \
-      (defined(__riscv) && __riscv_xlen == 64))
+	(defined(__riscv) && __riscv_xlen == 64) || \
+	(defined(__loongarch__) && __loongarch_grlen == 64))
 	test__skip();
 	return;
 #endif
Signature · 1 line
2.25.1
reviewer Acked-by on v4 3/4
#6 lore raw

Re: [PATCH v4 3/4] selftests/bpf: Enable kptr_xchg_inline test on LoongArch

Quoted 6 lines · Chenguang Zhao

From: Chenguang Zhao <[email protected]>

Run the kptr_xchg_inline functional test on LoongArch64 now that the bpf JIT can inline bpf_kptr_xchg() with correct memory ordering.

Quoted 22 lines
---
 tools/testing/selftests/bpf/prog_tests/kptr_xchg_inline.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/prog_tests/kptr_xchg_inline.c +2 -2
index 7def158da9eb..1215d6edd590 100644
@@ -14,7 +14,8 @@ void test_kptr_xchg_inline(void)
        int err;
 
 #if !(defined(__x86_64__) || defined(__aarch64__) || \
-      (defined(__riscv) && __riscv_xlen == 64))
+       (defined(__riscv) && __riscv_xlen == 64) || \
+       (defined(__loongarch__) && __loongarch_grlen == 64))
        test__skip();
        return;
 #endif
--

2.25.1

#7 lore raw

PATCH v4 4/4 selftests/bpf: Add kptr-xchg benchmark

In reply to Chenguang Zhao

From: Chenguang Zhao <[email protected]>

Add a bpf selftest benchmark that exercises bpf_kptr_xchg() in a tight loop so helper vs inlined JIT paths can be compared on supported architectures.

---
 tools/testing/selftests/bpf/Makefile          |  2 +
 tools/testing/selftests/bpf/bench.c           |  2 +
 .../selftests/bpf/benchs/bench_kptr_xchg.c    | 96 +++++++++++++++++++
 .../selftests/bpf/progs/kptr_xchg_bench.c     | 49 ++++++++++
 4 files changed, 149 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_kptr_xchg.c
 create mode 100644 tools/testing/selftests/bpf/progs/kptr_xchg_bench.c
tools/testing/selftests/bpf/Makefile +2 -0
index b642ee489ea6..6dfc81748584 100644
@@ -980,6 +980,7 @@ $(OUTPUT)/bench_lpm_trie_map.o: $(OUTPUT)/lpm_trie_bench.skel.h $(OUTPUT)/lpm_tr
 $(OUTPUT)/bench_bpf_nop.o: $(OUTPUT)/bpf_nop_bench.skel.h bench_bpf_timing.h
 $(OUTPUT)/bench_xdp_lb.o: $(OUTPUT)/xdp_lb_bench.skel.h bench_bpf_timing.h
 $(OUTPUT)/bench_bpf_timing.o: bench_bpf_timing.h
+$(OUTPUT)/bench_kptr_xchg.o: $(OUTPUT)/kptr_xchg_bench.skel.h
 $(OUTPUT)/bench.o: bench.h testing_helpers.h $(BPFOBJ)
 $(OUTPUT)/bench: LDLIBS += -lm
 $(OUTPUT)/bench: $(OUTPUT)/bench.o \
@@ -1005,6 +1006,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \
 		 $(OUTPUT)/bench_bpf_timing.o \
 		 $(OUTPUT)/bench_bpf_nop.o \
 		 $(OUTPUT)/bench_xdp_lb.o \
+		 $(OUTPUT)/bench_kptr_xchg.o \
 		 $(OUTPUT)/usdt_1.o \
 		 $(OUTPUT)/usdt_2.o \
 		 #
tools/testing/selftests/bpf/bench.c +2 -0
index 3d9d2cd7764b..550290d6701a 100644
@@ -582,6 +582,7 @@ extern const struct bench bench_lpm_trie_delete;
 extern const struct bench bench_lpm_trie_free;
 extern const struct bench bench_bpf_nop;
 extern const struct bench bench_xdp_lb;
+extern const struct bench bench_kptr_xchg;
 
 static const struct bench *benchs[] = {
 	&bench_count_global,
@@ -665,6 +666,7 @@ static const struct bench *benchs[] = {
 	&bench_lpm_trie_free,
 	&bench_bpf_nop,
 	&bench_xdp_lb,
+	&bench_kptr_xchg,
 };
 
 static void find_benchmark(void)
tools/testing/selftests/bpf/benchs/bench_kptr_xchg.c new file +96 -0

96 lines not on this page · raw message

tools/testing/selftests/bpf/progs/kptr_xchg_bench.c new file +49 -0

49 lines not on this page · raw message

Signature · 1 line
2.25.1
maintainer
#8 lore raw

Re: [PATCH v4 4/4] selftests/bpf: Add kptr-xchg benchmark

Quoted 6 lines · Chenguang Zhao

From: Chenguang Zhao <[email protected]>

Add a bpf selftest benchmark that exercises bpf_kptr_xchg() in a tight loop so helper vs inlined JIT paths can be compared on supported architectures.

Quoted 8 lines
---
  tools/testing/selftests/bpf/Makefile          |  2 +
  tools/testing/selftests/bpf/bench.c           |  2 +
  .../selftests/bpf/benchs/bench_kptr_xchg.c    | 96 +++++++++++++++++++
  .../selftests/bpf/progs/kptr_xchg_bench.c     | 49 ++++++++++
  4 files changed, 149 insertions(+)

create mode 100644 tools/testing/selftests/bpf/benchs/bench_kptr_xchg.c create mode 100644 tools/testing/selftests/bpf/progs/kptr_xchg_bench.c

Hi Chenguang,

Please send this generic selftest patch to the BPF mailing list. It should be routed through the bpf-next tree.

Also, please use the '[PATCH bpf-next]' subject prefix to specify that this is destined for the bpf-next tree.

For more info, please see:
https://www.kernel.org/doc/html/latest/bpf/bpf_devel_QA.html

Thanks,
Tiezhu

maintainer
#9 lore raw
In reply to Chenguang Zhao

Queued the first 3 patches, since the last one is not LoongArch-specific.

Huacai

Quoted 69 lines · Chenguang Zhao

From: Chenguang Zhao <[email protected]>

This series enables the BPF verifier to inline bpf_kptr_xchg() into BPF_XCHG on LoongArch64, and fixes the underlying JIT atomic ordering that makes such inlining safe.

The BPF verifier can lower bpf_kptr_xchg() to a single BPF_XCHG atomic when the JIT advertises ptr xchg support via bpf_jit_supports_ptr_xchg(). This removes helper-call overhead from the kptr exchange fast path. Inlining is only correct when the JITed exchange provides the same sequentially consistent ordering as the bpf_kptr_xchg() helper.

Patch 1 fixes memory ordering for all value-returning BPF atomic RMW operations emitted by the LoongArch bpf JIT. Per LKMM, value-returning atomic RMW must provide sequentially consistent ordering. Plain AMO instructions and bare ll/sc loops on LoongArch do not satisfy this by themselves:
- BPF_FETCH (ADD/AND/OR/XOR): switch to am*_db.{b,h,w,d}
- BPF_XCHG: switch to amswap_db.{b,h,w,d}
- BPF_CMPXCHG: emit dbar 0x700 after the ll/sc loop, matching __WEAK_LLSC_MB in cmpxchg.h
Non-value-returning RMW ops (plain BPF_ADD, BPF_AND, etc.) remain weakly ordered, consistent with LKMM. This fix is independent of kptr inlining and benefits all BPF programs using value-returning atomics on LoongArch.

Patch 2 implements bpf_jit_supports_ptr_xchg() so the verifier may inline bpf_kptr_xchg() on LoongArch64.

Patches 3 and 4 extend bpf selftests: functional coverage via kptr_xchg_inline, and an optional kptr-xchg benchmark to compare helper vs inlined paths.

Chenguang Zhao (4):
LoongArch: bpf: Fix memory ordering for value-returning atomics
LoongArch: bpf: Advertise JIT support for kptr xchg inline selftests/bpf: Enable kptr_xchg_inline test on LoongArch selftests/bpf: Add kptr-xchg benchmark

 arch/loongarch/include/asm/inst.h             | 18 ++++
 arch/loongarch/net/bpf_jit.c                  | 37 ++++---
 tools/testing/selftests/bpf/Makefile          |  2 +
 tools/testing/selftests/bpf/bench.c           |  2 +
 .../selftests/bpf/benchs/bench_kptr_xchg.c    | 96 +++++++++++++++++++
 .../bpf/prog_tests/kptr_xchg_inline.c         |  3 +-
 .../selftests/bpf/progs/kptr_xchg_bench.c     | 49 ++++++++++
 7 files changed, 192 insertions(+), 15 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_kptr_xchg.c
 create mode 100644 tools/testing/selftests/bpf/progs/kptr_xchg_bench.c

---
v4:
- Resend patches against new kernel.

v3:
- https://lore.kernel.org/all/[email protected]/

v2:
- https://lore.kernel.org/all/[email protected]/

v1:
- https://lore.kernel.org/all/[email protected]/

--
2.25.1

maintainer Acked-by Tested-by on the v4 series
#10 lore raw
In reply to Chenguang Zhao
Quoted 5 lines · Chenguang Zhao

From: Chenguang Zhao <[email protected]>

This series enables the BPF verifier to inline bpf_kptr_xchg() into BPF_XCHG on LoongArch64, and fixes the underlying JIT atomic ordering that makes such inlining safe.

The first three patches apply cleanly on top of the loongarch-next branch of linux-loongson.git.

The following selftest passed on LoongArch:

  # ./test_progs -t kptr_xchg_inline

Thanks,
Tiezhu