Skip to contentSkip to conversation

riscv, bpf: Fix signed operations and add 32 bit atomics

PATCH Superseded Kuan-Wei Chiu 2 participants 11 messages May 11 – Jun 22

Subsystems (3)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover riscv, bpf: Fix signed operations and add 32 bit atomics

Fix miscompiled signed operations and expands 32 bit atomic support in the RV32 BPF JIT.

The current implementation ignores the instruction offset field used by the bpf instruction set to specify BPF_SDIV/BPF_SMOD and BPF_MOVSX. This causes these operations to be treated as unsigned or zero-extended, leading to test_bpf failures. Fix this by checking the offset and emitting the correct instructions.

Additionally, we leverage the mandatory A extension to natively support 32 bit bpf atomics (and, or, xor, xchg) by mapping them directly to amo*.w instructions. BPF_CMPXCHG continues to fall back to the interpreter.

As a result, test_bpf.ko now runs with 0 failures, and the total number of successfully JIT'ed test cases increases from 843 to 902.
---
- Add missing Fixes tags.
- Fix memory ordering by emitting aq=1, rl=1

Kuan-Wei Chiu (3):
riscv, bpf: Fix support for BPF_SDIV and BPF_SMOD in RV32 JIT riscv, bpf: Fix support for BPF_MOVSX in RV32 JIT riscv, bpf: Add 32 bit atomic operations to RV32 JIT

 arch/riscv/net/bpf_jit_comp32.c | 101 +++++++++++++++++++++++++++-----
 1 file changed, 85 insertions(+), 16 deletions(-)
Signature · 1 line
2.54.0.563.g4f69b47b94-goog
#1 lore raw

PATCH v2 1/3 riscv, bpf: Fix support for BPF_SDIV and BPF_SMOD in RV32 JIT

The current rv32 bpf jit compiler incorrectly treats BPF_SDIV and BPF_SMOD as unsigned operations. The BPF instruction set allows signed division and modulo by reusing the BPF_DIV and BPF_MOD opcodes with the instruction offset set to 1.

Update the emit_alu_r32() function to accept an 'is_sdiv' variable and emit the correct div and rem instructions when the offset is 1.

Before this patch:

[   44.161771] test_bpf: #165 ALU_SDIV_X: -6 / 2 = -3 jited:1 ret 2147483645 != -3 (0x7ffffffd != 0xfffffffd)FAIL (1 times)
[   44.167385] test_bpf: #166 ALU_SDIV_K: -6 / 2 = -3 jited:1 ret 2147483645 != -3 (0x7ffffffd != 0xfffffffd)FAIL (1 times)
[   44.171053] test_bpf: #169 ALU_SMOD_X: -7 % 2 = -1 jited:1 ret 1 != -1 (0x1 != 0xffffffff)FAIL (1 times)
[   44.172081] test_bpf: #170 ALU_SMOD_K: -7 % 2 = -1 jited:1 ret 1 != -1 (0x1 != 0xffffffff)FAIL (1 times)

After this patch:

[   16.002192] test_bpf: #165 ALU_SDIV_X: -6 / 2 = -3 jited:1 95 PASS
[   16.002983] test_bpf: #166 ALU_SDIV_K: -6 / 2 = -3 jited:1 1059 PASS
[   16.017167] test_bpf: #169 ALU_SMOD_X: -7 % 2 = -1 jited:1 136 PASS
[   16.023002] test_bpf: #170 ALU_SMOD_K: -7 % 2 = -1 jited:1 109 PASS
Fixes ec0e2da95f72 ("bpf: Support new signed div/mod instructions.")
---
 arch/riscv/net/bpf_jit_comp32.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
arch/riscv/net/bpf_jit_comp32.c +14 -5
index 592dd86fbf81..7396899ea276 100644
@@ -509,7 +509,7 @@ static void emit_alu_r64(const s8 *dst, const s8 *src,
 }
 
 static void emit_alu_r32(const s8 *dst, const s8 *src,
-			 struct rv_jit_context *ctx, const u8 op)
+			 struct rv_jit_context *ctx, const u8 op, bool is_sdiv)
 {
 	const s8 *tmp1 = bpf2rv32[TMP_REG_1];
 	const s8 *tmp2 = bpf2rv32[TMP_REG_2];
@@ -539,10 +539,16 @@ static void emit_alu_r32(const s8 *dst, const s8 *src,
 		emit(rv_mul(lo(rd), lo(rd), lo(rs)), ctx);
 		break;
 	case BPF_DIV:
-		emit(rv_divu(lo(rd), lo(rd), lo(rs)), ctx);
+		if (is_sdiv)
+			emit(rv_div(lo(rd), lo(rd), lo(rs)), ctx);
+		else
+			emit(rv_divu(lo(rd), lo(rd), lo(rs)), ctx);
 		break;
 	case BPF_MOD:
-		emit(rv_remu(lo(rd), lo(rd), lo(rs)), ctx);
+		if (is_sdiv)
+			emit(rv_rem(lo(rd), lo(rd), lo(rs)), ctx);
+		else
+			emit(rv_remu(lo(rd), lo(rd), lo(rs)), ctx);
 		break;
 	case BPF_LSH:
 		emit(rv_sll(lo(rd), lo(rd), lo(rs)), ctx);
@@ -959,6 +965,7 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 	u8 code = insn->code;
 	s16 off = insn->off;
 	s32 imm = insn->imm;
+	bool is_sdiv = false;
 
 	const s8 *dst = bpf2rv32[insn->dst_reg];
 	const s8 *src = bpf2rv32[insn->src_reg];
@@ -1041,7 +1048,9 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 			emit_imm32(tmp2, imm, ctx);
 			src = tmp2;
 		}
-		emit_alu_r32(dst, src, ctx, BPF_OP(code));
+		if ((BPF_OP(code) == BPF_DIV || BPF_OP(code) == BPF_MOD) && insn->off == 1)
+			is_sdiv = true;
+		emit_alu_r32(dst, src, ctx, BPF_OP(code), is_sdiv);
 		break;
 
 	case BPF_ALU | BPF_MOV | BPF_K:
@@ -1065,7 +1074,7 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 		 * src is ignored---choose tmp2 as a dummy register since it
 		 * is not on the stack.
 		 */
-		emit_alu_r32(dst, tmp2, ctx, BPF_OP(code));
+		emit_alu_r32(dst, tmp2, ctx, BPF_OP(code), false);
 		break;
 
 	case BPF_ALU | BPF_END | BPF_FROM_LE:
Signature · 1 line
2.54.0.563.g4f69b47b94-goog
maintainer
#2 lore raw

Re: [PATCH bpf-next v2 1/3] riscv, bpf: Fix support for BPF_SDIV and BPF_SMOD in RV32 JIT

lgtm, just some nits

Quoted 31 lines · Kuan-Wei Chiu

The current rv32 bpf jit compiler incorrectly treats BPF_SDIV and BPF_SMOD as unsigned operations. The BPF instruction set allows signed division and modulo by reusing the BPF_DIV and BPF_MOD opcodes with the instruction offset set to 1.

Update the emit_alu_r32() function to accept an 'is_sdiv' variable and emit the correct div and rem instructions when the offset is 1.

Before this patch:

[   44.161771] test_bpf: #165 ALU_SDIV_X: -6 / 2 = -3 jited:1 ret 2147483645 != -3 (0x7ffffffd != 0xfffffffd)FAIL (1 times)
[   44.167385] test_bpf: #166 ALU_SDIV_K: -6 / 2 = -3 jited:1 ret 2147483645 != -3 (0x7ffffffd != 0xfffffffd)FAIL (1 times)
[   44.171053] test_bpf: #169 ALU_SMOD_X: -7 % 2 = -1 jited:1 ret 1 != -1 (0x1 != 0xffffffff)FAIL (1 times)
[   44.172081] test_bpf: #170 ALU_SMOD_K: -7 % 2 = -1 jited:1 ret 1 != -1 (0x1 != 0xffffffff)FAIL (1 times)

After this patch:

[   16.002192] test_bpf: #165 ALU_SDIV_X: -6 / 2 = -3 jited:1 95 PASS
[   16.002983] test_bpf: #166 ALU_SDIV_K: -6 / 2 = -3 jited:1 1059 PASS
[   16.017167] test_bpf: #169 ALU_SMOD_X: -7 % 2 = -1 jited:1 136 PASS
[   16.023002] test_bpf: #170 ALU_SMOD_K: -7 % 2 = -1 jited:1 109 PASS
Fixes ec0e2da95f72 ("bpf: Support new signed div/mod instructions.")
---
  arch/riscv/net/bpf_jit_comp32.c | 19 ++++++++++++++-----
  1 file changed, 14 insertions(+), 5 deletions(-)
arch/riscv/net/bpf_jit_comp32.c +0 -0
index 592dd86fbf81..7396899ea276 100644

arch/riscv/net/bpf_jit_comp32.c static void emit_alu_r64(const s8 *dst, const s8 *src, · line 512

Quoted 5 lines
arch/riscv/net/bpf_jit_comp32.c +1 -1
@@ -509,7 +509,7 @@ static void emit_alu_r64(const s8 *dst, const s8 *src,
  }
 
  static void emit_alu_r32(const s8 *dst, const s8 *src,
-			 struct rv_jit_context *ctx, const u8 op)
+			 struct rv_jit_context *ctx, const u8 op, bool is_sdiv)

let's combind op and is_sdiv to const struct bpf_insn *insn, and better to name is_signed = insn->off == 1

Quoted 9 lines
arch/riscv/net/bpf_jit_comp32.c +1 -1
  {
  	const s8 *tmp1 = bpf2rv32[TMP_REG_1];
  	const s8 *tmp2 = bpf2rv32[TMP_REG_2];
@@ -539,10 +539,16 @@ static void emit_alu_r32(const s8 *dst, const s8 *src,
  		emit(rv_mul(lo(rd), lo(rd), lo(rs)), ctx);
  		break;
  	case BPF_DIV:
-		emit(rv_divu(lo(rd), lo(rd), lo(rs)), ctx);
+		if (is_sdiv)

arch/riscv/net/bpf_jit_comp32.c static void emit_alu_r32(const s8 *dst, const s8 *src, · line 551

Quoted 10 lines
arch/riscv/net/bpf_jit_comp32.c +7 -1
+			emit(rv_div(lo(rd), lo(rd), lo(rs)), ctx);
+		else
+			emit(rv_divu(lo(rd), lo(rd), lo(rs)), ctx);
  		break;
  	case BPF_MOD:
-		emit(rv_remu(lo(rd), lo(rd), lo(rs)), ctx);
+		if (is_sdiv)
+			emit(rv_rem(lo(rd), lo(rd), lo(rs)), ctx);
+		else
+			emit(rv_remu(lo(rd), lo(rd), lo(rs)), ctx);

how about
emit(is_signed ? rv_rem(lo(rd) : rv_remu(lo(rd), lo(rd), lo(rs)), ctx)

Quoted 31 lines
arch/riscv/net/bpf_jit_comp32.c +5 -2
  		break;
  	case BPF_LSH:
  		emit(rv_sll(lo(rd), lo(rd), lo(rs)), ctx);
@@ -959,6 +965,7 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
  	u8 code = insn->code;
  	s16 off = insn->off;
  	s32 imm = insn->imm;
+	bool is_sdiv = false;
 
  	const s8 *dst = bpf2rv32[insn->dst_reg];
  	const s8 *src = bpf2rv32[insn->src_reg];
@@ -1041,7 +1048,9 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
  			emit_imm32(tmp2, imm, ctx);
  			src = tmp2;
  		}
-		emit_alu_r32(dst, src, ctx, BPF_OP(code));
+		if ((BPF_OP(code) == BPF_DIV || BPF_OP(code) == BPF_MOD) && insn->off == 1)
+			is_sdiv = true;
+		emit_alu_r32(dst, src, ctx, BPF_OP(code), is_sdiv);
  		break;
 
  	case BPF_ALU | BPF_MOV | BPF_K:
@@ -1065,7 +1074,7 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
  		 * src is ignored---choose tmp2 as a dummy register since it
  		 * is not on the stack.
  		 */
-		emit_alu_r32(dst, tmp2, ctx, BPF_OP(code));
+		emit_alu_r32(dst, tmp2, ctx, BPF_OP(code), false);
  		break;
 
  	case BPF_ALU | BPF_END | BPF_FROM_LE:
#3 lore raw

PATCH v2 2/3 riscv, bpf: Fix support for BPF_MOVSX in RV32 JIT

In reply to Kuan-Wei Chiu

The current rv32 bpf jit compiler incorrectly treats BPF_MOVSX as a standard zero-extended move operation. The bpf instruction set allows sign-extension moves by reusing the BPF_MOV opcode with the instruction offset set to 8, 16, or 32.

Update the bpf_jit_emit_insn() function to check the offset field for both ALU and ALU64 MOV operations. If the offset is non-zero, emit the correct slli and srai instructions to perform the sign extension.

Before this patch:

[   19.549705] test_bpf: #82 ALU_MOVSX | BPF_B jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
[   19.551354] test_bpf: #83 ALU_MOVSX | BPF_H jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
[   19.552576] test_bpf: #84 ALU64_MOVSX | BPF_B jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
[   19.553542] test_bpf: #85 ALU64_MOVSX | BPF_H jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
[   19.554807] test_bpf: #86 ALU64_MOVSX | BPF_W jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)

After this patch:

[   17.931172] test_bpf: #82 ALU_MOVSX | BPF_B jited:1 125 PASS
[   17.932198] test_bpf: #83 ALU_MOVSX | BPF_H jited:1 124 PASS
[   17.933039] test_bpf: #84 ALU64_MOVSX | BPF_B jited:1 124 PASS
[   17.933918] test_bpf: #85 ALU64_MOVSX | BPF_H jited:1 124 PASS
[   17.934751] test_bpf: #86 ALU64_MOVSX | BPF_W jited:1 122 PASS
Fixes 8100928c8814 ("bpf: Support new sign-extension mov insns")
---
 arch/riscv/net/bpf_jit_comp32.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
arch/riscv/net/bpf_jit_comp32.c +32 -0
index 7396899ea276..f8509950fed4 100644
@@ -974,6 +974,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 
 	switch (code) {
 	case BPF_ALU64 | BPF_MOV | BPF_X:
+		if (insn->off != 0) {
+			const s8 *rd = bpf_get_reg64(dst, tmp1, ctx);
+			const s8 *rs = bpf_get_reg64(src, tmp2, ctx);
+
+			if (insn->off == 8) {
+				emit(rv_slli(lo(rd), lo(rs), 24), ctx);
+				emit(rv_srai(lo(rd), lo(rd), 24), ctx);
+			} else if (insn->off == 16) {
+				emit(rv_slli(lo(rd), lo(rs), 16), ctx);
+				emit(rv_srai(lo(rd), lo(rd), 16), ctx);
+			} else {
+				emit(rv_addi(lo(rd), lo(rs), 0), ctx);
+			}
+			emit(rv_srai(hi(rd), lo(rd), 31), ctx);
+			bpf_put_reg64(dst, rd, ctx);
+			break;
+		}
+		fallthrough;
 
 	case BPF_ALU64 | BPF_ADD | BPF_X:
 	case BPF_ALU64 | BPF_ADD | BPF_K:
@@ -1024,6 +1042,20 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 			emit_zext64(dst, ctx);
 			break;
 		}
+		if (insn->off != 0) {
+			const s8 *rd = bpf_get_reg32(dst, tmp1, ctx);
+			const s8 *rs = bpf_get_reg32(src, tmp2, ctx);
+
+			if (insn->off == 8) {
+				emit(rv_slli(lo(rd), lo(rs), 24), ctx);
+				emit(rv_srai(lo(rd), lo(rd), 24), ctx);
+			} else if (insn->off == 16) {
+				emit(rv_slli(lo(rd), lo(rs), 16), ctx);
+				emit(rv_srai(lo(rd), lo(rd), 16), ctx);
+			}
+			bpf_put_reg32(dst, rd, ctx);
+			break;
+		}
 		fallthrough;
 
 	case BPF_ALU | BPF_ADD | BPF_X:
Signature · 1 line
2.54.0.563.g4f69b47b94-goog
maintainer Reviewed-by on v2 2/3
#4 lore raw

Re: [PATCH bpf-next v2 2/3] riscv, bpf: Fix support for BPF_MOVSX in RV32 JIT

Quoted 16 lines · Kuan-Wei Chiu

The current rv32 bpf jit compiler incorrectly treats BPF_MOVSX as a standard zero-extended move operation. The bpf instruction set allows sign-extension moves by reusing the BPF_MOV opcode with the instruction offset set to 8, 16, or 32.

Update the bpf_jit_emit_insn() function to check the offset field for both ALU and ALU64 MOV operations. If the offset is non-zero, emit the correct slli and srai instructions to perform the sign extension.

Before this patch:

[   19.549705] test_bpf: #82 ALU_MOVSX | BPF_B jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
[   19.551354] test_bpf: #83 ALU_MOVSX | BPF_H jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
[   19.552576] test_bpf: #84 ALU64_MOVSX | BPF_B jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
[   19.553542] test_bpf: #85 ALU64_MOVSX | BPF_H jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
[   19.554807] test_bpf: #86 ALU64_MOVSX | BPF_W jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
Quoted 8 lines

After this patch:

[   17.931172] test_bpf: #82 ALU_MOVSX | BPF_B jited:1 125 PASS
[   17.932198] test_bpf: #83 ALU_MOVSX | BPF_H jited:1 124 PASS
[   17.933039] test_bpf: #84 ALU64_MOVSX | BPF_B jited:1 124 PASS
[   17.933918] test_bpf: #85 ALU64_MOVSX | BPF_H jited:1 124 PASS
[   17.934751] test_bpf: #86 ALU64_MOVSX | BPF_W jited:1 122 PASS
Fixes 8100928c8814 ("bpf: Support new sign-extension mov insns")

not a fix, the fixes tag should remove. And the title should Add support for xxx

others lgtm,

Quoted 55 lines
---
  arch/riscv/net/bpf_jit_comp32.c | 32 ++++++++++++++++++++++++++++++++
  1 file changed, 32 insertions(+)
arch/riscv/net/bpf_jit_comp32.c +32 -0
index 7396899ea276..f8509950fed4 100644
@@ -974,6 +974,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 
  	switch (code) {
  	case BPF_ALU64 | BPF_MOV | BPF_X:
+		if (insn->off != 0) {
+			const s8 *rd = bpf_get_reg64(dst, tmp1, ctx);
+			const s8 *rs = bpf_get_reg64(src, tmp2, ctx);
+
+			if (insn->off == 8) {
+				emit(rv_slli(lo(rd), lo(rs), 24), ctx);
+				emit(rv_srai(lo(rd), lo(rd), 24), ctx);
+			} else if (insn->off == 16) {
+				emit(rv_slli(lo(rd), lo(rs), 16), ctx);
+				emit(rv_srai(lo(rd), lo(rd), 16), ctx);
+			} else {
+				emit(rv_addi(lo(rd), lo(rs), 0), ctx);
+			}
+			emit(rv_srai(hi(rd), lo(rd), 31), ctx);
+			bpf_put_reg64(dst, rd, ctx);
+			break;
+		}
+		fallthrough;
 
  	case BPF_ALU64 | BPF_ADD | BPF_X:
  	case BPF_ALU64 | BPF_ADD | BPF_K:
@@ -1024,6 +1042,20 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
  			emit_zext64(dst, ctx);
  			break;
  		}
+		if (insn->off != 0) {
+			const s8 *rd = bpf_get_reg32(dst, tmp1, ctx);
+			const s8 *rs = bpf_get_reg32(src, tmp2, ctx);
+
+			if (insn->off == 8) {
+				emit(rv_slli(lo(rd), lo(rs), 24), ctx);
+				emit(rv_srai(lo(rd), lo(rd), 24), ctx);
+			} else if (insn->off == 16) {
+				emit(rv_slli(lo(rd), lo(rs), 16), ctx);
+				emit(rv_srai(lo(rd), lo(rd), 16), ctx);
+			}
+			bpf_put_reg32(dst, rd, ctx);
+			break;
+		}
  		fallthrough;
 
  	case BPF_ALU | BPF_ADD | BPF_X:
#5 lore raw

Re: [PATCH bpf-next v2 2/3] riscv, bpf: Fix support for BPF_MOVSX in RV32 JIT

Hi Lehui,

Quoted 26 lines · Pu Lehui
Quoted 23 lines · Kuan-Wei Chiu

The current rv32 bpf jit compiler incorrectly treats BPF_MOVSX as a standard zero-extended move operation. The bpf instruction set allows sign-extension moves by reusing the BPF_MOV opcode with the instruction offset set to 8, 16, or 32.

Update the bpf_jit_emit_insn() function to check the offset field for both ALU and ALU64 MOV operations. If the offset is non-zero, emit the correct slli and srai instructions to perform the sign extension.

Before this patch:

[   19.549705] test_bpf: #82 ALU_MOVSX | BPF_B jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
[   19.551354] test_bpf: #83 ALU_MOVSX | BPF_H jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
[   19.552576] test_bpf: #84 ALU64_MOVSX | BPF_B jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
[   19.553542] test_bpf: #85 ALU64_MOVSX | BPF_H jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
[   19.554807] test_bpf: #86 ALU64_MOVSX | BPF_W jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)

After this patch:

[   17.931172] test_bpf: #82 ALU_MOVSX | BPF_B jited:1 125 PASS
[   17.932198] test_bpf: #83 ALU_MOVSX | BPF_H jited:1 124 PASS
[   17.933039] test_bpf: #84 ALU64_MOVSX | BPF_B jited:1 124 PASS
[   17.933918] test_bpf: #85 ALU64_MOVSX | BPF_H jited:1 124 PASS
[   17.934751] test_bpf: #86 ALU64_MOVSX | BPF_W jited:1 122 PASS
Quoted 4 lines
Quoted 1 line
Fixes 8100928c8814 ("bpf: Support new sign-extension mov insns")

not a fix, the fixes tag should remove. And the title should Add support for xxx

Thanks for the review.

I did consider using 'Add support' instead of 'Fix' for the title. I went with 'Fix' because when the JIT doesn't support a BPF instruction, it's supposed to return an error code to trigger a fallback to the interpreter. Right now, it doesn't return an error code and just executes it anyway, producing results that violate the spec and breaking bpf_test.

Regards,
Kuan-Wei

Quoted 57 lines

others lgtm,

Quoted 52 lines
---
  arch/riscv/net/bpf_jit_comp32.c | 32 ++++++++++++++++++++++++++++++++
  1 file changed, 32 insertions(+)
arch/riscv/net/bpf_jit_comp32.c +32 -0
index 7396899ea276..f8509950fed4 100644
@@ -974,6 +974,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
  	switch (code) {
  	case BPF_ALU64 | BPF_MOV | BPF_X:
+		if (insn->off != 0) {
+			const s8 *rd = bpf_get_reg64(dst, tmp1, ctx);
+			const s8 *rs = bpf_get_reg64(src, tmp2, ctx);
+
+			if (insn->off == 8) {
+				emit(rv_slli(lo(rd), lo(rs), 24), ctx);
+				emit(rv_srai(lo(rd), lo(rd), 24), ctx);
+			} else if (insn->off == 16) {
+				emit(rv_slli(lo(rd), lo(rs), 16), ctx);
+				emit(rv_srai(lo(rd), lo(rd), 16), ctx);
+			} else {
+				emit(rv_addi(lo(rd), lo(rs), 0), ctx);
+			}
+			emit(rv_srai(hi(rd), lo(rd), 31), ctx);
+			bpf_put_reg64(dst, rd, ctx);
+			break;
+		}
+		fallthrough;
  	case BPF_ALU64 | BPF_ADD | BPF_X:
  	case BPF_ALU64 | BPF_ADD | BPF_K:
@@ -1024,6 +1042,20 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
  			emit_zext64(dst, ctx);
  			break;
  		}
+		if (insn->off != 0) {
+			const s8 *rd = bpf_get_reg32(dst, tmp1, ctx);
+			const s8 *rs = bpf_get_reg32(src, tmp2, ctx);
+
+			if (insn->off == 8) {
+				emit(rv_slli(lo(rd), lo(rs), 24), ctx);
+				emit(rv_srai(lo(rd), lo(rd), 24), ctx);
+			} else if (insn->off == 16) {
+				emit(rv_slli(lo(rd), lo(rs), 16), ctx);
+				emit(rv_srai(lo(rd), lo(rd), 16), ctx);
+			}
+			bpf_put_reg32(dst, rd, ctx);
+			break;
+		}
  		fallthrough;
  	case BPF_ALU | BPF_ADD | BPF_X:
#6 lore raw

PATCH v2 3/3 riscv, bpf: Add 32 bit atomic operations to RV32 JIT

In reply to Kuan-Wei Chiu

The RV32 BPF JIT compiler currently only supports the BPF_ADD atomic operation. Other 32 bit atomic operations (and, or, xor, xchg) and their BPF_FETCH variants are not supported and gracefully fall back to the interpreter.

Since the RISC-V A extension is required for Linux on RV32, we can natively support these 32-bit BPF atomic operations by mapping them directly to the corresponding RISC-V amo*.w instructions.

Implement BPF_ADD, BPF_AND, BPF_OR, BPF_XOR, and BPF_XCHG with and without BPF_FETCH. BPF_CMPXCHG requires a more complex lr.w/sc.w loop and is left to fall back to the interpreter.

Before this patch:

[  138.862161] test_bpf: Summary: 1054 PASSED, 0 FAILED, [843/1042 JIT'ed]

After this patch:

[  157.024124] test_bpf: Summary: 1054 PASSED, 0 FAILED, [902/1042 JIT'ed]
---
 arch/riscv/net/bpf_jit_comp32.c | 50 +++++++++++++++++++++++++--------
 1 file changed, 39 insertions(+), 11 deletions(-)
arch/riscv/net/bpf_jit_comp32.c +39 -11
index f8509950fed4..7fd726a09b26 100644
@@ -877,7 +877,7 @@ static int emit_load_r64(const s8 *dst, const s8 *src, s16 off,
 
 static int emit_store_r64(const s8 *dst, const s8 *src, s16 off,
 			  struct rv_jit_context *ctx, const u8 size,
-			  const u8 mode)
+			  const u8 mode, s32 imm)
 {
 	const s8 *tmp1 = bpf2rv32[TMP_REG_1];
 	const s8 *tmp2 = bpf2rv32[TMP_REG_2];
@@ -902,11 +902,43 @@ static int emit_store_r64(const s8 *dst, const s8 *src, s16 off,
 		case BPF_MEM:
 			emit(rv_sw(RV_REG_T0, 0, lo(rs)), ctx);
 			break;
-		case BPF_ATOMIC: /* Only BPF_ADD supported */
-			emit(rv_amoadd_w(RV_REG_ZERO, lo(rs), RV_REG_T0, 0, 0),
-			     ctx);
+		case BPF_ATOMIC:
+		{
+			bool is_fetch = (imm & BPF_FETCH) || (imm == BPF_XCHG);
+			s8 fetch_reg = is_fetch ? lo(rs) : RV_REG_ZERO;
+			int aq = is_fetch ? 1 : 0;
+			int rl = is_fetch ? 1 : 0;
+
+			switch (imm) {
+			case BPF_ADD:
+			case BPF_ADD | BPF_FETCH:
+				emit(rv_amoadd_w(fetch_reg, lo(rs), RV_REG_T0, aq, rl), ctx);
+				break;
+			case BPF_AND:
+			case BPF_AND | BPF_FETCH:
+				emit(rv_amoand_w(fetch_reg, lo(rs), RV_REG_T0, aq, rl), ctx);
+				break;
+			case BPF_OR:
+			case BPF_OR | BPF_FETCH:
+				emit(rv_amoor_w(fetch_reg, lo(rs), RV_REG_T0, aq, rl), ctx);
+				break;
+			case BPF_XOR:
+			case BPF_XOR | BPF_FETCH:
+				emit(rv_amoxor_w(fetch_reg, lo(rs), RV_REG_T0, aq, rl), ctx);
+				break;
+			case BPF_XCHG:
+				emit(rv_amoswap_w(fetch_reg, lo(rs), RV_REG_T0, aq, rl), ctx);
+				break;
+			default:
+				return -1;
+			}
+			if (is_fetch) {
+				emit(rv_addi(hi(rs), RV_REG_ZERO, 0), ctx);
+				bpf_put_reg64(src, rs, ctx);
+			}
 			break;
 		}
+		}
 		break;
 	case BPF_DW:
 		emit(rv_sw(RV_REG_T0, 0, lo(rs)), ctx);
@@ -1308,20 +1340,16 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 		}
 
 		if (emit_store_r64(dst, src, off, ctx, BPF_SIZE(code),
-				   BPF_MODE(code)))
+				   BPF_MODE(code), 0))
 			return -1;
 		break;
 
 	case BPF_STX | BPF_ATOMIC | BPF_W:
-		if (insn->imm != BPF_ADD) {
-			pr_info_once(
-				"bpf-jit: not supported: atomic operation %02x ***\n",
-				insn->imm);
+		if (insn->imm == BPF_CMPXCHG)
 			return -EFAULT;
-		}
 
 		if (emit_store_r64(dst, src, off, ctx, BPF_SIZE(code),
-				   BPF_MODE(code)))
+				   BPF_MODE(code), insn->imm))
 			return -1;
 		break;
 
Signature · 1 line
2.54.0.563.g4f69b47b94-goog
maintainer
#7 lore raw

Re: [PATCH bpf-next v2 3/3] riscv, bpf: Add 32 bit atomic operations to RV32 JIT

Quoted 68 lines · Kuan-Wei Chiu

The RV32 BPF JIT compiler currently only supports the BPF_ADD atomic operation. Other 32 bit atomic operations (and, or, xor, xchg) and their BPF_FETCH variants are not supported and gracefully fall back to the interpreter.

Since the RISC-V A extension is required for Linux on RV32, we can natively support these 32-bit BPF atomic operations by mapping them directly to the corresponding RISC-V amo*.w instructions.

Implement BPF_ADD, BPF_AND, BPF_OR, BPF_XOR, and BPF_XCHG with and without BPF_FETCH. BPF_CMPXCHG requires a more complex lr.w/sc.w loop and is left to fall back to the interpreter.

Before this patch:

[  138.862161] test_bpf: Summary: 1054 PASSED, 0 FAILED, [843/1042 JIT'ed]

After this patch:

[  157.024124] test_bpf: Summary: 1054 PASSED, 0 FAILED, [902/1042 JIT'ed]
---
  arch/riscv/net/bpf_jit_comp32.c | 50 +++++++++++++++++++++++++--------
  1 file changed, 39 insertions(+), 11 deletions(-)
arch/riscv/net/bpf_jit_comp32.c +25 -4
index f8509950fed4..7fd726a09b26 100644
@@ -877,7 +877,7 @@ static int emit_load_r64(const s8 *dst, const s8 *src, s16 off,
 
  static int emit_store_r64(const s8 *dst, const s8 *src, s16 off,
  			  struct rv_jit_context *ctx, const u8 size,
-			  const u8 mode)
+			  const u8 mode, s32 imm)
  {
  	const s8 *tmp1 = bpf2rv32[TMP_REG_1];
  	const s8 *tmp2 = bpf2rv32[TMP_REG_2];
@@ -902,11 +902,43 @@ static int emit_store_r64(const s8 *dst, const s8 *src, s16 off,
  		case BPF_MEM:
  			emit(rv_sw(RV_REG_T0, 0, lo(rs)), ctx);
  			break;
-		case BPF_ATOMIC: /* Only BPF_ADD supported */
-			emit(rv_amoadd_w(RV_REG_ZERO, lo(rs), RV_REG_T0, 0, 0),
-			     ctx);
+		case BPF_ATOMIC:
+		{
+			bool is_fetch = (imm & BPF_FETCH) || (imm == BPF_XCHG);
+			s8 fetch_reg = is_fetch ? lo(rs) : RV_REG_ZERO;
+			int aq = is_fetch ? 1 : 0;
+			int rl = is_fetch ? 1 : 0;
+
+			switch (imm) {
+			case BPF_ADD:
+			case BPF_ADD | BPF_FETCH:
+				emit(rv_amoadd_w(fetch_reg, lo(rs), RV_REG_T0, aq, rl), ctx);
+				break;
+			case BPF_AND:
+			case BPF_AND | BPF_FETCH:
+				emit(rv_amoand_w(fetch_reg, lo(rs), RV_REG_T0, aq, rl), ctx);
+				break;
+			case BPF_OR:
+			case BPF_OR | BPF_FETCH:
+				emit(rv_amoor_w(fetch_reg, lo(rs), RV_REG_T0, aq, rl), ctx);
+				break;
+			case BPF_XOR:
+			case BPF_XOR | BPF_FETCH:
+				emit(rv_amoxor_w(fetch_reg, lo(rs), RV_REG_T0, aq, rl), ctx);
+				break;

arch/riscv/net/bpf_jit_comp32.c static int emit_store_r64(const s8 *dst, const s8 *src, s16 off, · line 938

Quoted 10 lines
arch/riscv/net/bpf_jit_comp32.c +9 -0
+			case BPF_XCHG:
+				emit(rv_amoswap_w(fetch_reg, lo(rs), RV_REG_T0, aq, rl), ctx);
+				break;
+			default:
+				return -1;
+			}
+			if (is_fetch) {
+				emit(rv_addi(hi(rs), RV_REG_ZERO, 0), ctx);
+				bpf_put_reg64(src, rs, ctx); > +			}
  			break;

rv32 code looks elegant, let's factor out an atomic function

Quoted 6 lines
arch/riscv/net/bpf_jit_comp32.c +1 -0
  		}
+		}
  		break;
  	case BPF_DW:
  		emit(rv_sw(RV_REG_T0, 0, lo(rs)), ctx);

arch/riscv/net/bpf_jit_comp32.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1316

Quoted 10 lines
arch/riscv/net/bpf_jit_comp32.c +1 -2
@@ -1308,20 +1340,16 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
  		}
 
  		if (emit_store_r64(dst, src, off, ctx, BPF_SIZE(code),
-				   BPF_MODE(code)))
+				   BPF_MODE(code), 0))
  			return -1;
  		break;
 
  	case BPF_STX | BPF_ATOMIC | BPF_W:
-		if (insn->imm != BPF_ADD) {

just insn->imm == BPF_CMPXCHG here, and not miss the warning message.

Quoted 13 lines
arch/riscv/net/bpf_jit_comp32.c +2 -5
-			pr_info_once(
-				"bpf-jit: not supported: atomic operation %02x ***\n",
-				insn->imm);
+		if (insn->imm == BPF_CMPXCHG)
  			return -EFAULT;
-		}
 
  		if (emit_store_r64(dst, src, off, ctx, BPF_SIZE(code),
-				   BPF_MODE(code)))
+				   BPF_MODE(code), insn->imm))
  			return -1;
  		break;
 
#8 lore raw
In reply to Kuan-Wei Chiu

Hi,

Quoted 9 lines · Kuan-Wei Chiu

Fix miscompiled signed operations and expands 32 bit atomic support in the RV32 BPF JIT.

The current implementation ignores the instruction offset field used by the bpf instruction set to specify BPF_SDIV/BPF_SMOD and BPF_MOVSX. This causes these operations to be treated as unsigned or zero-extended, leading to test_bpf failures. Fix this by checking the offset and emitting the correct instructions.

Quoted 7 lines

Additionally, we leverage the mandatory A extension to natively support 32 bit bpf atomics (and, or, xor, xchg) by mapping them directly to amo*.w instructions. BPF_CMPXCHG continues to fall back to the interpreter.

As a result, test_bpf.ko now runs with 0 failures, and the total number of successfully JIT'ed test cases increases from 843 to 902.

Is there anything else I need to do for this patchset to land?

Regards,
Kuan-wei

Quoted 15 lines

---
- Add missing Fixes tags.
- Fix memory ordering by emitting aq=1, rl=1

Kuan-Wei Chiu (3):
riscv, bpf: Fix support for BPF_SDIV and BPF_SMOD in RV32 JIT riscv, bpf: Fix support for BPF_MOVSX in RV32 JIT riscv, bpf: Add 32 bit atomic operations to RV32 JIT

 arch/riscv/net/bpf_jit_comp32.c | 101 +++++++++++++++++++++++++++-----
 1 file changed, 85 insertions(+), 16 deletions(-)
Signature · 1 line
2.54.0.563.g4f69b47b94-goog
maintainer
#9 lore raw
Quoted 17 lines · Kuan-Wei Chiu

Hi,

Quoted 14 lines · Kuan-Wei Chiu

Fix miscompiled signed operations and expands 32 bit atomic support in the RV32 BPF JIT.

The current implementation ignores the instruction offset field used by the bpf instruction set to specify BPF_SDIV/BPF_SMOD and BPF_MOVSX. This causes these operations to be treated as unsigned or zero-extended, leading to test_bpf failures. Fix this by checking the offset and emitting the correct instructions.

Additionally, we leverage the mandatory A extension to natively support 32 bit bpf atomics (and, or, xor, xchg) by mapping them directly to amo*.w instructions. BPF_CMPXCHG continues to fall back to the interpreter.

Quoted 4 lines
Quoted 2 lines

As a result, test_bpf.ko now runs with 0 failures, and the total number of successfully JIT'ed test cases increases from 843 to 902.

Is there anything else I need to do for this patchset to land?

Hi Kuan-wei, it seems Luke and Xi haven't been as active in the community for a while, I'll make time to cover the review and testing for this.

Regards,
Lehui

Quoted 24 lines

Regards,
Kuan-wei

Quoted 4 lines

---
- Add missing Fixes tags.
- Fix memory ordering by emitting aq=1, rl=1

Quoted 11 lines

Kuan-Wei Chiu (3):
riscv, bpf: Fix support for BPF_SDIV and BPF_SMOD in RV32 JIT riscv, bpf: Fix support for BPF_MOVSX in RV32 JIT riscv, bpf: Add 32 bit atomic operations to RV32 JIT

  arch/riscv/net/bpf_jit_comp32.c | 101 +++++++++++++++++++++++++++-----
  1 file changed, 85 insertions(+), 16 deletions(-)
Signature · 1 line
2.54.0.563.g4f69b47b94-goog

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#10 lore raw

Hi Lehui,

Quoted 20 lines · Pu Lehui
Quoted 17 lines · Kuan-Wei Chiu

Hi,

Quoted 14 lines · Kuan-Wei Chiu

Fix miscompiled signed operations and expands 32 bit atomic support in the RV32 BPF JIT.

The current implementation ignores the instruction offset field used by the bpf instruction set to specify BPF_SDIV/BPF_SMOD and BPF_MOVSX. This causes these operations to be treated as unsigned or zero-extended, leading to test_bpf failures. Fix this by checking the offset and emitting the correct instructions.

Additionally, we leverage the mandatory A extension to natively support 32 bit bpf atomics (and, or, xor, xchg) by mapping them directly to amo*.w instructions. BPF_CMPXCHG continues to fall back to the interpreter.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

As a result, test_bpf.ko now runs with 0 failures, and the total number of successfully JIT'ed test cases increases from 843 to 902.

Is there anything else I need to do for this patchset to land?

Hi Kuan-wei, it seems Luke and Xi haven't been as active in the community for a while, I'll make time to cover the review and testing for this.

Thank you for stepping in to help with the review and testing. I really appreciate it.

Since Luke and Xi haven't been active for a while, I was wondering if we should consider updating the MAINTAINERS file to reflect the current status?

Also, if you need extra hands, I'd be glad to help review and test future RV32 BPF patches.

Regards,
Kuan-Wei

Quoted 28 lines

Regards,
Lehui

Quoted 24 lines

Regards,
Kuan-wei

Quoted 4 lines

---
- Add missing Fixes tags.
- Fix memory ordering by emitting aq=1, rl=1

Quoted 11 lines

Kuan-Wei Chiu (3):
riscv, bpf: Fix support for BPF_SDIV and BPF_SMOD in RV32 JIT riscv, bpf: Fix support for BPF_MOVSX in RV32 JIT riscv, bpf: Add 32 bit atomic operations to RV32 JIT

  arch/riscv/net/bpf_jit_comp32.c | 101 +++++++++++++++++++++++++++-----
  1 file changed, 85 insertions(+), 16 deletions(-)
Signature · 1 line
2.54.0.563.g4f69b47b94-goog

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv