Implement cpuv4 support for s390x
Subsystems (8)
Ilya Leoshkevich
maintainercover Implement cpuv4 support for s390x
Hi,
This series adds the cpuv4 support to the s390x eBPF JIT. Patches 1-4 are preliminary bugfixes.
Patches 5-9 implement the new instructions. Patches 10-11 enable the tests.
Best regards,
Ilya
Ilya Leoshkevich (11):
bpf: Disable zero-extension for BPF_MEMSX
net: netfilter: Adjust timeouts of non-confirmed CTs in
bpf_ct_insert_entry()
selftests/bpf: Unmount the cgroup2 work directory selftests/bpf: Add big-endian support to the ldsx test s390/bpf: Implement BPF_MOV | BPF_X with sign-extension s390/bpf: Implement BPF_MEMSX
s390/bpf: Implement unconditional byte swap s390/bpf: Implement unconditional jump with 32-bit offset s390/bpf: Implement signed division selftests/bpf: Enable the cpuv4 tests for s390x selftests/bpf: Trim DENYLIST.s390x
arch/s390/net/bpf_jit_comp.c | 265 +++++++++++++----- kernel/bpf/verifier.c | 4 +- net/netfilter/nf_conntrack_bpf.c | 2 + tools/testing/selftests/bpf/DENYLIST.s390x | 25 -- tools/testing/selftests/bpf/cgroup_helpers.c | 33 ++- .../selftests/bpf/progs/test_ldsx_insn.c | 9 +- .../selftests/bpf/progs/verifier_bswap.c | 3 +- .../selftests/bpf/progs/verifier_gotol.c | 3 +- .../selftests/bpf/progs/verifier_ldsx.c | 149 ++++++---- .../selftests/bpf/progs/verifier_movsx.c | 3 +- .../selftests/bpf/progs/verifier_sdiv.c | 3 +- 11 files changed, 335 insertions(+), 164 deletions(-)
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH 1/11 bpf: Disable zero-extension for BPF_MEMSX
On the architectures that use bpf_jit_needs_zext(), e.g., s390x, the verifier incorrectly inserts a zero-extension after BPF_MEMSX, leading to miscompilations like the one below:
24: 89 1a ff fe 00 00 00 00 "r1 = *(s16 *)(r10 - 2);" # zext_dst set
0x3ff7fdb910e: lgh %r2,-2(%r13,%r0) # load halfword
0x3ff7fdb9114: llgfr %r2,%r2 # wrong!
25: 65 10 00 03 00 00 7f ff if r1 s> 32767 goto +3 <l0_1> # check_cond_jmp_op()Disable such zero-extensions. The JITs need to insert sign-extension themselves, if necessary.
--- kernel/bpf/verifier.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/bpf/verifier.c +3 -1
@@ -3110,7 +3110,9 @@ static void mark_insn_zext(struct bpf_verifier_env *env, {s32 def_idx = reg->subreg_def; -if (def_idx ==DEF_NOT_SUBREG ) +if (def_idx ==DEF_NOT_SUBREG || + (BPF_CLASS (env->prog->insnsi[def_idx -1 ].code) ==BPF_LDX && +BPF_MODE (env->prog->insnsi[def_idx -1 ].code) ==BPF_MEMSX ))return ; env->insn_aux_data[def_idx -1 ].zext_dst = true;
Signature · 1 line
2.41.0
Re: [PATCH bpf-next 01/11] bpf: Disable zero-extension for BPF_MEMSX
Quoted 9 lines · Ilya Leoshkevich
On the architectures that use
bpf_jit_needs_zext(), e.g., s390x, the verifier incorrectly inserts a zero-extension afterBPF_MEMSX, leading to miscompilations like the one below:24: 89 1a ff fe 00 00 00 00 "r1 = *(s16 *)(r10 - 2);" # zext_dst set 0x3ff7fdb910e: lgh %r2,-2(%r13,%r0) # load halfword 0x3ff7fdb9114: llgfr %r2,%r2 # wrong! 25: 65 10 00 03 00 00 7f ff if r1 s> 32767 goto +3 <l0_1> # check_cond_jmp_op()
Quoted 4 lines
Disable such zero-extensions. The JITs need to insert sign-extension themselves, if necessary.
Signed-off-by Ilya Leoshkevich <[email protected]>
Re: [PATCH bpf-next 01/11] bpf: Disable zero-extension for BPF_MEMSX
In reply to Ilya LeoshkevichHi Ilya
Quoted 23 lines · Ilya Leoshkevich
On the architectures that use
bpf_jit_needs_zext(), e.g., s390x, the verifier incorrectly inserts a zero-extension afterBPF_MEMSX, leading to miscompilations like the one below:24: 89 1a ff fe 00 00 00 00 "r1 = *(s16 *)(r10 - 2);" # zext_dst set 0x3ff7fdb910e: lgh %r2,-2(%r13,%r0) # load halfword 0x3ff7fdb9114: llgfr %r2,%r2 # wrong! 25: 65 10 00 03 00 00 7f ff if r1 s> 32767 goto +3 <l0_1> # check_cond_jmp_op()Disable such zero-extensions. The JITs need to insert sign-extension themselves, if necessary.
Signed-off-by Ilya Leoshkevich <[email protected]>--- kernel/bpf/verifier.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)kernel/bpf/verifier.c +0
-0
kernel/bpf/verifier.c static void mark_insn_zext(struct bpf_verifier_env *env, · line 3113
Quoted 4 lines
kernel/bpf/verifier.c +0
-1@@ -3110,7 +3110,9 @@ static void mark_insn_zext(struct bpf_verifier_env *env, {s32 def_idx = reg->subreg_def; -if (def_idx ==DEF_NOT_SUBREG )
The problem here is that reg->subreg_def should be set as DEF_NOT_SUBREG for registers that are used as destination registers of BPF_LDX | BPF_MEMSX. I am seeing
the same problem on ARM32 and was going to send a patch today.
The problem is that is_reg64() returns false for destination registers of BPF_LDX | BPF_MEMSX.
But BPF_LDX | BPF_MEMSX always loads a 64 bit value because of the sign extension sois_reg64() should return true.
I have written a patch that I will be sending as a reply to this. Please let me know if that makes sense.
Quoted 10 lines
kernel/bpf/verifier.c +3
-1+if (def_idx ==DEF_NOT_SUBREG || + (BPF_CLASS (env->prog->insnsi[def_idx -1 ].code) ==BPF_LDX && +BPF_MODE (env->prog->insnsi[def_idx -1 ].code) ==BPF_MEMSX ))return ; env->insn_aux_data[def_idx -1 ].zext_dst = true; --2.41.0
Thanks,
Puranjay
Re: [PATCH bpf-next 01/11] bpf: Disable zero-extension for BPF_MEMSX
Quoted 5 lines · Puranjay Mohan
The problem here is that
reg->subreg_defshould be set asDEF_NOT_SUBREGfor registers that are used as destination registers ofBPF_LDX|BPF_MEMSX. I am seeing
the same problem on ARM32 and was going to send a patch today.
Quoted 9 lines
The problem is that
is_reg64()returns false for destination registers ofBPF_LDX|BPF_MEMSX.
ButBPF_LDX|BPF_MEMSXalways loads a 64 bit value because of the sign extension sois_reg64()should return true.I have written a patch that I will be sending as a reply to this. Please let me know if that makes sense.
The check_reg_arg() function will mark reg->subreg_def = DEF_NOT_SUBREG for destination registers if is_reg64() returns true for these registers. My patch below make is_reg64() return true for destination registers of BPF_LDX with mod = BPF_MEMSX. I feel this is the correct way to fix this problem.
Here is my patch:
--- 8< ---
From cf1bf5282183cf721926ab14d968d3d4097b89b8 Mon Sep 17 00:00:00 2001
From: Puranjay Mohan <[email protected]>
Date: Fri, 1 Sep 2023 11:18:59 +0000
Subject: [PATCH bpf] bpf: verifier: mark destination of sign-extended load as
64 bit
The verifier can emit instructions to zero-extend destination registers when the register is being used to keep 32 bit values. This behaviour is enabled only when the JIT sets bpf_jit_needs_zext() -> true. In the case of a sign extended load instruction, the destination register always has a 64-bit value, therefore the verifier should not emit zero-extend instructions for it.
Change is_reg64() to return true if the register under consideration is a destination register of LDX instruction with mode = BPF_MEMSX.
--- kernel/bpf/verifier.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/bpf/verifier.c +1 -1
@@ -3029,7 +3029,7 @@ static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn,if (class ==BPF_LDX ) {if (t !=SRC_OP ) -return BPF_SIZE (code) ==BPF_DW ; +return (BPF_SIZE (code) ==BPF_DW ||BPF_MODE (code) ==BPF_MEMSX );/* LDX source must be ptr. */ return true; }
Signature · 1 line
2.39.2
Re: [PATCH bpf-next 01/11] bpf: Disable zero-extension for BPF_MEMSX
Quoted 53 lines · Puranjay Mohan
Quoted 5 lines · Puranjay Mohan
The problem here is that
reg->subreg_defshould be set asDEF_NOT_SUBREGfor registers that are used as destination registers ofBPF_LDX|BPF_MEMSX. I am seeing
the same problem on ARM32 and was going to send a patch today.Quoted 9 lines
The problem is that
is_reg64()returns false for destination registers ofBPF_LDX|BPF_MEMSX.
ButBPF_LDX|BPF_MEMSXalways loads a 64 bit value because of the sign extension sois_reg64()should return true.I have written a patch that I will be sending as a reply to this. Please let me know if that makes sense.
The
check_reg_arg()function will markreg->subreg_def=DEF_NOT_SUBREGfor destination registers ifis_reg64()returns true for these registers. My patch below makeis_reg64()return true for destination registers ofBPF_LDXwith mod =BPF_MEMSX. I feel this is the correct way to fix this problem.Here is my patch:
--- 8< ---
Fromcf1bf5282183cf721926ab14d968d3d4097b89b8Mon Sep 17 00:00:00 2001
From: Puranjay Mohan <[email protected]>
Date: Fri, 1 Sep 2023 11:18:59 +0000
Subject: [PATCH bpf] bpf: verifier: mark destination of sign-extended load as
64 bitThe verifier can emit instructions to zero-extend destination registers when the register is being used to keep 32 bit values. This behaviour is enabled only when the JIT sets
bpf_jit_needs_zext()-> true. In the case of a sign extended load instruction, the destination register always has a 64-bit value, therefore the verifier should not emit zero-extend instructions for it.Change
is_reg64()to return true if the register under consideration is a destination register of LDX instruction with mode =BPF_MEMSX.Fixes 1f9a1ea821ff ("bpf: Support new sign-extension load insns")Signed-off-by Puranjay Mohan <[email protected]>--- kernel/bpf/verifier.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/bpf/verifier.c +0
-0
kernel/bpf/verifier.c static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 3032
Quoted 5 lines
kernel/bpf/verifier.c +1
-1@@ -3029,7 +3029,7 @@ static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn,if (class ==BPF_LDX ) {if (t !=SRC_OP ) -return BPF_SIZE (code) ==BPF_DW ; +return (BPF_SIZE (code) ==BPF_DW ||BPF_MODE (code) ==BPF_MEMSX );
Looks like we have a bug here for normal LDX too. This 'if' condition was inserting unnecessary zext for LDX. It was harmless for LDX and broken for LDSX.
Both LDX and LDSX write all bits of 64-bit register.
I think the proper fix is to remove above two lines. wdyt?
Re: [PATCH bpf-next 01/11] bpf: Disable zero-extension for BPF_MEMSX
Quoted 65 lines · Alexei Starovoitov
Quoted 53 lines · Puranjay Mohan
Quoted 5 lines · Puranjay Mohan
The problem here is that
reg->subreg_defshould be set asDEF_NOT_SUBREGfor registers that are used as destination registers ofBPF_LDX|BPF_MEMSX. I am seeing
the same problem on ARM32 and was going to send a patch today.Quoted 9 lines
The problem is that
is_reg64()returns false for destination registers ofBPF_LDX|BPF_MEMSX.
ButBPF_LDX|BPF_MEMSXalways loads a 64 bit value because of the sign extension sois_reg64()should return true.I have written a patch that I will be sending as a reply to this. Please let me know if that makes sense.
The
check_reg_arg()function will markreg->subreg_def=DEF_NOT_SUBREGfor destination registers ifis_reg64()returns true for these registers. My patch below makeis_reg64()return true for destination registers ofBPF_LDXwith mod =BPF_MEMSX. I feel this is the correct way to fix this problem.Here is my patch:
--- 8< ---
Fromcf1bf5282183cf721926ab14d968d3d4097b89b8Mon Sep 17 00:00:00 2001
From: Puranjay Mohan <[email protected]>
Date: Fri, 1 Sep 2023 11:18:59 +0000
Subject: [PATCH bpf] bpf: verifier: mark destination of sign-extended load as
64 bitThe verifier can emit instructions to zero-extend destination registers when the register is being used to keep 32 bit values. This behaviour is enabled only when the JIT sets
bpf_jit_needs_zext()-> true. In the case of a sign extended load instruction, the destination register always has a 64-bit value, therefore the verifier should not emit zero-extend instructions for it.Change
is_reg64()to return true if the register under consideration is a destination register of LDX instruction with mode =BPF_MEMSX.Fixes 1f9a1ea821ff ("bpf: Support new sign-extension load insns")Signed-off-by Puranjay Mohan <[email protected]>--- kernel/bpf/verifier.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/bpf/verifier.c +0
-0kernel/bpf/verifier.c static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 3032
Quoted 5 lines
kernel/bpf/verifier.c +1
-1@@ -3029,7 +3029,7 @@ static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn,if (class ==BPF_LDX ) {if (t !=SRC_OP ) -return BPF_SIZE (code) ==BPF_DW ; +return (BPF_SIZE (code) ==BPF_DW ||BPF_MODE (code) ==BPF_MEMSX );Looks like we have a bug here for normal LDX too. This 'if' condition was inserting unnecessary zext for LDX. It was harmless for LDX and broken for LDSX.
Both LDX and LDSX write all bits of 64-bit register.
Quoted 2 lines
I think the proper fix is to remove above two lines. wdyt?
For LDX this returns true only if it is with a BPF_DW, for others it returns false. This means a zext is inserted for BPF_LDX | BPF_B/H/W.
This is not a bug because LDX writes 64 bits of the register only with BPF_DW. With BPF_B/H/W It only writes the lower 32bits and needs zext for upper 32 bits. On 32 bit architectures where a 64-bit BPF register is simulated with two 32-bit registers, explicit zext is required for BPF_LDX | BPF_B/H/W.
So, we should not remove this.
Thanks,
Puranjay
Re: [PATCH bpf-next 01/11] bpf: Disable zero-extension for BPF_MEMSX
Quoted 68 lines · Puranjay Mohan
Quoted 65 lines · Alexei Starovoitov
Quoted 53 lines · Puranjay Mohan
Quoted 5 lines · Puranjay Mohan
The problem here is that
reg->subreg_defshould be set asDEF_NOT_SUBREGfor registers that are used as destination registers ofBPF_LDX|BPF_MEMSX. I am seeing
the same problem on ARM32 and was going to send a patch today.Quoted 9 lines
The problem is that
is_reg64()returns false for destination registers ofBPF_LDX|BPF_MEMSX.
ButBPF_LDX|BPF_MEMSXalways loads a 64 bit value because of the sign extension sois_reg64()should return true.I have written a patch that I will be sending as a reply to this. Please let me know if that makes sense.
The
check_reg_arg()function will markreg->subreg_def=DEF_NOT_SUBREGfor destination registers ifis_reg64()returns true for these registers. My patch below makeis_reg64()return true for destination registers ofBPF_LDXwith mod =BPF_MEMSX. I feel this is the correct way to fix this problem.Here is my patch:
--- 8< ---
Fromcf1bf5282183cf721926ab14d968d3d4097b89b8Mon Sep 17 00:00:00 2001
From: Puranjay Mohan <[email protected]>
Date: Fri, 1 Sep 2023 11:18:59 +0000
Subject: [PATCH bpf] bpf: verifier: mark destination of sign-extended load as
64 bitThe verifier can emit instructions to zero-extend destination registers when the register is being used to keep 32 bit values. This behaviour is enabled only when the JIT sets
bpf_jit_needs_zext()-> true. In the case of a sign extended load instruction, the destination register always has a 64-bit value, therefore the verifier should not emit zero-extend instructions for it.Change
is_reg64()to return true if the register under consideration is a destination register of LDX instruction with mode =BPF_MEMSX.Fixes 1f9a1ea821ff ("bpf: Support new sign-extension load insns")Signed-off-by Puranjay Mohan <[email protected]>--- kernel/bpf/verifier.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/bpf/verifier.c +0
-0kernel/bpf/verifier.c static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 3032
Quoted 5 lines
kernel/bpf/verifier.c +1
-1@@ -3029,7 +3029,7 @@ static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn,if (class ==BPF_LDX ) {if (t !=SRC_OP ) -return BPF_SIZE (code) ==BPF_DW ; +return (BPF_SIZE (code) ==BPF_DW ||BPF_MODE (code) ==BPF_MEMSX );Looks like we have a bug here for normal LDX too. This 'if' condition was inserting unnecessary zext for LDX. It was harmless for LDX and broken for LDSX.
Both LDX and LDSX write all bits of 64-bit register.
Quoted 8 lines
Quoted 2 lines
I think the proper fix is to remove above two lines. wdyt?
For LDX this returns true only if it is with a
BPF_DW, for others it returns false. This means a zext is inserted forBPF_LDX| BPF_B/H/W.This is not a bug because LDX writes 64 bits of the register only with
BPF_DW. With BPF_B/H/W It only writes the lower 32bits and needs zext for upper 32 bits.
No. The interpreter writes all 64-bit for any LDX insn. All JITs must do it as well.
Quoted 2 lines
On 32 bit architectures where a 64-bit BPF register is simulated with two 32-bit registers, explicit zext is required for
BPF_LDX| BPF_B/H/W.
zext JIT-aid done by the verifier has nothing to do with 32-bit architecture. It's necessary on 64-bit as well when HW doesn't automatically zero out upper 32-bit like it does on arm64 and x86-64
Quoted 1 line
So, we should not remove this.
I still think we should.
Re: [PATCH bpf-next 01/11] bpf: Disable zero-extension for BPF_MEMSX
Quoted 81 lines · Alexei Starovoitov
Quoted 68 lines · Puranjay Mohan
Quoted 65 lines · Alexei Starovoitov
Quoted 53 lines · Puranjay Mohan
Quoted 5 lines · Puranjay Mohan
The problem here is that
reg->subreg_defshould be set asDEF_NOT_SUBREGfor registers that are used as destination registers ofBPF_LDX|BPF_MEMSX. I am seeing
the same problem on ARM32 and was going to send a patch today.Quoted 9 lines
The problem is that
is_reg64()returns false for destination registers ofBPF_LDX|BPF_MEMSX.
ButBPF_LDX|BPF_MEMSXalways loads a 64 bit value because of the sign extension sois_reg64()should return true.I have written a patch that I will be sending as a reply to this. Please let me know if that makes sense.
The
check_reg_arg()function will markreg->subreg_def=DEF_NOT_SUBREGfor destination registers ifis_reg64()returns true for these registers. My patch below makeis_reg64()return true for destination registers ofBPF_LDXwith mod =BPF_MEMSX. I feel this is the correct way to fix this problem.Here is my patch:
--- 8< ---
Fromcf1bf5282183cf721926ab14d968d3d4097b89b8Mon Sep 17 00:00:00 2001
From: Puranjay Mohan <[email protected]>
Date: Fri, 1 Sep 2023 11:18:59 +0000
Subject: [PATCH bpf] bpf: verifier: mark destination of sign-extended load as
64 bitThe verifier can emit instructions to zero-extend destination registers when the register is being used to keep 32 bit values. This behaviour is enabled only when the JIT sets
bpf_jit_needs_zext()-> true. In the case of a sign extended load instruction, the destination register always has a 64-bit value, therefore the verifier should not emit zero-extend instructions for it.Change
is_reg64()to return true if the register under consideration is a destination register of LDX instruction with mode =BPF_MEMSX.Fixes 1f9a1ea821ff ("bpf: Support new sign-extension load insns")Signed-off-by Puranjay Mohan <[email protected]>--- kernel/bpf/verifier.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/bpf/verifier.c +0
-0kernel/bpf/verifier.c static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 3032
Quoted 5 lines
kernel/bpf/verifier.c +1
-1@@ -3029,7 +3029,7 @@ static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn,if (class ==BPF_LDX ) {if (t !=SRC_OP ) -return BPF_SIZE (code) ==BPF_DW ; +return (BPF_SIZE (code) ==BPF_DW ||BPF_MODE (code) ==BPF_MEMSX );Looks like we have a bug here for normal LDX too. This 'if' condition was inserting unnecessary zext for LDX. It was harmless for LDX and broken for LDSX.
Both LDX and LDSX write all bits of 64-bit register.Quoted 8 lines
Quoted 2 lines
I think the proper fix is to remove above two lines. wdyt?
For LDX this returns true only if it is with a
BPF_DW, for others it returns false. This means a zext is inserted forBPF_LDX| BPF_B/H/W.This is not a bug because LDX writes 64 bits of the register only with
BPF_DW. With BPF_B/H/W It only writes the lower 32bits and needs zext for upper 32 bits.No. The interpreter writes all 64-bit for any LDX insn. All JITs must do it as well.
Quoted 6 lines
Quoted 2 lines
On 32 bit architectures where a 64-bit BPF register is simulated with two 32-bit registers, explicit zext is required for
BPF_LDX| BPF_B/H/W.zext JIT-aid done by the verifier has nothing to do with 32-bit architecture. It's necessary on 64-bit as well when HW doesn't automatically zero out upper 32-bit like it does on arm64 and x86-64
Yes, I agree that zext JIT-aid is required for all 32-bit architectures and some 64-bit architectures that can't automatically zero out the upper 32-bits.
Basically any architecture that sets bpf_jit_needs_zext() -> true.
Quoted 3 lines
Quoted 1 line
So, we should not remove this.
I still think we should.
If we remove this then some JITs will not zero extend the upper 32-bits for BPF_LDX | BPF_B/H/W.
My understanding is that Verifier sets prog->aux->verifier_zext if it emits zext instructions. If the verifier doesn't emit zext for LDX but sets prog->aux->verifier_zext that would cause wrong behavior for some JITs:
Example code from ARM32 jit doing BPF_LDX | BPF_MEM | BPF_B:
case BPF_B:
/* Load a Byte */ emit(ARM_LDRB_I(rd[1], rm, off), ctx); if (!ctx->prog->aux->verifier_zext) emit_a32_mov_i(rd[0], 0, ctx); break;
Here if ctx->prog->aux->verifier_zext is set by the verifier, and zext was not emitted for LDX, JIT will not zero the upper 32-bits.
RISCV32, PowerPC32, x86-32 JITs have similar code paths. Only MIPS32 JIT zero-extends for LDX without checking prog->aux->verifier_zext.
So, if we want to stop emitting zext for LDX then we would need to modify all these JITs to always zext for LDX.
Let me know if my understanding has some gaps, also if we decide to remove it, I am happy to send patches for it and fix the JITs that need modifications.
Thanks,
Puranjay
Re: [PATCH bpf-next 01/11] bpf: Disable zero-extension for BPF_MEMSX
Quoted 113 lines · Puranjay Mohan
Quoted 81 lines · Alexei Starovoitov
Quoted 68 lines · Puranjay Mohan
Quoted 65 lines · Alexei Starovoitov
Quoted 53 lines · Puranjay Mohan
Quoted 5 lines · Puranjay Mohan
The problem here is that
reg->subreg_defshould be set asDEF_NOT_SUBREGfor registers that are used as destination registers ofBPF_LDX|BPF_MEMSX. I am seeing
the same problem on ARM32 and was going to send a patch today.Quoted 9 lines
The problem is that
is_reg64()returns false for destination registers ofBPF_LDX|BPF_MEMSX.
ButBPF_LDX|BPF_MEMSXalways loads a 64 bit value because of the sign extension sois_reg64()should return true.I have written a patch that I will be sending as a reply to this. Please let me know if that makes sense.
The
check_reg_arg()function will markreg->subreg_def=DEF_NOT_SUBREGfor destination registers ifis_reg64()returns true for these registers. My patch below makeis_reg64()return true for destination registers ofBPF_LDXwith mod =BPF_MEMSX. I feel this is the correct way to fix this problem.Here is my patch:
--- 8< ---
Fromcf1bf5282183cf721926ab14d968d3d4097b89b8Mon Sep 17 00:00:00 2001
From: Puranjay Mohan <[email protected]>
Date: Fri, 1 Sep 2023 11:18:59 +0000
Subject: [PATCH bpf] bpf: verifier: mark destination of sign-extended load as
64 bitThe verifier can emit instructions to zero-extend destination registers when the register is being used to keep 32 bit values. This behaviour is enabled only when the JIT sets
bpf_jit_needs_zext()-> true. In the case of a sign extended load instruction, the destination register always has a 64-bit value, therefore the verifier should not emit zero-extend instructions for it.Change
is_reg64()to return true if the register under consideration is a destination register of LDX instruction with mode =BPF_MEMSX.Fixes 1f9a1ea821ff ("bpf: Support new sign-extension load insns")Signed-off-by Puranjay Mohan <[email protected]>--- kernel/bpf/verifier.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/bpf/verifier.c +0
-0kernel/bpf/verifier.c static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 3032
Quoted 5 lines
kernel/bpf/verifier.c +1
-1@@ -3029,7 +3029,7 @@ static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn,if (class ==BPF_LDX ) {if (t !=SRC_OP ) -return BPF_SIZE (code) ==BPF_DW ; +return (BPF_SIZE (code) ==BPF_DW ||BPF_MODE (code) ==BPF_MEMSX );Looks like we have a bug here for normal LDX too. This 'if' condition was inserting unnecessary zext for LDX. It was harmless for LDX and broken for LDSX.
Both LDX and LDSX write all bits of 64-bit register.Quoted 8 lines
Quoted 2 lines
I think the proper fix is to remove above two lines. wdyt?
For LDX this returns true only if it is with a
BPF_DW, for others it returns false. This means a zext is inserted forBPF_LDX| BPF_B/H/W.This is not a bug because LDX writes 64 bits of the register only with
BPF_DW. With BPF_B/H/W It only writes the lower 32bits and needs zext for upper 32 bits.No. The interpreter writes all 64-bit for any LDX insn. All JITs must do it as well.
Quoted 6 lines
Quoted 2 lines
On 32 bit architectures where a 64-bit BPF register is simulated with two 32-bit registers, explicit zext is required for
BPF_LDX| BPF_B/H/W.zext JIT-aid done by the verifier has nothing to do with 32-bit architecture. It's necessary on 64-bit as well when HW doesn't automatically zero out upper 32-bit like it does on arm64 and x86-64
Yes, I agree that zext JIT-aid is required for all 32-bit architectures and some 64-bit architectures that can't automatically zero out the upper 32-bits.
Basically any architecture that setsbpf_jit_needs_zext()-> true.Quoted 3 lines
Quoted 1 line
So, we should not remove this.
I still think we should.
If we remove this then some JITs will not zero extend the upper 32-bits for
BPF_LDX| BPF_B/H/W.My understanding is that Verifier sets
prog->aux->verifier_zextif it emits zext instructions. If the verifier doesn't emit zext for LDX but setsprog->aux->verifier_zextthat would cause wrong behavior for some JITs:Example code from ARM32 jit doing
BPF_LDX|BPF_MEM|BPF_B:case
BPF_B:/* Load a Byte */ emit(ARM_LDRB_I(rd[1], rm, off), ctx); if (!ctx->prog->aux->verifier_zext) emit_a32_mov_i(rd[0], 0, ctx); break;
Quoted 7 lines
Here if
ctx->prog->aux->verifier_zextis set by the verifier, and zext was not emitted for LDX, JIT will not zero the upper 32-bits.RISCV32, PowerPC32, x86-32 JITs have similar code paths. Only MIPS32 JIT zero-extends for LDX without checking
prog->aux->verifier_zext.So, if we want to stop emitting zext for LDX then we would need to modify all these JITs to always zext for LDX.
I guess we never clearly defined what 'needs_zext' is supposed to be, so it wouldn't be fair to call 32-bit JITs buggy.
But we better address this issue now.
This 32-bit zeroing after LDX hurts mips64, s390, ppc64, riscv64. I believe all 4 JITs emit proper zero extension into 64-bit register by using single cpu instruction,
but they also define bpf_jit_needs_zext() as true,
so extra BPF_ZEXT_REG() is added by the verifier
and it is a pure run-time overhead.
It's better to remove
if (t != SRC_OP)
return BPF_SIZE(code) == BPF_DW;
from is_reg64() to avoid adding BPF_ZEXT_REG() insn and fix 32-bit JITs at the same time.
RISCV32, PowerPC32, x86-32 JITs fixed in the first 3 patches to always zero upper 32-bit after LDX and
then 4th patch to remove these two lines.
Quoted 2 lines
Let me know if my understanding has some gaps, also if we decide to remove it, I am happy to send patches for it and fix the JITs that need modifications.
Thank you for working on it!
cc-ing JIT experts.
Re: [PATCH bpf-next 01/11] bpf: Disable zero-extension for BPF_MEMSX
Quoted 142 lines · Alexei Starovoitov
Quoted 113 lines · Puranjay Mohan
Quoted 81 lines · Alexei Starovoitov
Quoted 68 lines · Puranjay Mohan
Quoted 65 lines · Alexei Starovoitov
Quoted 53 lines · Puranjay Mohan
Quoted 5 lines · Puranjay Mohan
The problem here is that
reg->subreg_defshould be set asDEF_NOT_SUBREGfor registers that are used as destination registers ofBPF_LDX|BPF_MEMSX. I am seeing
the same problem on ARM32 and was going to send a patch today.Quoted 9 lines
The problem is that
is_reg64()returns false for destination registers ofBPF_LDX|BPF_MEMSX.
ButBPF_LDX|BPF_MEMSXalways loads a 64 bit value because of the sign extension sois_reg64()should return true.I have written a patch that I will be sending as a reply to this. Please let me know if that makes sense.
The
check_reg_arg()function will markreg->subreg_def=DEF_NOT_SUBREGfor destination registers ifis_reg64()returns true for these registers. My patch below makeis_reg64()return true for destination registers ofBPF_LDXwith mod =BPF_MEMSX. I feel this is the correct way to fix this problem.Here is my patch:
--- 8< ---
Fromcf1bf5282183cf721926ab14d968d3d4097b89b8Mon Sep 17 00:00:00 2001
From: Puranjay Mohan <[email protected]>
Date: Fri, 1 Sep 2023 11:18:59 +0000
Subject: [PATCH bpf] bpf: verifier: mark destination of sign-extended load as
64 bitThe verifier can emit instructions to zero-extend destination registers when the register is being used to keep 32 bit values. This behaviour is enabled only when the JIT sets
bpf_jit_needs_zext()-> true. In the case of a sign extended load instruction, the destination register always has a 64-bit value, therefore the verifier should not emit zero-extend instructions for it.Change
is_reg64()to return true if the register under consideration is a destination register of LDX instruction with mode =BPF_MEMSX.Fixes 1f9a1ea821ff ("bpf: Support new sign-extension load insns")Signed-off-by Puranjay Mohan <[email protected]>--- kernel/bpf/verifier.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/bpf/verifier.c +0
-0kernel/bpf/verifier.c static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 3032
Quoted 5 lines
kernel/bpf/verifier.c +1
-1@@ -3029,7 +3029,7 @@ static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn,if (class ==BPF_LDX ) {if (t !=SRC_OP ) -return BPF_SIZE (code) ==BPF_DW ; +return (BPF_SIZE (code) ==BPF_DW ||BPF_MODE (code) ==BPF_MEMSX );Looks like we have a bug here for normal LDX too. This 'if' condition was inserting unnecessary zext for LDX. It was harmless for LDX and broken for LDSX.
Both LDX and LDSX write all bits of 64-bit register.Quoted 8 lines
Quoted 2 lines
I think the proper fix is to remove above two lines. wdyt?
For LDX this returns true only if it is with a
BPF_DW, for others it returns false. This means a zext is inserted forBPF_LDX| BPF_B/H/W.This is not a bug because LDX writes 64 bits of the register only with
BPF_DW. With BPF_B/H/W It only writes the lower 32bits and needs zext for upper 32 bits.No. The interpreter writes all 64-bit for any LDX insn. All JITs must do it as well.
Quoted 6 lines
Quoted 2 lines
On 32 bit architectures where a 64-bit BPF register is simulated with two 32-bit registers, explicit zext is required for
BPF_LDX| BPF_B/H/W.zext JIT-aid done by the verifier has nothing to do with 32-bit architecture. It's necessary on 64-bit as well when HW doesn't automatically zero out upper 32-bit like it does on arm64 and x86-64
Yes, I agree that zext JIT-aid is required for all 32-bit architectures and some 64-bit architectures that can't automatically zero out the upper 32-bits.
Basically any architecture that setsbpf_jit_needs_zext()-> true.Quoted 3 lines
Quoted 1 line
So, we should not remove this.
I still think we should.
If we remove this then some JITs will not zero extend the upper 32-bits for
BPF_LDX| BPF_B/H/W.My understanding is that Verifier sets
prog->aux->verifier_zextif it emits zext instructions. If the verifier doesn't emit zext for LDX but setsprog->aux->verifier_zextthat would cause wrong behavior for some JITs:Example code from ARM32 jit doing
BPF_LDX|BPF_MEM|BPF_B:case
BPF_B:/* Load a Byte */ emit(ARM_LDRB_I(rd[1], rm, off), ctx); if (!ctx->prog->aux->verifier_zext) emit_a32_mov_i(rd[0], 0, ctx); break;Quoted 7 lines
Here if
ctx->prog->aux->verifier_zextis set by the verifier, and zext was not emitted for LDX, JIT will not zero the upper 32-bits.RISCV32, PowerPC32, x86-32 JITs have similar code paths. Only MIPS32 JIT zero-extends for LDX without checking
prog->aux->verifier_zext.So, if we want to stop emitting zext for LDX then we would need to modify all these JITs to always zext for LDX.
I guess we never clearly defined what '
needs_zext' is supposed to be, so it wouldn't be fair to call 32-bit JITs buggy.
But we better address this issue now.
This 32-bit zeroing after LDX hurts mips64, s390, ppc64, riscv64. I believe all 4 JITs emit proper zero extension into 64-bit register by using single cpu instruction,
but they also definebpf_jit_needs_zext()as true,
so extraBPF_ZEXT_REG()is added by the verifier
and it is a pure run-time overhead.It's better to remove
if (t !=SRC_OP)return BPF_SIZE(code) == BPF_DW;from
is_reg64()to avoid addingBPF_ZEXT_REG()insn and fix 32-bit JITs at the same time.
RISCV32, PowerPC32, x86-32 JITs fixed in the first 3 patches to always zero upper 32-bit after LDX and
then 4th patch to remove these two lines.
Quoted 6 lines
Quoted 2 lines
Let me know if my understanding has some gaps, also if we decide to remove it, I am happy to send patches for it and fix the JITs that need modifications.
Thank you for working on it!
cc-ing JIT experts.
Thanks for the detailed explanation. I agree with this approach. I will be sending the patches for this soon.
Thanks,
Puranjay
Re: [PATCH bpf-next 01/11] bpf: Disable zero-extension for BPF_MEMSX
In reply to Alexei StarovoitovHi Alexei,
[...]
Quoted 9 lines
I guess we never clearly defined what '
needs_zext' is supposed to be, so it wouldn't be fair to call 32-bit JITs buggy.
But we better address this issue now.
This 32-bit zeroing after LDX hurts mips64, s390, ppc64, riscv64. I believe all 4 JITs emit proper zero extension into 64-bit register by using single cpu instruction,
but they also definebpf_jit_needs_zext()as true,
so extraBPF_ZEXT_REG()is added by the verifier
and it is a pure run-time overhead.
I just realised that these zext instructions will not be a runtime overhead because the JITs ignore them.
Like
s390 does:
case BPF_LDX | BPF_MEM | BPF_B: /* dst = *(u8 *)(ul) (src + off) */ case BPF_LDX | BPF_PROBE_MEM | BPF_B:
/* llgc %dst,0(off,%src) */
EMIT6_DISP_LH(0xe3000000, 0x0090, dst_reg, src_reg, REG_0, off);
jit->seen |= SEEN_MEM;
if (insn_is_zext(&insn[1]))
insn_count = 2; /* this will skip the next zext instruction */
break;powerpc does after LDX:
if (size != BPF_DW && insn_is_zext(&insn[i + 1]))
addrs[++i] = ctx->idx * 4;
Quoted 8 lines
It's better to remove
if (t !=SRC_OP)return BPF_SIZE(code) == BPF_DW;from
is_reg64()to avoid addingBPF_ZEXT_REG()insn and fix 32-bit JITs at the same time.
RISCV32, PowerPC32, x86-32 JITs fixed in the first 3 patches to always zero upper 32-bit after LDX and
then 4th patch to remove these two lines.
I have sent the patches for above, although I think this optimization is useful because
zero extension after LDX is only required when the loaded value is later being used as
a 64-bit value. If it is not the case then the verifier will not emit the zext and 32-bit JITs will emit
1 less instruction because they expect the verifier to do the zext for them where required.
Link to patch series:
https://lore.kernel.org/bpf/[email protected]/T/#t
Thanks,
Puranjay
Re: [PATCH bpf-next 01/11] bpf: Disable zero-extension for BPF_MEMSX
Quoted 28 lines · Puranjay Mohan
Hi Alexei,
[...]
Quoted 9 lines
I guess we never clearly defined what '
needs_zext' is supposed to be, so it wouldn't be fair to call 32-bit JITs buggy.
But we better address this issue now.
This 32-bit zeroing after LDX hurts mips64, s390, ppc64, riscv64. I believe all 4 JITs emit proper zero extension into 64-bit register by using single cpu instruction,
but they also definebpf_jit_needs_zext()as true,
so extraBPF_ZEXT_REG()is added by the verifier
and it is a pure run-time overhead.I just realised that these zext instructions will not be a runtime overhead because the JITs ignore them.
Like
s390 does:
caseBPF_LDX|BPF_MEM|BPF_B: /* dst = *(u8 *)(ul) (src + off) */ caseBPF_LDX|BPF_PROBE_MEM|BPF_B:/* llgc %dst,0(off,%src) */ EMIT6_DISP_LH(0xe3000000, 0x0090, dst_reg, src_reg, REG_0, off); jit->seen |= SEEN_MEM; if (insn_is_zext(&insn[1])) insn_count = 2; /* this will skip the next zext instruction */ break;
Quoted 3 lines
powerpc does after LDX:
if (size !=BPF_DW&&insn_is_zext(&insn[i + 1]))addrs[++i] = ctx->idx * 4;
I see. Indeed the 64-bit JITs ignore this special zext insn after LDX.
Quoted 9 lines
Quoted 8 lines
It's better to remove
if (t !=SRC_OP)return BPF_SIZE(code) == BPF_DW;from
is_reg64()to avoid addingBPF_ZEXT_REG()insn and fix 32-bit JITs at the same time.
RISCV32, PowerPC32, x86-32 JITs fixed in the first 3 patches to always zero upper 32-bit after LDX and
then 4th patch to remove these two lines.
Quoted 8 lines
I have sent the patches for above, although I think this optimization is useful because
zero extension after LDX is only required when the loaded value is later being used as
a 64-bit value. If it is not the case then the verifier will not emit the zext and 32-bit JITs will emit
1 less instruction because they expect the verifier to do the zext for them where required.
You're correct.
Ok. Let's keep zext for LDX as-is.
Re: [PATCH bpf-next 01/11] bpf: Disable zero-extension for BPF_MEMSX
Quoted 55 lines · Alexei Starovoitov
Quoted 28 lines · Puranjay Mohan
Hi Alexei,
[...]
Quoted 9 lines
I guess we never clearly defined what '
needs_zext' is supposed to be, so it wouldn't be fair to call 32-bit JITs buggy.
But we better address this issue now.
This 32-bit zeroing after LDX hurts mips64, s390, ppc64, riscv64. I believe all 4 JITs emit proper zero extension into 64-bit register by using single cpu instruction,
but they also definebpf_jit_needs_zext()as true,
so extraBPF_ZEXT_REG()is added by the verifier
and it is a pure run-time overhead.I just realised that these zext instructions will not be a runtime overhead because the JITs ignore them.
Like
s390 does:
caseBPF_LDX|BPF_MEM|BPF_B: /* dst = *(u8 *)(ul) (src + off) */ caseBPF_LDX|BPF_PROBE_MEM|BPF_B:/* llgc %dst,0(off,%src) */ EMIT6_DISP_LH(0xe3000000, 0x0090, dst_reg, src_reg, REG_0, off); jit->seen |= SEEN_MEM; if (insn_is_zext(&insn[1])) insn_count = 2; /* this will skip the next zext instruction */ break;Quoted 3 lines
powerpc does after LDX:
if (size !=BPF_DW&&insn_is_zext(&insn[i + 1]))addrs[++i] = ctx->idx * 4;I see. Indeed the 64-bit JITs ignore this special zext insn after LDX.
Quoted 9 lines
Quoted 8 lines
It's better to remove
if (t !=SRC_OP)return BPF_SIZE(code) == BPF_DW;from
is_reg64()to avoid addingBPF_ZEXT_REG()insn and fix 32-bit JITs at the same time.
RISCV32, PowerPC32, x86-32 JITs fixed in the first 3 patches to always zero upper 32-bit after LDX and
then 4th patch to remove these two lines.Quoted 8 lines
I have sent the patches for above, although I think this optimization is useful because
zero extension after LDX is only required when the loaded value is later being used as
a 64-bit value. If it is not the case then the verifier will not emit the zext and 32-bit JITs will emit
1 less instruction because they expect the verifier to do the zext for them where required.
Quoted 2 lines
You're correct.
Ok. Let's keep zext for LDX as-is.
Yes,
let's do
if (class == BPF_LDX) {
if (t != SRC_OP)- return BPF_SIZE(code) == BPF_DW;
+ return (BPF_SIZE(code) == BPF_DW ||BPF_MODE(code) == BPF_MEMSX);
Thanks,
Puranjay
Re: [PATCH bpf-next 01/11] bpf: Disable zero-extension for BPF_MEMSX
Quoted 67 lines · Puranjay Mohan
Quoted 55 lines · Alexei Starovoitov
Quoted 28 lines · Puranjay Mohan
Hi Alexei,
[...]
Quoted 9 lines
I guess we never clearly defined what '
needs_zext' is supposed to be, so it wouldn't be fair to call 32-bit JITs buggy.
But we better address this issue now.
This 32-bit zeroing after LDX hurts mips64, s390, ppc64, riscv64. I believe all 4 JITs emit proper zero extension into 64-bit register by using single cpu instruction,
but they also definebpf_jit_needs_zext()as true,
so extraBPF_ZEXT_REG()is added by the verifier
and it is a pure run-time overhead.I just realised that these zext instructions will not be a runtime overhead because the JITs ignore them.
Like
s390 does:
caseBPF_LDX|BPF_MEM|BPF_B: /* dst = *(u8 *)(ul) (src + off) */ caseBPF_LDX|BPF_PROBE_MEM|BPF_B:/* llgc %dst,0(off,%src) */ EMIT6_DISP_LH(0xe3000000, 0x0090, dst_reg, src_reg, REG_0, off); jit->seen |= SEEN_MEM; if (insn_is_zext(&insn[1])) insn_count = 2; /* this will skip the next zext instruction */ break;Quoted 3 lines
powerpc does after LDX:
if (size !=BPF_DW&&insn_is_zext(&insn[i + 1]))addrs[++i] = ctx->idx * 4;I see. Indeed the 64-bit JITs ignore this special zext insn after LDX.
Quoted 9 lines
Quoted 8 lines
It's better to remove
if (t !=SRC_OP)return BPF_SIZE(code) == BPF_DW;from
is_reg64()to avoid addingBPF_ZEXT_REG()insn and fix 32-bit JITs at the same time.
RISCV32, PowerPC32, x86-32 JITs fixed in the first 3 patches to always zero upper 32-bit after LDX and
then 4th patch to remove these two lines.Quoted 8 lines
I have sent the patches for above, although I think this optimization is useful because
zero extension after LDX is only required when the loaded value is later being used as
a 64-bit value. If it is not the case then the verifier will not emit the zext and 32-bit JITs will emit
1 less instruction because they expect the verifier to do the zext for them where required.Quoted 2 lines
You're correct.
Ok. Let's keep zext for LDX as-is.Yes,
let's doif (class == BPF_LDX) { if (t != SRC_OP)(quoted diff) +1
-1- return BPF_SIZE(code) == BPF_DW; + return (BPF_SIZE(code) == BPF_DW ||
Quoted 1 line
BPF_MODE(code)==BPF_MEMSX);
Agree. imo that's a cleaner approach vs changing mark_insn_zext().
Re: [PATCH bpf-next 01/11] bpf: Disable zero-extension for BPF_MEMSX
In reply to Puranjay MohanLe 13/09/2023 à 02:22, Puranjay Mohan a écrit :
Quoted 66 lines
Quoted 55 lines · Alexei Starovoitov
Quoted 28 lines · Puranjay Mohan
Hi Alexei,
[...]
Quoted 9 lines
I guess we never clearly defined what '
needs_zext' is supposed to be, so it wouldn't be fair to call 32-bit JITs buggy.
But we better address this issue now.
This 32-bit zeroing after LDX hurts mips64, s390, ppc64, riscv64. I believe all 4 JITs emit proper zero extension into 64-bit register by using single cpu instruction,
but they also definebpf_jit_needs_zext()as true,
so extraBPF_ZEXT_REG()is added by the verifier
and it is a pure run-time overhead.I just realised that these zext instructions will not be a runtime overhead because the JITs ignore them.
Like
s390 does:
caseBPF_LDX|BPF_MEM|BPF_B: /* dst = *(u8 *)(ul) (src + off) */ caseBPF_LDX|BPF_PROBE_MEM|BPF_B:/* llgc %dst,0(off,%src) */ EMIT6_DISP_LH(0xe3000000, 0x0090, dst_reg, src_reg, REG_0, off); jit->seen |= SEEN_MEM; if (insn_is_zext(&insn[1])) insn_count = 2; /* this will skip the next zext instruction */ break;Quoted 3 lines
powerpc does after LDX:
if (size !=BPF_DW&&insn_is_zext(&insn[i + 1]))addrs[++i] = ctx->idx * 4;I see. Indeed the 64-bit JITs ignore this special zext insn after LDX.
Quoted 9 lines
Quoted 8 lines
It's better to remove
if (t !=SRC_OP)return BPF_SIZE(code) == BPF_DW;from
is_reg64()to avoid addingBPF_ZEXT_REG()insn and fix 32-bit JITs at the same time.
RISCV32, PowerPC32, x86-32 JITs fixed in the first 3 patches to always zero upper 32-bit after LDX and
then 4th patch to remove these two lines.Quoted 8 lines
I have sent the patches for above, although I think this optimization is useful because
zero extension after LDX is only required when the loaded value is later being used as
a 64-bit value. If it is not the case then the verifier will not emit the zext and 32-bit JITs will emit
1 less instruction because they expect the verifier to do the zext for them where required.Quoted 2 lines
You're correct.
Ok. Let's keep zext for LDX as-is.Yes,
let's doif (class == BPF_LDX) { if (t != SRC_OP)(quoted diff) +1
-1- return BPF_SIZE(code) == BPF_DW; + return (BPF_SIZE(code) == BPF_DW ||
Quoted 1 line
BPF_MODE(code)==BPF_MEMSX);
You don't need the parenthesis, just do
return BPF_SIZE(code) == BPF_DW || BPF_MODE(code) == BPF_MEMSX;
Christophe
Ilya Leoshkevich
maintainerRe: [PATCH bpf-next 01/11] bpf: Disable zero-extension for BPF_MEMSX
In reply to Puranjay MohanQuoted 63 lines · Puranjay Mohan
Hi Ilya
Quoted 30 lines · Ilya Leoshkevich
On the architectures that use
bpf_jit_needs_zext(), e.g., s390x, the
verifier incorrectly inserts a zero-extension afterBPF_MEMSX, leading
to miscompilations like the one below:24: 89 1a ff fe 00 00 00 00 "r1 = *(s16 *)(r10 - 2);" #
zext_dstset
0x3ff7fdb910e: lgh %r2,-
2(%r13,%r0) # load halfword 0x3ff7fdb9114: llgfr
%r2,%r2 # wrong!
25: 65 10 00 03 00 00 7f ff if r1 s> 32767 goto +3 <l0_1> #check_cond_jmp_op()Disable such zero-extensions. The JITs need to insert sign- extension
themselves, if necessary.Signed-off-by Ilya Leoshkevich <[email protected]>---
kernel/bpf/verifier.c | 4 +++-1 file changed, 3 insertions(+), 1 deletion(-)kernel/bpf/verifier.c +0
-0kernel/bpf/verifier.c static void mark_insn_zext(struct · line 3114
Quoted 5 lines
kernel/bpf/verifier.c +0
-1@@ -3110,7 +3110,9 @@ static void mark_insn_zext(structbpf_verifier_env *env, { s32 def_idx = reg->subreg_def; - if (def_idx == DEF_NOT_SUBREG)The problem here is that
reg->subreg_defshould be set asDEF_NOT_SUBREGfor
registers that are used as destination registers ofBPF_LDX|BPF_MEMSX. I am seeing
the same problem on ARM32 and was going to send a patch today.The problem is that
is_reg64()returns false for destination registers
ofBPF_LDX|BPF_MEMSX.
ButBPF_LDX|BPF_MEMSXalways loads a 64 bit value because of the sign extension sois_reg64()should return true.I have written a patch that I will be sending as a reply to this. Please let me know if that makes sense.
Quoted 7 lines
kernel/bpf/verifier.c +2
-0+ if (def_idx == DEF_NOT_SUBREG || + (BPF_CLASS(env->prog->insnsi[def_idx - 1].code) ==
BPF_LDX&&
+BPF_MODE(env->prog->insnsi[def_idx - 1].code)==BPF_MEMSX))
return;
Quoted 8 lines
Quoted 5 lines
env->insn_aux_data[def_idx - 1].zext_dst = true; --
2.41.0Thanks,
Puranjay
Hi,
I also considered doing this, and I think both approaches are functionally equivalent and work in practice.
However, I can envision that, just like we have the zext_dst optimization today, we might want a sext_dst optimization in the future. Therefore I think it's better to fix this by not setting zext_dst instead of not setting subreg_def.
Best regards,
Ilya
Ilya Leoshkevich
maintainerPATCH 2/11 net: netfilter: Adjust timeouts of non-confirmed CTs in bpf_ct_insert_entry()
In reply to Ilya Leoshkevichbpf_nf testcase fails on s390x: bpf_skb_ct_lookup() cannot find the entry that was added by bpf_ct_insert_entry() within the same BPF function.
The reason is that this entry is deleted by nf_ct_gc_expired().
The CT timeout starts ticking after the CT confirmation; therefore nf_conn.timeout is initially set to the timeout value, and __nf_conntrack_confirm() sets it to the deadline value. bpf_ct_insert_entry() sets IPS_CONFIRMED_BIT, but does not adjust the timeout, making its value meaningless and causing false positives.
Fix the problem by making bpf_ct_insert_entry() adjust the timeout, like __nf_conntrack_confirm().
--- net/netfilter/nf_conntrack_bpf.c | 2 ++ 1 file changed, 2 insertions(+)
net/netfilter/nf_conntrack_bpf.c +2 -0
@@ -381,6 +381,8 @@ __bpf_kfunc struct nf_conn *bpf_ct_insert_entry(struct nf_conn___init *nfct_i)struct nf_conn *nfct = (struct nf_conn *)nfct_i;int err; +if (!nf_ct_is_confirmed (nfct)) + nfct->timeout += nfct_time_stamp; nfct->status |=IPS_CONFIRMED ; err =nf_conntrack_hash_check_insert (nfct);if (err <0 ) {
Signature · 1 line
2.41.0
Daniel Borkmann
maintainerRe: [PATCH bpf-next 02/11] net: netfilter: Adjust timeouts of non-confirmed CTs in bpf_ct_insert_entry()
[ +Florian ]
Quoted 12 lines · Ilya Leoshkevich
bpf_nftestcase fails on s390x:bpf_skb_ct_lookup()cannot find the entry that was added bybpf_ct_insert_entry()within the same BPF function.The reason is that this entry is deleted by
nf_ct_gc_expired().The CT timeout starts ticking after the CT confirmation; therefore nf_conn.timeout is initially set to the timeout value, and
__nf_conntrack_confirm()sets it to the deadline value.bpf_ct_insert_entry()setsIPS_CONFIRMED_BIT, but does not adjust the timeout, making its value meaningless and causing false positives.
Quoted 4 lines
Fix the problem by making
bpf_ct_insert_entry()adjust the timeout, like__nf_conntrack_confirm().Signed-off-by Ilya Leoshkevich <[email protected]>
Should we route this fix via bpf tree instead? Also, could you reply with a Fixes tag?
Quoted 10 lines
--- net/netfilter/nf_conntrack_bpf.c | 2 ++ 1 file changed, 2 insertions(+)net/netfilter/nf_conntrack_bpf.c +0
-0
net/netfilter/nf_conntrack_bpf.c __bpf_kfunc struct nf_conn *bpf_ct_insert_entry(struct nf_conn___init *nfct_i) · line 388
Quoted 8 lines
net/netfilter/nf_conntrack_bpf.c +2
-0@@ -381,6 +381,8 @@ __bpf_kfunc struct nf_conn *bpf_ct_insert_entry(struct nf_conn___init *nfct_i)struct nf_conn *nfct = (struct nf_conn *)nfct_i;int err; +if (!nf_ct_is_confirmed (nfct)) + nfct->timeout += nfct_time_stamp; nfct->status |=IPS_CONFIRMED ; err =nf_conntrack_hash_check_insert (nfct);if (err <0 ) {
Thanks,
Daniel
Ilya Leoshkevich
maintainerRe: [PATCH bpf-next 02/11] net: netfilter: Adjust timeouts of non-confirmed CTs in bpf_ct_insert_entry()
Quoted 16 lines · Daniel Borkmann
[ +Florian ]
Quoted 13 lines · Ilya Leoshkevich
bpf_nftestcase fails on s390x:bpf_skb_ct_lookup()cannot find the entry that was added bybpf_ct_insert_entry()within the same BPF function.The reason is that this entry is deleted by
nf_ct_gc_expired().The CT timeout starts ticking after the CT confirmation; therefore nf_conn.timeout is initially set to the timeout value, and
__nf_conntrack_confirm()sets it to the deadline value.bpf_ct_insert_entry()setsIPS_CONFIRMED_BIT, but does not adjust the
timeout, making its value meaningless and causing false positives.
Quoted 8 lines
Quoted 4 lines
Fix the problem by making
bpf_ct_insert_entry()adjust the timeout, like__nf_conntrack_confirm().Signed-off-by Ilya Leoshkevich <[email protected]>Should we route this fix via bpf tree instead? Also, could you reply with
a Fixes tag?
Yes, putting this into the bpf tree makes sense to me. Should I resend with a different subject-prefix?
of nf_conntrack_hash_check_insert()")
[...]
Ilya Leoshkevich
maintainerPATCH 3/11 selftests/bpf: Unmount the cgroup2 work directory
In reply to Ilya Leoshkevichtest_progs -t bind_perm,bpf_obj_pinning/mounted-str-rel fails when the selftests directory is mounted under /mnt, which is a reasonable thing to do when sharing the selftests residing on the host with a virtual machine, e.g., using 9p.
The reason is that cgroup2 is mounted at /mnt and not unmounted, causing subsequent tests that need to access the selftests directory to fail.
Fix by unmounting it. The kernel maintains a mount stack, so this reveals what was mounted there before. Introduce cgroup_workdir_mounted in order to maintain idempotency. Make it thread-local in order to support test_progs -j.
--- tools/testing/selftests/bpf/cgroup_helpers.c | 33 +++++++++++++++----- 1 file changed, 26 insertions(+), 7 deletions(-)
tools/testing/selftests/bpf/cgroup_helpers.c +26 -7
@@ -49,6 +49,10 @@snprintf (buf,sizeof (buf),"%s%s" ,NETCLS_MOUNT_PATH , \ CGROUP_WORK_DIR) +static __threadbool cgroup_workdir_mounted; + +static void __cleanup_cgroup_environment (void ); +static int __enable_controllers (const char *cgroup_path,const char *controllers) {char path[PATH_MAX +1 ];
@@ -209,9 +213,10 @@ int setup_cgroup_environment(void)log_err ("mount cgroup2" );return 1 ; } + cgroup_workdir_mounted = true;/* Cleanup existing failed runs, now that the environment is setup */ -cleanup_cgroup_environment (); +__cleanup_cgroup_environment ();if (mkdir (cgroup_workdir,0777 ) && errno !=EEXIST ) {log_err ("mkdir cgroup work dir" );
@@ -305,11 +310,26 @@ int join_parent_cgroup(const char *relative_path)return join_cgroup_from_top (cgroup_path); } +/** +* __cleanup_cgroup_environment() - Delete temporary cgroups +* +* This is a helper for cleanup_cgroup_environment() that is responsible for +* deletion of all temporary cgroups that have been created during the test. +*/ +static void __cleanup_cgroup_environment (void ) +{ +char cgroup_workdir[PATH_MAX +1 ]; + +format_cgroup_path (cgroup_workdir,"" ); +join_cgroup_from_top (CGROUP_MOUNT_PATH ); +nftw (cgroup_workdir, nftwfunc,WALK_FD_LIMIT ,FTW_DEPTH |FTW_MOUNT ); +} +/** * cleanup_cgroup_environment() - Cleanup Cgroup Testing Environment * * This is an idempotent function to delete all temporary cgroups that -* have been created during the test, including the cgroup testing work +* have been created during the test and unmount the cgroup testing work * directory. * * At call time, it moves the calling process to the root cgroup, and then
@@ -320,11 +340,10 @@ int join_parent_cgroup(const char *relative_path)*/ void cleanup_cgroup_environment (void ) { -char cgroup_workdir[PATH_MAX +1 ]; - -format_cgroup_path (cgroup_workdir,"" ); -join_cgroup_from_top (CGROUP_MOUNT_PATH ); -nftw (cgroup_workdir, nftwfunc,WALK_FD_LIMIT ,FTW_DEPTH |FTW_MOUNT ); +__cleanup_cgroup_environment (); +if (cgroup_workdir_mounted &&umount (CGROUP_MOUNT_PATH )) +log_err ("umount cgroup2" ); + cgroup_workdir_mounted = false; }/**
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH 4/11 selftests/bpf: Add big-endian support to the ldsx test
In reply to Ilya LeoshkevichPrepare the ldsx test to run on big-endian systems by adding the necessary endianness checks around narrow memory accesses.
--- .../selftests/bpf/progs/test_ldsx_insn.c | 6 +- .../selftests/bpf/progs/verifier_ldsx.c | 146 ++++++++++-------- 2 files changed, 90 insertions(+), 62 deletions(-)
tools/testing/selftests/bpf/progs/test_ldsx_insn.c +5 -1
@@ -104,7 +104,11 @@ int _tc(volatile struct __sk_buff *skb)"%[tmp_mark] = r1" : [tmp_mark]"=r"(tmp_mark) : [ctx]"r"(skb), - [off_mark]"i"(offsetof(struct __sk_buff, mark)) + [off_mark]"i"(offsetof(struct __sk_buff, mark) +#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__ + +sizeof (skb->mark) -1 +#endif + ) : "r1");#else tmp_mark = (char )skb->mark;
tools/testing/selftests/bpf/progs/verifier_ldsx.c +85 -61
182 lines not on this page · raw message
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH 5/11 s390/bpf: Implement BPF_MOV | BPF_X with sign-extension
In reply to Ilya LeoshkevichImplement the cpuv4 register-to-register move with sign extension. It is distinguished from the normal moves by non-zero values in insn->off, which determine the source size. s390x has instructions to deal with all of them: lbr, lhr, lgbr, lghr and lgfr.
--- arch/s390/net/bpf_jit_comp.c | 48 ++++++++++++++++++++++++++++++------ 1 file changed, 40 insertions(+), 8 deletions(-)
arch/s390/net/bpf_jit_comp.c +40 -8
@@ -795,15 +795,47 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,/* * BPF_MOV */ -case BPF_ALU |BPF_MOV |BPF_X :/* dst = (u32) src */ -/* llgfr %dst,%src */ -EMIT4 (0xb9160000 , dst_reg, src_reg); -if (insn_is_zext (&insn[1 ])) - insn_count =2 ; +case BPF_ALU |BPF_MOV |BPF_X : +switch (insn->off) { +case 0 :/* DST = (u32) SRC */ +/* llgfr %dst,%src */ +EMIT4 (0xb9160000 , dst_reg, src_reg); +if (insn_is_zext (&insn[1 ])) + insn_count =2 ; +break ; +case 8 :/* DST = (u32)(s8) SRC */ +/* lbr %dst,%src */ +EMIT4 (0xb9260000 , dst_reg, src_reg); +/* llgfr %dst,%dst */ +EMIT4 (0xb9160000 , dst_reg, dst_reg); +break ; +case 16 :/* DST = (u32)(s16) SRC */ +/* lhr %dst,%src */ +EMIT4 (0xb9270000 , dst_reg, src_reg); +/* llgfr %dst,%dst */ +EMIT4 (0xb9160000 , dst_reg, dst_reg); +break ; + }break ; -case BPF_ALU64 |BPF_MOV |BPF_X :/* dst = src */ -/* lgr %dst,%src */ -EMIT4 (0xb9040000 , dst_reg, src_reg); +case BPF_ALU64 |BPF_MOV |BPF_X : +switch (insn->off) { +case 0 :/* DST = SRC */ +/* lgr %dst,%src */ +EMIT4 (0xb9040000 , dst_reg, src_reg); +break ; +case 8 :/* DST = (s8) SRC */ +/* lgbr %dst,%src */ +EMIT4 (0xb9060000 , dst_reg, src_reg); +break ; +case 16 :/* DST = (s16) SRC */ +/* lghr %dst,%src */ +EMIT4 (0xb9070000 , dst_reg, src_reg); +break ; +case 32 :/* DST = (s32) SRC */ +/* lgfr %dst,%src */ +EMIT4 (0xb9140000 , dst_reg, src_reg); +break ; + }break ;case BPF_ALU |BPF_MOV |BPF_K :/* dst = (u32) imm */ /* llilf %dst,imm */
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH 6/11 s390/bpf: Implement BPF_MEMSX
In reply to Ilya LeoshkevichImplement the cpuv4 load with sign-extension, which is encoded as BPF_MEMSX (and, for internal uses cases only, BPF_PROBE_MEMSX).
This is the same as BPF_MEM and BPF_PROBE_MEM, but with sign extension instead of zero extension, and s390x has the necessary
instructions: lgb, lgh and lgf.
--- arch/s390/net/bpf_jit_comp.c | 32 +++++++++++++++++++++++++++----- 1 file changed, 27 insertions(+), 5 deletions(-)
arch/s390/net/bpf_jit_comp.c +27 -5
@@ -670,15 +670,18 @@ static void bpf_jit_epilogue(struct bpf_jit *jit, u32 stack_depth)static int get_probe_mem_regno (const u8 *insn) {/* -* insn must point to llgc, llgh, llgf or lg, which have destination -* register at the same position. +* insn must point to llgc, llgh, llgf, lg, lgb, lgh or lgf, which have +* destination register at the same position. */ -if (insn[0 ] !=0xe3 )/* common llgc, llgh, llgf and lg prefix */ +if (insn[0 ] !=0xe3 )/* common prefix */ return -1 ;if (insn[5 ] !=0x90 &&/* llgc */ insn[5 ] !=0x91 &&/* llgh */ insn[5 ] !=0x16 &&/* llgf */ - insn[5 ] !=0x04 )/* lg */ + insn[5 ] !=0x04 &&/* lg */ + insn[5 ] !=0x77 &&/* lgb */ + insn[5 ] !=0x15 &&/* lgh */ + insn[5 ] !=0x14 )/* lgf */ return -1 ;return insn[1 ] >>4 ; }
@@ -788,7 +791,8 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,int err;if (BPF_CLASS (insn->code) ==BPF_LDX && -BPF_MODE (insn->code) ==BPF_PROBE_MEM ) + (BPF_MODE (insn->code) ==BPF_PROBE_MEM || +BPF_MODE (insn->code) ==BPF_PROBE_MEMSX )) probe_prg = jit->prg;switch (insn->code) {
@@ -1406,6 +1410,12 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,if (insn_is_zext (&insn[1 ])) insn_count =2 ;break ; +case BPF_LDX |BPF_MEMSX |BPF_B :/* dst = *(s8 *)(ul) (src + off) */ +case BPF_LDX |BPF_PROBE_MEMSX |BPF_B : +/* lgb %dst,0(off,%src) */ +EMIT6_DISP_LH (0xe3000000 ,0x0077 , dst_reg, src_reg,REG_0 , off); + jit->seen |=SEEN_MEM ; +break ;case BPF_LDX |BPF_MEM |BPF_H :/* dst = *(u16 *)(ul) (src + off) */ case BPF_LDX |BPF_PROBE_MEM |BPF_H :/* llgh %dst,0(off,%src) */
@@ -1414,6 +1424,12 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,if (insn_is_zext (&insn[1 ])) insn_count =2 ;break ; +case BPF_LDX |BPF_MEMSX |BPF_H :/* dst = *(s16 *)(ul) (src + off) */ +case BPF_LDX |BPF_PROBE_MEMSX |BPF_H : +/* lgh %dst,0(off,%src) */ +EMIT6_DISP_LH (0xe3000000 ,0x0015 , dst_reg, src_reg,REG_0 , off); + jit->seen |=SEEN_MEM ; +break ;case BPF_LDX |BPF_MEM |BPF_W :/* dst = *(u32 *)(ul) (src + off) */ case BPF_LDX |BPF_PROBE_MEM |BPF_W :/* llgf %dst,off(%src) */
@@ -1422,6 +1438,12 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,if (insn_is_zext (&insn[1 ])) insn_count =2 ;break ; +case BPF_LDX |BPF_MEMSX |BPF_W :/* dst = *(s32 *)(ul) (src + off) */ +case BPF_LDX |BPF_PROBE_MEMSX |BPF_W : +/* lgf %dst,off(%src) */ + jit->seen |=SEEN_MEM ; +EMIT6_DISP_LH (0xe3000000 ,0x0014 , dst_reg, src_reg,REG_0 , off); +break ;case BPF_LDX |BPF_MEM |BPF_DW :/* dst = *(u64 *)(ul) (src + off) */ case BPF_LDX |BPF_PROBE_MEM |BPF_DW :/* lg %dst,0(off,%src) */
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH 7/11 s390/bpf: Implement unconditional byte swap
In reply to Ilya LeoshkevichImplement the cpuv4 unconditional byte swap, which is encoded as BPF_ALU64 | BPF_END | BPF_FROM_LE. Since s390x is big-endian, it's the same as the existing BPF_ALU | BPF_END | BPF_FROM_LE.
--- arch/s390/net/bpf_jit_comp.c | 1 + 1 file changed, 1 insertion(+)
arch/s390/net/bpf_jit_comp.c +1 -0
@@ -1253,6 +1253,7 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, }break ;case BPF_ALU |BPF_END |BPF_FROM_LE : +case BPF_ALU64 |BPF_END |BPF_FROM_LE :switch (imm) {case 16 :/* dst = (u16) cpu_to_le16(dst) */ /* lrvr %dst,%dst */
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH 8/11 s390/bpf: Implement unconditional jump with 32-bit offset
In reply to Ilya LeoshkevichImplement the cpuv4 unconditional jump with 32-bit offset, which is encoded as BPF_JMP32 | BPF_JA and stores the offset in the imm field. Reuse the existing BPF_JMP | BPF_JA logic.
--- arch/s390/net/bpf_jit_comp.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
arch/s390/net/bpf_jit_comp.c +9 -3
@@ -779,6 +779,7 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, int i, bool extra_pass, u32 stack_depth) { struct bpf_insn *insn = &fp->insnsi[i]; +s16 branch_oc_off = insn->off;u32 dst_reg = insn->dst_reg;u32 src_reg = insn->src_reg;int last, insn_count =1 ;
@@ -1625,6 +1626,9 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,* instruction itself (loop) and for BPF with offset 0 we * branch to the instruction behind the branch. */ +case BPF_JMP32 |BPF_JA :/* if (true) */ + branch_oc_off = imm; + fallthrough;case BPF_JMP |BPF_JA :/* if (true) */ mask =0xf000 ;/* j */ goto branch_oc;
@@ -1793,14 +1797,16 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,break ; branch_oc:if (!is_first_pass (jit) && -can_use_rel (jit, addrs[i + off +1 ])) { +can_use_rel (jit, addrs[i + branch_oc_off +1 ])) {/* brc mask,off */ EMIT4_PCREL_RIC (0xa7040000 , - mask >>12 , addrs[i + off +1 ]); + mask >>12 , + addrs[i + branch_oc_off +1 ]); }else {/* brcl mask,off */ EMIT6_PCREL_RILC (0xc0040000 , - mask >>12 , addrs[i + off +1 ]); + mask >>12 , + addrs[i + branch_oc_off +1 ]); }break ; }
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH 9/11 s390/bpf: Implement signed division
In reply to Ilya LeoshkevichImplement the cpuv4 signed division. It is encoded as unsigned division, but with off field set to 1. s390x has the necessary
instructions: dsgfr, dsgf and dsgr.
--- arch/s390/net/bpf_jit_comp.c | 172 +++++++++++++++++++++++++---------- 1 file changed, 125 insertions(+), 47 deletions(-)
arch/s390/net/bpf_jit_comp.c +125 -47
@@ -949,66 +949,115 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,/* * BPF_DIV / BPF_MOD */ -case BPF_ALU |BPF_DIV |BPF_X :/* dst = (u32) dst / (u32) src */ -case BPF_ALU |BPF_MOD |BPF_X :/* dst = (u32) dst % (u32) src */ +case BPF_ALU |BPF_DIV |BPF_X : +case BPF_ALU |BPF_MOD |BPF_X : {int rc_reg =BPF_OP (insn->code) ==BPF_DIV ?REG_W1 :REG_W0 ; -/* lhi %w0,0 */ -EMIT4_IMM (0xa7080000 ,REG_W0 ,0 ); -/* lr %w1,%dst */ -EMIT2 (0x1800 ,REG_W1 , dst_reg); -/* dlr %w0,%src */ -EMIT4 (0xb9970000 ,REG_W0 , src_reg); +switch (off) { +case 0 :/* dst = (u32) dst {/,%} (u32) src */ +/* xr %w0,%w0 */ +EMIT2 (0x1700 ,REG_W0 ,REG_W0 ); +/* lr %w1,%dst */ +EMIT2 (0x1800 ,REG_W1 , dst_reg); +/* dlr %w0,%src */ +EMIT4 (0xb9970000 ,REG_W0 , src_reg); +break ; +case 1 :/* dst = (u32) ((s32) dst {/,%} (s32) src) */ +/* lgfr %r1,%dst */ +EMIT4 (0xb9140000 ,REG_W1 , dst_reg); +/* dsgfr %r0,%src */ +EMIT4 (0xb91d0000 ,REG_W0 , src_reg); +break ; + }/* llgfr %dst,%rc */ EMIT4 (0xb9160000 , dst_reg, rc_reg);if (insn_is_zext (&insn[1 ])) insn_count =2 ;break ; } -case BPF_ALU64 |BPF_DIV |BPF_X :/* dst = dst / src */ -case BPF_ALU64 |BPF_MOD |BPF_X :/* dst = dst % src */ +case BPF_ALU64 |BPF_DIV |BPF_X : +case BPF_ALU64 |BPF_MOD |BPF_X : {int rc_reg =BPF_OP (insn->code) ==BPF_DIV ?REG_W1 :REG_W0 ; -/* lghi %w0,0 */ -EMIT4_IMM (0xa7090000 ,REG_W0 ,0 ); -/* lgr %w1,%dst */ -EMIT4 (0xb9040000 ,REG_W1 , dst_reg); -/* dlgr %w0,%dst */ -EMIT4 (0xb9870000 ,REG_W0 , src_reg); +switch (off) { +case 0 :/* dst = dst {/,%} src */ +/* lghi %w0,0 */ +EMIT4_IMM (0xa7090000 ,REG_W0 ,0 ); +/* lgr %w1,%dst */ +EMIT4 (0xb9040000 ,REG_W1 , dst_reg); +/* dlgr %w0,%src */ +EMIT4 (0xb9870000 ,REG_W0 , src_reg); +break ; +case 1 :/* dst = (s64) dst {/,%} (s64) src */ +/* lgr %w1,%dst */ +EMIT4 (0xb9040000 ,REG_W1 , dst_reg); +/* dsgr %w0,%src */ +EMIT4 (0xb90d0000 ,REG_W0 , src_reg); +break ; + }/* lgr %dst,%rc */ EMIT4 (0xb9040000 , dst_reg, rc_reg);break ; } -case BPF_ALU |BPF_DIV |BPF_K :/* dst = (u32) dst / (u32) imm */ -case BPF_ALU |BPF_MOD |BPF_K :/* dst = (u32) dst % (u32) imm */ +case BPF_ALU |BPF_DIV |BPF_K : +case BPF_ALU |BPF_MOD |BPF_K : {int rc_reg =BPF_OP (insn->code) ==BPF_DIV ?REG_W1 :REG_W0 ;if (imm ==1 ) {if (BPF_OP (insn->code) ==BPF_MOD ) -/* lhgi %dst,0 */ +/* lghi %dst,0 */ EMIT4_IMM (0xa7090000 , dst_reg,0 );else EMIT_ZERO (dst_reg);break ; } -/* lhi %w0,0 */ -EMIT4_IMM (0xa7080000 ,REG_W0 ,0 ); -/* lr %w1,%dst */ -EMIT2 (0x1800 ,REG_W1 , dst_reg);if (!is_first_pass (jit) &&can_use_ldisp_for_lit32 (jit)) { -/* dl %w0,<d(imm)>(%l) */ -EMIT6_DISP_LH (0xe3000000 ,0x0097 ,REG_W0 ,REG_0 ,REG_L , -EMIT_CONST_U32 (imm)); +switch (off) { +case 0 :/* dst = (u32) dst {/,%} (u32) imm */ +/* xr %w0,%w0 */ +EMIT2 (0x1700 ,REG_W0 ,REG_W0 ); +/* lr %w1,%dst */ +EMIT2 (0x1800 ,REG_W1 , dst_reg); +/* dl %w0,<d(imm)>(%l) */ +EMIT6_DISP_LH (0xe3000000 ,0x0097 ,REG_W0 ,REG_0 , +REG_L ,EMIT_CONST_U32 (imm)); +break ; +case 1 :/* dst = (s32) dst {/,%} (s32) imm */ +/* lgfr %r1,%dst */ +EMIT4 (0xb9140000 ,REG_W1 , dst_reg); +/* dsgf %r0,<d(imm)>(%l) */ +EMIT6_DISP_LH (0xe3000000 ,0x001d ,REG_W0 ,REG_0 , +REG_L ,EMIT_CONST_U32 (imm)); +break ; + } }else { -/* lgfrl %dst,imm */ -EMIT6_PCREL_RILB (0xc40c0000 , dst_reg, -_EMIT_CONST_U32 (imm)); - jit->seen |=SEEN_LITERAL ; -/* dlr %w0,%dst */ -EMIT4 (0xb9970000 ,REG_W0 , dst_reg); +switch (off) { +case 0 :/* dst = (u32) dst {/,%} (u32) imm */ +/* xr %w0,%w0 */ +EMIT2 (0x1700 ,REG_W0 ,REG_W0 ); +/* lr %w1,%dst */ +EMIT2 (0x1800 ,REG_W1 , dst_reg); +/* lrl %dst,imm */ +EMIT6_PCREL_RILB (0xc40d0000 , dst_reg, +_EMIT_CONST_U32 (imm)); + jit->seen |=SEEN_LITERAL ; +/* dlr %w0,%dst */ +EMIT4 (0xb9970000 ,REG_W0 , dst_reg); +break ; +case 1 :/* dst = (s32) dst {/,%} (s32) imm */ +/* lgfr %w1,%dst */ +EMIT4 (0xb9140000 ,REG_W1 , dst_reg); +/* lgfrl %dst,imm */ +EMIT6_PCREL_RILB (0xc40c0000 , dst_reg, +_EMIT_CONST_U32 (imm)); + jit->seen |=SEEN_LITERAL ; +/* dsgr %w0,%dst */ +EMIT4 (0xb90d0000 ,REG_W0 , dst_reg); +break ; + } }/* llgfr %dst,%rc */ EMIT4 (0xb9160000 , dst_reg, rc_reg);
@@ -1016,8 +1065,8 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, insn_count =2 ;break ; } -case BPF_ALU64 |BPF_DIV |BPF_K :/* dst = dst / imm */ -case BPF_ALU64 |BPF_MOD |BPF_K :/* dst = dst % imm */ +case BPF_ALU64 |BPF_DIV |BPF_K : +case BPF_ALU64 |BPF_MOD |BPF_K : {int rc_reg =BPF_OP (insn->code) ==BPF_DIV ?REG_W1 :REG_W0 ;
@@ -1027,21 +1076,50 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,EMIT4_IMM (0xa7090000 , dst_reg,0 );break ; } -/* lghi %w0,0 */ -EMIT4_IMM (0xa7090000 ,REG_W0 ,0 ); -/* lgr %w1,%dst */ -EMIT4 (0xb9040000 ,REG_W1 , dst_reg);if (!is_first_pass (jit) &&can_use_ldisp_for_lit64 (jit)) { -/* dlg %w0,<d(imm)>(%l) */ -EMIT6_DISP_LH (0xe3000000 ,0x0087 ,REG_W0 ,REG_0 ,REG_L , -EMIT_CONST_U64 (imm)); +switch (off) { +case 0 :/* dst = dst {/,%} imm */ +/* lghi %w0,0 */ +EMIT4_IMM (0xa7090000 ,REG_W0 ,0 ); +/* lgr %w1,%dst */ +EMIT4 (0xb9040000 ,REG_W1 , dst_reg); +/* dlg %w0,<d(imm)>(%l) */ +EMIT6_DISP_LH (0xe3000000 ,0x0087 ,REG_W0 ,REG_0 , +REG_L ,EMIT_CONST_U64 (imm)); +break ; +case 1 :/* dst = (s64) dst {/,%} (s64) imm */ +/* lgr %w1,%dst */ +EMIT4 (0xb9040000 ,REG_W1 , dst_reg); +/* dsg %w0,<d(imm)>(%l) */ +EMIT6_DISP_LH (0xe3000000 ,0x000d ,REG_W0 ,REG_0 , +REG_L ,EMIT_CONST_U64 (imm)); +break ; + } }else { -/* lgrl %dst,imm */ -EMIT6_PCREL_RILB (0xc4080000 , dst_reg, -_EMIT_CONST_U64 (imm)); - jit->seen |=SEEN_LITERAL ; -/* dlgr %w0,%dst */ -EMIT4 (0xb9870000 ,REG_W0 , dst_reg); +switch (off) { +case 0 :/* dst = dst {/,%} imm */ +/* lghi %w0,0 */ +EMIT4_IMM (0xa7090000 ,REG_W0 ,0 ); +/* lgr %w1,%dst */ +EMIT4 (0xb9040000 ,REG_W1 , dst_reg); +/* lgrl %dst,imm */ +EMIT6_PCREL_RILB (0xc4080000 , dst_reg, +_EMIT_CONST_U64 (imm)); + jit->seen |=SEEN_LITERAL ; +/* dlgr %w0,%dst */ +EMIT4 (0xb9870000 ,REG_W0 , dst_reg); +break ; +case 1 :/* dst = (s64) dst {/,%} (s64) imm */ +/* lgr %w1,%dst */ +EMIT4 (0xb9040000 ,REG_W1 , dst_reg); +/* lgrl %dst,imm */ +EMIT6_PCREL_RILB (0xc4080000 , dst_reg, +_EMIT_CONST_U64 (imm)); + jit->seen |=SEEN_LITERAL ; +/* dsgr %w0,%dst */ +EMIT4 (0xb90d0000 ,REG_W0 , dst_reg); +break ; + } }/* lgr %dst,%rc */ EMIT4 (0xb9040000 , dst_reg, rc_reg);
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH 10/11 selftests/bpf: Enable the cpuv4 tests for s390x
In reply to Ilya LeoshkevichNow that all the cpuv4 support is in place, enable the tests.
--- tools/testing/selftests/bpf/progs/test_ldsx_insn.c | 3 ++- tools/testing/selftests/bpf/progs/verifier_bswap.c | 3 ++- tools/testing/selftests/bpf/progs/verifier_gotol.c | 3 ++- tools/testing/selftests/bpf/progs/verifier_ldsx.c | 3 ++- tools/testing/selftests/bpf/progs/verifier_movsx.c | 3 ++- tools/testing/selftests/bpf/progs/verifier_sdiv.c | 3 ++- 6 files changed, 12 insertions(+), 6 deletions(-)
tools/testing/selftests/bpf/progs/test_ldsx_insn.c +2 -1
@@ -6,7 +6,8 @@#include <bpf/bpf_tracing.h> #if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \ - (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 )) && __clang_major__ >=18 + (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 ) || \ + defined(__TARGET_ARCH_s390)) && __clang_major__ >=18 const volatile int skip =0 ;#else const volatile int skip =1 ;
tools/testing/selftests/bpf/progs/verifier_bswap.c +2 -1
@@ -5,7 +5,8 @@#include "bpf_misc.h" #if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \ - (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 )) && __clang_major__ >=18 + (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 ) || \ + defined(__TARGET_ARCH_s390)) && __clang_major__ >=18 SEC("socket")__description ("BSWAP, 16" )
tools/testing/selftests/bpf/progs/verifier_gotol.c +2 -1
@@ -5,7 +5,8 @@#include "bpf_misc.h" #if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \ - (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 )) && __clang_major__ >=18 + (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 ) || \ + defined(__TARGET_ARCH_s390)) && __clang_major__ >=18 SEC("socket")__description ("gotol, small_imm" )
tools/testing/selftests/bpf/progs/verifier_ldsx.c +2 -1
@@ -5,7 +5,8 @@#include "bpf_misc.h" #if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \ - (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 )) && __clang_major__ >=18 + (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 ) || \ + defined(__TARGET_ARCH_s390)) && __clang_major__ >=18 SEC("socket")__description ("LDSX, S8" )
tools/testing/selftests/bpf/progs/verifier_movsx.c +2 -1
@@ -5,7 +5,8 @@#include "bpf_misc.h" #if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \ - (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 )) && __clang_major__ >=18 + (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 ) || \ + defined(__TARGET_ARCH_s390)) && __clang_major__ >=18 SEC("socket")__description ("MOV32SX, S8" )
tools/testing/selftests/bpf/progs/verifier_sdiv.c +2 -1
@@ -5,7 +5,8 @@#include "bpf_misc.h" #if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \ - (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 )) && __clang_major__ >=18 + (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 ) || \ + defined(__TARGET_ARCH_s390)) && __clang_major__ >=18 SEC("socket")__description ("SDIV32, non-zero imm divisor, check 1" )
Signature · 1 line
2.41.0
Re: [PATCH bpf-next 10/11] selftests/bpf: Enable the cpuv4 tests for s390x
Quoted 3 lines · Ilya Leoshkevich
Now that all the cpuv4 support is in place, enable the tests.
Signed-off-by Ilya Leoshkevich <[email protected]>
Ilya Leoshkevich
maintainerPATCH 11/11 selftests/bpf: Trim DENYLIST.s390x
In reply to Ilya LeoshkevichEnable all selftests, except the 2 that have to do with the userspace unwinding, in the s390x CI.
--- tools/testing/selftests/bpf/DENYLIST.s390x | 25 ---------------------- 1 file changed, 25 deletions(-)
tools/testing/selftests/bpf/DENYLIST.s390x +0 -25
@@ -1,29 +1,4 @@ # TEMPORARY # Alphabetical order -bloom_filter_map # failed to find kernel BTF type ID of '__x64_sys_getpgid': -3 (?) -bpf_cookie # failed to open_and_load program: -524 (trampoline) -bpf_loop # attaches to __x64_sys_nanosleep -cgrp_local_storage # prog_attach unexpected error: -524 (trampoline) -dynptr/test_dynptr_skb_data -dynptr/test_skb_readonly -fexit_sleep # fexit_skel_load fexit skeleton failed (trampoline) get_stack_raw_tp # user_stack corrupted user stack (no backchain userspace) -iters/testmod_seq* # s390x doesn't support kfuncs in modules yet -kprobe_multi_bench_attach # bpf_program__attach_kprobe_multi_opts unexpected error: -95 -kprobe_multi_test # relies on fentry -ksyms_btf/weak_ksyms* # test_ksyms_weak__open_and_load unexpected error: -22 (kfunc) -ksyms_module # test_ksyms_module__open_and_load unexpected error: -9 (?) -ksyms_module_libbpf # JIT does not support calling kernel function (kfunc) -ksyms_module_lskel # test_ksyms_module_lskel__open_and_load unexpected error: -9 (?) -module_attach # skel_attach skeleton attach failed: -524 (trampoline) -ringbuf # skel_load skeleton load failed (?) stacktrace_build_id # compare_map_keys stackid_hmap vs. stackmap err -2 errno 2 (?) -test_lsm # attach unexpected error: -524 (trampoline) -trace_printk # trace_printk__load unexpected error: -2 (errno 2) (?) -trace_vprintk # trace_vprintk__open_and_load unexpected error: -9 (?) -unpriv_bpf_disabled # fentry -user_ringbuf # failed to find kernel BTF type ID of '__s390x_sys_prctl': -3 (?) -verif_stats # trace_vprintk__open_and_load unexpected error: -9 (?) -xdp_bonding # failed to auto-attach program 'trace_on_entry': -524 (trampoline) -xdp_metadata # JIT does not support calling kernel function (kfunc) -test_task_under_cgroup # JIT does not support calling kernel function (kfunc)
Signature · 1 line
2.41.0
This series was applied to bpf/bpf.git (master) by Daniel Borkmann <[email protected]>:
Hello:
This series was applied to bpf/bpf.git (master)
by Daniel Borkmann <[email protected]>:
Quoted 8 lines · On Wed, 30 Aug 2023 03:07:41 +0200 you wrote:
Hi,
This series adds the cpuv4 support to the s390x eBPF JIT. Patches 1-4 are preliminary bugfixes.
Patches 5-9 implement the new instructions. Patches 10-11 enable the tests.[...]
Here is the summary with links:
- [bpf-next,01/11] bpf: Disable zero-extension for BPF_MEMSX
(no matching commit)
- [bpf-next,02/11] net: netfilter: Adjust timeouts of non-confirmed CTs in bpf_ct_insert_entry()
https://git.kernel.org/bpf/bpf/c/6bd5bcb18f94
- [bpf-next,03/11] selftests/bpf: Unmount the cgroup2 work directory (no matching commit)
- [bpf-next,04/11] selftests/bpf: Add big-endian support to the ldsx test (no matching commit)
- [bpf-next,05/11] s390/bpf: Implement BPF_MOV | BPF_X with sign-extension (no matching commit)
- [bpf-next,06/11] s390/bpf: Implement BPF_MEMSX
(no matching commit)
- [bpf-next,07/11] s390/bpf: Implement unconditional byte swap (no matching commit)
- [bpf-next,08/11] s390/bpf: Implement unconditional jump with 32-bit offset (no matching commit)
- [bpf-next,09/11] s390/bpf: Implement signed division
(no matching commit)
- [bpf-next,10/11] selftests/bpf: Enable the cpuv4 tests for s390x (no matching commit)
- [bpf-next,11/11] selftests/bpf: Trim DENYLIST.s390x
(no matching commit)
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html