Implement cpuv4 support for s390x
Subsystems (6)
Ilya Leoshkevich
maintainerv2 cover Implement cpuv4 support for s390x
v1: https://lore.kernel.org/bpf/[email protected]/
v1 -> v2:
- Redo Disable zero-extension for BPF_MEMSX as Puranjay and Alexei suggested.
- Drop the bpf_ct_insert_entry() patch, it went in via the bpf tree.
- Rebase, don't apply A-bs because there were fixed conflicts.
Hi,
This series adds the cpuv4 support to the s390x eBPF JIT. Patches 1-3 are preliminary bugfixes.
Patches 4-8 implement the new instructions. Patches 9-10 enable the tests.
Best regards,
Ilya
Ilya Leoshkevich (10):
bpf: Disable zero-extension for BPF_MEMSX selftests/bpf: Unmount the cgroup2 work directory selftests/bpf: Add big-endian support to the ldsx test s390/bpf: Implement BPF_MOV | BPF_X with sign-extension s390/bpf: Implement BPF_MEMSX
s390/bpf: Implement unconditional byte swap s390/bpf: Implement unconditional jump with 32-bit offset s390/bpf: Implement signed division selftests/bpf: Enable the cpuv4 tests for s390x selftests/bpf: Trim DENYLIST.s390x
arch/s390/net/bpf_jit_comp.c | 265 +++++++++++++----- kernel/bpf/verifier.c | 2 +- tools/testing/selftests/bpf/DENYLIST.s390x | 25 -- tools/testing/selftests/bpf/cgroup_helpers.c | 33 ++- .../selftests/bpf/progs/test_ldsx_insn.c | 9 +- .../selftests/bpf/progs/verifier_bswap.c | 3 +- .../selftests/bpf/progs/verifier_gotol.c | 3 +- .../selftests/bpf/progs/verifier_ldsx.c | 149 ++++++---- .../selftests/bpf/progs/verifier_movsx.c | 3 +- .../selftests/bpf/progs/verifier_sdiv.c | 3 +- 10 files changed, 331 insertions(+), 164 deletions(-)
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH v2 1/10 bpf: Disable zero-extension for BPF_MEMSX
On the architectures that use bpf_jit_needs_zext(), e.g., s390x, the verifier incorrectly inserts a zero-extension after BPF_MEMSX, leading to miscompilations like the one below:
24: 89 1a ff fe 00 00 00 00 "r1 = *(s16 *)(r10 - 2);" # zext_dst set
0x3ff7fdb910e: lgh %r2,-2(%r13,%r0) # load halfword
0x3ff7fdb9114: llgfr %r2,%r2 # wrong!
25: 65 10 00 03 00 00 7f ff if r1 s> 32767 goto +3 <l0_1> # check_cond_jmp_op()Disable such zero-extensions. The JITs need to insert sign-extension themselves, if necessary.
--- kernel/bpf/verifier.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/bpf/verifier.c +1 -1
@@ -3114,7 +3114,7 @@ static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn,if (class ==BPF_LDX ) {if (t !=SRC_OP ) -return BPF_SIZE (code) ==BPF_DW ; +return BPF_SIZE (code) ==BPF_DW ||BPF_MODE (code) ==BPF_MEMSX ;/* LDX source must be ptr. */ return true; }
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH v2 2/10 selftests/bpf: Unmount the cgroup2 work directory
In reply to Ilya Leoshkevichtest_progs -t bind_perm,bpf_obj_pinning/mounted-str-rel fails when the selftests directory is mounted under /mnt, which is a reasonable thing to do when sharing the selftests residing on the host with a virtual machine, e.g., using 9p.
The reason is that cgroup2 is mounted at /mnt and not unmounted, causing subsequent tests that need to access the selftests directory to fail.
Fix by unmounting it. The kernel maintains a mount stack, so this reveals what was mounted there before. Introduce cgroup_workdir_mounted in order to maintain idempotency. Make it thread-local in order to support test_progs -j.
--- tools/testing/selftests/bpf/cgroup_helpers.c | 33 +++++++++++++++----- 1 file changed, 26 insertions(+), 7 deletions(-)
tools/testing/selftests/bpf/cgroup_helpers.c +26 -7
@@ -49,6 +49,10 @@snprintf (buf,sizeof (buf),"%s%s" ,NETCLS_MOUNT_PATH , \ CGROUP_WORK_DIR) +static __threadbool cgroup_workdir_mounted; + +static void __cleanup_cgroup_environment (void ); +static int __enable_controllers (const char *cgroup_path,const char *controllers) {char path[PATH_MAX +1 ];
@@ -209,9 +213,10 @@ int setup_cgroup_environment(void)log_err ("mount cgroup2" );return 1 ; } + cgroup_workdir_mounted = true;/* Cleanup existing failed runs, now that the environment is setup */ -cleanup_cgroup_environment (); +__cleanup_cgroup_environment ();if (mkdir (cgroup_workdir,0777 ) && errno !=EEXIST ) {log_err ("mkdir cgroup work dir" );
@@ -305,11 +310,26 @@ int join_parent_cgroup(const char *relative_path)return join_cgroup_from_top (cgroup_path); } +/** +* __cleanup_cgroup_environment() - Delete temporary cgroups +* +* This is a helper for cleanup_cgroup_environment() that is responsible for +* deletion of all temporary cgroups that have been created during the test. +*/ +static void __cleanup_cgroup_environment (void ) +{ +char cgroup_workdir[PATH_MAX +1 ]; + +format_cgroup_path (cgroup_workdir,"" ); +join_cgroup_from_top (CGROUP_MOUNT_PATH ); +nftw (cgroup_workdir, nftwfunc,WALK_FD_LIMIT ,FTW_DEPTH |FTW_MOUNT ); +} +/** * cleanup_cgroup_environment() - Cleanup Cgroup Testing Environment * * This is an idempotent function to delete all temporary cgroups that -* have been created during the test, including the cgroup testing work +* have been created during the test and unmount the cgroup testing work * directory. * * At call time, it moves the calling process to the root cgroup, and then
@@ -320,11 +340,10 @@ int join_parent_cgroup(const char *relative_path)*/ void cleanup_cgroup_environment (void ) { -char cgroup_workdir[PATH_MAX +1 ]; - -format_cgroup_path (cgroup_workdir,"" ); -join_cgroup_from_top (CGROUP_MOUNT_PATH ); -nftw (cgroup_workdir, nftwfunc,WALK_FD_LIMIT ,FTW_DEPTH |FTW_MOUNT ); +__cleanup_cgroup_environment (); +if (cgroup_workdir_mounted &&umount (CGROUP_MOUNT_PATH )) +log_err ("umount cgroup2" ); + cgroup_workdir_mounted = false; }/**
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH v2 3/10 selftests/bpf: Add big-endian support to the ldsx test
In reply to Ilya LeoshkevichPrepare the ldsx test to run on big-endian systems by adding the necessary endianness checks around narrow memory accesses.
--- .../selftests/bpf/progs/test_ldsx_insn.c | 6 +- .../selftests/bpf/progs/verifier_ldsx.c | 146 ++++++++++-------- 2 files changed, 90 insertions(+), 62 deletions(-)
tools/testing/selftests/bpf/progs/test_ldsx_insn.c +5 -1
@@ -104,7 +104,11 @@ int _tc(volatile struct __sk_buff *skb)"%[tmp_mark] = r1" : [tmp_mark]"=r"(tmp_mark) : [ctx]"r"(skb), - [off_mark]"i"(offsetof(struct __sk_buff, mark)) + [off_mark]"i"(offsetof(struct __sk_buff, mark) +#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__ + +sizeof (skb->mark) -1 +#endif + ) : "r1");#else tmp_mark = (char )skb->mark;
tools/testing/selftests/bpf/progs/verifier_ldsx.c +85 -61
182 lines not on this page · raw message
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH v2 4/10 s390/bpf: Implement BPF_MOV | BPF_X with sign-extension
In reply to Ilya LeoshkevichImplement the cpuv4 register-to-register move with sign extension. It is distinguished from the normal moves by non-zero values in insn->off, which determine the source size. s390x has instructions to deal with all of them: lbr, lhr, lgbr, lghr and lgfr.
--- arch/s390/net/bpf_jit_comp.c | 48 ++++++++++++++++++++++++++++++------ 1 file changed, 40 insertions(+), 8 deletions(-)
arch/s390/net/bpf_jit_comp.c +40 -8
@@ -795,15 +795,47 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,/* * BPF_MOV */ -case BPF_ALU |BPF_MOV |BPF_X :/* dst = (u32) src */ -/* llgfr %dst,%src */ -EMIT4 (0xb9160000 , dst_reg, src_reg); -if (insn_is_zext (&insn[1 ])) - insn_count =2 ; +case BPF_ALU |BPF_MOV |BPF_X : +switch (insn->off) { +case 0 :/* DST = (u32) SRC */ +/* llgfr %dst,%src */ +EMIT4 (0xb9160000 , dst_reg, src_reg); +if (insn_is_zext (&insn[1 ])) + insn_count =2 ; +break ; +case 8 :/* DST = (u32)(s8) SRC */ +/* lbr %dst,%src */ +EMIT4 (0xb9260000 , dst_reg, src_reg); +/* llgfr %dst,%dst */ +EMIT4 (0xb9160000 , dst_reg, dst_reg); +break ; +case 16 :/* DST = (u32)(s16) SRC */ +/* lhr %dst,%src */ +EMIT4 (0xb9270000 , dst_reg, src_reg); +/* llgfr %dst,%dst */ +EMIT4 (0xb9160000 , dst_reg, dst_reg); +break ; + }break ; -case BPF_ALU64 |BPF_MOV |BPF_X :/* dst = src */ -/* lgr %dst,%src */ -EMIT4 (0xb9040000 , dst_reg, src_reg); +case BPF_ALU64 |BPF_MOV |BPF_X : +switch (insn->off) { +case 0 :/* DST = SRC */ +/* lgr %dst,%src */ +EMIT4 (0xb9040000 , dst_reg, src_reg); +break ; +case 8 :/* DST = (s8) SRC */ +/* lgbr %dst,%src */ +EMIT4 (0xb9060000 , dst_reg, src_reg); +break ; +case 16 :/* DST = (s16) SRC */ +/* lghr %dst,%src */ +EMIT4 (0xb9070000 , dst_reg, src_reg); +break ; +case 32 :/* DST = (s32) SRC */ +/* lgfr %dst,%src */ +EMIT4 (0xb9140000 , dst_reg, src_reg); +break ; + }break ;case BPF_ALU |BPF_MOV |BPF_K :/* dst = (u32) imm */ /* llilf %dst,imm */
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH v2 5/10 s390/bpf: Implement BPF_MEMSX
In reply to Ilya LeoshkevichImplement the cpuv4 load with sign-extension, which is encoded as BPF_MEMSX (and, for internal uses cases only, BPF_PROBE_MEMSX).
This is the same as BPF_MEM and BPF_PROBE_MEM, but with sign extension instead of zero extension, and s390x has the necessary
instructions: lgb, lgh and lgf.
--- arch/s390/net/bpf_jit_comp.c | 32 +++++++++++++++++++++++++++----- 1 file changed, 27 insertions(+), 5 deletions(-)
arch/s390/net/bpf_jit_comp.c +27 -5
@@ -670,15 +670,18 @@ static void bpf_jit_epilogue(struct bpf_jit *jit, u32 stack_depth)static int get_probe_mem_regno (const u8 *insn) {/* -* insn must point to llgc, llgh, llgf or lg, which have destination -* register at the same position. +* insn must point to llgc, llgh, llgf, lg, lgb, lgh or lgf, which have +* destination register at the same position. */ -if (insn[0 ] !=0xe3 )/* common llgc, llgh, llgf and lg prefix */ +if (insn[0 ] !=0xe3 )/* common prefix */ return -1 ;if (insn[5 ] !=0x90 &&/* llgc */ insn[5 ] !=0x91 &&/* llgh */ insn[5 ] !=0x16 &&/* llgf */ - insn[5 ] !=0x04 )/* lg */ + insn[5 ] !=0x04 &&/* lg */ + insn[5 ] !=0x77 &&/* lgb */ + insn[5 ] !=0x15 &&/* lgh */ + insn[5 ] !=0x14 )/* lgf */ return -1 ;return insn[1 ] >>4 ; }
@@ -788,7 +791,8 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,int err;if (BPF_CLASS (insn->code) ==BPF_LDX && -BPF_MODE (insn->code) ==BPF_PROBE_MEM ) + (BPF_MODE (insn->code) ==BPF_PROBE_MEM || +BPF_MODE (insn->code) ==BPF_PROBE_MEMSX )) probe_prg = jit->prg;switch (insn->code) {
@@ -1406,6 +1410,12 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,if (insn_is_zext (&insn[1 ])) insn_count =2 ;break ; +case BPF_LDX |BPF_MEMSX |BPF_B :/* dst = *(s8 *)(ul) (src + off) */ +case BPF_LDX |BPF_PROBE_MEMSX |BPF_B : +/* lgb %dst,0(off,%src) */ +EMIT6_DISP_LH (0xe3000000 ,0x0077 , dst_reg, src_reg,REG_0 , off); + jit->seen |=SEEN_MEM ; +break ;case BPF_LDX |BPF_MEM |BPF_H :/* dst = *(u16 *)(ul) (src + off) */ case BPF_LDX |BPF_PROBE_MEM |BPF_H :/* llgh %dst,0(off,%src) */
@@ -1414,6 +1424,12 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,if (insn_is_zext (&insn[1 ])) insn_count =2 ;break ; +case BPF_LDX |BPF_MEMSX |BPF_H :/* dst = *(s16 *)(ul) (src + off) */ +case BPF_LDX |BPF_PROBE_MEMSX |BPF_H : +/* lgh %dst,0(off,%src) */ +EMIT6_DISP_LH (0xe3000000 ,0x0015 , dst_reg, src_reg,REG_0 , off); + jit->seen |=SEEN_MEM ; +break ;case BPF_LDX |BPF_MEM |BPF_W :/* dst = *(u32 *)(ul) (src + off) */ case BPF_LDX |BPF_PROBE_MEM |BPF_W :/* llgf %dst,off(%src) */
@@ -1422,6 +1438,12 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,if (insn_is_zext (&insn[1 ])) insn_count =2 ;break ; +case BPF_LDX |BPF_MEMSX |BPF_W :/* dst = *(s32 *)(ul) (src + off) */ +case BPF_LDX |BPF_PROBE_MEMSX |BPF_W : +/* lgf %dst,off(%src) */ + jit->seen |=SEEN_MEM ; +EMIT6_DISP_LH (0xe3000000 ,0x0014 , dst_reg, src_reg,REG_0 , off); +break ;case BPF_LDX |BPF_MEM |BPF_DW :/* dst = *(u64 *)(ul) (src + off) */ case BPF_LDX |BPF_PROBE_MEM |BPF_DW :/* lg %dst,0(off,%src) */
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH v2 6/10 s390/bpf: Implement unconditional byte swap
In reply to Ilya LeoshkevichImplement the cpuv4 unconditional byte swap, which is encoded as BPF_ALU64 | BPF_END | BPF_FROM_LE. Since s390x is big-endian, it's the same as the existing BPF_ALU | BPF_END | BPF_FROM_LE.
--- arch/s390/net/bpf_jit_comp.c | 1 + 1 file changed, 1 insertion(+)
arch/s390/net/bpf_jit_comp.c +1 -0
@@ -1253,6 +1253,7 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, }break ;case BPF_ALU |BPF_END |BPF_FROM_LE : +case BPF_ALU64 |BPF_END |BPF_FROM_LE :switch (imm) {case 16 :/* dst = (u16) cpu_to_le16(dst) */ /* lrvr %dst,%dst */
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH v2 7/10 s390/bpf: Implement unconditional jump with 32-bit offset
In reply to Ilya LeoshkevichImplement the cpuv4 unconditional jump with 32-bit offset, which is encoded as BPF_JMP32 | BPF_JA and stores the offset in the imm field. Reuse the existing BPF_JMP | BPF_JA logic.
--- arch/s390/net/bpf_jit_comp.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
arch/s390/net/bpf_jit_comp.c +9 -3
@@ -779,6 +779,7 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, int i, bool extra_pass, u32 stack_depth) { struct bpf_insn *insn = &fp->insnsi[i]; +s16 branch_oc_off = insn->off;u32 dst_reg = insn->dst_reg;u32 src_reg = insn->src_reg;int last, insn_count =1 ;
@@ -1625,6 +1626,9 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,* instruction itself (loop) and for BPF with offset 0 we * branch to the instruction behind the branch. */ +case BPF_JMP32 |BPF_JA :/* if (true) */ + branch_oc_off = imm; + fallthrough;case BPF_JMP |BPF_JA :/* if (true) */ mask =0xf000 ;/* j */ goto branch_oc;
@@ -1793,14 +1797,16 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,break ; branch_oc:if (!is_first_pass (jit) && -can_use_rel (jit, addrs[i + off +1 ])) { +can_use_rel (jit, addrs[i + branch_oc_off +1 ])) {/* brc mask,off */ EMIT4_PCREL_RIC (0xa7040000 , - mask >>12 , addrs[i + off +1 ]); + mask >>12 , + addrs[i + branch_oc_off +1 ]); }else {/* brcl mask,off */ EMIT6_PCREL_RILC (0xc0040000 , - mask >>12 , addrs[i + off +1 ]); + mask >>12 , + addrs[i + branch_oc_off +1 ]); }break ; }
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH v2 8/10 s390/bpf: Implement signed division
In reply to Ilya LeoshkevichImplement the cpuv4 signed division. It is encoded as unsigned division, but with off field set to 1. s390x has the necessary
instructions: dsgfr, dsgf and dsgr.
--- arch/s390/net/bpf_jit_comp.c | 172 +++++++++++++++++++++++++---------- 1 file changed, 125 insertions(+), 47 deletions(-)
arch/s390/net/bpf_jit_comp.c +125 -47
@@ -949,66 +949,115 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,/* * BPF_DIV / BPF_MOD */ -case BPF_ALU |BPF_DIV |BPF_X :/* dst = (u32) dst / (u32) src */ -case BPF_ALU |BPF_MOD |BPF_X :/* dst = (u32) dst % (u32) src */ +case BPF_ALU |BPF_DIV |BPF_X : +case BPF_ALU |BPF_MOD |BPF_X : {int rc_reg =BPF_OP (insn->code) ==BPF_DIV ?REG_W1 :REG_W0 ; -/* lhi %w0,0 */ -EMIT4_IMM (0xa7080000 ,REG_W0 ,0 ); -/* lr %w1,%dst */ -EMIT2 (0x1800 ,REG_W1 , dst_reg); -/* dlr %w0,%src */ -EMIT4 (0xb9970000 ,REG_W0 , src_reg); +switch (off) { +case 0 :/* dst = (u32) dst {/,%} (u32) src */ +/* xr %w0,%w0 */ +EMIT2 (0x1700 ,REG_W0 ,REG_W0 ); +/* lr %w1,%dst */ +EMIT2 (0x1800 ,REG_W1 , dst_reg); +/* dlr %w0,%src */ +EMIT4 (0xb9970000 ,REG_W0 , src_reg); +break ; +case 1 :/* dst = (u32) ((s32) dst {/,%} (s32) src) */ +/* lgfr %r1,%dst */ +EMIT4 (0xb9140000 ,REG_W1 , dst_reg); +/* dsgfr %r0,%src */ +EMIT4 (0xb91d0000 ,REG_W0 , src_reg); +break ; + }/* llgfr %dst,%rc */ EMIT4 (0xb9160000 , dst_reg, rc_reg);if (insn_is_zext (&insn[1 ])) insn_count =2 ;break ; } -case BPF_ALU64 |BPF_DIV |BPF_X :/* dst = dst / src */ -case BPF_ALU64 |BPF_MOD |BPF_X :/* dst = dst % src */ +case BPF_ALU64 |BPF_DIV |BPF_X : +case BPF_ALU64 |BPF_MOD |BPF_X : {int rc_reg =BPF_OP (insn->code) ==BPF_DIV ?REG_W1 :REG_W0 ; -/* lghi %w0,0 */ -EMIT4_IMM (0xa7090000 ,REG_W0 ,0 ); -/* lgr %w1,%dst */ -EMIT4 (0xb9040000 ,REG_W1 , dst_reg); -/* dlgr %w0,%dst */ -EMIT4 (0xb9870000 ,REG_W0 , src_reg); +switch (off) { +case 0 :/* dst = dst {/,%} src */ +/* lghi %w0,0 */ +EMIT4_IMM (0xa7090000 ,REG_W0 ,0 ); +/* lgr %w1,%dst */ +EMIT4 (0xb9040000 ,REG_W1 , dst_reg); +/* dlgr %w0,%src */ +EMIT4 (0xb9870000 ,REG_W0 , src_reg); +break ; +case 1 :/* dst = (s64) dst {/,%} (s64) src */ +/* lgr %w1,%dst */ +EMIT4 (0xb9040000 ,REG_W1 , dst_reg); +/* dsgr %w0,%src */ +EMIT4 (0xb90d0000 ,REG_W0 , src_reg); +break ; + }/* lgr %dst,%rc */ EMIT4 (0xb9040000 , dst_reg, rc_reg);break ; } -case BPF_ALU |BPF_DIV |BPF_K :/* dst = (u32) dst / (u32) imm */ -case BPF_ALU |BPF_MOD |BPF_K :/* dst = (u32) dst % (u32) imm */ +case BPF_ALU |BPF_DIV |BPF_K : +case BPF_ALU |BPF_MOD |BPF_K : {int rc_reg =BPF_OP (insn->code) ==BPF_DIV ?REG_W1 :REG_W0 ;if (imm ==1 ) {if (BPF_OP (insn->code) ==BPF_MOD ) -/* lhgi %dst,0 */ +/* lghi %dst,0 */ EMIT4_IMM (0xa7090000 , dst_reg,0 );else EMIT_ZERO (dst_reg);break ; } -/* lhi %w0,0 */ -EMIT4_IMM (0xa7080000 ,REG_W0 ,0 ); -/* lr %w1,%dst */ -EMIT2 (0x1800 ,REG_W1 , dst_reg);if (!is_first_pass (jit) &&can_use_ldisp_for_lit32 (jit)) { -/* dl %w0,<d(imm)>(%l) */ -EMIT6_DISP_LH (0xe3000000 ,0x0097 ,REG_W0 ,REG_0 ,REG_L , -EMIT_CONST_U32 (imm)); +switch (off) { +case 0 :/* dst = (u32) dst {/,%} (u32) imm */ +/* xr %w0,%w0 */ +EMIT2 (0x1700 ,REG_W0 ,REG_W0 ); +/* lr %w1,%dst */ +EMIT2 (0x1800 ,REG_W1 , dst_reg); +/* dl %w0,<d(imm)>(%l) */ +EMIT6_DISP_LH (0xe3000000 ,0x0097 ,REG_W0 ,REG_0 , +REG_L ,EMIT_CONST_U32 (imm)); +break ; +case 1 :/* dst = (s32) dst {/,%} (s32) imm */ +/* lgfr %r1,%dst */ +EMIT4 (0xb9140000 ,REG_W1 , dst_reg); +/* dsgf %r0,<d(imm)>(%l) */ +EMIT6_DISP_LH (0xe3000000 ,0x001d ,REG_W0 ,REG_0 , +REG_L ,EMIT_CONST_U32 (imm)); +break ; + } }else { -/* lgfrl %dst,imm */ -EMIT6_PCREL_RILB (0xc40c0000 , dst_reg, -_EMIT_CONST_U32 (imm)); - jit->seen |=SEEN_LITERAL ; -/* dlr %w0,%dst */ -EMIT4 (0xb9970000 ,REG_W0 , dst_reg); +switch (off) { +case 0 :/* dst = (u32) dst {/,%} (u32) imm */ +/* xr %w0,%w0 */ +EMIT2 (0x1700 ,REG_W0 ,REG_W0 ); +/* lr %w1,%dst */ +EMIT2 (0x1800 ,REG_W1 , dst_reg); +/* lrl %dst,imm */ +EMIT6_PCREL_RILB (0xc40d0000 , dst_reg, +_EMIT_CONST_U32 (imm)); + jit->seen |=SEEN_LITERAL ; +/* dlr %w0,%dst */ +EMIT4 (0xb9970000 ,REG_W0 , dst_reg); +break ; +case 1 :/* dst = (s32) dst {/,%} (s32) imm */ +/* lgfr %w1,%dst */ +EMIT4 (0xb9140000 ,REG_W1 , dst_reg); +/* lgfrl %dst,imm */ +EMIT6_PCREL_RILB (0xc40c0000 , dst_reg, +_EMIT_CONST_U32 (imm)); + jit->seen |=SEEN_LITERAL ; +/* dsgr %w0,%dst */ +EMIT4 (0xb90d0000 ,REG_W0 , dst_reg); +break ; + } }/* llgfr %dst,%rc */ EMIT4 (0xb9160000 , dst_reg, rc_reg);
@@ -1016,8 +1065,8 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, insn_count =2 ;break ; } -case BPF_ALU64 |BPF_DIV |BPF_K :/* dst = dst / imm */ -case BPF_ALU64 |BPF_MOD |BPF_K :/* dst = dst % imm */ +case BPF_ALU64 |BPF_DIV |BPF_K : +case BPF_ALU64 |BPF_MOD |BPF_K : {int rc_reg =BPF_OP (insn->code) ==BPF_DIV ?REG_W1 :REG_W0 ;
@@ -1027,21 +1076,50 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,EMIT4_IMM (0xa7090000 , dst_reg,0 );break ; } -/* lghi %w0,0 */ -EMIT4_IMM (0xa7090000 ,REG_W0 ,0 ); -/* lgr %w1,%dst */ -EMIT4 (0xb9040000 ,REG_W1 , dst_reg);if (!is_first_pass (jit) &&can_use_ldisp_for_lit64 (jit)) { -/* dlg %w0,<d(imm)>(%l) */ -EMIT6_DISP_LH (0xe3000000 ,0x0087 ,REG_W0 ,REG_0 ,REG_L , -EMIT_CONST_U64 (imm)); +switch (off) { +case 0 :/* dst = dst {/,%} imm */ +/* lghi %w0,0 */ +EMIT4_IMM (0xa7090000 ,REG_W0 ,0 ); +/* lgr %w1,%dst */ +EMIT4 (0xb9040000 ,REG_W1 , dst_reg); +/* dlg %w0,<d(imm)>(%l) */ +EMIT6_DISP_LH (0xe3000000 ,0x0087 ,REG_W0 ,REG_0 , +REG_L ,EMIT_CONST_U64 (imm)); +break ; +case 1 :/* dst = (s64) dst {/,%} (s64) imm */ +/* lgr %w1,%dst */ +EMIT4 (0xb9040000 ,REG_W1 , dst_reg); +/* dsg %w0,<d(imm)>(%l) */ +EMIT6_DISP_LH (0xe3000000 ,0x000d ,REG_W0 ,REG_0 , +REG_L ,EMIT_CONST_U64 (imm)); +break ; + } }else { -/* lgrl %dst,imm */ -EMIT6_PCREL_RILB (0xc4080000 , dst_reg, -_EMIT_CONST_U64 (imm)); - jit->seen |=SEEN_LITERAL ; -/* dlgr %w0,%dst */ -EMIT4 (0xb9870000 ,REG_W0 , dst_reg); +switch (off) { +case 0 :/* dst = dst {/,%} imm */ +/* lghi %w0,0 */ +EMIT4_IMM (0xa7090000 ,REG_W0 ,0 ); +/* lgr %w1,%dst */ +EMIT4 (0xb9040000 ,REG_W1 , dst_reg); +/* lgrl %dst,imm */ +EMIT6_PCREL_RILB (0xc4080000 , dst_reg, +_EMIT_CONST_U64 (imm)); + jit->seen |=SEEN_LITERAL ; +/* dlgr %w0,%dst */ +EMIT4 (0xb9870000 ,REG_W0 , dst_reg); +break ; +case 1 :/* dst = (s64) dst {/,%} (s64) imm */ +/* lgr %w1,%dst */ +EMIT4 (0xb9040000 ,REG_W1 , dst_reg); +/* lgrl %dst,imm */ +EMIT6_PCREL_RILB (0xc4080000 , dst_reg, +_EMIT_CONST_U64 (imm)); + jit->seen |=SEEN_LITERAL ; +/* dsgr %w0,%dst */ +EMIT4 (0xb90d0000 ,REG_W0 , dst_reg); +break ; + } }/* lgr %dst,%rc */ EMIT4 (0xb9040000 , dst_reg, rc_reg);
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH v2 9/10 selftests/bpf: Enable the cpuv4 tests for s390x
In reply to Ilya LeoshkevichNow that all the cpuv4 support is in place, enable the tests.
--- tools/testing/selftests/bpf/progs/test_ldsx_insn.c | 3 ++- tools/testing/selftests/bpf/progs/verifier_bswap.c | 3 ++- tools/testing/selftests/bpf/progs/verifier_gotol.c | 3 ++- tools/testing/selftests/bpf/progs/verifier_ldsx.c | 3 ++- tools/testing/selftests/bpf/progs/verifier_movsx.c | 3 ++- tools/testing/selftests/bpf/progs/verifier_sdiv.c | 3 ++- 6 files changed, 12 insertions(+), 6 deletions(-)
tools/testing/selftests/bpf/progs/test_ldsx_insn.c +2 -1
@@ -6,7 +6,8 @@#include <bpf/bpf_tracing.h> #if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \ - (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 )) && __clang_major__ >=18 + (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 ) || \ + defined(__TARGET_ARCH_s390)) && __clang_major__ >=18 const volatile int skip =0 ;#else const volatile int skip =1 ;
tools/testing/selftests/bpf/progs/verifier_bswap.c +2 -1
@@ -5,7 +5,8 @@#include "bpf_misc.h" #if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \ - (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 ) || defined(__TARGET_ARCH_arm)) && \ + (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 ) || \ + defined(__TARGET_ARCH_arm) || defined(__TARGET_ARCH_s390)) && \ __clang_major__ >=18 SEC ("socket" )
tools/testing/selftests/bpf/progs/verifier_gotol.c +2 -1
@@ -5,7 +5,8 @@#include "bpf_misc.h" #if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \ - (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 ) || defined(__TARGET_ARCH_arm)) && \ + (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 ) || \ + defined(__TARGET_ARCH_arm) || defined(__TARGET_ARCH_s390)) && \ __clang_major__ >=18 SEC ("socket" )
tools/testing/selftests/bpf/progs/verifier_ldsx.c +2 -1
@@ -5,7 +5,8 @@#include "bpf_misc.h" #if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \ - (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 ) || defined(__TARGET_ARCH_arm)) && \ + (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 ) || \ + defined(__TARGET_ARCH_arm) || defined(__TARGET_ARCH_s390)) && \ __clang_major__ >=18 SEC ("socket" )
tools/testing/selftests/bpf/progs/verifier_movsx.c +2 -1
@@ -5,7 +5,8 @@#include "bpf_misc.h" #if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \ - (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 ) || defined(__TARGET_ARCH_arm)) && \ + (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 ) || \ + defined(__TARGET_ARCH_arm) || defined(__TARGET_ARCH_s390)) && \ __clang_major__ >=18 SEC ("socket" )
tools/testing/selftests/bpf/progs/verifier_sdiv.c +2 -1
@@ -5,7 +5,8 @@#include "bpf_misc.h" #if (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \ - (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 ) || defined(__TARGET_ARCH_arm)) && \ + (defined(__TARGET_ARCH_riscv) && __riscv_xlen ==64 ) || \ + defined(__TARGET_ARCH_arm) || defined(__TARGET_ARCH_s390)) && \ __clang_major__ >=18 SEC ("socket" )
Signature · 1 line
2.41.0
Ilya Leoshkevich
maintainerPATCH v2 10/10 selftests/bpf: Trim DENYLIST.s390x
In reply to Ilya LeoshkevichEnable all selftests, except the 2 that have to do with the userspace unwinding, and the new exceptions test, in the s390x CI.
--- tools/testing/selftests/bpf/DENYLIST.s390x | 25 ---------------------- 1 file changed, 25 deletions(-)
tools/testing/selftests/bpf/DENYLIST.s390x +0 -25
@@ -1,30 +1,5 @@ # TEMPORARY # Alphabetical order -bloom_filter_map # failed to find kernel BTF type ID of '__x64_sys_getpgid': -3 (?) -bpf_cookie # failed to open_and_load program: -524 (trampoline) -bpf_loop # attaches to __x64_sys_nanosleep -cgrp_local_storage # prog_attach unexpected error: -524 (trampoline) -dynptr/test_dynptr_skb_data -dynptr/test_skb_readonly exceptions # JIT does not support calling kfunc bpf_throw (exceptions) -fexit_sleep # fexit_skel_load fexit skeleton failed (trampoline) get_stack_raw_tp # user_stack corrupted user stack (no backchain userspace) -iters/testmod_seq* # s390x doesn't support kfuncs in modules yet -kprobe_multi_bench_attach # bpf_program__attach_kprobe_multi_opts unexpected error: -95 -kprobe_multi_test # relies on fentry -ksyms_btf/weak_ksyms* # test_ksyms_weak__open_and_load unexpected error: -22 (kfunc) -ksyms_module # test_ksyms_module__open_and_load unexpected error: -9 (?) -ksyms_module_libbpf # JIT does not support calling kernel function (kfunc) -ksyms_module_lskel # test_ksyms_module_lskel__open_and_load unexpected error: -9 (?) -module_attach # skel_attach skeleton attach failed: -524 (trampoline) -ringbuf # skel_load skeleton load failed (?) stacktrace_build_id # compare_map_keys stackid_hmap vs. stackmap err -2 errno 2 (?) -test_lsm # attach unexpected error: -524 (trampoline) -trace_printk # trace_printk__load unexpected error: -2 (errno 2) (?) -trace_vprintk # trace_vprintk__open_and_load unexpected error: -9 (?) -unpriv_bpf_disabled # fentry -user_ringbuf # failed to find kernel BTF type ID of '__s390x_sys_prctl': -3 (?) -verif_stats # trace_vprintk__open_and_load unexpected error: -9 (?) -xdp_bonding # failed to auto-attach program 'trace_on_entry': -524 (trampoline) -xdp_metadata # JIT does not support calling kernel function (kfunc) -test_task_under_cgroup # JIT does not support calling kernel function (kfunc)
Signature · 1 line
2.41.0
Re: [PATCH bpf-next v2 01/10] bpf: Disable zero-extension for BPF_MEMSX
In reply to Ilya LeoshkevichQuoted 33 lines · Ilya Leoshkevich
On the architectures that use
bpf_jit_needs_zext(), e.g., s390x, the verifier incorrectly inserts a zero-extension afterBPF_MEMSX, leading to miscompilations like the one below:24: 89 1a ff fe 00 00 00 00 "r1 = *(s16 *)(r10 - 2);" # zext_dst set 0x3ff7fdb910e: lgh %r2,-2(%r13,%r0) # load halfword 0x3ff7fdb9114: llgfr %r2,%r2 # wrong! 25: 65 10 00 03 00 00 7f ff if r1 s> 32767 goto +3 <l0_1> # check_cond_jmp_op()Disable such zero-extensions. The JITs need to insert sign-extension themselves, if necessary.
Suggested-by Puranjay Mohan <[email protected]>Signed-off-by Ilya Leoshkevich <[email protected]>--- kernel/bpf/verifier.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/bpf/verifier.c +1
-1@@ -3114,7 +3114,7 @@ static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn,if (class ==BPF_LDX ) {if (t !=SRC_OP ) -return BPF_SIZE (code) ==BPF_DW ; +return BPF_SIZE (code) ==BPF_DW ||BPF_MODE (code) ==BPF_MEMSX ;/* LDX source must be ptr. */ return true; }Signature · 1 line
2.41.0
Thanks for this.
Puranjay
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 8 lines · On Tue, 19 Sep 2023 12:09:02 +0200 you wrote:
v1: https://lore.kernel.org/bpf/[email protected]/
v1 -> v2:
- Redo Disable zero-extension forBPF_MEMSXas Puranjay and Alexei suggested.
- Drop thebpf_ct_insert_entry()patch, it went in via the bpf tree.
- Rebase, don't apply A-bs because there were fixed conflicts.[...]
Here is the summary with links:
- [bpf-next,v2,01/10] bpf: Disable zero-extension for BPF_MEMSX
https://git.kernel.org/bpf/bpf-next/c/577c06af8188 - [bpf-next,v2,02/10] selftests/bpf: Unmount the cgroup2 work directory https://git.kernel.org/bpf/bpf-next/c/6cb66eca36f3 - [bpf-next,v2,03/10] selftests/bpf: Add big-endian support to the ldsx test https://git.kernel.org/bpf/bpf-next/c/9873ce2e9c68 - [bpf-next,v2,04/10] s390/bpf: Implement BPF_MOV | BPF_X with sign-extension https://git.kernel.org/bpf/bpf-next/c/3de55893f648 - [bpf-next,v2,05/10] s390/bpf: Implement BPF_MEMSX https://git.kernel.org/bpf/bpf-next/c/738476a079bd - [bpf-next,v2,06/10] s390/bpf: Implement unconditional byte swap https://git.kernel.org/bpf/bpf-next/c/90f426d35e01 - [bpf-next,v2,07/10] s390/bpf: Implement unconditional jump with 32-bit offset https://git.kernel.org/bpf/bpf-next/c/c690191e23d8 - [bpf-next,v2,08/10] s390/bpf: Implement signed division https://git.kernel.org/bpf/bpf-next/c/91d2ad78e90c - [bpf-next,v2,09/10] selftests/bpf: Enable the cpuv4 tests for s390x https://git.kernel.org/bpf/bpf-next/c/48c432382dd4 - [bpf-next,v2,10/10] selftests/bpf: Trim DENYLIST.s390x https://git.kernel.org/bpf/bpf-next/c/c29913bbf4ec
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html