bpf: arm64: Indirect jumps
Subsystems (6)
Puranjay Mohan
maintainercover bpf: arm64: Indirect jumps
This set adds the support of indirect jumps to the arm64 JIT. It involves calling bpf_prog_update_insn_ptrs() to support instructions array map. The second piece is supporting BPF_JMP|BPF_X|BPF_JA, SRC=0, DST=Rx, off=0, imm=0 instruction that is trivial to implement on arm64.
When running the selftests after doing the above changes, I found that on arm64 builds of llvm, a relocation section was being generated for .jumptables sections and it was making libbpf fail like:
libbpf: relocation against STT_SECTION in non-exec section is not supported!
Error: failed to link 'tools/testing/selftests/bpf/cpuv4/bpf_gotox.bpf.o': Invalid argument (22)
Which is due to:
Relocation section '.rel.jumptables' at offset 0x5b50 contains 263 entries:
Offset Info Type Symbol's Value Symbol's Name
0000000000000000 0000000300000002 R_BPF_64_ABS64 0000000000000000 syscall 0000000000000008 0000000300000002 R_BPF_64_ABS64 0000000000000000 syscall 0000000000000010 0000000300000002 R_BPF_64_ABS64 0000000000000000 syscall
This rel section is not generated by x86 builds of LLVM. The third patch of this set makes libbpf ignore relocation sections for .jumptables.
The final patch enables selftests on arm64:
[root@localhost bpf]# ./test_progs-cpuv4 -a "*gotox*"
#20/1 bpf_gotox/one-switch:OK #20/2 bpf_gotox/one-switch-non-zero-sec-offset:OK #20/3 bpf_gotox/two-switches:OK #20/4 bpf_gotox/big-jump-table:OK #20/5 bpf_gotox/static-global:OK #20/6 bpf_gotox/nonstatic-global:OK #20/7 bpf_gotox/other-sec:OK #20/8 bpf_gotox/static-global-other-sec:OK #20/9 bpf_gotox/nonstatic-global-other-sec:OK #20/10 bpf_gotox/one-jump-two-maps:OK #20/11 bpf_gotox/one-map-two-jumps:OK #20 bpf_gotox:OK #537/1 verifier_gotox/jump_table_ok:OK #537/2 verifier_gotox/jump_table_reserved_field_src_reg:OK #537/3 verifier_gotox/jump_table_reserved_field_non_zero_off:OK #537/4 verifier_gotox/jump_table_reserved_field_non_zero_imm:OK #537/5 verifier_gotox/jump_table_no_jump_table:OK #537/6 verifier_gotox/jump_table_incorrect_dst_reg_type:OK #537/7 verifier_gotox/jump_table_invalid_read_size_u32:OK #537/8 verifier_gotox/jump_table_invalid_read_size_u16:OK #537/9 verifier_gotox/jump_table_invalid_read_size_u8:OK #537/10 verifier_gotox/jump_table_misaligned_access:OK #537/11 verifier_gotox/jump_table_invalid_mem_acceess_pos:OK #537/12 verifier_gotox/jump_table_invalid_mem_acceess_neg:OK #537/13 verifier_gotox/jump_table_add_sub_ok:OK #537/14 verifier_gotox/jump_table_no_writes:OK #537/15 verifier_gotox/jump_table_use_reg_r0:OK #537/16 verifier_gotox/jump_table_use_reg_r1:OK #537/17 verifier_gotox/jump_table_use_reg_r2:OK #537/18 verifier_gotox/jump_table_use_reg_r3:OK #537/19 verifier_gotox/jump_table_use_reg_r4:OK #537/20 verifier_gotox/jump_table_use_reg_r5:OK #537/21 verifier_gotox/jump_table_use_reg_r6:OK #537/22 verifier_gotox/jump_table_use_reg_r7:OK #537/23 verifier_gotox/jump_table_use_reg_r8:OK #537/24 verifier_gotox/jump_table_use_reg_r9:OK #537/25 verifier_gotox/jump_table_outside_subprog:OK #537/26 verifier_gotox/jump_table_contains_non_unique_values:OK #537 verifier_gotox:OK
Summary: 2/37 PASSED, 0 SKIPPED, 0 FAILED
Puranjay Mohan (4):
bpf: arm64: Add support for instructions array
bpf: arm64: Add support for indirect jumps
libbpf: Ignore relocations for .jumptables sections
selftests: bpf: Enable gotox tests from arm64
arch/arm64/net/bpf_jit_comp.c | 11 +++++++++++ tools/lib/bpf/linker.c | 4 ++++ tools/testing/selftests/bpf/progs/verifier_gotox.c | 4 ++-- 3 files changed, 17 insertions(+), 2 deletions(-)
Signature · 1 line
2.47.3
Puranjay Mohan
maintainerPATCH 1/4 bpf: arm64: Add support for instructions array
Add support for the instructions array map type in the arm64 JIT by calling bpf_prog_update_insn_ptrs() with the offsets that map xlated_offset to the jited_offset in the final image. arm64 JIT already has this offset array which was being used for bpf_prog_fill_jited_linfo() and can be used directly for bpf_prog_update_insn_ptrs.
--- arch/arm64/net/bpf_jit_comp.c | 7 +++++++ 1 file changed, 7 insertions(+)
arch/arm64/net/bpf_jit_comp.c +7 -0
@@ -2231,6 +2231,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)for (i =0 ; i <= prog->len; i++) ctx.offset[i] *=AARCH64_INSN_SIZE ;bpf_prog_fill_jited_linfo (prog, ctx.offset +1 ); +/* +* The bpf_prog_update_insn_ptrs function expects offsets to +* point to the first byte of the jitted instruction (unlike +* the bpf_prog_fill_jited_linfo above, which, for historical +* reasons, expects to point to the next instruction) +*/ +bpf_prog_update_insn_ptrs (prog, ctx.offset, ctx.ro_image); out_off:if (!ro_header && priv_stack_ptr) {free_percpu (priv_stack_ptr);
Signature · 1 line
2.47.3
Puranjay Mohan
maintainerPATCH 2/4 bpf: arm64: Add support for indirect jumps
In reply to Puranjay MohanAdd support for a new instruction
BPF_JMP|BPF_X|BPF_JA, SRC=0, DST=Rx, off=0, imm=0
which does an indirect jump to a location stored in Rx. The register Rx should have type PTR_TO_INSN. This new type assures that the Rx register contains a value (or a range of values) loaded from a correct jump table – map of type instruction array.
ARM64 JIT supports indirect jumps to all registers through the A64_BR() macro, use it to implement this new instruction.
--- arch/arm64/net/bpf_jit_comp.c | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm64/net/bpf_jit_comp.c +4 -0
@@ -1452,6 +1452,10 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx,emit (A64_ASR (is64, dst, dst, imm), ctx);break ; +/* JUMP reg */ +case BPF_JMP |BPF_JA |BPF_X : +emit (A64_BR (dst), ctx); +break ;/* JUMP off */ case BPF_JMP |BPF_JA :case BPF_JMP32 |BPF_JA :
Signature · 1 line
2.47.3
Puranjay Mohan
maintainerPATCH 3/4 libbpf: Ignore relocations for .jumptables sections
In reply to Puranjay Mohan.jumptables sections generated by LLVM store byte offsets from the start of the section to the jump's target instruction. On arm64 builds of LLVM, it generates a rel section to add the section address to each entry in the jump table, therefore storing the absolute address of the jump target into the table. For BPF usage of .jumptables, we only care about the instruction offsets and calculate the final address much later after the JIT has finished, and called bpf_prog_update_insn_ptrs()
Without the change in this commit, compilation fails with:
libbpf: relocation against STT_SECTION in non-exec section is not supported! Error: failed to link 'tools/testing/selftests/bpf/cpuv4/bpf_gotox.bpf.o': Invalid argument (22)
This is due to the presence of a relocation section for .jumptables (non-exec) against syscall (STT_SECTION):
Relocation section '.rel.jumptables' at offset 0x5b50 contains 263 entries:
Offset Info Type Symbol's Value Symbol's Name
000000000000 0000000300000002 R_BPF_64_ABS64 0000000000000000 syscall 000000000008 0000000300000002 R_BPF_64_ABS64 0000000000000000 syscall
[...]
--- tools/lib/bpf/linker.c | 4 ++++ 1 file changed, 4 insertions(+)
tools/lib/bpf/linker.c +4 -0
@@ -634,6 +634,10 @@ static bool is_ignored_sec(struct src_sec *sec)if (strcmp (name,BTF_ELF_SEC ) ==0 ||strcmp (name,BTF_EXT_ELF_SEC ) ==0 )return true; + +/* .jumptables don't need relocations */ +if (strcmp (name,JUMPTABLES_SEC ) ==0 ) +return true; }return false;
Signature · 1 line
2.47.3
Puranjay Mohan
maintainerPATCH 4/4 selftests: bpf: Enable gotox tests from arm64
In reply to Puranjay Mohanarm64 JIT now supports gotox instruction and jumptables, so run tests in verifier_gotox.c for arm64.
--- tools/testing/selftests/bpf/progs/verifier_gotox.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/progs/verifier_gotox.c +2 -2
@@ -6,7 +6,7 @@#include "bpf_misc.h" #include "../../../include/linux/filter.h" -#ifdef __TARGET_ARCH_x86 +#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)#define DEFINE_SIMPLE_JUMP_TABLE_PROG (NAME ,SRC_REG ,OFF ,IMM ,OUTCOME ) \ \
@@ -384,6 +384,6 @@ jt0_%=: \ : __clobber_all); } -#endif /* __TARGET_ARCH_x86 */ +#endif /* __TARGET_ARCH_x86 || __TARGET_ARCH_arm64 */ char _license[] SEC("license") = "GPL";
Signature · 1 line
2.47.3
Quoted 14 lines · Puranjay Mohan
This set adds the support of indirect jumps to the arm64 JIT. It involves calling
bpf_prog_update_insn_ptrs()to support instructions array map. The second piece is supporting BPF_JMP|BPF_X|BPF_JA, SRC=0, DST=Rx, off=0, imm=0 instruction that is trivial to implement on arm64.When running the selftests after doing the above changes, I found that on arm64 builds of llvm, a relocation section was being generated for .jumptables sections and it was making libbpf fail like:
libbpf: relocation against
STT_SECTIONin non-exec section is not supported!
Error: failed to link 'tools/testing/selftests/bpf/cpuv4/bpf_gotox.bpf.o': Invalid argument (22)Which is due to:
Quoted 8 lines
Relocation section '.rel.jumptables' at offset
0x5b50contains 263 entries:
Offset Info Type Symbol's Value Symbol's Name0000000000000000 0000000300000002 R_BPF_64_ABS64 0000000000000000 syscall 0000000000000008 0000000300000002 R_BPF_64_ABS64 0000000000000000 syscall 0000000000000010 0000000300000002 R_BPF_64_ABS64 0000000000000000 syscallThis rel section is not generated by x86 builds of LLVM. The third patch of this set makes libbpf ignore relocation sections for .jumptables.
I added Yonghong to this thread. He had fixed this problem in
https://github.com/llvm/llvm-project/pull/166301 changes doesn't seem to be x86-specific...
Quoted 43 lines
The final patch enables selftests on arm64:
[root@localhost bpf]# ./test_progs-cpuv4 -a "*gotox*"
#20/1 bpf_gotox/one-switch:OK #20/2 bpf_gotox/one-switch-non-zero-sec-offset:OK #20/3 bpf_gotox/two-switches:OK #20/4 bpf_gotox/big-jump-table:OK #20/5 bpf_gotox/static-global:OK #20/6 bpf_gotox/nonstatic-global:OK #20/7 bpf_gotox/other-sec:OK #20/8 bpf_gotox/static-global-other-sec:OK #20/9 bpf_gotox/nonstatic-global-other-sec:OK #20/10 bpf_gotox/one-jump-two-maps:OK #20/11 bpf_gotox/one-map-two-jumps:OK #20 bpf_gotox:OK #537/1 verifier_gotox/jump_table_ok:OK #537/2 verifier_gotox/jump_table_reserved_field_src_reg:OK #537/3 verifier_gotox/jump_table_reserved_field_non_zero_off:OK #537/4 verifier_gotox/jump_table_reserved_field_non_zero_imm:OK #537/5 verifier_gotox/jump_table_no_jump_table:OK #537/6 verifier_gotox/jump_table_incorrect_dst_reg_type:OK #537/7 verifier_gotox/jump_table_invalid_read_size_u32:OK #537/8 verifier_gotox/jump_table_invalid_read_size_u16:OK #537/9 verifier_gotox/jump_table_invalid_read_size_u8:OK #537/10 verifier_gotox/jump_table_misaligned_access:OK #537/11 verifier_gotox/jump_table_invalid_mem_acceess_pos:OK #537/12 verifier_gotox/jump_table_invalid_mem_acceess_neg:OK #537/13 verifier_gotox/jump_table_add_sub_ok:OK #537/14 verifier_gotox/jump_table_no_writes:OK #537/15 verifier_gotox/jump_table_use_reg_r0:OK #537/16 verifier_gotox/jump_table_use_reg_r1:OK #537/17 verifier_gotox/jump_table_use_reg_r2:OK #537/18 verifier_gotox/jump_table_use_reg_r3:OK #537/19 verifier_gotox/jump_table_use_reg_r4:OK #537/20 verifier_gotox/jump_table_use_reg_r5:OK #537/21 verifier_gotox/jump_table_use_reg_r6:OK #537/22 verifier_gotox/jump_table_use_reg_r7:OK #537/23 verifier_gotox/jump_table_use_reg_r8:OK #537/24 verifier_gotox/jump_table_use_reg_r9:OK #537/25 verifier_gotox/jump_table_outside_subprog:OK #537/26 verifier_gotox/jump_table_contains_non_unique_values:OK #537 verifier_gotox:OK
Quoted 1 line
Summary: 2/37 PASSED, 0 SKIPPED, 0 FAILED
Cool!
Quoted 14 lines
Puranjay Mohan (4):
bpf: arm64: Add support for instructions array
bpf: arm64: Add support for indirect jumps
libbpf: Ignore relocations for .jumptables sections
selftests: bpf: Enable gotox tests from arm64arch/arm64/net/bpf_jit_comp.c | 11 +++++++++++ tools/lib/bpf/linker.c | 4 ++++ tools/testing/selftests/bpf/progs/verifier_gotox.c | 4 ++-- 3 files changed, 17 insertions(+), 2 deletions(-)Signature · 1 line
2.47.3
Re: [PATCH bpf-next 1/4] bpf: arm64: Add support for instructions array
In reply to Puranjay MohanQuoted 33 lines · Puranjay Mohan
Add support for the instructions array map type in the arm64 JIT by calling
bpf_prog_update_insn_ptrs()with the offsets that mapxlated_offsetto thejited_offsetin the final image. arm64 JIT already has this offset array which was being used forbpf_prog_fill_jited_linfo()and can be used directly forbpf_prog_update_insn_ptrs.Signed-off-by Puranjay Mohan <[email protected]>--- arch/arm64/net/bpf_jit_comp.c | 7 +++++++ 1 file changed, 7 insertions(+)arch/arm64/net/bpf_jit_comp.c +7
-0@@ -2231,6 +2231,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)for (i =0 ; i <= prog->len; i++) ctx.offset[i] *=AARCH64_INSN_SIZE ;bpf_prog_fill_jited_linfo (prog, ctx.offset +1 ); +/* +* The bpf_prog_update_insn_ptrs function expects offsets to +* point to the first byte of the jitted instruction (unlike +* the bpf_prog_fill_jited_linfo above, which, for historical +* reasons, expects to point to the next instruction) +*/ +bpf_prog_update_insn_ptrs (prog, ctx.offset, ctx.ro_image); out_off:if (!ro_header && priv_stack_ptr) {free_percpu (priv_stack_ptr);Signature · 1 line
2.47.3
Re: [PATCH bpf-next 2/4] bpf: arm64: Add support for indirect jumps
In reply to Puranjay MohanQuoted 35 lines · Puranjay Mohan
Add support for a new instruction
BPF_JMP|BPF_X|BPF_JA, SRC=0, DST=Rx, off=0, imm=0which does an indirect jump to a location stored in Rx. The register Rx should have type
PTR_TO_INSN. This new type assures that the Rx register contains a value (or a range of values) loaded from a correct jump table – map of type instruction array.ARM64 JIT supports indirect jumps to all registers through the
A64_BR()macro, use it to implement this new instruction.Signed-off-by Puranjay Mohan <[email protected]>--- arch/arm64/net/bpf_jit_comp.c | 4 ++++ 1 file changed, 4 insertions(+)arch/arm64/net/bpf_jit_comp.c +4
-0@@ -1452,6 +1452,10 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx,emit (A64_ASR (is64, dst, dst, imm), ctx);break ; +/* JUMP reg */ +case BPF_JMP |BPF_JA |BPF_X : +emit (A64_BR (dst), ctx); +break ;/* JUMP off */ case BPF_JMP |BPF_JA :case BPF_JMP32 |BPF_JA :Signature · 1 line
2.47.3
Re: [PATCH bpf-next 4/4] selftests: bpf: Enable gotox tests from arm64
In reply to Puranjay MohanQuoted 13 lines · Puranjay Mohan
arm64 JIT now supports gotox instruction and jumptables, so run tests in
verifier_gotox.cfor arm64.Signed-off-by Puranjay Mohan <[email protected]>--- tools/testing/selftests/bpf/progs/verifier_gotox.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)tools/testing/selftests/bpf/progs/verifier_gotox.c +0
-0
tools/testing/selftests/bpf/progs/verifier_gotox.c line 9
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c +1
-1@@ -6,7 +6,7 @@#include "bpf_misc.h" #include "../../../include/linux/filter.h" -#ifdef __TARGET_ARCH_x86 +#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
Great to see this :)
Quoted 13 lines
tools/testing/selftests/bpf/progs/verifier_gotox.c +1
-1#define DEFINE_SIMPLE_JUMP_TABLE_PROG (NAME ,SRC_REG ,OFF ,IMM ,OUTCOME ) \ \@@ -384,6 +384,6 @@ jt0_%=: \ : __clobber_all); } -#endif /* __TARGET_ARCH_x86 */ +#endif /* __TARGET_ARCH_x86 || __TARGET_ARCH_arm64 */ char _license[] SEC("license") = "GPL";Signature · 1 line
2.47.3
Puranjay Mohan
maintainerQuoted 21 lines · Anton Protopopov
Quoted 20 lines · Puranjay Mohan
This set adds the support of indirect jumps to the arm64 JIT. It involves calling
bpf_prog_update_insn_ptrs()to support instructions array map. The second piece is supporting BPF_JMP|BPF_X|BPF_JA, SRC=0, DST=Rx, off=0, imm=0 instruction that is trivial to implement on arm64.When running the selftests after doing the above changes, I found that on arm64 builds of llvm, a relocation section was being generated for .jumptables sections and it was making libbpf fail like:
libbpf: relocation against
STT_SECTIONin non-exec section is not supported!
Error: failed to link 'tools/testing/selftests/bpf/cpuv4/bpf_gotox.bpf.o': Invalid argument (22)Which is due to:
Relocation section '.rel.jumptables' at offset
0x5b50contains 263 entries:
Offset Info Type Symbol's Value Symbol's Name0000000000000000 0000000300000002 R_BPF_64_ABS64 0000000000000000 syscall 0000000000000008 0000000300000002 R_BPF_64_ABS64 0000000000000000 syscall 0000000000000010 0000000300000002 R_BPF_64_ABS64 0000000000000000 syscall
Quoted 6 lines
Quoted 2 lines
This rel section is not generated by x86 builds of LLVM. The third patch of this set makes libbpf ignore relocation sections for .jumptables.
I added Yonghong to this thread. He had fixed this problem in
https://github.com/llvm/llvm-project/pull/166301 changes doesn't seem to be x86-specific...
My arm64 build didn't have this change, that is why it was failing. After pulling the latest changes and building again, it works without the libbpf patch.
So, I will send v2 and drop the libbpf patch.
Thanks,
Puranjay