Add RISC-V (RV64G) BPF JIT
Subsystems (9)
v2 cover Add RISC-V (RV64G) BPF JIT
From: Björn Töpel <[email protected]>
Hi!
This v2 series adds an RV64G BPF JIT to the kernel.
At the moment the RISC-V Linux port does not support CONFIG_HAVE_KPROBES (Patrick Stählin sent out an RFC last year), which means that CONFIG_BPF_EVENTS is not supported. Thus, no tests involving BPF_PROG_TYPE_TRACEPOINT, BPF_PROG_TYPE_PERF_EVENT, BPF_PROG_TYPE_KPROBE and BPF_PROG_TYPE_RAW_TRACEPOINT passes.
The implementation does not support "far branching" (>4KiB).
Test results:
# modprobe test_bpf test_bpf: Summary: 378 PASSED, 0 FAILED, [366/366 JIT'ed]
# echo 1 > /proc/sys/kernel/unprivileged_bpf_disabled # ./test_verifier ... Summary: 761 PASSED, 507 SKIPPED, 2 FAILED
Note that "test_verifier" was run with one build with CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y and one without, otherwise many of the the tests that require unaligned access were skipped.
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y:
# echo 1 > /proc/sys/kernel/unprivileged_bpf_disabled # ./test_verifier | grep -c 'NOTE.*unknown align' 0
No CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS:
# echo 1 > /proc/sys/kernel/unprivileged_bpf_disabled # ./test_verifier | grep -c 'NOTE.*unknown align' 59
The two failing test_verifier tests are:
"ld_abs: vlan + abs, test 1"
"ld_abs: jump around ld_abs"
This is due to that "far branching" involved in those tests. All tests where done on QEMU emulator version 3.1.50 (v3.1.0-688-g8ae951fbc106). I'll test it on real hardware, when I get access to it.
I'm routing this patch via bpf-next/netdev mailing list (after a conversation with Palmer at FOSDEM), mainly because the other JITs went that path.
Again, thanks for all the comments!
Cheers,
Björn
v1 -> v2:
* Added JMP32 support. (Daniel)
* Add RISC-V to Documentation/sysctl/net.txt. (Daniel)
* Fixed seen_call() asymmetry. (Daniel)
* Fixed broken bpf_flush_icache() range. (Daniel)
* Added alignment annotations to some selftests.
RFCv1 -> v1:
* Cleaned up the Kconfig and net/Makefile. (Christoph)
* Removed the entry-stub and squashed the build/config changes to be part of the JIT implementation. (Christoph)
* Simplified the register tracking code. (Daniel)
* Removed unused macros. (Daniel)
* Added myself as maintainer and updated documentation. (Daniel)
* Removed HAVE_EFFICIENT_UNALIGNED_ACCESS. (Christoph, Palmer)
* Added tail-calls and cleaned up the code.
Björn Töpel (4):
bpf, riscv: add BPF JIT for RV64G
MAINTAINERS: add RISC-V BPF JIT maintainer
bpf, doc: add RISC-V JIT to BPF documentation selftests/bpf: add "any alignment" annotation for some tests
Documentation/networking/filter.txt | 16 +- Documentation/sysctl/net.txt | 1 + MAINTAINERS | 6 + arch/riscv/Kconfig | 1 + arch/riscv/Makefile | 2 +- arch/riscv/net/Makefile | 1 + arch/riscv/net/bpf_jit_comp.c | 1602 +++++++++++++++++ .../selftests/bpf/verifier/ctx_sk_msg.c | 1 + .../testing/selftests/bpf/verifier/ctx_skb.c | 1 + tools/testing/selftests/bpf/verifier/jmp32.c | 22 + tools/testing/selftests/bpf/verifier/jset.c | 2 + .../selftests/bpf/verifier/spill_fill.c | 1 + .../selftests/bpf/verifier/spin_lock.c | 2 + .../selftests/bpf/verifier/value_ptr_arith.c | 4 + 14 files changed, 1654 insertions(+), 8 deletions(-) create mode 100644 arch/riscv/net/Makefile create mode 100644 arch/riscv/net/bpf_jit_comp.c
Signature · 1 line
2.19.1
PATCH v2 1/4 bpf, riscv: add BPF JIT for RV64G
From: Björn Töpel <[email protected]>
This commit adds a BPF JIT for RV64G.
The JIT is a two-pass JIT, and has a dynamic prolog/epilogue (similar to the MIPS64 BPF JIT) instead of static ones (e.g. x86_64).
At the moment the RISC-V Linux port does not support CONFIG_HAVE_KPROBES, which means that CONFIG_BPF_EVENTS is not supported. Thus, no tests involving BPF_PROG_TYPE_TRACEPOINT, BPF_PROG_TYPE_PERF_EVENT, BPF_PROG_TYPE_KPROBE and BPF_PROG_TYPE_RAW_TRACEPOINT passes.
The implementation does not support "far branching" (>4KiB).
Test results:
# modprobe test_bpf test_bpf: Summary: 378 PASSED, 0 FAILED, [366/366 JIT'ed]
# echo 1 > /proc/sys/kernel/unprivileged_bpf_disabled # ./test_verifier ... Summary: 761 PASSED, 507 SKIPPED, 2 FAILED
Note that "test_verifier" was run with one build with CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y and one without, otherwise many of the the tests that require unaligned access were skipped.
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y:
# echo 1 > /proc/sys/kernel/unprivileged_bpf_disabled # ./test_verifier | grep -c 'NOTE.*unknown align' 0
No CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS:
# echo 1 > /proc/sys/kernel/unprivileged_bpf_disabled # ./test_verifier | grep -c 'NOTE.*unknown align' 59
The two failing test_verifier tests are:
"ld_abs: vlan + abs, test 1"
"ld_abs: jump around ld_abs"
This is due to that "far branching" involved in those tests.
All tests where done on QEMU (QEMU emulator version 3.1.50 (v3.1.0-688-g8ae951fbc106)).
--- arch/riscv/Kconfig | 1 + arch/riscv/Makefile | 2 +- arch/riscv/net/Makefile | 1 + arch/riscv/net/bpf_jit_comp.c | 1602 +++++++++++++++++++++++++++++++++ 4 files changed, 1605 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/net/Makefile create mode 100644 arch/riscv/net/bpf_jit_comp.c
arch/riscv/Kconfig +1 -0
@@ -49,6 +49,7 @@ config RISCV select RISCV_TIMER select GENERIC_IRQ_MULTI_HANDLER select ARCH_HAS_PTE_SPECIAL + select HAVE_EBPF_JIT if 64BIT config MMU def_bool y
arch/riscv/Makefile +1 -1
@@ -77,7 +77,7 @@ KBUILD_IMAGE := $(boot)/Image.gz head-y := arch/riscv/kernel/head.o -core-y += arch/riscv/kernel/ arch/riscv/mm/ +core-y += arch/riscv/kernel/ arch/riscv/mm/ arch/riscv/net/ libs-y += arch/riscv/lib/
arch/riscv/net/Makefile new file +1 -0
@@ -0,0 +1 @@ +obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o
arch/riscv/net/bpf_jit_comp.c new file +1602 -0
1,602 lines not on this page · raw message
Signature · 1 line
2.19.1
PATCH v2 2/4 MAINTAINERS: add RISC-V BPF JIT maintainer
In reply to [email protected]From: Björn Töpel <[email protected]>
Add Björn Töpel as RISC-V BPF JIT maintainer.
--- MAINTAINERS | 6 ++++++ 1 file changed, 6 insertions(+)
MAINTAINERS +6 -0
@@ -2907,6 +2907,12 @@ L: [email protected] S: Maintained F: arch/powerpc/net/ +BPF JIT for RISC-V (RV64G) +M: Björn Töpel <[email protected]> +L: [email protected] +S: Maintained +F: arch/riscv/net/ + BPF JIT for S390 M: Martin Schwidefsky <[email protected]> M: Heiko Carstens <[email protected]>
Signature · 1 line
2.19.1
PATCH v2 3/4 bpf, doc: add RISC-V JIT to BPF documentation
In reply to [email protected]From: Björn Töpel <[email protected]>
Update Documentation/networking/filter.txt andDocumentation/sysctl/net.txt to mention RISC-V.
--- Documentation/networking/filter.txt | 16 +++++++++------- Documentation/sysctl/net.txt | 1 + 2 files changed, 10 insertions(+), 7 deletions(-)
Documentation/networking/filter.txt +9 -7
@@ -464,10 +464,11 @@ breakpoints: 0 1 JIT compiler ------------ -The Linux kernel has a built-in BPF JIT compiler for x86_64, SPARC, PowerPC, -ARM, ARM64, MIPS and s390 and can be enabled through CONFIG_BPF_JIT. The JIT -compiler is transparently invoked for each attached filter from user space -or for internal kernel users if it has been previously enabled by root: +The Linux kernel has a built-in BPF JIT compiler for x86_64, SPARC, +PowerPC, ARM, ARM64, MIPS, RISC-V and s390 and can be enabled through +CONFIG_BPF_JIT. The JIT compiler is transparently invoked for each +attached filter from user space or for internal kernel users if it has +been previously enabled by root: echo 1 > /proc/sys/net/core/bpf_jit_enable
@@ -603,9 +604,10 @@ got from bpf_prog_create(), and 'ctx' the given context (e.g. skb pointer). All constraints and restrictions from bpf_check_classic() apply before a conversion to the new layout is being done behind the scenes! -Currently, the classic BPF format is being used for JITing on most 32-bit -architectures, whereas x86-64, aarch64, s390x, powerpc64, sparc64, arm32 perform -JIT compilation from eBPF instruction set. +Currently, the classic BPF format is being used for JITing on most +32-bit architectures, whereas x86-64, aarch64, s390x, powerpc64, +sparc64, arm32, riscv (RV64G) perform JIT compilation from eBPF +instruction set. Some core changes of the new internal format:
Documentation/sysctl/net.txt +1 -0
@@ -52,6 +52,7 @@ two flavors of JITs, the newer eBPF JIT currently supported on: - sparc64 - mips64 - s390x + - riscv And the older cBPF JIT supported on the following archs: - mips
Signature · 1 line
2.19.1
PATCH v2 4/4 selftests/bpf: add "any alignment" annotation for some tests
In reply to [email protected]From: Björn Töpel <[email protected]>
RISC-V does, in-general, not have "efficient unaligned access". When testing the RISC-V BPF JIT, some selftests failed in the verification due to misaligned access. Annotate these tests with the F_NEEDS_EFFICIENT_UNALIGNED_ACCESS flag.
--- .../selftests/bpf/verifier/ctx_sk_msg.c | 1 + .../testing/selftests/bpf/verifier/ctx_skb.c | 1 + tools/testing/selftests/bpf/verifier/jmp32.c | 22 +++++++++++++++++++ tools/testing/selftests/bpf/verifier/jset.c | 2 ++ .../selftests/bpf/verifier/spill_fill.c | 1 + .../selftests/bpf/verifier/spin_lock.c | 2 ++ .../selftests/bpf/verifier/value_ptr_arith.c | 4 ++++ 7 files changed, 33 insertions(+)
tools/testing/selftests/bpf/verifier/ctx_sk_msg.c +1 -0
@@ -100,6 +100,7 @@ .errstr = "invalid bpf_context access", .result = REJECT, .prog_type = BPF_PROG_TYPE_SK_MSG, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "invalid read past end of SK_MSG",
tools/testing/selftests/bpf/verifier/ctx_skb.c +1 -0
@@ -687,6 +687,7 @@ }, .errstr = "invalid bpf_context access", .result = REJECT, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "check skb->hash half load not permitted, unaligned 3",
tools/testing/selftests/bpf/verifier/jmp32.c +22 -0
@@ -27,6 +27,7 @@ .data64 = { 1ULL << 63 | 1, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jset32: BPF_X",
@@ -58,6 +59,7 @@ .data64 = { 1ULL << 63 | 1, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jset32: min/max deduction",
@@ -93,6 +95,7 @@ .data64 = { -1, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jeq32: BPF_X",
@@ -119,6 +122,7 @@ .data64 = { 1ULL << 63 | 1, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jeq32: min/max deduction",
@@ -154,6 +158,7 @@ .data64 = { -1, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jne32: BPF_X",
@@ -180,6 +185,7 @@ .data64 = { 1ULL << 63 | 2, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jne32: min/max deduction",
@@ -218,6 +224,7 @@ .data64 = { 0, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jge32: BPF_X",
@@ -244,6 +251,7 @@ .data64 = { (UINT_MAX - 1) | 2ULL << 32, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jge32: min/max deduction",
@@ -284,6 +292,7 @@ .data64 = { 0, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jgt32: BPF_X",
@@ -310,6 +319,7 @@ .data64 = { (UINT_MAX - 1) | 2ULL << 32, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jgt32: min/max deduction",
@@ -350,6 +360,7 @@ .data64 = { INT_MAX, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jle32: BPF_X",
@@ -376,6 +387,7 @@ .data64 = { UINT_MAX, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jle32: min/max deduction",
@@ -416,6 +428,7 @@ .data64 = { INT_MAX - 1, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jlt32: BPF_X",
@@ -442,6 +455,7 @@ .data64 = { (INT_MAX - 1) | 3ULL << 32, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jlt32: min/max deduction",
@@ -482,6 +496,7 @@ .data64 = { -2, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jsge32: BPF_X",
@@ -508,6 +523,7 @@ .data64 = { -2, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jsge32: min/max deduction",
@@ -548,6 +564,7 @@ .data64 = { 1, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jsgt32: BPF_X",
@@ -574,6 +591,7 @@ .data64 = { 0x7fffffff, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jsgt32: min/max deduction",
@@ -614,6 +632,7 @@ .data64 = { 1, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jsle32: BPF_X",
@@ -640,6 +659,7 @@ .data64 = { 0x7fffffff | 2ULL << 32, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jsle32: min/max deduction",
@@ -680,6 +700,7 @@ .data64 = { 1, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jslt32: BPF_X",
@@ -706,6 +727,7 @@ .data64 = { 0x7fffffff | 2ULL << 32, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jslt32: min/max deduction",
tools/testing/selftests/bpf/verifier/jset.c +2 -0
@@ -53,6 +53,7 @@ .data64 = { ~0ULL, } }, }, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "jset: sign-extend",
@@ -70,6 +71,7 @@ .result = ACCEPT, .retval = 2, .data = { 1, 0, 0, 0, 0, 0, 0, 1, }, + .flags =F_NEEDS_EFFICIENT_UNALIGNED_ACCESS , }, { "jset: known const compare",
tools/testing/selftests/bpf/verifier/spill_fill.c +1 -0
@@ -46,6 +46,7 @@ .errstr_unpriv = "attempt to corrupt spilled", .errstr = "R0 invalid mem access 'inv", .result = REJECT, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "check corrupted spill/fill, LSB",
tools/testing/selftests/bpf/verifier/spin_lock.c +2 -0
@@ -83,6 +83,7 @@ .result_unpriv = REJECT, .errstr_unpriv = "", .prog_type = BPF_PROG_TYPE_CGROUP_SKB, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "spin_lock: test4 direct ld/st",
@@ -112,6 +113,7 @@ .result_unpriv = REJECT, .errstr_unpriv = "", .prog_type = BPF_PROG_TYPE_CGROUP_SKB, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "spin_lock: test5 call within a locked region",
tools/testing/selftests/bpf/verifier/value_ptr_arith.c +4 -0
@@ -512,6 +512,7 @@ .fixup_map_array_48b = { 3 }, .result = ACCEPT, .retval = 0xabcdef12, + .flags =F_NEEDS_EFFICIENT_UNALIGNED_ACCESS , }, { "map access: unknown scalar += value_ptr, 3",
@@ -537,6 +538,7 @@ .result_unpriv = REJECT, .errstr_unpriv = "R0 pointer arithmetic of map value goes out of range", .retval = 0xabcdef12, + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "map access: unknown scalar += value_ptr, 4",
@@ -559,6 +561,7 @@ .result = REJECT, .errstr = "R1 max value is outside of the array range", .errstr_unpriv = "R1 pointer arithmetic of map value goes out of range", + .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "map access: value_ptr += unknown scalar, 1",
@@ -598,6 +601,7 @@ .fixup_map_array_48b = { 3 }, .result = ACCEPT, .retval = 0xabcdef12, + .flags =F_NEEDS_EFFICIENT_UNALIGNED_ACCESS , }, { "map access: value_ptr += unknown scalar, 3",
Signature · 1 line
2.19.1
Quoted 88 lines · On 02/05/2019 01:41 PM, [email protected] wrote:
From: Björn Töpel <[email protected]>
Hi!
This v2 series adds an RV64G BPF JIT to the kernel.
At the moment the RISC-V Linux port does not support
CONFIG_HAVE_KPROBES(Patrick Stählin sent out an RFC last year), which means thatCONFIG_BPF_EVENTSis not supported. Thus, no tests involvingBPF_PROG_TYPE_TRACEPOINT,BPF_PROG_TYPE_PERF_EVENT,BPF_PROG_TYPE_KPROBEandBPF_PROG_TYPE_RAW_TRACEPOINTpasses.The implementation does not support "far branching" (>4KiB).
Test results:
# modprobe test_bpf test_bpf: Summary: 378 PASSED, 0 FAILED, [366/366 JIT'ed]# echo 1 > /proc/sys/kernel/unprivileged_bpf_disabled # ./test_verifier ... Summary: 761 PASSED, 507 SKIPPED, 2 FAILEDNote that "
test_verifier" was run with one build withCONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=yand one without, otherwise many of the the tests that require unaligned access were skipped.
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y:# echo 1 > /proc/sys/kernel/unprivileged_bpf_disabled # ./test_verifier | grep -c 'NOTE.*unknown align' 0No
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS:# echo 1 > /proc/sys/kernel/unprivileged_bpf_disabled # ./test_verifier | grep -c 'NOTE.*unknown align' 59The two failing
test_verifiertests are:
"ld_abs: vlan + abs, test 1"
"ld_abs: jump aroundld_abs"This is due to that "far branching" involved in those tests. All tests where done on QEMU emulator version 3.1.50 (v3.1.0-688-g8ae951fbc106). I'll test it on real hardware, when I get access to it.
I'm routing this patch via bpf-next/netdev mailing list (after a conversation with Palmer at FOSDEM), mainly because the other JITs went that path.
Again, thanks for all the comments!
Cheers,
Björnv1 -> v2:
* Added JMP32 support. (Daniel)
* Add RISC-V toDocumentation/sysctl/net.txt. (Daniel)
* Fixedseen_call()asymmetry. (Daniel)
* Fixed brokenbpf_flush_icache()range. (Daniel)
* Added alignment annotations to some selftests.RFCv1 -> v1:
* Cleaned up theKconfigand net/Makefile. (Christoph)
* Removed the entry-stub and squashed the build/config changes to be part of the JIT implementation. (Christoph)
* Simplified the register tracking code. (Daniel)
* Removed unused macros. (Daniel)
* Added myself as maintainer and updated documentation. (Daniel)
* RemovedHAVE_EFFICIENT_UNALIGNED_ACCESS. (Christoph, Palmer)
* Added tail-calls and cleaned up the code.Björn Töpel (4):
bpf, riscv: add BPF JIT for RV64G
MAINTAINERS: add RISC-V BPF JIT maintainer
bpf, doc: add RISC-V JIT to BPF documentation selftests/bpf: add "any alignment" annotation for some testsDocumentation/networking/filter.txt | 16 +- Documentation/sysctl/net.txt | 1 + MAINTAINERS | 6 + arch/riscv/Kconfig | 1 + arch/riscv/Makefile | 2 +- arch/riscv/net/Makefile | 1 + arch/riscv/net/bpf_jit_comp.c | 1602 +++++++++++++++++ .../selftests/bpf/verifier/ctx_sk_msg.c | 1 + .../testing/selftests/bpf/verifier/ctx_skb.c | 1 +
Quoted 8 lines
tools/testing/selftests/bpf/verifier/jmp32.c | 22 + tools/testing/selftests/bpf/verifier/jset.c | 2 + .../selftests/bpf/verifier/spill_fill.c | 1 + .../selftests/bpf/verifier/spin_lock.c | 2 + .../selftests/bpf/verifier/value_ptr_arith.c | 4 + 14 files changed, 1654 insertions(+), 8 deletions(-) create mode 100644 arch/riscv/net/Makefile create mode 100644 arch/riscv/net/bpf_jit_comp.c
Applied, thanks!