RFC MIPS: eBPF: refactor code, add MIPS32 JIT

This patch series adds an eBPF JIT for MIPS32. The approach taken fixes and updates existing code to support MIPS64/MIPS32 systems, then refactors source into a common core and dependent MIPS64 JIT, and finally adds a MIPS32 eBPF JIT implementation using the common framework. This approach of developing MIPS64 and MIPS32 JITs in tandem has desirable benefits for consistency and long-term maintainability, and the iterative refactoring has helped identify several problems.

Overview
========

The initial code updates and refactoring exposed a number of problems in the existing MIPS64 JIT, which are fixed in patches #1 to #9. Patch #10 updates common code to support MIPS64/MIPS32 operation. Patch #12 separates the common core from the MIPS64 JIT code. Patches #13 and #14 add MIPS64 support for BPF_ATOMIC and BPF_JMP32 insns. Patch #15 adds a needed MIPS32 uasm opcode, while patch #16 adds the MIPS32 eBPF JIT.

Updates to the common core notably include support for bpf2bpf calls and making tailcalls from BPF subprograms (e.g. patch #11). Some lower priority features such as MIPS R1 ISA support, direct kernel calls (e.g. for TCP congestion control experiments) and PROBE_MEM support have been omitted.

On MIPS32, 64-bit BPF registers are mapped to 32-bit register pairs, and all 64-bit operations are built on 32-bit subregister ops. A few differences from the MIPS64 JIT include:

* BPF TAILCALL: counter stored on stack due to register pressure.
* BPF_JMP | JSET | BPF_K: drop bbit insns only usable on MIPS64 Octeon

Since MIPS32 does not include 64-bit div/mod or atomic opcodes, these BPF insns are implemented by directly calling the built-in kernel functions:
(with thanks to Luke Nelson for posting similar code online)

  * BPF_ATOMIC | BPF_DW | BPF_ADD (+BPF_FETCH)
  * BPF_ATOMIC | BPF_DW | BPF_AND (+BPF_FETCH)
  * BPF_ATOMIC | BPF_DW | BPF_XOR (+BPF_FETCH)
  * BPF_ATOMIC | BPF_DW | BPF_OR  (+BPF_FETCH)
  * BPF_ATOMIC | BPF_DW | BPF_XCHG
  * BPF_ATOMIC | BPF_DW | BPF_CMPXCHG
  * BPF_ALU64  | BPF_DIV | BPF_X
  * BPF_ALU64  | BPF_DIV | BPF_K
  * BPF_ALU64  | BPF_MOD | BPF_X
  * BPF_ALU64  | BPF_MOD | BPF_K

Testing
=======

Testing used LTS kernel 5.10.x and stable 5.13.x running on QEMU/OpenWRT. The test suite included the 'test_bpf' module, and 'test_verifier' and 'test_progs' from kselftests.

Using 'test_progs' from kselftests proved to be difficult in general since cross-compilation depends on libbpf/bpftool, which does not support cross-endian builds. A very hacked build was used, primarily for testing bpf2bpf calls and tailcalls.

The matrix of test configurations executed for this series covered the expected register sizes, MIPS ISA releases, and JIT settings:

WORDSIZE={64-bit,32-bit} x ISA={R2,R6} x JIT={off,on,hardened}

On MIPS32BE and MIPS32LE there was general parity between the results of interpreter vs. JIT-backed tests with respect to the numbers of PASSED, SKIPPED, and FAILED tests. The same was also true of MIPS64 retesting.

For example, the results below on MIPS32 are typical. Note that skipped test 885 is a "scale" test which results in OOM on the QEMU malta MIPS32 test systems used.

  root@OpenWrt:~# sysctl net.core.bpf_jit_enable=1
  root@OpenWrt:~# modprobe test_bpf
  ...
  test_bpf: Summary: 378 PASSED, 0 FAILED, [366/366 JIT'ed]
  root@OpenWrt:~# ./test_verifier 0 884
  ...
  Summary: 1231 PASSED, 0 SKIPPED, 20 FAILED
  root@OpenWrt:~# ./test_verifier 886 1184
  ...
  Summary: 459 PASSED, 1 SKIPPED, 2 FAILED
  root@OpenWrt:~# ./test_progs -n 105,106
  ...
  105 subprogs:OK
  106/1 tailcall_1:OK
  106/2 tailcall_2:OK
  106/3 tailcall_3:OK
  106/4 tailcall_4:OK
  106/5 tailcall_5:OK
  106/6 tailcall_bpf2bpf_1:OK
  106/7 tailcall_bpf2bpf_2:OK
  106/8 tailcall_bpf2bpf_3:OK
  106/9 tailcall_bpf2bpf_4:OK
  106 tailcalls:OK
  Summary: 2/9 PASSED, 0 SKIPPED, 0 FAILED

All feedback and suggestions are much appreciated!

---
Change History:

rfc v2:
* Implement all BPF_ATOMIC ops. For MIPS32 BPF_DW insns, call built-in 64-bit kernel functions.
* Add MIPS64 support for BPF_JMP32 conditionals.
* Support making tailcalls from bpf2bpf functions.
* Support bpf2bpf calls with an extra JIT pass to patch call addresses.
* Add JIT support for bpf_line_info via bpf_prog_fill_jited_linfo().
* Further code optimizations, cleanup and simplification.
* Update kernel docs.

rfc v1:
* Initial code proposal, focused on consistency and maintainability for both MIPS32/MIPS64.
* Several MIPS64 bugfixes and factoring out common shareable code.
* Addition of MIPS32 JIT, roughly matching MIPS64 capabilities.

---
Tony Ambardar (16):
MIPS: eBPF: support BPF_TAIL_CALL in JIT static analysis
MIPS: eBPF: mask 32-bit index for tail calls
MIPS: eBPF: fix BPF_ALU|ARSH handling in JIT static analysis
MIPS: eBPF: support BPF_JMP32 in JIT static analysis
MIPS: eBPF: fix system hang with verifier dead-code patching
MIPS: eBPF: fix JIT static analysis hang with bounded loops
MIPS: eBPF: fix MOD64 insn on R6 ISA
MIPS: eBPF: support long jump for BPF_JMP|EXIT
MIPS: eBPF: drop src_reg restriction in BPF_LD|BPF_DW|BPF_IMM
MIPS: eBPF: add core support for 32/64-bit systems
bpf: allow tailcalls in subprograms for MIPS64/MIPS32
MIPS: eBPF: refactor common 32/64-bit functions and headers
MIPS: eBPF64: support BPF_JMP32 conditionals
MIPS: eBPF64: implement all BPF_ATOMIC ops
MIPS: uasm: Enable muhu opcode for MIPS R6
MIPS: eBPF: add MIPS32 JIT

 Documentation/admin-guide/sysctl/net.rst |    6 +-
 Documentation/networking/filter.rst      |    6 +-
 arch/mips/Kconfig                        |    4 +-
 arch/mips/include/asm/uasm.h             |    1 +
 arch/mips/mm/uasm-mips.c                 |    4 +-
 arch/mips/mm/uasm.c                      |    3 +-
 arch/mips/net/Makefile                   |    8 +-
 arch/mips/net/ebpf_jit.c                 | 1938 ----------------------
 arch/mips/net/ebpf_jit.h                 |  297 ++++
 arch/mips/net/ebpf_jit_comp32.c          | 1398 ++++++++++++++++
 arch/mips/net/ebpf_jit_comp64.c          | 1085 ++++++++++++
 arch/mips/net/ebpf_jit_core.c            | 1193 +++++++++++++
 arch/x86/net/bpf_jit_comp.c              |    6 +
 include/linux/filter.h                   |    1 +
 kernel/bpf/core.c                        |    6 +
 kernel/bpf/verifier.c                    |    3 +-
 16 files changed, 4010 insertions(+), 1949 deletions(-)
 delete mode 100644 arch/mips/net/ebpf_jit.c
 create mode 100644 arch/mips/net/ebpf_jit.h
 create mode 100644 arch/mips/net/ebpf_jit_comp32.c
 create mode 100644 arch/mips/net/ebpf_jit_comp64.c
 create mode 100644 arch/mips/net/ebpf_jit_core.c

2.25.1