bpf: introduce function calls

First of all huge thank you to Daniel, John, Jakub, Edward and others who reviewed multiple iterations of this patch set over the last many months and to Dave and others who gave critical feedback during netconf/netdev.

The patch is solid enough and we thought through numerous corner cases, but it's not the end. More followups with code reorg and features to follow.

TLDR: Allow arbitrary function calls from bpf function to another bpf function.

Since the beginning of bpf all bpf programs were represented as a single function and program authors were forced to use always_inline for all functions in their C code. That was causing llvm to unnecessary inflate the code size and forcing developers to move code to header files with little code reuse.

With a bit of additional complexity teach verifier to recognize arbitrary function calls from one bpf function to another as long as all of functions are presented to the verifier as a single bpf program. Extended program layout:
..

r1 = ..    // arg1
r2 = ..    // arg2
call pc+1  // function call pc-relative

exit
.. = r1 // access arg1
.. = r2 // access arg2
..
call pc+20 // second level of function call
...

It allows for better optimized code and finally allows to introduce the core bpf libraries that can be reused in different projects, since programs are no longer limited by single elf file. With function calls bpf can be compiled into multiple .o files.

This patch is the first step. It detects programs that contain multiple functions and checks that calls between them are valid. It splits the sequence of bpf instructions (one program) into a set of bpf functions that call each other. Calls to only known functions are allowed. Since all functions are presented to the verifier at once conceptually it is 'static linking'.

Future plans:
- introduce BPF_PROG_TYPE_LIBRARY and allow a set of bpf functions to be loaded into the kernel that can be later linked to other programs with concrete program types. Aka 'dynamic linking'.

- introduce function pointer type and indirect calls to allow bpf functions call other dynamically loaded bpf functions while the caller bpf function is already executing. Aka 'runtime linking'. This will be more generic and more flexible alternative to bpf_tail_calls.

FAQ:
Q: Interpreter and JIT changes mean that new instruction is introduced ?
A: No. The call instruction technically stays the same. Now it can call
both kernel helpers and other bpf functions.
Calling convention stays the same as well.
From uapi point of view the call insn got new 'relocation' BPF_PSEUDO_CALL similar to BPF_PSEUDO_MAP_FD 'relocation' of bpf_ldimm64 insn.

Q: What had to change on LLVM side?
A: Trivial LLVM patch to allow calls was applied to upcoming 6.0 release:
https://reviews.llvm.org/rL318614
with few bugfixes as well.
Make sure to build the latest llvm to have bpf_call support.

More details in the patches.

Alexei Starovoitov (12):
bpf: introduce function calls (function boundaries)
bpf: introduce function calls (verification) selftests/bpf: add verifier tests for bpf_call
bpf: teach verifier to recognize zero initialized stack selftests/bpf: add tests for stack_zero tracking
libbpf: add support for bpf_call selftests/bpf: add bpf_call test selftests/bpf: add xdp noinline test
bpf: add support for bpf_call to interpreter
bpf: fix net.core.bpf_jit_enable race
bpf: x64: add JIT support for multi-function programs
bpf: arm64: add JIT support for multi-function programs

Daniel Borkmann (1):
selftests/bpf: additional bpf_call tests

 arch/arm/net/bpf_jit_32.c                        |    2 +-
 arch/arm64/net/bpf_jit_comp.c                    |   70 +-
 arch/mips/net/ebpf_jit.c                         |    2 +-
 arch/powerpc/net/bpf_jit_comp64.c                |    2 +-
 arch/s390/net/bpf_jit_comp.c                     |    2 +-
 arch/sparc/net/bpf_jit_comp_64.c                 |    2 +-
 arch/x86/net/bpf_jit_comp.c                      |   49 +-
 include/linux/bpf.h                              |    4 +
 include/linux/bpf_verifier.h                     |   45 +-
 include/linux/filter.h                           |   13 +-
 include/uapi/linux/bpf.h                         |    6 +
 kernel/bpf/core.c                                |  104 +-
 kernel/bpf/disasm.c                              |    8 +-
 kernel/bpf/syscall.c                             |    3 +-
 kernel/bpf/verifier.c                            | 1120 ++++++++++++---
 tools/include/uapi/linux/bpf.h                   |    6 +
 tools/lib/bpf/bpf.h                              |    2 +-
 tools/lib/bpf/libbpf.c                           |  170 ++-
 tools/testing/selftests/bpf/Makefile             |   12 +-
 tools/testing/selftests/bpf/test_l4lb_noinline.c |  473 +++++++
 tools/testing/selftests/bpf/test_progs.c         |   95 +-
 tools/testing/selftests/bpf/test_verifier.c      | 1624 +++++++++++++++++++++-
 tools/testing/selftests/bpf/test_xdp_noinline.c  |  833 +++++++++++
 23 files changed, 4378 insertions(+), 269 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/test_l4lb_noinline.c
 create mode 100644 tools/testing/selftests/bpf/test_xdp_noinline.c

2.9.5