RFC Unwinding for synchronous callbacks

A callback that a helper calls cannot unwind: the helper is a C frame in between, and it holds things only its epilogue drops.

Code that unwinds due to panics or exceptions doesn't often control where they can be invoked, and reusing the same code in callbacks would lead to verifier rejections since unwinding across such synchronous callbacks is not supported.

With this series, we allow a helper gets a pad of its own. A new macro bpf_call_cb() calls the callback from an asm goto with an extable entry at the return address that points to a label in the helper. unwind_exit of the callback lands there, the helper drops what it holds, and bpf_unwind_leave() sends it on to the pad after its own call, with the r0 the callback unwound with. The normal path pays nothing.

To the verifier such a call is a call of something that can unwind:
unwind_goto has to follow it, and the pad gets the r0 of the callback with its state. bpf_loop(), bpf_for_each_map_elem(), bpf_find_vma(), bpf_user_ringbuf_drain() and the comparator of bpf_rbtree_add() are opened up one by one. A callback may unwind with what it may store into callback_ctx already, and the fix for the pointers that don't outlive the helper [1] will apply to r0 as well.

Tested on x86-64 with clang, with and without kCFI, and gcc for the objects. The arm64 objects compile, but the arm64 run is left to CI.

[1]: https://lore.kernel.org/bpf/[email protected]/

Kumar Kartikeya Dwivedi (11):
bpf: Give a helper a way to call its callback with a pad bpf, arm64: Let the pad after a helper call have r0
bpf: Let a callback unwind into the pad of its helper
bpf: Give the inlined bpf_loop a pad
bpf: Let the callback of bpf_loop unwind
bpf: Let the callback of bpf_for_each_map_elem unwind
bpf: Let the callback of bpf_find_vma unwind
bpf: Let the callback of bpf_user_ringbuf_drain unwind
bpf: Let the comparator of bpf_rbtree_add unwind selftests/bpf: Test callbacks that unwind selftests/bpf: Test what a callback unwinds with

 arch/arm64/net/bpf_jit_comp.c                 |  129 +-
 arch/x86/net/bpf_jit_comp.c                   |   25 +-
 include/linux/bpf_unwind.h                    |  282 +++++
 include/linux/bpf_verifier.h                  |    6 +-
 include/linux/filter.h                        |    1 +
 kernel/bpf/arraymap.c                         |   11 +-
 kernel/bpf/backtrack.c                        |   21 +-
 kernel/bpf/bpf_iter.c                         |   32 +-
 kernel/bpf/core.c                             |   10 +
 kernel/bpf/fixups.c                           |   43 +-
 kernel/bpf/hashtab.c                          |   26 +-
 kernel/bpf/helpers.c                          |   50 +-
 kernel/bpf/liveness.c                         |    7 +
 kernel/bpf/ringbuf.c                          |   23 +-
 kernel/bpf/task_iter.c                        |   21 +-
 kernel/bpf/verifier.c                         |  162 ++-
 .../selftests/bpf/prog_tests/unwind_drain.c   |   71 ++
 .../selftests/bpf/prog_tests/verifier.c       |    2 +
 .../selftests/bpf/progs/unwind_drain.c        |   69 ++
 .../selftests/bpf/progs/verifier_unwind.c     |    3 +-
 .../selftests/bpf/progs/verifier_unwind_cb.c  | 1064 +++++++++++++++++
 21 files changed, 1974 insertions(+), 84 deletions(-)
 create mode 100644 include/linux/bpf_unwind.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/unwind_drain.c
 create mode 100644 tools/testing/selftests/bpf/progs/unwind_drain.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_unwind_cb.c

base-commit: 4169c9c90532df9585c0a2ae513337a58f5e59ab

2.53.0