Skip to content

BPF perf link and user-provided bpf_cookie

Andrii Nakryiko bpf 5 versions – Applied

Status from this message: This series was applied to bpf/bpf-next.git (refs/heads/master):

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v5 Latest16 patches 1 replyApplied 16/16
  2. v414 patches 0 repliesSuperseded
  3. v314 patches 11 repliesSuperseded
Earlier versions · 2
VersionPostedPatchesRepliesStatus
v214 patches28 repliesSuperseded
v114 patches1 replySuperseded

Changes in v5 from the cover letter

- rebase on latest bpf-next to resolve merge conflict;
- add ref_ctr_offset to uprobe_opts and corresponding selftest;

Patches in v5 All 16 applied

#Subject and filesSizeRepliesApplied
1/16bpf: refactor BPF_PROG_RUN into a functionDocumentation/networking/filter.rst drivers/media/rc/bpf-lirc.c drivers/net/ppp/ppp_generic.c +17 more+73 -600 repliesApplied
2/16bpf: refactor BPF_PROG_RUN_ARRAY family of macros into functionsinclude/linux/bpf.h include/linux/filter.h kernel/bpf/cgroup.c +1 more+124 -940 repliesApplied
3/16bpf: refactor perf_event_set_bpf_prog() to use struct bpf_prog inputkernel/events/core.c+28 -330 repliesApplied
4/16bpf: implement minimal BPF perf linkinclude/linux/bpf_types.h include/linux/trace_events.h include/uapi/linux/bpf.h +3 more+112 -130 repliesApplied
5/16bpf: allow to specify user-provided bpf_cookie for BPF perf linksdrivers/media/rc/bpf-lirc.c include/linux/bpf.h include/linux/perf_event.h +7 more+73 -281 replyApplied
6/16bpf: add bpf_get_attach_cookie() BPF helper to access bpf_cookie valueinclude/linux/bpf.h include/uapi/linux/bpf.h kernel/trace/bpf_trace.c +1 more+66 -40 repliesApplied
7/16libbpf: re-build libbpf.so when libbpf.map changestools/lib/bpf/Makefile+5 -50 repliesApplied
8/16libbpf: remove unused bpf_link's destroy operation, but add dealloctools/lib/bpf/libbpf.c+5 -40 repliesApplied
9/16libbpf: use BPF perf link when supported by kerneltools/lib/bpf/libbpf.c+90 -210 repliesApplied
10/16libbpf: add bpf_cookie support to bpf_link_create() APItools/lib/bpf/bpf.c tools/lib/bpf/bpf.h tools/lib/bpf/libbpf_internal.h+54 -180 repliesApplied
11/16libbpf: add bpf_cookie to perf_event, kprobe, uprobe, and tp attach APIstools/lib/bpf/libbpf.c tools/lib/bpf/libbpf.h tools/lib/bpf/libbpf.map+127 -250 repliesApplied
12/16selftests/bpf: test low-level perf BPF link APItools/testing/selftests/bpf/prog_tests/perf_link.c tools/testing/selftests/bpf/progs/test_perf_link.c+105 -00 repliesApplied
13/16selftests/bpf: extract uprobe-related helpers into trace_helpers.{c,h}tools/testing/selftests/bpf/prog_tests/attach_probe.c tools/testing/selftests/bpf/trace_helpers.c tools/testing/selftests/bpf/trace_helpers.h+70 -600 repliesApplied
14/16selftests/bpf: add bpf_cookie selftests for high-level APIstools/testing/selftests/bpf/prog_tests/bpf_cookie.c tools/testing/selftests/bpf/progs/test_bpf_cookie.c+339 -00 repliesApplied
15/16libbpf: add uprobe ref counter offset support for USDT semaphorestools/lib/bpf/libbpf.c tools/lib/bpf/libbpf.h+17 -40 repliesApplied
16/16selftests/bpf: add ref_ctr_offset selfteststools/testing/selftests/bpf/prog_tests/attach_probe.c tools/testing/selftests/bpf/trace_helpers.c tools/testing/selftests/bpf/trace_helpers.h+50 -110 repliesApplied

Reviews trailers per patch

7 of 16 patches have a Reviewed-by or Acked-by.

PatchYonghong SongPeter Zijlstra (Intel)
1/16A-b
2/16A-b
3/16A-b A-b
4/16A-b A-b
5/16A-b A-b
6/16A-b
7/16A-b
8/16
9/16
10/16
11/16
12/16
13/16
14/16
15/16
16/16

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v5 thread

Cover letter [PATCH v5 bpf-next 00/16] BPF perf link and user-provided bpf_cookie

This patch set implements an ability for users to specify custom black box u64 value for each BPF program attachment, bpf_cookie, which is available to BPF program at runtime. This is a feature that's critically missing for cases when some sort of generic processing needs to be done by the common BPF program logic (or even exactly the same BPF program) across multiple BPF hooks (e.g., many uniformly handled kprobes) and it's important to be able to distinguish between each BPF hook at runtime (e.g., for additional configuration lookup).

The choice of restricting this to a fixed-size 8-byte u64 value is an explicit design decision. Making this configurable by users adds unnecessary complexity (extra memory allocations, extra complications on the verifier side to validate accesses to variable-sized data area) while not really opening up new possibilities. If user's use case requires storing more data per attachment, it's possible to use either global array, or ARRAY/HASHMAP BPF maps, where bpf_cookie would be used as an index into respective storage, populated by user-space code before creating BPF link. This gives user all the flexibility and control while keeping BPF verifier and BPF helper API simple.

Currently, similar functionality can only be achieved through:
- code-generation and BPF program cloning, which is very complicated and

    unmaintainable;

- on-the-fly C code generation and further runtime compilation, which is what BCC uses and allows to do pretty simply. The big downside is a very heavy-weight Clang/LLVM dependency and inefficient memory usage (due to many BPF program clones and the compilation process itself);
- in some cases (kprobes and sometimes uprobes) it's possible to do function IP lookup to get function-specific configuration. This doesn't work for all the cases (e.g., when attaching uprobes to shared libraries) and has higher runtime overhead and additional programming complexity due to BPF_MAP_TYPE_HASHMAP lookups. Up until recently, before bpf_get_func_ip() BPF helper was added, it was also very complicated and unstable (API-wise) to get traced function's IP from fentry/fexit and kretprobe.

With libbpf and BPF CO-RE, runtime compilation is not an option, so to be able to build generic tracing tooling simply and efficiently, ability to provide additional bpf_cookie value for each *attachment* (as opposed to each BPF program) is extremely important. Two immediate users of this functionality are going to be libbpf-based USDT library (currently in development) and retsnoop ([0]), but I'm sure more applications will come once users get this feature in their kernels.

To achieve above described, all perf_event-based BPF hooks are made available through a new BPF_LINK_TYPE_PERF_EVENT BPF link, which allows to use common LINK_CREATE command for program attachments and generally brings perf_event-based attachments into a common BPF link infrastructure.

With that, LINK_CREATE gets ability to pass throught bpf_cookie value during link creation (BPF program attachment) time. bpf_get_attach_cookie() BPF helper is added to allow fetching this value at runtime from BPF program side. BPF cookie is stored either on struct perf_event itself and fetched from the BPF program context, or is passed through ambient BPF run context, added in c7603cfa04e7 ("bpf: Add ambient BPF runtime context stored in current").

On the libbpf side of things, BPF perf link is utilized whenever is supported by the kernel instead of using PERF_EVENT_IOC_SET_BPF ioctl on perf_event FD. All the tracing attach APIs are extended with OPTS and bpf_cookie is passed through corresponding opts structs.

Last part of the patch set adds few self-tests utilizing new APIs.

There are also a few refactorings along the way to make things cleaner and easier to work with, both in kernel (BPF_PROG_RUN and BPF_PROG_RUN_ARRAY), and throughout libbpf and selftests.

Follow-up patches will extend bpf_cookie to fentry/fexit programs.

While adding uprobe_opts, also extend it with ref_ctr_offset for specifying USDT semaphore (reference counter) offset. Update attach_probe selftests to validate its functionality. This is another feature (along with bpf_cookie) required for implementing libbpf-based USDT solution.

[0] https://github.com/anakryiko/retsnoop

Cc Peter Zijlstra <[email protected]> # for perf_event changes

v4->v5:
- rebase on latest bpf-next to resolve merge conflict;
- add ref_ctr_offset to uprobe_opts and corresponding selftest;
v3->v4:
- get rid of BPF_PROG_RUN macro in favor of bpf_prog_run() (Daniel);
- move #ifdef CONFIG_BPF_SYSCALL check into bpf_set_run_ctx (Daniel);
v2->v3:
- user_ctx -> bpf_cookie, bpf_get_user_ctx -> bpf_get_attach_cookie (Peter);
- fix BPF_LINK_TYPE_PERF_EVENT value fix (Jiri);
- use bpf_prog_run() from bpf_prog_run_pin_on_cpu() (Yonghong);
v1->v2:
- fix build failures on non-x86 arches by gating on CONFIG_PERF_EVENTS.

Andrii Nakryiko (16):
bpf: refactor BPF_PROG_RUN into a function
bpf: refactor BPF_PROG_RUN_ARRAY family of macros into functions
bpf: refactor perf_event_set_bpf_prog() to use struct bpf_prog input
bpf: implement minimal BPF perf link
bpf: allow to specify user-provided bpf_cookie for BPF perf links
bpf: add bpf_get_attach_cookie() BPF helper to access bpf_cookie value
libbpf: re-build libbpf.so when libbpf.map changes
libbpf: remove unused bpf_link's destroy operation, but add dealloc
libbpf: use BPF perf link when supported by kernel
libbpf: add bpf_cookie support to bpf_link_create() API
libbpf: add bpf_cookie to perf_event, kprobe, uprobe, and tp attach
APIs
selftests/bpf: test low-level perf BPF link API selftests/bpf: extract uprobe-related helpers into trace_helpers.{c,h} selftests/bpf: add bpf_cookie selftests for high-level APIs
libbpf: add uprobe ref counter offset support for USDT semaphores selftests/bpf: add ref_ctr_offset selftests

 Documentation/networking/filter.rst           |   4 +-
 drivers/media/rc/bpf-lirc.c                   |   6 +-
 drivers/net/ppp/ppp_generic.c                 |   8 +-
 drivers/net/team/team_mode_loadbalance.c      |   2 +-
 include/linux/bpf.h                           | 200 ++++++++------
 include/linux/bpf_types.h                     |   3 +
 include/linux/filter.h                        |  66 +++--
 include/linux/perf_event.h                    |   1 +
 include/linux/trace_events.h                  |   7 +-
 include/uapi/linux/bpf.h                      |  25 ++
 kernel/bpf/bpf_iter.c                         |   2 +-
 kernel/bpf/cgroup.c                           |  32 +--
 kernel/bpf/core.c                             |  31 ++-
 kernel/bpf/syscall.c                          | 105 +++++++-
 kernel/bpf/trampoline.c                       |   2 +-
 kernel/bpf/verifier.c                         |   2 +-
 kernel/events/core.c                          |  74 ++---
 kernel/trace/bpf_trace.c                      |  47 +++-
 lib/test_bpf.c                                |   2 +-
 net/bpf/test_run.c                            |   6 +-
 net/core/filter.c                             |   4 +-
 net/core/ptp_classifier.c                     |   2 +-
 net/netfilter/xt_bpf.c                        |   2 +-
 net/sched/act_bpf.c                           |   4 +-
 net/sched/cls_bpf.c                           |   4 +-
 tools/include/uapi/linux/bpf.h                |  25 ++
 tools/lib/bpf/Makefile                        |  10 +-
 tools/lib/bpf/bpf.c                           |  32 ++-
 tools/lib/bpf/bpf.h                           |   8 +-
 tools/lib/bpf/libbpf.c                        | 213 ++++++++++++---
 tools/lib/bpf/libbpf.h                        |  75 +++++-
 tools/lib/bpf/libbpf.map                      |   3 +
 tools/lib/bpf/libbpf_internal.h               |  32 ++-
 .../selftests/bpf/prog_tests/attach_probe.c   |  98 ++-----
 .../selftests/bpf/prog_tests/bpf_cookie.c     | 254 ++++++++++++++++++
 .../selftests/bpf/prog_tests/perf_link.c      |  89 ++++++
 .../selftests/bpf/progs/test_bpf_cookie.c     |  85 ++++++
 .../selftests/bpf/progs/test_perf_link.c      |  16 ++
 tools/testing/selftests/bpf/trace_helpers.c   |  87 ++++++
 tools/testing/selftests/bpf/trace_helpers.h   |   4 +
 40 files changed, 1315 insertions(+), 357 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_cookie.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/perf_link.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_bpf_cookie.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_perf_link.c
Signature · 1 line
2.30.2