Introduce pinnable bpf_link kernel abstraction
Subsystems (7)
v2 cover Introduce pinnable bpf_link kernel abstraction
This patch series adds bpf_link abstraction, analogous to libbpf's already existing bpf_link abstraction. This formalizes and makes more uniform existing bpf_link-like BPF program link (attachment) types (raw tracepoint and tracing links), which are FD-based objects that are automatically detached when last file reference is closed. These types of BPF program links are switched to using bpf_link framework.
FD-based bpf_link approach provides great safety guarantees, by ensuring there is not going to be an abandoned BPF program attached, if user process suddenly exits or forgets to clean up after itself. This is especially important in production environment and is what all the recent new BPF link types followed.
One of the previously existing inconveniences of FD-based approach, though, was the scenario in which user process wants to install BPF link and exit, but let attached BPF program run. Now, with bpf_link abstraction in place, it's easy to support pinning links in BPF FS, which is done as part of the same patch #1. This allows FD-based BPF program links to survive exit of a user process and original file descriptor being closed, by creating an file entry in BPF FS. This provides great safety by default, with simple way to opt out for cases where it's needed.
Corresponding libbpf APIs are added in the same patch set, as well as selftests for this functionality.
Other types of BPF program attachments (XDP, cgroup, perf_event, etc) are going to be converted in subsequent patches to follow similar approach.
v1->v2:
- use bpf_link_new_fd() uniformly (Alexei).
Andrii Nakryiko (3):
bpf: introduce pinnable bpf_link abstraction
libbpf: add bpf_link pinning/unpinning
selftests/bpf: add link pinning selftests
include/linux/bpf.h | 13 + kernel/bpf/inode.c | 42 +++- kernel/bpf/syscall.c | 223 ++++++++++++++---- tools/lib/bpf/libbpf.c | 131 +++++++--- tools/lib/bpf/libbpf.h | 5 + tools/lib/bpf/libbpf.map | 5 + .../selftests/bpf/prog_tests/link_pinning.c | 105 +++++++++ .../selftests/bpf/progs/test_link_pinning.c | 25 ++ 8 files changed, 476 insertions(+), 73 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/link_pinning.c create mode 100644 tools/testing/selftests/bpf/progs/test_link_pinning.c
Signature · 1 line
2.17.1
PATCH v2 1/3 bpf: introduce pinnable bpf_link abstraction
Introduce bpf_link abstraction, representing an attachment of BPF program to a BPF hook point (e.g., tracepoint, perf event, etc). bpf_link encapsulates ownership of attached BPF program, reference counting of a link itself, when reference from multiple anonymous inodes, as well as ensures that release callback will be called from a process context, so that users can safely take mutex locks and sleep.
Additionally, with a new abstraction it's now possible to generalize pinning of a link object in BPF FS, allowing to explicitly prevent BPF program detachment on process exit by pinning it in a BPF FS and let it open from independent other process to keep working with it.
Convert two existing bpf_link-like objects (raw tracepoint and tracing BPF program attachments) into utilizing bpf_link framework, making them pinnable in BPF FS. More FD-based bpf_links will be added in follow up patches.
--- include/linux/bpf.h | 13 +++ kernel/bpf/inode.c | 42 +++++++- kernel/bpf/syscall.c | 223 +++++++++++++++++++++++++++++++++++-------- 3 files changed, 232 insertions(+), 46 deletions(-)
include/linux/bpf.h +13 -0
@@ -1056,6 +1056,19 @@ extern int sysctl_unprivileged_bpf_disabled;int bpf_map_new_fd (struct bpf_map *map,int flags);int bpf_prog_new_fd (struct bpf_prog *prog); +struct bpf_link ; + +struct bpf_link_ops { +void (*release)(struct bpf_link *link); +}; + +void bpf_link_init (struct bpf_link *link,const struct bpf_link_ops *ops, +struct bpf_prog *prog); +void bpf_link_inc (struct bpf_link *link); +void bpf_link_put (struct bpf_link *link); +int bpf_link_new_fd (struct bpf_link *link); +struct bpf_link *bpf_link_get_from_fd (u32 ufd); +int bpf_obj_pin_user (u32 ufd,const char __user *pathname);int bpf_obj_get_user (const char __user *pathname,int flags);
kernel/bpf/inode.c +38 -4
106 lines not on this page · raw message
kernel/bpf/syscall.c +181 -42
305 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v2 2/3 libbpf: add bpf_link pinning/unpinning
In reply to Andrii NakryikoWith bpf_link abstraction supported by kernel explicitly, add pinning/unpinning API for links. Also allow to create (open) bpf_link from BPF FS file.
This API allows to have an "ephemeral" FD-based BPF links (like raw tracepoint or fexit/freplace attachments) surviving user process exit, by pinning them in a BPF FS, which is an important use case for long-running BPF programs.
As part of this, expose underlying FD for bpf_link. While legacy bpf_link's might not have a FD associated with them (which will be expressed as a bpf_link with fd=-1), kernel's abstraction is based around FD-based usage, so match it closely. This, subsequently, allows to have a generic pinning/unpinning API for generalized bpf_link. For some types of bpf_links kernel might not support pinning, in which case bpf_link__pin() will return error.
With FD being part of generic bpf_link, also get rid of bpf_link_fd in favor of using vanialla bpf_link.
--- tools/lib/bpf/libbpf.c | 131 +++++++++++++++++++++++++++++++-------- tools/lib/bpf/libbpf.h | 5 ++ tools/lib/bpf/libbpf.map | 5 ++ 3 files changed, 114 insertions(+), 27 deletions(-)
tools/lib/bpf/libbpf.c +104 -27
@@ -6931,6 +6931,8 @@ int bpf_prog_load_xattr(const struct bpf_prog_load_attr *attr,struct bpf_link {int (*detach)(struct bpf_link *link);int (*destroy)(struct bpf_link *link); +char *pin_path;/* NULL, if not pinned */ +int fd;/* hook FD, -1 if not applicable */ bool disconnected; };
@@ -6960,26 +6962,109 @@ int bpf_link__destroy(struct bpf_link *link) err = link->detach (link);if (link->destroy) link->destroy (link); +if (link->pin_path) +free (link->pin_path);free (link);return err; } -struct bpf_link_fd { -struct bpf_link link;/* has to be at the top of struct */ -int fd;/* hook FD */ -}; +int bpf_link__fd (const struct bpf_link *link) +{ +return link->fd; +} + +const char *bpf_link__pin_path (const struct bpf_link *link) +{ +return link->pin_path; +} + +static int bpf_link__detach_fd (struct bpf_link *link) +{ +return close (link->fd); +} + +struct bpf_link *bpf_link__open (const char *path) +{ +struct bpf_link *link; +int fd; + + fd =bpf_obj_get (path); +if (fd <0 ) { + fd = -errno; +pr_warn ("failed to open link at %s: %d\n" , path, fd); +return ERR_PTR (fd); + } + + link =calloc (1 ,sizeof (*link)); +if (!link) { +close (fd); +return ERR_PTR (-ENOMEM ); + } + link->detach = &bpf_link__detach_fd; + link->fd = fd; + + link->pin_path =strdup (path); +if (!link->pin_path) { +bpf_link__destroy (link); +return ERR_PTR (-ENOMEM ); + } + +return link; +} + +int bpf_link__pin (struct bpf_link *link,const char *path) +{ +int err; + +if (link->pin_path) +return -EBUSY ; + err =make_parent_dir (path); +if (err) +return err; + err =check_path (path); +if (err) +return err; + + link->pin_path =strdup (path); +if (!link->pin_path) +return -ENOMEM ; + +if (bpf_obj_pin (link->fd, link->pin_path)) { + err = -errno; +zfree (&link->pin_path); +return err; + } + +pr_debug ("link fd=%d: pinned at %s\n" , link->fd, link->pin_path); +return 0 ; +} + +int bpf_link__unpin (struct bpf_link *link) +{ +int err; + +if (!link->pin_path) +return -EINVAL ; + + err =unlink (link->pin_path); +if (err !=0 ) +return -errno; + +pr_debug ("link fd=%d: unpinned from %s\n" , link->fd, link->pin_path); +zfree (&link->pin_path); +return 0 ; +}static int bpf_link__detach_perf_event (struct bpf_link *link) { -struct bpf_link_fd *l = (void *)link;int err; - err =ioctl (l->fd,PERF_EVENT_IOC_DISABLE ,0 ); + err =ioctl (link->fd,PERF_EVENT_IOC_DISABLE ,0 );if (err) err = -errno; -close (l->fd); +close (link->fd);return err; }
@@ -6987,7 +7072,7 @@ struct bpf_link *bpf_program__attach_perf_event(struct bpf_program *prog, int pfd) {char errmsg[STRERR_BUFSIZE ]; -struct bpf_link_fd *link; +struct bpf_link *link;int prog_fd, err;if (pfd <0 ) {
@@ -7005,7 +7090,7 @@ struct bpf_link *bpf_program__attach_perf_event(struct bpf_program *prog, link =calloc (1 ,sizeof (*link));if (!link)return ERR_PTR (-ENOMEM ); - link->link.detach = &bpf_link__detach_perf_event; + link->detach = &bpf_link__detach_perf_event; link->fd = pfd;if (ioctl (pfd,PERF_EVENT_IOC_SET_BPF , prog_fd) <0 ) {
@@ -7024,7 +7109,7 @@ struct bpf_link *bpf_program__attach_perf_event(struct bpf_program *prog, libbpf_strerror_r(err, errmsg, sizeof(errmsg)));return ERR_PTR (err); } -return (struct bpf_link *)link; +return link; }/*
@@ -7312,18 +7397,11 @@ static struct bpf_link *attach_tp(const struct bpf_sec_def *sec,return link; } -static int bpf_link__detach_fd (struct bpf_link *link) -{ -struct bpf_link_fd *l = (void *)link; - -return close (l->fd); -} -struct bpf_link *bpf_program__attach_raw_tracepoint (struct bpf_program *prog,const char *tp_name) {char errmsg[STRERR_BUFSIZE ]; -struct bpf_link_fd *link; +struct bpf_link *link;int prog_fd, pfd; prog_fd =bpf_program__fd (prog);
@@ -7336,7 +7414,7 @@ struct bpf_link *bpf_program__attach_raw_tracepoint(struct bpf_program *prog, link =calloc (1 ,sizeof (*link));if (!link)return ERR_PTR (-ENOMEM ); - link->link.detach = &bpf_link__detach_fd; + link->detach = &bpf_link__detach_fd; pfd =bpf_raw_tracepoint_open (tp_name, prog_fd);if (pfd <0 ) {
@@ -7348,7 +7426,7 @@ struct bpf_link *bpf_program__attach_raw_tracepoint(struct bpf_program *prog,return ERR_PTR (pfd); } link->fd = pfd; -return (struct bpf_link *)link; +return link; } static struct bpf_link *attach_raw_tp(const struct bpf_sec_def *sec,
@@ -7362,7 +7440,7 @@ static struct bpf_link *attach_raw_tp(const struct bpf_sec_def *sec,struct bpf_link *bpf_program__attach_trace (struct bpf_program *prog) {char errmsg[STRERR_BUFSIZE ]; -struct bpf_link_fd *link; +struct bpf_link *link;int prog_fd, pfd; prog_fd =bpf_program__fd (prog);
@@ -7375,7 +7453,7 @@ struct bpf_link *bpf_program__attach_trace(struct bpf_program *prog) link =calloc (1 ,sizeof (*link));if (!link)return ERR_PTR (-ENOMEM ); - link->link.detach = &bpf_link__detach_fd; + link->detach = &bpf_link__detach_fd; pfd =bpf_raw_tracepoint_open (NULL , prog_fd);if (pfd <0 ) {
@@ -7409,10 +7487,9 @@ struct bpf_link *bpf_program__attach(struct bpf_program *prog)static int bpf_link__detach_struct_ops (struct bpf_link *link) { -struct bpf_link_fd *l = (void *)link;__u32 zero =0 ; -if (bpf_map_delete_elem (l->fd, &zero)) +if (bpf_map_delete_elem (link->fd, &zero))return -errno;return 0 ;
@@ -7421,7 +7498,7 @@ static int bpf_link__detach_struct_ops(struct bpf_link *link)struct bpf_link *bpf_map__attach_struct_ops (struct bpf_map *map) {struct bpf_struct_ops *st_ops; -struct bpf_link_fd *link; +struct bpf_link *link;__u32 i, zero =0 ;int err;
@@ -7453,10 +7530,10 @@ struct bpf_link *bpf_map__attach_struct_ops(struct bpf_map *map)return ERR_PTR (err); } - link->link.detach = bpf_link__detach_struct_ops; + link->detach = bpf_link__detach_struct_ops; link->fd = map->fd; -return (struct bpf_link *)link; +return link; }enum bpf_perf_event_ret
tools/lib/bpf/libbpf.h +5 -0
@@ -219,6 +219,11 @@ LIBBPF_API void bpf_program__unload(struct bpf_program *prog);struct bpf_link ; +LIBBPF_API struct bpf_link *bpf_link__open(const char *path); +LIBBPF_API int bpf_link__fd(const struct bpf_link *link); +LIBBPF_API const char *bpf_link__pin_path(const struct bpf_link *link); +LIBBPF_API int bpf_link__pin(struct bpf_link *link, const char *path); +LIBBPF_API int bpf_link__unpin(struct bpf_link *link); LIBBPF_API void bpf_link__disconnect(struct bpf_link *link); LIBBPF_API int bpf_link__destroy(struct bpf_link *link);
tools/lib/bpf/libbpf.map +5 -0
@@ -238,5 +238,10 @@ LIBBPF_0.0.7 { LIBBPF_0.0.8 { global: + bpf_link__fd; + bpf_link__open; + bpf_link__pin; + bpf_link__pin_path; + bpf_link__unpin; bpf_program__set_attach_target; } LIBBPF_0.0.7;
Signature · 1 line
2.17.1
PATCH v2 3/3 selftests/bpf: add link pinning selftests
In reply to Andrii NakryikoAdd selftests validating link pinning/unpinning and associated BPF link (attachment) lifetime.
--- .../selftests/bpf/prog_tests/link_pinning.c | 105 ++++++++++++++++++ .../selftests/bpf/progs/test_link_pinning.c | 25 +++++ 2 files changed, 130 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/link_pinning.c create mode 100644 tools/testing/selftests/bpf/progs/test_link_pinning.c
tools/testing/selftests/bpf/prog_tests/link_pinning.c new file +105 -0
@@ -0,0 +1,105 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2020 Facebook */ + +#include <test_progs.h> +#include <sys/stat.h> + +#include "test_link_pinning.skel.h" + +static int duration =0 ; + +void test_link_pinning_subtest (struct bpf_program *prog, +struct test_link_pinning__bss *bss) +{ +const char *link_pin_path ="/sys/fs/bpf/pinned_link_test" ; +struct stat statbuf = {}; +struct bpf_link *link; +int err, i; + + link =bpf_program__attach (prog); +if (CHECK (IS_ERR (link),"link_attach" ,"err: %ld\n" ,PTR_ERR (link))) + goto cleanup; + + bss->in =1 ; +usleep (1 ); +CHECK (bss->out !=1 ,"res_check1" ,"exp %d, got %d\n" ,1 , bss->out); + +/* pin link */ + err =bpf_link__pin (link, link_pin_path); +if (CHECK (err,"link_pin" ,"err: %d\n" , err)) + goto cleanup; + +CHECK (strcmp (link_pin_path,bpf_link__pin_path (link)),"pin_path1" , +"exp %s, got %s\n" , link_pin_path,bpf_link__pin_path (link)); + +/* check that link was pinned */ + err =stat (link_pin_path, &statbuf); +if (CHECK (err,"stat_link" ,"err %d errno %d\n" , err, errno)) + goto cleanup; + + bss->in =2 ; +usleep (1 ); +CHECK (bss->out !=2 ,"res_check2" ,"exp %d, got %d\n" ,2 , bss->out); + +/* destroy link, pinned link should keep program attached */ +bpf_link__destroy (link); + link =NULL ; + + bss->in =3 ; +usleep (1 ); +CHECK (bss->out !=3 ,"res_check3" ,"exp %d, got %d\n" ,3 , bss->out); + +/* re-open link from BPFFS */ + link =bpf_link__open (link_pin_path); +if (CHECK (IS_ERR (link),"link_open" ,"err: %ld\n" ,PTR_ERR (link))) + goto cleanup; + +CHECK (strcmp (link_pin_path,bpf_link__pin_path (link)),"pin_path2" , +"exp %s, got %s\n" , link_pin_path,bpf_link__pin_path (link)); + +/* unpin link from BPFFS, program still attached */ + err =bpf_link__unpin (link); +if (CHECK (err,"link_unpin" ,"err: %d\n" , err)) + goto cleanup; + +/* still active, as we have FD open now */ + bss->in =4 ; +usleep (1 ); +CHECK (bss->out !=4 ,"res_check4" ,"exp %d, got %d\n" ,4 , bss->out); + +bpf_link__destroy (link); + link =NULL ; + +/* Validate it's finally detached. +* Actual detachment might get delayed a bit, so there is no reliable +* way to validate it immediately here, let's count up for long enough +* and see if eventually output stops being updated +*/ +for (i =5 ; i <10000 ; i++) { + bss->in = i; +usleep (1 ); +if (bss->out == i -1 ) +break ; + } +CHECK (i ==10000 ,"link_attached" ,"got to iteration #%d\n" , i); + +cleanup: +if (!IS_ERR (link)) +bpf_link__destroy (link); +} + +void test_link_pinning (void ) +{ +struct test_link_pinning * skel; + + skel =test_link_pinning__open_and_load (); +if (CHECK (!skel,"skel_open" ,"failed to open skeleton\n" )) +return ; + +if (test__start_subtest ("pin_raw_tp" )) +test_link_pinning_subtest (skel->progs.raw_tp_prog, skel->bss); +if (test__start_subtest ("pin_tp_btf" )) +test_link_pinning_subtest (skel->progs.tp_btf_prog, skel->bss); + +test_link_pinning__destroy (skel); +}
tools/testing/selftests/bpf/progs/test_link_pinning.c new file +25 -0
@@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2020 Facebook */ + +#include <stdbool.h> +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> + +int in =0 ; +int out =0 ; + +SEC("raw_tp/sys_enter") +int raw_tp_prog (const void *ctx) +{ + out = in; +return 0 ; +} + +SEC("tp_btf/sys_enter") +int tp_btf_prog (const void *ctx) +{ + out = in; +return 0 ; +} + +char _license[] SEC("license") = "GPL";
Signature · 1 line
2.17.1
Alexei Starovoitov
AppliedQuoted 22 lines · Andrii Nakryiko
This patch series adds
bpf_linkabstraction, analogous to libbpf's already existingbpf_linkabstraction. This formalizes and makes more uniform existing bpf_link-like BPF program link (attachment) types (raw tracepoint and tracing links), which are FD-based objects that are automatically detached when last file reference is closed. These types of BPF program links are switched to usingbpf_linkframework.FD-based
bpf_linkapproach provides great safety guarantees, by ensuring there is not going to be an abandoned BPF program attached, if user process suddenly exits or forgets to clean up after itself. This is especially important in production environment and is what all the recent new BPF link types followed.One of the previously existing inconveniences of FD-based approach, though, was the scenario in which user process wants to install BPF link and exit, but let attached BPF program run. Now, with
bpf_linkabstraction in place, it's easy to support pinning links in BPF FS, which is done as part of the same patch #1. This allows FD-based BPF program links to survive exit of a user process and original file descriptor being closed, by creating an file entry in BPF FS. This provides great safety by default, with simple way to opt out for cases where it's needed.
Quoted 5 lines
Corresponding libbpf APIs are added in the same patch set, as well as selftests for this functionality.
Other types of BPF program attachments (XDP, cgroup,
perf_event, etc) are going to be converted in subsequent patches to follow similar approach.
Applied. Thanks.