Refactor perf_event sample user program with libbpf bpf_link
Subsystems (5)
v4 cover Refactor perf_event sample user program with libbpf bpf_link
Currently, some samples are using ioctl for enabling perf_event and attaching BPF programs to this event. However, the bpf_program__attach of libbpf(using bpf_link) is much more intuitive than the previous method using ioctl.
bpf_program__attach_perf_event manages the enable of perf_event and attach of BPF programs to it, so there's no neeed to do this directly with ioctl.
In addition, bpf_link provides consistency in the use of API because it allows disable (detach, destroy) for multiple events to be treated as one bpf_link__destroy.
To refactor samples with using this libbpf API, the bpf_load in the samples were removed and migrated to libbbpf. Because read_trace_pipe is used in bpf_load, multiple samples cannot be migrated to libbpf, this function was moved to trace_helpers.
Changes in v2:
- check memory allocation is successful
- clean up allocated memory on error
Changes in v3:
- Improve pointer error check (IS_ERR())
- change to calloc for easier destroy of bpf_link
- remove perf_event fd list since bpf_link handles fd
- use newer bpf_object__{open/load} API instead of bpf_prog_load
- perf_event for _SC_NPROCESSORS_ONLN instead of _SC_NPROCESSORS_CONF
- sample specific chagnes...
Changes in v4:
- bpf_link *, bpf_object * set NULL on init & err for easier destroy
- close bpf object with bpf_object__close()
Daniel T. Lee (2):
samples: bpf: move read_trace_pipe to trace_helpers
samples: bpf: refactor perf_event user program with libbpf bpf_link
samples/bpf/Makefile | 8 +- samples/bpf/bpf_load.c | 20 ---- samples/bpf/bpf_load.h | 1 - samples/bpf/sampleip_user.c | 98 +++++++++++------ samples/bpf/trace_event_user.c | 112 ++++++++++++++------ samples/bpf/tracex1_user.c | 1 + samples/bpf/tracex5_user.c | 1 + tools/testing/selftests/bpf/trace_helpers.c | 23 ++++ tools/testing/selftests/bpf/trace_helpers.h | 1 + 9 files changed, 171 insertions(+), 94 deletions(-)
Signature · 1 line
2.25.1
PATCH v4 1/2 samples: bpf: move read_trace_pipe to trace_helpers
To reduce the reliance of trace samples (trace*_user) on bpf_load, move read_trace_pipe to trace_helpers. By moving this bpf_loader helper elsewhere, trace functions can be easily migrated to libbbpf.
--- samples/bpf/Makefile | 4 ++-- samples/bpf/bpf_load.c | 20 ------------------ samples/bpf/bpf_load.h | 1 - samples/bpf/tracex1_user.c | 1 + samples/bpf/tracex5_user.c | 1 + tools/testing/selftests/bpf/trace_helpers.c | 23 +++++++++++++++++++++ tools/testing/selftests/bpf/trace_helpers.h | 1 + 7 files changed, 28 insertions(+), 23 deletions(-)
samples/bpf/Makefile +2 -2
@@ -64,11 +64,11 @@ fds_example-objs := fds_example.o sockex1-objs := sockex1_user.o sockex2-objs := sockex2_user.o sockex3-objs := bpf_load.o sockex3_user.o -tracex1-objs := bpf_load.o tracex1_user.o +tracex1-objs := bpf_load.o tracex1_user.o $(TRACE_HELPERS) tracex2-objs := bpf_load.o tracex2_user.o tracex3-objs := bpf_load.o tracex3_user.o tracex4-objs := bpf_load.o tracex4_user.o -tracex5-objs := bpf_load.o tracex5_user.o +tracex5-objs := bpf_load.o tracex5_user.o $(TRACE_HELPERS) tracex6-objs := bpf_load.o tracex6_user.o tracex7-objs := bpf_load.o tracex7_user.o test_probe_write_user-objs := bpf_load.o test_probe_write_user_user.o
samples/bpf/bpf_load.c +0 -20
@@ -665,23 +665,3 @@ int load_bpf_file_fixup_map(const char *path, fixup_map_cb fixup_map) {return do_load_bpf_file (path, fixup_map); } - -void read_trace_pipe (void ) -{ -int trace_fd; - - trace_fd =open (DEBUGFS "trace_pipe" ,O_RDONLY ,0 ); -if (trace_fd <0 ) -return ; - -while (1 ) { -static char buf[4096 ]; -ssize_t sz; - - sz =read (trace_fd, buf,sizeof (buf) -1 ); -if (sz >0 ) { - buf[sz] =0 ; -puts (buf); - } - } -}
samples/bpf/bpf_load.h +0 -1
@@ -53,6 +53,5 @@ extern int map_data_count;int load_bpf_file (char *path);int load_bpf_file_fixup_map (const char *path,fixup_map_cb fixup_map); -void read_trace_pipe (void );int bpf_set_link_xdp_fd (int ifindex,int fd,__u32 flags);#endif
samples/bpf/tracex1_user.c +1 -0
@@ -4,6 +4,7 @@#include <unistd.h> #include <bpf/bpf.h> #include "bpf_load.h" +#include "trace_helpers.h" int main (int ac,char **argv) {
samples/bpf/tracex5_user.c +1 -0
@@ -8,6 +8,7 @@#include <bpf/bpf.h> #include "bpf_load.h" #include <sys/resource.h> +#include "trace_helpers.h" /* install fake seccomp program to enable seccomp code path inside the kernel, * so that our kprobe attached to seccomp_phase1() can be triggered
tools/testing/selftests/bpf/trace_helpers.c +23 -0
@@ -4,12 +4,15 @@#include <string.h> #include <assert.h> #include <errno.h> +#include <fcntl.h> #include <poll.h> #include <unistd.h> #include <linux/perf_event.h> #include <sys/mman.h> #include "trace_helpers.h" +#define DEBUGFS "/sys/kernel/debug/tracing/" +#define MAX_SYMS 300000static struct ksym syms[MAX_SYMS ];static int sym_cnt;
@@ -86,3 +89,23 @@ long ksym_get_addr(const char *name)return 0 ; } + +void read_trace_pipe (void ) +{ +int trace_fd; + + trace_fd =open (DEBUGFS "trace_pipe" ,O_RDONLY ,0 ); +if (trace_fd <0 ) +return ; + +while (1 ) { +static char buf[4096 ]; +ssize_t sz; + + sz =read (trace_fd, buf,sizeof (buf) -1 ); +if (sz >0 ) { + buf[sz] =0 ; +puts (buf); + } + } +}
tools/testing/selftests/bpf/trace_helpers.h +1 -0
@@ -12,5 +12,6 @@ struct ksym {int load_kallsyms (void );struct ksym *ksym_search (long key);long ksym_get_addr (const char *name); +void read_trace_pipe (void );#endif
Signature · 1 line
2.25.1
PATCH v4 2/2 samples: bpf: refactor perf_event user program with libbpf bpf_link
In reply to Daniel T. LeeThe bpf_program__attach of libbpf(using bpf_link) is much more intuitive than the previous method using ioctl.
bpf_program__attach_perf_event manages the enable of perf_event and attach of BPF programs to it, so there's no neeed to do this directly with ioctl.
In addition, bpf_link provides consistency in the use of API because it allows disable (detach, destroy) for multiple events to be treated as one bpf_link__destroy. Also, bpf_link__destroy manages the close() of perf_event fd.
This commit refactors samples that attach the bpf program to perf_event by using libbbpf instead of ioctl. Also the bpf_load in the samples were removed and migrated to use libbbpf API.
---
Changes in v2:
- check memory allocation is successful
- clean up allocated memory on error
Changes in v3:
- Improve pointer error check (IS_ERR())
- change to calloc for easier destroy of bpf_link
- remove perf_event fd list since bpf_link handles fd
- use newer bpf_object__{open/load} API instead of bpf_prog_load
- perf_event for _SC_NPROCESSORS_ONLN instead of _SC_NPROCESSORS_CONF
- find program with name explicitly instead of bpf_program__next
- unconditional bpf_link__destroy() on cleanup
Changes in v4:
- bpf_link *, bpf_object * set NULL on init & err for easier destroy
- close bpf object with bpf_object__close()
samples/bpf/Makefile | 4 +- samples/bpf/sampleip_user.c | 98 +++++++++++++++++++---------- samples/bpf/trace_event_user.c | 112 ++++++++++++++++++++++----------- 3 files changed, 143 insertions(+), 71 deletions(-)
samples/bpf/Makefile +2 -2
@@ -88,8 +88,8 @@ xdp2-objs := xdp1_user.o xdp_router_ipv4-objs := xdp_router_ipv4_user.o test_current_task_under_cgroup-objs := bpf_load.o $(CGROUP_HELPERS) \ test_current_task_under_cgroup_user.o -trace_event-objs := bpf_load.o trace_event_user.o $(TRACE_HELPERS) -sampleip-objs := bpf_load.o sampleip_user.o $(TRACE_HELPERS) +trace_event-objs := trace_event_user.o $(TRACE_HELPERS) +sampleip-objs := sampleip_user.o $(TRACE_HELPERS) tc_l2_redirect-objs := bpf_load.o tc_l2_redirect_user.o lwt_len_hist-objs := bpf_load.o lwt_len_hist_user.o xdp_tx_iptunnel-objs := xdp_tx_iptunnel_user.o
samples/bpf/sampleip_user.c +64 -34
172 lines not on this page · raw message
samples/bpf/trace_event_user.c +77 -35
186 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH bpf-next v4 2/2] samples: bpf: refactor perf_event user program with libbpf bpf_link
Quoted 32 lines · Daniel T. Lee
The
bpf_program__attachoflibbpf(using bpf_link)is much more intuitive than the previous method using ioctl.
bpf_program__attach_perf_eventmanages the enable ofperf_eventand attach of BPF programs to it, so there's no neeed to do this directly with ioctl.In addition,
bpf_linkprovides consistency in the use of API because it allows disable (detach, destroy) for multiple events to be treated as onebpf_link__destroy. Also,bpf_link__destroymanages theclose()ofperf_eventfd.This commit refactors samples that attach the bpf program to
perf_eventby using libbbpf instead of ioctl. Also thebpf_loadin the samples were removed and migrated to use libbbpf API.Signed-off-by Daniel T. Lee <[email protected]>---
Changes in v2:
- check memory allocation is successful
- clean up allocated memory on errorChanges in v3:
- Improve pointer error check (IS_ERR())
- change to calloc for easier destroy ofbpf_link
- removeperf_eventfd list sincebpf_linkhandles fd
- use newer bpf_object__{open/load} API instead ofbpf_prog_load
-perf_eventfor_SC_NPROCESSORS_ONLNinstead of_SC_NPROCESSORS_CONF
- find program with name explicitly instead ofbpf_program__next
- unconditionalbpf_link__destroy()on cleanup
Quoted 9 lines
Changes in v4:
-bpf_link*,bpf_object* set NULL on init & err for easier destroy
- close bpf object withbpf_object__close()samples/bpf/Makefile | 4 +- samples/bpf/sampleip_user.c | 98 +++++++++++++++++++---------- samples/bpf/trace_event_user.c | 112 ++++++++++++++++++++++----------- 3 files changed, 143 insertions(+), 71 deletions(-)
Few more int_exit() problems, sorry I didn't catch it first few times, I'm not very familiar with all these bpf samples.
[...]
samples/bpf/trace_event_user.c static inline int generate_load(void) · line 180
Quoted 12 lines
all_cpu_err:samples/bpf/trace_event_user.c +4
-5-for (i--; i >=0 ; i--) { -ioctl (pmu_fd[i],PERF_EVENT_IOC_DISABLE ); -close (pmu_fd[i]); - } -free (pmu_fd); +for (i--; i >=0 ; i--) +bpf_link__destroy (links[i]); +err: +free (links);if (error)int_exit (0 );
if (error) you should exit with error, no?
Quoted 4 lines
}
static void
test_perf_event_task(struct perf_event_attr *attr){
[...]
samples/bpf/trace_event_user.c static void test_perf_event_task(struct perf_event_attr *attr) · line 215
Quoted 6 lines
err:
samples/bpf/trace_event_user.c +1
-2-ioctl (pmu_fd,PERF_EVENT_IOC_DISABLE ); -close (pmu_fd); +bpf_link__destroy (link);if (error)int_exit (0 );
same comment about exiting with error
Quoted 2 lines
}
(quoted diff) +0
-0@@ -282,7 +297,9 @@ static void test_bpf_perf_event(void)
[...]
samples/bpf/trace_event_user.c int main(int argc, char **argv) · line 350
Quoted 9 lines
samples/bpf/trace_event_user.c +4
-0@@ -305,6 +343,10 @@ int main(int argc, char **argv)return 0 ; }test_bpf_perf_event (); + error =0 ; + +cleanup: +bpf_object__close (obj);int_exit (0 );
here and in previous sample int_exit for whatever purpose sends KILL signal and exits with 0, that seems weird. Any idea why it was done that way?
Review int main(int argc, char **argv) · line 351
Quoted 2 lines
(quoted diff) +1
-1- return 0; + return error;
so with that int_ext() implementation you will never get to this error
Quoted 4 lines
(quoted diff) +0
-1} --2.25.1
Re: [PATCH bpf-next v4 2/2] samples: bpf: refactor perf_event user program with libbpf bpf_link
Quoted 38 lines · Andrii Nakryiko
Quoted 36 lines · Daniel T. Lee
The
bpf_program__attachoflibbpf(using bpf_link)is much more intuitive than the previous method using ioctl.
bpf_program__attach_perf_eventmanages the enable ofperf_eventand attach of BPF programs to it, so there's no neeed to do this directly with ioctl.In addition,
bpf_linkprovides consistency in the use of API because it allows disable (detach, destroy) for multiple events to be treated as onebpf_link__destroy. Also,bpf_link__destroymanages theclose()ofperf_eventfd.This commit refactors samples that attach the bpf program to
perf_eventby using libbbpf instead of ioctl. Also thebpf_loadin the samples were removed and migrated to use libbbpf API.Signed-off-by Daniel T. Lee <[email protected]>---
Changes in v2:
- check memory allocation is successful
- clean up allocated memory on errorChanges in v3:
- Improve pointer error check (IS_ERR())
- change to calloc for easier destroy ofbpf_link
- removeperf_eventfd list sincebpf_linkhandles fd
- use newer bpf_object__{open/load} API instead ofbpf_prog_load
-perf_eventfor_SC_NPROCESSORS_ONLNinstead of_SC_NPROCESSORS_CONF
- find program with name explicitly instead ofbpf_program__next
- unconditionalbpf_link__destroy()on cleanupChanges in v4:
-bpf_link*,bpf_object* set NULL on init & err for easier destroy
- close bpf object withbpf_object__close()
Quoted 9 lines
Quoted 5 lines
samples/bpf/Makefile | 4 +- samples/bpf/sampleip_user.c | 98 +++++++++++++++++++---------- samples/bpf/trace_event_user.c | 112 ++++++++++++++++++++++----------- 3 files changed, 143 insertions(+), 71 deletions(-)Few more
int_exit()problems, sorry I didn't catch it first few times, I'm not very familiar with all these bpf samples.
No, you've catch the exact problem.
In previous patch, it was int_exit(error) but I've revert back on this version. I'll explain more later.
Quoted 38 lines
[...]
samples/bpf/trace_event_user.c static inline int generate_load(void) · line 180
Quoted 12 lines
all_cpu_err:samples/bpf/trace_event_user.c +4
-5-for (i--; i >=0 ; i--) { -ioctl (pmu_fd[i],PERF_EVENT_IOC_DISABLE ); -close (pmu_fd[i]); - } -free (pmu_fd); +for (i--; i >=0 ; i--) +bpf_link__destroy (links[i]); +err: +free (links);if (error)int_exit (0 );if (error) you should exit with error, no?
Quoted 4 lines
}
static void
test_perf_event_task(struct perf_event_attr *attr){[...]
samples/bpf/trace_event_user.c static void test_perf_event_task(struct perf_event_attr *attr) · line 215
Quoted 6 lines
err:
samples/bpf/trace_event_user.c +1
-2-ioctl (pmu_fd,PERF_EVENT_IOC_DISABLE ); -close (pmu_fd); +bpf_link__destroy (link);if (error)int_exit (0 );same comment about exiting with error
Quoted 2 lines
}
(quoted diff) +0
-0@@ -282,7 +297,9 @@ static void test_bpf_perf_event(void)[...]
Quoted 14 lines
samples/bpf/trace_event_user.c int main(int argc, char **argv) · line 350
Quoted 9 lines
samples/bpf/trace_event_user.c +4
-0@@ -305,6 +343,10 @@ int main(int argc, char **argv)return 0 ; }test_bpf_perf_event (); + error =0 ; + +cleanup: +bpf_object__close (obj);int_exit (0 );here and in previous sample
int_exitfor whatever purpose sends KILL signal and exits with 0, that seems weird. Any idea why it was done that way?
I'm not sure why the code was written like that previously. However, IMHO, int_exit() is used as signal handler (not only this, other samples too) and this function is mainly used like this.
signal(SIGINT, int_exit);
When the signal occurs, the function will receive signal number as first parameter. So the reason why I've reverted int_exit(error) to int_exit(0) is,
Considering that this function is used as a signal handler, one function will be used for two purposes.
static void int_exit(int sig)
{
kill(0, SIGKILL);
exit(0);
}Passing error argument will make this function to indicate error on exit or to indicate signal on exit. Also it is weird to pass extra argument which is not signal to signal handler.
Actually when this int_exit() called, it will end before exit(0) and parent and child process will be killed with SIGKILL and the return code will be 137, which is 128 + 9 (SIGKILL).
# ./trace_event # echo $? 137
One option is to remove the int_exit() that was used to terminate during the process, not as a signal handler and change the logic to propagate the error to main. However, this can be somewhat awkward in semantics because a in trace_event_user.c function such as print_stacks() uses int_exit(). (Error propagated from printing stacks? not look good to me)
I not sure what is the best practice in this situation, so I just reverted it to original.
Any ideas or suggestion is welcome.
Quoted 9 lines
Review int main(int argc, char **argv) · line 351
Quoted 2 lines
(quoted diff) +1
-1- return 0; + return error;so with that
int_ext()implementation you will never get to this errorQuoted 4 lines
(quoted diff) +0
-1} --2.25.1
Thank you for your time and effort for the review :)
Best,
Daniel
Re: [PATCH bpf-next v4 2/2] samples: bpf: refactor perf_event user program with libbpf bpf_link
Quoted 145 lines · Daniel T. Lee
Quoted 38 lines · Andrii Nakryiko
Quoted 36 lines · Daniel T. Lee
The
bpf_program__attachoflibbpf(using bpf_link)is much more intuitive than the previous method using ioctl.
bpf_program__attach_perf_eventmanages the enable ofperf_eventand attach of BPF programs to it, so there's no neeed to do this directly with ioctl.In addition,
bpf_linkprovides consistency in the use of API because it allows disable (detach, destroy) for multiple events to be treated as onebpf_link__destroy. Also,bpf_link__destroymanages theclose()ofperf_eventfd.This commit refactors samples that attach the bpf program to
perf_eventby using libbbpf instead of ioctl. Also thebpf_loadin the samples were removed and migrated to use libbbpf API.Signed-off-by Daniel T. Lee <[email protected]>---
Changes in v2:
- check memory allocation is successful
- clean up allocated memory on errorChanges in v3:
- Improve pointer error check (IS_ERR())
- change to calloc for easier destroy ofbpf_link
- removeperf_eventfd list sincebpf_linkhandles fd
- use newer bpf_object__{open/load} API instead ofbpf_prog_load
-perf_eventfor_SC_NPROCESSORS_ONLNinstead of_SC_NPROCESSORS_CONF
- find program with name explicitly instead ofbpf_program__next
- unconditionalbpf_link__destroy()on cleanupChanges in v4:
-bpf_link*,bpf_object* set NULL on init & err for easier destroy
- close bpf object withbpf_object__close()Quoted 9 lines
Quoted 5 lines
samples/bpf/Makefile | 4 +- samples/bpf/sampleip_user.c | 98 +++++++++++++++++++---------- samples/bpf/trace_event_user.c | 112 ++++++++++++++++++++++----------- 3 files changed, 143 insertions(+), 71 deletions(-)Few more
int_exit()problems, sorry I didn't catch it first few times, I'm not very familiar with all these bpf samples.No, you've catch the exact problem.
In previous patch, it wasint_exit(error)but I've revert back on this version. I'll explain more later.Quoted 38 lines
[...]
samples/bpf/trace_event_user.c static inline int generate_load(void) · line 180
Quoted 12 lines
all_cpu_err:samples/bpf/trace_event_user.c +4
-5-for (i--; i >=0 ; i--) { -ioctl (pmu_fd[i],PERF_EVENT_IOC_DISABLE ); -close (pmu_fd[i]); - } -free (pmu_fd); +for (i--; i >=0 ; i--) +bpf_link__destroy (links[i]); +err: +free (links);if (error)int_exit (0 );if (error) you should exit with error, no?
Quoted 4 lines
}
static void
test_perf_event_task(struct perf_event_attr *attr){[...]
samples/bpf/trace_event_user.c static void test_perf_event_task(struct perf_event_attr *attr) · line 215
Quoted 6 lines
err:
samples/bpf/trace_event_user.c +1
-2-ioctl (pmu_fd,PERF_EVENT_IOC_DISABLE ); -close (pmu_fd); +bpf_link__destroy (link);if (error)int_exit (0 );same comment about exiting with error
Quoted 2 lines
}
(quoted diff) +0
-0@@ -282,7 +297,9 @@ static void test_bpf_perf_event(void)[...]
Quoted 14 lines
samples/bpf/trace_event_user.c int main(int argc, char **argv) · line 350
Quoted 9 lines
samples/bpf/trace_event_user.c +4
-0@@ -305,6 +343,10 @@ int main(int argc, char **argv)return 0 ; }test_bpf_perf_event (); + error =0 ; + +cleanup: +bpf_object__close (obj);int_exit (0 );here and in previous sample
int_exitfor whatever purpose sends KILL signal and exits with 0, that seems weird. Any idea why it was done that way?I'm not sure why the code was written like that previously. However, IMHO,
int_exit()is used as signal handler (not only this, other samples too) and this function is mainly used like this.
signal(SIGINT, int_exit);When the signal occurs, the function will receive signal number as first parameter. So the reason why I've reverted
int_exit(error)toint_exit(0)is,Considering that this function is used as a signal handler, one function will be used for two purposes.
static void
int_exit(int sig){ kill(0, SIGKILL); exit(0); }Passing error argument will make this function to indicate error on exit or to indicate signal on exit. Also it is weird to pass extra argument which is not signal to signal handler.
Actually when this
int_exit()called, it will end before exit(0) and parent and child process will be killed with SIGKILL and the return code will be 137, which is 128 + 9 (SIGKILL).# ./trace_event # echo $? 137One option is to remove the
int_exit()that was used to terminate during the process, not as a signal handler and change the logic to propagate the error to main. However, this can be somewhat awkward in semantics because a intrace_event_user.cfunction such asprint_stacks()usesint_exit(). (Error propagated from printing stacks? not look good to me)
Quoted 4 lines
I not sure what is the best practice in this situation, so I just reverted it to original.
Any ideas or suggestion is welcome.
I'd do it as simple as possible. Just exit with error (or return error, whatever makes more sense). I don't get why signal handler code has to be called here. SIGKILL also doesn't make any sense. Let's keep it simple.
Quoted 14 lines
Quoted 9 lines
Review int main(int argc, char **argv) · line 351
Quoted 2 lines
(quoted diff) +1
-1- return 0; + return error;so with that
int_ext()implementation you will never get to this errorQuoted 4 lines
(quoted diff) +0
-1} --2.25.1
Thank you for your time and effort for the review :)
Best,
Daniel