perf stat: Enable BPF counters with --for-each-cgroup
Namhyung Kim
maintainerv2 cover perf stat: Enable BPF counters with --for-each-cgroup
Hello,
This is to add BPF support for --for-each-cgroup to handle many cgroup events on big machines. You can use the --bpf-counters to enable the new behavior.
* changes v2
- remove incorrect use of BPF_F_PRESERVE_ELEMS
- add missing map elements after lookup
- handle cgroup v1
Basic idea is to use a single set of per-cpu events to count interested events and aggregate them to each cgroup. I used bperf mechanism to use a BPF program for cgroup-switches and save the results in a matching map element for given cgroups.
Without this, we need to have separate events for cgroups, and it creates unnecessary multiplexing overhead (and PMU programming) when tasks in different cgroups are switched. I saw this makes a big difference on 256 cpu machines with hundreds of cgroups.
Actually this is what I wanted to do it in the kernel [1], but IIUC with some limitations we can do the job using BPF.
Current limitations are:
* it doesn't support cgroup hierarchy
* there's no reliable way to trigger running the BPF program
Thanks,
Namhyung
[1] https://lore.kernel.org/lkml/[email protected]/
Namhyung Kim (3):
perf tools: Add read_cgroup_id() function perf tools: Add cgroup_is_v2() helper
perf stat: Enable BPF counter with --for-each-cgroup
tools/perf/Makefile.perf | 1 + tools/perf/util/Build | 1 + tools/perf/util/bpf_counter.c | 5 + tools/perf/util/bpf_counter_cgroup.c | 319 ++++++++++++++++++++ tools/perf/util/bpf_skel/bperf_cgroup.bpf.c | 146 +++++++++ tools/perf/util/cgroup.c | 46 +++ tools/perf/util/cgroup.h | 12 + 7 files changed, 530 insertions(+) create mode 100644 tools/perf/util/bpf_counter_cgroup.c create mode 100644 tools/perf/util/bpf_skel/bperf_cgroup.bpf.c
Signature · 1 line
2.32.0.272.g935e593368-goog
Namhyung Kim
maintainerPATCH v2 1/3 perf tools: Add read_cgroup_id() function
The read_cgroup_id() is to read a cgroup id from a file handle using name_to_handle_at(2) for the given cgroup. It'll be used by bperf cgroup stat later.
--- tools/perf/util/cgroup.c | 25 +++++++++++++++++++++++++ tools/perf/util/cgroup.h | 9 +++++++++ 2 files changed, 34 insertions(+)
tools/perf/util/cgroup.c +25 -0
@@ -45,6 +45,31 @@ static int open_cgroup(const char *name)return fd; } +#ifdef HAVE_FILE_HANDLE +int read_cgroup_id (struct cgroup *cgrp) +{ +char path[PATH_MAX +1 ]; +char mnt[PATH_MAX +1 ]; +struct { +struct file_handle fh; +uint64_t cgroup_id; + } handle; +int mount_id; + +if (cgroupfs_find_mountpoint (mnt,PATH_MAX +1 ,"perf_event" )) +return -1 ; + +scnprintf (path,PATH_MAX ,"%s/%s" , mnt, cgrp->name); + + handle.fh.handle_bytes =sizeof (handle.cgroup_id); +if (name_to_handle_at (AT_FDCWD , path, &handle.fh, &mount_id,0 ) <0 ) +return -1 ; + + cgrp->id = handle.cgroup_id; +return 0 ; +} +#endif /* HAVE_FILE_HANDLE */ +static struct cgroup *evlist__find_cgroup (struct evlist *evlist,const char *str) {struct evsel *counter;
tools/perf/util/cgroup.h +9 -0
@@ -38,4 +38,13 @@ struct cgroup *cgroup__find(struct perf_env *env, uint64_t id);void perf_env__purge_cgroups (struct perf_env *env); +#ifdef HAVE_FILE_HANDLE +int read_cgroup_id (struct cgroup *cgrp); +#else +int read_cgroup_id (struct cgroup *cgrp) +{ +return -1 ; +} +#endif /* HAVE_FILE_HANDLE */ +#endif /* __CGROUP_H__ */
Signature · 1 line
2.32.0.272.g935e593368-goog
Namhyung Kim
maintainerPATCH v2 2/3 perf tools: Add cgroup_is_v2() helper
In reply to Namhyung KimThe cgroup_is_v2() is to check if the given subsystem is mounted on cgroup v2 or not. It'll be used by BPF cgroup code later.
--- tools/perf/util/cgroup.c | 19 +++++++++++++++++++ tools/perf/util/cgroup.h | 2 ++ 2 files changed, 21 insertions(+)
tools/perf/util/cgroup.c +19 -0
@@ -9,6 +9,7 @@#include <linux/zalloc.h> #include <sys/types.h> #include <sys/stat.h> +#include <sys/statfs.h> #include <fcntl.h> #include <stdlib.h> #include <string.h>
@@ -70,6 +71,24 @@ int read_cgroup_id(struct cgroup *cgrp) }#endif /* HAVE_FILE_HANDLE */ +#ifndef CGROUP2_SUPER_MAGIC +#define CGROUP2_SUPER_MAGIC 0x63677270 +#endif + +int cgroup_is_v2 (const char *subsys) +{ +char mnt[PATH_MAX +1 ]; +struct statfs stbuf; + +if (cgroupfs_find_mountpoint (mnt,PATH_MAX +1 , subsys)) +return -1 ; + +if (statfs (mnt, stbuf) <0 ) +return -1 ; + +return (stbuf.f_type ==CGROUP2_SUPER_MAGIC ); +} +static struct cgroup *evlist__find_cgroup (struct evlist *evlist,const char *str) {struct evsel *counter;
tools/perf/util/cgroup.h +2 -0
@@ -47,4 +47,6 @@ int read_cgroup_id(struct cgroup *cgrp) }#endif /* HAVE_FILE_HANDLE */ +int cgroup_is_v2 (const char *subsys); +#endif /* __CGROUP_H__ */
Signature · 1 line
2.32.0.272.g935e593368-goog
Namhyung Kim
maintainerPATCH v2 3/3 perf stat: Enable BPF counter with --for-each-cgroup
In reply to Namhyung KimRecently bperf was added to use BPF to count perf events for various purposes. This is an extension for the approach and targetting to cgroup usages.
Unlike the other bperf, it doesn't share the events with other processes but it'd reduce unnecessary events (and the overhead of multiplexing) for each monitored cgroup within the perf session.
When --for-each-cgroup is used with --bpf-counters, it will open cgroup-switches event per cpu internally and attach the new BPF program to read given perf_events and to aggregate the results for cgroups. It's only called when task is switched to a task in a different cgroup.
It seems the bpf test run mechanism doens't work for programs attached the perf_event. So it's not guaranteed to get the accurate results using this. But in practice, perf tools tends to be in a separate cgroup than the actual works, so having an affinity setting loop at the end triggers cgroup switches on each cpu.
--- tools/perf/Makefile.perf | 1 + tools/perf/util/Build | 1 + tools/perf/util/bpf_counter.c | 5 + tools/perf/util/bpf_counter_cgroup.c | 319 ++++++++++++++++++++ tools/perf/util/bpf_skel/bperf_cgroup.bpf.c | 146 +++++++++ tools/perf/util/cgroup.c | 2 + tools/perf/util/cgroup.h | 1 + 7 files changed, 475 insertions(+) create mode 100644 tools/perf/util/bpf_counter_cgroup.c create mode 100644 tools/perf/util/bpf_skel/bperf_cgroup.bpf.c
tools/perf/Makefile.perf +1 -0
@@ -1015,6 +1015,7 @@ SKEL_OUT := $(abspath $(OUTPUT)util/bpf_skel) SKEL_TMP_OUT := $(abspath $(SKEL_OUT)/.tmp) SKELETONS := $(SKEL_OUT)/bpf_prog_profiler.skel.h SKELETONS += $(SKEL_OUT)/bperf_leader.skel.h $(SKEL_OUT)/bperf_follower.skel.h +SKELETONS += $(SKEL_OUT)/bperf_cgroup.skel.h ifdef BUILD_BPF_SKEL BPFTOOL := $(SKEL_TMP_OUT)/bootstrap/bpftool
tools/perf/util/Build +1 -0
@@ -140,6 +140,7 @@ perf-y += clockid.o perf-$(CONFIG_LIBBPF) += bpf-loader.o perf-$(CONFIG_LIBBPF) += bpf_map.o perf-$(CONFIG_PERF_BPF_SKEL) += bpf_counter.o +perf-$(CONFIG_PERF_BPF_SKEL) += bpf_counter_cgroup.o perf-$(CONFIG_BPF_PROLOGUE) += bpf-prologue.o perf-$(CONFIG_LIBELF) += symbol-elf.o perf-$(CONFIG_LIBELF) += probe-file.o
tools/perf/util/bpf_counter.c +5 -0
@@ -22,6 +22,7 @@#include "evsel.h" #include "evlist.h" #include "target.h" +#include "cgroup.h" #include "cpumap.h" #include "thread_map.h"
@@ -792,6 +793,8 @@ struct bpf_counter_ops bperf_ops = { .destroy = bperf__destroy, }; +extern struct bpf_counter_ops bperf_cgrp_ops; +static inline bool bpf_counter_skip (struct evsel *evsel) { return list_empty(&evsel->bpf_counter_list) &&
@@ -809,6 +812,8 @@ int bpf_counter__load(struct evsel *evsel, struct target *target) {if (target->bpf_str) evsel->bpf_counter_ops = &bpf_program_profiler_ops; +else if (cgrp_event_expanded && target->use_bpf) + evsel->bpf_counter_ops = &bperf_cgrp_ops;else if (target->use_bpf || evsel->bpf_counter ||evsel__match_bpf_counter_events (evsel->name)) evsel->bpf_counter_ops = &bperf_ops;
tools/perf/util/bpf_counter_cgroup.c new file +319 -0
319 lines not on this page · raw message
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +146 -0
146 lines not on this page · raw message
tools/perf/util/cgroup.c +2 -0
@@ -18,6 +18,7 @@#include <regex.h> int nr_cgroups; +bool cgrp_event_expanded;/* used to match cgroup name with patterns */ struct cgroup_name {
@@ -484,6 +485,7 @@ int evlist__expand_cgroup(struct evlist *evlist, const char *str, } ret =0 ; + cgrp_event_expanded = true; out_err:evlist__delete (orig_list);
tools/perf/util/cgroup.h +1 -0
@@ -17,6 +17,7 @@ struct cgroup { };extern int nr_cgroups;/* number of explicit cgroups defined */ +extern bool cgrp_event_expanded;struct cgroup *cgroup__get (struct cgroup *cgroup);void cgroup__put (struct cgroup *cgroup);
Signature · 1 line
2.32.0.272.g935e593368-goog
Peter Zijlstra
maintainerQuoted 2 lines · Namhyung Kim
Current limitations are:
* it doesn't support cgroup hierarchy
That seems unfortunate; there's no bpf helper to iterate cgroup hierarchy?
Quoted 1 line
* there's no reliable way to trigger running the BPF program
You can't attach to the PERF_COUNT_SW_CGROUP_SWITCHES event?
Namhyung Kim
maintainerHi Peter,
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Namhyung Kim
Current limitations are:
* it doesn't support cgroup hierarchyThat seems unfortunate; there's no bpf helper to iterate cgroup hierarchy?
I couldn't find one..
Quoted 4 lines
Quoted 2 lines
* there's no reliable way to trigger running the BPF program
You can't attach to the
PERF_COUNT_SW_CGROUP_SWITCHESevent?
I did it. But the BPF test run seems not to work with perf_event. So it needs to trigger a cgroup switch manually..
Thanks,
Namhyung
Peter Zijlstra
maintainerQuoted 4 lines · Namhyung Kim
Quoted 2 lines
That seems unfortunate; there's no bpf helper to iterate cgroup hierarchy?
I couldn't find one..
Song, is that something that would make sense to have?
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
* there's no reliable way to trigger running the BPF program
You can't attach to the
PERF_COUNT_SW_CGROUP_SWITCHESevent?I did it. But the BPF test run seems not to work with
perf_event. So it needs to trigger a cgroup switch manually..
AFAICT it should be possible to set a bpf prog on a software event. perf_event_set_bpf_prog() will take the first branch (!perf_event_is_tracing()) and call perf_event_set_bpf_handler().
That should then result in running the bpf program every time the event would generate a sample.
So if you configure the event to sample on every single event, it should then run your program every time.
This is all from looking at the code, because I really can't operate any of that for real. I suspect Song can help out.
The alternative is to attach a BPF program to the sched_switch tracepoint and do the cgroup filter in BPF.
Quoted 12 lines · Peter Zijlstra
Quoted 4 lines · Namhyung Kim
Quoted 2 lines
That seems unfortunate; there's no bpf helper to iterate cgroup hierarchy?
I couldn't find one..
Song, is that something that would make sense to have?
I think we can solve this with bpf_get_current_ancestor_cgroup_id and a bounded loop. Like:
/* get diff_reading, which is reading - prev_reading */
for (i = 0; i < 10 /* at most 10 levels */; i++) {
__u64 cgroup_id = bpf_get_current_ancestor_cgroup_id(i);
if (!cgroup_id)
break;
/* add diff_reading to cgroup_id */
}Quoted 15 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
* there's no reliable way to trigger running the BPF program
You can't attach to the
PERF_COUNT_SW_CGROUP_SWITCHESevent?I did it. But the BPF test run seems not to work with
perf_event. So it needs to trigger a cgroup switch manually..AFAICT it should be possible to set a bpf prog on a software event.
perf_event_set_bpf_prog()will take the first branch (!perf_event_is_tracing()) and callperf_event_set_bpf_handler().That should then result in running the bpf program every time the event would generate a sample.
Quoted 8 lines
So if you configure the event to sample on every single event, it should then run your program every time.
This is all from looking at the code, because I really can't operate any of that for real. I suspect Song can help out.
The alternative is to attach a BPF program to the
sched_switchtracepoint and do the cgroup filter in BPF.
We can create a raw_tp BPF program just for BPF_PROG_TEST_RUN (now also called BPF_PROG_RUN). The program should be the same as current on_switch program. We don't have to attach the program, just use BPF_PROG_RUN to trigger it.
Would something like this work?
Thanks,
Song
Namhyung Kim
maintainerHi Song,
Quoted 17 lines · Song Liu
Quoted 13 lines · Peter Zijlstra
Quoted 4 lines · Namhyung Kim
Quoted 2 lines
That seems unfortunate; there's no bpf helper to iterate cgroup hierarchy?
I couldn't find one..
Song, is that something that would make sense to have?
I think we can solve this with
bpf_get_current_ancestor_cgroup_idand a bounded loop. Like:
Quoted 8 lines
/* get diff_reading, which is reading - prev_reading */
for (i = 0; i < 10 /* at most 10 levels */; i++) {
__u64 cgroup_id = bpf_get_current_ancestor_cgroup_id(i);
if (!cgroup_id)
break;
/* add diff_reading to cgroup_id */
}OK, but I'm not sure 0 id is guaranteed.
Quoted 22 lines
Quoted 22 lines
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
* there's no reliable way to trigger running the BPF program
You can't attach to the
PERF_COUNT_SW_CGROUP_SWITCHESevent?I did it. But the BPF test run seems not to work with
perf_event. So it needs to trigger a cgroup switch manually..AFAICT it should be possible to set a bpf prog on a software event.
perf_event_set_bpf_prog()will take the first branch (!perf_event_is_tracing()) and callperf_event_set_bpf_handler().That should then result in running the bpf program every time the event would generate a sample.
So if you configure the event to sample on every single event, it should then run your program every time.
This is all from looking at the code, because I really can't operate any of that for real. I suspect Song can help out.
Quoted 8 lines
Quoted 2 lines
The alternative is to attach a BPF program to the
sched_switchtracepoint and do the cgroup filter in BPF.We can create a
raw_tpBPF program just forBPF_PROG_TEST_RUN(now also calledBPF_PROG_RUN). The program should be the same as currenton_switchprogram. We don't have to attach the program, just useBPF_PROG_RUNto trigger it.Would something like this work?
Oh, I think it'd work. Thanks for the suggestion!
Thanks,
Namhyung