Skip to contentSkip to conversation

perf stat: Enable BPF counters with --for-each-cgroup

PATCH Superseded Namhyung Kim 2 participants 12 messages Jun 22, 2021 – Jun 24, 2021 perf events

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Namhyung Kim

author maintainer
#0 lore raw

v3 cover perf stat: Enable BPF counters with --for-each-cgroup

Hello,

This is to add BPF support for --for-each-cgroup to handle many cgroup events on big machines. You can use the --bpf-counters to enable the new behavior.

* changes in v3
- support cgroup hierarchy with ancestor ids
- add and trigger raw_tp BPF program
- add a build rule for vmlinux.h

* changes in v2
- remove incorrect use of BPF_F_PRESERVE_ELEMS
- add missing map elements after lookup
- handle cgroup v1

Basic idea is to use a single set of per-cpu events to count interested events and aggregate them to each cgroup. I used bperf mechanism to use a BPF program for cgroup-switches and save the results in a matching map element for given cgroups.

Without this, we need to have separate events for cgroups, and it creates unnecessary multiplexing overhead (and PMU programming) when tasks in different cgroups are switched. I saw this makes a big difference on 256 cpu machines with hundreds of cgroups.

Actually this is what I wanted to do it in the kernel [1], but we can do the job using BPF!

Thanks,
Namhyung

[1] https://lore.kernel.org/lkml/[email protected]/

Namhyung Kim (3):
perf tools: Add read_cgroup_id() function perf tools: Add cgroup_is_v2() helper
perf stat: Enable BPF counter with --for-each-cgroup

 tools/perf/Makefile.perf                    |   7 +-
 tools/perf/util/Build                       |   1 +
 tools/perf/util/bpf_counter.c               |   5 +
 tools/perf/util/bpf_counter_cgroup.c        | 337 ++++++++++++++++++++
 tools/perf/util/bpf_skel/bperf_cgroup.bpf.c | 207 ++++++++++++
 tools/perf/util/cgroup.c                    |  46 +++
 tools/perf/util/cgroup.h                    |  12 +
 7 files changed, 614 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/util/bpf_counter_cgroup.c
 create mode 100644 tools/perf/util/bpf_skel/bperf_cgroup.bpf.c
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
author maintainer
#1 lore raw

PATCH v3 1/3 perf tools: Add read_cgroup_id() function

The read_cgroup_id() is to read a cgroup id from a file handle using name_to_handle_at(2) for the given cgroup. It'll be used by bperf cgroup stat later.

---
 tools/perf/util/cgroup.c | 25 +++++++++++++++++++++++++
 tools/perf/util/cgroup.h |  9 +++++++++
 2 files changed, 34 insertions(+)
tools/perf/util/cgroup.c +25 -0
index f24ab4585553..ef18c988c681 100644
@@ -45,6 +45,31 @@ static int open_cgroup(const char *name)
 	return fd;
 }
 
+#ifdef HAVE_FILE_HANDLE
+int read_cgroup_id(struct cgroup *cgrp)
+{
+	char path[PATH_MAX + 1];
+	char mnt[PATH_MAX + 1];
+	struct {
+		struct file_handle fh;
+		uint64_t cgroup_id;
+	} handle;
+	int mount_id;
+
+	if (cgroupfs_find_mountpoint(mnt, PATH_MAX + 1, "perf_event"))
+		return -1;
+
+	scnprintf(path, PATH_MAX, "%s/%s", mnt, cgrp->name);
+
+	handle.fh.handle_bytes = sizeof(handle.cgroup_id);
+	if (name_to_handle_at(AT_FDCWD, path, &handle.fh, &mount_id, 0) < 0)
+		return -1;
+
+	cgrp->id = handle.cgroup_id;
+	return 0;
+}
+#endif  /* HAVE_FILE_HANDLE */
+
 static struct cgroup *evlist__find_cgroup(struct evlist *evlist, const char *str)
 {
 	struct evsel *counter;
tools/perf/util/cgroup.h +9 -0
index 162906f3412a..707adbe25123 100644
@@ -38,4 +38,13 @@ struct cgroup *cgroup__find(struct perf_env *env, uint64_t id);
 
 void perf_env__purge_cgroups(struct perf_env *env);
 
+#ifdef HAVE_FILE_HANDLE
+int read_cgroup_id(struct cgroup *cgrp);
+#else
+int read_cgroup_id(struct cgroup *cgrp)
+{
+	return -1;
+}
+#endif  /* HAVE_FILE_HANDLE */
+
 #endif /* __CGROUP_H__ */
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
author maintainer
#2 lore raw

PATCH v3 2/3 perf tools: Add cgroup_is_v2() helper

In reply to Namhyung Kim

The cgroup_is_v2() is to check if the given subsystem is mounted on cgroup v2 or not. It'll be used by BPF cgroup code later.

---
 tools/perf/util/cgroup.c | 19 +++++++++++++++++++
 tools/perf/util/cgroup.h |  2 ++
 2 files changed, 21 insertions(+)
tools/perf/util/cgroup.c +19 -0
index ef18c988c681..48ec79211270 100644
@@ -9,6 +9,7 @@
 #include <linux/zalloc.h>
 #include <sys/types.h>
 #include <sys/stat.h>
+#include <sys/statfs.h>
 #include <fcntl.h>
 #include <stdlib.h>
 #include <string.h>
@@ -70,6 +71,24 @@ int read_cgroup_id(struct cgroup *cgrp)
 }
 #endif  /* HAVE_FILE_HANDLE */
 
+#ifndef CGROUP2_SUPER_MAGIC
+#define CGROUP2_SUPER_MAGIC  0x63677270
+#endif
+
+int cgroup_is_v2(const char *subsys)
+{
+	char mnt[PATH_MAX + 1];
+	struct statfs stbuf;
+
+	if (cgroupfs_find_mountpoint(mnt, PATH_MAX + 1, subsys))
+		return -1;
+
+	if (statfs(mnt, stbuf) < 0)
+		return -1;
+
+	return (stbuf.f_type == CGROUP2_SUPER_MAGIC);
+}
+
 static struct cgroup *evlist__find_cgroup(struct evlist *evlist, const char *str)
 {
 	struct evsel *counter;
tools/perf/util/cgroup.h +2 -0
index 707adbe25123..1549ec2fd348 100644
@@ -47,4 +47,6 @@ int read_cgroup_id(struct cgroup *cgrp)
 }
 #endif  /* HAVE_FILE_HANDLE */
 
+int cgroup_is_v2(const char *subsys);
+
 #endif /* __CGROUP_H__ */
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
author maintainer
#3 lore raw

PATCH v3 3/3 perf stat: Enable BPF counter with --for-each-cgroup

In reply to Namhyung Kim

Recently bperf was added to use BPF to count perf events for various purposes. This is an extension for the approach and targetting to cgroup usages.

Unlike the other bperf, it doesn't share the events with other processes but it'd reduce unnecessary events (and the overhead of multiplexing) for each monitored cgroup within the perf session.

When --for-each-cgroup is used with --bpf-counters, it will open cgroup-switches event per cpu internally and attach the new BPF program to read given perf_events and to aggregate the results for cgroups. It's only called when task is switched to a task in a different cgroup.

---
 tools/perf/Makefile.perf                    |   7 +-
 tools/perf/util/Build                       |   1 +
 tools/perf/util/bpf_counter.c               |   5 +
 tools/perf/util/bpf_counter_cgroup.c        | 337 ++++++++++++++++++++
 tools/perf/util/bpf_skel/bperf_cgroup.bpf.c | 207 ++++++++++++
 tools/perf/util/cgroup.c                    |   2 +
 tools/perf/util/cgroup.h                    |   1 +
 7 files changed, 559 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/util/bpf_counter_cgroup.c
 create mode 100644 tools/perf/util/bpf_skel/bperf_cgroup.bpf.c
tools/perf/Makefile.perf +6 -1
index e47f04e5b51e..786cba8f3798 100644
@@ -1015,6 +1015,7 @@ SKEL_OUT := $(abspath $(OUTPUT)util/bpf_skel)
 SKEL_TMP_OUT := $(abspath $(SKEL_OUT)/.tmp)
 SKELETONS := $(SKEL_OUT)/bpf_prog_profiler.skel.h
 SKELETONS += $(SKEL_OUT)/bperf_leader.skel.h $(SKEL_OUT)/bperf_follower.skel.h
+SKELETONS += $(SKEL_OUT)/bperf_cgroup.skel.h
 
 ifdef BUILD_BPF_SKEL
 BPFTOOL := $(SKEL_TMP_OUT)/bootstrap/bpftool
@@ -1032,7 +1033,11 @@ $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) | $(SKEL_TMP_OUT)
 	$(QUIET_CLANG)$(CLANG) -g -O2 -target bpf -Wall -Werror $(BPF_INCLUDE) \
 	  -c $(filter util/bpf_skel/%.bpf.c,$^) -o $@ && $(LLVM_STRIP) -g $@
 
-$(SKEL_OUT)/%.skel.h: $(SKEL_TMP_OUT)/%.bpf.o | $(BPFTOOL)
+$(SKEL_OUT)/vmlinux.h:
+	$(MAKE) -C ../bpf/bpftool OUTPUT=$(SKEL_TMP_OUT)/ $(SKEL_TMP_OUT)/vmlinux.h
+	$(Q)mv $(SKEL_TMP_OUT)/vmlinux.h $(SKEL_OUT)/vmlinux.h
+
+$(SKEL_OUT)/%.skel.h: $(SKEL_TMP_OUT)/%.bpf.o $(SKEL_OUT)/vmlinux.h | $(BPFTOOL)
 	$(QUIET_GENSKEL)$(BPFTOOL) gen skeleton $< > $@
 
 bpf-skel: $(SKELETONS)
tools/perf/util/Build +1 -0
index 95e15d1035ab..700d635448ff 100644
@@ -140,6 +140,7 @@ perf-y += clockid.o
 perf-$(CONFIG_LIBBPF) += bpf-loader.o
 perf-$(CONFIG_LIBBPF) += bpf_map.o
 perf-$(CONFIG_PERF_BPF_SKEL) += bpf_counter.o
+perf-$(CONFIG_PERF_BPF_SKEL) += bpf_counter_cgroup.o
 perf-$(CONFIG_BPF_PROLOGUE) += bpf-prologue.o
 perf-$(CONFIG_LIBELF) += symbol-elf.o
 perf-$(CONFIG_LIBELF) += probe-file.o
tools/perf/util/bpf_counter.c +5 -0
index 974f10e356f0..7812c5d9b826 100644
@@ -22,6 +22,7 @@
 #include "evsel.h"
 #include "evlist.h"
 #include "target.h"
+#include "cgroup.h"
 #include "cpumap.h"
 #include "thread_map.h"
 
@@ -792,6 +793,8 @@ struct bpf_counter_ops bperf_ops = {
 	.destroy    = bperf__destroy,
 };
 
+extern struct bpf_counter_ops bperf_cgrp_ops;
+
 static inline bool bpf_counter_skip(struct evsel *evsel)
 {
 	return list_empty(&evsel->bpf_counter_list) &&
@@ -809,6 +812,8 @@ int bpf_counter__load(struct evsel *evsel, struct target *target)
 {
 	if (target->bpf_str)
 		evsel->bpf_counter_ops = &bpf_program_profiler_ops;
+	else if (cgrp_event_expanded && target->use_bpf)
+		evsel->bpf_counter_ops = &bperf_cgrp_ops;
 	else if (target->use_bpf || evsel->bpf_counter ||
 		 evsel__match_bpf_counter_events(evsel->name))
 		evsel->bpf_counter_ops = &bperf_ops;
tools/perf/util/bpf_counter_cgroup.c new file +337 -0

337 lines not on this page · raw message

tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +207 -0

207 lines not on this page · raw message

tools/perf/util/cgroup.c +2 -0
index 48ec79211270..f7d07b365401 100644
@@ -18,6 +18,7 @@
 #include <regex.h>
 
 int nr_cgroups;
+bool cgrp_event_expanded;
 
 /* used to match cgroup name with patterns */
 struct cgroup_name {
@@ -484,6 +485,7 @@ int evlist__expand_cgroup(struct evlist *evlist, const char *str,
 	}
 
 	ret = 0;
+	cgrp_event_expanded = true;
 
 out_err:
 	evlist__delete(orig_list);
tools/perf/util/cgroup.h +1 -0
index 1549ec2fd348..21f7ccc566e1 100644
@@ -17,6 +17,7 @@ struct cgroup {
 };
 
 extern int nr_cgroups; /* number of explicit cgroups defined */
+extern bool cgrp_event_expanded;
 
 struct cgroup *cgroup__get(struct cgroup *cgroup);
 void cgroup__put(struct cgroup *cgroup);
Signature · 1 line
2.32.0.288.g62a8d224e6-goog
#4 lore raw

Re: [PATCH 3/3] perf stat: Enable BPF counter with --for-each-cgroup

Quoted 45 lines · Namhyung Kim

Recently bperf was added to use BPF to count perf events for various purposes. This is an extension for the approach and targetting to cgroup usages.

Unlike the other bperf, it doesn't share the events with other processes but it'd reduce unnecessary events (and the overhead of multiplexing) for each monitored cgroup within the perf session.

When --for-each-cgroup is used with --bpf-counters, it will open cgroup-switches event per cpu internally and attach the new BPF program to read given perf_events and to aggregate the results for cgroups. It's only called when task is switched to a task in a different cgroup.

---

tools/perf/Makefile.perf                    |   7 +-
tools/perf/util/Build                       |   1 +
tools/perf/util/bpf_counter.c               |   5 +
tools/perf/util/bpf_counter_cgroup.c        | 337 ++++++++++++++++++++
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c | 207 ++++++++++++
tools/perf/util/cgroup.c                    |   2 +
tools/perf/util/cgroup.h                    |   1 +
7 files changed, 559 insertions(+), 1 deletion(-)

create mode 100644 tools/perf/util/bpf_counter_cgroup.c create mode 100644 tools/perf/util/bpf_skel/bperf_cgroup.bpf.c

tools/perf/Makefile.perf +0 -0
index e47f04e5b51e..786cba8f3798 100644
@@ -1015,6 +1015,7 @@ SKEL_OUT := $(abspath $(OUTPUT)util/bpf_skel)
 SKEL_TMP_OUT := $(abspath $(SKEL_OUT)/.tmp)

SKELETONS := $(SKEL_OUT)/bpf_prog_profiler.skel.h
SKELETONS += $(SKEL_OUT)/bperf_leader.skel.h $(SKEL_OUT)/bperf_follower.skel.h +SKELETONS += $(SKEL_OUT)/bperf_cgroup.skel.h

ifdef BUILD_BPF_SKEL
BPFTOOL := $(SKEL_TMP_OUT)/bootstrap/bpftool

tools/perf/Makefile.perf $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) | $(SKEL_TMP_OUT) · line 1037

Quoted 6 lines
tools/perf/Makefile.perf +2 -1
@@ -1032,7 +1033,11 @@ $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) | $(SKEL_TMP_OUT)
 	$(QUIET_CLANG)$(CLANG) -g -O2 -target bpf -Wall -Werror $(BPF_INCLUDE) \
 	  -c $(filter util/bpf_skel/%.bpf.c,$^) -o $@ && $(LLVM_STRIP) -g $@
 
-$(SKEL_OUT)/%.skel.h: $(SKEL_TMP_OUT)/%.bpf.o | $(BPFTOOL)
+$(SKEL_OUT)/vmlinux.h:
+	$(MAKE) -C ../bpf/bpftool OUTPUT=$(SKEL_TMP_OUT)/ $(SKEL_TMP_OUT)/vmlinux.h

We build bpftool with $(BPFTOOL), which is a few lines above. Can we reuse some of that?

Quoted 48 lines
tools/perf/Makefile.perf +3 -0
+	$(Q)mv $(SKEL_TMP_OUT)/vmlinux.h $(SKEL_OUT)/vmlinux.h
+
+$(SKEL_OUT)/%.skel.h: $(SKEL_TMP_OUT)/%.bpf.o $(SKEL_OUT)/vmlinux.h | $(BPFTOOL)
 	$(QUIET_GENSKEL)$(BPFTOOL) gen skeleton $< > $@
 
 bpf-skel: $(SKELETONS)
tools/perf/util/Build +0 -0
index 95e15d1035ab..700d635448ff 100644
@@ -140,6 +140,7 @@ perf-y += clockid.o

perf-$(CONFIG_LIBBPF) += bpf-loader.o perf-$(CONFIG_LIBBPF) += bpf_map.o perf-$(CONFIG_PERF_BPF_SKEL) += bpf_counter.o +perf-$(CONFIG_PERF_BPF_SKEL) += bpf_counter_cgroup.o perf-$(CONFIG_BPF_PROLOGUE) += bpf-prologue.o perf-$(CONFIG_LIBELF) += symbol-elf.o perf-$(CONFIG_LIBELF) += probe-file.o

tools/perf/util/bpf_counter.c +5 -0
index 974f10e356f0..7812c5d9b826 100644
@@ -22,6 +22,7 @@
 #include "evsel.h"
 #include "evlist.h"
 #include "target.h"
+#include "cgroup.h"
 #include "cpumap.h"
 #include "thread_map.h"
 
@@ -792,6 +793,8 @@ struct bpf_counter_ops bperf_ops = {
 	.destroy    = bperf__destroy,
 };
 
+extern struct bpf_counter_ops bperf_cgrp_ops;
+
 static inline bool bpf_counter_skip(struct evsel *evsel)
 {
 	return list_empty(&evsel->bpf_counter_list) &&
@@ -809,6 +812,8 @@ int bpf_counter__load(struct evsel *evsel, struct target *target)
 {
 	if (target->bpf_str)
 		evsel->bpf_counter_ops = &bpf_program_profiler_ops;
+	else if (cgrp_event_expanded && target->use_bpf)
+		evsel->bpf_counter_ops = &bperf_cgrp_ops;
 	else if (target->use_bpf || evsel->bpf_counter ||
 		 evsel__match_bpf_counter_events(evsel->name))
 		evsel->bpf_counter_ops = &bperf_ops;

[...]

Quoted 23 lines
tools/perf/util/bpf_counter.c +23 -0
+
+#include "bpf_skel/bperf_cgroup.skel.h"
+
+static struct perf_event_attr cgrp_switch_attr = {
+	.type = PERF_TYPE_SOFTWARE,
+	.config = PERF_COUNT_SW_CGROUP_SWITCHES,
+	.size = sizeof(cgrp_switch_attr),
+	.sample_period = 1,
+	.disabled = 1,
+};
+
+static struct evsel *cgrp_switch;
+static struct xyarray *cgrp_prog_fds;
+static struct bperf_cgroup_bpf *skel;
+
+#define FD(evt, cpu) (*(int *)xyarray__entry(evt->core.fd, cpu, 0))
+#define PROG(cpu)    (*(int *)xyarray__entry(cgrp_prog_fds, cpu, 0))
+
+static void set_max_rlimit(void)
+{
+	struct rlimit rinf = { RLIM_INFINITY, RLIM_INFINITY };
+
+	setrlimit(RLIMIT_MEMLOCK, &rinf);

tools/perf/util/bpf_counter.c

Quoted 10 lines
tools/perf/util/bpf_counter.c +10 -0
+}
+
+static __u32 bpf_link_get_prog_id(int fd)
+{
+	struct bpf_link_info link_info = {0};
+	__u32 link_info_len = sizeof(link_info);
+
+	bpf_obj_get_info_by_fd(fd, &link_info, &link_info_len);
+	return link_info.prog_id;
+}

How about we move set_max_rlimit() and bpf_link_get_prog_id() to a header so we don't have to duplicate it?

Quoted 23 lines
tools/perf/util/bpf_counter.c +23 -0
+
+static int bperf_load_program(struct evlist *evlist)
+{
+	struct bpf_link *link;
+	struct evsel *evsel;
+	struct cgroup *cgrp, *leader_cgrp;
+	__u32 i, cpu, prog_id;
+	int nr_cpus = evlist->core.all_cpus->nr;
+	int map_size, map_fd;
+	int prog_fd, err;
+
+	skel = bperf_cgroup_bpf__open();
+	if (!skel) {
+		pr_err("Failed to open cgroup skeleton\n");
+		return -1;
+	}
+
+	skel->rodata->num_cpus = nr_cpus;
+	skel->rodata->num_events = evlist->core.nr_entries / nr_cgroups;
+
+	BUG_ON(evlist->core.nr_entries % nr_cgroups != 0);
+
+	/* we need one copy of events per cpu for reading */

tools/perf/util/bpf_counter.c

Quoted 10 lines
tools/perf/util/bpf_counter.c +10 -0
+	map_size = nr_cpus * evlist->core.nr_entries / nr_cgroups;
+	bpf_map__resize(skel->maps.events, map_size);
+	bpf_map__resize(skel->maps.cpu_idx, nr_cpus);
+	bpf_map__resize(skel->maps.cgrp_idx, nr_cgroups);
+	/* previous result is saved in a per-cpu array */
+	map_size = evlist->core.nr_entries / nr_cgroups;
+	bpf_map__resize(skel->maps.prev_readings, map_size);
+	/* cgroup result needs all events */
+	map_size = nr_cpus * evlist->core.nr_entries;
+	bpf_map__resize(skel->maps.cgrp_readings, map_size);

We are setting map_size back and forth here.

[...]

Quoted 26 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +20 -0
new file mode 100644
index 000000000000..6d74e93dd1f5
@@ -0,0 +1,207 @@
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+// Copyright (c) 2021 Facebook
+// Copyright (c) 2021 Google
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include <bpf/bpf_core_read.h>
+
+#define MAX_LEVELS  10  // max cgroup hierarchy level: arbitrary
+#define MAX_EVENTS  32  // max events per cgroup: arbitrary
+
+// NOTE: many of map and global data will be modified before loading
+//       from the userspace (perf tool) using the skeleton helpers.
+
+// single set of global perf events to measure
+struct {
+	__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
+	__uint(key_size, sizeof(__u32));
+	__uint(value_size, sizeof(int));
+	__uint(max_entries, 1);

tools/perf/util/bpf_skel/bperf_cgroup.bpf.c line 30

Quoted 10 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +10 -0
+} events SEC(".maps");
+
+// from logical cpu number to event index
+// useful when user wants to count subset of cpus
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__uint(key_size, sizeof(__u32));
+	__uint(value_size, sizeof(__u32));
+	__uint(max_entries, 1);
+} cpu_idx SEC(".maps");

How about we make cpu_idx a percpu array and use 0,1 for disable/enable profiling on this cpu?

Quoted 13 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +13 -0
+
+// from cgroup id to event index
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__uint(key_size, sizeof(__u64));
+	__uint(value_size, sizeof(__u32));
+	__uint(max_entries, 1);
+} cgrp_idx SEC(".maps");
+
+// per-cpu event snapshots to calculate delta
+struct {
+	__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+	__uint(key_size, sizeof(__u32));

tools/perf/util/bpf_skel/bperf_cgroup.bpf.c line 53

Quoted 10 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +10 -0
+	__uint(value_size, sizeof(struct bpf_perf_event_value));
+} prev_readings SEC(".maps");
+
+// aggregated event values for each cgroup
+// will be read from the user-space
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(key_size, sizeof(__u32));
+	__uint(value_size, sizeof(struct bpf_perf_event_value));
+} cgrp_readings SEC(".maps");

Maybe also make this a percpu array? This should make the BPF program faster.

Quoted 7 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +7 -0
+
+const volatile __u32 num_events = 1;
+const volatile __u32 num_cpus = 1;
+
+int enabled = 0;
+int use_cgroup_v2 = 0;
+

[...]

author maintainer
#5 lore raw

Re: [PATCH 3/3] perf stat: Enable BPF counter with --for-each-cgroup

Hi Song,

Quoted 49 lines · Song Liu
Quoted 49 lines · Namhyung Kim

Recently bperf was added to use BPF to count perf events for various purposes. This is an extension for the approach and targetting to cgroup usages.

Unlike the other bperf, it doesn't share the events with other processes but it'd reduce unnecessary events (and the overhead of multiplexing) for each monitored cgroup within the perf session.

When --for-each-cgroup is used with --bpf-counters, it will open cgroup-switches event per cpu internally and attach the new BPF program to read given perf_events and to aggregate the results for cgroups. It's only called when task is switched to a task in a different cgroup.

---

tools/perf/Makefile.perf                    |   7 +-
tools/perf/util/Build                       |   1 +
tools/perf/util/bpf_counter.c               |   5 +
tools/perf/util/bpf_counter_cgroup.c        | 337 ++++++++++++++++++++
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c | 207 ++++++++++++
tools/perf/util/cgroup.c                    |   2 +
tools/perf/util/cgroup.h                    |   1 +
7 files changed, 559 insertions(+), 1 deletion(-)

create mode 100644 tools/perf/util/bpf_counter_cgroup.c create mode 100644 tools/perf/util/bpf_skel/bperf_cgroup.bpf.c

tools/perf/Makefile.perf +0 -0
index e47f04e5b51e..786cba8f3798 100644
@@ -1015,6 +1015,7 @@ SKEL_OUT := $(abspath $(OUTPUT)util/bpf_skel)
 SKEL_TMP_OUT := $(abspath $(SKEL_OUT)/.tmp)

SKELETONS := $(SKEL_OUT)/bpf_prog_profiler.skel.h
SKELETONS += $(SKEL_OUT)/bperf_leader.skel.h $(SKEL_OUT)/bperf_follower.skel.h +SKELETONS += $(SKEL_OUT)/bperf_cgroup.skel.h

ifdef BUILD_BPF_SKEL
BPFTOOL := $(SKEL_TMP_OUT)/bootstrap/bpftool

tools/perf/Makefile.perf +0 -0
@@ -1032,7 +1033,11 @@ $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) | $(SKEL_TMP_OUT)
      $(QUIET_CLANG)$(CLANG) -g -O2 -target bpf -Wall -Werror $(BPF_INCLUDE) \
        -c $(filter util/bpf_skel/%.bpf.c,$^) -o $@ && $(LLVM_STRIP) -g $@
 
Quoted 6 lines

tools/perf/Makefile.perf $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) | $(SKEL_TMP_OUT) · line 1037

Quoted 3 lines
tools/perf/Makefile.perf +2 -1
-$(SKEL_OUT)/%.skel.h: $(SKEL_TMP_OUT)/%.bpf.o | $(BPFTOOL)
+$(SKEL_OUT)/vmlinux.h:
+     $(MAKE) -C ../bpf/bpftool OUTPUT=$(SKEL_TMP_OUT)/ $(SKEL_TMP_OUT)/vmlinux.h

We build bpftool with $(BPFTOOL), which is a few lines above. Can we reuse some of that?

Ok, will do.

Quoted 81 lines
Quoted 49 lines
tools/perf/Makefile.perf +3 -0
+     $(Q)mv $(SKEL_TMP_OUT)/vmlinux.h $(SKEL_OUT)/vmlinux.h
+
+$(SKEL_OUT)/%.skel.h: $(SKEL_TMP_OUT)/%.bpf.o $(SKEL_OUT)/vmlinux.h | $(BPFTOOL)
      $(QUIET_GENSKEL)$(BPFTOOL) gen skeleton $< > $@
 
 bpf-skel: $(SKELETONS)
tools/perf/util/Build +0 -0
index 95e15d1035ab..700d635448ff 100644
@@ -140,6 +140,7 @@ perf-y += clockid.o

perf-$(CONFIG_LIBBPF) += bpf-loader.o perf-$(CONFIG_LIBBPF) += bpf_map.o perf-$(CONFIG_PERF_BPF_SKEL) += bpf_counter.o +perf-$(CONFIG_PERF_BPF_SKEL) += bpf_counter_cgroup.o perf-$(CONFIG_BPF_PROLOGUE) += bpf-prologue.o perf-$(CONFIG_LIBELF) += symbol-elf.o perf-$(CONFIG_LIBELF) += probe-file.o

tools/perf/util/bpf_counter.c +5 -0
index 974f10e356f0..7812c5d9b826 100644
@@ -22,6 +22,7 @@
 #include "evsel.h"
 #include "evlist.h"
 #include "target.h"
+#include "cgroup.h"
 #include "cpumap.h"
 #include "thread_map.h"
 
@@ -792,6 +793,8 @@ struct bpf_counter_ops bperf_ops = {
      .destroy    = bperf__destroy,
 };
 
+extern struct bpf_counter_ops bperf_cgrp_ops;
+
 static inline bool bpf_counter_skip(struct evsel *evsel)
 {
      return list_empty(&evsel->bpf_counter_list) &&
@@ -809,6 +812,8 @@ int bpf_counter__load(struct evsel *evsel, struct target *target)
 {
      if (target->bpf_str)
              evsel->bpf_counter_ops = &bpf_program_profiler_ops;
+     else if (cgrp_event_expanded && target->use_bpf)
+             evsel->bpf_counter_ops = &bperf_cgrp_ops;
      else if (target->use_bpf || evsel->bpf_counter ||
               evsel__match_bpf_counter_events(evsel->name))
              evsel->bpf_counter_ops = &bperf_ops;

[...]

Quoted 28 lines
tools/perf/util/bpf_counter.c +28 -0
+
+#include "bpf_skel/bperf_cgroup.skel.h"
+
+static struct perf_event_attr cgrp_switch_attr = {
+     .type = PERF_TYPE_SOFTWARE,
+     .config = PERF_COUNT_SW_CGROUP_SWITCHES,
+     .size = sizeof(cgrp_switch_attr),
+     .sample_period = 1,
+     .disabled = 1,
+};
+
+static struct evsel *cgrp_switch;
+static struct xyarray *cgrp_prog_fds;
+static struct bperf_cgroup_bpf *skel;
+
+#define FD(evt, cpu) (*(int *)xyarray__entry(evt->core.fd, cpu, 0))
+#define PROG(cpu)    (*(int *)xyarray__entry(cgrp_prog_fds, cpu, 0))
+
+static void set_max_rlimit(void)
+{
+     struct rlimit rinf = { RLIM_INFINITY, RLIM_INFINITY };
+
+     setrlimit(RLIMIT_MEMLOCK, &rinf);
+}
+
+static __u32 bpf_link_get_prog_id(int fd)
+{
+     struct bpf_link_info link_info = {0};
Quoted 8 lines

tools/perf/util/bpf_counter.c

Quoted 5 lines
tools/perf/util/bpf_counter.c +5 -0
+     __u32 link_info_len = sizeof(link_info);
+
+     bpf_obj_get_info_by_fd(fd, &link_info, &link_info_len);
+     return link_info.prog_id;
+}

How about we move set_max_rlimit() and bpf_link_get_prog_id() to a header so we don't have to duplicate it?

Sounds good.

Quoted 28 lines
Quoted 28 lines
tools/perf/util/bpf_counter.c +27 -0
+
+static int bperf_load_program(struct evlist *evlist)
+{
+     struct bpf_link *link;
+     struct evsel *evsel;
+     struct cgroup *cgrp, *leader_cgrp;
+     __u32 i, cpu, prog_id;
+     int nr_cpus = evlist->core.all_cpus->nr;
+     int map_size, map_fd;
+     int prog_fd, err;
+
+     skel = bperf_cgroup_bpf__open();
+     if (!skel) {
+             pr_err("Failed to open cgroup skeleton\n");
+             return -1;
+     }
+
+     skel->rodata->num_cpus = nr_cpus;
+     skel->rodata->num_events = evlist->core.nr_entries / nr_cgroups;
+
+     BUG_ON(evlist->core.nr_entries % nr_cgroups != 0);
+
+     /* we need one copy of events per cpu for reading */
+     map_size = nr_cpus * evlist->core.nr_entries / nr_cgroups;
+     bpf_map__resize(skel->maps.events, map_size);
+     bpf_map__resize(skel->maps.cpu_idx, nr_cpus);
+     bpf_map__resize(skel->maps.cgrp_idx, nr_cgroups);
Quoted 8 lines

tools/perf/util/bpf_counter.c

Quoted 6 lines
tools/perf/util/bpf_counter.c +6 -0
+     /* previous result is saved in a per-cpu array */
+     map_size = evlist->core.nr_entries / nr_cgroups;
+     bpf_map__resize(skel->maps.prev_readings, map_size);
+     /* cgroup result needs all events */
+     map_size = nr_cpus * evlist->core.nr_entries;
+     bpf_map__resize(skel->maps.cgrp_readings, map_size);

We are setting map_size back and forth here.

But they are all different sizes.

Quoted 35 lines

[...]

Quoted 31 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +25 -0
new file mode 100644
index 000000000000..6d74e93dd1f5
@@ -0,0 +1,207 @@
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+// Copyright (c) 2021 Facebook
+// Copyright (c) 2021 Google
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include <bpf/bpf_core_read.h>
+
+#define MAX_LEVELS  10  // max cgroup hierarchy level: arbitrary
+#define MAX_EVENTS  32  // max events per cgroup: arbitrary
+
+// NOTE: many of map and global data will be modified before loading
+//       from the userspace (perf tool) using the skeleton helpers.
+
+// single set of global perf events to measure
+struct {
+     __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(int));
+     __uint(max_entries, 1);
+} events SEC(".maps");
+
+// from logical cpu number to event index
+// useful when user wants to count subset of cpus
+struct {
Quoted 8 lines

tools/perf/util/bpf_skel/bperf_cgroup.bpf.c line 30

Quoted 5 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +5 -0
+     __uint(type, BPF_MAP_TYPE_HASH);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(__u32));
+     __uint(max_entries, 1);
+} cpu_idx SEC(".maps");

How about we make cpu_idx a percpu array and use 0,1 for disable/enable profiling on this cpu?

No, it's to calculate an index to the cgrp_readings map which has the event x cpu x cgroup number of elements.

It controls enabling events with a global (bss) variable.

Quoted 19 lines
Quoted 19 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +18 -0
+
+// from cgroup id to event index
+struct {
+     __uint(type, BPF_MAP_TYPE_HASH);
+     __uint(key_size, sizeof(__u64));
+     __uint(value_size, sizeof(__u32));
+     __uint(max_entries, 1);
+} cgrp_idx SEC(".maps");
+
+// per-cpu event snapshots to calculate delta
+struct {
+     __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(struct bpf_perf_event_value));
+} prev_readings SEC(".maps");
+
+// aggregated event values for each cgroup
+// will be read from the user-space
Quoted 8 lines

tools/perf/util/bpf_skel/bperf_cgroup.bpf.c line 53

Quoted 5 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +5 -0
+struct {
+     __uint(type, BPF_MAP_TYPE_ARRAY);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(struct bpf_perf_event_value));
+} cgrp_readings SEC(".maps");

Maybe also make this a percpu array? This should make the BPF program faster.

Maybe. But I don't know how to access the elements in a per-cpu map from userspace.

Thanks,
Namhyung

Quoted 10 lines
Quoted 8 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +7 -0
+
+const volatile __u32 num_events = 1;
+const volatile __u32 num_cpus = 1;
+
+int enabled = 0;
+int use_cgroup_v2 = 0;
+

[...]

#6 lore raw

Re: [PATCH 3/3] perf stat: Enable BPF counter with --for-each-cgroup

Quoted 3 lines · Namhyung Kim

[...]

Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
tools/perf/util/bpf_counter_cgroup.c new file +11 -0
+
+     skel->rodata->num_cpus = nr_cpus;
+     skel->rodata->num_events = evlist->core.nr_entries / nr_cgroups;
+
+     BUG_ON(evlist->core.nr_entries % nr_cgroups != 0);
+
+     /* we need one copy of events per cpu for reading */
+     map_size = nr_cpus * evlist->core.nr_entries / nr_cgroups;
+     bpf_map__resize(skel->maps.events, map_size);
+     bpf_map__resize(skel->maps.cpu_idx, nr_cpus);
+     bpf_map__resize(skel->maps.cgrp_idx, nr_cgroups);
Quoted 8 lines

tools/perf/util/bpf_counter_cgroup.c line 96

Quoted 6 lines
tools/perf/util/bpf_counter_cgroup.c new file +6 -0
+     /* previous result is saved in a per-cpu array */
+     map_size = evlist->core.nr_entries / nr_cgroups;
+     bpf_map__resize(skel->maps.prev_readings, map_size);
+     /* cgroup result needs all events */
+     map_size = nr_cpus * evlist->core.nr_entries;
+     bpf_map__resize(skel->maps.cgrp_readings, map_size);

We are setting map_size back and forth here.

Quoted 1 line

But they are all different sizes.

Right. I misread the code.

Quoted 45 lines
Quoted 36 lines

[...]

Quoted 31 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +25 -0
new file mode 100644
index 000000000000..6d74e93dd1f5
@@ -0,0 +1,207 @@
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+// Copyright (c) 2021 Facebook
+// Copyright (c) 2021 Google
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include <bpf/bpf_core_read.h>
+
+#define MAX_LEVELS  10  // max cgroup hierarchy level: arbitrary
+#define MAX_EVENTS  32  // max events per cgroup: arbitrary
+
+// NOTE: many of map and global data will be modified before loading
+//       from the userspace (perf tool) using the skeleton helpers.
+
+// single set of global perf events to measure
+struct {
+     __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(int));
+     __uint(max_entries, 1);
+} events SEC(".maps");
+
+// from logical cpu number to event index
+// useful when user wants to count subset of cpus
+struct {
Quoted 8 lines

tools/perf/util/bpf_skel/bperf_cgroup.bpf.c line 30

Quoted 5 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +5 -0
+     __uint(type, BPF_MAP_TYPE_HASH);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(__u32));
+     __uint(max_entries, 1);
+} cpu_idx SEC(".maps");

How about we make cpu_idx a percpu array and use 0,1 for disable/enable profiling on this cpu?

Quoted 4 lines

No, it's to calculate an index to the cgrp_readings map which has the event x cpu x cgroup number of elements.

It controls enabling events with a global (bss) variable.

If we make cgrp_idx a per cpu array, we probably don't need the cpu_idx map?

Quoted 29 lines
Quoted 20 lines
Quoted 20 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +18 -0
+
+// from cgroup id to event index
+struct {
+     __uint(type, BPF_MAP_TYPE_HASH);
+     __uint(key_size, sizeof(__u64));
+     __uint(value_size, sizeof(__u32));
+     __uint(max_entries, 1);
+} cgrp_idx SEC(".maps");
+
+// per-cpu event snapshots to calculate delta
+struct {
+     __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(struct bpf_perf_event_value));
+} prev_readings SEC(".maps");
+
+// aggregated event values for each cgroup
+// will be read from the user-space
Quoted 8 lines

tools/perf/util/bpf_skel/bperf_cgroup.bpf.c line 53

Quoted 5 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +5 -0
+struct {
+     __uint(type, BPF_MAP_TYPE_ARRAY);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(struct bpf_perf_event_value));
+} cgrp_readings SEC(".maps");

Maybe also make this a percpu array? This should make the BPF program faster.

Quoted 2 lines

Maybe. But I don't know how to access the elements in a per-cpu map from userspace.

Please refer to bperf__read() reading accum_readings. Basically, we read one index of all CPUs with one bpf_map_lookup_elem().

Thanks,
Song

author maintainer
#7 lore raw

Re: [PATCH 3/3] perf stat: Enable BPF counter with --for-each-cgroup

Quoted 21 lines · Song Liu
Quoted 21 lines
Quoted 20 lines
Quoted 7 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +7 -0
+
+// single set of global perf events to measure
+struct {
+     __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(int));
+     __uint(max_entries, 1);

tools/perf/util/bpf_skel/bperf_cgroup.bpf.c line 30

Quoted 10 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +10 -0
+} events SEC(".maps");
+
+// from logical cpu number to event index
+// useful when user wants to count subset of cpus
+struct {
+     __uint(type, BPF_MAP_TYPE_HASH);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(__u32));
+     __uint(max_entries, 1);
+} cpu_idx SEC(".maps");

How about we make cpu_idx a percpu array and use 0,1 for disable/enable profiling on this cpu?

Quoted 7 lines
Quoted 4 lines

No, it's to calculate an index to the cgrp_readings map which has the event x cpu x cgroup number of elements.

It controls enabling events with a global (bss) variable.

If we make cgrp_idx a per cpu array, we probably don't need the cpu_idx map?

Right.

Quoted 30 lines
Quoted 30 lines
Quoted 29 lines
Quoted 16 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +13 -0
+
+// from cgroup id to event index
+struct {
+     __uint(type, BPF_MAP_TYPE_HASH);
+     __uint(key_size, sizeof(__u64));
+     __uint(value_size, sizeof(__u32));
+     __uint(max_entries, 1);
+} cgrp_idx SEC(".maps");
+
+// per-cpu event snapshots to calculate delta
+struct {
+     __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+     __uint(key_size, sizeof(__u32));

tools/perf/util/bpf_skel/bperf_cgroup.bpf.c line 53

Quoted 10 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +10 -0
+     __uint(value_size, sizeof(struct bpf_perf_event_value));
+} prev_readings SEC(".maps");
+
+// aggregated event values for each cgroup
+// will be read from the user-space
+struct {
+     __uint(type, BPF_MAP_TYPE_ARRAY);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(struct bpf_perf_event_value));
+} cgrp_readings SEC(".maps");

Maybe also make this a percpu array? This should make the BPF program faster.

Quoted 5 lines
Quoted 2 lines

Maybe. But I don't know how to access the elements in a per-cpu map from userspace.

Please refer to bperf__read() reading accum_readings. Basically, we read one index of all CPUs with one bpf_map_lookup_elem().

Thanks! So when I use a per-cpu array with 3 elements, I can access to cpu/elem entries in a row like below, right?

0/0, 0/1, 0/2, 1/0, 1/1, 1/2, 2/0, 2/1, 2/2, 3/0, ...

Thanks,
Namhyung

#8 lore raw

Re: [PATCH 3/3] perf stat: Enable BPF counter with --for-each-cgroup

Quoted 69 lines · Namhyung Kim
Quoted 21 lines · Song Liu
Quoted 21 lines
Quoted 20 lines
Quoted 7 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +7 -0
+
+// single set of global perf events to measure
+struct {
+     __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(int));
+     __uint(max_entries, 1);

tools/perf/util/bpf_skel/bperf_cgroup.bpf.c line 30

Quoted 10 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +10 -0
+} events SEC(".maps");
+
+// from logical cpu number to event index
+// useful when user wants to count subset of cpus
+struct {
+     __uint(type, BPF_MAP_TYPE_HASH);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(__u32));
+     __uint(max_entries, 1);
+} cpu_idx SEC(".maps");

How about we make cpu_idx a percpu array and use 0,1 for disable/enable profiling on this cpu?

Quoted 7 lines
Quoted 4 lines

No, it's to calculate an index to the cgrp_readings map which has the event x cpu x cgroup number of elements.

It controls enabling events with a global (bss) variable.

If we make cgrp_idx a per cpu array, we probably don't need the cpu_idx map?

Right.

Quoted 33 lines
Quoted 32 lines
Quoted 21 lines
Quoted 21 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +18 -0
+
+// from cgroup id to event index
+struct {
+     __uint(type, BPF_MAP_TYPE_HASH);
+     __uint(key_size, sizeof(__u64));
+     __uint(value_size, sizeof(__u32));
+     __uint(max_entries, 1);
+} cgrp_idx SEC(".maps");
+
+// per-cpu event snapshots to calculate delta
+struct {
+     __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(struct bpf_perf_event_value));
+} prev_readings SEC(".maps");
+
+// aggregated event values for each cgroup
+// will be read from the user-space
Quoted 8 lines

tools/perf/util/bpf_skel/bperf_cgroup.bpf.c line 53

Quoted 5 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +5 -0
+struct {
+     __uint(type, BPF_MAP_TYPE_ARRAY);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(struct bpf_perf_event_value));
+} cgrp_readings SEC(".maps");

Maybe also make this a percpu array? This should make the BPF program faster.

Maybe. But I don't know how to access the elements in a per-cpu map from userspace.

Quoted 7 lines
Quoted 2 lines

Please refer to bperf__read() reading accum_readings. Basically, we read one index of all CPUs with one bpf_map_lookup_elem().

Thanks! So when I use a per-cpu array with 3 elements, I can access to cpu/elem entries in a row like below, right?

0/0, 0/1, 0/2, 1/0, 1/1, 1/2, 2/0, 2/1, 2/2, 3/0, ...

I am not sure I am following here.

Say the system have 10 cpus, and the array has 3 elements. We can do:

	__u32 values[10];  /* assuming both key and value are __u32 */
	__u32 elem;
	int cpu;

	for (elem = 0; elem < 3; elem++) {
		bpf_map_lookup_elem(map_fd, &elem, values);
		for (cpu = 0; cpu < 10; cpu++)
			values[cpu] /* this is the value for cpu/elem */
	}

Thanks,
Song

author maintainer
#9 lore raw

Re: [PATCH 3/3] perf stat: Enable BPF counter with --for-each-cgroup

Quoted 30 lines · Song Liu
Quoted 30 lines · Namhyung Kim
Quoted 24 lines · Song Liu
Quoted 24 lines
Quoted 12 lines
Quoted 12 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +12 -0
+
+// single set of global perf events to measure
+struct {
+     __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(int));
+     __uint(max_entries, 1);
+} events SEC(".maps");
+
+// from logical cpu number to event index
+// useful when user wants to count subset of cpus
+struct {
Quoted 8 lines

tools/perf/util/bpf_skel/bperf_cgroup.bpf.c line 30

Quoted 5 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +5 -0
+     __uint(type, BPF_MAP_TYPE_HASH);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(__u32));
+     __uint(max_entries, 1);
+} cpu_idx SEC(".maps");

How about we make cpu_idx a percpu array and use 0,1 for disable/enable profiling on this cpu?

No, it's to calculate an index to the cgrp_readings map which has the event x cpu x cgroup number of elements.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

It controls enabling events with a global (bss) variable.

If we make cgrp_idx a per cpu array, we probably don't need the cpu_idx map?

Right.

Maybe not. Sometimes we want to profile a subset of cpus only. In that case, cpu != idx then I think we still need this.

Quoted 46 lines
Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 22 lines
Quoted 22 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +18 -0
+
+// from cgroup id to event index
+struct {
+     __uint(type, BPF_MAP_TYPE_HASH);
+     __uint(key_size, sizeof(__u64));
+     __uint(value_size, sizeof(__u32));
+     __uint(max_entries, 1);
+} cgrp_idx SEC(".maps");
+
+// per-cpu event snapshots to calculate delta
+struct {
+     __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(struct bpf_perf_event_value));
+} prev_readings SEC(".maps");
+
+// aggregated event values for each cgroup
+// will be read from the user-space
Quoted 8 lines

tools/perf/util/bpf_skel/bperf_cgroup.bpf.c line 53

Quoted 5 lines
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c new file +5 -0
+struct {
+     __uint(type, BPF_MAP_TYPE_ARRAY);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(struct bpf_perf_event_value));
+} cgrp_readings SEC(".maps");

Maybe also make this a percpu array? This should make the BPF program faster.

Maybe. But I don't know how to access the elements in a per-cpu map from userspace.

Quoted 7 lines
Quoted 2 lines

Please refer to bperf__read() reading accum_readings. Basically, we read one index of all CPUs with one bpf_map_lookup_elem().

Thanks! So when I use a per-cpu array with 3 elements, I can access to cpu/elem entries in a row like below, right?

0/0, 0/1, 0/2, 1/0, 1/1, 1/2, 2/0, 2/1, 2/2, 3/0, ...

I am not sure I am following here.

Say the system have 10 cpus, and the array has 3 elements. We can do:

Quoted 9 lines
        __u32 values[10];  /* assuming both key and value are __u32 */
        __u32 elem;
        int cpu;

        for (elem = 0; elem < 3; elem++) {
                bpf_map_lookup_elem(map_fd, &elem, values);
                for (cpu = 0; cpu < 10; cpu++)
                        values[cpu] /* this is the value for cpu/elem */
        }

Thanks for the explanation, I didn't think that way. I thought it like below:

    __u32 elem, value;

    for (elem = 0; elem < 3 * 10; elem++) {
        bpf_map_lookup_elem(map_fd, &elem, &value);
    }

So in this case, the actual value size is like below, right?

value-size = map-value-size * number-of-cpu

Thanks,
Namhyung

#10 lore raw

Re: [PATCH 3/3] perf stat: Enable BPF counter with --for-each-cgroup

Quoted 37 lines · Namhyung Kim
Quoted 32 lines · Song Liu
Quoted 32 lines · Namhyung Kim
Quoted 26 lines · Song Liu
Quoted 25 lines
Quoted 12 lines
Quoted 12 lines
(quoted diff) +12 -0
+
+// single set of global perf events to measure
+struct {
+     __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(int));
+     __uint(max_entries, 1);
+} events SEC(".maps");
+
+// from logical cpu number to event index
+// useful when user wants to count subset of cpus
+struct {
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+     __uint(type, BPF_MAP_TYPE_HASH);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(__u32));
+     __uint(max_entries, 1);
+} cpu_idx SEC(".maps");

How about we make cpu_idx a percpu array and use 0,1 for disable/enable profiling on this cpu?

No, it's to calculate an index to the cgrp_readings map which has the event x cpu x cgroup number of elements.

It controls enabling events with a global (bss) variable.

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

If we make cgrp_idx a per cpu array, we probably don't need the cpu_idx map?

Right.

Maybe not. Sometimes we want to profile a subset of cpus only. In that case, cpu != idx then I think we still need this.

We can only attach the bpf program on selected CPUs. Say, we want CPUs 1, 3, 5. We just do

	for (i in [1, 3, 5]) {
		link = bpf_program__attach_perf_event(skel->progs.on_switch,
						      FD(cgrp_switch, i));
		/* */
	}

The value arrays are still for all cpu, but they will just report zero for CPU 0, 2, 4, ....

Would this work?

[...]

Quoted 34 lines
Quoted 15 lines
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines

Maybe. But I don't know how to access the elements in a per-cpu map from userspace.

Please refer to bperf__read() reading accum_readings. Basically, we read one index of all CPUs with one bpf_map_lookup_elem().

Thanks! So when I use a per-cpu array with 3 elements, I can access to cpu/elem entries in a row like below, right?

0/0, 0/1, 0/2, 1/0, 1/1, 1/2, 2/0, 2/1, 2/2, 3/0, ...

I am not sure I am following here.

Say the system have 10 cpus, and the array has 3 elements. We can do:

Quoted 9 lines
       __u32 values[10];  /* assuming both key and value are __u32 */
       __u32 elem;
       int cpu;

       for (elem = 0; elem < 3; elem++) {
               bpf_map_lookup_elem(map_fd, &elem, values);
               for (cpu = 0; cpu < 10; cpu++)
                       values[cpu] /* this is the value for cpu/elem */
       }

Thanks for the explanation, I didn't think that way. I thought it like below:

   __u32 elem, value;

   for (elem = 0; elem < 3 * 10; elem++) {
       bpf_map_lookup_elem(map_fd, &elem, &value);
   }
Quoted 3 lines

So in this case, the actual value size is like below, right?

value-size = map-value-size * number-of-cpu

This is right (for user space).

Thanks,
Song

author maintainer
#11 lore raw

Re: [PATCH 3/3] perf stat: Enable BPF counter with --for-each-cgroup

Quoted 55 lines · Song Liu
Quoted 38 lines · Namhyung Kim
Quoted 32 lines · Song Liu
Quoted 32 lines · Namhyung Kim
Quoted 26 lines · Song Liu
Quoted 25 lines
Quoted 12 lines
Quoted 12 lines
(quoted diff) +12 -0
+
+// single set of global perf events to measure
+struct {
+     __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(int));
+     __uint(max_entries, 1);
+} events SEC(".maps");
+
+// from logical cpu number to event index
+// useful when user wants to count subset of cpus
+struct {
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+     __uint(type, BPF_MAP_TYPE_HASH);
+     __uint(key_size, sizeof(__u32));
+     __uint(value_size, sizeof(__u32));
+     __uint(max_entries, 1);
+} cpu_idx SEC(".maps");

How about we make cpu_idx a percpu array and use 0,1 for disable/enable profiling on this cpu?

No, it's to calculate an index to the cgrp_readings map which has the event x cpu x cgroup number of elements.

It controls enabling events with a global (bss) variable.

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

If we make cgrp_idx a per cpu array, we probably don't need the cpu_idx map?

Right.

Maybe not. Sometimes we want to profile a subset of cpus only. In that case, cpu != idx then I think we still need this.

We can only attach the bpf program on selected CPUs. Say, we want CPUs 1, 3, 5. We just do

        for (i in [1, 3, 5]) {
                link = bpf_program__attach_perf_event(skel->progs.on_switch,
                                                      FD(cgrp_switch, i));
                /* */
        }
Quoted 4 lines

The value arrays are still for all cpu, but they will just report zero for CPU 0, 2, 4, ....

Would this work?

Yeah, that's exactly what I do, and I'd like to have a compact map eliminating the unused entries (cpus). But now I think that I can keep it with a full cpus and just don't use them.

Quoted 35 lines
Quoted 35 lines
Quoted 16 lines
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines

Maybe. But I don't know how to access the elements in a per-cpu map from userspace.

Please refer to bperf__read() reading accum_readings. Basically, we read one index of all CPUs with one bpf_map_lookup_elem().

Thanks! So when I use a per-cpu array with 3 elements, I can access to cpu/elem entries in a row like below, right?

0/0, 0/1, 0/2, 1/0, 1/1, 1/2, 2/0, 2/1, 2/2, 3/0, ...

I am not sure I am following here.

Say the system have 10 cpus, and the array has 3 elements. We can do:

Quoted 9 lines
       __u32 values[10];  /* assuming both key and value are __u32 */
       __u32 elem;
       int cpu;

       for (elem = 0; elem < 3; elem++) {
               bpf_map_lookup_elem(map_fd, &elem, values);
               for (cpu = 0; cpu < 10; cpu++)
                       values[cpu] /* this is the value for cpu/elem */
       }

Thanks for the explanation, I didn't think that way. I thought it like below:

   __u32 elem, value;

   for (elem = 0; elem < 3 * 10; elem++) {
       bpf_map_lookup_elem(map_fd, &elem, &value);
   }
Quoted 5 lines
Quoted 3 lines

So in this case, the actual value size is like below, right?

value-size = map-value-size * number-of-cpu

This is right (for user space).

Thanks for your clarification!

Namhyung