perf: Improve cgroup profiling (v5)
Namhyung Kim
maintainercover perf: Improve cgroup profiling (v5)
Hello,
This work is to improve cgroup profiling in perf. Currently it only supports profiling tasks in a specific cgroup and there's no way to identify which cgroup the current sample belongs to. So I added PERF_SAMPLE_CGROUP to add cgroup id into each sample. It's a 64-bit integer having file handle of the cgroup. And kernel also generates PERF_RECORD_CGROUP event for new groups to correlate the cgroup id and cgroup name (path in the cgroup filesystem). The cgroup id can be read from userspace by name_to_handle_at() system call so it can synthesize the CGROUP event for existing groups.
So why do we want this? Systems running a large number of jobs in different cgroups want to profiling such jobs precisely. This includes container hosting systems widely used today. Currently perf supports namespace tracking but the systems may not use (cgroup) namespace for their jobs. Also it'd be more intuitive to see cgroup names (as they're given by user or sysadmin) rather than numeric cgroup/namespace id even if they use the namespaces.
From Stephane Eranian:
Quoted 6 lines
In data centers you care about attributing samples to a job not such much to a process. A job may have multiple processes which may come and go. The cgroup on the other hand stays around for the entire lifetime of the job. It is much easier to map a cgroup name to a particular job than it is to map a pid back to a job name, especially for offline post-processing.
Note that this only works for "perf_event" cgroups (obviously) so if users are still using cgroup-v1 interface, they need to have same hierarchy for subsystem(s) want to profile with it.
* Changes from v4:
- use CONFIG_CGROUP_PERF
- move cgroup tree to perf_env
- move cgroup fs utility function to tools/lib/api/fs
- use a local buffer and check its size for cgroup systhesis
* Changes from v3:
- staticize perf_event_cgroup()
- fix sample parsing test
* Changes from v2:
- remove path_len from cgroup_event
- bail out if kernel doesn't support cgroup sampling
- add some description in the Kconfig
* Changes from v1:
- use new cgroup id (= file handle)
The code is available at perf/cgroup-v5 branch in
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git
The testing result looks something like this:
[root@qemu build]# ./perf record --all-cgroups ./cgtest
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.009 MB perf.data (150 samples) ]
[root@qemu build]# ./perf report -s cgroup,comm --stdio
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 150 of event 'cpu-clock:pppH'
# Event count (approx.): 37500000
#
# Overhead cgroup Command
# ........ .......... .......
#
32.00% /sub/cgrp2 looper2
28.00% /sub/cgrp1 looper1
25.33% /sub looper0
4.00% / cgtest
4.00% /sub cgtest
3.33% /sub/cgrp2 cgtest
2.67% /sub/cgrp1 cgtest
0.67% / looper0The test program (cgtest) follows.
Thanks,
Namhyung
-------8<-----------------------------------------8<---------------- #define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <sched.h>
#include <errno.h>
#include <sys/stat.h>
#include <sys/wait.h>
#include <sys/prctl.h>
#include <sys/mount.h>
char cgbase[] = "/sys/fs/cgroup/perf_event";
void mkcgrp(char *name) {
char buf[256];
snprintf(buf, sizeof(buf), "%s%s", cgbase, name); if (mkdir(buf, 0755) < 0)
perror("mkdir");
}void rmcgrp(char *name) {
char buf[256];
snprintf(buf, sizeof(buf), "%s%s", cgbase, name); if (rmdir(buf) < 0)
perror("rmdir");
}void setcgrp(char *name) {
char buf[256];
int fd;
snprintf(buf, sizeof(buf), "%s%s/tasks", cgbase, name);
fd = open(buf, O_WRONLY);
if (fd > 0) {
if (write(fd, "0\n", 2) != 2)
perror("write");
close(fd); }
}
void create_sub_cgroup(int idx) {
char buf[128];
snprintf(buf, sizeof(buf), "/sub/cgrp%d", idx);
mkcgrp(buf);
}
void remove_sub_cgroup(int idx) {
char buf[128];
snprintf(buf, sizeof(buf), "/sub/cgrp%d", idx);
rmcgrp(buf);
}
void set_sub_cgroup(int idx) {
char buf[128];
snprintf(buf, sizeof(buf), "/sub/cgrp%d", idx);
setcgrp(buf);
}
void set_task_name(int idx) {
char buf[16];
snprintf(buf, sizeof(buf), "looper%d", idx);
prctl(PR_SET_NAME, buf, 0, 0, 0);
}
void loop(unsigned max) {
volatile unsigned int count = 1;
while (count++ != max) {
asm volatile ("pause"); }
}
void worker(int idx, unsigned cnt, int new_ns) {
int oldns;
create_sub_cgroup(idx);
set_sub_cgroup(idx);
if (new_ns) {
if (unshare(CLONE_NEWCGROUP) < 0)
perror("unshare");#if 0 /* FIXME */
if (unshare(CLONE_NEWNS) < 0)
perror("unshare");
if (mount("none", "/sys", NULL, MS_REMOUNT | MS_REC | MS_SLAVE, NULL) < 0)
perror("mount --make-rslave");
sleep(1);
if (umount("/sys/fs/cgroup/perf_event") < 0)
perror("umount");
if (mount("cgroup", "/sys/fs/cgroup/perf_event", "cgroup",
MS_NODEV | MS_NOEXEC | MS_NOSUID, "perf_event") < 0)
perror("mount again");#endif
}
if (fork() == 0) {
set_task_name(idx);
loop(cnt);
exit(0); }
wait(NULL);
}
int main(int argc, char *argv[])
{
int i, nr = 2;
int new_ns = 1;
unsigned cnt = 1000000;
int fd;
if (argc > 1)
nr = atoi(argv[1]);
if (argc > 2)
cnt = atoi(argv[2]);
if (argc > 3)
new_ns = atoi(argv[3]);
mkcgrp("/sub");
setcgrp("/sub");
for (i = 0; i < nr; i++) {
if (fork() == 0) {
worker(i+1, cnt, new_ns);
exit(0);
}
}
set_task_name(0);
loop(cnt);
for (i = 0; i < nr; i++)
wait(NULL);
for (i = 0; i < nr; i++)
remove_sub_cgroup(i+1);
setcgrp("/");
rmcgrp("/sub"); return 0;
} -------8<-----------------------------------------8<----------------
Namhyung Kim (10):
perf/core: Add PERF_RECORD_CGROUP event
perf/core: Add PERF_SAMPLE_CGROUP feature
tools lib api fs: Move cgroupsfs__mountpoint()
perf tools: Basic support for CGROUP event
perf tools: Maintain cgroup hierarchy
perf report: Add 'cgroup' sort key
perf record: Support synthesizing cgroup events
perf record: Add --all-cgroups option
perf top: Add --all-cgroups option
perf script: Add --show-cgroup-events option
include/linux/perf_event.h | 1 + include/uapi/linux/perf_event.h | 16 ++- init/Kconfig | 3 +- kernel/events/core.c | 133 ++++++++++++++++++++ tools/include/uapi/linux/perf_event.h | 16 ++- tools/lib/api/fs/Build | 1 + tools/lib/api/fs/cgroup.c | 67 ++++++++++ tools/lib/api/fs/fs.h | 2 + tools/lib/perf/include/perf/event.h | 7 ++ tools/perf/Documentation/perf-record.txt | 5 +- tools/perf/Documentation/perf-report.txt | 1 + tools/perf/Documentation/perf-script.txt | 3 + tools/perf/Documentation/perf-top.txt | 4 + tools/perf/builtin-diff.c | 1 + tools/perf/builtin-record.c | 10 ++ tools/perf/builtin-report.c | 1 + tools/perf/builtin-script.c | 41 +++++++ tools/perf/builtin-top.c | 9 ++ tools/perf/tests/sample-parsing.c | 6 +- tools/perf/util/cgroup.c | 143 +++++++++++++--------- tools/perf/util/cgroup.h | 18 ++- tools/perf/util/env.c | 2 + tools/perf/util/env.h | 6 + tools/perf/util/event.c | 19 +++ tools/perf/util/event.h | 6 + tools/perf/util/evsel.c | 17 ++- tools/perf/util/evsel.h | 1 + tools/perf/util/hist.c | 12 ++ tools/perf/util/hist.h | 1 + tools/perf/util/machine.c | 19 +++ tools/perf/util/machine.h | 3 + tools/perf/util/perf_event_attr_fprintf.c | 2 + tools/perf/util/record.h | 1 + tools/perf/util/session.c | 4 + tools/perf/util/sort.c | 37 ++++++ tools/perf/util/sort.h | 2 + tools/perf/util/synthetic-events.c | 121 ++++++++++++++++++ tools/perf/util/synthetic-events.h | 1 + tools/perf/util/tool.h | 2 + 39 files changed, 671 insertions(+), 73 deletions(-) create mode 100644 tools/lib/api/fs/cgroup.c
Signature · 1 line
2.25.0.265.gbab2e86ba0-goog
Namhyung Kim
maintainerPATCH 1/10 perf/core: Add PERF_RECORD_CGROUP event
To support cgroup tracking, add CGROUP event to save a link between cgroup path and id number. This is needed since cgroups can go away when userspace tries to read the cgroup info (from the id) later. The attr.cgroup bit was also added to enable cgroup tracking from userspace.
This event will be generated when a new cgroup becomes active. Userspace might need to synthesize those events for existing cgroups.
--- include/uapi/linux/perf_event.h | 13 +++- kernel/events/core.c | 111 ++++++++++++++++++++++++++++++++ 2 files changed, 123 insertions(+), 1 deletion(-)
include/uapi/linux/perf_event.h +12 -1
@@ -377,7 +377,8 @@ struct perf_event_attr { ksymbol : 1, /* include ksymbol events */ bpf_event : 1, /* include bpf events */ aux_output : 1, /* generate AUX records instead of events */ - __reserved_1 : 32; + cgroup : 1, /* include cgroup events */ + __reserved_1 : 31;union {__u32 wakeup_events;/* wakeup every n events */
@@ -1006,6 +1007,16 @@ enum perf_event_type {*/ PERF_RECORD_BPF_EVENT =18 , +/* +* struct { +* struct perf_event_header header; +* u64 id; +* char path[]; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_CGROUP =19 , + PERF_RECORD_MAX, /* non-ABI */ };
kernel/events/core.c +111 -0
147 lines not on this page · raw message
Signature · 1 line
2.25.0.265.gbab2e86ba0-goog
Namhyung Kim
maintainerPATCH 2/10 perf/core: Add PERF_SAMPLE_CGROUP feature
In reply to Namhyung KimThe PERF_SAMPLE_CGROUP bit is to save (perf_event) cgroup information in the sample. It will add a 64-bit id to identify current cgroup and it's the file handle in the cgroup file system. Userspace should use this information with PERF_RECORD_CGROUP event to match which cgroup it belongs.
I put it before PERF_SAMPLE_AUX for simplicity since it just needs a 64-bit word. But if we want bigger samples, I can work on that direction too.
--- include/linux/perf_event.h | 1 + include/uapi/linux/perf_event.h | 3 ++- init/Kconfig | 3 ++- kernel/events/core.c | 22 ++++++++++++++++++++++ 4 files changed, 27 insertions(+), 2 deletions(-)
include/linux/perf_event.h +1 -0
@@ -1001,6 +1001,7 @@ struct perf_sample_data {u64 stack_user_size;u64 phys_addr; +u64 cgroup; }____cacheline_aligned ;/* default value for data source */
include/uapi/linux/perf_event.h +2 -1
@@ -142,8 +142,9 @@ enum perf_event_sample_format {PERF_SAMPLE_REGS_INTR =1U <<18 ,PERF_SAMPLE_PHYS_ADDR =1U <<19 ,PERF_SAMPLE_AUX =1U <<20 , +PERF_SAMPLE_CGROUP =1U <<21 , -PERF_SAMPLE_MAX =1U <<21 ,/* non-ABI */ +PERF_SAMPLE_MAX =1U <<22 ,/* non-ABI */ __PERF_SAMPLE_CALLCHAIN_EARLY = 1ULL << 63, /* non-ABI; internal use */ };
init/Kconfig +2 -1
@@ -1026,7 +1026,8 @@ config CGROUP_PERF help This option extends the perf per-cpu mode to restrict monitoring to threads which belong to the cgroup specified and run on the - designated cpu. + designated cpu. Or this can be used to have cgroup ID in samples + so that it can monitor performance events among cgroups. Say N if unsure.
kernel/events/core.c +22 -0
@@ -1754,6 +1754,9 @@ static void __perf_event_header_size(struct perf_event *event, u64 sample_type)if (sample_type &PERF_SAMPLE_PHYS_ADDR ) size +=sizeof (data->phys_addr); +if (sample_type &PERF_SAMPLE_CGROUP ) + size +=sizeof (data->cgroup); + event->header_size = size; }
@@ -6699,6 +6702,9 @@ void perf_output_sample(struct perf_output_handle *handle,if (sample_type &PERF_SAMPLE_PHYS_ADDR )perf_output_put (handle, data->phys_addr); +if (sample_type &PERF_SAMPLE_CGROUP ) +perf_output_put (handle, data->cgroup); +if (sample_type &PERF_SAMPLE_AUX ) {perf_output_put (handle, data->aux_size);
@@ -6895,6 +6901,16 @@ void perf_prepare_sample(struct perf_event_header *header,if (sample_type &PERF_SAMPLE_PHYS_ADDR ) data->phys_addr =perf_virt_to_phys (data->addr); +#ifdef CONFIG_CGROUP_PERF +if (sample_type &PERF_SAMPLE_CGROUP ) { +struct cgroup *cgrp; + +/* protected by RCU */ + cgrp =task_css_check (current, perf_event_cgrp_id,1 )->cgroup; + data->cgroup =cgroup_id (cgrp); + } +#endif +if (sample_type &PERF_SAMPLE_AUX ) {u64 size;
@@ -11090,6 +11106,12 @@ static int perf_copy_attr(struct perf_event_attr __user *uattr,if (attr->sample_type &PERF_SAMPLE_REGS_INTR ) ret =perf_reg_validate (attr->sample_regs_intr); + +#ifndef CONFIG_CGROUP_PERF +if (attr->sample_type &PERF_SAMPLE_CGROUP ) +return -EINVAL ; +#endif + out:return ret;
Signature · 1 line
2.25.0.265.gbab2e86ba0-goog
Namhyung Kim
maintainerPATCH 3/10 tools lib api fs: Move cgroupsfs__mountpoint()
In reply to Namhyung KimMove it from tools/perf/util/cgroup.c as it can be used by other places. Note that cgroup filesystem is different from others since it's usually mounted separately (in v1) for each subsystem.
I just copied the code with a little modification to pass a name of subsystem and renamed it to follow other APIs.
--- tools/lib/api/fs/Build | 1 + tools/lib/api/fs/cgroup.c | 67 +++++++++++++++++++++++++++++++++++++++ tools/lib/api/fs/fs.h | 2 ++ tools/perf/util/cgroup.c | 63 ++---------------------------------- 4 files changed, 72 insertions(+), 61 deletions(-) create mode 100644 tools/lib/api/fs/cgroup.c
tools/lib/api/fs/Build +1 -0
@@ -1,2 +1,3 @@ libapi-y += fs.o libapi-y += tracing_path.o +libapi-y += cgroup.o
tools/lib/api/fs/cgroup.c new file +67 -0
67 lines not on this page · raw message
tools/lib/api/fs/fs.h +2 -0
@@ -28,6 +28,8 @@ FS(bpf_fs)#undef FS +int cgroupfs__mountpoint (char *buf,size_t maxlen,const char *subsys); +int filename__read_int (const char *filename,int *value);int filename__read_ull (const char *filename,unsigned long long *value);int filename__read_xll (const char *filename,unsigned long long *value);
tools/perf/util/cgroup.c +2 -61
84 lines not on this page · raw message
Signature · 1 line
2.25.0.265.gbab2e86ba0-goog
Namhyung Kim
maintainerPATCH 4/10 perf tools: Basic support for CGROUP event
In reply to Namhyung KimImplement basic functionality to support cgroup tracking. Each cgroup can be identified by inode number which can be read from userspace too. The actual cgroup processing will come in the later patch.
--- tools/include/uapi/linux/perf_event.h | 16 ++++++++++++++-- tools/lib/perf/include/perf/event.h | 7 +++++++ tools/perf/builtin-diff.c | 1 + tools/perf/builtin-report.c | 1 + tools/perf/tests/sample-parsing.c | 6 +++++- tools/perf/util/event.c | 18 ++++++++++++++++++ tools/perf/util/event.h | 6 ++++++ tools/perf/util/evsel.c | 6 ++++++ tools/perf/util/machine.c | 12 ++++++++++++ tools/perf/util/machine.h | 3 +++ tools/perf/util/perf_event_attr_fprintf.c | 2 ++ tools/perf/util/session.c | 4 ++++ tools/perf/util/synthetic-events.c | 8 ++++++++ tools/perf/util/tool.h | 1 + 14 files changed, 88 insertions(+), 3 deletions(-)
tools/include/uapi/linux/perf_event.h +14 -2
@@ -142,8 +142,9 @@ enum perf_event_sample_format {PERF_SAMPLE_REGS_INTR =1U <<18 ,PERF_SAMPLE_PHYS_ADDR =1U <<19 ,PERF_SAMPLE_AUX =1U <<20 , +PERF_SAMPLE_CGROUP =1U <<21 , -PERF_SAMPLE_MAX =1U <<21 ,/* non-ABI */ +PERF_SAMPLE_MAX =1U <<22 ,/* non-ABI */ __PERF_SAMPLE_CALLCHAIN_EARLY = 1ULL << 63, /* non-ABI; internal use */ };
@@ -377,7 +378,8 @@ struct perf_event_attr { ksymbol : 1, /* include ksymbol events */ bpf_event : 1, /* include bpf events */ aux_output : 1, /* generate AUX records instead of events */ - __reserved_1 : 32; + cgroup : 1, /* include cgroup events */ + __reserved_1 : 31;union {__u32 wakeup_events;/* wakeup every n events */
@@ -1006,6 +1008,16 @@ enum perf_event_type {*/ PERF_RECORD_BPF_EVENT =18 , +/* +* struct { +* struct perf_event_header header; +* u64 id; +* char path[]; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_CGROUP =19 , + PERF_RECORD_MAX, /* non-ABI */ };
tools/lib/perf/include/perf/event.h +7 -0
@@ -105,6 +105,12 @@ struct perf_record_bpf_event {__u8 tag[BPF_TAG_SIZE ];// prog tag }; +struct perf_record_cgroup { +struct perf_event_header header; +__u64 id; +char path[PATH_MAX ]; +}; +struct perf_record_sample {struct perf_event_header header;__u64 array[];
@@ -352,6 +358,7 @@ union perf_event {struct perf_record_mmap2 mmap2;struct perf_record_comm comm;struct perf_record_namespaces namespaces; +struct perf_record_cgroup cgroup;struct perf_record_fork fork;struct perf_record_lost lost;struct perf_record_lost_samples lost_samples;
tools/perf/builtin-diff.c +1 -0
@@ -455,6 +455,7 @@ static struct perf_diff pdiff = { .fork = perf_event__process_fork, .lost = perf_event__process_lost, .namespaces = perf_event__process_namespaces, + .cgroup = perf_event__process_cgroup, .ordered_events = true, .ordering_requires_timestamps = true, },
tools/perf/builtin-report.c +1 -0
@@ -1094,6 +1094,7 @@ int cmd_report(int argc, const char **argv) .mmap2 = perf_event__process_mmap2, .comm = perf_event__process_comm, .namespaces = perf_event__process_namespaces, + .cgroup = perf_event__process_cgroup, .exit = perf_event__process_exit, .fork = perf_event__process_fork, .lost = perf_event__process_lost,
tools/perf/tests/sample-parsing.c +5 -1
@@ -150,6 +150,9 @@ static bool samples_same(const struct perf_sample *s1,if (type &PERF_SAMPLE_PHYS_ADDR )COMP (phys_addr); +if (type &PERF_SAMPLE_CGROUP ) +COMP (cgroup); +if (type &PERF_SAMPLE_AUX ) {COMP (aux_sample.size); if (memcmp(s1->aux_sample.data, s2->aux_sample.data,
@@ -228,6 +231,7 @@ static int do_test(u64 sample_type, u64 sample_regs, u64 read_format) .regs = regs, }, .phys_addr = 113, + .cgroup = 114, .aux_sample = { .size = sizeof(aux_data), .data = (void *)aux_data,
@@ -331,7 +335,7 @@ int test__sample_parsing(struct test *test __maybe_unused, int subtest __maybe_u* were added. Please actually update the test rather than just change * the condition below. */ -if (PERF_SAMPLE_MAX >PERF_SAMPLE_AUX <<1 ) { +if (PERF_SAMPLE_MAX >PERF_SAMPLE_CGROUP <<1 ) {pr_debug ("sample format has changed, some new PERF_SAMPLE_ bit was introduced - test needs updating\n" );return -1 ; }
tools/perf/util/event.c +18 -0
@@ -54,6 +54,7 @@ static const char *perf_event__names[] = { [PERF_RECORD_NAMESPACES] = "NAMESPACES", [PERF_RECORD_KSYMBOL] = "KSYMBOL", [PERF_RECORD_BPF_EVENT] = "BPF_EVENT", + [PERF_RECORD_CGROUP] = "CGROUP", [PERF_RECORD_HEADER_ATTR] = "ATTR", [PERF_RECORD_HEADER_EVENT_TYPE] = "EVENT_TYPE", [PERF_RECORD_HEADER_TRACING_DATA] = "TRACING_DATA",
@@ -180,6 +181,12 @@ size_t perf_event__fprintf_namespaces(union perf_event *event, FILE *fp)return ret; } +size_t perf_event__fprintf_cgroup (union perf_event *event,FILE *fp) +{ +return fprintf (fp," cgroup: %" PRI_lu64" %s\n" , + event->cgroup.id, event->cgroup.path); +} + int perf_event__process_comm(struct perf_tool *tool __maybe_unused, union perf_event *event, struct perf_sample *sample,
@@ -196,6 +203,14 @@ int perf_event__process_namespaces(struct perf_tool *tool __maybe_unused,return machine__process_namespaces_event (machine, event, sample); } +int perf_event__process_cgroup (struct perf_tool *tool__maybe_unused , +union perf_event *event, +struct perf_sample *sample, +struct machine *machine) +{ +return machine__process_cgroup_event (machine, event, sample); +} + int perf_event__process_lost(struct perf_tool *tool __maybe_unused, union perf_event *event, struct perf_sample *sample,
@@ -417,6 +432,9 @@ size_t perf_event__fprintf(union perf_event *event, FILE *fp)case PERF_RECORD_NAMESPACES : ret +=perf_event__fprintf_namespaces (event, fp);break ; +case PERF_RECORD_CGROUP : + ret +=perf_event__fprintf_cgroup (event, fp); +break ;case PERF_RECORD_MMAP2 : ret +=perf_event__fprintf_mmap2 (event, fp);break ;
tools/perf/util/event.h +6 -0
@@ -135,6 +135,7 @@ struct perf_sample {u32 raw_size;u64 data_src;u64 phys_addr; +u64 cgroup;u32 flags;u16 insn_len;u8 cpumode;
@@ -321,6 +322,10 @@ int perf_event__process_namespaces(struct perf_tool *tool,union perf_event *event, struct perf_sample *sample, struct machine *machine); +int perf_event__process_cgroup (struct perf_tool *tool, +union perf_event *event, +struct perf_sample *sample, +struct machine *machine); int perf_event__process_mmap(struct perf_tool *tool, union perf_event *event, struct perf_sample *sample,
@@ -376,6 +381,7 @@ size_t perf_event__fprintf_switch(union perf_event *event, FILE *fp);size_t perf_event__fprintf_thread_map (union perf_event *event,FILE *fp);size_t perf_event__fprintf_cpu_map (union perf_event *event,FILE *fp);size_t perf_event__fprintf_namespaces (union perf_event *event,FILE *fp); +size_t perf_event__fprintf_cgroup (union perf_event *event,FILE *fp);size_t perf_event__fprintf_ksymbol (union perf_event *event,FILE *fp);size_t perf_event__fprintf_bpf (union perf_event *event,FILE *fp);size_t perf_event__fprintf (union perf_event *event,FILE *fp);
tools/perf/util/evsel.c +6 -0
@@ -2252,6 +2252,12 @@ int perf_evsel__parse_sample(struct evsel *evsel, union perf_event *event, array++; } + data->cgroup =0 ; +if (type &PERF_SAMPLE_CGROUP ) { + data->cgroup = *array; + array++; + } +if (type &PERF_SAMPLE_AUX ) {OVERFLOW_CHECK_u64 (array); sz = *array++;
tools/perf/util/machine.c +12 -0
@@ -654,6 +654,16 @@ int machine__process_namespaces_event(struct machine *machine __maybe_unused,return err; } +int machine__process_cgroup_event (struct machine *machine__maybe_unused , +union perf_event *event, +struct perf_sample *sample__maybe_unused ) +{ +if (dump_trace) +perf_event__fprintf_cgroup (event, stdout); + +return 0 ; +} +int machine__process_lost_event (struct machine *machine__maybe_unused ,union perf_event *event,struct perf_sample *sample__maybe_unused ) {
@@ -1880,6 +1890,8 @@ int machine__process_event(struct machine *machine, union perf_event *event, ret =machine__process_mmap_event (machine, event, sample);break ;case PERF_RECORD_NAMESPACES : ret =machine__process_namespaces_event (machine, event, sample);break ; +case PERF_RECORD_CGROUP : + ret =machine__process_cgroup_event (machine, event, sample);break ;case PERF_RECORD_MMAP2 : ret =machine__process_mmap2_event (machine, event, sample);break ;case PERF_RECORD_FORK :
tools/perf/util/machine.h +3 -0
@@ -128,6 +128,9 @@ int machine__process_switch_event(struct machine *machine,int machine__process_namespaces_event (struct machine *machine,union perf_event *event,struct perf_sample *sample); +int machine__process_cgroup_event (struct machine *machine, +union perf_event *event, +struct perf_sample *sample);int machine__process_mmap_event (struct machine *machine,union perf_event *event,struct perf_sample *sample); int machine__process_mmap2_event(struct machine *machine, union perf_event *event,
tools/perf/util/perf_event_attr_fprintf.c +2 -0
@@ -35,6 +35,7 @@ static void __p_sample_type(char *buf, size_t size, u64 value)bit_name (BRANCH_STACK ),bit_name (REGS_USER ),bit_name (STACK_USER ),bit_name (IDENTIFIER ),bit_name (REGS_INTR ),bit_name (DATA_SRC ),bit_name (WEIGHT ),bit_name (PHYS_ADDR ),bit_name (AUX ), +bit_name (CGROUP ), { .name = NULL, } };#undef bit_name
@@ -131,6 +132,7 @@ int perf_event_attr__fprintf(FILE *fp, struct perf_event_attr *attr,PRINT_ATTRf (ksymbol, p_unsigned);PRINT_ATTRf (bpf_event, p_unsigned);PRINT_ATTRf (aux_output, p_unsigned); +PRINT_ATTRf (cgroup, p_unsigned);PRINT_ATTRn ("{ wakeup_events, wakeup_watermark }" , wakeup_events, p_unsigned);PRINT_ATTRf (bp_type, p_unsigned);
tools/perf/util/session.c +4 -0
@@ -471,6 +471,8 @@ void perf_tool__fill_defaults(struct perf_tool *tool) tool->comm = process_event_stub;if (tool->namespaces ==NULL ) tool->namespaces = process_event_stub; +if (tool->cgroup ==NULL ) + tool->cgroup = process_event_stub;if (tool->fork ==NULL ) tool->fork = process_event_stub;if (tool->exit ==NULL )
@@ -1434,6 +1436,8 @@ static int machines__deliver_event(struct machines *machines,return tool->comm (tool, event, sample, machine);case PERF_RECORD_NAMESPACES :return tool->namespaces (tool, event, sample, machine); +case PERF_RECORD_CGROUP : +return tool->cgroup (tool, event, sample, machine);case PERF_RECORD_FORK :return tool->fork (tool, event, sample, machine);case PERF_RECORD_EXIT :
tools/perf/util/synthetic-events.c +8 -0
@@ -1228,6 +1228,9 @@ size_t perf_event__sample_event_size(const struct perf_sample *sample, u64 type,if (type &PERF_SAMPLE_PHYS_ADDR ) result +=sizeof (u64); +if (type &PERF_SAMPLE_CGROUP ) + result +=sizeof (u64); +if (type &PERF_SAMPLE_AUX ) { result +=sizeof (u64); result += sample->aux_sample.size;
@@ -1401,6 +1404,11 @@ int perf_event__synthesize_sample(union perf_event *event, u64 type, u64 read_fo array++; } +if (type &PERF_SAMPLE_CGROUP ) { + *array = sample->cgroup; + array++; + } +if (type &PERF_SAMPLE_AUX ) { sz = sample->aux_sample.size; *array++ = sz;
tools/perf/util/tool.h +1 -0
@@ -46,6 +46,7 @@ struct perf_tool { mmap2, comm, namespaces, + cgroup, fork, exit, lost,
Signature · 1 line
2.25.0.265.gbab2e86ba0-goog
Namhyung Kim
maintainerPATCH 5/10 perf tools: Maintain cgroup hierarchy
In reply to Namhyung KimEach cgroup is kept in the perf_env's cgroup_tree sorted by the cgroup id. Hist entries have cgroup id can compare it directly and later it can be used to find a group name using this tree.
--- tools/perf/util/cgroup.c | 80 +++++++++++++++++++++++++++++++++++++++ tools/perf/util/cgroup.h | 17 +++++++-- tools/perf/util/env.c | 2 + tools/perf/util/env.h | 6 +++ tools/perf/util/machine.c | 9 ++++- 5 files changed, 109 insertions(+), 5 deletions(-)
tools/perf/util/cgroup.c +80 -0
@@ -191,3 +191,83 @@ int parse_cgroups(const struct option *opt, const char *str, }return 0 ; } + +static struct cgroup *__cgroup__findnew (struct rb_root *root,uint64_t id, +bool create,const char *path) +{ +struct rb_node **p = &root->rb_node; +struct rb_node *parent =NULL ; +struct cgroup *cgrp; + +while (*p !=NULL ) { + parent = *p; + cgrp = rb_entry(parent, struct cgroup, node); + +if (cgrp->id == id) +return cgrp; + +if (cgrp->id < id) + p = &(*p)->rb_left; +else + p = &(*p)->rb_right; + } + +if (!create) +return NULL ; + + cgrp =malloc (sizeof (*cgrp)); +if (cgrp ==NULL ) +return NULL ; + + cgrp->name =strdup (path); +if (cgrp->name ==NULL ) { +free (cgrp); +return NULL ; + } + + cgrp->fd =-1 ; + cgrp->id = id; +refcount_set (&cgrp->refcnt,1 ); + +rb_link_node (&cgrp->node, parent, p); +rb_insert_color (&cgrp->node, root); + +return cgrp; +} + +struct cgroup *cgroup__findnew (struct perf_env *env,uint64_t id, +const char *path) +{ +struct cgroup *cgrp; + +down_write (&env->cgroups.lock); + cgrp =__cgroup__findnew (&env->cgroups.tree, id, true, path); +up_write (&env->cgroups.lock); +return cgrp; +} + +struct cgroup *cgroup__find (struct perf_env *env,uint64_t id) +{ +struct cgroup *cgrp; + +down_read (&env->cgroups.lock); + cgrp =__cgroup__findnew (&env->cgroups.tree, id, false,NULL ); +up_read (&env->cgroups.lock); +return cgrp; +} + +void perf_env__purge_cgroups (struct perf_env *env) +{ +struct rb_node *node; +struct cgroup *cgrp; + +down_write (&env->cgroups.lock); +while (!RB_EMPTY_ROOT (&env->cgroups.tree)) { + node =rb_first (&env->cgroups.tree); + cgrp = rb_entry(node, struct cgroup, node); + +rb_erase (node, &env->cgroups.tree); +cgroup__put (cgrp); + } +up_write (&env->cgroups.lock); +}
tools/perf/util/cgroup.h +13 -4
@@ -3,16 +3,19 @@#define __CGROUP_H__#include <linux/refcount.h> +#include <linux/rbtree.h> +#include "util/env.h" struct option ;struct cgroup { -char *name; -int fd; -refcount_t refcnt; +struct rb_node node; +u64 id; +char *name; +int fd; +refcount_t refcnt; }; -extern int nr_cgroups;/* number of explicit cgroups defined */ struct cgroup *cgroup__get (struct cgroup *cgroup);
@@ -26,4 +29,10 @@ void evlist__set_default_cgroup(struct evlist *evlist, struct cgroup *cgroup);int parse_cgroups (const struct option *opt,const char *str,int unset); +struct cgroup *cgroup__findnew (struct perf_env *env,uint64_t id, +const char *path); +struct cgroup *cgroup__find (struct perf_env *env,uint64_t id); + +void perf_env__purge_cgroups (struct perf_env *env); +#endif /* __CGROUP_H__ */
tools/perf/util/env.c +2 -0
@@ -6,6 +6,7 @@#include <linux/ctype.h> #include <linux/zalloc.h> #include "bpf-event.h" +#include "cgroup.h" #include <errno.h> #include <sys/utsname.h> #include <bpf/libbpf.h>
@@ -168,6 +169,7 @@ void perf_env__exit(struct perf_env *env)int i;perf_env__purge_bpf (env); +perf_env__purge_cgroups (env);zfree (&env->hostname);zfree (&env->os_release);zfree (&env->version);
tools/perf/util/env.h +6 -0
@@ -88,6 +88,12 @@ struct perf_env {u32 btfs_cnt; } bpf_progs; +/* same reason as above (for perf-top) */ +struct { +struct rw_semaphore lock; +struct rb_root tree; + } cgroups; +/* For fast cpu to numa node lookup via perf_env__numa_node */ int *numa_map;int nr_numa_map;
tools/perf/util/machine.c +8 -1
@@ -33,6 +33,7 @@#include "asm/bug.h" #include "bpf-event.h" #include <internal/lib.h> // page_size +#include "cgroup.h" #include <linux/ctype.h> #include <symbol/kallsyms.h>
@@ -654,13 +655,19 @@ int machine__process_namespaces_event(struct machine *machine __maybe_unused,return err; } -int machine__process_cgroup_event (struct machine *machine__maybe_unused , +int machine__process_cgroup_event (struct machine *machine,union perf_event *event,struct perf_sample *sample__maybe_unused ) { +struct cgroup *cgrp; +if (dump_trace)perf_event__fprintf_cgroup (event, stdout); + cgrp =cgroup__findnew (machine->env, event->cgroup.id, event->cgroup.path); +if (cgrp ==NULL ) +return -ENOMEM ; +return 0 ; }
Signature · 1 line
2.25.0.265.gbab2e86ba0-goog
Namhyung Kim
maintainerPATCH 6/10 perf report: Add 'cgroup' sort key
In reply to Namhyung KimThe cgroup sort key is to show cgroup membership of each task. Currently it shows full path in the cgroupfs (not relative to the root of cgroup namespace) since it'd be more intuitive IMHO. Otherwise root cgroup in different namespaces will all show same name - "/".
The cgroup sort key should come before cgroup_id otherwise sort_dimension__add() will match it to cgroup_id as it only matches with the given substring.
For example it will look like following. Note that record patch adding --all-cgroups patch will come later.
$ perf record -a --namespace --all-cgroups cgtest [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.208 MB perf.data (4090 samples) ]
$ perf report -s cgroup_id,cgroup,pid
...
# Overhead cgroup id (dev/inode) cgroup Pid:Command
# ........ ..................... .......... ...............
#
93.96% 0/0x0 / 0:swapper
1.25% 3/0xeffffffb / 278:looper0
0.86% 3/0xf000015f /sub/cgrp1 280:cgtest
0.37% 3/0xf0000160 /sub/cgrp2 281:cgtest
0.34% 3/0xf0000163 /sub/cgrp3 282:cgtest
0.22% 3/0xeffffffb /sub 278:looper0
0.20% 3/0xeffffffb / 280:cgtest
0.15% 3/0xf0000163 /sub/cgrp3 285:looper3--- tools/perf/Documentation/perf-report.txt | 1 + tools/perf/util/hist.c | 12 ++++++++ tools/perf/util/hist.h | 1 + tools/perf/util/sort.c | 37 ++++++++++++++++++++++++ tools/perf/util/sort.h | 2 ++ 5 files changed, 53 insertions(+)
tools/perf/Documentation/perf-report.txt +1 -0
@@ -95,6 +95,7 @@ OPTIONS abort cost. This is the global weight. - local_weight: Local weight version of the weight above. - cgroup_id: ID derived from cgroup namespace device and inode numbers. + - cgroup: cgroup pathname in the cgroupfs. - transaction: Transaction abort flags. - overhead: Overhead percentage of sample - overhead_sys: Overhead percentage of sample running in system mode
tools/perf/util/hist.c +12 -0
@@ -10,6 +10,7 @@#include "mem-events.h" #include "session.h" #include "namespaces.h" +#include "cgroup.h" #include "sort.h" #include "units.h" #include "evlist.h"
@@ -222,6 +223,16 @@ void hists__calc_col_len(struct hists *hists, struct hist_entry *h)if (h->trace_output)hists__new_col_len (hists,HISTC_TRACE ,strlen (h->trace_output)); + +if (h->cgroup) { +const char *cgrp_name ="unknown" ; +struct cgroup *cgrp =cgroup__find (h->ms.maps->machine->env, + h->cgroup); +if (cgrp !=NULL ) + cgrp_name = cgrp->name; + +hists__new_col_len (hists,HISTC_CGROUP ,strlen (cgrp_name)); + } }void hists__output_recalc_col_len (struct hists *hists,int max_rows)
@@ -691,6 +702,7 @@ __hists__add_entry(struct hists *hists, .dev = ns ? ns->link_info[CGROUP_NS_INDEX].dev : 0, .ino = ns ? ns->link_info[CGROUP_NS_INDEX].ino : 0, }, + .cgroup = sample->cgroup, .ms = { .maps = al->maps, .map = al->map,
tools/perf/util/hist.h +1 -0
@@ -38,6 +38,7 @@ enum hist_column {HISTC_THREAD ,HISTC_COMM ,HISTC_CGROUP_ID , +HISTC_CGROUP ,HISTC_PARENT ,HISTC_CPU , HISTC_SOCKET,
tools/perf/util/sort.c +37 -0
@@ -12,6 +12,7 @@#include "cacheline.h" #include "comm.h" #include "map.h" +#include "maps.h" #include "symbol.h" #include "map_symbol.h" #include "branch.h"
@@ -25,6 +26,8 @@#include "mem-events.h" #include "annotate.h" #include "time-utils.h" +#include "cgroup.h" +#include "machine.h" #include <linux/kernel.h> #include <linux/string.h>
@@ -634,6 +637,39 @@ struct sort_entry sort_cgroup_id = { .se_width_idx = HISTC_CGROUP_ID, }; +/* --sort cgroup */ + +static int64_t +sort__cgroup_cmp (struct hist_entry *left,struct hist_entry *right) +{ +return right->cgroup - left->cgroup; +} + +static int hist_entry__cgroup_snprintf (struct hist_entry *he, +char *bf,size_t size, +unsigned int width__maybe_unused ) +{ +const char *cgrp_name ="N/A" ; + +if (he->cgroup) { +struct cgroup *cgrp =cgroup__find (he->ms.maps->machine->env, + he->cgroup); +if (cgrp !=NULL ) + cgrp_name = cgrp->name; +else + cgrp_name ="unknown" ; + } + +return repsep_snprintf (bf, size,"%s" , cgrp_name); +} + +struct sort_entry sort_cgroup = { + .se_header ="cgroup" , + .se_cmp = sort__cgroup_cmp, + .se_snprintf = hist_entry__cgroup_snprintf, + .se_width_idx =HISTC_CGROUP , +}; +/* --sort socket */ static int64_t
@@ -1658,6 +1694,7 @@ static struct sort_dimension common_sort_dimensions[] = {DIM (SORT_TRACE ,"trace" , sort_trace),DIM (SORT_SYM_SIZE ,"symbol_size" , sort_sym_size),DIM (SORT_DSO_SIZE ,"dso_size" , sort_dso_size), +DIM (SORT_CGROUP ,"cgroup" , sort_cgroup),DIM (SORT_CGROUP_ID ,"cgroup_id" , sort_cgroup_id),DIM (SORT_SYM_IPC_NULL ,"ipc_null" , sort_sym_ipc_null), DIM(SORT_TIME, "time", sort_time),
tools/perf/util/sort.h +2 -0
@@ -101,6 +101,7 @@ struct hist_entry {struct thread *thread;struct comm *comm;struct namespace_id cgroup_id; +u64 cgroup;u64 ip;u64 transaction;s32 socket;
@@ -224,6 +225,7 @@ enum sort_type {SORT_TRACE ,SORT_SYM_SIZE ,SORT_DSO_SIZE , +SORT_CGROUP ,SORT_CGROUP_ID ,SORT_SYM_IPC_NULL , SORT_TIME,
Signature · 1 line
2.25.0.265.gbab2e86ba0-goog
Re: [PATCH 06/10] perf report: Add 'cgroup' sort key
On Mon, Feb 24, 2020 at 01:37:45PM +0900, Namhyung Kim wrote:
SNIP
Quoted 10 lines
tools/perf/util/sort.c +6
-0@@ -634,6 +637,39 @@ struct sort_entry sort_cgroup_id = { .se_width_idx = HISTC_CGROUP_ID, }; +/* --sort cgroup */ + +static int64_t +sort__cgroup_cmp (struct hist_entry *left,struct hist_entry *right) +{ +return right->cgroup - left->cgroup;
tools/perf/util/sort.c struct sort_entry sort_cgroup_id = { · line 655
Quoted 10 lines
tools/perf/util/sort.c +10
-0+} + +static int hist_entry__cgroup_snprintf (struct hist_entry *he, +char *bf,size_t size, +unsigned int width__maybe_unused ) +{ +const char *cgrp_name ="N/A" ; + +if (he->cgroup) { + struct cgroup *cgrp = cgroup__find(he->ms.maps->machine->env,
eveything is connected :)) great that this one works
jirka
Namhyung Kim
maintainerPATCH 7/10 perf record: Support synthesizing cgroup events
In reply to Namhyung KimSynthesize cgroup events by iterating cgroup filesystem directories. The cgroup event only saves the portion of cgroup path after the mount point and the cgroup id (which actually is a file handle).
--- tools/perf/builtin-record.c | 5 ++ tools/perf/util/cgroup.h | 1 + tools/perf/util/event.c | 1 + tools/perf/util/synthetic-events.c | 113 +++++++++++++++++++++++++++++ tools/perf/util/synthetic-events.h | 1 + tools/perf/util/tool.h | 1 + 6 files changed, 122 insertions(+)
tools/perf/builtin-record.c +5 -0
@@ -1397,6 +1397,11 @@ static int record__synthesize(struct record *rec, bool tail)if (err <0 )pr_warning ("Couldn't synthesize bpf events.\n" ); + err =perf_event__synthesize_cgroups (tool, process_synthesized_event, + machine); +if (err <0 ) +pr_warning ("Couldn't synthesize cgroup events.\n" ); + err =__machine__synthesize_threads (machine, tool, &opts->target, rec->evlist->core.threads, process_synthesized_event, opts->sample_address,1 );
tools/perf/util/cgroup.h +1 -0
@@ -18,6 +18,7 @@ struct cgroup {extern int nr_cgroups;/* number of explicit cgroups defined */ +int cgroupfs_find_mountpoint (char *buf,size_t maxlen);struct cgroup *cgroup__get (struct cgroup *cgroup);void cgroup__put (struct cgroup *cgroup);
tools/perf/util/event.c +1 -0
@@ -33,6 +33,7 @@#include "bpf-event.h" #include "tool.h" #include "../perf.h" +#include "cgroup.h" static const char *perf_event__names[] = { [0 ] ="TOTAL" ,
tools/perf/util/synthetic-events.c +113 -0
125 lines not on this page · raw message
tools/perf/util/synthetic-events.h +1 -0
@@ -45,6 +45,7 @@ int perf_event__synthesize_kernel_mmap(struct perf_tool *tool, perf_event__handlint perf_event__synthesize_mmap_events (struct perf_tool *tool,union perf_event *event,pid_t pid,pid_t tgid,perf_event__handler_t process,struct machine *machine,bool mmap_data);int perf_event__synthesize_modules (struct perf_tool *tool,perf_event__handler_t process,struct machine *machine);int perf_event__synthesize_namespaces (struct perf_tool *tool,union perf_event *event,pid_t pid,pid_t tgid,perf_event__handler_t process,struct machine *machine); +int perf_event__synthesize_cgroups (struct perf_tool *tool,perf_event__handler_t process,struct machine *machine);int perf_event__synthesize_sample (union perf_event *event,u64 type,u64 read_format,const struct perf_sample *sample);int perf_event__synthesize_stat_config (struct perf_tool *tool,struct perf_stat_config *config,perf_event__handler_t process,struct machine *machine);int perf_event__synthesize_stat_events (struct perf_stat_config *config,struct perf_tool *tool,struct evlist *evlist,perf_event__handler_t process,bool attrs);
tools/perf/util/tool.h +1 -0
@@ -79,6 +79,7 @@ struct perf_tool {bool ordered_events;bool ordering_requires_timestamps;bool namespace_events; +bool cgroup_events;bool no_warn;enum show_feature_header show_feat_hdr; };
Signature · 1 line
2.25.0.265.gbab2e86ba0-goog
Re: [PATCH 07/10] perf record: Support synthesizing cgroup events
On Mon, Feb 24, 2020 at 01:37:46PM +0900, Namhyung Kim wrote:
SNIP
Quoted 39 lines
tools/perf/util/synthetic-events.c +39
-0+ d =opendir (path); +if (d ==NULL ) { +pr_debug ("failed to open directory: %s\n" , path); +return -1 ; + } + +while ((dent =readdir (d)) !=NULL ) { +if (dent->d_type !=DT_DIR ) +continue ; +if (!strcmp (dent->d_name,"." ) || + !strcmp (dent->d_name,".." )) +continue ; + +/* any sane path should be less than PATH_MAX */ +if (strlen (path) +strlen (dent->d_name) +1 >=PATH_MAX ) +continue ; + +if (path[pos -1 ] !='/' ) +strcat (path,"/" ); +strcat (path, dent->d_name); + + ret =perf_event__walk_cgroup_tree (tool, event, path, + mount_len, process, machine); +if (ret <0 ) +break ; + + path[pos] ='\0' ; + } + +closedir (d); +return ret; +} + +int perf_event__synthesize_cgroups (struct perf_tool *tool, +perf_event__handler_t process, +struct machine *machine) +{ +union perf_event event; +char cgrp_root[PATH_MAX ];
tools/perf/util/synthetic-events.c int perf_event__synthesize_mmap_events(struct perf_tool *tool, · line 520
Quoted 10 lines
tools/perf/util/synthetic-events.c +10
-0+size_t mount_len;/* length of mount point in the path */ + +if (cgroupfs__mountpoint (cgrp_root,PATH_MAX ,"perf_event" ) <0 ) { +pr_debug ("cannot find cgroup mount point\n" ); +return -1 ; + } + + mount_len =strlen (cgrp_root); +/* make sure the path starts with a slash (after mount point) */ +strcat (cgrp_root,"/" );
the code above checks on this and seems to add '/' if needed, is this strcat necessary?
jirka
Re: [PATCH 07/10] perf record: Support synthesizing cgroup events
Quoted 48 lines · Jiri Olsa
On Mon, Feb 24, 2020 at 01:37:46PM +0900, Namhyung Kim wrote:
SNIP
Quoted 44 lines
tools/perf/util/synthetic-events.c +44
-0+ d =opendir (path); +if (d ==NULL ) { +pr_debug ("failed to open directory: %s\n" , path); +return -1 ; + } + +while ((dent =readdir (d)) !=NULL ) { +if (dent->d_type !=DT_DIR ) +continue ; +if (!strcmp (dent->d_name,"." ) || + !strcmp (dent->d_name,".." )) +continue ; + +/* any sane path should be less than PATH_MAX */ +if (strlen (path) +strlen (dent->d_name) +1 >=PATH_MAX ) +continue ; + +if (path[pos -1 ] !='/' ) +strcat (path,"/" ); +strcat (path, dent->d_name); + + ret =perf_event__walk_cgroup_tree (tool, event, path, + mount_len, process, machine); +if (ret <0 ) +break ; + + path[pos] ='\0' ; + } + +closedir (d); +return ret; +} + +int perf_event__synthesize_cgroups (struct perf_tool *tool, +perf_event__handler_t process, +struct machine *machine) +{ +union perf_event event; +char cgrp_root[PATH_MAX ]; +size_t mount_len;/* length of mount point in the path */ + +if (cgroupfs__mountpoint (cgrp_root,PATH_MAX ,"perf_event" ) <0 ) { +pr_debug ("cannot find cgroup mount point\n" ); +return -1 ;
Quoted 8 lines
tools/perf/util/synthetic-events.c int perf_event__synthesize_mmap_events(struct perf_tool *tool, · line 520
Quoted 5 lines
tools/perf/util/synthetic-events.c +5
-0+ } + + mount_len =strlen (cgrp_root); +/* make sure the path starts with a slash (after mount point) */ +strcat (cgrp_root,"/" );the code above checks on this and seems to add '/' if needed, is this strcat necessary?
nah nevermind.. it's the recursive call check above, ok
jirka
Namhyung Kim
maintainerPATCH 8/10 perf record: Add --all-cgroups option
In reply to Namhyung KimThe --all-cgroups option is to enable cgroup profiling support. It tells kernel to record CGROUP events in the ring buffer so that perf report can identify task/cgroup association later.
--- tools/perf/Documentation/perf-record.txt | 5 ++++- tools/perf/builtin-record.c | 5 +++++ tools/perf/util/evsel.c | 11 ++++++++++- tools/perf/util/evsel.h | 1 + tools/perf/util/record.h | 1 + 5 files changed, 21 insertions(+), 2 deletions(-)
tools/perf/Documentation/perf-record.txt +4 -1
@@ -385,7 +385,10 @@ displayed with the weight and local_weight sort keys. This currently works for abort events and some memory events in precise mode on modern Intel CPUs. --namespaces:: -Record events of type PERF_RECORD_NAMESPACES. +Record events of type PERF_RECORD_NAMESPACES. This enables 'cgroup_id' sort key. + +--all-cgroups:: +Record events of type PERF_RECORD_CGROUP. This enables 'cgroup' sort key. --transaction:: Record transaction flags for transaction related events.
tools/perf/builtin-record.c +5 -0
@@ -1433,6 +1433,9 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)if (rec->opts.record_namespaces) tool->namespace_events = true; +if (rec->opts.record_cgroup) + tool->cgroup_events = true; +if (rec->opts.auxtrace_snapshot_mode || rec->switch_output.enabled) {signal (SIGUSR2 , snapshot_sig_handler);if (rec->opts.auxtrace_snapshot_mode)
@@ -2363,6 +2366,8 @@ static struct option __record_options[] = { "per thread proc mmap processing timeout in ms"),OPT_BOOLEAN (0 ,"namespaces" , &record.opts.record_namespaces,"Record namespaces events" ), +OPT_BOOLEAN (0 ,"all-cgroups" , &record.opts.record_cgroup, +"Record cgroup events" ),OPT_BOOLEAN (0 ,"switch-events" , &record.opts.record_switch_events,"Record context switch events" ), OPT_BOOLEAN_FLAG(0, "all-kernel", &record.opts.all_kernel,
tools/perf/util/evsel.c +10 -1
@@ -1102,6 +1102,11 @@ void perf_evsel__config(struct evsel *evsel, struct record_opts *opts,if (opts->record_namespaces) attr->namespaces = track; +if (opts->record_cgroup) { + attr->cgroup = track && !perf_missing_features.cgroup; +perf_evsel__set_sample_bit (evsel,CGROUP ); + } +if (opts->record_switch_events) attr->context_switch = track;
@@ -1784,7 +1789,11 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,* Must probe features in the order they were added to the * perf_event_attr interface. */ -if (!perf_missing_features.aux_output && evsel->core.attr.aux_output) { +if (!perf_missing_features.cgroup && evsel->core.attr.cgroup) { + perf_missing_features.cgroup = true; +pr_debug2_peo ("Kernel has no cgroup sampling support, bailing out\n" ); + goto out_close; + }else if (!perf_missing_features.aux_output && evsel->core.attr.aux_output) { perf_missing_features.aux_output = true;pr_debug2_peo ("Kernel has no attr.aux_output support, bailing out\n" ); goto out_close;
tools/perf/util/evsel.h +1 -0
@@ -119,6 +119,7 @@ struct perf_missing_features {bool ksymbol;bool bpf;bool aux_output; +bool cgroup; };extern struct perf_missing_features perf_missing_features;
tools/perf/util/record.h +1 -0
@@ -34,6 +34,7 @@ struct record_opts {bool auxtrace_snapshot_on_exit;bool auxtrace_sample_mode;bool record_namespaces; +bool record_cgroup;bool record_switch_events;bool all_kernel;bool all_user;
Signature · 1 line
2.25.0.265.gbab2e86ba0-goog
Namhyung Kim
maintainerPATCH 9/10 perf top: Add --all-cgroups option
In reply to Namhyung KimThe --all-cgroups option is to enable cgroup profiling support. It tells kernel to record CGROUP events in the ring buffer so that perf report can identify task/cgroup association later.
--- tools/perf/Documentation/perf-top.txt | 4 ++++ tools/perf/builtin-top.c | 9 +++++++++ 2 files changed, 13 insertions(+)
tools/perf/Documentation/perf-top.txt +4 -0
@@ -272,6 +272,10 @@ Default is to monitor all CPUS. Record events of type PERF_RECORD_NAMESPACES and display it with the 'cgroup_id' sort key. +--all-cgroups:: + Record events of type PERF_RECORD_CGROUP and display it with the + 'cgroup' sort key. + --switch-on EVENT_NAME:: Only consider events after this event is found.
tools/perf/builtin-top.c +9 -0
@@ -1244,6 +1244,8 @@ static int __cmd_top(struct perf_top *top)if (opts->record_namespaces) top->tool.namespace_events = true; +if (opts->record_cgroup) + top->tool.cgroup_events = true; ret = perf_event__synthesize_bpf_events(top->session, perf_event__process, &top->session->machines.host,
@@ -1251,6 +1253,11 @@ static int __cmd_top(struct perf_top *top)if (ret <0 )pr_debug ("Couldn't synthesize BPF events: Pre-existing BPF programs won't have symbols resolved.\n" ); + ret =perf_event__synthesize_cgroups (&top->tool, perf_event__process, + &top->session->machines.host); +if (ret <0 ) +pr_debug ("Couldn't synthesize cgroup events.\n" ); +machine__synthesize_threads (&top->session->machines.host, &opts->target, top->evlist->core.threads, false, top->nr_threads_synthesize);
@@ -1543,6 +1550,8 @@ int cmd_top(int argc, const char **argv) "number of thread to run event synthesize"),OPT_BOOLEAN (0 ,"namespaces" , &opts->record_namespaces,"Record namespaces events" ), +OPT_BOOLEAN (0 ,"all-cgroups" , &opts->record_cgroup, +"Record cgroup events" ),OPTS_EVSWITCH (&top.evswitch), OPT_END() };
Signature · 1 line
2.25.0.265.gbab2e86ba0-goog
Namhyung Kim
maintainerPATCH 10/10 perf script: Add --show-cgroup-events option
In reply to Namhyung KimThe --show-cgroup-events option is to print CGROUP events in the output like others.
--- tools/perf/Documentation/perf-script.txt | 3 ++ tools/perf/builtin-script.c | 41 ++++++++++++++++++++++++ 2 files changed, 44 insertions(+)
tools/perf/Documentation/perf-script.txt +3 -0
@@ -319,6 +319,9 @@ OPTIONS --show-bpf-events Display bpf events i.e. events of type PERF_RECORD_KSYMBOL and PERF_RECORD_BPF_EVENT. +--show-cgroup-events + Display cgroup events i.e. events of type PERF_RECORD_CGROUP. + --demangle:: Demangle symbol names to human readable form. It's enabled by default, disable with --no-demangle.
tools/perf/builtin-script.c +41 -0
71 lines not on this page · raw message
Signature · 1 line
2.25.0.265.gbab2e86ba0-goog
Quoted 32 lines · Namhyung Kim
Hello,
This work is to improve cgroup profiling in perf. Currently it only supports profiling tasks in a specific cgroup and there's no way to identify which cgroup the current sample belongs to. So I added
PERF_SAMPLE_CGROUPto add cgroup id into each sample. It's a 64-bit integer having file handle of the cgroup. And kernel also generatesPERF_RECORD_CGROUPevent for new groups to correlate the cgroup id and cgroup name (path in the cgroup filesystem). The cgroup id can be read from userspace byname_to_handle_at()system call so it can synthesize the CGROUP event for existing groups.So why do we want this? Systems running a large number of jobs in different cgroups want to profiling such jobs precisely. This includes container hosting systems widely used today. Currently perf supports namespace tracking but the systems may not use (cgroup) namespace for their jobs. Also it'd be more intuitive to see cgroup names (as they're given by user or sysadmin) rather than numeric cgroup/namespace id even if they use the namespaces.
From Stephane Eranian:
Quoted 6 lines
In data centers you care about attributing samples to a job not such much to a process. A job may have multiple processes which may come and go. The cgroup on the other hand stays around for the entire lifetime of the job. It is much easier to map a cgroup name to a particular job than it is to map a pid back to a job name, especially for offline post-processing.
Note that this only works for "
perf_event" cgroups (obviously) so if users are still using cgroup-v1 interface, they need to have same hierarchy for subsystem(s) want to profile with it.
Quoted 5 lines
* Changes from v4:
- useCONFIG_CGROUP_PERF
- move cgroup tree toperf_env
- move cgroup fs utility function totools/lib/api/fs
- use a local buffer and check its size for cgroup systhesis
the perf top tui should all cgroup id as 0 and the headers are misaligned
Samples Overhead cgroup id (dev/inode Pid:Command 83.78% 0/0x0 N/A 6508:perf 8.82% 0/0x0 N/A 0:swapper 2.59% 0/0x0 N/A 6466:perf 1.69% 0/0x0 N/A 6509:perf-top-UI 0.56% 0/0x0 N/A 12:rcu_sched 0.29% 0/0x0 N/A 429:kworker/0:2-mm_ 0.15% 0/0x0 N/A 1416:sshd 0.12% 0/0x0 N/A 187:migration/35
jirka
Namhyung Kim
maintainerHello Peter, Tejun and folks.
Do you have any other comments on the kernel side? If not, can I get your Ack?
Thanks
Namhyung
Quoted 311 lines · Namhyung Kim
Hello,
This work is to improve cgroup profiling in perf. Currently it only supports profiling tasks in a specific cgroup and there's no way to identify which cgroup the current sample belongs to. So I added
PERF_SAMPLE_CGROUPto add cgroup id into each sample. It's a 64-bit integer having file handle of the cgroup. And kernel also generatesPERF_RECORD_CGROUPevent for new groups to correlate the cgroup id and cgroup name (path in the cgroup filesystem). The cgroup id can be read from userspace byname_to_handle_at()system call so it can synthesize the CGROUP event for existing groups.So why do we want this? Systems running a large number of jobs in different cgroups want to profiling such jobs precisely. This includes container hosting systems widely used today. Currently perf supports namespace tracking but the systems may not use (cgroup) namespace for their jobs. Also it'd be more intuitive to see cgroup names (as they're given by user or sysadmin) rather than numeric cgroup/namespace id even if they use the namespaces.
From Stephane Eranian:
Quoted 6 lines
In data centers you care about attributing samples to a job not such much to a process. A job may have multiple processes which may come and go. The cgroup on the other hand stays around for the entire lifetime of the job. It is much easier to map a cgroup name to a particular job than it is to map a pid back to a job name, especially for offline post-processing.
Note that this only works for "
perf_event" cgroups (obviously) so if users are still using cgroup-v1 interface, they need to have same hierarchy for subsystem(s) want to profile with it.* Changes from v4:
- useCONFIG_CGROUP_PERF
- move cgroup tree toperf_env
- move cgroup fs utility function totools/lib/api/fs
- use a local buffer and check its size for cgroup systhesis* Changes from v3:
- staticizeperf_event_cgroup()
- fix sample parsing test* Changes from v2:
- removepath_lenfromcgroup_event
- bail out if kernel doesn't support cgroup sampling
- add some description in theKconfig* Changes from v1:
- use new cgroup id (= file handle)The code is available at perf/cgroup-v5 branch in
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.gitThe testing result looks something like this:
[root@qemu build]# ./perf record --all-cgroups ./cgtest
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.009 MB perf.data (150 samples) ][root@qemu build]# ./perf report -s cgroup,comm --stdio # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 150 of event 'cpu-clock:pppH' # Event count (approx.): 37500000 # # Overhead cgroup Command # ........ .......... ....... # 32.00% /sub/cgrp2 looper2 28.00% /sub/cgrp1 looper1 25.33% /sub looper0 4.00% / cgtest 4.00% /sub cgtest 3.33% /sub/cgrp2 cgtest 2.67% /sub/cgrp1 cgtest 0.67% / looper0The test program (cgtest) follows.
Thanks,
NamhyungCc Tejun Heo <[email protected]>Cc Li Zefan <[email protected]>Cc Johannes Weiner <[email protected]>Cc Adrian Hunter <[email protected]>-------8<-----------------------------------------8<---------------- #define
_GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <sched.h>
#include <errno.h>
#include <sys/stat.h>
#include <sys/wait.h>
#include <sys/prctl.h>
#include <sys/mount.h>char cgbase[] = "
/sys/fs/cgroup/perf_event";void
mkcgrp(char *name){
char buf[256];
snprintf(buf, sizeof(buf), "%s%s", cgbase, name); if (mkdir(buf, 0755)< 0)perror("mkdir"); }void
rmcgrp(char *name){
char buf[256];
snprintf(buf, sizeof(buf), "%s%s", cgbase, name); if (rmdir(buf) < 0)perror("rmdir"); }void
setcgrp(char *name){
char buf[256];
int fd;snprintf(buf, sizeof(buf), "%s%s/tasks", cgbase, name);fd =
open(buf, O_WRONLY);if (fd > 0) { if (write(fd, "0\n", 2) != 2) perror("write"); close(fd);}
}void
create_sub_cgroup(int idx){
char buf[128];
snprintf(buf, sizeof(buf), "/sub/cgrp%d", idx);
mkcgrp(buf);
}void
remove_sub_cgroup(int idx){
char buf[128];
snprintf(buf, sizeof(buf), "/sub/cgrp%d", idx);
rmcgrp(buf);
}void
set_sub_cgroup(int idx){
char buf[128];
snprintf(buf, sizeof(buf), "/sub/cgrp%d", idx);
setcgrp(buf);
}void
set_task_name(int idx){
char buf[16];
snprintf(buf, sizeof(buf), "looper%d", idx);
prctl(PR_SET_NAME, buf, 0, 0, 0);
}void loop(unsigned max) {
volatile unsigned int count = 1;while (count++ != max) { asm volatile ("pause");}
}void
worker(int idx, unsigned cnt, int new_ns){
int oldns;create_sub_cgroup(idx); set_sub_cgroup(idx); if (new_ns) { if (unshare(CLONE_NEWCGROUP) < 0) perror("unshare");#if 0 /* FIXME */ if (unshare(CLONE_NEWNS) < 0) perror("unshare"); if (mount("none", "/sys", NULL, MS_REMOUNT | MS_REC | MS_SLAVE, NULL) < 0) perror("mount --make-rslave"); sleep(1); if (umount("/sys/fs/cgroup/perf_event") < 0) perror("umount"); if (mount("cgroup", "/sys/fs/cgroup/perf_event", "cgroup", MS_NODEV | MS_NOEXEC | MS_NOSUID, "perf_event") < 0) perror("mount again");#endif
}if (fork() == 0) { set_task_name(idx); loop(cnt); exit(0);}
wait(NULL);
}int
main(int argc, char *argv[])
{
int i, nr = 2;
intnew_ns= 1;
unsigned cnt = 1000000;
int fd;if (argc > 1) nr = atoi(argv[1]); if (argc > 2) cnt = atoi(argv[2]); if (argc > 3) new_ns = atoi(argv[3]); mkcgrp("/sub"); setcgrp("/sub"); for (i = 0; i < nr; i++) { if (fork() == 0) { worker(i+1, cnt, new_ns); exit(0); } } set_task_name(0); loop(cnt); for (i = 0; i < nr; i++) wait(NULL); for (i = 0; i < nr; i++) remove_sub_cgroup(i+1); setcgrp("/"); rmcgrp("/sub");return 0;
} -------8<-----------------------------------------8<----------------Namhyung Kim (10):
perf/core: AddPERF_RECORD_CGROUPevent
perf/core: AddPERF_SAMPLE_CGROUPfeature
tools lib api fs: Movecgroupsfs__mountpoint()
perf tools: Basic support for CGROUP event
perf tools: Maintain cgroup hierarchy
perf report: Add 'cgroup' sort key
perf record: Support synthesizing cgroup events
perf record: Add --all-cgroups option
perf top: Add --all-cgroups option
perf script: Add --show-cgroup-events optioninclude/linux/perf_event.h | 1 + include/uapi/linux/perf_event.h | 16 ++- init/Kconfig | 3 +- kernel/events/core.c | 133 ++++++++++++++++++++ tools/include/uapi/linux/perf_event.h | 16 ++- tools/lib/api/fs/Build | 1 + tools/lib/api/fs/cgroup.c | 67 ++++++++++ tools/lib/api/fs/fs.h | 2 + tools/lib/perf/include/perf/event.h | 7 ++ tools/perf/Documentation/perf-record.txt | 5 +- tools/perf/Documentation/perf-report.txt | 1 + tools/perf/Documentation/perf-script.txt | 3 + tools/perf/Documentation/perf-top.txt | 4 + tools/perf/builtin-diff.c | 1 + tools/perf/builtin-record.c | 10 ++ tools/perf/builtin-report.c | 1 + tools/perf/builtin-script.c | 41 +++++++ tools/perf/builtin-top.c | 9 ++ tools/perf/tests/sample-parsing.c | 6 +- tools/perf/util/cgroup.c | 143 +++++++++++++--------- tools/perf/util/cgroup.h | 18 ++- tools/perf/util/env.c | 2 + tools/perf/util/env.h | 6 + tools/perf/util/event.c | 19 +++ tools/perf/util/event.h | 6 + tools/perf/util/evsel.c | 17 ++- tools/perf/util/evsel.h | 1 + tools/perf/util/hist.c | 12 ++ tools/perf/util/hist.h | 1 + tools/perf/util/machine.c | 19 +++ tools/perf/util/machine.h | 3 + tools/perf/util/perf_event_attr_fprintf.c | 2 + tools/perf/util/record.h | 1 + tools/perf/util/session.c | 4 + tools/perf/util/sort.c | 37 ++++++ tools/perf/util/sort.h | 2 + tools/perf/util/synthetic-events.c | 121 ++++++++++++++++++ tools/perf/util/synthetic-events.h | 1 + tools/perf/util/tool.h | 2 + 39 files changed, 671 insertions(+), 73 deletions(-) create mode 100644 tools/lib/api/fs/cgroup.c--
2.25.0.265.gbab2e86ba0-goog
Tejun Heo
Acked-byQuoted 4 lines · Namhyung Kim
Hello Peter, Tejun and folks.
Do you have any other comments on the kernel side? If not, can I get your Ack?
Everything looks good from cgroup side. I think I acked all cgroup parts already but if not please feel free to add
Thanks.
Signature · 1 line
tejun
Quoted 10 lines · Tejun Heo
Quoted 4 lines · Namhyung Kim
Hello Peter, Tejun and folks.
Do you have any other comments on the kernel side? If not, can I get your Ack?
Everything looks good from cgroup side. I think I acked all cgroup parts already but if not please feel free to add
Acked-by Tejun Heo <[email protected]>
Yeah, looks good to me too. Since it's mostly userspace patches, will you route it Arnaldo?
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Tejun Heo
Quoted 4 lines · Namhyung Kim
Hello Peter, Tejun and folks.
Do you have any other comments on the kernel side? If not, can I get your Ack?
Everything looks good from cgroup side. I think I acked all cgroup
parts already
Quoted 8 lines
Quoted 3 lines
but if not please feel free to add
Acked-by Tejun Heo <[email protected]>Yeah, looks good to me too. Since it's mostly userspace patches, will you route it Arnaldo?
Thanks for the acks, I'll process it tomorrow, so tests passing it'll go Ingo's way for 5.7 tomorrow.
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Namhyung Kim
maintainerQuoted 12 lines · Arnaldo Melo
Quoted 11 lines · Peter Zijlstra
Quoted 7 lines · Tejun Heo
Quoted 4 lines · Namhyung Kim
Hello Peter, Tejun and folks.
Do you have any other comments on the kernel side? If not, can I get your Ack?
Everything looks good from cgroup side. I think I acked all cgroup
parts already
Quoted 2 lines
but if not please feel free to add
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Acked-by Tejun Heo <[email protected]>Yeah, looks good to me too. Since it's mostly userspace patches, will you route it Arnaldo?
Thanks for the acks, I'll process it tomorrow, so tests passing it'll go Ingo's way for 5.7 tomorrow.
Thanks all for the acks,
Arnaldo, as it's floating around on the list about a month already I guess it needs a rebase. Also I need to check TUI as Jiri reported. Will post a next version soon.
Thanks
Namhyung
Em Wed, Mar 25, 2020 at 07:24:37PM +0900, Namhyung Kim escreveu:
Quoted 20 lines · Arnaldo Melo
Quoted 19 lines · Peter Zijlstra
Quoted 17 lines · Tejun Heo
Quoted 7 lines · Namhyung Kim
Quoted 4 lines
Hello Peter, Tejun and folks.
Do you have any other comments on the kernel side? If not, can I get your Ack?
Everything looks good from cgroup side. I think I acked all cgroup
parts already
Quoted 3 lines
but if not please feel free to add
Acked-by Tejun Heo <[email protected]>Yeah, looks good to me too. Since it's mostly userspace patches, will you route it Arnaldo?
Quoted 7 lines
Quoted 1 line
Thanks for the acks, I'll process it tomorrow, so tests passing it'll go Ingo's way for 5.7 tomorrow.
Thanks all for the acks,
Arnaldo, as it's floating around on the list about a month already I guess it needs a rebase. Also I need to check TUI as Jiri reported. Will post a next version soon.
Thanks, I'll wait then.
- Arnaldo