Skip to content

RFC perf_events: add support for per-cpu per-cgroup monitoring (v2)

Stephane Eranian lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest2 patches 4 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/2perf_events: add support for per-cpu per-cgroup monitoring (v2)include/linux/cgroup.h include/linux/cgroup_subsys.h include/linux/perf_event.h +2 more+471 -1094 replies
2/2perf_events: add support for per-cpu per-cgroup monitoring (v2)tools/perf/Documentation/perf-record.txt tools/perf/Documentation/perf-stat.txt tools/perf/Makefile +4 more+189 -20 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 0/2] perf_events: add support for per-cpu per-cgroup monitoring (v2)

This series of patches adds per-container (cgroup) filtering capability to per-cpu monitoring. In other words, we can monitor all threads belonging to a specific cgroup and running on a specific CPU.

This is useful to measure what is going on inside a cgroup. Something that cannot easily and cheaply be achieved with either per-thread or per-cpu mode. Cgroups can span multiple CPUs. CPUs can be shared between cgroups. Cgroups can have lots of threads. Threads can come and go during a measurement.

To measure per-cgroup today requires using per-thread mode and attaching to all the current threads inside a cgroup and tracking new threads. That would require scanning of /proc/PID, which is subject to race conditions, and creating an event for each thread, each event requiring kernel memory.

The approach taken by this patch is to leverage the per-cpu mode by simply adding a filtering capability on context switch only when necessary. That way the amount of kernel memory used remains bound by the number of CPUs. We also do not have to scan /proc. We are only interested in cgroup level counts, not per-thread.

The cgroup to monitor is designated by passing a file descriptor opened on a new per-cgroup file in the cgroup filesystem (perf_event.perf). The option must be activated by setting perf_event_attr.cgroup=1 and passing a valid file descriptor in perf_event_attr.cgroup_fd. Those are the only two ABI extensions.

The patch also includes changes to the perf tool to make use of cgroup filtering. Both perf stat and perf record have been extended to support cgroup via a new -G option. The cgroup is specified per event:

$ perf stat -B -a -e cycles:u,cycles:u,cycles:u -G test1,,test2 -- sleep 1
 Performance counter stats for 'sleep 1':
      2,368,667,414  cycles                   test1
      2,369,661,459  cycles                  
      <not counted>  cycles                   test2
        1.001856890  seconds time elapsed

Here, we measure cycles in 3 different cgroups. When a cgroup is omitted, the "root" cgroup is used, i.e., all threads executing on the monitored CPUs are measured.

In this second version, time tracking has been updated. In cgroup mode, time_enabled tracks the time during which the cgroup was active, i.e., threads from the cgroup executed on the monitored CPU. The meaning of time_running is unchanged. In non-cgroup mode, time_enabled stills tracks wall-clock time for per-cpu events. Here is an example:

In one shell I do:

$ echo $$ >/cgroup/test1/perf_events.perf
$ taskset -c 1 noploop 600
In another shell I do:
$ taskset -c 1 noploop 600

Both noploops are competing on CPU1 (part of cgroup test1)

$ perf stat -B -a -e cycles:u,cycles:u,cycles:u -G test1,,test2 -- sleep 1
 Performance counter stats for 'sleep 1':
      1,190,595,954  cycles                   test1
      2,372,471,023  cycles                  
      <not counted>  cycles                   test2
        1.001845567  seconds time elapsed

The second count reflects activity across all CPUs and cgroups. The first count reflects was happened inside cgroup test1. As shown, the noploop running inside test1, only got half the CPU time.

PATCH 0/2: introduction
PATCH 1/2: kernel changes
PATCH 2/2: perf tool changes