Multi-PMU groups with the perf tool

Greetings,

I've been recently asked about using "uncore-like" PMUs with "perf record" command. The issue is that those PMUs are generally non-sampling, so one can not simply do

# perf record -a -e ccn/cycles/,ccn/xp_valid_flit,xp=1,port=0,vc=1,dir=1/ sleep 1

as the event_init function will return -EINVAL. In most cases, stat provides a valid equivalent:

# perf stat -a -e ccn/cycles/,ccn/xp_valid_flit,xp=1,port=0,vc=1,dir=1/ sleep 1 

providing diff between counter values "before" and "after", but what if you're looking at something more complex than "sleep 1" and you want to observe memory system behaviour over time? Of course no one expects the samples to be correlated with a particular PC, but sampling and visualising the data with, say, 1ms period can be useful for certain use cases.

My initial answer was: create a group with a 1ms cpu-clock (so hrtimer) as a leader and, with PERF_SAMPLE_READ, attach "uncore" children to it. That way, you'll get uncore counters read every ms.

I know that this works from the kernel point of view, because my custom tool does exactly this, but I wanted to provide an example using the standard perf tool (the user, obviously, already had it). So I spent some time trying to create such a group with the perf tool and, after going as far as reading the code :-) must admit defeat. What I would like to do is something along the lines of (with the parenthesis group syntax being just a product of my imagination ;-)

# perf record -F 1000 -a \
	-e cpu-clock(ccn/cycles/,ccn/xp_valid_flit,xp=1,port=0,vc=1,dir=1/) \
	sleep 1 

Have I missed something (very likely, given the number of "not-really-obvious" features of the tool ;-), or is there no way of creating such a custom group with the perf tool today? It's not a critique, merely a question.

Cheers!

Pawel

I'm sure there's a medical name for answering own questions, particularly after 6 months, but hey, it may help someone else :-)

By pure accident I finally realised it is - in principle - possible with the existing tool. The incantation would look like this (skipped the "ccn/xp_valid_flit.../" event for the sake of line length):

# perf record -a -e '{cpu-clock,ccn/cycles/}:S' sleep 1

only it still doesn't work, failing -EINVAL. After some digging I've realised the reason - the tool tries to create such a group on each of the CPUs, because I've asked for it with -a, right? Only that "uncore" events like "ccn/cycles/" are to be pinned to a single CPU and to communicate this they export "cpumask" sysfs attribute. Let's have a look:

# cat /sys/bus/event_source/devices/ccn/cpumask
3

And now, if we try to run the command above with some verbosity:

# perf record -vv -a -e '{cpu-clock,ccn/cycles/}:S' sleep 1
------------------------------------------------------------
perf_event_attr:
  type                             1
  size                             112
  { sample_period, sample_freq }   4000
  sample_type                      IP|TID|TIME|READ|ID|CPU|PERIOD
  read_format                      ID|GROUP
  disabled                         1
  mmap                             1
  comm                             1
  freq                             1
  task                             1
  sample_id_all                    1
  exclude_guest                    1
  mmap2                            1
  comm_exec                        1
------------------------------------------------------------
sys_perf_event_open: pid -1  cpu 0  group_fd -1  flags 0x8
sys_perf_event_open: pid -1  cpu 1  group_fd -1  flags 0x8
sys_perf_event_open: pid -1  cpu 2  group_fd -1  flags 0x8
sys_perf_event_open: pid -1  cpu 3  group_fd -1  flags 0x8
sys_perf_event_open: pid -1  cpu 4  group_fd -1  flags 0x8
sys_perf_event_open: pid -1  cpu 5  group_fd -1  flags 0x8
sys_perf_event_open: pid -1  cpu 6  group_fd -1  flags 0x8
sys_perf_event_open: pid -1  cpu 7  group_fd -1  flags 0x8
------------------------------------------------------------
perf_event_attr:
  type                             7
  size                             112
  config                           0xff00
  sample_type                      IP|TID|TIME|READ|ID|CPU|PERIOD
  read_format                      ID|GROUP
  freq                             1
  sample_id_all                    1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid -1  cpu 3  group_fd 4  flags 0x8
sys_perf_event_open failed, error -22

The tool seemingly did the right thing and requested the "ccn" event on CPU3 for group 4, being the "cpu-clock" event on CPU0, only it's not allowed. The perf core has this check:

			/*
			 * Make sure we're both events for the same CPU;
			 * grouping events for different CPUs is broken; since
			 * you can never concurrently schedule them anyhow.
			 */
			if (group_leader->cpu != event->cpu)
				goto err_context;

If I now, instead of doing "-a" request two CPUs in the right order, I get the proof:

# perf record -vv -C3,0 -e '{cpu-clock,ccn/cycles/}:S' sleep 1
------------------------------------------------------------
perf_event_attr:
  type                             1
  size                             112
  { sample_period, sample_freq }   4000
  sample_type                      IP|TID|TIME|READ|ID|CPU|PERIOD
  read_format                      ID|GROUP
  disabled                         1
  mmap                             1
  comm                             1
  freq                             1
  task                             1
  sample_id_all                    1
  exclude_guest                    1
  mmap2                            1
  comm_exec                        1
------------------------------------------------------------
sys_perf_event_open: pid -1  cpu 3  group_fd -1  flags 0x8
sys_perf_event_open: pid -1  cpu 0  group_fd -1  flags 0x8
------------------------------------------------------------
perf_event_attr:
  type                             7
  size                             112
  config                           0xff00
  sample_type                      IP|TID|TIME|READ|ID|CPU|PERIOD
  read_format                      ID|GROUP
  freq                             1
  sample_id_all                    1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid -1  cpu 3  group_fd 4  flags 0x8
sys_perf_event_open: pid -1  cpu 0  group_fd 5  flags 0x8
sys_perf_event_open failed, error -22

Everything works fine for the first group, created on CPU3 and then fails for the second one. Interestingly it seems that in this case the tool ignores "cpumask" and tries to create the "ccn" event on CPU0. The thing is that the "cpu" argument gets overridden by the driver to match the "cpumask" value and we're back in square one.

So finally, if I only run record on the single, correct CPU, it works:

# perf record -C3 -e '{cpu-clock,ccn/cycles/}:S' sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.330 MB perf.data (1874 samples) ]

Only it's not exactly what I wanted (I want system wide CPU data - with other events in the list - with the uncore events included in it at regular intervals)... The obvious "fix" for it would be making sure the tool does not create "cpumask"ed events in group belonging to a "wrong" CPU, but whether it's the correct solution, I'm not sure yet.

Comments welcome, silence won't be taken as an offence ;-)

Pawel