Skip to contentSkip to conversation

perf test fail :: "perf stat --bpf-counters --for-each-cgroup test"

DISCUSSION Michael Petlan 2 participants 6 messages Jul 19, 2024 – Nov 4, 2024 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Hello Namhyung,

we were investigating some test failures of the testcase mentioned in $subj. We have narrowed it down to:

    # perf stat -C 0,1 --for-each-cgroup system.slice,user.slice -e cycles -- taskset -c 1 perf test -w thloop

Performance counter stats for 'CPU(s) 0,1':
<not counted> cycles system.slice
3,020,401,084 cycles user.slice

1.009787097 seconds time elapsed

As seen, the system.slice is not counted properly in our case. It happens even without bpf-counters being involved.

There were rumours that it might be caused due to too small system load, but it apparently happens even when the load was replaced by "thloop" workload from perf-test's workload library. However, even so, if the load was insufficient, we'd see a value – 0 instead of "not counted". The "<not counted>" result is printed if the counter wasn't properly enabled and running.

Have you encountered this problem? What could cause it?

Thanks.
Michael

A message that is not in the archive (sent off-list or not yet received): CA+JHD90TkDVHPw4jqxMX2guqsg-8xrqD2iiEfZ_akixvVYZKZg@mail.gmail.com

#2 lore raw
Quoted 24 lines · Arnaldo Carvalho de Melo

On Fri, Jul 19, 2024, 6:50 AM Michael Petlan <[email protected]> wrote:
Hello Namhyung,

we were investigating some test failures of the testcase mentioned in $subj. We have narrowed it down to:

    # perf stat -C 0,1 --for-each-cgroup system.slice,user.slice -e cycles -- taskset -c 1 perf test -w thloop

    Performance counter stats for 'CPU(s) 0,1':
         <not counted>      cycles                           system.slice 3,020,401,084      cycles                           user.slice

         1.009787097 seconds time elapsed

As seen, the system.slice is not counted properly in our case. It happens even without bpf-counters being involved.

There were rumours that it might be caused due to too small system load, but it apparently happens even when the load was replaced by "thloop" workload from perf-test's workload library. However, even so, if the load was insufficient, we'd see a value – 0 instead of "not counted". The "<not counted>" result is printed if the counter wasn't properly enabled and running.

Quoted 4 lines

Have you encountered this problem? What could cause it?

What does running with -vvv says? Some inconclusive error coming from the kernel?

Nothing obvious:

# perf stat -vvv -C 0,1 --for-each-cgroup system.slice,user.slice -e cpu-clock taskset -c 0 perf test -w thloop
Using CPUID GenuineIntel-6-6A-6
Control descriptor is not initialized
Opening: cpu-clock 
------------------------------------------------------------
perf_event_attr:   
  type                             1 (PERF_TYPE_SOFTWARE)
  size                             136
  config                           0 (PERF_COUNT_SW_CPU_CLOCK)
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
  disabled                         1
  inherit                          1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid 3  cpu 0  group_fd -1  flags 0xc = 5
Opening: cpu-clock 
------------------------------------------------------------
perf_event_attr:   
  type                             1 (PERF_TYPE_SOFTWARE)
  size                             136
  config                           0 (PERF_COUNT_SW_CPU_CLOCK)
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
  disabled                         1
  inherit                          1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid 4  cpu 0  group_fd -1  flags 0xc = 6
Opening: cpu-clock
------------------------------------------------------------
perf_event_attr:
  type                             1 (PERF_TYPE_SOFTWARE)
  size                             136
  config                           0 (PERF_COUNT_SW_CPU_CLOCK)
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
  disabled                         1
  inherit                          1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid 3  cpu 1  group_fd -1  flags 0xc = 7
Opening: cpu-clock
------------------------------------------------------------
perf_event_attr:   
  type                             1 (PERF_TYPE_SOFTWARE)
  size                             136
  config                           0 (PERF_COUNT_SW_CPU_CLOCK)
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
  disabled                         1
  inherit                          1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid 4  cpu 1  group_fd -1  flags 0xc = 9
cpu-clock: 0: 0 0 0
cpu-clock: 0: 1004758163 1004761145 1004761145
cpu-clock: 1: 0 0 0
cpu-clock: 1: 60896 62271 62271
cpu-clock: 0 0 0   
cpu-clock: 1004819059 1004823416 1004823416

Performance counter stats for 'CPU(s) 0,1':

<not counted> msec cpu-clock system.slice
1,004.82 msec cpu-clock user.slice # 0.999 CPUs utilized

1.005824026 seconds time elapsed

Some events weren't counted. Try disabling the NMI watchdog:
echo 0 > /proc/sys/kernel/nmi_watchdog perf stat ...
echo 1 > /proc/sys/kernel/nmi_watchdog

....

The nmi_watchdog message is irrelevant, it does not work no matter what is set there.

Quoted 1 line

Maybe retsnoop can narrow it down?

Will try. Thanks.

Quoted 4 lines

https://github.com/anakryiko/retsnoop

- Arnaldo

Michael

Quoted 8 lines

Thanks.
Michael

#3 lore raw
Quoted 25 lines · Michael Petlan
Quoted 24 lines · Arnaldo Carvalho de Melo

On Fri, Jul 19, 2024, 6:50 AM Michael Petlan <[email protected]> wrote:
Hello Namhyung,

we were investigating some test failures of the testcase mentioned in $subj. We have narrowed it down to:

    # perf stat -C 0,1 --for-each-cgroup system.slice,user.slice -e cycles -- taskset -c 1 perf test -w thloop

    Performance counter stats for 'CPU(s) 0,1':
         <not counted>      cycles                           system.slice 3,020,401,084      cycles                           user.slice

         1.009787097 seconds time elapsed

As seen, the system.slice is not counted properly in our case. It happens even without bpf-counters being involved.

There were rumours that it might be caused due to too small system load, but it apparently happens even when the load was replaced by "thloop" workload from perf-test's workload library. However, even so, if the load was insufficient, we'd see a value – 0 instead of "not counted". The "<not counted>" result is printed if the counter wasn't properly enabled and running.

Quoted 5 lines
Quoted 4 lines

Have you encountered this problem? What could cause it?

What does running with -vvv says? Some inconclusive error coming from the kernel?

Hello!

We have been investigating this issue a bit more again and we have come to conclusion that everything is probably OK, except of the testcase which in short relies on the fact that taskset can force any system.slice workload to happen on a particular CPU, which in my opinion does not apply, being rather random and that's why the test sometimes fails.

To summarize the problem a bit:

1) The $subj testcase sometimes fails.

2) It consists of two parts, one performs counting system-wide and the second limits the counting to CPUs 0 and 1. The second one sometimes fails, while the first (systemwide) passes always.

3) The reason why the test fails is because system.slice may get <not counted> result.

4) There is another problem with this testcase on single-cpu boxes, since there is no "cpu 1", so we decided to try having "-C 0" and "taskset -c 0" on such boxes. The problems with getting "<not counted>" disappeared!

---------------------------

So... The systemwide tracing test works:

# perf stat --for-each-cgroup system.slice,user.slice -e cycles -a -- sleep 3
 Performance counter stats for 'system wide':
     8,884,593      cycles            system.slice
     5,645,624      cycles            user.slice

3.004137451 seconds time elapsed

When we pin the workload AND tracing to particular CPU, it might fail:

# perf stat -C 0 --for-each-cgroup system.slice,user.slice -e cycles -- taskset -c 0 true

Performance counter stats for 'CPU(s) 0':

     <not counted>  cycles            system.slice
     2,722,263      cycles            user.slice

0.004184686 seconds time elapsed

Namhyung said that there might be not enough load, which finally appears to be the problem. But not in the manner that replacing true by something more "heavy" would help, but in the fact that system.slice didn't run on CPU 0 at all during the perf stat counting.

Taskset can pin the process to some CPU, but even without it (or when we pin it to CPU 3 for example), _some_ user.slice content is always run on cpu 0, so we get values.

However, there is no guarantee that system.slice will run there. We would probably need to load more the content that systemd decides to put under "system.slice" and hope that it will get a chance to run on CPU 0 or 1 or whatever we use in the testcase.

Of course, the more CPUs the machine has, the higher chance to get the <not counted> result for system.slice is. That's why -a works and also that's why it works on a single-CPU machine.

............

Thus, I think that we should simply remove the taskset part of the testcase and leave only the systemwide part.

Thoughts?

Michael

#4 lore raw

Hello Michael,

Quoted 96 lines · Michael Petlan
Quoted 25 lines · Michael Petlan
Quoted 24 lines · Arnaldo Carvalho de Melo

On Fri, Jul 19, 2024, 6:50 AM Michael Petlan <[email protected]> wrote:
Hello Namhyung,

we were investigating some test failures of the testcase mentioned in $subj. We have narrowed it down to:

    # perf stat -C 0,1 --for-each-cgroup system.slice,user.slice -e cycles -- taskset -c 1 perf test -w thloop

    Performance counter stats for 'CPU(s) 0,1':
         <not counted>      cycles                           system.slice 3,020,401,084      cycles                           user.slice

         1.009787097 seconds time elapsed

As seen, the system.slice is not counted properly in our case. It happens even without bpf-counters being involved.

There were rumours that it might be caused due to too small system load, but it apparently happens even when the load was replaced by "thloop" workload from perf-test's workload library. However, even so, if the load was insufficient, we'd see a value – 0 instead of "not counted". The "<not counted>" result is printed if the counter wasn't properly enabled and running.

Quoted 5 lines
Quoted 4 lines

Have you encountered this problem? What could cause it?

What does running with -vvv says? Some inconclusive error coming from the kernel?

Hello!

We have been investigating this issue a bit more again and we have come to conclusion that everything is probably OK, except of the testcase which in short relies on the fact that taskset can force any system.slice workload to happen on a particular CPU, which in my opinion does not apply, being rather random and that's why the test sometimes fails.

To summarize the problem a bit:

1) The $subj testcase sometimes fails.

2) It consists of two parts, one performs counting system-wide and the second limits the counting to CPUs 0 and 1. The second one sometimes fails, while the first (systemwide) passes always.

3) The reason why the test fails is because system.slice may get <not counted> result.

4) There is another problem with this testcase on single-cpu boxes, since there is no "cpu 1", so we decided to try having "-C 0" and "taskset -c 0" on such boxes. The problems with getting "<not counted>" disappeared!

---------------------------

So... The systemwide tracing test works:

# perf stat --for-each-cgroup system.slice,user.slice -e cycles -a -- sleep 3
 Performance counter stats for 'system wide':
     8,884,593      cycles            system.slice
     5,645,624      cycles            user.slice
   3.004137451 seconds time elapsed

When we pin the workload AND tracing to particular CPU, it might fail:

# perf stat -C 0 --for-each-cgroup system.slice,user.slice -e cycles -- taskset -c 0 true

Performance counter stats for 'CPU(s) 0':

     <not counted>  cycles            system.slice
     2,722,263      cycles            user.slice
   0.004184686 seconds time elapsed

Namhyung said that there might be not enough load, which finally appears to be the problem. But not in the manner that replacing true by something more "heavy" would help, but in the fact that system.slice didn't run on CPU 0 at all during the perf stat counting.

Taskset can pin the process to some CPU, but even without it (or when we pin it to CPU 3 for example), _some_ user.slice content is always run on cpu 0, so we get values.

However, there is no guarantee that system.slice will run there. We would probably need to load more the content that systemd decides to put under "system.slice" and hope that it will get a chance to run on CPU 0 or 1 or whatever we use in the testcase.

Of course, the more CPUs the machine has, the higher chance to get the <not counted> result for system.slice is. That's why -a works and also that's why it works on a single-CPU machine.

Quoted 6 lines

............

Thus, I think that we should simply remove the taskset part of the testcase and leave only the systemwide part.

Thoughts?

Thanks for taking a look at this. I admit that the CPU list can make troubles. We need a better cgroup testing infra. Anyway let's get rid of the problematic test for now.

Thanks,
Namhyung

#5 lore raw
In reply to Michael Petlan

Hello,

Quoted 17 lines · Michael Petlan

Hello Namhyung,

we were investigating some test failures of the testcase mentioned in $subj. We have narrowed it down to:

    # perf stat -C 0,1 --for-each-cgroup system.slice,user.slice -e cycles -- taskset -c 1 perf test -w thloop

Performance counter stats for 'CPU(s) 0,1':
<not counted> cycles system.slice
3,020,401,084 cycles user.slice

1.009787097 seconds time elapsed

As seen, the system.slice is not counted properly in our case. It happens even without bpf-counters being involved.

Quoted 8 lines

There were rumours that it might be caused due to too small system load, but it apparently happens even when the load was replaced by "thloop" workload from perf-test's workload library. However, even so, if the load was insufficient, we'd see a value – 0 instead of "not counted". The "<not counted>" result is printed if the counter wasn't properly enabled and running.

Have you encountered this problem? What could cause it?

Sometimes, it depends on the system activity. I assumed there would be some system operation running on the given CPUs at any time. If not, it can fail..

Thanks,
Namhyung

#6 lore raw
Quoted 27 lines · Namhyung Kim

Hello,

Quoted 24 lines · Michael Petlan

Hello Namhyung,

we were investigating some test failures of the testcase mentioned in $subj. We have narrowed it down to:

    # perf stat -C 0,1 --for-each-cgroup system.slice,user.slice -e cycles -- taskset -c 1 perf test -w thloop

Performance counter stats for 'CPU(s) 0,1':
<not counted> cycles system.slice
3,020,401,084 cycles user.slice

1.009787097 seconds time elapsed

As seen, the system.slice is not counted properly in our case. It happens even without bpf-counters being involved.

There were rumours that it might be caused due to too small system load, but it apparently happens even when the load was replaced by "thloop" workload from perf-test's workload library. However, even so, if the load was insufficient, we'd see a value – 0 instead of "not counted". The "<not counted>" result is printed if the counter wasn't properly enabled and running.

Quoted 5 lines
Quoted 1 line

Have you encountered this problem? What could cause it?

Sometimes, it depends on the system activity. I assumed there would be some system operation running on the given CPUs at any time. If not, it can fail..

Thanks for response. However, in such case, I'd expect "0" value instead of "<not counted>". As far as I understand, "<not supported>" occurs when an event cannot be even opened, "<not counted>" appears when the event can be opened, but for example there aren't enough free hardware counters, so it does not run any counting actually. But for cases when there is not enough load, we get simply low numerical values, or 0, don't we...

Michael

Quoted 5 lines

Thanks,
Namhyung