Skip to contentSkip to conversation

perf stat: Fix false NMI watchdog warning in aggregation modes

PATCH Applied Chun-Tse Shao 4 participants 4 messages Jun 11 – Jun 12 perf events

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 perf stat: Fix false NMI watchdog warning in aggregation modes

In aggregation modes (e.g. --per-socket, --per-die, etc.), a counter might not be scheduled or counted on specific aggregate groups if it was not assigned to the CPUs belonging to those groups. However, the printout() check triggers the "print_free_counters_hint" logic unconditionally for any supported counter with a missing count. This results in a false "Some events weren't counted. Try disabling the NMI watchdog" warning.

Furthermore, the NMI watchdog only reserves performance counters on core PMUs. Uncore PMU events (e.g. CHA, IMC) are not affected by the NMI watchdog, but their failures also falsely triggered this warning.

This warning was originally introduced in commit 02d492e5dcb7 ("perf stat: Issue a HW watchdog disable hint").

To fix this, restrict setting of print_free_counters_hint to only trigger for core PMU events by checking counter->pmu and counter->pmu->is_core.

Example before/after:

$ perf stat -M lpm_miss_lat --metric-only --per-socket -a -- sleep 1

Before:
Performance counter stats for 'system wide':

ns lpm_miss_lat_rem ns lpm_miss_lat_loc

S0      126                202.3               207.9
S1      126                231.9               259.3
       1.006029831 seconds time elapsed

Some events weren't counted. Try disabling the NMI watchdog:
echo 0 > /proc/sys/kernel/nmi_watchdog perf stat ...
echo 1 > /proc/sys/kernel/nmi_watchdog

After:
Performance counter stats for 'system wide':

ns lpm_miss_lat_rem ns lpm_miss_lat_loc

S0      126                202.3               207.9
S1      126                231.9               259.3
       1.006029831 seconds time elapsed
Assisted-by Gemini:gemini-next
---
 tools/perf/util/stat-display.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/util/stat-display.c +2 -2
index 2b69d238858c..0a5750bb59fa 100644
@@ -821,9 +821,9 @@ static void printout(struct perf_stat_config *config, struct outstate *os,
 		ok = false;
 
 		if (counter->supported) {
-			if (!evlist__has_hybrid_pmus(counter->evlist)) {
+			if (!evlist__has_hybrid_pmus(counter->evlist) &&
+			    counter->pmu && counter->pmu->is_core)
 				config->print_free_counters_hint = 1;
-			}
 		}
 	}
 
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Directly accessing `counter->pmu` without lazy initialization suppresses the NMI watchdog hint…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Directly accessing counter->pmu without lazy initialization suppresses the NMI watchdog hint during perf stat report. --

commit 8ecd592e1137cf0841158570884e6514049ebf6f
Author: Chun-Tse Shao <[email protected]>

perf stat: Fix false NMI watchdog warning in aggregation modes

This commit fixes false NMI watchdog warnings in aggregation modes by restricting the hint to core PMU events. It prevents the warning from triggering on uncore PMU events by checking the event's PMU and its core flag.

Quoted 4 lines
tools/perf/util/stat-display.c +0 -0
index 993f4c4b8f442..b576776917353 100644

[ ... ]

tools/perf/util/stat-display.c static void printout(struct perf_stat_config *config, struct outstate *os, · line 827

Quoted 7 lines
tools/perf/util/stat-display.c +2 -1
@@ -823,9 +823,9 @@ static void printout(struct perf_stat_config *config, struct outstate *os,
 		ok = false;
 
 		if (counter->supported) {
-			if (!evlist__has_hybrid_pmus(counter->evlist)) {
+			if (!evlist__has_hybrid_pmus(counter->evlist) &&
+			    counter->pmu && counter->pmu->is_core)

[Severity: Medium]
During a perf stat report on non-hybrid machines, will counter->pmu be null here because it is lazily initialized when constructed from perf.data?

Since evlist__has_hybrid_pmus() immediately returns false on non-hybrid machines without iterating over the events, counter->pmu would remain uninitialized.

Does this code safely handle lazy initialization, or should it use evsel__find_pmu(counter) to guarantee the PMU is resolved before checking is_core?

Quoted 4 lines
tools/perf/util/stat-display.c +0 -1
 				config->print_free_counters_hint = 1;
-			}
 		}
 	}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
reviewer Reviewed-by on v2 1/1
#2 lore raw
In reply to Chun-Tse Shao
Quoted 43 lines · Chun-Tse Shao

In aggregation modes (e.g. --per-socket, --per-die, etc.), a counter might not be scheduled or counted on specific aggregate groups if it was not assigned to the CPUs belonging to those groups. However, the printout() check triggers the "print_free_counters_hint" logic unconditionally for any supported counter with a missing count. This results in a false "Some events weren't counted. Try disabling the NMI watchdog" warning.

Furthermore, the NMI watchdog only reserves performance counters on core PMUs. Uncore PMU events (e.g. CHA, IMC) are not affected by the NMI watchdog, but their failures also falsely triggered this warning.

This warning was originally introduced in commit 02d492e5dcb7 ("perf stat: Issue a HW watchdog disable hint").

To fix this, restrict setting of print_free_counters_hint to only trigger for core PMU events by checking counter->pmu and counter->pmu->is_core.

Example before/after:

$ perf stat -M lpm_miss_lat --metric-only --per-socket -a -- sleep 1

Before:
Performance counter stats for 'system wide':

ns lpm_miss_lat_rem ns lpm_miss_lat_loc

S0      126                202.3               207.9
S1      126                231.9               259.3
       1.006029831 seconds time elapsed

Some events weren't counted. Try disabling the NMI watchdog:
echo 0 > /proc/sys/kernel/nmi_watchdog perf stat ...
echo 1 > /proc/sys/kernel/nmi_watchdog

After:
Performance counter stats for 'system wide':

Quoted 8 lines

ns lpm_miss_lat_rem ns lpm_miss_lat_loc

S0      126                202.3               207.9
S1      126                231.9               259.3
       1.006029831 seconds time elapsed
Assisted-by Gemini:gemini-next

The suggestion for something like:

```
pmu = evsel__find_pmu(counter);
if (pmu && pmu->is_core)
```

isn't really necessary because we nearly always set the PMU. Also, a case where we lack a pmu has historically been for a core PMU, making the whole thing contradictory. The patch as-is is clear.

Thanks,
Ian

Quoted 23 lines
---
 tools/perf/util/stat-display.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/util/stat-display.c +2 -3
index 2b69d238858c..0a5750bb59fa 100644
@@ -821,9 +821,9 @@ static void printout(struct perf_stat_config *config, struct outstate *os,
                ok = false;
 
                if (counter->supported) {
-                       if (!evlist__has_hybrid_pmus(counter->evlist)) {
+                       if (!evlist__has_hybrid_pmus(counter->evlist) &&
+                           counter->pmu && counter->pmu->is_core)
                                config->print_free_counters_hint = 1;
-                       }
                }
        }
 
--

2.54.0.1136.gdb2ca164c4-goog

maintainer Applied
#3 lore raw
Quoted 10 lines · Ian Rogers
Quoted 6 lines · Chun-Tse Shao

To fix this, restrict setting of print_free_counters_hint to only trigger for core PMU events by checking counter->pmu and counter->pmu->is_core.

Assisted-by Gemini:gemini-next
Quoted 8 lines

The suggestion for something like:

```
pmu = evsel__find_pmu(counter);
if (pmu && pmu->is_core)
```

isn't really necessary because we nearly always set the PMU. Also, a case where we lack a pmu has historically been for a core PMU, making the whole thing contradictory. The patch as-is is clear.

Thanks, applied to perf-tools-next, for v7.2.

- Arnaldo