Skip to contentSkip to conversation

x86/aperfmperf: Refresh stale sample via IPI for busy NOHZ_FULL CPUs

PATCH In review Jing Wu 2 participants 6 messages Jul 28 – Jul 29 x86

lore
Share & tools
lore

Jing Wu

author
#0 lore raw

PATCH 1/1 x86/aperfmperf: Refresh stale sample via IPI for busy NOHZ_FULL CPUs

An isolated CPU covered by nohz_full stops its periodic tick once it has only one runnable task, since sched_can_stop_tick() only checks scheduling-class fairness and has no notion of cpufreq reporting needs. arch_scale_freq_tick() runs only from scheduler_tick(), so cpu_samples for that CPU is never refreshed again.

arch_freq_get_on_cpu() then permanently hits its staleness check and falls back to cpufreq_quick_get(), which returns whatever policy->cur was left at (typically the P-state floor). This happens even though HWP hardware keeps running the CPU at full turbo autonomously, as confirmed by turbostat and by directly reading APERF/MPERF.

Reproduce on an isolated, nohz_full CPU with intel_pstate/HWP by loading it and watching scaling_cur_freq stay pinned at the floor:

    taskset -c $CPU stress --cpu 1 &
    for i in $(seq 10); do
        cat /sys/devices/system/cpu/cpu$CPU/cpufreq/scaling_cur_freq
        sleep 0.5
    done
    turbostat --cpu $CPU --interval 1 --num_iterations 5

scaling_cur_freq stays at the floor for the whole run, while turbostat's Bzy_MHz confirms the CPU is actually at full turbo.

Refresh the stale sample with one on-demand arch_scale_freq_tick() via IPI before falling back, but only when the target CPU is online and not idle. APERF/MPERF both stop advancing during idle (C1+), so a delta computed over an arbitrarily long stale window still yields a correct busy-time frequency average.

Fixes 7d84c1ebf9dd ("x86/aperfmperf: Replace aperfmperf_get_khz()")
---
 arch/x86/kernel/cpu/aperfmperf.c | 34 +++++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/aperfmperf.c +33 -1
index 7ffc78d5ebf21..e51544db9f9bb 100644
@@ -12,6 +12,7 @@
 #include <linux/math64.h>
 #include <linux/percpu.h>
 #include <linux/rcupdate.h>
+#include <linux/sched.h>
 #include <linux/sched/isolation.h>
 #include <linux/sched/topology.h>
 #include <linux/smp.h>
@@ -503,16 +504,41 @@ void arch_scale_freq_tick(void)
  */
 #define MAX_SAMPLE_AGE	((unsigned long)HZ / 50)
 
+static void aperfmperf_snapshot_cpu_ipi(void *info)
+{
+	arch_scale_freq_tick();
+}
+
+/*
+ * A NOHZ_FULL CPU with a single runnable task (e.g. an isolated CPU running
+ * a pinned PMD/busy-poll workload) stops its periodic tick, so nothing ever
+ * calls arch_scale_freq_tick() for it again and cpu_samples goes stale
+ * forever, not just for one MAX_SAMPLE_AGE window. Force one on-demand
+ * sample via IPI so a deliberate frequency read doesn't report the P-state
+ * floor from before isolation took effect. Skip idle CPUs: their frequency
+ * genuinely doesn't matter and there is no point poking them with an IPI.
+ */
+static bool aperfmperf_refresh_stale_sample(int cpu)
+{
+	if (!cpu_online(cpu) || idle_cpu(cpu))
+		return false;
+
+	smp_call_function_single(cpu, aperfmperf_snapshot_cpu_ipi, NULL, 1);
+	return true;
+}
+
 int arch_freq_get_on_cpu(int cpu)
 {
 	struct aperfmperf *s = per_cpu_ptr(&cpu_samples, cpu);
 	unsigned int seq, freq;
 	unsigned long last;
+	bool refreshed = false;
 	u64 acnt, mcnt;
 
 	if (!cpu_feature_enabled(X86_FEATURE_APERFMPERF))
 		goto fallback;
 
+again:
 	do {
 		seq = raw_read_seqcount_begin(&s->seq);
 		last = s->last_update;
@@ -524,8 +550,14 @@ int arch_freq_get_on_cpu(int cpu)
 	 * Bail on invalid count and when the last update was too long ago,
 	 * which covers idle and NOHZ full CPUs.
 	 */
-	if (!mcnt || (jiffies - last) > MAX_SAMPLE_AGE)
+	if (!mcnt || (jiffies - last) > MAX_SAMPLE_AGE) {
+		if (!refreshed) {
+			refreshed = true;
+			if (aperfmperf_refresh_stale_sample(cpu))
+				goto again;
+		}
 		goto fallback;
+	}
 
 	return div64_u64((cpu_khz * acnt), mcnt);
 

---
base-commit: 502d801f0ab03e4f32f9a33d203154ce84887921
change-id: 20260728-bug-isolatecpu-cpufreq-864a5fba85b7

Best regards,

Signature · 1 line
Jing Wu <[email protected]>
#1 lore raw
Quoted 5 lines · Jing Wu

An isolated CPU covered by nohz_full stops its periodic tick once it has only one runnable task, since sched_can_stop_tick() only checks scheduling-class fairness and has no notion of cpufreq reporting needs. arch_scale_freq_tick() runs only from scheduler_tick(), so cpu_samples for that CPU is never refreshed again.

Sending IPIs to NOHZ_FULL Cpus should be a no-no.

#2 lore raw
In reply to Jing Wu
Quoted 22 lines · Jing Wu

An isolated CPU covered by nohz_full stops its periodic tick once it has only one runnable task, since sched_can_stop_tick() only checks scheduling-class fairness and has no notion of cpufreq reporting needs. arch_scale_freq_tick() runs only from scheduler_tick(), so cpu_samples for that CPU is never refreshed again.

arch_freq_get_on_cpu() then permanently hits its staleness check and falls back to cpufreq_quick_get(), which returns whatever policy->cur was left at (typically the P-state floor). This happens even though HWP hardware keeps running the CPU at full turbo autonomously, as confirmed by turbostat and by directly reading APERF/MPERF.

Reproduce on an isolated, nohz_full CPU with intel_pstate/HWP by loading it and watching scaling_cur_freq stay pinned at the floor:

    taskset -c $CPU stress --cpu 1 &
    for i in $(seq 10); do
        cat /sys/devices/system/cpu/cpu$CPU/cpufreq/scaling_cur_freq
        sleep 0.5
    done
    turbostat --cpu $CPU --interval 1 --num_iterations 5
Quoted 8 lines

scaling_cur_freq stays at the floor for the whole run, while turbostat's Bzy_MHz confirms the CPU is actually at full turbo.

Refresh the stale sample with one on-demand arch_scale_freq_tick() via IPI before falling back, but only when the target CPU is online and not idle. APERF/MPERF both stop advancing during idle (C1+), so a delta computed over an arbitrarily long stale window still yields a correct busy-time frequency average.

Aside from the fact that sending IPIs to NOHZ_FULL is just plain wrong, this whole thing makes no sense.

When the CPU is isolated, nothing should care about the ratio anyway. Just set the thing to '1' (1024) when the CPU enters NOHZ_FULL mode and ensure it isn't ever modified.

author
#3 lore raw
Quoted 6 lines · Peter Zijlstra

Aside from the fact that sending IPIs to NOHZ_FULL is just plain wrong, this whole thing makes no sense.

When the CPU is isolated, nothing should care about the ratio anyway. Just set the thing to '1' (1024) when the CPU enters NOHZ_FULL mode and ensure it isn't ever modified.

Fair, understood.

For context on why I went looking in the first place: stressing an isolated, nohz_full CPU shows both /proc/cpuinfo's "cpu MHz" and /sys/devices/system/cpu/cpuN/cpufreq/scaling_cur_freq stuck at the P-state floor (e.g. 800MHz) for as long as the CPU stays busy and isolated, while turbostat confirms the hardware is actually running at full turbo (e.g. 3.2GHz) the whole time. Both interfaces go through arch_freq_get_on_cpu(), so whatever affects one affects both.

Getting the exact value would need an on-demand rdmsr on the target CPU - which is what turbostat itself does via /dev/cpu/N/msr's rdmsr_safe_regs_on_cpu(), i.e. the same smp_call_function_single() IPI, just triggered manually by a human running a diagnostic tool instead of sitting behind a commonly-polled sysfs file.

I looked for a way around that: PCU mailbox telemetry can expose a per-core P-state on some Xeon uncores without touching the target CPU, and HFI publishes a shared table too, but that's a per-core performance/efficiency class, not an achieved clock, and PCU access is uncore/generation-specific, not a general mechanism. So as far as I can tell there's no way to get the exact value for an isolated CPU without an IPI of some form.

Thanks,
Jing Wu

#4 lore raw
Quoted 24 lines · Jing Wu
Quoted 6 lines · Peter Zijlstra

Aside from the fact that sending IPIs to NOHZ_FULL is just plain wrong, this whole thing makes no sense.

When the CPU is isolated, nothing should care about the ratio anyway. Just set the thing to '1' (1024) when the CPU enters NOHZ_FULL mode and ensure it isn't ever modified.

Fair, understood.

For context on why I went looking in the first place: stressing an isolated, nohz_full CPU shows both /proc/cpuinfo's "cpu MHz" and /sys/devices/system/cpu/cpuN/cpufreq/scaling_cur_freq stuck at the P-state floor (e.g. 800MHz) for as long as the CPU stays busy and isolated, while turbostat confirms the hardware is actually running at full turbo (e.g. 3.2GHz) the whole time. Both interfaces go through arch_freq_get_on_cpu(), so whatever affects one affects both.

Getting the exact value would need an on-demand rdmsr on the target CPU - which is what turbostat itself does via /dev/cpu/N/msr's rdmsr_safe_regs_on_cpu(), i.e. the same smp_call_function_single() IPI, just triggered manually by a human running a diagnostic tool instead of sitting behind a commonly-polled sysfs file.

Quoted 7 lines

I looked for a way around that: PCU mailbox telemetry can expose a per-core P-state on some Xeon uncores without touching the target CPU, and HFI publishes a shared table too, but that's a per-core performance/efficiency class, not an achieved clock, and PCU access is uncore/generation-specific, not a general mechanism. So as far as I can tell there's no way to get the exact value for an isolated CPU without an IPI of some form.

Oh, you care about the silly sysfs files? I though this was about the scheduler use of aperf/mperf ratio.

Both are driven from the same source, but the scheduler use makes no sense when isolated/NOHZ_FULL. And I would argue that keeping the CPU isolated is more important than having the silly number 'accurate'.

Something like so perhaps?

arch/x86/kernel/cpu/aperfmperf.c +5 -1
index 7ffc78d5ebf2..cedb40e6b5e6 100644
@@ -438,7 +438,8 @@ static void scale_freq_tick(u64 acnt, u64 mcnt)
 {
 	u64 freq_scale, freq_ratio;
 
-	if (!arch_scale_freq_invariant())
+	if (!arch_scale_freq_invariant() ||
+	    !housekeeping_cpu(smp_processor_id(), HK_TYPE_TICK))
 		return;
 
 	if (check_shl_overflow(acnt, 2*SCHED_CAPACITY_SHIFT, &acnt))
@@ -510,6 +511,9 @@ int arch_freq_get_on_cpu(int cpu)
 	unsigned long last;
 	u64 acnt, mcnt;
 
+	if (!housekeeping_cpu(cpu, HK_TYPE_TICK))
+		return -EOPNOTSUPP;
+
 	if (!cpu_feature_enabled(X86_FEATURE_APERFMPERF))
 		goto fallback;
 
author
#5 lore raw
Quoted 6 lines · Peter Zijlstra

Oh, you care about the silly sysfs files? I though this was about the scheduler use of aperf/mperf ratio.

Both are driven from the same source, but the scheduler use makes no sense when isolated/NOHZ_FULL. And I would argue that keeping the CPU isolated is more important than having the silly number 'accurate'.

Yes, sorry for the confusion - it's the sysfs files. To be concrete about why: when someone is chasing a real incident (packet latency in an OVS/DPDK PMD pipeline pinned to one of these cores, say), reading scaling_cur_freq/cpuinfo is one of the first things they check. Seeing the P-state floor while the core is genuinely at full turbo sends that investigation down the wrong path and burns real engineering time.

-EOPNOTSUPP doesn't fix that particular problem: the operator still gets nothing useful from the standard interface and has to fall back to turbostat regardless, so the diagnostic workflow ends up no better off than it is today.

Given that, I'd like to make the case for going back to something closer to the original approach: it's a small, self-contained change, and it reports the actual value instead of an estimate or nothing. The concern was IPIs turning periodic if something scrapes this file on a schedule - but for the isolated cores we run PMD workloads on, nothing polls per-core scaling_cur_freq/cpuinfo on any kind of schedule; it only gets read by a person during an investigation, which is a rare, deliberate action, not unlike running turbostat by hand.

Would you reconsider on that basis, or is there a middle ground you'd be open to?