arm64/cpufreq: report and track frequencies above 4.19 GHz
Subsystems (6)
cover arm64/cpufreq: report and track frequencies above 4.19 GHz
The Snapdragon X2 Elite (Glymur) is the first arm64 laptop part I have seen whose boost OPP, 4723200 kHz, sits above 4194304 kHz. Two independent problems become visible there, both of which make the kernel believe a boosted CPU is running slower than it is.
Patch 1 fixes an overflow in arch_freq_get_on_cpu(): the u64 product of the frequency scale and the reference frequency is truncated to unsigned int before being shifted back down, which wraps for any reference frequency above 2^32 / SCHED_CAPACITY_SCALE = 4194304 kHz.
Patch 2 makes capacity_freq_ref follow the boost state. It is latched once on CPUFREQ_CREATE_POLICY, and boost frequencies are excluded from policy->cpuinfo.max_freq while boost is off, so on a machine that boots with boost disabled it keeps the sustained maximum forever. On arm64 that saturates the AMU frequency scale at SCHED_CAPACITY_SCALE, so the scheduler cannot distinguish a boosted CPU from one at the sustained maximum, and arch_freq_get_on_cpu() cannot report above it.
The order matters: patch 2 is what raises capacity_freq_ref past 4194304 kHz on this machine, so patch 1 has to land with or before it.
Measured on a Lenovo Yoga Slim 7x Gen 11 (Glymur, 4032000 kHz sustained, 4723200 kHz boost), pinning a policy to a single OPP and timing a fixed workload on one of its CPUs:
requested OPP time cpuinfo_avg_freq
---------------------------------------------------------
4032000 kHz 2.011s 4031325 (0.02% low) 4723200 kHz 1.726s 524283 before 4723200 kHz 1.726s 4032000 with patch 1 only 4723200 kHz 1.726s 4718587 with both (0.10% low)
The timings never change: 2.011 / 1.726 = 1.165 against a frequency ratio of 4723200 / 4032000 = 1.171, so the hardware was running at the requested frequency throughout. Only the kernel's view of it was wrong.
Note that cpuinfo_cur_freq still reports 4032000 kHz at the boost OPP on this machine. That is a separate path -- scmi_dvfs_freq_get() asking firmware for the current performance level -- with no clamp in the kernel, and it is not addressed here.
Oleg Keri (2):
arm64: topology: fix arch_freq_get_on_cpu() overflow above 4.19 GHz
cpufreq: update capacity_freq_ref when the boost state changes
arch/arm/include/asm/topology.h | 1 + arch/arm64/include/asm/topology.h | 1 + arch/arm64/kernel/topology.c | 5 +++-- arch/riscv/include/asm/topology.h | 1 + drivers/base/arch_topology.c | 19 +++++++++++++------ drivers/cpufreq/cpufreq.c | 2 ++ include/linux/arch_topology.h | 1 + include/linux/cpufreq.h | 7 +++++++ 8 files changed, 29 insertions(+), 8 deletions(-)
base-commit: 9d80aa4617b32f5054c5aa471d06b66704854935
Signature · 1 line
2.55.0
PATCH 1/2 arm64: topology: fix arch_freq_get_on_cpu() overflow above 4.19 GHz
arch_freq_get_on_cpu() computes the product of the frequency scale and the reference frequency as a u64, but assigns it to an unsigned int before shifting it back down:
freq = scale * arch_scale_freq_ref(cpu); freq >>= SCHED_CAPACITY_SHIFT;
The product is truncated to 32 bits before the shift, so the result wraps once arch_scale_freq_ref() exceeds 2^32 / SCHED_CAPACITY_SCALE, i.e. 4194304 kHz.
On a Snapdragon X2 Elite (Glymur) laptop, whose boost OPP is 4723200 kHz, cpuinfo_avg_freq reports 524283 kHz instead of ~4723200 kHz while the CPU demonstrably runs at the boost frequency: a fixed workload completes in 1.72 s at the 4723200 kHz OPP versus 2.01 s at 4032000 kHz, matching the 1.171 frequency ratio.
Keep the arithmetic in 64 bits until after the shift.
--- arch/arm64/kernel/topology.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
arch/arm64/kernel/topology.c +3 -2
@@ -245,8 +245,9 @@ int arch_freq_get_on_cpu(int cpu)* (see amu_scale_freq_tick for details) */ scale =arch_scale_freq_capacity (cpu); - freq = scale *arch_scale_freq_ref (cpu); - freq >>=SCHED_CAPACITY_SHIFT ; + scale *=arch_scale_freq_ref (cpu); + scale >>=SCHED_CAPACITY_SHIFT ; + freq = scale;return freq; }
Signature · 1 line
2.55.0
Re: [PATCH v1 1/2] arm64: topology: fix arch_freq_get_on_cpu() overflow above 4.19 GHz
Quoted 30 lines · On Sun, 6 Sep 2026 18:37:14 +0200 Oleg Keri <[email protected]> wrote:
arch_freq_get_on_cpu()computes the product of the frequency scale and the reference frequency as a u64, but assigns it to an unsigned int before shifting it back down:freq = scale * arch_scale_freq_ref(cpu); freq >>= SCHED_CAPACITY_SHIFT;The product is truncated to 32 bits before the shift, so the result wraps once
arch_scale_freq_ref()exceeds 2^32 /SCHED_CAPACITY_SCALE, i.e. 4194304 kHz.On a Snapdragon X2 Elite (Glymur) laptop, whose boost OPP is 4723200 kHz,
cpuinfo_avg_freqreports 524283 kHz instead of ~4723200 kHz while the CPU demonstrably runs at the boost frequency: a fixed workload completes in 1.72 s at the 4723200 kHz OPP versus 2.01 s at 4032000 kHz, matching the 1.171 frequency ratio.Keep the arithmetic in 64 bits until after the shift.
Fixes 16d1e27475f6 ("arm64: Provide an AMU-based version of arch_freq_get_on_cpu")Signed-off-by Oleg Keri <[email protected]>--- arch/arm64/kernel/topology.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)arch/arm64/kernel/topology.c +0
-0
arch/arm64/kernel/topology.c int arch_freq_get_on_cpu(int cpu) · line 249
Quoted 7 lines
arch/arm64/kernel/topology.c +2
-2@@ -245,8 +245,9 @@ int arch_freq_get_on_cpu(int cpu)* (see amu_scale_freq_tick for details) */ scale =arch_scale_freq_capacity (cpu); - freq = scale *arch_scale_freq_ref (cpu); - freq >>=SCHED_CAPACITY_SHIFT ; + scale *=arch_scale_freq_ref (cpu); + scale >>=SCHED_CAPACITY_SHIFT ;
Maybe just do it all one line?
freq = (scale * arch_scale_freq_ref(cpu)) >> SCHED_CAPACITY_SHIFT;
I was going to suggest that you roll in the arch_scale_freq_capacity() as well but then we'd need a cast to ensure maths was done in 64 bits and that was uglier.
I think this should end up the same as what you have.
Thanks,
Jonathan
Quoted 4 lines
arch/arm64/kernel/topology.c +1
-0+ freq = scale;return freq; }
Re: [PATCH v1 1/2] arm64: topology: fix arch_freq_get_on_cpu() overflow above 4.19 GHz
Good point - that reads much better than reusing scale as scratch. Taken for v2:
scale = arch_scale_freq_capacity(cpu); freq = (scale * arch_scale_freq_ref(cpu)) >> SCHED_CAPACITY_SHIFT;
Same semantics, and it makes it obvious that both the multiply and the shift stay in 64 bits and only the final value is narrowed - which is the whole point of the fix.
Agreed on leaving arch_scale_freq_capacity() out of the expression too. This file is arm64 only, so unsigned long is 64 bits and it would be safe here, but it stops being self-evident to the reader and the cast that would make it evident is worse than keeping the temporary.
Thanks for the review.
PATCH 2/2 cpufreq: update capacity_freq_ref when the boost state changes
In reply to Oleg Kericapacity_freq_ref is latched from policy->cpuinfo.max_freq by init_cpu_capacity_callback() on CPUFREQ_CREATE_POLICY, and never updated afterwards.
cpufreq_frequency_table_cpuinfo() excludes CPUFREQ_BOOST_FREQ entries while boost is disabled, so on a system that boots with boost off the latched value is the non-boost maximum. Enabling boost later raises policy->cpuinfo.max_freq but leaves capacity_freq_ref behind.
Two things then go wrong on arm64, where the AMU drives frequency invariance. amu_scale_freq_tick() caps the computed scale at SCHED_CAPACITY_SCALE, so a CPU running above capacity_freq_ref saturates at 1024: the scheduler cannot tell a boosted CPU from one at the sustained maximum, and utilisation is underestimated. And arch_freq_get_on_cpu(), which reverses that computation, cannot report more than capacity_freq_ref, so cpuinfo_avg_freq is pinned to the non-boost maximum.
On a Snapdragon X2 Elite (Glymur) laptop with a 4032000 kHz sustained and a 4723200 kHz boost OPP, cpuinfo_avg_freq reads exactly 4032000 while the CPU runs at 4723200; a fixed workload completes in 1.72 s rather than the 2.01 s that frequency would imply.
Factor the update out of init_cpu_capacity_callback() into topology_update_freq_ref() and call it from policy_set_boost(), which is the common path for the global boost knob, the per-policy one, and the CPU online path.
--- arch/arm/include/asm/topology.h | 1 + arch/arm64/include/asm/topology.h | 1 + arch/riscv/include/asm/topology.h | 1 + drivers/base/arch_topology.c | 19 +++++++++++++------ drivers/cpufreq/cpufreq.c | 2 ++ include/linux/arch_topology.h | 1 + include/linux/cpufreq.h | 7 +++++++ 7 files changed, 26 insertions(+), 6 deletions(-)
arch/arm/include/asm/topology.h +1 -0
@@ -11,6 +11,7 @@#ifndef CONFIG_BL_SWITCHER /* Replace task scheduler's default frequency-invariant accounting */ #define arch_set_freq_scale topology_set_freq_scale +#define arch_update_freq_ref topology_update_freq_ref#define arch_scale_freq_capacity topology_get_freq_scale#define arch_scale_freq_invariant topology_scale_freq_invariant#define arch_scale_freq_ref topology_get_freq_ref
arch/arm64/include/asm/topology.h +1 -0
@@ -22,6 +22,7 @@ void update_freq_counters_refs(void);/* Replace task scheduler's default frequency-invariant accounting */ #define arch_scale_freq_tick topology_scale_freq_tick#define arch_set_freq_scale topology_set_freq_scale +#define arch_update_freq_ref topology_update_freq_ref#define arch_scale_freq_capacity topology_get_freq_scale#define arch_scale_freq_invariant topology_scale_freq_invariant#define arch_scale_freq_ref topology_get_freq_ref
arch/riscv/include/asm/topology.h +1 -0
@@ -11,6 +11,7 @@/* Replace task scheduler's default frequency-invariant accounting */ #define arch_scale_freq_tick topology_scale_freq_tick#define arch_set_freq_scale topology_set_freq_scale +#define arch_update_freq_ref topology_update_freq_ref#define arch_scale_freq_capacity topology_get_freq_scale#define arch_scale_freq_invariant topology_scale_freq_invariant#define arch_scale_freq_ref topology_get_freq_ref
drivers/base/arch_topology.c +13 -6
@@ -317,6 +317,18 @@ bool __init topology_parse_cpu_capacity(struct device_node *cpu_node, int cpu)return !ret; } +void topology_update_freq_ref (const struct cpumask *cpus,unsigned int max_freq) +{ +int cpu; + +for_each_cpu (cpu, cpus) { +per_cpu (capacity_freq_ref, cpu) = max_freq; +freq_inv_set_max_ratio (cpu, +per_cpu (capacity_freq_ref, cpu) *HZ_PER_KHZ ); + } +} +EXPORT_SYMBOL_GPL (topology_update_freq_ref); +void __weak freq_inv_set_max_ratio (int cpu,u64 max_rate) { }
@@ -392,7 +404,6 @@ init_cpu_capacity_callback(struct notifier_block *nb, void *data) {struct cpufreq_policy *policy = data; -int cpu;if (val !=CPUFREQ_CREATE_POLICY )return 0 ;
@@ -403,11 +414,7 @@ init_cpu_capacity_callback(struct notifier_block *nb,cpumask_andnot (cpus_to_visit, cpus_to_visit, policy->related_cpus); -for_each_cpu (cpu, policy->related_cpus) { -per_cpu (capacity_freq_ref, cpu) = policy->cpuinfo.max_freq; -freq_inv_set_max_ratio (cpu, -per_cpu (capacity_freq_ref, cpu) *HZ_PER_KHZ ); - } +topology_update_freq_ref (policy->related_cpus, policy->cpuinfo.max_freq);if (cpumask_empty (cpus_to_visit)) {if (raw_capacity) {
drivers/cpufreq/cpufreq.c +2 -0
@@ -594,6 +594,8 @@ static int policy_set_boost(struct cpufreq_policy *policy, bool enable)return ret; } +arch_update_freq_ref (policy->related_cpus, policy->cpuinfo.max_freq); +return 0 ; }
include/linux/arch_topology.h +1 -0
@@ -31,6 +31,7 @@ static inline unsigned long topology_get_freq_scale(int cpu)void topology_set_freq_scale (const struct cpumask *cpus,unsigned long cur_freq,unsigned long max_freq); +void topology_update_freq_ref (const struct cpumask *cpus,unsigned int max_freq);bool topology_scale_freq_invariant (void );enum scale_freq_source {
include/linux/cpufreq.h +7 -0
@@ -1235,6 +1235,13 @@ void arch_set_freq_scale(const struct cpumask *cpus, }#endif +#ifndef arch_update_freq_ref +static __always_inline +void arch_update_freq_ref(const struct cpumask *cpus, unsigned int max_freq) +{ +} +#endif +/* the following are really really optional */ extern struct freq_attr cpufreq_freq_attr_scaling_available_freqs;extern struct freq_attr cpufreq_freq_attr_scaling_boost_freqs;
Signature · 1 line
2.55.0