x86/cpu: Consolidate APERF/MPERF code
Subsystems (4)
cover x86/cpu: Consolidate APERF/MPERF code
APERF/MPERF is utilized in two ways:
1) Ad hoc readout of CPU frequency which requires IPIs
2) Frequency scale calculation for frequency invariant scheduling which reads APERF/MPERF on every tick.
These are completely independent code parts. Eric observed long latencies when reading /proc/cpuinfo which reads out CPU frequency via #1 and proposed to replace the per CPU single IPI with a broadcast IPI.
While this makes the latency smaller, it is not necessary at all because #2 samples APERF/MPERF periodically, except on idle or isolated NOHZ full CPUs which are excluded from IPI already.
It could be argued that not all APERF/MPERF capable systems have the required BIOS information to enable frequency invariance support, but in practice most of them do. So the APERF/MPERF sampling can be made unconditional and just the frequency scale calculation for the scheduler excluded.
The following series consolidates that.
Thanks,
tglx
--- arch/x86/include/asm/cpu.h | 2 arch/x86/include/asm/topology.h | 17 - arch/x86/kernel/acpi/cppc.c | 28 -- arch/x86/kernel/cpu/aperfmperf.c | 474 +++++++++++++++++++++++++++++++-------- arch/x86/kernel/cpu/proc.c | 2 arch/x86/kernel/smpboot.c | 358 ----------------------------- fs/proc/cpuinfo.c | 6 include/linux/cpufreq.h | 1 8 files changed, 405 insertions(+), 483 deletions(-)
PATCH 1/10 x86/aperfmperf: Dont wake idle CPUs in arch_freq_get_on_cpu()
aperfmperf_get_khz() already excludes idle CPUs from APERF/MPERF sampling and that's a reasonable decision. There is no point in sending up to two IPIs to an idle CPU just because someone reads a sysfs file.
--- arch/x86/kernel/cpu/aperfmperf.c | 3 +++ 1 file changed, 3 insertions(+)
arch/x86/kernel/cpu/aperfmperf.c +3 -0
@@ -139,6 +139,9 @@ unsigned int arch_freq_get_on_cpu(int cpif (!housekeeping_cpu (cpu,HK_TYPE_MISC ))return 0 ; +if (rcu_is_idle_cpu (cpu)) +return 0 ; +if (aperfmperf_snapshot_cpu (cpu,ktime_get (), true))return per_cpu (samples.khz, cpu);
Re: [patch 01/10] x86/aperfmperf: Dont wake idle CPUs in arch_freq_get_on_cpu()
Quoted 6 lines · Thomas Gleixner
aperfmperf_get_khz()already excludes idle CPUs from APERF/MPERF sampling and that's a reasonable decision. There is no point in sending up to two IPIs to an idle CPU just because someone reads a sysfs file.Signed-off-by Thomas Gleixner <[email protected]>
Quoted 17 lines
--- arch/x86/kernel/cpu/aperfmperf.c | 3 +++ 1 file changed, 3 insertions(+)arch/x86/kernel/cpu/aperfmperf.c +3
-0@@ -139,6 +139,9 @@ unsigned int arch_freq_get_on_cpu(int cpif (!housekeeping_cpu (cpu,HK_TYPE_MISC ))return 0 ; +if (rcu_is_idle_cpu (cpu)) +return 0 ; +if (aperfmperf_snapshot_cpu (cpu,ktime_get (), true))return per_cpu (samples.khz, cpu);
[tip: x86/cleanups] x86/aperfmperf: Dont wake idle CPUs in arch_freq_get_on_cpu()
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: 6d108c96bf23598cc3b4f91d60e9b7694abcd2a7 Gitweb: https://git.kernel.org/tip/6d108c96bf23598cc3b4f91d60e9b7694abcd2a7 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 15 Apr 2022 21:19:50 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 15:51:08 +02:00
x86/aperfmperf: Dont wake idle CPUs in arch_freq_get_on_cpu()
aperfmperf_get_khz() already excludes idle CPUs from APERF/MPERF sampling and that's a reasonable decision. There is no point in sending up to two IPIs to an idle CPU just because someone reads a sysfs file.
--- arch/x86/kernel/cpu/aperfmperf.c | 3 +++ 1 file changed, 3 insertions(+)
arch/x86/kernel/cpu/aperfmperf.c +3 -0
@@ -139,6 +139,9 @@ unsigned int arch_freq_get_on_cpu(int cpu)if (!housekeeping_cpu (cpu,HK_TYPE_MISC ))return 0 ; +if (rcu_is_idle_cpu (cpu)) +return 0 ; +if (aperfmperf_snapshot_cpu (cpu,ktime_get (), true))return per_cpu (samples.khz, cpu);
PATCH 2/10 x86/smp: Move APERF/MPERF code where it belongs
In reply to Thomas Gleixneras this can share code with the preexisting APERF/MPERF code.
No functional change.
--- arch/x86/kernel/cpu/aperfmperf.c | 366 ++++++++++++++++++++++++++++++++++++++- arch/x86/kernel/smpboot.c | 355 ------------------------------------- 2 files changed, 362 insertions(+), 359 deletions(-)
arch/x86/kernel/cpu/aperfmperf.c +362 -4
@@ -6,15 +6,19 @@* Copyright (C) 2017 Intel Corp. * Author: Len Brown <[email protected]> */ - +#include <linux/cpufreq.h> #include <linux/delay.h> #include <linux/ktime.h> #include <linux/math64.h> #include <linux/percpu.h> -#include <linux/cpufreq.h> -#include <linux/smp.h> -#include <linux/sched/isolation.h> #include <linux/rcupdate.h> +#include <linux/sched/isolation.h> +#include <linux/sched/topology.h> +#include <linux/smp.h> +#include <linux/syscore_ops.h> + +#include <asm/cpu_device_id.h> +#include <asm/intel-family.h> #include "cpu.h"
@@ -152,3 +156,357 @@ unsigned int arch_freq_get_on_cpu(int cpreturn per_cpu (samples.khz, cpu); } + +#if defined(CONFIG_X86_64 ) && defined(CONFIG_SMP ) +/* +* APERF/MPERF frequency ratio computation. +* +* The scheduler wants to do frequency invariant accounting and needs a <1 +* ratio to account for the 'current' frequency, corresponding to +* freq_curr / freq_max. +* +* Since the frequency freq_curr on x86 is controlled by micro-controller and +* our P-state setting is little more than a request/hint, we need to observe +* the effective frequency 'BusyMHz', i.e. the average frequency over a time +* interval after discarding idle time. This is given by: +* +* BusyMHz = delta_APERF / delta_MPERF * freq_base +* +* where freq_base is the max non-turbo P-state. +* +* The freq_max term has to be set to a somewhat arbitrary value, because we +* can't know which turbo states will be available at a given point in time: +* it all depends on the thermal headroom of the entire package. We set it to +* the turbo level with 4 cores active. +* +* Benchmarks show that's a good compromise between the 1C turbo ratio +* (freq_curr/freq_max would rarely reach 1) and something close to freq_base, +* which would ignore the entire turbo range (a conspicuous part, making +* freq_curr/freq_max always maxed out). +* +* An exception to the heuristic above is the Atom uarch, where we choose the +* highest turbo level for freq_max since Atom's are generally oriented towards +* power efficiency. +* +* Setting freq_max to anything less than the 1C turbo ratio makes the ratio +* freq_curr / freq_max to eventually grow >1, in which case we clip it to 1. +*/ + +DEFINE_STATIC_KEY_FALSE (arch_scale_freq_key); + +static DEFINE_PER_CPU(u64, arch_prev_aperf); +static DEFINE_PER_CPU(u64, arch_prev_mperf); +static u64 arch_turbo_freq_ratio =SCHED_CAPACITY_SCALE ; +static u64 arch_max_freq_ratio =SCHED_CAPACITY_SCALE ; + +void arch_set_max_freq_ratio (bool turbo_disabled) +{ + arch_max_freq_ratio = turbo_disabled ?SCHED_CAPACITY_SCALE : + arch_turbo_freq_ratio; +} +EXPORT_SYMBOL_GPL (arch_set_max_freq_ratio); + +static bool turbo_disabled (void ) +{ +u64 misc_en; +int err; + + err =rdmsrl_safe (MSR_IA32_MISC_ENABLE , &misc_en); +if (err) +return false; + +return (misc_en &MSR_IA32_MISC_ENABLE_TURBO_DISABLE ); +} + +static bool slv_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq) +{ +int err; + + err =rdmsrl_safe (MSR_ATOM_CORE_RATIOS , base_freq); +if (err) +return false; + + err =rdmsrl_safe (MSR_ATOM_CORE_TURBO_RATIOS , turbo_freq); +if (err) +return false; + + *base_freq = (*base_freq >>16 ) &0x3F ;/* max P state */ + *turbo_freq = *turbo_freq &0x3F ;/* 1C turbo */ + +return true; +} + +#define X86_MATCH (model) \ + X86_MATCH_VENDOR_FAM_MODEL_FEATURE(INTEL, 6, \ + INTEL_FAM6_##model, X86_FEATURE_APERFMPERF, NULL) + +static const struct x86_cpu_id has_knl_turbo_ratio_limits[] = { +X86_MATCH (XEON_PHI_KNL ), +X86_MATCH (XEON_PHI_KNM ), + {} +}; + +static const struct x86_cpu_id has_skx_turbo_ratio_limits[] = { +X86_MATCH (SKYLAKE_X ), + {} +}; + +static const struct x86_cpu_id has_glm_turbo_ratio_limits[] = { +X86_MATCH (ATOM_GOLDMONT ), +X86_MATCH (ATOM_GOLDMONT_D ), +X86_MATCH (ATOM_GOLDMONT_PLUS ), + {} +}; + +static bool knl_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq, +int num_delta_fratio) +{ +int fratio, delta_fratio, found; +int err, i; +u64 msr; + + err =rdmsrl_safe (MSR_PLATFORM_INFO , base_freq); +if (err) +return false; + + *base_freq = (*base_freq >>8 ) &0xFF ;/* max P state */ + + err =rdmsrl_safe (MSR_TURBO_RATIO_LIMIT , &msr); +if (err) +return false; + + fratio = (msr >>8 ) &0xFF ; + i =16 ; + found =0 ; +do { +if (found >= num_delta_fratio) { + *turbo_freq = fratio; +return true; + } + + delta_fratio = (msr >> (i +5 )) &0x7 ; + +if (delta_fratio) { + found +=1 ; + fratio -= delta_fratio; + } + + i +=8 ; + }while (i <64 ); + +return true; +} + +static bool skx_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq,int size) +{ +u64 ratios, counts; +u32 group_size; +int err, i; + + err =rdmsrl_safe (MSR_PLATFORM_INFO , base_freq); +if (err) +return false; + + *base_freq = (*base_freq >>8 ) &0xFF ;/* max P state */ + + err =rdmsrl_safe (MSR_TURBO_RATIO_LIMIT , &ratios); +if (err) +return false; + + err =rdmsrl_safe (MSR_TURBO_RATIO_LIMIT1 , &counts); +if (err) +return false; + +for (i =0 ; i <64 ; i +=8 ) { + group_size = (counts >> i) &0xFF ; +if (group_size >= size) { + *turbo_freq = (ratios >> i) &0xFF ; +return true; + } + } + +return false; +} + +static bool core_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq) +{ +u64 msr; +int err; + + err =rdmsrl_safe (MSR_PLATFORM_INFO , base_freq); +if (err) +return false; + + err =rdmsrl_safe (MSR_TURBO_RATIO_LIMIT , &msr); +if (err) +return false; + + *base_freq = (*base_freq >>8 ) &0xFF ;/* max P state */ + *turbo_freq = (msr >>24 ) &0xFF ;/* 4C turbo */ + +/* The CPU may have less than 4 cores */ +if (!*turbo_freq) + *turbo_freq = msr &0xFF ;/* 1C turbo */ + +return true; +} + +static bool intel_set_max_freq_ratio (void ) +{ +u64 base_freq, turbo_freq; +u64 turbo_ratio; + +if (slv_set_max_freq_ratio (&base_freq, &turbo_freq)) + goto out; + +if (x86_match_cpu (has_glm_turbo_ratio_limits) && +skx_set_max_freq_ratio (&base_freq, &turbo_freq,1 )) + goto out; + +if (x86_match_cpu (has_knl_turbo_ratio_limits) && +knl_set_max_freq_ratio (&base_freq, &turbo_freq,1 )) + goto out; + +if (x86_match_cpu (has_skx_turbo_ratio_limits) && +skx_set_max_freq_ratio (&base_freq, &turbo_freq,4 )) + goto out; + +if (core_set_max_freq_ratio (&base_freq, &turbo_freq)) + goto out; + +return false; + +out: +/* +* Some hypervisors advertise X86_FEATURE_APERFMPERF +* but then fill all MSR's with zeroes. +* Some CPUs have turbo boost but don't declare any turbo ratio +* in MSR_TURBO_RATIO_LIMIT. +*/ +if (!base_freq || !turbo_freq) { +pr_debug ("Couldn't determine cpu base or turbo frequency, necessary for scale-invariant accounting.\n" ); +return false; + } + + turbo_ratio =div_u64 (turbo_freq *SCHED_CAPACITY_SCALE , base_freq); +if (!turbo_ratio) { +pr_debug ("Non-zero turbo and base frequencies led to a 0 ratio.\n" ); +return false; + } + + arch_turbo_freq_ratio = turbo_ratio; +arch_set_max_freq_ratio (turbo_disabled ()); + +return true; +} + +static void init_counter_refs (void ) +{ +u64 aperf, mperf; + +rdmsrl (MSR_IA32_APERF , aperf); +rdmsrl (MSR_IA32_MPERF , mperf); + +this_cpu_write (arch_prev_aperf, aperf); +this_cpu_write (arch_prev_mperf, mperf); +} + +#ifdef CONFIG_PM_SLEEP +static struct syscore_ops freq_invariance_syscore_ops = { + .resume = init_counter_refs, +}; + +static void register_freq_invariance_syscore_ops (void ) +{ +/* Bail out if registered already. */ +if (freq_invariance_syscore_ops.node.prev) +return ; + +register_syscore_ops (&freq_invariance_syscore_ops); +} +#else +static inline void register_freq_invariance_syscore_ops (void ) {} +#endif + +void init_freq_invariance (bool secondary,bool cppc_ready) +{ +bool ret = false; + +if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) +return ; + +if (secondary) { +if (static_branch_likely (&arch_scale_freq_key)) { +init_counter_refs (); + } +return ; + } + +if (boot_cpu_data.x86_vendor ==X86_VENDOR_INTEL ) + ret =intel_set_max_freq_ratio (); +else if (boot_cpu_data.x86_vendor ==X86_VENDOR_AMD ) { +if (!cppc_ready) { +return ; + } + ret =amd_set_max_freq_ratio (&arch_turbo_freq_ratio); + } + +if (ret) { +init_counter_refs (); +static_branch_enable (&arch_scale_freq_key); +register_freq_invariance_syscore_ops (); +pr_info ("Estimated ratio of average max frequency by base frequency (times 1024): %llu\n" , arch_max_freq_ratio); + }else { +pr_debug ("Couldn't determine max cpu frequency, necessary for scale-invariant accounting.\n" ); + } +} + +static void disable_freq_invariance_workfn (struct work_struct *work) +{ +static_branch_disable (&arch_scale_freq_key); +} + +static DECLARE_WORK(disable_freq_invariance_work, + disable_freq_invariance_workfn); + +DEFINE_PER_CPU (unsigned long ,arch_freq_scale ) =SCHED_CAPACITY_SCALE ; + +void arch_scale_freq_tick (void ) +{ +u64 freq_scale; +u64 aperf, mperf; +u64 acnt, mcnt; + +if (!arch_scale_freq_invariant ()) +return ; + +rdmsrl (MSR_IA32_APERF , aperf); +rdmsrl (MSR_IA32_MPERF , mperf); + + acnt = aperf -this_cpu_read (arch_prev_aperf); + mcnt = mperf -this_cpu_read (arch_prev_mperf); + +this_cpu_write (arch_prev_aperf, aperf); +this_cpu_write (arch_prev_mperf, mperf); + +if (check_shl_overflow (acnt,2 *SCHED_CAPACITY_SHIFT , &acnt)) + goto error; + +if (check_mul_overflow (mcnt, arch_max_freq_ratio, &mcnt) || !mcnt) + goto error; + + freq_scale =div64_u64 (acnt, mcnt); +if (!freq_scale) + goto error; + +if (freq_scale >SCHED_CAPACITY_SCALE ) + freq_scale =SCHED_CAPACITY_SCALE ; + +this_cpu_write (arch_freq_scale, freq_scale); +return ; + +error: +pr_warn ("Scheduler frequency invariance went wobbly, disabling!\n" ); +schedule_work (&disable_freq_invariance_work); +} +#endif /* CONFIG_X86_64 && CONFIG_SMP */
arch/x86/kernel/smpboot.c +0 -355
364 lines not on this page · raw message
Re: [patch 02/10] x86/smp: Move APERF/MPERF code where it belongs
Quoted 6 lines · Thomas Gleixner
as this can share code with the preexisting APERF/MPERF code.
No functional change.
Signed-off-by Thomas Gleixner <[email protected]>
Quoted 758 lines
--- arch/x86/kernel/cpu/aperfmperf.c | 366 ++++++++++++++++++++++++++++++++++++++- arch/x86/kernel/smpboot.c | 355 ------------------------------------- 2 files changed, 362 insertions(+), 359 deletions(-)arch/x86/kernel/cpu/aperfmperf.c +362
-4@@ -6,15 +6,19 @@* Copyright (C) 2017 Intel Corp. * Author: Len Brown <[email protected]> */ - +#include <linux/cpufreq.h> #include <linux/delay.h> #include <linux/ktime.h> #include <linux/math64.h> #include <linux/percpu.h> -#include <linux/cpufreq.h> -#include <linux/smp.h> -#include <linux/sched/isolation.h> #include <linux/rcupdate.h> +#include <linux/sched/isolation.h> +#include <linux/sched/topology.h> +#include <linux/smp.h> +#include <linux/syscore_ops.h> + +#include <asm/cpu_device_id.h> +#include <asm/intel-family.h> #include "cpu.h" @@ -152,3 +156,357 @@ unsigned int arch_freq_get_on_cpu(int cpreturn per_cpu (samples.khz, cpu); } + +#if defined(CONFIG_X86_64 ) && defined(CONFIG_SMP ) +/* +* APERF/MPERF frequency ratio computation. +* +* The scheduler wants to do frequency invariant accounting and needs a <1 +* ratio to account for the 'current' frequency, corresponding to +* freq_curr / freq_max. +* +* Since the frequency freq_curr on x86 is controlled by micro-controller and +* our P-state setting is little more than a request/hint, we need to observe +* the effective frequency 'BusyMHz', i.e. the average frequency over a time +* interval after discarding idle time. This is given by: +* +* BusyMHz = delta_APERF / delta_MPERF * freq_base +* +* where freq_base is the max non-turbo P-state. +* +* The freq_max term has to be set to a somewhat arbitrary value, because we +* can't know which turbo states will be available at a given point in time: +* it all depends on the thermal headroom of the entire package. We set it to +* the turbo level with 4 cores active. +* +* Benchmarks show that's a good compromise between the 1C turbo ratio +* (freq_curr/freq_max would rarely reach 1) and something close to freq_base, +* which would ignore the entire turbo range (a conspicuous part, making +* freq_curr/freq_max always maxed out). +* +* An exception to the heuristic above is the Atom uarch, where we choose the +* highest turbo level for freq_max since Atom's are generally oriented towards +* power efficiency. +* +* Setting freq_max to anything less than the 1C turbo ratio makes the ratio +* freq_curr / freq_max to eventually grow >1, in which case we clip it to 1. +*/ + +DEFINE_STATIC_KEY_FALSE (arch_scale_freq_key); + +static DEFINE_PER_CPU(u64, arch_prev_aperf); +static DEFINE_PER_CPU(u64, arch_prev_mperf); +static u64 arch_turbo_freq_ratio =SCHED_CAPACITY_SCALE ; +static u64 arch_max_freq_ratio =SCHED_CAPACITY_SCALE ; + +void arch_set_max_freq_ratio (bool turbo_disabled) +{ + arch_max_freq_ratio = turbo_disabled ?SCHED_CAPACITY_SCALE : + arch_turbo_freq_ratio; +} +EXPORT_SYMBOL_GPL (arch_set_max_freq_ratio); + +static bool turbo_disabled (void ) +{ +u64 misc_en; +int err; + + err =rdmsrl_safe (MSR_IA32_MISC_ENABLE , &misc_en); +if (err) +return false; + +return (misc_en &MSR_IA32_MISC_ENABLE_TURBO_DISABLE ); +} + +static bool slv_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq) +{ +int err; + + err =rdmsrl_safe (MSR_ATOM_CORE_RATIOS , base_freq); +if (err) +return false; + + err =rdmsrl_safe (MSR_ATOM_CORE_TURBO_RATIOS , turbo_freq); +if (err) +return false; + + *base_freq = (*base_freq >>16 ) &0x3F ;/* max P state */ + *turbo_freq = *turbo_freq &0x3F ;/* 1C turbo */ + +return true; +} + +#define X86_MATCH (model) \ + X86_MATCH_VENDOR_FAM_MODEL_FEATURE(INTEL, 6, \ + INTEL_FAM6_##model, X86_FEATURE_APERFMPERF, NULL) + +static const struct x86_cpu_id has_knl_turbo_ratio_limits[] = { +X86_MATCH (XEON_PHI_KNL ), +X86_MATCH (XEON_PHI_KNM ), + {} +}; + +static const struct x86_cpu_id has_skx_turbo_ratio_limits[] = { +X86_MATCH (SKYLAKE_X ), + {} +}; + +static const struct x86_cpu_id has_glm_turbo_ratio_limits[] = { +X86_MATCH (ATOM_GOLDMONT ), +X86_MATCH (ATOM_GOLDMONT_D ), +X86_MATCH (ATOM_GOLDMONT_PLUS ), + {} +}; + +static bool knl_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq, +int num_delta_fratio) +{ +int fratio, delta_fratio, found; +int err, i; +u64 msr; + + err =rdmsrl_safe (MSR_PLATFORM_INFO , base_freq); +if (err) +return false; + + *base_freq = (*base_freq >>8 ) &0xFF ;/* max P state */ + + err =rdmsrl_safe (MSR_TURBO_RATIO_LIMIT , &msr); +if (err) +return false; + + fratio = (msr >>8 ) &0xFF ; + i =16 ; + found =0 ; +do { +if (found >= num_delta_fratio) { + *turbo_freq = fratio; +return true; + } + + delta_fratio = (msr >> (i +5 )) &0x7 ; + +if (delta_fratio) { + found +=1 ; + fratio -= delta_fratio; + } + + i +=8 ; + }while (i <64 ); + +return true; +} + +static bool skx_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq,int size) +{ +u64 ratios, counts; +u32 group_size; +int err, i; + + err =rdmsrl_safe (MSR_PLATFORM_INFO , base_freq); +if (err) +return false; + + *base_freq = (*base_freq >>8 ) &0xFF ;/* max P state */ + + err =rdmsrl_safe (MSR_TURBO_RATIO_LIMIT , &ratios); +if (err) +return false; + + err =rdmsrl_safe (MSR_TURBO_RATIO_LIMIT1 , &counts); +if (err) +return false; + +for (i =0 ; i <64 ; i +=8 ) { + group_size = (counts >> i) &0xFF ; +if (group_size >= size) { + *turbo_freq = (ratios >> i) &0xFF ; +return true; + } + } + +return false; +} + +static bool core_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq) +{ +u64 msr; +int err; + + err =rdmsrl_safe (MSR_PLATFORM_INFO , base_freq); +if (err) +return false; + + err =rdmsrl_safe (MSR_TURBO_RATIO_LIMIT , &msr); +if (err) +return false; + + *base_freq = (*base_freq >>8 ) &0xFF ;/* max P state */ + *turbo_freq = (msr >>24 ) &0xFF ;/* 4C turbo */ + +/* The CPU may have less than 4 cores */ +if (!*turbo_freq) + *turbo_freq = msr &0xFF ;/* 1C turbo */ + +return true; +} + +static bool intel_set_max_freq_ratio (void ) +{ +u64 base_freq, turbo_freq; +u64 turbo_ratio; + +if (slv_set_max_freq_ratio (&base_freq, &turbo_freq)) + goto out; + +if (x86_match_cpu (has_glm_turbo_ratio_limits) && +skx_set_max_freq_ratio (&base_freq, &turbo_freq,1 )) + goto out; + +if (x86_match_cpu (has_knl_turbo_ratio_limits) && +knl_set_max_freq_ratio (&base_freq, &turbo_freq,1 )) + goto out; + +if (x86_match_cpu (has_skx_turbo_ratio_limits) && +skx_set_max_freq_ratio (&base_freq, &turbo_freq,4 )) + goto out; + +if (core_set_max_freq_ratio (&base_freq, &turbo_freq)) + goto out; + +return false; + +out: +/* +* Some hypervisors advertise X86_FEATURE_APERFMPERF +* but then fill all MSR's with zeroes. +* Some CPUs have turbo boost but don't declare any turbo ratio +* in MSR_TURBO_RATIO_LIMIT. +*/ +if (!base_freq || !turbo_freq) { +pr_debug ("Couldn't determine cpu base or turbo frequency, necessary for scale-invariant accounting.\n" ); +return false; + } + + turbo_ratio =div_u64 (turbo_freq *SCHED_CAPACITY_SCALE , base_freq); +if (!turbo_ratio) { +pr_debug ("Non-zero turbo and base frequencies led to a 0 ratio.\n" ); +return false; + } + + arch_turbo_freq_ratio = turbo_ratio; +arch_set_max_freq_ratio (turbo_disabled ()); + +return true; +} + +static void init_counter_refs (void ) +{ +u64 aperf, mperf; + +rdmsrl (MSR_IA32_APERF , aperf); +rdmsrl (MSR_IA32_MPERF , mperf); + +this_cpu_write (arch_prev_aperf, aperf); +this_cpu_write (arch_prev_mperf, mperf); +} + +#ifdef CONFIG_PM_SLEEP +static struct syscore_ops freq_invariance_syscore_ops = { + .resume = init_counter_refs, +}; + +static void register_freq_invariance_syscore_ops (void ) +{ +/* Bail out if registered already. */ +if (freq_invariance_syscore_ops.node.prev) +return ; + +register_syscore_ops (&freq_invariance_syscore_ops); +} +#else +static inline void register_freq_invariance_syscore_ops (void ) {} +#endif + +void init_freq_invariance (bool secondary,bool cppc_ready) +{ +bool ret = false; + +if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) +return ; + +if (secondary) { +if (static_branch_likely (&arch_scale_freq_key)) { +init_counter_refs (); + } +return ; + } + +if (boot_cpu_data.x86_vendor ==X86_VENDOR_INTEL ) + ret =intel_set_max_freq_ratio (); +else if (boot_cpu_data.x86_vendor ==X86_VENDOR_AMD ) { +if (!cppc_ready) { +return ; + } + ret =amd_set_max_freq_ratio (&arch_turbo_freq_ratio); + } + +if (ret) { +init_counter_refs (); +static_branch_enable (&arch_scale_freq_key); +register_freq_invariance_syscore_ops (); +pr_info ("Estimated ratio of average max frequency by base frequency (times 1024): %llu\n" , arch_max_freq_ratio); + }else { +pr_debug ("Couldn't determine max cpu frequency, necessary for scale-invariant accounting.\n" ); + } +} + +static void disable_freq_invariance_workfn (struct work_struct *work) +{ +static_branch_disable (&arch_scale_freq_key); +} + +static DECLARE_WORK(disable_freq_invariance_work, + disable_freq_invariance_workfn); + +DEFINE_PER_CPU (unsigned long ,arch_freq_scale ) =SCHED_CAPACITY_SCALE ; + +void arch_scale_freq_tick (void ) +{ +u64 freq_scale; +u64 aperf, mperf; +u64 acnt, mcnt; + +if (!arch_scale_freq_invariant ()) +return ; + +rdmsrl (MSR_IA32_APERF , aperf); +rdmsrl (MSR_IA32_MPERF , mperf); + + acnt = aperf -this_cpu_read (arch_prev_aperf); + mcnt = mperf -this_cpu_read (arch_prev_mperf); + +this_cpu_write (arch_prev_aperf, aperf); +this_cpu_write (arch_prev_mperf, mperf); + +if (check_shl_overflow (acnt,2 *SCHED_CAPACITY_SHIFT , &acnt)) + goto error; + +if (check_mul_overflow (mcnt, arch_max_freq_ratio, &mcnt) || !mcnt) + goto error; + + freq_scale =div64_u64 (acnt, mcnt); +if (!freq_scale) + goto error; + +if (freq_scale >SCHED_CAPACITY_SCALE ) + freq_scale =SCHED_CAPACITY_SCALE ; + +this_cpu_write (arch_freq_scale, freq_scale); +return ; + +error: +pr_warn ("Scheduler frequency invariance went wobbly, disabling!\n" ); +schedule_work (&disable_freq_invariance_work); +} +#endif /* CONFIG_X86_64 && CONFIG_SMP */ arch/x86/kernel/smpboot.c +0
-355@@ -56,7 +56,6 @@#include <linux/numa.h> #include <linux/pgtable.h> #include <linux/overflow.h> -#include <linux/syscore_ops.h> #include <asm/acpi.h> #include <asm/desc.h> @@ -1847,357 +1846,3 @@ void native_play_dead(void) }#endif - -#ifdef CONFIG_X86_64 -/* -* APERF/MPERF frequency ratio computation. -* -* The scheduler wants to do frequency invariant accounting and needs a <1 -* ratio to account for the 'current' frequency, corresponding to -* freq_curr / freq_max. -* -* Since the frequency freq_curr on x86 is controlled by micro-controller and -* our P-state setting is little more than a request/hint, we need to observe -* the effective frequency 'BusyMHz', i.e. the average frequency over a time -* interval after discarding idle time. This is given by: -* -* BusyMHz = delta_APERF / delta_MPERF * freq_base -* -* where freq_base is the max non-turbo P-state. -* -* The freq_max term has to be set to a somewhat arbitrary value, because we -* can't know which turbo states will be available at a given point in time: -* it all depends on the thermal headroom of the entire package. We set it to -* the turbo level with 4 cores active. -* -* Benchmarks show that's a good compromise between the 1C turbo ratio -* (freq_curr/freq_max would rarely reach 1) and something close to freq_base, -* which would ignore the entire turbo range (a conspicuous part, making -* freq_curr/freq_max always maxed out). -* -* An exception to the heuristic above is the Atom uarch, where we choose the -* highest turbo level for freq_max since Atom's are generally oriented towards -* power efficiency. -* -* Setting freq_max to anything less than the 1C turbo ratio makes the ratio -* freq_curr / freq_max to eventually grow >1, in which case we clip it to 1. -*/ - -DEFINE_STATIC_KEY_FALSE (arch_scale_freq_key); - -static DEFINE_PER_CPU(u64, arch_prev_aperf); -static DEFINE_PER_CPU(u64, arch_prev_mperf); -static u64 arch_turbo_freq_ratio =SCHED_CAPACITY_SCALE ; -static u64 arch_max_freq_ratio =SCHED_CAPACITY_SCALE ; - -void arch_set_max_freq_ratio (bool turbo_disabled) -{ - arch_max_freq_ratio = turbo_disabled ?SCHED_CAPACITY_SCALE : - arch_turbo_freq_ratio; -} -EXPORT_SYMBOL_GPL (arch_set_max_freq_ratio); - -static bool turbo_disabled (void ) -{ -u64 misc_en; -int err; - - err =rdmsrl_safe (MSR_IA32_MISC_ENABLE , &misc_en); -if (err) -return false; - -return (misc_en &MSR_IA32_MISC_ENABLE_TURBO_DISABLE ); -} - -static bool slv_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq) -{ -int err; - - err =rdmsrl_safe (MSR_ATOM_CORE_RATIOS , base_freq); -if (err) -return false; - - err =rdmsrl_safe (MSR_ATOM_CORE_TURBO_RATIOS , turbo_freq); -if (err) -return false; - - *base_freq = (*base_freq >>16 ) &0x3F ;/* max P state */ - *turbo_freq = *turbo_freq &0x3F ;/* 1C turbo */ - -return true; -} - -#define X86_MATCH (model) \ - X86_MATCH_VENDOR_FAM_MODEL_FEATURE(INTEL, 6, \ - INTEL_FAM6_##model, X86_FEATURE_APERFMPERF, NULL) - -static const struct x86_cpu_id has_knl_turbo_ratio_limits[] = { -X86_MATCH (XEON_PHI_KNL ), -X86_MATCH (XEON_PHI_KNM ), - {} -}; - -static const struct x86_cpu_id has_skx_turbo_ratio_limits[] = { -X86_MATCH (SKYLAKE_X ), - {} -}; - -static const struct x86_cpu_id has_glm_turbo_ratio_limits[] = { -X86_MATCH (ATOM_GOLDMONT ), -X86_MATCH (ATOM_GOLDMONT_D ), -X86_MATCH (ATOM_GOLDMONT_PLUS ), - {} -}; - -static bool knl_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq, -int num_delta_fratio) -{ -int fratio, delta_fratio, found; -int err, i; -u64 msr; - - err =rdmsrl_safe (MSR_PLATFORM_INFO , base_freq); -if (err) -return false; - - *base_freq = (*base_freq >>8 ) &0xFF ;/* max P state */ - - err =rdmsrl_safe (MSR_TURBO_RATIO_LIMIT , &msr); -if (err) -return false; - - fratio = (msr >>8 ) &0xFF ; - i =16 ; - found =0 ; -do { -if (found >= num_delta_fratio) { - *turbo_freq = fratio; -return true; - } - - delta_fratio = (msr >> (i +5 )) &0x7 ; - -if (delta_fratio) { - found +=1 ; - fratio -= delta_fratio; - } - - i +=8 ; - }while (i <64 ); - -return true; -} - -static bool skx_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq,int size) -{ -u64 ratios, counts; -u32 group_size; -int err, i; - - err =rdmsrl_safe (MSR_PLATFORM_INFO , base_freq); -if (err) -return false; - - *base_freq = (*base_freq >>8 ) &0xFF ;/* max P state */ - - err =rdmsrl_safe (MSR_TURBO_RATIO_LIMIT , &ratios); -if (err) -return false; - - err =rdmsrl_safe (MSR_TURBO_RATIO_LIMIT1 , &counts); -if (err) -return false; - -for (i =0 ; i <64 ; i +=8 ) { - group_size = (counts >> i) &0xFF ; -if (group_size >= size) { - *turbo_freq = (ratios >> i) &0xFF ; -return true; - } - } - -return false; -} - -static bool core_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq) -{ -u64 msr; -int err; - - err =rdmsrl_safe (MSR_PLATFORM_INFO , base_freq); -if (err) -return false; - - err =rdmsrl_safe (MSR_TURBO_RATIO_LIMIT , &msr); -if (err) -return false; - - *base_freq = (*base_freq >>8 ) &0xFF ;/* max P state */ - *turbo_freq = (msr >>24 ) &0xFF ;/* 4C turbo */ - -/* The CPU may have less than 4 cores */ -if (!*turbo_freq) - *turbo_freq = msr &0xFF ;/* 1C turbo */ - -return true; -} - -static bool intel_set_max_freq_ratio (void ) -{ -u64 base_freq, turbo_freq; -u64 turbo_ratio; - -if (slv_set_max_freq_ratio (&base_freq, &turbo_freq)) - goto out; - -if (x86_match_cpu (has_glm_turbo_ratio_limits) && -skx_set_max_freq_ratio (&base_freq, &turbo_freq,1 )) - goto out; - -if (x86_match_cpu (has_knl_turbo_ratio_limits) && -knl_set_max_freq_ratio (&base_freq, &turbo_freq,1 )) - goto out; - -if (x86_match_cpu (has_skx_turbo_ratio_limits) && -skx_set_max_freq_ratio (&base_freq, &turbo_freq,4 )) - goto out; - -if (core_set_max_freq_ratio (&base_freq, &turbo_freq)) - goto out; - -return false; - -out: -/* -* Some hypervisors advertise X86_FEATURE_APERFMPERF -* but then fill all MSR's with zeroes. -* Some CPUs have turbo boost but don't declare any turbo ratio -* in MSR_TURBO_RATIO_LIMIT. -*/ -if (!base_freq || !turbo_freq) { -pr_debug ("Couldn't determine cpu base or turbo frequency, necessary for scale-invariant accounting.\n" ); -return false; - } - - turbo_ratio =div_u64 (turbo_freq *SCHED_CAPACITY_SCALE , base_freq); -if (!turbo_ratio) { -pr_debug ("Non-zero turbo and base frequencies led to a 0 ratio.\n" ); -return false; - } - - arch_turbo_freq_ratio = turbo_ratio; -arch_set_max_freq_ratio (turbo_disabled ()); - -return true; -} - -static void init_counter_refs (void ) -{ -u64 aperf, mperf; - -rdmsrl (MSR_IA32_APERF , aperf); -rdmsrl (MSR_IA32_MPERF , mperf); - -this_cpu_write (arch_prev_aperf, aperf); -this_cpu_write (arch_prev_mperf, mperf); -} - -#ifdef CONFIG_PM_SLEEP -static struct syscore_ops freq_invariance_syscore_ops = { - .resume = init_counter_refs, -}; - -static void register_freq_invariance_syscore_ops (void ) -{ -/* Bail out if registered already. */ -if (freq_invariance_syscore_ops.node.prev) -return ; - -register_syscore_ops (&freq_invariance_syscore_ops); -} -#else -static inline void register_freq_invariance_syscore_ops (void ) {} -#endif - -void init_freq_invariance (bool secondary,bool cppc_ready) -{ -bool ret = false; - -if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) -return ; - -if (secondary) { -if (static_branch_likely (&arch_scale_freq_key)) { -init_counter_refs (); - } -return ; - } - -if (boot_cpu_data.x86_vendor ==X86_VENDOR_INTEL ) - ret =intel_set_max_freq_ratio (); -else if (boot_cpu_data.x86_vendor ==X86_VENDOR_AMD ) { -if (!cppc_ready) { -return ; - } - ret =amd_set_max_freq_ratio (&arch_turbo_freq_ratio); - } - -if (ret) { -init_counter_refs (); -static_branch_enable (&arch_scale_freq_key); -register_freq_invariance_syscore_ops (); -pr_info ("Estimated ratio of average max frequency by base frequency (times 1024): %llu\n" , arch_max_freq_ratio); - }else { -pr_debug ("Couldn't determine max cpu frequency, necessary for scale-invariant accounting.\n" ); - } -} - -static void disable_freq_invariance_workfn (struct work_struct *work) -{ -static_branch_disable (&arch_scale_freq_key); -} - -static DECLARE_WORK(disable_freq_invariance_work, - disable_freq_invariance_workfn); - -DEFINE_PER_CPU (unsigned long ,arch_freq_scale ) =SCHED_CAPACITY_SCALE ; - -void arch_scale_freq_tick (void ) -{ -u64 freq_scale; -u64 aperf, mperf; -u64 acnt, mcnt; - -if (!arch_scale_freq_invariant ()) -return ; - -rdmsrl (MSR_IA32_APERF , aperf); -rdmsrl (MSR_IA32_MPERF , mperf); - - acnt = aperf -this_cpu_read (arch_prev_aperf); - mcnt = mperf -this_cpu_read (arch_prev_mperf); - -this_cpu_write (arch_prev_aperf, aperf); -this_cpu_write (arch_prev_mperf, mperf); - -if (check_shl_overflow (acnt,2 *SCHED_CAPACITY_SHIFT , &acnt)) - goto error; - -if (check_mul_overflow (mcnt, arch_max_freq_ratio, &mcnt) || !mcnt) - goto error; - - freq_scale =div64_u64 (acnt, mcnt); -if (!freq_scale) - goto error; - -if (freq_scale >SCHED_CAPACITY_SCALE ) - freq_scale =SCHED_CAPACITY_SCALE ; - -this_cpu_write (arch_freq_scale, freq_scale); -return ; - -error: -pr_warn ("Scheduler frequency invariance went wobbly, disabling!\n" ); -schedule_work (&disable_freq_invariance_work); -} -#endif /* CONFIG_X86_64 */
[tip: x86/cleanups] x86/smp: Move APERF/MPERF code where it belongs
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: 55cb0b70749361d7f82a979768c77ac301f07da9 Gitweb: https://git.kernel.org/tip/55cb0b70749361d7f82a979768c77ac301f07da9 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 15 Apr 2022 21:19:51 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 15:51:08 +02:00
x86/smp: Move APERF/MPERF code where it belongs
as this can share code with the preexisting APERF/MPERF code.
No functional change.
--- arch/x86/kernel/cpu/aperfmperf.c | 366 +++++++++++++++++++++++++++++- arch/x86/kernel/smpboot.c | 355 +----------------------------- 2 files changed, 362 insertions(+), 359 deletions(-)
arch/x86/kernel/cpu/aperfmperf.c +362 -4
@@ -6,15 +6,19 @@* Copyright (C) 2017 Intel Corp. * Author: Len Brown <[email protected]> */ - +#include <linux/cpufreq.h> #include <linux/delay.h> #include <linux/ktime.h> #include <linux/math64.h> #include <linux/percpu.h> -#include <linux/cpufreq.h> -#include <linux/smp.h> -#include <linux/sched/isolation.h> #include <linux/rcupdate.h> +#include <linux/sched/isolation.h> +#include <linux/sched/topology.h> +#include <linux/smp.h> +#include <linux/syscore_ops.h> + +#include <asm/cpu_device_id.h> +#include <asm/intel-family.h> #include "cpu.h"
@@ -152,3 +156,357 @@ unsigned int arch_freq_get_on_cpu(int cpu)return per_cpu (samples.khz, cpu); } + +#if defined(CONFIG_X86_64 ) && defined(CONFIG_SMP ) +/* +* APERF/MPERF frequency ratio computation. +* +* The scheduler wants to do frequency invariant accounting and needs a <1 +* ratio to account for the 'current' frequency, corresponding to +* freq_curr / freq_max. +* +* Since the frequency freq_curr on x86 is controlled by micro-controller and +* our P-state setting is little more than a request/hint, we need to observe +* the effective frequency 'BusyMHz', i.e. the average frequency over a time +* interval after discarding idle time. This is given by: +* +* BusyMHz = delta_APERF / delta_MPERF * freq_base +* +* where freq_base is the max non-turbo P-state. +* +* The freq_max term has to be set to a somewhat arbitrary value, because we +* can't know which turbo states will be available at a given point in time: +* it all depends on the thermal headroom of the entire package. We set it to +* the turbo level with 4 cores active. +* +* Benchmarks show that's a good compromise between the 1C turbo ratio +* (freq_curr/freq_max would rarely reach 1) and something close to freq_base, +* which would ignore the entire turbo range (a conspicuous part, making +* freq_curr/freq_max always maxed out). +* +* An exception to the heuristic above is the Atom uarch, where we choose the +* highest turbo level for freq_max since Atom's are generally oriented towards +* power efficiency. +* +* Setting freq_max to anything less than the 1C turbo ratio makes the ratio +* freq_curr / freq_max to eventually grow >1, in which case we clip it to 1. +*/ + +DEFINE_STATIC_KEY_FALSE (arch_scale_freq_key); + +static DEFINE_PER_CPU(u64, arch_prev_aperf); +static DEFINE_PER_CPU(u64, arch_prev_mperf); +static u64 arch_turbo_freq_ratio =SCHED_CAPACITY_SCALE ; +static u64 arch_max_freq_ratio =SCHED_CAPACITY_SCALE ; + +void arch_set_max_freq_ratio (bool turbo_disabled) +{ + arch_max_freq_ratio = turbo_disabled ?SCHED_CAPACITY_SCALE : + arch_turbo_freq_ratio; +} +EXPORT_SYMBOL_GPL (arch_set_max_freq_ratio); + +static bool turbo_disabled (void ) +{ +u64 misc_en; +int err; + + err =rdmsrl_safe (MSR_IA32_MISC_ENABLE , &misc_en); +if (err) +return false; + +return (misc_en &MSR_IA32_MISC_ENABLE_TURBO_DISABLE ); +} + +static bool slv_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq) +{ +int err; + + err =rdmsrl_safe (MSR_ATOM_CORE_RATIOS , base_freq); +if (err) +return false; + + err =rdmsrl_safe (MSR_ATOM_CORE_TURBO_RATIOS , turbo_freq); +if (err) +return false; + + *base_freq = (*base_freq >>16 ) &0x3F ;/* max P state */ + *turbo_freq = *turbo_freq &0x3F ;/* 1C turbo */ + +return true; +} + +#define X86_MATCH (model) \ + X86_MATCH_VENDOR_FAM_MODEL_FEATURE(INTEL, 6, \ + INTEL_FAM6_##model, X86_FEATURE_APERFMPERF, NULL) + +static const struct x86_cpu_id has_knl_turbo_ratio_limits[] = { +X86_MATCH (XEON_PHI_KNL ), +X86_MATCH (XEON_PHI_KNM ), + {} +}; + +static const struct x86_cpu_id has_skx_turbo_ratio_limits[] = { +X86_MATCH (SKYLAKE_X ), + {} +}; + +static const struct x86_cpu_id has_glm_turbo_ratio_limits[] = { +X86_MATCH (ATOM_GOLDMONT ), +X86_MATCH (ATOM_GOLDMONT_D ), +X86_MATCH (ATOM_GOLDMONT_PLUS ), + {} +}; + +static bool knl_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq, +int num_delta_fratio) +{ +int fratio, delta_fratio, found; +int err, i; +u64 msr; + + err =rdmsrl_safe (MSR_PLATFORM_INFO , base_freq); +if (err) +return false; + + *base_freq = (*base_freq >>8 ) &0xFF ;/* max P state */ + + err =rdmsrl_safe (MSR_TURBO_RATIO_LIMIT , &msr); +if (err) +return false; + + fratio = (msr >>8 ) &0xFF ; + i =16 ; + found =0 ; +do { +if (found >= num_delta_fratio) { + *turbo_freq = fratio; +return true; + } + + delta_fratio = (msr >> (i +5 )) &0x7 ; + +if (delta_fratio) { + found +=1 ; + fratio -= delta_fratio; + } + + i +=8 ; + }while (i <64 ); + +return true; +} + +static bool skx_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq,int size) +{ +u64 ratios, counts; +u32 group_size; +int err, i; + + err =rdmsrl_safe (MSR_PLATFORM_INFO , base_freq); +if (err) +return false; + + *base_freq = (*base_freq >>8 ) &0xFF ;/* max P state */ + + err =rdmsrl_safe (MSR_TURBO_RATIO_LIMIT , &ratios); +if (err) +return false; + + err =rdmsrl_safe (MSR_TURBO_RATIO_LIMIT1 , &counts); +if (err) +return false; + +for (i =0 ; i <64 ; i +=8 ) { + group_size = (counts >> i) &0xFF ; +if (group_size >= size) { + *turbo_freq = (ratios >> i) &0xFF ; +return true; + } + } + +return false; +} + +static bool core_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq) +{ +u64 msr; +int err; + + err =rdmsrl_safe (MSR_PLATFORM_INFO , base_freq); +if (err) +return false; + + err =rdmsrl_safe (MSR_TURBO_RATIO_LIMIT , &msr); +if (err) +return false; + + *base_freq = (*base_freq >>8 ) &0xFF ;/* max P state */ + *turbo_freq = (msr >>24 ) &0xFF ;/* 4C turbo */ + +/* The CPU may have less than 4 cores */ +if (!*turbo_freq) + *turbo_freq = msr &0xFF ;/* 1C turbo */ + +return true; +} + +static bool intel_set_max_freq_ratio (void ) +{ +u64 base_freq, turbo_freq; +u64 turbo_ratio; + +if (slv_set_max_freq_ratio (&base_freq, &turbo_freq)) + goto out; + +if (x86_match_cpu (has_glm_turbo_ratio_limits) && +skx_set_max_freq_ratio (&base_freq, &turbo_freq,1 )) + goto out; + +if (x86_match_cpu (has_knl_turbo_ratio_limits) && +knl_set_max_freq_ratio (&base_freq, &turbo_freq,1 )) + goto out; + +if (x86_match_cpu (has_skx_turbo_ratio_limits) && +skx_set_max_freq_ratio (&base_freq, &turbo_freq,4 )) + goto out; + +if (core_set_max_freq_ratio (&base_freq, &turbo_freq)) + goto out; + +return false; + +out: +/* +* Some hypervisors advertise X86_FEATURE_APERFMPERF +* but then fill all MSR's with zeroes. +* Some CPUs have turbo boost but don't declare any turbo ratio +* in MSR_TURBO_RATIO_LIMIT. +*/ +if (!base_freq || !turbo_freq) { +pr_debug ("Couldn't determine cpu base or turbo frequency, necessary for scale-invariant accounting.\n" ); +return false; + } + + turbo_ratio =div_u64 (turbo_freq *SCHED_CAPACITY_SCALE , base_freq); +if (!turbo_ratio) { +pr_debug ("Non-zero turbo and base frequencies led to a 0 ratio.\n" ); +return false; + } + + arch_turbo_freq_ratio = turbo_ratio; +arch_set_max_freq_ratio (turbo_disabled ()); + +return true; +} + +static void init_counter_refs (void ) +{ +u64 aperf, mperf; + +rdmsrl (MSR_IA32_APERF , aperf); +rdmsrl (MSR_IA32_MPERF , mperf); + +this_cpu_write (arch_prev_aperf, aperf); +this_cpu_write (arch_prev_mperf, mperf); +} + +#ifdef CONFIG_PM_SLEEP +static struct syscore_ops freq_invariance_syscore_ops = { + .resume = init_counter_refs, +}; + +static void register_freq_invariance_syscore_ops (void ) +{ +/* Bail out if registered already. */ +if (freq_invariance_syscore_ops.node.prev) +return ; + +register_syscore_ops (&freq_invariance_syscore_ops); +} +#else +static inline void register_freq_invariance_syscore_ops (void ) {} +#endif + +void init_freq_invariance (bool secondary,bool cppc_ready) +{ +bool ret = false; + +if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) +return ; + +if (secondary) { +if (static_branch_likely (&arch_scale_freq_key)) { +init_counter_refs (); + } +return ; + } + +if (boot_cpu_data.x86_vendor ==X86_VENDOR_INTEL ) + ret =intel_set_max_freq_ratio (); +else if (boot_cpu_data.x86_vendor ==X86_VENDOR_AMD ) { +if (!cppc_ready) { +return ; + } + ret =amd_set_max_freq_ratio (&arch_turbo_freq_ratio); + } + +if (ret) { +init_counter_refs (); +static_branch_enable (&arch_scale_freq_key); +register_freq_invariance_syscore_ops (); +pr_info ("Estimated ratio of average max frequency by base frequency (times 1024): %llu\n" , arch_max_freq_ratio); + }else { +pr_debug ("Couldn't determine max cpu frequency, necessary for scale-invariant accounting.\n" ); + } +} + +static void disable_freq_invariance_workfn (struct work_struct *work) +{ +static_branch_disable (&arch_scale_freq_key); +} + +static DECLARE_WORK(disable_freq_invariance_work, + disable_freq_invariance_workfn); + +DEFINE_PER_CPU (unsigned long ,arch_freq_scale ) =SCHED_CAPACITY_SCALE ; + +void arch_scale_freq_tick (void ) +{ +u64 freq_scale; +u64 aperf, mperf; +u64 acnt, mcnt; + +if (!arch_scale_freq_invariant ()) +return ; + +rdmsrl (MSR_IA32_APERF , aperf); +rdmsrl (MSR_IA32_MPERF , mperf); + + acnt = aperf -this_cpu_read (arch_prev_aperf); + mcnt = mperf -this_cpu_read (arch_prev_mperf); + +this_cpu_write (arch_prev_aperf, aperf); +this_cpu_write (arch_prev_mperf, mperf); + +if (check_shl_overflow (acnt,2 *SCHED_CAPACITY_SHIFT , &acnt)) + goto error; + +if (check_mul_overflow (mcnt, arch_max_freq_ratio, &mcnt) || !mcnt) + goto error; + + freq_scale =div64_u64 (acnt, mcnt); +if (!freq_scale) + goto error; + +if (freq_scale >SCHED_CAPACITY_SCALE ) + freq_scale =SCHED_CAPACITY_SCALE ; + +this_cpu_write (arch_freq_scale, freq_scale); +return ; + +error: +pr_warn ("Scheduler frequency invariance went wobbly, disabling!\n" ); +schedule_work (&disable_freq_invariance_work); +} +#endif /* CONFIG_X86_64 && CONFIG_SMP */
arch/x86/kernel/smpboot.c +0 -355
364 lines not on this page · raw message
PATCH 3/10 x86/aperfmperf: Separate AP/BP frequency invariance init
In reply to Thomas GleixnerThis code is convoluted and because it can be invoked post init via the ACPI/CPPC code, all of the initialization functionality is built in instead of being part of init text and init data.
As a first step create separate calls for the boot and the application processors.
--- arch/x86/include/asm/topology.h | 12 +++++------- arch/x86/kernel/acpi/cppc.c | 3 ++- arch/x86/kernel/cpu/aperfmperf.c | 23 +++++++++++------------ arch/x86/kernel/smpboot.c | 4 ++-- 4 files changed, 20 insertions(+), 22 deletions(-)
arch/x86/include/asm/topology.h +5 -7
@@ -216,14 +216,12 @@ extern void arch_scale_freq_tick(void);#define arch_scale_freq_tick arch_scale_freq_tickextern void arch_set_max_freq_ratio (bool turbo_disabled); -void init_freq_invariance (bool secondary,bool cppc_ready); +extern void bp_init_freq_invariance (bool cppc_ready); +extern void ap_init_freq_invariance (void );#else -static inline void arch_set_max_freq_ratio (bool turbo_disabled) -{ -} -static inline void init_freq_invariance (bool secondary,bool cppc_ready) -{ -} +static inline void arch_set_max_freq_ratio (bool turbo_disabled) { } +static inline void bp_init_freq_invariance (bool cppc_ready) { } +static inline void ap_init_freq_invariance (void ) { }#endif #ifdef CONFIG_ACPI_CPPC_LIB
arch/x86/kernel/acpi/cppc.c +2 -1
@@ -96,7 +96,8 @@ void init_freq_invariance_cppc(void)mutex_lock (&freq_invariance_lock); -init_freq_invariance (secondary, true); +if (!secondary) +bp_init_freq_invariance (true); secondary = true;mutex_unlock (&freq_invariance_lock);
arch/x86/kernel/cpu/aperfmperf.c +11 -12
@@ -428,31 +428,24 @@ static void register_freq_invariance_sysstatic inline void register_freq_invariance_syscore_ops (void ) {}#endif -void init_freq_invariance (bool secondary,bool cppc_ready) +void bp_init_freq_invariance (bool cppc_ready) { -bool ret = false; +bool ret; -if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) +if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF ))return ; -if (secondary) { -if (static_branch_likely (&arch_scale_freq_key)) { -init_counter_refs (); - } -return ; - } +init_counter_refs ();if (boot_cpu_data.x86_vendor ==X86_VENDOR_INTEL ) ret =intel_set_max_freq_ratio ();else if (boot_cpu_data.x86_vendor ==X86_VENDOR_AMD ) { -if (!cppc_ready) { +if (!cppc_ready)return ; - } ret =amd_set_max_freq_ratio (&arch_turbo_freq_ratio); }if (ret) { -init_counter_refs ();static_branch_enable (&arch_scale_freq_key);register_freq_invariance_syscore_ops ();pr_info ("Estimated ratio of average max frequency by base frequency (times 1024): %llu\n" , arch_max_freq_ratio);
@@ -461,6 +454,12 @@ void init_freq_invariance(bool secondary } } +void ap_init_freq_invariance (void ) +{ +if (cpu_feature_enabled (X86_FEATURE_APERFMPERF )) +init_counter_refs (); +} +static void disable_freq_invariance_workfn (struct work_struct *work) {static_branch_disable (&arch_scale_freq_key);
arch/x86/kernel/smpboot.c +2 -2
@@ -186,7 +186,7 @@ static void smp_callin(void)*/ set_cpu_sibling_map (raw_smp_processor_id ()); -init_freq_invariance (true, false); +ap_init_freq_invariance ();/* * Get our bogomips.
@@ -1396,7 +1396,7 @@ void __init native_smp_prepare_cpus(unsi {smp_prepare_cpus_common (); -init_freq_invariance (false, false); +bp_init_freq_invariance (false);smp_sanity_check ();switch (apic_intr_mode) {
Re: [patch 03/10] x86/aperfmperf: Separate AP/BP frequency invariance init
Quoted 90 lines · Thomas Gleixner
This code is convoluted and because it can be invoked post init via the ACPI/CPPC code, all of the initialization functionality is built in instead of being part of init text and init data.
As a first step create separate calls for the boot and the application processors.
Signed-off-by Thomas Gleixner <[email protected]>--- arch/x86/include/asm/topology.h | 12 +++++------- arch/x86/kernel/acpi/cppc.c | 3 ++- arch/x86/kernel/cpu/aperfmperf.c | 23 +++++++++++------------ arch/x86/kernel/smpboot.c | 4 ++-- 4 files changed, 20 insertions(+), 22 deletions(-)arch/x86/include/asm/topology.h +5
-7@@ -216,14 +216,12 @@ extern void arch_scale_freq_tick(void);#define arch_scale_freq_tick arch_scale_freq_tickextern void arch_set_max_freq_ratio (bool turbo_disabled); -void init_freq_invariance (bool secondary,bool cppc_ready); +extern void bp_init_freq_invariance (bool cppc_ready); +extern void ap_init_freq_invariance (void );#else -static inline void arch_set_max_freq_ratio (bool turbo_disabled) -{ -} -static inline void init_freq_invariance (bool secondary,bool cppc_ready) -{ -} +static inline void arch_set_max_freq_ratio (bool turbo_disabled) { } +static inline void bp_init_freq_invariance (bool cppc_ready) { } +static inline void ap_init_freq_invariance (void ) { }#endif #ifdef CONFIG_ACPI_CPPC_LIB arch/x86/kernel/acpi/cppc.c +2
-1@@ -96,7 +96,8 @@ void init_freq_invariance_cppc(void)mutex_lock (&freq_invariance_lock); -init_freq_invariance (secondary, true); +if (!secondary) +bp_init_freq_invariance (true); secondary = true;mutex_unlock (&freq_invariance_lock);arch/x86/kernel/cpu/aperfmperf.c +5
-12@@ -428,31 +428,24 @@ static void register_freq_invariance_sysstatic inline void register_freq_invariance_syscore_ops (void ) {}#endif -void init_freq_invariance (bool secondary,bool cppc_ready) +void bp_init_freq_invariance (bool cppc_ready) { -bool ret = false; +bool ret; -if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) +if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF ))return ; -if (secondary) { -if (static_branch_likely (&arch_scale_freq_key)) { -init_counter_refs (); - } -return ; - } +init_counter_refs ();if (boot_cpu_data.x86_vendor ==X86_VENDOR_INTEL ) ret =intel_set_max_freq_ratio ();else if (boot_cpu_data.x86_vendor ==X86_VENDOR_AMD ) { -if (!cppc_ready) { +if (!cppc_ready)return ; - } ret =amd_set_max_freq_ratio (&arch_turbo_freq_ratio); }if (ret) { -init_counter_refs ();static_branch_enable (&arch_scale_freq_key);register_freq_invariance_syscore_ops ();pr_info ("Estimated ratio of average max frequency by base frequency (times 1024): %llu\n" , arch_max_freq_ratio);
arch/x86/kernel/cpu/aperfmperf.c void init_freq_invariance(bool secondary · line 460
Quoted 7 lines
arch/x86/kernel/cpu/aperfmperf.c +4
-0@@ -461,6 +454,12 @@ void init_freq_invariance(bool secondary } } +void ap_init_freq_invariance (void ) +{ +if (cpu_feature_enabled (X86_FEATURE_APERFMPERF )) +init_counter_refs ();
This doesn't check arch_scale_freq_key now which may be a good thing to mention in the changelog.
I don't see anything questionable in the patch, though, so
Quoted 26 lines
arch/x86/kernel/cpu/aperfmperf.c +2
-0+} +static void disable_freq_invariance_workfn (struct work_struct *work) {static_branch_disable (&arch_scale_freq_key);arch/x86/kernel/smpboot.c +2
-2@@ -186,7 +186,7 @@ static void smp_callin(void)*/ set_cpu_sibling_map (raw_smp_processor_id ()); -init_freq_invariance (true, false); +ap_init_freq_invariance ();/* * Get our bogomips. @@ -1396,7 +1396,7 @@ void __init native_smp_prepare_cpus(unsi {smp_prepare_cpus_common (); -init_freq_invariance (false, false); +bp_init_freq_invariance (false);smp_sanity_check ();switch (apic_intr_mode) {
[tip: x86/cleanups] x86/aperfmperf: Separate AP/BP frequency invariance init
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: 138a7f9c6beae8d652113b8e7a44994b4200bbcd Gitweb: https://git.kernel.org/tip/138a7f9c6beae8d652113b8e7a44994b4200bbcd Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 15 Apr 2022 21:19:53 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 15:51:08 +02:00
x86/aperfmperf: Separate AP/BP frequency invariance init
This code is convoluted and because it can be invoked post init via the ACPI/CPPC code, all of the initialization functionality is built in instead of being part of init text and init data.
As a first step create separate calls for the boot and the application processors.
--- arch/x86/include/asm/topology.h | 12 +++++------- arch/x86/kernel/acpi/cppc.c | 3 ++- arch/x86/kernel/cpu/aperfmperf.c | 23 +++++++++++------------ arch/x86/kernel/smpboot.c | 4 ++-- 4 files changed, 20 insertions(+), 22 deletions(-)
arch/x86/include/asm/topology.h +5 -7
@@ -216,14 +216,12 @@ extern void arch_scale_freq_tick(void);#define arch_scale_freq_tick arch_scale_freq_tickextern void arch_set_max_freq_ratio (bool turbo_disabled); -void init_freq_invariance (bool secondary,bool cppc_ready); +extern void bp_init_freq_invariance (bool cppc_ready); +extern void ap_init_freq_invariance (void );#else -static inline void arch_set_max_freq_ratio (bool turbo_disabled) -{ -} -static inline void init_freq_invariance (bool secondary,bool cppc_ready) -{ -} +static inline void arch_set_max_freq_ratio (bool turbo_disabled) { } +static inline void bp_init_freq_invariance (bool cppc_ready) { } +static inline void ap_init_freq_invariance (void ) { }#endif #ifdef CONFIG_ACPI_CPPC_LIB
arch/x86/kernel/acpi/cppc.c +2 -1
@@ -96,7 +96,8 @@ void init_freq_invariance_cppc(void)mutex_lock (&freq_invariance_lock); -init_freq_invariance (secondary, true); +if (!secondary) +bp_init_freq_invariance (true); secondary = true;mutex_unlock (&freq_invariance_lock);
arch/x86/kernel/cpu/aperfmperf.c +11 -12
@@ -428,31 +428,24 @@ static void register_freq_invariance_syscore_ops(void)static inline void register_freq_invariance_syscore_ops (void ) {}#endif -void init_freq_invariance (bool secondary,bool cppc_ready) +void bp_init_freq_invariance (bool cppc_ready) { -bool ret = false; +bool ret; -if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) +if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF ))return ; -if (secondary) { -if (static_branch_likely (&arch_scale_freq_key)) { -init_counter_refs (); - } -return ; - } +init_counter_refs ();if (boot_cpu_data.x86_vendor ==X86_VENDOR_INTEL ) ret =intel_set_max_freq_ratio ();else if (boot_cpu_data.x86_vendor ==X86_VENDOR_AMD ) { -if (!cppc_ready) { +if (!cppc_ready)return ; - } ret =amd_set_max_freq_ratio (&arch_turbo_freq_ratio); }if (ret) { -init_counter_refs ();static_branch_enable (&arch_scale_freq_key);register_freq_invariance_syscore_ops ();pr_info ("Estimated ratio of average max frequency by base frequency (times 1024): %llu\n" , arch_max_freq_ratio);
@@ -461,6 +454,12 @@ void init_freq_invariance(bool secondary, bool cppc_ready) } } +void ap_init_freq_invariance (void ) +{ +if (cpu_feature_enabled (X86_FEATURE_APERFMPERF )) +init_counter_refs (); +} +static void disable_freq_invariance_workfn (struct work_struct *work) {static_branch_disable (&arch_scale_freq_key);
arch/x86/kernel/smpboot.c +2 -2
@@ -186,7 +186,7 @@ static void smp_callin(void)*/ set_cpu_sibling_map (raw_smp_processor_id ()); -init_freq_invariance (true, false); +ap_init_freq_invariance ();/* * Get our bogomips.
@@ -1396,7 +1396,7 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) {smp_prepare_cpus_common (); -init_freq_invariance (false, false); +bp_init_freq_invariance (false);smp_sanity_check ();switch (apic_intr_mode) {
PATCH 4/10 x86/aperfmperf: Untangle Intel and AMD frequency invariance init
In reply to Thomas GleixnerAMD boot CPU initialization happens late via ACPI/CPPC which prevents the Intel parts from being marked __init.
Split out the common code and provide a dedicated interface for the AMD initialization and mark the Intel specific code and data __init.
The remaining text size is almost cut in half:
text: 2614 -> 1350
init.text: 0 -> 786
--- arch/x86/include/asm/topology.h | 13 ++------ arch/x86/kernel/acpi/cppc.c | 29 +++++++----------- arch/x86/kernel/cpu/aperfmperf.c | 62 ++++++++++++++++++++------------------- arch/x86/kernel/smpboot.c | 2 - 4 files changed, 49 insertions(+), 57 deletions(-)
arch/x86/include/asm/topology.h +4 -9
@@ -216,24 +216,19 @@ extern void arch_scale_freq_tick(void);#define arch_scale_freq_tick arch_scale_freq_tickextern void arch_set_max_freq_ratio (bool turbo_disabled); -extern void bp_init_freq_invariance (bool cppc_ready); +extern void freq_invariance_set_perf_ratio (u64 ratio,bool turbo_disabled); +extern void bp_init_freq_invariance (void );extern void ap_init_freq_invariance (void );#else static inline void arch_set_max_freq_ratio (bool turbo_disabled) { } -static inline void bp_init_freq_invariance (bool cppc_ready) { } +static inline void freq_invariance_set_perf_ratio (u64 ratio,bool turbo_disabled) { } +static inline void bp_init_freq_invariance (void ) { }static inline void ap_init_freq_invariance (void ) { }#endif #ifdef CONFIG_ACPI_CPPC_LIB void init_freq_invariance_cppc (void );#define arch_init_invariance_cppc init_freq_invariance_cppc - -bool amd_set_max_freq_ratio (u64 *ratio); -#else -static inline bool amd_set_max_freq_ratio (u64 *ratio) -{ -return false; -}#endif #endif /* _ASM_X86_TOPOLOGY_H */
arch/x86/kernel/acpi/cppc.c +12 -17
@@ -50,20 +50,17 @@ int cpc_write_ffh(int cpunum, struct cpcreturn err; } -bool amd_set_max_freq_ratio (u64 *ratio) +static void amd_set_max_freq_ratio (void ) {struct cppc_perf_caps perf_caps;u64 highest_perf, nominal_perf;u64 perf_ratio;int rc; -if (!ratio) -return false; - rc =cppc_get_perf_caps (0 , &perf_caps);if (rc) {pr_debug ("Could not retrieve perf counters (%d)\n" , rc); -return false; +return ; } highest_perf =amd_get_highest_perf ();
@@ -71,7 +68,7 @@ bool amd_set_max_freq_ratio(u64 *ratio)if (!highest_perf || !nominal_perf) {pr_debug ("Could not retrieve highest or nominal performance\n" ); -return false; +return ; } perf_ratio =div_u64 (highest_perf *SCHED_CAPACITY_SCALE , nominal_perf);
@@ -79,26 +76,24 @@ bool amd_set_max_freq_ratio(u64 *ratio) perf_ratio = (perf_ratio +SCHED_CAPACITY_SCALE ) >>1 ;if (!perf_ratio) {pr_debug ("Non-zero highest/nominal perf values led to a 0 ratio\n" ); -return false; +return ; } - *ratio = perf_ratio; -arch_set_max_freq_ratio (false); - -return true; +freq_invariance_set_perf_ratio (perf_ratio, false); }static DEFINE_MUTEX (freq_invariance_lock);void init_freq_invariance_cppc (void ) { -static bool secondary; +static bool init_done; -mutex_lock (&freq_invariance_lock); - -if (!secondary) -bp_init_freq_invariance (true); - secondary = true; +if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF )) +return ; +mutex_lock (&freq_invariance_lock); +if (!init_done) +amd_set_max_freq_ratio (); + init_done = true;mutex_unlock (&freq_invariance_lock); }
arch/x86/kernel/cpu/aperfmperf.c +32 -30
133 lines not on this page · raw message
arch/x86/kernel/smpboot.c +1 -1
@@ -1396,7 +1396,7 @@ void __init native_smp_prepare_cpus(unsi {smp_prepare_cpus_common (); -bp_init_freq_invariance (false); +bp_init_freq_invariance ();smp_sanity_check ();switch (apic_intr_mode) {
Re: [patch 04/10] x86/aperfmperf: Untangle Intel and AMD frequency invariance init
Quoted 7 lines · Thomas Gleixner
AMD boot CPU initialization happens late via ACPI/CPPC which prevents the Intel parts from being marked
__init.Split out the common code and provide a dedicated interface for the AMD initialization and mark the Intel specific code and data
__init.
Quoted 6 lines
The remaining text size is almost cut in half:
text: 2614 -> 1350
init.text: 0 -> 786Signed-off-by Thomas Gleixner <[email protected]>
All good AFAICS:
Quoted 262 lines
--- arch/x86/include/asm/topology.h | 13 ++------ arch/x86/kernel/acpi/cppc.c | 29 +++++++----------- arch/x86/kernel/cpu/aperfmperf.c | 62 ++++++++++++++++++++------------------- arch/x86/kernel/smpboot.c | 2 - 4 files changed, 49 insertions(+), 57 deletions(-)arch/x86/include/asm/topology.h +4
-9@@ -216,24 +216,19 @@ extern void arch_scale_freq_tick(void);#define arch_scale_freq_tick arch_scale_freq_tickextern void arch_set_max_freq_ratio (bool turbo_disabled); -extern void bp_init_freq_invariance (bool cppc_ready); +extern void freq_invariance_set_perf_ratio (u64 ratio,bool turbo_disabled); +extern void bp_init_freq_invariance (void );extern void ap_init_freq_invariance (void );#else static inline void arch_set_max_freq_ratio (bool turbo_disabled) { } -static inline void bp_init_freq_invariance (bool cppc_ready) { } +static inline void freq_invariance_set_perf_ratio (u64 ratio,bool turbo_disabled) { } +static inline void bp_init_freq_invariance (void ) { }static inline void ap_init_freq_invariance (void ) { }#endif #ifdef CONFIG_ACPI_CPPC_LIB void init_freq_invariance_cppc (void );#define arch_init_invariance_cppc init_freq_invariance_cppc - -bool amd_set_max_freq_ratio (u64 *ratio); -#else -static inline bool amd_set_max_freq_ratio (u64 *ratio) -{ -return false; -}#endif #endif /* _ASM_X86_TOPOLOGY_H */ arch/x86/kernel/acpi/cppc.c +12
-17@@ -50,20 +50,17 @@ int cpc_write_ffh(int cpunum, struct cpcreturn err; } -bool amd_set_max_freq_ratio (u64 *ratio) +static void amd_set_max_freq_ratio (void ) {struct cppc_perf_caps perf_caps;u64 highest_perf, nominal_perf;u64 perf_ratio;int rc; -if (!ratio) -return false; - rc =cppc_get_perf_caps (0 , &perf_caps);if (rc) {pr_debug ("Could not retrieve perf counters (%d)\n" , rc); -return false; +return ; } highest_perf =amd_get_highest_perf ();@@ -71,7 +68,7 @@ bool amd_set_max_freq_ratio(u64 *ratio)if (!highest_perf || !nominal_perf) {pr_debug ("Could not retrieve highest or nominal performance\n" ); -return false; +return ; } perf_ratio =div_u64 (highest_perf *SCHED_CAPACITY_SCALE , nominal_perf);@@ -79,26 +76,24 @@ bool amd_set_max_freq_ratio(u64 *ratio) perf_ratio = (perf_ratio +SCHED_CAPACITY_SCALE ) >>1 ;if (!perf_ratio) {pr_debug ("Non-zero highest/nominal perf values led to a 0 ratio\n" ); -return false; +return ; } - *ratio = perf_ratio; -arch_set_max_freq_ratio (false); - -return true; +freq_invariance_set_perf_ratio (perf_ratio, false); }static DEFINE_MUTEX (freq_invariance_lock);void init_freq_invariance_cppc (void ) { -static bool secondary; +static bool init_done; -mutex_lock (&freq_invariance_lock); - -if (!secondary) -bp_init_freq_invariance (true); - secondary = true; +if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF )) +return ; +mutex_lock (&freq_invariance_lock); +if (!init_done) +amd_set_max_freq_ratio (); + init_done = true;mutex_unlock (&freq_invariance_lock); }arch/x86/kernel/cpu/aperfmperf.c +32
-30@@ -206,7 +206,7 @@ void arch_set_max_freq_ratio(bool turbo_ }EXPORT_SYMBOL_GPL (arch_set_max_freq_ratio); -static bool turbo_disabled (void ) +static bool __init turbo_disabled (void ) {u64 misc_en;int err;@@ -218,7 +218,7 @@ static bool turbo_disabled(void)return (misc_en &MSR_IA32_MISC_ENABLE_TURBO_DISABLE ); } -static bool slv_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq) +static bool __init slv_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq) {int err;@@ -240,26 +240,26 @@ static bool slv_set_max_freq_ratio(u64 *X86_MATCH_VENDOR_FAM_MODEL_FEATURE (INTEL ,6 , \ INTEL_FAM6_##model, X86_FEATURE_APERFMPERF, NULL) -static const struct x86_cpu_id has_knl_turbo_ratio_limits[] = { +static const struct x86_cpu_id has_knl_turbo_ratio_limits[]__initconst = {X86_MATCH (XEON_PHI_KNL ),X86_MATCH (XEON_PHI_KNM ), {} }; -static const struct x86_cpu_id has_skx_turbo_ratio_limits[] = { +static const struct x86_cpu_id has_skx_turbo_ratio_limits[]__initconst = {X86_MATCH (SKYLAKE_X ), {} }; -static const struct x86_cpu_id has_glm_turbo_ratio_limits[] = { +static const struct x86_cpu_id has_glm_turbo_ratio_limits[]__initconst = {X86_MATCH (ATOM_GOLDMONT ),X86_MATCH (ATOM_GOLDMONT_D ),X86_MATCH (ATOM_GOLDMONT_PLUS ), {} }; -static bool knl_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq, -int num_delta_fratio) +static bool __init knl_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq, +int num_delta_fratio) {int fratio, delta_fratio, found;int err, i;@@ -297,7 +297,7 @@ static bool knl_set_max_freq_ratio(u64 *return true; } -static bool skx_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq,int size) +static bool __init skx_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq,int size) {u64 ratios, counts;u32 group_size;@@ -328,7 +328,7 @@ static bool skx_set_max_freq_ratio(u64 *return false; } -static bool core_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq) +static bool __init core_set_max_freq_ratio (u64 *base_freq,u64 *turbo_freq) {u64 msr;int err;@@ -351,7 +351,7 @@ static bool core_set_max_freq_ratio(u64return true; } -static bool intel_set_max_freq_ratio (void ) +static bool __init intel_set_max_freq_ratio (void ) {u64 base_freq, turbo_freq;u64 turbo_ratio;@@ -418,40 +418,42 @@ static struct syscore_ops freq_invariancstatic void register_freq_invariance_syscore_ops (void ) { -/* Bail out if registered already. */ -if (freq_invariance_syscore_ops.node.prev) -return ; -register_syscore_ops (&freq_invariance_syscore_ops); }#else static inline void register_freq_invariance_syscore_ops (void ) {}#endif -void bp_init_freq_invariance (bool cppc_ready) +static void freq_invariance_enable (void ) +{ +if (static_branch_unlikely (&arch_scale_freq_key)) { +WARN_ON_ONCE (1 ); +return ; + } +static_branch_enable (&arch_scale_freq_key); +register_freq_invariance_syscore_ops (); +pr_info ("Estimated ratio of average max frequency by base frequency (times 1024): %llu\n" , arch_max_freq_ratio); +} + +void freq_invariance_set_perf_ratio (u64 ratio,bool turbo_disabled) { -bool ret; + arch_turbo_freq_ratio = ratio; +arch_set_max_freq_ratio (turbo_disabled); +freq_invariance_enable (); +} +void __init bp_init_freq_invariance (void ) +{if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF ))return ;init_counter_refs (); -if (boot_cpu_data.x86_vendor ==X86_VENDOR_INTEL ) - ret =intel_set_max_freq_ratio (); -else if (boot_cpu_data.x86_vendor ==X86_VENDOR_AMD ) { -if (!cppc_ready) -return ; - ret =amd_set_max_freq_ratio (&arch_turbo_freq_ratio); - } +if (boot_cpu_data.x86_vendor !=X86_VENDOR_INTEL ) +return ; -if (ret) { -static_branch_enable (&arch_scale_freq_key); -register_freq_invariance_syscore_ops (); -pr_info ("Estimated ratio of average max frequency by base frequency (times 1024): %llu\n" , arch_max_freq_ratio); - }else { -pr_debug ("Couldn't determine max cpu frequency, necessary for scale-invariant accounting.\n" ); - } +if (intel_set_max_freq_ratio ()) +freq_invariance_enable (); }void ap_init_freq_invariance (void )arch/x86/kernel/smpboot.c +1
-1@@ -1396,7 +1396,7 @@ void __init native_smp_prepare_cpus(unsi {smp_prepare_cpus_common (); -bp_init_freq_invariance (false); +bp_init_freq_invariance ();smp_sanity_check ();switch (apic_intr_mode) {
[tip: x86/cleanups] x86/aperfmperf: Untangle Intel and AMD frequency invariance init
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: 39a80f50743fd38d4a96603ca3ddbe4fc8723c01 Gitweb: https://git.kernel.org/tip/39a80f50743fd38d4a96603ca3ddbe4fc8723c01 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 15 Apr 2022 21:19:54 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 15:51:08 +02:00
x86/aperfmperf: Untangle Intel and AMD frequency invariance init
AMD boot CPU initialization happens late via ACPI/CPPC which prevents the Intel parts from being marked __init.
Split out the common code and provide a dedicated interface for the AMD initialization and mark the Intel specific code and data __init.
The remaining text size is almost cut in half:
text: 2614 -> 1350
init.text: 0 -> 786
--- arch/x86/include/asm/topology.h | 13 ++----- arch/x86/kernel/acpi/cppc.c | 29 ++++++--------- arch/x86/kernel/cpu/aperfmperf.c | 62 ++++++++++++++++--------------- arch/x86/kernel/smpboot.c | 2 +- 4 files changed, 49 insertions(+), 57 deletions(-)
arch/x86/include/asm/topology.h +4 -9
@@ -216,24 +216,19 @@ extern void arch_scale_freq_tick(void);#define arch_scale_freq_tick arch_scale_freq_tickextern void arch_set_max_freq_ratio (bool turbo_disabled); -extern void bp_init_freq_invariance (bool cppc_ready); +extern void freq_invariance_set_perf_ratio (u64 ratio,bool turbo_disabled); +extern void bp_init_freq_invariance (void );extern void ap_init_freq_invariance (void );#else static inline void arch_set_max_freq_ratio (bool turbo_disabled) { } -static inline void bp_init_freq_invariance (bool cppc_ready) { } +static inline void freq_invariance_set_perf_ratio (u64 ratio,bool turbo_disabled) { } +static inline void bp_init_freq_invariance (void ) { }static inline void ap_init_freq_invariance (void ) { }#endif #ifdef CONFIG_ACPI_CPPC_LIB void init_freq_invariance_cppc (void );#define arch_init_invariance_cppc init_freq_invariance_cppc - -bool amd_set_max_freq_ratio (u64 *ratio); -#else -static inline bool amd_set_max_freq_ratio (u64 *ratio) -{ -return false; -}#endif #endif /* _ASM_X86_TOPOLOGY_H */
arch/x86/kernel/acpi/cppc.c +12 -17
@@ -50,20 +50,17 @@ int cpc_write_ffh(int cpunum, struct cpc_reg *reg, u64 val)return err; } -bool amd_set_max_freq_ratio (u64 *ratio) +static void amd_set_max_freq_ratio (void ) {struct cppc_perf_caps perf_caps;u64 highest_perf, nominal_perf;u64 perf_ratio;int rc; -if (!ratio) -return false; - rc =cppc_get_perf_caps (0 , &perf_caps);if (rc) {pr_debug ("Could not retrieve perf counters (%d)\n" , rc); -return false; +return ; } highest_perf =amd_get_highest_perf ();
@@ -71,7 +68,7 @@ bool amd_set_max_freq_ratio(u64 *ratio)if (!highest_perf || !nominal_perf) {pr_debug ("Could not retrieve highest or nominal performance\n" ); -return false; +return ; } perf_ratio =div_u64 (highest_perf *SCHED_CAPACITY_SCALE , nominal_perf);
@@ -79,26 +76,24 @@ bool amd_set_max_freq_ratio(u64 *ratio) perf_ratio = (perf_ratio +SCHED_CAPACITY_SCALE ) >>1 ;if (!perf_ratio) {pr_debug ("Non-zero highest/nominal perf values led to a 0 ratio\n" ); -return false; +return ; } - *ratio = perf_ratio; -arch_set_max_freq_ratio (false); - -return true; +freq_invariance_set_perf_ratio (perf_ratio, false); }static DEFINE_MUTEX (freq_invariance_lock);void init_freq_invariance_cppc (void ) { -static bool secondary; +static bool init_done; -mutex_lock (&freq_invariance_lock); - -if (!secondary) -bp_init_freq_invariance (true); - secondary = true; +if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF )) +return ; +mutex_lock (&freq_invariance_lock); +if (!init_done) +amd_set_max_freq_ratio (); + init_done = true;mutex_unlock (&freq_invariance_lock); }
arch/x86/kernel/cpu/aperfmperf.c +32 -30
133 lines not on this page · raw message
arch/x86/kernel/smpboot.c +1 -1
@@ -1396,7 +1396,7 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) {smp_prepare_cpus_common (); -bp_init_freq_invariance (false); +bp_init_freq_invariance ();smp_sanity_check ();switch (apic_intr_mode) {
[tip: x86/cleanups] x86/aperfmperf: Untangle Intel and AMD frequency invariance init
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: 0dfaf3f6ecc0c7f4f876255aa82e8959d3721365 Gitweb: https://git.kernel.org/tip/0dfaf3f6ecc0c7f4f876255aa82e8959d3721365 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 15 Apr 2022 21:19:54 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 20:22:19 +02:00
x86/aperfmperf: Untangle Intel and AMD frequency invariance init
AMD boot CPU initialization happens late via ACPI/CPPC which prevents the Intel parts from being marked __init.
Split out the common code and provide a dedicated interface for the AMD initialization and mark the Intel specific code and data __init.
The remaining text size is almost cut in half:
text: 2614 -> 1350
init.text: 0 -> 786
--- arch/x86/include/asm/topology.h | 13 ++----- arch/x86/kernel/acpi/cppc.c | 30 +++++++-------- arch/x86/kernel/cpu/aperfmperf.c | 62 ++++++++++++++++--------------- arch/x86/kernel/smpboot.c | 2 +- 4 files changed, 51 insertions(+), 56 deletions(-)
arch/x86/include/asm/topology.h +4 -9
@@ -216,24 +216,19 @@ extern void arch_scale_freq_tick(void);#define arch_scale_freq_tick arch_scale_freq_tickextern void arch_set_max_freq_ratio (bool turbo_disabled); -extern void bp_init_freq_invariance (bool cppc_ready); +extern void freq_invariance_set_perf_ratio (u64 ratio,bool turbo_disabled); +extern void bp_init_freq_invariance (void );extern void ap_init_freq_invariance (void );#else static inline void arch_set_max_freq_ratio (bool turbo_disabled) { } -static inline void bp_init_freq_invariance (bool cppc_ready) { } +static inline void freq_invariance_set_perf_ratio (u64 ratio,bool turbo_disabled) { } +static inline void bp_init_freq_invariance (void ) { }static inline void ap_init_freq_invariance (void ) { }#endif #ifdef CONFIG_ACPI_CPPC_LIB void init_freq_invariance_cppc (void );#define arch_init_invariance_cppc init_freq_invariance_cppc - -bool amd_set_max_freq_ratio (u64 *ratio); -#else -static inline bool amd_set_max_freq_ratio (u64 *ratio) -{ -return false; -}#endif #endif /* _ASM_X86_TOPOLOGY_H */
arch/x86/kernel/acpi/cppc.c +14 -16
@@ -50,20 +50,17 @@ int cpc_write_ffh(int cpunum, struct cpc_reg *reg, u64 val)return err; } -bool amd_set_max_freq_ratio (u64 *ratio) +static void amd_set_max_freq_ratio (void ) {struct cppc_perf_caps perf_caps;u64 highest_perf, nominal_perf;u64 perf_ratio;int rc; -if (!ratio) -return false; - rc =cppc_get_perf_caps (0 , &perf_caps);if (rc) {pr_debug ("Could not retrieve perf counters (%d)\n" , rc); -return false; +return ; } highest_perf =amd_get_highest_perf ();
@@ -71,7 +68,7 @@ bool amd_set_max_freq_ratio(u64 *ratio)if (!highest_perf || !nominal_perf) {pr_debug ("Could not retrieve highest or nominal performance\n" ); -return false; +return ; } perf_ratio =div_u64 (highest_perf *SCHED_CAPACITY_SCALE , nominal_perf);
@@ -79,26 +76,27 @@ bool amd_set_max_freq_ratio(u64 *ratio) perf_ratio = (perf_ratio +SCHED_CAPACITY_SCALE ) >>1 ;if (!perf_ratio) {pr_debug ("Non-zero highest/nominal perf values led to a 0 ratio\n" ); -return false; +return ; } - *ratio = perf_ratio; -arch_set_max_freq_ratio (false); - -return true; +freq_invariance_set_perf_ratio (perf_ratio, false); }static DEFINE_MUTEX (freq_invariance_lock);void init_freq_invariance_cppc (void ) { -static bool secondary; +static bool init_done; -mutex_lock (&freq_invariance_lock); +if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF )) +return ; -if (!secondary) -bp_init_freq_invariance (true); - secondary = true; +if (boot_cpu_data.x86_vendor !=X86_VENDOR_AMD ) +return ; +mutex_lock (&freq_invariance_lock); +if (!init_done) +amd_set_max_freq_ratio (); + init_done = true;mutex_unlock (&freq_invariance_lock); }
arch/x86/kernel/cpu/aperfmperf.c +32 -30
133 lines not on this page · raw message
arch/x86/kernel/smpboot.c +1 -1
@@ -1396,7 +1396,7 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) {smp_prepare_cpus_common (); -bp_init_freq_invariance (false); +bp_init_freq_invariance ();smp_sanity_check ();switch (apic_intr_mode) {
PATCH 5/10 x86/aperfmperf: Put frequency invariance aperf/mperf data into a struct
In reply to Thomas GleixnerPreparation for sharing code with the CPU frequency portion of the aperf/mperf code.
No functional change.
--- arch/x86/kernel/cpu/aperfmperf.c | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-)
arch/x86/kernel/cpu/aperfmperf.c +15 -11
@@ -22,6 +22,13 @@#include "cpu.h" +struct aperfmperf { +u64 aperf; +u64 mperf; +}; + +static DEFINE_PER_CPU_SHARED_ALIGNED(struct aperfmperf, cpu_samples); +struct aperfmperf_sample {unsigned int khz;atomic_t scfpending;
@@ -194,8 +201,6 @@ unsigned int arch_freq_get_on_cpu(int cpDEFINE_STATIC_KEY_FALSE (arch_scale_freq_key); -static DEFINE_PER_CPU(u64, arch_prev_aperf); -static DEFINE_PER_CPU(u64, arch_prev_mperf);static u64 arch_turbo_freq_ratio =SCHED_CAPACITY_SCALE ;static u64 arch_max_freq_ratio =SCHED_CAPACITY_SCALE ;
@@ -407,8 +412,8 @@ static void init_counter_refs(void)rdmsrl (MSR_IA32_APERF , aperf);rdmsrl (MSR_IA32_MPERF , mperf); -this_cpu_write (arch_prev_aperf, aperf); -this_cpu_write (arch_prev_mperf, mperf); +this_cpu_write (cpu_samples.aperf, aperf); +this_cpu_write (cpu_samples.mperf, mperf); }#ifdef CONFIG_PM_SLEEP
@@ -474,9 +479,8 @@ DEFINE_PER_CPU(unsigned long, arch_freq_void arch_scale_freq_tick (void ) { -u64 freq_scale; -u64 aperf, mperf; -u64 acnt, mcnt; +struct aperfmperf *s =this_cpu_ptr (&cpu_samples); +u64 aperf, mperf, acnt, mcnt, freq_scale;if (!arch_scale_freq_invariant ())return ;
@@ -484,11 +488,11 @@ void arch_scale_freq_tick(void)rdmsrl (MSR_IA32_APERF , aperf);rdmsrl (MSR_IA32_MPERF , mperf); - acnt = aperf -this_cpu_read (arch_prev_aperf); - mcnt = mperf -this_cpu_read (arch_prev_mperf); + acnt = aperf - s->aperf; + mcnt = mperf - s->mperf; -this_cpu_write (arch_prev_aperf, aperf); -this_cpu_write (arch_prev_mperf, mperf); + s->aperf = aperf; + s->mperf = mperf;if (check_shl_overflow (acnt,2 *SCHED_CAPACITY_SHIFT , &acnt)) goto error;
Re: [patch 05/10] x86/aperfmperf: Put frequency invariance aperf/mperf data into a struct
Quoted 7 lines · Thomas Gleixner
Preparation for sharing code with the CPU frequency portion of the aperf/mperf code.
No functional change.
Signed-off-by Thomas Gleixner <[email protected]>
All good AFAICS:
Quoted 69 lines
--- arch/x86/kernel/cpu/aperfmperf.c | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-)arch/x86/kernel/cpu/aperfmperf.c +15
-11@@ -22,6 +22,13 @@#include "cpu.h" +struct aperfmperf { +u64 aperf; +u64 mperf; +}; + +static DEFINE_PER_CPU_SHARED_ALIGNED(struct aperfmperf, cpu_samples); +struct aperfmperf_sample {unsigned int khz;atomic_t scfpending;@@ -194,8 +201,6 @@ unsigned int arch_freq_get_on_cpu(int cpDEFINE_STATIC_KEY_FALSE (arch_scale_freq_key); -static DEFINE_PER_CPU(u64, arch_prev_aperf); -static DEFINE_PER_CPU(u64, arch_prev_mperf);static u64 arch_turbo_freq_ratio =SCHED_CAPACITY_SCALE ;static u64 arch_max_freq_ratio =SCHED_CAPACITY_SCALE ;@@ -407,8 +412,8 @@ static void init_counter_refs(void)rdmsrl (MSR_IA32_APERF , aperf);rdmsrl (MSR_IA32_MPERF , mperf); -this_cpu_write (arch_prev_aperf, aperf); -this_cpu_write (arch_prev_mperf, mperf); +this_cpu_write (cpu_samples.aperf, aperf); +this_cpu_write (cpu_samples.mperf, mperf); }#ifdef CONFIG_PM_SLEEP @@ -474,9 +479,8 @@ DEFINE_PER_CPU(unsigned long, arch_freq_void arch_scale_freq_tick (void ) { -u64 freq_scale; -u64 aperf, mperf; -u64 acnt, mcnt; +struct aperfmperf *s =this_cpu_ptr (&cpu_samples); +u64 aperf, mperf, acnt, mcnt, freq_scale;if (!arch_scale_freq_invariant ())return ;@@ -484,11 +488,11 @@ void arch_scale_freq_tick(void)rdmsrl (MSR_IA32_APERF , aperf);rdmsrl (MSR_IA32_MPERF , mperf); - acnt = aperf -this_cpu_read (arch_prev_aperf); - mcnt = mperf -this_cpu_read (arch_prev_mperf); + acnt = aperf - s->aperf; + mcnt = mperf - s->mperf; -this_cpu_write (arch_prev_aperf, aperf); -this_cpu_write (arch_prev_mperf, mperf); + s->aperf = aperf; + s->mperf = mperf;if (check_shl_overflow (acnt,2 *SCHED_CAPACITY_SHIFT , &acnt)) goto error;
[tip: x86/cleanups] x86/aperfmperf: Put frequency invariance aperf/mperf data into a struct
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: d512d9a127285226c370ec7e92efb9ef61a6515b Gitweb: https://git.kernel.org/tip/d512d9a127285226c370ec7e92efb9ef61a6515b Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 15 Apr 2022 21:19:56 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 15:51:08 +02:00
x86/aperfmperf: Put frequency invariance aperf/mperf data into a struct
Preparation for sharing code with the CPU frequency portion of the aperf/mperf code.
No functional change.
--- arch/x86/kernel/cpu/aperfmperf.c | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-)
arch/x86/kernel/cpu/aperfmperf.c +15 -11
@@ -22,6 +22,13 @@#include "cpu.h" +struct aperfmperf { +u64 aperf; +u64 mperf; +}; + +static DEFINE_PER_CPU_SHARED_ALIGNED(struct aperfmperf, cpu_samples); +struct aperfmperf_sample {unsigned int khz;atomic_t scfpending;
@@ -194,8 +201,6 @@ unsigned int arch_freq_get_on_cpu(int cpu)DEFINE_STATIC_KEY_FALSE (arch_scale_freq_key); -static DEFINE_PER_CPU(u64, arch_prev_aperf); -static DEFINE_PER_CPU(u64, arch_prev_mperf);static u64 arch_turbo_freq_ratio =SCHED_CAPACITY_SCALE ;static u64 arch_max_freq_ratio =SCHED_CAPACITY_SCALE ;
@@ -407,8 +412,8 @@ static void init_counter_refs(void)rdmsrl (MSR_IA32_APERF , aperf);rdmsrl (MSR_IA32_MPERF , mperf); -this_cpu_write (arch_prev_aperf, aperf); -this_cpu_write (arch_prev_mperf, mperf); +this_cpu_write (cpu_samples.aperf, aperf); +this_cpu_write (cpu_samples.mperf, mperf); }#ifdef CONFIG_PM_SLEEP
@@ -474,9 +479,8 @@ DEFINE_PER_CPU(unsigned long, arch_freq_scale) = SCHED_CAPACITY_SCALE;void arch_scale_freq_tick (void ) { -u64 freq_scale; -u64 aperf, mperf; -u64 acnt, mcnt; +struct aperfmperf *s =this_cpu_ptr (&cpu_samples); +u64 aperf, mperf, acnt, mcnt, freq_scale;if (!arch_scale_freq_invariant ())return ;
@@ -484,11 +488,11 @@ void arch_scale_freq_tick(void)rdmsrl (MSR_IA32_APERF , aperf);rdmsrl (MSR_IA32_MPERF , mperf); - acnt = aperf -this_cpu_read (arch_prev_aperf); - mcnt = mperf -this_cpu_read (arch_prev_mperf); + acnt = aperf - s->aperf; + mcnt = mperf - s->mperf; -this_cpu_write (arch_prev_aperf, aperf); -this_cpu_write (arch_prev_mperf, mperf); + s->aperf = aperf; + s->mperf = mperf;if (check_shl_overflow (acnt,2 *SCHED_CAPACITY_SHIFT , &acnt)) goto error;
[tip: x86/cleanups] x86/aperfmperf: Put frequency invariance aperf/mperf data into a struct
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: 24620d94a52adc0cafe65dc65bed1d586ca04a6e Gitweb: https://git.kernel.org/tip/24620d94a52adc0cafe65dc65bed1d586ca04a6e Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 15 Apr 2022 21:19:56 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 20:22:19 +02:00
x86/aperfmperf: Put frequency invariance aperf/mperf data into a struct
Preparation for sharing code with the CPU frequency portion of the aperf/mperf code.
No functional change.
--- arch/x86/kernel/cpu/aperfmperf.c | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-)
arch/x86/kernel/cpu/aperfmperf.c +15 -11
@@ -22,6 +22,13 @@#include "cpu.h" +struct aperfmperf { +u64 aperf; +u64 mperf; +}; + +static DEFINE_PER_CPU_SHARED_ALIGNED(struct aperfmperf, cpu_samples); +struct aperfmperf_sample {unsigned int khz;atomic_t scfpending;
@@ -194,8 +201,6 @@ unsigned int arch_freq_get_on_cpu(int cpu)DEFINE_STATIC_KEY_FALSE (arch_scale_freq_key); -static DEFINE_PER_CPU(u64, arch_prev_aperf); -static DEFINE_PER_CPU(u64, arch_prev_mperf);static u64 arch_turbo_freq_ratio =SCHED_CAPACITY_SCALE ;static u64 arch_max_freq_ratio =SCHED_CAPACITY_SCALE ;
@@ -407,8 +412,8 @@ static void init_counter_refs(void)rdmsrl (MSR_IA32_APERF , aperf);rdmsrl (MSR_IA32_MPERF , mperf); -this_cpu_write (arch_prev_aperf, aperf); -this_cpu_write (arch_prev_mperf, mperf); +this_cpu_write (cpu_samples.aperf, aperf); +this_cpu_write (cpu_samples.mperf, mperf); }#ifdef CONFIG_PM_SLEEP
@@ -474,9 +479,8 @@ DEFINE_PER_CPU(unsigned long, arch_freq_scale) = SCHED_CAPACITY_SCALE;void arch_scale_freq_tick (void ) { -u64 freq_scale; -u64 aperf, mperf; -u64 acnt, mcnt; +struct aperfmperf *s =this_cpu_ptr (&cpu_samples); +u64 aperf, mperf, acnt, mcnt, freq_scale;if (!arch_scale_freq_invariant ())return ;
@@ -484,11 +488,11 @@ void arch_scale_freq_tick(void)rdmsrl (MSR_IA32_APERF , aperf);rdmsrl (MSR_IA32_MPERF , mperf); - acnt = aperf -this_cpu_read (arch_prev_aperf); - mcnt = mperf -this_cpu_read (arch_prev_mperf); + acnt = aperf - s->aperf; + mcnt = mperf - s->mperf; -this_cpu_write (arch_prev_aperf, aperf); -this_cpu_write (arch_prev_mperf, mperf); + s->aperf = aperf; + s->mperf = mperf;if (check_shl_overflow (acnt,2 *SCHED_CAPACITY_SHIFT , &acnt)) goto error;
PATCH 6/10 x86/aperfmperf: Restructure arch_scale_freq_tick()
In reply to Thomas GleixnerPreparation for sharing code with the CPU frequency portion of the aperf/mperf code.
No functional change.
--- arch/x86/kernel/cpu/aperfmperf.c | 36 +++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 15 deletions(-)
arch/x86/kernel/cpu/aperfmperf.c +21 -15
@@ -477,22 +477,9 @@ static DECLARE_WORK(disable_freq_invaria DEFINE_PER_CPU(unsigned long, arch_freq_scale) = SCHED_CAPACITY_SCALE; -void arch_scale_freq_tick (void ) +static void scale_freq_tick (u64 acnt,u64 mcnt) { -struct aperfmperf *s =this_cpu_ptr (&cpu_samples); -u64 aperf, mperf, acnt, mcnt, freq_scale; - -if (!arch_scale_freq_invariant ()) -return ; - -rdmsrl (MSR_IA32_APERF , aperf); -rdmsrl (MSR_IA32_MPERF , mperf); - - acnt = aperf - s->aperf; - mcnt = mperf - s->mperf; - - s->aperf = aperf; - s->mperf = mperf; +u64 freq_scale;if (check_shl_overflow (acnt,2 *SCHED_CAPACITY_SHIFT , &acnt)) goto error;
@@ -514,4 +501,23 @@ void arch_scale_freq_tick(void)pr_warn ("Scheduler frequency invariance went wobbly, disabling!\n" ); schedule_work(&disable_freq_invariance_work); } + +void arch_scale_freq_tick (void ) +{ +struct aperfmperf *s =this_cpu_ptr (&cpu_samples); +u64 acnt, mcnt, aperf, mperf; + +if (!arch_scale_freq_invariant ()) +return ; + +rdmsrl (MSR_IA32_APERF , aperf); +rdmsrl (MSR_IA32_MPERF , mperf); + acnt = aperf - s->aperf; + mcnt = mperf - s->mperf; + + s->aperf = aperf; + s->mperf = mperf; + +scale_freq_tick (acnt, mcnt); +}#endif /* CONFIG_X86_64 && CONFIG_SMP */
Re: [patch 06/10] x86/aperfmperf: Restructure arch_scale_freq_tick()
Quoted 7 lines · Thomas Gleixner
Preparation for sharing code with the CPU frequency portion of the aperf/mperf code.
No functional change.
Signed-off-by Thomas Gleixner <[email protected]>
All good AFAICS:
Quoted 56 lines
--- arch/x86/kernel/cpu/aperfmperf.c | 36 +++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 15 deletions(-)arch/x86/kernel/cpu/aperfmperf.c +21
-15@@ -477,22 +477,9 @@ static DECLARE_WORK(disable_freq_invaria DEFINE_PER_CPU(unsigned long, arch_freq_scale) = SCHED_CAPACITY_SCALE; -void arch_scale_freq_tick (void ) +static void scale_freq_tick (u64 acnt,u64 mcnt) { -struct aperfmperf *s =this_cpu_ptr (&cpu_samples); -u64 aperf, mperf, acnt, mcnt, freq_scale; - -if (!arch_scale_freq_invariant ()) -return ; - -rdmsrl (MSR_IA32_APERF , aperf); -rdmsrl (MSR_IA32_MPERF , mperf); - - acnt = aperf - s->aperf; - mcnt = mperf - s->mperf; - - s->aperf = aperf; - s->mperf = mperf; +u64 freq_scale;if (check_shl_overflow (acnt,2 *SCHED_CAPACITY_SHIFT , &acnt)) goto error;@@ -514,4 +501,23 @@ void arch_scale_freq_tick(void)pr_warn ("Scheduler frequency invariance went wobbly, disabling!\n" ); schedule_work(&disable_freq_invariance_work); } + +void arch_scale_freq_tick (void ) +{ +struct aperfmperf *s =this_cpu_ptr (&cpu_samples); +u64 acnt, mcnt, aperf, mperf; + +if (!arch_scale_freq_invariant ()) +return ; + +rdmsrl (MSR_IA32_APERF , aperf); +rdmsrl (MSR_IA32_MPERF , mperf); + acnt = aperf - s->aperf; + mcnt = mperf - s->mperf; + + s->aperf = aperf; + s->mperf = mperf; + +scale_freq_tick (acnt, mcnt); +}#endif /* CONFIG_X86_64 && CONFIG_SMP */
[tip: x86/cleanups] x86/aperfmperf: Restructure arch_scale_freq_tick()
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: 76225ca3666c5b05afb8dcc18a73f002b097af24 Gitweb: https://git.kernel.org/tip/76225ca3666c5b05afb8dcc18a73f002b097af24 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 15 Apr 2022 21:19:57 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 15:51:08 +02:00
x86/aperfmperf: Restructure arch_scale_freq_tick()
Preparation for sharing code with the CPU frequency portion of the aperf/mperf code.
No functional change.
--- arch/x86/kernel/cpu/aperfmperf.c | 36 ++++++++++++++++++------------- 1 file changed, 21 insertions(+), 15 deletions(-)
arch/x86/kernel/cpu/aperfmperf.c +21 -15
@@ -477,22 +477,9 @@ static DECLARE_WORK(disable_freq_invariance_work, DEFINE_PER_CPU(unsigned long, arch_freq_scale) = SCHED_CAPACITY_SCALE; -void arch_scale_freq_tick (void ) +static void scale_freq_tick (u64 acnt,u64 mcnt) { -struct aperfmperf *s =this_cpu_ptr (&cpu_samples); -u64 aperf, mperf, acnt, mcnt, freq_scale; - -if (!arch_scale_freq_invariant ()) -return ; - -rdmsrl (MSR_IA32_APERF , aperf); -rdmsrl (MSR_IA32_MPERF , mperf); - - acnt = aperf - s->aperf; - mcnt = mperf - s->mperf; - - s->aperf = aperf; - s->mperf = mperf; +u64 freq_scale;if (check_shl_overflow (acnt,2 *SCHED_CAPACITY_SHIFT , &acnt)) goto error;
@@ -514,4 +501,23 @@ error:pr_warn ("Scheduler frequency invariance went wobbly, disabling!\n" ); schedule_work(&disable_freq_invariance_work); } + +void arch_scale_freq_tick (void ) +{ +struct aperfmperf *s =this_cpu_ptr (&cpu_samples); +u64 acnt, mcnt, aperf, mperf; + +if (!arch_scale_freq_invariant ()) +return ; + +rdmsrl (MSR_IA32_APERF , aperf); +rdmsrl (MSR_IA32_MPERF , mperf); + acnt = aperf - s->aperf; + mcnt = mperf - s->mperf; + + s->aperf = aperf; + s->mperf = mperf; + +scale_freq_tick (acnt, mcnt); +}#endif /* CONFIG_X86_64 && CONFIG_SMP */
[tip: x86/cleanups] x86/aperfmperf: Restructure arch_scale_freq_tick()
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: 73a5fa7d51366a549a9f2e3ee875ae51aa0b5580 Gitweb: https://git.kernel.org/tip/73a5fa7d51366a549a9f2e3ee875ae51aa0b5580 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 15 Apr 2022 21:19:57 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 20:22:19 +02:00
x86/aperfmperf: Restructure arch_scale_freq_tick()
Preparation for sharing code with the CPU frequency portion of the aperf/mperf code.
No functional change.
--- arch/x86/kernel/cpu/aperfmperf.c | 36 ++++++++++++++++++------------- 1 file changed, 21 insertions(+), 15 deletions(-)
arch/x86/kernel/cpu/aperfmperf.c +21 -15
@@ -477,22 +477,9 @@ static DECLARE_WORK(disable_freq_invariance_work, DEFINE_PER_CPU(unsigned long, arch_freq_scale) = SCHED_CAPACITY_SCALE; -void arch_scale_freq_tick (void ) +static void scale_freq_tick (u64 acnt,u64 mcnt) { -struct aperfmperf *s =this_cpu_ptr (&cpu_samples); -u64 aperf, mperf, acnt, mcnt, freq_scale; - -if (!arch_scale_freq_invariant ()) -return ; - -rdmsrl (MSR_IA32_APERF , aperf); -rdmsrl (MSR_IA32_MPERF , mperf); - - acnt = aperf - s->aperf; - mcnt = mperf - s->mperf; - - s->aperf = aperf; - s->mperf = mperf; +u64 freq_scale;if (check_shl_overflow (acnt,2 *SCHED_CAPACITY_SHIFT , &acnt)) goto error;
@@ -514,4 +501,23 @@ error:pr_warn ("Scheduler frequency invariance went wobbly, disabling!\n" ); schedule_work(&disable_freq_invariance_work); } + +void arch_scale_freq_tick (void ) +{ +struct aperfmperf *s =this_cpu_ptr (&cpu_samples); +u64 acnt, mcnt, aperf, mperf; + +if (!arch_scale_freq_invariant ()) +return ; + +rdmsrl (MSR_IA32_APERF , aperf); +rdmsrl (MSR_IA32_MPERF , mperf); + acnt = aperf - s->aperf; + mcnt = mperf - s->mperf; + + s->aperf = aperf; + s->mperf = mperf; + +scale_freq_tick (acnt, mcnt); +}#endif /* CONFIG_X86_64 && CONFIG_SMP */
PATCH 7/10 x86/aperfmperf: Make parts of the frequency invariance code unconditional
In reply to Thomas GleixnerThe frequency invariance support is currently limited to x86/64 and SMP, which is the vast majority of machines.
arch_scale_freq_tick() is called every tick on all CPUs and reads the APERF and MPERF MSRs. The CPU frequency getters function do the same via dedicated IPIs.
While it could be argued that on systems where frequency invariance support is disabled (32bit, !SMP) the per tick read of the APERF and MPERF MSRs can be avoided, it does not make sense to keep the extra code and the resulting runtime issues of mass IPIs around.
As a first step split out the non frequency invariance specific initialization code and the read MSR portion of arch_scale_freq_tick(). The rest of the code is still conditional and guarded with a static key.
--- arch/x86/include/asm/cpu.h | 2 + arch/x86/include/asm/topology.h | 4 -- arch/x86/kernel/cpu/aperfmperf.c | 63 +++++++++++++++++++++++---------------- arch/x86/kernel/smpboot.c | 3 - 4 files changed, 41 insertions(+), 31 deletions(-)
arch/x86/include/asm/cpu.h +2 -0
@@ -36,6 +36,8 @@ extern int _debug_hotplug_cpu(int cpu, i#endif #endif +extern void ap_init_aperfmperf (void ); +int mwait_usable (const struct cpuinfo_x86 *);unsigned int x86_family (unsigned int sig);
arch/x86/include/asm/topology.h +0 -4
@@ -217,13 +217,9 @@ extern void arch_scale_freq_tick(void);extern void arch_set_max_freq_ratio (bool turbo_disabled);extern void freq_invariance_set_perf_ratio (u64 ratio,bool turbo_disabled); -extern void bp_init_freq_invariance (void ); -extern void ap_init_freq_invariance (void );#else static inline void arch_set_max_freq_ratio (bool turbo_disabled) { }static inline void freq_invariance_set_perf_ratio (u64 ratio,bool turbo_disabled) { } -static inline void bp_init_freq_invariance (void ) { } -static inline void ap_init_freq_invariance (void ) { }#endif #ifdef CONFIG_ACPI_CPPC_LIB
arch/x86/kernel/cpu/aperfmperf.c +38 -25
115 lines not on this page · raw message
arch/x86/kernel/smpboot.c +1 -2
@@ -186,7 +186,7 @@ static void smp_callin(void)*/ set_cpu_sibling_map (raw_smp_processor_id ()); -ap_init_freq_invariance (); +ap_init_aperfmperf ();/* * Get our bogomips.
@@ -1396,7 +1396,6 @@ void __init native_smp_prepare_cpus(unsi {smp_prepare_cpus_common (); -bp_init_freq_invariance ();smp_sanity_check ();switch (apic_intr_mode) {
Re: [patch 07/10] x86/aperfmperf: Make parts of the frequency invariance code unconditional
Quoted 13 lines · Thomas Gleixner
The frequency invariance support is currently limited to x86/64 and SMP, which is the vast majority of machines.
arch_scale_freq_tick()is called every tick on all CPUs and reads the APERF and MPERF MSRs. The CPU frequency getters function do the same via dedicated IPIs.While it could be argued that on systems where frequency invariance support is disabled (32bit, !SMP) the per tick read of the APERF and MPERF MSRs can be avoided, it does not make sense to keep the extra code and the resulting runtime issues of mass IPIs around.
Quoted 5 lines
As a first step split out the non frequency invariance specific initialization code and the read MSR portion of
arch_scale_freq_tick(). The rest of the code is still conditional and guarded with a static key.Signed-off-by Thomas Gleixner <[email protected]>
All good AFAICS:
Quoted 179 lines
--- arch/x86/include/asm/cpu.h | 2 + arch/x86/include/asm/topology.h | 4 -- arch/x86/kernel/cpu/aperfmperf.c | 63 +++++++++++++++++++++++---------------- arch/x86/kernel/smpboot.c | 3 - 4 files changed, 41 insertions(+), 31 deletions(-)arch/x86/include/asm/cpu.h +2
-0@@ -36,6 +36,8 @@ extern int _debug_hotplug_cpu(int cpu, i#endif #endif +extern void ap_init_aperfmperf (void ); +int mwait_usable (const struct cpuinfo_x86 *);unsigned int x86_family (unsigned int sig);arch/x86/include/asm/topology.h +0
-4@@ -217,13 +217,9 @@ extern void arch_scale_freq_tick(void);extern void arch_set_max_freq_ratio (bool turbo_disabled);extern void freq_invariance_set_perf_ratio (u64 ratio,bool turbo_disabled); -extern void bp_init_freq_invariance (void ); -extern void ap_init_freq_invariance (void );#else static inline void arch_set_max_freq_ratio (bool turbo_disabled) { }static inline void freq_invariance_set_perf_ratio (u64 ratio,bool turbo_disabled) { } -static inline void bp_init_freq_invariance (void ) { } -static inline void ap_init_freq_invariance (void ) { }#endif #ifdef CONFIG_ACPI_CPPC_LIB arch/x86/kernel/cpu/aperfmperf.c +38
-25@@ -17,6 +17,7 @@#include <linux/smp.h> #include <linux/syscore_ops.h> +#include <asm/cpu.h> #include <asm/cpu_device_id.h> #include <asm/intel-family.h> @@ -164,6 +165,17 @@ unsigned int arch_freq_get_on_cpu(int cpreturn per_cpu (samples.khz, cpu); } +static void init_counter_refs (void ) +{ +u64 aperf, mperf; + +rdmsrl (MSR_IA32_APERF , aperf); +rdmsrl (MSR_IA32_MPERF , mperf); + +this_cpu_write (cpu_samples.aperf, aperf); +this_cpu_write (cpu_samples.mperf, mperf); +} +#if defined(CONFIG_X86_64 ) && defined(CONFIG_SMP )/* * APERF/MPERF frequency ratio computation. @@ -405,17 +417,6 @@ static bool __init intel_set_max_freq_rareturn true; } -static void init_counter_refs (void ) -{ -u64 aperf, mperf; - -rdmsrl (MSR_IA32_APERF , aperf); -rdmsrl (MSR_IA32_MPERF , mperf); - -this_cpu_write (cpu_samples.aperf, aperf); -this_cpu_write (cpu_samples.mperf, mperf); -} -#ifdef CONFIG_PM_SLEEP static struct syscore_ops freq_invariance_syscore_ops = { .resume = init_counter_refs,@@ -447,13 +448,8 @@ void freq_invariance_set_perf_ratio(u64freq_invariance_enable (); } -void __init bp_init_freq_invariance (void ) +static void __init bp_init_freq_invariance (void ) { -if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF )) -return ; - -init_counter_refs (); -if (boot_cpu_data.x86_vendor !=X86_VENDOR_INTEL )return ;@@ -461,12 +457,6 @@ void __init bp_init_freq_invariance(voidfreq_invariance_enable (); } -void ap_init_freq_invariance (void ) -{ -if (cpu_feature_enabled (X86_FEATURE_APERFMPERF )) -init_counter_refs (); -} -static void disable_freq_invariance_workfn (struct work_struct *work) {static_branch_disable (&arch_scale_freq_key);@@ -481,6 +471,9 @@ static void scale_freq_tick(u64 acnt, u6 {u64 freq_scale; +if (!arch_scale_freq_invariant ()) +return ; +if (check_shl_overflow (acnt,2 *SCHED_CAPACITY_SHIFT , &acnt)) goto error;@@ -501,13 +494,17 @@ static void scale_freq_tick(u64 acnt, u6pr_warn ("Scheduler frequency invariance went wobbly, disabling!\n" ); schedule_work(&disable_freq_invariance_work); } +#else +static inline void bp_init_freq_invariance (void ) { } +static inline void scale_freq_tick (u64 acnt,u64 mcnt) { } +#endif /* CONFIG_X86_64 && CONFIG_SMP */ void arch_scale_freq_tick (void ) {struct aperfmperf *s =this_cpu_ptr (&cpu_samples);u64 acnt, mcnt, aperf, mperf; -if (!arch_scale_freq_invariant ()) +if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF ))return ;rdmsrl (MSR_IA32_APERF , aperf);@@ -520,4 +517,20 @@ void arch_scale_freq_tick(void)scale_freq_tick (acnt, mcnt); } -#endif /* CONFIG_X86_64 && CONFIG_SMP */ + +static int __init bp_init_aperfmperf (void ) +{ +if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF )) +return 0 ; + +init_counter_refs (); +bp_init_freq_invariance (); +return 0 ; +} +early_initcall (bp_init_aperfmperf); + +void ap_init_aperfmperf (void ) +{ +if (cpu_feature_enabled (X86_FEATURE_APERFMPERF )) +init_counter_refs (); +}arch/x86/kernel/smpboot.c +1
-2@@ -186,7 +186,7 @@ static void smp_callin(void)*/ set_cpu_sibling_map (raw_smp_processor_id ()); -ap_init_freq_invariance (); +ap_init_aperfmperf ();/* * Get our bogomips. @@ -1396,7 +1396,6 @@ void __init native_smp_prepare_cpus(unsi {smp_prepare_cpus_common (); -bp_init_freq_invariance ();smp_sanity_check ();switch (apic_intr_mode) {
[tip: x86/cleanups] x86/aperfmperf: Make parts of the frequency invariance code unconditional
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: 93989bbee7b21fa779ab22343b0463d395d020fd Gitweb: https://git.kernel.org/tip/93989bbee7b21fa779ab22343b0463d395d020fd Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 15 Apr 2022 21:19:59 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 15:51:08 +02:00
x86/aperfmperf: Make parts of the frequency invariance code unconditional
The frequency invariance support is currently limited to x86/64 and SMP, which is the vast majority of machines.
arch_scale_freq_tick() is called every tick on all CPUs and reads the APERF and MPERF MSRs. The CPU frequency getters function do the same via dedicated IPIs.
While it could be argued that on systems where frequency invariance support is disabled (32bit, !SMP) the per tick read of the APERF and MPERF MSRs can be avoided, it does not make sense to keep the extra code and the resulting runtime issues of mass IPIs around.
As a first step split out the non frequency invariance specific initialization code and the read MSR portion of arch_scale_freq_tick(). The rest of the code is still conditional and guarded with a static key.
--- arch/x86/include/asm/cpu.h | 2 +- arch/x86/include/asm/topology.h | 4 +-- arch/x86/kernel/cpu/aperfmperf.c | 63 ++++++++++++++++++------------- arch/x86/kernel/smpboot.c | 3 +- 4 files changed, 41 insertions(+), 31 deletions(-)
arch/x86/include/asm/cpu.h +2 -0
@@ -36,6 +36,8 @@ extern int _debug_hotplug_cpu(int cpu, int action);#endif #endif +extern void ap_init_aperfmperf (void ); +int mwait_usable (const struct cpuinfo_x86 *);unsigned int x86_family (unsigned int sig);
arch/x86/include/asm/topology.h +0 -4
@@ -217,13 +217,9 @@ extern void arch_scale_freq_tick(void);extern void arch_set_max_freq_ratio (bool turbo_disabled);extern void freq_invariance_set_perf_ratio (u64 ratio,bool turbo_disabled); -extern void bp_init_freq_invariance (void ); -extern void ap_init_freq_invariance (void );#else static inline void arch_set_max_freq_ratio (bool turbo_disabled) { }static inline void freq_invariance_set_perf_ratio (u64 ratio,bool turbo_disabled) { } -static inline void bp_init_freq_invariance (void ) { } -static inline void ap_init_freq_invariance (void ) { }#endif #ifdef CONFIG_ACPI_CPPC_LIB
arch/x86/kernel/cpu/aperfmperf.c +38 -25
115 lines not on this page · raw message
arch/x86/kernel/smpboot.c +1 -2
@@ -186,7 +186,7 @@ static void smp_callin(void)*/ set_cpu_sibling_map (raw_smp_processor_id ()); -ap_init_freq_invariance (); +ap_init_aperfmperf ();/* * Get our bogomips.
@@ -1396,7 +1396,6 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) {smp_prepare_cpus_common (); -bp_init_freq_invariance ();smp_sanity_check ();switch (apic_intr_mode) {
[tip: x86/cleanups] x86/aperfmperf: Make parts of the frequency invariance code unconditional
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: bb6e89df9028b2fab0ce6ac71cd9ef25b6ada32d Gitweb: https://git.kernel.org/tip/bb6e89df9028b2fab0ce6ac71cd9ef25b6ada32d Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 15 Apr 2022 21:19:59 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 20:22:19 +02:00
x86/aperfmperf: Make parts of the frequency invariance code unconditional
The frequency invariance support is currently limited to x86/64 and SMP, which is the vast majority of machines.
arch_scale_freq_tick() is called every tick on all CPUs and reads the APERF and MPERF MSRs. The CPU frequency getters function do the same via dedicated IPIs.
While it could be argued that on systems where frequency invariance support is disabled (32bit, !SMP) the per tick read of the APERF and MPERF MSRs can be avoided, it does not make sense to keep the extra code and the resulting runtime issues of mass IPIs around.
As a first step split out the non frequency invariance specific initialization code and the read MSR portion of arch_scale_freq_tick(). The rest of the code is still conditional and guarded with a static key.
--- arch/x86/include/asm/cpu.h | 2 +- arch/x86/include/asm/topology.h | 4 +-- arch/x86/kernel/cpu/aperfmperf.c | 63 ++++++++++++++++++------------- arch/x86/kernel/smpboot.c | 3 +- 4 files changed, 41 insertions(+), 31 deletions(-)
arch/x86/include/asm/cpu.h +2 -0
@@ -36,6 +36,8 @@ extern int _debug_hotplug_cpu(int cpu, int action);#endif #endif +extern void ap_init_aperfmperf (void ); +int mwait_usable (const struct cpuinfo_x86 *);unsigned int x86_family (unsigned int sig);
arch/x86/include/asm/topology.h +0 -4
@@ -217,13 +217,9 @@ extern void arch_scale_freq_tick(void);extern void arch_set_max_freq_ratio (bool turbo_disabled);extern void freq_invariance_set_perf_ratio (u64 ratio,bool turbo_disabled); -extern void bp_init_freq_invariance (void ); -extern void ap_init_freq_invariance (void );#else static inline void arch_set_max_freq_ratio (bool turbo_disabled) { }static inline void freq_invariance_set_perf_ratio (u64 ratio,bool turbo_disabled) { } -static inline void bp_init_freq_invariance (void ) { } -static inline void ap_init_freq_invariance (void ) { }#endif #ifdef CONFIG_ACPI_CPPC_LIB
arch/x86/kernel/cpu/aperfmperf.c +38 -25
115 lines not on this page · raw message
arch/x86/kernel/smpboot.c +1 -2
@@ -186,7 +186,7 @@ static void smp_callin(void)*/ set_cpu_sibling_map (raw_smp_processor_id ()); -ap_init_freq_invariance (); +ap_init_aperfmperf ();/* * Get our bogomips.
@@ -1396,7 +1396,6 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) {smp_prepare_cpus_common (); -bp_init_freq_invariance ();smp_sanity_check ();switch (apic_intr_mode) {
PATCH 8/10 x86/aperfmperf: Store aperf/mperf data for cpu frequency reads
In reply to Thomas GleixnerNow that the MSR readout is unconditional, store the results in the per CPU data structure along with a jiffies timestamp for the CPU frequency readout code.
--- arch/x86/kernel/cpu/aperfmperf.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/aperfmperf.c +13 -1
@@ -24,11 +24,17 @@#include "cpu.h" struct aperfmperf { +seqcount_t seq; +unsigned long last_update; +u64 acnt; +u64 mcnt;u64 aperf;u64 mperf; }; -static DEFINE_PER_CPU_SHARED_ALIGNED(struct aperfmperf, cpu_samples); +static DEFINE_PER_CPU_SHARED_ALIGNED(struct aperfmperf, cpu_samples) = { + .seq =SEQCNT_ZERO (cpu_samples.seq) +};struct aperfmperf_sample {unsigned int khz;
@@ -515,6 +521,12 @@ void arch_scale_freq_tick(void) s->aperf = aperf; s->mperf = mperf; +raw_write_seqcount_begin (&s->seq); + s->last_update = jiffies; + s->acnt = acnt; + s->mcnt = mcnt; +raw_write_seqcount_end (&s->seq); +scale_freq_tick (acnt, mcnt); }
Re: [patch 08/10] x86/aperfmperf: Store aperf/mperf data for cpu frequency reads
Quoted 6 lines · Thomas Gleixner
Now that the MSR readout is unconditional, store the results in the per CPU data structure along with a jiffies timestamp for the CPU frequency readout code.
Signed-off-by Thomas Gleixner <[email protected]>
Quoted 39 lines
--- arch/x86/kernel/cpu/aperfmperf.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)arch/x86/kernel/cpu/aperfmperf.c +13
-1@@ -24,11 +24,17 @@#include "cpu.h" struct aperfmperf { +seqcount_t seq; +unsigned long last_update; +u64 acnt; +u64 mcnt;u64 aperf;u64 mperf; }; -static DEFINE_PER_CPU_SHARED_ALIGNED(struct aperfmperf, cpu_samples); +static DEFINE_PER_CPU_SHARED_ALIGNED(struct aperfmperf, cpu_samples) = { + .seq =SEQCNT_ZERO (cpu_samples.seq) +};struct aperfmperf_sample {unsigned int khz;@@ -515,6 +521,12 @@ void arch_scale_freq_tick(void) s->aperf = aperf; s->mperf = mperf; +raw_write_seqcount_begin (&s->seq); + s->last_update = jiffies; + s->acnt = acnt; + s->mcnt = mcnt; +raw_write_seqcount_end (&s->seq); +scale_freq_tick (acnt, mcnt); }
[tip: x86/cleanups] x86/aperfmperf: Store aperf/mperf data for cpu frequency reads
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: 3b1fc17f635164d74934f67d3bb46cdf877fb67f Gitweb: https://git.kernel.org/tip/3b1fc17f635164d74934f67d3bb46cdf877fb67f Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 15 Apr 2022 21:20:01 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 15:51:08 +02:00
x86/aperfmperf: Store aperf/mperf data for cpu frequency reads
Now that the MSR readout is unconditional, store the results in the per CPU data structure along with a jiffies timestamp for the CPU frequency readout code.
--- arch/x86/kernel/cpu/aperfmperf.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/aperfmperf.c +13 -1
@@ -24,11 +24,17 @@#include "cpu.h" struct aperfmperf { +seqcount_t seq; +unsigned long last_update; +u64 acnt; +u64 mcnt;u64 aperf;u64 mperf; }; -static DEFINE_PER_CPU_SHARED_ALIGNED(struct aperfmperf, cpu_samples); +static DEFINE_PER_CPU_SHARED_ALIGNED(struct aperfmperf, cpu_samples) = { + .seq =SEQCNT_ZERO (cpu_samples.seq) +};struct aperfmperf_sample {unsigned int khz;
@@ -515,6 +521,12 @@ void arch_scale_freq_tick(void) s->aperf = aperf; s->mperf = mperf; +raw_write_seqcount_begin (&s->seq); + s->last_update = jiffies; + s->acnt = acnt; + s->mcnt = mcnt; +raw_write_seqcount_end (&s->seq); +scale_freq_tick (acnt, mcnt); }
[tip: x86/cleanups] x86/aperfmperf: Store aperf/mperf data for cpu frequency reads
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: cd8c0e142daf9de9ce594e61b75509b0af7bfb26 Gitweb: https://git.kernel.org/tip/cd8c0e142daf9de9ce594e61b75509b0af7bfb26 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 15 Apr 2022 21:20:01 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 20:22:19 +02:00
x86/aperfmperf: Store aperf/mperf data for cpu frequency reads
Now that the MSR readout is unconditional, store the results in the per CPU data structure along with a jiffies timestamp for the CPU frequency readout code.
--- arch/x86/kernel/cpu/aperfmperf.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/aperfmperf.c +13 -1
@@ -24,11 +24,17 @@#include "cpu.h" struct aperfmperf { +seqcount_t seq; +unsigned long last_update; +u64 acnt; +u64 mcnt;u64 aperf;u64 mperf; }; -static DEFINE_PER_CPU_SHARED_ALIGNED(struct aperfmperf, cpu_samples); +static DEFINE_PER_CPU_SHARED_ALIGNED(struct aperfmperf, cpu_samples) = { + .seq =SEQCNT_ZERO (cpu_samples.seq) +};struct aperfmperf_sample {unsigned int khz;
@@ -515,6 +521,12 @@ void arch_scale_freq_tick(void) s->aperf = aperf; s->mperf = mperf; +raw_write_seqcount_begin (&s->seq); + s->last_update = jiffies; + s->acnt = acnt; + s->mcnt = mcnt; +raw_write_seqcount_end (&s->seq); +scale_freq_tick (acnt, mcnt); }
PATCH 9/10 x86/aperfmperf: Replace aperfmperf_get_khz()
In reply to Thomas GleixnerThe frequency invariance infrastructure provides the APERF/MPERF samples already. Utilize them for the cpu frequency display in /proc/cpuinfo.
The sample is considered valid for 20ms. So for idle or isolated NOHZ full CPUs the function returns 0, which is matching the previous behaviour.
This gets rid of the mass IPIs and a delay of 20ms for stabilizing observed by Eric when reading /proc/cpuinfo.
--- arch/x86/kernel/cpu/aperfmperf.c | 77 +++++++++++++++++---------------------- fs/proc/cpuinfo.c | 6 --- include/linux/cpufreq.h | 1 3 files changed, 35 insertions(+), 49 deletions(-)
arch/x86/kernel/cpu/aperfmperf.c +34 -43
@@ -100,49 +100,6 @@ static bool aperfmperf_snapshot_cpu(intreturn time_delta <=APERFMPERF_STALE_THRESHOLD_MS ; } -unsigned int aperfmperf_get_khz (int cpu) -{ -if (!cpu_khz) -return 0 ; - -if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) -return 0 ; - -if (!housekeeping_cpu (cpu,HK_TYPE_MISC )) -return 0 ; - -if (rcu_is_idle_cpu (cpu)) -return 0 ;/* Idle CPUs are completely uninteresting. */ - -aperfmperf_snapshot_cpu (cpu,ktime_get (), true); -return per_cpu (samples.khz, cpu); -} - -void arch_freq_prepare_all (void ) -{ -ktime_t now =ktime_get (); -bool wait = false; -int cpu; - -if (!cpu_khz) -return ; - -if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) -return ; - -for_each_online_cpu (cpu) { -if (!housekeeping_cpu (cpu,HK_TYPE_MISC )) -continue ; -if (rcu_is_idle_cpu (cpu)) -continue ;/* Idle CPUs are completely uninteresting. */ -if (!aperfmperf_snapshot_cpu (cpu, now, false)) - wait = true; - } - -if (wait) -msleep (APERFMPERF_REFRESH_DELAY_MS ); -} -unsigned int arch_freq_get_on_cpu (int cpu) {struct aperfmperf_sample *s =per_cpu_ptr (&samples, cpu);
@@ -529,6 +486,40 @@ void arch_scale_freq_tick(void)scale_freq_tick (acnt, mcnt); } +/* +* Discard samples older than the define maximum sample age of 20ms. There +* is no point in sending IPIs in such a case. If the scheduler tick was +* not running then the CPU is either idle or isolated. +*/ +#define MAX_SAMPLE_AGE ((unsigned long)HZ / 50) + +unsigned int aperfmperf_get_khz (int cpu) +{ +struct aperfmperf *s =per_cpu_ptr (&cpu_samples, cpu); +unsigned long last; +unsigned int seq; +u64 acnt, mcnt; + +if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF )) +return 0 ; + +do { + seq =raw_read_seqcount_begin (&s->seq); + last = s->last_update; + acnt = s->acnt; + mcnt = s->mcnt; + }while (read_seqcount_retry (&s->seq, seq)); + +/* +* Bail on invalid count and when the last update was too long ago, +* which covers idle and NOHZ full CPUs. +*/ +if (!mcnt || (jiffies - last) >MAX_SAMPLE_AGE ) +return 0 ; + +return div64_u64 ((cpu_khz * acnt), mcnt); +} +static int __init bp_init_aperfmperf (void ) {if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF ))
fs/proc/cpuinfo.c +1 -5
@@ -5,14 +5,10 @@#include <linux/proc_fs.h> #include <linux/seq_file.h> -__weak void arch_freq_prepare_all (void ) -{ -} -extern const struct seq_operations cpuinfo_op; +static int cpuinfo_open (struct inode *inode,struct file *file) { -arch_freq_prepare_all ();return seq_open (file, &cpuinfo_op); }
include/linux/cpufreq.h +0 -1
@@ -1199,7 +1199,6 @@ static inline void sched_cpufreq_governo struct cpufreq_governor *old_gov) { }#endif -extern void arch_freq_prepare_all (void );extern unsigned int arch_freq_get_on_cpu (int cpu);#ifndef arch_set_freq_scale
Re: [patch 09/10] x86/aperfmperf: Replace aperfmperf_get_khz()
Quoted 4 lines · Thomas Gleixner
The frequency invariance infrastructure provides the APERF/MPERF samples already. Utilize them for the cpu frequency display in
/proc/cpuinfo.
Quoted 8 lines
The sample is considered valid for 20ms. So for idle or isolated NOHZ full CPUs the function returns 0, which is matching the previous behaviour.
This gets rid of the mass IPIs and a delay of 20ms for stabilizing observed by Eric when reading
/proc/cpuinfo.Reported-by Eric Dumazet <[email protected]>Signed-off-by Thomas Gleixner <[email protected]>
All fine IMV, one minor nit below.
Quoted 81 lines
--- arch/x86/kernel/cpu/aperfmperf.c | 77 +++++++++++++++++---------------------- fs/proc/cpuinfo.c | 6 --- include/linux/cpufreq.h | 1 3 files changed, 35 insertions(+), 49 deletions(-)arch/x86/kernel/cpu/aperfmperf.c +19
-43@@ -100,49 +100,6 @@ static bool aperfmperf_snapshot_cpu(intreturn time_delta <=APERFMPERF_STALE_THRESHOLD_MS ; } -unsigned int aperfmperf_get_khz (int cpu) -{ -if (!cpu_khz) -return 0 ; - -if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) -return 0 ; - -if (!housekeeping_cpu (cpu,HK_TYPE_MISC )) -return 0 ; - -if (rcu_is_idle_cpu (cpu)) -return 0 ;/* Idle CPUs are completely uninteresting. */ - -aperfmperf_snapshot_cpu (cpu,ktime_get (), true); -return per_cpu (samples.khz, cpu); -} - -void arch_freq_prepare_all (void ) -{ -ktime_t now =ktime_get (); -bool wait = false; -int cpu; - -if (!cpu_khz) -return ; - -if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) -return ; - -for_each_online_cpu (cpu) { -if (!housekeeping_cpu (cpu,HK_TYPE_MISC )) -continue ; -if (rcu_is_idle_cpu (cpu)) -continue ;/* Idle CPUs are completely uninteresting. */ -if (!aperfmperf_snapshot_cpu (cpu, now, false)) - wait = true; - } - -if (wait) -msleep (APERFMPERF_REFRESH_DELAY_MS ); -} -unsigned int arch_freq_get_on_cpu (int cpu) {struct aperfmperf_sample *s =per_cpu_ptr (&samples, cpu);@@ -529,6 +486,40 @@ void arch_scale_freq_tick(void)scale_freq_tick (acnt, mcnt); } +/* +* Discard samples older than the define maximum sample age of 20ms. There +* is no point in sending IPIs in such a case. If the scheduler tick was +* not running then the CPU is either idle or isolated. +*/ +#define MAX_SAMPLE_AGE ((unsigned long)HZ / 50) + +unsigned int aperfmperf_get_khz (int cpu) +{ +struct aperfmperf *s =per_cpu_ptr (&cpu_samples, cpu); +unsigned long last; +unsigned int seq; +u64 acnt, mcnt; + +if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF )) +return 0 ; + + do { + seq = raw_read_seqcount_begin(&s->seq);
arch/x86/kernel/cpu/aperfmperf.c void arch_scale_freq_tick(void) · line 517
Quoted 10 lines
arch/x86/kernel/cpu/aperfmperf.c +10
-0+ last = s->last_update; + acnt = s->acnt; + mcnt = s->mcnt; + }while (read_seqcount_retry (&s->seq, seq)); + +/* +* Bail on invalid count and when the last update was too long ago, +* which covers idle and NOHZ full CPUs. +*/ +if (!mcnt || (jiffies - last) >MAX_SAMPLE_AGE )
The inner parens are not needed here.
Quoted 37 lines
arch/x86/kernel/cpu/aperfmperf.c +5
-0+return 0 ; + +return div64_u64 ((cpu_khz * acnt), mcnt); +} +static int __init bp_init_aperfmperf (void ) {if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF ))fs/proc/cpuinfo.c +1
-5@@ -5,14 +5,10 @@#include <linux/proc_fs.h> #include <linux/seq_file.h> -__weak void arch_freq_prepare_all (void ) -{ -} -extern const struct seq_operations cpuinfo_op; +static int cpuinfo_open (struct inode *inode,struct file *file) { -arch_freq_prepare_all ();return seq_open (file, &cpuinfo_op); }include/linux/cpufreq.h +0
-1@@ -1199,7 +1199,6 @@ static inline void sched_cpufreq_governo struct cpufreq_governor *old_gov) { }#endif -extern void arch_freq_prepare_all (void );extern unsigned int arch_freq_get_on_cpu (int cpu);#ifndef arch_set_freq_scale
[tip: x86/cleanups] x86/aperfmperf: Replace aperfmperf_get_khz()
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: 61551f094837f77952eba2fdf8b913bb5b191ced Gitweb: https://git.kernel.org/tip/61551f094837f77952eba2fdf8b913bb5b191ced Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 15 Apr 2022 21:20:02 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 15:51:08 +02:00
x86/aperfmperf: Replace aperfmperf_get_khz()
The frequency invariance infrastructure provides the APERF/MPERF samples already. Utilize them for the cpu frequency display in /proc/cpuinfo.
The sample is considered valid for 20ms. So for idle or isolated NOHZ full CPUs the function returns 0, which is matching the previous behaviour.
This gets rid of the mass IPIs and a delay of 20ms for stabilizing observed by Eric when reading /proc/cpuinfo.
--- arch/x86/kernel/cpu/aperfmperf.c | 77 +++++++++++++------------------ fs/proc/cpuinfo.c | 6 +-- include/linux/cpufreq.h | 1 +- 3 files changed, 35 insertions(+), 49 deletions(-)
arch/x86/kernel/cpu/aperfmperf.c +34 -43
@@ -101,49 +101,6 @@ static bool aperfmperf_snapshot_cpu(int cpu, ktime_t now, bool wait)return time_delta <=APERFMPERF_STALE_THRESHOLD_MS ; } -unsigned int aperfmperf_get_khz (int cpu) -{ -if (!cpu_khz) -return 0 ; - -if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) -return 0 ; - -if (!housekeeping_cpu (cpu,HK_TYPE_MISC )) -return 0 ; - -if (rcu_is_idle_cpu (cpu)) -return 0 ;/* Idle CPUs are completely uninteresting. */ - -aperfmperf_snapshot_cpu (cpu,ktime_get (), true); -return per_cpu (samples.khz, cpu); -} - -void arch_freq_prepare_all (void ) -{ -ktime_t now =ktime_get (); -bool wait = false; -int cpu; - -if (!cpu_khz) -return ; - -if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) -return ; - -for_each_online_cpu (cpu) { -if (!housekeeping_cpu (cpu,HK_TYPE_MISC )) -continue ; -if (rcu_is_idle_cpu (cpu)) -continue ;/* Idle CPUs are completely uninteresting. */ -if (!aperfmperf_snapshot_cpu (cpu, now, false)) - wait = true; - } - -if (wait) -msleep (APERFMPERF_REFRESH_DELAY_MS ); -} -unsigned int arch_freq_get_on_cpu (int cpu) {struct aperfmperf_sample *s =per_cpu_ptr (&samples, cpu);
@@ -530,6 +487,40 @@ void arch_scale_freq_tick(void)scale_freq_tick (acnt, mcnt); } +/* +* Discard samples older than the define maximum sample age of 20ms. There +* is no point in sending IPIs in such a case. If the scheduler tick was +* not running then the CPU is either idle or isolated. +*/ +#define MAX_SAMPLE_AGE ((unsigned long)HZ / 50) + +unsigned int aperfmperf_get_khz (int cpu) +{ +struct aperfmperf *s =per_cpu_ptr (&cpu_samples, cpu); +unsigned long last; +unsigned int seq; +u64 acnt, mcnt; + +if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF )) +return 0 ; + +do { + seq =raw_read_seqcount_begin (&s->seq); + last = s->last_update; + acnt = s->acnt; + mcnt = s->mcnt; + }while (read_seqcount_retry (&s->seq, seq)); + +/* +* Bail on invalid count and when the last update was too long ago, +* which covers idle and NOHZ full CPUs. +*/ +if (!mcnt || (jiffies - last) >MAX_SAMPLE_AGE ) +return 0 ; + +return div64_u64 ((cpu_khz * acnt), mcnt); +} +static int __init bp_init_aperfmperf (void ) {if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF ))
fs/proc/cpuinfo.c +1 -5
@@ -5,14 +5,10 @@#include <linux/proc_fs.h> #include <linux/seq_file.h> -__weak void arch_freq_prepare_all (void ) -{ -} -extern const struct seq_operations cpuinfo_op; +static int cpuinfo_open (struct inode *inode,struct file *file) { -arch_freq_prepare_all ();return seq_open (file, &cpuinfo_op); }
include/linux/cpufreq.h +0 -1
@@ -1199,7 +1199,6 @@ static inline void sched_cpufreq_governor_change(struct cpufreq_policy *policy, struct cpufreq_governor *old_gov) { }#endif -extern void arch_freq_prepare_all (void );extern unsigned int arch_freq_get_on_cpu (int cpu);#ifndef arch_set_freq_scale
[tip: x86/cleanups] x86/aperfmperf: Replace aperfmperf_get_khz()
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: 7d84c1ebf9ddafca27b481e6da7d24a023dacaa2 Gitweb: https://git.kernel.org/tip/7d84c1ebf9ddafca27b481e6da7d24a023dacaa2 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 15 Apr 2022 21:20:02 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 20:22:19 +02:00
x86/aperfmperf: Replace aperfmperf_get_khz()
The frequency invariance infrastructure provides the APERF/MPERF samples already. Utilize them for the cpu frequency display in /proc/cpuinfo.
The sample is considered valid for 20ms. So for idle or isolated NOHZ full CPUs the function returns 0, which is matching the previous behaviour.
This gets rid of the mass IPIs and a delay of 20ms for stabilizing observed by Eric when reading /proc/cpuinfo.
--- arch/x86/kernel/cpu/aperfmperf.c | 77 +++++++++++++------------------ fs/proc/cpuinfo.c | 6 +-- include/linux/cpufreq.h | 1 +- 3 files changed, 35 insertions(+), 49 deletions(-)
arch/x86/kernel/cpu/aperfmperf.c +34 -43
@@ -101,49 +101,6 @@ static bool aperfmperf_snapshot_cpu(int cpu, ktime_t now, bool wait)return time_delta <=APERFMPERF_STALE_THRESHOLD_MS ; } -unsigned int aperfmperf_get_khz (int cpu) -{ -if (!cpu_khz) -return 0 ; - -if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) -return 0 ; - -if (!housekeeping_cpu (cpu,HK_TYPE_MISC )) -return 0 ; - -if (rcu_is_idle_cpu (cpu)) -return 0 ;/* Idle CPUs are completely uninteresting. */ - -aperfmperf_snapshot_cpu (cpu,ktime_get (), true); -return per_cpu (samples.khz, cpu); -} - -void arch_freq_prepare_all (void ) -{ -ktime_t now =ktime_get (); -bool wait = false; -int cpu; - -if (!cpu_khz) -return ; - -if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) -return ; - -for_each_online_cpu (cpu) { -if (!housekeeping_cpu (cpu,HK_TYPE_MISC )) -continue ; -if (rcu_is_idle_cpu (cpu)) -continue ;/* Idle CPUs are completely uninteresting. */ -if (!aperfmperf_snapshot_cpu (cpu, now, false)) - wait = true; - } - -if (wait) -msleep (APERFMPERF_REFRESH_DELAY_MS ); -} -unsigned int arch_freq_get_on_cpu (int cpu) {struct aperfmperf_sample *s =per_cpu_ptr (&samples, cpu);
@@ -530,6 +487,40 @@ void arch_scale_freq_tick(void)scale_freq_tick (acnt, mcnt); } +/* +* Discard samples older than the define maximum sample age of 20ms. There +* is no point in sending IPIs in such a case. If the scheduler tick was +* not running then the CPU is either idle or isolated. +*/ +#define MAX_SAMPLE_AGE ((unsigned long)HZ / 50) + +unsigned int aperfmperf_get_khz (int cpu) +{ +struct aperfmperf *s =per_cpu_ptr (&cpu_samples, cpu); +unsigned long last; +unsigned int seq; +u64 acnt, mcnt; + +if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF )) +return 0 ; + +do { + seq =raw_read_seqcount_begin (&s->seq); + last = s->last_update; + acnt = s->acnt; + mcnt = s->mcnt; + }while (read_seqcount_retry (&s->seq, seq)); + +/* +* Bail on invalid count and when the last update was too long ago, +* which covers idle and NOHZ full CPUs. +*/ +if (!mcnt || (jiffies - last) >MAX_SAMPLE_AGE ) +return 0 ; + +return div64_u64 ((cpu_khz * acnt), mcnt); +} +static int __init bp_init_aperfmperf (void ) {if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF ))
fs/proc/cpuinfo.c +1 -5
@@ -5,14 +5,10 @@#include <linux/proc_fs.h> #include <linux/seq_file.h> -__weak void arch_freq_prepare_all (void ) -{ -} -extern const struct seq_operations cpuinfo_op; +static int cpuinfo_open (struct inode *inode,struct file *file) { -arch_freq_prepare_all ();return seq_open (file, &cpuinfo_op); }
include/linux/cpufreq.h +0 -1
@@ -1199,7 +1199,6 @@ static inline void sched_cpufreq_governor_change(struct cpufreq_policy *policy, struct cpufreq_governor *old_gov) { }#endif -extern void arch_freq_prepare_all (void );extern unsigned int arch_freq_get_on_cpu (int cpu);#ifndef arch_set_freq_scale
PATCH 10/10 x86/aperfmperf: Replace arch_freq_get_on_cpu()
In reply to Thomas GleixnerReading the current CPU frequency from /sys/..../scaling_cur_freq involves in the worst case two IPIs due to the ad hoc sampling.
The frequency invariance infrastructure provides the APERF/MPERF samples already. Utilize them and consolidate this with the /proc/cpuinfo readout.
The sample is considered valid for 20ms. So for idle or isolated NOHZ full CPUs the function returns 0, which is matching the previous behaviour.
The resulting text size vs. the original APERF/MPERF plus the separate frequency invariance code:
text: 2411 -> 723
init.text: 0 -> 767
--- arch/x86/kernel/cpu/aperfmperf.c | 94 --------------------------------------- arch/x86/kernel/cpu/proc.c | 2 2 files changed, 2 insertions(+), 94 deletions(-)
arch/x86/kernel/cpu/aperfmperf.c +1 -93
@@ -35,98 +35,6 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(str .seq = SEQCNT_ZERO(cpu_samples.seq) }; -struct aperfmperf_sample { -unsigned int khz; -atomic_t scfpending; -ktime_t time; -u64 aperf; -u64 mperf; -}; - -static DEFINE_PER_CPU(struct aperfmperf_sample, samples); - -#define APERFMPERF_CACHE_THRESHOLD_MS 10 -#define APERFMPERF_REFRESH_DELAY_MS 10 -#define APERFMPERF_STALE_THRESHOLD_MS 1000 - -/* -* aperfmperf_snapshot_khz() -* On the current CPU, snapshot APERF, MPERF, and jiffies -* unless we already did it within 10ms -* calculate kHz, save snapshot -*/ -static void aperfmperf_snapshot_khz (void *dummy) -{ -u64 aperf, aperf_delta; -u64 mperf, mperf_delta; -struct aperfmperf_sample *s =this_cpu_ptr (&samples); -unsigned long flags; - -local_irq_save (flags); -rdmsrl (MSR_IA32_APERF , aperf); -rdmsrl (MSR_IA32_MPERF , mperf); -local_irq_restore (flags); - - aperf_delta = aperf - s->aperf; - mperf_delta = mperf - s->mperf; - -/* -* There is no architectural guarantee that MPERF -* increments faster than we can read it. -*/ -if (mperf_delta ==0 ) -return ; - - s->time =ktime_get (); - s->aperf = aperf; - s->mperf = mperf; - s->khz =div64_u64 ((cpu_khz * aperf_delta), mperf_delta); -atomic_set_release (&s->scfpending,0 ); -} - -static bool aperfmperf_snapshot_cpu (int cpu,ktime_t now,bool wait) -{ -s64 time_delta =ktime_ms_delta (now,per_cpu (samples.time, cpu)); -struct aperfmperf_sample *s =per_cpu_ptr (&samples, cpu); - -/* Don't bother re-computing within the cache threshold time. */ -if (time_delta <APERFMPERF_CACHE_THRESHOLD_MS ) -return true; - -if (!atomic_xchg (&s->scfpending,1 ) || wait) -smp_call_function_single (cpu, aperfmperf_snapshot_khz,NULL , wait); - -/* Return false if the previous iteration was too long ago. */ -return time_delta <=APERFMPERF_STALE_THRESHOLD_MS ; -} - -unsigned int arch_freq_get_on_cpu (int cpu) -{ -struct aperfmperf_sample *s =per_cpu_ptr (&samples, cpu); - -if (!cpu_khz) -return 0 ; - -if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) -return 0 ; - -if (!housekeeping_cpu (cpu,HK_TYPE_MISC )) -return 0 ; - -if (rcu_is_idle_cpu (cpu)) -return 0 ; - -if (aperfmperf_snapshot_cpu (cpu,ktime_get (), true)) -return per_cpu (samples.khz, cpu); - -msleep (APERFMPERF_REFRESH_DELAY_MS ); -atomic_set (&s->scfpending,1 ); -smp_mb ();/* ->scfpending before smp_call_function_single(). */ -smp_call_function_single (cpu, aperfmperf_snapshot_khz,NULL ,1 ); - -return per_cpu (samples.khz, cpu); -} -static void init_counter_refs (void ) {u64 aperf, mperf;
@@ -493,7 +401,7 @@ void arch_scale_freq_tick(void)*/ #define MAX_SAMPLE_AGE ((unsigned long)HZ / 50) -unsigned int aperfmperf_get_khz (int cpu) +unsigned int arch_freq_get_on_cpu (int cpu) {struct aperfmperf *s =per_cpu_ptr (&cpu_samples, cpu);unsigned long last;
arch/x86/kernel/cpu/proc.c +1 -1
@@ -84,7 +84,7 @@ static int show_cpuinfo(struct seq_fileseq_printf (m,"microcode\t: 0x%x\n" , c->microcode);if (cpu_has (c,X86_FEATURE_TSC )) { -unsigned int freq =aperfmperf_get_khz (cpu); +unsigned int freq =arch_freq_get_on_cpu (cpu);if (!freq) freq =cpufreq_quick_get (cpu);
Re: [patch 10/10] x86/aperfmperf: Replace arch_freq_get_on_cpu()
Quoted 10 lines · Thomas Gleixner
Reading the current CPU frequency from
/sys/..../scaling_cur_freqinvolves in the worst case two IPIs due to the ad hoc sampling.The frequency invariance infrastructure provides the APERF/MPERF samples already. Utilize them and consolidate this with the
/proc/cpuinforeadout.The sample is considered valid for 20ms. So for idle or isolated NOHZ full CPUs the function returns 0, which is matching the previous behaviour.
Quoted 7 lines
The resulting text size vs. the original APERF/MPERF plus the separate frequency invariance code:
text: 2411 -> 723 init.text: 0 -> 767Signed-off-by Thomas Gleixner <[email protected]>
All good AFAICS.
Quoted 127 lines
--- arch/x86/kernel/cpu/aperfmperf.c | 94 --------------------------------------- arch/x86/kernel/cpu/proc.c | 2 2 files changed, 2 insertions(+), 94 deletions(-)arch/x86/kernel/cpu/aperfmperf.c +1
-93@@ -35,98 +35,6 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(str .seq = SEQCNT_ZERO(cpu_samples.seq) }; -struct aperfmperf_sample { -unsigned int khz; -atomic_t scfpending; -ktime_t time; -u64 aperf; -u64 mperf; -}; - -static DEFINE_PER_CPU(struct aperfmperf_sample, samples); - -#define APERFMPERF_CACHE_THRESHOLD_MS 10 -#define APERFMPERF_REFRESH_DELAY_MS 10 -#define APERFMPERF_STALE_THRESHOLD_MS 1000 - -/* -* aperfmperf_snapshot_khz() -* On the current CPU, snapshot APERF, MPERF, and jiffies -* unless we already did it within 10ms -* calculate kHz, save snapshot -*/ -static void aperfmperf_snapshot_khz (void *dummy) -{ -u64 aperf, aperf_delta; -u64 mperf, mperf_delta; -struct aperfmperf_sample *s =this_cpu_ptr (&samples); -unsigned long flags; - -local_irq_save (flags); -rdmsrl (MSR_IA32_APERF , aperf); -rdmsrl (MSR_IA32_MPERF , mperf); -local_irq_restore (flags); - - aperf_delta = aperf - s->aperf; - mperf_delta = mperf - s->mperf; - -/* -* There is no architectural guarantee that MPERF -* increments faster than we can read it. -*/ -if (mperf_delta ==0 ) -return ; - - s->time =ktime_get (); - s->aperf = aperf; - s->mperf = mperf; - s->khz =div64_u64 ((cpu_khz * aperf_delta), mperf_delta); -atomic_set_release (&s->scfpending,0 ); -} - -static bool aperfmperf_snapshot_cpu (int cpu,ktime_t now,bool wait) -{ -s64 time_delta =ktime_ms_delta (now,per_cpu (samples.time, cpu)); -struct aperfmperf_sample *s =per_cpu_ptr (&samples, cpu); - -/* Don't bother re-computing within the cache threshold time. */ -if (time_delta <APERFMPERF_CACHE_THRESHOLD_MS ) -return true; - -if (!atomic_xchg (&s->scfpending,1 ) || wait) -smp_call_function_single (cpu, aperfmperf_snapshot_khz,NULL , wait); - -/* Return false if the previous iteration was too long ago. */ -return time_delta <=APERFMPERF_STALE_THRESHOLD_MS ; -} - -unsigned int arch_freq_get_on_cpu (int cpu) -{ -struct aperfmperf_sample *s =per_cpu_ptr (&samples, cpu); - -if (!cpu_khz) -return 0 ; - -if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) -return 0 ; - -if (!housekeeping_cpu (cpu,HK_TYPE_MISC )) -return 0 ; - -if (rcu_is_idle_cpu (cpu)) -return 0 ; - -if (aperfmperf_snapshot_cpu (cpu,ktime_get (), true)) -return per_cpu (samples.khz, cpu); - -msleep (APERFMPERF_REFRESH_DELAY_MS ); -atomic_set (&s->scfpending,1 ); -smp_mb ();/* ->scfpending before smp_call_function_single(). */ -smp_call_function_single (cpu, aperfmperf_snapshot_khz,NULL ,1 ); - -return per_cpu (samples.khz, cpu); -} -static void init_counter_refs (void ) {u64 aperf, mperf;@@ -493,7 +401,7 @@ void arch_scale_freq_tick(void)*/ #define MAX_SAMPLE_AGE ((unsigned long)HZ / 50) -unsigned int aperfmperf_get_khz (int cpu) +unsigned int arch_freq_get_on_cpu (int cpu) {struct aperfmperf *s =per_cpu_ptr (&cpu_samples, cpu);unsigned long last;arch/x86/kernel/cpu/proc.c +1
-1@@ -84,7 +84,7 @@ static int show_cpuinfo(struct seq_fileseq_printf (m,"microcode\t: 0x%x\n" , c->microcode);if (cpu_has (c,X86_FEATURE_TSC )) { -unsigned int freq =aperfmperf_get_khz (cpu); +unsigned int freq =arch_freq_get_on_cpu (cpu);if (!freq) freq =cpufreq_quick_get (cpu);
[tip: x86/cleanups] x86/aperfmperf: Replace arch_freq_get_on_cpu()
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: a86a1356ee79da6bc4abfb9e0499f963ff6da9ae Gitweb: https://git.kernel.org/tip/a86a1356ee79da6bc4abfb9e0499f963ff6da9ae Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 15 Apr 2022 21:20:04 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 15:51:09 +02:00
x86/aperfmperf: Replace arch_freq_get_on_cpu()
Reading the current CPU frequency from /sys/..../scaling_cur_freq involves in the worst case two IPIs due to the ad hoc sampling.
The frequency invariance infrastructure provides the APERF/MPERF samples already. Utilize them and consolidate this with the /proc/cpuinfo readout.
The sample is considered valid for 20ms. So for idle or isolated NOHZ full CPUs the function returns 0, which is matching the previous behaviour.
The resulting text size vs. the original APERF/MPERF plus the separate frequency invariance code:
text: 2411 -> 723
init.text: 0 -> 767
--- arch/x86/kernel/cpu/aperfmperf.c | 94 +------------------------------- arch/x86/kernel/cpu/proc.c | 2 +- 2 files changed, 2 insertions(+), 94 deletions(-)
arch/x86/kernel/cpu/aperfmperf.c +1 -93
@@ -36,98 +36,6 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(struct aperfmperf, cpu_samples) = { .seq = SEQCNT_ZERO(cpu_samples.seq) }; -struct aperfmperf_sample { -unsigned int khz; -atomic_t scfpending; -ktime_t time; -u64 aperf; -u64 mperf; -}; - -static DEFINE_PER_CPU(struct aperfmperf_sample, samples); - -#define APERFMPERF_CACHE_THRESHOLD_MS 10 -#define APERFMPERF_REFRESH_DELAY_MS 10 -#define APERFMPERF_STALE_THRESHOLD_MS 1000 - -/* -* aperfmperf_snapshot_khz() -* On the current CPU, snapshot APERF, MPERF, and jiffies -* unless we already did it within 10ms -* calculate kHz, save snapshot -*/ -static void aperfmperf_snapshot_khz (void *dummy) -{ -u64 aperf, aperf_delta; -u64 mperf, mperf_delta; -struct aperfmperf_sample *s =this_cpu_ptr (&samples); -unsigned long flags; - -local_irq_save (flags); -rdmsrl (MSR_IA32_APERF , aperf); -rdmsrl (MSR_IA32_MPERF , mperf); -local_irq_restore (flags); - - aperf_delta = aperf - s->aperf; - mperf_delta = mperf - s->mperf; - -/* -* There is no architectural guarantee that MPERF -* increments faster than we can read it. -*/ -if (mperf_delta ==0 ) -return ; - - s->time =ktime_get (); - s->aperf = aperf; - s->mperf = mperf; - s->khz =div64_u64 ((cpu_khz * aperf_delta), mperf_delta); -atomic_set_release (&s->scfpending,0 ); -} - -static bool aperfmperf_snapshot_cpu (int cpu,ktime_t now,bool wait) -{ -s64 time_delta =ktime_ms_delta (now,per_cpu (samples.time, cpu)); -struct aperfmperf_sample *s =per_cpu_ptr (&samples, cpu); - -/* Don't bother re-computing within the cache threshold time. */ -if (time_delta <APERFMPERF_CACHE_THRESHOLD_MS ) -return true; - -if (!atomic_xchg (&s->scfpending,1 ) || wait) -smp_call_function_single (cpu, aperfmperf_snapshot_khz,NULL , wait); - -/* Return false if the previous iteration was too long ago. */ -return time_delta <=APERFMPERF_STALE_THRESHOLD_MS ; -} - -unsigned int arch_freq_get_on_cpu (int cpu) -{ -struct aperfmperf_sample *s =per_cpu_ptr (&samples, cpu); - -if (!cpu_khz) -return 0 ; - -if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) -return 0 ; - -if (!housekeeping_cpu (cpu,HK_TYPE_MISC )) -return 0 ; - -if (rcu_is_idle_cpu (cpu)) -return 0 ; - -if (aperfmperf_snapshot_cpu (cpu,ktime_get (), true)) -return per_cpu (samples.khz, cpu); - -msleep (APERFMPERF_REFRESH_DELAY_MS ); -atomic_set (&s->scfpending,1 ); -smp_mb ();/* ->scfpending before smp_call_function_single(). */ -smp_call_function_single (cpu, aperfmperf_snapshot_khz,NULL ,1 ); - -return per_cpu (samples.khz, cpu); -} -static void init_counter_refs (void ) {u64 aperf, mperf;
@@ -494,7 +402,7 @@ void arch_scale_freq_tick(void)*/ #define MAX_SAMPLE_AGE ((unsigned long)HZ / 50) -unsigned int aperfmperf_get_khz (int cpu) +unsigned int arch_freq_get_on_cpu (int cpu) {struct aperfmperf *s =per_cpu_ptr (&cpu_samples, cpu);unsigned long last;
arch/x86/kernel/cpu/proc.c +1 -1
@@ -84,7 +84,7 @@ static int show_cpuinfo(struct seq_file *m, void *v)seq_printf (m,"microcode\t: 0x%x\n" , c->microcode);if (cpu_has (c,X86_FEATURE_TSC )) { -unsigned int freq =aperfmperf_get_khz (cpu); +unsigned int freq =arch_freq_get_on_cpu (cpu);if (!freq) freq =cpufreq_quick_get (cpu);
[tip: x86/cleanups] x86/aperfmperf: Replace arch_freq_get_on_cpu()
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: f3eca381bd49d708073ba1a9af4fa6ea5d5810a6 Gitweb: https://git.kernel.org/tip/f3eca381bd49d708073ba1a9af4fa6ea5d5810a6 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 15 Apr 2022 21:20:04 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 20:22:19 +02:00
x86/aperfmperf: Replace arch_freq_get_on_cpu()
Reading the current CPU frequency from /sys/..../scaling_cur_freq involves in the worst case two IPIs due to the ad hoc sampling.
The frequency invariance infrastructure provides the APERF/MPERF samples already. Utilize them and consolidate this with the /proc/cpuinfo readout.
The sample is considered valid for 20ms. So for idle or isolated NOHZ full CPUs the function returns 0, which is matching the previous behaviour.
The resulting text size vs. the original APERF/MPERF plus the separate frequency invariance code:
text: 2411 -> 723
init.text: 0 -> 767
--- arch/x86/kernel/cpu/aperfmperf.c | 94 +------------------------------- arch/x86/kernel/cpu/proc.c | 2 +- 2 files changed, 2 insertions(+), 94 deletions(-)
arch/x86/kernel/cpu/aperfmperf.c +1 -93
@@ -36,98 +36,6 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(struct aperfmperf, cpu_samples) = { .seq = SEQCNT_ZERO(cpu_samples.seq) }; -struct aperfmperf_sample { -unsigned int khz; -atomic_t scfpending; -ktime_t time; -u64 aperf; -u64 mperf; -}; - -static DEFINE_PER_CPU(struct aperfmperf_sample, samples); - -#define APERFMPERF_CACHE_THRESHOLD_MS 10 -#define APERFMPERF_REFRESH_DELAY_MS 10 -#define APERFMPERF_STALE_THRESHOLD_MS 1000 - -/* -* aperfmperf_snapshot_khz() -* On the current CPU, snapshot APERF, MPERF, and jiffies -* unless we already did it within 10ms -* calculate kHz, save snapshot -*/ -static void aperfmperf_snapshot_khz (void *dummy) -{ -u64 aperf, aperf_delta; -u64 mperf, mperf_delta; -struct aperfmperf_sample *s =this_cpu_ptr (&samples); -unsigned long flags; - -local_irq_save (flags); -rdmsrl (MSR_IA32_APERF , aperf); -rdmsrl (MSR_IA32_MPERF , mperf); -local_irq_restore (flags); - - aperf_delta = aperf - s->aperf; - mperf_delta = mperf - s->mperf; - -/* -* There is no architectural guarantee that MPERF -* increments faster than we can read it. -*/ -if (mperf_delta ==0 ) -return ; - - s->time =ktime_get (); - s->aperf = aperf; - s->mperf = mperf; - s->khz =div64_u64 ((cpu_khz * aperf_delta), mperf_delta); -atomic_set_release (&s->scfpending,0 ); -} - -static bool aperfmperf_snapshot_cpu (int cpu,ktime_t now,bool wait) -{ -s64 time_delta =ktime_ms_delta (now,per_cpu (samples.time, cpu)); -struct aperfmperf_sample *s =per_cpu_ptr (&samples, cpu); - -/* Don't bother re-computing within the cache threshold time. */ -if (time_delta <APERFMPERF_CACHE_THRESHOLD_MS ) -return true; - -if (!atomic_xchg (&s->scfpending,1 ) || wait) -smp_call_function_single (cpu, aperfmperf_snapshot_khz,NULL , wait); - -/* Return false if the previous iteration was too long ago. */ -return time_delta <=APERFMPERF_STALE_THRESHOLD_MS ; -} - -unsigned int arch_freq_get_on_cpu (int cpu) -{ -struct aperfmperf_sample *s =per_cpu_ptr (&samples, cpu); - -if (!cpu_khz) -return 0 ; - -if (!boot_cpu_has (X86_FEATURE_APERFMPERF )) -return 0 ; - -if (!housekeeping_cpu (cpu,HK_TYPE_MISC )) -return 0 ; - -if (rcu_is_idle_cpu (cpu)) -return 0 ; - -if (aperfmperf_snapshot_cpu (cpu,ktime_get (), true)) -return per_cpu (samples.khz, cpu); - -msleep (APERFMPERF_REFRESH_DELAY_MS ); -atomic_set (&s->scfpending,1 ); -smp_mb ();/* ->scfpending before smp_call_function_single(). */ -smp_call_function_single (cpu, aperfmperf_snapshot_khz,NULL ,1 ); - -return per_cpu (samples.khz, cpu); -} -static void init_counter_refs (void ) {u64 aperf, mperf;
@@ -494,7 +402,7 @@ void arch_scale_freq_tick(void)*/ #define MAX_SAMPLE_AGE ((unsigned long)HZ / 50) -unsigned int aperfmperf_get_khz (int cpu) +unsigned int arch_freq_get_on_cpu (int cpu) {struct aperfmperf *s =per_cpu_ptr (&cpu_samples, cpu);unsigned long last;
arch/x86/kernel/cpu/proc.c +1 -1
@@ -84,7 +84,7 @@ static int show_cpuinfo(struct seq_file *m, void *v)seq_printf (m,"microcode\t: 0x%x\n" , c->microcode);if (cpu_has (c,X86_FEATURE_TSC )) { -unsigned int freq =aperfmperf_get_khz (cpu); +unsigned int freq =arch_freq_get_on_cpu (cpu);if (!freq) freq =cpufreq_quick_get (cpu);
Quoted 16 lines · Thomas Gleixner
APERF/MPERF is utilized in two ways:
1) Ad hoc readout of CPU frequency which requires IPIs
2) Frequency scale calculation for frequency invariant scheduling which reads APERF/MPERF on every tick.
These are completely independent code parts. Eric observed long latencies when reading
/proc/cpuinfowhich reads out CPU frequency via #1 and proposed to replace the per CPU single IPI with a broadcast IPI.While this makes the latency smaller, it is not necessary at all because #2 samples APERF/MPERF periodically, except on idle or isolated NOHZ full CPUs which are excluded from IPI already.
Quoted 8 lines
It could be argued that not all APERF/MPERF capable systems have the required BIOS information to enable frequency invariance support, but in practice most of them do. So the APERF/MPERF sampling can be made unconditional and just the frequency scale calculation for the scheduler excluded.
The following series consolidates that.
Thanks a lot for working on that Thomas.
I am not sure I will be able to backport this to a Google prodkernel, as I guess there will be many merge conflicts.
Do you have by any chance this work available in a git branch ?
Thanks.
Quoted 15 lines
Thanks,
tglx
--- arch/x86/include/asm/cpu.h | 2 arch/x86/include/asm/topology.h | 17 - arch/x86/kernel/acpi/cppc.c | 28 -- arch/x86/kernel/cpu/aperfmperf.c | 474 +++++++++++++++++++++++++++++++-------- arch/x86/kernel/cpu/proc.c | 2 arch/x86/kernel/smpboot.c | 358 ----------------------------- fs/proc/cpuinfo.c | 6 include/linux/cpufreq.h | 1 8 files changed, 405 insertions(+), 483 deletions(-)
Eric,
Quoted 7 lines · Eric Dumazet
Quoted 6 lines · Thomas Gleixner
It could be argued that not all APERF/MPERF capable systems have the required BIOS information to enable frequency invariance support, but in practice most of them do. So the APERF/MPERF sampling can be made unconditional and just the frequency scale calculation for the scheduler excluded.
Quoted 7 lines
Quoted 2 lines
The following series consolidates that.
Thanks a lot for working on that Thomas.
I am not sure I will be able to backport this to a Google prodkernel, as I guess there will be many merge conflicts.
:)
Quoted 1 line
Do you have by any chance this work available in a git branch ?
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git x86/amperf
Thanks,
tglx
Eric Dumazet
Tested-byQuoted 4 lines · Thomas Gleixner
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.gitx86/amperf
Excellent, things look fine to me.
Before:
# grep MHz /proc/cpuinfo | sort | uniq -c
255 cpu MHz : 2249.998
1 cpu MHz : 3297.719# grep MHz /proc/cpuinfo | sort|uniq -c
1 cpu MHz : 1590.400
1 cpu MHz : 1684.772
1 cpu MHz : 1693.890
1 cpu MHz : 1780.072
1 cpu MHz : 1784.513
1 cpu MHz : 1831.106
1 cpu MHz : 1880.344
1 cpu MHz : 1953.481
1 cpu MHz : 1980.636
1 cpu MHz : 2013.620
1 cpu MHz : 2219.617
240 cpu MHz : 2250.173
1 cpu MHz : 3292.206
1 cpu MHz : 3294.956
1 cpu MHz : 3297.653
1 cpu MHz : 3298.385
1 cpu MHz : 3300.197Quoted 10 lines · Thomas Gleixner
--- arch/x86/include/asm/cpu.h | 2 arch/x86/include/asm/topology.h | 17 - arch/x86/kernel/acpi/cppc.c | 28 -- arch/x86/kernel/cpu/aperfmperf.c | 474 +++++++++++++++++++++++++++++++-------- arch/x86/kernel/cpu/proc.c | 2 arch/x86/kernel/smpboot.c | 358 ----------------------------- fs/proc/cpuinfo.c | 6 include/linux/cpufreq.h | 1 8 files changed, 405 insertions(+), 483 deletions(-)
Hi Thomas,
Quoted 15 lines · On 2022.04.15 12:20 Thomas Gleixner wrote:
APERF/MPERF is utilized in two ways:
1) Ad hoc readout of CPU frequency which requires IPIs
2) Frequency scale calculation for frequency invariant scheduling which reads APERF/MPERF on every tick.
These are completely independent code parts. Eric observed long latencies when reading
/proc/cpuinfowhich reads out CPU frequency via #1 and proposed to replace the per CPU single IPI with a broadcast IPI.While this makes the latency smaller, it is not necessary at all because #2 samples APERF/MPERF periodically, except on idle or isolated NOHZ full CPUs which are excluded from IPI already.
Quoted 7 lines
It could be argued that not all APERF/MPERF capable systems have the required BIOS information to enable frequency invariance support, but in practice most of them do. So the APERF/MPERF sampling can be made unconditional and just the frequency scale calculation for the scheduler excluded.
The following series consolidates that.
I have used this patch set with the acpi-cpufreq, intel_cpufreq (passive), and intel_pstate (active) CPU frequency scaling drivers and various governors. Additionally, with HWP both enabled and disabled.
For intel_pstate (active), both HWP enabled or disabled, the behaviour of scaling_cur_freq is inconsistent with prior to this patch set and other scaling driver governor combinations.
Note there is no issue with " grep MHz /proc/cpuinfo" for any combination.
Examples:
No-HWP:
active/powersave:
doug@s19:~/freq-scalers/trace$ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq:2300418 /sys/devices/system/cpu/cpu10/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu11/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu5/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu6/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu7/cpufreq/scaling_cur_freq:2300006 /sys/devices/system/cpu/cpu8/cpufreq/scaling_cur_freq:2300005 /sys/devices/system/cpu/cpu9/cpufreq/scaling_cur_freq:0
active/performance:
doug@s19:~$ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu10/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu11/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu5/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu6/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu7/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu8/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu9/cpufreq/scaling_cur_freq:0
HWP:
active/powersave:
doug@s19:~$ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu10/cpufreq/scaling_cur_freq:799993 /sys/devices/system/cpu/cpu11/cpufreq/scaling_cur_freq:800069 /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq:800131 /sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu5/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu6/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu7/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu8/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu9/cpufreq/scaling_cur_freq:799844
active/performance:
doug@s19:~$ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu10/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu11/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq:4800186 /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq:4800016 /sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu5/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu6/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu7/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu8/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu9/cpufreq/scaling_cur_freq:0
Other configurations:intel_cpufreq /schedutil (no HWP), for example:
doug@s19:~$ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq:800000 /sys/devices/system/cpu/cpu10/cpufreq/scaling_cur_freq:1067573 /sys/devices/system/cpu/cpu11/cpufreq/scaling_cur_freq:800000 /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq:800000 /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq:800000 /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq:800011 /sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq:800000 /sys/devices/system/cpu/cpu5/cpufreq/scaling_cur_freq:800109 /sys/devices/system/cpu/cpu6/cpufreq/scaling_cur_freq:800000 /sys/devices/system/cpu/cpu7/cpufreq/scaling_cur_freq:800000 /sys/devices/system/cpu/cpu8/cpufreq/scaling_cur_freq:800000 /sys/devices/system/cpu/cpu9/cpufreq/scaling_cur_freq:800000
Processor: Intel(R) Core(TM) i5-10600K CPU @ 4.10GHz
Quoted 13 lines
Thanks,
tglx
---arch/x86/include/asm/cpu.h | 2 arch/x86/include/asm/topology.h | 17 - arch/x86/kernel/acpi/cppc.c | 28 -- arch/x86/kernel/cpu/aperfmperf.c | 474 +++++++++++++++++++++++++++++++-------- arch/x86/kernel/cpu/proc.c | 2 arch/x86/kernel/smpboot.c | 358 ----------------------------- fs/proc/cpuinfo.c | 6 include/linux/cpufreq.h | 1 8 files changed, 405 insertions(+), 483 deletions(-)
Rafael J. Wysocki
maintainerQuoted 44 lines · Doug Smythies
Hi Thomas,
Quoted 15 lines · On 2022.04.15 12:20 Thomas Gleixner wrote:
APERF/MPERF is utilized in two ways:
1) Ad hoc readout of CPU frequency which requires IPIs
2) Frequency scale calculation for frequency invariant scheduling which reads APERF/MPERF on every tick.
These are completely independent code parts. Eric observed long latencies when reading
/proc/cpuinfowhich reads out CPU frequency via #1 and proposed to replace the per CPU single IPI with a broadcast IPI.While this makes the latency smaller, it is not necessary at all because #2 samples APERF/MPERF periodically, except on idle or isolated NOHZ full CPUs which are excluded from IPI already.
Quoted 7 lines
It could be argued that not all APERF/MPERF capable systems have the required BIOS information to enable frequency invariance support, but in practice most of them do. So the APERF/MPERF sampling can be made unconditional and just the frequency scale calculation for the scheduler excluded.
The following series consolidates that.
I have used this patch set with the acpi-cpufreq,
intel_cpufreq(passive), andintel_pstate(active) CPU frequency scaling drivers and various governors. Additionally, with HWP both enabled and disabled.For
intel_pstate(active), both HWP enabled or disabled, the behaviour ofscaling_cur_freqis inconsistent with prior to this patch set and other scaling driver governor combinations.Note there is no issue with " grep MHz
/proc/cpuinfo" for any combination.Examples:
No-HWP:
active/powersave:
Quoted 13 lines
doug@s19:~/freq-scalers/trace$ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq:2300418 /sys/devices/system/cpu/cpu10/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu11/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu5/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu6/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu7/cpufreq/scaling_cur_freq:2300006 /sys/devices/system/cpu/cpu8/cpufreq/scaling_cur_freq:2300005 /sys/devices/system/cpu/cpu9/cpufreq/scaling_cur_freq:0
That's because after the changes in this series scaling_cur_freq returns 0 if the given CPU is idle.
I guess it could return the last known result, but that wouldn't be more meaningful.
Quoted 27 lines · Rafael J. Wysocki
Quoted 12 lines · Doug Smythies
For
intel_pstate(active), both HWP enabled or disabled, the behaviour ofscaling_cur_freqis inconsistent with prior to this patch set and other scaling driver governor combinations.Note there is no issue with " grep MHz
/proc/cpuinfo" for any combination.Examples:
No-HWP:
active/powersave:
Quoted 13 lines
doug@s19:~/freq-scalers/trace$ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq:2300418 /sys/devices/system/cpu/cpu10/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu11/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu5/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu6/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu7/cpufreq/scaling_cur_freq:2300006 /sys/devices/system/cpu/cpu8/cpufreq/scaling_cur_freq:2300005 /sys/devices/system/cpu/cpu9/cpufreq/scaling_cur_freq:0
Quoted 2 lines
That's because after the changes in this series
scaling_cur_freqreturns 0 if the given CPU is idle.
Which is sensible IMO as there is really no point in waking an idle CPU just to read those MSRs, then wait 20ms wake it up again to read those MSRs again.
Quoted 2 lines
I guess it could return the last known result, but that wouldn't be more meaningful.
Right.
Thanks,
tglx
Hi Thomas, Rafael,
Thank you for your replies.
Quoted 28 lines · On 2022.04.19 14:11 Thomas Gleixner wrote:
Quoted 27 lines · Rafael J. Wysocki
Quoted 25 lines · Doug Smythies
For
intel_pstate(active), both HWP enabled or disabled, the behaviour ofscaling_cur_freqis inconsistent with prior to this patch set and other scaling driver governor combinations.Note there is no issue with " grep MHz
/proc/cpuinfo" for any combination.Examples:
No-HWP:
active/powersave:
doug@s19:~/freq-scalers/trace$ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq:2300418 /sys/devices/system/cpu/cpu10/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu11/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu5/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu6/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu7/cpufreq/scaling_cur_freq:2300006 /sys/devices/system/cpu/cpu8/cpufreq/scaling_cur_freq:2300005 /sys/devices/system/cpu/cpu9/cpufreq/scaling_cur_freq:0
Quoted 6 lines
Quoted 2 lines
That's because after the changes in this series
scaling_cur_freqreturns 0 if the given CPU is idle.Which is sensible IMO as there is really no point in waking an idle CPU just to read those MSRs, then wait 20ms wake it up again to read those MSRs again.
I totally agree.
It is the inconsistency for what is displayed as a function of driver/governor that is my concern.
Quoted 5 lines
Quoted 3 lines
I guess it could return the last known result, but that wouldn't be more meaningful.
Right.
How about something like this, which I realize might break something else, but just to demonstrate:
doug@s19:~/kernel/linux$ git diff
drivers/cpufreq/cpufreq.c +1 -1
@@ -710,7 +710,7 @@ static ssize_t show_scaling_cur_freq(struct cpufreq_policy *policy, char *buf)else if (cpufreq_driver->setpolicy && cpufreq_driver->get) ret =sprintf (buf,"%u\n" , cpufreq_driver->get (policy->cpu));else - ret =sprintf (buf,"%u\n" , policy->cur); + ret =sprintf (buf,"%u\n" , freq);return ret; }
Note: I left the other 0 return condition, because I do not know what uses it.
Which gives:
acpi-cpufreq/schedutil
doug@s19:~/kernel/linux$ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu10/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu11/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu5/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu6/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu7/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu8/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu9/cpufreq/scaling_cur_freq:4100723
intel_pstate/powersave (no-HWP)
doug@s19:~/kernel/linux$ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu10/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu11/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu5/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu6/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu7/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu8/cpufreq/scaling_cur_freq:800295 /sys/devices/system/cpu/cpu9/cpufreq/scaling_cur_freq:800015
intel_cpufreq/schedutil (no-HWP)
doug@s19:~/kernel/linux$ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu10/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu11/cpufreq/scaling_cur_freq:1971265 /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu5/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu6/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu7/cpufreq/scaling_cur_freq:0 /sys/devices/system/cpu/cpu8/cpufreq/scaling_cur_freq:2785446 /sys/devices/system/cpu/cpu9/cpufreq/scaling_cur_freq:0
Which I suggest is more consistent.
Note: because it was deleted from this thread,
and just for reference, I'll repost the previous
intel_cpufreq/schedutil (no-HWP) output:
doug@s19:~$ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq:800000 /sys/devices/system/cpu/cpu10/cpufreq/scaling_cur_freq:1067573 /sys/devices/system/cpu/cpu11/cpufreq/scaling_cur_freq:800000 /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq:800000 /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq:800000 /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq:800011 /sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq:800000 /sys/devices/system/cpu/cpu5/cpufreq/scaling_cur_freq:800109 /sys/devices/system/cpu/cpu6/cpufreq/scaling_cur_freq:800000 /sys/devices/system/cpu/cpu7/cpufreq/scaling_cur_freq:800000 /sys/devices/system/cpu/cpu8/cpufreq/scaling_cur_freq:800000 /sys/devices/system/cpu/cpu9/cpufreq/scaling_cur_freq:800000
... Doug
Quoted 11 lines · Doug Smythies
Quoted 6 lines · On 2022.04.19 14:11 Thomas Gleixner wrote:
Quoted 2 lines
That's because after the changes in this series
scaling_cur_freqreturns 0 if the given CPU is idle.Which is sensible IMO as there is really no point in waking an idle CPU just to read those MSRs, then wait 20ms wake it up again to read those MSRs again.
I totally agree.
It is the inconsistency for what is displayed as a function of driver/governor that is my concern.
Raphael suggested to move the show_cpuinfo() logic into the a/mperf code. See below.
Thanks,
tglx
---
Subject: x86/aperfmperf: Integrate the fallback code from show_cpuinfo()
From: Thomas Gleixner <[email protected]>
Date: Mon, 25 Apr 2022 15:19:29 +0200
Due to the avoidance of IPIs to idle CPUs arch_freq_get_on_cpu() can return 0 when the last sample was too long ago.
show_cpuinfo() has a fallback to cpufreq_quick_get() and if that fails to return cpu_khz, but the readout code for the per CPU scaling frequency in sysfs does not.
Move that fallback into arch_freq_get_on_cpu() so the behaviour is the same when reading /proc/cpuinfo and /sys/..../cur_scaling_freq.
--- arch/x86/kernel/cpu/aperfmperf.c | 10 +++++++--- arch/x86/kernel/cpu/proc.c | 7 +------ 2 files changed, 8 insertions(+), 9 deletions(-)
arch/x86/kernel/cpu/aperfmperf.c +7 -3
@@ -405,12 +405,12 @@ void arch_scale_freq_tick(void)unsigned int arch_freq_get_on_cpu (int cpu) {struct aperfmperf *s =per_cpu_ptr (&cpu_samples, cpu); +unsigned int seq, freq;unsigned long last; -unsigned int seq;u64 acnt, mcnt;if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF )) -return 0 ; + goto fallback; do { seq = raw_read_seqcount_begin(&s->seq);
@@ -424,9 +424,13 @@ unsigned int arch_freq_get_on_cpu(int cp* which covers idle and NOHZ full CPUs. */ if (!mcnt || (jiffies - last) >MAX_SAMPLE_AGE ) -return 0 ; + goto fallback;return div64_u64 ((cpu_khz * acnt), mcnt); + +fallback: + freq =cpufreq_quick_get (cpu); +return freq ? freq : cpu_khz; }static int __init bp_init_aperfmperf (void )
arch/x86/kernel/cpu/proc.c +1 -6
@@ -86,12 +86,7 @@ static int show_cpuinfo(struct seq_fileif (cpu_has (c,X86_FEATURE_TSC )) {unsigned int freq =arch_freq_get_on_cpu (cpu); -if (!freq) - freq =cpufreq_quick_get (cpu); -if (!freq) - freq = cpu_khz; -seq_printf (m,"cpu MHz\t\t: %u.%03u\n" , - freq /1000 , (freq %1000 )); +seq_printf (m,"cpu MHz\t\t: %u.%03u\n" , freq /1000 , (freq %1000 )); }/* Cache size */
Quoted 9 lines · Thomas Gleixner
Quoted 8 lines · Doug Smythies
Quoted 6 lines · On 2022.04.19 14:11 Thomas Gleixner wrote:
Quoted 2 lines
That's because after the changes in this series
scaling_cur_freqreturns 0 if the given CPU is idle.Which is sensible IMO as there is really no point in waking an idle CPU just to read those MSRs, then wait 20ms wake it up again to read those MSRs again.
Quoted 6 lines
Quoted 3 lines
I totally agree.
It is the inconsistency for what is displayed as a function of driver/governor that is my concern.Raphael suggested to move the
show_cpuinfo()logic into the a/mperf code. See below.
Hi Thomas,
I tested the patch on top of your 10 patch set on kernel 5.18-rc3. It addresses my consistency concerns.
Thank you
... Doug
Quoted 5 lines
---
Subject: x86/aperfmperf: Integrate the fallback code fromshow_cpuinfo()
From: Thomas Gleixner <[email protected]>
Date: Mon, 25 Apr 2022 15:19:29 +0200
...
[tip: x86/cleanups] x86/aperfmperf: Integrate the fallback code from show_cpuinfo()
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: e696cabf5da2b4ed104508674de6125b860f3c9f Gitweb: https://git.kernel.org/tip/e696cabf5da2b4ed104508674de6125b860f3c9f Author: Thomas Gleixner <[email protected]> AuthorDate: Mon, 25 Apr 2022 17:45:42 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 15:51:09 +02:00
x86/aperfmperf: Integrate the fallback code from show_cpuinfo()
Due to the avoidance of IPIs to idle CPUs arch_freq_get_on_cpu() can return 0 when the last sample was too long ago.
show_cpuinfo() has a fallback to cpufreq_quick_get() and if that fails to return cpu_khz, but the readout code for the per CPU scaling frequency in sysfs does not.
Move that fallback into arch_freq_get_on_cpu() so the behaviour is the same when reading /proc/cpuinfo and /sys/..../cur_scaling_freq.
--- arch/x86/kernel/cpu/aperfmperf.c | 10 +++++++--- arch/x86/kernel/cpu/proc.c | 7 +------ 2 files changed, 8 insertions(+), 9 deletions(-)
arch/x86/kernel/cpu/aperfmperf.c +7 -3
@@ -405,12 +405,12 @@ void arch_scale_freq_tick(void)unsigned int arch_freq_get_on_cpu (int cpu) {struct aperfmperf *s =per_cpu_ptr (&cpu_samples, cpu); +unsigned int seq, freq;unsigned long last; -unsigned int seq;u64 acnt, mcnt;if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF )) -return 0 ; + goto fallback; do { seq = raw_read_seqcount_begin(&s->seq);
@@ -424,9 +424,13 @@ unsigned int arch_freq_get_on_cpu(int cpu)* which covers idle and NOHZ full CPUs. */ if (!mcnt || (jiffies - last) >MAX_SAMPLE_AGE ) -return 0 ; + goto fallback;return div64_u64 ((cpu_khz * acnt), mcnt); + +fallback: + freq =cpufreq_quick_get (cpu); +return freq ? freq : cpu_khz; }static int __init bp_init_aperfmperf (void )
arch/x86/kernel/cpu/proc.c +1 -6
@@ -86,12 +86,7 @@ static int show_cpuinfo(struct seq_file *m, void *v)if (cpu_has (c,X86_FEATURE_TSC )) {unsigned int freq =arch_freq_get_on_cpu (cpu); -if (!freq) - freq =cpufreq_quick_get (cpu); -if (!freq) - freq = cpu_khz; -seq_printf (m,"cpu MHz\t\t: %u.%03u\n" , - freq /1000 , (freq %1000 )); +seq_printf (m,"cpu MHz\t\t: %u.%03u\n" , freq /1000 , (freq %1000 )); }/* Cache size */
[tip: x86/cleanups] x86/aperfmperf: Integrate the fallback code from show_cpuinfo()
In reply to Thomas GleixnerThe following commit has been merged into the x86/cleanups branch of tip:
The following commit has been merged into the x86/cleanups branch of tip:
Commit-ID: fb4c77c21aba03677f283acda3cae748ef866abf Gitweb: https://git.kernel.org/tip/fb4c77c21aba03677f283acda3cae748ef866abf Author: Thomas Gleixner <[email protected]> AuthorDate: Mon, 25 Apr 2022 17:45:42 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 27 Apr 2022 20:22:20 +02:00
x86/aperfmperf: Integrate the fallback code from show_cpuinfo()
Due to the avoidance of IPIs to idle CPUs arch_freq_get_on_cpu() can return 0 when the last sample was too long ago.
show_cpuinfo() has a fallback to cpufreq_quick_get() and if that fails to return cpu_khz, but the readout code for the per CPU scaling frequency in sysfs does not.
Move that fallback into arch_freq_get_on_cpu() so the behaviour is the same when reading /proc/cpuinfo and /sys/..../cur_scaling_freq.
--- arch/x86/kernel/cpu/aperfmperf.c | 10 +++++++--- arch/x86/kernel/cpu/proc.c | 7 +------ 2 files changed, 8 insertions(+), 9 deletions(-)
arch/x86/kernel/cpu/aperfmperf.c +7 -3
@@ -405,12 +405,12 @@ void arch_scale_freq_tick(void)unsigned int arch_freq_get_on_cpu (int cpu) {struct aperfmperf *s =per_cpu_ptr (&cpu_samples, cpu); +unsigned int seq, freq;unsigned long last; -unsigned int seq;u64 acnt, mcnt;if (!cpu_feature_enabled (X86_FEATURE_APERFMPERF )) -return 0 ; + goto fallback; do { seq = raw_read_seqcount_begin(&s->seq);
@@ -424,9 +424,13 @@ unsigned int arch_freq_get_on_cpu(int cpu)* which covers idle and NOHZ full CPUs. */ if (!mcnt || (jiffies - last) >MAX_SAMPLE_AGE ) -return 0 ; + goto fallback;return div64_u64 ((cpu_khz * acnt), mcnt); + +fallback: + freq =cpufreq_quick_get (cpu); +return freq ? freq : cpu_khz; }static int __init bp_init_aperfmperf (void )
arch/x86/kernel/cpu/proc.c +1 -6
@@ -86,12 +86,7 @@ static int show_cpuinfo(struct seq_file *m, void *v)if (cpu_has (c,X86_FEATURE_TSC )) {unsigned int freq =arch_freq_get_on_cpu (cpu); -if (!freq) - freq =cpufreq_quick_get (cpu); -if (!freq) - freq = cpu_khz; -seq_printf (m,"cpu MHz\t\t: %u.%03u\n" , - freq /1000 , (freq %1000 )); +seq_printf (m,"cpu MHz\t\t: %u.%03u\n" , freq /1000 , (freq %1000 )); }/* Cache size */
Quoted 15 lines · Thomas Gleixner
APERF/MPERF is utilized in two ways:
1) Ad hoc readout of CPU frequency which requires IPIs
2) Frequency scale calculation for frequency invariant scheduling which reads APERF/MPERF on every tick.
These are completely independent code parts. Eric observed long latencies when reading
/proc/cpuinfowhich reads out CPU frequency via #1 and proposed to replace the per CPU single IPI with a broadcast IPI.While this makes the latency smaller, it is not necessary at all because #2 samples APERF/MPERF periodically, except on idle or isolated NOHZ full CPUs which are excluded from IPI already.
Quoted 7 lines
It could be argued that not all APERF/MPERF capable systems have the required BIOS information to enable frequency invariance support, but in practice most of them do. So the APERF/MPERF sampling can be made unconditional and just the frequency scale calculation for the scheduler excluded.
The following series consolidates that.
Quoted 15 lines
Thanks,
tglx
--- arch/x86/include/asm/cpu.h | 2 arch/x86/include/asm/topology.h | 17 - arch/x86/kernel/acpi/cppc.c | 28 -- arch/x86/kernel/cpu/aperfmperf.c | 474 +++++++++++++++++++++++++++++++-------- arch/x86/kernel/cpu/proc.c | 2 arch/x86/kernel/smpboot.c | 358 ----------------------------- fs/proc/cpuinfo.c | 6 include/linux/cpufreq.h | 1 8 files changed, 405 insertions(+), 483 deletions(-)