Fix three racy updates to MSR_K7_HWCR
Subsystems (3)
v2 cover Fix three racy updates to MSR_K7_HWCR
I was backporting commit 65f55a301766 ("x86/CPU/AMD: Add CPUID faulting support") to a local branch based on Linux v6.12, when our internal Sashiko asked:
Quoted 6 lines
Can this corrupt
MSR_K7_HWCR? disable_cpuid()->set_cpuid_faulting() is called with preemption disabled, but interrupts are still enabled. Sincemsr_set_bit()performs a read-modify-write without disabling interrupts, if an IPI arrives between the read and write and modifiesMSR_K7_HWCR(e.g. acpi-cpufreq toggling Core Performance Boost), the IPI's update will be lost.
To confirm that this wasn't just AI slop, I set up an empirical test on a Turin system. First, I replaced the amd-pstate cpufreq driver with acpi-cpufreq. Then I ran a test program, where one thread repeatedly reads CPU0's HWCR, toggles /sys/devices/system/cpu/cpufreq/boost, reads CPU0's HWCR again, and then verifies that the CPB_DIS bit has flipped. A second thread, pinned to CPU0, repeatedly calls arch_prctl(ARCH_SET_CPUID, <val>), where <val> alternates between 0 and 1. With the second thread running, the first thread soon fails the verification step, indicating that the CPB_DIS bit change is, in fact, lost.
Per Boris's review of v1, this version hoists the HWCR update logic out into a new helper, amd_update_hwcr(), which performs the read-modify-write with interrupts disabled, and converts the three runtime (non-initialization) HWCR read-modify-write sites to use it:
* set_cpuid_faulting(), fixing the race demonstrated above (patch 1);
* toggle_hw_mce_inject(), which previously performed the
read-modify-write as two independent crosscalls (patch 2);
* boost_set_msr() in acpi-cpufreq, whose process-context invocation on the cpufreq policy teardown path can race with an HWCR update made by the MCE injector's crosscall (patch 3).
Initialization-time HWCR updates are left alone for now, to avoid excessive churn.
v1 -> v2:
- Add some of the cover letter details to the first patch
- Hoist the HWCR update logic out into amd_update_hwcr() [Boris]
- Use the helper in toggle_hw_mce_inject(), collapsing the split read/write crosscalls into one
- Use the helper in boost_set_msr() [Boris]
v1: https://lore.kernel.org/all/[email protected]/
Jim Mattson (3):
x86/CPU/AMD: Avoid racy updates to MSR_K7_HWCR in set_cpuid_faulting() x86/mce/inject: Avoid racy updates to MSR_K7_HWCR in
toggle_hw_mce_inject()
cpufreq: ACPI: Avoid racy updates to MSR_K7_HWCR in boost_set_msr()
arch/x86/include/asm/msr-index.h | 3 +++ arch/x86/include/asm/processor.h | 2 ++ arch/x86/kernel/cpu/amd.c | 42 ++++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/mce/inject.c | 34 +++++++++++++++++++------- arch/x86/kernel/process.c | 4 +-- drivers/cpufreq/acpi-cpufreq.c | 7 +++--- 6 files changed, 78 insertions(+), 14 deletions(-)
base-commit: 2b414a95b8f7307d42173ba9e580d6d3e2bcbfce
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
PATCH v2 1/3 x86/CPU/AMD: Avoid racy updates to MSR_K7_HWCR in set_cpuid_faulting()
Since msr_set_bit() and msr_clear_bit() perform a non-atomic update to an MSR, they can race with a write to the same MSR from interrupt context.
On AMD CPUs, set_cpuid_faulting() uses these functions to modify MSR_K7_HWCR from process context, with preemption disabled but interrupts enabled. The acpi-cpufreq driver's boost_set_msr() modifies HWCR from interrupt context. If a crosscall IPI arrives between set_cpuid_faulting()'s read and write of MSR_K7_HWCR and toggles the Core Performance Boost disable bit (CPB_DIS), the IPI's update is lost.
This race has been observed empirically on a Turin system running the acpi-cpufreq driver with a synthetic test. One thread repeatedly toggles /sys/devices/system/cpu/cpufreq/boost and verifies CPB_DIS on CPU0 after each write. A second thread pinned to CPU0 calls arch_prctl(ARCH_SET_CPUID, <val>), with alternating <val>s of 0 and 1. CPB_DIS bit changes are sometimes lost.
Introduce amd_update_hwcr() to perform an interrupt-safe read-modify-write of MSR_K7_HWCR, and use it in set_cpuid_faulting() to prevent races with HWCR updates in interrupt context. Note that when set_cpuid_faulting() is called from __switch_to_xtra(), interrupts are already disabled, so the race is only possible on the arch_prctl() paths.
--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/include/asm/processor.h | 2 ++ arch/x86/kernel/cpu/amd.c | 42 ++++++++++++++++++++++++++++++++ arch/x86/kernel/process.c | 4 +-- 4 files changed, 47 insertions(+), 2 deletions(-)
arch/x86/include/asm/msr-index.h +1 -0
@@ -899,6 +899,7 @@#define MSR_K7_HWCR_IRPERF_EN_BIT 30#define MSR_K7_HWCR_IRPERF_EN BIT_ULL(MSR_K7_HWCR_IRPERF_EN_BIT)#define MSR_K7_HWCR_CPUID_USER_DIS_BIT 35 +#define MSR_K7_HWCR_CPUID_USER_DIS BIT_ULL(MSR_K7_HWCR_CPUID_USER_DIS_BIT)#define MSR_K7_FID_VID_CTL 0xc0010041#define MSR_K7_FID_VID_STATUS 0xc0010042#define MSR_K7_HWCR_CPB_DIS_BIT 25
arch/x86/include/asm/processor.h +2 -0
@@ -716,9 +716,11 @@ static __always_inline void amd_clear_divider(void) }extern void amd_check_microcode (void ); +extern int amd_update_hwcr (u64 clear,u64 set);#else static inline void amd_clear_divider (void ) { }static inline void amd_check_microcode (void ) { } +static inline int amd_update_hwcr (u64 clear,u64 set) {return -ENODEV ; }#endif extern unsigned long arch_align_stack (unsigned long sp);
arch/x86/kernel/cpu/amd.c +42 -0
48 lines not on this page · raw message
arch/x86/kernel/process.c +2 -2
@@ -355,9 +355,9 @@ static void set_cpuid_faulting(bool on)wrmsrq (MSR_MISC_FEATURES_ENABLES , msrval); }else if (boot_cpu_data.x86_vendor ==X86_VENDOR_AMD ) {if (on) -msr_set_bit (MSR_K7_HWCR ,MSR_K7_HWCR_CPUID_USER_DIS_BIT ); +amd_update_hwcr (0 ,MSR_K7_HWCR_CPUID_USER_DIS );else -msr_clear_bit (MSR_K7_HWCR ,MSR_K7_HWCR_CPUID_USER_DIS_BIT ); +amd_update_hwcr (MSR_K7_HWCR_CPUID_USER_DIS ,0 ); } }
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Re: [PATCH v2 1/3] x86/CPU/AMD: Avoid racy updates to MSR_K7_HWCR in set_cpuid_faulting()
Quoted 81 lines · Jim Mattson
Since
msr_set_bit()andmsr_clear_bit()perform a non-atomic update to an MSR, they can race with a write to the same MSR from interrupt context.On AMD CPUs,
set_cpuid_faulting()uses these functions to modifyMSR_K7_HWCRfrom process context, with preemption disabled but interrupts enabled. The acpi-cpufreq driver'sboost_set_msr()modifies HWCR from interrupt context. If a crosscall IPI arrives between set_cpuid_faulting()'s read and write ofMSR_K7_HWCRand toggles the Core Performance Boost disable bit (CPB_DIS), the IPI's update is lost.This race has been observed empirically on a Turin system running the acpi-cpufreq driver with a synthetic test. One thread repeatedly toggles
/sys/devices/system/cpu/cpufreq/boostand verifiesCPB_DISon CPU0 after each write. A second thread pinned to CPU0 callsarch_prctl(ARCH_SET_CPUID, <val>), with alternating <val>s of 0 and 1.CPB_DISbit changes are sometimes lost.Introduce
amd_update_hwcr()to perform an interrupt-safe read-modify-write ofMSR_K7_HWCR, and use it inset_cpuid_faulting()to prevent races with HWCR updates in interrupt context. Note that whenset_cpuid_faulting()is called from__switch_to_xtra(), interrupts are already disabled, so the race is only possible on thearch_prctl()paths.Reported-by Sashiko (gemini/gemini-3.1-pro-preview)Suggested-by Borislav Petkov <[email protected]>Fixes 65f55a301766 ("x86/CPU/AMD: Add CPUID faulting support")Signed-off-by Jim Mattson <[email protected]>--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/include/asm/processor.h | 2 ++ arch/x86/kernel/cpu/amd.c | 42 ++++++++++++++++++++++++++++++++ arch/x86/kernel/process.c | 4 +-- 4 files changed, 47 insertions(+), 2 deletions(-)arch/x86/include/asm/msr-index.h +1
-0@@ -899,6 +899,7 @@#define MSR_K7_HWCR_IRPERF_EN_BIT 30#define MSR_K7_HWCR_IRPERF_EN BIT_ULL(MSR_K7_HWCR_IRPERF_EN_BIT)#define MSR_K7_HWCR_CPUID_USER_DIS_BIT 35 +#define MSR_K7_HWCR_CPUID_USER_DIS BIT_ULL(MSR_K7_HWCR_CPUID_USER_DIS_BIT)#define MSR_K7_FID_VID_CTL 0xc0010041#define MSR_K7_FID_VID_STATUS 0xc0010042#define MSR_K7_HWCR_CPB_DIS_BIT 25arch/x86/include/asm/processor.h +2
-0@@ -716,9 +716,11 @@ static __always_inline void amd_clear_divider(void) }extern void amd_check_microcode (void ); +extern int amd_update_hwcr (u64 clear,u64 set);#else static inline void amd_clear_divider (void ) { }static inline void amd_check_microcode (void ) { } +static inline int amd_update_hwcr (u64 clear,u64 set) {return -ENODEV ; }#endif extern unsigned long arch_align_stack (unsigned long sp);arch/x86/kernel/cpu/amd.c +8
-0@@ -1319,6 +1319,48 @@ void amd_check_microcode(void)on_each_cpu (zenbleed_check_cpu,NULL ,1 ); } +/** +* amd_update_hwcr - Update MSR_K7_HWCR on the executing CPU +* @clear: Bits to clear +* @set: Bits to set +* +* MSR_K7_HWCR is written from both process context (e.g. CPUID faulting +* updates via arch_prctl(ARCH_SET_CPUID)) and interrupt context (e.g. +* Core Performance Boost updates IPI'd by the acpi-cpufreq driver), so
arch/x86/kernel/cpu/amd.c void amd_check_microcode(void) · line 1339
Quoted 10 lines
arch/x86/kernel/cpu/amd.c +10
-0+* a read-modify-write of the MSR must be performed with interrupts +* disabled to avoid losing an update made by an intervening interrupt. +* All runtime (non-initialization) updates of MSR_K7_HWCR should go +* through this helper. +* +* Bits in @set take precedence over bits in @clear. +* +* Context: Any context except NMI. Disabling interrupts does not +* serialize against an NMI, so NMI handlers must not write +* MSR_K7_HWCR.
Might be more useful to WARN if in_nmi()?
Quoted 8 lines
arch/x86/kernel/cpu/amd.c +8
-0+* +* Return: 0 on success, negative error code if an MSR access faults. +*/ +int amd_update_hwcr (u64 clear,u64 set) +{ +unsigned long flags; +u64 oldval, newval; +int ret;
arch/x86/kernel/cpu/amd.c void amd_check_microcode(void) · line 1357
Quoted 10 lines
arch/x86/kernel/cpu/amd.c +10
-0+ +local_irq_save (flags); + ret =rdmsrq_safe (MSR_K7_HWCR , &oldval); +if (ret) + goto out; + + newval = (oldval & ~clear) | set; + +if (newval != oldval) + ret =wrmsrq_safe (MSR_K7_HWCR , newval);
Can we reuse msr_set_bit() and msr_clear_bit() here?
Quoted 28 lines
arch/x86/kernel/cpu/amd.c +6
-0+out: +local_irq_restore (flags); +return ret; +} +EXPORT_SYMBOL_GPL (amd_update_hwcr); +static const char *const s5_reset_reason_txt[] = { [0 ] ="thermal pin BP_THERMTRIP_L was tripped" , [1 ] ="power button was pressed for 4 seconds" ,arch/x86/kernel/process.c +2
-3@@ -355,9 +355,9 @@ static void set_cpuid_faulting(bool on)wrmsrq (MSR_MISC_FEATURES_ENABLES , msrval); }else if (boot_cpu_data.x86_vendor ==X86_VENDOR_AMD ) {if (on) -msr_set_bit (MSR_K7_HWCR ,MSR_K7_HWCR_CPUID_USER_DIS_BIT ); +amd_update_hwcr (0 ,MSR_K7_HWCR_CPUID_USER_DIS );else -msr_clear_bit (MSR_K7_HWCR ,MSR_K7_HWCR_CPUID_USER_DIS_BIT ); +amd_update_hwcr (MSR_K7_HWCR_CPUID_USER_DIS ,0 ); } } --2.54.0.1136.gdb2ca164c4-goog
Re: [PATCH v2 1/3] x86/CPU/AMD: Avoid racy updates to MSR_K7_HWCR in set_cpuid_faulting()
Quoted 2 lines · Yosry Ahmed
On Fri, Jun 12, 2026 at 2:57 PM Jim Mattson <[email protected]> wrote:
...
Quoted 5 lines
Quoted 3 lines
+ * Context: Any context except NMI. Disabling interrupts does not
+ * serialize against an NMI, so NMI handlers must not write
+ *MSR_K7_HWCR.Might be more useful to WARN if
in_nmi()?
I concur.
Quoted 12 lines
Quoted 12 lines
+ *
+ * Return: 0 on success, negative error code if an MSR access faults.
+ */arch/x86/kernel/cpu/amd.c +9
-0+int amd_update_hwcr (u64 clear,u64 set) +{ +unsigned long flags; +u64 oldval, newval; +int ret; + +local_irq_save (flags); + ret =rdmsrq_safe (MSR_K7_HWCR , &oldval); +if (ret)
Quoted 8 lines
arch/x86/kernel/cpu/amd.c void amd_check_microcode(void) · line 1357
Quoted 6 lines
arch/x86/kernel/cpu/amd.c +6
-0+ goto out; + + newval = (oldval & ~clear) | set; + +if (newval != oldval) + ret =wrmsrq_safe (MSR_K7_HWCR , newval);Can we reuse
msr_set_bit()andmsr_clear_bit()here?
Yes. I tried to make it more general, but none of the current users want to modify more than one bit.
I'll try to get v3 out this week.
PATCH v2 2/3 x86/mce/inject: Avoid racy updates to MSR_K7_HWCR in toggle_hw_mce_inject()
In reply to Jim Mattsontoggle_hw_mce_inject() performs a read-modify-write of MSR_K7_HWCR as two independent crosscalls: one to read the MSR and one to write it. Another HWCR update on the target CPU can be lost if it occurs CPU between the two crosscalls.
Replace the two crosscalls with a single crosscall that performs the read-modify-write using the amd_update_hwcr() helper.
Opportunistically, replace the open-coded BIT(18) with a new MSR_K7_HWCR_MCSTATUSWREN macro.
--- arch/x86/include/asm/msr-index.h | 2 ++ arch/x86/kernel/cpu/mce/inject.c | 34 +++++++++++++++++++++++--------- 2 files changed, 27 insertions(+), 9 deletions(-)
arch/x86/include/asm/msr-index.h +2 -0
@@ -900,6 +900,8 @@#define MSR_K7_HWCR_IRPERF_EN BIT_ULL(MSR_K7_HWCR_IRPERF_EN_BIT)#define MSR_K7_HWCR_CPUID_USER_DIS_BIT 35#define MSR_K7_HWCR_CPUID_USER_DIS BIT_ULL(MSR_K7_HWCR_CPUID_USER_DIS_BIT) +#define MSR_K7_HWCR_MCSTATUSWREN_BIT 18 +#define MSR_K7_HWCR_MCSTATUSWREN BIT_ULL(MSR_K7_HWCR_MCSTATUSWREN_BIT)#define MSR_K7_FID_VID_CTL 0xc0010041#define MSR_K7_FID_VID_STATUS 0xc0010042#define MSR_K7_HWCR_CPB_DIS_BIT 25
arch/x86/kernel/cpu/mce/inject.c +25 -9
@@ -30,6 +30,7 @@#include <asm/mce.h> #include <asm/msr.h> #include <asm/nmi.h> +#include <asm/processor.h> #include <asm/smp.h> #include "internal.h"
@@ -310,28 +311,43 @@ static struct notifier_block inject_nb = { .notifier_call = mce_inject_raise, }; +struct hwcr_update_info { +u64 clear; +u64 set; +int err; +}; + +static void ipi_update_hwcr (void *info) +{ +struct hwcr_update_info *ui = info; + + ui->err =amd_update_hwcr (ui->clear, ui->set); +} +/* * Caller needs to be make sure this cpu doesn't disappear * from under us, i.e.: get_cpu/put_cpu. */ static int toggle_hw_mce_inject (unsigned int cpu,bool enable) { -u32 l, h; +struct hwcr_update_info ui = { + .clear = enable ?0 :MSR_K7_HWCR_MCSTATUSWREN , + .set = enable ?MSR_K7_HWCR_MCSTATUSWREN :0 , + };int err; - err =rdmsr_on_cpu (cpu,MSR_K7_HWCR , &l, &h); + err =smp_call_function_single (cpu, ipi_update_hwcr, &ui,1 );if (err) { -pr_err ("%s: error reading HWCR\n" , __func__); +pr_err ("%s: error calling ipi_update_hwcr on CPU %d\n" , __func__, cpu);return err; } - enable ? (l |=BIT (18 )) : (l &= ~BIT (18 )); - - err =wrmsr_on_cpu (cpu,MSR_K7_HWCR , l, h); -if (err) -pr_err ("%s: error writing HWCR\n" , __func__); +if (ui.err) { +pr_err ("%s: error updating HWCR on CPU %d\n" , __func__, cpu); +return ui.err; + } -return err; +return 0 ; }static int __set_inj (const char *buf)
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
PATCH v2 3/3 cpufreq: ACPI: Avoid racy updates to MSR_K7_HWCR in boost_set_msr()
In reply to Jim MattsonOn AMD and Hygon CPUs, use amd_update_hwcr() in boost_set_msr() so that all runtime updates of MSR_K7_HWCR go through the common interrupt-safe helper.
boost_set_msr() usually runs with interrupts disabled, but the call via cpufreq_boost_down_prep() on the cpufreq policy teardown path runs in process context with interrupts enabled. The MCE injector's toggle_hw_mce_inject() modifies the same MSR via a crosscall that can target any CPU. If such a crosscall lands between the open-coded read and write of MSR_K7_HWCR, the injector's McStatusWrEn update is lost.
--- drivers/cpufreq/acpi-cpufreq.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/cpufreq/acpi-cpufreq.c +4 -3
@@ -103,9 +103,10 @@ static int boost_set_msr(bool enable)break ;case X86_VENDOR_HYGON :case X86_VENDOR_AMD : - msr_addr =MSR_K7_HWCR ; - msr_mask =MSR_K7_HWCR_CPB_DIS ; -break ; +if (enable) +return amd_update_hwcr (MSR_K7_HWCR_CPB_DIS ,0 ); +else +return amd_update_hwcr (0 ,MSR_K7_HWCR_CPB_DIS );default :return -EINVAL ; }
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Quoted 3 lines
hoists the HWCR update logic out
into a new helper,amd_update_hwcr(), which performs the read-modify-write with interrupts disabled
Add some perf to your test scenario and these bugs will reappear, this time caused by NMIs.
I don't have any good suggestions. Core scope MSRs are horrible because you can't update them atomically.
At boot time you can fix it by ensuring the RMW will cause both threads to write the same value (i.e. the modify is not dependent on the read value, and that there is exactly one WRMSR in the entire kernel for this MSR).
For runtime with multiple agendas going on, you need some kind of spinlock because it's a plain concurrency problem.
~Andrew
Quoted 10 lines · Andrew Cooper
Quoted 4 lines
hoists the HWCR update logic out
into a new helper,amd_update_hwcr(), which performs the read-modify-write with interrupts disabledAdd some
perfto your test scenario and these bugs will reappear, this time caused by NMIs.I don't have any good suggestions. Core scope MSRs are horrible because you can't update them atomically.
I agree, but I spot-checked a couple of (Zen1+) PPRs, and I thought they said that HWCR has thread scope. For example, in
https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/programmer-references/56255_OSRR.pdf
has:
Quoted 1 line
_lthree[1:0]_core[3:0]_thread[1:0]; MSRC0010015
And, earlier:
Quoted 1 line
The absence of the instance parameter _thread[1:0] signifies that there is not a specific instance of said register per thread and thus the register is shared between thread 1 and thread 0.
Was it core-shared on older platforms?