Hitting AUTOIBRS WARN_ON_ONCE() in init_amd() booting 32-bit kernel under KVM
Hi Boris and x86 + KVM folks,
I got access to a new box that has an EPYC 9454P in it and I noticed that I hit the warning from
/*
* Make sure EFER[AIBRSE - Automatic IBRS Enable] is set. The APs are brought up
* using the trampoline code and as part of it, MSR_EFER gets prepared there in
* order to be replicated onto them. Regardless, set it here again, if not set,
* to protect against any future refactoring/code reorganization which might
* miss setting this important bit.
*/
if (spectre_v2_in_eibrs_mode(spectre_v2_enabled) &&
cpu_has(c, X86_FEATURE_AUTOIBRS))
WARN_ON_ONCE(msr_set_bit(MSR_EFER, _EFER_AUTOIBRS));
that was added by commit 8cc68c9c9e92 ("x86/CPU/AMD: Make sure EFER[AIBRSE] is set") when booting a 32-bit kernel in QEMU with KVM. I do not see this without KVM, so maybe this has something to do with commit 8c19b6f257fa ("KVM: x86: Propagate the AMD Automatic IBRS feature to the guest") as well?
I've included as much information as I can below for reproduction. If there is anything more I can provide or test, I am more than happy to do so. If this is expected given it is a 32-bit guest or something else, I can just ignore the warning in my report infrastructure. I have not checked if this is a recent regression since I have only had this box for a couple of days.
$ uname -r 6.13.0-rc1-debug
$ lscpu
...
Model name: AMD EPYC 9454P 48-Core Processor
CPU family: 25
Model: 17
...
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Reg file data sampling: Not affected
Retbleed: Not affected
Spec rstack overflow: Mitigation; Safe RET
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Srbds: Not affected
Tsx async abort: Not affected$ qemu-system-i386 --version | head -1 QEMU emulator version 9.1.2
$ git show --format='%h ("%s")' -s
896d8946da97 ("Merge tag 'net-6.13-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net")$ make -skj"$(nproc)" ARCH=i386 CROSS_COMPILE=i386-linux- mrproper defconfig bzImage
$ curl -LSs https://github.com/ClangBuiltLinux/boot-utils/releases/download/20241120-044434/x86-rootfs.cpio.zst | zstd -d >rootfs.cpio
$ qemu-system-i386 \
-display none \
-nodefaults \
-M q35 \
-d unimp,guest_errors \
-append 'console=ttyS0 earlycon=uart8250,io,0x3f8' \
-kernel arch/x86/boot/bzImage \
-initrd rootfs.cpio \
-cpu host \
-enable-kvm \
-m 512m \
-smp 8 \
-serial mon:stdio
[ 0.000000] Linux version 6.13.0-rc1-00170-g896d8946da97 (nathan@ax162) (i386-linux-gcc (GCC) 14.2.0, GNU ld (GNU Binutils) 2.42) #1 SMP PREEMPT_DYNAMIC Thu Dec 5 13:54:30 MST 2024
...
[ 0.096072] smp: Bringing up secondary CPUs ...
[ 0.096731] smpboot: x86: Booting SMP configuration:
[ 0.097004] .... node #0, CPUs: #1
[ 0.009830] ------------[ cut here ]------------
[ 0.009830] WARNING: CPU: 1 PID: 0 at arch/x86/kernel/cpu/amd.c:1068 init_amd+0x50f/0xa20
[ 0.009830] Modules linked in:
[ 0.009830] CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Not tainted 6.13.0-rc1-00170-g896d8946da97 #1
[ 0.009830] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS Arch Linux 1.16.3-1-1 04/01/2014
[ 0.009830] EIP: init_amd+0x50f/0xa20
[ 0.009830] Code: 4d 00 f0 80 4f 78 04 e9 b4 fd ff ff 8d b4 26 00 00 00 00 ba 15 00 00 00 b8 80 00 00 c0 e8 79 65 4d 00 85 c0 0f 84 01 fe ff ff <0f> 0b e9 fa fd ff ff 2e 8d b4 26 00 00 00 00 66 90 8b 47 38 85 c0
[ 0.009830] EAX: 00000001 EBX: 00000011 ECX: c0000080 EDX: 00000000
[ 0.009830] ESI: df5850a1 EDI: df585020 EBP: c1157f54 ESP: c1157f04
[ 0.009830] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00210002
[ 0.009830] CR0: 80050033 CR2: 00000000 CR3: 0d6ce000 CR4: 00350e90
[ 0.009830] Call Trace:
[ 0.009830] ? show_regs.cold+0x16/0x1e
[ 0.009830] ? __warn.cold+0xbf/0x114
[ 0.009830] ? init_amd+0x50f/0xa20
[ 0.009830] ? init_amd+0x50f/0xa20
[ 0.009830] ? report_bug+0x116/0x150
[ 0.009830] ? init_amd+0x510/0xa20
[ 0.009830] ? exc_overflow+0x50/0x50
[ 0.009830] ? handle_bug+0x56/0x90
[ 0.009830] ? exc_invalid_op+0x1b/0x70
[ 0.009830] ? handle_exception+0x14b/0x14b
[ 0.009830] ? exc_overflow+0x50/0x50
[ 0.009830] ? init_amd+0x50f/0xa20
[ 0.009830] ? exc_overflow+0x50/0x50
[ 0.009830] ? init_amd+0x50f/0xa20
[ 0.009830] identify_cpu+0x29e/0x700
[ 0.009830] identify_secondary_cpu+0xf/0x70
[ 0.009830] smp_store_cpu_info+0x5a/0x70
[ 0.009830] start_secondary+0x6e/0x100
[ 0.009830] startup_32_smp+0x151/0x154
[ 0.009830] ---[ end trace 0000000000000000 ]---
...Cheers,
Nathan
Sean Christopherson
reviewerQuoted 16 lines · Nathan Chancellor
Hi Boris and x86 + KVM folks,
I got access to a new box that has an EPYC 9454P in it and I noticed that I hit the warning from
/*
* Make sure EFER[AIBRSE - Automatic IBRS Enable] is set. The APs are brought up
* using the trampoline code and as part of it,MSR_EFERgets prepared there in
* order to be replicated onto them. Regardless, set it here again, if not set,
* to protect against any future refactoring/code reorganization which might
* miss setting this important bit.
*/
if (spectre_v2_in_eibrs_mode(spectre_v2_enabled)&&
cpu_has(c, X86_FEATURE_AUTOIBRS))
WARN_ON_ONCE(msr_set_bit(MSR_EFER, _EFER_AUTOIBRS));
Quoted 5 lines
that was added by commit
8cc68c9c9e92("x86/CPU/AMD: Make sure EFER[AIBRSE] is set") when booting a 32-bit kernel in QEMU with KVM. I do not see this without KVM, so maybe this has something to do with commit8c19b6f257fa("KVM: x86: Propagate the AMD Automatic IBRS feature to the guest") as well?
This is a bug in the above code. msr_set_bit() returns '1' on a successful write. Presumably spectre_v2_select_mitigation() sets EFER.AUTOIBRS when booting on bare metal, in which case msr_set_bit() returns '0' because the bit is already set.
--
From: Sean Christopherson <[email protected]>
Date: Fri, 6 Dec 2024 08:14:45 -0800
Subject: [PATCH] x86/CPU/AMD: WARN when setting EFER.AUTOIBRS if and only if
the WRMSR fails
When ensuring EFER.AUTOIBRS is set, WARN only on a negative return code from msr_set_bit(), as '1' is used to indicate the WRMSR was successful ('0' indicates the MSR bit was already set).
--- arch/x86/kernel/cpu/amd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/amd.c +1 -1
@@ -1065,7 +1065,7 @@ static void init_amd(struct cpuinfo_x86 *c)*/ if (spectre_v2_in_eibrs_mode (spectre_v2_enabled) &&cpu_has (c,X86_FEATURE_AUTOIBRS )) -WARN_ON_ONCE (msr_set_bit (MSR_EFER , _EFER_AUTOIBRS)); +WARN_ON_ONCE (msr_set_bit (MSR_EFER , _EFER_AUTOIBRS) <0 );/* AMD CPUs don't need fencing after x2APIC/TSC_DEADLINE MSR writes. */ clear_cpu_cap (c,X86_FEATURE_APIC_MSRS_FENCE );
base-commit: b8f52214c61a5b99a54168145378e91b40d10c90
[tip: x86/urgent] x86/CPU/AMD: WARN when setting EFER.AUTOIBRS if and only if the WRMSR fails
The following commit has been merged into the x86/urgent branch of tip:
The following commit has been merged into the x86/urgent branch of tip:
Commit-ID: 492077668fb453b8b16c842fcf3fafc2ebc190e9 Gitweb: https://git.kernel.org/tip/492077668fb453b8b16c842fcf3fafc2ebc190e9 Author: Sean Christopherson <[email protected]> AuthorDate: Fri, 06 Dec 2024 08:20:06 -08:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 06 Dec 2024 19:57:05 +01:00
x86/CPU/AMD: WARN when setting EFER.AUTOIBRS if and only if the WRMSR fails
When ensuring EFER.AUTOIBRS is set, WARN only on a negative return code from msr_set_bit(), as '1' is used to indicate the WRMSR was successful ('0' indicates the MSR bit was already set).
--- arch/x86/kernel/cpu/amd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/amd.c +1 -1
@@ -1065,7 +1065,7 @@ static void init_amd(struct cpuinfo_x86 *c)*/ if (spectre_v2_in_eibrs_mode (spectre_v2_enabled) &&cpu_has (c,X86_FEATURE_AUTOIBRS )) -WARN_ON_ONCE (msr_set_bit (MSR_EFER , _EFER_AUTOIBRS)); +WARN_ON_ONCE (msr_set_bit (MSR_EFER , _EFER_AUTOIBRS) <0 );/* AMD CPUs don't need fencing after x2APIC/TSC_DEADLINE MSR writes. */ clear_cpu_cap (c,X86_FEATURE_APIC_MSRS_FENCE );
Quoted 5 lines · Sean Christopherson
From: Sean Christopherson <[email protected]>
Date: Fri, 6 Dec 2024 08:14:45 -0800
Subject: [PATCH] x86/CPU/AMD: WARN when setting EFER.AUTOIBRS if and only if
the WRMSR fails
Quoted 8 lines
When ensuring EFER.AUTOIBRS is set, WARN only on a negative return code from
msr_set_bit(), as '1' is used to indicate the WRMSR was successful ('0' indicates the MSR bit was already set).Fixes 8cc68c9c9e92 ("x86/CPU/AMD: Make sure EFER[AIBRSE] is set")Reported-by Nathan Chancellor <[email protected]>Signed-off-by Sean Christopherson <[email protected]>
LGTM, but please post as a proper patch in its own thread so the -tip maintainers are more likely to see it.
Signature · 1 line
Josh
Sean Christopherson
reviewerQuoted 10 lines · Josh Poimboeuf
Quoted 9 lines · Sean Christopherson
From: Sean Christopherson <[email protected]>
Date: Fri, 6 Dec 2024 08:14:45 -0800
Subject: [PATCH] x86/CPU/AMD: WARN when setting EFER.AUTOIBRS if and only if
the WRMSR failsWhen ensuring EFER.AUTOIBRS is set, WARN only on a negative return code from
msr_set_bit(), as '1' is used to indicate the WRMSR was successful ('0' indicates the MSR bit was already set).
Quoted 7 lines
Quoted 4 lines
Fixes 8cc68c9c9e92 ("x86/CPU/AMD: Make sure EFER[AIBRSE] is set")Reported-by Nathan Chancellor <[email protected]>Signed-off-by Sean Christopherson <[email protected]>LGTM, but please post as a proper patch in its own thread so the -tip maintainers are more likely to see it.
In Boris I trust :-)
Already in Linus' tree, commit 492077668fb4 ("x86/CPU/AMD: WARN when setting EFER.AUTOIBRS if and only if the WRMSR fails").
Borislav Petkov
maintainerQuoted 1 line · Sean Christopherson
In Boris I trust :-)
Oh wow. Like I'm on money and stuff. :-P
It should actually be
s/Boris/Ingo/
who fished it out.
:-)
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette