Small question about reserved bits in MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR
Hi!
We recently saw a failure in one of the aws VM instances that causes the following error during the guest boot:
0.480051] unchecked MSR access error: WRMSR to 0xc0000302 (tried to write 0x040000000000001f) at rIP: 0xffffffff96c093e2 (amd_pmu_cpu_reset.constprop.0+0x42/0x80)
I investigated the issue and I see that the hypervisor does expose PerfmonV2, but not the LBRv2 support:
# cpuid -1 -l 0x80000022
CPU:
Extended Performance Monitoring and Debugging (0x80000022):
AMD performance monitoring V2 = true
AMD LBR V2 = false
AMD LBR stack & PMC freezing = false
number of core perf ctrs = 0x5 (5)
number of LBR stack entries = 0x0 (0)
number of avail Northbridge perf ctrs = 0x0 (0)
number of available UMC PMCs = 0x0 (0)
active UMCs bitmask = 0x0I also verified that I can write 0x1f to 0xc0000302 but not 0x040000000000001f:
# wrmsr 0xc0000302 0x1f # wrmsr 0xc0000302 0x040000000000001f wrmsr: CPU 0 cannot set MSR 0xc0000302 to 0x040000000000001f #
The AMD's APM is not clear on what should happen if unsupported bits are attempted to be cleared using this MSR.
Also I noticed that amd_pmu_v2_handle_irq writes 0xffffffffffffffff to this msrs. It has the following code:
WARN_ON(status > 0); /* Clear overflow and freeze bits */ amd_pmu_ack_global_status(~status);
This implies that it is OK to set all bits in this MSR.
Can you please take a look?
Thanks in advance,
Best regards,
Maxim Levitsky
Quoted 35 lines · Maxim Levitsky
Hi!
We recently saw a failure in one of the aws VM instances that causes the following error during the guest boot:
0.480051] unchecked MSR access error: WRMSR to
0xc0000302(tried to write0x040000000000001f) at rIP:0xffffffff96c093e2(amd_pmu_cpu_reset.constprop.0+0x42/0x80)I investigated the issue and I see that the hypervisor does expose PerfmonV2, but not the LBRv2 support:
# cpuid -1 -l
0x80000022
CPU:
Extended Performance Monitoring and Debugging (0x80000022):AMD performance monitoring V2 = true AMD LBR V2 = false AMD LBR stack & PMC freezing = false number of core perf ctrs = 0x5 (5) number of LBR stack entries = 0x0 (0) number of avail Northbridge perf ctrs = 0x0 (0) number of available UMC PMCs = 0x0 (0) active UMCs bitmask = 0x0I also verified that I can write
0x1fto0xc0000302but not0x040000000000001f:# wrmsr 0xc0000302 0x1f # wrmsr 0xc0000302 0x040000000000001f wrmsr: CPU 0 cannot set MSR 0xc0000302 to 0x040000000000001f #The AMD's APM is not clear on what should happen if unsupported bits are attempted to be cleared using this MSR.
Also I noticed that
amd_pmu_v2_handle_irqwrites0xffffffffffffffffto this msrs. It has the following code:
Quoted 8 lines
WARN_ON(status > 0); /* Clear overflow and freeze bits */ amd_pmu_ack_global_status(~status);This implies that it is OK to set all bits in this MSR.
To share my data point on QEMU+KVM: I am not able to reproduce with the most recent QEMU (not AWS) + below patch.
[PATCH v2 2/4] i386/cpu: Add PerfMonV2 feature bit
https://lore.kernel.org/all/69905b486218f8287b9703d1a9001175d04c2f02.1723068946.git.babu.moger@amd.com/
Both my VM and KVM are 6.10.
vm# cpuid -1 -l 0x80000022
CPU:
Extended Performance Monitoring and Debugging (0x80000022):
AMD performance monitoring V2 = true
AMD LBR V2 = false
AMD LBR stack & PMC freezing = false
number of core perf ctrs = 0x6 (6)
number of LBR stack entries = 0x0 (0)
number of avail Northbridge perf ctrs = 0x0 (0)
number of available UMC PMCs = 0x0 (0)
active UMCs bitmask = 0x0Both writes are passed.
vm# wrmsr 0xc0000302 0x1f
vm# wrmsr 0xc0000302 0x040000000000001f
Here is bcc output. Both writes are good.
kvm# /usr/share/bcc/tools/trace -t -C 'kvm_pmu_set_msr "%x", retval' ... ...
4.748614 19 43545 43550 CPU 0/KVM kvm_pmu_set_msr 0 10.97396 19 43545 43550 CPU 0/KVM kvm_pmu_set_msr 0
Dongli Zhang
Quoted 11 lines · On 9/17/2024 2:11 AM, [email protected] wrote:
Quoted 9 lines · Maxim Levitsky
Hi!
We recently saw a failure in one of the aws VM instances that causes the following error during the guest boot:
0.480051] unchecked MSR access error: WRMSR to
0xc0000302(tried to write0x040000000000001f) at rIP:0xffffffff96c093e2(amd_pmu_cpu_reset.constprop.0+0x42/0x80)I investigated the issue and I see that the hypervisor does expose PerfmonV2, but not the LBRv2 support:
Quoted 12 lines
Quoted 12 lines
# cpuid -1 -l
0x80000022
CPU:
Extended Performance Monitoring and Debugging (0x80000022):AMD performance monitoring V2 = true AMD LBR V2 = false AMD LBR stack & PMC freezing = false number of core perf ctrs = 0x5 (5) number of LBR stack entries = 0x0 (0) number of avail Northbridge perf ctrs = 0x0 (0) number of available UMC PMCs = 0x0 (0) active UMCs bitmask = 0x0
That's expected. LBRv2 is currently not available to KVM guests. However, PerfMonV2 should be the only feature bit required to indicate the availability of MSRs 0xc0000300..0xc0000303
Quoted 14 lines
Quoted 14 lines
I also verified that I can write
0x1fto0xc0000302but not0x040000000000001f:# wrmsr 0xc0000302 0x1f # wrmsr 0xc0000302 0x040000000000001f wrmsr: CPU 0 cannot set MSR 0xc0000302 to 0x040000000000001f #The AMD's APM is not clear on what should happen if unsupported bits are attempted to be cleared using this MSR.
Also I noticed that
amd_pmu_v2_handle_irqwrites0xffffffffffffffffto this msrs. It has the following code:
Quoted 8 lines
Quoted 8 lines
WARN_ON(status > 0); /* Clear overflow and freeze bits */ amd_pmu_ack_global_status(~status);This implies that it is OK to set all bits in this MSR.
It is, but writes to the reserved bits are ignored.
Quoted 27 lines
To share my data point on QEMU+KVM: I am not able to reproduce with the most recent QEMU (not AWS) + below patch.
[PATCH v2 2/4] i386/cpu: Add PerfMonV2 feature bit
https://lore.kernel.org/all/69905b486218f8287b9703d1a9001175d04c2f02.1723068946.git.babu.moger@amd.com/Both my VM and KVM are 6.10.
vm# cpuid -1 -l
0x80000022
CPU:
Extended Performance Monitoring and Debugging (0x80000022):AMD performance monitoring V2 = true AMD LBR V2 = false AMD LBR stack & PMC freezing = false number of core perf ctrs = 0x6 (6) number of LBR stack entries = 0x0 (0) number of avail Northbridge perf ctrs = 0x0 (0) number of available UMC PMCs = 0x0 (0) active UMCs bitmask = 0x0Both writes are passed.
vm# wrmsr
0xc00003020x1f
vm# wrmsr0xc00003020x040000000000001f
Quoted 7 lines
Here is bcc output. Both writes are good.
kvm#
/usr/share/bcc/tools/trace-t -C 'kvm_pmu_set_msr"%x", retval' ... ...4.748614 19 43545 43550 CPU 0/KVM kvm_pmu_set_msr 0 10.97396 19 43545 43550 CPU 0/KVM kvm_pmu_set_msr 0
Thanks for testing. I cannot replicate this either with an upstream kernel.
- Sandipan
Quoted 80 lines · Sandipan Das
Quoted 10 lines · On 9/17/2024 2:11 AM, [email protected] wrote:
Quoted 9 lines · Maxim Levitsky
Hi!
We recently saw a failure in one of the aws VM instances that causes the following error during the guest boot:
0.480051] unchecked MSR access error: WRMSR to
0xc0000302(tried to write0x040000000000001f) at rIP:0xffffffff96c093e2(amd_pmu_cpu_reset.constprop.0+0x42/0x80)I investigated the issue and I see that the hypervisor does expose PerfmonV2, but not the LBRv2 support:
Quoted 12 lines
Quoted 12 lines
# cpuid -1 -l
0x80000022
CPU:
Extended Performance Monitoring and Debugging (0x80000022):AMD performance monitoring V2 = true AMD LBR V2 = false AMD LBR stack & PMC freezing = false number of core perf ctrs = 0x5 (5) number of LBR stack entries = 0x0 (0) number of avail Northbridge perf ctrs = 0x0 (0) number of available UMC PMCs = 0x0 (0) active UMCs bitmask = 0x0That's expected. LBRv2 is currently not available to KVM guests. However, PerfMonV2 should be the only feature bit required to indicate the availability of MSRs 0xc0000300..0xc0000303
Quoted 14 lines
Quoted 14 lines
I also verified that I can write
0x1fto0xc0000302but not0x040000000000001f:# wrmsr 0xc0000302 0x1f # wrmsr 0xc0000302 0x040000000000001f wrmsr: CPU 0 cannot set MSR 0xc0000302 to 0x040000000000001f #The AMD's APM is not clear on what should happen if unsupported bits are attempted to be cleared using this MSR.
Also I noticed that
amd_pmu_v2_handle_irqwrites0xffffffffffffffffto this msr. It has the following code:Quoted 8 lines
Quoted 8 lines
WARN_ON(status > 0); /* Clear overflow and freeze bits */ amd_pmu_ack_global_status(~status);This implies that it is OK to set all bits in this MSR.
It is, but writes to the reserved bits are ignored.
Quoted 28 lines
To share my data point on QEMU+KVM: I am not able to reproduce with the most recent QEMU (not AWS) + below patch.
[PATCH v2 2/4] i386/cpu: Add PerfMonV2 feature bit
https://lore.kernel.org/all/69905b486218f8287b9703d1a9001175d04c2f02.1723068946.git.babu.moger@amd.com/Both my VM and KVM are 6.10.
vm# cpuid -1 -l
0x80000022
CPU:
Extended Performance Monitoring and Debugging (0x80000022):AMD performance monitoring V2 = true AMD LBR V2 = false AMD LBR stack & PMC freezing = false number of core perf ctrs = 0x6 (6) number of LBR stack entries = 0x0 (0) number of avail Northbridge perf ctrs = 0x0 (0) number of available UMC PMCs = 0x0 (0) active UMCs bitmask = 0x0Both writes are passed.
vm# wrmsr
0xc00003020x1f
vm# wrmsr0xc00003020x040000000000001fHere is bcc output. Both writes are good.
Quoted 7 lines
Quoted 5 lines
kvm#
/usr/share/bcc/tools/trace-t -C 'kvm_pmu_set_msr"%x", retval' ... ...4.748614 19 43545 43550 CPU 0/KVM kvm_pmu_set_msr 0 10.97396 19 43545 43550 CPU 0/KVM kvm_pmu_set_msr 0Thanks for testing. I cannot replicate this either with an upstream kernel.
Hi,
I also tested on bare metal Zen4 system just now, and I also see that MSR 0xc0000302 can be set to any value.
So this is a hypervisor bug, I'll report it to AWS.
Best regards,
Maxim Levitsky
Quoted 3 lines
- Sandipan
Quoted 4 lines · Maxim Levitsky
I also tested on bare metal Zen4 system just now, and I also see that MSR
0xc0000302can be set to any value.So this is a hypervisor bug, I'll report it to AWS.
Have you gotten any response from AWS? I'm still seeing this bug on current c7a instances.
- Josh Triplett