Skip to contentSkip to conversation

Small question about reserved bits in MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR

DISCUSSION Maxim Levitsky 4 participants 5 messages Sep 16, 2024 – Nov 11, 2024 KVM

lore
Share & tools
lore
#0 lore raw

Hi!

We recently saw a failure in one of the aws VM instances that causes the following error during the guest boot:

0.480051] unchecked MSR access error: WRMSR to 0xc0000302 (tried to write 0x040000000000001f) at rIP: 0xffffffff96c093e2 (amd_pmu_cpu_reset.constprop.0+0x42/0x80)

I investigated the issue and I see that the hypervisor does expose PerfmonV2, but not the LBRv2 support:

# cpuid -1 -l 0x80000022
CPU:
Extended Performance Monitoring and Debugging (0x80000022):

      AMD performance monitoring V2         = true
      AMD LBR V2                            = false
      AMD LBR stack & PMC freezing          = false
      number of core perf ctrs              = 0x5 (5)
      number of LBR stack entries           = 0x0 (0)
      number of avail Northbridge perf ctrs = 0x0 (0)
      number of available UMC PMCs          = 0x0 (0)
      active UMCs bitmask                   = 0x0

I also verified that I can write 0x1f to 0xc0000302 but not 0x040000000000001f:

# wrmsr 0xc0000302 0x1f
# wrmsr 0xc0000302 0x040000000000001f
wrmsr: CPU 0 cannot set MSR 0xc0000302 to 0x040000000000001f
#

The AMD's APM is not clear on what should happen if unsupported bits are attempted to be cleared using this MSR.

Also I noticed that amd_pmu_v2_handle_irq writes 0xffffffffffffffff to this msrs. It has the following code:

	WARN_ON(status > 0);

	/* Clear overflow and freeze bits */
	amd_pmu_ack_global_status(~status);

This implies that it is OK to set all bits in this MSR.

Can you please take a look?

Thanks in advance,
Best regards,
Maxim Levitsky

#1 lore raw
Quoted 35 lines · Maxim Levitsky

Hi!

We recently saw a failure in one of the aws VM instances that causes the following error during the guest boot:

0.480051] unchecked MSR access error: WRMSR to 0xc0000302 (tried to write 0x040000000000001f) at rIP: 0xffffffff96c093e2 (amd_pmu_cpu_reset.constprop.0+0x42/0x80)

I investigated the issue and I see that the hypervisor does expose PerfmonV2, but not the LBRv2 support:

# cpuid -1 -l 0x80000022
CPU:
Extended Performance Monitoring and Debugging (0x80000022):

      AMD performance monitoring V2         = true
      AMD LBR V2                            = false
      AMD LBR stack & PMC freezing          = false
      number of core perf ctrs              = 0x5 (5)
      number of LBR stack entries           = 0x0 (0)
      number of avail Northbridge perf ctrs = 0x0 (0)
      number of available UMC PMCs          = 0x0 (0)
      active UMCs bitmask                   = 0x0

I also verified that I can write 0x1f to 0xc0000302 but not 0x040000000000001f:

# wrmsr 0xc0000302 0x1f
# wrmsr 0xc0000302 0x040000000000001f
wrmsr: CPU 0 cannot set MSR 0xc0000302 to 0x040000000000001f
#

The AMD's APM is not clear on what should happen if unsupported bits are attempted to be cleared using this MSR.

Also I noticed that amd_pmu_v2_handle_irq writes 0xffffffffffffffff to this msrs. It has the following code:

Quoted 8 lines
	WARN_ON(status > 0);

	/* Clear overflow and freeze bits */
	amd_pmu_ack_global_status(~status);

This implies that it is OK to set all bits in this MSR.

To share my data point on QEMU+KVM: I am not able to reproduce with the most recent QEMU (not AWS) + below patch.

[PATCH v2 2/4] i386/cpu: Add PerfMonV2 feature bit
https://lore.kernel.org/all/69905b486218f8287b9703d1a9001175d04c2f02.1723068946.git.babu.moger@amd.com/

Both my VM and KVM are 6.10.

vm# cpuid -1 -l 0x80000022
CPU:
Extended Performance Monitoring and Debugging (0x80000022):

      AMD performance monitoring V2         = true
      AMD LBR V2                            = false
      AMD LBR stack & PMC freezing          = false
      number of core perf ctrs              = 0x6 (6)
      number of LBR stack entries           = 0x0 (0)
      number of avail Northbridge perf ctrs = 0x0 (0)
      number of available UMC PMCs          = 0x0 (0)
      active UMCs bitmask                   = 0x0

Both writes are passed.

vm# wrmsr 0xc0000302 0x1f
vm# wrmsr 0xc0000302 0x040000000000001f

Here is bcc output. Both writes are good.

kvm# /usr/share/bcc/tools/trace -t -C 'kvm_pmu_set_msr "%x", retval' ... ...

4.748614 19  43545   43550   CPU 0/KVM       kvm_pmu_set_msr  0
10.97396 19  43545   43550   CPU 0/KVM       kvm_pmu_set_msr  0

Dongli Zhang

#2 lore raw
Quoted 11 lines · On 9/17/2024 2:11 AM, [email protected] wrote:
Quoted 9 lines · Maxim Levitsky

Hi!

We recently saw a failure in one of the aws VM instances that causes the following error during the guest boot:

0.480051] unchecked MSR access error: WRMSR to 0xc0000302 (tried to write 0x040000000000001f) at rIP: 0xffffffff96c093e2 (amd_pmu_cpu_reset.constprop.0+0x42/0x80)

I investigated the issue and I see that the hypervisor does expose PerfmonV2, but not the LBRv2 support:

Quoted 12 lines
Quoted 12 lines

# cpuid -1 -l 0x80000022
CPU:
Extended Performance Monitoring and Debugging (0x80000022):

      AMD performance monitoring V2         = true
      AMD LBR V2                            = false
      AMD LBR stack & PMC freezing          = false
      number of core perf ctrs              = 0x5 (5)
      number of LBR stack entries           = 0x0 (0)
      number of avail Northbridge perf ctrs = 0x0 (0)
      number of available UMC PMCs          = 0x0 (0)
      active UMCs bitmask                   = 0x0

That's expected. LBRv2 is currently not available to KVM guests. However, PerfMonV2 should be the only feature bit required to indicate the availability of MSRs 0xc0000300..0xc0000303

Quoted 14 lines
Quoted 14 lines

I also verified that I can write 0x1f to 0xc0000302 but not 0x040000000000001f:

# wrmsr 0xc0000302 0x1f
# wrmsr 0xc0000302 0x040000000000001f
wrmsr: CPU 0 cannot set MSR 0xc0000302 to 0x040000000000001f
#

The AMD's APM is not clear on what should happen if unsupported bits are attempted to be cleared using this MSR.

Also I noticed that amd_pmu_v2_handle_irq writes 0xffffffffffffffff to this msrs. It has the following code:

Quoted 8 lines
Quoted 8 lines
	WARN_ON(status > 0);

	/* Clear overflow and freeze bits */
	amd_pmu_ack_global_status(~status);

This implies that it is OK to set all bits in this MSR.

It is, but writes to the reserved bits are ignored.

Quoted 27 lines

To share my data point on QEMU+KVM: I am not able to reproduce with the most recent QEMU (not AWS) + below patch.

[PATCH v2 2/4] i386/cpu: Add PerfMonV2 feature bit
https://lore.kernel.org/all/69905b486218f8287b9703d1a9001175d04c2f02.1723068946.git.babu.moger@amd.com/

Both my VM and KVM are 6.10.

vm# cpuid -1 -l 0x80000022
CPU:
Extended Performance Monitoring and Debugging (0x80000022):

      AMD performance monitoring V2         = true
      AMD LBR V2                            = false
      AMD LBR stack & PMC freezing          = false
      number of core perf ctrs              = 0x6 (6)
      number of LBR stack entries           = 0x0 (0)
      number of avail Northbridge perf ctrs = 0x0 (0)
      number of available UMC PMCs          = 0x0 (0)
      active UMCs bitmask                   = 0x0

Both writes are passed.

vm# wrmsr 0xc0000302 0x1f
vm# wrmsr 0xc0000302 0x040000000000001f

Quoted 7 lines

Here is bcc output. Both writes are good.

kvm# /usr/share/bcc/tools/trace -t -C 'kvm_pmu_set_msr "%x", retval' ... ...

4.748614 19  43545   43550   CPU 0/KVM       kvm_pmu_set_msr  0
10.97396 19  43545   43550   CPU 0/KVM       kvm_pmu_set_msr  0

Thanks for testing. I cannot replicate this either with an upstream kernel.

- Sandipan

#3 lore raw
Quoted 80 lines · Sandipan Das
Quoted 10 lines · On 9/17/2024 2:11 AM, [email protected] wrote:
Quoted 9 lines · Maxim Levitsky

Hi!

We recently saw a failure in one of the aws VM instances that causes the following error during the guest boot:

0.480051] unchecked MSR access error: WRMSR to 0xc0000302 (tried to write 0x040000000000001f) at rIP: 0xffffffff96c093e2 (amd_pmu_cpu_reset.constprop.0+0x42/0x80)

I investigated the issue and I see that the hypervisor does expose PerfmonV2, but not the LBRv2 support:

Quoted 12 lines
Quoted 12 lines

# cpuid -1 -l 0x80000022
CPU:
Extended Performance Monitoring and Debugging (0x80000022):

      AMD performance monitoring V2         = true
      AMD LBR V2                            = false
      AMD LBR stack & PMC freezing          = false
      number of core perf ctrs              = 0x5 (5)
      number of LBR stack entries           = 0x0 (0)
      number of avail Northbridge perf ctrs = 0x0 (0)
      number of available UMC PMCs          = 0x0 (0)
      active UMCs bitmask                   = 0x0

That's expected. LBRv2 is currently not available to KVM guests. However, PerfMonV2 should be the only feature bit required to indicate the availability of MSRs 0xc0000300..0xc0000303

Quoted 14 lines
Quoted 14 lines

I also verified that I can write 0x1f to 0xc0000302 but not 0x040000000000001f:

# wrmsr 0xc0000302 0x1f
# wrmsr 0xc0000302 0x040000000000001f
wrmsr: CPU 0 cannot set MSR 0xc0000302 to 0x040000000000001f
#

The AMD's APM is not clear on what should happen if unsupported bits are attempted to be cleared using this MSR.

Also I noticed that amd_pmu_v2_handle_irq writes 0xffffffffffffffff to this msr. It has the following code:

Quoted 8 lines
Quoted 8 lines
	WARN_ON(status > 0);

	/* Clear overflow and freeze bits */
	amd_pmu_ack_global_status(~status);

This implies that it is OK to set all bits in this MSR.

It is, but writes to the reserved bits are ignored.

Quoted 28 lines

To share my data point on QEMU+KVM: I am not able to reproduce with the most recent QEMU (not AWS) + below patch.

[PATCH v2 2/4] i386/cpu: Add PerfMonV2 feature bit
https://lore.kernel.org/all/69905b486218f8287b9703d1a9001175d04c2f02.1723068946.git.babu.moger@amd.com/

Both my VM and KVM are 6.10.

vm# cpuid -1 -l 0x80000022
CPU:
Extended Performance Monitoring and Debugging (0x80000022):

      AMD performance monitoring V2         = true
      AMD LBR V2                            = false
      AMD LBR stack & PMC freezing          = false
      number of core perf ctrs              = 0x6 (6)
      number of LBR stack entries           = 0x0 (0)
      number of avail Northbridge perf ctrs = 0x0 (0)
      number of available UMC PMCs          = 0x0 (0)
      active UMCs bitmask                   = 0x0

Both writes are passed.

vm# wrmsr 0xc0000302 0x1f
vm# wrmsr 0xc0000302 0x040000000000001f

Here is bcc output. Both writes are good.

Quoted 7 lines
Quoted 5 lines

kvm# /usr/share/bcc/tools/trace -t -C 'kvm_pmu_set_msr "%x", retval' ... ...

4.748614 19  43545   43550   CPU 0/KVM       kvm_pmu_set_msr  0
10.97396 19  43545   43550   CPU 0/KVM       kvm_pmu_set_msr  0

Thanks for testing. I cannot replicate this either with an upstream kernel.

Hi,

I also tested on bare metal Zen4 system just now, and I also see that MSR 0xc0000302 can be set to any value.

So this is a hypervisor bug, I'll report it to AWS.

Best regards,
Maxim Levitsky

Quoted 3 lines

- Sandipan

#4 lore raw
Quoted 4 lines · Maxim Levitsky

I also tested on bare metal Zen4 system just now, and I also see that MSR 0xc0000302 can be set to any value.

So this is a hypervisor bug, I'll report it to AWS.

Have you gotten any response from AWS? I'm still seeing this bug on current c7a instances.

- Josh Triplett