Skip to contentSkip to conversation

perf/x86/intel: KVM: Mask PEBS_ENABLE loaded for guest with vCPU's value.

PATCH Applied Sean Christopherson 4 participants 8 messages Apr 26, 2025 – May 9, 2025

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 perf/x86/intel: KVM: Mask PEBS_ENABLE loaded for guest with vCPU's value.

When generating the MSR_IA32_PEBS_ENABLE value that will be loaded on VM-Entry to a KVM guest, mask the value with the vCPU's desired PEBS_ENABLE value. Consulting only the host kernel's host vs. guest masks results in running the guest with PEBS enabled even when the guest doesn't want to use PEBS. Because KVM uses perf events to proxy the guest virtual PMU, simply looking at exclude_host can't differentiate between events created by host userspace, and events created by KVM on behalf of the guest.

Running the guest with PEBS unexpectedly enabled typically manifests as crashes due to a near-infinite stream of #PFs. E.g. if the guest hasn't written MSR_IA32_DS_AREA, the CPU will hit page faults on address '0' when trying to record PEBS events.

The issue is most easily reproduced by running perf kvm top from before commit 7b100989b4f6 ("perf evlist: Remove __evlist__add_default") (after
which, perf kvm top effectively stopped using PEBS). The userspace side
of perf creates a guest-only PEBS event, which intel_guest_get_msrs() misconstrues a guest-*owned* PEBS event.

Arguably, this is a userspace bug, as enabling PEBS on guest-only events simply cannot work, and userspace can kill VMs in many other ways (there is no danger to the host). However, even if this is considered to be bad userspace behavior, there's zero downside to perf/KVM restricting PEBS to guest-owned events.

Note, commit 854250329c02 ("KVM: x86/pmu: Disable guest PEBS temporarily in two rare situations") fixed the case where host userspace is profiling KVM *and* userspace, but missed the case where userspace is profiling only KVM.

Fixes c59a1f106f5c ("KVM: x86/pmu: Add IA32_PEBS_ENABLE MSR emulation for extended PEBS")
Cc Dapeng Mi <[email protected]>
---
 arch/x86/events/intel/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/events/intel/core.c +1 -1
index cd6329207311..75a376478b21 100644
@@ -4625,7 +4625,7 @@ static struct perf_guest_switch_msr *intel_guest_get_msrs(int *nr, void *data)
 	arr[pebs_enable] = (struct perf_guest_switch_msr){
 		.msr = MSR_IA32_PEBS_ENABLE,
 		.host = cpuc->pebs_enabled & ~cpuc->intel_ctrl_guest_mask,
-		.guest = pebs_mask & ~cpuc->intel_ctrl_host_mask,
+		.guest = pebs_mask & ~cpuc->intel_ctrl_host_mask & kvm_pmu->pebs_enable,
 	};
 
 	if (arr[pebs_enable].host) {

base-commit: 2492e5aba2be064d0604ae23ae0770ecc0168192

Signature · 1 line
2.49.0.850.g28803427d3-goog
Reviewed-by on v1 1/1
#1 lore raw
Quoted 45 lines · Sean Christopherson

When generating the MSR_IA32_PEBS_ENABLE value that will be loaded on VM-Entry to a KVM guest, mask the value with the vCPU's desired PEBS_ENABLE value. Consulting only the host kernel's host vs. guest masks results in running the guest with PEBS enabled even when the guest doesn't want to use PEBS. Because KVM uses perf events to proxy the guest virtual PMU, simply looking at exclude_host can't differentiate between events created by host userspace, and events created by KVM on behalf of the guest.

Running the guest with PEBS unexpectedly enabled typically manifests as crashes due to a near-infinite stream of #PFs. E.g. if the guest hasn't written MSR_IA32_DS_AREA, the CPU will hit page faults on address '0' when trying to record PEBS events.

The issue is most easily reproduced by running perf kvm top from before commit 7b100989b4f6 ("perf evlist: Remove __evlist__add_default") (after
which, perf kvm top effectively stopped using PEBS). The userspace side
of perf creates a guest-only PEBS event, which intel_guest_get_msrs() misconstrues a guest-*owned* PEBS event.

Arguably, this is a userspace bug, as enabling PEBS on guest-only events simply cannot work, and userspace can kill VMs in many other ways (there is no danger to the host). However, even if this is considered to be bad userspace behavior, there's zero downside to perf/KVM restricting PEBS to guest-owned events.

Note, commit 854250329c02 ("KVM: x86/pmu: Disable guest PEBS temporarily in two rare situations") fixed the case where host userspace is profiling KVM *and* userspace, but missed the case where userspace is profiling only KVM.

Fixes c59a1f106f5c ("KVM: x86/pmu: Add IA32_PEBS_ENABLE MSR emulation for extended PEBS")
Cc Dapeng Mi <[email protected]>
---
 arch/x86/events/intel/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/events/intel/core.c +0 -0
index cd6329207311..75a376478b21 100644

arch/x86/events/intel/core.c static struct perf_guest_switch_msr *intel_guest_get_msrs(int *nr, void *data) · line 4628

Quoted 5 lines
arch/x86/events/intel/core.c +1 -1
@@ -4625,7 +4625,7 @@ static struct perf_guest_switch_msr *intel_guest_get_msrs(int *nr, void *data)
 	arr[pebs_enable] = (struct perf_guest_switch_msr){
 		.msr = MSR_IA32_PEBS_ENABLE,
 		.host = cpuc->pebs_enabled & ~cpuc->intel_ctrl_guest_mask,
-		.guest = pebs_mask & ~cpuc->intel_ctrl_host_mask,
+		.guest = pebs_mask & ~cpuc->intel_ctrl_host_mask & kvm_pmu->pebs_enable,

Although I can't reproduce the guest crash issue on local SPR server, the logic is correct.

Furthermore,  the below global_ctrl guest value calculation  seems incorrect as well.

    u64 pebs_mask = cpuc->pebs_enabled & x86_pmu.pebs_capable;

    arr[global_ctrl] = (struct perf_guest_switch_msr){
        .msr = MSR_CORE_PERF_GLOBAL_CTRL,
        .host = intel_ctrl & ~cpuc->intel_ctrl_guest_mask, .guest = intel_ctrl & ~cpuc->intel_ctrl_host_mask & ~pebs_mask, };

For guest PEBS events, its HW counter idx would be set into cpuc->pebs_enabled as well, so the calculated guest value would clear the corresponding bit of the guest PEBS events. Thus the guest PEBS events should not be enabled in non-root mode, but coincidentally since the calculated host and guest value of global_ctrl MSR is same, atomic_switch_perf_msrs() doesn't really write the guest value into GLOBAL_CTRL MSR before vm-entry and GLOBAL_CTRL MSR still keeps the default value (all HW counters are enabled). That's why we see guest PEBS still works.

for (i = 0; i < nr_msrs; i++)
        if (msrs[i].host == msrs[i].guest) clear_atomic_switch_msr(vmx, msrs[i].msr); else
            add_atomic_switch_msr(vmx, msrs[i].msr, msrs[i].guest, msrs[i].host, false);

Currently we have this Sean's fix, only the guest PEBS event bits of IA32_PEBS_ENABLE MSR are enabled in non-root mode, suppose we can simply change global_ctrl guest value calculation to this.

arch/x86/events/intel/core.c +0 -0
index 09d2d66c9f21..5bc56bb616ec 100644
@@ -4342,9 +4342,12 @@ static struct perf_guest_switch_msr

*intel_guest_get_msrs(int *nr, void *data)
        arr[global_ctrl] = (struct perf_guest_switch_msr){
                .msr = MSR_CORE_PERF_GLOBAL_CTRL,
                .host = intel_ctrl & ~cpuc->intel_ctrl_guest_mask, -               .guest = intel_ctrl & ~cpuc->intel_ctrl_host_mask & ~pebs_mask, +               .guest = intel_ctrl & ~cpuc->intel_ctrl_host_mask, };

Quoted 5 lines
arch/x86/events/intel/core.c +0 -0
 	};
 
 	if (arr[pebs_enable].host) {

base-commit: 2492e5aba2be064d0604ae23ae0770ecc0168192

#2 lore raw
Quoted 10 lines · Dapeng Mi

On 4/26/2025 8:13 AM, Sean Christopherson wrote:
Currently we have this Sean's fix, only the guest PEBS event bits of IA32_PEBS_ENABLE MSR are enabled in non-root mode, suppose we can simply change global_ctrl guest value calculation to this.

arch/x86/events/intel/core.c +0 -0
index 09d2d66c9f21..5bc56bb616ec 100644

arch/x86/events/intel/core.c static struct perf_guest_switch_msr · line 4347

Quoted 7 lines
arch/x86/events/intel/core.c +1 -1
@@ -4342,9 +4342,12 @@ static struct perf_guest_switch_msr
 *intel_guest_get_msrs(int *nr, void *data)
         arr[global_ctrl] = (struct perf_guest_switch_msr){
                 .msr = MSR_CORE_PERF_GLOBAL_CTRL,
                 .host = intel_ctrl & ~cpuc->intel_ctrl_guest_mask,
-               .guest = intel_ctrl & ~cpuc->intel_ctrl_host_mask & ~pebs_mask,
+               .guest = intel_ctrl & ~cpuc->intel_ctrl_host_mask,
         };

Hmm, that's not as clear cut. PEBS needs to be disabled because leaving it enabled will crash the guest. For the counter itself, unless leaving it enabled breaks perf and/or degrades the sampling, I don't think there's an obvious right/wrong approach.

E.g. if the host wants to profile host and guest, then keeping the count running while the guest is active might be a good thing. It's still far, far from perfect, as a counter that overflows when the guest is active won't generate a PEBS record, but without digging further, it's not obvious that even that flaw is overall worse than always disabling the counter.

#3 lore raw
Quoted 23 lines · Sean Christopherson
Quoted 10 lines · Dapeng Mi

On 4/26/2025 8:13 AM, Sean Christopherson wrote:
Currently we have this Sean's fix, only the guest PEBS event bits of IA32_PEBS_ENABLE MSR are enabled in non-root mode, suppose we can simply change global_ctrl guest value calculation to this.

arch/x86/events/intel/core.c +0 -0
index 09d2d66c9f21..5bc56bb616ec 100644

arch/x86/events/intel/core.c static struct perf_guest_switch_msr · line 4347

Quoted 7 lines
arch/x86/events/intel/core.c +1 -1
@@ -4342,9 +4342,12 @@ static struct perf_guest_switch_msr
 *intel_guest_get_msrs(int *nr, void *data)
         arr[global_ctrl] = (struct perf_guest_switch_msr){
                 .msr = MSR_CORE_PERF_GLOBAL_CTRL,
                 .host = intel_ctrl & ~cpuc->intel_ctrl_guest_mask,
-               .guest = intel_ctrl & ~cpuc->intel_ctrl_host_mask & ~pebs_mask,
+               .guest = intel_ctrl & ~cpuc->intel_ctrl_host_mask,
         };

Hmm, that's not as clear cut. PEBS needs to be disabled because leaving it enabled will crash the guest. For the counter itself, unless leaving it enabled breaks perf and/or degrades the sampling, I don't think there's an obvious right/wrong approach.

Quoted 5 lines

E.g. if the host wants to profile host and guest, then keeping the count running while the guest is active might be a good thing. It's still far, far from perfect, as a counter that overflows when the guest is active won't generate a PEBS record, but without digging further, it's not obvious that even that flaw is overall worse than always disabling the counter.

Hmm, if the host PEBS event only samples host side, whether the HW counter or the PEBS engine would be disabled once VM enters non-root mode, the KVM PEBS implementation is correct. But for the host PEBS events which sampling both host and guest, the implementation seems incorrect.

As the below code shows, as long as there are host PEBS events regardless of the host PEBS events only sample guest or both host and guest, the host PEBS events would be disabled on both HW counters and PEBS engine once VM enters non-root mode.

    arr[global_ctrl] = (struct perf_guest_switch_msr){
        .msr = MSR_CORE_PERF_GLOBAL_CTRL,
        .host = intel_ctrl & ~cpuc->intel_ctrl_guest_mask, .guest = intel_ctrl & ~cpuc->intel_ctrl_host_mask & ~pebs_mask, };

    if (arr[pebs_enable].host) {
        /* Disable guest PEBS if host PEBS is enabled. */ arr[pebs_enable].guest = 0;

    }

So the host PEBS events which hopes to sample both host and guest only samples host side in fact. This is unexpected.

I agree it's not enough to just remove the "~pebs_mask" for guest global_ctrl, a simple way to fix this issue could be to skip all PEBS related MSR switch at VM-exit/VM-entry and always keep the host vlaue in these PEBS MSRS regardless of in root or non-root mode.

#4 lore raw
Quoted 31 lines · Dapeng Mi
Quoted 23 lines · Sean Christopherson
Quoted 10 lines · Dapeng Mi

On 4/26/2025 8:13 AM, Sean Christopherson wrote:
Currently we have this Sean's fix, only the guest PEBS event bits of IA32_PEBS_ENABLE MSR are enabled in non-root mode, suppose we can simply change global_ctrl guest value calculation to this.

arch/x86/events/intel/core.c +0 -0
index 09d2d66c9f21..5bc56bb616ec 100644

arch/x86/events/intel/core.c static struct perf_guest_switch_msr · line 4347

Quoted 7 lines
arch/x86/events/intel/core.c +1 -1
@@ -4342,9 +4342,12 @@ static struct perf_guest_switch_msr
 *intel_guest_get_msrs(int *nr, void *data)
         arr[global_ctrl] = (struct perf_guest_switch_msr){
                 .msr = MSR_CORE_PERF_GLOBAL_CTRL,
                 .host = intel_ctrl & ~cpuc->intel_ctrl_guest_mask,
-               .guest = intel_ctrl & ~cpuc->intel_ctrl_host_mask & ~pebs_mask,
+               .guest = intel_ctrl & ~cpuc->intel_ctrl_host_mask,
         };

Hmm, that's not as clear cut. PEBS needs to be disabled because leaving it enabled will crash the guest. For the counter itself, unless leaving it enabled breaks perf and/or degrades the sampling, I don't think there's an obvious right/wrong approach.

Quoted 5 lines

E.g. if the host wants to profile host and guest, then keeping the count running while the guest is active might be a good thing. It's still far, far from perfect, as a counter that overflows when the guest is active won't generate a PEBS record, but without digging further, it's not obvious that even that flaw is overall worse than always disabling the counter.

Quoted 4 lines

Hmm, if the host PEBS event only samples host side, whether the HW counter or the PEBS engine would be disabled once VM enters non-root mode, the KVM PEBS implementation is correct. But for the host PEBS events which sampling both host and guest, the implementation seems incorrect.

Well, yeah, because there is no correct implementation.

Quoted 11 lines

As the below code shows, as long as there are host PEBS events regardless of the host PEBS events only sample guest or both host and guest, the host PEBS events would be disabled on both HW counters and PEBS engine once VM enters non-root mode.

    arr[global_ctrl] = (struct perf_guest_switch_msr){
        .msr = MSR_CORE_PERF_GLOBAL_CTRL,
        .host = intel_ctrl & ~cpuc->intel_ctrl_guest_mask, .guest = intel_ctrl & ~cpuc->intel_ctrl_host_mask & ~pebs_mask, };

Quoted 8 lines

    if (arr[pebs_enable].host) {
        /* Disable guest PEBS if host PEBS is enabled. */ arr[pebs_enable].guest = 0;

    }

So the host PEBS events which hopes to sample both host and guest only samples host side in fact. This is unexpected.

It's only unexpected in the sense that it's probably not well documented. Because the DS buffer is virtually address, there simply isn't a sane way to enable PEBS (or any feature that utizies the DS buffer) while running a KVM guest that isn't enlightened to explicitly allow profiling via host PEBS (and AFAIK, no such guest exists).

Even when KVM is using shadowing paging, i.e. fully controls the page tables used while the guest is running, enabling PEBS isn't feasible as KVM has no way to prevent the guest from using the virtual address. E.g. KVM could shove in mappings for the DS buffer, but that DoS the guest if the guest wants to use the same virtual address range for its own purposes, and would be a massive data leak to the guest since the guest could read host data from the buffer.

#5 lore raw
Quoted 61 lines · Sean Christopherson
Quoted 29 lines · Dapeng Mi
Quoted 23 lines · Sean Christopherson
Quoted 10 lines · Dapeng Mi

On 4/26/2025 8:13 AM, Sean Christopherson wrote:
Currently we have this Sean's fix, only the guest PEBS event bits of IA32_PEBS_ENABLE MSR are enabled in non-root mode, suppose we can simply change global_ctrl guest value calculation to this.

arch/x86/events/intel/core.c +0 -0
index 09d2d66c9f21..5bc56bb616ec 100644

arch/x86/events/intel/core.c static struct perf_guest_switch_msr · line 4347

Quoted 7 lines
arch/x86/events/intel/core.c +1 -1
@@ -4342,9 +4342,12 @@ static struct perf_guest_switch_msr
 *intel_guest_get_msrs(int *nr, void *data)
         arr[global_ctrl] = (struct perf_guest_switch_msr){
                 .msr = MSR_CORE_PERF_GLOBAL_CTRL,
                 .host = intel_ctrl & ~cpuc->intel_ctrl_guest_mask,
-               .guest = intel_ctrl & ~cpuc->intel_ctrl_host_mask & ~pebs_mask,
+               .guest = intel_ctrl & ~cpuc->intel_ctrl_host_mask,
         };

Hmm, that's not as clear cut. PEBS needs to be disabled because leaving it enabled will crash the guest. For the counter itself, unless leaving it enabled breaks perf and/or degrades the sampling, I don't think there's an obvious right/wrong approach.

Quoted 5 lines

E.g. if the host wants to profile host and guest, then keeping the count running while the guest is active might be a good thing. It's still far, far from perfect, as a counter that overflows when the guest is active won't generate a PEBS record, but without digging further, it's not obvious that even that flaw is overall worse than always disabling the counter.

Quoted 4 lines

Hmm, if the host PEBS event only samples host side, whether the HW counter or the PEBS engine would be disabled once VM enters non-root mode, the KVM PEBS implementation is correct. But for the host PEBS events which sampling both host and guest, the implementation seems incorrect.

Well, yeah, because there is no correct implementation.

Quoted 11 lines

As the below code shows, as long as there are host PEBS events regardless of the host PEBS events only sample guest or both host and guest, the host PEBS events would be disabled on both HW counters and PEBS engine once VM enters non-root mode.

    arr[global_ctrl] = (struct perf_guest_switch_msr){
        .msr = MSR_CORE_PERF_GLOBAL_CTRL,
        .host = intel_ctrl & ~cpuc->intel_ctrl_guest_mask, .guest = intel_ctrl & ~cpuc->intel_ctrl_host_mask & ~pebs_mask, };

Quoted 8 lines

    if (arr[pebs_enable].host) {
        /* Disable guest PEBS if host PEBS is enabled. */ arr[pebs_enable].guest = 0;

    }

So the host PEBS events which hopes to sample both host and guest only samples host side in fact. This is unexpected.

It's only unexpected in the sense that it's probably not well documented. Because the DS buffer is virtually address, there simply isn't a sane way to enable PEBS (or any feature that utizies the DS buffer) while running a KVM guest that isn't enlightened to explicitly allow profiling via host PEBS (and AFAIK, no such guest exists).

Quoted 6 lines

Even when KVM is using shadowing paging, i.e. fully controls the page tables used while the guest is running, enabling PEBS isn't feasible as KVM has no way to prevent the guest from using the virtual address. E.g. KVM could shove in mappings for the DS buffer, but that DoS the guest if the guest wants to use the same virtual address range for its own purposes, and would be a massive data leak to the guest since the guest could read host data from the buffer.

Yeah, that's true. Thanks for the explanation.

Quoted 1 line
Tested-by on v1 1/1
#6 lore raw
In reply to Sean Christopherson
Quoted 30 lines · Sean Christopherson

When generating the MSR_IA32_PEBS_ENABLE value that will be loaded on VM-Entry to a KVM guest, mask the value with the vCPU's desired PEBS_ENABLE value. Consulting only the host kernel's host vs. guest masks results in running the guest with PEBS enabled even when the guest doesn't want to use PEBS. Because KVM uses perf events to proxy the guest virtual PMU, simply looking at exclude_host can't differentiate between events created by host userspace, and events created by KVM on behalf of the guest.

Running the guest with PEBS unexpectedly enabled typically manifests as crashes due to a near-infinite stream of #PFs. E.g. if the guest hasn't written MSR_IA32_DS_AREA, the CPU will hit page faults on address '0' when trying to record PEBS events.

The issue is most easily reproduced by running perf kvm top from before commit 7b100989b4f6 ("perf evlist: Remove __evlist__add_default") (after
which, perf kvm top effectively stopped using PEBS). The userspace side
of perf creates a guest-only PEBS event, which intel_guest_get_msrs() misconstrues a guest-*owned* PEBS event.

Arguably, this is a userspace bug, as enabling PEBS on guest-only events simply cannot work, and userspace can kill VMs in many other ways (there is no danger to the host). However, even if this is considered to be bad userspace behavior, there's zero downside to perf/KVM restricting PEBS to guest-owned events.

Note, commit 854250329c02 ("KVM: x86/pmu: Disable guest PEBS temporarily in two rare situations") fixed the case where host userspace is profiling KVM *and* userspace, but missed the case where userspace is profiling only KVM.

Quoted 6 lines
Fixes c59a1f106f5c ("KVM: x86/pmu: Add IA32_PEBS_ENABLE MSR emulation for extended PEBS")
Cc Dapeng Mi <[email protected]>
#7 lore raw
In reply to Sean Christopherson
The following commit has been merged into the perf/urgent branch of tip:

The following commit has been merged into the perf/urgent branch of tip:

Commit-ID:     58f6217e5d0132a9f14e401e62796916aa055c1b
Gitweb:        https://git.kernel.org/tip/58f6217e5d0132a9f14e401e62796916aa055c1b
Author:        Sean Christopherson <[email protected]>
AuthorDate:    Fri, 25 Apr 2025 17:13:55 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Wed, 30 Apr 2025 13:58:29 +02:00

perf/x86/intel: KVM: Mask PEBS_ENABLE loaded for guest with vCPU's value.

When generating the MSR_IA32_PEBS_ENABLE value that will be loaded on VM-Entry to a KVM guest, mask the value with the vCPU's desired PEBS_ENABLE value. Consulting only the host kernel's host vs. guest masks results in running the guest with PEBS enabled even when the guest doesn't want to use PEBS. Because KVM uses perf events to proxy the guest virtual PMU, simply looking at exclude_host can't differentiate between events created by host userspace, and events created by KVM on behalf of the guest.

Running the guest with PEBS unexpectedly enabled typically manifests as crashes due to a near-infinite stream of #PFs. E.g. if the guest hasn't written MSR_IA32_DS_AREA, the CPU will hit page faults on address '0' when trying to record PEBS events.

The issue is most easily reproduced by running perf kvm top from before commit 7b100989b4f6 ("perf evlist: Remove __evlist__add_default") (after
which, perf kvm top effectively stopped using PEBS). The userspace side
of perf creates a guest-only PEBS event, which intel_guest_get_msrs() misconstrues a guest-*owned* PEBS event.

Arguably, this is a userspace bug, as enabling PEBS on guest-only events simply cannot work, and userspace can kill VMs in many other ways (there is no danger to the host). However, even if this is considered to be bad userspace behavior, there's zero downside to perf/KVM restricting PEBS to guest-owned events.

Note, commit 854250329c02 ("KVM: x86/pmu: Disable guest PEBS temporarily in two rare situations") fixed the case where host userspace is profiling KVM *and* userspace, but missed the case where userspace is profiling only KVM.

---
 arch/x86/events/intel/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/events/intel/core.c +1 -1
index 00dfe48..c5f3854 100644
@@ -4395,7 +4395,7 @@ static struct perf_guest_switch_msr *intel_guest_get_msrs(int *nr, void *data)
 	arr[pebs_enable] = (struct perf_guest_switch_msr){
 		.msr = MSR_IA32_PEBS_ENABLE,
 		.host = cpuc->pebs_enabled & ~cpuc->intel_ctrl_guest_mask,
-		.guest = pebs_mask & ~cpuc->intel_ctrl_host_mask,
+		.guest = pebs_mask & ~cpuc->intel_ctrl_host_mask & kvm_pmu->pebs_enable,
 	};
 
 	if (arr[pebs_enable].host) {