perf/KVM: Support PMU partitioning for x86 platforms
Subsystems (7)
cover perf/KVM: Support PMU partitioning for x86 platforms
Introduction
============
PMU partitioning allows a VMM to assign a subset of PMU resources to a guest while retaining the remainder for host use, enabling concurrent host and guest PMU usage while the guest is running.
Intel PerfMon masking provides architectural support for PMU partitioning. This VMX extension provides fine-grained, bit-level control over global PerfMon MSRs and RDPMC, allowing a guest to natively access PMU resources while hardware ensures that guest accesses cannot affect host-owned resources. This eliminates the need to intercept those accesses for PMU isolation.
The feature is enumerated by IA32_VMX_PROCBASED_CTLS3[17] and enabled via the new PerfMon masking VM-execution control in tertiary processor-based VM-execution controls.
A new 64-bit VMCS field, PERFMON_MASK (encoding 0x2054/0x2055), defines the guest-owned PMU resources. The mask follows the IA32_PERF_GLOBAL_STATUS bit layout.
When enabled, hardware enforces guest access to selected PerfMon MSRs and RDPMC according to PERFMON_MASK, reducing the need for software emulation and interception.
The full description of PerfMon masking can be found at Intel ISA, 319433-062 [1]. Diamond Rapids is the first platform to support this feature.
Design
======
1. KVM
This series builds PMU partitioning on top of the mediated vPMU, which allows guest access to PerfMon MSRs without emulation, making it a natural foundation for PerfMon masking.
A new Intel-specific KVM module parameter, perfmon_mask, defines the host-wide PMU partition. The mask uses the PERFMON_MASK VMCS encoding, where set bits identify guest-owned PMU resources and cleared bits identify host-owned resources. The setting is shared by all mediated vPMU guests on the host.
KVM constrains its PMU capability based on the perfmon_mask parameter, so only minimal additional changes are needed to manage PMU capability advertisement and MSR emulation.
Upon VMX transition, unlike the non-partitioned mediated vPMU, KVM always preserves the host-owned bits in the global MSRs.
This series does not yet implement the full PerfMon masking software guideline in section 10.5 of [1]. For example, the RELOAD_{PMCx,FIXED_CTRx} and IA32_PMC_GPn_CFG_{A,B,C} MSRs are not yet supported by KVM.
2. perf core
The perf scheduler is taught to honor the PMU partition mask:
- Loading guest context: reschedule system-wide events onto host-owned counters.
- Putting guest context: schedule out system-wide events before rescheduling in so that they can be scheduled on now more available counters.
- In guest context, don't schedule in exclude_guest events, but allow !exclude_guest to be scheduled in.
As a current limitation, !exclude_guest events that rely on exclusive PMU facilities (e.g. Intel PT, LBR, or PEBS) are not supported under PMU partitioning and are rejected.
3. perf/x86
Apply PMU partitioning constraints to PMC allocation: while a partitioned guest is running, guest-owned counters are reserved for guest use and are unavailable to host event scheduling. Outside guest execution, all counters remain available to the host.
In PMI handling in guest mode, handle only the host-owned bits that are set in IA32_PERF_GLOBAL_STATUS, and leave the guest-owned bits for KVM.
4. PMI design
Unlike mediated vPMU, PMU partitioning does not switch LVTPC to a dedicated guest PMI vector. Host-owned counters remain active while the guest is running and may trigger PMIs.
PMIs therefore remain routed through the host NMI handler. PMI handling is split: host-induced counter overflows are handled by the host perf NMI handler as usual, while guest-induced overflows are left pending and KVM injects the corresponding PMI into the guest.
Related Work
============
ARM's PMU Partitioning series [2] solves a similar problem. It relies on MDCR_EL2.HPMN to enforce the host/guest counter boundary and lets the guest access its counters without trapping.
Similar to the perfmon_mask parameter, ARM's series has a module parameter, reserved_host_counters, that defines the maximum number of guest counters; userspace VMMs may then configure a smaller number of counters for each guest, up to that host-wide maximum.
Probably the biggest difference is that ARM's series chooses to context-switch guest/host PMU state only at vcpu_{load,put}(), and evict host events from would-be guest-owned counters if needed, while this series continues the mediated vPMU design and performs it on every VM-entry/VM-exit.
Also, while a guest is running, this series allows host events to be scheduled on exactly the complement of the effective guest counter mask, whereas ARM's series uses the single, global reserved_host_counters value, regardless of how many counters any given guest is actually using.
Tests
=====
Tested in the engineering environment with various perfmon_mask configurations.
1. Passed KVM selftests on L0 and L1.
2. No issues related to PMU partitioning were found by perf test running on L0 and L1.
3. Host exclude_guest events do not count non-root activity:
# The following host command reports similar counts with or without
# guest workloads. $ perf stat -e instructions:H,cycles:H,branches:H -a -I 1000
4. Host system-wide !exclude_guest events count non-root activity. # The following host command captures counts generated by guest
# workloads. $ perf stat -e instructions:HG,cycles:HG,branches:HG -a -I 1000
5. Put test 3 and 4 together.
# Guest idle.
$ stat -C 1 -e cycles:GH,instructions:G,branches:H -a Performance counter stats for 'system wide':
113,933,309 cycles:GH (100.00%)
135,454,451 instructions:G (100.00%)
26,966,970 branches:H1.959020017 seconds time elapsed
# Ran sysbench on the guest.
$ stat -C 1 -e cycles:GH,instructions:G,branches:H -a Performance counter stats for 'system wide':
5,012,158,432 cycles:GH (99.94%)
5,468,621,724 instructions:G (99.94%)
26,654,821 branches:H1.948625192 seconds time elapsed
6. Host system-wide !exclude_guest sampling events observe guest-induced PMIs.
# When running a guest workload with a known PMI rate, the following
# host command captures the corresponding overflow activity.
# Guest idle.
$ perf record -e instructions:H,cycles:HG,branches:HG -c 1000000 -a -- sleep 10
[ perf record: Captured and wrote 0.322 MB perf.data (1432 samples) ]
SAMPLE events: 1432 (30.4%)
SAMPLE events: 730
SAMPLE events: 549
SAMPLE events: 153# Independantly ran workload on the guest to know the expected PMIs.
$ perf record -e instructions,cycles,branches -c1000000 tchain_edit
[ perf record: Captured and wrote 0.176 MB perf.data (4536 samples) ]
SAMPLE events: 4536 (99.0%)
SAMPLE events: 2037
SAMPLE events: 1821
SAMPLE events: 678 # Profile this on host again while ran tchain_edit on the guest.
$ perf record -e instructions:H,cycles:HG,branches:HG -c 1000000 -a -- sleep 10
[ perf record: Captured and wrote 0.433 MB perf.data (3959 samples) ]
SAMPLE events: 3959 (55.0%)
SAMPLE events: 762
SAMPLE events: 2362
SAMPLE events: 8357. Perf Metrics tests.
# If perf_metrics is assigned to the guests, guests can run slots and
# metrics events. Host exclude_host slots and metrics events are not
# scheduled in non-root mode.
$ perf stat -e '{topdown-bad-spec,topdown-retiring,topdown-be-bound,slots}:G' -a --sleep 1
Performance counter stats for 'system wide': 0 slots (18.11%)
0 topdown-bad-spec (18.11%)
0 topdown-retiring (18.11%)
0 topdown-be-bound (18.11%)1.001590434 seconds time elapsed
# Same events run on the host if perf_metrics is not assigned to the
# guests.
$ perf stat -e '{topdown-bad-spec,topdown-retiring,topdown-be-bound,slots}:G' -a --sleep 1
Performance counter stats for 'system wide': 14,632,663 slots (100.00%)
918,127 topdown-bad-spec (100.00%)
1,319,808 topdown-retiring (100.00%)
3,514,335 topdown-be-bound (100.00%)1.001938038 seconds time elapsed
8. Some other sanity tests.
Patch Summary
=============
1. perf/x86 core enablement (patches 1-8)
Introduce guest-state tracking, host/guest PMI splitting, and the counter/event constraints the mask depends on.
2. perf core scheduling (patches 9-11)
Propagate the partition mask into mediated-PMU VMs and teach the generic perf scheduler to honor it when loading/putting guest context.
3. KVM PerfMon masking (patches 12-20)
perfmon_mask module parameter, PERFMON_MASK VMCS setup, MSR intercept relaxation, and GLOBAL_CTRL/GLOBAL_STATUS/GLOBAL_INUSE handling for the mediated vPMU under PerfMon masking.
4. Enablement (patch 21)
Turn on PerfMon masking on supported Intel platforms.
5. Selftests (patches 22-23)
Many thanks to Kan Liang for the original PerfMon masking design, Andi Kleen for his guidance, and Dapeng Mi for the many discussions, reviews, and suggestions that helped shape this series.
Rebased to kvm-x86/next: 1b731e5ded48
Thie series depends on the PMU v5 series [3] and Topdown metrics series [4].
Code also available at [5].
[1]: https://www.intel.com/content/www/us/en/content-details/922690/intel-architecture-instruction-set-extensions-programming-reference.html
[2]: https://lore.kernel.org/lkml/[email protected]/T/
[3]: https://lore.kernel.org/kvm/[email protected]/
[4]: https://lore.kernel.org/kvm/[email protected]/
[5]: https://github.com/ZideChen0/linux-kernel/tree/pmu_partitioning_v1
Kan Liang (3):
perf/x86: Restrict !exclude_guest events to host-owned counters
perf: Skip exclude_guest events on PMU partitioned counters
perf: Reschedule events across PMU partition transitions
Zide Chen (20):
perf/x86/intel: Guard counter masks against zero counters perf, perf/x86: Pass partition mask from KVM to perf/x86 perf/x86: Add GUEST_PMU states for PMU partitioning perf/x86: Split host/guest PMI handling under PMU partitioning perf/x86: Allow exclude_host events to run in non-root mode perf/x86: Apply PMU partition mask on static constraints perf/x86: Export available PMU counters to sysfs
perf, perf/x86: Allow host !exclude_guest events in PMU partitioning
KVM: x86/pmu: Add the perfmon_mask module parameter
KVM: x86/pmu: Set up the PERFMON_MASK VMCS field
KVM: x86/pmu, perf/x86: Update effective PMU partition mask
KVM: x86/pmu: Relax MSR intercept policy under PerfMon masking
KVM: x86/pmu: Handle FIXED_CTR_CTRL under PerfMon masking
KVM: x86/pmu: Handle GLOBAL_CTRL under PerfMon masking
KVM: x86/pmu: Handle GLOBAL_STATUS MSRs under PerfMon masking
KVM: x86/pmu: Always intercept GLOBAL_INUSE under PerfMon masking
KVM: x86/pmu: Request guest PMI for guest-induced PMIs
KVM: x86/pmu: Enable PerfMon masking
KVM: selftests: Fix PERF_METRICS test by checking FC3 availability
KVM: selftests: Allow no general purpose counters on the host
.../sysfs-bus-event_source-devices-caps | 5 + .../admin-guide/kernel-parameters.txt | 30 +++ arch/x86/events/core.c | 227 +++++++++++++++++- arch/x86/events/intel/core.c | 185 ++++++++++++-- arch/x86/events/perf_event.h | 20 ++ arch/x86/include/asm/kvm_host.h | 1 + arch/x86/include/asm/perf_event.h | 3 + arch/x86/include/asm/vmx.h | 3 + arch/x86/include/asm/vmxfeatures.h | 1 + arch/x86/kvm/pmu.c | 53 +++- arch/x86/kvm/pmu.h | 13 +- arch/x86/kvm/svm/pmu.c | 2 +- arch/x86/kvm/vmx/capabilities.h | 6 + arch/x86/kvm/vmx/pmu_intel.c | 206 +++++++++++++++- arch/x86/kvm/vmx/vmx.c | 86 ++++++- arch/x86/kvm/vmx/vmx.h | 3 +- arch/x86/kvm/x86.c | 6 +- include/linux/perf_event.h | 8 +- kernel/events/core.c | 122 ++++++++-- .../selftests/kvm/x86/pmu_counters_test.c | 29 ++- 20 files changed, 929 insertions(+), 80 deletions(-)
PATCH 1/23 perf/x86/intel: Guard counter masks against zero counters
When running as a guest with a vPMU exposed, the host may expose zero general-purpose counters or zero fixed counters. In that case, GENMASK_ULL() is called with a negative high-bit argument, resulting in an out-of-range shift and undefined behavior.
On systems with PMU partitioning enabled, this configuration is more likely. Change GENMASK_ULL() to BIT_ULL() to guard mask generation against zero counters.
--- arch/x86/events/intel/core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/events/intel/core.c +2 -2
@@ -7905,7 +7905,7 @@ __init int intel_pmu_init(void) x86_pmu = intel_pmu; x86_pmu.version = version; - x86_pmu.cntr_mask64 =GENMASK_ULL (eax.split.num_counters -1 ,0 ); + x86_pmu.cntr_mask64 =BIT_ULL (eax.split.num_counters) -1 ; x86_pmu.cntval_bits = eax.split.bit_width; x86_pmu.cntval_mask = (1ULL << eax.split.bit_width) -1 ;
@@ -7924,7 +7924,7 @@ __init int intel_pmu_init(void)int assume =3 * !boot_cpu_has (X86_FEATURE_HYPERVISOR ); x86_pmu.fixed_cntr_mask64 = -GENMASK_ULL (max ((int )edx.split.num_counters_fixed, assume) -1 ,0 ); +BIT_ULL (max ((int )edx.split.num_counters_fixed, assume)) -1 ; }else if (version >=5 ) x86_pmu.fixed_cntr_mask64 = fixed_mask;
Signature · 1 line
2.55.0
Re: [PATCH 01/23] perf/x86/intel: Guard counter masks against zero counters
Quoted 38 lines · Zide Chen
When running as a guest with a vPMU exposed, the host may expose zero general-purpose counters or zero fixed counters. In that case,
GENMASK_ULL()is called with a negative high-bit argument, resulting in an out-of-range shift and undefined behavior.On systems with PMU partitioning enabled, this configuration is more likely. Change
GENMASK_ULL()toBIT_ULL()to guard mask generation against zero counters.Fixes 722e42e45c2f ("perf/x86: Support counter mask")Signed-off-by Zide Chen <[email protected]>--- arch/x86/events/intel/core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)arch/x86/events/intel/core.c +2
-2@@ -7905,7 +7905,7 @@ __init int intel_pmu_init(void) x86_pmu = intel_pmu; x86_pmu.version = version; - x86_pmu.cntr_mask64 =GENMASK_ULL (eax.split.num_counters -1 ,0 ); + x86_pmu.cntr_mask64 =BIT_ULL (eax.split.num_counters) -1 ; x86_pmu.cntval_bits = eax.split.bit_width; x86_pmu.cntval_mask = (1ULL << eax.split.bit_width) -1 ;@@ -7924,7 +7924,7 @@ __init int intel_pmu_init(void)int assume =3 * !boot_cpu_has (X86_FEATURE_HYPERVISOR ); x86_pmu.fixed_cntr_mask64 = -GENMASK_ULL (max ((int )edx.split.num_counters_fixed, assume) -1 ,0 ); +BIT_ULL (max ((int )edx.split.num_counters_fixed, assume)) -1 ; }else if (version >=5 ) x86_pmu.fixed_cntr_mask64 = fixed_mask;
PATCH 2/23 perf, perf/x86: Pass partition mask from KVM to perf/x86
In reply to Zide Chenx86 PMU partitioning support requires mediated vPMU and is supported only on Intel platforms. KVM is responsible for determining if PMU partitioning can be enabled, and if so, it passes the mask to perf core when creating a mediated PMU VM. The mask is host-wide state shared by all mediated PMU guests.
Perf core passes the mask down to perf/x86 via the new callback arch_perf_set_pmu_partition_mask(), which sanitizes it before applying it to the newly added x86_pmu. The mask is cleared when the last mediated PMU VM is released.
Note: PMU partitioning technically allows the host to run certain !exclude_guest events while partitioned guests are running, as long as such events don't use exclusive resources reserved for the guests. The current behavior is kept: reject the creation of a partitioned guest as long as any !exclude_guest host events exist.
Introduce PERF_PMU_CAP_PMU_PARTITION to indicate that the pmu supports PMU partitioning. Note that virtualization of Intel PT and BTS is currently unsupported; even if it were, this flag would not apply to those PMUs, since their exclusive resources can't be shared between host and guest, and their events are not subject to PMU partitioning scheduling.
perf_create_mediated_pmu() is called with 0 for now; this will be replaced with the actual partition_mask in a later patch.
--- arch/x86/events/core.c | 58 ++++++++++++++++++++++++++++++++++++ arch/x86/events/perf_event.h | 9 ++++++ arch/x86/kvm/x86.c | 2 +- include/linux/perf_event.h | 4 ++- kernel/events/core.c | 19 +++++++++--- 5 files changed, 86 insertions(+), 6 deletions(-)
arch/x86/events/core.c +58 -0
@@ -2773,6 +2773,64 @@ static bool x86_pmu_filter(struct pmu *pmu, int cpu)return ret; } +/** +* arch_perf_set_pmu_partition_mask - Validate and set the VM-owned counter +* mask for mediated vPMU +* @partition_mask: Bitmask of PMU counters or other hardware resources +* to hand over to mediated vPMU guests. 0 disables PMU +* partitioning, as in the legacy model. +* +* Called via perf_create_mediated_pmu() to validate @partition_mask and, +* if valid, record it in x86_pmu.partition_mask for use by the +* scheduler, and set PERF_PMU_CAP_PMU_PARTITION on the generic PMU so +* perf core can check it without reaching into x86-private state. +* +* Return: 0 on success, -errno otherwise. +*/ +int arch_perf_set_pmu_partition_mask (u64 partition_mask) +{ +u64 current_mask =READ_ONCE (x86_pmu.partition_mask); +struct pmu *pmu; + +/* Non-paritioned mediated guests fall into this case. */ +if (current_mask == partition_mask) +return 0 ; + +/* +* AMD does not yet implement the hardware support for PMU partitioning +* between host and guest. Thus limit it to Intel platforms with the +* PerfMon masking VMX extension. Leave it to KVM to check the VMX +* feature. KVM doesn't support vPMU on Hybrid CPUs at all. +*/ +if (boot_cpu_data.x86_vendor !=X86_VENDOR_INTEL ||is_hybrid ()) +return -EOPNOTSUPP ; + + pmu =x86_get_pmu (raw_smp_processor_id ()); +if (!(pmu->capabilities &PERF_PMU_CAP_MEDIATED_VPMU )) +return -EOPNOTSUPP ; + +/* +* If a mediated-PMU VM was already created, the configured mask +* cannot be changed. +*/ +if (current_mask && partition_mask) +return -EINVAL ; + +/* +* perf/core guarantees that this path is reached only after all +* partitioned guests have been released. +*/ +if (!partition_mask) { +WRITE_ONCE (x86_pmu.partition_mask,0 ); + pmu->capabilities &= ~PERF_PMU_CAP_PMU_PARTITION ; +return 0 ; + } + +WRITE_ONCE (x86_pmu.partition_mask, partition_mask); + pmu->capabilities |=PERF_PMU_CAP_PMU_PARTITION ; +return 0 ; +} +static struct pmu pmu = { .pmu_enable = x86_pmu_enable, .pmu_disable = x86_pmu_disable,
arch/x86/events/perf_event.h +9 -0
@@ -883,6 +883,15 @@ struct x86_pmu {int events_mask_len;int apic;u64 max_period; + +/* +* Bitmask of PMU resources that may be assigned to a guest. +* +* The mask is set when the first mediated vPMU is created and is +* cleared when the last mediated vPMU is torn down. +*/ +u64 partition_mask; +struct event_constraint * (*get_event_constraints)(struct cpu_hw_events *cpuc, int idx,
arch/x86/kvm/x86.c +1 -1
@@ -9328,7 +9328,7 @@ int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id)if (enable_mediated_pmu && kvm->arch.enable_pmu && !kvm->arch.created_mediated_pmu) {if (irqchip_in_kernel (kvm)) { - r =perf_create_mediated_pmu (); + r =perf_create_mediated_pmu (0 );if (r) {pr_warn_ratelimited (PERF_MEDIATED_PMU_MSG );return r;
include/linux/perf_event.h +3 -1
@@ -306,6 +306,7 @@ struct perf_event_pmu_context;#define PERF_PMU_CAP_AUX_PAUSE 0x0200#define PERF_PMU_CAP_AUX_PREFER_LARGE 0x0400#define PERF_PMU_CAP_MEDIATED_VPMU 0x0800 +#define PERF_PMU_CAP_PMU_PARTITION 0x1000/** * pmu::scope
@@ -1931,10 +1932,11 @@ extern int perf_event_period(struct perf_event *event, u64 value);extern u64 perf_event_pause (struct perf_event *event,bool reset);#ifdef CONFIG_PERF_GUEST_MEDIATED_PMU -int perf_create_mediated_pmu (void ); +int perf_create_mediated_pmu (u64 pmu_partition_mask);void perf_release_mediated_pmu (void );void perf_load_guest_context (void );void perf_put_guest_context (void ); +int arch_perf_set_pmu_partition_mask (u64 pmu_partition_mask);#endif #else /* !CONFIG_PERF_EVENTS: */
kernel/events/core.c +15 -4
@@ -6340,6 +6340,11 @@ static atomic_t nr_include_guest_events __read_mostly;static atomic_t nr_mediated_pmu_vms__read_mostly ;static DEFINE_MUTEX (perf_mediated_pmu_mutex); +int __weak arch_perf_set_pmu_partition_mask (u64 partition_mask) +{ +return partition_mask ? -EOPNOTSUPP :0 ; +} +/* !exclude_guest event of PMU with PERF_PMU_CAP_MEDIATED_VPMU */ static inline bool is_include_guest_event (struct perf_event *event) {
@@ -6387,15 +6392,18 @@ static void mediated_pmu_unaccount_event(struct perf_event *event)* No impact for the PMU without PERF_PMU_CAP_MEDIATED_VPMU. The perf * still owns all the PMU resources. */ -int perf_create_mediated_pmu (void ) +int perf_create_mediated_pmu (u64 partition_mask) { -if (atomic_inc_not_zero (&nr_mediated_pmu_vms)) -return 0 ; +int ret;guard (mutex)(&perf_mediated_pmu_mutex);if (atomic_read (&nr_include_guest_events))return -EBUSY ; + ret =arch_perf_set_pmu_partition_mask (partition_mask); +if (ret) +return ret; +atomic_inc (&nr_mediated_pmu_vms);return 0 ; }
@@ -6403,10 +6411,13 @@ EXPORT_SYMBOL_FOR_KVM(perf_create_mediated_pmu);void perf_release_mediated_pmu (void ) { +guard (mutex)(&perf_mediated_pmu_mutex); +if (WARN_ON_ONCE (!atomic_read (&nr_mediated_pmu_vms)))return ; -atomic_dec (&nr_mediated_pmu_vms); +if (atomic_dec_and_test (&nr_mediated_pmu_vms)) +arch_perf_set_pmu_partition_mask (0 ); }EXPORT_SYMBOL_FOR_KVM (perf_release_mediated_pmu);
Signature · 1 line
2.55.0
PATCH 3/23 perf/x86: Add GUEST_PMU states for PMU partitioning
In reply to Zide ChenCurrently, KVM loads a guest's mediated PMU context in a strict order:perf_load_guest_context() runs first (it must execute in host context), then PERF_GLOBAL_CTRL is cleared (to avoid spurious PMIs), then perf_load_guest_lvtpc() switches the LVTPC hardware vector, and finally the guest's PMCs are loaded.
Under PMU partitioning, host-owned counters remain available to host events. When loading guest context, the perf scheduler must reschedule host events off guest-owned counters.
Add GUEST_PMU_PARTITION_PRELOAD to represent the intermediate state in which PMU partitioning constraints are needed for scheduling, but PMU partition PMI handling is not yet active. Enter this state through the new perf_pmu_partition_preload() helper.
Add GUEST_PMU_PARTITION_NMI for the state after the above _PRELOAD state, i.e. the host context transition is complete and guest context loading may proceed. LVTPC remains routed to NMI, and the PMU partition mask can be applied in the PMI handler.
Add GUEST_PMU_MEDIATED to indicate that the LVTPC is set to the fixed PERF_GUEST_MEDIATED_PMI_VECTOR, either in the PMU non-partitioned case, or the partition setup where no host events are scheduled and no need to share NMI with the host.
--- arch/x86/events/core.c | 94 ++++++++++++++++++++++++++++--- arch/x86/events/perf_event.h | 2 + arch/x86/include/asm/perf_event.h | 1 + arch/x86/kvm/pmu.c | 2 + 4 files changed, 91 insertions(+), 8 deletions(-)
arch/x86/events/core.c +86 -8
@@ -57,7 +57,38 @@ DEFINE_PER_CPU(struct cpu_hw_events, cpu_hw_events) = { .pmu = &pmu, }; -static DEFINE_PER_CPU(bool, guest_lvtpc_loaded); +/* +* GUEST_PMU_NONE - No guest mediated PMU context is loaded. +* +* GUEST_PMU_MEDIATED - LVTPC is routed to the dedicated mediated PMI vector +* instead of NMI, so any PMI in this state can only be guest-induced. +* This covers both the non-partitioned mediated vPMU model and the PMU +* partitioning case where the host currently has no events scheduled on +* this CPU (see perf_load_guest_lvtpc()). +* +* GUEST_PMU_PARTITION_PRELOAD - Entering PMU partitioning guest mode, the host +* context is still loaded. PMU partitioning constraints must be applied +* so that host events can be rescheduled onto host-owned counters. +* +* GUEST_PMU_PARTITION_NMI - PMU partitioning is enabled, host event +* rescheduling is complete, and the host currently has events scheduled +* on this CPU, so LVTPC is routed to NMI and shared between host- and +* guest-owned counters: a PMI in this state may be host- or +* guest-induced. +* +* _PARTITION_PRELOAD and _PARTITION_NMI are distinct because PMU +* partitioning constraints must be visible to the scheduler before host +* events are rescheduled, while PMU partition masking can be applied to PMI +* handling only after the rescheduling completes. +*/ +enum guest_pmu_mode { +GUEST_PMU_NONE , +GUEST_PMU_MEDIATED , +GUEST_PMU_PARTITION_PRELOAD , +GUEST_PMU_PARTITION_NMI , +}; + +static DEFINE_PER_CPU(enum guest_pmu_mode, guest_pmu_state);DEFINE_STATIC_KEY_FALSE (rdpmc_never_available_key);DEFINE_STATIC_KEY_FALSE (rdpmc_always_available_key);
@@ -1769,21 +1800,63 @@ void perf_events_lapic_init(void)apic_write (APIC_LVTPC ,APIC_DM_NMI ); } +bool pmu_partition_configured (void ) +{ +return READ_ONCE (x86_pmu.partition_mask) !=0 ; +} +#ifdef CONFIG_PERF_GUEST_MEDIATED_PMU +/* +* Mark this CPU as running a PMU partitioned guest. Guest PMU partition +* constraints apply from this point, even if host PMU context remains loaded. +*/ +void perf_pmu_partition_preload (void ) +{ +if (pmu_partition_configured ()) +this_cpu_write (guest_pmu_state,GUEST_PMU_PARTITION_PRELOAD ); +} +EXPORT_SYMBOL_FOR_KVM (perf_pmu_partition_preload); +void perf_load_guest_lvtpc (u32 guest_lvtpc) { -u32 masked = guest_lvtpc &APIC_LVT_MASKED ; +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); +bool is_pmu_partitioned =pmu_partition_configured (); +bool use_nmi; -apic_write (APIC_LVTPC , -APIC_DM_FIXED |PERF_GUEST_MEDIATED_PMI_VECTOR | masked); -this_cpu_write (guest_lvtpc_loaded, true); +if (is_pmu_partitioned) +WARN_ON_ONCE (this_cpu_read (guest_pmu_state) != +GUEST_PMU_PARTITION_PRELOAD ); + +/* +* If the host has events scheduled on this CPU, a PMI could be host- +* or guest-induced, so share NMI with the guest. Otherwise, route +* LVTPC to the dedicated mediated PMI vector for better +* performance and simpler handling. +*/ + use_nmi = is_pmu_partitioned && cpuc->n_events; +if (!use_nmi) +apic_write (APIC_LVTPC ,APIC_DM_FIXED | +PERF_GUEST_MEDIATED_PMI_VECTOR | + (guest_lvtpc &APIC_LVT_MASKED )); + +this_cpu_write (guest_pmu_state, + use_nmi ?GUEST_PMU_PARTITION_NMI :GUEST_PMU_MEDIATED ); }EXPORT_SYMBOL_FOR_KVM (perf_load_guest_lvtpc);void perf_put_guest_lvtpc (void ) { -this_cpu_write (guest_lvtpc_loaded, false); -apic_write (APIC_LVTPC ,APIC_DM_NMI ); +enum guest_pmu_mode state =this_cpu_read (guest_pmu_state); + +this_cpu_write (guest_pmu_state,GUEST_PMU_NONE ); + +/* +* LVTPC needs restoring to NMI unless it's already routed there, i.e. +* unless LVTPC was left routed to the dedicated mediated PMI vector +* (see perf_load_guest_lvtpc()). +*/ +if (state ==GUEST_PMU_MEDIATED ) +apic_write (APIC_LVTPC ,APIC_DM_NMI ); }EXPORT_SYMBOL_FOR_KVM (perf_put_guest_lvtpc);#endif /* CONFIG_PERF_GUEST_MEDIATED_PMU */
@@ -1802,8 +1875,13 @@ perf_event_nmi_handler(unsigned int cmd, struct pt_regs *regs)* loaded will generate false positives and clobber guest state. Note, * the LVTPC is switched to/from the dedicated mediated PMI IRQ vector * while host events are quiesced. +* +* GUEST_PMU_PARTITION_NMI is intentionally excluded here: LVTPC stays +* routed to NMI in that state, and an NMI there can be host- or +* guest-induced. GUEST_PMU_PARTITION_PRELOAD is likewise excluded, as +* PMU is still loaded with host context. */ -if (this_cpu_read (guest_lvtpc_loaded)) +if (this_cpu_read (guest_pmu_state) ==GUEST_PMU_MEDIATED )return NMI_DONE ;/*
arch/x86/events/perf_event.h +2 -0
@@ -1602,6 +1602,8 @@ static inline int is_pebs_pt(struct perf_event *event)return !!(event->hw.flags &PERF_X86_EVENT_PEBS_VIA_PT ); } +bool pmu_partition_configured (void ); +#ifdef CONFIG_CPU_SUP_INTEL static inline bool intel_pmu_has_bts_period (struct perf_event *event,u64 period)
arch/x86/include/asm/perf_event.h +1 -0
@@ -790,6 +790,7 @@ static inline void perf_check_microcode(void) { }#endif #ifdef CONFIG_PERF_GUEST_MEDIATED_PMU +extern void perf_pmu_partition_preload (void );extern void perf_load_guest_lvtpc (u32 guest_lvtpc);extern void perf_put_guest_lvtpc (void );#endif
arch/x86/kvm/pmu.c +2 -0
@@ -1388,6 +1388,8 @@ void kvm_mediated_pmu_load(struct kvm_vcpu *vcpu)lockdep_assert_irqs_disabled (); +perf_pmu_partition_preload (); +perf_load_guest_context ();/*
Signature · 1 line
2.55.0
PATCH 4/23 perf/x86: Split host/guest PMI handling under PMU partitioning
In reply to Zide ChenIf PMU partitioning is enabled, LVTPC could route to NMI if any host events are scheduled in this CPU. Thus, PMIs that fire in guest context can be host- or guest-induced.
The host NMI handler processes host-owned PMIs as usual. Guest-induced PMIs are marked as handled to avoid unknown NMI warnings, and their GLOBAL_STATUS bits deliberately remain untouched in hardware; it's KVM's responsibility to inject the corresponding PMIs into the guest.
Other places that write to global MSRs in NMI context also need to distinguish host-owned and guest-owned bits, so that they won't clobber guest-owned bits. The GLOBAL_CTRL is an exception: its guest value is expected to be saved by VMX on VM exit and restored by KVM on VM Entry.
Add x86_pmu_partition_nmi_active() to identify when PMU partitioning is active and LVTPC is routed to NMI, to help determine when the partition mask needs to be applied.
The effective mask may differ across guests. Add a per-cpu partition_mask field to struct cpu_hw_events, to be updated by KVM whenever the effective mask changes on the CPU. This could differ from the static x86_pmu.partition_mask, of which the effective mask is a subset.
--- arch/x86/events/core.c | 19 +++++++++++- arch/x86/events/intel/core.c | 58 ++++++++++++++++++++++++++++++++++-- arch/x86/events/perf_event.h | 8 +++++ 3 files changed, 82 insertions(+), 3 deletions(-)
arch/x86/events/core.c +18 -1
@@ -1805,6 +1805,19 @@ bool pmu_partition_configured(void)return READ_ONCE (x86_pmu.partition_mask) !=0 ; } +bool x86_pmu_partition_nmi_active (void ) +{ +enum guest_pmu_mode state =this_cpu_read (guest_pmu_state); + +return pmu_partition_configured () && + state ==GUEST_PMU_PARTITION_NMI ; +} + +u64 x86_pmu_current_partition_mask (void ) +{ +return this_cpu_ptr (&cpu_hw_events)->partition_mask; +} +#ifdef CONFIG_PERF_GUEST_MEDIATED_PMU /* * Mark this CPU as running a PMU partitioned guest. Guest PMU partition
@@ -1887,8 +1900,12 @@ perf_event_nmi_handler(unsigned int cmd, struct pt_regs *regs)/* * All PMUs/events that share this PMI handler should make sure to * increment active_events for their events. +* +* If PMU partitioning is enabled, guest-induced PMIs need to be marked +* as handled to avoid unknown NMI warnings. */ -if (!atomic_read (&active_events)) +if (!atomic_read (&active_events) && + !x86_pmu_partition_nmi_active ())return NMI_DONE ; start_clock =sched_clock ();
arch/x86/events/intel/core.c +56 -2
124 lines not on this page · raw message
arch/x86/events/perf_event.h +8 -0
@@ -300,6 +300,12 @@ struct cpu_hw_events {unsigned int txn_flags;int is_fake; +/* +* The PMU resources owned by the vCPU currently scheduled on this +* CPU, which is a subset of x86_pmu.partition_mask. +*/ +u64 partition_mask; +/* * Intel DebugStore bits */
@@ -1603,6 +1609,8 @@ static inline int is_pebs_pt(struct perf_event *event) }bool pmu_partition_configured (void ); +bool x86_pmu_partition_nmi_active (void ); +u64 x86_pmu_current_partition_mask (void );#ifdef CONFIG_CPU_SUP_INTEL
Signature · 1 line
2.55.0
PATCH 5/23 perf/x86: Allow exclude_host events to run in non-root mode
In reply to Zide ChenOn Intel, exclude_host events are prevented from running by masking the Global Ctrl bits with ~cpuc->intel_ctrl_guest_mask.
When PMU partitioning is enabled, the host is allowed to run exclude_host events in non-root mode, while such events still need to be masked in root mode, since there are no other checkpoints to stop them from being scheduled in.
Add x86_pmu_partition_loaded() to distinguish scheduling constraints from x86_pmu_partition_nmi_active(), as PMU partitioning constraints apply in both the _PARTITION_PRELOAD and _PARTITION_NMI states.
--- arch/x86/events/core.c | 11 +++++++++++ arch/x86/events/intel/core.c | 6 ++++-- arch/x86/events/perf_event.h | 1 + 3 files changed, 16 insertions(+), 2 deletions(-)
arch/x86/events/core.c +11 -0
@@ -1813,6 +1813,17 @@ bool x86_pmu_partition_nmi_active(void) state ==GUEST_PMU_PARTITION_NMI ; } +/* +* Skip for "fake" cpuc used during event validation, where +* per-CPU state like guest_pmu_state is meaningless. +*/ +bool x86_pmu_partition_loaded (struct cpu_hw_events *cpuc) +{ +return !cpuc->is_fake && +pmu_partition_configured () && +this_cpu_read (guest_pmu_state) !=GUEST_PMU_NONE ; +} +u64 x86_pmu_current_partition_mask (void ) {return this_cpu_ptr (&cpu_hw_events)->partition_mask;
arch/x86/events/intel/core.c +4 -2
@@ -2812,8 +2812,10 @@ static void __intel_pmu_enable_all(int added, bool pmi) cpuc->active_fixed_ctrl_val = cpuc->fixed_ctrl_val; } -wrmsrq (MSR_CORE_PERF_GLOBAL_CTRL , - intel_ctrl & ~cpuc->intel_ctrl_guest_mask); +if (!x86_pmu_partition_loaded (cpuc)) + intel_ctrl &= ~cpuc->intel_ctrl_guest_mask; + +wrmsrq (MSR_CORE_PERF_GLOBAL_CTRL , intel_ctrl);if (test_bit (INTEL_PMC_IDX_FIXED_BTS , cpuc->active_mask)) { struct perf_event *event =
arch/x86/events/perf_event.h +1 -0
@@ -1610,6 +1610,7 @@ static inline int is_pebs_pt(struct perf_event *event)bool pmu_partition_configured (void );bool x86_pmu_partition_nmi_active (void ); +bool x86_pmu_partition_loaded (struct cpu_hw_events *cpuc);u64 x86_pmu_current_partition_mask (void );#ifdef CONFIG_CPU_SUP_INTEL
Signature · 1 line
2.55.0
PATCH 6/23 perf/x86: Restrict !exclude_guest events to host-owned counters
In reply to Zide ChenFrom: Kan Liang <[email protected]>
When PMU partitioning is enabled and a guest is running, !exclude_guest events must be scheduled only on host-owned counters.
After guest exit, PMU partition constraints are no longer applied and host events may use the full counter set.
--- arch/x86/events/intel/core.c | 45 +++++++++++++++++++++++++++++++++--- 1 file changed, 42 insertions(+), 3 deletions(-)
arch/x86/events/intel/core.c +42 -3
@@ -2814,6 +2814,13 @@ static void __intel_pmu_enable_all(int added, bool pmi)if (!x86_pmu_partition_loaded (cpuc)) intel_ctrl &= ~cpuc->intel_ctrl_guest_mask; +else if (pmi) +/* +* Guest-owned bits are excluded here, not because of clobbering +* global ctrl guest bits in non-root mode, but because guest +* counter MSRs have not yet been saved by KVM in NMI context. +*/ + intel_ctrl &= ~x86_pmu_current_partition_mask ();wrmsrq (MSR_CORE_PERF_GLOBAL_CTRL , intel_ctrl);
@@ -4448,6 +4455,31 @@ dyn_constraint(struct cpu_hw_events *cpuc, struct event_constraint *c, int idx)return c; } +/* +* Mask out guest-owned counters from a constraint when PMU partition has been +* entered, so !exclude_guest host events are not scheduled onto them while +* the CPU is in non-root mode. +*/ +static struct event_constraint * +part_constraint (struct cpu_hw_events *cpuc,int idx, +struct perf_event *event,struct event_constraint *c) +{ +/* +* Skip &emptyconstraint: dyn_constraint() would clone it, breaking +* the "c == &emptyconstraint" pointer checks callers rely on. +*/ +if (!c->weight || c == &emptyconstraint) +return c; + +if (x86_pmu_partition_loaded (cpuc)) { + c =dyn_constraint (cpuc, c, idx); + c->idxmsk64 &= ~x86_pmu_current_partition_mask (); + c->weight =hweight64 (c->idxmsk64); + } + +return c; +} +static struct event_constraint *intel_get_excl_constraints (struct cpu_hw_events *cpuc,struct perf_event *event,int idx,struct event_constraint *c)
@@ -4569,8 +4601,14 @@ intel_get_event_constraints(struct cpu_hw_events *cpuc, int idx, c2 = c1; } -if (cpuc->excl_cntrs) +/* +* No platform supports both PMU_FL_EXCL_CNTRS and PerfMon masking +* simultaneously, so part_constraint() is not needed on this path. +*/ +if (cpuc->excl_cntrs) { +WARN_ON_ONCE (x86_pmu_partition_loaded (cpuc));return intel_get_excl_constraints (cpuc, event, idx, c2); + }if (event->hw.dyn_constraint != ~0ULL ) { c2 =dyn_constraint (cpuc, c2, idx);
@@ -4578,7 +4616,7 @@ intel_get_event_constraints(struct cpu_hw_events *cpuc, int idx, c2->weight =hweight64 (c2->idxmsk64); } -return c2; +return part_constraint (cpuc, idx, event, c2); } static void intel_put_excl_constraints(struct cpu_hw_events *cpuc,
@@ -5948,7 +5986,8 @@ int intel_cpuc_prepare(struct cpu_hw_events *cpuc, int cpu) goto err; } -if (x86_pmu.flags & (PMU_FL_EXCL_CNTRS |PMU_FL_TFA |PMU_FL_DYN_CONSTRAINT )) { +if (x86_pmu.flags & (PMU_FL_EXCL_CNTRS |PMU_FL_TFA |PMU_FL_DYN_CONSTRAINT ) || + (x86_get_pmu (cpu)->capabilities &PERF_PMU_CAP_MEDIATED_VPMU )) {size_t sz =X86_PMC_IDX_MAX *sizeof (struct event_constraint ); cpuc->constraint_list =kzalloc_node (sz,GFP_KERNEL ,cpu_to_node (cpu));
Signature · 1 line
2.55.0
PATCH 7/23 perf/x86: Apply PMU partition mask on static constraints
In reply to Zide ChenStatic, counter-specific constraints used for attr.precise_ip == 3, or other special cases bypass intel_get_event_constraints(), which is where partition_mask is applied via dyn_constraint().
As a result, such a host !exclude_guest event can still be scheduled onto a counter that partition_mask reserves for the guest, causing host and guest to share the same hardware counter.
Some of these static constraint paths, e.g. glp or cmt, are currently only reachable on platforms that don't support PerfMon masking. Still, apply part_constraint() to all of them uniformly, so future platforms that combine PerfMon masking with these constraint paths are not silently exposed to this bug.
--- arch/x86/events/intel/core.c | 39 +++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 14 deletions(-)
arch/x86/events/intel/core.c +25 -14
@@ -4459,6 +4459,9 @@ dyn_constraint(struct cpu_hw_events *cpuc, struct event_constraint *c, int idx)* Mask out guest-owned counters from a constraint when PMU partition has been * entered, so !exclude_guest host events are not scheduled onto them while * the CPU is in non-root mode. +* +* This is also used by PMU-specific get_event_constraints() wrappers +* that hard-code a static, counter-specific constraint. */ static struct event_constraint * part_constraint(struct cpu_hw_events *cpuc, int idx,
@@ -5568,7 +5571,7 @@ hsw_get_event_constraints(struct cpu_hw_events *cpuc, int idx,/* Handle special quirk on in_tx_checkpointed only in counter 2 */ if (event->hw.config &HSW_IN_TX_CHECKPOINTED ) {if (c->idxmsk64 & (1U <<2 )) -return &counter2_constraint; +return part_constraint (cpuc, idx, event, &counter2_constraint);return &emptyconstraint; }
@@ -5585,7 +5588,7 @@ icl_get_event_constraints(struct cpu_hw_events *cpuc, int idx,*/ if ((event->attr.precise_ip ==3 ) &&constraint_match (&fixed0_constraint, event->hw.config)) -return &fixed0_constraint; +return part_constraint (cpuc, idx, event, &fixed0_constraint);return hsw_get_event_constraints (cpuc, idx, event); }
@@ -5607,7 +5610,7 @@ glc_get_event_constraints(struct cpu_hw_events *cpuc, int idx,if ((event->attr.precise_ip ==3 ) && !constraint_match (&fixed0_constraint, event->hw.config)) {if (c->idxmsk64 &BIT_ULL (0 )) -return &counter0_constraint; +return part_constraint (cpuc, idx, event, &counter0_constraint);return &emptyconstraint; }
@@ -5623,7 +5626,7 @@ glp_get_event_constraints(struct cpu_hw_events *cpuc, int idx,/* :ppp means to do reduced skid PEBS which is PMC0 only. */ if (event->attr.precise_ip ==3 ) -return &counter0_constraint; +return part_constraint (cpuc, idx, event, &counter0_constraint); c =intel_get_event_constraints (cpuc, idx, event);
@@ -5645,9 +5648,9 @@ tnt_get_event_constraints(struct cpu_hw_events *cpuc, int idx,if (event->attr.precise_ip ==3 ) {/* Force instruction:ppp on PMC0 and Fixed counter 0 */ if (constraint_match (&fixed0_constraint, event->hw.config)) -return &fixed0_counter0_constraint; +return part_constraint (cpuc, idx, event, &fixed0_counter0_constraint); -return &counter0_constraint; +return part_constraint (cpuc, idx, event, &counter0_constraint); }return c;
@@ -5705,22 +5708,30 @@ cmt_get_event_constraints(struct cpu_hw_events *cpuc, int idx,if (event->attr.precise_ip ==3 ) {/* Force instruction:ppp on PMC0, 1 and Fixed counter 0 */ if (constraint_match (&fixed0_constraint, event->hw.config)) { + c = &fixed0_counter0_1_constraint; +/* The fixed counter 0 doesn't support LBR event logging. */ if (branch_sample_counters (event)) -return &counter0_1_constraint; -else -return &fixed0_counter0_1_constraint; + c = &counter0_1_constraint; + +return part_constraint (cpuc, idx, event, c); }switch (c->idxmsk64 &0x3ull ) {case 0x1 : -return &counter0_constraint; + c = &counter0_constraint; +break ;case 0x2 : -return &counter1_constraint; + c = &counter1_constraint; +break ;case 0x3 : -return &counter0_1_constraint; + c = &counter0_1_constraint; +break ; +default : + c = &emptyconstraint; +break ; } -return &emptyconstraint; +return part_constraint (cpuc, idx, event, c); }return c;
@@ -5744,7 +5755,7 @@ rwc_get_event_constraints(struct cpu_hw_events *cpuc, int idx,*/ if (event->attr.precise_ip ==3 )return &emptyconstraint; -return &counters_1_7_constraint; +return part_constraint (cpuc, idx, event, &counters_1_7_constraint); }return c;
Signature · 1 line
2.55.0
Re: [PATCH 07/23] perf/x86: Apply PMU partition mask on static constraints
It seems not a good practice for me to modify all the places calling these static constraints. We could have to add new get_event_constraints() helpers for future platforms, and we have to duplicate these part_constraint() code.
Could we leverage the dynamic constraint mechanism? We can set the event->hw.dyn_constraint to be host counters mask, then we don't need any specific change for these static constraints.
Thanks.
Quoted 128 lines · Zide Chen
Static, counter-specific constraints used for attr.precise_ip == 3, or other special cases bypass
intel_get_event_constraints(), which is wherepartition_maskis applied viadyn_constraint().As a result, such a host !exclude_guest event can still be scheduled onto a counter that
partition_maskreserves for the guest, causing host and guest to share the same hardware counter.Some of these static constraint paths, e.g. glp or cmt, are currently only reachable on platforms that don't support PerfMon masking. Still, apply
part_constraint()to all of them uniformly, so future platforms that combine PerfMon masking with these constraint paths are not silently exposed to this bug.Signed-off-by Zide Chen <[email protected]>--- arch/x86/events/intel/core.c | 39 +++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 14 deletions(-)arch/x86/events/intel/core.c +25
-14@@ -4459,6 +4459,9 @@ dyn_constraint(struct cpu_hw_events *cpuc, struct event_constraint *c, int idx)* Mask out guest-owned counters from a constraint when PMU partition has been * entered, so !exclude_guest host events are not scheduled onto them while * the CPU is in non-root mode. +* +* This is also used by PMU-specific get_event_constraints() wrappers +* that hard-code a static, counter-specific constraint. */ static struct event_constraint * part_constraint(struct cpu_hw_events *cpuc, int idx,@@ -5568,7 +5571,7 @@ hsw_get_event_constraints(struct cpu_hw_events *cpuc, int idx,/* Handle special quirk on in_tx_checkpointed only in counter 2 */ if (event->hw.config &HSW_IN_TX_CHECKPOINTED ) {if (c->idxmsk64 & (1U <<2 )) -return &counter2_constraint; +return part_constraint (cpuc, idx, event, &counter2_constraint);return &emptyconstraint; }@@ -5585,7 +5588,7 @@ icl_get_event_constraints(struct cpu_hw_events *cpuc, int idx,*/ if ((event->attr.precise_ip ==3 ) &&constraint_match (&fixed0_constraint, event->hw.config)) -return &fixed0_constraint; +return part_constraint (cpuc, idx, event, &fixed0_constraint);return hsw_get_event_constraints (cpuc, idx, event); }@@ -5607,7 +5610,7 @@ glc_get_event_constraints(struct cpu_hw_events *cpuc, int idx,if ((event->attr.precise_ip ==3 ) && !constraint_match (&fixed0_constraint, event->hw.config)) {if (c->idxmsk64 &BIT_ULL (0 )) -return &counter0_constraint; +return part_constraint (cpuc, idx, event, &counter0_constraint);return &emptyconstraint; }@@ -5623,7 +5626,7 @@ glp_get_event_constraints(struct cpu_hw_events *cpuc, int idx,/* :ppp means to do reduced skid PEBS which is PMC0 only. */ if (event->attr.precise_ip ==3 ) -return &counter0_constraint; +return part_constraint (cpuc, idx, event, &counter0_constraint); c =intel_get_event_constraints (cpuc, idx, event);@@ -5645,9 +5648,9 @@ tnt_get_event_constraints(struct cpu_hw_events *cpuc, int idx,if (event->attr.precise_ip ==3 ) {/* Force instruction:ppp on PMC0 and Fixed counter 0 */ if (constraint_match (&fixed0_constraint, event->hw.config)) -return &fixed0_counter0_constraint; +return part_constraint (cpuc, idx, event, &fixed0_counter0_constraint); -return &counter0_constraint; +return part_constraint (cpuc, idx, event, &counter0_constraint); }return c;@@ -5705,22 +5708,30 @@ cmt_get_event_constraints(struct cpu_hw_events *cpuc, int idx,if (event->attr.precise_ip ==3 ) {/* Force instruction:ppp on PMC0, 1 and Fixed counter 0 */ if (constraint_match (&fixed0_constraint, event->hw.config)) { + c = &fixed0_counter0_1_constraint; +/* The fixed counter 0 doesn't support LBR event logging. */ if (branch_sample_counters (event)) -return &counter0_1_constraint; -else -return &fixed0_counter0_1_constraint; + c = &counter0_1_constraint; + +return part_constraint (cpuc, idx, event, c); }switch (c->idxmsk64 &0x3ull ) {case 0x1 : -return &counter0_constraint; + c = &counter0_constraint; +break ;case 0x2 : -return &counter1_constraint; + c = &counter1_constraint; +break ;case 0x3 : -return &counter0_1_constraint; + c = &counter0_1_constraint; +break ; +default : + c = &emptyconstraint; +break ; } -return &emptyconstraint; +return part_constraint (cpuc, idx, event, c); }return c;@@ -5744,7 +5755,7 @@ rwc_get_event_constraints(struct cpu_hw_events *cpuc, int idx,*/ if (event->attr.precise_ip ==3 )return &emptyconstraint; -return &counters_1_7_constraint; +return part_constraint (cpuc, idx, event, &counters_1_7_constraint); }return c;
Re: [PATCH 07/23] perf/x86: Apply PMU partition mask on static constraints
Quoted 7 lines · Dapeng
It seems not a good practice for me to modify all the places calling these static constraints. We could have to add new
get_event_constraints()helpers for future platforms, and we have to duplicate thesepart_constraint()code.
Could we leverage the dynamic constraint mechanism? We can set theevent->hw.dyn_constraintto be host counters mask, then we don't need any specific change for these static constraints.
In all code paths that use these static constraints, for example, counter0_constraint, it is assumed that GP counter 0 is available on the host. However, with PMU partitioning, any counter may be unavailable, even in the model-specific code path, and this assumption is broken.
So, adding a wrapper seems unavoidable.
Additionally, part_constraint() already makes use of dyn_constraint, right?
Quoted 131 lines
Thanks.
Quoted 128 lines · Zide Chen
Static, counter-specific constraints used for attr.precise_ip == 3, or other special cases bypass
intel_get_event_constraints(), which is wherepartition_maskis applied viadyn_constraint().As a result, such a host !exclude_guest event can still be scheduled onto a counter that
partition_maskreserves for the guest, causing host and guest to share the same hardware counter.Some of these static constraint paths, e.g. glp or cmt, are currently only reachable on platforms that don't support PerfMon masking. Still, apply
part_constraint()to all of them uniformly, so future platforms that combine PerfMon masking with these constraint paths are not silently exposed to this bug.Signed-off-by Zide Chen <[email protected]>--- arch/x86/events/intel/core.c | 39 +++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 14 deletions(-)arch/x86/events/intel/core.c +25
-14@@ -4459,6 +4459,9 @@ dyn_constraint(struct cpu_hw_events *cpuc, struct event_constraint *c, int idx)* Mask out guest-owned counters from a constraint when PMU partition has been * entered, so !exclude_guest host events are not scheduled onto them while * the CPU is in non-root mode. +* +* This is also used by PMU-specific get_event_constraints() wrappers +* that hard-code a static, counter-specific constraint. */ static struct event_constraint * part_constraint(struct cpu_hw_events *cpuc, int idx,@@ -5568,7 +5571,7 @@ hsw_get_event_constraints(struct cpu_hw_events *cpuc, int idx,/* Handle special quirk on in_tx_checkpointed only in counter 2 */ if (event->hw.config &HSW_IN_TX_CHECKPOINTED ) {if (c->idxmsk64 & (1U <<2 )) -return &counter2_constraint; +return part_constraint (cpuc, idx, event, &counter2_constraint);return &emptyconstraint; }@@ -5585,7 +5588,7 @@ icl_get_event_constraints(struct cpu_hw_events *cpuc, int idx,*/ if ((event->attr.precise_ip ==3 ) &&constraint_match (&fixed0_constraint, event->hw.config)) -return &fixed0_constraint; +return part_constraint (cpuc, idx, event, &fixed0_constraint);return hsw_get_event_constraints (cpuc, idx, event); }@@ -5607,7 +5610,7 @@ glc_get_event_constraints(struct cpu_hw_events *cpuc, int idx,if ((event->attr.precise_ip ==3 ) && !constraint_match (&fixed0_constraint, event->hw.config)) {if (c->idxmsk64 &BIT_ULL (0 )) -return &counter0_constraint; +return part_constraint (cpuc, idx, event, &counter0_constraint);return &emptyconstraint; }@@ -5623,7 +5626,7 @@ glp_get_event_constraints(struct cpu_hw_events *cpuc, int idx,/* :ppp means to do reduced skid PEBS which is PMC0 only. */ if (event->attr.precise_ip ==3 ) -return &counter0_constraint; +return part_constraint (cpuc, idx, event, &counter0_constraint); c =intel_get_event_constraints (cpuc, idx, event);@@ -5645,9 +5648,9 @@ tnt_get_event_constraints(struct cpu_hw_events *cpuc, int idx,if (event->attr.precise_ip ==3 ) {/* Force instruction:ppp on PMC0 and Fixed counter 0 */ if (constraint_match (&fixed0_constraint, event->hw.config)) -return &fixed0_counter0_constraint; +return part_constraint (cpuc, idx, event, &fixed0_counter0_constraint); -return &counter0_constraint; +return part_constraint (cpuc, idx, event, &counter0_constraint); }return c;@@ -5705,22 +5708,30 @@ cmt_get_event_constraints(struct cpu_hw_events *cpuc, int idx,if (event->attr.precise_ip ==3 ) {/* Force instruction:ppp on PMC0, 1 and Fixed counter 0 */ if (constraint_match (&fixed0_constraint, event->hw.config)) { + c = &fixed0_counter0_1_constraint; +/* The fixed counter 0 doesn't support LBR event logging. */ if (branch_sample_counters (event)) -return &counter0_1_constraint; -else -return &fixed0_counter0_1_constraint; + c = &counter0_1_constraint; + +return part_constraint (cpuc, idx, event, c); }switch (c->idxmsk64 &0x3ull ) {case 0x1 : -return &counter0_constraint; + c = &counter0_constraint; +break ;case 0x2 : -return &counter1_constraint; + c = &counter1_constraint; +break ;case 0x3 : -return &counter0_1_constraint; + c = &counter0_1_constraint; +break ; +default : + c = &emptyconstraint; +break ; } -return &emptyconstraint; +return part_constraint (cpuc, idx, event, c); }return c;@@ -5744,7 +5755,7 @@ rwc_get_event_constraints(struct cpu_hw_events *cpuc, int idx,*/ if (event->attr.precise_ip ==3 )return &emptyconstraint; -return &counters_1_7_constraint; +return part_constraint (cpuc, idx, event, &counters_1_7_constraint); }return c;
Re: [PATCH 07/23] perf/x86: Apply PMU partition mask on static constraints
Quoted 9 lines · Zide
Quoted 7 lines · Dapeng
It seems not a good practice for me to modify all the places calling these static constraints. We could have to add new
get_event_constraints()helpers for future platforms, and we have to duplicate thesepart_constraint()code.
Could we leverage the dynamic constraint mechanism? We can set theevent->hw.dyn_constraintto be host counters mask, then we don't need any specific change for these static constraints.
Quoted 8 lines
In all code paths that use these static constraints, for example,
counter0_constraint, it is assumed that GP counter 0 is available on the host. However, with PMU partitioning, any counter may be unavailable, even in the model-specific code path, and this assumption is broken.So, adding a wrapper seems unavoidable.
Additionally,
part_constraint()already makes use ofdyn_constraint, right?
Could we add a wrapper helper like below?
arch/x86/events/core.c +0 -0
@@ -1033,6 +1033,17 @@ int perf_assign_events(struct event_constraint
**constraints, int n,
}
EXPORT_SYMBOL_GPL(perf_assign_events);
+static struct event_constraint * +x86_get_event_constraints(struct cpu_hw_events *cpuc, int idx,
+ struct perf_event *event)
+{
+ struct event_constraint *c;
+
+ c = static_call(x86_pmu_get_event_constraints)(cpuc, idx, event); +
+ return part_constraint(cpuc, idx, event, c);
+}
+
int x86_schedule_events(struct cpu_hw_events *cpuc, int n, int *assign) {
struct event_constraint *c;
@@ -1069,7 +1080,7 @@ int x86_schedule_events(struct cpu_hw_events *cpuc, int n, int *assign)
* change due to external factors (sibling state, allow_tfa). */
if (!c || (c->flags & PERF_X86_EVENT_DYNAMIC)) { - c = static_call(x86_pmu_get_event_constraints)(cpuc, i, cpuc->event_list[i]); + c = x86_get_event_constraints(cpuc, i, cpuc->event_list[i]);
cpuc->event_constraint[i] = c;
}
Quoted 132 lines
Quoted 132 lines
Thanks.
Quoted 128 lines · Zide Chen
Static, counter-specific constraints used for attr.precise_ip == 3, or other special cases bypass
intel_get_event_constraints(), which is wherepartition_maskis applied viadyn_constraint().As a result, such a host !exclude_guest event can still be scheduled onto a counter that
partition_maskreserves for the guest, causing host and guest to share the same hardware counter.Some of these static constraint paths, e.g. glp or cmt, are currently only reachable on platforms that don't support PerfMon masking. Still, apply
part_constraint()to all of them uniformly, so future platforms that combine PerfMon masking with these constraint paths are not silently exposed to this bug.Signed-off-by Zide Chen <[email protected]>--- arch/x86/events/intel/core.c | 39 +++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 14 deletions(-)arch/x86/events/intel/core.c +25
-14@@ -4459,6 +4459,9 @@ dyn_constraint(struct cpu_hw_events *cpuc, struct event_constraint *c, int idx)* Mask out guest-owned counters from a constraint when PMU partition has been * entered, so !exclude_guest host events are not scheduled onto them while * the CPU is in non-root mode. +* +* This is also used by PMU-specific get_event_constraints() wrappers +* that hard-code a static, counter-specific constraint. */ static struct event_constraint * part_constraint(struct cpu_hw_events *cpuc, int idx,@@ -5568,7 +5571,7 @@ hsw_get_event_constraints(struct cpu_hw_events *cpuc, int idx,/* Handle special quirk on in_tx_checkpointed only in counter 2 */ if (event->hw.config &HSW_IN_TX_CHECKPOINTED ) {if (c->idxmsk64 & (1U <<2 )) -return &counter2_constraint; +return part_constraint (cpuc, idx, event, &counter2_constraint);return &emptyconstraint; }@@ -5585,7 +5588,7 @@ icl_get_event_constraints(struct cpu_hw_events *cpuc, int idx,*/ if ((event->attr.precise_ip ==3 ) &&constraint_match (&fixed0_constraint, event->hw.config)) -return &fixed0_constraint; +return part_constraint (cpuc, idx, event, &fixed0_constraint);return hsw_get_event_constraints (cpuc, idx, event); }@@ -5607,7 +5610,7 @@ glc_get_event_constraints(struct cpu_hw_events *cpuc, int idx,if ((event->attr.precise_ip ==3 ) && !constraint_match (&fixed0_constraint, event->hw.config)) {if (c->idxmsk64 &BIT_ULL (0 )) -return &counter0_constraint; +return part_constraint (cpuc, idx, event, &counter0_constraint);return &emptyconstraint; }@@ -5623,7 +5626,7 @@ glp_get_event_constraints(struct cpu_hw_events *cpuc, int idx,/* :ppp means to do reduced skid PEBS which is PMC0 only. */ if (event->attr.precise_ip ==3 ) -return &counter0_constraint; +return part_constraint (cpuc, idx, event, &counter0_constraint); c =intel_get_event_constraints (cpuc, idx, event);@@ -5645,9 +5648,9 @@ tnt_get_event_constraints(struct cpu_hw_events *cpuc, int idx,if (event->attr.precise_ip ==3 ) {/* Force instruction:ppp on PMC0 and Fixed counter 0 */ if (constraint_match (&fixed0_constraint, event->hw.config)) -return &fixed0_counter0_constraint; +return part_constraint (cpuc, idx, event, &fixed0_counter0_constraint); -return &counter0_constraint; +return part_constraint (cpuc, idx, event, &counter0_constraint); }return c;@@ -5705,22 +5708,30 @@ cmt_get_event_constraints(struct cpu_hw_events *cpuc, int idx,if (event->attr.precise_ip ==3 ) {/* Force instruction:ppp on PMC0, 1 and Fixed counter 0 */ if (constraint_match (&fixed0_constraint, event->hw.config)) { + c = &fixed0_counter0_1_constraint; +/* The fixed counter 0 doesn't support LBR event logging. */ if (branch_sample_counters (event)) -return &counter0_1_constraint; -else -return &fixed0_counter0_1_constraint; + c = &counter0_1_constraint; + +return part_constraint (cpuc, idx, event, c); }switch (c->idxmsk64 &0x3ull ) {case 0x1 : -return &counter0_constraint; + c = &counter0_constraint; +break ;case 0x2 : -return &counter1_constraint; + c = &counter1_constraint; +break ;case 0x3 : -return &counter0_1_constraint; + c = &counter0_1_constraint; +break ; +default : + c = &emptyconstraint; +break ; } -return &emptyconstraint; +return part_constraint (cpuc, idx, event, c); }return c;@@ -5744,7 +5755,7 @@ rwc_get_event_constraints(struct cpu_hw_events *cpuc, int idx,*/ if (event->attr.precise_ip ==3 )return &emptyconstraint; -return &counters_1_7_constraint; +return part_constraint (cpuc, idx, event, &counters_1_7_constraint); }return c;
PATCH 8/23 perf/x86: Export available PMU counters to sysfs
In reply to Zide ChenWith PMU partitioning support, the set of PMU counters available to a PMU can vary from the architectural maximum. While this information is available through CPUID, it is not easily consumable from scripts.
Add two new sysfs ABI files, gp_counters and fixed_counters, under the PMU caps group to export the available counters. The fixed_counters file is omitted when the PMU does not support fixed-function counters.
$ grep . /sys/devices/cpu/caps/*_counters /sys/devices/cpu/caps/fixed_counters:0-3 /sys/devices/cpu/caps/gp_counters:0-7
--- .../sysfs-bus-event_source-devices-caps | 5 +++ arch/x86/events/core.c | 37 +++++++++++++++++++ 2 files changed, 42 insertions(+)
Documentation/ABI/testing/sysfs-bus-event_source-devices-caps +5 -0
@@ -22,3 +22,8 @@ Description: of PERF_SAMPLE_BRANCH_COUNTERS, while the "branch_counter_width" exposes the width of each counter. Both of them can be used by the perf tool to parse the logged counters in each branch. + + The "gp_counters" and "fixed_counters" attributes expose the + available general-purpose and fixed-function PMU counters as + a range list, e.g. "0-7". The "fixed_counters" attribute is + omitted if the PMU does not support fixed-function counters.
arch/x86/events/core.c +37 -0
@@ -2814,14 +2814,51 @@ static ssize_t max_precise_show(struct device *cdev,static DEVICE_ATTR_RO (max_precise); +static ssize_t gp_counters_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct pmu *pmu =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%*pbl\n" ,INTEL_PMC_MAX_GENERIC , +hybrid (pmu, cntr_mask)); +} + +static ssize_t fixed_counters_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct pmu *pmu =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%*pbl\n" ,INTEL_PMC_MAX_FIXED , +hybrid (pmu, fixed_cntr_mask)); +} + +static DEVICE_ATTR_RO (gp_counters); +static DEVICE_ATTR_RO (fixed_counters); + +static umode_t x86_pmu_caps_is_visible (struct kobject *kobj, +struct attribute *attr,int n) +{ +struct device *dev =kobj_to_dev (kobj); +struct pmu *pmu =dev_get_drvdata (dev); + +if (attr == &dev_attr_fixed_counters.attr && + !hybrid (pmu, fixed_cntr_mask64)) +return 0 ; + +return attr->mode; +} +static struct attribute *x86_pmu_caps_attrs[] = { &dev_attr_max_precise.attr, + &dev_attr_gp_counters.attr, + &dev_attr_fixed_counters.attr,NULL };static struct attribute_group x86_pmu_caps_group__ro_after_init = { .name ="caps" , .attrs = x86_pmu_caps_attrs, + .is_visible = x86_pmu_caps_is_visible, };static const struct attribute_group *x86_pmu_attr_groups[] = {
Signature · 1 line
2.55.0
PATCH 9/23 perf: Skip exclude_guest events on PMU partitioned counters
In reply to Zide ChenFrom: Kan Liang <[email protected]>
Without PMU partitioning, a PMU with PERF_PMU_CAP_MEDIATED_VPMU blocks host event scheduling while the guest PMU context is loaded.
When PMU partitioning is enabled, host-owned counters remain available. Skip exclude_guest events, but continue to allow other events to be scheduled on the remaining host-owned counters.
--- kernel/events/core.c | 35 ++++++++++++++++++++++++++++------- 1 file changed, 28 insertions(+), 7 deletions(-)
kernel/events/core.c +28 -7
@@ -4105,6 +4105,32 @@ struct merge_sched_data {enum event_type_t event_type; }; +static bool +perf_mediated_pmu_skip_event (struct perf_event *event,enum event_type_t event_type) +{ +if (event->pmu_ctx->pmu->capabilities &PERF_PMU_CAP_PMU_PARTITION ) { +/* +* EVENT_GUEST && !guest_ctx_loaded: perf_load_guest_context() +* reschedules !exclude_guest events. +* +* !EVENT_GUEST && guest_ctx_loaded: Guest context loaded. +*/ +if (event->attr.exclude_guest && + !!(event_type &EVENT_GUEST ) !=is_guest_mediated_pmu_loaded ()) +return true; + }else { +/* +* Don't schedule in any host events because no counters +* are available. +*/ +if (!(event_type &EVENT_GUEST ) && +is_guest_mediated_pmu_loaded ()) +return true; + } + +return false; +} +static int merge_sched_in (struct perf_event *event,void *data) {struct perf_event_context *ctx = event->ctx;
@@ -4116,13 +4142,8 @@ static int merge_sched_in(struct perf_event *event, void *data)if (!event_filter_match (event))return 0 ; -/* -* Don't schedule in any host events from PMU with -* PERF_PMU_CAP_MEDIATED_VPMU, while a guest is running. -*/ -if (is_guest_mediated_pmu_loaded () && - event->pmu_ctx->pmu->capabilities &PERF_PMU_CAP_MEDIATED_VPMU && - !(msd->event_type &EVENT_GUEST )) +if ((event->pmu_ctx->pmu->capabilities &PERF_PMU_CAP_MEDIATED_VPMU ) && +perf_mediated_pmu_skip_event (event, msd->event_type))return 0 ;if (group_can_go_on (event, msd->can_add_hw)) {
Signature · 1 line
2.55.0
PATCH 10/23 perf: Reschedule events across PMU partition transitions
In reply to Zide ChenFrom: Kan Liang <[email protected]>
When entering a guest with PMU partitioning enabled, some counters remain available to the host. Similar to the non-partitioning setup, schedule out all events because a counter may become guest-owned.
Unlike the non-partitioned case, host events must then be rescheduled so that !exclude_guest events can be scheduled onto the remaining host-owned counters, while keeping exclude_guest events off. Since PMU partitioning shrinks the counters available to host events, add a lightweight PMU-context reschedule cpuctx_sched_in_all() instead of rebuilding the entire CPU perf scheduling state.
When exiting the guest, schedule out host events so that they can be rescheduled against the expanded set of host-owned counters. Similar to guest entry, only PMU-level event placement needs to be rebuilt, so add ctx_sched_out_all() instead of using the existing full CPU-context scheduling API.
These context switches rely on the architectural PMU partition mask being configured for the target context. On guest entry, the guest PMU partition mask is expected to be active before events are rescheduled. On guest exit, it is expected to be disabled so that events are rescheduled with host counter constraints.
This differs from the perf core guest context, where guest_ctx_loaded is updated at the end of the load/put guest context. As a result, guest_ctx_loaded does not yet reflect the target context when the rescheduling occurs.
Add pmu_partition_enabled to perf_{load,put}_guest_context(), and hardcode to false temporarily until later patches.
--- arch/x86/kvm/pmu.c | 4 +-- include/linux/perf_event.h | 4 +-- kernel/events/core.c | 52 +++++++++++++++++++++++++++++++++++--- 3 files changed, 53 insertions(+), 7 deletions(-)
arch/x86/kvm/pmu.c +2 -2
@@ -1390,7 +1390,7 @@ void kvm_mediated_pmu_load(struct kvm_vcpu *vcpu)perf_pmu_partition_preload (); -perf_load_guest_context (); +perf_load_guest_context (false);/* * Explicitly clear PERF_GLOBAL_CTRL, as "loading" the guest's context
@@ -1463,5 +1463,5 @@ void kvm_mediated_pmu_put(struct kvm_vcpu *vcpu)perf_put_guest_lvtpc (); -perf_put_guest_context (); +perf_put_guest_context (false); }
include/linux/perf_event.h +2 -2
@@ -1934,8 +1934,8 @@ extern u64 perf_event_pause(struct perf_event *event, bool reset);#ifdef CONFIG_PERF_GUEST_MEDIATED_PMU int perf_create_mediated_pmu (u64 pmu_partition_mask);void perf_release_mediated_pmu (void ); -void perf_load_guest_context (void ); -void perf_put_guest_context (void ); +void perf_load_guest_context (bool pmu_partition_enabled); +void perf_put_guest_context (bool pmu_partition_enabled);int arch_perf_set_pmu_partition_mask (u64 pmu_partition_mask);#endif
kernel/events/core.c +49 -3
76 lines not on this page · raw message
Signature · 1 line
2.55.0
PATCH 11/23 perf, perf/x86: Allow host !exclude_guest events in PMU partitioning
In reply to Zide ChenWith PMU partitioning, the host is allowed to create !exclude_guest events because it no longer yields all PMU resources to the guest while the guest is running.
However, Perf Metrics, LBR, BTS, and PEBS cannot be shared between host and guest. Host events that rely on these exclusive facilities must not be scheduled in if the facilities are guest-owned.
Intel PT PMU is special: Since it can't be partitioned, supporting Intel PT passthrough requires heterogeneous mediated vPMUs. Additional work is needed to handle nr_include_guest_events accounting.
--- arch/x86/events/intel/core.c | 35 ++++++++++++++++++++++++++++++- arch/x86/include/asm/perf_event.h | 1 + kernel/events/core.c | 16 ++++++++++++-- 3 files changed, 49 insertions(+), 3 deletions(-)
arch/x86/events/intel/core.c +34 -1
@@ -4455,10 +4455,40 @@ dyn_constraint(struct cpu_hw_events *cpuc, struct event_constraint *c, int idx)return c; } +static bool event_uses_guest_owned_facility (struct perf_event *event) +{ +/* +* For Intel platforms, PMU partition mask shares the same bit layout +* as IA32_PERF_GLOBAL_STATUS. +*/ +u64 partition_mask =x86_pmu_current_partition_mask (); + +if ((partition_mask &GLOBAL_STATUS_PERF_METRICS_OVF ) && +is_topdown_event (event)) +return true; + +if ((partition_mask &GLOBAL_STATUS_LBRS_FROZEN ) && +needs_branch_stack (event)) +return true; + +if (partition_mask & + (GLOBAL_STATUS_BUFFER_OVF |GLOBAL_STATUS_ARCH_PEBS_THRESHOLD )) { +if (event->attr.precise_ip ||is_pebs_counter_event_group (event)) +return true; + +if ((partition_mask &GLOBAL_STATUS_BUFFER_OVF ) && +intel_pmu_has_bts (event)) +return true; + } + +return false; +} +/* * Mask out guest-owned counters from a constraint when PMU partition has been * entered, so !exclude_guest host events are not scheduled onto them while -* the CPU is in non-root mode. +* the CPU is in non-root mode. Reject the event when a guest is currently +* loaded and it needs a guest-owned exclusive facility. * * This is also used by PMU-specific get_event_constraints() wrappers * that hard-code a static, counter-specific constraint.
@@ -4475,6 +4505,9 @@ part_constraint(struct cpu_hw_events *cpuc, int idx,return c;if (x86_pmu_partition_loaded (cpuc)) { +if (event_uses_guest_owned_facility (event)) +return &emptyconstraint; + c =dyn_constraint (cpuc, c, idx); c->idxmsk64 &= ~x86_pmu_current_partition_mask (); c->weight =hweight64 (c->idxmsk64);
arch/x86/include/asm/perf_event.h +1 -0
@@ -453,6 +453,7 @@ static inline bool is_topdown_idx(int idx)#define GLOBAL_STATUS_ARCH_PEBS_THRESHOLD_BIT 54#define GLOBAL_STATUS_ARCH_PEBS_THRESHOLD BIT_ULL(GLOBAL_STATUS_ARCH_PEBS_THRESHOLD_BIT)#define GLOBAL_STATUS_PERF_METRICS_OVF_BIT 48 +#define GLOBAL_STATUS_PERF_METRICS_OVF BIT_ULL(GLOBAL_STATUS_PERF_METRICS_OVF_BIT)#define GLOBAL_CTRL_EN_PERF_METRICS BIT_ULL(48)/*
kernel/events/core.c +14 -2
@@ -6381,11 +6381,23 @@ static int mediated_pmu_account_event(struct perf_event *event)if (!is_include_guest_event (event))return 0 ; +/* +* This lockless fast path assumes that heterogeneous mediated vPMUs +* are not supported, i.e. a mix of PERF_PMU_CAP_MEDIATED_VPMU PMUs +* with and without PERF_PMU_CAP_PMU_PARTITION. +*/ if (atomic_inc_not_zero (&nr_include_guest_events))return 0 ;guard (mutex)(&perf_mediated_pmu_mutex); -if (atomic_read (&nr_mediated_pmu_vms)) + +/* +* PMU partitioning allows scheduling !exclude_guest events while a +* guest is running. However, it is up to the PMU driver to validate +* whether the facilities needed by the event are available on the host. +*/ +if (atomic_read (&nr_mediated_pmu_vms) && + !(event->pmu->capabilities &PERF_PMU_CAP_PMU_PARTITION ))return -EOPNOTSUPP ;atomic_inc (&nr_include_guest_events);
@@ -6418,7 +6430,7 @@ int perf_create_mediated_pmu(u64 partition_mask)int ret;guard (mutex)(&perf_mediated_pmu_mutex); -if (atomic_read (&nr_include_guest_events)) +if (atomic_read (&nr_include_guest_events) && !partition_mask)return -EBUSY ; ret =arch_perf_set_pmu_partition_mask (partition_mask);
Signature · 1 line
2.55.0
PATCH 12/23 KVM: x86/pmu: Add the perfmon_mask module parameter
In reply to Zide ChenOnly Intel CPUs support PerfMon masking, so this new parameter is Intel-specific. Mediated vPMU must be enabled for PerfMon masking.
PerfMon masking lets a VMM partition PMU resources between host and
guest: each mask bit determines whether the guest (set) or host (clear) owns a counter's MSR(s), RDPMC access, and the corresponding bit in the global MSRs (e.g. IA32_PERF_GLOBAL_CTRL) in non-root mode.
The setting is system-wide and caps what any individual guest may be given; each guest may configure a subset of it.
The parameter is configured with a text-based, semicolon-separated list of terms. Currently supported for a mediated vPMU guest:
guest_gp=<list> general purpose counters (0-based) owned by the guest
guest_fixed=<list> fixed counters (0-based) owned by the guest
perf_metrics PERF_METRICS is owned by the guest
For example:
kvm-intel.perfmon_mask=guest_gp=0-3;guest_fixed=0,2-3;perf_metrics
Internally, the parsed terms are folded into a 64-bit perfmon_mask variable that uses the same layout as the PERFMON_MASK VMCS field, i.e. the IA32_PERF_GLOBAL_STATUS layout. An empty (default) value behaves the same as a plain mediated vPMU.
Using IA32_PERF_GLOBAL_CTRL in the generic VM-exit MSR-store area while the PerfMon masking VM-execution control is set is undefined, so PerfMon masking requires the dedicated Save-IA32_PERF_GLOBAL_CTRL VM-exit control.
cpu_has_vmx_perfmon_mask() is hardcoded to false temporarily until later patches.
--- .../admin-guide/kernel-parameters.txt | 30 ++++++++ arch/x86/kvm/pmu.c | 3 + arch/x86/kvm/pmu.h | 1 + arch/x86/kvm/vmx/capabilities.h | 5 ++ arch/x86/kvm/vmx/pmu_intel.c | 68 +++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 54 +++++++++++++++ arch/x86/kvm/vmx/vmx.h | 1 + arch/x86/kvm/x86.c | 2 +- 8 files changed, 163 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +30 -0
@@ -3284,6 +3284,36 @@ Kernel parameters [KVM,Intel] Control nested virtualization feature in KVM/VMX. Default is 1 (enabled). + kvm-intel.perfmon_mask= + [KVM,Intel] Defines the host-wide PMU resource + partition between guest and host. Resources assigned + to the guest are unavailable to the host, and vice + versa, while a guest is running. + + The value is a semicolon-separated list of terms: + + guest_gp=<list> General purpose counters + (0-based) assigned to the guest. + guest_fixed=<list> Fixed counters (0-based) assigned + to the guest. + perf_metrics PERF_METRICS is assigned to the + guest. Requires fixed counter 3 + to also be assigned to the + guest. + + <list> is a comma-separated list of numbers and/or + ranges, e.g. "0-2,5". Terms and the resources they + don't mention default to being host-owned. For + example: + + kvm-intel.perfmon_mask=guest_gp=0-3;guest_fixed=0-1,3;perf_metrics + + assigns general purpose counters 0-3, fixed counters + 0, 1 and 3, and PERF_METRICS to the guest, while all + other resources remain with the host. + + Default is "" (disabled). + kvm-intel.unrestricted_guest= [KVM,Intel] Control KVM's use of unrestricted guest feature (virtualized real and unpaged mode). Default
arch/x86/kvm/pmu.c +3 -0
@@ -43,6 +43,9 @@ module_param(enable_pmu, bool, 0444);bool __read_mostly enable_mediated_pmu;EXPORT_SYMBOL_FOR_KVM_INTERNAL (enable_mediated_pmu); +u64 __read_mostly perfmon_mask; +EXPORT_SYMBOL_FOR_KVM_INTERNAL (perfmon_mask); +struct kvm_x86_pmu_event_filter {__u32 action;__u32 nevents;
arch/x86/kvm/pmu.h +1 -0
@@ -64,6 +64,7 @@ struct kvm_pmu_ops {extern bool enable_pmu;extern bool enable_mediated_pmu; +extern u64 perfmon_mask;void kvm_pmu_ops_update (const struct kvm_pmu_ops *pmu_ops);
arch/x86/kvm/vmx/capabilities.h +5 -0
@@ -296,6 +296,11 @@ static inline bool cpu_has_vmx_ipiv(void)return vmcs_config.cpu_based_3rd_exec_ctrl &TERTIARY_EXEC_IPI_VIRT ; } +static inline bool cpu_has_vmx_perfmon_mask (void ) +{ +return false; +} +static inline bool cpu_has_vmx_flexpriority (void ) { return cpu_has_vmx_tpr_shadow() &&
arch/x86/kvm/vmx/pmu_intel.c +68 -0
74 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.c +54 -0
78 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.h +1 -0
@@ -667,6 +667,7 @@ static __always_inline struct vcpu_vmx *to_vmx(struct kvm_vcpu *vcpu)void intel_pmu_cross_mapped_check (struct kvm_pmu *pmu);int intel_pmu_create_guest_lbr_event (struct kvm_vcpu *vcpu);void vmx_passthrough_lbr_msrs (struct kvm_vcpu *vcpu); +void intel_pmu_perfmon_mask_setup (void );struct vmcs *alloc_vmcs_cpu (bool shadow,int cpu,gfp_t flags);void free_vmcs (struct vmcs *vmcs);
arch/x86/kvm/x86.c +1 -1
@@ -9328,7 +9328,7 @@ int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id)if (enable_mediated_pmu && kvm->arch.enable_pmu && !kvm->arch.created_mediated_pmu) {if (irqchip_in_kernel (kvm)) { - r =perf_create_mediated_pmu (0 ); + r =perf_create_mediated_pmu (perfmon_mask);if (r) {pr_warn_ratelimited (PERF_MEDIATED_PMU_MSG );return r;
Signature · 1 line
2.55.0
Re: [PATCH 12/23] KVM: x86/pmu: Add the perfmon_mask module parameter
I don't remember why to set the perfmon_mask to be a kvm_intel module parameter, but per my understanding, it seems better and more straightforward to set perfmon_mask to be a perf/x86 parameter. It's also consistent with current perf/x86 behavior. Host exposes all PMU capabilities to KVM instead of the opposite.
Thanks.
Quoted 320 lines · Zide Chen
Only Intel CPUs support PerfMon masking, so this new parameter is Intel-specific. Mediated vPMU must be enabled for PerfMon masking.
PerfMon masking lets a VMM partition PMU resources between host and
guest: each mask bit determines whether the guest (set) or host (clear) owns a counter's MSR(s), RDPMC access, and the corresponding bit in the global MSRs (e.g.IA32_PERF_GLOBAL_CTRL) in non-root mode.The setting is system-wide and caps what any individual guest may be given; each guest may configure a subset of it.
The parameter is configured with a text-based, semicolon-separated list of terms. Currently supported for a mediated vPMU guest:
guest_gp=<list> general purpose counters (0-based) owned by the guest
guest_fixed=<list> fixed counters (0-based) owned by the guest
perf_metricsPERF_METRICSis owned by the guestFor example:
kvm-intel.perfmon_mask=guest_gp=0-3;guest_fixed=0,2-3;perf_metrics
Internally, the parsed terms are folded into a 64-bit
perfmon_maskvariable that uses the same layout as thePERFMON_MASKVMCS field, i.e. theIA32_PERF_GLOBAL_STATUSlayout. An empty (default) value behaves the same as a plain mediated vPMU.Using
IA32_PERF_GLOBAL_CTRLin the generic VM-exit MSR-store area while the PerfMon masking VM-execution control is set is undefined, so PerfMon masking requires the dedicated Save-IA32_PERF_GLOBAL_CTRL VM-exit control.
cpu_has_vmx_perfmon_mask()is hardcoded to false temporarily until later patches.Signed-off-by Zide Chen <[email protected]>--- .../admin-guide/kernel-parameters.txt | 30 ++++++++ arch/x86/kvm/pmu.c | 3 + arch/x86/kvm/pmu.h | 1 + arch/x86/kvm/vmx/capabilities.h | 5 ++ arch/x86/kvm/vmx/pmu_intel.c | 68 +++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 54 +++++++++++++++ arch/x86/kvm/vmx/vmx.h | 1 + arch/x86/kvm/x86.c | 2 +- 8 files changed, 163 insertions(+), 1 deletion(-)Documentation/admin-guide/kernel-parameters.txt +30
-0@@ -3284,6 +3284,36 @@ Kernel parameters [KVM,Intel] Control nested virtualization feature in KVM/VMX. Default is 1 (enabled). + kvm-intel.perfmon_mask= + [KVM,Intel] Defines the host-wide PMU resource + partition between guest and host. Resources assigned + to the guest are unavailable to the host, and vice + versa, while a guest is running. + + The value is a semicolon-separated list of terms: + + guest_gp=<list> General purpose counters + (0-based) assigned to the guest. + guest_fixed=<list> Fixed counters (0-based) assigned + to the guest. + perf_metrics PERF_METRICS is assigned to the + guest. Requires fixed counter 3 + to also be assigned to the + guest. + + <list> is a comma-separated list of numbers and/or + ranges, e.g. "0-2,5". Terms and the resources they + don't mention default to being host-owned. For + example: + + kvm-intel.perfmon_mask=guest_gp=0-3;guest_fixed=0-1,3;perf_metrics + + assigns general purpose counters 0-3, fixed counters + 0, 1 and 3, and PERF_METRICS to the guest, while all + other resources remain with the host. + + Default is "" (disabled). + kvm-intel.unrestricted_guest= [KVM,Intel] Control KVM's use of unrestricted guest feature (virtualized real and unpaged mode). Defaultarch/x86/kvm/pmu.c +3
-0@@ -43,6 +43,9 @@ module_param(enable_pmu, bool, 0444);bool __read_mostly enable_mediated_pmu;EXPORT_SYMBOL_FOR_KVM_INTERNAL (enable_mediated_pmu); +u64 __read_mostly perfmon_mask; +EXPORT_SYMBOL_FOR_KVM_INTERNAL (perfmon_mask); +struct kvm_x86_pmu_event_filter {__u32 action;__u32 nevents;arch/x86/kvm/pmu.h +1
-0@@ -64,6 +64,7 @@ struct kvm_pmu_ops {extern bool enable_pmu;extern bool enable_mediated_pmu; +extern u64 perfmon_mask;void kvm_pmu_ops_update (const struct kvm_pmu_ops *pmu_ops);arch/x86/kvm/vmx/capabilities.h +5
-0@@ -296,6 +296,11 @@ static inline bool cpu_has_vmx_ipiv(void)return vmcs_config.cpu_based_3rd_exec_ctrl &TERTIARY_EXEC_IPI_VIRT ; } +static inline bool cpu_has_vmx_perfmon_mask (void ) +{ +return false; +} +static inline bool cpu_has_vmx_flexpriority (void ) { return cpu_has_vmx_tpr_shadow() &&arch/x86/kvm/vmx/pmu_intel.c +68
-0@@ -955,6 +955,74 @@ static void intel_mediated_pmu_put(struct kvm_vcpu *vcpu) } } +static bool intel_pmu_validate_perfmon_mask (void ) +{ +u64 guest_fixed_mask, guest_gp_mask; + +/* +* Combining VM-exit MSR-store with PerfMon masking produces +* undefined behavior, so it requires hardware support for this +* dedicated save control. +*/ +if (!cpu_has_save_perf_global_ctrl ()) +return false; + + guest_gp_mask = perfmon_mask &GENMASK_ULL (INTEL_PMC_MAX_GENERIC -1 ,0 ); + guest_fixed_mask = perfmon_mask >>INTEL_PMC_IDX_FIXED ; + guest_fixed_mask &=GENMASK_ULL (INTEL_PMC_MAX_FIXED -1 ,0 ); + +if ((guest_fixed_mask & ~kvm_pmu_cap.fixed_cntr_mask64) || + (guest_gp_mask & ~kvm_pmu_cap.cntr_mask64)) +return false; + +/* +* Without KVM Arch PerfMon extension support, the guest cannot own +* non-contiguous GP counters. +*/ +if (guest_gp_mask & (guest_gp_mask +1 )) +return false; + +if ((perfmon_mask &GLOBAL_STATUS_PERF_METRICS_OVF ) && + !(kvm_host.perf_capabilities &PERF_CAP_PERF_METRICS )) +return false; + +/* +* PERF_METRICS and fixed counter 3 must both be host-owned or both +* guest-owned. +*/ +if (!!(perfmon_mask &BIT_ULL (GLOBAL_STATUS_PERF_METRICS_OVF_BIT )) != + !!(perfmon_mask &BIT_ULL (INTEL_PMC_IDX_FIXED +3 ))) +return false; + +/* +* The guest must not own all PMU counters. Otherwise, the configuration +* degenerates into plain mediated vPMU and adds unnecessary complexity +* to the perf scheduler. +*/ +if ((guest_fixed_mask == kvm_host_pmu.fixed_cntr_mask64) && + (guest_gp_mask == kvm_host_pmu.cntr_mask64)) +return false; + +return true; +} + +void intel_pmu_perfmon_mask_setup (void ) +{ +if (!perfmon_mask) +return ; + +if (!enable_mediated_pmu || !cpu_has_vmx_perfmon_mask ()) { + perfmon_mask =0 ; +return ; + } + +if (!intel_pmu_validate_perfmon_mask ()) { +pr_warn ("Invalid perfmon_mask=%#llx, disabling PerfMon masking\n" , + perfmon_mask); + perfmon_mask =0 ; + } +} +struct kvm_pmu_ops intel_pmu_ops__initdata = { .emulate_rdpmc = intel_emulate_rdpmc, .msr_idx_to_pmc = intel_msr_idx_to_pmc,arch/x86/kvm/vmx/vmx.c +54
-0@@ -14,6 +14,7 @@*/ #define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#include <linux/bitmap.h> #include <linux/highmem.h> #include <linux/hrtimer.h> #include <linux/kernel.h> @@ -25,6 +26,7 @@#include <linux/sched.h> #include <linux/sched/smt.h> #include <linux/slab.h> +#include <linux/string.h> #include <linux/tboot.h> #include <linux/trace_events.h> @@ -164,6 +166,56 @@ module_param(allow_smaller_maxphyaddr, bool, S_IRUGO);module_param (enable_mediated_pmu, bool,0444 ); +/* +* See the "kvm-intel.perfmon_mask" entry in +* Documentation/admin-guide/kernel-parameters.txt for the full syntax. +* Example: kvm-intel.perfmon_mask=guest_gp=0-3;guest_fixed=0-1,3;perf_metrics +*/ +static int perfmon_mask_set (const char *val,const struct kernel_param *kp) +{ +unsigned long gp_bitmap =0 , fixed_bitmap =0 ; +char *buf, *orig, *tok; +size_t prefix_len; +u64 mask =0 ; +int r =0 ; + + buf = orig =kstrdup (val,GFP_KERNEL ); +if (!buf) +return -ENOMEM ; + +while ((tok =strsep (&buf,";" )) !=NULL ) { +if (!*tok) +continue ; + +if (!strcmp (tok,"perf_metrics" )) { + mask |=GLOBAL_STATUS_PERF_METRICS_OVF ; + }else if ((prefix_len =str_has_prefix (tok,"guest_gp=" ))) { + r =bitmap_parselist (tok + prefix_len, &gp_bitmap, +INTEL_PMC_MAX_GENERIC ); + }else if ((prefix_len =str_has_prefix (tok,"guest_fixed=" ))) { + r =bitmap_parselist (tok + prefix_len, &fixed_bitmap, +INTEL_PMC_MAX_FIXED ); + }else { + r = -EINVAL ; + } + +if (r) + goto out; + } + + mask |= gp_bitmap | ((u64 )fixed_bitmap <<INTEL_PMC_IDX_FIXED ); + *(u64 *)kp->arg = mask; +out: +kfree (orig); +return r; +} + +static const struct kernel_param_ops perfmon_mask_ops = { + .set = perfmon_mask_set, + .get = param_get_ullong, +}; +module_param_cb (perfmon_mask, &perfmon_mask_ops, &perfmon_mask,0444 ); +#define KVM_VM_CR0_ALWAYS_OFF (X86_CR0_NW | X86_CR0_CD)#define KVM_VM_CR0_ALWAYS_ON_UNRESTRICTED_GUEST X86_CR0_NE#define KVM_VM_CR0_ALWAYS_ON \@@ -8824,6 +8876,8 @@ __init int vmx_hardware_setup(void)else vt_init_ops.handle_intel_pt_intr =NULL ; +intel_pmu_perfmon_mask_setup (); +setup_default_sgx_lepubkeyhash ();vmx_set_cpu_caps ();arch/x86/kvm/vmx/vmx.h +1
-0@@ -667,6 +667,7 @@ static __always_inline struct vcpu_vmx *to_vmx(struct kvm_vcpu *vcpu)void intel_pmu_cross_mapped_check (struct kvm_pmu *pmu);int intel_pmu_create_guest_lbr_event (struct kvm_vcpu *vcpu);void vmx_passthrough_lbr_msrs (struct kvm_vcpu *vcpu); +void intel_pmu_perfmon_mask_setup (void );struct vmcs *alloc_vmcs_cpu (bool shadow,int cpu,gfp_t flags);void free_vmcs (struct vmcs *vmcs);arch/x86/kvm/x86.c +1
-1@@ -9328,7 +9328,7 @@ int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id)if (enable_mediated_pmu && kvm->arch.enable_pmu && !kvm->arch.created_mediated_pmu) {if (irqchip_in_kernel (kvm)) { - r =perf_create_mediated_pmu (0 ); + r =perf_create_mediated_pmu (perfmon_mask);if (r) {pr_warn_ratelimited (PERF_MEDIATED_PMU_MSG );return r;
Re: [PATCH 12/23] KVM: x86/pmu: Add the perfmon_mask module parameter
Quoted 5 lines · Dapeng
I don't remember why to set the
perfmon_maskto be akvm_intelmodule parameter, but per my understanding, it seems better and more straightforward to setperfmon_maskto be a perf/x86 parameter. It's also consistent with current perf/x86 behavior. Host exposes all PMU capabilities to KVM instead of the opposite.
Perfmon Masking can be enabled only after perf_create_mediated_pmu() succeeds, but the following conditions are not determined by perf/x86:
- Host VMX capability support.
- mediated PMU is enabled.
- perf/core does not block the mediated guest creation.
If perf/x86 owns the parameter, but whether the parameter takes effect depends on actions taken by a loaded module at runtime, it does not sound reasonable.
BTW, in this approach, an additional interface between KVM and perf/x86 would be needed to communicate the enablement of Perfmon Masking.
Additionally, this is less flexible, as the mask cannot be changed without rebooting the host. I might be wrong, but not sure whether different masks may be needed to accommodate different guests for live migration.
Also, given that the format of this parameter is complicated, it is not surprising that an administrator may provide an incorrect or undesired value at boot time, and repeatedly rebooting the host to correct it may not be convenient.
Quoted 323 lines
Thanks.
Quoted 320 lines · Zide Chen
Only Intel CPUs support PerfMon masking, so this new parameter is Intel-specific. Mediated vPMU must be enabled for PerfMon masking.
PerfMon masking lets a VMM partition PMU resources between host and
guest: each mask bit determines whether the guest (set) or host (clear) owns a counter's MSR(s), RDPMC access, and the corresponding bit in the global MSRs (e.g.IA32_PERF_GLOBAL_CTRL) in non-root mode.The setting is system-wide and caps what any individual guest may be given; each guest may configure a subset of it.
The parameter is configured with a text-based, semicolon-separated list of terms. Currently supported for a mediated vPMU guest:
guest_gp=<list> general purpose counters (0-based) owned by the guest
guest_fixed=<list> fixed counters (0-based) owned by the guest
perf_metricsPERF_METRICSis owned by the guestFor example:
kvm-intel.perfmon_mask=guest_gp=0-3;guest_fixed=0,2-3;perf_metrics
Internally, the parsed terms are folded into a 64-bit
perfmon_maskvariable that uses the same layout as thePERFMON_MASKVMCS field, i.e. theIA32_PERF_GLOBAL_STATUSlayout. An empty (default) value behaves the same as a plain mediated vPMU.Using
IA32_PERF_GLOBAL_CTRLin the generic VM-exit MSR-store area while the PerfMon masking VM-execution control is set is undefined, so PerfMon masking requires the dedicated Save-IA32_PERF_GLOBAL_CTRL VM-exit control.
cpu_has_vmx_perfmon_mask()is hardcoded to false temporarily until later patches.Signed-off-by Zide Chen <[email protected]>--- .../admin-guide/kernel-parameters.txt | 30 ++++++++ arch/x86/kvm/pmu.c | 3 + arch/x86/kvm/pmu.h | 1 + arch/x86/kvm/vmx/capabilities.h | 5 ++ arch/x86/kvm/vmx/pmu_intel.c | 68 +++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 54 +++++++++++++++ arch/x86/kvm/vmx/vmx.h | 1 + arch/x86/kvm/x86.c | 2 +- 8 files changed, 163 insertions(+), 1 deletion(-)Documentation/admin-guide/kernel-parameters.txt +30
-0@@ -3284,6 +3284,36 @@ Kernel parameters [KVM,Intel] Control nested virtualization feature in KVM/VMX. Default is 1 (enabled). + kvm-intel.perfmon_mask= + [KVM,Intel] Defines the host-wide PMU resource + partition between guest and host. Resources assigned + to the guest are unavailable to the host, and vice + versa, while a guest is running. + + The value is a semicolon-separated list of terms: + + guest_gp=<list> General purpose counters + (0-based) assigned to the guest. + guest_fixed=<list> Fixed counters (0-based) assigned + to the guest. + perf_metrics PERF_METRICS is assigned to the + guest. Requires fixed counter 3 + to also be assigned to the + guest. + + <list> is a comma-separated list of numbers and/or + ranges, e.g. "0-2,5". Terms and the resources they + don't mention default to being host-owned. For + example: + + kvm-intel.perfmon_mask=guest_gp=0-3;guest_fixed=0-1,3;perf_metrics + + assigns general purpose counters 0-3, fixed counters + 0, 1 and 3, and PERF_METRICS to the guest, while all + other resources remain with the host. + + Default is "" (disabled). + kvm-intel.unrestricted_guest= [KVM,Intel] Control KVM's use of unrestricted guest feature (virtualized real and unpaged mode). Defaultarch/x86/kvm/pmu.c +3
-0@@ -43,6 +43,9 @@ module_param(enable_pmu, bool, 0444);bool __read_mostly enable_mediated_pmu;EXPORT_SYMBOL_FOR_KVM_INTERNAL (enable_mediated_pmu); +u64 __read_mostly perfmon_mask; +EXPORT_SYMBOL_FOR_KVM_INTERNAL (perfmon_mask); +struct kvm_x86_pmu_event_filter {__u32 action;__u32 nevents;arch/x86/kvm/pmu.h +1
-0@@ -64,6 +64,7 @@ struct kvm_pmu_ops {extern bool enable_pmu;extern bool enable_mediated_pmu; +extern u64 perfmon_mask;void kvm_pmu_ops_update (const struct kvm_pmu_ops *pmu_ops);arch/x86/kvm/vmx/capabilities.h +5
-0@@ -296,6 +296,11 @@ static inline bool cpu_has_vmx_ipiv(void)return vmcs_config.cpu_based_3rd_exec_ctrl &TERTIARY_EXEC_IPI_VIRT ; } +static inline bool cpu_has_vmx_perfmon_mask (void ) +{ +return false; +} +static inline bool cpu_has_vmx_flexpriority (void ) { return cpu_has_vmx_tpr_shadow() &&arch/x86/kvm/vmx/pmu_intel.c +68
-0@@ -955,6 +955,74 @@ static void intel_mediated_pmu_put(struct kvm_vcpu *vcpu) } } +static bool intel_pmu_validate_perfmon_mask (void ) +{ +u64 guest_fixed_mask, guest_gp_mask; + +/* +* Combining VM-exit MSR-store with PerfMon masking produces +* undefined behavior, so it requires hardware support for this +* dedicated save control. +*/ +if (!cpu_has_save_perf_global_ctrl ()) +return false; + + guest_gp_mask = perfmon_mask &GENMASK_ULL (INTEL_PMC_MAX_GENERIC -1 ,0 ); + guest_fixed_mask = perfmon_mask >>INTEL_PMC_IDX_FIXED ; + guest_fixed_mask &=GENMASK_ULL (INTEL_PMC_MAX_FIXED -1 ,0 ); + +if ((guest_fixed_mask & ~kvm_pmu_cap.fixed_cntr_mask64) || + (guest_gp_mask & ~kvm_pmu_cap.cntr_mask64)) +return false; + +/* +* Without KVM Arch PerfMon extension support, the guest cannot own +* non-contiguous GP counters. +*/ +if (guest_gp_mask & (guest_gp_mask +1 )) +return false; + +if ((perfmon_mask &GLOBAL_STATUS_PERF_METRICS_OVF ) && + !(kvm_host.perf_capabilities &PERF_CAP_PERF_METRICS )) +return false; + +/* +* PERF_METRICS and fixed counter 3 must both be host-owned or both +* guest-owned. +*/ +if (!!(perfmon_mask &BIT_ULL (GLOBAL_STATUS_PERF_METRICS_OVF_BIT )) != + !!(perfmon_mask &BIT_ULL (INTEL_PMC_IDX_FIXED +3 ))) +return false; + +/* +* The guest must not own all PMU counters. Otherwise, the configuration +* degenerates into plain mediated vPMU and adds unnecessary complexity +* to the perf scheduler. +*/ +if ((guest_fixed_mask == kvm_host_pmu.fixed_cntr_mask64) && + (guest_gp_mask == kvm_host_pmu.cntr_mask64)) +return false; + +return true; +} + +void intel_pmu_perfmon_mask_setup (void ) +{ +if (!perfmon_mask) +return ; + +if (!enable_mediated_pmu || !cpu_has_vmx_perfmon_mask ()) { + perfmon_mask =0 ; +return ; + } + +if (!intel_pmu_validate_perfmon_mask ()) { +pr_warn ("Invalid perfmon_mask=%#llx, disabling PerfMon masking\n" , + perfmon_mask); + perfmon_mask =0 ; + } +} +struct kvm_pmu_ops intel_pmu_ops__initdata = { .emulate_rdpmc = intel_emulate_rdpmc, .msr_idx_to_pmc = intel_msr_idx_to_pmc,arch/x86/kvm/vmx/vmx.c +54
-0@@ -14,6 +14,7 @@*/ #define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#include <linux/bitmap.h> #include <linux/highmem.h> #include <linux/hrtimer.h> #include <linux/kernel.h> @@ -25,6 +26,7 @@#include <linux/sched.h> #include <linux/sched/smt.h> #include <linux/slab.h> +#include <linux/string.h> #include <linux/tboot.h> #include <linux/trace_events.h> @@ -164,6 +166,56 @@ module_param(allow_smaller_maxphyaddr, bool, S_IRUGO);module_param (enable_mediated_pmu, bool,0444 ); +/* +* See the "kvm-intel.perfmon_mask" entry in +* Documentation/admin-guide/kernel-parameters.txt for the full syntax. +* Example: kvm-intel.perfmon_mask=guest_gp=0-3;guest_fixed=0-1,3;perf_metrics +*/ +static int perfmon_mask_set (const char *val,const struct kernel_param *kp) +{ +unsigned long gp_bitmap =0 , fixed_bitmap =0 ; +char *buf, *orig, *tok; +size_t prefix_len; +u64 mask =0 ; +int r =0 ; + + buf = orig =kstrdup (val,GFP_KERNEL ); +if (!buf) +return -ENOMEM ; + +while ((tok =strsep (&buf,";" )) !=NULL ) { +if (!*tok) +continue ; + +if (!strcmp (tok,"perf_metrics" )) { + mask |=GLOBAL_STATUS_PERF_METRICS_OVF ; + }else if ((prefix_len =str_has_prefix (tok,"guest_gp=" ))) { + r =bitmap_parselist (tok + prefix_len, &gp_bitmap, +INTEL_PMC_MAX_GENERIC ); + }else if ((prefix_len =str_has_prefix (tok,"guest_fixed=" ))) { + r =bitmap_parselist (tok + prefix_len, &fixed_bitmap, +INTEL_PMC_MAX_FIXED ); + }else { + r = -EINVAL ; + } + +if (r) + goto out; + } + + mask |= gp_bitmap | ((u64 )fixed_bitmap <<INTEL_PMC_IDX_FIXED ); + *(u64 *)kp->arg = mask; +out: +kfree (orig); +return r; +} + +static const struct kernel_param_ops perfmon_mask_ops = { + .set = perfmon_mask_set, + .get = param_get_ullong, +}; +module_param_cb (perfmon_mask, &perfmon_mask_ops, &perfmon_mask,0444 ); +#define KVM_VM_CR0_ALWAYS_OFF (X86_CR0_NW | X86_CR0_CD)#define KVM_VM_CR0_ALWAYS_ON_UNRESTRICTED_GUEST X86_CR0_NE#define KVM_VM_CR0_ALWAYS_ON \@@ -8824,6 +8876,8 @@ __init int vmx_hardware_setup(void)else vt_init_ops.handle_intel_pt_intr =NULL ; +intel_pmu_perfmon_mask_setup (); +setup_default_sgx_lepubkeyhash ();vmx_set_cpu_caps ();arch/x86/kvm/vmx/vmx.h +1
-0@@ -667,6 +667,7 @@ static __always_inline struct vcpu_vmx *to_vmx(struct kvm_vcpu *vcpu)void intel_pmu_cross_mapped_check (struct kvm_pmu *pmu);int intel_pmu_create_guest_lbr_event (struct kvm_vcpu *vcpu);void vmx_passthrough_lbr_msrs (struct kvm_vcpu *vcpu); +void intel_pmu_perfmon_mask_setup (void );struct vmcs *alloc_vmcs_cpu (bool shadow,int cpu,gfp_t flags);void free_vmcs (struct vmcs *vmcs);arch/x86/kvm/x86.c +1
-1@@ -9328,7 +9328,7 @@ int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id)if (enable_mediated_pmu && kvm->arch.enable_pmu && !kvm->arch.created_mediated_pmu) {if (irqchip_in_kernel (kvm)) { - r =perf_create_mediated_pmu (0 ); + r =perf_create_mediated_pmu (perfmon_mask);if (r) {pr_warn_ratelimited (PERF_MEDIATED_PMU_MSG );return r;
Re: [PATCH 12/23] KVM: x86/pmu: Add the perfmon_mask module parameter
Quoted 7 lines · Zide
Quoted 5 lines · Dapeng
I don't remember why to set the
perfmon_maskto be akvm_intelmodule parameter, but per my understanding, it seems better and more straightforward to setperfmon_maskto be a perf/x86 parameter. It's also consistent with current perf/x86 behavior. Host exposes all PMU capabilities to KVM instead of the opposite.
Quoted 6 lines
Perfmon Masking can be enabled only after
perf_create_mediated_pmu()succeeds, but the following conditions are not determined by perf/x86:- Host VMX capability support.
- mediated PMU is enabled.
- perf/core does not block the mediated guest creation.
Ok, strictly speaking, whether mediated vPMU is enabled has nothing to do with the PMU partition, but it's true that only KVM knows if VMX supports perfmon_mask.
If we decide to define the perfmon_mask as a kvm_intel parameter, the host events need to be forced scheduling out and scheduling when the parameter changes. But suppose it's fine since it should be not frequent to change the parameter.
Thanks.
Quoted 340 lines
If perf/x86 owns the parameter, but whether the parameter takes effect depends on actions taken by a loaded module at runtime, it does not sound reasonable.
BTW, in this approach, an additional interface between KVM and perf/x86 would be needed to communicate the enablement of Perfmon Masking.
Additionally, this is less flexible, as the mask cannot be changed without rebooting the host. I might be wrong, but not sure whether different masks may be needed to accommodate different guests for live migration.
Also, given that the format of this parameter is complicated, it is not surprising that an administrator may provide an incorrect or undesired value at boot time, and repeatedly rebooting the host to correct it may not be convenient.
Quoted 323 lines
Thanks.
Quoted 320 lines · Zide Chen
Only Intel CPUs support PerfMon masking, so this new parameter is Intel-specific. Mediated vPMU must be enabled for PerfMon masking.
PerfMon masking lets a VMM partition PMU resources between host and
guest: each mask bit determines whether the guest (set) or host (clear) owns a counter's MSR(s), RDPMC access, and the corresponding bit in the global MSRs (e.g.IA32_PERF_GLOBAL_CTRL) in non-root mode.The setting is system-wide and caps what any individual guest may be given; each guest may configure a subset of it.
The parameter is configured with a text-based, semicolon-separated list of terms. Currently supported for a mediated vPMU guest:
guest_gp=<list> general purpose counters (0-based) owned by the guest
guest_fixed=<list> fixed counters (0-based) owned by the guest
perf_metricsPERF_METRICSis owned by the guestFor example:
kvm-intel.perfmon_mask=guest_gp=0-3;guest_fixed=0,2-3;perf_metrics
Internally, the parsed terms are folded into a 64-bit
perfmon_maskvariable that uses the same layout as thePERFMON_MASKVMCS field, i.e. theIA32_PERF_GLOBAL_STATUSlayout. An empty (default) value behaves the same as a plain mediated vPMU.Using
IA32_PERF_GLOBAL_CTRLin the generic VM-exit MSR-store area while the PerfMon masking VM-execution control is set is undefined, so PerfMon masking requires the dedicated Save-IA32_PERF_GLOBAL_CTRL VM-exit control.
cpu_has_vmx_perfmon_mask()is hardcoded to false temporarily until later patches.Signed-off-by Zide Chen <[email protected]>--- .../admin-guide/kernel-parameters.txt | 30 ++++++++ arch/x86/kvm/pmu.c | 3 + arch/x86/kvm/pmu.h | 1 + arch/x86/kvm/vmx/capabilities.h | 5 ++ arch/x86/kvm/vmx/pmu_intel.c | 68 +++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 54 +++++++++++++++ arch/x86/kvm/vmx/vmx.h | 1 + arch/x86/kvm/x86.c | 2 +- 8 files changed, 163 insertions(+), 1 deletion(-)Documentation/admin-guide/kernel-parameters.txt +30
-0@@ -3284,6 +3284,36 @@ Kernel parameters [KVM,Intel] Control nested virtualization feature in KVM/VMX. Default is 1 (enabled). + kvm-intel.perfmon_mask= + [KVM,Intel] Defines the host-wide PMU resource + partition between guest and host. Resources assigned + to the guest are unavailable to the host, and vice + versa, while a guest is running. + + The value is a semicolon-separated list of terms: + + guest_gp=<list> General purpose counters + (0-based) assigned to the guest. + guest_fixed=<list> Fixed counters (0-based) assigned + to the guest. + perf_metrics PERF_METRICS is assigned to the + guest. Requires fixed counter 3 + to also be assigned to the + guest. + + <list> is a comma-separated list of numbers and/or + ranges, e.g. "0-2,5". Terms and the resources they + don't mention default to being host-owned. For + example: + + kvm-intel.perfmon_mask=guest_gp=0-3;guest_fixed=0-1,3;perf_metrics + + assigns general purpose counters 0-3, fixed counters + 0, 1 and 3, and PERF_METRICS to the guest, while all + other resources remain with the host. + + Default is "" (disabled). + kvm-intel.unrestricted_guest= [KVM,Intel] Control KVM's use of unrestricted guest feature (virtualized real and unpaged mode). Defaultarch/x86/kvm/pmu.c +3
-0@@ -43,6 +43,9 @@ module_param(enable_pmu, bool, 0444);bool __read_mostly enable_mediated_pmu;EXPORT_SYMBOL_FOR_KVM_INTERNAL (enable_mediated_pmu); +u64 __read_mostly perfmon_mask; +EXPORT_SYMBOL_FOR_KVM_INTERNAL (perfmon_mask); +struct kvm_x86_pmu_event_filter {__u32 action;__u32 nevents;arch/x86/kvm/pmu.h +1
-0@@ -64,6 +64,7 @@ struct kvm_pmu_ops {extern bool enable_pmu;extern bool enable_mediated_pmu; +extern u64 perfmon_mask;void kvm_pmu_ops_update (const struct kvm_pmu_ops *pmu_ops);arch/x86/kvm/vmx/capabilities.h +5
-0@@ -296,6 +296,11 @@ static inline bool cpu_has_vmx_ipiv(void)return vmcs_config.cpu_based_3rd_exec_ctrl &TERTIARY_EXEC_IPI_VIRT ; } +static inline bool cpu_has_vmx_perfmon_mask (void ) +{ +return false; +} +static inline bool cpu_has_vmx_flexpriority (void ) { return cpu_has_vmx_tpr_shadow() &&arch/x86/kvm/vmx/pmu_intel.c +68
-0@@ -955,6 +955,74 @@ static void intel_mediated_pmu_put(struct kvm_vcpu *vcpu) } } +static bool intel_pmu_validate_perfmon_mask (void ) +{ +u64 guest_fixed_mask, guest_gp_mask; + +/* +* Combining VM-exit MSR-store with PerfMon masking produces +* undefined behavior, so it requires hardware support for this +* dedicated save control. +*/ +if (!cpu_has_save_perf_global_ctrl ()) +return false; + + guest_gp_mask = perfmon_mask &GENMASK_ULL (INTEL_PMC_MAX_GENERIC -1 ,0 ); + guest_fixed_mask = perfmon_mask >>INTEL_PMC_IDX_FIXED ; + guest_fixed_mask &=GENMASK_ULL (INTEL_PMC_MAX_FIXED -1 ,0 ); + +if ((guest_fixed_mask & ~kvm_pmu_cap.fixed_cntr_mask64) || + (guest_gp_mask & ~kvm_pmu_cap.cntr_mask64)) +return false; + +/* +* Without KVM Arch PerfMon extension support, the guest cannot own +* non-contiguous GP counters. +*/ +if (guest_gp_mask & (guest_gp_mask +1 )) +return false; + +if ((perfmon_mask &GLOBAL_STATUS_PERF_METRICS_OVF ) && + !(kvm_host.perf_capabilities &PERF_CAP_PERF_METRICS )) +return false; + +/* +* PERF_METRICS and fixed counter 3 must both be host-owned or both +* guest-owned. +*/ +if (!!(perfmon_mask &BIT_ULL (GLOBAL_STATUS_PERF_METRICS_OVF_BIT )) != + !!(perfmon_mask &BIT_ULL (INTEL_PMC_IDX_FIXED +3 ))) +return false; + +/* +* The guest must not own all PMU counters. Otherwise, the configuration +* degenerates into plain mediated vPMU and adds unnecessary complexity +* to the perf scheduler. +*/ +if ((guest_fixed_mask == kvm_host_pmu.fixed_cntr_mask64) && + (guest_gp_mask == kvm_host_pmu.cntr_mask64)) +return false; + +return true; +} + +void intel_pmu_perfmon_mask_setup (void ) +{ +if (!perfmon_mask) +return ; + +if (!enable_mediated_pmu || !cpu_has_vmx_perfmon_mask ()) { + perfmon_mask =0 ; +return ; + } + +if (!intel_pmu_validate_perfmon_mask ()) { +pr_warn ("Invalid perfmon_mask=%#llx, disabling PerfMon masking\n" , + perfmon_mask); + perfmon_mask =0 ; + } +} +struct kvm_pmu_ops intel_pmu_ops__initdata = { .emulate_rdpmc = intel_emulate_rdpmc, .msr_idx_to_pmc = intel_msr_idx_to_pmc,arch/x86/kvm/vmx/vmx.c +54
-0@@ -14,6 +14,7 @@*/ #define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#include <linux/bitmap.h> #include <linux/highmem.h> #include <linux/hrtimer.h> #include <linux/kernel.h> @@ -25,6 +26,7 @@#include <linux/sched.h> #include <linux/sched/smt.h> #include <linux/slab.h> +#include <linux/string.h> #include <linux/tboot.h> #include <linux/trace_events.h> @@ -164,6 +166,56 @@ module_param(allow_smaller_maxphyaddr, bool, S_IRUGO);module_param (enable_mediated_pmu, bool,0444 ); +/* +* See the "kvm-intel.perfmon_mask" entry in +* Documentation/admin-guide/kernel-parameters.txt for the full syntax. +* Example: kvm-intel.perfmon_mask=guest_gp=0-3;guest_fixed=0-1,3;perf_metrics +*/ +static int perfmon_mask_set (const char *val,const struct kernel_param *kp) +{ +unsigned long gp_bitmap =0 , fixed_bitmap =0 ; +char *buf, *orig, *tok; +size_t prefix_len; +u64 mask =0 ; +int r =0 ; + + buf = orig =kstrdup (val,GFP_KERNEL ); +if (!buf) +return -ENOMEM ; + +while ((tok =strsep (&buf,";" )) !=NULL ) { +if (!*tok) +continue ; + +if (!strcmp (tok,"perf_metrics" )) { + mask |=GLOBAL_STATUS_PERF_METRICS_OVF ; + }else if ((prefix_len =str_has_prefix (tok,"guest_gp=" ))) { + r =bitmap_parselist (tok + prefix_len, &gp_bitmap, +INTEL_PMC_MAX_GENERIC ); + }else if ((prefix_len =str_has_prefix (tok,"guest_fixed=" ))) { + r =bitmap_parselist (tok + prefix_len, &fixed_bitmap, +INTEL_PMC_MAX_FIXED ); + }else { + r = -EINVAL ; + } + +if (r) + goto out; + } + + mask |= gp_bitmap | ((u64 )fixed_bitmap <<INTEL_PMC_IDX_FIXED ); + *(u64 *)kp->arg = mask; +out: +kfree (orig); +return r; +} + +static const struct kernel_param_ops perfmon_mask_ops = { + .set = perfmon_mask_set, + .get = param_get_ullong, +}; +module_param_cb (perfmon_mask, &perfmon_mask_ops, &perfmon_mask,0444 ); +#define KVM_VM_CR0_ALWAYS_OFF (X86_CR0_NW | X86_CR0_CD)#define KVM_VM_CR0_ALWAYS_ON_UNRESTRICTED_GUEST X86_CR0_NE#define KVM_VM_CR0_ALWAYS_ON \@@ -8824,6 +8876,8 @@ __init int vmx_hardware_setup(void)else vt_init_ops.handle_intel_pt_intr =NULL ; +intel_pmu_perfmon_mask_setup (); +setup_default_sgx_lepubkeyhash ();vmx_set_cpu_caps ();arch/x86/kvm/vmx/vmx.h +1
-0@@ -667,6 +667,7 @@ static __always_inline struct vcpu_vmx *to_vmx(struct kvm_vcpu *vcpu)void intel_pmu_cross_mapped_check (struct kvm_pmu *pmu);int intel_pmu_create_guest_lbr_event (struct kvm_vcpu *vcpu);void vmx_passthrough_lbr_msrs (struct kvm_vcpu *vcpu); +void intel_pmu_perfmon_mask_setup (void );struct vmcs *alloc_vmcs_cpu (bool shadow,int cpu,gfp_t flags);void free_vmcs (struct vmcs *vmcs);arch/x86/kvm/x86.c +1
-1@@ -9328,7 +9328,7 @@ int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id)if (enable_mediated_pmu && kvm->arch.enable_pmu && !kvm->arch.created_mediated_pmu) {if (irqchip_in_kernel (kvm)) { - r =perf_create_mediated_pmu (0 ); + r =perf_create_mediated_pmu (perfmon_mask);if (r) {pr_warn_ratelimited (PERF_MEDIATED_PMU_MSG );return r;
PATCH 13/23 KVM: x86/pmu: Set up the PERFMON_MASK VMCS field
In reply to Zide ChenProgram the VMCS PERFMON_MASK field when PerfMon masking is enabled. The field uses the same bit definitions as IA32_PERF_GLOBAL_STATUS and can therefore be derived from ~pmu->global_status_rsvd.
Expose IA32_PERF_CAPABILITIES.PERF_METRICS_AVAILABLE[15] only when PERFMON_MASK[48] is available to the guest.
When PerfMon masking is enabled, guest RDMSR/WRMSR accesses to PMU global MSRs are filtered by the mask, writes outside the mask cause an #GP(0), and RDPMC returns only guest-owned counter values.
Intentionally defer enabling PerfMon masking in VM-execution control until the remaining PerfMon masking support is in place, so that intermediate commits remain functional during bisection.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/include/asm/vmx.h | 2 ++ arch/x86/kvm/pmu.c | 4 ++-- arch/x86/kvm/pmu.h | 6 ++++++ arch/x86/kvm/vmx/pmu_intel.c | 26 ++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 6 +++++- 6 files changed, 42 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -582,6 +582,7 @@ struct kvm_pmu {u64 eventsel_rsvd;u64 raw_event_mask;u64 perf_metrics; +u64 perfmon_mask;struct kvm_pmc gp_counters[KVM_MAX_NR_GP_COUNTERS ];struct kvm_pmc fixed_counters[KVM_MAX_NR_FIXED_COUNTERS ];
arch/x86/include/asm/vmx.h +2 -0
@@ -275,6 +275,8 @@ enum vmcs_field {SHARED_EPT_POINTER =0x0000203C ,PID_POINTER_TABLE =0x00002042 ,PID_POINTER_TABLE_HIGH =0x00002043 , +PERFMON_MASK =0x00002054 , +PERFMON_MASK_HIGH =0x00002055 ,GUEST_PHYSICAL_ADDRESS =0x00002400 ,GUEST_PHYSICAL_ADDRESS_HIGH =0x00002401 , VMCS_LINK_POINTER = 0x00002800,
arch/x86/kvm/pmu.c +2 -2
@@ -1393,7 +1393,7 @@ void kvm_mediated_pmu_load(struct kvm_vcpu *vcpu)perf_pmu_partition_preload (); -perf_load_guest_context (false); +perf_load_guest_context (kvm_vcpu_has_perfmon_mask (vcpu));/* * Explicitly clear PERF_GLOBAL_CTRL, as "loading" the guest's context
@@ -1466,5 +1466,5 @@ void kvm_mediated_pmu_put(struct kvm_vcpu *vcpu)perf_put_guest_lvtpc (); -perf_put_guest_context (false); +perf_put_guest_context (kvm_vcpu_has_perfmon_mask (vcpu)); }
arch/x86/kvm/pmu.h +6 -0
@@ -89,6 +89,12 @@ static inline bool kvm_vcpu_has_mediated_pmu(struct kvm_vcpu *vcpu)return enable_mediated_pmu &&vcpu_to_pmu (vcpu)->version; } +static inline bool kvm_vcpu_has_perfmon_mask (struct kvm_vcpu *vcpu) +{ +return kvm_vcpu_has_mediated_pmu (vcpu) && +vcpu_to_pmu (vcpu)->perfmon_mask; +} +static inline unsigned long kvm_gp_pmc_mask (struct kvm_pmu *pmu) { return pmu->pmc_exists64 &
arch/x86/kvm/vmx/pmu_intel.c +26 -0
@@ -703,6 +703,21 @@ static void intel_pmu_refresh(struct kvm_vcpu *vcpu) pmu->pebs_enable_rsvd = ~kvm_gp_pmc_mask (pmu); } } + +if (kvm_vcpu_has_mediated_pmu (vcpu) && perfmon_mask) { + pmu->perfmon_mask = ~pmu->global_status_rsvd; + +/* +* The PerfMon mask for a particular guest must be a subset +* of the module-wide mask. This masks out the global bits +* (e.g. GLOBAL_STATUS_COND_CHG) that must be handled by the +* host and were removed from global_status_rsvd without +* checking perfmon_mask, and defends in depth against any +* other bits inadvertently granted to the guest. +*/ + pmu->perfmon_mask &= perfmon_mask; +vmcs_write64 (PERFMON_MASK , pmu->perfmon_mask); + } }static void intel_pmu_init (struct kvm_vcpu *vcpu)
@@ -739,6 +754,7 @@ static void intel_pmu_reset(struct kvm_vcpu *vcpu)struct kvm_pmu *pmu =vcpu_to_pmu (vcpu); pmu->perf_metrics =0 ; + pmu->perfmon_mask =0 ;intel_pmu_release_guest_lbr_event (vcpu); }
@@ -1021,6 +1037,16 @@ void intel_pmu_perfmon_mask_setup(void) perfmon_mask); perfmon_mask =0 ; } + +/* +* perfmon_mask represents the maximum resources that any guest may +* have. KVM chooses to expose fewer hardware resources to guests. +*/ +if (perfmon_mask) { + kvm_pmu_cap.cntr_mask64 &= perfmon_mask; + kvm_pmu_cap.fixed_cntr_mask64 &= + (perfmon_mask >>INTEL_PMC_IDX_FIXED ); + } }struct kvm_pmu_ops intel_pmu_ops__initdata = {
arch/x86/kvm/vmx/vmx.c +5 -1
@@ -4294,6 +4294,9 @@ static void vmx_recalc_pmu_msr_intercepts(struct kvm_vcpu *vcpu)if (!cpu_has_save_perf_global_ctrl ()) { vm_exit_controls_bits &= ~VM_EXIT_SAVE_IA32_PERF_GLOBAL_CTRL ; +/* Module parameter validation should already prevent this. */ +WARN_ON_ONCE (kvm_vcpu_has_perfmon_mask (vcpu)); +if (has_mediated_pmu)vmx_add_autostore_msr (vmx,MSR_CORE_PERF_GLOBAL_CTRL );else
@@ -8133,7 +8136,8 @@ static __init u64 vmx_get_perf_capabilities(void) perf_cap &= ~PERF_CAP_PEBS_BASELINE ; } -if (enable_mediated_pmu) +if (enable_mediated_pmu && + (!perfmon_mask || (perfmon_mask &GLOBAL_STATUS_PERF_METRICS_OVF ))) perf_cap |= kvm_host.perf_capabilities &PERF_CAP_PERF_METRICS ;return perf_cap;
Signature · 1 line
2.55.0
PATCH 14/23 KVM: x86/pmu, perf/x86: Update effective PMU partition mask
In reply to Zide ChenThe static system-wide mask x86_pmu.partition_mask defines the global maximum guest capability, while a guest may have its own mask, which is a subset of x86_pmu.partition_mask.
The per-guest mask, which on Intel is backed by the PERFMON_MASK VMCS field, is the effective mask while a vCPU is loaded. Whenever it changes -- e.g. during kvm_arch_vcpu_{load,put}() -- perf/x86 needs to be told so it can keep its own per-CPU copy in sync.
--- arch/x86/events/core.c | 8 ++++++++ arch/x86/include/asm/perf_event.h | 1 + arch/x86/kvm/pmu.c | 13 +++++++++++++ arch/x86/kvm/pmu.h | 3 +++ arch/x86/kvm/x86.c | 4 ++++ 5 files changed, 29 insertions(+)
arch/x86/events/core.c +8 -0
@@ -1883,6 +1883,14 @@ void perf_put_guest_lvtpc(void)apic_write (APIC_LVTPC ,APIC_DM_NMI ); }EXPORT_SYMBOL_FOR_KVM (perf_put_guest_lvtpc); + +void perf_set_current_partition_mask (u64 mask) +{ +struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); + + cpuc->partition_mask = mask &READ_ONCE (x86_pmu.partition_mask); +} +EXPORT_SYMBOL_FOR_KVM (perf_set_current_partition_mask);#endif /* CONFIG_PERF_GUEST_MEDIATED_PMU */ static int
arch/x86/include/asm/perf_event.h +1 -0
@@ -794,6 +794,7 @@ static inline void perf_check_microcode(void) { }extern void perf_pmu_partition_preload (void );extern void perf_load_guest_lvtpc (u32 guest_lvtpc);extern void perf_put_guest_lvtpc (void ); +extern void perf_set_current_partition_mask (u64 mask);#endif #if defined(CONFIG_PERF_EVENTS ) && defined(CONFIG_CPU_SUP_INTEL )
arch/x86/kvm/pmu.c +13 -0
@@ -46,6 +46,19 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(enable_mediated_pmu);u64 __read_mostly perfmon_mask;EXPORT_SYMBOL_FOR_KVM_INTERNAL (perfmon_mask); +void kvm_pmu_vcpu_load (struct kvm_vcpu *vcpu) +{ +struct kvm_pmu *pmu =vcpu_to_pmu (vcpu); + +perf_set_current_partition_mask (kvm_vcpu_has_perfmon_mask (vcpu) ? + pmu->perfmon_mask :0 ); +} + +void kvm_pmu_vcpu_put (struct kvm_vcpu *vcpu) +{ +perf_set_current_partition_mask (0 ); +} +struct kvm_x86_pmu_event_filter {__u32 action;__u32 nevents;
arch/x86/kvm/pmu.h +3 -0
@@ -70,6 +70,9 @@ void kvm_pmu_ops_update(const struct kvm_pmu_ops *pmu_ops);void kvm_handle_guest_mediated_pmi (void ); +void kvm_pmu_vcpu_load (struct kvm_vcpu *vcpu); +void kvm_pmu_vcpu_put (struct kvm_vcpu *vcpu); +static inline bool kvm_pmu_has_perf_global_ctrl (struct kvm_pmu *pmu) {/*
arch/x86/kvm/x86.c +4 -0
@@ -2538,6 +2538,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)kvm_request_l1tf_flush_l1d (); +kvm_pmu_vcpu_load (vcpu); +if (vcpu->scheduled_out && pmu->version && pmu->event_count) { pmu->need_cleanup = true;kvm_make_request (KVM_REQ_PMU , vcpu);
@@ -2686,6 +2688,8 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)kvm_x86_call (vcpu_put)(vcpu); vcpu->arch.last_host_tsc =rdtsc (); + +kvm_pmu_vcpu_put (vcpu); } static int kvm_vcpu_ioctl_get_lapic(struct kvm_vcpu *vcpu,
Signature · 1 line
2.55.0
PATCH 15/23 KVM: x86/pmu: Relax MSR intercept policy under PerfMon masking
In reply to Zide ChenWhen a PerfMon mask is configured for the guest, hardware already delivers #GP(0) on RDPMC for any counter or PERF_METRICS whose corresponding PerfMon mask bit is clear, so KVM does not need to intercept RDPMC, nor separately gate access in software, to enforce the same restriction.
No changes are needed in intel_emulate_rdpmc(): the existing checks already fully reflect the PerfMon mask restrictions for GP counters, fixed counters, and PERF_METRICS respectively.
Similarly, guest reads of IA32_PERF_GLOBAL_CTRL are subject to a logical AND with the PerfMon mask, and writes cause a #GP(0) if the guest attempts to set a bit whose corresponding bit in the PerfMon mask is clear.
--- arch/x86/kvm/pmu.c | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/x86/kvm/pmu.c +8 -0
@@ -808,6 +808,14 @@ static bool kvm_need_any_pmc_intercept(struct kvm_vcpu *vcpu)if (!kvm_vcpu_has_mediated_pmu (vcpu))return true; +/* +* When PerfMon mask is enabled, KVM need not intercept RDPMC or +* accesses to IA32_PERF_GLOBAL_CTRL, as hardware blocks or filters +* out access to non-guest-owned RDPMC indices or GLOBAL_CTRL bits. +*/ +if (kvm_vcpu_has_perfmon_mask (vcpu)) +return false; +/* * Note! Check *host* PMU capabilities, not KVM's PMU capabilities, as * KVM's capabilities are constrained based on KVM support, i.e. KVM's
Signature · 1 line
2.55.0
PATCH 16/23 KVM: x86/pmu: Handle FIXED_CTR_CTRL under PerfMon masking
In reply to Zide ChenPerfMon masking affects reads from and writes to this MSR. The effective mask is defined as bits 4n+3:4n and bits 4n+35:4n+32 being set if and only if bit 32+n of the PerfMon mask is set (for 0 <= n <= 7).
KVM doesn't support some fixed-counter features, and pmu->fixed_ctr_ctrl_rsvd is a subset of the effective mask above, so the existing gating in intel_pmu_set_msr() is sufficient even under PerfMon masking.
Upon guest PMU context load, OR the guest-owned bits with whatever the hardware currently holds for every fixed-counter index the guest does not own, before writing the result to the MSR.
During PMU context put, host-owned fixed counters may still be scheduled for system-wide host events, so the host-owned bits should remain unchanged.
--- arch/x86/kvm/vmx/pmu_intel.c | 40 +++++++++++++++++++++++++++++++++--- 1 file changed, 37 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/pmu_intel.c +37 -3
@@ -920,6 +920,33 @@ static void intel_pmu_write_global_ctrl(u64 global_ctrl)vmcs_write64 (GUEST_IA32_PERF_GLOBAL_CTRL , global_ctrl); } +static u64 intel_fixed_ctrl_host_bits (struct kvm_pmu *pmu) +{ +unsigned long fixed_mask; +u64 fixed_ctl; +int i; + +if (!kvm_vcpu_has_perfmon_mask (pmu_to_vcpu (pmu))) +return 0 ; + + fixed_mask =kvm_fixed_pmc_mask (pmu); + +rdmsrq (MSR_CORE_PERF_FIXED_CTR_CTRL , fixed_ctl); + +/* +* Use the full per-counter nibbles (bits 4n+3:4n and 4n+35:4n+32) to +* strip all bits belonging to guest-owned counters. +* +* pmu->fixed_ctr_ctrl_rsvd can't be used here since it can't gate bits +* that are not supported by KVM. +*/ +kvm_for_each_fixed_counter (i, fixed_mask) + fixed_ctl &= ~intel_fixed_bits_by_idx (i,GENMASK_ULL (3 ,0 ) | +GENMASK_ULL (35 ,32 )); + +return fixed_ctl; +} +static void intel_mediated_pmu_load (struct kvm_vcpu *vcpu) {struct kvm_pmu *pmu =vcpu_to_pmu (vcpu);
@@ -935,7 +962,8 @@ static void intel_mediated_pmu_load(struct kvm_vcpu *vcpu)if (pmu->global_status & toggle)wrmsrq (MSR_CORE_PERF_GLOBAL_STATUS_SET , pmu->global_status & toggle); -wrmsrq (MSR_CORE_PERF_FIXED_CTR_CTRL , pmu->fixed_ctr_ctrl_hw); +wrmsrq (MSR_CORE_PERF_FIXED_CTR_CTRL , + pmu->fixed_ctr_ctrl_hw |intel_fixed_ctrl_host_bits (pmu)); }static void intel_mediated_pmu_put (struct kvm_vcpu *vcpu)
@@ -953,9 +981,15 @@ static void intel_mediated_pmu_put(struct kvm_vcpu *vcpu)* Clear hardware FIXED_CTR_CTRL MSR to avoid information leakage and * also to avoid accidentally enabling fixed counters (based on guest * state) while running in the host, e.g. when setting global ctrl. +* +* Keep the host-owned counters unchanged. */ -if (pmu->fixed_ctr_ctrl_hw) -wrmsrq (MSR_CORE_PERF_FIXED_CTR_CTRL ,0 ); +if (pmu->fixed_ctr_ctrl_hw) { +u64 fixed_ctl =intel_fixed_ctrl_host_bits (pmu); + + fixed_ctl &= ~pmu->fixed_ctr_ctrl_hw; +wrmsrq (MSR_CORE_PERF_FIXED_CTR_CTRL , fixed_ctl); + }if (kvm_vcpu_has_perf_metrics (vcpu)) { pmu->perf_metrics =rdpmc (INTEL_PMC_FIXED_RDPMC_METRICS );
Signature · 1 line
2.55.0
Re: [PATCH 16/23] KVM: x86/pmu: Handle FIXED_CTR_CTRL under PerfMon masking
Quoted 46 lines · Zide Chen
PerfMon masking affects reads from and writes to this MSR. The effective mask is defined as bits 4n+3:4n and bits 4n+35:4n+32 being set if and only if bit 32+n of the PerfMon mask is set (for 0 <= n <= 7).
KVM doesn't support some fixed-counter features, and
pmu->fixed_ctr_ctrl_rsvdis a subset of the effective mask above, so the existing gating inintel_pmu_set_msr()is sufficient even under PerfMon masking.Upon guest PMU context load, OR the guest-owned bits with whatever the hardware currently holds for every fixed-counter index the guest does not own, before writing the result to the MSR.
During PMU context put, host-owned fixed counters may still be scheduled for system-wide host events, so the host-owned bits should remain unchanged.
Signed-off-by Zide Chen <[email protected]>--- arch/x86/kvm/vmx/pmu_intel.c | 40 +++++++++++++++++++++++++++++++++--- 1 file changed, 37 insertions(+), 3 deletions(-)arch/x86/kvm/vmx/pmu_intel.c +16
-0@@ -920,6 +920,33 @@ static void intel_pmu_write_global_ctrl(u64 global_ctrl)vmcs_write64 (GUEST_IA32_PERF_GLOBAL_CTRL , global_ctrl); } +static u64 intel_fixed_ctrl_host_bits (struct kvm_pmu *pmu) +{ +unsigned long fixed_mask; +u64 fixed_ctl; +int i; + +if (!kvm_vcpu_has_perfmon_mask (pmu_to_vcpu (pmu))) +return 0 ; + + fixed_mask =kvm_fixed_pmc_mask (pmu); + +rdmsrq (MSR_CORE_PERF_FIXED_CTR_CTRL , fixed_ctl); + +/* +* Use the full per-counter nibbles (bits 4n+3:4n and 4n+35:4n+32) to +* strip all bits belonging to guest-owned counters.
arch/x86/kvm/vmx/pmu_intel.c static void intel_pmu_write_global_ctrl(u64 global_ctrl) · line 948
Quoted 10 lines
arch/x86/kvm/vmx/pmu_intel.c +10
-0+* +* pmu->fixed_ctr_ctrl_rsvd can't be used here since it can't gate bits +* that are not supported by KVM. +*/ +kvm_for_each_fixed_counter (i, fixed_mask) + fixed_ctl &= ~intel_fixed_bits_by_idx (i,GENMASK_ULL (3 ,0 ) | +GENMASK_ULL (35 ,32 )); + +return fixed_ctl; +}
Strictly speaking, this should be a perf/x86/intel function and then exposed to KVM. Perf host side may also need this function.
Quoted 32 lines
arch/x86/kvm/vmx/pmu_intel.c +11
-3+static void intel_mediated_pmu_load (struct kvm_vcpu *vcpu) {struct kvm_pmu *pmu =vcpu_to_pmu (vcpu);@@ -935,7 +962,8 @@ static void intel_mediated_pmu_load(struct kvm_vcpu *vcpu)if (pmu->global_status & toggle)wrmsrq (MSR_CORE_PERF_GLOBAL_STATUS_SET , pmu->global_status & toggle); -wrmsrq (MSR_CORE_PERF_FIXED_CTR_CTRL , pmu->fixed_ctr_ctrl_hw); +wrmsrq (MSR_CORE_PERF_FIXED_CTR_CTRL , + pmu->fixed_ctr_ctrl_hw |intel_fixed_ctrl_host_bits (pmu)); }static void intel_mediated_pmu_put (struct kvm_vcpu *vcpu)@@ -953,9 +981,15 @@ static void intel_mediated_pmu_put(struct kvm_vcpu *vcpu)* Clear hardware FIXED_CTR_CTRL MSR to avoid information leakage and * also to avoid accidentally enabling fixed counters (based on guest * state) while running in the host, e.g. when setting global ctrl. +* +* Keep the host-owned counters unchanged. */ -if (pmu->fixed_ctr_ctrl_hw) -wrmsrq (MSR_CORE_PERF_FIXED_CTR_CTRL ,0 ); +if (pmu->fixed_ctr_ctrl_hw) { +u64 fixed_ctl =intel_fixed_ctrl_host_bits (pmu); + + fixed_ctl &= ~pmu->fixed_ctr_ctrl_hw; +wrmsrq (MSR_CORE_PERF_FIXED_CTR_CTRL , fixed_ctl); + }if (kvm_vcpu_has_perf_metrics (vcpu)) { pmu->perf_metrics =rdpmc (INTEL_PMC_FIXED_RDPMC_METRICS );
Re: [PATCH 16/23] KVM: x86/pmu: Handle FIXED_CTR_CTRL under PerfMon masking
Quoted 53 lines · Dapeng
Quoted 51 lines · Zide Chen
PerfMon masking affects reads from and writes to this MSR. The effective mask is defined as bits 4n+3:4n and bits 4n+35:4n+32 being set if and only if bit 32+n of the PerfMon mask is set (for 0 <= n <= 7).
KVM doesn't support some fixed-counter features, and
pmu->fixed_ctr_ctrl_rsvdis a subset of the effective mask above, so the existing gating inintel_pmu_set_msr()is sufficient even under PerfMon masking.Upon guest PMU context load, OR the guest-owned bits with whatever the hardware currently holds for every fixed-counter index the guest does not own, before writing the result to the MSR.
During PMU context put, host-owned fixed counters may still be scheduled for system-wide host events, so the host-owned bits should remain unchanged.
Signed-off-by Zide Chen <[email protected]>--- arch/x86/kvm/vmx/pmu_intel.c | 40 +++++++++++++++++++++++++++++++++--- 1 file changed, 37 insertions(+), 3 deletions(-)arch/x86/kvm/vmx/pmu_intel.c +21
-0@@ -920,6 +920,33 @@ static void intel_pmu_write_global_ctrl(u64 global_ctrl)vmcs_write64 (GUEST_IA32_PERF_GLOBAL_CTRL , global_ctrl); } +static u64 intel_fixed_ctrl_host_bits (struct kvm_pmu *pmu) +{ +unsigned long fixed_mask; +u64 fixed_ctl; +int i; + +if (!kvm_vcpu_has_perfmon_mask (pmu_to_vcpu (pmu))) +return 0 ; + + fixed_mask =kvm_fixed_pmc_mask (pmu); + +rdmsrq (MSR_CORE_PERF_FIXED_CTR_CTRL , fixed_ctl); + +/* +* Use the full per-counter nibbles (bits 4n+3:4n and 4n+35:4n+32) to +* strip all bits belonging to guest-owned counters. +* +* pmu->fixed_ctr_ctrl_rsvd can't be used here since it can't gate bits +* that are not supported by KVM. +*/ +kvm_for_each_fixed_counter (i, fixed_mask)
Quoted 8 lines
arch/x86/kvm/vmx/pmu_intel.c static void intel_pmu_write_global_ctrl(u64 global_ctrl) · line 948
Quoted 5 lines
arch/x86/kvm/vmx/pmu_intel.c +5
-0+ fixed_ctl &= ~intel_fixed_bits_by_idx (i,GENMASK_ULL (3 ,0 ) | +GENMASK_ULL (35 ,32 )); + +return fixed_ctl; +}Strictly speaking, this should be a perf/x86/intel function and then exposed to KVM. Perf host side may also need this function.
The code is slightly different. For example, fc0 is bit 0 in kvm_fixed_pmc_mask(), while it is bit 32 in the host case (x86_pmu_current_partition_mask()).
It does not seem worthwhile to refactor the code for reuse for such a small API.
PATCH 17/23 KVM: x86/pmu: Handle GLOBAL_CTRL under PerfMon masking
In reply to Zide ChenGuest writes must be gated, and guest reads masked, with pmu->global_ctrl_rsvd rather than pmu->perfmon_mask, because bits 63:52 in global_ctrl_rsvd are guaranteed to be cleared, and bits 51:0 are the complement between the two masks.
Since the host could schedule !exclude_guest events on host-owned resources in non-root mode, OR the host-owned bits into GUEST_IA32_PERF_GLOBAL_CTRL when loading guest PMU state. The host- owned bits are supposed to be set during perf_load_guest_context().
Add a host_global_ctrl parameter to the mediated_load() callback so that the host value can be available before the register is cleared. This allows intel_mediated_pmu_load() to preserve host-owned bits.
When caching pmu->global_ctrl at VM exit, mask the guest value with pmu->global_ctrl_rsvd to retain guest-owned bits only.
--- arch/x86/kvm/pmu.c | 7 ++++++- arch/x86/kvm/pmu.h | 2 +- arch/x86/kvm/svm/pmu.c | 2 +- arch/x86/kvm/vmx/pmu_intel.c | 13 ++++++++++++- arch/x86/kvm/vmx/vmx.c | 4 ++++ 5 files changed, 24 insertions(+), 4 deletions(-)
arch/x86/kvm/pmu.c +6 -1
@@ -1406,6 +1406,8 @@ static void kvm_pmu_load_guest_pmcs(struct kvm_vcpu *vcpu)void kvm_mediated_pmu_load (struct kvm_vcpu *vcpu) { +u64 host_global_ctrl =0 ; +if (!kvm_vcpu_has_mediated_pmu (vcpu) ||KVM_BUG_ON (!lapic_in_kernel (vcpu), vcpu->kvm))return ;
@@ -1429,13 +1431,16 @@ void kvm_mediated_pmu_load(struct kvm_vcpu *vcpu)* even for SVM to minimize the damage if a perf event is left enabled, * and to ensure a consistent starting state. */ +if (kvm_vcpu_has_perfmon_mask (vcpu)) +rdmsrq (kvm_pmu_ops.PERF_GLOBAL_CTRL, host_global_ctrl); +wrmsrq (kvm_pmu_ops.PERF_GLOBAL_CTRL,0 );perf_load_guest_lvtpc (kvm_lapic_get_reg (vcpu->arch.apic,APIC_LVTPC ));kvm_pmu_load_guest_pmcs (vcpu); -kvm_pmu_call (mediated_load)(vcpu); +kvm_pmu_call (mediated_load)(vcpu, host_global_ctrl); }static void kvm_pmu_put_guest_pmcs (struct kvm_vcpu *vcpu)
arch/x86/kvm/pmu.h +1 -1
@@ -39,7 +39,7 @@ struct kvm_pmu_ops {bool (*pmc_is_disabled_in_current_mode)(struct kvm_pmc *pmc);bool (*is_mediated_pmu_supported)(struct x86_pmu_capability *host_pmu); -void (*mediated_load)(struct kvm_vcpu *vcpu); +void (*mediated_load)(struct kvm_vcpu *vcpu,u64 host_global_ctrl);void (*mediated_put)(struct kvm_vcpu *vcpu);void (*write_global_ctrl)(u64 global_ctrl);
arch/x86/kvm/svm/pmu.c +1 -1
@@ -246,7 +246,7 @@ static bool amd_pmu_is_mediated_pmu_supported(struct x86_pmu_capability *host_pmreturn host_pmu->version >=2 ; } -static void amd_mediated_pmu_load (struct kvm_vcpu *vcpu) +static void amd_mediated_pmu_load (struct kvm_vcpu *vcpu,u64 host_global_ctrl) {struct kvm_pmu *pmu =vcpu_to_pmu (vcpu);u64 global_status;
arch/x86/kvm/vmx/pmu_intel.c +12 -1
@@ -947,11 +947,22 @@ static u64 intel_fixed_ctrl_host_bits(struct kvm_pmu *pmu)return fixed_ctl; } -static void intel_mediated_pmu_load (struct kvm_vcpu *vcpu) +static void intel_mediated_pmu_load (struct kvm_vcpu *vcpu,u64 host_global_ctrl) {struct kvm_pmu *pmu =vcpu_to_pmu (vcpu);u64 global_status, toggle; +/* +* Preserve host-owned bits: perf may schedule !exclude_guest events on +* host-owned counters in non-root mode. +* PerfMon masking requires VM_EXIT_SAVE_IA32_PERF_GLOBAL_CTRL, so the +* MSR-store/load path does not apply here. +*/ +if (kvm_vcpu_has_perfmon_mask (vcpu)) { + host_global_ctrl &= pmu->global_ctrl_rsvd; +intel_pmu_write_global_ctrl (pmu->global_ctrl | host_global_ctrl); + } +if (kvm_vcpu_has_perf_metrics (vcpu))wrmsrq (MSR_PERF_METRICS , pmu->perf_metrics);
arch/x86/kvm/vmx/vmx.c +4 -0
@@ -7467,6 +7467,10 @@ static void vmx_refresh_guest_perf_global_control(struct kvm_vcpu *vcpu) } pmu->global_ctrl =vmcs_read64 (GUEST_IA32_PERF_GLOBAL_CTRL ); + +/* Strip host-owned bits that were ORed into the VMCS on VMX entry. */ +if (kvm_vcpu_has_perfmon_mask (vcpu)) + pmu->global_ctrl &= ~pmu->global_ctrl_rsvd; } void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp)
Signature · 1 line
2.55.0
PATCH 18/23 KVM: x86/pmu: Handle GLOBAL_STATUS MSRs under PerfMon masking
In reply to Zide ChenFor IA32_PERF_GLOBAL_STATUS{,_SET,_RESET}, the effective mask under PerfMon masking is perfmon_mask. Writes that set bits outside the effective mask must #GP, so use ~perfmon_mask as the effective reserved-bit mask instead of global_status_rsvd.
At guest context load, keep host-owned GLOBAL_STATUS bits unchanged because system-wide events may be scheduled on host-owned resources while running in VMX non-root mode.
Likewise, at guest context put, preserve host-owned bits and clear only the guest-owned subset from hardware GLOBAL_STATUS, leaving pmu->global_status containing only guest-owned bits.
--- arch/x86/kvm/pmu.c | 13 ++++++++----- arch/x86/kvm/vmx/pmu_intel.c | 17 ++++++++++++++++- 2 files changed, 24 insertions(+), 6 deletions(-)
arch/x86/kvm/pmu.c +8 -5
@@ -922,12 +922,15 @@ int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)struct kvm_pmu *pmu =vcpu_to_pmu (vcpu);u32 msr = msr_info->index;u64 data = msr_info->data; -u64 diff; +u64 global_status_rsvd, diff;/* * Note, AMD ignores writes to reserved bits and read-only PMU MSRs, * whereas Intel generates #GP on attempts to write reserved/RO MSRs. */ + global_status_rsvd =kvm_vcpu_has_perfmon_mask (vcpu) ? + ~pmu->perfmon_mask : pmu->global_status_rsvd; +switch (msr) {case MSR_CORE_PERF_GLOBAL_STATUS :if (!msr_info->host_initiated)
@@ -938,7 +941,7 @@ int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)if (!msr_info->host_initiated)break ; -if (data & pmu->global_status_rsvd) +if (data & global_status_rsvd)return 1 ; pmu->global_status = data;
@@ -967,7 +970,7 @@ int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)* GLOBAL_OVF_CTRL, a.k.a. GLOBAL STATUS_RESET, clears bits in * GLOBAL_STATUS, and so the set of reserved bits is the same. */ -if (data & pmu->global_status_rsvd) +if (data & global_status_rsvd)return 1 ; fallthrough;case MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR :
@@ -975,14 +978,14 @@ int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) pmu->global_status &= ~data;break ;case MSR_CORE_PERF_GLOBAL_STATUS_SET : -if (data & pmu->global_status_rsvd) +if (data & global_status_rsvd)return 1 ;if (!msr_info->host_initiated) pmu->global_status |= data;break ;case MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_SET :if (!msr_info->host_initiated) - pmu->global_status |= data & ~pmu->global_status_rsvd; + pmu->global_status |= data & ~global_status_rsvd;break ;default :kvm_pmu_mark_pmc_in_use (vcpu, msr_info->index);
arch/x86/kvm/vmx/pmu_intel.c +16 -1
@@ -968,6 +968,14 @@ static void intel_mediated_pmu_load(struct kvm_vcpu *vcpu, u64 host_global_ctrl)rdmsrq (MSR_CORE_PERF_GLOBAL_STATUS , global_status); toggle = pmu->global_status ^ global_status; + +/* +* Restrict OVF_CTRL/STATUS_SET writes to guest-owned bits under +* PerfMon masking. +*/ +if (kvm_vcpu_has_perfmon_mask (vcpu)) + toggle &= pmu->perfmon_mask; +if (global_status & toggle)wrmsrq (MSR_CORE_PERF_GLOBAL_OVF_CTRL , global_status & toggle);if (pmu->global_status & toggle)
@@ -984,7 +992,14 @@ static void intel_mediated_pmu_put(struct kvm_vcpu *vcpu)/* MSR_CORE_PERF_GLOBAL_CTRL is already saved at VM-exit. */ rdmsrq (MSR_CORE_PERF_GLOBAL_STATUS , pmu->global_status); -/* Clear hardware MSR_CORE_PERF_GLOBAL_STATUS MSR, if non-zero. */ +/* +* Clear only the guest-owned bits from the hardware GLOBAL_STATUS +* if any are set. pmu->global_status is then left holding just the +* guest-owned subset. +*/ +if (kvm_vcpu_has_perfmon_mask (vcpu)) + pmu->global_status &= pmu->perfmon_mask; +if (pmu->global_status)wrmsrq (MSR_CORE_PERF_GLOBAL_OVF_CTRL , pmu->global_status);
Signature · 1 line
2.55.0
PATCH 19/23 KVM: x86/pmu: Always intercept GLOBAL_INUSE under PerfMon masking
In reply to Zide ChenUnlike IA32_PERF_GLOBAL_CTRL, hardware does not filter GLOBAL_INUSE reads based on the PerfMon mask at all, so always intercept this MSR and filter out host-owned bits in KVM.
--- arch/x86/kvm/vmx/vmx.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +8 -1
@@ -4339,8 +4339,15 @@ static void vmx_recalc_pmu_msr_intercepts(struct kvm_vcpu *vcpu) MSR_TYPE_RW, intercept);vmx_set_intercept_for_msr (vcpu,MSR_CORE_PERF_GLOBAL_STATUS_SET ,MSR_TYPE_RW , intercept || pmu->version <4 ); + +/* +* IA32_PERF_GLOBAL_INUSE reads are not subject to PerfMon mask +* filtering. +*/ vmx_set_intercept_for_msr (vcpu,MSR_CORE_PERF_GLOBAL_INUSE , -MSR_TYPE_RW , intercept || pmu->version <4 ); +MSR_TYPE_RW , + intercept ||kvm_vcpu_has_perfmon_mask (vcpu) || + pmu->version <4 ); intercept = !has_mediated_pmu || !kvm_vcpu_has_perf_metrics (vcpu); vmx_set_intercept_for_msr(vcpu, MSR_PERF_METRICS,
Signature · 1 line
2.55.0
PATCH 20/23 KVM: x86/pmu: Request guest PMI for guest-induced PMIs
In reply to Zide ChenWhen PMU partitioning is enabled, guest-induced PMIs that occur in guest mode may be delivered to the host perf PMI handler, resulting in a VM-Exit.
intel_mediated_pmu_put() reads the hardware GLOBAL_STATUS MSR into pmu->global_status and masks it to the subset owned by the guest. Request a guest PMI when the saved status indicates a pending guest PMI, i.e. when
- an overflow status bit (47:0) is set for a counter whose PMI-enable bit is enabled; or
- a miscellaneous status bit without a corresponding PMI-enable control is set.
--- arch/x86/kvm/pmu.c | 3 ++- arch/x86/kvm/pmu.h | 1 + arch/x86/kvm/vmx/pmu_intel.c | 42 ++++++++++++++++++++++++++++++++++++ 3 files changed, 45 insertions(+), 1 deletion(-)
arch/x86/kvm/pmu.c +2 -1
@@ -1112,7 +1112,7 @@ void kvm_pmu_destroy(struct kvm_vcpu *vcpu)kvm_pmu_reset (vcpu); } -static bool pmc_is_pmi_enabled (struct kvm_pmc *pmc) +bool pmc_is_pmi_enabled (struct kvm_pmc *pmc) {u8 fixed_ctr_ctrl;
@@ -1123,6 +1123,7 @@ static bool pmc_is_pmi_enabled(struct kvm_pmc *pmc) pmc->idx - KVM_FIXED_PMC_BASE_IDX);return fixed_ctr_ctrl &INTEL_FIXED_0_ENABLE_PMI ; } +EXPORT_SYMBOL_FOR_KVM_INTERNAL (pmc_is_pmi_enabled);static void kvm_pmu_incr_counter (struct kvm_pmc *pmc) {
arch/x86/kvm/pmu.h +1 -0
@@ -336,6 +336,7 @@ static inline bool kvm_vcpu_has_perf_metrics(struct kvm_vcpu *vcpu)return kvm_vcpu_get_perf_caps (vcpu) &PERF_CAP_PERF_METRICS ; } +bool pmc_is_pmi_enabled (struct kvm_pmc *pmc);void kvm_pmu_deliver_pmi (struct kvm_vcpu *vcpu);int kvm_pmu_rdpmc (struct kvm_vcpu *vcpu,unsigned pmc,u64 *data);int kvm_pmu_check_rdpmc_early (struct kvm_vcpu *vcpu,unsigned int idx);
arch/x86/kvm/vmx/pmu_intel.c +42 -0
54 lines not on this page · raw message
Signature · 1 line
2.55.0
PATCH 21/23 KVM: x86/pmu: Enable PerfMon masking
In reply to Zide ChenPerfMon masking is enumerated by bit [17] of IA32_VMX_PROCBASED_CTLS3 (MSR 0x492), and it can be enabled by the new tertiary VM-execution control bit 17.
Add TERTIARY_EXEC_PERFMON_MASK_ENABLE to KVM's optional tertiary VM-execution controls, allowing cpu_has_vmx_perfmon_mask() to return true on PerfMon masking capable platforms.
With this in place, a valid perfmon_mask module parameter is no longer forced to zero, pmu->perfmon_mask can be configured as intended, and PerfMon masking can be enabled in hardware.
perf_create_mediated_pmu() can now pass a valid PerfMon mask to the perf subsystem, enabling PMU partitioning aware event scheduling.
--- arch/x86/include/asm/vmx.h | 1 + arch/x86/include/asm/vmxfeatures.h | 1 + arch/x86/kvm/vmx/capabilities.h | 3 ++- arch/x86/kvm/vmx/vmx.c | 13 +++++++++++++ arch/x86/kvm/vmx/vmx.h | 2 +- 5 files changed, 18 insertions(+), 2 deletions(-)
arch/x86/include/asm/vmx.h +1 -0
@@ -96,6 +96,7 @@ struct vmcs {* Definitions of Tertiary Processor-Based VM-Execution Controls. */ #define TERTIARY_EXEC_IPI_VIRT VMCS_CONTROL_BIT(IPI_VIRT) +#define TERTIARY_EXEC_PERFMON_MASK_ENABLE VMCS_CONTROL_BIT(PERFMON_MASK_ENABLE)#define PIN_BASED_EXT_INTR_MASK VMCS_CONTROL_BIT(INTR_EXITING)#define PIN_BASED_NMI_EXITING VMCS_CONTROL_BIT(NMI_EXITING)
arch/x86/include/asm/vmxfeatures.h +1 -0
@@ -90,4 +90,5 @@/* Tertiary Processor-Based VM-Execution Controls, word 3 */ #define VMX_FEATURE_IPI_VIRT ( 3*32+ 4)/* "ipi_virt" Enable IPI virtualization */ +#define VMX_FEATURE_PERFMON_MASK_ENABLE ( 3*32+ 17)/* "perfmon_mask_enable" Enable PERFMON_MASK */ #endif /* _ASM_X86_VMXFEATURES_H */
arch/x86/kvm/vmx/capabilities.h +2 -1
@@ -298,7 +298,8 @@ static inline bool cpu_has_vmx_ipiv(void)static inline bool cpu_has_vmx_perfmon_mask (void ) { -return false; +return vmcs_config.cpu_based_3rd_exec_ctrl & +TERTIARY_EXEC_PERFMON_MASK_ENABLE ; }static inline bool cpu_has_vmx_flexpriority (void )
arch/x86/kvm/vmx/vmx.c +13 -0
@@ -4288,6 +4288,14 @@ static void vmx_recalc_pmu_msr_intercepts(struct kvm_vcpu *vcpu)bool intercept = !has_mediated_pmu;int i; +/* +* Keep the PerfMon masking tertiary VM-execution control in sync +* regardless of enable_mediated_pmu. +*/ +if (cpu_has_vmx_perfmon_mask ()) +tertiary_exec_controls_changebit (vmx,TERTIARY_EXEC_PERFMON_MASK_ENABLE , +kvm_vcpu_has_perfmon_mask (vcpu)); +if (!enable_mediated_pmu)return ;
@@ -4766,6 +4774,9 @@ static u64 vmx_tertiary_exec_control(struct vcpu_vmx *vmx)if (!enable_ipiv || !kvm_vcpu_apicv_active (&vmx->vcpu)) exec_control &= ~TERTIARY_EXEC_IPI_VIRT ; +if (!kvm_vcpu_has_perfmon_mask (&vmx->vcpu)) + exec_control &= ~TERTIARY_EXEC_PERFMON_MASK_ENABLE ; +return exec_control; }
@@ -6754,6 +6765,8 @@ void dump_vmcs(struct kvm_vcpu *vcpu)if (secondary_exec_control &SECONDARY_EXEC_ENABLE_VPID )pr_err ("Virtual processor ID = 0x%04x\n" ,vmcs_read16 (VIRTUAL_PROCESSOR_ID )); +if (tertiary_exec_control &TERTIARY_EXEC_PERFMON_MASK_ENABLE ) +pr_err ("PERFMON_MASK = 0x%016llx\n" ,vmcs_read64 (PERFMON_MASK ));if (secondary_exec_control &SECONDARY_EXEC_EPT_VIOLATION_VE ) {struct vmx_ve_information *ve_info = vmx->ve_info;u64 ve_info_pa =vmcs_read64 (VE_INFORMATION_ADDRESS );
arch/x86/kvm/vmx/vmx.h +1 -1
@@ -577,7 +577,7 @@ static inline u8 vmx_get_rvi(void)#define KVM_REQUIRED_VMX_TERTIARY_VM_EXEC_CONTROL 0#define KVM_OPTIONAL_VMX_TERTIARY_VM_EXEC_CONTROL \ - (TERTIARY_EXEC_IPI_VIRT) + (TERTIARY_EXEC_IPI_VIRT | TERTIARY_EXEC_PERFMON_MASK_ENABLE)#define BUILD_CONTROLS_SHADOW (lname, uname, bits) \ static inline void lname##_controls_set(struct vcpu_vmx *vmx, u##bits val) \
Signature · 1 line
2.55.0
PATCH 22/23 KVM: selftests: Fix PERF_METRICS test by checking FC3 availability
In reply to Zide ChenIA32_PERF_CAPABILITIES.PERF_METRICS_AVAILABLE[bit 15] is not tied to a particular PMU version, and userspace can enable it as long as it is advertised.
On a PERF_METRICS capable host that supports fixed counter 3 through a non-contiguous fixed counter bitmap, userspace may enable this capability while configuring a guest PMU version lower than 5.
This scenario is unlikely on bare-metal systems, but it can occur with PMU partitioning. For example, when kvm-intel is loaded with:
perfmon_mask=guest_fixed=1-3;perf_metrics
CPUID.0AH:ECX[4:0] is 0, and thus fixed counter 3 is not available when the guest PMU version is below 5. Skip the test in this case.
--- tools/testing/selftests/kvm/x86/pmu_counters_test.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
tools/testing/selftests/kvm/x86/pmu_counters_test.c +2 -1
@@ -678,7 +678,8 @@ static void __guest_test_perf_metrics(void)if ((guest_get_pmu_version () <2 ) ||/* Does guest have GLOBAL_CTRL? */ !this_cpu_has (X86_FEATURE_PDCM ) || - !(rdmsr (MSR_IA32_PERF_CAPABILITIES ) &PERF_CAP_PERF_METRICS )) + !(rdmsr (MSR_IA32_PERF_CAPABILITIES ) &PERF_CAP_PERF_METRICS ) || + !this_pmu_has (X86_PMU_FEATURE_TOPDOWN_SLOTS_FIXED ))return ;wrmsr (MSR_CORE_PERF_GLOBAL_CTRL ,0 );
Signature · 1 line
2.55.0
PATCH 23/23 KVM: selftests: Allow no general purpose counters on the host
In reply to Zide ChenDo not GUEST_ASSERT(nr_gp_counters). Instead, skip the architectural event tests that require general-purpose counters when none are available.
In theory, guest CPUID.0AH:EAX[15:08] can be configured to report zero when the selftest runs on L1. In practice, this scenario is more likely to occur when PMU partitioning limits the number of available general-purpose counters.
--- .../selftests/kvm/x86/pmu_counters_test.c | 26 +++++++++++++------ 1 file changed, 18 insertions(+), 8 deletions(-)
tools/testing/selftests/kvm/x86/pmu_counters_test.c +18 -8
@@ -282,19 +282,16 @@ static void __guest_test_arch_event(u8 idx, u32 pmc, u32 pmc_msr,GUEST_TEST_EVENT (idx, pmc, pmc_msr, ctrl_msr, ctrl_msr_value,KVM_FEP ); } -static void guest_test_arch_event (u8 idx) +static void guest_test_arch_gp_event (u8 idx,bool guest_has_perf_global_ctrl) {u32 nr_gp_counters =this_cpu_property (X86_PROPERTY_PMU_NR_GP_COUNTERS ); -u32 pmu_version =guest_get_pmu_version (); -/* PERF_GLOBAL_CTRL exists only for Architectural PMU Version 2+. */ -bool guest_has_perf_global_ctrl = pmu_version >=2 ; -struct kvm_x86_pmu_feature gp_event, fixed_event; +struct kvm_x86_pmu_feature gp_event;u32 base_pmc_msr;unsigned int i;u64 eventsel; -/* The host side shouldn't invoke this without a guest PMU. */ -GUEST_ASSERT (pmu_version); +if (!nr_gp_counters) +return ;if (this_cpu_has (X86_FEATURE_PDCM ) &&rdmsr (MSR_IA32_PERF_CAPABILITIES ) &PMU_CAP_FW_WRITES )
@@ -305,7 +302,6 @@ static void guest_test_arch_event(u8 idx) gp_event =intel_event_to_feature (idx).gp_event;GUEST_ASSERT_EQ (idx, gp_event.f.bit); -GUEST_ASSERT (nr_gp_counters); i =kvm_random_u32_in_range (&kvm_rng,0 , nr_gp_counters -1 ); eventsel = ARCH_PERFMON_EVENTSEL_OS | ARCH_PERFMON_EVENTSEL_ENABLE |
@@ -316,6 +312,20 @@ static void guest_test_arch_event(u8 idx)wrmsr (MSR_CORE_PERF_GLOBAL_CTRL ,BIT_ULL (i));__guest_test_arch_event (idx, i, base_pmc_msr + i,MSR_P6_EVNTSEL0 + i, eventsel); +} + +static void guest_test_arch_event (u8 idx) +{ +u32 pmu_version =guest_get_pmu_version (); +/* PERF_GLOBAL_CTRL exists only for Architectural PMU Version 2+. */ +bool guest_has_perf_global_ctrl = pmu_version >=2 ; +struct kvm_x86_pmu_feature fixed_event; +unsigned int i; + +/* The host side shouldn't invoke this without a guest PMU. */ +GUEST_ASSERT (pmu_version); + +guest_test_arch_gp_event (idx, guest_has_perf_global_ctrl);if (!guest_has_perf_global_ctrl)return ;
Signature · 1 line
2.55.0
Quoted 68 lines · Zide Chen
Introduction
============PMU partitioning allows a VMM to assign a subset of PMU resources to a guest while retaining the remainder for host use, enabling concurrent host and guest PMU usage while the guest is running.
Intel PerfMon masking provides architectural support for PMU partitioning. This VMX extension provides fine-grained, bit-level control over global PerfMon MSRs and RDPMC, allowing a guest to natively access PMU resources while hardware ensures that guest accesses cannot affect host-owned resources. This eliminates the need to intercept those accesses for PMU isolation.
The feature is enumerated by IA32_VMX_PROCBASED_CTLS3[17] and enabled via the new PerfMon masking VM-execution control in tertiary processor-based VM-execution controls.
A new 64-bit VMCS field,
PERFMON_MASK(encoding 0x2054/0x2055), defines the guest-owned PMU resources. The mask follows theIA32_PERF_GLOBAL_STATUSbit layout.When enabled, hardware enforces guest access to selected PerfMon MSRs and RDPMC according to
PERFMON_MASK, reducing the need for software emulation and interception.The full description of PerfMon masking can be found at Intel ISA, 319433-062 [1]. Diamond Rapids is the first platform to support this feature.
Design
======1. KVM
This series builds PMU partitioning on top of the mediated vPMU, which allows guest access to PerfMon MSRs without emulation, making it a natural foundation for PerfMon masking.
A new Intel-specific KVM module parameter,
perfmon_mask, defines the host-wide PMU partition. The mask uses thePERFMON_MASKVMCS encoding, where set bits identify guest-owned PMU resources and cleared bits identify host-owned resources. The setting is shared by all mediated vPMU guests on the host.KVM constrains its PMU capability based on the
perfmon_maskparameter, so only minimal additional changes are needed to manage PMU capability advertisement and MSR emulation.Upon VMX transition, unlike the non-partitioned mediated vPMU, KVM always preserves the host-owned bits in the global MSRs.
This series does not yet implement the full PerfMon masking software guideline in section 10.5 of [1]. For example, the RELOAD_{PMCx,FIXED_CTRx} and IA32_PMC_GPn_CFG_{A,B,C} MSRs are not yet supported by KVM.
2. perf core
The perf scheduler is taught to honor the PMU partition mask:
- Loading guest context: reschedule system-wide events onto host-owned counters.
- Putting guest context: schedule out system-wide events before rescheduling in so that they can be scheduled on now more available counters.
- In guest context, don't schedule inexclude_guestevents, but allow !exclude_guest to be scheduled in.
Quoted 3 lines
As a current limitation, !exclude_guest events that rely on exclusive PMU facilities (e.g. Intel PT, LBR, or PEBS) are not supported under PMU partitioning and are rejected.
Hmm, scheduling events on each vmx transaction (especially twice events scheduling) would introduce performance overhead. Strictly speaking, current design looks like a dynamic PMU partition, host events can be scheduled to the counters which are allocated to guest in root mode.
It's fine on functionality, but I'm not sure how much benefits can be gotten from the host events rescheduling on each vmx transaction by paying such performance penalty. According to what I know, most scenarios could only need 1 or 2 monitor events on host and it would be good enough to keep 1 or 2 counters for host. In such case, the events scheduling on each vmx transaction becomes unnecessary.
An alternative way is statically separate the PMU counters for host and guest, e.g. host occupies the counters 0 and 1, and guest has the left counters 2 - 7. Host events can only be scheduled on events 0 or 1 and guest events own the counters 2 -7. In this way, the host events scheduling can be thoroughly eliminated and the whole design and implementation can become simpler.
Thanks.
Quoted 230 lines
3. perf/x86
Apply PMU partitioning constraints to PMC allocation: while a partitioned guest is running, guest-owned counters are reserved for guest use and are unavailable to host event scheduling. Outside guest execution, all counters remain available to the host.
In PMI handling in guest mode, handle only the host-owned bits that are set in
IA32_PERF_GLOBAL_STATUS, and leave the guest-owned bits for KVM.4. PMI design
Unlike mediated vPMU, PMU partitioning does not switch LVTPC to a dedicated guest PMI vector. Host-owned counters remain active while the guest is running and may trigger PMIs.
PMIs therefore remain routed through the host NMI handler. PMI handling is split: host-induced counter overflows are handled by the host perf NMI handler as usual, while guest-induced overflows are left pending and KVM injects the corresponding PMI into the guest.
Related Work
============ARM's PMU Partitioning series [2] solves a similar problem. It relies on MDCR_EL2.HPMN to enforce the host/guest counter boundary and lets the guest access its counters without trapping.
Similar to the
perfmon_maskparameter, ARM's series has a module parameter,reserved_host_counters, that defines the maximum number of guest counters; userspace VMMs may then configure a smaller number of counters for each guest, up to that host-wide maximum.Probably the biggest difference is that ARM's series chooses to context-switch guest/host PMU state only at vcpu_{load,put}(), and evict host events from would-be guest-owned counters if needed, while this series continues the mediated vPMU design and performs it on every VM-entry/VM-exit.
Also, while a guest is running, this series allows host events to be scheduled on exactly the complement of the effective guest counter mask, whereas ARM's series uses the single, global
reserved_host_countersvalue, regardless of how many counters any given guest is actually using.Tests
=====Tested in the engineering environment with various
perfmon_maskconfigurations.1. Passed KVM selftests on L0 and L1.
2. No issues related to PMU partitioning were found by
perf testrunning on L0 and L1.3. Host
exclude_guestevents do not count non-root activity:
# The following host command reports similar counts with or without# guest workloads. $ perf stat -e instructions:H,cycles:H,branches:H -a -I 10004. Host system-wide !exclude_guest events count non-root activity. # The following host command captures counts generated by guest
# workloads. $ perf stat -e instructions:HG,cycles:HG,branches:HG -a -I 10005. Put test 3 and 4 together.
# Guest idle.$ stat -C 1 -e cycles:GH,instructions:G,branches:H -a Performance counter stats for 'system wide':113,933,309 cycles:GH (100.00%) 135,454,451 instructions:G (100.00%) 26,966,970 branches:H1.959020017 seconds time elapsed# Ran sysbench on the guest.
$ stat -C 1 -e cycles:GH,instructions:G,branches:H -a Performance counter stats for 'system wide':5,012,158,432 cycles:GH (99.94%) 5,468,621,724 instructions:G (99.94%) 26,654,821 branches:H1.948625192 seconds time elapsed6. Host system-wide !exclude_guest sampling events observe guest-induced PMIs.
# When running a guest workload with a known PMI rate, the following# host command captures the corresponding overflow activity.# Guest idle.
$ perf record -e instructions:H,cycles:HG,branches:HG -c 1000000 -a -- sleep 10 [ perf record: Captured and wrote 0.322 MB perf.data (1432 samples) ] SAMPLE events: 1432 (30.4%) SAMPLE events: 730 SAMPLE events: 549 SAMPLE events: 153# Independantly ran workload on the guest to know the expected PMIs.
$ perf record -e instructions,cycles,branches -c1000000 tchain_edit [ perf record: Captured and wrote 0.176 MB perf.data (4536 samples) ] SAMPLE events: 4536 (99.0%) SAMPLE events: 2037 SAMPLE events: 1821 SAMPLE events: 678# Profile this on host again while ran
tchain_editon the guest.$ perf record -e instructions:H,cycles:HG,branches:HG -c 1000000 -a -- sleep 10 [ perf record: Captured and wrote 0.433 MB perf.data (3959 samples) ] SAMPLE events: 3959 (55.0%) SAMPLE events: 762 SAMPLE events: 2362 SAMPLE events: 8357. Perf Metrics tests.
# Ifperf_metricsis assigned to the guests, guests can run slots and# metrics events. Host exclude_host slots and metrics events are not # scheduled in non-root mode. $ perf stat -e '{topdown-bad-spec,topdown-retiring,topdown-be-bound,slots}:G' -a --sleep 1 Performance counter stats for 'system wide':0 slots (18.11%) 0 topdown-bad-spec (18.11%) 0 topdown-retiring (18.11%) 0 topdown-be-bound (18.11%)1.001590434 seconds time elapsed# Same events run on the host if
perf_metricsis not assigned to the# guests. $ perf stat -e '{topdown-bad-spec,topdown-retiring,topdown-be-bound,slots}:G' -a --sleep 1 Performance counter stats for 'system wide':14,632,663 slots (100.00%) 918,127 topdown-bad-spec (100.00%) 1,319,808 topdown-retiring (100.00%) 3,514,335 topdown-be-bound (100.00%)1.001938038 seconds time elapsed8. Some other sanity tests.
Patch Summary
=============
1. perf/x86 core enablement (patches 1-8)
Introduce guest-state tracking, host/guest PMI splitting, and the counter/event constraints the mask depends on.2. perf core scheduling (patches 9-11)
Propagate the partition mask into mediated-PMU VMs and teach the generic perf scheduler to honor it when loading/putting guest context.3. KVM PerfMon masking (patches 12-20)
perfmon_maskmodule parameter,PERFMON_MASKVMCS setup, MSR intercept relaxation, and GLOBAL_CTRL/GLOBAL_STATUS/GLOBAL_INUSE handling for the mediated vPMU under PerfMon masking.4. Enablement (patch 21)
Turn on PerfMon masking on supported Intel platforms.5. Selftests (patches 22-23)
Many thanks to Kan Liang for the original PerfMon masking design, Andi Kleen for his guidance, and Dapeng Mi for the many discussions, reviews, and suggestions that helped shape this series.
Rebased to kvm-x86/next:
1b731e5ded48
Thie series depends on the PMU v5 series [3] and Topdown metrics series [4].Code also available at [5].
[1]: https://www.intel.com/content/www/us/en/content-details/922690/intel-architecture-instruction-set-extensions-programming-reference.html
[2]: https://lore.kernel.org/lkml/[email protected]/T/
[3]: https://lore.kernel.org/kvm/[email protected]/
[4]: https://lore.kernel.org/kvm/[email protected]/
[5]: https://github.com/ZideChen0/linux-kernel/tree/pmu_partitioning_v1Kan Liang (3):
perf/x86: Restrict !exclude_guest events to host-owned counters
perf: Skipexclude_guestevents on PMU partitioned counters
perf: Reschedule events across PMU partition transitionsZide Chen (20):
perf/x86/intel: Guard counter masks against zero counters perf, perf/x86: Pass partition mask from KVM to perf/x86 perf/x86: AddGUEST_PMUstates for PMU partitioning perf/x86: Split host/guest PMI handling under PMU partitioning perf/x86: Allowexclude_hostevents to run in non-root mode perf/x86: Apply PMU partition mask on static constraints perf/x86: Export available PMU counters to sysfs
perf, perf/x86: Allow host !exclude_guest events in PMU partitioning
KVM: x86/pmu: Add theperfmon_maskmodule parameter
KVM: x86/pmu: Set up thePERFMON_MASKVMCS field
KVM: x86/pmu, perf/x86: Update effective PMU partition mask
KVM: x86/pmu: Relax MSR intercept policy under PerfMon masking
KVM: x86/pmu: HandleFIXED_CTR_CTRLunder PerfMon masking
KVM: x86/pmu: HandleGLOBAL_CTRLunder PerfMon masking
KVM: x86/pmu: HandleGLOBAL_STATUSMSRs under PerfMon masking
KVM: x86/pmu: Always interceptGLOBAL_INUSEunder PerfMon masking
KVM: x86/pmu: Request guest PMI for guest-induced PMIs
KVM: x86/pmu: Enable PerfMon masking
KVM: selftests: FixPERF_METRICStest by checking FC3 availability
KVM: selftests: Allow no general purpose counters on the host.../sysfs-bus-event_source-devices-caps | 5 + .../admin-guide/kernel-parameters.txt | 30 +++ arch/x86/events/core.c | 227 +++++++++++++++++- arch/x86/events/intel/core.c | 185 ++++++++++++-- arch/x86/events/perf_event.h | 20 ++ arch/x86/include/asm/kvm_host.h | 1 + arch/x86/include/asm/perf_event.h | 3 + arch/x86/include/asm/vmx.h | 3 + arch/x86/include/asm/vmxfeatures.h | 1 + arch/x86/kvm/pmu.c | 53 +++- arch/x86/kvm/pmu.h | 13 +- arch/x86/kvm/svm/pmu.c | 2 +- arch/x86/kvm/vmx/capabilities.h | 6 + arch/x86/kvm/vmx/pmu_intel.c | 206 +++++++++++++++- arch/x86/kvm/vmx/vmx.c | 86 ++++++- arch/x86/kvm/vmx/vmx.h | 3 +- arch/x86/kvm/x86.c | 6 +- include/linux/perf_event.h | 8 +- kernel/events/core.c | 122 ++++++++-- .../selftests/kvm/x86/pmu_counters_test.c | 29 ++- 20 files changed, 929 insertions(+), 80 deletions(-)