RFC KVM: x86/pmu: Let userspace disable SW accounting of emulated instructions
Get series with b4
b4 am [email protected]Versions newest first
- v1 RFC Latest2 patches 11 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/2 | KVM: x86/pmu: Add CAP to disable SW accounting of emulated instructionsarch/ arch/ arch/ +2 more | +53 | 5 replies | |
| 2/2 | KVM: Documentation: Document KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTINGDocumentation/ | +32 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.
Cover letter [RFC PATCH 0/2] KVM: x86/pmu: Let userspace disable SW accounting of emulated instructions
On hosts that run guests with an *emulated* vPMU -- guest PMU MSR accesses trap into KVM and each guest counter is backed by a host perf_event -- the per-emulated-instruction PMU accounting added by commit 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions") is pathologically expensive.
Whenever a guest programs a counter for retired instructions (0xc0) or retired branches (0xc2) -- routine for any profiler-style workload -- every instruction KVM emulates triggers a software increment plus a batched KVM_REQ_PMU reprogram. The reprogram is drained on the next VM-entry regardless of which exit preceded it, so ordinary exits (most importantly the guest's 1kHz timer tick) absorb a full reprogram_counter() cycle: ctx->mutex, ctx_resched(), and a burst of serialized PMU-MSR writes. That inflates timer-interrupt handling into the hundreds of microseconds and, under load, can escalate to a guest CSD lockup.
This series adds KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING, a VM-scoped capability that takes a bitmask of PERF_COUNT_HW_* event IDs and makes KVM skip the software accounting of emulated instructions for those events. It is opt-in and defaults off; the only cost to an opted-in guest is that instructions-retired / branches-retired counters undercount by the instructions KVM emulates in host context, i.e. the behavior that predates that accounting. Hardware-executed guest instructions are still counted by the backing perf_event and its overflow/PMI path is untouched.
Patch 1 adds the capability and the accounting short-circuit. Patch 2 documents it in Documentation/virt/kvm/api.rst.
Measured benefit
================
Host: AMD EPYC 7R13 "Milan", KVM, v7.2-rc4 + this series. Guest: QEMU, -cpu host,pmu=on, 8 vCPUs. The guest workload keeps an instructions- retired counter resident on every vCPU and periodically reprograms the PMU, driving the accounting storm above, while the vCPUs are kept busy so the 1kHz tick fires. The guest's local-APIC timer interrupt handler (__sysvec_apic_timer_interrupt) is traced with ftrace function_graph for 30s (~160k samples per run). The capability is toggled per-VM via KVM_ENABLE_CAP; nothing else changes between runs.
Timer-interrupt handler duration:
metric accounting ON (baseline) accounting OFF (CAP) speedup
------ ------------------------ -------------------- -------
mean 209.8 us 3.16 us 66x
p50 256.4 us 3.14 us 82x
p99 463.3 us 4.45 us 104x
p99.9 915.2 us 5.03 us 182x
max 929.7 us 22.55 us 41xSome workloads genuinely need the extra precision that the emulated- instruction accounting provides, and for them the default behavior is the right trade-off. But for many others the accuracy gained is not worth the cost it imposes on every exit, and this capability lets those users opt out of paying it.
Luka Absandze (2):
KVM: x86/pmu: Add CAP to disable SW accounting of emulated
instructions
KVM: Documentation: Document KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING
Documentation/virt/kvm/api.rst | 24 ++++++++++++++++++++++++ arch/x86/include/asm/kvm_host.h | 7 +++++++ arch/x86/kvm/pmu.c | 6 ++++++ arch/x86/kvm/pmu.h | 9 +++++++++ arch/x86/kvm/x86.c | 10 ++++++++++ include/uapi/linux/kvm.h | 1 + 6 files changed, 57 insertions(+)
Signature · 1 line
2.47.3