Skip to content

RFC KVM: x86/pmu: Let userspace disable SW accounting of emulated instructions

Luka Absandze kvm linux-doc lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest2 patches 11 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/2KVM: x86/pmu: Add CAP to disable SW accounting of emulated instructionsarch/x86/include/asm/kvm_host.h arch/x86/kvm/pmu.c arch/x86/kvm/pmu.h +2 more+53 -05 replies
2/2KVM: Documentation: Document KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTINGDocumentation/virt/kvm/api.rst+32 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 0/2] KVM: x86/pmu: Let userspace disable SW accounting of emulated instructions

On hosts that run guests with an *emulated* vPMU -- guest PMU MSR accesses trap into KVM and each guest counter is backed by a host perf_event -- the per-emulated-instruction PMU accounting added by commit 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions") is pathologically expensive.

Whenever a guest programs a counter for retired instructions (0xc0) or retired branches (0xc2) -- routine for any profiler-style workload -- every instruction KVM emulates triggers a software increment plus a batched KVM_REQ_PMU reprogram. The reprogram is drained on the next VM-entry regardless of which exit preceded it, so ordinary exits (most importantly the guest's 1kHz timer tick) absorb a full reprogram_counter() cycle: ctx->mutex, ctx_resched(), and a burst of serialized PMU-MSR writes. That inflates timer-interrupt handling into the hundreds of microseconds and, under load, can escalate to a guest CSD lockup.

This series adds KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING, a VM-scoped capability that takes a bitmask of PERF_COUNT_HW_* event IDs and makes KVM skip the software accounting of emulated instructions for those events. It is opt-in and defaults off; the only cost to an opted-in guest is that instructions-retired / branches-retired counters undercount by the instructions KVM emulates in host context, i.e. the behavior that predates that accounting. Hardware-executed guest instructions are still counted by the backing perf_event and its overflow/PMI path is untouched.

Patch 1 adds the capability and the accounting short-circuit. Patch 2 documents it in Documentation/virt/kvm/api.rst.

Measured benefit
================

Host: AMD EPYC 7R13 "Milan", KVM, v7.2-rc4 + this series. Guest: QEMU, -cpu host,pmu=on, 8 vCPUs. The guest workload keeps an instructions- retired counter resident on every vCPU and periodically reprograms the PMU, driving the accounting storm above, while the vCPUs are kept busy so the 1kHz tick fires. The guest's local-APIC timer interrupt handler (__sysvec_apic_timer_interrupt) is traced with ftrace function_graph for 30s (~160k samples per run). The capability is toggled per-VM via KVM_ENABLE_CAP; nothing else changes between runs.

Timer-interrupt handler duration:

    metric   accounting ON (baseline)   accounting OFF (CAP)   speedup
    ------   ------------------------   --------------------   -------
    mean            209.8 us                   3.16 us           66x
    p50             256.4 us                   3.14 us           82x
    p99             463.3 us                   4.45 us          104x
    p99.9           915.2 us                   5.03 us          182x
    max             929.7 us                  22.55 us           41x

Some workloads genuinely need the extra precision that the emulated- instruction accounting provides, and for them the default behavior is the right trade-off. But for many others the accuracy gained is not worth the cost it imposes on every exit, and this capability lets those users opt out of paying it.

Luka Absandze (2):
KVM: x86/pmu: Add CAP to disable SW accounting of emulated
instructions
KVM: Documentation: Document KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING

 Documentation/virt/kvm/api.rst  | 24 ++++++++++++++++++++++++
 arch/x86/include/asm/kvm_host.h |  7 +++++++
 arch/x86/kvm/pmu.c              |  6 ++++++
 arch/x86/kvm/pmu.h              |  9 +++++++++
 arch/x86/kvm/x86.c              | 10 ++++++++++
 include/uapi/linux/kvm.h        |  1 +
 6 files changed, 57 insertions(+)
Signature · 1 line
2.47.3