Skip to contentSkip to conversation

KVM: x86: per-VM C-state policy enforcement (KVM_CAP_CSTATE_POLICY)

RFC PATCH In review Anthony Harivel 3 participants 7 messages Sep 14 – Sep 15

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v3 cover KVM: x86: per-VM C-state policy enforcement (KVM_CAP_CSTATE_POLICY)

This is v3 of the RFC for KVM_CAP_CSTATE_POLICY, a new VM-scoped capability that lets userspace set a maximum C-state ceiling per VM.

Changes since v2 (RFC, design questions only):
- Implemented Option A from v2 design discussion: temporarily toggle cpuidle states_usage[].disable on the pinned pCPU before/after kvm_vcpu_block(). Uses CPUIDLE_STATE_DISABLED_BY_DRIVER flag.
- Added KVM selftest (7 tests, all passing on bare metal).
- Added QEMU prototype (not in this series -- separate tree).
- Answered design questions from v2 based on implementation experience.

Changes since v1:
- Corrected internal flow: enforcement goes through kvm_vcpu_halt() + cpuidle constraint, not inline MWAIT execution.
- Expanded design questions with concrete enforcement options.

== Problem ==

Cloud operators running NFV workloads want to reduce energy consumption by disabling halt-polling (halt_poll_ns=0). This lets vCPUs enter real idle states instead of busy-looping, saving power.

However, disabling halt-polling is system-wide and ALL VMs on the host lose halt-polling. The kernel cpuidle governor then picks deep C-states (C6, ~133us wakeup) by default, which hurts latency-sensitive VMs.

There is no per-VM mechanism to say: "save power on batch VMs, but keep wakeup latency low on this latency-sensitive VM."

Existing knobs are all system-wide or not VM-aware:
- intel_idle.max_cstate: system-wide kernel parameter
- /sys/devices/system/cpu/cpuN/cpuidle/stateN/disable: per-CPU, not VM-aware
- halt_poll_ns: system-wide, all or nothing

== Solution ==

KVM_CAP_CSTATE_POLICY: a VM-scoped ioctl (same pattern as KVM_CAP_HALT_POLL) that constrains cpuidle state selection when vCPUs halt.

args[0] = max_cstate:
-1 = disabled (default, no constraint)
0 = force C0 (HLT returns immediately, no idle) 1 = cap at C1 (~2us wakeup)
6 = cap at C6 (effectively unrestricted)

When a vCPU enters kvm_vcpu_block(), KVM disables cpuidle states deeper than max_cstate on the current pCPU (preempt-disabled), then restores after wakeup. The capability is re-callable at runtime -- policy changes take effect on the next vCPU HLT without VM restart.

== Test results ==

Tested on Dell R640 (Intel Xeon Gold 5118, intel_idle driver with POLL/C1/C1E/C6 states).

KVM selftest (7/7 pass):

ok 1 cap_supported
ok 2 valid_policies
ok 3 invalid_policies
ok 4 policy_change
ok 5 cpuidle_no_policy
ok 6 cpuidle_enforcement
ok 7 policy_c0_no_idle

Selftest cpuidle counters (1000 guest HLTs per test):

                  No policy     Policy=C1     Policy=C0
  POLL usage:     +0            +0            +56,743
  C1 usage:       +3            +1,073        +0
  C1E usage:      +70           +0            +0
  C6 usage:       +1,003        +0            +0

Multi-VM demo (2 VMs, 60s, same host, turbostat):

                  VM-A (policy=C1)    VM-B (no policy)
  C1 residency:   98-99%              0%
  C6 residency:   0%                  54-99%

The enforcement is clear: VM-A stays in C1 while VM-B on the same host goes deep into C6.

== Design decisions (answers to v2 questions) ==

(a) Enforcement mechanism: Option A (toggle states_usage[].disable) works correctly for pinned-core NFV configurations. The disable flags are set/cleared with preemption disabled, so no migration race. A WARN_ON_ONCE guards against unexpected CPU changes.

(b) VMCS toggling: not needed for v1 -- enforcement works within the existing HLT exit path. MWAIT interception is a separate concern for a future series if cpu-pm=on integration is desired.

(c) MWAIT hint mapping: not needed for v1 -- the policy operates on cpuidle state indices, which the intel_idle driver maps to hardware C-states. This is the same abstraction sysfs uses.

(d) halt_poll_ns interaction: orthogonal as expected. halt_poll_ns controls spin duration before idle; max_cstate controls idle depth after polling. Both knobs work together.

(e) Per-vCPU vs per-VM: starting with per-VM. Per-vCPU can be added later as a separate capability if needed.

== Scope ==

This v3 is scoped to pinned-core configurations (NFV, DPDK). A per-task cpuidle constraint (Option C from v2) would cover all cases including overcommit, but requires cpuidle subsystem changes. Happy to pursue that if maintainers prefer, but Option A covers the immediate customer need.

== QEMU prototype (separate tree) ==

A working QEMU patch adds -accel kvm,cstate-policy=N. Available at:
https://github.com/aharivel/qemu/tree/kvm-cstate-policy

== Series ==

[1/3] cpuidle: export cpuidle_devices for KVM module access [2/3] KVM: x86: add KVM_CAP_CSTATE_POLICY capability [3/3] KVM: selftests: add cstate_policy_test

Anthony Harivel (3):
cpuidle: export cpuidle_devices for KVM C-state policy enforcement
KVM: x86: add KVM_CAP_CSTATE_POLICY for per-VM C-state enforcement
KVM: selftests: add cstate_policy_test for KVM_CAP_CSTATE_POLICY

 drivers/cpuidle/cpuidle.c                     |   1 +
 include/linux/kvm_host.h                      |   2 +
 include/uapi/linux/kvm.h                      |   1 +
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/x86/cstate_policy_test.c    | 517 ++++++++++++++++++
 virt/kvm/kvm_main.c                           | 101 ++++
 6 files changed, 623 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/cstate_policy_test.c
Signature · 1 line
2.55.0
#1 lore raw

RFC PATCH v3 1/3 cpuidle: export cpuidle_devices for KVM C-state policy enforcement

Export the per-CPU cpuidle_devices symbol so that KVM can access cpuidle device and driver state to temporarily constrain idle state selection when enforcing per-VM C-state policies.

This is needed by the upcoming KVM_CAP_CSTATE_POLICY capability, which disables deep C-states on the current pCPU before a vCPU blocks in kvm_vcpu_block().

---
 drivers/cpuidle/cpuidle.c | 1 +
 1 file changed, 1 insertion(+)
drivers/cpuidle/cpuidle.c +1 -0
index 2d2f40a2cb81..13f8acde31a4 100644
@@ -31,6 +31,7 @@
 #include "cpuidle.h"
 
 DEFINE_PER_CPU(struct cpuidle_device *, cpuidle_devices);
+EXPORT_PER_CPU_SYMBOL_GPL(cpuidle_devices);
 DEFINE_PER_CPU(struct cpuidle_device, cpuidle_dev);
 
 DEFINE_MUTEX(cpuidle_lock);
Signature · 1 line
2.55.0
#2 lore raw

RFC PATCH v3 2/3 KVM: x86: add KVM_CAP_CSTATE_POLICY for per-VM C-state enforcement

In reply to Anthony Harivel

Add a new VM-scoped capability that allows userspace to set a maximum C-state ceiling for host cpuidle when vCPUs halt.

When a vCPU enters kvm_vcpu_block(), KVM temporarily disables cpuidle states deeper than max_cstate on the current pCPU using the existing states_usage[].disable mechanism (CPUIDLE_STATE_DISABLED_BY_DRIVER). After wakeup, the original disable flags are restored.

The capability follows the same pattern as KVM_CAP_HALT_POLL:
- VM-scoped ioctl via KVM_ENABLE_CAP
- args[0] = max_cstate (-1 to 6, -1 disables the policy)
- Re-callable at runtime without VM restart
- Memory ordering via smp_wmb/rmb

This fills an operational gap for NFV and latency-sensitive deployments where the host operator needs per-VM control over idle depth without requiring guest cooperation. The enforcement is scoped to pinned-core configurations where the disable flags do not race with other tasks.

Measured VM exit overhead (Intel Xeon, ftrace):
Median: 867 ns — negligible relative to C-state exit latencies (C1E: 10us, C3: 33us, C6: 133us).

---
 include/linux/kvm_host.h |   2 +
 include/uapi/linux/kvm.h |   1 +
 virt/kvm/kvm_main.c      | 101 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 104 insertions(+)
include/linux/kvm_host.h +2 -0
index ab8cfaec82d3..3eca172e62ff 100644
@@ -863,6 +863,8 @@ struct kvm {
 	pid_t userspace_pid;
 	bool override_halt_poll_ns;
 	unsigned int max_halt_poll_ns;
+	bool override_cstate_policy;
+	int max_cstate;
 	u32 dirty_ring_size;
 	bool dirty_ring_with_bitmap;
 	bool vm_bugged;
include/uapi/linux/kvm.h +1 -0
index 70e36e6a0ad4..559d23e01127 100644
@@ -998,6 +998,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_S390_VSIE_ESAMODE 248
 #define KVM_CAP_S390_HPAGE_2G 249
 #define KVM_CAP_PPC_COMPAT_CAPS 250
+#define KVM_CAP_CSTATE_POLICY 251
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
virt/kvm/kvm_main.c +101 -0

142 lines not on this page · raw message

Signature · 1 line
2.55.0
#3 lore raw

Re: [PATCH RFC v3 2/3] KVM: x86: add KVM_CAP_CSTATE_POLICY for per-VM C-state enforcement

Quoted 14 lines · Anthony Harivel

Add a new VM-scoped capability that allows userspace to set a maximum C-state ceiling for host cpuidle when vCPUs halt.

When a vCPU enters kvm_vcpu_block(), KVM temporarily disables cpuidle states deeper than max_cstate on the current pCPU using the existing states_usage[].disable mechanism (CPUIDLE_STATE_DISABLED_BY_DRIVER). After wakeup, the original disable flags are restored.

The capability follows the same pattern as KVM_CAP_HALT_POLL:
- VM-scoped ioctl via KVM_ENABLE_CAP
- args[0] = max_cstate (-1 to 6, -1 disables the policy)
- Re-callable at runtime without VM restart
- Memory ordering via smp_wmb/rmb

Quoted 4 lines

This fills an operational gap for NFV and latency-sensitive deployments where the host operator needs per-VM control over idle depth without requiring guest cooperation. The enforcement is scoped to pinned-core configurations where the disable flags do not race with other tasks.

Sorry, NAK, this doesn't belong in KVM. Given that the only way this can work is if vCPU are pinned 1:1 to pCPUs, then it should be very doable for the cpuidle subystem to provide an interface to let (privileged?) userspace restrict the maximum C-state on a per-CPU basis.

My apologies for not responding to v1 or v2, I am guilty of Jim's axiom that upstream doesn't respond to RFCs without code.

Pulling in the other options here:

+    The enforcement point is kvm_vcpu_halt() → kvm_vcpu_block() →
+    schedule() → cpuidle. The problem: cpuidle has no per-task
+    C-state constraint. states_usage[].disable is per-CPU, and
+    forced_idle_latency_limit_ns is also per-CPU.

I don't understand why per-CPU controls are a bad thing. A task-based scheme can really only work if vCPUs are pinned to pCPUs, i.e. you effectively need per-CPU controls anyways. And explicit per-CPU controls would allow for more relaxed scheduling too, e.g. would allow affining vCPUs to a set of pCPUs without needing to have strict 1:1 pinning.

+
+ Three options I see:
+

+      Option A: Temporarily toggle states_usage[i].disable on the
+      pinned pCPU before/after kvm_vcpu_block(). Set
+      CPUIDLE_STATE_DISABLED_BY_DRIVER for states > max_cstate,
+      restore after wakeup. Simple, works with existing API, but
+      only correct with dedicated pinning — overcommit with mixed
+      policies would race on the disable flags.
+
+      Option B: Use forced_idle_latency_limit_ns on the pCPU.
+      Same per-CPU limitation, and latency-based rather than
+      state-index-based — less precise.

Conceptually, (b) seems like the right approach. Per-task will be a mess because similar to a KVM-based interface, it can probably only work if tasks are pinned to pCPUs.

And isn't abstracting away the exact C-state via forced_idle_latency_limit_ns a *good* thing? Without that, userspace will need to tune its configuration for each individual uarch based on the properties of various C-states for a given CPU.

+

+      Option C: Propose a new cpuidle API for per-task idle
+      constraints (e.g. a per-task_struct annotation checked by
+      the governor during select()). Correct for all cases, but
+      bigger scope and needs cpuidle maintainer buy-in.
maintainer
#4 lore raw

Re: [PATCH RFC v3 2/3] KVM: x86: add KVM_CAP_CSTATE_POLICY for per-VM C-state enforcement

Il lun 14 set 2026, 17:24 Sean Christopherson <[email protected]> ha scritto:

Quoted 8 lines
(quoted diff) +3 -0
+      Option B: Use forced_idle_latency_limit_ns on the pCPU.
+      Same per-CPU limitation, and latency-based rather than
+      state-index-based — less precise.

Conceptually, (b) seems like the right approach. Per-task will be a mess because similar to a KVM-based interface, it can probably only work if tasks are pinned to pCPUs.

Yes, per-task (option C) doesn't make much sense. If you want to bound the tasks' wake-up latency they're either pinned, or all I/O bound. But if they are I/O bound and happen to contend, what happens if they migrate? The destination pCPU won't have the right limit set, so you're back to pCPU limits.

And doing it in KVM doesn't remove the issue.

Quoted 3 lines

And isn't abstracting away the exact C-state via forced_idle_latency_limit_ns a *good* thing? Without that, userspace will need to tune its configuration for each individual uarch based on the properties of various C-states for a given CPU.

Yes, even if you had a KVM interface it should allow nanosecond setup, C-state indices really don't mean much.

Paolo

Quoted 5 lines
(quoted diff) +4 -0
+      Option C: Propose a new cpuidle API for per-task idle
+      constraints (e.g. a per-task_struct annotation checked by
+      the governor during select()). Correct for all cases, but
+      bigger scope and needs cpuidle maintainer buy-in.
#5 lore raw

Re: [PATCH RFC v3 2/3] KVM: x86: add KVM_CAP_CSTATE_POLICY for per-VM C-state enforcement

In reply to Sean Christopherson
Quoted 8 lines

Sorry, NAK, this doesn't belong in KVM. Given that the only way this can work is if vCPU are pinned 1:1 to pCPUs, then it should be very doable for the cpuidle subystem to provide an interface to let (privileged?) userspace restrict the maximum C-state on a per-CPU basis.

My apologies for not responding to v1 or v2, I am guilty of Jim's axiom that upstream doesn't respond to RFCs without code.

Thanks for the review Sean, and no worries about v1/v2, I understand the "no code, no review" reality.

I agree this belongs in cpuidle rather than KVM. A per-CPU userspace interface is the right level of abstraction; it works for our pinned vCPU use case and is useful beyond KVM (DPDK, bare-metal NFV, etc.).

Quoted 11 lines
(quoted diff) +3 -0
+      Option B: Use forced_idle_latency_limit_ns on the pCPU.
+      Same per-CPU limitation, and latency-based rather than
+      state-index-based, less precise.

Conceptually, (b) seems like the right approach. Per-task will be a mess because similar to a KVM-based interface, it can probably only work if tasks are pinned to pCPUs.

And isn't abstracting away the exact C-state via forced_idle_latency_limit_ns a *good* thing? Without that, userspace will need to tune its configuration for each individual uarch based on the properties of various C-states for a given CPU.

I also agree that latency-based is better than C-state-index-based. I mean "Max 10us
wakeup latency" is portable across Intel/AMD/ARM without needing to know the C-state table for each uarch and that's a real plus.

I looked at forced_idle_latency_limit_ns as a starting point, but it doesn't quite fit as-is. Currently it is:

1. Kernel-internal only, set by cpuidle_use_deepest_state(), called exclusively from play_idle_precise() (idle injection for thermal).

2. A governor bypass, not a governor constraint, when set, the idle path skips the normal governor select() entirely and calls cpuidle_find_deepest_state() directly (in kernel/sched/idle.c). This forces the deepest state *within* the limit rather than letting the governor choose while *capping* at the limit.

3. Not exposed via sysfs. No userspace interface exists.

For our use case, I want the governor to keep making its normal decisions (considering load, predicted idle duration, etc.) but with an upper bound on the exit latency it can select. More of a ceiling than a "force deepest" override.

I see two paths forward:

(a) Add a new per-CPU sysfs attribute under cpuidle, e.g. /sys/devices/system/cpu/cpuN/cpuidle/latency_limit_ns that sets a governor-respected ceiling. The governor's select() would filter out states with exit_latency_ns > limit. This is similar to how PM QoS cpu_latency works, but per-CPU rather than system-wide.

(b) Reuse forced_idle_latency_limit_ns but change its semantics to be a governor ceiling rather than a governor bypass. This would affect the idle injection path though, so might not be desirable.

I tend to choose (a). I've already done a PoC that is working as expected. I'll send a patch queue to the cpuidle maintainers (Rafael Wysocki, Daniel Lezcano).

Thanks,
Anthony

#6 lore raw

RFC PATCH v3 3/3 KVM: selftests: add cstate_policy_test for KVM_CAP_CSTATE_POLICY

In reply to Anthony Harivel

Add a KVM selftest that validates the KVM_CAP_CSTATE_POLICY capability:

1. cap_supported: capability is advertised
2. valid_policies: max_cstate -1 through 6 accepted
3. invalid_policies: out-of-range values and bad flags rejected
4. policy_change: policy can be changed at runtime on a live VM
5. cpuidle_no_policy: control test — without policy, deep C-states (C6) are entered during guest HLT (confirms test methodology)
6. cpuidle_enforcement: with policy=C1, deep C-states are blocked — C6 usage delta is near zero while C1 accumulates
7. policy_c0_no_idle: with policy=C0, no idle states are entered

Tests 5-7 pin a vCPU to a specific pCPU and compare host cpuidle usage counters before and after 1000 guest HLT cycles. The guest uses a periodic APIC timer to wake from HLT.

Tested on Dell R640 (Intel Xeon Gold 5118, intel_idle driver with POLL/C1/C1E/C6 states). Results:

  - No policy:    C6 entered 1003 times (control)
  - Policy=C1:    C6 entered 0 times, C1 entered 1073 times
  - Policy=C0:    POLL entered 56743 times, no real idle
---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/x86/cstate_policy_test.c    | 517 ++++++++++++++++++
 2 files changed, 518 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/cstate_policy_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
index 6fc34e9bf8e1..09e58d6c71e3 100644
@@ -151,6 +151,7 @@ TEST_GEN_PROGS_x86 += x86/max_vcpuid_cap_test
 TEST_GEN_PROGS_x86 += x86/triple_fault_event_test
 TEST_GEN_PROGS_x86 += x86/recalc_apic_map_test
 TEST_GEN_PROGS_x86 += x86/aperfmperf_test
+TEST_GEN_PROGS_x86 += x86/cstate_policy_test
 TEST_GEN_PROGS_x86 += access_tracking_perf_test
 TEST_GEN_PROGS_x86 += coalesced_io_test
 TEST_GEN_PROGS_x86 += dirty_log_perf_test
tools/testing/selftests/kvm/x86/cstate_policy_test.c new file +517 -0

517 lines not on this page · raw message

Signature · 1 line
2.55.0