KVM: enable halt poll shrink parameter
Subsystems (2)
cover KVM: enable halt poll shrink parameter
KVM halt polling interval growth and shrink behavior has evolved since its inception. The current mechanism adjusts the polling interval based on whether vcpu wakeup was received or not during polling interval using grow and shrink parameter values. Though grow parameter is logically set to 2 by default, shrink parameter is kept disabled (set to 0).
Disabled shrink has two issues:
1) Resets polling interval to 0 on every un-successful poll assuming it is less likely to receive a vcpu wakeup in further shrunk intervals.
2) Even on successful poll, if total block time is greater or equal to current poll_ns value, polling interval is reset to 0 instead shrinking gradually.
These aspects reduce the chances receiving valid wakeup during polling and lose potential performance benefits for VM workloads.
Below is the summary of experiments conducted to assess performance and power impact by enabling the halt_poll_ns_shrink parameter(value set to 2).
Performance Test Summary: (Higher is better)
--------------------------------------------
Platform Details: Chrome Brya platform
CPU - Alder Lake (12th Gen Intel CPU i7-1255U) Host kernel version - 5.15.127-20371-g710a1611ad33
Android VM workload (Score) Base Shrink Enabled (value 2) Delta
---------------------------------------------------------------------------
GeekBench Multi-core(CPU) 5754 5856 2% 3D Mark Slingshot(CPU+GPU) 15486 15885 3% Stream (handopt)(Memory) 20566 21594 5% fio seq-read (Storage) 727 747 3% fio seq-write (Storage) 331 343 3% fio rand-read (Storage) 690 732 6% fio rand-write (Storage) 299 300 1%
Steam Gaming VM (Avg FPS) Base Shrink Enabled (value 2) Delta
---------------------------------------------------------------------------
Metro Redux (OpenGL) 54.80 59.60 9% Dota 2 (Open GL) 48.74 51.40 5% Dota 2 (Vulkan) 20.80 21.10 1% SpaceShip (Vulkan) 20.40 21.52 6%
With Shrink enabled, majority of workloads show higher % of successful polling. Reduced latency of returning control back to VM and avoided overhead of vm_exit contribute to these performance gains.
Power Impact Assessment Summary: (Lower is better)
--------------------------------------------------
Method : DAQ measurements of CPU and Memory rails
CPU+Memory (Watt) Base Shrink Enabled (value 2) Delta
---------------------------------------------------------------------------
Idle* (Host) 0.636 0.631 -0.8% Video Playback (Host) 2.225 2.210 -0.7% Tomb Raider (VM) 17.261 17.175 -0.5% SpaceShip Benchmark(VM) 17.079 17.123 0.3%
*Idle power - Idle system with no application running, Android and Borealis VMs enabled running no workload. Duration 180 sec.
Power measurements done for Chrome idle scenario and active Gaming VM workload show negligible power overhead since additional polling creates very short duration bursts which are less likely to have gone to a complete idle CPU state.
NOTE: No tests are conducted on non-x86 platform with this changed config
The default values of grow and shrink parameters get commonly used by various VM deployments unless specifically tuned for performance. Hence referring to performance and power measurements results shown above, it is recommended to have shrink enabled (with value 2) by default so that there is no need to explicitly set this parameter through kernel cmdline or by other means.
Parshuram Sangle (2):
KVM: enable halt polling shrink parameter by default
KVM: documentation update to halt polling
Documentation/virt/kvm/halt-polling.rst | 26 +++++++++++++------------ virt/kvm/kvm_main.c | 4 ++-- 2 files changed, 16 insertions(+), 14 deletions(-)
base-commit: 2b3f2325e71f09098723727d665e2e8003d455dc
Signature · 1 line
2.17.1
PATCH 1/2 KVM: enable halt polling shrink parameter by default
Default halt_poll_ns_shrink value of 0 always resets polling interval to 0 on an un-successful poll where vcpu wakeup is not received. This is mostly to avoid pointless polling for more number of shorter intervals. But disabled shrink assumes vcpu wakeup is less likely to be received in subsequent shorter polling intervals. Another side effect of 0 shrink value is that, even on a successful poll if total block time was greater than current polling interval, the polling interval starts over from 0 instead of shrinking by a factor.
Enabling shrink with value of 2 allows the polling interval to gradually decrement in case of un-successful poll events as well. This gives a fair chance for successful polling events in subsequent polling intervals rather than resetting it to 0 and starting over from grow_start.
Below kvm stat log snippet shows interleaved growth and shrinking of polling interval:
87162647182125: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 10000 (grow 0) 87162647637763: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (grow 10000) 87162649627943: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 40000 (grow 20000) 87162650892407: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (shrink 40000) 87162651540378: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 40000 (grow 20000) 87162652276768: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (shrink 40000) 87162652515037: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 40000 (grow 20000) 87162653383787: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (shrink 40000) 87162653627670: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 10000 (shrink 20000) 87162653796321: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (grow 10000) 87162656171645: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 10000 (shrink 20000) 87162661607487: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 0 (shrink 10000)
Having both grow and shrink enabled creates a balance in polling interval growth and shrink behavior. Tests show improved successful polling attempt ratio which contribute to VM performance. Power penalty is quite negligible as shrunk polling intervals create bursts of very short durations.
Performance assessment results show 3-6% improvements in CPU+GPU, Memory and Storage Android VM workloads whereas 5-9% improvement in average FPS of gaming VM workloads.
Power penalty is below 1% where host OS is either idle or running a native workload having 2 VMs enabled. CPU/GPU intensive gaming workloads as well do not show any increased power overhead with shrink enabled.
--- Documentation/virt/kvm/halt-polling.rst | 2 +- virt/kvm/kvm_main.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
Documentation/virt/kvm/halt-polling.rst +1 -1
@@ -105,7 +105,7 @@ powerpc kvm-hv case. | | grow_halt_poll_ns() | | | | function. | | +-----------------------+---------------------------+-------------------------+ -|halt_poll_ns_shrink | The value by which the | 0 | +|halt_poll_ns_shrink | The value by which the | 2 | | | halt polling interval is | | | | divided in the | | | | shrink_halt_poll_ns() | |
virt/kvm/kvm_main.c +2 -2
@@ -91,8 +91,8 @@ unsigned int halt_poll_ns_grow_start = 10000; /* 10us */module_param (halt_poll_ns_grow_start, uint,0644 );EXPORT_SYMBOL_GPL (halt_poll_ns_grow_start); -/* Default resets per-vcpu halt_poll_ns . */ -unsigned int halt_poll_ns_shrink; +/* Default halves per-vcpu halt_poll_ns. */ +unsigned int halt_poll_ns_shrink =2 ;module_param (halt_poll_ns_shrink, uint,0644 );EXPORT_SYMBOL_GPL (halt_poll_ns_shrink);
Signature · 1 line
2.17.1
PATCH 2/2 KVM: documentation update to halt polling
In reply to Parshuram SangleCorrected kvm documentation to reflect current handling of polling interval on successful and un-successful polling events. Also updated the description about newly added halt_poll_ns_grow_start parameter.
--- Documentation/virt/kvm/halt-polling.rst | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-)
Documentation/virt/kvm/halt-polling.rst +13 -11
@@ -44,12 +44,14 @@ or in the case of powerpc kvm-hv, in the vcore struct: Thus this is a per vcpu (or vcore) value. -During polling if a wakeup source is received within the halt polling interval, -the interval is left unchanged. In the event that a wakeup source isn't -received during the polling interval (and thus schedule is invoked) there are -two options, either the polling interval and total block time[0] were less than -the global max polling interval (see module params below), or the total block -time was greater than the global max polling interval. +During polling if a wakeup source is not received within the halt polling +interval (and thus schedule is invoked), the interval is always shrunk in +shrink_halt_poll_ns(). In the event that a wakeup source is received during +the polling interval, polling interval is left unchanged if total block time +is below or equal to current interval value otherwise there are two options, +either the polling interval and total block time[0] were less than the global +max polling interval (see module params below), or the total block time was +greater than the global max polling interval. In the event that both the polling interval and total block time were less than the global max polling interval then the polling interval can be increased in
@@ -79,11 +81,11 @@ adjustment of the polling interval. Module Parameters ================= -The kvm module has 3 tuneable module parameters to adjust the global max -polling interval as well as the rate at which the polling interval is grown and -shrunk. These variables are defined in include/linux/kvm_host.h and as module -parameters in virt/kvm/kvm_main.c, or arch/powerpc/kvm/book3s_hv.c in the -powerpc kvm-hv case. +The kvm module has 4 tunable module parameters to adjust the global max +polling interval, initial value (to grow from 0) as well as the rate at which +the polling interval is grown and shrunk. These variables are defined in +include/linux/kvm_host.h and as module parameters in virt/kvm/kvm_main.c, or +arch/powerpc/kvm/book3s_hv.c in the powerpc kvm-hv case. +-----------------------+---------------------------+-------------------------+ |Module Parameter | Description | Default Value |
Signature · 1 line
2.17.1
Sean Christopherson
reviewerRe: [PATCH 2/2] KVM: documentation update to halt polling
Quoted 19 lines · Parshuram Sangle
Corrected kvm documentation to reflect current handling of polling interval on successful and un-successful polling events. Also updated the description about newly added
halt_poll_ns_grow_startparameter.Co-developed-by Rajendran Jaishankar <[email protected]>Signed-off-by Rajendran Jaishankar <[email protected]>Signed-off-by Parshuram Sangle <[email protected]>--- Documentation/virt/kvm/halt-polling.rst | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-)Documentation/virt/kvm/halt-polling.rst +0
-0@@ -44,12 +44,14 @@ or in the case of powerpc kvm-hv, in the vcore struct: Thus this is a per vcpu (or vcore) value.
Documentation/virt/kvm/halt-polling.rst or in the case of powerpc kvm-hv, in the vcore struct: · line 49
Quoted 10 lines
Documentation/virt/kvm/halt-polling.rst +3
-6-During polling if a wakeup source is received within the halt polling interval, -the interval is left unchanged. In the event that a wakeup source isn't -received during the polling interval (and thus schedule is invoked) there are -two options, either the polling interval and total block time[0] were less than -the global max polling interval (see module params below), or the total block -time was greater than the global max polling interval. +During polling if a wakeup source is not received within the halt polling +interval (and thus schedule is invoked), the interval is always shrunk in +shrink_halt_poll_ns().
This is wrong. KVM grows the interval if the wakeup event arrives after KVM stops polling if the total halt time is less than the max allowed interval.
The existing wording is a bit odd, and it fails to document the per-VM capability, but I didn't find anything in the existing documentation that is incorrect (ignoring the Module Parameters section below).
If we wanted to clean things up, I would vote for something like the below. For this patch, I'm going to drop this update and just keep the Module Parameters fixes.
During polling if a wakeup source is received within the halt polling interval, the interval is left unchanged (the total block time[0] is less than or equal to the current interval). If the block time is greater than the interval, there are two possibilities: either the polling interval and total block time were less than the VM's max polling interval (see module params and KVM_CAP_HALT_POLL below), or the total block time was greater than the VM's max polling interval.
In the event that both the polling interval and total block time were less than the max polling interval then the polling interval can be increased in the hope that next time during the longer polling interval the wake up source will be received while the host is polling and the latency benefits will be received. The polling interval is grown in the function grow_halt_poll_ns() and is multiplied by the module parameters halt_poll_ns_grow and halt_poll_ns_grow_start.
In the event that the total block time was greater than the VM's max polling interval then the host will never poll for long enough to wakeup during the polling interval (KVM will always poll for less time than the block time), so the polling internal is shrunk in order to avoid pointless polling. The polling interval is shrunk in the function shrink_halt_poll_ns() and is divided by the module parameter halt_poll_ns_shrink, or set to 0 iff halt_poll_ns_shrink == 0.
It is worth noting that this adjustment process attempts to hone in on some steady state polling interval but will only really do a good job for wakeups which come at an approximately constant rate, otherwise there will be constant adjustment of the polling interval.
Soft reminder for patch review
Sean Christopherson
reviewerQuoted 64 lines · Parshuram Sangle
KVM halt polling interval growth and shrink behavior has evolved since its inception. The current mechanism adjusts the polling interval based on whether vcpu wakeup was received or not during polling interval using grow and shrink parameter values. Though grow parameter is logically set to 2 by default, shrink parameter is kept disabled (set to 0).
Disabled shrink has two issues:
1) Resets polling interval to 0 on every un-successful poll assuming it is less likely to receive a vcpu wakeup in further shrunk intervals.
2) Even on successful poll, if total block time is greater or equal to currentpoll_nsvalue, polling interval is reset to 0 instead shrinking gradually.These aspects reduce the chances receiving valid wakeup during polling and lose potential performance benefits for VM workloads.
Below is the summary of experiments conducted to assess performance and power impact by enabling the
halt_poll_ns_shrinkparameter(value set to 2).Performance Test Summary: (Higher is better)
--------------------------------------------
Platform Details: Chrome Brya platform
CPU - Alder Lake (12th Gen Intel CPU i7-1255U) Host kernel version - 5.15.127-20371-g710a1611ad33Android VM workload (Score) Base Shrink Enabled (value 2) Delta
---------------------------------------------------------------------------GeekBench Multi-core(CPU) 5754 5856 2% 3D Mark Slingshot(CPU+GPU) 15486 15885 3% Stream (handopt)(Memory) 20566 21594 5% fio seq-read (Storage) 727 747 3% fio seq-write (Storage) 331 343 3% fio rand-read (Storage) 690 732 6% fio rand-write (Storage) 299 300 1%Steam Gaming VM (Avg FPS) Base Shrink Enabled (value 2) Delta
---------------------------------------------------------------------------Metro Redux (OpenGL) 54.80 59.60 9% Dota 2 (Open GL) 48.74 51.40 5% Dota 2 (Vulkan) 20.80 21.10 1% SpaceShip (Vulkan) 20.40 21.52 6%With Shrink enabled, majority of workloads show higher % of successful polling. Reduced latency of returning control back to VM and avoided overhead of
vm_exitcontribute to these performance gains.Power Impact Assessment Summary: (Lower is better)
--------------------------------------------------
Method : DAQ measurements of CPU and Memory railsCPU+Memory (Watt) Base Shrink Enabled (value 2) Delta
---------------------------------------------------------------------------Idle* (Host) 0.636 0.631 -0.8% Video Playback (Host) 2.225 2.210 -0.7% Tomb Raider (VM) 17.261 17.175 -0.5% SpaceShip Benchmark(VM) 17.079 17.123 0.3%*Idle power - Idle system with no application running, Android and Borealis VMs enabled running no workload. Duration 180 sec.
Power measurements done for Chrome idle scenario and active Gaming VM workload show negligible power overhead since additional polling creates very short duration bursts which are less likely to have gone to a complete idle CPU state.
Quoted 8 lines
NOTE: No tests are conducted on non-x86 platform with this changed config
The default values of grow and shrink parameters get commonly used by various VM deployments unless specifically tuned for performance. Hence referring to performance and power measurements results shown above, it is recommended to have shrink enabled (with value 2) by default so that there is no need to explicitly set this parameter through kernel cmdline or by other means.
I am by no means an expert on halt polling or power management, but all of this seems like a reasonable tradeoff. And even without the numbers you provided, starting from scratch after a single failure is rather odd.
So unless someone objects, I'll plan on applying this for 6.11 in a few weeks (after the 6.10 merge window closes).
Quoted 6 lines · Parshuram Sangle
KVM halt polling interval growth and shrink behavior has evolved since its inception. The current mechanism adjusts the polling interval based on whether vcpu wakeup was received or not during polling interval using grow and shrink parameter values. Though grow parameter is logically set to 2 by default, shrink parameter is kept disabled (set to 0).
Quoted 7 lines
Disabled shrink has two issues:
1) Resets polling interval to 0 on every un-successful poll assuming it is less likely to receive a vcpu wakeup in further shrunk intervals.
2) Even on successful poll, if total block time is greater or equal to currentpoll_nsvalue, polling interval is reset to 0 instead shrinking gradually.[...]
Applied to kvm-x86 generic, with a reduced version of the doc update as described in response to patch 2. Thanks!
[1/2] KVM: enable halt polling shrink parameter by default
https://github.com/kvm-x86/linux/commit/aeb1b22a3ac8 [2/2] KVM: documentation update to halt polling https://github.com/kvm-x86/linux/commit/f8aadead1971