Skip to contentSkip to conversation

KVM: enable halt poll shrink parameter

PATCH Applied Parshuram Sangle 2 participants 7 messages Nov 2, 2023 – Jun 4, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover KVM: enable halt poll shrink parameter

KVM halt polling interval growth and shrink behavior has evolved since its inception. The current mechanism adjusts the polling interval based on whether vcpu wakeup was received or not during polling interval using grow and shrink parameter values. Though grow parameter is logically set to 2 by default, shrink parameter is kept disabled (set to 0).

Disabled shrink has two issues:
1) Resets polling interval to 0 on every un-successful poll assuming it is less likely to receive a vcpu wakeup in further shrunk intervals.
2) Even on successful poll, if total block time is greater or equal to current poll_ns value, polling interval is reset to 0 instead shrinking gradually.

These aspects reduce the chances receiving valid wakeup during polling and lose potential performance benefits for VM workloads.

Below is the summary of experiments conducted to assess performance and power impact by enabling the halt_poll_ns_shrink parameter(value set to 2).

Performance Test Summary: (Higher is better)
--------------------------------------------
Platform Details: Chrome Brya platform
CPU - Alder Lake (12th Gen Intel CPU i7-1255U) Host kernel version - 5.15.127-20371-g710a1611ad33

Android VM workload (Score) Base Shrink Enabled (value 2) Delta
---------------------------------------------------------------------------

GeekBench Multi-core(CPU)     5754      5856                        2%
3D Mark Slingshot(CPU+GPU)    15486     15885                       3%
Stream (handopt)(Memory)      20566     21594                       5%
fio seq-read (Storage)        727       747                         3%
fio seq-write (Storage)       331       343                         3%
fio rand-read (Storage)       690       732                         6%
fio rand-write (Storage)      299       300                         1%

Steam Gaming VM (Avg FPS) Base Shrink Enabled (value 2) Delta
---------------------------------------------------------------------------

Metro Redux (OpenGL)          54.80     59.60                       9%
Dota 2 (Open GL)              48.74     51.40                       5%
Dota 2 (Vulkan)               20.80     21.10                       1%
SpaceShip (Vulkan)            20.40     21.52                       6%

With Shrink enabled, majority of workloads show higher % of successful polling. Reduced latency of returning control back to VM and avoided overhead of vm_exit contribute to these performance gains.

Power Impact Assessment Summary: (Lower is better)
--------------------------------------------------
Method : DAQ measurements of CPU and Memory rails

CPU+Memory (Watt) Base Shrink Enabled (value 2) Delta
---------------------------------------------------------------------------

Idle* (Host)                  0.636     0.631                       -0.8%
Video Playback (Host)         2.225     2.210                       -0.7%
Tomb Raider (VM)              17.261    17.175                      -0.5%
SpaceShip Benchmark(VM)       17.079    17.123                       0.3%

*Idle power - Idle system with no application running, Android and Borealis VMs enabled running no workload. Duration 180 sec.

Power measurements done for Chrome idle scenario and active Gaming VM workload show negligible power overhead since additional polling creates very short duration bursts which are less likely to have gone to a complete idle CPU state.

NOTE: No tests are conducted on non-x86 platform with this changed config

The default values of grow and shrink parameters get commonly used by various VM deployments unless specifically tuned for performance. Hence referring to performance and power measurements results shown above, it is recommended to have shrink enabled (with value 2) by default so that there is no need to explicitly set this parameter through kernel cmdline or by other means.

Parshuram Sangle (2):
KVM: enable halt polling shrink parameter by default
KVM: documentation update to halt polling

 Documentation/virt/kvm/halt-polling.rst | 26 +++++++++++++------------
 virt/kvm/kvm_main.c                     |  4 ++--
 2 files changed, 16 insertions(+), 14 deletions(-)

base-commit: 2b3f2325e71f09098723727d665e2e8003d455dc

Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/2 KVM: enable halt polling shrink parameter by default

Default halt_poll_ns_shrink value of 0 always resets polling interval to 0 on an un-successful poll where vcpu wakeup is not received. This is mostly to avoid pointless polling for more number of shorter intervals. But disabled shrink assumes vcpu wakeup is less likely to be received in subsequent shorter polling intervals. Another side effect of 0 shrink value is that, even on a successful poll if total block time was greater than current polling interval, the polling interval starts over from 0 instead of shrinking by a factor.

Enabling shrink with value of 2 allows the polling interval to gradually decrement in case of un-successful poll events as well. This gives a fair chance for successful polling events in subsequent polling intervals rather than resetting it to 0 and starting over from grow_start.

Below kvm stat log snippet shows interleaved growth and shrinking of polling interval:
87162647182125: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 10000 (grow 0) 87162647637763: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (grow 10000) 87162649627943: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 40000 (grow 20000) 87162650892407: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (shrink 40000) 87162651540378: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 40000 (grow 20000) 87162652276768: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (shrink 40000) 87162652515037: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 40000 (grow 20000) 87162653383787: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (shrink 40000) 87162653627670: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 10000 (shrink 20000) 87162653796321: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (grow 10000) 87162656171645: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 10000 (shrink 20000) 87162661607487: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 0 (shrink 10000)

Having both grow and shrink enabled creates a balance in polling interval growth and shrink behavior. Tests show improved successful polling attempt ratio which contribute to VM performance. Power penalty is quite negligible as shrunk polling intervals create bursts of very short durations.

Performance assessment results show 3-6% improvements in CPU+GPU, Memory and Storage Android VM workloads whereas 5-9% improvement in average FPS of gaming VM workloads.

Power penalty is below 1% where host OS is either idle or running a native workload having 2 VMs enabled. CPU/GPU intensive gaming workloads as well do not show any increased power overhead with shrink enabled.

---
 Documentation/virt/kvm/halt-polling.rst | 2 +-
 virt/kvm/kvm_main.c                     | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
Documentation/virt/kvm/halt-polling.rst +1 -1
index c82a04b709b4..64f32a81133f 100644
@@ -105,7 +105,7 @@ powerpc kvm-hv case.
 |			| grow_halt_poll_ns()	    |			      |
 |			| function.		    |			      |
 +-----------------------+---------------------------+-------------------------+
-|halt_poll_ns_shrink	| The value by which the    | 0			      |
+|halt_poll_ns_shrink	| The value by which the    | 2			      |
 |			| halt polling interval is  |			      |
 |			| divided in the	    |			      |
 |			| shrink_halt_poll_ns()	    |			      |
virt/kvm/kvm_main.c +2 -2
index 486800a7024b..cfc474558660 100644
@@ -91,8 +91,8 @@ unsigned int halt_poll_ns_grow_start = 10000; /* 10us */
 module_param(halt_poll_ns_grow_start, uint, 0644);
 EXPORT_SYMBOL_GPL(halt_poll_ns_grow_start);
 
-/* Default resets per-vcpu halt_poll_ns . */
-unsigned int halt_poll_ns_shrink;
+/* Default halves per-vcpu halt_poll_ns. */
+unsigned int halt_poll_ns_shrink = 2;
 module_param(halt_poll_ns_shrink, uint, 0644);
 EXPORT_SYMBOL_GPL(halt_poll_ns_shrink);
 
Signature · 1 line
2.17.1
#2 lore raw

PATCH 2/2 KVM: documentation update to halt polling

In reply to Parshuram Sangle

Corrected kvm documentation to reflect current handling of polling interval on successful and un-successful polling events. Also updated the description about newly added halt_poll_ns_grow_start parameter.

---
 Documentation/virt/kvm/halt-polling.rst | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)
Documentation/virt/kvm/halt-polling.rst +13 -11
index 64f32a81133f..cff388d9dc72 100644
@@ -44,12 +44,14 @@ or in the case of powerpc kvm-hv, in the vcore struct:
 
 Thus this is a per vcpu (or vcore) value.
 
-During polling if a wakeup source is received within the halt polling interval,
-the interval is left unchanged. In the event that a wakeup source isn't
-received during the polling interval (and thus schedule is invoked) there are
-two options, either the polling interval and total block time[0] were less than
-the global max polling interval (see module params below), or the total block
-time was greater than the global max polling interval.
+During polling if a wakeup source is not received within the halt polling
+interval (and thus schedule is invoked), the interval is always shrunk in
+shrink_halt_poll_ns(). In the event that a wakeup source is received during
+the polling interval, polling interval is left unchanged if total block time
+is below or equal to current interval value otherwise there are two options,
+either the polling interval and total block time[0] were less than the global
+max polling interval (see module params below), or the total block time was
+greater than the global max polling interval.
 
 In the event that both the polling interval and total block time were less than
 the global max polling interval then the polling interval can be increased in
@@ -79,11 +81,11 @@ adjustment of the polling interval.
 Module Parameters
 =================
 
-The kvm module has 3 tuneable module parameters to adjust the global max
-polling interval as well as the rate at which the polling interval is grown and
-shrunk. These variables are defined in include/linux/kvm_host.h and as module
-parameters in virt/kvm/kvm_main.c, or arch/powerpc/kvm/book3s_hv.c in the
-powerpc kvm-hv case.
+The kvm module has 4 tunable module parameters to adjust the global max
+polling interval, initial value (to grow from 0) as well as the rate at which
+the polling interval is grown and shrunk. These variables are defined in
+include/linux/kvm_host.h and as module parameters in virt/kvm/kvm_main.c, or
+arch/powerpc/kvm/book3s_hv.c in the powerpc kvm-hv case.
 
 +-----------------------+---------------------------+-------------------------+
 |Module Parameter	|   Description		    |	     Default Value    |
Signature · 1 line
2.17.1
#3 lore raw

Re: [PATCH 2/2] KVM: documentation update to halt polling

Quoted 19 lines · Parshuram Sangle

Corrected kvm documentation to reflect current handling of polling interval on successful and un-successful polling events. Also updated the description about newly added halt_poll_ns_grow_start parameter.

---
 Documentation/virt/kvm/halt-polling.rst | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)
Documentation/virt/kvm/halt-polling.rst +0 -0
index 64f32a81133f..cff388d9dc72 100644
@@ -44,12 +44,14 @@ or in the case of powerpc kvm-hv, in the vcore struct:
 
 Thus this is a per vcpu (or vcore) value.

Documentation/virt/kvm/halt-polling.rst or in the case of powerpc kvm-hv, in the vcore struct: · line 49

Quoted 10 lines
Documentation/virt/kvm/halt-polling.rst +3 -6
 
-During polling if a wakeup source is received within the halt polling interval,
-the interval is left unchanged. In the event that a wakeup source isn't
-received during the polling interval (and thus schedule is invoked) there are
-two options, either the polling interval and total block time[0] were less than
-the global max polling interval (see module params below), or the total block
-time was greater than the global max polling interval.
+During polling if a wakeup source is not received within the halt polling
+interval (and thus schedule is invoked), the interval is always shrunk in
+shrink_halt_poll_ns().

This is wrong. KVM grows the interval if the wakeup event arrives after KVM stops polling if the total halt time is less than the max allowed interval.

The existing wording is a bit odd, and it fails to document the per-VM capability, but I didn't find anything in the existing documentation that is incorrect (ignoring the Module Parameters section below).

If we wanted to clean things up, I would vote for something like the below. For this patch, I'm going to drop this update and just keep the Module Parameters fixes.

During polling if a wakeup source is received within the halt polling interval, the interval is left unchanged (the total block time[0] is less than or equal to the current interval). If the block time is greater than the interval, there are two possibilities: either the polling interval and total block time were less than the VM's max polling interval (see module params and KVM_CAP_HALT_POLL below), or the total block time was greater than the VM's max polling interval.

In the event that both the polling interval and total block time were less than the max polling interval then the polling interval can be increased in the hope that next time during the longer polling interval the wake up source will be received while the host is polling and the latency benefits will be received. The polling interval is grown in the function grow_halt_poll_ns() and is multiplied by the module parameters halt_poll_ns_grow and halt_poll_ns_grow_start.

In the event that the total block time was greater than the VM's max polling interval then the host will never poll for long enough to wakeup during the polling interval (KVM will always poll for less time than the block time), so the polling internal is shrunk in order to avoid pointless polling. The polling interval is shrunk in the function shrink_halt_poll_ns() and is divided by the module parameter halt_poll_ns_shrink, or set to 0 iff halt_poll_ns_shrink == 0.

It is worth noting that this adjustment process attempts to hone in on some steady state polling interval but will only really do a good job for wakeups which come at an approximately constant rate, otherwise there will be constant adjustment of the polling interval.

#5 lore raw
In reply to Parshuram Sangle
Quoted 64 lines · Parshuram Sangle

KVM halt polling interval growth and shrink behavior has evolved since its inception. The current mechanism adjusts the polling interval based on whether vcpu wakeup was received or not during polling interval using grow and shrink parameter values. Though grow parameter is logically set to 2 by default, shrink parameter is kept disabled (set to 0).

Disabled shrink has two issues:
1) Resets polling interval to 0 on every un-successful poll assuming it is less likely to receive a vcpu wakeup in further shrunk intervals.
2) Even on successful poll, if total block time is greater or equal to current poll_ns value, polling interval is reset to 0 instead shrinking gradually.

These aspects reduce the chances receiving valid wakeup during polling and lose potential performance benefits for VM workloads.

Below is the summary of experiments conducted to assess performance and power impact by enabling the halt_poll_ns_shrink parameter(value set to 2).

Performance Test Summary: (Higher is better)
--------------------------------------------
Platform Details: Chrome Brya platform
CPU - Alder Lake (12th Gen Intel CPU i7-1255U) Host kernel version - 5.15.127-20371-g710a1611ad33

Android VM workload (Score) Base Shrink Enabled (value 2) Delta
---------------------------------------------------------------------------

GeekBench Multi-core(CPU)     5754      5856                        2%
3D Mark Slingshot(CPU+GPU)    15486     15885                       3%
Stream (handopt)(Memory)      20566     21594                       5%
fio seq-read (Storage)        727       747                         3%
fio seq-write (Storage)       331       343                         3%
fio rand-read (Storage)       690       732                         6%
fio rand-write (Storage)      299       300                         1%

Steam Gaming VM (Avg FPS) Base Shrink Enabled (value 2) Delta
---------------------------------------------------------------------------

Metro Redux (OpenGL)          54.80     59.60                       9%
Dota 2 (Open GL)              48.74     51.40                       5%
Dota 2 (Vulkan)               20.80     21.10                       1%
SpaceShip (Vulkan)            20.40     21.52                       6%

With Shrink enabled, majority of workloads show higher % of successful polling. Reduced latency of returning control back to VM and avoided overhead of vm_exit contribute to these performance gains.

Power Impact Assessment Summary: (Lower is better)
--------------------------------------------------
Method : DAQ measurements of CPU and Memory rails

CPU+Memory (Watt) Base Shrink Enabled (value 2) Delta
---------------------------------------------------------------------------

Idle* (Host)                  0.636     0.631                       -0.8%
Video Playback (Host)         2.225     2.210                       -0.7%
Tomb Raider (VM)              17.261    17.175                      -0.5%
SpaceShip Benchmark(VM)       17.079    17.123                       0.3%

*Idle power - Idle system with no application running, Android and Borealis VMs enabled running no workload. Duration 180 sec.

Power measurements done for Chrome idle scenario and active Gaming VM workload show negligible power overhead since additional polling creates very short duration bursts which are less likely to have gone to a complete idle CPU state.

Quoted 8 lines

NOTE: No tests are conducted on non-x86 platform with this changed config

The default values of grow and shrink parameters get commonly used by various VM deployments unless specifically tuned for performance. Hence referring to performance and power measurements results shown above, it is recommended to have shrink enabled (with value 2) by default so that there is no need to explicitly set this parameter through kernel cmdline or by other means.

I am by no means an expert on halt polling or power management, but all of this seems like a reasonable tradeoff. And even without the numbers you provided, starting from scratch after a single failure is rather odd.

So unless someone objects, I'll plan on applying this for 6.11 in a few weeks (after the 6.10 merge window closes).

reviewer Applied
#6 lore raw
In reply to Parshuram Sangle
Quoted 6 lines · Parshuram Sangle

KVM halt polling interval growth and shrink behavior has evolved since its inception. The current mechanism adjusts the polling interval based on whether vcpu wakeup was received or not during polling interval using grow and shrink parameter values. Though grow parameter is logically set to 2 by default, shrink parameter is kept disabled (set to 0).

Quoted 7 lines

Disabled shrink has two issues:
1) Resets polling interval to 0 on every un-successful poll assuming it is less likely to receive a vcpu wakeup in further shrunk intervals.
2) Even on successful poll, if total block time is greater or equal to current poll_ns value, polling interval is reset to 0 instead shrinking gradually.

[...]

Applied to kvm-x86 generic, with a reduced version of the doc update as described in response to patch 2. Thanks!

[1/2] KVM: enable halt polling shrink parameter by default

      https://github.com/kvm-x86/linux/commit/aeb1b22a3ac8
[2/2] KVM: documentation update to halt polling
      https://github.com/kvm-x86/linux/commit/f8aadead1971

--
https://github.com/kvm-x86/linux/tree/next