Skip to content

RFC Make vCPUs that are HLT state candidates for load balancing

Masanori Misono kvm lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest1 patch 3 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/1KVM: x86: Don't set preempted when vCPU does HLT VMEXITarch/x86/kvm/x86.c+11 -40 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH RFC 0/1] Make vCPUs that are HLT state candidates for load balancing

Hi,

I observed performance degradation when running some parallel programs on a VM that has (1) KVM_FEATURE_PV_UNHALT, (2) KVM_FEATURE_STEAL_TIME, and (3) multi-core architecture. The benchmark results are shown at the bottom. An example of libvirt XML for creating such VM is

```
[...]
  <vcpu placement='static'>8</vcpu>
  <cpu mode='host-model'>
    <topology sockets='1' cores='8' threads='1'/>
  </cpu>
  <qemu:commandline>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='host,l3-cache=on,+kvm-pv-unhalt,+kvm-steal-time'/>
  </qemu:commandline>
[...]
```

I investigate the cause and found that the problem occurs in the following ways:

- vCPU1 schedules thread A, and vCPU2 schedules thread B. vCPU1 and vCPU2 share LLC.
- Thread A tries to acquire a lock but fails, resulting in a sleep state (via futex.)
- vCPU1 becomes idle because there are no runnable threads and does HLT, which leads to HLT VMEXIT (if idle=halt, and KVM doesn't disable HLT VMEXIT using KVM_CAP_X86_DISABLE_EXITS).
- KVM sets vCPU1's st->preempted as 1 in kvm_steal_time_set_preempted().
- Thread C wakes on vCPU2. vCPU2 tries to do load balancing in select_idle_core(). Although vCPU1 is idle, vCPU1 is not a candidate for load balancing because is_vcpu_preempted(vCPU1) is true, hence available_idle_cpu(vPCU1) is false.
- As a result, both thread B and thread C stay in the vCPU2's runqueue, and vCPU1 is not utilized.

The patch changes kvm_arch_cpu_put() so that it does not set st->preempted as 1 when a vCPU does HLT VMEXIT. As a result, is_vcpu_preempted(vCPU) becomes 0, and the vCPU becomes a candidate for CFS load balancing.

The followings are parts of benchmark results of NPB-OMP
(https://www.nas.nasa.gov/publications/npb.html), which contains several parallel computing programs. My machine has two nodes, and each CPU has 24 cores (Intel Xeon Platinum 8160, hyper-threading disabled.) I created a VM with 48 vCPU, and each vCPU is pinned to the corresponding pCPU. I also created virtual NUMA so that the guest environment became as close as the host. Values in the tables are execution time (seconds; lower is better).

| environmnent \ benchmark name | lu.C   | mg.C  | bt.C  | cg.C  |
|-------------------------------+--------+-------+-------+-------|
| host (Linux v5.13-rc3)        | 50.67  | 14.67 | 54.77 | 20.08 |
| VM (sockets=48, cores=1)      | 51.37  | 14.88 | 55.99 | 20.05 |
| VM (sockets=2, cores=24)      | 170.12 | 23.86 | 75.95 | 40.15 |
|   w/ this patch               | 48.92  | 14.95 | 55.23 | 20.09 |

is_vcpu_preempted() is also used in PV spinlock implementations to mitigate lock holder preemption problems, etc. A vCPU holding a lock does not do HLT, so I think this patch doesn't affect them. However, pCPU may be running the host's thread that has higher priority than a vCPU thread, and in that case, is_vcpu_preempted() should return 0 ideally. I guess its implementation would be a bit complicated, so I wonder if this patch approach is acceptable.

Thanks,

Masanori Misono (1):
KVM: x86: Don't set preempted when vCPU does HLT VMEXIT

 arch/x86/kvm/x86.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

base-commit: c4681547bcce777daf576925a966ffa824edd09d

Signature · 1 line
2.31.1