Skip to content

Optimize KVM/ARM for VHE systems

Christoffer Dall kvm kvmarm linux-arm-kernel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v5 Latest40 patches 5 repliesIn review

Changes in v5 from the cover letter

- Rebased on v4.16-rc3 and other patches queued for kvmarm/next.
- Mostly cosmetic, detailed changelogs are in the individual patches.
- Patch 13 introduces an additional ISB on return from the VHE run function which we lost now when we no longer call the kvm_call_hyp() wrapper which used to take care of that for us.

Patches in v5 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/40KVM: arm/arm64: Avoid vcpu_load for other vcpu ioctls than KVM_RUNarch/arm64/kvm/guest.c virt/kvm/arm/arm.c+0 -121 reply
2/40KVM: arm/arm64: Move vcpu_load call after kvm_vcpu_first_run_initvirt/kvm/arm/arch_timer.c virt/kvm/arm/arm.c virt/kvm/arm/vgic/vgic-init.c+8 -290 replies
3/40KVM: arm64: Avoid storing the vcpu pointer on the stackarch/arm64/include/asm/kvm_asm.h arch/arm64/include/asm/kvm_host.h arch/arm64/kernel/asm-offsets.c +4 more+48 -271 reply
4/40KVM: arm64: Rework hyp_panic for VHE and non-VHEarch/arm64/kvm/hyp/switch.c+23 -190 replies
5/40KVM: arm64: Move HCR_INT_OVERRIDE to default HCR_EL2 guest flagarch/arm64/include/asm/kvm_arm.h arch/arm64/kvm/hyp/switch.c+2 -50 replies
6/40KVM: arm/arm64: Get rid of vcpu->arch.irq_linesarch/arm/include/asm/kvm_emulate.h arch/arm/include/asm/kvm_host.h arch/arm/kvm/emulate.c +7 more+16 -370 replies
7/40KVM: arm/arm64: Add kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregsarch/arm/include/asm/kvm_host.h arch/arm64/include/asm/kvm_host.h arch/arm64/kvm/hyp/sysreg-sr.c +1 more+38 -00 replies
8/40KVM: arm/arm64: Introduce vcpu_el1_is_32bitarch/arm64/include/asm/kvm_emulate.h arch/arm64/kvm/hyp/switch.c arch/arm64/kvm/hyp/sysreg-sr.c +1 more+17 -120 replies
9/40KVM: arm64: Move debug dirty flag calculation out of world switcharch/arm64/kvm/debug.c arch/arm64/kvm/hyp/debug-sr.c+5 -60 replies
10/40KVM: arm64: Slightly improve debug save/restore functionsarch/arm64/kvm/hyp/debug-sr.c+12 -140 replies
11/40KVM: arm64: Improve debug register save/restore flowarch/arm64/include/asm/kvm_hyp.h arch/arm64/kvm/hyp/debug-sr.c arch/arm64/kvm/hyp/switch.c+42 -300 replies
12/40KVM: arm64: Factor out fault info population and gic workaroundsarch/arm64/kvm/hyp/switch.c+57 -470 replies
13/40KVM: arm64: Introduce VHE-specific kvm_vcpu_runarch/arm/include/asm/kvm_asm.h arch/arm/kvm/hyp/switch.c arch/arm64/include/asm/kvm_asm.h +3 more+87 -91 reply
14/40KVM: arm64: Remove kern_hyp_va() use in VHE switch functionarch/arm64/kvm/hyp/switch.c+1 -30 replies
15/40KVM: arm64: Don't deactivate VM on VHE systemsarch/arm64/kvm/hyp/switch.c+3 -50 replies
16/40KVM: arm64: Remove noop calls to timer save/restore from VHE switcharch/arm64/kvm/hyp/switch.c virt/kvm/arm/hyp/timer-sr.c+22 -240 replies
17/40KVM: arm64: Move userspace system registers into separate functionarch/arm64/kvm/hyp/sysreg-sr.c+35 -131 reply
18/40KVM: arm64: Rewrite sysreg alternatives to static keysarch/arm64/kvm/hyp/sysreg-sr.c+4 -130 replies
19/40KVM: arm64: Introduce separate VHE/non-VHE sysreg save/restore functionsarch/arm64/include/asm/kvm_hyp.h arch/arm64/kvm/hyp/switch.c arch/arm64/kvm/hyp/sysreg-sr.c+50 -220 replies
20/40KVM: arm/arm64: Remove leftover comment from kvm_vcpu_run_vhearch/arm64/kvm/hyp/switch.c+0 -40 replies
21/40KVM: arm64: Unify non-VHE host/guest sysreg save and restore functionsarch/arm64/include/asm/kvm_hyp.h arch/arm64/kvm/hyp/switch.c arch/arm64/kvm/hyp/sysreg-sr.c+9 -250 replies
22/40KVM: arm64: Don't save the host ELR_EL2 and SPSR_EL2 on VHE systemsarch/arm64/kvm/hyp/sysreg-sr.c+13 -00 replies
23/40KVM: arm64: Change 32-bit handling of VM system registersarch/arm64/include/asm/kvm_host.h arch/arm64/kvm/sys_regs.c+15 -130 replies
24/40KVM: arm64: Rewrite system register accessors to read/write functionsarch/arm64/include/asm/kvm_emulate.h arch/arm64/include/asm/kvm_host.h arch/arm64/include/asm/kvm_mmu.h +6 more+101 -760 replies
25/40KVM: arm64: Introduce framework for accessing deferred sysregsarch/arm64/include/asm/kvm_host.h arch/arm64/kvm/sys_regs.c+39 -20 replies
26/40KVM: arm/arm64: Prepare to handle deferred save/restore of SPSR_EL1arch/arm/include/asm/kvm_emulate.h arch/arm/kvm/emulate.c arch/arm64/include/asm/kvm_emulate.h +2 more+51 -100 replies
27/40KVM: arm64: Prepare to handle deferred save/restore of ELR_EL1arch/arm64/include/asm/kvm_emulate.h arch/arm64/kvm/inject_fault.c+19 -30 replies
28/40KVM: arm64: Defer saving/restoring 64-bit sysregs to vcpu load/put on VHEarch/arm64/kvm/hyp/sysreg-sr.c arch/arm64/kvm/sys_regs.c+80 -80 replies
29/40KVM: arm64: Prepare to handle deferred save/restore of 32-bit registersarch/arm64/include/asm/kvm_emulate.h arch/arm64/kvm/regmap.c+59 -400 replies
30/40KVM: arm64: Defer saving/restoring 32-bit sysregs to vcpu load/putarch/arm64/kvm/hyp/switch.c arch/arm64/kvm/hyp/sysreg-sr.c arch/arm64/kvm/sys_regs.c+27 -110 replies
31/40KVM: arm64: Move common VHE/non-VHE trap config in separate functionsarch/arm64/kvm/hyp/switch.c+46 -310 replies
32/40KVM: arm64: Directly call VHE and non-VHE FPSIMD enabled functionsarch/arm64/kvm/hyp/switch.c+3 -120 replies
33/40KVM: arm64: Configure c15, PMU, and debug register traps on cpu load/put for VHEarch/arm64/include/asm/kvm_hyp.h arch/arm64/kvm/hyp/switch.c arch/arm64/kvm/hyp/sysreg-sr.c+29 -90 replies
34/40KVM: arm64: Cleanup __activate_traps and __deactive_traps for VHE and non-VHEarch/arm64/kvm/hyp/switch.c+10 -120 replies
35/40KVM: arm/arm64: Get rid of vgic_elrsrinclude/kvm/arm_vgic.h virt/kvm/arm/hyp/vgic-v2-sr.c virt/kvm/arm/hyp/vgic-v3-sr.c +2 more+10 -280 replies
36/40KVM: arm/arm64: Handle VGICv2 save/restore from the main VGIC codearch/arm/kvm/hyp/switch.c arch/arm64/include/asm/kvm_hyp.h arch/arm64/kvm/hyp/switch.c +4 more+84 -761 reply
37/40KVM: arm/arm64: Move arm64-only vgic-v2-sr.c file to arm64arch/arm/kvm/hyp/Makefile arch/arm64/kvm/hyp/Makefile arch/arm64/kvm/hyp/vgic-v2-cpuif-proxy.c+1 -40 replies
38/40KVM: arm/arm64: Handle VGICv3 save/restore from the main VGIC code on VHEarch/arm64/kvm/hyp/switch.c virt/kvm/arm/arm.c virt/kvm/arm/vgic/vgic.c+26 -90 replies
39/40KVM: arm/arm64: Move VGIC APR save/restore to vgic put/loadarch/arm/include/asm/kvm_hyp.h arch/arm64/include/asm/kvm_hyp.h virt/kvm/arm/hyp/vgic-v3-sr.c +2 more+78 -620 replies
40/40KVM: arm/arm64: Avoid VGICv3 save/restore on VHE with no IRQsarch/arm/include/asm/kvm_hyp.h arch/arm/kvm/hyp/switch.c arch/arm64/include/asm/kvm_hyp.h +4 more+103 -520 replies

Reviews trailers per patch

38 of 40 patches have a Reviewed-by or Acked-by.

PatchJulien GrallMarc ZyngierAndrew JonesJulien ThierryAndre Przywara
1/40R-b R-b R-b
2/40R-b R-b
3/40R-b R-b R-b
4/40A-b R-b
5/40R-b
6/40A-b R-b R-b
7/40A-b R-b
8/40R-b A-b R-b R-b
9/40R-b R-b R-b
10/40R-b R-b
11/40R-b R-b
12/40R-b R-b
13/40R-b R-b
14/40R-b R-b
15/40A-b R-b
16/40A-b R-b
17/40R-b R-b R-b
18/40A-b R-b
19/40R-b R-b
20/40A-b R-b
21/40R-b R-b
22/40R-b R-b
23/40R-b R-b
24/40A-b R-b
25/40R-b R-b
26/40R-b R-b
27/40R-b R-b
28/40R-b
29/40R-b R-b
30/40R-b R-b
31/40R-b R-b
32/40R-b R-b
33/40R-b R-b
34/40R-b
35/40R-b
36/40R-b
37/40R-b
38/40
39/40R-b
40/40

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 5 reviewers.

Read the v5 thread

Cover letter [PATCH v5 00/40] Optimize KVM/ARM for VHE systems

This series redesigns parts of KVM/ARM to optimize the performance on VHE systems. The general approach is to try to do as little work as possible when transitioning between the VM and the hypervisor. This has the benefit of lower latency when waiting for interrupts and delivering virtual interrupts, and reduces the overhead of emulating behavior and I/O in the host kernel.

Patches 01 through 06 are not VHE specific, but rework parts of KVM/ARM that can be generally improved. We then add infrastructure to move more logic into vcpu_load and vcpu_put, we improve handling of VFP and debug registers.

We then introduce a new world-switch function for VHE systems, which we can tweak and optimize for VHE systems. To do that, we rework a lot of the system register save/restore handling and emulation code that may need access to system registers, so that we can defer as many system register save/restore operations to vcpu_load and vcpu_put, and move this logic out of the VHE world switch function.

We then optimize the configuration of traps. On non-VHE systems, both the host and VM kernels run in EL1, but because the host kernel should have full access to the underlying hardware, but the VM kernel should not, we essentially make the host kernel more privileged than the VM kernel despite them both running at the same privilege level by enabling VE traps when entering the VM and disabling those traps when exiting the VM. On VHE systems, the host kernel runs in EL2 and has full access to the hardware (as much as allowed by secure side software), and is unaffected by the trap configuration. That means we can configure the traps for VMs running in EL1 once, and don't have to switch them on and off for every entry/exit to/from the VM.

Finally, we improve our VGIC handling by moving all save/restore logic out of the VHE world-switch, and we make it possible to truly only evaluate if the AP list is empty and not do *any* VGIC work if that is the case, and only do the minimal amount of work required in the course of the VGIC processing when we have virtual interrupts in flight.

The patches are based on v4.16-rc3 with kvmarm/next and kvmarm/master applied on top.

I've given the patches a fair amount of testing on Thunder-X, Mustang, Seattle, and TC2 (32-bit) for non-VHE testing, and tested VHE functionality on TX2 and on the foundation model.

The patches are also available in the vhe-optimize-v5 branch on my kernel.org repository [1]. The vhe-optimize-v5-base branch contains prerequisites of this series.

Patches 36, 38, and 40 are missing Acked-by or Reviewed-by tags.

[1]: git://git.kernel.org/pub/scm/linux/kernel/git/cdall/linux.git vhe-optimize-v5

Changes since v4:
- Rebased on v4.16-rc3 and other patches queued for kvmarm/next.
- Mostly cosmetic, detailed changelogs are in the individual patches.
- Patch 13 introduces an additional ISB on return from the VHE run function which we lost now when we no longer call the kvm_call_hyp() wrapper which used to take care of that for us.

Changes since v3:
- Rebased on v4.16-rc1 (fun!)
- Removed VFP optimizations because it was discovered that the deferred approach taken in v3 was buggy
(https://lists.cs.columbia.edu/pipermail/kvmarm/2018-February/029838.html)
This causes a fair amount of changes throughout and I've removed affected reviewed-by and other tags as best I could to take this into account.
- Used a switch-statement to handle deferred system registers instead of the macro approach taken in v3.
- Addressed other review coments (tried to keep track of this with logs of changes in individual patches).
- I ran Yury's IPI benchark test and other tests on TX2 and could not observe a performance regression, but rather an improvement of around 65%. I suspect the previous regression was due to the timer WFI problem solved for v4.16-rc1.
- I haven't included Tomasz' reviewed-by, because I figured too much of the series has changed since v3.

Changes since v2:
- Rebased on v4.15-rc3.
- Includes two additional patches that only does vcpu_load after kvm_vcpu_first_run_init and only for KVM_RUN.
- Addressed review comments from v2 (detailed changelogs are in the individual patches).

Thanks,
-Christoffer

Christoffer Dall (39):
KVM: arm/arm64: Avoid vcpu_load for other vcpu ioctls than KVM_RUN
KVM: arm/arm64: Move vcpu_load call after kvm_vcpu_first_run_init
KVM: arm64: Avoid storing the vcpu pointer on the stack
KVM: arm64: Rework hyp_panic for VHE and non-VHE
KVM: arm/arm64: Get rid of vcpu->arch.irq_lines
KVM: arm/arm64: Add kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs
KVM: arm/arm64: Introduce vcpu_el1_is_32bit
KVM: arm64: Move debug dirty flag calculation out of world switch
KVM: arm64: Slightly improve debug save/restore functions
KVM: arm64: Improve debug register save/restore flow
KVM: arm64: Factor out fault info population and gic workarounds
KVM: arm64: Introduce VHE-specific kvm_vcpu_run
KVM: arm64: Remove kern_hyp_va() use in VHE switch function
KVM: arm64: Don't deactivate VM on VHE systems
KVM: arm64: Remove noop calls to timer save/restore from VHE switch
KVM: arm64: Move userspace system registers into separate function
KVM: arm64: Rewrite sysreg alternatives to static keys
KVM: arm64: Introduce separate VHE/non-VHE sysreg save/restore
functions
KVM: arm/arm64: Remove leftover comment from kvm_vcpu_run_vhe
KVM: arm64: Unify non-VHE host/guest sysreg save and restore functions
KVM: arm64: Don't save the host ELR_EL2 and SPSR_EL2 on VHE systems
KVM: arm64: Change 32-bit handling of VM system registers
KVM: arm64: Rewrite system register accessors to read/write functions
KVM: arm64: Introduce framework for accessing deferred sysregs
KVM: arm/arm64: Prepare to handle deferred save/restore of SPSR_EL1
KVM: arm64: Prepare to handle deferred save/restore of ELR_EL1
KVM: arm64: Defer saving/restoring 64-bit sysregs to vcpu load/put on
VHE
KVM: arm64: Prepare to handle deferred save/restore of 32-bit
registers
KVM: arm64: Defer saving/restoring 32-bit sysregs to vcpu load/put
KVM: arm64: Move common VHE/non-VHE trap config in separate functions
KVM: arm64: Directly call VHE and non-VHE FPSIMD enabled functions
KVM: arm64: Configure c15, PMU, and debug register traps on cpu
load/put for VHE
KVM: arm64: Cleanup __activate_traps and __deactive_traps for VHE and
non-VHE
KVM: arm/arm64: Get rid of vgic_elrsr
KVM: arm/arm64: Handle VGICv2 save/restore from the main VGIC code
KVM: arm/arm64: Move arm64-only vgic-v2-sr.c file to arm64
KVM: arm/arm64: Handle VGICv3 save/restore from the main VGIC code on
VHE
KVM: arm/arm64: Move VGIC APR save/restore to vgic put/load
KVM: arm/arm64: Avoid VGICv3 save/restore on VHE with no IRQs

Shih-Wei Li (1):
KVM: arm64: Move HCR_INT_OVERRIDE to default HCR_EL2 guest flag

 arch/arm/include/asm/kvm_asm.h                     |   5 +-
 arch/arm/include/asm/kvm_emulate.h                 |  21 +-
 arch/arm/include/asm/kvm_host.h                    |   6 +-
 arch/arm/include/asm/kvm_hyp.h                     |   4 +
 arch/arm/kvm/emulate.c                             |   4 +-
 arch/arm/kvm/hyp/Makefile                          |   1 -
 arch/arm/kvm/hyp/switch.c                          |  16 +-
 arch/arm64/include/asm/kvm_arm.h                   |   4 +-
 arch/arm64/include/asm/kvm_asm.h                   |  19 +-
 arch/arm64/include/asm/kvm_emulate.h               |  74 ++--
 arch/arm64/include/asm/kvm_host.h                  |  53 ++-
 arch/arm64/include/asm/kvm_hyp.h                   |  29 +-
 arch/arm64/include/asm/kvm_mmu.h                   |   2 +-
 arch/arm64/kernel/asm-offsets.c                    |   1 +
 arch/arm64/kvm/debug.c                             |  29 +-
 arch/arm64/kvm/guest.c                             |   3 -
 arch/arm64/kvm/hyp/Makefile                        |   2 +-
 arch/arm64/kvm/hyp/debug-sr.c                      |  88 ++---
 arch/arm64/kvm/hyp/entry.S                         |   6 +-
 arch/arm64/kvm/hyp/hyp-entry.S                     |  28 +-
 arch/arm64/kvm/hyp/switch.c                        | 378 ++++++++++++---------
 arch/arm64/kvm/hyp/sysreg-sr.c                     | 172 ++++++++--
 .../arm64/kvm/hyp/vgic-v2-cpuif-proxy.c            |  81 -----
 arch/arm64/kvm/inject_fault.c                      |  24 +-
 arch/arm64/kvm/regmap.c                            |  67 +++-
 arch/arm64/kvm/sys_regs.c                          | 173 ++++++++--
 arch/arm64/kvm/sys_regs.h                          |   4 +-
 arch/arm64/kvm/sys_regs_generic_v8.c               |   4 +-
 include/kvm/arm_vgic.h                             |   2 -
 virt/kvm/arm/aarch32.c                             |   2 +-
 virt/kvm/arm/arch_timer.c                          |   4 -
 virt/kvm/arm/arm.c                                 |  57 ++--
 virt/kvm/arm/hyp/timer-sr.c                        |  44 +--
 virt/kvm/arm/hyp/vgic-v3-sr.c                      | 244 +++++++------
 virt/kvm/arm/mmu.c                                 |   6 +-
 virt/kvm/arm/pmu.c                                 |  36 +-
 virt/kvm/arm/vgic/vgic-init.c                      |  11 -
 virt/kvm/arm/vgic/vgic-v2.c                        |  61 +++-
 virt/kvm/arm/vgic/vgic-v3.c                        |  12 +-
 virt/kvm/arm/vgic/vgic.c                           |  33 ++
 virt/kvm/arm/vgic/vgic.h                           |   3 +
 41 files changed, 1136 insertions(+), 677 deletions(-)
 rename virt/kvm/arm/hyp/vgic-v2-sr.c => arch/arm64/kvm/hyp/vgic-v2-cpuif-proxy.c (51%)
Signature · 1 line
2.14.2