Skip to content

Add migration support to the MSHV accelerator

Magnus Kulke kvm qemu-devel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest34 patches 19 repliesIn review

Changes since RFC from the cover letter

- Added CET SS/IBT MSR migration
- Assert 64bit padding on CPUX86State->sysenter_cs statically

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/34target/i386/mshv: use arch_load/store_reg fnsaccel/mshv/mshv-all.c include/system/mshv_int.h target/i386/mshv/mshv-cpu.c+23 -370 replies
2/34target/i386/mshv: use generic FPU/xcr0 stateinclude/system/mshv_int.h target/i386/mshv/mshv-cpu.c+47 -440 replies
3/34target/i386/mshv: impl init/load/store_vcpu_stateaccel/mshv/mshv-all.c include/system/mshv_int.h target/i386/mshv/mshv-cpu.c+185 -1840 replies
4/34accel/accel-irq: add AccelRouteChange abstractionaccel/accel-irq.c accel/kvm/kvm-all.c accel/stubs/kvm-stub.c +8 more+41 -240 replies
5/34accel/accel-irq: add generic begin_route_changesaccel/accel-irq.c hw/misc/ivshmem-pci.c hw/vfio/pci.c +4 more+30 -140 replies
6/34accel/accel-irq: add generic commit_route_changesaccel/accel-irq.c hw/misc/ivshmem-pci.c hw/vfio/pci.c +2 more+7 -190 replies
7/34accel/mshv: add irq_routes to stateaccel/mshv/irq.c accel/mshv/mshv-all.c include/system/mshv.h +1 more+24 -40 replies
8/34accel/mshv: update s->irq_routes in add_msi_routeaccel/accel-irq.c accel/mshv/irq.c accel/stubs/mshv-stub.c +1 more+84 -100 replies
9/34accel/mshv: update s->irq_routes in update_msi_routeaccel/mshv/irq.c+38 -490 replies
10/34accel/mshv: update s->irq_routes in release_virqaccel/accel-irq.c accel/mshv/irq.c accel/stubs/mshv-stub.c +1 more+17 -320 replies
11/34accel/mshv: use s->irq_routes in commit_routesaccel/accel-irq.c accel/mshv/irq.c accel/stubs/mshv-stub.c +1 more+7 -60 replies
12/34accel/mshv: reserve ioapic routes on s->irq_routesaccel/mshv/irq.c accel/mshv/mshv-all.c include/system/mshv_int.h+18 -1030 replies
13/34accel/mshv: remove redundant msi controlleraccel/mshv/irq.c accel/mshv/mshv-all.c include/system/mshv_int.h+0 -160 replies
14/34target/i386/mshv: move apic logic into own fileinclude/system/mshv_int.h target/i386/mshv/meson.build target/i386/mshv/mshv-apic.c+87 -00 replies
15/34target/i386/mshv: remove redundant apic helperstarget/i386/mshv/mshv-cpu.c+3 -850 replies
16/34target/i386/mshv: migrate LAPIC statehw/intc/apic_common.c include/hw/i386/apic_internal.h target/i386/mshv/mshv-cpu.c+67 -210 replies
17/34target/i386/mshv: move msr code to archaccel/mshv/meson.build target/i386/mshv/meson.build target/i386/mshv/msr.c+1 -10 replies
18/34accel/mshv: store partition proc featuresaccel/mshv/mshv-all.c include/hw/hyperv/hvhdk.h include/system/mshv_int.h+209 -00 replies
19/34target/i386/mshv: expose msvh_get_generic_regsinclude/system/mshv_int.h target/i386/mshv/mshv-cpu.c+8 -90 replies
20/34target/i386/mshv: migrate MSRsinclude/hw/hyperv/hvgdk_mini.h include/system/mshv_int.h target/i386/mshv/mshv-cpu.c +1 more+259 -3680 replies
21/34target/i386/mshv: migrate MTRR MSRstarget/i386/mshv/msr.c+129 -70 replies
22/34target/i386/mshv: migrate Synic SINT MSRstarget/i386/mshv/msr.c+41 -00 replies
23/34target/i386/mshv: migrate CET/SS MSRsinclude/hw/hyperv/hvgdk_mini.h target/i386/mshv/msr.c+41 -00 replies
24/34target/i386/mshv: migrate SIMP and SIEFP stateinclude/system/mshv_int.h target/i386/cpu.h target/i386/machine.c +4 more+260 -50 replies
25/34target/i386/mshv: migrate STIMER stateinclude/system/mshv_int.h target/i386/cpu.h target/i386/machine.c +2 more+90 -00 replies
26/34accel/mshv: introduce SaveVMHandleraccel/mshv/mshv-all.c+75 -05 replies
27/34accel/mshv: write synthetic MSRs after migrationaccel/mshv/mshv-all.c include/hw/hyperv/hvgdk_mini.h include/system/mshv_int.h +1 more+26 -00 replies
28/34accel/mshv: migrate REFERENCE_TIMEaccel/mshv/mshv-all.c+69 -10 replies
29/34target/i386/mshv: migrate pending ints/excsinclude/hw/hyperv/hvgdk_mini.h include/system/mshv_int.h target/i386/mshv/mshv-cpu.c+184 -00 replies
30/34target/i386: add de/compaction to xsave_helpertarget/i386/cpu.h target/i386/xsave_helper.c+257 -03 replies
31/34target/i386/mshv: migrate XSAVE statetarget/i386/cpu.h target/i386/mshv/mshv-cpu.c+100 -20 replies
32/34target/i386/mshv: reconstruct hflags after loadtarget/i386/mshv/mshv-cpu.c+13 -00 replies
33/34target/i386/mshv: migrate MP_STATEaccel/mshv/mshv-all.c include/system/mshv_int.h target/i386/mshv/mshv-cpu.c+91 -00 replies
34/34accel/mshv: enable dirty page trackingaccel/mshv/mem.c accel/mshv/mshv-all.c include/system/mshv_int.h+219 -00 replies

Reviews trailers per patch

1 of 34 patches has a Reviewed-by or Acked-by.

PatchMohamed Mediouni
1/34
2/34
3/34
4/34
5/34
6/34
7/34
8/34
9/34
10/34
11/34
12/34
13/34
14/34
15/34
16/34
17/34
18/34
19/34
20/34
21/34
22/34
23/34
24/34
25/34
26/34
27/34
28/34
29/34
30/34R-b
31/34
32/34
33/34
34/34

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH 00/34] Add migration support to the MSHV accelerator

Hey all, This is a patch series for live migration support in the MSHV accelerator. Since this is somewhat invasive and touches various parts outside of the accel's folder hierarchies I'm sending an RFC series to collect early feedback.

This patches are sent in sent in isolation, but for LM to be fully functional we will require require the recently submitted patch series "Support QEMU cpu models in MSHV accelerator" (at v6 currently) to be merged, since on some hosts we will need the CPUID infra to disable features that we currently are not able to migrate yet (e.g. AMX tiles).

In this series we perform some preperatory refactorings and introduce new abstractions where required, particulary for irqchips and MSR logic. We also have to introduce some generic logic for XSAVE de/compaction to allow migration of XSAVE state.

Note: there is some pending optimization to avoid a buffer copy and rework existing xsave helper code to be generic over a compacted or standard layout. We'll address this in a later revision or in a discrete patch.

The guest state components that are covered by migration are:

- standard regs
- special regs
- xcr0
- (legacy) FPU regs
- XSAVE
- LAPIC
- MSRs
- SynIC state (SIMP, SIEFP, STIMER)
- pending interrupts/exceptions
- MP state (AP cpu modes)

Finally, routines for dirty-page tracking to reduce migration downtime have beend added and integrated in the respective hooks.

best,

magnus

Changes since RFC:

- Added CET SS/IBT MSR migration
- Assert 64bit padding on CPUX86State->sysenter_cs statically

Magnus Kulke (34):
target/i386/mshv: use arch_load/store_reg fns target/i386/mshv: use generic FPU/xcr0 state target/i386/mshv: impl init/load/store_vcpu_state accel/accel-irq: add AccelRouteChange abstraction accel/accel-irq: add generic begin_route_changes accel/accel-irq: add generic commit_route_changes accel/mshv: add irq_routes to state accel/mshv: update s->irq_routes in add_msi_route accel/mshv: update s->irq_routes in update_msi_route accel/mshv: update s->irq_routes in release_virq accel/mshv: use s->irq_routes in commit_routes accel/mshv: reserve ioapic routes on s->irq_routes accel/mshv: remove redundant msi controller target/i386/mshv: move apic logic into own file target/i386/mshv: remove redundant apic helpers target/i386/mshv: migrate LAPIC state target/i386/mshv: move msr code to arch accel/mshv: store partition proc features target/i386/mshv: expose msvh_get_generic_regs target/i386/mshv: migrate MSRs target/i386/mshv: migrate MTRR MSRs target/i386/mshv: migrate Synic SINT MSRs target/i386/mshv: migrate CET/SS MSRs target/i386/mshv: migrate SIMP and SIEFP state target/i386/mshv: migrate STIMER state accel/mshv: introduce SaveVMHandler accel/mshv: write synthetic MSRs after migration accel/mshv: migrate REFERENCE_TIME target/i386/mshv: migrate pending ints/excs target/i386: add de/compaction to xsave_helper target/i386/mshv: migrate XSAVE state target/i386/mshv: reconstruct hflags after load target/i386/mshv: migrate MP_STATE accel/mshv: enable dirty page tracking

 accel/accel-irq.c               |  41 +-
 accel/kvm/kvm-all.c             |   6 +-
 accel/mshv/irq.c                | 360 ++++++------
 accel/mshv/mem.c                | 211 +++++++
 accel/mshv/meson.build          |   1 -
 accel/mshv/mshv-all.c           | 243 +++++++-
 accel/mshv/msr.c                | 375 -------------
 accel/stubs/kvm-stub.c          |   2 +-
 accel/stubs/mshv-stub.c         |   6 +-
 hw/intc/apic_common.c           |   3 +
 hw/misc/ivshmem-pci.c           |   8 +-
 hw/vfio/pci.c                   |  11 +-
 hw/virtio/virtio-pci.c          |   3 +-
 include/accel/accel-route.h     |  17 +
 include/hw/hyperv/hvgdk_mini.h  |  33 ++
 include/hw/hyperv/hvhdk.h       | 150 +++++
 include/hw/i386/apic_internal.h |   5 +
 include/system/accel-irq.h      |   6 +-
 include/system/kvm.h            |  23 +-
 include/system/mshv.h           |  15 +-
 include/system/mshv_int.h       |  89 +--
 target/i386/cpu.h               |  14 +-
 target/i386/kvm/kvm.c           |   5 +-
 target/i386/machine.c           |  46 ++
 target/i386/mshv/meson.build    |   3 +
 target/i386/mshv/mshv-apic.c    |  78 +++
 target/i386/mshv/mshv-cpu.c     | 958 +++++++++++++++++++++++---------
 target/i386/mshv/msr.c          | 467 ++++++++++++++++
 target/i386/mshv/synic.c        | 206 +++++++
 target/i386/xsave_helper.c      | 255 +++++++++
 30 files changed, 2684 insertions(+), 956 deletions(-)
 delete mode 100644 accel/mshv/msr.c
 create mode 100644 include/accel/accel-route.h
 create mode 100644 target/i386/mshv/mshv-apic.c
 create mode 100644 target/i386/mshv/msr.c
 create mode 100644 target/i386/mshv/synic.c
Signature · 1 line
2.34.1