Skip to content

RFC x86/sev: KEXEC/KDUMP support for SEV-ES guests

Joerg Roedel kvm linux-coco lkml stable virtualization 1 version New

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest12 patches 0 repliesNew

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/12kexec: Allow architecture code to opt-out at runtimeinclude/linux/kexec.h kernel/kexec.c kernel/kexec_file.c+25 -00 replies
2/12x86/kexec/64: Forbid kexec when running as an SEV-ES guestarch/x86/kernel/machine_kexec_64.c+8 -00 replies
3/12x86/sev: Save and print negotiated GHCB protocol versionarch/x86/boot/compressed/sev.c arch/x86/kernel/sev-shared.c arch/x86/kernel/sev.c+34 -30 replies
4/12x86/sev: Do not hardcode GHCB protocol versionarch/x86/boot/compressed/sev.c arch/x86/kernel/sev-shared.c arch/x86/kernel/sev.c+14 -10 replies
5/12x86/sev: Use GHCB protocol version 2 if supportedarch/x86/boot/compressed/sev.c arch/x86/include/asm/sev.h arch/x86/kernel/sev-shared.c+24 -70 replies
6/12x86/sev: Cache AP Jump Table Addressarch/x86/kernel/sev.c+14 -120 replies
7/12x86/sev: Setup code to park APs in the AP Jump Tablearch/x86/include/asm/realmode.h arch/x86/include/asm/sev-ap-jumptable.h arch/x86/kernel/sev.c +5 more+341 -10 replies
8/12x86/sev: Park APs on AP Jump Table with GHCB protocol version 2arch/x86/include/asm/realmode.h arch/x86/kernel/sev.c arch/x86/realmode/rm/Makefile +2 more+144 -100 replies
9/12x86/sev: Use AP Jump Table blob to stop CPUarch/x86/include/asm/sev.h arch/x86/kernel/process.c arch/x86/kernel/sev.c+25 -10 replies
10/12x86/sev: Add MMIO handling support to boot/compressed/ codearch/x86/boot/compressed/sev.c arch/x86/kernel/sev-shared.c arch/x86/kernel/sev.c +2 more+1116 -10980 replies
11/12x86/sev: Handle CLFLUSH MMIO eventsarch/x86/kernel/sev-shared.c+9 -00 replies
12/12x86/sev: Support kexec under SEV-ES with AP Jump Table blobarch/x86/include/asm/sev.h arch/x86/kernel/machine_kexec_64.c arch/x86/kernel/sev.c+19 -10 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 00/12] x86/sev: KEXEC/KDUMP support for SEV-ES guests

From: Joerg Roedel <[email protected]>

Hi,

here are changes to enable kexec/kdump in SEV-ES guests. The biggest problem for supporting kexec/kdump under SEV-ES is to find a way to hand the non-boot CPUs (APs) from one kernel to another.

Without SEV-ES the first kernel parks the CPUs in a HLT loop until they get reset by the kexec'ed kernel via an INIT-SIPI-SIPI sequence. For virtual machines the CPU reset is emulated by the hypervisor, which sets the vCPU registers back to reset state.

This does not work under SEV-ES, because the hypervisor has no access to the vCPU registers and can't make modifications to them. So an SEV-ES guest needs to reset the vCPU itself and park it using the AP-reset-hold protocol. Upon wakeup the guest needs to jump to real-mode and to the reset-vector configured in the AP-Jump-Table.

The code to do this is the main part of this patch-set. It works by placing code on the AP Jump-Table page itself to park the vCPU and for jumping to the reset vector upon wakeup. The code on the AP Jump Table runs in 16-bit protected mode with segment base set to the beginning of the page. The AP Jump-Table is usually not within the first 1MB of memory, so the code can't run in real-mode.

The AP Jump-Table is the best place to put the parking code, because the memory is owned, but read-only by the firmware and writeable by the OS. Only the first 4 bytes are used for the reset-vector, leaving the rest of the page for code/data/stack to park a vCPU. The code can't be in kernel memory because by the time the vCPU wakes up the memory will be owned by the new kernel, which might have overwritten it already.

The other patches add initial GHCB Version 2 protocol support, because kexec/kdump need the MSR-based (without a GHCB) AP-reset-hold VMGEXT, which is a GHCB protocol version 2 feature.

The kexec'ed kernel is also entered via the decompressor and needs MMIO support there, so this patch-set also adds MMIO #VC support to the decompressor and support for handling CLFLUSH instructions.

Finally there is also code to disable kexec/kdump support at runtime when the environment does not support it (e.g. no GHCB protocol version 2 support or AP Jump Table over 4GB).

The diffstat looks big, but most of it is moving code for MMIO #VC support around to make it available to the decompressor.

There is also a video showing the code in action:

https://www.youtube.com/watch?v=j1AUJANP7Mk

Please review.

Thanks,

Joerg

Joerg Roedel (12):
kexec: Allow architecture code to opt-out at runtime x86/kexec/64: Forbid kexec when running as an SEV-ES guest x86/sev: Save and print negotiated GHCB protocol version x86/sev: Do not hardcode GHCB protocol version x86/sev: Use GHCB protocol version 2 if supported x86/sev: Cache AP Jump Table Address
x86/sev: Setup code to park APs in the AP Jump Table x86/sev: Park APs on AP Jump Table with GHCB protocol version 2 x86/sev: Use AP Jump Table blob to stop CPU
x86/sev: Add MMIO handling support to boot/compressed/ code x86/sev: Handle CLFLUSH MMIO events
x86/sev: Support kexec under SEV-ES with AP Jump Table blob

 arch/x86/boot/compressed/sev.c          |  56 +-
 arch/x86/include/asm/realmode.h         |   5 +
 arch/x86/include/asm/sev-ap-jumptable.h |  25 +
 arch/x86/include/asm/sev.h              |  13 +-
 arch/x86/kernel/machine_kexec_64.c      |  12 +
 arch/x86/kernel/process.c               |   8 +
 arch/x86/kernel/sev-shared.c            | 333 +++++++++-
 arch/x86/kernel/sev.c                   | 494 ++++++---------
 arch/x86/lib/insn-eval-shared.c         | 805 ++++++++++++++++++++++++
 arch/x86/lib/insn-eval.c                | 802 +----------------------
 arch/x86/realmode/Makefile              |   9 +-
 arch/x86/realmode/rm/Makefile           |  11 +-
 arch/x86/realmode/rm/header.S           |   3 +
 arch/x86/realmode/rm/sev_ap_park.S      |  89 +++
 arch/x86/realmode/rmpiggy.S             |   6 +
 arch/x86/realmode/sev/Makefile          |  41 ++
 arch/x86/realmode/sev/ap_jump_table.S   | 130 ++++
 arch/x86/realmode/sev/ap_jump_table.lds |  24 +
 include/linux/kexec.h                   |   2 +
 kernel/kexec.c                          |  14 +
 kernel/kexec_file.c                     |   9 +
 21 files changed, 1765 insertions(+), 1126 deletions(-)
 create mode 100644 arch/x86/include/asm/sev-ap-jumptable.h
 create mode 100644 arch/x86/lib/insn-eval-shared.c
 create mode 100644 arch/x86/realmode/rm/sev_ap_park.S
 create mode 100644 arch/x86/realmode/sev/Makefile
 create mode 100644 arch/x86/realmode/sev/ap_jump_table.S
 create mode 100644 arch/x86/realmode/sev/ap_jump_table.lds

base-commit: 8d9d46bbf3b6b7ff8edcac33603ab45c29e0e07f

Signature · 1 line
2.31.1