liveupdate: kvm: Guest_memfd preservation
Subsystems (8)
v4 cover liveupdate: kvm: Guest_memfd preservation
Hello,
This is v4 of guest_memfd preservation during liveupdate. As per the discussion from bi-weekly guest-memfd meeting, This series will be go to linux via kvm tree. And Planning to get it merged in upcoming merge window. (7.2-rc1 for release in 7.3)
The changes are rebased on:
kvm/next + liveupdate/next (merge) + [4] + [5] Where,
[4]: luo: APIs to retrieve file internally from session [5]: selftests: liveupdate sefltests library
Here is the github repo:
https://github.com/tar-unix/linux/tree/gmem-pre
Patches [4] and [5] are prequisite of this series and they are rebased/merged on liveupdate/next. kvm/next is behind few patches from liveupdate/next, hence direct cherry-pick of [4] and [5] is not possible. On Linux 7.2 they will be aligned. If we succeed in merging it before I can request liveupdate maintainer to provide the tag.
Steps to test:
1. Compile Kernel with CONFIG_LIVEUPDATE_GUEST_MEMFD=y
2. boot kernel with command line: kho=on liveupdate=on
3. run the following kselftest
$ .selftests/kvm/guest_memfd_preservation_test --s 1 $ <kexec> --reuse-cmdline $ .selftests/kvm/guest_memfd_preservation_test --s 2
NOTE: Assert the following:
$ ls /dev/liveupdate $ ls /dev/kvm $ dmesg | grep liveupdate # (should have kvm_vm_luo && # guest_memfd_luo handler registered)
V4 <- V3
1. Split the refactoring patch into multiple patches [2, 3, 4]
2. Updated commits msg as per the kvm documentation and suggestions
3. Removed KHOSER_PTR and using raw pointer, as series [3] not yet decided to go in the next cycle
4. Implemented more tests in guest_memfd_preservation_test allocation during frozen gmem_inode, non-guest_memfd preservation
5. Moved the vm_create_from_fd to kvm_util.c from the guest_memfd_preservation_test.c
5. Minor updates as per the suggestions
V3 <- RFC V2 [2]
1. Finalize the design
2. resolve sashiko reported bugs
3. Use of KHOSER_PTR instead of raw serialized_data as per [3]
RFC V2 [2] <- RFC V1 [1]
1. Removed mem_attr_array as it is not needed for fully-shared
2. Removed pre-faulted condition
3. Added vm_type preservation for ARM64.
4. Removed liveupdate_get_file_incoming api patch as it is sent separately [4] by Samiullah.
**GUEST MEMFD PRESERVATION**
Below is the cover letter of the series which explains the design.
SCOPE:
1. Fully Shared Guest_memfd
2. Guest_memfd backed by PAGE_SIZE pages
Any VM whose memory is backed by such guest_memfd and satisfy the above conditions can be preserved across liveupdate.
To Add support for liveupdate of any subsystem, we need to implement the following calls: (e.g: subsystem: guest_memfd)
guest_memfd_luo_can_preserve: used by luo subsystem to filter the
guest_memfd filesguest_memfd_luo_preserve: triggered from userspace by PRESERVE ioctl
If can_preserve says the file is preservable by guest_memfd_luo_preserveguest_memfd_luo_retrieve: triggered from userspace by RETRIEVE ioctl,
recognised by the passed token from userspace. Used in new kernel after kexec.guest_memfd_luo_unpreserve: triggered from userspace when a session is
closed, which inherently calls unpreserve of all preserved files.guest_memfd_luo_freeze: triggered just before kexec automatically in
kernel. no userspace involvement.guest_memfd_luo_finish: triggered by userspace when all files are
retrieved and want to finish the session retrieval.
*PART A: KVM VM preservation*
Introdcued the kvm_luo.c to preserve the vm_file during liveupdate. Currently it only preserve vm_type. This can be expanded to preserve more details in future as need arises for example data related to cocoVM (SEPT etc.).
int kvm_fd = open("/dev/kvm", O_RDWR);
int vm_fd = ioctl(kvm_fd, KVM_CREATE_VM, 0);
preserve_arg = {
.size = sizeof(preserve_arg),
.fd = vm_fd,
.token = VM_TOKEN,
};
ioctl(session_fd, LIVEUPDATE_SESSION_PRESERVE_FD, &preserve_arg);This VM_TOKEN is used by guest_memfd preservation. guest_memfd preservation save this token (fetched internally by luo apis [4]) to associates the guest_memfd with its vm_file. On retrieval, guest_memfd retrieves the vm_file using this token and use its struct kvm to retrieve itself.
PART B: Guest_memfd preservation
Introduce guest_memfd_luo.c to preserve the guest_memfd file during liveupdate. Currently it only preserve guest_memfd files that have:
1. INIT_SHARED flag ensures intially all the memory is shared
2. !kvm_arch_has_private_mem() which ensures no shared pages can be converted to private after [6] lands.
Preservation first freezes the gmem_inode introduced in PATCH [07/11] and preserve each allocated folios from guest_memfd page_cache using KHO.
luo_freeze (not the gmem_inode freeze PATCH[07/11]) call update the vm_token which is preserved across kexec.
Retrieval retrieves the vm_file using vm_token and create the guest_memfd with vm_file->kvm and populate the preserved folios to guest_memfd page_cache.
Added the Documentation for VMM on guest_memfd preservation guidelines To know more about, PATCH[10/11] can be referred which has the more detailed documentation.
Future Plan:
1. Make preservation immune to new allocation and fallocate calls so that we can remove freeze call
2. In future, When IOMMU will use guest_memfd and can allocate page tables for DMA in that area, Later it will be a problem if other user of guest_memfd do punch hole before preservation, that will make the pages to be lost after preservation. This will be taken care of when IOMMU will add support for guest_memfd.
3. guest_memfd preservation for private pages + in-place conversion.
4. In future, when guest_memfd will have hugetlb page support, add support its preservation.
[1]: https://lore.kernel.org/all/[email protected]/
[2]: https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/c054ba0fb2639932bbe354420d3f4f84cce84905.1780676742.git.tarunsahu@google.com/ [3]: https://lore.kernel.org/all/[email protected]/ [4]: https://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git/commit/?h=next&id=f4d2e4f0d12d88155cd1128ed3294f7827bf4c5d [5]: https://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git/commit/?h=next&id=2f3f53a3735a7e67bbe9e580cb49372bf9261967 https://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git/commit/?h=next&id=df1a0d068d481b7d175c5af3970043206409a062
[6]: https://lore.kernel.org/all/[email protected]/
Tarun Sahu (11):
liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option
KVM: Introduce kvm_create_vm_file() helper
KVM: Export kvm_uevent_notify_vm_create()
KVM: Track weak reference to vm_file in struct kvm
KVM: LUO: Support VM preservation across live updates
KVM: guest_memfd: Move internal definitions to internal header
KVM: guest_memfd: Add support for freezing mappings
KVM: guest_memfd: Add support for preservation via LUO
docs: liveupdate: Add documentation for VM and guest_memfd
preservation
KVM: selftests: Split ____vm_create() and add vm_create_from_fd()
KVM: selftests: Add guest_memfd_preservation_test
Documentation/core-api/liveupdate.rst | 1 + Documentation/liveupdate/vmm.rst | 107 ++++ MAINTAINERS | 14 + include/linux/kho/abi/kvm.h | 106 ++++ include/linux/kvm_host.h | 14 + kernel/liveupdate/Kconfig | 15 + tools/testing/selftests/kvm/Makefile.kvm | 6 +- .../kvm/guest_memfd_preservation_test.c | 357 ++++++++++++ .../testing/selftests/kvm/include/kvm_util.h | 3 + tools/testing/selftests/kvm/lib/kvm_util.c | 49 +- virt/kvm/Makefile.kvm | 1 + virt/kvm/guest_memfd.c | 185 +++++-- virt/kvm/guest_memfd.h | 44 ++ virt/kvm/guest_memfd_luo.c | 515 ++++++++++++++++++ virt/kvm/kvm_luo.c | 195 +++++++ virt/kvm/kvm_main.c | 94 +++- virt/kvm/kvm_mm.h | 15 + 17 files changed, 1640 insertions(+), 81 deletions(-) create mode 100644 Documentation/liveupdate/vmm.rst create mode 100644 include/linux/kho/abi/kvm.h create mode 100644 tools/testing/selftests/kvm/guest_memfd_preservation_test.c create mode 100644 virt/kvm/guest_memfd.h create mode 100644 virt/kvm/guest_memfd_luo.c create mode 100644 virt/kvm/kvm_luo.c
base-commit: a204badd8432f93b7e862e7dac6db0fe3d65f370
Signature · 1 line
2.55.0.229.g6434b31f56-goog
PATCH v4 1/11 liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option
Introduce the LIVEUPDATE_GUEST_MEMFD Kconfig option. This option enables live update support for KVM guest_memfd files, enabling guest_memfd-backed memory preservation across kernel upgrades.
Currently this support only guest_memfd files that are full-shared (INIT_SHARED and no support for private_mem).
--- kernel/liveupdate/Kconfig | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
kernel/liveupdate/Kconfig +15 -0
@@ -86,4 +86,19 @@ config LIVEUPDATE_MEMFD If unsure, say N. +config LIVEUPDATE_GUEST_MEMFD + bool "Live update support for guest_memfd" + depends on LIVEUPDATE + depends on KVM_GUEST_MEMFD + default LIVEUPDATE + help + Enable live update support for KVM guest_memfd files. This allows + preserving VM Memory backed by guest_memfd file across kernel live + updates. + + This can only be used for the guest_memfd that are fully-shared + and pre-faulted. + + If unsure, say N. + endmenu
Signature · 1 line
2.55.0.229.g6434b31f56-goog
[email protected]
AI reviewRe: [PATCH v4 01/11] liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The patch introduces the `LIVEUPDATE_GUEST_MEMFD` Kconfig option claiming to enable a new…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The patch introduces the LIVEUPDATE_GUEST_MEMFD Kconfig option claiming to enable a new feature, but there is no actual implementation provided. --
--- Patch [1]: [PATCH v4 01/11] liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option --- commit e0c3fde657d56c548cd9a7467d2143f0f42c5242
Author: Tarun Sahu <[email protected]>
liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option
This commit introduces the LIVEUPDATE_GUEST_MEMFD Kconfig option, which aims to enable live update support for KVM guest_memfd files.
kernel/liveupdate/Kconfig config LIVEUPDATE_MEMFD · line 95
Quoted 13 lines
kernel/liveupdate/Kconfig +5
-0@@ -88,4 +88,19 @@ config LIVEUPDATE_MEMFD If unsure, say N. +config LIVEUPDATE_GUEST_MEMFD + bool "Live update support for guest_memfd" + depends on LIVEUPDATE + depends on KVM_GUEST_MEMFD + default LIVEUPDATE
[Severity: Medium]
Is the implementation for this feature missing from this patch?
The commit message notes that this patch enables live update support for KVM guest_memfd files, but the newly added LIVEUPDATE_GUEST_MEMFD symbol appears to be unreferenced in the tree. Since there are no Makefiles, C files, or preprocessor checks that use it, this currently acts as dead code.
Could the implementation be missing, or was this intended to be introduced as part of a larger series where the symbol is used in a subsequent patch?
Quoted 11 lines
kernel/liveupdate/Kconfig +10
-0+ help + Enable live update support for KVM guest_memfd files. This allows + preserving VM Memory backed by guest_memfd file across kernel live + updates. + + This can only be used for the guest_memfd that are fully-shared + and pre-faulted. + + If unsure, say N. + endmenu
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Re: [PATCH v4 01/11] liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option
Quoted 30 lines · [email protected] writes:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The patch introduces theLIVEUPDATE_GUEST_MEMFDKconfigoption claiming to enable a new feature, but there is no actual implementation provided. ----- Patch [1]: [PATCH v4 01/11] liveupdate: Add
LIVEUPDATE_GUEST_MEMFDconfig option --- commite0c3fde657d56c548cd9a7467d2143f0f42c5242
Author: Tarun Sahu <[email protected]>liveupdate: Add
LIVEUPDATE_GUEST_MEMFDconfig optionThis commit introduces the
LIVEUPDATE_GUEST_MEMFDKconfigoption, which aims to enable live update support for KVMguest_memfdfiles.kernel/liveupdate/Kconfig config LIVEUPDATE_MEMFD · line 95
Quoted 13 lines
kernel/liveupdate/Kconfig +5
-0@@ -88,4 +88,19 @@ config LIVEUPDATE_MEMFD If unsure, say N. +config LIVEUPDATE_GUEST_MEMFD + bool "Live update support for guest_memfd" + depends on LIVEUPDATE + depends on KVM_GUEST_MEMFD + default LIVEUPDATE[Severity: Medium]
Is the implementation for this feature missing from this patch?
Quoted 8 lines
The commit message notes that this patch enables live update support for KVM
guest_memfdfiles, but the newly addedLIVEUPDATE_GUEST_MEMFDsymbol appears to be unreferenced in the tree. Since there are no Makefiles, C files, or preprocessor checks that use it, this currently acts as dead code.Could the implementation be missing, or was this intended to be introduced as part of a larger series where the symbol is used in a subsequent patch?
Sashiko is right here, I think this patch should be squashed with the first usage.
Quoted 14 lines
kernel/liveupdate/Kconfig config LIVEUPDATE_MEMFD · line 104
Quoted 11 lines
kernel/liveupdate/Kconfig +10
-0+ help + Enable live update support for KVM guest_memfd files. This allows + preserving VM Memory backed by guest_memfd file across kernel live + updates. + + This can only be used for the guest_memfd that are fully-shared + and pre-faulted. + + If unsure, say N. + endmenu--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Re: [PATCH v4 01/11] liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option
Thanks for reviewing it.
Quoted 17 lines · Ackerley Tng
Quoted 15 lines · [email protected] writes:
kernel/liveupdate/Kconfig config LIVEUPDATE_MEMFD · line 93
Quoted 6 lines
kernel/liveupdate/Kconfig +5
-0+config LIVEUPDATE_GUEST_MEMFD + bool "Live update support for guest_memfd" + depends on LIVEUPDATE + depends on KVM_GUEST_MEMFD + default LIVEUPDATE[Severity: Medium]
Is the implementation for this feature missing from this patch?The commit message notes that this patch enables live update support for KVM
guest_memfdfiles, but the newly addedLIVEUPDATE_GUEST_MEMFDsymbol appears to be unreferenced in the tree. Since there are no Makefiles, C files, or preprocessor checks that use it, this currently acts as dead code.
Quoted 6 lines
Quoted 3 lines
Could the implementation be missing, or was this intended to be introduced as part of a larger series where the symbol is used in a subsequent patch?
Sashiko is right here, I think this patch should be squashed with the first usage.
Sure, will update it.
Quoted 15 lines
Quoted 15 lines
kernel/liveupdate/Kconfig config LIVEUPDATE_MEMFD · line 104
Quoted 12 lines
kernel/liveupdate/Kconfig +10
-0+ help + Enable live update support for KVM guest_memfd files. This allows + preserving VM Memory backed by guest_memfd file across kernel live + updates. + + This can only be used for the guest_memfd that are fully-shared + and pre-faulted. + + If unsure, say N. + endmenu--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Sean Christopherson
maintainerRe: [PATCH v4 01/11] liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option
In reply to Tarun SahuQuoted 6 lines · Tarun Sahu
Introduce the
LIVEUPDATE_GUEST_MEMFDKconfigoption. This option enables live update support for KVMguest_memfdfiles, enabling guest_memfd-backed memory preservation across kernel upgrades.Currently this support only
guest_memfdfiles that are full-shared (INIT_SHAREDand no support forprivate_mem).
Why? Specifically, why do we need/want a separate Kconfig for guest_memfd liveupdate versus KVM liveupdate? I can _maybe_ see a KVM-wide Kconfig, but even that seems like a stretch given that (AFAIK) hypervisors are the one and only use-case for liveupdate.
Re: [PATCH v4 01/11] liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option
Quoted 5 lines · Sean Christopherson
Quoted 4 lines · Tarun Sahu
Introduce the
LIVEUPDATE_GUEST_MEMFDKconfigoption. This option enables live update support for KVMguest_memfdfiles, enabling guest_memfd-backed memory preservation across kernel upgrades.
Quoted 7 lines
Quoted 2 lines
Currently this support only
guest_memfdfiles that are full-shared (INIT_SHAREDand no support forprivate_mem).Why? Specifically, why do we need/want a separate
Kconfigforguest_memfdliveupdate versus KVM liveupdate? I can _maybe_ see a KVM-wideKconfig, but even that seems like a stretch given that (AFAIK) hypervisors are the one and only use-case for liveupdate.
LIVEUPDATE is used in many subsystem. It is possible to enable liveupdate and kvm without enabling KVM_GUEST_MEMFD. So GUEST_MEMFD preservation must depend on CONFIG_LIVEUPDATE and KVM_GUEST_MEMFD both. LIVEUPDATE_GUEST_MEMFD Kconfig is wrapper on both of them. It need not to be visible in menu, So I can make it hidden. WDYT?
Also, KVM VM_FILE preservation and GUEST_MEMFD preservation both depends on LIVEUPDATE_GUEST_MEMFD in this patch series. [PATCH 05/11]* Which is wrong. So I will update KVM VM_FILE preservation to depend only on CONFIG_LIVEUPDATE.
*https://lore.kernel.org/all/[email protected]/#Z31virt:kvm:Makefile.kvm
~Tarun
Sean Christopherson
maintainerRe: [PATCH v4 01/11] liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option
Quoted 21 lines · On Tue, Aug 11, 2026, [email protected] wrote:
Quoted 5 lines · Sean Christopherson
Quoted 4 lines · Tarun Sahu
Introduce the
LIVEUPDATE_GUEST_MEMFDKconfigoption. This option enables live update support for KVMguest_memfdfiles, enabling guest_memfd-backed memory preservation across kernel upgrades.Quoted 7 lines
Quoted 2 lines
Currently this support only
guest_memfdfiles that are full-shared (INIT_SHAREDand no support forprivate_mem).Why? Specifically, why do we need/want a separate
Kconfigforguest_memfdliveupdate versus KVM liveupdate? I can _maybe_ see a KVM-wideKconfig, but even that seems like a stretch given that (AFAIK) hypervisors are the one and only use-case for liveupdate.LIVEUPDATE is used in many subsystem. It is possible to enable liveupdate and kvm without enabling
KVM_GUEST_MEMFD. SoGUEST_MEMFDpreservation must depend onCONFIG_LIVEUPDATEandKVM_GUEST_MEMFDboth.LIVEUPDATE_GUEST_MEMFDKconfigis wrapper on both of them. It need not to be visible in menu, So I can make it hidden. WDYT?
Quoted 4 lines
Also, KVM
VM_FILEpreservation andGUEST_MEMFDpreservation both depends onLIVEUPDATE_GUEST_MEMFDin this patch series. [PATCH 05/11]* Which is wrong. So I will update KVMVM_FILEpreservation to depend only onCONFIG_LIVEUPDATE.
Or just handle it all in the makefile? E.g.
kvm/Makefile.kvm +5 -0
@@ -13,3 +13,8 @@ kvm-$(CONFIG_HAVE_KVM_IRQ_ROUTING) += $(KVM)/irqchip.o kvm-$(CONFIG_HAVE_KVM_DIRTY_RING) += $(KVM)/dirty_ring.o kvm-$(CONFIG_HAVE_KVM_PFNCACHE) += $(KVM)/pfncache.o kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd.o + +ifdef CONFIG_LIVEUPDATE +kvm-y += $(KVM)/kvm_luo.o +kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd_luo.o +endif
Re: [PATCH v4 01/11] liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option
Quoted 34 lines · Sean Christopherson
Quoted 21 lines · On Tue, Aug 11, 2026, [email protected] wrote:
Quoted 5 lines · Sean Christopherson
Quoted 4 lines · Tarun Sahu
Introduce the
LIVEUPDATE_GUEST_MEMFDKconfigoption. This option enables live update support for KVMguest_memfdfiles, enabling guest_memfd-backed memory preservation across kernel upgrades.Quoted 7 lines
Quoted 2 lines
Currently this support only
guest_memfdfiles that are full-shared (INIT_SHAREDand no support forprivate_mem).Why? Specifically, why do we need/want a separate
Kconfigforguest_memfdliveupdate versus KVM liveupdate? I can _maybe_ see a KVM-wideKconfig, but even that seems like a stretch given that (AFAIK) hypervisors are the one and only use-case for liveupdate.LIVEUPDATE is used in many subsystem. It is possible to enable liveupdate and kvm without enabling
KVM_GUEST_MEMFD. SoGUEST_MEMFDpreservation must depend onCONFIG_LIVEUPDATEandKVM_GUEST_MEMFDboth.LIVEUPDATE_GUEST_MEMFDKconfigis wrapper on both of them. It need not to be visible in menu, So I can make it hidden. WDYT?Quoted 4 lines
Also, KVM
VM_FILEpreservation andGUEST_MEMFDpreservation both depends onLIVEUPDATE_GUEST_MEMFDin this patch series. [PATCH 05/11]* Which is wrong. So I will update KVMVM_FILEpreservation to depend only onCONFIG_LIVEUPDATE.Or just handle it all in the makefile? E.g.
kvm/Makefile.kvm +0
-0
kvm/Makefile.kvm kvm-$(CONFIG_HAVE_KVM_IRQ_ROUTING) += $(KVM)/irqchip.o · line 20
Quoted 8 lines
kvm/Makefile.kvm +5
-0@@ -13,3 +13,8 @@ kvm-$(CONFIG_HAVE_KVM_IRQ_ROUTING) += $(KVM)/irqchip.o kvm-$(CONFIG_HAVE_KVM_DIRTY_RING) += $(KVM)/dirty_ring.o kvm-$(CONFIG_HAVE_KVM_PFNCACHE) += $(KVM)/pfncache.o kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd.o + +ifdef CONFIG_LIVEUPDATE +kvm-y += $(KVM)/kvm_luo.o +kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd_luo.o +endif
Sounds Good to me. Will take care of it. Thanks.
PATCH v4 2/11 KVM: Introduce kvm_create_vm_file() helper
In reply to Tarun Sahu- Move ITOA_MAX_LEN to kvm_mm.h for reuse by upcoming kvm_luo code.
- Introduce kvm_create_vm_file() to wrap kvm_create_vm() and anon_inode_getfile(), providing a unified VM file creation API.
--- include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 48 ++++++++++++++++++++++------------------ virt/kvm/kvm_mm.h | 3 +++ 3 files changed, 31 insertions(+), 21 deletions(-)
include/linux/kvm_host.h +1 -0
@@ -1074,6 +1074,7 @@ void kvm_get_kvm(struct kvm *kvm);bool kvm_get_kvm_safe (struct kvm *kvm);void kvm_put_kvm (struct kvm *kvm);bool file_is_kvm (struct file *file); +struct file *kvm_create_vm_file (unsigned long type,const char *fdname);void kvm_put_kvm_no_destroy (struct kvm *kvm);static inline struct kvm_memslots *__kvm_memslots (struct kvm *kvm,int as_id)
virt/kvm/kvm_main.c +27 -21
78 lines not on this page · raw message
virt/kvm/kvm_mm.h +3 -0
@@ -6,6 +6,9 @@#include <linux/kvm.h> #include <linux/kvm_types.h> +/* Worst case buffer size needed for holding an integer as a string. */ +#define ITOA_MAX_LEN 12 +/* * Architectures can choose whether to use an rwlock or spinlock * for the mmu_lock. These macros, for use in common code
Signature · 1 line
2.55.0.229.g6434b31f56-goog
Re: [PATCH v4 02/11] KVM: Introduce kvm_create_vm_file() helper
Quoted 4 lines · Tarun Sahu
- Move
ITOA_MAX_LENtokvm_mm.hfor reuse by upcomingkvm_luocode.
- Introducekvm_create_vm_file()to wrapkvm_create_vm()andanon_inode_getfile(), providing a unified VM file creation API.
I think these should be 2 separate patches.
Quoted 3 lines
[...snip...]
Re: [PATCH v4 02/11] KVM: Introduce kvm_create_vm_file() helper
Quoted 8 lines · Ackerley Tng
Quoted 4 lines · Tarun Sahu
- Move
ITOA_MAX_LENtokvm_mm.hfor reuse by upcomingkvm_luocode.
- Introducekvm_create_vm_file()to wrapkvm_create_vm()andanon_inode_getfile(), providing a unified VM file creation API.I think these should be 2 separate patches.
Okay, I will split them. Thanks
Quoted 4 lines
Quoted 4 lines
[...snip...]
Sean Christopherson
maintainerRe: [PATCH v4 02/11] KVM: Introduce kvm_create_vm_file() helper
In reply to Tarun SahuQuoted 3 lines · Tarun Sahu
- Move
ITOA_MAX_LENtokvm_mm.hfor reuse by upcomingkvm_luocode.
- Introducekvm_create_vm_file()to wrapkvm_create_vm()andanon_inode_getfile(), providing a unified VM file creation API.
As Ackerley pointed out, changelogs that are literally a list pretty much always needs to be split.
These also need to explain *why*. "providing a unified VM file creation API" isn't sufficient justification. In general, all of these changelogs need to provide more of the "why"; I can figure out what the patches are doing easily enough, but in my speed read of the series, it's hard to figure out the "why".
Re: [PATCH v4 02/11] KVM: Introduce kvm_create_vm_file() helper
Quoted 5 lines · Sean Christopherson
Quoted 3 lines · Tarun Sahu
- Move
ITOA_MAX_LENtokvm_mm.hfor reuse by upcomingkvm_luocode.
- Introducekvm_create_vm_file()to wrapkvm_create_vm()andanon_inode_getfile(), providing a unified VM file creation API.
Quoted 7 lines
As Ackerley pointed out, changelogs that are literally a list pretty much always needs to be split.
These also need to explain *why*. "providing a unified VM file creation API" isn't sufficient justification. In general, all of these changelogs need to provide more of the "why"; I can figure out what the patches are doing easily enough, but in my speed read of the series, it's hard to figure out the "why".
Okay, Will update them.
PATCH v4 3/11 KVM: Export kvm_uevent_notify_vm_create()
In reply to Tarun Sahu- Export kvm_uevent_notify_vm_create() to allow triggering KVM event notifications during VM creation.
--- include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 5 +++++ 2 files changed, 6 insertions(+)
include/linux/kvm_host.h +1 -0
@@ -1076,6 +1076,7 @@ void kvm_put_kvm(struct kvm *kvm);bool file_is_kvm (struct file *file);struct file *kvm_create_vm_file (unsigned long type,const char *fdname);void kvm_put_kvm_no_destroy (struct kvm *kvm); +void kvm_uevent_notify_vm_create (struct kvm *kvm);static inline struct kvm_memslots *__kvm_memslots (struct kvm *kvm,int as_id) {
virt/kvm/kvm_main.c +5 -0
@@ -6348,6 +6348,11 @@ static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm)kfree (env); } +void kvm_uevent_notify_vm_create (struct kvm *kvm) +{ +kvm_uevent_notify_change (KVM_EVENT_CREATE_VM , kvm); +} +static void kvm_init_debug (void ) {const struct file_operations *fops;
Signature · 1 line
2.55.0.229.g6434b31f56-goog
[email protected]
AI reviewRe: [PATCH v4 03/11] KVM: Export kvm_uevent_notify_vm_create()
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] The patch claims to export `kvm_uevent_notify_vm_create()` but fails to include the…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The patch claims to export kvm_uevent_notify_vm_create() but fails to include the EXPORT_SYMBOL_GPL() or EXPORT_SYMBOL_FOR_KVM_INTERNAL() macro. --
--- Patch [3]: [PATCH v4 03/11] KVM: Export kvm_uevent_notify_vm_create() --- commit 5e642ea16f0d267e622a352de60bfe397ecf79df
Author: Tarun Sahu <[email protected]>
KVM: Export kvm_uevent_notify_vm_create()
This commit exposes the kvm_uevent_notify_vm_create function to allow triggering KVM event notifications during VM creation.
[ ... ]
virt/kvm/kvm_main.c static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm) · line 6354
Quoted 11 lines
virt/kvm/kvm_main.c +4
-0@@ -6348,6 +6348,11 @@ static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm)kfree (env); } +void kvm_uevent_notify_vm_create (struct kvm *kvm) +{ +kvm_uevent_notify_change (KVM_EVENT_CREATE_VM , kvm); +}
[Severity: Low]
The commit message mentions exporting this function, but is it missing an EXPORT_SYMBOL_GPL or EXPORT_SYMBOL_FOR_KVM_INTERNAL declaration?
Without the macro, won't dynamically loaded architecture-specific modules like kvm-intel.ko hit undefined symbol linking errors when they try to call it?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Re: [PATCH v4 03/11] KVM: Export kvm_uevent_notify_vm_create()
Quoted 31 lines · [email protected] writes:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The patch claims to exportkvm_uevent_notify_vm_create()but fails to include theEXPORT_SYMBOL_GPL()orEXPORT_SYMBOL_FOR_KVM_INTERNAL()macro. ----- Patch [3]: [PATCH v4 03/11] KVM: Export
kvm_uevent_notify_vm_create()--- commit5e642ea16f0d267e622a352de60bfe397ecf79df
Author: Tarun Sahu <[email protected]>KVM: Export
kvm_uevent_notify_vm_create()This commit exposes the
kvm_uevent_notify_vm_createfunction to allow triggering KVM event notifications during VM creation.[ ... ]
virt/kvm/kvm_main.c static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm) · line 6354
Quoted 11 lines
virt/kvm/kvm_main.c +4
-0@@ -6348,6 +6348,11 @@ static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm)kfree (env); } +void kvm_uevent_notify_vm_create (struct kvm *kvm) +{ +kvm_uevent_notify_change (KVM_EVENT_CREATE_VM , kvm); +}[Severity: Low]
The commit message mentions exporting this function, but is it missing anEXPORT_SYMBOL_GPLorEXPORT_SYMBOL_FOR_KVM_INTERNALdeclaration?
Quoted 4 lines
Without the macro, won't dynamically loaded architecture-specific modules like kvm-intel.ko hit undefined symbol linking errors when they try to call it?
I'm not 100% sure about the use of EXPORT_SYMBOL... here but usually I compile kvm as a module to try and catch these types of linking issues. Maybe Sean will have a better method/explanation for this.
This brings about another question I have though. How does luo work if the new kernel's deserialization of the persisted data structures depends on a module? Like in this case, KVM is a module, if KVM isn't loaded in the new kernel, do the data structures stick around until KVM is loaded? Or is KVM required to be loaded as a module in the new kernel?
Quoted 2 lines
--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Sean Christopherson
maintainerRe: [PATCH v4 03/11] KVM: Export kvm_uevent_notify_vm_create()
Quoted 28 lines · Ackerley Tng
Quoted 27 lines · [email protected] writes:
--- Patch [3]: [PATCH v4 03/11] KVM: Export
kvm_uevent_notify_vm_create()--- commit5e642ea16f0d267e622a352de60bfe397ecf79df
Author: Tarun Sahu <[email protected]>KVM: Export
kvm_uevent_notify_vm_create()This commit exposes the
kvm_uevent_notify_vm_createfunction to allow triggering KVM event notifications during VM creation.[ ... ]
virt/kvm/kvm_main.c static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm) · line 6354
Quoted 11 lines
virt/kvm/kvm_main.c +4
-0@@ -6348,6 +6348,11 @@ static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm)kfree (env); } +void kvm_uevent_notify_vm_create (struct kvm *kvm) +{ +kvm_uevent_notify_change (KVM_EVENT_CREATE_VM , kvm); +}[Severity: Low]
The commit message mentions exporting this function, but is it missing anEXPORT_SYMBOL_GPLorEXPORT_SYMBOL_FOR_KVM_INTERNALdeclaration?
Quoted 7 lines
Quoted 3 lines
Without the macro, won't dynamically loaded architecture-specific modules like kvm-intel.ko hit undefined symbol linking errors when they try to call it?
I'm not 100% sure about the use of
EXPORT_SYMBOL... here but usually I compile kvm as a module to try and catch these types of linking issues. Maybe Sean will have a better method/explanation for this.
The issue is the shortlog, not the code. The intent is to make the function globally visible, not to truly export it (the eventual usage is code that's linked into kvm.ko, i.e. doesn't need an export).
Re: [PATCH v4 03/11] KVM: Export kvm_uevent_notify_vm_create()
In reply to Ackerley TngQuoted 33 lines · Ackerley Tng
Quoted 31 lines · [email protected] writes:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The patch claims to exportkvm_uevent_notify_vm_create()but fails to include theEXPORT_SYMBOL_GPL()orEXPORT_SYMBOL_FOR_KVM_INTERNAL()macro. ----- Patch [3]: [PATCH v4 03/11] KVM: Export
kvm_uevent_notify_vm_create()--- commit5e642ea16f0d267e622a352de60bfe397ecf79df
Author: Tarun Sahu <[email protected]>KVM: Export
kvm_uevent_notify_vm_create()This commit exposes the
kvm_uevent_notify_vm_createfunction to allow triggering KVM event notifications during VM creation.[ ... ]
virt/kvm/kvm_main.c static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm) · line 6354
Quoted 11 lines
virt/kvm/kvm_main.c +4
-0@@ -6348,6 +6348,11 @@ static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm)kfree (env); } +void kvm_uevent_notify_vm_create (struct kvm *kvm) +{ +kvm_uevent_notify_change (KVM_EVENT_CREATE_VM , kvm); +}[Severity: Low]
The commit message mentions exporting this function, but is it missing anEXPORT_SYMBOL_GPLorEXPORT_SYMBOL_FOR_KVM_INTERNALdeclaration?
Quoted 8 lines
Quoted 4 lines
Without the macro, won't dynamically loaded architecture-specific modules like kvm-intel.ko hit undefined symbol linking errors when they try to call it?
I'm not 100% sure about the use of
EXPORT_SYMBOL... here but usually I compile kvm as a module to try and catch these types of linking issues. Maybe Sean will have a better method/explanation for this.
The Intention is just to make it global to kvm module files. it is only used in kvm module so need to use EXPORT symbol as Sean also mentioned.
Quoted 7 lines
This brings about another question I have though. How does luo work if the new kernel's deserialization of the persisted data structures depends on a module? Like in this case, KVM is a module, if KVM isn't loaded in the new kernel, do the data structures stick around until KVM is loaded? Or is KVM required to be loaded as a module in the new kernel?
If kvm module is not loaded then guest_memfd_luo/kvm_luo handler will also not be registered. So luo_finish call will leak this memory and not be freed. This is aligned with LUO design.
I have raised the saperate thread with LUO list to discuss if LUO_CORE should throw a warning in such cases. [1]
[1] https://lore.kernel.org/all/[email protected]/
~Tarun
Quoted 3 lines
Quoted 3 lines
--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
PATCH v4 4/11 KVM: Track weak reference to vm_file in struct kvm
In reply to Tarun Sahu- Add a weak reference 'vm_file' in struct kvm under CONFIG_LIVEUPDATE_GUEST_MEMFD to enable reverse file resolution without circular lifetime dependencies.
--- include/linux/kvm_host.h | 12 ++++++++++++ virt/kvm/kvm_main.c | 26 ++++++++++++++++++++++++++ 2 files changed, 38 insertions(+)
include/linux/kvm_host.h +12 -0
@@ -874,6 +874,18 @@ struct kvm {#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES /* Protected by slots_lock (for writes) and RCU (for reads) */ struct xarray mem_attr_array; +#endif +#ifdef CONFIG_LIVEUPDATE_GUEST_MEMFD +/* +* Weak reference to the VFS file backing this KVM instance. Stored +* without incrementing the file refcount to prevent a circular lifetime +* dependency (since file->private_data already pins this struct kvm). +* Used exclusively to resolve the file pointer back from struct kvm. +* +* Written/cleared via rcu_assign_pointer() and read locklessly under +* RCU (e.g. via get_file_active() to prevent ABA races). +*/ +struct file __rcu *vm_file;#endif char stats_id[KVM_STATS_NAME_SIZE ]; };
virt/kvm/kvm_main.c +26 -0
@@ -1346,6 +1346,19 @@ static int kvm_vm_release(struct inode *inode, struct file *filp) {struct kvm *kvm = filp->private_data; +#ifdef CONFIG_LIVEUPDATE_GUEST_MEMFD +/* +* Clear the weak reference of the vm file. +* In case vm file is closed by userspace, but kvm still has +* other users like vCPUs, clearing this pointer ensures +* that we don't have a dangling pointer to a closed file. +* +* Cleared via rcu_assign_pointer() to ensure proper memory visibility +* for concurrent lockless readers under RCU. +*/ +rcu_assign_pointer (kvm->vm_file,NULL ); +#endif +kvm_irqfd_release (kvm);kvm_put_kvm (kvm);
@@ -5488,6 +5501,19 @@ struct file *kvm_create_vm_file(unsigned long type, const char *fdname)return file; } +#ifdef CONFIG_LIVEUPDATE_GUEST_MEMFD +/* +* Weak reference to the file (without get_file()) to prevent a circular +* dependency. Safe because the file's release path clears this pointer +* and drops its reference to the VM. +* +* Written via rcu_assign_pointer() because the pointer can be read +* locklessly under RCU (e.g., in kvm_gmem_luo_preserve() via +* get_file_active() to prevent lockless ABA races). +*/ +rcu_assign_pointer (kvm->vm_file, file); +#endif +/* * Don't call kvm_put_kvm anymore at this point; file->f_op is * already set, with ->release() being kvm_vm_release(). In error
Signature · 1 line
2.55.0.229.g6434b31f56-goog
Sean Christopherson
maintainerRe: [PATCH v4 04/11] KVM: Track weak reference to vm_file in struct kvm
Quoted 1 line · Tarun Sahu
- Add a weak reference '
vm_file' in struct kvm under
"vm_file" is effectively already claimed by vm_area_struct.vm_file. *If* we go this route, I would very strongly prefer to take a dependency on "vfio: Use file-based reference counting for KVM"[*]. I'll poke that thread, we really should land that one particular patch long before the s390-is-arm64 insanity.
Empashish on "if" because this all feels backwards. It's not at all clear to me why liveupdate goes directly to guest_memfd, but then requires a token associated with the VM.
Re: [PATCH v4 04/11] KVM: Track weak reference to vm_file in struct kvm
Quoted 8 lines · Sean Christopherson
Quoted 1 line · Tarun Sahu
- Add a weak reference '
vm_file' in struct kvm under"
vm_file" is effectively already claimed by vm_area_struct.vm_file. *If* we go this route, I would very strongly prefer to take a dependency on "vfio: Use file-based reference counting for KVM"[*]. I'll poke that thread, we really should land that one particular patch long before the s390-is-arm64 insanity.
[*] Seem interesting way to solve this. So we will also have vm_file directly in guest_memfd instead of struct kvm and refcounting on vm_file instead struct kvm?
struct gmem_file {
- struct kvm *kvm;
+ struct file *vm_file; struct xarray bindings; struct list_head entry; };
Quoted 6 lines
Empashish on "if" because this all feels backwards.
It's not at all clear to
me why liveupdate goes directly toguest_memfd, but then requires a token associated with the VM.
guest_memfd can be recreated without having the struct kvm, Which there is no way to pass it on retrieval path. So the function kvm_gmem_luo_retrieve() gets this VM token, find the vm_file (hence struct kvm) associated with the preserved guest_memfd. Now it create a fresh guest_memfd by calling kvm_gmem_create(..., kvm). And populate preserved flags and folios to this guest_memfd.
Quoted 2 lines
PATCH v4 5/11 KVM: LUO: Support VM preservation across live updates
In reply to Tarun SahuRegister a Live Update Orchestrator (LUO) file handler for KVM VM files to serialize and deserialize VM state across kexec live updates.
Currently, Only VM type (e.g. arch.vm_type on x86) is preserved as part of VM preservation. On retrieval, kvm_luo_retrieve() recreates the KVM VM file via kvm_create_vm_file() and use an atomically incremented ID for the internal fdname, as the final fdname assigned by userspace is not yet known during retrieval. As this fdname is only used in debugfs infra, This will not break any UAPI.
This infrastructure establishes the foundation for preserving guest_memfd instances across live updates, and can be expanded in the future to preserve additional VM state.
Also updates MAINTAINERS to include virt/kvm/kvm_luo.c and include/linux/kho/abi/kvm.h.
--- MAINTAINERS | 11 ++ include/linux/kho/abi/kvm.h | 39 ++++++++ virt/kvm/Makefile.kvm | 1 + virt/kvm/kvm_luo.c | 195 ++++++++++++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 8 ++ virt/kvm/kvm_mm.h | 8 ++ 6 files changed, 262 insertions(+) create mode 100644 include/linux/kho/abi/kvm.h create mode 100644 virt/kvm/kvm_luo.c
MAINTAINERS +11 -0
@@ -14539,6 +14539,17 @@ S: Maintained F: Documentation/devicetree/bindings/leds/backlight/kinetic,ktz8866.yaml F: drivers/video/backlight/ktz8866.c +KVM LIVE UPDATE +M: Pasha Tatashin <[email protected]> +M: Mike Rapoport <[email protected]> +M: Pratyush Yadav <[email protected]> +R: Tarun Sahu <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +T: git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git +F: virt/kvm/kvm_luo.c + KVM PARAVIRT (KVM/paravirt) M: Paolo Bonzini <[email protected]> R: Vitaly Kuznetsov <[email protected]>
include/linux/kho/abi/kvm.h new file +39 -0
@@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (c) 2026, Google LLC. +* Tarun Sahu <[email protected]> +* +* KVM Preservation ABI for Live Update Orchestrator (LUO) +*/ +#ifndef _LINUX_KHO_ABI_KVM_H +#define _LINUX_KHO_ABI_KVM_H + +#include <linux/types.h> +#include <linux/kho/abi/kexec_handover.h> + +/** +* DOC: KVM Live Update ABI +* +* KVM uses the ABI defined below for preserving its state +* across a kexec reboot using the LUO. +* +* The state is serialized into a packed structure `struct kvm_luo_ser` +* which is handed over to the next kernel via the KHO mechanism. +* +* This interface is a contract. Any modification to the structure layout +* constitutes a breaking change. Such changes require incrementing the +* version number in the KVM_LUO_FH_COMPATIBLE compatibility string. +*/ + +/** +* struct kvm_luo_ser - Main serialization structure for a KVM VM. +* @type: The type of VM. +*/ +struct kvm_luo_ser { +u64 type; +} __packed; + +/* The compatibility string for KVM VM file handler */ +#define KVM_LUO_FH_COMPATIBLE "kvm_vm_luo_v1" + +#endif /* _LINUX_KHO_ABI_KVM_H */
virt/kvm/Makefile.kvm +1 -0
@@ -13,3 +13,4 @@ kvm-$(CONFIG_HAVE_KVM_IRQ_ROUTING) += $(KVM)/irqchip.o kvm-$(CONFIG_HAVE_KVM_DIRTY_RING) += $(KVM)/dirty_ring.o kvm-$(CONFIG_HAVE_KVM_PFNCACHE) += $(KVM)/pfncache.o kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd.o +kvm-$(CONFIG_LIVEUPDATE_GUEST_MEMFD) += $(KVM)/kvm_luo.o
virt/kvm/kvm_luo.c new file +195 -0
195 lines not on this page · raw message
virt/kvm/kvm_main.c +8 -0
@@ -6577,6 +6577,10 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module)if (r) goto err_virt; + r =kvm_luo_init (); +if (r) + goto err_luo; +/* * Registration _must_ be the very last thing done, as this exposes * /dev/kvm to userspace, i.e. all infrastructure must be setup!
@@ -6590,6 +6594,8 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module)return 0 ; err_register: +kvm_luo_exit (); +err_luo:kvm_uninit_virtualization (); err_virt:kvm_gmem_exit ();
@@ -6619,6 +6625,8 @@ void kvm_exit(void)*/ misc_deregister (&kvm_dev); +kvm_luo_exit (); +kvm_uninit_virtualization ();debugfs_remove_recursive (kvm_debugfs_dir);
virt/kvm/kvm_mm.h +8 -0
@@ -100,4 +100,12 @@ static inline void kvm_gmem_unbind(struct kvm_memory_slot *slot) }#endif /* CONFIG_KVM_GUEST_MEMFD */ +#ifdef CONFIG_LIVEUPDATE_GUEST_MEMFD +int kvm_luo_init (void ); +void kvm_luo_exit (void ); +#else +static inline int kvm_luo_init (void ) {return 0 ; } +static inline void kvm_luo_exit (void ) {} +#endif /* CONFIG_LIVEUPDATE_GUEST_MEMFD */ +#endif /* __KVM_MM_H__ */
Signature · 1 line
2.55.0.229.g6434b31f56-goog
[email protected]
AI reviewRe: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] The generated `fdname` for a restored VM can collide with the file descriptor of a newly created…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] The generated fdname for a restored VM can collide with the file descriptor of a newly created VM in the same process, causing debugfs directory creation to fail silently for the new VM.
- [Low] The MAINTAINERS file was not updated to include the new ABI header include/linux/kho/abi/kvm.h, contrary to the commit message.
--
--- Patch [5]: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates --- commit 650c9d9cd02da64dbc0cdbd5b8073f5b00dadf99
Author: Tarun Sahu <[email protected]>
KVM: LUO: Support VM preservation across live updates
Register a Live Update Orchestrator (LUO) file handler for KVM VM files to serialize and deserialize VM state across kexec live updates. Preserves VM type and recreates the KVM VM file during retrieval with an internally generated fdname.
Quoted 2 lines
Also updates
MAINTAINERSto includevirt/kvm/kvm_luo.candinclude/linux/kho/abi/kvm.h.
[Severity: Low]
This isn't a bug, but the commit message states that the MAINTAINERS file is updated to include include/linux/kho/abi/kvm.h, but the diff only adds virt/kvm/kvm_luo.c. Should the header file be added to the KVM LIVE UPDATE section?
Quoted 5 lines
virt/kvm/kvm_luo.c new file +0
-0
[ ... ]
Quoted 7 lines
virt/kvm/kvm_luo.c new file +7
-0+static atomic_t restored_vm_id =ATOMIC_INIT (0 ); + +static int kvm_luo_retrieve (struct liveupdate_file_op_args *args) +{ +char fdname[ITOA_MAX_LEN +1 ]; +struct kvm_luo_ser *ser; +struct file *file;
virt/kvm/kvm_luo.c line 107
Quoted 10 lines
virt/kvm/kvm_luo.c new file +10
-0+struct kvm *kvm; +int err =0 ; + +if (!args->serialized_data) +return -EINVAL ; + + ser =phys_to_virt (args->serialized_data); + +snprintf (fdname,sizeof (fdname),"%d" , +atomic_inc_return (&restored_vm_id));
[Severity: Medium]
Could this generated fdname format collide with the file descriptor of a newly created VM in the same process?
Since this uses a simple "%d" format based on a global restored_vm_id counter, it shares the identical naming format as normal VM file descriptors.
If a process restores VMs using LUO and then creates a new VM that happens to get assigned a file descriptor matching one of the restored VM IDs, would the debugfs directory creation for the new VM fail silently and result in missing VM stats?
Quoted 2 lines
virt/kvm/kvm_luo.c new file +2
-0+ + file =kvm_create_vm_file (ser->type, fdname);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Sean Christopherson
maintainerRe: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
In reply to Tarun SahuQuoted 5 lines · Tarun Sahu
Register a Live Update Orchestrator (LUO) file handler for KVM VM files to serialize and deserialize VM state across kexec live updates.
Currently, Only VM type (e.g. arch.vm_type on x86) is preserved as part of VM preservation.
Why?
Quoted 9 lines
On retrieval,
kvm_luo_retrieve()recreates the KVM VM file viakvm_create_vm_file()and use an atomically incremented ID for the internal fdname, as the final fdname assigned by userspace is not yet known during retrieval. As this fdname is only used in debugfs infra, This will not break any UAPI.This infrastructure establishes the foundation for preserving
guest_memfdinstances across live updates, and can be expanded in the future to preserve additional VM state.
Uh, why guest_memfd? As much as I want to push guest_memfd adoption, it seems guest_memfd should be the _last_ thing we support, not the first. As evidenced by the last two decades, it's very doable to have KVM VMs without guest_memfd, but it's rather hard to have VMs without vCPUs.
Quoted 22 lines
Also updates
MAINTAINERSto includevirt/kvm/kvm_luo.candinclude/linux/kho/abi/kvm.h.Signed-off-by Tarun Sahu <[email protected]>--- MAINTAINERS | 11 ++ include/linux/kho/abi/kvm.h | 39 ++++++++ virt/kvm/Makefile.kvm | 1 + virt/kvm/kvm_luo.c | 195 ++++++++++++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 8 ++ virt/kvm/kvm_mm.h | 8 ++ 6 files changed, 262 insertions(+) create mode 100644 include/linux/kho/abi/kvm.h create mode 100644 virt/kvm/kvm_luo.cMAINTAINERS +0
-0@@ -14539,6 +14539,17 @@ S: Maintained F: Documentation/devicetree/bindings/leds/backlight/kinetic,ktz8866.yaml F: drivers/video/backlight/ktz8866.c
MAINTAINERS S: Maintained · line 14550
Quoted 10 lines
MAINTAINERS +9
-0+KVM LIVE UPDATE +M: Pasha Tatashin <[email protected]> +M: Mike Rapoport <[email protected]> +M: Pratyush Yadav <[email protected]> +R: Tarun Sahu <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +T: git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git
NAK on taking changes through a different tree. This is KVM code, period.
In general, I'm skeptical of the dedicated MAINTAINERS entry. It's extremely difficult to tell since this series is little more than a skeleton (either that or liveupdate is way simpler that I was expecting), but I suspect that maintaining liveupdate for KVM (or for any subsystem) will require more subsystem-specific knowledge than liveupdate knowledge.
E.g. the LUO APIs seem pretty straightforward; I assume the bulk of the complexity is going to be in knowing what to save/restore, and how, which is much more about KVM than it is about liveupdate.
Quoted 77 lines
MAINTAINERS +2
-0+F: virt/kvm/kvm_luo.c + KVM PARAVIRT (KVM/paravirt) M: Paolo Bonzini <[email protected]> R: Vitaly Kuznetsov <[email protected]>include/linux/kho/abi/kvm.h new file +39
-0@@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (c) 2026, Google LLC. +* Tarun Sahu <[email protected]> +* +* KVM Preservation ABI for Live Update Orchestrator (LUO) +*/ +#ifndef _LINUX_KHO_ABI_KVM_H +#define _LINUX_KHO_ABI_KVM_H + +#include <linux/types.h> +#include <linux/kho/abi/kexec_handover.h> + +/** +* DOC: KVM Live Update ABI +* +* KVM uses the ABI defined below for preserving its state +* across a kexec reboot using the LUO. +* +* The state is serialized into a packed structure `struct kvm_luo_ser` +* which is handed over to the next kernel via the KHO mechanism. +* +* This interface is a contract. Any modification to the structure layout +* constitutes a breaking change. Such changes require incrementing the +* version number in the KVM_LUO_FH_COMPATIBLE compatibility string. +*/ + +/** +* struct kvm_luo_ser - Main serialization structure for a KVM VM. +* @type: The type of VM. +*/ +struct kvm_luo_ser { +u64 type; +} __packed; + +/* The compatibility string for KVM VM file handler */ +#define KVM_LUO_FH_COMPATIBLE "kvm_vm_luo_v1" + +#endif /* _LINUX_KHO_ABI_KVM_H */ virt/kvm/Makefile.kvm +1
-0@@ -13,3 +13,4 @@ kvm-$(CONFIG_HAVE_KVM_IRQ_ROUTING) += $(KVM)/irqchip.o kvm-$(CONFIG_HAVE_KVM_DIRTY_RING) += $(KVM)/dirty_ring.o kvm-$(CONFIG_HAVE_KVM_PFNCACHE) += $(KVM)/pfncache.o kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd.o +kvm-$(CONFIG_LIVEUPDATE_GUEST_MEMFD) += $(KVM)/kvm_luo.ovirt/kvm/kvm_luo.c new file +12
-0@@ -0,0 +1,195 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* +* Copyright (c) 2026, Google LLC. +* Tarun Sahu <[email protected]> +* +* KVM VM Preservation for Live Update Orchestrator (LUO) +*/ + +/** +* DOC: KVM VM Preservation via LUO +*
virt/kvm/kvm_luo.c line 22
Quoted 10 lines
virt/kvm/kvm_luo.c new file +10
-0+* Overview +* ======== +* +* KVM virtual machines (VMs) can be preserved over a kexec reboot using the +* Live Update Orchestrator (LUO) file preservation. This allows userspace +* to preserve KVM VM state across kexec reboots. +* +* The preservation is not intended to be fully transparent. Only specific +* VM configuration and state are preserved, while other aspects of the VM +* must be re-established or re-configured by userspace after retrieval.
IMO, this is not helpful documentation. Anyone with passing knowledge of LUO already knows the above. What people like me don't know are the exact details of how preserving a KVM guest is expected to work.
Quoted 11 lines
virt/kvm/kvm_luo.c new file +11
-0+* Preserved Properties +* ==================== +* +* The following properties of the KVM VM are preserved across kexec: +* +* VM Type +* The VM type (e.g., on x86 architecture, the vm_type parameter) is +* preserved. +* +* Non-Preserved Properties +* ========================
virt/kvm/kvm_luo.c line 44
Quoted 10 lines
virt/kvm/kvm_luo.c new file +10
-0+* +* The preservation does not cover: +* +* - vCPUs and vCPU states +* - Memspots / Memory slot layout (memslots) +* - Interrupt controllers and IRQ routings +* - Coalesced MMIO zones +* - Device bindings (VFIO/Eventfds) +* - Active paging or guest registers state +* - etc
So... what's the plan? Bluntly, this series isn't going anywhere without a clear plan of how all of this is going to fit together.
virt/kvm/kvm_luo.c line 67
Quoted 4 lines
virt/kvm/kvm_luo.c new file +4
-0+static int kvm_luo_preserve (struct liveupdate_file_op_args *args) +{ +struct kvm *kvm = args->file->private_data; +struct kvm_luo_ser *ser;
I *really* dislike the "ser" nomenclature. The abbreviation isn't common, and IMO it's not intuitive. This also needs to be more specific, becuase I'm guessing we'll end up with more than one "kvm" LUO structure.
Maybe something like kvm_vm_luo_state?
Quoted 8 lines
virt/kvm/kvm_luo.c new file +8
-0+ +if (kvm->vm_dead || kvm->vm_bugged) +return -EINVAL ; + + ser =kho_alloc_preserve (sizeof (*ser)); +if (IS_ERR (ser)) +return PTR_ERR (ser); +
virt/kvm/kvm_luo.c line 85
Quoted 10 lines
virt/kvm/kvm_luo.c new file +10
-0+#if defined(CONFIG_X86 ) + ser->type = kvm->arch.vm_type; +#elif defined(CONFIG_ARM64 ) + ser->type =kvm_phys_shift (&kvm->arch.mmu); +if (kvm_vm_is_protected (kvm)) + ser->type |=KVM_VM_TYPE_ARM_PROTECTED ; + +#else + ser->type =0 ; +#endif
This needs to be properly supported with arch callbacks. And per-arch enabling needs to be done in separate patches.
Quoted 9 lines
virt/kvm/kvm_luo.c new file +9
-0+ + args->serialized_data =virt_to_phys (ser); +return 0 ; +} + +static atomic_t restored_vm_id =ATOMIC_INIT (0 ); + +static int kvm_luo_retrieve (struct liveupdate_file_op_args *args) +{
virt/kvm/kvm_luo.c line 104
Quoted 10 lines
virt/kvm/kvm_luo.c new file +10
-0+char fdname[ITOA_MAX_LEN +1 ]; +struct kvm_luo_ser *ser; +struct file *file; +struct kvm *kvm; +int err =0 ; + +if (!args->serialized_data) +return -EINVAL ; + + ser =phys_to_virt (args->serialized_data);
KVM generally uses __va() and __pa(), is there a reason to diverge?
Pratyush Yadav
maintainerRe: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
Quoted 15 lines · Sean Christopherson
Quoted 5 lines · Tarun Sahu
Register a Live Update Orchestrator (LUO) file handler for KVM VM files to serialize and deserialize VM state across kexec live updates.
Currently, Only VM type (e.g. arch.vm_type on x86) is preserved as part of VM preservation.
Why?
Quoted 6 lines
On retrieval,
kvm_luo_retrieve()recreates the KVM VM file viakvm_create_vm_file()and use an atomically incremented ID for the internal fdname, as the final fdname assigned by userspace is not yet known during retrieval. As this fdname is only used in debugfs infra, This will not break any UAPI.
Quoted 8 lines
Quoted 3 lines
This infrastructure establishes the foundation for preserving
guest_memfdinstances across live updates, and can be expanded in the future to preserve additional VM state.Uh, why
guest_memfd? As much as I want to pushguest_memfdadoption, it seemsguest_memfdshould be the _last_ thing we support, not the first. As evidenced by the last two decades, it's very doable to have KVM VMs withoutguest_memfd, but it's rather hard to have VMs without vCPUs.
You _can_ preserve vCPUs today using KVM_{GET,SET}_REGS, they just won't run in the background during the reboot. This series can save you from dumping VM memory to disk if it is backed by guest_memfd.
Quoted 36 lines
Quoted 23 lines
Also updates
MAINTAINERSto includevirt/kvm/kvm_luo.candinclude/linux/kho/abi/kvm.h.Signed-off-by Tarun Sahu <[email protected]>--- MAINTAINERS | 11 ++ include/linux/kho/abi/kvm.h | 39 ++++++++ virt/kvm/Makefile.kvm | 1 + virt/kvm/kvm_luo.c | 195 ++++++++++++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 8 ++ virt/kvm/kvm_mm.h | 8 ++ 6 files changed, 262 insertions(+) create mode 100644 include/linux/kho/abi/kvm.h create mode 100644 virt/kvm/kvm_luo.cMAINTAINERS +0
-0@@ -14539,6 +14539,17 @@ S: Maintained F: Documentation/devicetree/bindings/leds/backlight/kinetic,ktz8866.yaml F: drivers/video/backlight/ktz8866.cMAINTAINERS S: Maintained · line 14550
Quoted 10 lines
MAINTAINERS +9
-0+KVM LIVE UPDATE +M: Pasha Tatashin <[email protected]> +M: Mike Rapoport <[email protected]> +M: Pratyush Yadav <[email protected]> +R: Tarun Sahu <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +T: git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.gitNAK on taking changes through a different tree. This is KVM code, period.
Quoted 3 lines
In general, I'm skeptical of the dedicated
MAINTAINERSentry. It's extremely difficult to tell since this series is little more than a skeleton (either that or liveupdate is way simpler that I was expecting), but I suspect that maintaining
It's a bit of both. This series of course doesn't support everything that guest_memfd can. At the same time, I also keep being (pleasantly) surprised at preservation being relatively simple. For example, the code to preserve a shmem file (via memfd) is roughly 600 lines, a big chunk of which is comments. The code of course has some limitations, but it is good enough for use in production.
Quoted 6 lines
liveupdate for KVM (or for any subsystem) will require more subsystem-specific knowledge than liveupdate knowledge.
E.g. the LUO APIs seem pretty straightforward; I assume the bulk of the complexity is going to be in knowing what to save/restore, and how, which is much more about KVM than it is about liveupdate.
I think it is fine if you want to take these changes through the KVM tree, but I would like live update maintainers to be listed as reviewers at least.
For one, we care about ABI breakages and versioning. The serialized state is a part of live update ABI and changes to it should be ACKed by us. For another, how the file handlers interact with their dependencies can affect the behaviour that VMMs observe. Those changes should also pass by some live update eyes.
So I think we should have a separate entry for KVM LIVE UPDATE that lists the live update maintainers (or perhaps only kexec@) as a reviewer. The patches can still flow through KVM tree but we'd get a chance to ACK/NACK them.
Of course this all can evolve later as we see fit but I think this is a good starting point.
Quoted 6 lines
Quoted 6 lines
MAINTAINERS +2
-0+F: virt/kvm/kvm_luo.c + KVM PARAVIRT (KVM/paravirt) M: Paolo Bonzini <[email protected]> R: Vitaly Kuznetsov <[email protected]>
[...]
Signature · 2 lines
Regards, Pratyush Yadav
Sean Christopherson
maintainerRe: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
Quoted 21 lines · Pratyush Yadav
Quoted 19 lines · Sean Christopherson
Quoted 5 lines · Tarun Sahu
Register a Live Update Orchestrator (LUO) file handler for KVM VM files to serialize and deserialize VM state across kexec live updates.
Currently, Only VM type (e.g. arch.vm_type on x86) is preserved as part of VM preservation.
Why?
Quoted 9 lines
On retrieval,
kvm_luo_retrieve()recreates the KVM VM file viakvm_create_vm_file()and use an atomically incremented ID for the internal fdname, as the final fdname assigned by userspace is not yet known during retrieval. As this fdname is only used in debugfs infra, This will not break any UAPI.This infrastructure establishes the foundation for preserving
guest_memfdinstances across live updates, and can be expanded in the future to preserve additional VM state.
Quoted 7 lines
Quoted 4 lines
Uh, why
guest_memfd? As much as I want to pushguest_memfdadoption, it seemsguest_memfdshould be the _last_ thing we support, not the first. As evidenced by the last two decades, it's very doable to have KVM VMs withoutguest_memfd, but it's rather hard to have VMs without vCPUs.You _can_ preserve vCPUs today using KVM_{GET,SET}_REGS, they just won't run in the background during the reboot.
What about x86 CoCo VMs? Which are quite literally _the_ reason guest_memfd was created in the first place.
Quoted 2 lines
This series can save you from dumping VM memory to disk if it is backed by
guest_memfd.
Or to word it another way, one _can_ save guest_memfd, it's just slower. My point is that this series needs to provide a _lot_ more information about the bigger KVM picture. For those of us that are on the very fringes of live update, it's practically impossible to review because, to us, it seems very arbitrary.
The part that's especially confusing is the saving of the VM type. That comes straight from userspace, so it's super bizarre to automatically save/restore that, but nothing else.
Quoted 15 lines
Quoted 15 lines
Review
Quoted 9 lines
(quoted diff) +9
-0+KVM LIVE UPDATE +M: Pasha Tatashin <[email protected]> +M: Mike Rapoport <[email protected]> +M: Pratyush Yadav <[email protected]> +R: Tarun Sahu <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +T: git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.gitNAK on taking changes through a different tree. This is KVM code, period.
In general, I'm skeptical of the dedicated
MAINTAINERSentry. It's extremely difficult to tell since this series is little more than a skeleton (either that or liveupdate is way simpler that I was expecting), but I suspect that maintaining
...
Quoted 7 lines
Quoted 3 lines
E.g. the LUO APIs seem pretty straightforward; I assume the bulk of the complexity is going to be in knowing what to save/restore, and how, which is much more about KVM than it is about liveupdate.
I think it is fine if you want to take these changes through the KVM tree, but I would like live update maintainers to be listed as reviewers at least.
Why not simply add a file pattern match to the LIVE UPDATE entry?
MAINTAINERS +1 -1
@@ -15052,8 +15052,8 @@ F: include/linux/liveupdate.h F: include/uapi/linux/liveupdate.h F: kernel/liveupdate/ F: lib/tests/liveupdate.c -F: mm/memfd_luo.c F: tools/testing/selftests/liveupdate/ +N: [^a-z]luo LLC (802.2) L: [email protected]
Quoted 7 lines
At the same time, I also keep being (pleasantly)
surprised at preservation being relatively simple. For example, the code to preserve a shmem file (via memfd) is roughly 600 lines, a big chunk of which is comments. The code of course has some limitations, but it is good enough for use in production.For one, we care about ABI breakages and versioning.
Which is amusing to me because that implies KVM does not, and I would hazard to guess that KVM has the biggest ABI surface of any subsystem in the kernel by a country mile (though I'm probably wildly underestimating the effective ABI surface of filesystems).
Quoted 2 lines
The serialized state is a part of live update ABI and changes to it should be ACKed by us.
Meh, "Don't break userspace" is a universal rule in the kernel, I genuinely don't see why liveupdate needs special treatment.
Quoted 3 lines
For another, how the file handlers interact with their dependencies can affect the behaviour that VMMs observe. Those changes should also pass by some live update eyes.
Perhaps in the short term, but IMO, that's not a winning strategy in the long term. From my perspective, that like saying the PAGE CACHE maintainers should review every usage of the filemap APIs, because how the APIs are used impacts the page cache and affects userspace-visible behavior. There are myriad analogies like that throughout the kernel.
Yes, liveupdate is new and shiny, but IMO for it to be successful and maintainable, it needs to be treated like any other core infrastructure in the kernel, not a special snowflake whose details are known only by a handful of people. Because I think it's likely liveupdate goes one of two ways: either liveupdate becomes a very niche thing that is used sparingly throughout the kernel, or it becomes a broadly used feature that is supported by many filesystems and subsystems.
If liveupdate is relegated to niche status, then it probably isn't going to see a significant amount of ongoing development, at which point the folks working on liveupdate will naturally migrate to other projects, and maintenance will largely be left to subsystem maintainers.
If liveupdate is broadly used, then having a single group of people maintain every subsystem's usage won't scale, and maintenance will again largely fall on the shoulder of subsystem maintainers. Which is totally fine and working as intended, because that's exactly what subystem maintainers are signing up for by merging support for liveupdate.
Pratyush Yadav
maintainerRe: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
Quoted 27 lines · Sean Christopherson
Quoted 26 lines · Pratyush Yadav
Quoted 23 lines · Sean Christopherson
Quoted 5 lines · Tarun Sahu
Register a Live Update Orchestrator (LUO) file handler for KVM VM files to serialize and deserialize VM state across kexec live updates.
Currently, Only VM type (e.g. arch.vm_type on x86) is preserved as part of VM preservation.
Why?
Quoted 9 lines
On retrieval,
kvm_luo_retrieve()recreates the KVM VM file viakvm_create_vm_file()and use an atomically incremented ID for the internal fdname, as the final fdname assigned by userspace is not yet known during retrieval. As this fdname is only used in debugfs infra, This will not break any UAPI.This infrastructure establishes the foundation for preserving
guest_memfdinstances across live updates, and can be expanded in the future to preserve additional VM state.Uh, why
guest_memfd? As much as I want to pushguest_memfdadoption, it seemsguest_memfdshould be the _last_ thing we support, not the first. As evidenced by the last two decades, it's very doable to have KVM VMs withoutguest_memfd, but it's rather hard to have VMs without vCPUs.
Quoted 5 lines
Quoted 2 lines
You _can_ preserve vCPUs today using KVM_{GET,SET}_REGS, they just won't run in the background during the reboot.
What about x86 CoCo VMs? Which are quite literally _the_ reason
guest_memfdwas created in the first place.
I don't know much about the history but I thought these days guest_memfd is used for more than just encrypted memory. I have seen talk of it being used for non-confidential VMs. For example these patches [0][1][2].
The CoCo parts can follow, but IIUC guest_memfd is being used to back guest memory on non-CoCo VMs too.
[0] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Quoted 5 lines
Quoted 3 lines
This series can save you from dumping VM memory to disk if it is backed by
guest_memfd.Or to word it another way, one _can_ save
guest_memfd, it's just slower. My
Sure. But making things faster is the entire point of live update.
Guest memory is one piece of that puzzle. Devices are another, and that's why there is a lot of work going on with VFIO, PCI, and IOMMU preservation. vCPUs are also a piece, but I think those will be the hardest to live update.
Quoted 3 lines
point is that this series needs to provide a _lot_ more information about the bigger KVM picture. For those of us that are on the very fringes of live update, it's practically impossible to review because, to us, it seems very arbitrary.
That's valid criticism. This series should do a better job of laying out the high level plan and where things fit.
Quoted 38 lines
The part that's especially confusing is the saving of the VM type. That comes straight from userspace, so it's super bizarre to automatically save/restore that, but nothing else.
Quoted 15 lines
Quoted 15 lines
Review
Quoted 9 lines
(quoted diff) +9
-0+KVM LIVE UPDATE +M: Pasha Tatashin <[email protected]> +M: Mike Rapoport <[email protected]> +M: Pratyush Yadav <[email protected]> +R: Tarun Sahu <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +T: git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.gitNAK on taking changes through a different tree. This is KVM code, period.
In general, I'm skeptical of the dedicated
MAINTAINERSentry. It's extremely difficult to tell since this series is little more than a skeleton (either that or liveupdate is way simpler that I was expecting), but I suspect that maintaining...
Quoted 7 lines
Quoted 3 lines
E.g. the LUO APIs seem pretty straightforward; I assume the bulk of the complexity is going to be in knowing what to save/restore, and how, which is much more about KVM than it is about liveupdate.
I think it is fine if you want to take these changes through the KVM tree, but I would like live update maintainers to be listed as reviewers at least.
Why not simply add a file pattern match to the LIVE UPDATE entry?
MAINTAINERS +0
-0
MAINTAINERS F: include/linux/liveupdate.h · line 15059
Quoted 9 lines
MAINTAINERS +1
-1@@ -15052,8 +15052,8 @@ F: include/linux/liveupdate.h F: include/uapi/linux/liveupdate.h F: kernel/liveupdate/ F: lib/tests/liveupdate.c -F: mm/memfd_luo.c F: tools/testing/selftests/liveupdate/ +N: [^a-z]luo LLC (802.2) L: [email protected]
This would list us as maintainers of kvm_luo.c and the tree as liveupdate.git, both of which is something you're saying you _don't_ want.
Quoted 10 lines
Quoted 8 lines
At the same time, I also keep being (pleasantly)
surprised at preservation being relatively simple. For example, the code to preserve a shmem file (via memfd) is roughly 600 lines, a big chunk of which is comments. The code of course has some limitations, but it is good enough for use in production.For one, we care about ABI breakages and versioning.
Which is amusing to me because that implies KVM does not, and I would hazard to
No, it doesn't. What I'm saying is I care about changes to the _live update ABI_. Just like you probably care about changes to KVM ABI but not so much about BPF for example.
Quoted 9 lines
guess that KVM has the biggest ABI surface of any subsystem in the kernel by a country mile (though I'm probably wildly underestimating the effective ABI surface of filesystems).
Quoted 2 lines
The serialized state is a part of live update ABI and changes to it should be ACKed by us.
Meh, "Don't break userspace" is a universal rule in the kernel, I genuinely don't see why liveupdate needs special treatment.
Ironically enough, you miss my point. I'm not talking about userspace ABI. We all know not to break that. I am talking about live update _serialization ABI_. See the stuff under include/linux/kho/abi. This series also adds things there.
This is ABI between kernels. It needs to be stable-ish so you can move from one kernel version to another. At the same time, unlike userspace ABI, it can change. Today we don't have any rules and let you change things freely as long as you do a version bump. But at a later point, the plan is to add some stability requirements to the ABI so you can actually upgrade the kernel across major versions.
So at least for ABI changes, there should be an explicit ACK from the live update group. For code changes, I'd be flexible if you'd prefer that. More on it below.
Quoted 10 lines
Quoted 4 lines
For another, how the file handlers interact with their dependencies can affect the behaviour that VMMs observe. Those changes should also pass by some live update eyes.
Perhaps in the short term, but IMO, that's not a winning strategy in the long term. From my perspective, that like saying the PAGE CACHE maintainers should review every usage of the filemap APIs, because how the APIs are used impacts the page cache and affects userspace-visible behavior. There are myriad analogies like that throughout the kernel.
I've heard kernel maintainers complain many times about people or companies throwing code over the wall and not staying around to deal with the mess it might make.
I'd like to avoid that with live update and help you maintain this. As you've said, you are not as familiar with live update, and perhaps you might not even be as interested in it. So why not let the people who are help review the code?
Ultimately it is your subsystem so it is your call. And as I've said before, if you want to take it through the KVM tree and have a veto I think that is perfectly fine.
As an alternate example, with memfd_luo, the MM folks are rarely involved and the maintenance and review is done largely by me because I wrote that code. Mainly because the contents in memfd_luo.c are all live update related and don't matter much to core MM or memfd. I have heard similar desire for the HugeTLB live update work I am doing.
We can figure out what works best for KVM.
Quoted 18 lines
Yes, liveupdate is new and shiny, but IMO for it to be successful and maintainable, it needs to be treated like any other core infrastructure in the kernel, not a special snowflake whose details are known only by a handful of people. Because I think it's likely liveupdate goes one of two ways: either liveupdate becomes a very niche thing that is used sparingly throughout the kernel, or it becomes a broadly used feature that is supported by many filesystems and subsystems.
If liveupdate is relegated to niche status, then it probably isn't going to see a significant amount of ongoing development, at which point the folks working on liveupdate will naturally migrate to other projects, and maintenance will largely be left to subsystem maintainers.
If liveupdate is broadly used, then having a single group of people maintain every subsystem's usage won't scale, and maintenance will again largely fall on the shoulder of subsystem maintainers. Which is totally fine and working as intended, because that's exactly what subystem maintainers are signing up for by merging support for liveupdate.
Signature · 2 lines
Regards, Pratyush Yadav
Sean Christopherson
maintainerRe: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
Quoted 32 lines · Pratyush Yadav
Quoted 30 lines · Sean Christopherson
Quoted 28 lines · Pratyush Yadav
Quoted 15 lines · Sean Christopherson
Quoted 5 lines · Tarun Sahu
Register a Live Update Orchestrator (LUO) file handler for KVM VM files to serialize and deserialize VM state across kexec live updates.
Currently, Only VM type (e.g. arch.vm_type on x86) is preserved as part of VM preservation.
Why?
Quoted 6 lines
On retrieval,
kvm_luo_retrieve()recreates the KVM VM file viakvm_create_vm_file()and use an atomically incremented ID for the internal fdname, as the final fdname assigned by userspace is not yet known during retrieval. As this fdname is only used in debugfs infra, This will not break any UAPI.Quoted 8 lines
Quoted 3 lines
This infrastructure establishes the foundation for preserving
guest_memfdinstances across live updates, and can be expanded in the future to preserve additional VM state.Uh, why
guest_memfd? As much as I want to pushguest_memfdadoption, it seemsguest_memfdshould be the _last_ thing we support, not the first. As evidenced by the last two decades, it's very doable to have KVM VMs withoutguest_memfd, but it's rather hard to have VMs without vCPUs.You _can_ preserve vCPUs today using KVM_{GET,SET}_REGS, they just won't run in the background during the reboot.
Quoted 6 lines
Quoted 2 lines
What about x86 CoCo VMs? Which are quite literally _the_ reason
guest_memfdwas created in the first place.I don't know much about the history but I thought these days
guest_memfdis used for more than just encrypted memory. I have seen talk of it being used for non-confidential VMs. For example these patches [0][1][2].
It's getting there, but it's all still very nascent.
Quoted 2 lines
The CoCo parts can follow, but IIUC
guest_memfdis being used to back guest memory on non-CoCo VMs too.
Not using upstream code, though there are most definitely folks using guest_memfd in production with out-of-tree patches.
But that's all beside the point. What I'm saying is that SNP and TDX *must* use guest_memfd, whereas guest_memfd is optional for all other VM types. And so adding LUO support for guest_memfd without even sketching out a plan for CoCo VMs feels backwards.
I'm not necessarily opposed to gradual guest_memfd support, but there needs to be a clear plan of how all of this is going to fit together. It doesn't need to be perfect, and I'm sure we'll make mistakes along the way, but I want to at least try not to paint ourselves into a corner, especially with respect to the ABI.
Quoted 22 lines
Quoted 22 lines
Quoted 7 lines
Quoted 3 lines
E.g. the LUO APIs seem pretty straightforward; I assume the bulk of the complexity is going to be in knowing what to save/restore, and how, which is much more about KVM than it is about liveupdate.
I think it is fine if you want to take these changes through the KVM tree, but I would like live update maintainers to be listed as reviewers at least.
Why not simply add a file pattern match to the LIVE UPDATE entry?
MAINTAINERS +1
-1@@ -15052,8 +15052,8 @@ F: include/linux/liveupdate.h F: include/uapi/linux/liveupdate.h F: kernel/liveupdate/ F: lib/tests/liveupdate.c -F: mm/memfd_luo.c F: tools/testing/selftests/liveupdate/ +N: [^a-z]luo
Quoted 4 lines
MAINTAINERS F: include/linux/liveupdate.h · line 15059
Quoted 2 lines
MAINTAINERS +0
-0LLC (802.2) L: [email protected]This would list us as maintainers of
kvm_luo.cand the tree as liveupdate.git,
It would list both "LIVE UPDATE" and "KERNEL VIRTUAL MACHINE (KVM)", as KVM would
still cover the files via "F: virt/kvm/*". And my read of MAINTAINERS is that
N: and K: entries are "secondary" if the files/scope is covered by an F: entry.
E.g. arch/x86/kvm/vmx/tdx.c is covered by "KERNEL VIRTUAL MACHINE FOR X86 (KVM/x86)"
via "F: arch/x86/kvm/*/", and also by "X86 TRUST DOMAIN EXTENSIONS (TDX)" via
"N: tdx" (or maybe "K: \b(tdx)"? I haven't bothered to check which one
triggers). And while I don't think there was ever any formal discussion, AFAIK everyone reads the situation as KVM still being the primary maintainer/tree for that code.
Quoted 1 line
both of which is something you're saying you _don't_ want.
No, what I don't want is a dedicated "KVM LIVE UPDATE" entry, because I think most
people would read "F: virt/kvm/kvm_luo.c" and "F: virt/kvm/guest_memfd_luo.c"
as being more precise than KVM's "F: virt/kvm/*" and thus would read things as
"KVM LIVE UPDATE" being the primary maintainer.
In other words, I'm more than ok with LUO being looped in on KVM LUO changes and having the authority to object to problematic changes, but I'm not ok with LUO taking primary ownership of KVM code.
I realize there's more than a bit of nuance in my interpretation of N: and K:, but again my experience with TDX is that so long as the maintainers are aligned on expectations, it's a non-issue in practice.
Quoted 21 lines
Quoted 9 lines
Quoted 7 lines
At the same time, I also keep being (pleasantly)
surprised at preservation being relatively simple. For example, the code to preserve a shmem file (via memfd) is roughly 600 lines, a big chunk of which is comments. The code of course has some limitations, but it is good enough for use in production.For one, we care about ABI breakages and versioning.
Which is amusing to me because that implies KVM does not, and I would hazard to
No, it doesn't. What I'm saying is I care about changes to the
_liveupdateABI_. Just like you probably care about changes to KVM ABI but not so much about BPF for example.Quoted 7 lines
guess that KVM has the biggest ABI surface of any subsystem in the kernel by a country mile (though I'm probably wildly underestimating the effective ABI surface of filesystems).
Quoted 2 lines
The serialized state is a part of live update ABI and changes to it should be ACKed by us.
Quoted 7 lines
Quoted 2 lines
Meh, "Don't break userspace" is a universal rule in the kernel, I genuinely don't see why liveupdate needs special treatment.
Ironically enough, you miss my point. I'm not talking about userspace ABI. We all know not to break that. I am talking about live update
_serializationABI_. See the stuff underinclude/linux/kho/abi. This series also adds things there.
No, I understand exactly what ABI you're talking about.
Quoted 3 lines
This is ABI between kernels. It needs to be stable-ish so you can move from one kernel version to another. At the same time, unlike userspace ABI, it can change.
Uh, yeah, so KVM has been managing such immutable ABI for practically its entire existence. KVM's save/restore uAPI has exactly what you're describing: serialization ABI that needs to be backwards and forwards compatible between different kernels in order to support both upgrade and rollback scenarios via live migration.
KVM also has immutable ABI between itself and guest kernels, including implicit "ABI" in the form of not changing guest-visible behavior.
Quoted 7 lines
Today we don't have any rules and let you change things freely as long as you do a version bump. But at a later point, the plan is to add some stability requirements to the ABI so you can actually upgrade the kernel across major versions.
So at least for ABI changes, there should be an explicit ACK from the live update group.
I don't entirely agree. I get where you're coming from, and I 100% agree that the more eyeballs on changes that may affect ABI, the better.
Where I have problems with the above is that it doesn't account for the nuances of save/restore across different kernels. The literal format of the serialized data is the most obvious form of ABI, but it's absolutely possible to change ABI without changing the data format.
E.g. say there's a flags field in some serialization structure, and flags X and Y are mutually exclusive in current kernels. If a future kernel relaxes that restriction for whatever reason, then the effective ABI has been broken because state created and saved on new kernels can't be restored on old kernels. And this is not a theoretical concern, KVM has run afoul of this a few times (though thankfully very rarely).
I don't think it's reasonable to expect LUO maintainers to gain enough expertise in each subsystem to be able to ensure changes are forwards and backwards compatible. The only way I see LUO being successful in the long term is to get subsystem maintainers/contributors to understand *and buy-in* to the LUO model and rules, so that each subsystem can largely be self-sustatining. I.e. setting yourselves up as literal gatekeepers will help prevent blatant breakage, but it's less likely to help guard against more subtle breakage, and in my experience, subtle breakage is by far harder to detect and more painful to deal with.
Somewhat of a side topic: in my experience, using monotonically increasing version numbers is a horrible way to enumerate features/content. So for me, allowing KVM's LUO ABI to change with a verson bump is probably a non-starter. I.e. whatever gets merged needs to be more future-proof than "we'll deal with it later".
Pratyush Yadav
maintainerRe: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
Quoted 2 lines · Sean Christopherson
Quoted 1 line · Pratyush Yadav
On Tue, Aug 11 2026, Sean Christopherson wrote:
[...]
Quoted 9 lines
But that's all beside the point. What I'm saying is that SNP and TDX *must* use
guest_memfd, whereasguest_memfdis optional for all other VM types. And so adding LUO support forguest_memfdwithout even sketching out a plan for CoCo VMs feels backwards.I'm not necessarily opposed to gradual
guest_memfdsupport, but there needs to be a clear plan of how all of this is going to fit together. It doesn't need to be perfect, and I'm sure we'll make mistakes along the way, but I want to at least try not to paint ourselves into a corner, especially with respect to the ABI.
Right. I think landing shared guest_memfd is the first step. It is simple enough and doesn't need to preserve the more complicated architecture specific things like secure EPT, etc. The CoCo support can build on top of this.
There is work underway already for an RFC of TDX live update, but Tarun knows the details better than me so I'll let him fill in the details for the next version of this series.
Quoted 41 lines
Quoted 23 lines
Quoted 23 lines
Quoted 8 lines
Quoted 4 lines
E.g. the LUO APIs seem pretty straightforward; I assume the bulk of the complexity is going to be in knowing what to save/restore, and how, which is much more about KVM than it is about liveupdate.
I think it is fine if you want to take these changes through the KVM tree, but I would like live update maintainers to be listed as reviewers at least.
Why not simply add a file pattern match to the LIVE UPDATE entry?
MAINTAINERS +1
-1@@ -15052,8 +15052,8 @@ F: include/linux/liveupdate.h F: include/uapi/linux/liveupdate.h F: kernel/liveupdate/ F: lib/tests/liveupdate.c -F: mm/memfd_luo.c F: tools/testing/selftests/liveupdate/ +N: [^a-z]luoQuoted 4 lines
MAINTAINERS F: include/linux/liveupdate.h · line 15059
Quoted 2 lines
MAINTAINERS +0
-0LLC (802.2) L: [email protected]This would list us as maintainers of
kvm_luo.cand the tree as liveupdate.git,It would list both "LIVE UPDATE" and "KERNEL VIRTUAL MACHINE (KVM)", as KVM would
still cover the files via "F:virt/kvm/*". And my read ofMAINTAINERSis that
N: and K: entries are "secondary" if the files/scope is covered by an F: entry.E.g.
arch/x86/kvm/vmx/tdx.cis covered by "KERNEL VIRTUAL MACHINE FOR X86 (KVM/x86)"
via "F:arch/x86/kvm/*/", and also by "X86 TRUST DOMAIN EXTENSIONS (TDX)" via
"N: tdx" (or maybe "K: \b(tdx)"? I haven't bothered to check which one
triggers). And while I don't think there was ever any formal discussion, AFAIK everyone reads the situation as KVM still being the primary maintainer/tree for that code.Quoted 1 line
both of which is something you're saying you _don't_ want.
Quoted 8 lines
No, what I don't want is a dedicated "KVM LIVE UPDATE" entry, because I think most
people would read "F:virt/kvm/kvm_luo.c" and "F:virt/kvm/guest_memfd_luo.c"
as being more precise than KVM's "F:virt/kvm/*" and thus would read things as
"KVM LIVE UPDATE" being the primary maintainer.In other words, I'm more than ok with LUO being looped in on KVM LUO changes and having the authority to object to problematic changes, but I'm not ok with LUO taking primary ownership of KVM code.
Sure, that sounds good to me.
Quoted 4 lines
I realize there's more than a bit of nuance in my interpretation of N: and K:, but again my experience with TDX is that so long as the maintainers are aligned on expectations, it's a non-issue in practice.
Yep, as long as we agree among ourselves I don't think the red tape around which entry has higher priority matters much.
Quoted 32 lines
Quoted 22 lines
Quoted 10 lines
Quoted 8 lines
At the same time, I also keep being (pleasantly)
surprised at preservation being relatively simple. For example, the code to preserve a shmem file (via memfd) is roughly 600 lines, a big chunk of which is comments. The code of course has some limitations, but it is good enough for use in production.For one, we care about ABI breakages and versioning.
Which is amusing to me because that implies KVM does not, and I would hazard to
No, it doesn't. What I'm saying is I care about changes to the
_liveupdateABI_. Just like you probably care about changes to KVM ABI but not so much about BPF for example.Quoted 7 lines
guess that KVM has the biggest ABI surface of any subsystem in the kernel by a country mile (though I'm probably wildly underestimating the effective ABI surface of filesystems).
Quoted 2 lines
The serialized state is a part of live update ABI and changes to it should be ACKed by us.
Quoted 7 lines
Quoted 2 lines
Meh, "Don't break userspace" is a universal rule in the kernel, I genuinely don't see why liveupdate needs special treatment.
Ironically enough, you miss my point. I'm not talking about userspace ABI. We all know not to break that. I am talking about live update
_serializationABI_. See the stuff underinclude/linux/kho/abi. This series also adds things there.No, I understand exactly what ABI you're talking about.
Quoted 8 lines
Quoted 3 lines
This is ABI between kernels. It needs to be stable-ish so you can move from one kernel version to another. At the same time, unlike userspace ABI, it can change.
Uh, yeah, so KVM has been managing such immutable ABI for practically its entire existence. KVM's save/restore uAPI has exactly what you're describing: serialization ABI that needs to be backwards and forwards compatible between different kernels in order to support both upgrade and rollback scenarios via live migration.
I think there is a slight difference between KVM's save/resture uAPI and live update's ABI. With KVM's uAPI, you need to maintain strict backwards compatibility because userspace reads what you output. So if you change the layout, userspace will interpret it wrong and might break.
With live update, the ABI never gets to userspace. It is used to talk between kernels directly. So if you do break that, your userspace keeps working fine, you just might not be able to live update to the incompatible kernel.
So with live update, we don't need to keep backwards compatibility in the ABI forever. Of course, it is good to minimize changes, but we have more freedom to change it.
The current idea is we change the ABI whenever needed. This is because live update is still in development so we are likely to see a lot more changes before we settle down into something that mostly works. At some point, we want to start being more strict between incompatible ABI changes. The current idea is that we would like to keep backwards compatibility across LTS kernels. But none of that is set in stone yet.
That's why I would like to have an explicit ACK from the live update group for ABI changes. I would like to enforce these compatibility windows.
Quoted 20 lines
KVM also has immutable ABI between itself and guest kernels, including implicit "ABI" in the form of not changing guest-visible behavior.
Quoted 7 lines
Today we don't have any rules and let you change things freely as long as you do a version bump. But at a later point, the plan is to add some stability requirements to the ABI so you can actually upgrade the kernel across major versions.
So at least for ABI changes, there should be an explicit ACK from the live update group.
I don't entirely agree. I get where you're coming from, and I 100% agree that the more eyeballs on changes that may affect ABI, the better.
Where I have problems with the above is that it doesn't account for the nuances of save/restore across different kernels. The literal format of the serialized data is the most obvious form of ABI, but it's absolutely possible to change ABI without changing the data format.
Quoted 6 lines
E.g. say there's a flags field in some serialization structure, and flags X and Y are mutually exclusive in current kernels. If a future kernel relaxes that restriction for whatever reason, then the effective ABI has been broken because state created and saved on new kernels can't be restored on old kernels. And this is not a theoretical concern, KVM has run afoul of this a few times (though thankfully very rarely).
That's a bug, and should be fixed.
I am talking here about intentional ABI changes. As I mentioned above we _can_ change the ABI, we just need to be careful about it.
Quoted 9 lines
I don't think it's reasonable to expect LUO maintainers to gain enough expertise in each subsystem to be able to ensure changes are forwards and backwards compatible. The only way I see LUO being successful in the long term is to get subsystem maintainers/contributors to understand *and buy-in* to the LUO model and rules, so that each subsystem can largely be self-sustatining. I.e. setting yourselves up as literal gatekeepers will help prevent blatant breakage, but it's less likely to help guard against more subtle breakage, and in my experience, subtle breakage is by far harder to detect and more painful to deal with.
Outside of enforcing the ABI compatibility windows, I think this makes sense.
In principle at least, though from my experience the maintainer appetite to care about live update varies. The feeling I've got from MM for example is more along the lines of "you break it, you buy it". Which is also a valid model IMO. The person who wrote the live update code for a subsystem is well equipped to understand these nuances.
Quoted 5 lines
Somewhat of a side topic: in my experience, using monotonically increasing version numbers is a horrible way to enumerate features/content. So for me, allowing KVM's LUO ABI to change with a verson bump is probably a non-starter. I.e. whatever gets merged needs to be more future-proof than "we'll deal with it later".
As I wrote above, the compatibility model for live update allows breaking ABI with version bumps. Userspace keeps working fine, it only restricts the set of kernels that you can go to. So the "we'll deal with it later" is not as harmful because our choices here are not permanent.
Also, there is work underway to allow transitions between versions [0].
But still, I am not opposed to a more flexible guest_memfd ABI. I think we can reserve some space for feature flags to let us add things in the future.
[0] https://lore.kernel.org/kexec/[email protected]/
Signature · 2 lines
Regards, Pratyush Yadav
Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
Quoted 17 lines · Pratyush Yadav
Quoted 9 lines · Sean Christopherson
Quoted 3 lines · Pratyush Yadav
This is ABI between kernels. It needs to be stable-ish so you can move from one kernel version to another. At the same time, unlike userspace ABI, it can change.
Uh, yeah, so KVM has been managing such immutable ABI for practically its entire existence. KVM's save/restore uAPI has exactly what you're describing: serialization ABI that needs to be backwards and forwards compatible between different kernels in order to support both upgrade and rollback scenarios via live migration.
I think there is a slight difference between KVM's save/resture uAPI and live update's ABI. With KVM's uAPI, you need to maintain strict backwards compatibility because userspace reads what you output. So if you change the layout, userspace will interpret it wrong and might break.
Quoted 8 lines
With live update, the ABI never gets to userspace. It is used to talk between kernels directly. So if you do break that, your userspace keeps working fine, you just might not be able to live update to the incompatible kernel.
So with live update, we don't need to keep backwards compatibility in the ABI forever. Of course, it is good to minimize changes, but we have more freedom to change it.
Ah. So this is the heart of the disconnect. I very strongly disagree with the statement that live update doesn't need to support backwards compatibility. I can totally believe that the folks working on live update are ok breaking backwards compatibility because their use cases are "fine" with such breakage. And I can also believe live update as an upstream kernel feature being developed by those same folks is also ok with breaking backwards compatibility.
But with my upstream KVM maintainer hat on, I am not ok with that. I did not agree to support a world where KVM is allowed to break backwards compatibility, so long as it's done "carefully" or whatever. If y'all want to deal with the resulting complexity, that's fine by me, but you'll be doing it without KVM.
I totally understand that exploratory work and initial development is best done in private and/or in a small working groups. But decisions that will significantly impact multiple subsystems need to be made *with* those subystems. I mean, obviously it's possible to make a decision in a small group and then "publish" the result later, but then as is happening here, the community and subsystem maintainers like me may refuse to play ball if they disagree.
To be very clear, for any KVM live update support that doesn't guarantee backwards compatibility (allowing that bugs happen), a very firm:
NAK
Pratyush Yadav
maintainerRe: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
Hi Sean,
Quoted 34 lines · Sean Christopherson
Quoted 17 lines · Pratyush Yadav
Quoted 9 lines · Sean Christopherson
Quoted 3 lines · Pratyush Yadav
This is ABI between kernels. It needs to be stable-ish so you can move from one kernel version to another. At the same time, unlike userspace ABI, it can change.
Uh, yeah, so KVM has been managing such immutable ABI for practically its entire existence. KVM's save/restore uAPI has exactly what you're describing: serialization ABI that needs to be backwards and forwards compatible between different kernels in order to support both upgrade and rollback scenarios via live migration.
I think there is a slight difference between KVM's save/resture uAPI and live update's ABI. With KVM's uAPI, you need to maintain strict backwards compatibility because userspace reads what you output. So if you change the layout, userspace will interpret it wrong and might break.
Quoted 8 lines
With live update, the ABI never gets to userspace. It is used to talk between kernels directly. So if you do break that, your userspace keeps working fine, you just might not be able to live update to the incompatible kernel.
So with live update, we don't need to keep backwards compatibility in the ABI forever. Of course, it is good to minimize changes, but we have more freedom to change it.
Ah. So this is the heart of the disconnect. I very strongly disagree with the statement that live update doesn't need to support backwards compatibility. I can totally believe that the folks working on live update are ok breaking backwards compatibility because their use cases are "fine" with such breakage. And I can also believe live update as an upstream kernel feature being developed by those same folks is also ok with breaking backwards compatibility.
Quoted 4 lines
But with my upstream KVM maintainer hat on, I am not ok with that. I did not agree to support a world where KVM is allowed to break backwards compatibility, so long as it's done "carefully" or whatever. If y'all want to deal with the resulting complexity, that's fine by me, but you'll be doing it without KVM.
Let's step back a bit. I don't think backwards compatibility in the _ABI_ is all that important in live update's context. What is important is that our users are able to live update from kernel version X to Y. The line format the kernel uses to describe its state is an implementation detail. Our users never see it.
The high level idea is that when you need to make a change to the ABI so the kernel can better describe the objects/resources/files it is passing, you create a new version of the ABI and allow transitions from older versions.
Say you make some changes and need an ABI version v5. You don't get rid of v4. You keep it around. LUO core will facilitate picking the right version for the next kernel. So it will be possible to seamlessly upgrade your kernel that speaks v4 to a new kernel that speaks v4 and v5. Now this kernel can start speaking v5 if its successor speaks v5 too. And so on for going to v6 and v7, etc.
After a "reasonable" time given for upgrades, you deprecate v4. v4 has been around long enough and our users have had a chance to go to kernels speaking newer versions. That's when you remove the code for v4 from the kernel.
We can argue what "reasonable" means, but the core idea stays.
It will still be possible to go back to v4 or earlier, but you'd need an extra stop along the way. And of course, vendors can keep a wider support matrix downstream if they see the need for it.
Does this idea of "backwards compatibility" sound acceptable to you, at least at a high level?
Now coming back to today's reality. We don't yet support this version transition. I proposed the idea at LPC 2025 [0]. Logan Odell has taken over the work since I have other things on my plate, but it is something being actively developed. Logan recently sent some RFCs [1] for this too, though TBH I haven't yet gotten to those patches. We also have proposed a talk about this at LPC 2026, it would be great if you could attend (if the talk gets accepted).
If you say that we should figure this version transition out and land it upstream before we land the KVM code, I think that's a fair ask. I can work with that.
But I think we first need to agree on the principles of the compatibility model I described above.
[0] https://lpc.events/event/19/contributions/2049/
[1] https://lore.kernel.org/kexec/[email protected]/T/#u
Quoted 7 lines
I totally understand that exploratory work and initial development is best done in private and/or in a small working groups. But decisions that will significantly impact multiple subsystems need to be made *with* those subystems. I mean, obviously it's possible to make a decision in a small group and then "publish" the result later, but then as is happening here, the community and subsystem maintainers like me may refuse to play ball if they disagree.
The design of live update was discussed and agreed upon in the open. David Rientjes (and now Pasha) has been hosting the live update biweeklies [2] from the very start of the project. The series is open to everyone and notes shared for those who couldn't join. This meeting was attended by a lot of people from different companies and different subsystem backgrounds.
There have also been a microconference at LPC 2025 where topics around live update in PCI, MM, VFIO, IOMMU, etc. were discussed, along with presentations at subsystem tracks in other conferences.
Of course, it is impossible to get all maintainers of all subsystems to attend these sessions or conferences. That's fine.
But I think it is quite unfair to say that live update is developed in private or in small groups.
[2] https://lore.kernel.org/all/?q=s%3A%22%5BHypervisor+Live+Update%5D%22
Signature · 2 lines
Regards, Pratyush Yadav
Sean Christopherson
maintainerRe: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
Quoted 23 lines · Pratyush Yadav
Hi Sean,
Quoted 14 lines · Sean Christopherson
Quoted 5 lines · Pratyush Yadav
Quoted 1 line · Sean Christopherson
On Wed, Aug 12, 2026, Pratyush Yadav wrote:
So with live update, we don't need to keep backwards compatibility in the ABI forever. Of course, it is good to minimize changes, but we have more freedom to change it.
Ah. So this is the heart of the disconnect. I very strongly disagree with the statement that live update doesn't need to support backwards compatibility. I can totally believe that the folks working on live update are ok breaking backwards compatibility because their use cases are "fine" with such breakage. And I can also believe live update as an upstream kernel feature being developed by those same folks is also ok with breaking backwards compatibility.
Quoted 4 lines
But with my upstream KVM maintainer hat on, I am not ok with that. I did not agree to support a world where KVM is allowed to break backwards compatibility, so long as it's done "carefully" or whatever. If y'all want to deal with the resulting complexity, that's fine by me, but you'll be doing it without KVM.
Quoted 5 lines
Let's step back a bit. I don't think backwards compatibility in the _ABI_ is all that important in live update's context. What is important is that our users are able to live update from kernel version X to Y. The line format the kernel uses to describe its state is an implementation detail. Our users never see it.
The rule is thou shalt not break userspace. Whether or not the breakage is the result of an explicit ABI change is irrelevant.
Quoted 17 lines
The high level idea is that when you need to make a change to the ABI so the kernel can better describe the objects/resources/files it is passing, you create a new version of the ABI and allow transitions from older versions.
Say you make some changes and need an ABI version v5. You don't get rid of v4. You keep it around. LUO core will facilitate picking the right version for the next kernel. So it will be possible to seamlessly upgrade your kernel that speaks v4 to a new kernel that speaks v4 and v5. Now this kernel can start speaking v5 if its successor speaks v5 too. And so on for going to v6 and v7, etc.
After a "reasonable" time given for upgrades, you deprecate v4. v4 has been around long enough and our users have had a chance to go to kernels speaking newer versions. That's when you remove the code for v4 from the kernel.
Quoted 8 lines
We can argue what "reasonable" means, but the core idea stays.
It will still be possible to go back to v4 or earlier, but you'd need an extra stop along the way. And of course, vendors can keep a wider support matrix downstream if they see the need for it.
Does this idea of "backwards compatibility" sound acceptable to you, at least at a high level?
No.
It's probably fine for Google and other large companies that tightly control their kernels and use cases, and have the resources to juggle the resulting complexity, e.g. have kernel engineers on staff to track feature and dependencies, coordinate and plan kernel upgrades, etc.
It's not acceptable for upstream, where downstream consumers often run a distro kernel, have much more varied use cases, and don't always have a horde of kernel engineers on staff to help them thread the needle you describe above. And if supporting live update as a general feature for all users of the kernel isn't being factored into design considerations, then that needs to change, otherwise this is all dead in the water.
I also don't see the point. Maintaining a rigid save/restore ABI is annoying, but it's not _hard_ (or at least, not _that_ hard), especially if there's a set of well-documented best known practices that subsystems can follow, e.g. so that individual subsystems don't need to learn painful lessons first-hand. I genuinely believe that maintaining the version hell you describe above would be more costly in the long run than simply committing to full backwards compatibility within a given subsystem. I can imagine that enumerating what subsystems' information is in the payload will require a different scheme, but for a given subsystem, I don't see any reason to aim for anything less than full backwards compatibility.
Pratyush Yadav
maintainerRe: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
Hi Sean,
Quoted 73 lines · Sean Christopherson
Quoted 23 lines · Pratyush Yadav
Hi Sean,
Quoted 14 lines · Sean Christopherson
Quoted 5 lines · Pratyush Yadav
Quoted 1 line · Sean Christopherson
On Wed, Aug 12, 2026, Pratyush Yadav wrote:
So with live update, we don't need to keep backwards compatibility in the ABI forever. Of course, it is good to minimize changes, but we have more freedom to change it.
Ah. So this is the heart of the disconnect. I very strongly disagree with the statement that live update doesn't need to support backwards compatibility. I can totally believe that the folks working on live update are ok breaking backwards compatibility because their use cases are "fine" with such breakage. And I can also believe live update as an upstream kernel feature being developed by those same folks is also ok with breaking backwards compatibility.
Quoted 4 lines
But with my upstream KVM maintainer hat on, I am not ok with that. I did not agree to support a world where KVM is allowed to break backwards compatibility, so long as it's done "carefully" or whatever. If y'all want to deal with the resulting complexity, that's fine by me, but you'll be doing it without KVM.
Quoted 5 lines
Let's step back a bit. I don't think backwards compatibility in the _ABI_ is all that important in live update's context. What is important is that our users are able to live update from kernel version X to Y. The line format the kernel uses to describe its state is an implementation detail. Our users never see it.
The rule is thou shalt not break userspace. Whether or not the breakage is the result of an explicit ABI change is irrelevant.
Quoted 17 lines
The high level idea is that when you need to make a change to the ABI so the kernel can better describe the objects/resources/files it is passing, you create a new version of the ABI and allow transitions from older versions.
Say you make some changes and need an ABI version v5. You don't get rid of v4. You keep it around. LUO core will facilitate picking the right version for the next kernel. So it will be possible to seamlessly upgrade your kernel that speaks v4 to a new kernel that speaks v4 and v5. Now this kernel can start speaking v5 if its successor speaks v5 too. And so on for going to v6 and v7, etc.
After a "reasonable" time given for upgrades, you deprecate v4. v4 has been around long enough and our users have had a chance to go to kernels speaking newer versions. That's when you remove the code for v4 from the kernel.
Quoted 8 lines
We can argue what "reasonable" means, but the core idea stays.
It will still be possible to go back to v4 or earlier, but you'd need an extra stop along the way. And of course, vendors can keep a wider support matrix downstream if they see the need for it.
Does this idea of "backwards compatibility" sound acceptable to you, at least at a high level?
No.
It's probably fine for Google and other large companies that tightly control their kernels and use cases, and have the resources to juggle the resulting complexity, e.g. have kernel engineers on staff to track feature and dependencies, coordinate and plan kernel upgrades, etc.
It's not acceptable for upstream, where downstream consumers often run a distro kernel, have much more varied use cases, and don't always have a horde of kernel engineers on staff to help them thread the needle you describe above. And if supporting live update as a general feature for all users of the kernel isn't being factored into design considerations, then that needs to change, otherwise this is all dead in the water.
Quoted 9 lines
I also don't see the point. Maintaining a rigid save/restore ABI is annoying, but it's not _hard_ (or at least, not _that_ hard), especially if there's a set of well-documented best known practices that subsystems can follow, e.g. so that individual subsystems don't need to learn painful lessons first-hand. I genuinely believe that maintaining the version hell you describe above would be more costly in the long run than simply committing to full backwards compatibility within a given subsystem. I can imagine that enumerating what subsystems' information is in the payload will require a different scheme, but for a given subsystem, I don't see any reason to aim for anything less than full backwards compatibility.
Let's say for argument's sake that we commit for a fully stable backwards compatible ABI. Even then, you have to deal with multiple ABI versions.
Live update's ABI is more complex compared to KVM's save/restore ABI. For the KVM save/restore uAPI, you are largely describing architectural state like CPU registers, etc. These things don't evolve as fast and more or less stay the same.
Live update needs to describe the state of kernel objects. These are more complex and evolve faster.
For example, say you merge guest_memfd preservation today. Some time later, someone comes up with a more efficient data structure to track the folios in the file. You _have_ to make a backwards-incompatible ABI change to use this data structure.
Or say you add a new memory backend (like the HugeTLB patches in flight). That likely will need a different ABI to describe the state of the guest_memfd.
So you will end up with multiple ABI versions that aren't always backwards compatible.
If you refuse that idea too, then KVM live update will be dead in the water for a different reason. It will be damn near useless because it can't keep up with an evolving subsystem.
Now once you get multiple ABI versions and you can seamlessly go from old to new one, say you have a version that was superseded 5 years ago. It would be completely reasonable to say that this version is old enough and no one should be going from a 5 year old kernel to a modern one. So you deprecate this ABI. Deprecating old unused uAPIs is not unprecedented.
I think we are better off formalizing this deprecation period from the get go.
A somewhat tangential example is BPF kfuncs. My BPF program that works in kernel X might not work in kernel Y because the kfunc has changed or been removed.
The argument they make in kfuncs.rst is that kfuncs "provide a kernel <-> kernel API, and thus are not bound by any of the strict stability restrictions associated with kernel <-> user UAPIs". For LUO as well, this is a kernel -> kernel API. Users can also still do a regular kexec or reboot. They just won't get the performance optimization of LUO.
Regardless of if you agree with the last bit about deprecating old versions, ABIs evolving with the subsystem is a ground reality of live update and it would be foolish to think we can do with only backwards-compatible ABI changes forever.
Signature · 2 lines
Regards, Pratyush Yadav
Sean Christopherson
maintainerRe: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
Quoted 30 lines · Pratyush Yadav
Hi Sean,
Quoted 17 lines · Sean Christopherson
Quoted 2 lines
Does this idea of "backwards compatibility" sound acceptable to you, at least at a high level?
No.
It's probably fine for Google and other large companies that tightly control their kernels and use cases, and have the resources to juggle the resulting complexity, e.g. have kernel engineers on staff to track feature and dependencies, coordinate and plan kernel upgrades, etc.
It's not acceptable for upstream, where downstream consumers often run a distro kernel, have much more varied use cases, and don't always have a horde of kernel engineers on staff to help them thread the needle you describe above. And if supporting live update as a general feature for all users of the kernel isn't being factored into design considerations, then that needs to change, otherwise this is all dead in the water.
Quoted 9 lines
I also don't see the point. Maintaining a rigid save/restore ABI is annoying, but it's not _hard_ (or at least, not _that_ hard), especially if there's a set of well-documented best known practices that subsystems can follow, e.g. so that individual subsystems don't need to learn painful lessons first-hand. I genuinely believe that maintaining the version hell you describe above would be more costly in the long run than simply committing to full backwards compatibility within a given subsystem. I can imagine that enumerating what subsystems' information is in the payload will require a different scheme, but for a given subsystem, I don't see any reason to aim for anything less than full backwards compatibility.
Quoted 8 lines
Let's say for argument's sake that we commit for a fully stable backwards compatible ABI. Even then, you have to deal with multiple ABI versions.
Live update's ABI is more complex compared to KVM's save/restore ABI. For the KVM save/restore uAPI, you are largely describing architectural state like CPU registers, etc. These things don't evolve as fast and more or less stay the same.
Right, because nothing meaningful has changed in any architecture in the 20+ years since KVM has provided save/restore support, whereas guest_memfd looks nothing like it did when it was introduced three years ago.
Quoted 2 lines
Live update needs to describe the state of kernel objects. These are more complex
LOL, you might be the first to claim x86 virtualization isn't all that complex.
Quoted 1 line
and evolve faster.
The speed at which things change doesn't automatically mean we shouldn't strive for backwards compatibility. Yes, providing backwards compatibility requires additional care and planning, and over time *might* lead to an ABI that is difficult to maintain. But IMO, that just makes it all the more important to get the design right the first time, not that we shouldn't even try because it's hard.
Quoted 4 lines
For example, say you merge
guest_memfdpreservation today. Some time later, someone comes up with a more efficient data structure to track the folios in the file. You _have_ to make a backwards-incompatible ABI change to use this data structure.
Only if those details bleed into the ABI/contract. I actually have a concrete KVM (well, virtualization) example for this.
Intel's VMX architecture disallows direct memory accesses to the VMCS, and instead requires software to access the VMCS via dedicated ISA, using architectural encoding numbers to reference VMCS fields. I.e. VMX decouples how data is stored in memory (the data structures) from the ABI/contract with software (VMCS field encodings).
This allows Intel to optimize the data structures to be more efficient and performant for each microarchitecture based on the features and properties of each uarch, all without breaking backwards/forwards compatibility with software.
My favorite esoteric example is AR_BYTES packing. For Haswell, Intel added an optimization in ucode to allow saving/loading segment register state in a single uop (IIRC). The optimization was especially valuable for virtualization as it shaved cycles off the VM-Enter/VM-Exit hot paths. A key piece of the optimization was it required the AR_BYTES metadata to be stored in 16 bits, but existing CPUs stored AR_BYTES using 32 bits in an "unpacked" format.
Fortunately, because the in-memory representation was decoupled from the contract with software, Intel could pack AR_BYTES into 16 bits for Haswell+ and pack/unpack the data on VMWRITE/VMREAD, so that the format presented to software remained unchanged.
Does VMX's decoupling of the in-memory represntation of a VMCS have downsides? Absolutely. Most notably, it incurs extra complexity (in software and hardware) to achieve comparable performance to directly accessible data structures (AMD's VMCB and Hyper-V's eVMCS) for nested virtualization. And I'm sure it has placed contraints on Intel's designs, and obviously introduces complexity into the overall system by adding a layer of indirection. But IMO the VMX architecture has been a huge win overall for Intel.
Quoted 6 lines
Or say you add a new memory backend (like the HugeTLB patches in flight). That likely will need a different ABI to describe the state of the
guest_memfd.So you will end up with multiple ABI versions that aren't always backwards compatible.
No, you end up with *features* that aren't backwards compatible. I can't imagine anyone will argue that we should never add new features because then we can't rollback to an older kernel.
But adding a new feature shouldn't break the existing ABI. E.g. adding support for HugeTLB in guest_memfd shouldn't prevent rolling back to an older kernel when the HugeTLB functionality isn't being used.
Using AMD's VMCB and Intel's VMCS as examples, literally every major new AMD/Intel uarch extends the VMC{B,S} in some way, but without fail it's always done in a way that is backwards compatible with existing software. I.e. AMD and Intel ship new features, but existing software continues to work, and VMs continue to be migratable across CPU generations[*], with the obvious restriction that migrating a VM using a feature introduced on generation N to a generation N-1 CPU isn't a smart idea.
[*] There are exceptions. E.g. Intel removed MPX, and so VMs with MPX can't be
migrated to newer CPUs. Migrating between CPUs with different MAXPHYADDR is sketchy (and simply not done by some CSPs) because neither AMD nor Intel virtualizes MAXPHYADDR. But those exceptions are absolutely Big Deals that undergo significant scrutiny, from all parties involved.
Quoted 8 lines
If you refuse that idea too, then KVM live update will be dead in the water for a different reason. It will be damn near useless because it can't keep up with an evolving subsystem.
Now once you get multiple ABI versions and you can seamlessly go from old to new one, say you have a version that was superseded 5 years ago. It would be completely reasonable to say that this version is old enough and no one should be going from a 5 year old kernel to a modern one.
LOL, Google literally does this. Granted, the extreme cases only happen for stragglers, and I think we do force VMs to bounce through a "middle" kernel in those cases, but I doubt Google is the only company that runs frankenkernels for an absurd number of years for a variety of reasons. E.g. 4.4 LTS was officially supported for 6 years, and I'll bet dollars to donuts people ran it for much longer than that.
Quoted 1 line
So you deprecate this ABI. Deprecating old unused uAPIs is not unprecedented.
When there are provably no users, or we can convince existing userspace to migrate to an alternative.
Quoted 2 lines
I think we are better off formalizing this deprecation period from the get go.
Why? What does it buy us? Because all I see is potential abuse and an excuse for not spending time getting the designs right.
Quoted 7 lines
A somewhat tangential example is BPF kfuncs. My BPF program that works in kernel X might not work in kernel Y because the kfunc has changed or been removed.
The argument they make in
kfuncs.rstis that kfuncs "provide a kernel <-> kernel API, and thus are not bound by any of the strict stability restrictions associated with kernel <-> user UAPIs".
BPF's documentation isn't arguing anything, it's merely reiterating Linux's long-standing policy that there is no such thing as a stable kernel ABI (in upstream).
Quoted 1 line
For LUO as well, this is a kernel -> kernel API.
Stating the obvious, I disagree with this. As I said before, if this is the stance LUO wants to take, then so be it, but my NAK stands.
Quoted 2 lines
Users can also still do a regular kexec or reboot. They just won't get the performance optimization of LUO.
The amount of time, energy, and money poured into minimizing VM downtime on live migration suggests the overwhelming majority of LUO's targeted users aren't going to take kindly to this stance.
Quoted 4 lines
Regardless of if you agree with the last bit about deprecating old versions, ABIs evolving with the subsystem is a ground reality of live update and it would be foolish to think we can do with only backwards-compatible ABI changes forever.
I never said the ABI is immutable, I said it needs to be backwards/forwards compatible.
Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
Quoted 173 lines · Sean Christopherson
Quoted 30 lines · Pratyush Yadav
Hi Sean,
Quoted 17 lines · Sean Christopherson
Quoted 2 lines
Does this idea of "backwards compatibility" sound acceptable to you, at least at a high level?
No.
It's probably fine for Google and other large companies that tightly control their kernels and use cases, and have the resources to juggle the resulting complexity, e.g. have kernel engineers on staff to track feature and dependencies, coordinate and plan kernel upgrades, etc.
It's not acceptable for upstream, where downstream consumers often run a distro kernel, have much more varied use cases, and don't always have a horde of kernel engineers on staff to help them thread the needle you describe above. And if supporting live update as a general feature for all users of the kernel isn't being factored into design considerations, then that needs to change, otherwise this is all dead in the water.
Quoted 9 lines
I also don't see the point. Maintaining a rigid save/restore ABI is annoying, but it's not _hard_ (or at least, not _that_ hard), especially if there's a set of well-documented best known practices that subsystems can follow, e.g. so that individual subsystems don't need to learn painful lessons first-hand. I genuinely believe that maintaining the version hell you describe above would be more costly in the long run than simply committing to full backwards compatibility within a given subsystem. I can imagine that enumerating what subsystems' information is in the payload will require a different scheme, but for a given subsystem, I don't see any reason to aim for anything less than full backwards compatibility.
Quoted 8 lines
Let's say for argument's sake that we commit for a fully stable backwards compatible ABI. Even then, you have to deal with multiple ABI versions.
Live update's ABI is more complex compared to KVM's save/restore ABI. For the KVM save/restore uAPI, you are largely describing architectural state like CPU registers, etc. These things don't evolve as fast and more or less stay the same.
Right, because nothing meaningful has changed in any architecture in the 20+ years since KVM has provided save/restore support, whereas
guest_memfdlooks nothing like it did when it was introduced three years ago.Quoted 2 lines
Live update needs to describe the state of kernel objects. These are more complex
LOL, you might be the first to claim x86 virtualization isn't all that complex.
Quoted 1 line
and evolve faster.
The speed at which things change doesn't automatically mean we shouldn't strive for backwards compatibility. Yes, providing backwards compatibility requires additional care and planning, and over time *might* lead to an ABI that is difficult to maintain. But IMO, that just makes it all the more important to get the design right the first time, not that we shouldn't even try because it's hard.
Quoted 4 lines
For example, say you merge
guest_memfdpreservation today. Some time later, someone comes up with a more efficient data structure to track the folios in the file. You _have_ to make a backwards-incompatible ABI change to use this data structure.Only if those details bleed into the ABI/contract. I actually have a concrete KVM (well, virtualization) example for this.
Intel's VMX architecture disallows direct memory accesses to the VMCS, and instead requires software to access the VMCS via dedicated ISA, using architectural encoding numbers to reference VMCS fields. I.e. VMX decouples how data is stored in memory (the data structures) from the ABI/contract with software (VMCS field encodings).
This allows Intel to optimize the data structures to be more efficient and performant for each microarchitecture based on the features and properties of each uarch, all without breaking backwards/forwards compatibility with software.
My favorite esoteric example is
AR_BYTESpacking. For Haswell, Intel added an optimization in ucode to allow saving/loading segment register state in a single uop (IIRC). The optimization was especially valuable for virtualization as it shaved cycles off the VM-Enter/VM-Exit hot paths. A key piece of the optimization was it required theAR_BYTESmetadata to be stored in 16 bits, but existing CPUs storedAR_BYTESusing 32 bits in an "unpacked" format.Fortunately, because the in-memory representation was decoupled from the contract with software, Intel could pack
AR_BYTESinto 16 bits for Haswell+ and pack/unpack the data on VMWRITE/VMREAD, so that the format presented to software remained unchanged.Does VMX's decoupling of the in-memory represntation of a VMCS have downsides? Absolutely. Most notably, it incurs extra complexity (in software and hardware) to achieve comparable performance to directly accessible data structures (AMD's VMCB and Hyper-V's eVMCS) for nested virtualization. And I'm sure it has placed contraints on Intel's designs, and obviously introduces complexity into the overall system by adding a layer of indirection. But IMO the VMX architecture has been a huge win overall for Intel.
Quoted 6 lines
Or say you add a new memory backend (like the HugeTLB patches in flight). That likely will need a different ABI to describe the state of the
guest_memfd.So you will end up with multiple ABI versions that aren't always backwards compatible.
No, you end up with *features* that aren't backwards compatible. I can't imagine anyone will argue that we should never add new features because then we can't rollback to an older kernel.
But adding a new feature shouldn't break the existing ABI. E.g. adding support for HugeTLB in
guest_memfdshouldn't prevent rolling back to an older kernel when the HugeTLB functionality isn't being used.Using AMD's VMCB and Intel's VMCS as examples, literally every major new AMD/Intel uarch extends the VMC{B,S} in some way, but without fail it's always done in a way that is backwards compatible with existing software. I.e. AMD and Intel ship new features, but existing software continues to work, and VMs continue to be migratable across CPU generations[*], with the obvious restriction that migrating a VM using a feature introduced on generation N to a generation N-1 CPU isn't a smart idea.
[*] There are exceptions. E.g. Intel removed MPX, and so VMs with MPX can't be
migrated to newer CPUs. Migrating between CPUs with different MAXPHYADDR is sketchy (and simply not done by some CSPs) because neither AMD nor Intel virtualizes MAXPHYADDR. But those exceptions are absolutely Big Deals that undergo significant scrutiny, from all parties involved.Quoted 8 lines
If you refuse that idea too, then KVM live update will be dead in the water for a different reason. It will be damn near useless because it can't keep up with an evolving subsystem.
Now once you get multiple ABI versions and you can seamlessly go from old to new one, say you have a version that was superseded 5 years ago. It would be completely reasonable to say that this version is old enough and no one should be going from a 5 year old kernel to a modern one.
LOL, Google literally does this. Granted, the extreme cases only happen for stragglers, and I think we do force VMs to bounce through a "middle" kernel in those cases, but I doubt Google is the only company that runs frankenkernels for an absurd number of years for a variety of reasons. E.g. 4.4 LTS was officially supported for 6 years, and I'll bet dollars to donuts people ran it for much longer than that.
Quoted 1 line
So you deprecate this ABI. Deprecating old unused uAPIs is not unprecedented.
When there are provably no users, or we can convince existing userspace to migrate to an alternative.
Quoted 2 lines
I think we are better off formalizing this deprecation period from the get go.
Why? What does it buy us? Because all I see is potential abuse and an excuse for not spending time getting the designs right.
Quoted 7 lines
A somewhat tangential example is BPF kfuncs. My BPF program that works in kernel X might not work in kernel Y because the kfunc has changed or been removed.
The argument they make in
kfuncs.rstis that kfuncs "provide a kernel <-> kernel API, and thus are not bound by any of the strict stability restrictions associated with kernel <-> user UAPIs".BPF's documentation isn't arguing anything, it's merely reiterating Linux's long-standing policy that there is no such thing as a stable kernel ABI (in upstream).
Quoted 1 line
For LUO as well, this is a kernel -> kernel API.
Stating the obvious, I disagree with this. As I said before, if this is the stance LUO wants to take, then so be it, but my NAK stands.
Quoted 2 lines
Users can also still do a regular kexec or reboot. They just won't get the performance optimization of LUO.
The amount of time, energy, and money poured into minimizing VM downtime on live migration suggests the overwhelming majority of LUO's targeted users aren't going to take kindly to this stance.
Quoted 7 lines
Quoted 4 lines
Regardless of if you agree with the last bit about deprecating old versions, ABIs evolving with the subsystem is a ground reality of live update and it would be foolish to think we can do with only backwards-compatible ABI changes forever.
I never said the ABI is immutable, I said it needs to be backwards/forwards compatible.
Thanks Sean, Pratyush. @loganodell has recently sent the RFC[1] on backward compatibility for liveupdate. This discussion is important so I propose to move the backward compatibility discussion on [1].
For guest_memfd preservation, I will sent v5 soon with the following changes:
1. Suggestions from across the patches
2. tentative Plan for future Guest_memfd preservation
3. tentative Plan for VM preservation.
4. Backward compatibility built on top of RFC [1]
What do you think?
Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
In reply to Sean ChristophersonQuoted 24 lines · Sean Christopherson
Quoted 23 lines · Pratyush Yadav
Hi Sean,
Quoted 18 lines · Sean Christopherson
Quoted 5 lines · Pratyush Yadav
Quoted 1 line · Sean Christopherson
On Wed, Aug 12, 2026, Pratyush Yadav wrote:
So with live update, we don't need to keep backwards compatibility in the ABI forever. Of course, it is good to minimize changes, but we have more freedom to change it.
Ah. So this is the heart of the disconnect. I very strongly disagree with the statement that live update doesn't need to support backwards compatibility. I can totally believe that the folks working on live update are ok breaking backwards compatibility because their use cases are "fine" with such breakage. And I can also believe live update as an upstream kernel feature being developed by those same folks is also ok with breaking backwards compatibility.
But with my upstream KVM maintainer hat on, I am not ok with that. I did not agree to support a world where KVM is allowed to break backwards compatibility, so long as it's done "carefully" or whatever. If y'all want to deal with the resulting complexity, that's fine by me, but you'll be doing it without KVM.
Quoted 8 lines
Quoted 5 lines
Let's step back a bit. I don't think backwards compatibility in the _ABI_ is all that important in live update's context. What is important is that our users are able to live update from kernel version X to Y. The line format the kernel uses to describe its state is an implementation detail. Our users never see it.
The rule is thou shalt not break userspace. Whether or not the breakage is the result of an explicit ABI change is irrelevant.
I argued strongly for this simpler approach, I will write here the reasoning I gave.
Firstly, "live update" is not uABI per-say. It is not "userspace breaking", and it is not a "regression". It is an internal kernel mechanism to allow the kernel to self-upgrade. In the industry it is typical any of these hitless update/patch/etc schemes to only work between a few tightly controlled version pairs.
Asking upstream to carry a full matrix of every single version pair ever released is massive over-engineering and cost on upstream maintainers. Refusing to do this is not a uABI breakage, it is not a regression, it is simply a lack of a feature.
I think upstream should start by agreeing to only support forward going upgrades within a single stable branch. If live update becomes a success then maybe we could do from a stable branch to the immediate next stable branch too. I don't know.
This alone is already much more powerful than live patching.
The balancing issue here is the impact on the kernel subsystems adopting luo. Every time you change anything about the internal function of the subystem you now have to go test a massive version pair matrix to make sure everything works? No thanks.
Luo is very invasive and a huge PITA at the best of times. It doesn't need to be even worse.
Quoted 5 lines
It's probably fine for Google and other large companies that tightly control their kernels and use cases, and have the resources to juggle the resulting complexity, e.g. have kernel engineers on staff to track feature and dependencies, coordinate and plan kernel upgrades, etc.
Every downstream that wants to support luo is going to necessarily severely restrict the version pairs that can work. Like a RH type distro may only support it for 9.1.x -> 9.1.x+1 - they can carry the cost of figuring out how to manage their patching and compatability matrix on their own with the tools upstream provides.
Quoted 4 lines
engineers on staff to help them thread the needle you describe above. And if supporting live update as a general feature for all users of the kernel isn't being factored into design considerations, then that needs to change, otherwise this is all dead in the water.
General uses can use upstream and do CVE upgrades within a single stable branch. That's good enough, lets start there. We don't need to boil the ocean.
Quoted 3 lines
I also don't see the point. Maintaining a rigid save/restore ABI is annoying, but it's not _hard_ (or at least, not _that_ hard), especially if there's a set
I was told KVM had the smallest luo footprint of everything, so perhaps your perspective is different.
In other places luo becomes coupled to the internal datastructures of the kernel, we literally have to preserve lots of kernel memory utterly unchanged without any kind of serialization at all. This inevitably creates restrictions on the evolution of the kernel in general if we can no longer change in-kernel architecture because it would make some data structure incompatible with a kernel 10 years old.
There is also a need for *downgrade*. Meaning the N+1 kernel has to strictly emulate the limitations and capabilities of the N kernel so it can be rolled back. Can you imagine the nightmare of trying to codify the feature progression for every single kernel release forever in some impossible scheme to enable this? Again, no thanks.
So, what upstream can reasonably do, that still provides real value without severely burdening every subsystem, is upgrades within a stable kernel branch only.
If some distro or CSP wants to do more, they can deal with it. They have a far, far simpler problem because they can exactly lock down their version pairs to a very small universe. They don't need a single kernel that can emulate 10 different releases of behaviors concurrently.
Jason
Sean Christopherson
maintainerRe: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
Quoted 18 lines · Jason Gunthorpe
Quoted 2 lines · Sean Christopherson
The rule is thou shalt not break userspace. Whether or not the breakage is the result of an explicit ABI change is irrelevant.
I argued strongly for this simpler approach, I will write here the reasoning I gave.
Firstly, "live update" is not uABI per-say. It is not "userspace breaking", and it is not a "regression". It is an internal kernel mechanism to allow the kernel to self-upgrade. In the industry it is typical any of these hitless update/patch/etc schemes to only work between a few tightly controlled version pairs.
Asking upstream to carry a full matrix of every single version pair ever released is massive over-engineering and cost on upstream maintainers. Refusing to do this is not a uABI breakage, it is not a regression, it is simply a lack of a feature.
Quoted 4 lines
I think upstream should start by agreeing to only support forward going upgrades within a single stable branch. If live update becomes a success then maybe we could do from a stable branch to the immediate next stable branch too. I don't know.
From that perspective, what I am proposing actually goes a step further. I'm saying don't commit to supporting *any* specific versions in upstream. Express the feature requirements in the serialization payloads, and let userspace sort out what kernels are compatible based on their actual usage. The kernel may need to provide additional discovery mechanisms, e.g. so that userspace can probe to see what is supported, but discovery is usually fairly simple to implement and maintain.
Quoted 6 lines
This alone is already much more powerful than live patching.
The balancing issue here is the impact on the kernel subsystems adopting luo. Every time you change anything about the internal function of the subystem you now have to go test a massive version pair matrix to make sure everything works? No thanks.
No, the subsystem just needs to make sure that it serializes its data using the defined ABI (where ABI here means the format of the payload and the meaning of any flags in the header). That should be *easier* for maintainers to handle than trying to support arbitrary versions, because it eliminates subjectivity and having to make judgment calls or remember magic version numbers.
Quoted 15 lines
Luo is very invasive and a huge PITA at the best of times. It doesn't need to be even worse.
Quoted 5 lines
It's probably fine for Google and other large companies that tightly control their kernels and use cases, and have the resources to juggle the resulting complexity, e.g. have kernel engineers on staff to track feature and dependencies, coordinate and plan kernel upgrades, etc.
Every downstream that wants to support luo is going to necessarily severely restrict the version pairs that can work. Like a RH type distro may only support it for 9.1.x -> 9.1.x+1 - they can carry the cost of figuring out how to manage their patching and compatability matrix on their own with the tools upstream provides.
Quoted 8 lines
Quoted 4 lines
engineers on staff to help them thread the needle you describe above. And if supporting live update as a general feature for all users of the kernel isn't being factored into design considerations, then that needs to change, otherwise this is all dead in the water.
General uses can use upstream and do CVE upgrades within a single stable branch. That's good enough, lets start there. We don't need to boil the ocean.
We're in violent agreement on this point, I think we just disagree on how to express compatibility.
Quoted 6 lines
Quoted 3 lines
I also don't see the point. Maintaining a rigid save/restore ABI is annoying, but it's not _hard_ (or at least, not _that_ hard), especially if there's a set
I was told KVM had the smallest luo footprint of everything, so perhaps your perspective is different.
Only because KVM already has a massive ABI surface for save/restore. If you want to convince me that magic version numbers are the right approach, then show me how KVM's existing save/restore support would be made "better" and easier to maintain by throwing away all of KVM's save/restore uAPI and replacing it with a versioning scheme.
Quoted 3 lines
In other places luo becomes coupled to the internal datastructures of the kernel, we literally have to preserve lots of kernel memory utterly unchanged without any kind of serialization at all.
Have to, or choose to? I have a very, very hard time believing that it's infeasible to define a serialization format that is decoupled from kernel internals.
Quoted 3 lines
This inevitably creates restrictions on the evolution of the kernel in general if we can no longer change in-kernel architecture because it would make some data structure incompatible with a kernel 10 years old.
Only if the relevant subsystem defined a poor save/restore ABI in the first place.
Quoted 17 lines
There is also a need for *downgrade*. Meaning the N+1 kernel has to strictly emulate the limitations and capabilities of the N kernel so it can be rolled back. Can you imagine the nightmare of trying to codify the feature progression for every single kernel release forever in some impossible scheme to enable this? Again, no thanks.
So, what upstream can reasonably do, that still provides real value without severely burdening every subsystem, is upgrades within a stable kernel branch only.
If some distro or CSP wants to do more, they can deal with it. They have a far, far simpler problem because they can exactly lock down their version pairs to a very small universe. They don't need a single kernel that can emulate 10 different releases of behaviors concurrently.
Jason
Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
Quoted 7 lines · Sean Christopherson
From that perspective, what I am proposing actually goes a step further. I'm saying don't commit to supporting *any* specific versions in upstream. Express the feature requirements in the serialization payloads, and let userspace sort out what kernels are compatible based on their actual usage. The kernel may need to provide additional discovery mechanisms, e.g. so that userspace can probe to see what is supported, but discovery is usually fairly simple to implement and maintain.
I'm not so fussed about the version number scheme itself. Alot of different schemes have been proposed over the last year and half, and some were very complicated. Nobody came with a more granular proposal that also didn't have alot of complexity attached to it.
FWWI this started out with creating device tree fragments with a full YAML schema, and the same kind of perfect ABI like you are talking about. Yet it did not figure out how to make it discoverable, it was super inefficient and very hard to code for.
It has to be discoverable from the ELF, restrictable on the export side, easy on maintainers, and not complex to implement.
Quoted 5 lines
No, the subsystem just needs to make sure that it serializes its data using the defined ABI (where ABI here means the format of the payload and the meaning of any flags in the header). That should be *easier* for maintainers to handle than trying to support arbitrary versions, because it eliminates subjectivity and having to make judgment calls or remember magic version numbers.
I'm pretty sure I have something like PTSD from ABI definitions adventures on the uAPI side. :( Please don't call it easier!
I really don't want more of those in the kernel process. I think Linus's non-stable-api-nonsense is really a good thing for community health.
With the simplification that upstream supports only one version at once, a simple "id" to represent that ABI was the simplest, easiest on maintainers thing. There is never a fight. Someone has a new idea, great no worries, change the ID. Done.
I guess I should say my perspective is to prioritize not burdening the maintainers.
Quoted 4 lines
Quoted 2 lines
I was told KVM had the smallest luo footprint of everything, so perhaps your perspective is different.
Only because KVM already has a massive ABI surface for save/restore.
Yeah, you are lucky, other subsystems haven't done that. Honestly, was it easy to create?
Quoted 4 lines
If you want to convince me that magic version numbers are the right approach, then show me how KVM's existing save/restore support would be made "better" and easier to maintain by throwing away all of KVM's save/restore uAPI and replacing it with a versioning scheme.
I don't know about KVM, but how do I manage something like serializing an iommu page table?
I'm replacing all the iommu page table code. It behaves differently. It supports different things that old kernels don't understand. iommu page tables have to be under continuous active DMA during kexec, so they cannot be serialized.
Setting the old stuff as V1 and the new suff as V2 is so easy and is unconditionally correct. To my point about supporting only stable branches this achieves it with minimal maintainer effort.
Yes, we could do some comprehensive analysis and try to determine everything that changed and make micro feature bits, and some thing to map the current layout to those bits and then hope all of this is correct.. But that's *a lot* of work, probably will have gaps since it will never be tested. Seriously, why bother? Explain to me why I should spend my time on this and who benifits?
IMHO your KVM example has a much clearer answer to that question. It is UAPI so you have to do it anyhow, and you have a robust open source ecosystem with alot of VMMs that will consume it. Yeah, I'm on board, makes sense.
Quoted 3 lines
Have to, or choose to? I have a very, very hard time believing that it's infeasible to define a serialization format that is decoupled from kernel internals.
Reduced perfectly everything should become grounded in either HW or kernel UAPI definitions. Things are not perfect, the kernel has limitations, it doesn't support every HW feature, stuff leaks in.. Like my page table example, the new stuff supports ARM CONT, the old stuff does not. That is kexec ABI breaking and is only happening because of kernel internal details. In the page table work alone there are probably ~10-20 micro features like that that would have to be identified and delt with. Frankly I'm not confident I could even capture them all.
Also there is the push to make kexec downtime lower. That puts pressure to just retain kernel things exactly as is, and maybe even keep kernel memory as-is. ie a sloppy job serializing to get better performance.
Quoted 1 line
Only if the relevant subsystem defined a poor save/restore ABI in the first place.
I think KVM is the only subsystem that has save/restore :)
Jason
Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
In reply to Pratyush YadavQuoted 15 lines · Pratyush Yadav
Quoted 2 lines · Sean Christopherson
Quoted 1 line · Pratyush Yadav
On Tue, Aug 11 2026, Sean Christopherson wrote:
[...]
Quoted 9 lines
But that's all beside the point. What I'm saying is that SNP and TDX *must* use
guest_memfd, whereasguest_memfdis optional for all other VM types. And so adding LUO support forguest_memfdwithout even sketching out a plan for CoCo VMs feels backwards.I'm not necessarily opposed to gradual
guest_memfdsupport, but there needs to be a clear plan of how all of this is going to fit together. It doesn't need to be perfect, and I'm sure we'll make mistakes along the way, but I want to at least try not to paint ourselves into a corner, especially with respect to the ABI.
Quoted 8 lines
Right. I think landing shared
guest_memfdis the first step. It is simple enough and doesn't need to preserve the more complicated architecture specific things like secure EPT, etc. The CoCo support can build on top of this.There is work underway already for an RFC of TDX live update, but Tarun knows the details better than me so I'll let him fill in the details for the next version of this series.
Right, I am keeping things minimal for this version just addressing the non-cocovm guest_memfd to be preserved. On top of it, We will buidl support for the cocoVM preservation. And Sagi ([email protected]) is already working on it.
Why not implement both in the same series:
1. This series focus on finalizing the design, How guest_memfd will be preserved, So Any user (Not now) but in future will use guest_memfd in non cocoVM case, This alone feature will be used.
2. guest_memfd cocoVM preservation support will be built on top of in-place conversion series and I preferred not to complicate two topics in single series, Instead to develop gradually.
I agree to your answer for have Borader Roadmap. I tried to answer in Cover-letter. Which I am re-stating it here:
This series support guest_memfd preservation for non-COCOVM only. cocoVM Support will be built on top of that. Which includes the following stages.
non cocoVM guest_memfd preservation -> TDX preservation -> in-place convertable private guest_memfd preservation -> in-place convertable private guest_memfd preservation backed by hugetlb.
End Goal is to support guest_memfd preservation on cocoVM backed by hugetlb.
The question remains:
Why we are preserving KVM and only vm_type there.
I will try to answer this on the thread of VM preservation to not loose the context.
Quoted 54 lines
Quoted 42 lines
Quoted 24 lines
Quoted 24 lines
Quoted 9 lines
Quoted 5 lines
E.g. the LUO APIs seem pretty straightforward; I assume the bulk of the complexity is going to be in knowing what to save/restore, and how, which is much more about KVM than it is about liveupdate.
I think it is fine if you want to take these changes through the KVM tree, but I would like live update maintainers to be listed as reviewers at least.
Why not simply add a file pattern match to the LIVE UPDATE entry?
MAINTAINERS +1
-1@@ -15052,8 +15052,8 @@ F: include/linux/liveupdate.h F: include/uapi/linux/liveupdate.h F: kernel/liveupdate/ F: lib/tests/liveupdate.c -F: mm/memfd_luo.c F: tools/testing/selftests/liveupdate/ +N: [^a-z]luoQuoted 4 lines
MAINTAINERS F: include/linux/liveupdate.h · line 15059
Quoted 2 lines
MAINTAINERS +0
-0LLC (802.2) L: [email protected]This would list us as maintainers of
kvm_luo.cand the tree as liveupdate.git,It would list both "LIVE UPDATE" and "KERNEL VIRTUAL MACHINE (KVM)", as KVM would
still cover the files via "F:virt/kvm/*". And my read ofMAINTAINERSis that
N: and K: entries are "secondary" if the files/scope is covered by an F: entry.E.g.
arch/x86/kvm/vmx/tdx.cis covered by "KERNEL VIRTUAL MACHINE FOR X86 (KVM/x86)"
via "F:arch/x86/kvm/*/", and also by "X86 TRUST DOMAIN EXTENSIONS (TDX)" via
"N: tdx" (or maybe "K: \b(tdx)"? I haven't bothered to check which one
triggers). And while I don't think there was ever any formal discussion, AFAIK everyone reads the situation as KVM still being the primary maintainer/tree for that code.Quoted 1 line
both of which is something you're saying you _don't_ want.
Quoted 8 lines
No, what I don't want is a dedicated "KVM LIVE UPDATE" entry, because I think most
people would read "F:virt/kvm/kvm_luo.c" and "F:virt/kvm/guest_memfd_luo.c"
as being more precise than KVM's "F:virt/kvm/*" and thus would read things as
"KVM LIVE UPDATE" being the primary maintainer.In other words, I'm more than ok with LUO being looped in on KVM LUO changes and having the authority to object to problematic changes, but I'm not ok with LUO taking primary ownership of KVM code.
Sure, that sounds good to me.
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
I realize there's more than a bit of nuance in my interpretation of N: and K:, but again my experience with TDX is that so long as the maintainers are aligned on expectations, it's a non-issue in practice.
Yep, as long as we agree among ourselves I don't think the red tape around which entry has higher priority matters much.
Agree!
Quoted 57 lines
Quoted 33 lines
Quoted 23 lines
Quoted 11 lines
Quoted 9 lines
At the same time, I also keep being (pleasantly)
surprised at preservation being relatively simple. For example, the code to preserve a shmem file (via memfd) is roughly 600 lines, a big chunk of which is comments. The code of course has some limitations, but it is good enough for use in production.For one, we care about ABI breakages and versioning.
Which is amusing to me because that implies KVM does not, and I would hazard to
No, it doesn't. What I'm saying is I care about changes to the
_liveupdateABI_. Just like you probably care about changes to KVM ABI but not so much about BPF for example.Quoted 7 lines
guess that KVM has the biggest ABI surface of any subsystem in the kernel by a country mile (though I'm probably wildly underestimating the effective ABI surface of filesystems).
Quoted 2 lines
The serialized state is a part of live update ABI and changes to it should be ACKed by us.
Quoted 7 lines
Quoted 2 lines
Meh, "Don't break userspace" is a universal rule in the kernel, I genuinely don't see why liveupdate needs special treatment.
Ironically enough, you miss my point. I'm not talking about userspace ABI. We all know not to break that. I am talking about live update
_serializationABI_. See the stuff underinclude/linux/kho/abi. This series also adds things there.No, I understand exactly what ABI you're talking about.
Quoted 8 lines
Quoted 3 lines
This is ABI between kernels. It needs to be stable-ish so you can move from one kernel version to another. At the same time, unlike userspace ABI, it can change.
Uh, yeah, so KVM has been managing such immutable ABI for practically its entire existence. KVM's save/restore uAPI has exactly what you're describing: serialization ABI that needs to be backwards and forwards compatible between different kernels in order to support both upgrade and rollback scenarios via live migration.
I think there is a slight difference between KVM's save/resture uAPI and live update's ABI. With KVM's uAPI, you need to maintain strict backwards compatibility because userspace reads what you output. So if you change the layout, userspace will interpret it wrong and might break.
With live update, the ABI never gets to userspace. It is used to talk between kernels directly. So if you do break that, your userspace keeps working fine, you just might not be able to live update to the incompatible kernel.
So with live update, we don't need to keep backwards compatibility in the ABI forever. Of course, it is good to minimize changes, but we have more freedom to change it.
Quoted 7 lines
The current idea is we change the ABI whenever needed. This is because live update is still in development so we are likely to see a lot more changes before we settle down into something that mostly works. At some point, we want to start being more strict between incompatible ABI changes. The current idea is that we would like to keep backwards compatibility across LTS kernels. But none of that is set in stone yet.
True, I also understand Sean's Concern. I will try to explain why bacward compatibility with padding does not solve the problem here.
Kernel V1 -> Kernel V2guest_memfd_luo ABI v1 -> guest_memfd_luo ABI v2
(No Userspace involvement here: userspace only determine if the upcoming kernel V2 can support the V1 preserved files, if return true: liveupdate()
elif: no_liveupdate())
When Kernel V1, preserve the data, it has preserved based on ABI v1. then serializes it to make it transferable to next kernel via fdt.
Now Kernel V2 boots up, and it needs to deserialize the data and restore the file which was preserved before. But this kernel V2 expect guest_memfd preservation ABI V2. These ABI V2 might be because of multiple reasons, for e.g.
1. A new feature in guest_memfd_luo (preserving more flags, or the private/shared state of folios etc)
2. A new feature in guest_memfd itself, guest_memfd kernel APIs (guest_memfd_create(..., new_args) or a new init call for guest_memfd after guest_memfd_create(), or an older field/flog is dropped). So new kernel should be able to retrieve old guest_memfd and populate in new guest_memfd for backward compatibility.
Now, What will happen to these new fields or arguments that are needed, Kernel V2->guest_memfd_luo V2 will take care of setting them to default.
Is setting them to default acceptable? Not always? So that is where the negotiation comes in picture: Kernel V2 should know if it can support retrieving kernel V1 or otherwise userspace must not transition to V2 using liveupdate if V2 does not have compatibility with V1.
Now to support this backward compatibility, the retrieval APIs and deserialization APIs in V2 kernel should be able to provide the deserialization/retrieval for V1 preserved data.
Kernel V1 -> serialize data ---kexec ----> kernel V2 -> deserialization -> retrieve
Now deserialization/retrival is aware of V1/V2 comptatibility then it can set the new fields, parse the preserved data of V1 into V2 (or directly give it to guest_memfd APIs with setting any new expected fields to default.)
Now having a padding fields in kernel guest_memfd_luo ABIs does not help here. As support for deserialization and parsing from V1-> V2 in kernel V2 will determine if kernel V2 is backward compatible with V1. which can also take care of setting the extra needed fields in V2 to default value or remove the unwanted flags/fields not needed in kernel V2.
Now Questions is, if there isa need for rollback to previous kernel (V2 -> V1)?
There are two options,
1. kernel V2 is aware that it is going to get kexec to V1. then serialization function on preservation will serialize the data in guest_memfd_luo ABIs V1 compatible format.
2. kernel V1 guest_memfd_luo ABIs had the padding in their ABIs structures which even if kernel V2 will set those field, will be ignored by kernel V1. This will not require saperate deserialization/preservation mechanism for guest_memfd_luo V1 in kernel V2. This will not work in every case, As there might be different parsers are in guest_memfd_luo V2 and V1. then serialized data preserved by guest_memfd_luo V2 can not be parsed by guest_memfd_luo V1 after kexec.
So,
Kernel V2 _must have_ to support backward compatibility with kernel V1:
1. guest_memfd_luo V1 context aware deserialization/retrieval mechanism
2. guest_memfd_luo V1 context aware serialization/preservation mechanism
Kernel V1/V2 may have:
1. padding (hard to determine, can have heuristically) to ease the job of deserialization/retrieval/serialization/preservation apis. But does not solve backward compatibility alone.
Once we will keep updating the host, There will be a point when we are at V15 but we dont want V1 backward compatiblity anymore, So we drop the deserialization/retrieval/serialization/preservation for V1 from V15.
Would love to hear your thoughts?
Looping Logan ([email protected]) for his insights on this as well.
~Tarun
Quoted 77 lines
That's why I would like to have an explicit ACK from the live update group for ABI changes. I would like to enforce these compatibility windows.
Quoted 20 lines
KVM also has immutable ABI between itself and guest kernels, including implicit "ABI" in the form of not changing guest-visible behavior.
Quoted 7 lines
Today we don't have any rules and let you change things freely as long as you do a version bump. But at a later point, the plan is to add some stability requirements to the ABI so you can actually upgrade the kernel across major versions.
So at least for ABI changes, there should be an explicit ACK from the live update group.
I don't entirely agree. I get where you're coming from, and I 100% agree that the more eyeballs on changes that may affect ABI, the better.
Where I have problems with the above is that it doesn't account for the nuances of save/restore across different kernels. The literal format of the serialized data is the most obvious form of ABI, but it's absolutely possible to change ABI without changing the data format.
Quoted 6 lines
E.g. say there's a flags field in some serialization structure, and flags X and Y are mutually exclusive in current kernels. If a future kernel relaxes that restriction for whatever reason, then the effective ABI has been broken because state created and saved on new kernels can't be restored on old kernels. And this is not a theoretical concern, KVM has run afoul of this a few times (though thankfully very rarely).
That's a bug, and should be fixed.
I am talking here about intentional ABI changes. As I mentioned above we _can_ change the ABI, we just need to be careful about it.
Quoted 9 lines
I don't think it's reasonable to expect LUO maintainers to gain enough expertise in each subsystem to be able to ensure changes are forwards and backwards compatible. The only way I see LUO being successful in the long term is to get subsystem maintainers/contributors to understand *and buy-in* to the LUO model and rules, so that each subsystem can largely be self-sustatining. I.e. setting yourselves up as literal gatekeepers will help prevent blatant breakage, but it's less likely to help guard against more subtle breakage, and in my experience, subtle breakage is by far harder to detect and more painful to deal with.
Outside of enforcing the ABI compatibility windows, I think this makes sense.
In principle at least, though from my experience the maintainer appetite to care about live update varies. The feeling I've got from MM for example is more along the lines of "you break it, you buy it". Which is also a valid model IMO. The person who wrote the live update code for a subsystem is well equipped to understand these nuances.
Quoted 5 lines
Somewhat of a side topic: in my experience, using monotonically increasing version numbers is a horrible way to enumerate features/content. So for me, allowing KVM's LUO ABI to change with a verson bump is probably a non-starter. I.e. whatever gets merged needs to be more future-proof than "we'll deal with it later".
As I wrote above, the compatibility model for live update allows breaking ABI with version bumps. Userspace keeps working fine, it only restricts the set of kernels that you can go to. So the "we'll deal with it later" is not as harmful because our choices here are not permanent.
Also, there is work underway to allow transitions between versions [0].
But still, I am not opposed to a more flexible
guest_memfdABI. I think we can reserve some space for feature flags to let us add things in the future.[0] https://lore.kernel.org/kexec/[email protected]/
Signature · 2 lines
Regards, Pratyush Yadav
Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates
In reply to Sean ChristophersonQuoted 36 lines · Sean Christopherson
Quoted 21 lines · Pratyush Yadav
Quoted 19 lines · Sean Christopherson
Quoted 5 lines · Tarun Sahu
Register a Live Update Orchestrator (LUO) file handler for KVM VM files to serialize and deserialize VM state across kexec live updates.
Currently, Only VM type (e.g. arch.vm_type on x86) is preserved as part of VM preservation.
Why?
Quoted 9 lines
On retrieval,
kvm_luo_retrieve()recreates the KVM VM file viakvm_create_vm_file()and use an atomically incremented ID for the internal fdname, as the final fdname assigned by userspace is not yet known during retrieval. As this fdname is only used in debugfs infra, This will not break any UAPI.This infrastructure establishes the foundation for preserving
guest_memfdinstances across live updates, and can be expanded in the future to preserve additional VM state.Quoted 7 lines
Quoted 4 lines
Uh, why
guest_memfd? As much as I want to pushguest_memfdadoption, it seemsguest_memfdshould be the _last_ thing we support, not the first. As evidenced by the last two decades, it's very doable to have KVM VMs withoutguest_memfd, but it's rather hard to have VMs without vCPUs.You _can_ preserve vCPUs today using KVM_{GET,SET}_REGS, they just won't run in the background during the reboot.
What about x86 CoCo VMs? Which are quite literally _the_ reason
guest_memfdwas created in the first place.Quoted 2 lines
This series can save you from dumping VM memory to disk if it is backed by
guest_memfd.
Quoted 6 lines
Or to word it another way, one _can_ save
guest_memfd, it's just slower.My point is that this series needs to provide a _lot_ more information about the bigger KVM picture.
Yes, I agree. I will try to layout the plan.
End goal of this series is to preserve VM memory which is backed by guest_memfd. Not all VM are backed by guest_memfd. So preservation of KVM (vm_file) is independent of preservation of guest_memfd. But guest_memefd can not be preserved without preserving the KVM (vm_file). So I agree to your suggestion:
kvm/Makefile.kvm +5 -0
@@ -13,3 +13,8 @@ kvm-$(CONFIG_HAVE_KVM_IRQ_ROUTING) += $(KVM)/irqchip.o kvm-$(CONFIG_HAVE_KVM_DIRTY_RING) += $(KVM)/dirty_ring.o kvm-$(CONFIG_HAVE_KVM_PFNCACHE) += $(KVM)/pfncache.o kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd.o + +ifdef CONFIG_LIVEUPDATE +kvm-y += $(KVM)/kvm_luo.o +kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd_luo.o +endif
Guest_memfd can't be created alone without struct kvm (kvm_gmem_create() and KVM_GMEM_CREATE ioctl). This is how guest_memfd has been designed. I don't want to break this design which has been accepted upstream after lot of discussion.
So while LUO preserve guest_memfd's data, It just preserve the PFNs value, few flags that belongs to the guest_memfd.
On restore side in new kernel, These PFNs, flags will be populated to a _newly_ created guest_memfd. that is how preservation and retrieval works. So To create this new guest_memfd, We need struct kvm (vm_file). this vm_file must be the same VM which had this guest_memfd in old kernel, So we preserve the vm_file, get its TOKEN preserve with guest_memfd and during restore, we create the guest_memfd with the same VM (vm_file/struct kvm).
I agree, I did not do good job explaining things in commit message. I will make sure to update them in next revision.
Quoted 6 lines
For those of us that are on the very fringes of live update,
it's practically impossible to review because, to us, it seems very arbitrary.The part that's especially confusing is the saving of the VM type. That comes straight from userspace, so it's super bizarre to automatically save/restore that, but nothing else.
Like, guest_memfd needs struct kvm to create itself. struct kvm (vm_file) needs vm_type to create itself (kvm_create_vm() or KVM_CREATE_VM IOCTL). LUO does not provde functionality to pass any subsystem specific arguments. So vm_type needs to be preserved even though, userspace is aware about it.
So Why do we preserve only vm_type: To keep things simple for this series, As target is guest_memfd.
Currently I dont have discreet plan on what else will
,in future, be needed to be preserved. Which I agree not a absoulute right way to approach. I will layout a rough plan on KVM side preservation.
Having this need for backward compatiblity: I responded here:
https://lore.kernel.org/all/[email protected]/
Quoted 90 lines
Quoted 16 lines
Quoted 16 lines
Review
Quoted 10 lines
(quoted diff) +9
-0+KVM LIVE UPDATE +M: Pasha Tatashin <[email protected]> +M: Mike Rapoport <[email protected]> +M: Pratyush Yadav <[email protected]> +R: Tarun Sahu <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +T: git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.gitNAK on taking changes through a different tree. This is KVM code, period.
In general, I'm skeptical of the dedicated
MAINTAINERSentry. It's extremely difficult to tell since this series is little more than a skeleton (either that or liveupdate is way simpler that I was expecting), but I suspect that maintaining...
Quoted 7 lines
Quoted 3 lines
E.g. the LUO APIs seem pretty straightforward; I assume the bulk of the complexity is going to be in knowing what to save/restore, and how, which is much more about KVM than it is about liveupdate.
I think it is fine if you want to take these changes through the KVM tree, but I would like live update maintainers to be listed as reviewers at least.
Why not simply add a file pattern match to the LIVE UPDATE entry?
MAINTAINERS +1
-1@@ -15052,8 +15052,8 @@ F: include/linux/liveupdate.h F: include/uapi/linux/liveupdate.h F: kernel/liveupdate/ F: lib/tests/liveupdate.c -F: mm/memfd_luo.c F: tools/testing/selftests/liveupdate/ +N: [^a-z]luo LLC (802.2) L: [email protected]Quoted 8 lines
At the same time, I also keep being (pleasantly)
surprised at preservation being relatively simple. For example, the code to preserve a shmem file (via memfd) is roughly 600 lines, a big chunk of which is comments. The code of course has some limitations, but it is good enough for use in production.For one, we care about ABI breakages and versioning.
Which is amusing to me because that implies KVM does not, and I would hazard to guess that KVM has the biggest ABI surface of any subsystem in the kernel by a country mile (though I'm probably wildly underestimating the effective ABI surface of filesystems).
Quoted 2 lines
The serialized state is a part of live update ABI and changes to it should be ACKed by us.
Meh, "Don't break userspace" is a universal rule in the kernel, I genuinely don't see why liveupdate needs special treatment.
Quoted 3 lines
For another, how the file handlers interact with their dependencies can affect the behaviour that VMMs observe. Those changes should also pass by some live update eyes.
Perhaps in the short term, but IMO, that's not a winning strategy in the long term. From my perspective, that like saying the PAGE CACHE maintainers should review every usage of the filemap APIs, because how the APIs are used impacts the page cache and affects userspace-visible behavior. There are myriad analogies like that throughout the kernel.
Yes, liveupdate is new and shiny, but IMO for it to be successful and maintainable, it needs to be treated like any other core infrastructure in the kernel, not a special snowflake whose details are known only by a handful of people. Because I think it's likely liveupdate goes one of two ways: either liveupdate becomes a very niche thing that is used sparingly throughout the kernel, or it becomes a broadly used feature that is supported by many filesystems and subsystems.
If liveupdate is relegated to niche status, then it probably isn't going to see a significant amount of ongoing development, at which point the folks working on liveupdate will naturally migrate to other projects, and maintenance will largely be left to subsystem maintainers.
If liveupdate is broadly used, then having a single group of people maintain every subsystem's usage won't scale, and maintenance will again largely fall on the shoulder of subsystem maintainers. Which is totally fine and working as intended, because that's exactly what subystem maintainers are signing up for by merging support for liveupdate.
PATCH v4 6/11 KVM: guest_memfd: Move internal definitions to internal header
In reply to Tarun SahuExtract 'struct gmem_file', 'struct gmem_inode', and GMEM_I() from virt/kvm/guest_memfd.c into a new internal header virt/kvm/guest_memfd.h. Also split __kvm_gmem_create() to expose a non-static __kvm_gmem_create_file() helper that returns a 'struct file *' instead of an fd.
These internal definitions and helpers allow upcoming guest_memfd Live Update Orchestrator (LUO) preservation code to access guest_memfd internals and reconstruct guest_memfd file instances from preserved state without installing them into a file descriptor table up front.
--- virt/kvm/guest_memfd.c | 68 +++++++++++++++++------------------------- virt/kvm/guest_memfd.h | 39 ++++++++++++++++++++++++ 2 files changed, 67 insertions(+), 40 deletions(-) create mode 100644 virt/kvm/guest_memfd.h
virt/kvm/guest_memfd.c +28 -40
@@ -7,38 +7,12 @@#include <linux/mempolicy.h> #include <linux/pseudo_fs.h> #include <linux/pagemap.h> +#include "guest_memfd.h" #include "kvm_mm.h" static struct vfsmount *kvm_gmem_mnt; -/* -* A guest_memfd instance can be associated multiple VMs, each with its own -* "view" of the underlying physical memory. -* -* The gmem's inode is effectively the raw underlying physical storage, and is -* used to track properties of the physical memory, while each gmem file is -* effectively a single VM's view of that storage, and is used to track assets -* specific to its associated VM, e.g. memslots=>gmem bindings. -*/ -struct gmem_file { -struct kvm *kvm; -struct xarray bindings; -struct list_head entry; -}; - -struct gmem_inode { -struct shared_policy policy; -struct inode vfs_inode; -struct list_head gmem_file_list; - -u64 flags; -}; - -static __always_inline struct gmem_inode *GMEM_I(struct inode *inode) -{ - return container_of(inode, struct gmem_inode, vfs_inode); -}#define kvm_gmem_for_each_file (f, inode) \ list_for_each_entry(f, &GMEM_I(inode)->gmem_file_list, entry)
@@ -557,23 +531,17 @@ bool __weak kvm_arch_supports_gmem_init_shared(struct kvm *kvm)return true; } -static int __kvm_gmem_create (struct kvm *kvm,loff_t size,u64 flags) +struct file *__kvm_gmem_create_file (struct kvm *kvm,loff_t size,u64 flags) {static const char *name ="[kvm-gmem]" ;struct gmem_file *f;struct inode *inode;struct file *file; -int fd, err; - - fd =get_unused_fd_flags (0 ); -if (fd <0 ) -return fd; +int err; f =kzalloc_obj (*f); -if (!f) { - err = -ENOMEM ; - goto err_fd; - } +if (!f) +return ERR_PTR (-ENOMEM );/* __fput() will take care of fops_put(). */ if (!fops_get (&kvm_gmem_fops)) {
@@ -612,8 +580,7 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)xa_init (&f->bindings);list_add (&f->entry, &GMEM_I (inode)->gmem_file_list); -fd_install (fd, file); -return fd; +return file; err_inode:iput (inode);
@@ -621,7 +588,28 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)fops_put (&kvm_gmem_fops); err_gmem:kfree (f); -err_fd: +return ERR_PTR (err); +} + +static int __kvm_gmem_create (struct kvm *kvm,loff_t size,u64 flags) +{ +struct file *file; +int fd, err; + + fd =get_unused_fd_flags (0 ); +if (fd <0 ) +return fd; + + file =__kvm_gmem_create_file (kvm, size, flags); +if (IS_ERR (file)) { + err =PTR_ERR (file); + goto err_put_fd; + } + +fd_install (fd, file); +return fd; + +err_put_fd:put_unused_fd (fd);return err; }
virt/kvm/guest_memfd.h new file +39 -0
@@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_GUEST_MEMFD_H__ +#define __KVM_GUEST_MEMFD_H__ 1 + +#include <linux/kvm_host.h> +#include <linux/fs.h> +#include <linux/mempolicy.h> + +/* +* A guest_memfd instance can be associated multiple VMs, each with its own +* "view" of the underlying physical memory. +* +* The gmem's inode is effectively the raw underlying physical storage, and is +* used to track properties of the physical memory, while each gmem file is +* effectively a single VM's view of that storage, and is used to track assets +* specific to its associated VM, e.g. memslots=>gmem bindings. +*/ +struct gmem_file { +struct kvm *kvm; +struct xarray bindings; +struct list_head entry; +}; + +struct gmem_inode { +struct shared_policy policy; +struct inode vfs_inode; +struct list_head gmem_file_list; + +u64 flags; +}; + +static inline struct gmem_inode *GMEM_I (struct inode *inode) +{ + return container_of(inode, struct gmem_inode, vfs_inode); +} + +struct file *__kvm_gmem_create_file (struct kvm *kvm,loff_t size,u64 flags); + +#endif /* __KVM_GUEST_MEMFD_H__ */
Signature · 1 line
2.55.0.229.g6434b31f56-goog
Re: [PATCH v4 06/11] KVM: guest_memfd: Move internal definitions to internal header
Quoted 6 lines · Tarun Sahu
Extract 'struct
gmem_file', 'structgmem_inode', andGMEM_I()fromvirt/kvm/guest_memfd.cinto a new internal headervirt/kvm/guest_memfd.h. Also split__kvm_gmem_create()to expose a non-static__kvm_gmem_create_file()helper that returns a 'struct file *' instead of an fd.
This is done basically to support a new virt/kvm/guest_memfd_luo.c file.
Would like to know what Sean thinks of this! I'd like to know for future guest_memfd work too.
Quoted 171 lines
These internal definitions and helpers allow upcoming
guest_memfdLive Update Orchestrator (LUO) preservation code to accessguest_memfdinternals and reconstructguest_memfdfile instances from preserved state without installing them into a file descriptor table up front.Signed-off-by Tarun Sahu <[email protected]>--- virt/kvm/guest_memfd.c | 68 +++++++++++++++++------------------------- virt/kvm/guest_memfd.h | 39 ++++++++++++++++++++++++ 2 files changed, 67 insertions(+), 40 deletions(-) create mode 100644 virt/kvm/guest_memfd.hvirt/kvm/guest_memfd.c +28
-40@@ -7,38 +7,12 @@#include <linux/mempolicy.h> #include <linux/pseudo_fs.h> #include <linux/pagemap.h> +#include "guest_memfd.h" #include "kvm_mm.h" static struct vfsmount *kvm_gmem_mnt; -/* -* A guest_memfd instance can be associated multiple VMs, each with its own -* "view" of the underlying physical memory. -* -* The gmem's inode is effectively the raw underlying physical storage, and is -* used to track properties of the physical memory, while each gmem file is -* effectively a single VM's view of that storage, and is used to track assets -* specific to its associated VM, e.g. memslots=>gmem bindings. -*/ -struct gmem_file { -struct kvm *kvm; -struct xarray bindings; -struct list_head entry; -}; - -struct gmem_inode { -struct shared_policy policy; -struct inode vfs_inode; -struct list_head gmem_file_list; - -u64 flags; -}; - -static __always_inline struct gmem_inode *GMEM_I(struct inode *inode) -{ - return container_of(inode, struct gmem_inode, vfs_inode); -}#define kvm_gmem_for_each_file (f, inode) \ list_for_each_entry(f, &GMEM_I(inode)->gmem_file_list, entry)@@ -557,23 +531,17 @@ bool __weak kvm_arch_supports_gmem_init_shared(struct kvm *kvm)return true; } -static int __kvm_gmem_create (struct kvm *kvm,loff_t size,u64 flags) +struct file *__kvm_gmem_create_file (struct kvm *kvm,loff_t size,u64 flags) {static const char *name ="[kvm-gmem]" ;struct gmem_file *f;struct inode *inode;struct file *file; -int fd, err; - - fd =get_unused_fd_flags (0 ); -if (fd <0 ) -return fd; +int err; f =kzalloc_obj (*f); -if (!f) { - err = -ENOMEM ; - goto err_fd; - } +if (!f) +return ERR_PTR (-ENOMEM );/* __fput() will take care of fops_put(). */ if (!fops_get (&kvm_gmem_fops)) {@@ -612,8 +580,7 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)xa_init (&f->bindings);list_add (&f->entry, &GMEM_I (inode)->gmem_file_list); -fd_install (fd, file); -return fd; +return file; err_inode:iput (inode);@@ -621,7 +588,28 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)fops_put (&kvm_gmem_fops); err_gmem:kfree (f); -err_fd: +return ERR_PTR (err); +} + +static int __kvm_gmem_create (struct kvm *kvm,loff_t size,u64 flags) +{ +struct file *file; +int fd, err; + + fd =get_unused_fd_flags (0 ); +if (fd <0 ) +return fd; + + file =__kvm_gmem_create_file (kvm, size, flags); +if (IS_ERR (file)) { + err =PTR_ERR (file); + goto err_put_fd; + } + +fd_install (fd, file); +return fd; + +err_put_fd:put_unused_fd (fd);return err; }virt/kvm/guest_memfd.h new file +39
-1@@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_GUEST_MEMFD_H__ +#define __KVM_GUEST_MEMFD_H__ 1 + +#include <linux/kvm_host.h> +#include <linux/fs.h> +#include <linux/mempolicy.h> + +/* +* A guest_memfd instance can be associated multiple VMs, each with its own +* "view" of the underlying physical memory. +* +* The gmem's inode is effectively the raw underlying physical storage, and is +* used to track properties of the physical memory, while each gmem file is +* effectively a single VM's view of that storage, and is used to track assets +* specific to its associated VM, e.g. memslots=>gmem bindings. +*/ +struct gmem_file { +struct kvm *kvm; +struct xarray bindings; +struct list_head entry; +}; + +struct gmem_inode { +struct shared_policy policy; +struct inode vfs_inode; +struct list_head gmem_file_list; + +u64 flags; +}; + +static inline struct gmem_inode *GMEM_I (struct inode *inode) +{ + return container_of(inode, struct gmem_inode, vfs_inode); +} + +struct file *__kvm_gmem_create_file (struct kvm *kvm,loff_t size,u64 flags); + +#endif /* __KVM_GUEST_MEMFD_H__ */ --2.55.0.229.g6434b31f56-goog
Pratyush Yadav
maintainerRe: [PATCH v4 06/11] KVM: guest_memfd: Move internal definitions to internal header
Quoted 11 lines · Ackerley Tng
Quoted 6 lines · Tarun Sahu
Extract 'struct
gmem_file', 'structgmem_inode', andGMEM_I()fromvirt/kvm/guest_memfd.cinto a new internal headervirt/kvm/guest_memfd.h. Also split__kvm_gmem_create()to expose a non-static__kvm_gmem_create_file()helper that returns a 'struct file *' instead of an fd.This is done basically to support a new
virt/kvm/guest_memfd_luo.cfile.
My $0.02 on this: I think it is useful to split out the live update code into a separate file. It makes for easier reviews, and puts a clear boundary between core and live update. We have done it for memfd/shmem for example (see mm/memfd_luo.c) and for other live update series in flight like PCI or IOMMU.
Quoted 10 lines
Would like to know what Sean thinks of this! I'd like to know for future
guest_memfdwork too.Quoted 6 lines
These internal definitions and helpers allow upcoming
guest_memfdLive Update Orchestrator (LUO) preservation code to accessguest_memfdinternals and reconstructguest_memfdfile instances from preserved state without installing them into a file descriptor table up front.Signed-off-by Tarun Sahu <[email protected]>
[...]
Signature · 2 lines
Regards, Pratyush Yadav
PATCH v4 7/11 KVM: guest_memfd: Add support for freezing mappings
In reply to Tarun SahuIntroduce kvm_gmem_freeze() to freeze a guest_memfd inode's mapping, which prevents fallocate() operations and new page fault allocations during preservation.
Use a global SRCU (kvm_gmem_freeze_srcu) to synchronize freeze state checkers without incurring per-fault locking overhead or risking per-CPU memory exhaustion (as per-CPU structure not counted in cgroup) from per-inode SRCU structures by faulty/compromised VMM.
--- virt/kvm/guest_memfd.c | 117 +++++++++++++++++++++++++++++++++++++---- virt/kvm/guest_memfd.h | 5 ++ 2 files changed, 111 insertions(+), 11 deletions(-)
virt/kvm/guest_memfd.c +106 -11
@@ -7,11 +7,13 @@#include <linux/mempolicy.h> #include <linux/pseudo_fs.h> #include <linux/pagemap.h> +#include <linux/srcu.h> #include "guest_memfd.h" #include "kvm_mm.h" static struct vfsmount *kvm_gmem_mnt; +static struct srcu_struct kvm_gmem_freeze_srcu;#define kvm_gmem_for_each_file (f, inode) \
@@ -96,6 +98,7 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)/* TODO: Support huge pages. */ struct mempolicy *policy;struct folio *folio; +int idx;/* * Fast-path: See if folio is already present in mapping to avoid
@@ -105,12 +108,20 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)if (!IS_ERR (folio))return folio; + idx =srcu_read_lock (&kvm_gmem_freeze_srcu); +if (kvm_gmem_is_frozen (inode)) { +srcu_read_unlock (&kvm_gmem_freeze_srcu, idx); +return ERR_PTR (-EPERM ); + } + policy =mpol_shared_policy_lookup (&GMEM_I (inode)->policy, index); folio =__filemap_get_folio_mpol (inode->i_mapping, index,FGP_LOCK |FGP_CREAT ,mapping_gfp_mask (inode->i_mapping), policy);mpol_cond_put (policy); +srcu_read_unlock (&kvm_gmem_freeze_srcu, idx); +/* * External interfaces like kvm_gmem_get_pfn() support dealing * with hugepages to a degree, but internally, guest_memfd currently
@@ -273,16 +284,30 @@ static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)static long kvm_gmem_fallocate (struct file *file,int mode,loff_t offset,loff_t len) { +struct inode *inode =file_inode (file);int ret; +int idx; -if (!(mode &FALLOC_FL_KEEP_SIZE )) -return -EOPNOTSUPP ; + idx =srcu_read_lock (&kvm_gmem_freeze_srcu); +if (kvm_gmem_is_frozen (inode)) { +srcu_read_unlock (&kvm_gmem_freeze_srcu, idx); +return -EPERM ; + } -if (mode & ~(FALLOC_FL_KEEP_SIZE |FALLOC_FL_PUNCH_HOLE )) -return -EOPNOTSUPP ; +if (!(mode &FALLOC_FL_KEEP_SIZE )) { + ret = -EOPNOTSUPP ; + goto out; + } -if (!PAGE_ALIGNED (offset) || !PAGE_ALIGNED (len)) -return -EINVAL ; +if (mode & ~(FALLOC_FL_KEEP_SIZE |FALLOC_FL_PUNCH_HOLE )) { + ret = -EOPNOTSUPP ; + goto out; + } + +if (!PAGE_ALIGNED (offset) || !PAGE_ALIGNED (len)) { + ret = -EINVAL ; + goto out; + }if (mode &FALLOC_FL_PUNCH_HOLE ) ret =kvm_gmem_punch_hole (file_inode (file), offset, len);
@@ -291,6 +316,9 @@ static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,if (!ret)file_modified (file); + +out: +srcu_read_unlock (&kvm_gmem_freeze_srcu, idx);return ret; }
@@ -948,7 +976,9 @@ static void kvm_gmem_destroy_inode(struct inode *inode)static void kvm_gmem_free_inode (struct inode *inode) { -kmem_cache_free (kvm_gmem_inode_cachep,GMEM_I (inode)); +struct gmem_inode *gi =GMEM_I (inode); + +kmem_cache_free (kvm_gmem_inode_cachep, gi); }static const struct super_operations kvm_gmem_super_operations = {
@@ -1003,12 +1033,21 @@ int kvm_gmem_init(struct module *module)if (!kvm_gmem_inode_cachep)return -ENOMEM ; + ret =init_srcu_struct (&kvm_gmem_freeze_srcu); +if (ret) + goto err_cache; + ret =kvm_gmem_init_mount (); -if (ret) { -kmem_cache_destroy (kvm_gmem_inode_cachep); -return ret; - } +if (ret) + goto err_srcu; +return 0 ; + +err_srcu: +cleanup_srcu_struct (&kvm_gmem_freeze_srcu); +err_cache: +kmem_cache_destroy (kvm_gmem_inode_cachep); +return ret; }void kvm_gmem_exit (void )
@@ -1016,5 +1055,61 @@ void kvm_gmem_exit(void)kern_unmount (kvm_gmem_mnt); kvm_gmem_mnt =NULL ;rcu_barrier (); +cleanup_srcu_struct (&kvm_gmem_freeze_srcu);kmem_cache_destroy (kvm_gmem_inode_cachep); } + +/** +* kvm_gmem_freeze - Freeze or unfreeze a guest_memfd inode mapping. +* @inode: The guest_memfd inode. +* @freeze: True to freeze, false to unfreeze. +* +* This API is used strictly during the live update / preservation transition +* window to prevent host userspace and guest-side faults from making any +* mapping modifications (such as fallocate or page fault allocation) +* to the guest_memfd page cache. +* +* Synchronization Strategy (Sleepable RCU): +* To avoid high-contention VFS locks (like inode_lock or +* filemap_invalidate_lock) on the vCPU page fault hot paths, this subsystem +* implements a lightweight, system-wide Sleepable RCU (SRCU) mechanism +* (`kvm_gmem_freeze_srcu`): +* +* Global vs. Per-Inode SRCU +* ====================== +* A single system-wide global static `srcu_struct` is used instead of a +* per-inode SRCU structure to completely prevent unprivileged users from +* exhausting the host's per-CPU memory allocator. Because +* `init_srcu_struct()` allocates per-CPU memory via `alloc_percpu()`, which +* is not accounted by memory cgroups (memcg), +* a per-inode SRCU structure would allow a tenant to bypass cgroup limits and +* trigger a system-wide Out-of-Memory (OOM) crash simply by spawning a large +* number of guest_memfd file descriptors (bounded only by RLIMIT_NOFILE). +* +* Flag Modification Note: +* Since `GUEST_MEMFD_F_MAPPING_FROZEN` is the ONLY flag in +* `GMEM_I(inode)->flags` that is mutated dynamically at runtime (all other +* flags are creation-time flags which remain strictly read-only), there is +* no possibility of concurrent bit-modification races. Therefore, a standard +* `WRITE_ONCE` is fully safe and does not require complex `cmpxchg` +* synchronization loops. +*/ +void kvm_gmem_freeze (struct inode *inode,bool freeze) +{ +u64 flags =READ_ONCE (GMEM_I (inode)->flags); + +if (freeze) + flags |=GUEST_MEMFD_F_MAPPING_FROZEN ; +else + flags &= ~GUEST_MEMFD_F_MAPPING_FROZEN ; + +WRITE_ONCE (GMEM_I (inode)->flags, flags); + +if (freeze) +synchronize_srcu (&kvm_gmem_freeze_srcu); +} + +bool kvm_gmem_is_frozen (struct inode *inode) +{ +return READ_ONCE (GMEM_I (inode)->flags) &GUEST_MEMFD_F_MAPPING_FROZEN ; +}
virt/kvm/guest_memfd.h +5 -0
@@ -29,11 +29,16 @@ struct gmem_inode {u64 flags; }; +/* Internal kernel-only flags (must not overlap with UAPI flags) */ +#define GUEST_MEMFD_F_MAPPING_FROZEN (1ULL << 63) +static inline struct gmem_inode *GMEM_I (struct inode *inode) { return container_of(inode, struct gmem_inode, vfs_inode); }struct file *__kvm_gmem_create_file (struct kvm *kvm,loff_t size,u64 flags); +void kvm_gmem_freeze (struct inode *inode,bool freeze); +bool kvm_gmem_is_frozen (struct inode *inode);#endif /* __KVM_GUEST_MEMFD_H__ */
Signature · 1 line
2.55.0.229.g6434b31f56-goog
[email protected]
AI reviewRe: [PATCH v4 07/11] KVM: guest_memfd: Add support for freezing mappings
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Holding the global `kvm_gmem_freeze_srcu` read lock during unbounded `fallocate` operations allows…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Holding the global kvm_gmem_freeze_srcu read lock during unbounded fallocate operations allows an unprivileged tenant to block kvm_gmem_freeze() indefinitely for other VMs. --
--- Patch [7]: [PATCH v4 07/11] KVM: guest_memfd: Add support for freezing mappings --- commit de8f141a41c06197ca510e03554977beeeb4fcdf
Author: Tarun Sahu <[email protected]>
KVM: guest_memfd: Add support for freezing mappings
This commit introduces kvm_gmem_freeze() to freeze a guest_memfd inode's mapping to prevent fallocate and page fault allocations during preservation. It uses a global SRCU to synchronize freeze state checkers without incurring per-fault locking overhead or per-CPU memory exhaustion.
Quoted 4 lines
virt/kvm/guest_memfd.c +0
-0
[ ... ]
Quoted 15 lines
virt/kvm/guest_memfd.c +7
-2@@ -273,16 +284,30 @@ static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)static long kvm_gmem_fallocate (struct file *file,int mode,loff_t offset,loff_t len) { +struct inode *inode =file_inode (file);int ret; +int idx; -if (!(mode &FALLOC_FL_KEEP_SIZE )) -return -EOPNOTSUPP ; + idx =srcu_read_lock (&kvm_gmem_freeze_srcu); +if (kvm_gmem_is_frozen (inode)) { +srcu_read_unlock (&kvm_gmem_freeze_srcu, idx); +return -EPERM ; + }
[ ... ]
virt/kvm/guest_memfd.c static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset, · line 323
Quoted 12 lines
virt/kvm/guest_memfd.c +3
-0if (mode &FALLOC_FL_PUNCH_HOLE ) ret =kvm_gmem_punch_hole (file_inode (file), offset, len);else ret =kvm_gmem_allocate (file_inode (file), offset, len);if (!ret)file_modified (file); + +out: +srcu_read_unlock (&kvm_gmem_freeze_srcu, idx);return ret; }
[Severity: High]
Does this code allow an unprivileged tenant to stall the system by holding the global kvm_gmem_freeze_srcu read lock across unbounded fallocate operations?
If a tenant requests a massive hole punch or allocation, kvm_gmem_fallocate() holds the read lock for the entire duration of kvm_gmem_punch_hole() or kvm_gmem_allocate().
If another VM triggers kvm_gmem_freeze(..., true) during this time, it will block in synchronize_srcu(&kvm_gmem_freeze_srcu) until the massive operation completes.
Could this lead to a cross-tenant denial of service where one VM blocks live updates and preservation operations for all other VMs on the same hypervisor?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Re: [PATCH v4 07/11] KVM: guest_memfd: Add support for freezing mappings
In reply to Tarun SahuQuoted 9 lines · Tarun Sahu
Introduce
kvm_gmem_freeze()to freeze aguest_memfdinode's mapping, which preventsfallocate()operations and new page fault allocations during preservation.Use a global SRCU (
kvm_gmem_freeze_srcu) to synchronize freeze state checkers without incurring per-fault locking overhead or risking per-CPU memory exhaustion (as per-CPU structure not counted in cgroup) from per-inode SRCU structures by faulty/compromised VMM.
I wonder if everything to do with freezing should be gated behind CONFIG_LIVEUPDATE_GUEST_MEMFD. What do people think?
Quoted 4 lines
Signed-off-by Tarun Sahu <[email protected]>[...snip...]
Re: [PATCH v4 07/11] KVM: guest_memfd: Add support for freezing mappings
Quoted 6 lines · Ackerley Tng
Quoted 4 lines · Tarun Sahu
Introduce
kvm_gmem_freeze()to freeze aguest_memfdinode's mapping, which preventsfallocate()operations and new page fault allocations during preservation.
Quoted 8 lines
Quoted 5 lines
Use a global SRCU (
kvm_gmem_freeze_srcu) to synchronize freeze state checkers without incurring per-fault locking overhead or risking per-CPU memory exhaustion (as per-CPU structure not counted in cgroup) from per-inode SRCU structures by faulty/compromised VMM.I wonder if everything to do with freezing should be gated behind
CONFIG_LIVEUPDATE_GUEST_MEMFD. What do people think?
Right now, Only liveupdate is the usecase. There was a discussion around the future usecase when IOMMU FD can use few pages from guest_memfd and will have to prevent punch_hole operation. But that will also avoid new faults. So freezing is not the solution here.
A SEAL type solution like in memfd can be explored later. But this is very far in future so Plenty of time to discuss more ideas.
On gating the freeze across CONFIG_LIVEUPDATE_GUEST_MEMFD sounds good
Idea. As in future, there is a plan to remove the freeze at all from
the preservation path. Before LIVEUPDATE, there was no usecase and in future, that usecase will also be gone. So gating it against the config makes sense.
Would love to know what do people think?
~Tarun
Quoted 5 lines
Quoted 5 lines
Signed-off-by Tarun Sahu <[email protected]>[...snip...]
Re: [PATCH v4 07/11] KVM: guest_memfd: Add support for freezing mappings
In reply to Tarun SahuQuoted 28 lines · Tarun Sahu
[...snip...]
virt/kvm/guest_memfd.c +11
-6@@ -273,16 +284,30 @@ static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)static long kvm_gmem_fallocate (struct file *file,int mode,loff_t offset,loff_t len) { +struct inode *inode =file_inode (file);int ret; +int idx; -if (!(mode &FALLOC_FL_KEEP_SIZE )) -return -EOPNOTSUPP ; + idx =srcu_read_lock (&kvm_gmem_freeze_srcu); +if (kvm_gmem_is_frozen (inode)) { +srcu_read_unlock (&kvm_gmem_freeze_srcu, idx); +return -EPERM ; + } -if (mode & ~(FALLOC_FL_KEEP_SIZE |FALLOC_FL_PUNCH_HOLE )) -return -EOPNOTSUPP ; +if (!(mode &FALLOC_FL_KEEP_SIZE )) { + ret = -EOPNOTSUPP ; + goto out; + } -if (!PAGE_ALIGNED (offset) || !PAGE_ALIGNED (len)) -return -EINVAL ;
virt/kvm/guest_memfd.c static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len) · line 311
Quoted 10 lines
virt/kvm/guest_memfd.c +9
-0+if (mode & ~(FALLOC_FL_KEEP_SIZE |FALLOC_FL_PUNCH_HOLE )) { + ret = -EOPNOTSUPP ; + goto out; + } + +if (!PAGE_ALIGNED (offset) || !PAGE_ALIGNED (len)) { + ret = -EINVAL ; + goto out; + }
Continuing from [1], this guard is actually meant for PUNCH_HOLE... I wonder if there could be a deeper refactoring to check when actually punching a hole, basically the equivalent place as kvm_gmem_get_folio() in the flow of fallocating.
Will srcu_read_lock() be taken in a nested way for the fallocate case? It can be taken in a nested way with some intricacies, but maybe if the freeze check can be done just as the folio is about to be removed (as described in the above paragraph), it would avoid this nested locking here.
Quoted 15 lines
virt/kvm/guest_memfd.c +3
-0if (mode &FALLOC_FL_PUNCH_HOLE ) ret =kvm_gmem_punch_hole (file_inode (file), offset, len);@@ -291,6 +316,9 @@ static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,if (!ret)file_modified (file); + +out: +srcu_read_unlock (&kvm_gmem_freeze_srcu, idx);return ret; }
Re: [PATCH v4 07/11] KVM: guest_memfd: Add support for freezing mappings
Quoted 46 lines · Ackerley Tng
Quoted 28 lines · Tarun Sahu
[...snip...]
virt/kvm/guest_memfd.c +11
-6@@ -273,16 +284,30 @@ static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)static long kvm_gmem_fallocate (struct file *file,int mode,loff_t offset,loff_t len) { +struct inode *inode =file_inode (file);int ret; +int idx; -if (!(mode &FALLOC_FL_KEEP_SIZE )) -return -EOPNOTSUPP ; + idx =srcu_read_lock (&kvm_gmem_freeze_srcu); +if (kvm_gmem_is_frozen (inode)) { +srcu_read_unlock (&kvm_gmem_freeze_srcu, idx); +return -EPERM ; + } -if (mode & ~(FALLOC_FL_KEEP_SIZE |FALLOC_FL_PUNCH_HOLE )) -return -EOPNOTSUPP ; +if (!(mode &FALLOC_FL_KEEP_SIZE )) { + ret = -EOPNOTSUPP ; + goto out; + } -if (!PAGE_ALIGNED (offset) || !PAGE_ALIGNED (len)) -return -EINVAL ;virt/kvm/guest_memfd.c static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len) · line 311
Quoted 10 lines
virt/kvm/guest_memfd.c +9
-0+if (mode & ~(FALLOC_FL_KEEP_SIZE |FALLOC_FL_PUNCH_HOLE )) { + ret = -EOPNOTSUPP ; + goto out; + } + +if (!PAGE_ALIGNED (offset) || !PAGE_ALIGNED (len)) { + ret = -EINVAL ; + goto out; + }Continuing from [1], this guard is actually meant for
PUNCH_HOLE... I wonder if there could be a deeper refactoring to check when actually punching a hole, basically the equivalent place askvm_gmem_get_folio()in the flow of fallocating.
Quoted 7 lines
Will
srcu_read_lock()be taken in a nested way for the fallocate case? It can be taken in a nested way with some intricacies, but maybe if the freeze check can be done just as the folio is about to be removed (as described in the above paragraph), it would avoid this nested locking here.
Instead of avoiding on remove folio, We can just add the freeze check across the PUNCH_HOLE, So there will be no nested lock anymore. Nested lock is just with fallocate populate -> kvm_gmem_get_pfn. So this nested ness will be removed with this update.
~Tarun
Quoted 16 lines
Quoted 16 lines
virt/kvm/guest_memfd.c +3
-0if (mode &FALLOC_FL_PUNCH_HOLE ) ret =kvm_gmem_punch_hole (file_inode (file), offset, len);@@ -291,6 +316,9 @@ static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,if (!ret)file_modified (file); + +out: +srcu_read_unlock (&kvm_gmem_freeze_srcu, idx);return ret; }
Sean Christopherson
maintainerRe: [PATCH v4 07/11] KVM: guest_memfd: Add support for freezing mappings
In reply to Tarun SahuQuoted 13 lines · Tarun Sahu
+/**
+ *kvm_gmem_freeze- Freeze or unfreeze aguest_memfdinode mapping.
+ * @inode: Theguest_memfdinode.
+ * @freeze: True to freeze, false to unfreeze.
+ *
+ * This API is used strictly during the live update / preservation transition
+ * window to prevent host userspace and guest-side faults from making any
+ * mapping modifications (such as fallocate or page fault allocation)
+ * to theguest_memfdpage cache.
+ *
+ * Synchronization Strategy (Sleepable RCU):
+ * To avoid high-contention VFS locks (likeinode_lockor
+ *filemap_invalidate_lock) on the vCPU page fault hot paths, this subsystem
Quoted 8 lines
+ * implements a lightweight, system-wide Sleepable RCU (SRCU) mechanism
+ * (kvm_gmem_freeze_srcu):
+ *
+ * Global vs. Per-Inode SRCU
+ * ======================
+ * A single system-wide global staticsrcu_structis used instead of a
+ * per-inode SRCU structure to completely prevent unprivileged users from
+ * exhausting the host's per-CPU memory allocator.
This argument doesn't hold up given that each kvm structure has two SRCU structs.
Re: [PATCH v4 07/11] KVM: guest_memfd: Add support for freezing mappings
Quoted 23 lines · Sean Christopherson
Quoted 13 lines · Tarun Sahu
+/**
+ *kvm_gmem_freeze- Freeze or unfreeze aguest_memfdinode mapping.
+ * @inode: Theguest_memfdinode.
+ * @freeze: True to freeze, false to unfreeze.
+ *
+ * This API is used strictly during the live update / preservation transition
+ * window to prevent host userspace and guest-side faults from making any
+ * mapping modifications (such as fallocate or page fault allocation)
+ * to theguest_memfdpage cache.
+ *
+ * Synchronization Strategy (Sleepable RCU):
+ * To avoid high-contention VFS locks (likeinode_lockor
+ *filemap_invalidate_lock) on the vCPU page fault hot paths, this subsystemQuoted 8 lines
+ * implements a lightweight, system-wide Sleepable RCU (SRCU) mechanism
+ * (kvm_gmem_freeze_srcu):
+ *
+ * Global vs. Per-Inode SRCU
+ * ======================
+ * A single system-wide global staticsrcu_structis used instead of a
+ * per-inode SRCU structure to completely prevent unprivileged users from
+ * exhausting the host's per-CPU memory allocator.
Quoted 1 line
This argument doesn't hold up given that each kvm structure has two SRCU structs.
Ohh okay. I was very skeptic that per-cpu structure are not counted in cgroups and liveupdate is once in a while usecase So avoided making it per-cpu. But otherwise I am fine with going with per-cpu. Will take care of it in next revision.
Thanks
~Tarun
PATCH v4 8/11 KVM: guest_memfd: Add support for preservation via LUO
In reply to Tarun SahuRegister a Live Update Orchestrator (LUO) file handler for guest_memfd files to preserve guest memory across kexec reboots.
Freeze the guest_memfd inode via kvm_gmem_freeze() during preservation to prevent concurrent fallocate() calls or page fault allocations. Walk the inode's page cache folios to serialize their metadata and physical PFNs for kexec handover.
On retrieval, reconstruct the guest_memfd file with __kvm_gmem_create_file(), re-insert the preserved folios into the page cache via filemap_add_folio(), and associate the file back to its preserved KVM VM via the VM token.
Currently, this supports fully shared guest_memfd instances backed by PAGE_SIZE pages. Also update MAINTAINERS for guest_memfd LUO preservation.
--- MAINTAINERS | 1 + include/linux/kho/abi/kvm.h | 79 +++++- virt/kvm/Makefile.kvm | 2 +- virt/kvm/guest_memfd_luo.c | 515 ++++++++++++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 7 + virt/kvm/kvm_mm.h | 4 + 6 files changed, 601 insertions(+), 7 deletions(-) create mode 100644 virt/kvm/guest_memfd_luo.c
MAINTAINERS +1 -0
@@ -14548,6 +14548,7 @@ L: [email protected] L: [email protected] S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git +F: virt/kvm/guest_memfd_luo.c F: virt/kvm/kvm_luo.c KVM PARAVIRT (KVM/paravirt)
include/linux/kho/abi/kvm.h +73 -6
97 lines not on this page · raw message
virt/kvm/Makefile.kvm +1 -1
@@ -13,4 +13,4 @@ kvm-$(CONFIG_HAVE_KVM_IRQ_ROUTING) += $(KVM)/irqchip.o kvm-$(CONFIG_HAVE_KVM_DIRTY_RING) += $(KVM)/dirty_ring.o kvm-$(CONFIG_HAVE_KVM_PFNCACHE) += $(KVM)/pfncache.o kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd.o -kvm-$(CONFIG_LIVEUPDATE_GUEST_MEMFD) += $(KVM)/kvm_luo.o +kvm-$(CONFIG_LIVEUPDATE_GUEST_MEMFD) += $(KVM)/guest_memfd_luo.o $(KVM)/kvm_luo.o
virt/kvm/guest_memfd_luo.c new file +515 -0
515 lines not on this page · raw message
virt/kvm/kvm_main.c +7 -0
@@ -6581,6 +6581,10 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module)if (r) goto err_luo; + r =kvm_gmem_luo_init (); +if (r) + goto err_gmem_luo; +/* * Registration _must_ be the very last thing done, as this exposes * /dev/kvm to userspace, i.e. all infrastructure must be setup!
@@ -6594,6 +6598,8 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module)return 0 ; err_register: +kvm_gmem_luo_exit (); +err_gmem_luo:kvm_luo_exit (); err_luo:kvm_uninit_virtualization ();
@@ -6625,6 +6631,7 @@ void kvm_exit(void)*/ misc_deregister (&kvm_dev); +kvm_gmem_luo_exit ();kvm_luo_exit ();kvm_uninit_virtualization ();
virt/kvm/kvm_mm.h +4 -0
@@ -103,9 +103,13 @@ static inline void kvm_gmem_unbind(struct kvm_memory_slot *slot)#ifdef CONFIG_LIVEUPDATE_GUEST_MEMFD int kvm_luo_init (void );void kvm_luo_exit (void ); +int kvm_gmem_luo_init (void ); +void kvm_gmem_luo_exit (void );#else static inline int kvm_luo_init (void ) {return 0 ; }static inline void kvm_luo_exit (void ) {} +static inline int kvm_gmem_luo_init (void ) {return 0 ; } +static inline void kvm_gmem_luo_exit (void ) {}#endif /* CONFIG_LIVEUPDATE_GUEST_MEMFD */ #endif /* __KVM_MM_H__ */
Signature · 1 line
2.55.0.229.g6434b31f56-goog
[email protected]
AI reviewRe: [PATCH v4 08/11] KVM: guest_memfd: Add support for preservation via LUO
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] `kvm_gmem_luo_finish` incorrectly performs cleanup on successful retrieval, resulting in…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] kvm_gmem_luo_finish incorrectly performs cleanup on successful retrieval, resulting in use-after-free and double-frees of the serialization state and premature freeing of page cache folios.
- [High] Missing kho_is_enabled() check before invoking KHO serialization APIs, risking a NULL pointer dereference.
--
--- Patch [8]: [PATCH v4 08/11] KVM: guest_memfd: Add support for preservation via LUO --- commit 8d81cd3b20bca211fdf1860f352a2bbcf1096ec9
Author: Tarun Sahu <[email protected]>
KVM: guest_memfd: Add support for preservation via LUO
This commit registers a Live Update Orchestrator file handler for guest_memfd files to preserve guest memory across kexec reboots. It freezes the guest_memfd inode during preservation, walks the page cache to serialize metadata, and reconstructs the file and its folios on retrieval.
Quoted 6 lines
virt/kvm/guest_memfd_luo.c new file +0
-0@@ -0,0 +1,515 @@
[ ... ]
Quoted 5 lines
virt/kvm/guest_memfd_luo.c new file +5
-0+static int kvm_gmem_luo_preserve (struct liveupdate_file_op_args *args) +{ +struct guest_memfd_luo_folio_ser *folios_ser =NULL ; +u64 count =0 , gmem_flags, abi_flags =0 ; +struct guest_memfd_luo_ser *ser;
[ ... ]
virt/kvm/guest_memfd_luo.c line 185
Quoted 3 lines
virt/kvm/guest_memfd_luo.c new file +3
-0+ end_index = size >>PAGE_SHIFT ; + + ser =kho_alloc_preserve (sizeof (*ser));
[Severity: High]
Does this path need to verify if the KHO subsystem is actually enabled?
If the kernel is booted with KHO disabled, calling kho_alloc_preserve() and other KHO APIs unconditionally can cause a NULL pointer dereference.
Could we check kho_is_enabled() either here or in kvm_gmem_luo_init() before registering the handler?
[ ... ]
Quoted 5 lines
virt/kvm/guest_memfd_luo.c new file +5
-0+static void kvm_gmem_luo_finish (struct liveupdate_file_op_args *args) +{ +struct guest_memfd_luo_ser *ser; +struct guest_memfd_luo_folio_ser *folios_ser; +
virt/kvm/guest_memfd_luo.c line 471
Quoted 10 lines
virt/kvm/guest_memfd_luo.c new file +10
-0+/* Nothing to be done here, if retrieve_status was successful or errored, +* Cleanup is taken care of in retrieval call. +*/ +if (args->retrieve_status) +return ; + +if (!args->serialized_data) +return ; + + ser =phys_to_virt (args->serialized_data);
[Severity: High]
Is the check on args->retrieve_status inverted here?
When kvm_gmem_luo_retrieve() succeeds, it returns 0 and frees the ser pointer:
kvm_gmem_luo_retrieve()
args->file = file;
kho_restore_free(ser);
return 0;Because retrieve_status is 0 on success, this finish function will fall through the if statement instead of returning early. It will then access the freed ser pointer and mistakenly discard all the successfully restored folios.
Would checking if (!args->retrieve_status) avoid this use-after-free regression?
Quoted 11 lines
virt/kvm/guest_memfd_luo.c new file +11
-0+ +if (ser->nr_folios) { + folios_ser =kho_restore_vmalloc (&ser->folios); +if (folios_ser) { +kvm_gmem_luo_discard_folios (folios_ser, ser->nr_folios,0 ); +vfree (folios_ser); + } + } + +kho_restore_free (ser); +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Re: [PATCH v4 08/11] KVM: guest_memfd: Add support for preservation via LUO
In reply to Tarun SahuQuoted 12 lines · Tarun Sahu
[...snip...]
virt/kvm/guest_memfd_luo.c new file +9
-0+static bool kvm_gmem_luo_can_preserve (struct liveupdate_file_handler *handler,struct file *file) +{ +struct inode *inode =file_inode (file); +struct gmem_file *gmem_file; +struct kvm *kvm; + +if (inode->i_sb->s_magic !=GUEST_MEMFD_MAGIC ) +return false; +
virt/kvm/guest_memfd_luo.c line 124
Quoted 10 lines
virt/kvm/guest_memfd_luo.c new file +10
-0+ gmem_file = file->private_data; +if (!gmem_file) +return false; + +/* +* Only Fully-shared guest_memfd preservation is supported +*/ +if (!(GMEM_I (inode)->flags &GUEST_MEMFD_FLAG_INIT_SHARED )) +return false; +
The above check can probably be removed. I think the core thing about this series is "we can preserve any memory that doesn't have any encryption state that goes along with it", and so the check for kvm_arch_has_private_mem() below would be sufficient to gate that.
virt/kvm/guest_memfd_luo.c line 133
Quoted 9 lines
virt/kvm/guest_memfd_luo.c new file +9
-0+/* +* It makes sure that no memory can be converted to private +* even if it was initially fully shared (in-place conversions are +* prevented). +*/ + kvm = gmem_file->kvm; +if (kvm_arch_has_private_mem (kvm)) +return false; +
For now, has_private_mem == has some encryption state that we can't preserve yet.
This also gates off KVM_X86_SW_PROTECTED_VM, but I guess that is intended, since when work is done for preserving encryption state, we can then test using KVM_X86_SW_PROTECTED_VM, and only then, remove this check?
Quoted 9 lines
virt/kvm/guest_memfd_luo.c new file +6
-0+if (mapping_large_folio_support (inode->i_mapping)) +return false; + +return true; +} +
Re: [PATCH v4 08/11] KVM: guest_memfd: Add support for preservation via LUO
Quoted 21 lines · Ackerley Tng
Quoted 19 lines · Tarun Sahu
[...snip...]
virt/kvm/guest_memfd_luo.c new file +16
-0+static bool kvm_gmem_luo_can_preserve (struct liveupdate_file_handler *handler,struct file *file) +{ +struct inode *inode =file_inode (file); +struct gmem_file *gmem_file; +struct kvm *kvm; + +if (inode->i_sb->s_magic !=GUEST_MEMFD_MAGIC ) +return false; + + gmem_file = file->private_data; +if (!gmem_file) +return false; + +/* +* Only Fully-shared guest_memfd preservation is supported +*/
Quoted 8 lines
virt/kvm/guest_memfd_luo.c line 124
Quoted 3 lines
virt/kvm/guest_memfd_luo.c new file +3
-0+if (!(GMEM_I (inode)->flags &GUEST_MEMFD_FLAG_INIT_SHARED )) +return false; +The above check can probably be removed. I think the core thing about this series is "we can preserve any memory that doesn't have any encryption state that goes along with it", and so the check for
kvm_arch_has_private_mem()below would be sufficient to gate that.
Sounds Good. Will remove it in next version. I will rephrase the
Line: "Support only fully shared guest_memfd" => "Support only guest_memfd that does not support private pages and also can not be converted to private"
Quoted 14 lines
virt/kvm/guest_memfd_luo.c line 133
Quoted 10 lines
virt/kvm/guest_memfd_luo.c new file +9
-0+/* +* It makes sure that no memory can be converted to private +* even if it was initially fully shared (in-place conversions are +* prevented). +*/ + kvm = gmem_file->kvm; +if (kvm_arch_has_private_mem (kvm)) +return false; +For now,
has_private_mem== has some encryption state that we can't preserve yet.
Quoted 4 lines
This also gates off
KVM_X86_SW_PROTECTED_VM, but I guess that is intended, since when work is done for preserving encryption state, we can then test usingKVM_X86_SW_PROTECTED_VM, and only then, remove this check?
Yes, It will be enabled when private memory preservation will be supported.
Quoted 10 lines
Quoted 10 lines
virt/kvm/guest_memfd_luo.c new file +6
-0+if (mapping_large_folio_support (inode->i_mapping)) +return false; + +return true; +} +
PATCH v4 9/11 docs: liveupdate: Add documentation for VM and guest_memfd preservation
In reply to Tarun SahuAdd Documentation/liveupdate/vmm.rst detailing VMM requirements, scope, ordering constraints, and kernel-doc inclusions for KVM VM and guest_memfd preservation via Live Update Orchestrator (LUO).
Also update Documentation/core-api/liveupdate.rst and MAINTAINERS.
--- Documentation/core-api/liveupdate.rst | 1 + Documentation/liveupdate/vmm.rst | 107 ++++++++++++++++++++++++++ MAINTAINERS | 1 + 3 files changed, 109 insertions(+) create mode 100644 Documentation/liveupdate/vmm.rst
Documentation/core-api/liveupdate.rst +1 -0
@@ -34,6 +34,7 @@ The following types of file descriptors can be preserved :maxdepth: 1 ../mm/memfd_preservation + ../liveupdate/vmm Public API ==========
Documentation/liveupdate/vmm.rst new file +107 -0
107 lines not on this page · raw message
MAINTAINERS +1 -0
@@ -14548,6 +14548,7 @@ L: [email protected] L: [email protected] S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git +F: Documentation/liveupdate/vmm.rst F: virt/kvm/guest_memfd_luo.c F: virt/kvm/kvm_luo.c
Signature · 1 line
2.55.0.229.g6434b31f56-goog
[email protected]
AI reviewRe: [PATCH v4 09/11] docs: liveupdate: Add documentation for VM and guest_memfd preservation
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Mismatched kernel-doc section titles cause Sphinx build warnings and missing documentation. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Mismatched kernel-doc section titles cause Sphinx build warnings and missing documentation. --
--- Patch [9]: [PATCH v4 09/11] docs: liveupdate: Add documentation for VM and guest_memfd preservation --- commit a930d2f58a9edc06bdcfd10fb383e97d7fe3996c
Author: Tarun Sahu <[email protected]>
docs: liveupdate: Add documentation for VM and guest_memfd preservation
Add Documentation/liveupdate/vmm.rst detailing VMM requirements, scope, ordering constraints, and kernel-doc inclusions for KVM VM and guest_memfd preservation via Live Update Orchestrator (LUO).
Quoted 7 lines
Documentation/liveupdate/vmm.rst new file +1
-0@@ -0,0 +1,107 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later
Documentation/liveupdate/vmm.rst line 11
Quoted 10 lines
Documentation/liveupdate/vmm.rst new file +10
-0+ +============================= +VM & Guest_Memfd Preservation +============================= + +.. kernel-doc:: virt/kvm/kvm_luo.c + :doc: KVM VM Preservation via LUO + +.. kernel-doc:: virt/kvm/guest_memfd_luo.c + :doc: Guest_Memfd Preservation via LUO
[Severity: Low]
Does this section title exactly match the DOC: title in virt/kvm/guest_memfd_luo.c?
If the source file uses "Guestmemfd Preservation via LUO" without the underscore, this directive will fail to find the documentation and generate a Sphinx build warning.
[ ... ]
Documentation/liveupdate/vmm.rst line 98
Quoted 5 lines
Documentation/liveupdate/vmm.rst new file +5
-0+VM & Guest_Memfd Preservation ABI +================================= + +.. kernel-doc:: include/linux/kho/abi/kvm.h + :doc: DOC: guest_memfd Live Update ABI
[Severity: Low]
Does this section title match the actual title in include/linux/kho/abi/kvm.h?
If the source file uses "KVM and guest_memfd Live Update ABI", this mismatch will cause empty sections in the generated documentation.
Also, should the literal "DOC: " prefix be omitted here? The directive typically only expects the text that follows the prefix.
Quoted 5 lines
Documentation/liveupdate/vmm.rst new file +5
-0+ +.. kernel-doc:: include/linux/kho/abi/kvm.h + :internal: + +See Also
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
Re: [PATCH v4 09/11] docs: liveupdate: Add documentation for VM and guest_memfd preservation
Quoted 36 lines · [email protected] writes:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Mismatched kernel-doc section titles cause Sphinx build warnings and missing documentation. ----- Patch [9]: [PATCH v4 09/11] docs: liveupdate: Add documentation for VM and
guest_memfdpreservation --- commita930d2f58a9edc06bdcfd10fb383e97d7fe3996c
Author: Tarun Sahu <[email protected]>docs: liveupdate: Add documentation for VM and
guest_memfdpreservationAdd
Documentation/liveupdate/vmm.rstdetailing VMM requirements, scope, ordering constraints, and kernel-doc inclusions for KVM VM andguest_memfdpreservation via Live Update Orchestrator (LUO).Quoted 7 lines
Documentation/liveupdate/vmm.rst new file +1
-0@@ -0,0 +1,107 @@ +.. SPDX-License-Identifier: GPL-2.0-or-laterDocumentation/liveupdate/vmm.rst line 11
Quoted 10 lines
Documentation/liveupdate/vmm.rst new file +10
-0+ +============================= +VM & Guest_Memfd Preservation +============================= + +.. kernel-doc:: virt/kvm/kvm_luo.c + :doc: KVM VM Preservation via LUO + +.. kernel-doc:: virt/kvm/guest_memfd_luo.c + :doc: Guest_Memfd Preservation via LUO[Severity: Low]
Does this section title exactly match the DOC: title invirt/kvm/guest_memfd_luo.c?
Quoted 3 lines
If the source file uses "Guestmemfd Preservation via LUO" without the underscore, this directive will fail to find the documentation and generate a Sphinx build warning.
I will correct it. Good find!
Quoted 26 lines
[ ... ]
Documentation/liveupdate/vmm.rst line 98
Quoted 5 lines
Documentation/liveupdate/vmm.rst new file +5
-0+VM & Guest_Memfd Preservation ABI +================================= + +.. kernel-doc:: include/linux/kho/abi/kvm.h + :doc: DOC: guest_memfd Live Update ABI[Severity: Low]
Does this section title match the actual title ininclude/linux/kho/abi/kvm.h?If the source file uses "KVM and
guest_memfdLive Update ABI", this mismatch will cause empty sections in the generated documentation.Also, should the literal "DOC: " prefix be omitted here? The directive typically only expects the text that follows the prefix.
Quoted 5 lines
Documentation/liveupdate/vmm.rst new file +5
-0+ +.. kernel-doc:: include/linux/kho/abi/kvm.h + :internal: + +See AlsoSignature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
PATCH v4 10/11 KVM: selftests: Split ____vm_create() and add vm_create_from_fd()
In reply to Tarun SahuRefactor ____vm_create() in the KVM selftest library to extract its initialization steps into separate helpers vm_init_fields() and vm_init_memory_properties(). Also introduce vm_create_from_fd() to instantiate a struct kvm_vm from an existing VM file descriptor.
This allows test setups (such as guest_memfd preservation tests) to reconstruct and initialize a struct kvm_vm when restoring preserved VMs across live updates. No functional change is introduced for existing tests.
--- .../testing/selftests/kvm/include/kvm_util.h | 3 ++ tools/testing/selftests/kvm/lib/kvm_util.c | 49 ++++++++++++++++--- 2 files changed, 44 insertions(+), 8 deletions(-)
tools/testing/selftests/kvm/include/kvm_util.h +3 -0
@@ -471,6 +471,9 @@ const char *vm_guest_mode_string(u32 i);void kvm_vm_free (struct kvm_vm *vmp);void kvm_vm_restart (struct kvm_vm *vmp); +void vm_init_fields (struct kvm_vm *vm,struct vm_shape shape); +void vm_init_memory_properties (struct kvm_vm *vm); +struct kvm_vm *vm_create_from_fd (int vm_fd,struct vm_shape shape);void kvm_vm_release (struct kvm_vm *vmp);void kvm_vm_elf_load (struct kvm_vm *vm,const char *filename);int kvm_memfd_alloc (size_t size,bool hugepages);
tools/testing/selftests/kvm/lib/kvm_util.c +41 -8
73 lines not on this page · raw message
Signature · 1 line
2.55.0.229.g6434b31f56-goog
PATCH v4 11/11 KVM: selftests: Add guest_memfd_preservation_test
In reply to Tarun SahuAdd guest_memfd_preservation_test to verify that KVM VM state and guest memory backed by guest_memfd are preserved across a kexec reboot via Live Update Orchestrator (LUO).
This test includes multiple tests
- (A) Test to preserve non-guest_memfd, non-memfd which must fail (/dev/null)
- (B) Test to preserve guest_memfd with no INIT_SHARED flag which must fail
- (C) Test to allocate the guest_memfd memory after preservation because preservation makes the guest_memfd frozen. Which must fail as well.
- (D) Test to verify guest_memfd with INIT_SHARED flag preservation across
kexec: Below is the details of stages in this test.
In Stage 1 (pre-kexec), create a VM with a guest_memfd memory slot, run guest code to write a deterministic pattern into guest_memfd, and preserve both the VM and guest_memfd file descriptors via LUO. This stage also tests A, B and C. A, B and C do not require stage2 so they will finish in stage 1 only.
In Stage 2 (post-kexec), retrieve the preserved VM and guest_memfd file descriptors, reconstruct the VM using vm_create_from_fd(), re-map the retrieved guest_memfd, and execute guest code to verify data integrity against the original pattern.
Example usage:
$ ./guest_memfd_preservation_test --s 1 $ <kexec> $ ./guest_memfd_preservation_test --s 2
--- MAINTAINERS | 1 + tools/testing/selftests/kvm/Makefile.kvm | 6 +- .../kvm/guest_memfd_preservation_test.c | 357 ++++++++++++++++++ 3 files changed, 363 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/kvm/guest_memfd_preservation_test.c
MAINTAINERS +1 -0
@@ -14549,6 +14549,7 @@ L: [email protected] S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git F: Documentation/liveupdate/vmm.rst +F: tools/testing/selftests/kvm/guest_memfd_preservation_test.c F: virt/kvm/guest_memfd_luo.c F: virt/kvm/kvm_luo.c
tools/testing/selftests/kvm/Makefile.kvm +5 -1
@@ -165,6 +165,8 @@ TEST_GEN_PROGS_x86 += pre_fault_memory_test # Compiled outputs used by test targets TEST_GEN_PROGS_EXTENDED_x86 += x86/nx_huge_pages_test +# Manual test that forks a persistent background daemon; skip auto CI run +TEST_GEN_PROGS_EXTENDED_x86 += guest_memfd_preservation_test TEST_GEN_PROGS_arm64 = $(TEST_GEN_PROGS_COMMON) TEST_GEN_PROGS_arm64 += arm64/aarch32_id_regs
@@ -259,6 +261,7 @@ OVERRIDE_TARGETS = 1 # which causes the environment variable to override the makefile). include ../lib.mk include ../cgroup/lib/libcgroup.mk +include ../liveupdate/lib/libliveupdate.mk INSTALL_HDR_PATH = $(top_srcdir)/usr LINUX_HDR_PATH = $(INSTALL_HDR_PATH)/include/
@@ -313,7 +316,8 @@ LIBKVM_S := $(filter %.S,$(LIBKVM)) LIBKVM_C_OBJ := $(patsubst %.c, $(OUTPUT)/%.o, $(LIBKVM_C)) LIBKVM_S_OBJ := $(patsubst %.S, $(OUTPUT)/%.o, $(LIBKVM_S)) LIBKVM_STRING_OBJ := $(patsubst %.c, $(OUTPUT)/%.o, $(LIBKVM_STRING)) -LIBKVM_OBJS = $(LIBKVM_C_OBJ) $(LIBKVM_S_OBJ) $(LIBKVM_STRING_OBJ) $(LIBCGROUP_O) +LIBKVM_OBJS = $(LIBKVM_C_OBJ) $(LIBKVM_S_OBJ) $(LIBKVM_STRING_OBJ) \ + $(LIBCGROUP_O) $(LIBLIVEUPDATE_O) SPLIT_TEST_GEN_PROGS := $(patsubst %, $(OUTPUT)/%, $(SPLIT_TESTS)) SPLIT_TEST_GEN_OBJ := $(patsubst %, $(OUTPUT)/$(ARCH)/%.o, $(SPLIT_TESTS))
tools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +357 -0
357 lines not on this page · raw message
Signature · 1 line
2.55.0.229.g6434b31f56-goog
Re: [PATCH v4 11/11] KVM: selftests: Add guest_memfd_preservation_test
Quoted 4 lines · Tarun Sahu
Add
guest_memfd_preservation_testto verify that KVM VM state and guest memory backed byguest_memfdare preserved across a kexec reboot via Live Update Orchestrator (LUO).
Quoted 8 lines
This test includes multiple tests
- (A) Test to preserve non-guest_memfd, non-memfd which must fail (/dev/null)
- (B) Test to preserveguest_memfdwith noINIT_SHAREDflag which must fail
- (C) Test to allocate theguest_memfdmemory after preservation because preservation makes theguest_memfdfrozen. Which must fail as well.
- (D) Test to verifyguest_memfdwithINIT_SHAREDflag preservation across
kexec: Below is the details of stages in this test.
I think we should split this into multiple patches, where each patch introduces 1 test.
Quoted 9 lines
[...snip...]
+++
b/tools/testing/selftests/kvm/guest_memfd_preservation_test.ctools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +4
-0@@ -0,0 +1,357 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (C) 2026, Google LLC. +*
tools/testing/selftests/kvm/guest_memfd_preservation_test.c line 14
Quoted 10 lines
tools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +10
-0+* Author: Tarun Sahu <[email protected]> +* +* Test for VM and guest_memfd preservation across kexec (Live Update) via LUO. +* +* NOTE: This is a MANUAL test and is excluded from automated CI/testing +* frameworks because Stage 1 daemonizes into the background to pin resources +* and requires a human operator to manually trigger kexec before Stage 2 +* is executed. Running Stage 1 automatically would leak the background daemon +* and cause CI runners to falsely interpret it as a passed test. +*
Usually there aren't many printfs in tests, but since this is a MANUAL test, maybe that's ok?
Would like to know what Sean thinks of this, and printfs in general.
Quoted 3 lines
[...snip...]
Re: [PATCH v4 11/11] KVM: selftests: Add guest_memfd_preservation_test
Quoted 15 lines · Ackerley Tng
Quoted 4 lines · Tarun Sahu
Add
guest_memfd_preservation_testto verify that KVM VM state and guest memory backed byguest_memfdare preserved across a kexec reboot via Live Update Orchestrator (LUO).Quoted 8 lines
This test includes multiple tests
- (A) Test to preserve non-guest_memfd, non-memfd which must fail (/dev/null)
- (B) Test to preserveguest_memfdwith noINIT_SHAREDflag which must fail
- (C) Test to allocate theguest_memfdmemory after preservation because preservation makes theguest_memfdfrozen. Which must fail as well.
- (D) Test to verifyguest_memfdwithINIT_SHAREDflag preservation across
kexec: Below is the details of stages in this test.
Quoted 2 lines
I think we should split this into multiple patches, where each patch introduces 1 test.
Got it. I will split them. I thought that this test already spawn a VM and other test can just reuse the same VM.
Quoted 29 lines
Quoted 10 lines
[...snip...]
+++
b/tools/testing/selftests/kvm/guest_memfd_preservation_test.ctools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +4
-0@@ -0,0 +1,357 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (C) 2026, Google LLC. +* tools/testing/selftests/kvm/guest_memfd_preservation_test.c line 14
Quoted 10 lines
tools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +10
-0+* Author: Tarun Sahu <[email protected]> +* +* Test for VM and guest_memfd preservation across kexec (Live Update) via LUO. +* +* NOTE: This is a MANUAL test and is excluded from automated CI/testing +* frameworks because Stage 1 daemonizes into the background to pin resources +* and requires a human operator to manually trigger kexec before Stage 2 +* is executed. Running Stage 1 automatically would leak the background daemon +* and cause CI runners to falsely interpret it as a passed test. +* Usually there aren't many printfs in tests, but since this is a MANUAL test, maybe that's ok?
Would like to know what Sean thinks of this, and printfs in general.
Quoted 3 lines
[...snip...]
Re: [PATCH v4 11/11] KVM: selftests: Add guest_memfd_preservation_test
Quoted 17 lines · Tarun Sahu
Quoted 15 lines · Ackerley Tng
Quoted 12 lines · Tarun Sahu
Add
guest_memfd_preservation_testto verify that KVM VM state and guest memory backed byguest_memfdare preserved across a kexec reboot via Live Update Orchestrator (LUO).This test includes multiple tests
- (A) Test to preserve non-guest_memfd, non-memfd which must fail (/dev/null)
- (B) Test to preserveguest_memfdwith noINIT_SHAREDflag which must fail
- (C) Test to allocate theguest_memfdmemory after preservation because preservation makes theguest_memfdfrozen. Which must fail as well.
- (D) Test to verifyguest_memfdwithINIT_SHAREDflag preservation across
kexec: Below is the details of stages in this test.
Quoted 5 lines
Quoted 2 lines
I think we should split this into multiple patches, where each patch introduces 1 test.
Got it. I will split them. I thought that this test already spawn a VM and other test can just reuse the same VM.
From upstream hypervisor liveupdate meeting, I realized you meant to be same file but saperate commits. Sound good to me. I will take care of that.
Quoted 30 lines
Quoted 30 lines
Quoted 11 lines
[...snip...]
+++
b/tools/testing/selftests/kvm/guest_memfd_preservation_test.ctools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +4
-0@@ -0,0 +1,357 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (C) 2026, Google LLC. +* tools/testing/selftests/kvm/guest_memfd_preservation_test.c line 14
Quoted 10 lines
tools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +10
-0+* Author: Tarun Sahu <[email protected]> +* +* Test for VM and guest_memfd preservation across kexec (Live Update) via LUO. +* +* NOTE: This is a MANUAL test and is excluded from automated CI/testing +* frameworks because Stage 1 daemonizes into the background to pin resources +* and requires a human operator to manually trigger kexec before Stage 2 +* is executed. Running Stage 1 automatically would leak the background daemon +* and cause CI runners to falsely interpret it as a passed test. +* Usually there aren't many printfs in tests, but since this is a MANUAL test, maybe that's ok?
Would like to know what Sean thinks of this, and printfs in general.
Quoted 3 lines
[...snip...]
Pratyush Yadav
maintainerRe: [PATCH v4 11/11] KVM: selftests: Add guest_memfd_preservation_test
In reply to Ackerley TngQuoted 13 lines · Ackerley Tng
Quoted 11 lines · Tarun Sahu
Add
guest_memfd_preservation_testto verify that KVM VM state and guest memory backed byguest_memfdare preserved across a kexec reboot via Live Update Orchestrator (LUO).This test includes multiple tests
- (A) Test to preserve non-guest_memfd, non-memfd which must fail (/dev/null)
- (B) Test to preserveguest_memfdwith noINIT_SHAREDflag which must fail
- (C) Test to allocate theguest_memfdmemory after preservation because preservation makes theguest_memfdfrozen. Which must fail as well.
- (D) Test to verifyguest_memfdwithINIT_SHAREDflag preservation across
kexec: Below is the details of stages in this test.
[...]
Quoted 8 lines
Quoted 8 lines
+++
b/tools/testing/selftests/kvm/guest_memfd_preservation_test.ctools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +6
-0@@ -0,0 +1,357 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (C) 2026, Google LLC. +* +* Author: Tarun Sahu <[email protected]> +*
Quoted 8 lines
tools/testing/selftests/kvm/guest_memfd_preservation_test.c line 14
Quoted 8 lines
tools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +8
-0+* Test for VM and guest_memfd preservation across kexec (Live Update) via LUO. +* +* NOTE: This is a MANUAL test and is excluded from automated CI/testing +* frameworks because Stage 1 daemonizes into the background to pin resources +* and requires a human operator to manually trigger kexec before Stage 2 +* is executed. Running Stage 1 automatically would leak the background daemon +* and cause CI runners to falsely interpret it as a passed test. +*
I think this is fine as a first step but we really should automate these.
We have some infrastructure to automate other live update tests across both stages already by launching a VM under qemu and making the test process as the init task. See tools/testing/selftests/kho/vmtest.sh. For KVM tests you'd need the runner to be able to run nested VMs.
The script right now is pretty bare bones, but I really think we should spend time improving it to the point where we can automate all our tests using it.
Quoted 5 lines
Usually there aren't many printfs in tests, but since this is a MANUAL test, maybe that's ok?
Would like to know what Sean thinks of this, and printfs in general.
Signature · 2 lines
Regards, Pratyush Yadav
Hello,
As per the discussion on the series, I will try to layout the future plan of guest_memfd preservation and VM preservation via LUO. Though guest_memfd itself is in development phase, Deducing the exact plan and ABI beforehand is quite impossible. So I will propose tentative plan as per my current understanding.
Introduction:
Guest_memfd was developed only for CocoVM use-case, But now its target is to unify guest memory for cocoVM and non-cocoVM. Though not all user will be onboarded on using guest_memfd with non-CocoVM, I see current development in guest_memfd for e.g multiple backends will provide its usefulness for non-CocoVM (Maybe get rid of hugetlbfs - traditional PITA).
This preservation series and future development targets the preservation for both cocoVM and non-cocoVM.
Tentative Plan (Overview):
I plan on supporting the guest_memfd preservation on incremental steps. This series builds the foundation for it and only support fully shared guest_memfd backed by PAGE_SIZE pages. Currently, For this base series, Only potential user would be non-CocoVM. But due to performance reasons, at this stage it can not be used (as it is backed by PAGE_SIZE).
Incremental Steps:
A. This base foundation series (Fully shared backed by PAGE_SIZE pages) also require VM preservation [1]. but currently preserving single field only (VM_TYPE).
This sets the foundation for VM preservation via LUO as well. Present KVM uAPIs can support preservation for non-CocoVM backed by non-guest_memfd memory. But they are not sufficient to preserve the cocoVM [2].
B. Next Step, Private guest_memfd preservation on top of in-place conversion series (merged in KVM x86 cocoVM tree) again backed by PAGE_SIZE pages. This will also include VM preservation, for example the data structures that are TDX protected can not be preserved with current KVM uAPIs [2].
C. Parallel to B, guest_memfd preservation for non-CocoVM backed by hugetlb. This work will be on top of guest_memfd support for hugetlb backend [3][4]. Currently, How this support will be developed is in upstream discussion. But The work on preservation will pick it as soon as hugetlb support is integrated.
D. On Top of C, Extends guest_memfd preservation for hugetlb backend for cocoVM (Integrating private guest_memfd support)
E. Preservation for any other backend that will be supported in future.
VM Preservation:
Guest_memfd preservation can not be supported alone, it needs VM preservation as well [1][2]. Even though there are no current user, But there are users who want guest_memfd preservation for non-cocoVM and cocoVM both. And that opens the door for VM preservation as well because present KVM uAPIs are not sufficient (for cocoVM case [2]). This will unify preservation mechanism for VM (and its memory) in linux.
Presently, for non-CocoVM, KVM provides uAPIs to preserve the state of the VM for e.g. vCPU + device state through the (KVM_GET/SET_REGS, SREGS2, XSAVE, MSRS, LAPIC, VCPU_EVENTS, MP_STATE, NESTED_STATE, CLOCK, ...))
Now, for cocoVM, we need to preserve the TD state of the VM and global TDX module state. For which preservation of the following will be needed (Not everything included, it is as much I could think at this time, Also I am not expert on TDX, I will try)
**Global State (Will use LUO FLB objects)**:
1. TDMRs + PAMT locations
2. In-use HKID bitmap
**Per-TD/VM (part of per-VM preservation, It can open door to unify the place of preservation for other fields as well. But again open for discussion)**
3. VM_TYPE (already preserved)
4. TDR Page
5. TDCS Page
6. HKID
7. S-EPT non-leaf pages
8. S-EPT leaf maps
9. per VM memory attributes (unrelated to TDX)
**per-vCPUs (preserved via vCPU fd)**
10. vCPU ID
11. TDVPR page
12. TDCX page
etc.
Note: Also, why not extends
the support for full preservation of vCPUs ignoring existing KVM uAPIs to unify everything at one place: Open to discussion and can be discussed thoroughly once RFC for TDX preservation will be sent.
Guest_memfd Preservation:
1. per gmem memory attributes
2. folios preservation (size/number of pages per folio, index etc)
3. State of hugetlb preservation (again LUO FLB object will be needed to preserve system shared pool information)
4. SPLIT information for hugetlb for shared pages
5. backing type of guest_memfd
6. gmem related flags
7. mem policy
etc.
The above plan is currently based on my current understanding which can be changed as guest_memfd evolve and new findings come to the light while during the work/review of RFC of individual features.
[1]: https://lore.kernel.org/all/[email protected]/#:~:text=Guest_memfd%20can%27t%20be,KVM%20side%20preservation.
[2]
KVM uAPI fetch data to userspace and restore it back in new kernel. for
TDX case, that is not possible becuase of 2 major reasons:
A. all control data structures are protected and encrypted,
Host cannot access them if they do not belong to host.
While LUO just have to handover the PFN to new kernel,
So LUO does not access the underlying data.
B. TDX module can not be freshly initialized, it will lose
the state of the protected VMs and their relation data.
So preserving using KVM uAPIs and reinitializing it and
populating it via userspace is not an option. It has to be
done from kernel. and kexec handover/LUO can handle it very
well by updating TDX initializing sequence in kernel (This
is one of the Idea currently in my mind, topic is open for
exploration).
[3]: https://lore.kernel.org/all/[email protected]/
[4]: https://lore.kernel.org/all/[email protected]/
I wanted to send this before LPC so that we can pick up this for discussion on Tues in either KVM MC (13:00 KVM and Live Update) Slot [5] or Live Update (16:10 Guest_Memfd Preservation For Live Update) Slot [6] or for general discussion on liveupdate ABI (15:50 Live Update Compatibility) Slot [7].
[5]: https://lpc.events/event/20/contributions/2632/
[6]: https://lpc.events/event/20/contributions/2610/
[7]: https://lpc.events/event/20/contributions/2613/
[5] is before [6] & [7], So, We can also have any other slots as per [8] on wed.
[8]: https://lore.kernel.org/all/[email protected]/#r
~Tarun
Quoted 202 lines · Tarun Sahu
Hello,
This is v4 of
guest_memfdpreservation during liveupdate. As per the discussion from bi-weekly guest-memfd meeting, This series will be go to linux via kvm tree. And Planning to get it merged in upcoming merge window. (7.2-rc1 for release in 7.3)The changes are rebased on:
kvm/next + liveupdate/next (merge) + [4] + [5] Where,
[4]: luo: APIs to retrieve file internally from session [5]: selftests: liveupdate sefltests library
Here is the github repo:
https://github.com/tar-unix/linux/tree/gmem-prePatches [4] and [5] are prequisite of this series and they are rebased/merged on liveupdate/next. kvm/next is behind few patches from liveupdate/next, hence direct cherry-pick of [4] and [5] is not possible. On Linux 7.2 they will be aligned. If we succeed in merging it before I can request liveupdate maintainer to provide the tag.
Steps to test:
1. Compile Kernel withCONFIG_LIVEUPDATE_GUEST_MEMFD=y
2. boot kernel with command line: kho=on liveupdate=on
3. run the following kselftest$ .selftests/kvm/guest_memfd_preservation_test --s 1 $ <kexec> --reuse-cmdline $ .selftests/kvm/guest_memfd_preservation_test --s 2NOTE: Assert the following:
$ ls /dev/liveupdate $ ls /dev/kvm $ dmesg | grep liveupdate # (should have kvm_vm_luo && # guest_memfd_luo handler registered)V4 <- V3
1. Split the refactoring patch into multiple patches [2, 3, 4]
2. Updated commits msg as per the kvm documentation and suggestions
3. RemovedKHOSER_PTRand using raw pointer, as series [3] not yet decided to go in the next cycle
4. Implemented more tests inguest_memfd_preservation_testallocation during frozengmem_inode, non-guest_memfd preservation
5. Moved thevm_create_from_fdtokvm_util.cfrom theguest_memfd_preservation_test.c
5. Minor updates as per the suggestionsV3 <- RFC V2 [2]
1. Finalize the design
2. resolve sashiko reported bugs
3. Use ofKHOSER_PTRinstead of rawserialized_dataas per [3]RFC V2 [2] <- RFC V1 [1]
1. Removedmem_attr_arrayas it is not needed for fully-shared
2. Removed pre-faulted condition
3. Addedvm_typepreservation for ARM64.
4. Removedliveupdate_get_file_incomingapi patch as it is sent separately [4] by Samiullah.**GUEST MEMFD PRESERVATION**
Below is the cover letter of the series which explains the design.SCOPE:
1. Fully SharedGuest_memfd
2.Guest_memfdbacked byPAGE_SIZEpagesAny VM whose memory is backed by such
guest_memfdand satisfy the above conditions can be preserved across liveupdate.To Add support for liveupdate of any subsystem, we need to implement the following calls: (e.g: subsystem:
guest_memfd)
guest_memfd_luo_can_preserve: used by luo subsystem to filter the
guest_memfdfilesguest_memfd_luo_preserve: triggered from userspace by PRESERVE ioctl
Ifcan_preservesays the file is preservable byguest_memfd_luo_preserveguest_memfd_luo_retrieve: triggered from userspace by RETRIEVE ioctl,
recognised by the passed token from userspace. Used in new kernel after kexec.guest_memfd_luo_unpreserve: triggered from userspace when a session is
closed, which inherently calls unpreserve of all preserved files.guest_memfd_luo_freeze: triggered just before kexec automatically in
kernel. no userspace involvement.guest_memfd_luo_finish: triggered by userspace when all files are
retrieved and want to finish the session retrieval.*PART A: KVM VM preservation*
Introdcued thekvm_luo.cto preserve thevm_fileduring liveupdate. Currently it only preservevm_type. This can be expanded to preserve more details in future as need arises for example data related to cocoVM (SEPT etc.).int kvm_fd = open("/dev/kvm", O_RDWR); int vm_fd = ioctl(kvm_fd, KVM_CREATE_VM, 0); preserve_arg = { .size = sizeof(preserve_arg), .fd = vm_fd, .token = VM_TOKEN, }; ioctl(session_fd, LIVEUPDATE_SESSION_PRESERVE_FD, &preserve_arg);This
VM_TOKENis used byguest_memfdpreservation.guest_memfdpreservation save this token (fetched internally by luo apis [4]) to associates theguest_memfdwith itsvm_file. On retrieval,guest_memfdretrieves thevm_fileusing this token and use its struct kvm to retrieve itself.PART B:
Guest_memfdpreservation
Introduceguest_memfd_luo.cto preserve theguest_memfdfile during liveupdate. Currently it only preserveguest_memfdfiles that have:1.
INIT_SHAREDflag ensures intially all the memory is shared
2. !kvm_arch_has_private_mem() which ensures no shared pages can be converted to private after [6] lands.Preservation first freezes the
gmem_inodeintroduced in PATCH [07/11] and preserve each allocated folios fromguest_memfdpage_cacheusing KHO.
luo_freeze(not thegmem_inodefreeze PATCH[07/11]) call update thevm_tokenwhich is preserved across kexec.Retrieval retrieves the
vm_fileusingvm_tokenand create theguest_memfdwithvm_file->kvmand populate the preserved folios toguest_memfdpage_cache.Added the Documentation for VMM on
guest_memfdpreservation guidelines To know more about, PATCH[10/11] can be referred which has the more detailed documentation.Future Plan:
1. Make preservation immune to new allocation and fallocate calls so that we can remove freeze call
2. In future, When IOMMU will useguest_memfdand can allocate page tables for DMA in that area, Later it will be a problem if other user ofguest_memfddo punch hole before preservation, that will make the pages to be lost after preservation. This will be taken care of when IOMMU will add support forguest_memfd.
3.guest_memfdpreservation for private pages + in-place conversion.
4. In future, whenguest_memfdwill have hugetlb page support, add support its preservation.[1]: https://lore.kernel.org/all/[email protected]/
[2]: https://lore.kernel.org/all/[email protected]/https://lore.kernel.org/all/c054ba0fb2639932bbe354420d3f4f84cce84905.1780676742.git.tarunsahu@google.com/ [3]: https://lore.kernel.org/all/[email protected]/ [4]: https://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git/commit/?h=next&id=f4d2e4f0d12d88155cd1128ed3294f7827bf4c5d [5]: https://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git/commit/?h=next&id=2f3f53a3735a7e67bbe9e580cb49372bf9261967 https://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git/commit/?h=next&id=df1a0d068d481b7d175c5af3970043206409a062[6]: https://lore.kernel.org/all/[email protected]/
Tarun Sahu (11):
liveupdate: AddLIVEUPDATE_GUEST_MEMFDconfig option
KVM: Introducekvm_create_vm_file()helper
KVM: Exportkvm_uevent_notify_vm_create()
KVM: Track weak reference tovm_filein struct kvm
KVM: LUO: Support VM preservation across live updates
KVM:guest_memfd: Move internal definitions to internal header
KVM:guest_memfd: Add support for freezing mappings
KVM:guest_memfd: Add support for preservation via LUO
docs: liveupdate: Add documentation for VM andguest_memfd
preservation
KVM: selftests: Split____vm_create()and addvm_create_from_fd()
KVM: selftests: Addguest_memfd_preservation_testDocumentation/core-api/liveupdate.rst | 1 + Documentation/liveupdate/vmm.rst | 107 ++++ MAINTAINERS | 14 + include/linux/kho/abi/kvm.h | 106 ++++ include/linux/kvm_host.h | 14 + kernel/liveupdate/Kconfig | 15 + tools/testing/selftests/kvm/Makefile.kvm | 6 +- .../kvm/guest_memfd_preservation_test.c | 357 ++++++++++++ .../testing/selftests/kvm/include/kvm_util.h | 3 + tools/testing/selftests/kvm/lib/kvm_util.c | 49 +- virt/kvm/Makefile.kvm | 1 + virt/kvm/guest_memfd.c | 185 +++++-- virt/kvm/guest_memfd.h | 44 ++ virt/kvm/guest_memfd_luo.c | 515 ++++++++++++++++++ virt/kvm/kvm_luo.c | 195 +++++++ virt/kvm/kvm_main.c | 94 +++- virt/kvm/kvm_mm.h | 15 + 17 files changed, 1640 insertions(+), 81 deletions(-) create mode 100644 Documentation/liveupdate/vmm.rst create mode 100644 include/linux/kho/abi/kvm.h create mode 100644 tools/testing/selftests/kvm/guest_memfd_preservation_test.c create mode 100644 virt/kvm/guest_memfd.h create mode 100644 virt/kvm/guest_memfd_luo.c create mode 100644 virt/kvm/kvm_luo.cbase-commit:
a204badd8432f93b7e862e7dac6db0fe3d65f370--
2.55.0.229.g6434b31f56-goog