Skip to contentSkip to conversation

liveupdate: kvm: Guest_memfd preservation

PATCH NAK Tarun Sahu 6 participants 66 messages Jul 28 – Sep 29

Subsystems (8)
lore
Share & tools
lore

Tarun Sahu

author
#0 lore raw

v4 cover liveupdate: kvm: Guest_memfd preservation

Hello,

This is v4 of guest_memfd preservation during liveupdate. As per the discussion from bi-weekly guest-memfd meeting, This series will be go to linux via kvm tree. And Planning to get it merged in upcoming merge window. (7.2-rc1 for release in 7.3)

The changes are rebased on:
kvm/next + liveupdate/next (merge) + [4] + [5] Where,
[4]: luo: APIs to retrieve file internally from session [5]: selftests: liveupdate sefltests library
Here is the github repo:
https://github.com/tar-unix/linux/tree/gmem-pre

Patches [4] and [5] are prequisite of this series and they are rebased/merged on liveupdate/next. kvm/next is behind few patches from liveupdate/next, hence direct cherry-pick of [4] and [5] is not possible. On Linux 7.2 they will be aligned. If we succeed in merging it before I can request liveupdate maintainer to provide the tag.

Steps to test:
1. Compile Kernel with CONFIG_LIVEUPDATE_GUEST_MEMFD=y
2. boot kernel with command line: kho=on liveupdate=on
3. run the following kselftest

	$ .selftests/kvm/guest_memfd_preservation_test --s 1
	$ <kexec> --reuse-cmdline
	$ .selftests/kvm/guest_memfd_preservation_test --s 2

NOTE: Assert the following:

	$ ls /dev/liveupdate
	$ ls /dev/kvm
	$ dmesg | grep liveupdate # (should have kvm_vm_luo &&
		# guest_memfd_luo handler registered)

V4 <- V3
1. Split the refactoring patch into multiple patches [2, 3, 4]
2. Updated commits msg as per the kvm documentation and suggestions
3. Removed KHOSER_PTR and using raw pointer, as series [3] not yet decided to go in the next cycle
4. Implemented more tests in guest_memfd_preservation_test allocation during frozen gmem_inode, non-guest_memfd preservation
5. Moved the vm_create_from_fd to kvm_util.c from the guest_memfd_preservation_test.c
5. Minor updates as per the suggestions

V3 <- RFC V2 [2]
1. Finalize the design
2. resolve sashiko reported bugs
3. Use of KHOSER_PTR instead of raw serialized_data as per [3]

RFC V2 [2] <- RFC V1 [1]
1. Removed mem_attr_array as it is not needed for fully-shared
2. Removed pre-faulted condition
3. Added vm_type preservation for ARM64.
4. Removed liveupdate_get_file_incoming api patch as it is sent separately [4] by Samiullah.

**GUEST MEMFD PRESERVATION**
Below is the cover letter of the series which explains the design.

SCOPE:
1. Fully Shared Guest_memfd
2. Guest_memfd backed by PAGE_SIZE pages

Any VM whose memory is backed by such guest_memfd and satisfy the above conditions can be preserved across liveupdate.

To Add support for liveupdate of any subsystem, we need to implement the following calls: (e.g: subsystem: guest_memfd)

guest_memfd_luo_can_preserve: used by luo subsystem to filter the
guest_memfd files
guest_memfd_luo_preserve: triggered from userspace by PRESERVE ioctl
If can_preserve says the file is preservable by guest_memfd_luo_preserve
guest_memfd_luo_retrieve: triggered from userspace by RETRIEVE ioctl,
recognised by the passed token from userspace. Used in new kernel after kexec.
guest_memfd_luo_unpreserve: triggered from userspace when a session is
closed, which inherently calls unpreserve of all preserved files.
guest_memfd_luo_freeze: triggered just before kexec automatically in
kernel. no userspace involvement.
guest_memfd_luo_finish: triggered by userspace when all files are
retrieved and want to finish the session retrieval.

*PART A: KVM VM preservation*
Introdcued the kvm_luo.c to preserve the vm_file during liveupdate. Currently it only preserve vm_type. This can be expanded to preserve more details in future as need arises for example data related to cocoVM (SEPT etc.).

	int kvm_fd = open("/dev/kvm", O_RDWR);
	int vm_fd = ioctl(kvm_fd, KVM_CREATE_VM, 0);
	preserve_arg = {
    	.size = sizeof(preserve_arg),
	    .fd = vm_fd,
	    .token = VM_TOKEN,
	};
	ioctl(session_fd, LIVEUPDATE_SESSION_PRESERVE_FD, &preserve_arg);

This VM_TOKEN is used by guest_memfd preservation. guest_memfd preservation save this token (fetched internally by luo apis [4]) to associates the guest_memfd with its vm_file. On retrieval, guest_memfd retrieves the vm_file using this token and use its struct kvm to retrieve itself.

PART B: Guest_memfd preservation
Introduce guest_memfd_luo.c to preserve the guest_memfd file during liveupdate. Currently it only preserve guest_memfd files that have:

1. INIT_SHARED flag ensures intially all the memory is shared
2. !kvm_arch_has_private_mem() which ensures no shared pages can be converted to private after [6] lands.

Preservation first freezes the gmem_inode introduced in PATCH [07/11] and preserve each allocated folios from guest_memfd page_cache using KHO.

luo_freeze (not the gmem_inode freeze PATCH[07/11]) call update the vm_token which is preserved across kexec.

Retrieval retrieves the vm_file using vm_token and create the guest_memfd with vm_file->kvm and populate the preserved folios to guest_memfd page_cache.

Added the Documentation for VMM on guest_memfd preservation guidelines To know more about, PATCH[10/11] can be referred which has the more detailed documentation.

Future Plan:
1. Make preservation immune to new allocation and fallocate calls so that we can remove freeze call
2. In future, When IOMMU will use guest_memfd and can allocate page tables for DMA in that area, Later it will be a problem if other user of guest_memfd do punch hole before preservation, that will make the pages to be lost after preservation. This will be taken care of when IOMMU will add support for guest_memfd.
3. guest_memfd preservation for private pages + in-place conversion.
4. In future, when guest_memfd will have hugetlb page support, add support its preservation.

[1]: https://lore.kernel.org/all/[email protected]/
[2]: https://lore.kernel.org/all/[email protected]/

     https://lore.kernel.org/all/c054ba0fb2639932bbe354420d3f4f84cce84905.1780676742.git.tarunsahu@google.com/
[3]: https://lore.kernel.org/all/[email protected]/
[4]: https://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git/commit/?h=next&id=f4d2e4f0d12d88155cd1128ed3294f7827bf4c5d
[5]: https://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git/commit/?h=next&id=2f3f53a3735a7e67bbe9e580cb49372bf9261967
     https://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git/commit/?h=next&id=df1a0d068d481b7d175c5af3970043206409a062

[6]: https://lore.kernel.org/all/[email protected]/

Tarun Sahu (11):
liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option
KVM: Introduce kvm_create_vm_file() helper
KVM: Export kvm_uevent_notify_vm_create()
KVM: Track weak reference to vm_file in struct kvm
KVM: LUO: Support VM preservation across live updates
KVM: guest_memfd: Move internal definitions to internal header
KVM: guest_memfd: Add support for freezing mappings
KVM: guest_memfd: Add support for preservation via LUO
docs: liveupdate: Add documentation for VM and guest_memfd
preservation
KVM: selftests: Split ____vm_create() and add vm_create_from_fd()
KVM: selftests: Add guest_memfd_preservation_test

 Documentation/core-api/liveupdate.rst         |   1 +
 Documentation/liveupdate/vmm.rst              | 107 ++++
 MAINTAINERS                                   |  14 +
 include/linux/kho/abi/kvm.h                   | 106 ++++
 include/linux/kvm_host.h                      |  14 +
 kernel/liveupdate/Kconfig                     |  15 +
 tools/testing/selftests/kvm/Makefile.kvm      |   6 +-
 .../kvm/guest_memfd_preservation_test.c       | 357 ++++++++++++
 .../testing/selftests/kvm/include/kvm_util.h  |   3 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |  49 +-
 virt/kvm/Makefile.kvm                         |   1 +
 virt/kvm/guest_memfd.c                        | 185 +++++--
 virt/kvm/guest_memfd.h                        |  44 ++
 virt/kvm/guest_memfd_luo.c                    | 515 ++++++++++++++++++
 virt/kvm/kvm_luo.c                            | 195 +++++++
 virt/kvm/kvm_main.c                           |  94 +++-
 virt/kvm/kvm_mm.h                             |  15 +
 17 files changed, 1640 insertions(+), 81 deletions(-)
 create mode 100644 Documentation/liveupdate/vmm.rst
 create mode 100644 include/linux/kho/abi/kvm.h
 create mode 100644 tools/testing/selftests/kvm/guest_memfd_preservation_test.c
 create mode 100644 virt/kvm/guest_memfd.h
 create mode 100644 virt/kvm/guest_memfd_luo.c
 create mode 100644 virt/kvm/kvm_luo.c

base-commit: a204badd8432f93b7e862e7dac6db0fe3d65f370

Signature · 1 line
2.55.0.229.g6434b31f56-goog
author
#1 lore raw

PATCH v4 1/11 liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option

Introduce the LIVEUPDATE_GUEST_MEMFD Kconfig option. This option enables live update support for KVM guest_memfd files, enabling guest_memfd-backed memory preservation across kernel upgrades.

Currently this support only guest_memfd files that are full-shared (INIT_SHARED and no support for private_mem).

---
 kernel/liveupdate/Kconfig | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
kernel/liveupdate/Kconfig +15 -0
index c13af38ba23a..2490f9a9b838 100644
@@ -86,4 +86,19 @@ config LIVEUPDATE_MEMFD
 
 	  If unsure, say N.
 
+config LIVEUPDATE_GUEST_MEMFD
+	bool "Live update support for guest_memfd"
+	depends on LIVEUPDATE
+	depends on KVM_GUEST_MEMFD
+	default LIVEUPDATE
+	help
+	  Enable live update support for KVM guest_memfd files. This allows
+	  preserving VM Memory backed by guest_memfd file across kernel live
+	  updates.
+
+	  This can only be used for the guest_memfd that are fully-shared
+	  and pre-faulted.
+
+	  If unsure, say N.
+
 endmenu
Signature · 1 line
2.55.0.229.g6434b31f56-goog
#2 lore raw

Re: [PATCH v4 01/11] liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The patch introduces the `LIVEUPDATE_GUEST_MEMFD` Kconfig option claiming to enable a new…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The patch introduces the LIVEUPDATE_GUEST_MEMFD Kconfig option claiming to enable a new feature, but there is no actual implementation provided. --

--- Patch [1]: [PATCH v4 01/11] liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option --- commit e0c3fde657d56c548cd9a7467d2143f0f42c5242
Author: Tarun Sahu <[email protected]>

liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option

This commit introduces the LIVEUPDATE_GUEST_MEMFD Kconfig option, which aims to enable live update support for KVM guest_memfd files.

kernel/liveupdate/Kconfig config LIVEUPDATE_MEMFD · line 95

Quoted 13 lines
kernel/liveupdate/Kconfig +5 -0
index 1a8513f16ef7c..0bbc4037192e9 100644
@@ -88,4 +88,19 @@ config LIVEUPDATE_MEMFD
 
 	  If unsure, say N.
 
+config LIVEUPDATE_GUEST_MEMFD
+	bool "Live update support for guest_memfd"
+	depends on LIVEUPDATE
+	depends on KVM_GUEST_MEMFD
+	default LIVEUPDATE

[Severity: Medium]
Is the implementation for this feature missing from this patch?

The commit message notes that this patch enables live update support for KVM guest_memfd files, but the newly added LIVEUPDATE_GUEST_MEMFD symbol appears to be unreferenced in the tree. Since there are no Makefiles, C files, or preprocessor checks that use it, this currently acts as dead code.

Could the implementation be missing, or was this intended to be introduced as part of a larger series where the symbol is used in a subsequent patch?

Quoted 11 lines
kernel/liveupdate/Kconfig +10 -0
+	help
+	  Enable live update support for KVM guest_memfd files. This allows
+	  preserving VM Memory backed by guest_memfd file across kernel live
+	  updates.
+
+	  This can only be used for the guest_memfd that are fully-shared
+	  and pre-faulted.
+
+	  If unsure, say N.
+
 endmenu
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#3 lore raw

Re: [PATCH v4 01/11] liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option

Quoted 30 lines · [email protected] writes:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The patch introduces the LIVEUPDATE_GUEST_MEMFD Kconfig option claiming to enable a new feature, but there is no actual implementation provided. --

--- Patch [1]: [PATCH v4 01/11] liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option --- commit e0c3fde657d56c548cd9a7467d2143f0f42c5242
Author: Tarun Sahu <[email protected]>

liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option

This commit introduces the LIVEUPDATE_GUEST_MEMFD Kconfig option, which aims to enable live update support for KVM guest_memfd files.

kernel/liveupdate/Kconfig config LIVEUPDATE_MEMFD · line 95

Quoted 13 lines
kernel/liveupdate/Kconfig +5 -0
index 1a8513f16ef7c..0bbc4037192e9 100644
@@ -88,4 +88,19 @@ config LIVEUPDATE_MEMFD
 
 	  If unsure, say N.
 
+config LIVEUPDATE_GUEST_MEMFD
+	bool "Live update support for guest_memfd"
+	depends on LIVEUPDATE
+	depends on KVM_GUEST_MEMFD
+	default LIVEUPDATE

[Severity: Medium]
Is the implementation for this feature missing from this patch?

Quoted 8 lines

The commit message notes that this patch enables live update support for KVM guest_memfd files, but the newly added LIVEUPDATE_GUEST_MEMFD symbol appears to be unreferenced in the tree. Since there are no Makefiles, C files, or preprocessor checks that use it, this currently acts as dead code.

Could the implementation be missing, or was this intended to be introduced as part of a larger series where the symbol is used in a subsequent patch?

Sashiko is right here, I think this patch should be squashed with the first usage.

Quoted 14 lines

kernel/liveupdate/Kconfig config LIVEUPDATE_MEMFD · line 104

Quoted 11 lines
kernel/liveupdate/Kconfig +10 -0
+	help
+	  Enable live update support for KVM guest_memfd files. This allows
+	  preserving VM Memory backed by guest_memfd file across kernel live
+	  updates.
+
+	  This can only be used for the guest_memfd that are fully-shared
+	  and pre-faulted.
+
+	  If unsure, say N.
+
 endmenu

--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1

#4 lore raw

Re: [PATCH v4 01/11] liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option

Thanks for reviewing it.

Quoted 17 lines · Ackerley Tng
Quoted 15 lines · [email protected] writes:

kernel/liveupdate/Kconfig config LIVEUPDATE_MEMFD · line 93

Quoted 6 lines
kernel/liveupdate/Kconfig +5 -0
+config LIVEUPDATE_GUEST_MEMFD
+	bool "Live update support for guest_memfd"
+	depends on LIVEUPDATE
+	depends on KVM_GUEST_MEMFD
+	default LIVEUPDATE

[Severity: Medium]
Is the implementation for this feature missing from this patch?

The commit message notes that this patch enables live update support for KVM guest_memfd files, but the newly added LIVEUPDATE_GUEST_MEMFD symbol appears to be unreferenced in the tree. Since there are no Makefiles, C files, or preprocessor checks that use it, this currently acts as dead code.

Quoted 6 lines
Quoted 3 lines

Could the implementation be missing, or was this intended to be introduced as part of a larger series where the symbol is used in a subsequent patch?

Sashiko is right here, I think this patch should be squashed with the first usage.

Sure, will update it.

Quoted 15 lines
Quoted 15 lines

kernel/liveupdate/Kconfig config LIVEUPDATE_MEMFD · line 104

Quoted 12 lines
kernel/liveupdate/Kconfig +10 -0
+	help
+	  Enable live update support for KVM guest_memfd files. This allows
+	  preserving VM Memory backed by guest_memfd file across kernel live
+	  updates.
+
+	  This can only be used for the guest_memfd that are fully-shared
+	  and pre-faulted.
+
+	  If unsure, say N.
+
 endmenu

--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1

#5 lore raw

Re: [PATCH v4 01/11] liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option

In reply to Tarun Sahu
Quoted 6 lines · Tarun Sahu

Introduce the LIVEUPDATE_GUEST_MEMFD Kconfig option. This option enables live update support for KVM guest_memfd files, enabling guest_memfd-backed memory preservation across kernel upgrades.

Currently this support only guest_memfd files that are full-shared (INIT_SHARED and no support for private_mem).

Why? Specifically, why do we need/want a separate Kconfig for guest_memfd liveupdate versus KVM liveupdate? I can _maybe_ see a KVM-wide Kconfig, but even that seems like a stretch given that (AFAIK) hypervisors are the one and only use-case for liveupdate.

#6 lore raw

Re: [PATCH v4 01/11] liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option

Quoted 5 lines · Sean Christopherson
Quoted 4 lines · Tarun Sahu

Introduce the LIVEUPDATE_GUEST_MEMFD Kconfig option. This option enables live update support for KVM guest_memfd files, enabling guest_memfd-backed memory preservation across kernel upgrades.

Quoted 7 lines
Quoted 2 lines

Currently this support only guest_memfd files that are full-shared (INIT_SHARED and no support for private_mem).

Why? Specifically, why do we need/want a separate Kconfig for guest_memfd liveupdate versus KVM liveupdate? I can _maybe_ see a KVM-wide Kconfig, but even that seems like a stretch given that (AFAIK) hypervisors are the one and only use-case for liveupdate.

LIVEUPDATE is used in many subsystem. It is possible to enable liveupdate and kvm without enabling KVM_GUEST_MEMFD. So GUEST_MEMFD preservation must depend on CONFIG_LIVEUPDATE and KVM_GUEST_MEMFD both. LIVEUPDATE_GUEST_MEMFD Kconfig is wrapper on both of them. It need not to be visible in menu, So I can make it hidden. WDYT?

Also, KVM VM_FILE preservation and GUEST_MEMFD preservation both depends on LIVEUPDATE_GUEST_MEMFD in this patch series. [PATCH 05/11]* Which is wrong. So I will update KVM VM_FILE preservation to depend only on CONFIG_LIVEUPDATE.

*https://lore.kernel.org/all/[email protected]/#Z31virt:kvm:Makefile.kvm

~Tarun

#7 lore raw

Re: [PATCH v4 01/11] liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option

Quoted 21 lines · On Tue, Aug 11, 2026, [email protected] wrote:
Quoted 5 lines · Sean Christopherson
Quoted 4 lines · Tarun Sahu

Introduce the LIVEUPDATE_GUEST_MEMFD Kconfig option. This option enables live update support for KVM guest_memfd files, enabling guest_memfd-backed memory preservation across kernel upgrades.

Quoted 7 lines
Quoted 2 lines

Currently this support only guest_memfd files that are full-shared (INIT_SHARED and no support for private_mem).

Why? Specifically, why do we need/want a separate Kconfig for guest_memfd liveupdate versus KVM liveupdate? I can _maybe_ see a KVM-wide Kconfig, but even that seems like a stretch given that (AFAIK) hypervisors are the one and only use-case for liveupdate.

LIVEUPDATE is used in many subsystem. It is possible to enable liveupdate and kvm without enabling KVM_GUEST_MEMFD. So GUEST_MEMFD preservation must depend on CONFIG_LIVEUPDATE and KVM_GUEST_MEMFD both. LIVEUPDATE_GUEST_MEMFD Kconfig is wrapper on both of them. It need not to be visible in menu, So I can make it hidden. WDYT?

Quoted 4 lines

Also, KVM VM_FILE preservation and GUEST_MEMFD preservation both depends on LIVEUPDATE_GUEST_MEMFD in this patch series. [PATCH 05/11]* Which is wrong. So I will update KVM VM_FILE preservation to depend only on CONFIG_LIVEUPDATE.

Or just handle it all in the makefile? E.g.

kvm/Makefile.kvm +5 -0
index d047d4cf58c9..e6f098498795 100644
@@ -13,3 +13,8 @@ kvm-$(CONFIG_HAVE_KVM_IRQ_ROUTING) += $(KVM)/irqchip.o
 kvm-$(CONFIG_HAVE_KVM_DIRTY_RING) += $(KVM)/dirty_ring.o
 kvm-$(CONFIG_HAVE_KVM_PFNCACHE) += $(KVM)/pfncache.o
 kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd.o
+
+ifdef CONFIG_LIVEUPDATE
+kvm-y += $(KVM)/kvm_luo.o
+kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd_luo.o
+endif
#8 lore raw

Re: [PATCH v4 01/11] liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option

Quoted 34 lines · Sean Christopherson
Quoted 21 lines · On Tue, Aug 11, 2026, [email protected] wrote:
Quoted 5 lines · Sean Christopherson
Quoted 4 lines · Tarun Sahu

Introduce the LIVEUPDATE_GUEST_MEMFD Kconfig option. This option enables live update support for KVM guest_memfd files, enabling guest_memfd-backed memory preservation across kernel upgrades.

Quoted 7 lines
Quoted 2 lines

Currently this support only guest_memfd files that are full-shared (INIT_SHARED and no support for private_mem).

Why? Specifically, why do we need/want a separate Kconfig for guest_memfd liveupdate versus KVM liveupdate? I can _maybe_ see a KVM-wide Kconfig, but even that seems like a stretch given that (AFAIK) hypervisors are the one and only use-case for liveupdate.

LIVEUPDATE is used in many subsystem. It is possible to enable liveupdate and kvm without enabling KVM_GUEST_MEMFD. So GUEST_MEMFD preservation must depend on CONFIG_LIVEUPDATE and KVM_GUEST_MEMFD both. LIVEUPDATE_GUEST_MEMFD Kconfig is wrapper on both of them. It need not to be visible in menu, So I can make it hidden. WDYT?

Quoted 4 lines

Also, KVM VM_FILE preservation and GUEST_MEMFD preservation both depends on LIVEUPDATE_GUEST_MEMFD in this patch series. [PATCH 05/11]* Which is wrong. So I will update KVM VM_FILE preservation to depend only on CONFIG_LIVEUPDATE.

Or just handle it all in the makefile? E.g.

kvm/Makefile.kvm +0 -0
index d047d4cf58c9..e6f098498795 100644

kvm/Makefile.kvm kvm-$(CONFIG_HAVE_KVM_IRQ_ROUTING) += $(KVM)/irqchip.o · line 20

Quoted 8 lines
kvm/Makefile.kvm +5 -0
@@ -13,3 +13,8 @@ kvm-$(CONFIG_HAVE_KVM_IRQ_ROUTING) += $(KVM)/irqchip.o
 kvm-$(CONFIG_HAVE_KVM_DIRTY_RING) += $(KVM)/dirty_ring.o
 kvm-$(CONFIG_HAVE_KVM_PFNCACHE) += $(KVM)/pfncache.o
 kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd.o
+
+ifdef CONFIG_LIVEUPDATE
+kvm-y += $(KVM)/kvm_luo.o
+kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd_luo.o
+endif

Sounds Good to me. Will take care of it. Thanks.

author
#9 lore raw

PATCH v4 2/11 KVM: Introduce kvm_create_vm_file() helper

In reply to Tarun Sahu

- Move ITOA_MAX_LEN to kvm_mm.h for reuse by upcoming kvm_luo code.
- Introduce kvm_create_vm_file() to wrap kvm_create_vm() and anon_inode_getfile(), providing a unified VM file creation API.

---
 include/linux/kvm_host.h |  1 +
 virt/kvm/kvm_main.c      | 48 ++++++++++++++++++++++------------------
 virt/kvm/kvm_mm.h        |  3 +++
 3 files changed, 31 insertions(+), 21 deletions(-)
include/linux/kvm_host.h +1 -0
index ab8cfaec82d3..79b084887cdb 100644
@@ -1074,6 +1074,7 @@ void kvm_get_kvm(struct kvm *kvm);
 bool kvm_get_kvm_safe(struct kvm *kvm);
 void kvm_put_kvm(struct kvm *kvm);
 bool file_is_kvm(struct file *file);
+struct file *kvm_create_vm_file(unsigned long type, const char *fdname);
 void kvm_put_kvm_no_destroy(struct kvm *kvm);
 
 static inline struct kvm_memslots *__kvm_memslots(struct kvm *kvm, int as_id)
virt/kvm/kvm_main.c +27 -21

78 lines not on this page · raw message

virt/kvm/kvm_mm.h +3 -0
index 7510ca915dd1..624161793fec 100644
@@ -6,6 +6,9 @@
 #include <linux/kvm.h>
 #include <linux/kvm_types.h>
 
+/* Worst case buffer size needed for holding an integer as a string. */
+#define ITOA_MAX_LEN 12
+
 /*
  * Architectures can choose whether to use an rwlock or spinlock
  * for the mmu_lock.  These macros, for use in common code
Signature · 1 line
2.55.0.229.g6434b31f56-goog
#10 lore raw

Re: [PATCH v4 02/11] KVM: Introduce kvm_create_vm_file() helper

Quoted 4 lines · Tarun Sahu

- Move ITOA_MAX_LEN to kvm_mm.h for reuse by upcoming kvm_luo code.
- Introduce kvm_create_vm_file() to wrap kvm_create_vm() and anon_inode_getfile(), providing a unified VM file creation API.

I think these should be 2 separate patches.

Quoted 3 lines

[...snip...]

#11 lore raw

Re: [PATCH v4 02/11] KVM: Introduce kvm_create_vm_file() helper

Quoted 8 lines · Ackerley Tng
Quoted 4 lines · Tarun Sahu

- Move ITOA_MAX_LEN to kvm_mm.h for reuse by upcoming kvm_luo code.
- Introduce kvm_create_vm_file() to wrap kvm_create_vm() and anon_inode_getfile(), providing a unified VM file creation API.

I think these should be 2 separate patches.

Okay, I will split them. Thanks

Quoted 4 lines
Quoted 4 lines

[...snip...]

#12 lore raw

Re: [PATCH v4 02/11] KVM: Introduce kvm_create_vm_file() helper

In reply to Tarun Sahu
Quoted 3 lines · Tarun Sahu

- Move ITOA_MAX_LEN to kvm_mm.h for reuse by upcoming kvm_luo code.
- Introduce kvm_create_vm_file() to wrap kvm_create_vm() and anon_inode_getfile(), providing a unified VM file creation API.

As Ackerley pointed out, changelogs that are literally a list pretty much always needs to be split.

These also need to explain *why*. "providing a unified VM file creation API" isn't sufficient justification. In general, all of these changelogs need to provide more of the "why"; I can figure out what the patches are doing easily enough, but in my speed read of the series, it's hard to figure out the "why".

#13 lore raw

Re: [PATCH v4 02/11] KVM: Introduce kvm_create_vm_file() helper

Quoted 5 lines · Sean Christopherson
Quoted 3 lines · Tarun Sahu

- Move ITOA_MAX_LEN to kvm_mm.h for reuse by upcoming kvm_luo code.
- Introduce kvm_create_vm_file() to wrap kvm_create_vm() and anon_inode_getfile(), providing a unified VM file creation API.

Quoted 7 lines

As Ackerley pointed out, changelogs that are literally a list pretty much always needs to be split.

These also need to explain *why*. "providing a unified VM file creation API" isn't sufficient justification. In general, all of these changelogs need to provide more of the "why"; I can figure out what the patches are doing easily enough, but in my speed read of the series, it's hard to figure out the "why".

Okay, Will update them.

author
#14 lore raw

PATCH v4 3/11 KVM: Export kvm_uevent_notify_vm_create()

In reply to Tarun Sahu

- Export kvm_uevent_notify_vm_create() to allow triggering KVM event notifications during VM creation.

---
 include/linux/kvm_host.h | 1 +
 virt/kvm/kvm_main.c      | 5 +++++
 2 files changed, 6 insertions(+)
include/linux/kvm_host.h +1 -0
index 79b084887cdb..d9908c19e42f 100644
@@ -1076,6 +1076,7 @@ void kvm_put_kvm(struct kvm *kvm);
 bool file_is_kvm(struct file *file);
 struct file *kvm_create_vm_file(unsigned long type, const char *fdname);
 void kvm_put_kvm_no_destroy(struct kvm *kvm);
+void kvm_uevent_notify_vm_create(struct kvm *kvm);
 
 static inline struct kvm_memslots *__kvm_memslots(struct kvm *kvm, int as_id)
 {
virt/kvm/kvm_main.c +5 -0
index 3339b8fcd59f..701d4596a91c 100644
@@ -6348,6 +6348,11 @@ static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm)
 	kfree(env);
 }
 
+void kvm_uevent_notify_vm_create(struct kvm *kvm)
+{
+	kvm_uevent_notify_change(KVM_EVENT_CREATE_VM, kvm);
+}
+
 static void kvm_init_debug(void)
 {
 	const struct file_operations *fops;
Signature · 1 line
2.55.0.229.g6434b31f56-goog
#15 lore raw

Re: [PATCH v4 03/11] KVM: Export kvm_uevent_notify_vm_create()

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] The patch claims to export `kvm_uevent_notify_vm_create()` but fails to include the…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The patch claims to export kvm_uevent_notify_vm_create() but fails to include the EXPORT_SYMBOL_GPL() or EXPORT_SYMBOL_FOR_KVM_INTERNAL() macro. --

--- Patch [3]: [PATCH v4 03/11] KVM: Export kvm_uevent_notify_vm_create() --- commit 5e642ea16f0d267e622a352de60bfe397ecf79df
Author: Tarun Sahu <[email protected]>

KVM: Export kvm_uevent_notify_vm_create()

This commit exposes the kvm_uevent_notify_vm_create function to allow triggering KVM event notifications during VM creation.

[ ... ]

virt/kvm/kvm_main.c static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm) · line 6354

Quoted 11 lines
virt/kvm/kvm_main.c +4 -0
@@ -6348,6 +6348,11 @@ static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm)
 	kfree(env);
 }
 
+void kvm_uevent_notify_vm_create(struct kvm *kvm)
+{
+	kvm_uevent_notify_change(KVM_EVENT_CREATE_VM, kvm);
+}

[Severity: Low]
The commit message mentions exporting this function, but is it missing an EXPORT_SYMBOL_GPL or EXPORT_SYMBOL_FOR_KVM_INTERNAL declaration?

Without the macro, won't dynamically loaded architecture-specific modules like kvm-intel.ko hit undefined symbol linking errors when they try to call it?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#16 lore raw

Re: [PATCH v4 03/11] KVM: Export kvm_uevent_notify_vm_create()

Quoted 31 lines · [email protected] writes:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The patch claims to export kvm_uevent_notify_vm_create() but fails to include the EXPORT_SYMBOL_GPL() or EXPORT_SYMBOL_FOR_KVM_INTERNAL() macro. --

--- Patch [3]: [PATCH v4 03/11] KVM: Export kvm_uevent_notify_vm_create() --- commit 5e642ea16f0d267e622a352de60bfe397ecf79df
Author: Tarun Sahu <[email protected]>

KVM: Export kvm_uevent_notify_vm_create()

This commit exposes the kvm_uevent_notify_vm_create function to allow triggering KVM event notifications during VM creation.

[ ... ]

virt/kvm/kvm_main.c static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm) · line 6354

Quoted 11 lines
virt/kvm/kvm_main.c +4 -0
@@ -6348,6 +6348,11 @@ static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm)
 	kfree(env);
 }
 
+void kvm_uevent_notify_vm_create(struct kvm *kvm)
+{
+	kvm_uevent_notify_change(KVM_EVENT_CREATE_VM, kvm);
+}

[Severity: Low]
The commit message mentions exporting this function, but is it missing an EXPORT_SYMBOL_GPL or EXPORT_SYMBOL_FOR_KVM_INTERNAL declaration?

Quoted 4 lines

Without the macro, won't dynamically loaded architecture-specific modules like kvm-intel.ko hit undefined symbol linking errors when they try to call it?

I'm not 100% sure about the use of EXPORT_SYMBOL... here but usually I compile kvm as a module to try and catch these types of linking issues. Maybe Sean will have a better method/explanation for this.

This brings about another question I have though. How does luo work if the new kernel's deserialization of the persisted data structures depends on a module? Like in this case, KVM is a module, if KVM isn't loaded in the new kernel, do the data structures stick around until KVM is loaded? Or is KVM required to be loaded as a module in the new kernel?

Quoted 2 lines

--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3

#17 lore raw

Re: [PATCH v4 03/11] KVM: Export kvm_uevent_notify_vm_create()

Quoted 28 lines · Ackerley Tng
Quoted 27 lines · [email protected] writes:

--- Patch [3]: [PATCH v4 03/11] KVM: Export kvm_uevent_notify_vm_create() --- commit 5e642ea16f0d267e622a352de60bfe397ecf79df
Author: Tarun Sahu <[email protected]>

KVM: Export kvm_uevent_notify_vm_create()

This commit exposes the kvm_uevent_notify_vm_create function to allow triggering KVM event notifications during VM creation.

[ ... ]

virt/kvm/kvm_main.c static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm) · line 6354

Quoted 11 lines
virt/kvm/kvm_main.c +4 -0
@@ -6348,6 +6348,11 @@ static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm)
 	kfree(env);
 }
 
+void kvm_uevent_notify_vm_create(struct kvm *kvm)
+{
+	kvm_uevent_notify_change(KVM_EVENT_CREATE_VM, kvm);
+}

[Severity: Low]
The commit message mentions exporting this function, but is it missing an EXPORT_SYMBOL_GPL or EXPORT_SYMBOL_FOR_KVM_INTERNAL declaration?

Quoted 7 lines
Quoted 3 lines

Without the macro, won't dynamically loaded architecture-specific modules like kvm-intel.ko hit undefined symbol linking errors when they try to call it?

I'm not 100% sure about the use of EXPORT_SYMBOL... here but usually I compile kvm as a module to try and catch these types of linking issues. Maybe Sean will have a better method/explanation for this.

The issue is the shortlog, not the code. The intent is to make the function globally visible, not to truly export it (the eventual usage is code that's linked into kvm.ko, i.e. doesn't need an export).

#18 lore raw

Re: [PATCH v4 03/11] KVM: Export kvm_uevent_notify_vm_create()

In reply to Ackerley Tng
Quoted 33 lines · Ackerley Tng
Quoted 31 lines · [email protected] writes:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The patch claims to export kvm_uevent_notify_vm_create() but fails to include the EXPORT_SYMBOL_GPL() or EXPORT_SYMBOL_FOR_KVM_INTERNAL() macro. --

--- Patch [3]: [PATCH v4 03/11] KVM: Export kvm_uevent_notify_vm_create() --- commit 5e642ea16f0d267e622a352de60bfe397ecf79df
Author: Tarun Sahu <[email protected]>

KVM: Export kvm_uevent_notify_vm_create()

This commit exposes the kvm_uevent_notify_vm_create function to allow triggering KVM event notifications during VM creation.

[ ... ]

virt/kvm/kvm_main.c static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm) · line 6354

Quoted 11 lines
virt/kvm/kvm_main.c +4 -0
@@ -6348,6 +6348,11 @@ static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm)
 	kfree(env);
 }
 
+void kvm_uevent_notify_vm_create(struct kvm *kvm)
+{
+	kvm_uevent_notify_change(KVM_EVENT_CREATE_VM, kvm);
+}

[Severity: Low]
The commit message mentions exporting this function, but is it missing an EXPORT_SYMBOL_GPL or EXPORT_SYMBOL_FOR_KVM_INTERNAL declaration?

Quoted 8 lines
Quoted 4 lines

Without the macro, won't dynamically loaded architecture-specific modules like kvm-intel.ko hit undefined symbol linking errors when they try to call it?

I'm not 100% sure about the use of EXPORT_SYMBOL... here but usually I compile kvm as a module to try and catch these types of linking issues. Maybe Sean will have a better method/explanation for this.

The Intention is just to make it global to kvm module files. it is only used in kvm module so need to use EXPORT symbol as Sean also mentioned.

Quoted 7 lines

This brings about another question I have though. How does luo work if the new kernel's deserialization of the persisted data structures depends on a module? Like in this case, KVM is a module, if KVM isn't loaded in the new kernel, do the data structures stick around until KVM is loaded? Or is KVM required to be loaded as a module in the new kernel?

If kvm module is not loaded then guest_memfd_luo/kvm_luo handler will also not be registered. So luo_finish call will leak this memory and not be freed. This is aligned with LUO design.

I have raised the saperate thread with LUO list to discuss if LUO_CORE should throw a warning in such cases. [1]

[1] https://lore.kernel.org/all/[email protected]/

~Tarun

Quoted 3 lines
Quoted 3 lines

--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3

author
#19 lore raw

PATCH v4 4/11 KVM: Track weak reference to vm_file in struct kvm

In reply to Tarun Sahu

- Add a weak reference 'vm_file' in struct kvm under CONFIG_LIVEUPDATE_GUEST_MEMFD to enable reverse file resolution without circular lifetime dependencies.

---
 include/linux/kvm_host.h | 12 ++++++++++++
 virt/kvm/kvm_main.c      | 26 ++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)
include/linux/kvm_host.h +12 -0
index d9908c19e42f..8da9589425ee 100644
@@ -874,6 +874,18 @@ struct kvm {
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
 	/* Protected by slots_lock (for writes) and RCU (for reads) */
 	struct xarray mem_attr_array;
+#endif
+#ifdef CONFIG_LIVEUPDATE_GUEST_MEMFD
+	/*
+	 * Weak reference to the VFS file backing this KVM instance. Stored
+	 * without incrementing the file refcount to prevent a circular lifetime
+	 * dependency (since file->private_data already pins this struct kvm).
+	 * Used exclusively to resolve the file pointer back from struct kvm.
+	 *
+	 * Written/cleared via rcu_assign_pointer() and read locklessly under
+	 * RCU (e.g. via get_file_active() to prevent ABA races).
+	 */
+	struct file __rcu *vm_file;
 #endif
 	char stats_id[KVM_STATS_NAME_SIZE];
 };
virt/kvm/kvm_main.c +26 -0
index 701d4596a91c..14c32541ae34 100644
@@ -1346,6 +1346,19 @@ static int kvm_vm_release(struct inode *inode, struct file *filp)
 {
 	struct kvm *kvm = filp->private_data;
 
+#ifdef CONFIG_LIVEUPDATE_GUEST_MEMFD
+	/*
+	 * Clear the weak reference of the vm file.
+	 * In case vm file is closed by userspace, but kvm still has
+	 * other users like vCPUs, clearing this pointer ensures
+	 * that we don't have a dangling pointer to a closed file.
+	 *
+	 * Cleared via rcu_assign_pointer() to ensure proper memory visibility
+	 * for concurrent lockless readers under RCU.
+	 */
+	rcu_assign_pointer(kvm->vm_file, NULL);
+#endif
+
 	kvm_irqfd_release(kvm);
 
 	kvm_put_kvm(kvm);
@@ -5488,6 +5501,19 @@ struct file *kvm_create_vm_file(unsigned long type, const char *fdname)
 		return file;
 	}
 
+#ifdef CONFIG_LIVEUPDATE_GUEST_MEMFD
+	/*
+	 * Weak reference to the file (without get_file()) to prevent a circular
+	 * dependency. Safe because the file's release path clears this pointer
+	 * and drops its reference to the VM.
+	 *
+	 * Written via rcu_assign_pointer() because the pointer can be read
+	 * locklessly under RCU (e.g., in kvm_gmem_luo_preserve() via
+	 * get_file_active() to prevent lockless ABA races).
+	 */
+	rcu_assign_pointer(kvm->vm_file, file);
+#endif
+
 	/*
 	 * Don't call kvm_put_kvm anymore at this point; file->f_op is
 	 * already set, with ->release() being kvm_vm_release().  In error
Signature · 1 line
2.55.0.229.g6434b31f56-goog
#20 lore raw

Re: [PATCH v4 04/11] KVM: Track weak reference to vm_file in struct kvm

Quoted 1 line · Tarun Sahu

- Add a weak reference 'vm_file' in struct kvm under

"vm_file" is effectively already claimed by vm_area_struct.vm_file. *If* we go this route, I would very strongly prefer to take a dependency on "vfio: Use file-based reference counting for KVM"[*]. I'll poke that thread, we really should land that one particular patch long before the s390-is-arm64 insanity.

Empashish on "if" because this all feels backwards. It's not at all clear to me why liveupdate goes directly to guest_memfd, but then requires a token associated with the VM.

https://lore.kernel.org/all/[email protected]

#21 lore raw

Re: [PATCH v4 04/11] KVM: Track weak reference to vm_file in struct kvm

Quoted 8 lines · Sean Christopherson
Quoted 1 line · Tarun Sahu

- Add a weak reference 'vm_file' in struct kvm under

"vm_file" is effectively already claimed by vm_area_struct.vm_file. *If* we go this route, I would very strongly prefer to take a dependency on "vfio: Use file-based reference counting for KVM"[*]. I'll poke that thread, we really should land that one particular patch long before the s390-is-arm64 insanity.

[*] Seem interesting way to solve this. So we will also have vm_file directly in guest_memfd instead of struct kvm and refcounting on vm_file instead struct kvm?

struct gmem_file {
- struct kvm *kvm;

+	struct file *vm_file;
	struct xarray bindings;
	struct list_head entry;
};
Quoted 6 lines

Empashish on "if" because this all feels backwards.

It's not at all clear to
me why liveupdate goes directly to guest_memfd, but then requires a token associated with the VM.

guest_memfd can be recreated without having the struct kvm, Which there is no way to pass it on retrieval path. So the function kvm_gmem_luo_retrieve() gets this VM token, find the vm_file (hence struct kvm) associated with the preserved guest_memfd. Now it create a fresh guest_memfd by calling kvm_gmem_create(..., kvm). And populate preserved flags and folios to this guest_memfd.

Quoted 2 lines

https://lore.kernel.org/all/[email protected]

author
#22 lore raw

PATCH v4 5/11 KVM: LUO: Support VM preservation across live updates

In reply to Tarun Sahu

Register a Live Update Orchestrator (LUO) file handler for KVM VM files to serialize and deserialize VM state across kexec live updates.

Currently, Only VM type (e.g. arch.vm_type on x86) is preserved as part of VM preservation. On retrieval, kvm_luo_retrieve() recreates the KVM VM file via kvm_create_vm_file() and use an atomically incremented ID for the internal fdname, as the final fdname assigned by userspace is not yet known during retrieval. As this fdname is only used in debugfs infra, This will not break any UAPI.

This infrastructure establishes the foundation for preserving guest_memfd instances across live updates, and can be expanded in the future to preserve additional VM state.

Also updates MAINTAINERS to include virt/kvm/kvm_luo.c and include/linux/kho/abi/kvm.h.

---
 MAINTAINERS                 |  11 ++
 include/linux/kho/abi/kvm.h |  39 ++++++++
 virt/kvm/Makefile.kvm       |   1 +
 virt/kvm/kvm_luo.c          | 195 ++++++++++++++++++++++++++++++++++++
 virt/kvm/kvm_main.c         |   8 ++
 virt/kvm/kvm_mm.h           |   8 ++
 6 files changed, 262 insertions(+)
 create mode 100644 include/linux/kho/abi/kvm.h
 create mode 100644 virt/kvm/kvm_luo.c
MAINTAINERS +11 -0
index a3ed337e827d..0283f0fd6ef4 100644
@@ -14539,6 +14539,17 @@ S:	Maintained
 F:	Documentation/devicetree/bindings/leds/backlight/kinetic,ktz8866.yaml
 F:	drivers/video/backlight/ktz8866.c
 
+KVM LIVE UPDATE
+M:	Pasha Tatashin <[email protected]>
+M:	Mike Rapoport <[email protected]>
+M:	Pratyush Yadav <[email protected]>
+R:	Tarun Sahu <[email protected]>
+L:	[email protected]
+L:	[email protected]
+S:	Maintained
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git
+F:	virt/kvm/kvm_luo.c
+
 KVM PARAVIRT (KVM/paravirt)
 M:	Paolo Bonzini <[email protected]>
 R:	Vitaly Kuznetsov <[email protected]>
include/linux/kho/abi/kvm.h new file +39 -0
new file mode 100644
index 000000000000..718db68a541a
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Tarun Sahu <[email protected]>
+ *
+ * KVM Preservation ABI for Live Update Orchestrator (LUO)
+ */
+#ifndef _LINUX_KHO_ABI_KVM_H
+#define _LINUX_KHO_ABI_KVM_H
+
+#include <linux/types.h>
+#include <linux/kho/abi/kexec_handover.h>
+
+/**
+ * DOC: KVM Live Update ABI
+ *
+ * KVM uses the ABI defined below for preserving its state
+ * across a kexec reboot using the LUO.
+ *
+ * The state is serialized into a packed structure `struct kvm_luo_ser`
+ * which is handed over to the next kernel via the KHO mechanism.
+ *
+ * This interface is a contract. Any modification to the structure layout
+ * constitutes a breaking change. Such changes require incrementing the
+ * version number in the KVM_LUO_FH_COMPATIBLE compatibility string.
+ */
+
+/**
+ * struct kvm_luo_ser - Main serialization structure for a KVM VM.
+ * @type:         The type of VM.
+ */
+struct kvm_luo_ser {
+	u64 type;
+} __packed;
+
+/* The compatibility string for KVM VM file handler */
+#define KVM_LUO_FH_COMPATIBLE	"kvm_vm_luo_v1"
+
+#endif /* _LINUX_KHO_ABI_KVM_H */
virt/kvm/Makefile.kvm +1 -0
index d047d4cf58c9..c1a962159264 100644
@@ -13,3 +13,4 @@ kvm-$(CONFIG_HAVE_KVM_IRQ_ROUTING) += $(KVM)/irqchip.o
 kvm-$(CONFIG_HAVE_KVM_DIRTY_RING) += $(KVM)/dirty_ring.o
 kvm-$(CONFIG_HAVE_KVM_PFNCACHE) += $(KVM)/pfncache.o
 kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd.o
+kvm-$(CONFIG_LIVEUPDATE_GUEST_MEMFD) += $(KVM)/kvm_luo.o
virt/kvm/kvm_luo.c new file +195 -0

195 lines not on this page · raw message

virt/kvm/kvm_main.c +8 -0
index 14c32541ae34..d9c3dd1afb05 100644
@@ -6577,6 +6577,10 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module)
 	if (r)
 		goto err_virt;
 
+	r = kvm_luo_init();
+	if (r)
+		goto err_luo;
+
 	/*
 	 * Registration _must_ be the very last thing done, as this exposes
 	 * /dev/kvm to userspace, i.e. all infrastructure must be setup!
@@ -6590,6 +6594,8 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module)
 	return 0;
 
 err_register:
+	kvm_luo_exit();
+err_luo:
 	kvm_uninit_virtualization();
 err_virt:
 	kvm_gmem_exit();
@@ -6619,6 +6625,8 @@ void kvm_exit(void)
 	 */
 	misc_deregister(&kvm_dev);
 
+	kvm_luo_exit();
+
 	kvm_uninit_virtualization();
 
 	debugfs_remove_recursive(kvm_debugfs_dir);
virt/kvm/kvm_mm.h +8 -0
index 624161793fec..87198715fb01 100644
@@ -100,4 +100,12 @@ static inline void kvm_gmem_unbind(struct kvm_memory_slot *slot)
 }
 #endif /* CONFIG_KVM_GUEST_MEMFD */
 
+#ifdef CONFIG_LIVEUPDATE_GUEST_MEMFD
+int kvm_luo_init(void);
+void kvm_luo_exit(void);
+#else
+static inline int kvm_luo_init(void) { return 0; }
+static inline void kvm_luo_exit(void) {}
+#endif /* CONFIG_LIVEUPDATE_GUEST_MEMFD */
+
 #endif /* __KVM_MM_H__ */
Signature · 1 line
2.55.0.229.g6434b31f56-goog
#23 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] The generated `fdname` for a restored VM can collide with the file descriptor of a newly created…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] The generated fdname for a restored VM can collide with the file descriptor of a newly created VM in the same process, causing debugfs directory creation to fail silently for the new VM.
- [Low] The MAINTAINERS file was not updated to include the new ABI header include/linux/kho/abi/kvm.h, contrary to the commit message.
--

--- Patch [5]: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates --- commit 650c9d9cd02da64dbc0cdbd5b8073f5b00dadf99
Author: Tarun Sahu <[email protected]>

KVM: LUO: Support VM preservation across live updates

Register a Live Update Orchestrator (LUO) file handler for KVM VM files to serialize and deserialize VM state across kexec live updates. Preserves VM type and recreates the KVM VM file during retrieval with an internally generated fdname.

Quoted 2 lines

Also updates MAINTAINERS to include virt/kvm/kvm_luo.c and include/linux/kho/abi/kvm.h.

[Severity: Low]
This isn't a bug, but the commit message states that the MAINTAINERS file is updated to include include/linux/kho/abi/kvm.h, but the diff only adds virt/kvm/kvm_luo.c. Should the header file be added to the KVM LIVE UPDATE section?

Quoted 5 lines
virt/kvm/kvm_luo.c new file +0 -0
new file mode 100644
index 0000000000000..025b53151b6a0

[ ... ]

Quoted 7 lines
virt/kvm/kvm_luo.c new file +7 -0
+static atomic_t restored_vm_id = ATOMIC_INIT(0);
+
+static int kvm_luo_retrieve(struct liveupdate_file_op_args *args)
+{
+	char fdname[ITOA_MAX_LEN + 1];
+	struct kvm_luo_ser *ser;
+	struct file *file;

virt/kvm/kvm_luo.c line 107

Quoted 10 lines
virt/kvm/kvm_luo.c new file +10 -0
+	struct kvm *kvm;
+	int err = 0;
+
+	if (!args->serialized_data)
+		return -EINVAL;
+
+	ser = phys_to_virt(args->serialized_data);
+
+	snprintf(fdname, sizeof(fdname), "%d",
+		 atomic_inc_return(&restored_vm_id));

[Severity: Medium]
Could this generated fdname format collide with the file descriptor of a newly created VM in the same process?

Since this uses a simple "%d" format based on a global restored_vm_id counter, it shares the identical naming format as normal VM file descriptors.

If a process restores VMs using LUO and then creates a new VM that happens to get assigned a file descriptor matching one of the restored VM IDs, would the debugfs directory creation for the new VM fail silently and result in missing VM stats?

Quoted 2 lines
virt/kvm/kvm_luo.c new file +2 -0
+
+	file = kvm_create_vm_file(ser->type, fdname);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
#24 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

In reply to Tarun Sahu
Quoted 5 lines · Tarun Sahu

Register a Live Update Orchestrator (LUO) file handler for KVM VM files to serialize and deserialize VM state across kexec live updates.

Currently, Only VM type (e.g. arch.vm_type on x86) is preserved as part of VM preservation.

Why?

Quoted 9 lines

On retrieval, kvm_luo_retrieve() recreates the KVM VM file via kvm_create_vm_file() and use an atomically incremented ID for the internal fdname, as the final fdname assigned by userspace is not yet known during retrieval. As this fdname is only used in debugfs infra, This will not break any UAPI.

This infrastructure establishes the foundation for preserving guest_memfd instances across live updates, and can be expanded in the future to preserve additional VM state.

Uh, why guest_memfd? As much as I want to push guest_memfd adoption, it seems guest_memfd should be the _last_ thing we support, not the first. As evidenced by the last two decades, it's very doable to have KVM VMs without guest_memfd, but it's rather hard to have VMs without vCPUs.

Quoted 22 lines

Also updates MAINTAINERS to include virt/kvm/kvm_luo.c and include/linux/kho/abi/kvm.h.

---
 MAINTAINERS                 |  11 ++
 include/linux/kho/abi/kvm.h |  39 ++++++++
 virt/kvm/Makefile.kvm       |   1 +
 virt/kvm/kvm_luo.c          | 195 ++++++++++++++++++++++++++++++++++++
 virt/kvm/kvm_main.c         |   8 ++
 virt/kvm/kvm_mm.h           |   8 ++
 6 files changed, 262 insertions(+)
 create mode 100644 include/linux/kho/abi/kvm.h
 create mode 100644 virt/kvm/kvm_luo.c
MAINTAINERS +0 -0
index a3ed337e827d..0283f0fd6ef4 100644
@@ -14539,6 +14539,17 @@ S:	Maintained
 F:	Documentation/devicetree/bindings/leds/backlight/kinetic,ktz8866.yaml
 F:	drivers/video/backlight/ktz8866.c

MAINTAINERS S: Maintained · line 14550

Quoted 10 lines
MAINTAINERS +9 -0
 
+KVM LIVE UPDATE
+M:	Pasha Tatashin <[email protected]>
+M:	Mike Rapoport <[email protected]>
+M:	Pratyush Yadav <[email protected]>
+R:	Tarun Sahu <[email protected]>
+L:	[email protected]
+L:	[email protected]
+S:	Maintained
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git

NAK on taking changes through a different tree. This is KVM code, period.

In general, I'm skeptical of the dedicated MAINTAINERS entry. It's extremely difficult to tell since this series is little more than a skeleton (either that or liveupdate is way simpler that I was expecting), but I suspect that maintaining liveupdate for KVM (or for any subsystem) will require more subsystem-specific knowledge than liveupdate knowledge.

E.g. the LUO APIs seem pretty straightforward; I assume the bulk of the complexity is going to be in knowing what to save/restore, and how, which is much more about KVM than it is about liveupdate.

Quoted 77 lines
MAINTAINERS +2 -0
+F:	virt/kvm/kvm_luo.c
+
 KVM PARAVIRT (KVM/paravirt)
 M:	Paolo Bonzini <[email protected]>
 R:	Vitaly Kuznetsov <[email protected]>
include/linux/kho/abi/kvm.h new file +39 -0
new file mode 100644
index 000000000000..718db68a541a
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Tarun Sahu <[email protected]>
+ *
+ * KVM Preservation ABI for Live Update Orchestrator (LUO)
+ */
+#ifndef _LINUX_KHO_ABI_KVM_H
+#define _LINUX_KHO_ABI_KVM_H
+
+#include <linux/types.h>
+#include <linux/kho/abi/kexec_handover.h>
+
+/**
+ * DOC: KVM Live Update ABI
+ *
+ * KVM uses the ABI defined below for preserving its state
+ * across a kexec reboot using the LUO.
+ *
+ * The state is serialized into a packed structure `struct kvm_luo_ser`
+ * which is handed over to the next kernel via the KHO mechanism.
+ *
+ * This interface is a contract. Any modification to the structure layout
+ * constitutes a breaking change. Such changes require incrementing the
+ * version number in the KVM_LUO_FH_COMPATIBLE compatibility string.
+ */
+
+/**
+ * struct kvm_luo_ser - Main serialization structure for a KVM VM.
+ * @type:         The type of VM.
+ */
+struct kvm_luo_ser {
+	u64 type;
+} __packed;
+
+/* The compatibility string for KVM VM file handler */
+#define KVM_LUO_FH_COMPATIBLE	"kvm_vm_luo_v1"
+
+#endif /* _LINUX_KHO_ABI_KVM_H */
virt/kvm/Makefile.kvm +1 -0
index d047d4cf58c9..c1a962159264 100644
@@ -13,3 +13,4 @@ kvm-$(CONFIG_HAVE_KVM_IRQ_ROUTING) += $(KVM)/irqchip.o
 kvm-$(CONFIG_HAVE_KVM_DIRTY_RING) += $(KVM)/dirty_ring.o
 kvm-$(CONFIG_HAVE_KVM_PFNCACHE) += $(KVM)/pfncache.o
 kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd.o
+kvm-$(CONFIG_LIVEUPDATE_GUEST_MEMFD) += $(KVM)/kvm_luo.o
virt/kvm/kvm_luo.c new file +12 -0
new file mode 100644
index 000000000000..025b53151b6a
@@ -0,0 +1,195 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Copyright (c) 2026, Google LLC.
+ * Tarun Sahu <[email protected]>
+ *
+ * KVM VM Preservation for Live Update Orchestrator (LUO)
+ */
+
+/**
+ * DOC: KVM VM Preservation via LUO
+ *

virt/kvm/kvm_luo.c line 22

Quoted 10 lines
virt/kvm/kvm_luo.c new file +10 -0
+ * Overview
+ * ========
+ *
+ * KVM virtual machines (VMs) can be preserved over a kexec reboot using the
+ * Live Update Orchestrator (LUO) file preservation. This allows userspace
+ * to preserve KVM VM state across kexec reboots.
+ *
+ * The preservation is not intended to be fully transparent. Only specific
+ * VM configuration and state are preserved, while other aspects of the VM
+ * must be re-established or re-configured by userspace after retrieval.

IMO, this is not helpful documentation. Anyone with passing knowledge of LUO already knows the above. What people like me don't know are the exact details of how preserving a KVM guest is expected to work.

Quoted 11 lines
virt/kvm/kvm_luo.c new file +11 -0
+ * Preserved Properties
+ * ====================
+ *
+ * The following properties of the KVM VM are preserved across kexec:
+ *
+ * VM Type
+ *   The VM type (e.g., on x86 architecture, the vm_type parameter) is
+ *   preserved.
+ *
+ * Non-Preserved Properties
+ * ========================

virt/kvm/kvm_luo.c line 44

Quoted 10 lines
virt/kvm/kvm_luo.c new file +10 -0
+ *
+ * The preservation does not cover:
+ *
+ * - vCPUs and vCPU states
+ * - Memspots / Memory slot layout (memslots)
+ * - Interrupt controllers and IRQ routings
+ * - Coalesced MMIO zones
+ * - Device bindings (VFIO/Eventfds)
+ * - Active paging or guest registers state
+ * - etc

So... what's the plan? Bluntly, this series isn't going anywhere without a clear plan of how all of this is going to fit together.

virt/kvm/kvm_luo.c line 67

Quoted 4 lines
virt/kvm/kvm_luo.c new file +4 -0
+static int kvm_luo_preserve(struct liveupdate_file_op_args *args)
+{
+	struct kvm *kvm = args->file->private_data;
+	struct kvm_luo_ser *ser;

I *really* dislike the "ser" nomenclature. The abbreviation isn't common, and IMO it's not intuitive. This also needs to be more specific, becuase I'm guessing we'll end up with more than one "kvm" LUO structure.

Maybe something like kvm_vm_luo_state?

Quoted 8 lines
virt/kvm/kvm_luo.c new file +8 -0
+
+	if (kvm->vm_dead || kvm->vm_bugged)
+		return -EINVAL;
+
+	ser = kho_alloc_preserve(sizeof(*ser));
+	if (IS_ERR(ser))
+		return PTR_ERR(ser);
+

virt/kvm/kvm_luo.c line 85

Quoted 10 lines
virt/kvm/kvm_luo.c new file +10 -0
+#if defined(CONFIG_X86)
+	ser->type = kvm->arch.vm_type;
+#elif defined(CONFIG_ARM64)
+	ser->type = kvm_phys_shift(&kvm->arch.mmu);
+	if (kvm_vm_is_protected(kvm))
+		ser->type |= KVM_VM_TYPE_ARM_PROTECTED;
+
+#else
+	ser->type = 0;
+#endif

This needs to be properly supported with arch callbacks. And per-arch enabling needs to be done in separate patches.

Quoted 9 lines
virt/kvm/kvm_luo.c new file +9 -0
+
+	args->serialized_data = virt_to_phys(ser);
+	return 0;
+}
+
+static atomic_t restored_vm_id = ATOMIC_INIT(0);
+
+static int kvm_luo_retrieve(struct liveupdate_file_op_args *args)
+{

virt/kvm/kvm_luo.c line 104

Quoted 10 lines
virt/kvm/kvm_luo.c new file +10 -0
+	char fdname[ITOA_MAX_LEN + 1];
+	struct kvm_luo_ser *ser;
+	struct file *file;
+	struct kvm *kvm;
+	int err = 0;
+
+	if (!args->serialized_data)
+		return -EINVAL;
+
+	ser = phys_to_virt(args->serialized_data);

KVM generally uses __va() and __pa(), is there a reason to diverge?

maintainer
#25 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

Quoted 15 lines · Sean Christopherson
Quoted 5 lines · Tarun Sahu

Register a Live Update Orchestrator (LUO) file handler for KVM VM files to serialize and deserialize VM state across kexec live updates.

Currently, Only VM type (e.g. arch.vm_type on x86) is preserved as part of VM preservation.

Why?

Quoted 6 lines

On retrieval, kvm_luo_retrieve() recreates the KVM VM file via kvm_create_vm_file() and use an atomically incremented ID for the internal fdname, as the final fdname assigned by userspace is not yet known during retrieval. As this fdname is only used in debugfs infra, This will not break any UAPI.

Quoted 8 lines
Quoted 3 lines

This infrastructure establishes the foundation for preserving guest_memfd instances across live updates, and can be expanded in the future to preserve additional VM state.

Uh, why guest_memfd? As much as I want to push guest_memfd adoption, it seems guest_memfd should be the _last_ thing we support, not the first. As evidenced by the last two decades, it's very doable to have KVM VMs without guest_memfd, but it's rather hard to have VMs without vCPUs.

You _can_ preserve vCPUs today using KVM_{GET,SET}_REGS, they just won't run in the background during the reboot. This series can save you from dumping VM memory to disk if it is backed by guest_memfd.

Quoted 36 lines
Quoted 23 lines

Also updates MAINTAINERS to include virt/kvm/kvm_luo.c and include/linux/kho/abi/kvm.h.

---
 MAINTAINERS                 |  11 ++
 include/linux/kho/abi/kvm.h |  39 ++++++++
 virt/kvm/Makefile.kvm       |   1 +
 virt/kvm/kvm_luo.c          | 195 ++++++++++++++++++++++++++++++++++++
 virt/kvm/kvm_main.c         |   8 ++
 virt/kvm/kvm_mm.h           |   8 ++
 6 files changed, 262 insertions(+)
 create mode 100644 include/linux/kho/abi/kvm.h
 create mode 100644 virt/kvm/kvm_luo.c
MAINTAINERS +0 -0
index a3ed337e827d..0283f0fd6ef4 100644
@@ -14539,6 +14539,17 @@ S:	Maintained
 F:	Documentation/devicetree/bindings/leds/backlight/kinetic,ktz8866.yaml
 F:	drivers/video/backlight/ktz8866.c

MAINTAINERS S: Maintained · line 14550

Quoted 10 lines
MAINTAINERS +9 -0
 
+KVM LIVE UPDATE
+M:	Pasha Tatashin <[email protected]>
+M:	Mike Rapoport <[email protected]>
+M:	Pratyush Yadav <[email protected]>
+R:	Tarun Sahu <[email protected]>
+L:	[email protected]
+L:	[email protected]
+S:	Maintained
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git

NAK on taking changes through a different tree. This is KVM code, period.

Quoted 3 lines

In general, I'm skeptical of the dedicated MAINTAINERS entry. It's extremely difficult to tell since this series is little more than a skeleton (either that or liveupdate is way simpler that I was expecting), but I suspect that maintaining

It's a bit of both. This series of course doesn't support everything that guest_memfd can. At the same time, I also keep being (pleasantly) surprised at preservation being relatively simple. For example, the code to preserve a shmem file (via memfd) is roughly 600 lines, a big chunk of which is comments. The code of course has some limitations, but it is good enough for use in production.

Quoted 6 lines

liveupdate for KVM (or for any subsystem) will require more subsystem-specific knowledge than liveupdate knowledge.

E.g. the LUO APIs seem pretty straightforward; I assume the bulk of the complexity is going to be in knowing what to save/restore, and how, which is much more about KVM than it is about liveupdate.

I think it is fine if you want to take these changes through the KVM tree, but I would like live update maintainers to be listed as reviewers at least.

For one, we care about ABI breakages and versioning. The serialized state is a part of live update ABI and changes to it should be ACKed by us. For another, how the file handlers interact with their dependencies can affect the behaviour that VMMs observe. Those changes should also pass by some live update eyes.

So I think we should have a separate entry for KVM LIVE UPDATE that lists the live update maintainers (or perhaps only kexec@) as a reviewer. The patches can still flow through KVM tree but we'd get a chance to ACK/NACK them.

Of course this all can evolve later as we see fit but I think this is a good starting point.

Quoted 6 lines
Quoted 6 lines
MAINTAINERS +2 -0
+F:	virt/kvm/kvm_luo.c
+
 KVM PARAVIRT (KVM/paravirt)
 M:	Paolo Bonzini <[email protected]>
 R:	Vitaly Kuznetsov <[email protected]>

[...]

Signature · 2 lines
Regards,
Pratyush Yadav
#26 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

Quoted 21 lines · Pratyush Yadav
Quoted 19 lines · Sean Christopherson
Quoted 5 lines · Tarun Sahu

Register a Live Update Orchestrator (LUO) file handler for KVM VM files to serialize and deserialize VM state across kexec live updates.

Currently, Only VM type (e.g. arch.vm_type on x86) is preserved as part of VM preservation.

Why?

Quoted 9 lines

On retrieval, kvm_luo_retrieve() recreates the KVM VM file via kvm_create_vm_file() and use an atomically incremented ID for the internal fdname, as the final fdname assigned by userspace is not yet known during retrieval. As this fdname is only used in debugfs infra, This will not break any UAPI.

This infrastructure establishes the foundation for preserving guest_memfd instances across live updates, and can be expanded in the future to preserve additional VM state.

Quoted 7 lines
Quoted 4 lines

Uh, why guest_memfd? As much as I want to push guest_memfd adoption, it seems guest_memfd should be the _last_ thing we support, not the first. As evidenced by the last two decades, it's very doable to have KVM VMs without guest_memfd, but it's rather hard to have VMs without vCPUs.

You _can_ preserve vCPUs today using KVM_{GET,SET}_REGS, they just won't run in the background during the reboot.

What about x86 CoCo VMs? Which are quite literally _the_ reason guest_memfd was created in the first place.

Quoted 2 lines

This series can save you from dumping VM memory to disk if it is backed by guest_memfd.

Or to word it another way, one _can_ save guest_memfd, it's just slower. My point is that this series needs to provide a _lot_ more information about the bigger KVM picture. For those of us that are on the very fringes of live update, it's practically impossible to review because, to us, it seems very arbitrary.

The part that's especially confusing is the saving of the VM type. That comes straight from userspace, so it's super bizarre to automatically save/restore that, but nothing else.

Quoted 15 lines
Quoted 15 lines

Review

Quoted 9 lines
(quoted diff) +9 -0
+KVM LIVE UPDATE
+M:	Pasha Tatashin <[email protected]>
+M:	Mike Rapoport <[email protected]>
+M:	Pratyush Yadav <[email protected]>
+R:	Tarun Sahu <[email protected]>
+L:	[email protected]
+L:	[email protected]
+S:	Maintained
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git

NAK on taking changes through a different tree. This is KVM code, period.

In general, I'm skeptical of the dedicated MAINTAINERS entry. It's extremely difficult to tell since this series is little more than a skeleton (either that or liveupdate is way simpler that I was expecting), but I suspect that maintaining

...

Quoted 7 lines
Quoted 3 lines

E.g. the LUO APIs seem pretty straightforward; I assume the bulk of the complexity is going to be in knowing what to save/restore, and how, which is much more about KVM than it is about liveupdate.

I think it is fine if you want to take these changes through the KVM tree, but I would like live update maintainers to be listed as reviewers at least.

Why not simply add a file pattern match to the LIVE UPDATE entry?

MAINTAINERS +1 -1
index 8014b9f8253e..2eb57b22c37f 100644
@@ -15052,8 +15052,8 @@ F:      include/linux/liveupdate.h
 F:     include/uapi/linux/liveupdate.h
 F:     kernel/liveupdate/
 F:     lib/tests/liveupdate.c
-F:     mm/memfd_luo.c
 F:     tools/testing/selftests/liveupdate/
+N:     [^a-z]luo
 
 LLC (802.2)
 L:     [email protected]
Quoted 7 lines

At the same time, I also keep being (pleasantly)
surprised at preservation being relatively simple. For example, the code to preserve a shmem file (via memfd) is roughly 600 lines, a big chunk of which is comments. The code of course has some limitations, but it is good enough for use in production.

For one, we care about ABI breakages and versioning.

Which is amusing to me because that implies KVM does not, and I would hazard to guess that KVM has the biggest ABI surface of any subsystem in the kernel by a country mile (though I'm probably wildly underestimating the effective ABI surface of filesystems).

Quoted 2 lines

The serialized state is a part of live update ABI and changes to it should be ACKed by us.

Meh, "Don't break userspace" is a universal rule in the kernel, I genuinely don't see why liveupdate needs special treatment.

Quoted 3 lines

For another, how the file handlers interact with their dependencies can affect the behaviour that VMMs observe. Those changes should also pass by some live update eyes.

Perhaps in the short term, but IMO, that's not a winning strategy in the long term. From my perspective, that like saying the PAGE CACHE maintainers should review every usage of the filemap APIs, because how the APIs are used impacts the page cache and affects userspace-visible behavior. There are myriad analogies like that throughout the kernel.

Yes, liveupdate is new and shiny, but IMO for it to be successful and maintainable, it needs to be treated like any other core infrastructure in the kernel, not a special snowflake whose details are known only by a handful of people. Because I think it's likely liveupdate goes one of two ways: either liveupdate becomes a very niche thing that is used sparingly throughout the kernel, or it becomes a broadly used feature that is supported by many filesystems and subsystems.

If liveupdate is relegated to niche status, then it probably isn't going to see a significant amount of ongoing development, at which point the folks working on liveupdate will naturally migrate to other projects, and maintenance will largely be left to subsystem maintainers.

If liveupdate is broadly used, then having a single group of people maintain every subsystem's usage won't scale, and maintenance will again largely fall on the shoulder of subsystem maintainers. Which is totally fine and working as intended, because that's exactly what subystem maintainers are signing up for by merging support for liveupdate.

maintainer
#27 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

Quoted 27 lines · Sean Christopherson
Quoted 26 lines · Pratyush Yadav
Quoted 23 lines · Sean Christopherson
Quoted 5 lines · Tarun Sahu

Register a Live Update Orchestrator (LUO) file handler for KVM VM files to serialize and deserialize VM state across kexec live updates.

Currently, Only VM type (e.g. arch.vm_type on x86) is preserved as part of VM preservation.

Why?

Quoted 9 lines

On retrieval, kvm_luo_retrieve() recreates the KVM VM file via kvm_create_vm_file() and use an atomically incremented ID for the internal fdname, as the final fdname assigned by userspace is not yet known during retrieval. As this fdname is only used in debugfs infra, This will not break any UAPI.

This infrastructure establishes the foundation for preserving guest_memfd instances across live updates, and can be expanded in the future to preserve additional VM state.

Uh, why guest_memfd? As much as I want to push guest_memfd adoption, it seems guest_memfd should be the _last_ thing we support, not the first. As evidenced by the last two decades, it's very doable to have KVM VMs without guest_memfd, but it's rather hard to have VMs without vCPUs.

Quoted 5 lines
Quoted 2 lines

You _can_ preserve vCPUs today using KVM_{GET,SET}_REGS, they just won't run in the background during the reboot.

What about x86 CoCo VMs? Which are quite literally _the_ reason guest_memfd was created in the first place.

I don't know much about the history but I thought these days guest_memfd is used for more than just encrypted memory. I have seen talk of it being used for non-confidential VMs. For example these patches [0][1][2].

The CoCo parts can follow, but IIUC guest_memfd is being used to back guest memory on non-CoCo VMs too.

[0] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

Quoted 5 lines
Quoted 3 lines

This series can save you from dumping VM memory to disk if it is backed by guest_memfd.

Or to word it another way, one _can_ save guest_memfd, it's just slower. My

Sure. But making things faster is the entire point of live update.

Guest memory is one piece of that puzzle. Devices are another, and that's why there is a lot of work going on with VFIO, PCI, and IOMMU preservation. vCPUs are also a piece, but I think those will be the hardest to live update.

Quoted 3 lines

point is that this series needs to provide a _lot_ more information about the bigger KVM picture. For those of us that are on the very fringes of live update, it's practically impossible to review because, to us, it seems very arbitrary.

That's valid criticism. This series should do a better job of laying out the high level plan and where things fit.

Quoted 38 lines

The part that's especially confusing is the saving of the VM type. That comes straight from userspace, so it's super bizarre to automatically save/restore that, but nothing else.

Quoted 15 lines
Quoted 15 lines

Review

Quoted 9 lines
(quoted diff) +9 -0
+KVM LIVE UPDATE
+M:	Pasha Tatashin <[email protected]>
+M:	Mike Rapoport <[email protected]>
+M:	Pratyush Yadav <[email protected]>
+R:	Tarun Sahu <[email protected]>
+L:	[email protected]
+L:	[email protected]
+S:	Maintained
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git

NAK on taking changes through a different tree. This is KVM code, period.

In general, I'm skeptical of the dedicated MAINTAINERS entry. It's extremely difficult to tell since this series is little more than a skeleton (either that or liveupdate is way simpler that I was expecting), but I suspect that maintaining

...

Quoted 7 lines
Quoted 3 lines

E.g. the LUO APIs seem pretty straightforward; I assume the bulk of the complexity is going to be in knowing what to save/restore, and how, which is much more about KVM than it is about liveupdate.

I think it is fine if you want to take these changes through the KVM tree, but I would like live update maintainers to be listed as reviewers at least.

Why not simply add a file pattern match to the LIVE UPDATE entry?

MAINTAINERS +0 -0
index 8014b9f8253e..2eb57b22c37f 100644

MAINTAINERS F: include/linux/liveupdate.h · line 15059

Quoted 9 lines
MAINTAINERS +1 -1
@@ -15052,8 +15052,8 @@ F:      include/linux/liveupdate.h
 F:     include/uapi/linux/liveupdate.h
 F:     kernel/liveupdate/
 F:     lib/tests/liveupdate.c
-F:     mm/memfd_luo.c
 F:     tools/testing/selftests/liveupdate/
+N:     [^a-z]luo
 
 LLC (802.2)
 L:     [email protected]

This would list us as maintainers of kvm_luo.c and the tree as liveupdate.git, both of which is something you're saying you _don't_ want.

Quoted 10 lines
Quoted 8 lines

At the same time, I also keep being (pleasantly)
surprised at preservation being relatively simple. For example, the code to preserve a shmem file (via memfd) is roughly 600 lines, a big chunk of which is comments. The code of course has some limitations, but it is good enough for use in production.

For one, we care about ABI breakages and versioning.

Which is amusing to me because that implies KVM does not, and I would hazard to

No, it doesn't. What I'm saying is I care about changes to the _live update ABI_. Just like you probably care about changes to KVM ABI but not so much about BPF for example.

Quoted 9 lines

guess that KVM has the biggest ABI surface of any subsystem in the kernel by a country mile (though I'm probably wildly underestimating the effective ABI surface of filesystems).

Quoted 2 lines

The serialized state is a part of live update ABI and changes to it should be ACKed by us.

Meh, "Don't break userspace" is a universal rule in the kernel, I genuinely don't see why liveupdate needs special treatment.

Ironically enough, you miss my point. I'm not talking about userspace ABI. We all know not to break that. I am talking about live update _serialization ABI_. See the stuff under include/linux/kho/abi. This series also adds things there.

This is ABI between kernels. It needs to be stable-ish so you can move from one kernel version to another. At the same time, unlike userspace ABI, it can change. Today we don't have any rules and let you change things freely as long as you do a version bump. But at a later point, the plan is to add some stability requirements to the ABI so you can actually upgrade the kernel across major versions.

So at least for ABI changes, there should be an explicit ACK from the live update group. For code changes, I'd be flexible if you'd prefer that. More on it below.

Quoted 10 lines
Quoted 4 lines

For another, how the file handlers interact with their dependencies can affect the behaviour that VMMs observe. Those changes should also pass by some live update eyes.

Perhaps in the short term, but IMO, that's not a winning strategy in the long term. From my perspective, that like saying the PAGE CACHE maintainers should review every usage of the filemap APIs, because how the APIs are used impacts the page cache and affects userspace-visible behavior. There are myriad analogies like that throughout the kernel.

I've heard kernel maintainers complain many times about people or companies throwing code over the wall and not staying around to deal with the mess it might make.

I'd like to avoid that with live update and help you maintain this. As you've said, you are not as familiar with live update, and perhaps you might not even be as interested in it. So why not let the people who are help review the code?

Ultimately it is your subsystem so it is your call. And as I've said before, if you want to take it through the KVM tree and have a veto I think that is perfectly fine.

As an alternate example, with memfd_luo, the MM folks are rarely involved and the maintenance and review is done largely by me because I wrote that code. Mainly because the contents in memfd_luo.c are all live update related and don't matter much to core MM or memfd. I have heard similar desire for the HugeTLB live update work I am doing.

We can figure out what works best for KVM.

Quoted 18 lines

Yes, liveupdate is new and shiny, but IMO for it to be successful and maintainable, it needs to be treated like any other core infrastructure in the kernel, not a special snowflake whose details are known only by a handful of people. Because I think it's likely liveupdate goes one of two ways: either liveupdate becomes a very niche thing that is used sparingly throughout the kernel, or it becomes a broadly used feature that is supported by many filesystems and subsystems.

If liveupdate is relegated to niche status, then it probably isn't going to see a significant amount of ongoing development, at which point the folks working on liveupdate will naturally migrate to other projects, and maintenance will largely be left to subsystem maintainers.

If liveupdate is broadly used, then having a single group of people maintain every subsystem's usage won't scale, and maintenance will again largely fall on the shoulder of subsystem maintainers. Which is totally fine and working as intended, because that's exactly what subystem maintainers are signing up for by merging support for liveupdate.

Signature · 2 lines
Regards,
Pratyush Yadav
#28 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

Quoted 32 lines · Pratyush Yadav
Quoted 30 lines · Sean Christopherson
Quoted 28 lines · Pratyush Yadav
Quoted 15 lines · Sean Christopherson
Quoted 5 lines · Tarun Sahu

Register a Live Update Orchestrator (LUO) file handler for KVM VM files to serialize and deserialize VM state across kexec live updates.

Currently, Only VM type (e.g. arch.vm_type on x86) is preserved as part of VM preservation.

Why?

Quoted 6 lines

On retrieval, kvm_luo_retrieve() recreates the KVM VM file via kvm_create_vm_file() and use an atomically incremented ID for the internal fdname, as the final fdname assigned by userspace is not yet known during retrieval. As this fdname is only used in debugfs infra, This will not break any UAPI.

Quoted 8 lines
Quoted 3 lines

This infrastructure establishes the foundation for preserving guest_memfd instances across live updates, and can be expanded in the future to preserve additional VM state.

Uh, why guest_memfd? As much as I want to push guest_memfd adoption, it seems guest_memfd should be the _last_ thing we support, not the first. As evidenced by the last two decades, it's very doable to have KVM VMs without guest_memfd, but it's rather hard to have VMs without vCPUs.

You _can_ preserve vCPUs today using KVM_{GET,SET}_REGS, they just won't run in the background during the reboot.

Quoted 6 lines
Quoted 2 lines

What about x86 CoCo VMs? Which are quite literally _the_ reason guest_memfd was created in the first place.

I don't know much about the history but I thought these days guest_memfd is used for more than just encrypted memory. I have seen talk of it being used for non-confidential VMs. For example these patches [0][1][2].

It's getting there, but it's all still very nascent.

Quoted 2 lines

The CoCo parts can follow, but IIUC guest_memfd is being used to back guest memory on non-CoCo VMs too.

Not using upstream code, though there are most definitely folks using guest_memfd in production with out-of-tree patches.

But that's all beside the point. What I'm saying is that SNP and TDX *must* use guest_memfd, whereas guest_memfd is optional for all other VM types. And so adding LUO support for guest_memfd without even sketching out a plan for CoCo VMs feels backwards.

I'm not necessarily opposed to gradual guest_memfd support, but there needs to be a clear plan of how all of this is going to fit together. It doesn't need to be perfect, and I'm sure we'll make mistakes along the way, but I want to at least try not to paint ourselves into a corner, especially with respect to the ABI.

Quoted 22 lines
Quoted 22 lines
Quoted 7 lines
Quoted 3 lines

E.g. the LUO APIs seem pretty straightforward; I assume the bulk of the complexity is going to be in knowing what to save/restore, and how, which is much more about KVM than it is about liveupdate.

I think it is fine if you want to take these changes through the KVM tree, but I would like live update maintainers to be listed as reviewers at least.

Why not simply add a file pattern match to the LIVE UPDATE entry?

MAINTAINERS +1 -1
index 8014b9f8253e..2eb57b22c37f 100644
@@ -15052,8 +15052,8 @@ F:      include/linux/liveupdate.h
 F:     include/uapi/linux/liveupdate.h
 F:     kernel/liveupdate/
 F:     lib/tests/liveupdate.c
-F:     mm/memfd_luo.c
 F:     tools/testing/selftests/liveupdate/
+N:     [^a-z]luo
 
Quoted 4 lines

MAINTAINERS F: include/linux/liveupdate.h · line 15059

Quoted 2 lines
MAINTAINERS +0 -0
 LLC (802.2)
 L:     [email protected]

This would list us as maintainers of kvm_luo.c and the tree as liveupdate.git,

It would list both "LIVE UPDATE" and "KERNEL VIRTUAL MACHINE (KVM)", as KVM would
still cover the files via "F: virt/kvm/*". And my read of MAINTAINERS is that
N: and K: entries are "secondary" if the files/scope is covered by an F: entry.

E.g. arch/x86/kvm/vmx/tdx.c is covered by "KERNEL VIRTUAL MACHINE FOR X86 (KVM/x86)"
via "F: arch/x86/kvm/*/", and also by "X86 TRUST DOMAIN EXTENSIONS (TDX)" via
"N: tdx" (or maybe "K: \b(tdx)"? I haven't bothered to check which one
triggers). And while I don't think there was ever any formal discussion, AFAIK everyone reads the situation as KVM still being the primary maintainer/tree for that code.

Quoted 1 line

both of which is something you're saying you _don't_ want.

No, what I don't want is a dedicated "KVM LIVE UPDATE" entry, because I think most
people would read "F: virt/kvm/kvm_luo.c" and "F: virt/kvm/guest_memfd_luo.c"
as being more precise than KVM's "F: virt/kvm/*" and thus would read things as
"KVM LIVE UPDATE" being the primary maintainer.

In other words, I'm more than ok with LUO being looped in on KVM LUO changes and having the authority to object to problematic changes, but I'm not ok with LUO taking primary ownership of KVM code.

I realize there's more than a bit of nuance in my interpretation of N: and K:, but again my experience with TDX is that so long as the maintainers are aligned on expectations, it's a non-issue in practice.

Quoted 21 lines
Quoted 9 lines
Quoted 7 lines

At the same time, I also keep being (pleasantly)
surprised at preservation being relatively simple. For example, the code to preserve a shmem file (via memfd) is roughly 600 lines, a big chunk of which is comments. The code of course has some limitations, but it is good enough for use in production.

For one, we care about ABI breakages and versioning.

Which is amusing to me because that implies KVM does not, and I would hazard to

No, it doesn't. What I'm saying is I care about changes to the _live update ABI_. Just like you probably care about changes to KVM ABI but not so much about BPF for example.

Quoted 7 lines

guess that KVM has the biggest ABI surface of any subsystem in the kernel by a country mile (though I'm probably wildly underestimating the effective ABI surface of filesystems).

Quoted 2 lines

The serialized state is a part of live update ABI and changes to it should be ACKed by us.

Quoted 7 lines
Quoted 2 lines

Meh, "Don't break userspace" is a universal rule in the kernel, I genuinely don't see why liveupdate needs special treatment.

Ironically enough, you miss my point. I'm not talking about userspace ABI. We all know not to break that. I am talking about live update _serialization ABI_. See the stuff under include/linux/kho/abi. This series also adds things there.

No, I understand exactly what ABI you're talking about.

Quoted 3 lines

This is ABI between kernels. It needs to be stable-ish so you can move from one kernel version to another. At the same time, unlike userspace ABI, it can change.

Uh, yeah, so KVM has been managing such immutable ABI for practically its entire existence. KVM's save/restore uAPI has exactly what you're describing: serialization ABI that needs to be backwards and forwards compatible between different kernels in order to support both upgrade and rollback scenarios via live migration.

KVM also has immutable ABI between itself and guest kernels, including implicit "ABI" in the form of not changing guest-visible behavior.

Quoted 7 lines

Today we don't have any rules and let you change things freely as long as you do a version bump. But at a later point, the plan is to add some stability requirements to the ABI so you can actually upgrade the kernel across major versions.

So at least for ABI changes, there should be an explicit ACK from the live update group.

I don't entirely agree. I get where you're coming from, and I 100% agree that the more eyeballs on changes that may affect ABI, the better.

Where I have problems with the above is that it doesn't account for the nuances of save/restore across different kernels. The literal format of the serialized data is the most obvious form of ABI, but it's absolutely possible to change ABI without changing the data format.

E.g. say there's a flags field in some serialization structure, and flags X and Y are mutually exclusive in current kernels. If a future kernel relaxes that restriction for whatever reason, then the effective ABI has been broken because state created and saved on new kernels can't be restored on old kernels. And this is not a theoretical concern, KVM has run afoul of this a few times (though thankfully very rarely).

I don't think it's reasonable to expect LUO maintainers to gain enough expertise in each subsystem to be able to ensure changes are forwards and backwards compatible. The only way I see LUO being successful in the long term is to get subsystem maintainers/contributors to understand *and buy-in* to the LUO model and rules, so that each subsystem can largely be self-sustatining. I.e. setting yourselves up as literal gatekeepers will help prevent blatant breakage, but it's less likely to help guard against more subtle breakage, and in my experience, subtle breakage is by far harder to detect and more painful to deal with.

Somewhat of a side topic: in my experience, using monotonically increasing version numbers is a horrible way to enumerate features/content. So for me, allowing KVM's LUO ABI to change with a verson bump is probably a non-starter. I.e. whatever gets merged needs to be more future-proof than "we'll deal with it later".

maintainer
#29 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

Quoted 2 lines · Sean Christopherson
Quoted 1 line · Pratyush Yadav

On Tue, Aug 11 2026, Sean Christopherson wrote:

[...]

Quoted 9 lines

But that's all beside the point. What I'm saying is that SNP and TDX *must* use guest_memfd, whereas guest_memfd is optional for all other VM types. And so adding LUO support for guest_memfd without even sketching out a plan for CoCo VMs feels backwards.

I'm not necessarily opposed to gradual guest_memfd support, but there needs to be a clear plan of how all of this is going to fit together. It doesn't need to be perfect, and I'm sure we'll make mistakes along the way, but I want to at least try not to paint ourselves into a corner, especially with respect to the ABI.

Right. I think landing shared guest_memfd is the first step. It is simple enough and doesn't need to preserve the more complicated architecture specific things like secure EPT, etc. The CoCo support can build on top of this.

There is work underway already for an RFC of TDX live update, but Tarun knows the details better than me so I'll let him fill in the details for the next version of this series.

Quoted 41 lines
Quoted 23 lines
Quoted 23 lines
Quoted 8 lines
Quoted 4 lines

E.g. the LUO APIs seem pretty straightforward; I assume the bulk of the complexity is going to be in knowing what to save/restore, and how, which is much more about KVM than it is about liveupdate.

I think it is fine if you want to take these changes through the KVM tree, but I would like live update maintainers to be listed as reviewers at least.

Why not simply add a file pattern match to the LIVE UPDATE entry?

MAINTAINERS +1 -1
index 8014b9f8253e..2eb57b22c37f 100644
@@ -15052,8 +15052,8 @@ F:      include/linux/liveupdate.h
 F:     include/uapi/linux/liveupdate.h
 F:     kernel/liveupdate/
 F:     lib/tests/liveupdate.c
-F:     mm/memfd_luo.c
 F:     tools/testing/selftests/liveupdate/
+N:     [^a-z]luo
 
Quoted 4 lines

MAINTAINERS F: include/linux/liveupdate.h · line 15059

Quoted 2 lines
MAINTAINERS +0 -0
 LLC (802.2)
 L:     [email protected]

This would list us as maintainers of kvm_luo.c and the tree as liveupdate.git,

It would list both "LIVE UPDATE" and "KERNEL VIRTUAL MACHINE (KVM)", as KVM would
still cover the files via "F: virt/kvm/*". And my read of MAINTAINERS is that
N: and K: entries are "secondary" if the files/scope is covered by an F: entry.

E.g. arch/x86/kvm/vmx/tdx.c is covered by "KERNEL VIRTUAL MACHINE FOR X86 (KVM/x86)"
via "F: arch/x86/kvm/*/", and also by "X86 TRUST DOMAIN EXTENSIONS (TDX)" via
"N: tdx" (or maybe "K: \b(tdx)"? I haven't bothered to check which one
triggers). And while I don't think there was ever any formal discussion, AFAIK everyone reads the situation as KVM still being the primary maintainer/tree for that code.

Quoted 1 line

both of which is something you're saying you _don't_ want.

Quoted 8 lines

No, what I don't want is a dedicated "KVM LIVE UPDATE" entry, because I think most
people would read "F: virt/kvm/kvm_luo.c" and "F: virt/kvm/guest_memfd_luo.c"
as being more precise than KVM's "F: virt/kvm/*" and thus would read things as
"KVM LIVE UPDATE" being the primary maintainer.

In other words, I'm more than ok with LUO being looped in on KVM LUO changes and having the authority to object to problematic changes, but I'm not ok with LUO taking primary ownership of KVM code.

Sure, that sounds good to me.

Quoted 4 lines

I realize there's more than a bit of nuance in my interpretation of N: and K:, but again my experience with TDX is that so long as the maintainers are aligned on expectations, it's a non-issue in practice.

Yep, as long as we agree among ourselves I don't think the red tape around which entry has higher priority matters much.

Quoted 32 lines
Quoted 22 lines
Quoted 10 lines
Quoted 8 lines

At the same time, I also keep being (pleasantly)
surprised at preservation being relatively simple. For example, the code to preserve a shmem file (via memfd) is roughly 600 lines, a big chunk of which is comments. The code of course has some limitations, but it is good enough for use in production.

For one, we care about ABI breakages and versioning.

Which is amusing to me because that implies KVM does not, and I would hazard to

No, it doesn't. What I'm saying is I care about changes to the _live update ABI_. Just like you probably care about changes to KVM ABI but not so much about BPF for example.

Quoted 7 lines

guess that KVM has the biggest ABI surface of any subsystem in the kernel by a country mile (though I'm probably wildly underestimating the effective ABI surface of filesystems).

Quoted 2 lines

The serialized state is a part of live update ABI and changes to it should be ACKed by us.

Quoted 7 lines
Quoted 2 lines

Meh, "Don't break userspace" is a universal rule in the kernel, I genuinely don't see why liveupdate needs special treatment.

Ironically enough, you miss my point. I'm not talking about userspace ABI. We all know not to break that. I am talking about live update _serialization ABI_. See the stuff under include/linux/kho/abi. This series also adds things there.

No, I understand exactly what ABI you're talking about.

Quoted 8 lines
Quoted 3 lines

This is ABI between kernels. It needs to be stable-ish so you can move from one kernel version to another. At the same time, unlike userspace ABI, it can change.

Uh, yeah, so KVM has been managing such immutable ABI for practically its entire existence. KVM's save/restore uAPI has exactly what you're describing: serialization ABI that needs to be backwards and forwards compatible between different kernels in order to support both upgrade and rollback scenarios via live migration.

I think there is a slight difference between KVM's save/resture uAPI and live update's ABI. With KVM's uAPI, you need to maintain strict backwards compatibility because userspace reads what you output. So if you change the layout, userspace will interpret it wrong and might break.

With live update, the ABI never gets to userspace. It is used to talk between kernels directly. So if you do break that, your userspace keeps working fine, you just might not be able to live update to the incompatible kernel.

So with live update, we don't need to keep backwards compatibility in the ABI forever. Of course, it is good to minimize changes, but we have more freedom to change it.

The current idea is we change the ABI whenever needed. This is because live update is still in development so we are likely to see a lot more changes before we settle down into something that mostly works. At some point, we want to start being more strict between incompatible ABI changes. The current idea is that we would like to keep backwards compatibility across LTS kernels. But none of that is set in stone yet.

That's why I would like to have an explicit ACK from the live update group for ABI changes. I would like to enforce these compatibility windows.

Quoted 20 lines

KVM also has immutable ABI between itself and guest kernels, including implicit "ABI" in the form of not changing guest-visible behavior.

Quoted 7 lines

Today we don't have any rules and let you change things freely as long as you do a version bump. But at a later point, the plan is to add some stability requirements to the ABI so you can actually upgrade the kernel across major versions.

So at least for ABI changes, there should be an explicit ACK from the live update group.

I don't entirely agree. I get where you're coming from, and I 100% agree that the more eyeballs on changes that may affect ABI, the better.

Where I have problems with the above is that it doesn't account for the nuances of save/restore across different kernels. The literal format of the serialized data is the most obvious form of ABI, but it's absolutely possible to change ABI without changing the data format.

Quoted 6 lines

E.g. say there's a flags field in some serialization structure, and flags X and Y are mutually exclusive in current kernels. If a future kernel relaxes that restriction for whatever reason, then the effective ABI has been broken because state created and saved on new kernels can't be restored on old kernels. And this is not a theoretical concern, KVM has run afoul of this a few times (though thankfully very rarely).

That's a bug, and should be fixed.

I am talking here about intentional ABI changes. As I mentioned above we _can_ change the ABI, we just need to be careful about it.

Quoted 9 lines

I don't think it's reasonable to expect LUO maintainers to gain enough expertise in each subsystem to be able to ensure changes are forwards and backwards compatible. The only way I see LUO being successful in the long term is to get subsystem maintainers/contributors to understand *and buy-in* to the LUO model and rules, so that each subsystem can largely be self-sustatining. I.e. setting yourselves up as literal gatekeepers will help prevent blatant breakage, but it's less likely to help guard against more subtle breakage, and in my experience, subtle breakage is by far harder to detect and more painful to deal with.

Outside of enforcing the ABI compatibility windows, I think this makes sense.

In principle at least, though from my experience the maintainer appetite to care about live update varies. The feeling I've got from MM for example is more along the lines of "you break it, you buy it". Which is also a valid model IMO. The person who wrote the live update code for a subsystem is well equipped to understand these nuances.

Quoted 5 lines

Somewhat of a side topic: in my experience, using monotonically increasing version numbers is a horrible way to enumerate features/content. So for me, allowing KVM's LUO ABI to change with a verson bump is probably a non-starter. I.e. whatever gets merged needs to be more future-proof than "we'll deal with it later".

As I wrote above, the compatibility model for live update allows breaking ABI with version bumps. Userspace keeps working fine, it only restricts the set of kernels that you can go to. So the "we'll deal with it later" is not as harmful because our choices here are not permanent.

Also, there is work underway to allow transitions between versions [0].

But still, I am not opposed to a more flexible guest_memfd ABI. I think we can reserve some space for feature flags to let us add things in the future.

[0] https://lore.kernel.org/kexec/[email protected]/

Signature · 2 lines
Regards,
Pratyush Yadav
maintainer NAK
#30 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

Quoted 17 lines · Pratyush Yadav
Quoted 9 lines · Sean Christopherson
Quoted 3 lines · Pratyush Yadav

This is ABI between kernels. It needs to be stable-ish so you can move from one kernel version to another. At the same time, unlike userspace ABI, it can change.

Uh, yeah, so KVM has been managing such immutable ABI for practically its entire existence. KVM's save/restore uAPI has exactly what you're describing: serialization ABI that needs to be backwards and forwards compatible between different kernels in order to support both upgrade and rollback scenarios via live migration.

I think there is a slight difference between KVM's save/resture uAPI and live update's ABI. With KVM's uAPI, you need to maintain strict backwards compatibility because userspace reads what you output. So if you change the layout, userspace will interpret it wrong and might break.

Quoted 8 lines

With live update, the ABI never gets to userspace. It is used to talk between kernels directly. So if you do break that, your userspace keeps working fine, you just might not be able to live update to the incompatible kernel.

So with live update, we don't need to keep backwards compatibility in the ABI forever. Of course, it is good to minimize changes, but we have more freedom to change it.

Ah. So this is the heart of the disconnect. I very strongly disagree with the statement that live update doesn't need to support backwards compatibility. I can totally believe that the folks working on live update are ok breaking backwards compatibility because their use cases are "fine" with such breakage. And I can also believe live update as an upstream kernel feature being developed by those same folks is also ok with breaking backwards compatibility.

But with my upstream KVM maintainer hat on, I am not ok with that. I did not agree to support a world where KVM is allowed to break backwards compatibility, so long as it's done "carefully" or whatever. If y'all want to deal with the resulting complexity, that's fine by me, but you'll be doing it without KVM.

I totally understand that exploratory work and initial development is best done in private and/or in a small working groups. But decisions that will significantly impact multiple subsystems need to be made *with* those subystems. I mean, obviously it's possible to make a decision in a small group and then "publish" the result later, but then as is happening here, the community and subsystem maintainers like me may refuse to play ball if they disagree.

To be very clear, for any KVM live update support that doesn't guarantee backwards compatibility (allowing that bugs happen), a very firm:

NAK

maintainer
#31 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

Hi Sean,

Quoted 34 lines · Sean Christopherson
Quoted 17 lines · Pratyush Yadav
Quoted 9 lines · Sean Christopherson
Quoted 3 lines · Pratyush Yadav

This is ABI between kernels. It needs to be stable-ish so you can move from one kernel version to another. At the same time, unlike userspace ABI, it can change.

Uh, yeah, so KVM has been managing such immutable ABI for practically its entire existence. KVM's save/restore uAPI has exactly what you're describing: serialization ABI that needs to be backwards and forwards compatible between different kernels in order to support both upgrade and rollback scenarios via live migration.

I think there is a slight difference between KVM's save/resture uAPI and live update's ABI. With KVM's uAPI, you need to maintain strict backwards compatibility because userspace reads what you output. So if you change the layout, userspace will interpret it wrong and might break.

Quoted 8 lines

With live update, the ABI never gets to userspace. It is used to talk between kernels directly. So if you do break that, your userspace keeps working fine, you just might not be able to live update to the incompatible kernel.

So with live update, we don't need to keep backwards compatibility in the ABI forever. Of course, it is good to minimize changes, but we have more freedom to change it.

Ah. So this is the heart of the disconnect. I very strongly disagree with the statement that live update doesn't need to support backwards compatibility. I can totally believe that the folks working on live update are ok breaking backwards compatibility because their use cases are "fine" with such breakage. And I can also believe live update as an upstream kernel feature being developed by those same folks is also ok with breaking backwards compatibility.

Quoted 4 lines

But with my upstream KVM maintainer hat on, I am not ok with that. I did not agree to support a world where KVM is allowed to break backwards compatibility, so long as it's done "carefully" or whatever. If y'all want to deal with the resulting complexity, that's fine by me, but you'll be doing it without KVM.

Let's step back a bit. I don't think backwards compatibility in the _ABI_ is all that important in live update's context. What is important is that our users are able to live update from kernel version X to Y. The line format the kernel uses to describe its state is an implementation detail. Our users never see it.

The high level idea is that when you need to make a change to the ABI so the kernel can better describe the objects/resources/files it is passing, you create a new version of the ABI and allow transitions from older versions.

Say you make some changes and need an ABI version v5. You don't get rid of v4. You keep it around. LUO core will facilitate picking the right version for the next kernel. So it will be possible to seamlessly upgrade your kernel that speaks v4 to a new kernel that speaks v4 and v5. Now this kernel can start speaking v5 if its successor speaks v5 too. And so on for going to v6 and v7, etc.

After a "reasonable" time given for upgrades, you deprecate v4. v4 has been around long enough and our users have had a chance to go to kernels speaking newer versions. That's when you remove the code for v4 from the kernel.

We can argue what "reasonable" means, but the core idea stays.

It will still be possible to go back to v4 or earlier, but you'd need an extra stop along the way. And of course, vendors can keep a wider support matrix downstream if they see the need for it.

Does this idea of "backwards compatibility" sound acceptable to you, at least at a high level?

Now coming back to today's reality. We don't yet support this version transition. I proposed the idea at LPC 2025 [0]. Logan Odell has taken over the work since I have other things on my plate, but it is something being actively developed. Logan recently sent some RFCs [1] for this too, though TBH I haven't yet gotten to those patches. We also have proposed a talk about this at LPC 2026, it would be great if you could attend (if the talk gets accepted).

If you say that we should figure this version transition out and land it upstream before we land the KVM code, I think that's a fair ask. I can work with that.

But I think we first need to agree on the principles of the compatibility model I described above.

[0] https://lpc.events/event/19/contributions/2049/
[1] https://lore.kernel.org/kexec/[email protected]/T/#u

Quoted 7 lines

I totally understand that exploratory work and initial development is best done in private and/or in a small working groups. But decisions that will significantly impact multiple subsystems need to be made *with* those subystems. I mean, obviously it's possible to make a decision in a small group and then "publish" the result later, but then as is happening here, the community and subsystem maintainers like me may refuse to play ball if they disagree.

The design of live update was discussed and agreed upon in the open. David Rientjes (and now Pasha) has been hosting the live update biweeklies [2] from the very start of the project. The series is open to everyone and notes shared for those who couldn't join. This meeting was attended by a lot of people from different companies and different subsystem backgrounds.

There have also been a microconference at LPC 2025 where topics around live update in PCI, MM, VFIO, IOMMU, etc. were discussed, along with presentations at subsystem tracks in other conferences.

Of course, it is impossible to get all maintainers of all subsystems to attend these sessions or conferences. That's fine.

But I think it is quite unfair to say that live update is developed in private or in small groups.

[2] https://lore.kernel.org/all/?q=s%3A%22%5BHypervisor+Live+Update%5D%22

Signature · 2 lines
Regards,
Pratyush Yadav
#32 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

Quoted 23 lines · Pratyush Yadav

Hi Sean,

Quoted 14 lines · Sean Christopherson
Quoted 5 lines · Pratyush Yadav
Quoted 1 line · Sean Christopherson

On Wed, Aug 12, 2026, Pratyush Yadav wrote:

So with live update, we don't need to keep backwards compatibility in the ABI forever. Of course, it is good to minimize changes, but we have more freedom to change it.

Ah. So this is the heart of the disconnect. I very strongly disagree with the statement that live update doesn't need to support backwards compatibility. I can totally believe that the folks working on live update are ok breaking backwards compatibility because their use cases are "fine" with such breakage. And I can also believe live update as an upstream kernel feature being developed by those same folks is also ok with breaking backwards compatibility.

Quoted 4 lines

But with my upstream KVM maintainer hat on, I am not ok with that. I did not agree to support a world where KVM is allowed to break backwards compatibility, so long as it's done "carefully" or whatever. If y'all want to deal with the resulting complexity, that's fine by me, but you'll be doing it without KVM.

Quoted 5 lines

Let's step back a bit. I don't think backwards compatibility in the _ABI_ is all that important in live update's context. What is important is that our users are able to live update from kernel version X to Y. The line format the kernel uses to describe its state is an implementation detail. Our users never see it.

The rule is thou shalt not break userspace. Whether or not the breakage is the result of an explicit ABI change is irrelevant.

Quoted 17 lines

The high level idea is that when you need to make a change to the ABI so the kernel can better describe the objects/resources/files it is passing, you create a new version of the ABI and allow transitions from older versions.

Say you make some changes and need an ABI version v5. You don't get rid of v4. You keep it around. LUO core will facilitate picking the right version for the next kernel. So it will be possible to seamlessly upgrade your kernel that speaks v4 to a new kernel that speaks v4 and v5. Now this kernel can start speaking v5 if its successor speaks v5 too. And so on for going to v6 and v7, etc.

After a "reasonable" time given for upgrades, you deprecate v4. v4 has been around long enough and our users have had a chance to go to kernels speaking newer versions. That's when you remove the code for v4 from the kernel.

Quoted 8 lines

We can argue what "reasonable" means, but the core idea stays.

It will still be possible to go back to v4 or earlier, but you'd need an extra stop along the way. And of course, vendors can keep a wider support matrix downstream if they see the need for it.

Does this idea of "backwards compatibility" sound acceptable to you, at least at a high level?

No.

It's probably fine for Google and other large companies that tightly control their kernels and use cases, and have the resources to juggle the resulting complexity, e.g. have kernel engineers on staff to track feature and dependencies, coordinate and plan kernel upgrades, etc.

It's not acceptable for upstream, where downstream consumers often run a distro kernel, have much more varied use cases, and don't always have a horde of kernel engineers on staff to help them thread the needle you describe above. And if supporting live update as a general feature for all users of the kernel isn't being factored into design considerations, then that needs to change, otherwise this is all dead in the water.

I also don't see the point. Maintaining a rigid save/restore ABI is annoying, but it's not _hard_ (or at least, not _that_ hard), especially if there's a set of well-documented best known practices that subsystems can follow, e.g. so that individual subsystems don't need to learn painful lessons first-hand. I genuinely believe that maintaining the version hell you describe above would be more costly in the long run than simply committing to full backwards compatibility within a given subsystem. I can imagine that enumerating what subsystems' information is in the payload will require a different scheme, but for a given subsystem, I don't see any reason to aim for anything less than full backwards compatibility.

maintainer
#33 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

Hi Sean,

Quoted 73 lines · Sean Christopherson
Quoted 23 lines · Pratyush Yadav

Hi Sean,

Quoted 14 lines · Sean Christopherson
Quoted 5 lines · Pratyush Yadav
Quoted 1 line · Sean Christopherson

On Wed, Aug 12, 2026, Pratyush Yadav wrote:

So with live update, we don't need to keep backwards compatibility in the ABI forever. Of course, it is good to minimize changes, but we have more freedom to change it.

Ah. So this is the heart of the disconnect. I very strongly disagree with the statement that live update doesn't need to support backwards compatibility. I can totally believe that the folks working on live update are ok breaking backwards compatibility because their use cases are "fine" with such breakage. And I can also believe live update as an upstream kernel feature being developed by those same folks is also ok with breaking backwards compatibility.

Quoted 4 lines

But with my upstream KVM maintainer hat on, I am not ok with that. I did not agree to support a world where KVM is allowed to break backwards compatibility, so long as it's done "carefully" or whatever. If y'all want to deal with the resulting complexity, that's fine by me, but you'll be doing it without KVM.

Quoted 5 lines

Let's step back a bit. I don't think backwards compatibility in the _ABI_ is all that important in live update's context. What is important is that our users are able to live update from kernel version X to Y. The line format the kernel uses to describe its state is an implementation detail. Our users never see it.

The rule is thou shalt not break userspace. Whether or not the breakage is the result of an explicit ABI change is irrelevant.

Quoted 17 lines

The high level idea is that when you need to make a change to the ABI so the kernel can better describe the objects/resources/files it is passing, you create a new version of the ABI and allow transitions from older versions.

Say you make some changes and need an ABI version v5. You don't get rid of v4. You keep it around. LUO core will facilitate picking the right version for the next kernel. So it will be possible to seamlessly upgrade your kernel that speaks v4 to a new kernel that speaks v4 and v5. Now this kernel can start speaking v5 if its successor speaks v5 too. And so on for going to v6 and v7, etc.

After a "reasonable" time given for upgrades, you deprecate v4. v4 has been around long enough and our users have had a chance to go to kernels speaking newer versions. That's when you remove the code for v4 from the kernel.

Quoted 8 lines

We can argue what "reasonable" means, but the core idea stays.

It will still be possible to go back to v4 or earlier, but you'd need an extra stop along the way. And of course, vendors can keep a wider support matrix downstream if they see the need for it.

Does this idea of "backwards compatibility" sound acceptable to you, at least at a high level?

No.

It's probably fine for Google and other large companies that tightly control their kernels and use cases, and have the resources to juggle the resulting complexity, e.g. have kernel engineers on staff to track feature and dependencies, coordinate and plan kernel upgrades, etc.

It's not acceptable for upstream, where downstream consumers often run a distro kernel, have much more varied use cases, and don't always have a horde of kernel engineers on staff to help them thread the needle you describe above. And if supporting live update as a general feature for all users of the kernel isn't being factored into design considerations, then that needs to change, otherwise this is all dead in the water.

Quoted 9 lines

I also don't see the point. Maintaining a rigid save/restore ABI is annoying, but it's not _hard_ (or at least, not _that_ hard), especially if there's a set of well-documented best known practices that subsystems can follow, e.g. so that individual subsystems don't need to learn painful lessons first-hand. I genuinely believe that maintaining the version hell you describe above would be more costly in the long run than simply committing to full backwards compatibility within a given subsystem. I can imagine that enumerating what subsystems' information is in the payload will require a different scheme, but for a given subsystem, I don't see any reason to aim for anything less than full backwards compatibility.

Let's say for argument's sake that we commit for a fully stable backwards compatible ABI. Even then, you have to deal with multiple ABI versions.

Live update's ABI is more complex compared to KVM's save/restore ABI. For the KVM save/restore uAPI, you are largely describing architectural state like CPU registers, etc. These things don't evolve as fast and more or less stay the same.

Live update needs to describe the state of kernel objects. These are more complex and evolve faster.

For example, say you merge guest_memfd preservation today. Some time later, someone comes up with a more efficient data structure to track the folios in the file. You _have_ to make a backwards-incompatible ABI change to use this data structure.

Or say you add a new memory backend (like the HugeTLB patches in flight). That likely will need a different ABI to describe the state of the guest_memfd.

So you will end up with multiple ABI versions that aren't always backwards compatible.

If you refuse that idea too, then KVM live update will be dead in the water for a different reason. It will be damn near useless because it can't keep up with an evolving subsystem.

Now once you get multiple ABI versions and you can seamlessly go from old to new one, say you have a version that was superseded 5 years ago. It would be completely reasonable to say that this version is old enough and no one should be going from a 5 year old kernel to a modern one. So you deprecate this ABI. Deprecating old unused uAPIs is not unprecedented.

I think we are better off formalizing this deprecation period from the get go.

A somewhat tangential example is BPF kfuncs. My BPF program that works in kernel X might not work in kernel Y because the kfunc has changed or been removed.

The argument they make in kfuncs.rst is that kfuncs "provide a kernel <-> kernel API, and thus are not bound by any of the strict stability restrictions associated with kernel <-> user UAPIs". For LUO as well, this is a kernel -> kernel API. Users can also still do a regular kexec or reboot. They just won't get the performance optimization of LUO.

Regardless of if you agree with the last bit about deprecating old versions, ABIs evolving with the subsystem is a ground reality of live update and it would be foolish to think we can do with only backwards-compatible ABI changes forever.

Signature · 2 lines
Regards,
Pratyush Yadav
#34 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

Quoted 30 lines · Pratyush Yadav

Hi Sean,

Quoted 17 lines · Sean Christopherson
Quoted 2 lines

Does this idea of "backwards compatibility" sound acceptable to you, at least at a high level?

No.

It's probably fine for Google and other large companies that tightly control their kernels and use cases, and have the resources to juggle the resulting complexity, e.g. have kernel engineers on staff to track feature and dependencies, coordinate and plan kernel upgrades, etc.

It's not acceptable for upstream, where downstream consumers often run a distro kernel, have much more varied use cases, and don't always have a horde of kernel engineers on staff to help them thread the needle you describe above. And if supporting live update as a general feature for all users of the kernel isn't being factored into design considerations, then that needs to change, otherwise this is all dead in the water.

Quoted 9 lines

I also don't see the point. Maintaining a rigid save/restore ABI is annoying, but it's not _hard_ (or at least, not _that_ hard), especially if there's a set of well-documented best known practices that subsystems can follow, e.g. so that individual subsystems don't need to learn painful lessons first-hand. I genuinely believe that maintaining the version hell you describe above would be more costly in the long run than simply committing to full backwards compatibility within a given subsystem. I can imagine that enumerating what subsystems' information is in the payload will require a different scheme, but for a given subsystem, I don't see any reason to aim for anything less than full backwards compatibility.

Quoted 8 lines

Let's say for argument's sake that we commit for a fully stable backwards compatible ABI. Even then, you have to deal with multiple ABI versions.

Live update's ABI is more complex compared to KVM's save/restore ABI. For the KVM save/restore uAPI, you are largely describing architectural state like CPU registers, etc. These things don't evolve as fast and more or less stay the same.

Right, because nothing meaningful has changed in any architecture in the 20+ years since KVM has provided save/restore support, whereas guest_memfd looks nothing like it did when it was introduced three years ago.

Quoted 2 lines

Live update needs to describe the state of kernel objects. These are more complex

LOL, you might be the first to claim x86 virtualization isn't all that complex.

Quoted 1 line

and evolve faster.

The speed at which things change doesn't automatically mean we shouldn't strive for backwards compatibility. Yes, providing backwards compatibility requires additional care and planning, and over time *might* lead to an ABI that is difficult to maintain. But IMO, that just makes it all the more important to get the design right the first time, not that we shouldn't even try because it's hard.

Quoted 4 lines

For example, say you merge guest_memfd preservation today. Some time later, someone comes up with a more efficient data structure to track the folios in the file. You _have_ to make a backwards-incompatible ABI change to use this data structure.

Only if those details bleed into the ABI/contract. I actually have a concrete KVM (well, virtualization) example for this.

Intel's VMX architecture disallows direct memory accesses to the VMCS, and instead requires software to access the VMCS via dedicated ISA, using architectural encoding numbers to reference VMCS fields. I.e. VMX decouples how data is stored in memory (the data structures) from the ABI/contract with software (VMCS field encodings).

This allows Intel to optimize the data structures to be more efficient and performant for each microarchitecture based on the features and properties of each uarch, all without breaking backwards/forwards compatibility with software.

My favorite esoteric example is AR_BYTES packing. For Haswell, Intel added an optimization in ucode to allow saving/loading segment register state in a single uop (IIRC). The optimization was especially valuable for virtualization as it shaved cycles off the VM-Enter/VM-Exit hot paths. A key piece of the optimization was it required the AR_BYTES metadata to be stored in 16 bits, but existing CPUs stored AR_BYTES using 32 bits in an "unpacked" format.

Fortunately, because the in-memory representation was decoupled from the contract with software, Intel could pack AR_BYTES into 16 bits for Haswell+ and pack/unpack the data on VMWRITE/VMREAD, so that the format presented to software remained unchanged.

Does VMX's decoupling of the in-memory represntation of a VMCS have downsides? Absolutely. Most notably, it incurs extra complexity (in software and hardware) to achieve comparable performance to directly accessible data structures (AMD's VMCB and Hyper-V's eVMCS) for nested virtualization. And I'm sure it has placed contraints on Intel's designs, and obviously introduces complexity into the overall system by adding a layer of indirection. But IMO the VMX architecture has been a huge win overall for Intel.

Quoted 6 lines

Or say you add a new memory backend (like the HugeTLB patches in flight). That likely will need a different ABI to describe the state of the guest_memfd.

So you will end up with multiple ABI versions that aren't always backwards compatible.

No, you end up with *features* that aren't backwards compatible. I can't imagine anyone will argue that we should never add new features because then we can't rollback to an older kernel.

But adding a new feature shouldn't break the existing ABI. E.g. adding support for HugeTLB in guest_memfd shouldn't prevent rolling back to an older kernel when the HugeTLB functionality isn't being used.

Using AMD's VMCB and Intel's VMCS as examples, literally every major new AMD/Intel uarch extends the VMC{B,S} in some way, but without fail it's always done in a way that is backwards compatible with existing software. I.e. AMD and Intel ship new features, but existing software continues to work, and VMs continue to be migratable across CPU generations[*], with the obvious restriction that migrating a VM using a feature introduced on generation N to a generation N-1 CPU isn't a smart idea.

[*] There are exceptions. E.g. Intel removed MPX, and so VMs with MPX can't be
migrated to newer CPUs. Migrating between CPUs with different MAXPHYADDR is sketchy (and simply not done by some CSPs) because neither AMD nor Intel virtualizes MAXPHYADDR. But those exceptions are absolutely Big Deals that undergo significant scrutiny, from all parties involved.

Quoted 8 lines

If you refuse that idea too, then KVM live update will be dead in the water for a different reason. It will be damn near useless because it can't keep up with an evolving subsystem.

Now once you get multiple ABI versions and you can seamlessly go from old to new one, say you have a version that was superseded 5 years ago. It would be completely reasonable to say that this version is old enough and no one should be going from a 5 year old kernel to a modern one.

LOL, Google literally does this. Granted, the extreme cases only happen for stragglers, and I think we do force VMs to bounce through a "middle" kernel in those cases, but I doubt Google is the only company that runs frankenkernels for an absurd number of years for a variety of reasons. E.g. 4.4 LTS was officially supported for 6 years, and I'll bet dollars to donuts people ran it for much longer than that.

Quoted 1 line

So you deprecate this ABI. Deprecating old unused uAPIs is not unprecedented.

When there are provably no users, or we can convince existing userspace to migrate to an alternative.

Quoted 2 lines

I think we are better off formalizing this deprecation period from the get go.

Why? What does it buy us? Because all I see is potential abuse and an excuse for not spending time getting the designs right.

Quoted 7 lines

A somewhat tangential example is BPF kfuncs. My BPF program that works in kernel X might not work in kernel Y because the kfunc has changed or been removed.

The argument they make in kfuncs.rst is that kfuncs "provide a kernel <-> kernel API, and thus are not bound by any of the strict stability restrictions associated with kernel <-> user UAPIs".

BPF's documentation isn't arguing anything, it's merely reiterating Linux's long-standing policy that there is no such thing as a stable kernel ABI (in upstream).

Quoted 1 line

For LUO as well, this is a kernel -> kernel API.

Stating the obvious, I disagree with this. As I said before, if this is the stance LUO wants to take, then so be it, but my NAK stands.

Quoted 2 lines

Users can also still do a regular kexec or reboot. They just won't get the performance optimization of LUO.

The amount of time, energy, and money poured into minimizing VM downtime on live migration suggests the overwhelming majority of LUO's targeted users aren't going to take kindly to this stance.

Quoted 4 lines

Regardless of if you agree with the last bit about deprecating old versions, ABIs evolving with the subsystem is a ground reality of live update and it would be foolish to think we can do with only backwards-compatible ABI changes forever.

I never said the ABI is immutable, I said it needs to be backwards/forwards compatible.

#35 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

Quoted 173 lines · Sean Christopherson
Quoted 30 lines · Pratyush Yadav

Hi Sean,

Quoted 17 lines · Sean Christopherson
Quoted 2 lines

Does this idea of "backwards compatibility" sound acceptable to you, at least at a high level?

No.

It's probably fine for Google and other large companies that tightly control their kernels and use cases, and have the resources to juggle the resulting complexity, e.g. have kernel engineers on staff to track feature and dependencies, coordinate and plan kernel upgrades, etc.

It's not acceptable for upstream, where downstream consumers often run a distro kernel, have much more varied use cases, and don't always have a horde of kernel engineers on staff to help them thread the needle you describe above. And if supporting live update as a general feature for all users of the kernel isn't being factored into design considerations, then that needs to change, otherwise this is all dead in the water.

Quoted 9 lines

I also don't see the point. Maintaining a rigid save/restore ABI is annoying, but it's not _hard_ (or at least, not _that_ hard), especially if there's a set of well-documented best known practices that subsystems can follow, e.g. so that individual subsystems don't need to learn painful lessons first-hand. I genuinely believe that maintaining the version hell you describe above would be more costly in the long run than simply committing to full backwards compatibility within a given subsystem. I can imagine that enumerating what subsystems' information is in the payload will require a different scheme, but for a given subsystem, I don't see any reason to aim for anything less than full backwards compatibility.

Quoted 8 lines

Let's say for argument's sake that we commit for a fully stable backwards compatible ABI. Even then, you have to deal with multiple ABI versions.

Live update's ABI is more complex compared to KVM's save/restore ABI. For the KVM save/restore uAPI, you are largely describing architectural state like CPU registers, etc. These things don't evolve as fast and more or less stay the same.

Right, because nothing meaningful has changed in any architecture in the 20+ years since KVM has provided save/restore support, whereas guest_memfd looks nothing like it did when it was introduced three years ago.

Quoted 2 lines

Live update needs to describe the state of kernel objects. These are more complex

LOL, you might be the first to claim x86 virtualization isn't all that complex.

Quoted 1 line

and evolve faster.

The speed at which things change doesn't automatically mean we shouldn't strive for backwards compatibility. Yes, providing backwards compatibility requires additional care and planning, and over time *might* lead to an ABI that is difficult to maintain. But IMO, that just makes it all the more important to get the design right the first time, not that we shouldn't even try because it's hard.

Quoted 4 lines

For example, say you merge guest_memfd preservation today. Some time later, someone comes up with a more efficient data structure to track the folios in the file. You _have_ to make a backwards-incompatible ABI change to use this data structure.

Only if those details bleed into the ABI/contract. I actually have a concrete KVM (well, virtualization) example for this.

Intel's VMX architecture disallows direct memory accesses to the VMCS, and instead requires software to access the VMCS via dedicated ISA, using architectural encoding numbers to reference VMCS fields. I.e. VMX decouples how data is stored in memory (the data structures) from the ABI/contract with software (VMCS field encodings).

This allows Intel to optimize the data structures to be more efficient and performant for each microarchitecture based on the features and properties of each uarch, all without breaking backwards/forwards compatibility with software.

My favorite esoteric example is AR_BYTES packing. For Haswell, Intel added an optimization in ucode to allow saving/loading segment register state in a single uop (IIRC). The optimization was especially valuable for virtualization as it shaved cycles off the VM-Enter/VM-Exit hot paths. A key piece of the optimization was it required the AR_BYTES metadata to be stored in 16 bits, but existing CPUs stored AR_BYTES using 32 bits in an "unpacked" format.

Fortunately, because the in-memory representation was decoupled from the contract with software, Intel could pack AR_BYTES into 16 bits for Haswell+ and pack/unpack the data on VMWRITE/VMREAD, so that the format presented to software remained unchanged.

Does VMX's decoupling of the in-memory represntation of a VMCS have downsides? Absolutely. Most notably, it incurs extra complexity (in software and hardware) to achieve comparable performance to directly accessible data structures (AMD's VMCB and Hyper-V's eVMCS) for nested virtualization. And I'm sure it has placed contraints on Intel's designs, and obviously introduces complexity into the overall system by adding a layer of indirection. But IMO the VMX architecture has been a huge win overall for Intel.

Quoted 6 lines

Or say you add a new memory backend (like the HugeTLB patches in flight). That likely will need a different ABI to describe the state of the guest_memfd.

So you will end up with multiple ABI versions that aren't always backwards compatible.

No, you end up with *features* that aren't backwards compatible. I can't imagine anyone will argue that we should never add new features because then we can't rollback to an older kernel.

But adding a new feature shouldn't break the existing ABI. E.g. adding support for HugeTLB in guest_memfd shouldn't prevent rolling back to an older kernel when the HugeTLB functionality isn't being used.

Using AMD's VMCB and Intel's VMCS as examples, literally every major new AMD/Intel uarch extends the VMC{B,S} in some way, but without fail it's always done in a way that is backwards compatible with existing software. I.e. AMD and Intel ship new features, but existing software continues to work, and VMs continue to be migratable across CPU generations[*], with the obvious restriction that migrating a VM using a feature introduced on generation N to a generation N-1 CPU isn't a smart idea.

[*] There are exceptions. E.g. Intel removed MPX, and so VMs with MPX can't be
migrated to newer CPUs. Migrating between CPUs with different MAXPHYADDR is sketchy (and simply not done by some CSPs) because neither AMD nor Intel virtualizes MAXPHYADDR. But those exceptions are absolutely Big Deals that undergo significant scrutiny, from all parties involved.

Quoted 8 lines

If you refuse that idea too, then KVM live update will be dead in the water for a different reason. It will be damn near useless because it can't keep up with an evolving subsystem.

Now once you get multiple ABI versions and you can seamlessly go from old to new one, say you have a version that was superseded 5 years ago. It would be completely reasonable to say that this version is old enough and no one should be going from a 5 year old kernel to a modern one.

LOL, Google literally does this. Granted, the extreme cases only happen for stragglers, and I think we do force VMs to bounce through a "middle" kernel in those cases, but I doubt Google is the only company that runs frankenkernels for an absurd number of years for a variety of reasons. E.g. 4.4 LTS was officially supported for 6 years, and I'll bet dollars to donuts people ran it for much longer than that.

Quoted 1 line

So you deprecate this ABI. Deprecating old unused uAPIs is not unprecedented.

When there are provably no users, or we can convince existing userspace to migrate to an alternative.

Quoted 2 lines

I think we are better off formalizing this deprecation period from the get go.

Why? What does it buy us? Because all I see is potential abuse and an excuse for not spending time getting the designs right.

Quoted 7 lines

A somewhat tangential example is BPF kfuncs. My BPF program that works in kernel X might not work in kernel Y because the kfunc has changed or been removed.

The argument they make in kfuncs.rst is that kfuncs "provide a kernel <-> kernel API, and thus are not bound by any of the strict stability restrictions associated with kernel <-> user UAPIs".

BPF's documentation isn't arguing anything, it's merely reiterating Linux's long-standing policy that there is no such thing as a stable kernel ABI (in upstream).

Quoted 1 line

For LUO as well, this is a kernel -> kernel API.

Stating the obvious, I disagree with this. As I said before, if this is the stance LUO wants to take, then so be it, but my NAK stands.

Quoted 2 lines

Users can also still do a regular kexec or reboot. They just won't get the performance optimization of LUO.

The amount of time, energy, and money poured into minimizing VM downtime on live migration suggests the overwhelming majority of LUO's targeted users aren't going to take kindly to this stance.

Quoted 7 lines
Quoted 4 lines

Regardless of if you agree with the last bit about deprecating old versions, ABIs evolving with the subsystem is a ground reality of live update and it would be foolish to think we can do with only backwards-compatible ABI changes forever.

I never said the ABI is immutable, I said it needs to be backwards/forwards compatible.

Thanks Sean, Pratyush. @loganodell has recently sent the RFC[1] on backward compatibility for liveupdate. This discussion is important so I propose to move the backward compatibility discussion on [1].

For guest_memfd preservation, I will sent v5 soon with the following changes:
1. Suggestions from across the patches
2. tentative Plan for future Guest_memfd preservation
3. tentative Plan for VM preservation.
4. Backward compatibility built on top of RFC [1]

What do you think?

[1]: https://lore.kernel.org/all/[email protected]/

#36 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

In reply to Sean Christopherson
Quoted 24 lines · Sean Christopherson
Quoted 23 lines · Pratyush Yadav

Hi Sean,

Quoted 18 lines · Sean Christopherson
Quoted 5 lines · Pratyush Yadav
Quoted 1 line · Sean Christopherson

On Wed, Aug 12, 2026, Pratyush Yadav wrote:

So with live update, we don't need to keep backwards compatibility in the ABI forever. Of course, it is good to minimize changes, but we have more freedom to change it.

Ah. So this is the heart of the disconnect. I very strongly disagree with the statement that live update doesn't need to support backwards compatibility. I can totally believe that the folks working on live update are ok breaking backwards compatibility because their use cases are "fine" with such breakage. And I can also believe live update as an upstream kernel feature being developed by those same folks is also ok with breaking backwards compatibility.

But with my upstream KVM maintainer hat on, I am not ok with that. I did not agree to support a world where KVM is allowed to break backwards compatibility, so long as it's done "carefully" or whatever. If y'all want to deal with the resulting complexity, that's fine by me, but you'll be doing it without KVM.

Quoted 8 lines
Quoted 5 lines

Let's step back a bit. I don't think backwards compatibility in the _ABI_ is all that important in live update's context. What is important is that our users are able to live update from kernel version X to Y. The line format the kernel uses to describe its state is an implementation detail. Our users never see it.

The rule is thou shalt not break userspace. Whether or not the breakage is the result of an explicit ABI change is irrelevant.

I argued strongly for this simpler approach, I will write here the reasoning I gave.

Firstly, "live update" is not uABI per-say. It is not "userspace breaking", and it is not a "regression". It is an internal kernel mechanism to allow the kernel to self-upgrade. In the industry it is typical any of these hitless update/patch/etc schemes to only work between a few tightly controlled version pairs.

Asking upstream to carry a full matrix of every single version pair ever released is massive over-engineering and cost on upstream maintainers. Refusing to do this is not a uABI breakage, it is not a regression, it is simply a lack of a feature.

I think upstream should start by agreeing to only support forward going upgrades within a single stable branch. If live update becomes a success then maybe we could do from a stable branch to the immediate next stable branch too. I don't know.

This alone is already much more powerful than live patching.

The balancing issue here is the impact on the kernel subsystems adopting luo. Every time you change anything about the internal function of the subystem you now have to go test a massive version pair matrix to make sure everything works? No thanks.

Luo is very invasive and a huge PITA at the best of times. It doesn't need to be even worse.

Quoted 5 lines

It's probably fine for Google and other large companies that tightly control their kernels and use cases, and have the resources to juggle the resulting complexity, e.g. have kernel engineers on staff to track feature and dependencies, coordinate and plan kernel upgrades, etc.

Every downstream that wants to support luo is going to necessarily severely restrict the version pairs that can work. Like a RH type distro may only support it for 9.1.x -> 9.1.x+1 - they can carry the cost of figuring out how to manage their patching and compatability matrix on their own with the tools upstream provides.

Quoted 4 lines

engineers on staff to help them thread the needle you describe above. And if supporting live update as a general feature for all users of the kernel isn't being factored into design considerations, then that needs to change, otherwise this is all dead in the water.

General uses can use upstream and do CVE upgrades within a single stable branch. That's good enough, lets start there. We don't need to boil the ocean.

Quoted 3 lines

I also don't see the point. Maintaining a rigid save/restore ABI is annoying, but it's not _hard_ (or at least, not _that_ hard), especially if there's a set

I was told KVM had the smallest luo footprint of everything, so perhaps your perspective is different.

In other places luo becomes coupled to the internal datastructures of the kernel, we literally have to preserve lots of kernel memory utterly unchanged without any kind of serialization at all. This inevitably creates restrictions on the evolution of the kernel in general if we can no longer change in-kernel architecture because it would make some data structure incompatible with a kernel 10 years old.

There is also a need for *downgrade*. Meaning the N+1 kernel has to strictly emulate the limitations and capabilities of the N kernel so it can be rolled back. Can you imagine the nightmare of trying to codify the feature progression for every single kernel release forever in some impossible scheme to enable this? Again, no thanks.

So, what upstream can reasonably do, that still provides real value without severely burdening every subsystem, is upgrades within a stable kernel branch only.

If some distro or CSP wants to do more, they can deal with it. They have a far, far simpler problem because they can exactly lock down their version pairs to a very small universe. They don't need a single kernel that can emulate 10 different releases of behaviors concurrently.

Jason

#37 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

Quoted 18 lines · Jason Gunthorpe
Quoted 2 lines · Sean Christopherson

The rule is thou shalt not break userspace. Whether or not the breakage is the result of an explicit ABI change is irrelevant.

I argued strongly for this simpler approach, I will write here the reasoning I gave.

Firstly, "live update" is not uABI per-say. It is not "userspace breaking", and it is not a "regression". It is an internal kernel mechanism to allow the kernel to self-upgrade. In the industry it is typical any of these hitless update/patch/etc schemes to only work between a few tightly controlled version pairs.

Asking upstream to carry a full matrix of every single version pair ever released is massive over-engineering and cost on upstream maintainers. Refusing to do this is not a uABI breakage, it is not a regression, it is simply a lack of a feature.

Quoted 4 lines

I think upstream should start by agreeing to only support forward going upgrades within a single stable branch. If live update becomes a success then maybe we could do from a stable branch to the immediate next stable branch too. I don't know.

From that perspective, what I am proposing actually goes a step further. I'm saying don't commit to supporting *any* specific versions in upstream. Express the feature requirements in the serialization payloads, and let userspace sort out what kernels are compatible based on their actual usage. The kernel may need to provide additional discovery mechanisms, e.g. so that userspace can probe to see what is supported, but discovery is usually fairly simple to implement and maintain.

Quoted 6 lines

This alone is already much more powerful than live patching.

The balancing issue here is the impact on the kernel subsystems adopting luo. Every time you change anything about the internal function of the subystem you now have to go test a massive version pair matrix to make sure everything works? No thanks.

No, the subsystem just needs to make sure that it serializes its data using the defined ABI (where ABI here means the format of the payload and the meaning of any flags in the header). That should be *easier* for maintainers to handle than trying to support arbitrary versions, because it eliminates subjectivity and having to make judgment calls or remember magic version numbers.

Quoted 15 lines

Luo is very invasive and a huge PITA at the best of times. It doesn't need to be even worse.

Quoted 5 lines

It's probably fine for Google and other large companies that tightly control their kernels and use cases, and have the resources to juggle the resulting complexity, e.g. have kernel engineers on staff to track feature and dependencies, coordinate and plan kernel upgrades, etc.

Every downstream that wants to support luo is going to necessarily severely restrict the version pairs that can work. Like a RH type distro may only support it for 9.1.x -> 9.1.x+1 - they can carry the cost of figuring out how to manage their patching and compatability matrix on their own with the tools upstream provides.

Quoted 8 lines
Quoted 4 lines

engineers on staff to help them thread the needle you describe above. And if supporting live update as a general feature for all users of the kernel isn't being factored into design considerations, then that needs to change, otherwise this is all dead in the water.

General uses can use upstream and do CVE upgrades within a single stable branch. That's good enough, lets start there. We don't need to boil the ocean.

We're in violent agreement on this point, I think we just disagree on how to express compatibility.

Quoted 6 lines
Quoted 3 lines

I also don't see the point. Maintaining a rigid save/restore ABI is annoying, but it's not _hard_ (or at least, not _that_ hard), especially if there's a set

I was told KVM had the smallest luo footprint of everything, so perhaps your perspective is different.

Only because KVM already has a massive ABI surface for save/restore. If you want to convince me that magic version numbers are the right approach, then show me how KVM's existing save/restore support would be made "better" and easier to maintain by throwing away all of KVM's save/restore uAPI and replacing it with a versioning scheme.

Quoted 3 lines

In other places luo becomes coupled to the internal datastructures of the kernel, we literally have to preserve lots of kernel memory utterly unchanged without any kind of serialization at all.

Have to, or choose to? I have a very, very hard time believing that it's infeasible to define a serialization format that is decoupled from kernel internals.

Quoted 3 lines

This inevitably creates restrictions on the evolution of the kernel in general if we can no longer change in-kernel architecture because it would make some data structure incompatible with a kernel 10 years old.

Only if the relevant subsystem defined a poor save/restore ABI in the first place.

Quoted 17 lines

There is also a need for *downgrade*. Meaning the N+1 kernel has to strictly emulate the limitations and capabilities of the N kernel so it can be rolled back. Can you imagine the nightmare of trying to codify the feature progression for every single kernel release forever in some impossible scheme to enable this? Again, no thanks.

So, what upstream can reasonably do, that still provides real value without severely burdening every subsystem, is upgrades within a stable kernel branch only.

If some distro or CSP wants to do more, they can deal with it. They have a far, far simpler problem because they can exactly lock down their version pairs to a very small universe. They don't need a single kernel that can emulate 10 different releases of behaviors concurrently.

Jason

#38 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

Quoted 7 lines · Sean Christopherson

From that perspective, what I am proposing actually goes a step further. I'm saying don't commit to supporting *any* specific versions in upstream. Express the feature requirements in the serialization payloads, and let userspace sort out what kernels are compatible based on their actual usage. The kernel may need to provide additional discovery mechanisms, e.g. so that userspace can probe to see what is supported, but discovery is usually fairly simple to implement and maintain.

I'm not so fussed about the version number scheme itself. Alot of different schemes have been proposed over the last year and half, and some were very complicated. Nobody came with a more granular proposal that also didn't have alot of complexity attached to it.

FWWI this started out with creating device tree fragments with a full YAML schema, and the same kind of perfect ABI like you are talking about. Yet it did not figure out how to make it discoverable, it was super inefficient and very hard to code for.

It has to be discoverable from the ELF, restrictable on the export side, easy on maintainers, and not complex to implement.

Quoted 5 lines

No, the subsystem just needs to make sure that it serializes its data using the defined ABI (where ABI here means the format of the payload and the meaning of any flags in the header). That should be *easier* for maintainers to handle than trying to support arbitrary versions, because it eliminates subjectivity and having to make judgment calls or remember magic version numbers.

I'm pretty sure I have something like PTSD from ABI definitions adventures on the uAPI side. :( Please don't call it easier!

I really don't want more of those in the kernel process. I think Linus's non-stable-api-nonsense is really a good thing for community health.

With the simplification that upstream supports only one version at once, a simple "id" to represent that ABI was the simplest, easiest on maintainers thing. There is never a fight. Someone has a new idea, great no worries, change the ID. Done.

I guess I should say my perspective is to prioritize not burdening the maintainers.

Quoted 4 lines
Quoted 2 lines

I was told KVM had the smallest luo footprint of everything, so perhaps your perspective is different.

Only because KVM already has a massive ABI surface for save/restore.

Yeah, you are lucky, other subsystems haven't done that. Honestly, was it easy to create?

Quoted 4 lines

If you want to convince me that magic version numbers are the right approach, then show me how KVM's existing save/restore support would be made "better" and easier to maintain by throwing away all of KVM's save/restore uAPI and replacing it with a versioning scheme.

I don't know about KVM, but how do I manage something like serializing an iommu page table?

I'm replacing all the iommu page table code. It behaves differently. It supports different things that old kernels don't understand. iommu page tables have to be under continuous active DMA during kexec, so they cannot be serialized.

Setting the old stuff as V1 and the new suff as V2 is so easy and is unconditionally correct. To my point about supporting only stable branches this achieves it with minimal maintainer effort.

Yes, we could do some comprehensive analysis and try to determine everything that changed and make micro feature bits, and some thing to map the current layout to those bits and then hope all of this is correct.. But that's *a lot* of work, probably will have gaps since it will never be tested. Seriously, why bother? Explain to me why I should spend my time on this and who benifits?

IMHO your KVM example has a much clearer answer to that question. It is UAPI so you have to do it anyhow, and you have a robust open source ecosystem with alot of VMMs that will consume it. Yeah, I'm on board, makes sense.

Quoted 3 lines

Have to, or choose to? I have a very, very hard time believing that it's infeasible to define a serialization format that is decoupled from kernel internals.

Reduced perfectly everything should become grounded in either HW or kernel UAPI definitions. Things are not perfect, the kernel has limitations, it doesn't support every HW feature, stuff leaks in.. Like my page table example, the new stuff supports ARM CONT, the old stuff does not. That is kexec ABI breaking and is only happening because of kernel internal details. In the page table work alone there are probably ~10-20 micro features like that that would have to be identified and delt with. Frankly I'm not confident I could even capture them all.

Also there is the push to make kexec downtime lower. That puts pressure to just retain kernel things exactly as is, and maybe even keep kernel memory as-is. ie a sloppy job serializing to get better performance.

Quoted 1 line

Only if the relevant subsystem defined a poor save/restore ABI in the first place.

I think KVM is the only subsystem that has save/restore :)

Jason

#39 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

In reply to Pratyush Yadav
Quoted 15 lines · Pratyush Yadav
Quoted 2 lines · Sean Christopherson
Quoted 1 line · Pratyush Yadav

On Tue, Aug 11 2026, Sean Christopherson wrote:

[...]

Quoted 9 lines

But that's all beside the point. What I'm saying is that SNP and TDX *must* use guest_memfd, whereas guest_memfd is optional for all other VM types. And so adding LUO support for guest_memfd without even sketching out a plan for CoCo VMs feels backwards.

I'm not necessarily opposed to gradual guest_memfd support, but there needs to be a clear plan of how all of this is going to fit together. It doesn't need to be perfect, and I'm sure we'll make mistakes along the way, but I want to at least try not to paint ourselves into a corner, especially with respect to the ABI.

Quoted 8 lines

Right. I think landing shared guest_memfd is the first step. It is simple enough and doesn't need to preserve the more complicated architecture specific things like secure EPT, etc. The CoCo support can build on top of this.

There is work underway already for an RFC of TDX live update, but Tarun knows the details better than me so I'll let him fill in the details for the next version of this series.

Right, I am keeping things minimal for this version just addressing the non-cocovm guest_memfd to be preserved. On top of it, We will buidl support for the cocoVM preservation. And Sagi ([email protected]) is already working on it.

Why not implement both in the same series:
1. This series focus on finalizing the design, How guest_memfd will be preserved, So Any user (Not now) but in future will use guest_memfd in non cocoVM case, This alone feature will be used.

2. guest_memfd cocoVM preservation support will be built on top of in-place conversion series and I preferred not to complicate two topics in single series, Instead to develop gradually.

I agree to your answer for have Borader Roadmap. I tried to answer in Cover-letter. Which I am re-stating it here:

This series support guest_memfd preservation for non-COCOVM only. cocoVM Support will be built on top of that. Which includes the following stages.
non cocoVM guest_memfd preservation -> TDX preservation -> in-place convertable private guest_memfd preservation -> in-place convertable private guest_memfd preservation backed by hugetlb.

End Goal is to support guest_memfd preservation on cocoVM backed by hugetlb.

The question remains:
Why we are preserving KVM and only vm_type there.

I will try to answer this on the thread of VM preservation to not loose the context.

Quoted 54 lines
Quoted 42 lines
Quoted 24 lines
Quoted 24 lines
Quoted 9 lines
Quoted 5 lines

E.g. the LUO APIs seem pretty straightforward; I assume the bulk of the complexity is going to be in knowing what to save/restore, and how, which is much more about KVM than it is about liveupdate.

I think it is fine if you want to take these changes through the KVM tree, but I would like live update maintainers to be listed as reviewers at least.

Why not simply add a file pattern match to the LIVE UPDATE entry?

MAINTAINERS +1 -1
index 8014b9f8253e..2eb57b22c37f 100644
@@ -15052,8 +15052,8 @@ F:      include/linux/liveupdate.h
 F:     include/uapi/linux/liveupdate.h
 F:     kernel/liveupdate/
 F:     lib/tests/liveupdate.c
-F:     mm/memfd_luo.c
 F:     tools/testing/selftests/liveupdate/
+N:     [^a-z]luo
 
Quoted 4 lines

MAINTAINERS F: include/linux/liveupdate.h · line 15059

Quoted 2 lines
MAINTAINERS +0 -0
 LLC (802.2)
 L:     [email protected]

This would list us as maintainers of kvm_luo.c and the tree as liveupdate.git,

It would list both "LIVE UPDATE" and "KERNEL VIRTUAL MACHINE (KVM)", as KVM would
still cover the files via "F: virt/kvm/*". And my read of MAINTAINERS is that
N: and K: entries are "secondary" if the files/scope is covered by an F: entry.

E.g. arch/x86/kvm/vmx/tdx.c is covered by "KERNEL VIRTUAL MACHINE FOR X86 (KVM/x86)"
via "F: arch/x86/kvm/*/", and also by "X86 TRUST DOMAIN EXTENSIONS (TDX)" via
"N: tdx" (or maybe "K: \b(tdx)"? I haven't bothered to check which one
triggers). And while I don't think there was ever any formal discussion, AFAIK everyone reads the situation as KVM still being the primary maintainer/tree for that code.

Quoted 1 line

both of which is something you're saying you _don't_ want.

Quoted 8 lines

No, what I don't want is a dedicated "KVM LIVE UPDATE" entry, because I think most
people would read "F: virt/kvm/kvm_luo.c" and "F: virt/kvm/guest_memfd_luo.c"
as being more precise than KVM's "F: virt/kvm/*" and thus would read things as
"KVM LIVE UPDATE" being the primary maintainer.

In other words, I'm more than ok with LUO being looped in on KVM LUO changes and having the authority to object to problematic changes, but I'm not ok with LUO taking primary ownership of KVM code.

Sure, that sounds good to me.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

I realize there's more than a bit of nuance in my interpretation of N: and K:, but again my experience with TDX is that so long as the maintainers are aligned on expectations, it's a non-issue in practice.

Yep, as long as we agree among ourselves I don't think the red tape around which entry has higher priority matters much.

Agree!

Quoted 57 lines
Quoted 33 lines
Quoted 23 lines
Quoted 11 lines
Quoted 9 lines

At the same time, I also keep being (pleasantly)
surprised at preservation being relatively simple. For example, the code to preserve a shmem file (via memfd) is roughly 600 lines, a big chunk of which is comments. The code of course has some limitations, but it is good enough for use in production.

For one, we care about ABI breakages and versioning.

Which is amusing to me because that implies KVM does not, and I would hazard to

No, it doesn't. What I'm saying is I care about changes to the _live update ABI_. Just like you probably care about changes to KVM ABI but not so much about BPF for example.

Quoted 7 lines

guess that KVM has the biggest ABI surface of any subsystem in the kernel by a country mile (though I'm probably wildly underestimating the effective ABI surface of filesystems).

Quoted 2 lines

The serialized state is a part of live update ABI and changes to it should be ACKed by us.

Quoted 7 lines
Quoted 2 lines

Meh, "Don't break userspace" is a universal rule in the kernel, I genuinely don't see why liveupdate needs special treatment.

Ironically enough, you miss my point. I'm not talking about userspace ABI. We all know not to break that. I am talking about live update _serialization ABI_. See the stuff under include/linux/kho/abi. This series also adds things there.

No, I understand exactly what ABI you're talking about.

Quoted 8 lines
Quoted 3 lines

This is ABI between kernels. It needs to be stable-ish so you can move from one kernel version to another. At the same time, unlike userspace ABI, it can change.

Uh, yeah, so KVM has been managing such immutable ABI for practically its entire existence. KVM's save/restore uAPI has exactly what you're describing: serialization ABI that needs to be backwards and forwards compatible between different kernels in order to support both upgrade and rollback scenarios via live migration.

I think there is a slight difference between KVM's save/resture uAPI and live update's ABI. With KVM's uAPI, you need to maintain strict backwards compatibility because userspace reads what you output. So if you change the layout, userspace will interpret it wrong and might break.

With live update, the ABI never gets to userspace. It is used to talk between kernels directly. So if you do break that, your userspace keeps working fine, you just might not be able to live update to the incompatible kernel.

So with live update, we don't need to keep backwards compatibility in the ABI forever. Of course, it is good to minimize changes, but we have more freedom to change it.

Quoted 7 lines

The current idea is we change the ABI whenever needed. This is because live update is still in development so we are likely to see a lot more changes before we settle down into something that mostly works. At some point, we want to start being more strict between incompatible ABI changes. The current idea is that we would like to keep backwards compatibility across LTS kernels. But none of that is set in stone yet.

True, I also understand Sean's Concern. I will try to explain why bacward compatibility with padding does not solve the problem here.

Kernel V1 -> Kernel V2
guest_memfd_luo ABI v1 -> guest_memfd_luo ABI v2

(No Userspace involvement here: userspace only determine if the upcoming kernel V2 can support the V1 preserved files, if return true: liveupdate()
elif: no_liveupdate())

When Kernel V1, preserve the data, it has preserved based on ABI v1. then serializes it to make it transferable to next kernel via fdt.

Now Kernel V2 boots up, and it needs to deserialize the data and restore the file which was preserved before. But this kernel V2 expect guest_memfd preservation ABI V2. These ABI V2 might be because of multiple reasons, for e.g.
1. A new feature in guest_memfd_luo (preserving more flags, or the private/shared state of folios etc)

2. A new feature in guest_memfd itself, guest_memfd kernel APIs (guest_memfd_create(..., new_args) or a new init call for guest_memfd after guest_memfd_create(), or an older field/flog is dropped). So new kernel should be able to retrieve old guest_memfd and populate in new guest_memfd for backward compatibility.

Now, What will happen to these new fields or arguments that are needed, Kernel V2->guest_memfd_luo V2 will take care of setting them to default.

Is setting them to default acceptable? Not always? So that is where the negotiation comes in picture: Kernel V2 should know if it can support retrieving kernel V1 or otherwise userspace must not transition to V2 using liveupdate if V2 does not have compatibility with V1.

Now to support this backward compatibility, the retrieval APIs and deserialization APIs in V2 kernel should be able to provide the deserialization/retrieval for V1 preserved data.

Kernel V1 -> serialize data ---kexec ----> kernel V2 -> deserialization -> retrieve

Now deserialization/retrival is aware of V1/V2 comptatibility then it can set the new fields, parse the preserved data of V1 into V2 (or directly give it to guest_memfd APIs with setting any new expected fields to default.)

Now having a padding fields in kernel guest_memfd_luo ABIs does not help here. As support for deserialization and parsing from V1-> V2 in kernel V2 will determine if kernel V2 is backward compatible with V1. which can also take care of setting the extra needed fields in V2 to default value or remove the unwanted flags/fields not needed in kernel V2.

Now Questions is, if there isa need for rollback to previous kernel (V2 -> V1)?

There are two options,
1. kernel V2 is aware that it is going to get kexec to V1. then serialization function on preservation will serialize the data in guest_memfd_luo ABIs V1 compatible format.

2. kernel V1 guest_memfd_luo ABIs had the padding in their ABIs structures which even if kernel V2 will set those field, will be ignored by kernel V1. This will not require saperate deserialization/preservation mechanism for guest_memfd_luo V1 in kernel V2. This will not work in every case, As there might be different parsers are in guest_memfd_luo V2 and V1. then serialized data preserved by guest_memfd_luo V2 can not be parsed by guest_memfd_luo V1 after kexec.

So,

Kernel V2 _must have_ to support backward compatibility with kernel V1:
1. guest_memfd_luo V1 context aware deserialization/retrieval mechanism
2. guest_memfd_luo V1 context aware serialization/preservation mechanism

Kernel V1/V2 may have:
1. padding (hard to determine, can have heuristically) to ease the job of deserialization/retrieval/serialization/preservation apis. But does not solve backward compatibility alone.

Once we will keep updating the host, There will be a point when we are at V15 but we dont want V1 backward compatiblity anymore, So we drop the deserialization/retrieval/serialization/preservation for V1 from V15.

Would love to hear your thoughts?

Looping Logan ([email protected]) for his insights on this as well.

~Tarun

Quoted 77 lines

That's why I would like to have an explicit ACK from the live update group for ABI changes. I would like to enforce these compatibility windows.

Quoted 20 lines

KVM also has immutable ABI between itself and guest kernels, including implicit "ABI" in the form of not changing guest-visible behavior.

Quoted 7 lines

Today we don't have any rules and let you change things freely as long as you do a version bump. But at a later point, the plan is to add some stability requirements to the ABI so you can actually upgrade the kernel across major versions.

So at least for ABI changes, there should be an explicit ACK from the live update group.

I don't entirely agree. I get where you're coming from, and I 100% agree that the more eyeballs on changes that may affect ABI, the better.

Where I have problems with the above is that it doesn't account for the nuances of save/restore across different kernels. The literal format of the serialized data is the most obvious form of ABI, but it's absolutely possible to change ABI without changing the data format.

Quoted 6 lines

E.g. say there's a flags field in some serialization structure, and flags X and Y are mutually exclusive in current kernels. If a future kernel relaxes that restriction for whatever reason, then the effective ABI has been broken because state created and saved on new kernels can't be restored on old kernels. And this is not a theoretical concern, KVM has run afoul of this a few times (though thankfully very rarely).

That's a bug, and should be fixed.

I am talking here about intentional ABI changes. As I mentioned above we _can_ change the ABI, we just need to be careful about it.

Quoted 9 lines

I don't think it's reasonable to expect LUO maintainers to gain enough expertise in each subsystem to be able to ensure changes are forwards and backwards compatible. The only way I see LUO being successful in the long term is to get subsystem maintainers/contributors to understand *and buy-in* to the LUO model and rules, so that each subsystem can largely be self-sustatining. I.e. setting yourselves up as literal gatekeepers will help prevent blatant breakage, but it's less likely to help guard against more subtle breakage, and in my experience, subtle breakage is by far harder to detect and more painful to deal with.

Outside of enforcing the ABI compatibility windows, I think this makes sense.

In principle at least, though from my experience the maintainer appetite to care about live update varies. The feeling I've got from MM for example is more along the lines of "you break it, you buy it". Which is also a valid model IMO. The person who wrote the live update code for a subsystem is well equipped to understand these nuances.

Quoted 5 lines

Somewhat of a side topic: in my experience, using monotonically increasing version numbers is a horrible way to enumerate features/content. So for me, allowing KVM's LUO ABI to change with a verson bump is probably a non-starter. I.e. whatever gets merged needs to be more future-proof than "we'll deal with it later".

As I wrote above, the compatibility model for live update allows breaking ABI with version bumps. Userspace keeps working fine, it only restricts the set of kernels that you can go to. So the "we'll deal with it later" is not as harmful because our choices here are not permanent.

Also, there is work underway to allow transitions between versions [0].

But still, I am not opposed to a more flexible guest_memfd ABI. I think we can reserve some space for feature flags to let us add things in the future.

[0] https://lore.kernel.org/kexec/[email protected]/

Signature · 2 lines
Regards,
Pratyush Yadav
#40 lore raw

Re: [PATCH v4 05/11] KVM: LUO: Support VM preservation across live updates

In reply to Sean Christopherson
Quoted 36 lines · Sean Christopherson
Quoted 21 lines · Pratyush Yadav
Quoted 19 lines · Sean Christopherson
Quoted 5 lines · Tarun Sahu

Register a Live Update Orchestrator (LUO) file handler for KVM VM files to serialize and deserialize VM state across kexec live updates.

Currently, Only VM type (e.g. arch.vm_type on x86) is preserved as part of VM preservation.

Why?

Quoted 9 lines

On retrieval, kvm_luo_retrieve() recreates the KVM VM file via kvm_create_vm_file() and use an atomically incremented ID for the internal fdname, as the final fdname assigned by userspace is not yet known during retrieval. As this fdname is only used in debugfs infra, This will not break any UAPI.

This infrastructure establishes the foundation for preserving guest_memfd instances across live updates, and can be expanded in the future to preserve additional VM state.

Quoted 7 lines
Quoted 4 lines

Uh, why guest_memfd? As much as I want to push guest_memfd adoption, it seems guest_memfd should be the _last_ thing we support, not the first. As evidenced by the last two decades, it's very doable to have KVM VMs without guest_memfd, but it's rather hard to have VMs without vCPUs.

You _can_ preserve vCPUs today using KVM_{GET,SET}_REGS, they just won't run in the background during the reboot.

What about x86 CoCo VMs? Which are quite literally _the_ reason guest_memfd was created in the first place.

Quoted 2 lines

This series can save you from dumping VM memory to disk if it is backed by guest_memfd.

Quoted 6 lines

Or to word it another way, one _can_ save guest_memfd, it's just slower.

My point is that this series needs to provide a _lot_ more information about the bigger KVM picture.

Yes, I agree. I will try to layout the plan.

End goal of this series is to preserve VM memory which is backed by guest_memfd. Not all VM are backed by guest_memfd. So preservation of KVM (vm_file) is independent of preservation of guest_memfd. But guest_memefd can not be preserved without preserving the KVM (vm_file). So I agree to your suggestion:

kvm/Makefile.kvm +5 -0
index d047d4cf58c9..e6f098498795 100644
@@ -13,3 +13,8 @@ kvm-$(CONFIG_HAVE_KVM_IRQ_ROUTING) += $(KVM)/irqchip.o
 kvm-$(CONFIG_HAVE_KVM_DIRTY_RING) += $(KVM)/dirty_ring.o
 kvm-$(CONFIG_HAVE_KVM_PFNCACHE) += $(KVM)/pfncache.o
 kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd.o
+
+ifdef CONFIG_LIVEUPDATE
+kvm-y += $(KVM)/kvm_luo.o
+kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd_luo.o
+endif

Guest_memfd can't be created alone without struct kvm (kvm_gmem_create() and KVM_GMEM_CREATE ioctl). This is how guest_memfd has been designed. I don't want to break this design which has been accepted upstream after lot of discussion.

So while LUO preserve guest_memfd's data, It just preserve the PFNs value, few flags that belongs to the guest_memfd.

On restore side in new kernel, These PFNs, flags will be populated to a _newly_ created guest_memfd. that is how preservation and retrieval works. So To create this new guest_memfd, We need struct kvm (vm_file). this vm_file must be the same VM which had this guest_memfd in old kernel, So we preserve the vm_file, get its TOKEN preserve with guest_memfd and during restore, we create the guest_memfd with the same VM (vm_file/struct kvm).

I agree, I did not do good job explaining things in commit message. I will make sure to update them in next revision.

Quoted 6 lines

For those of us that are on the very fringes of live update,
it's practically impossible to review because, to us, it seems very arbitrary.

The part that's especially confusing is the saving of the VM type. That comes straight from userspace, so it's super bizarre to automatically save/restore that, but nothing else.

Like, guest_memfd needs struct kvm to create itself. struct kvm (vm_file) needs vm_type to create itself (kvm_create_vm() or KVM_CREATE_VM IOCTL). LUO does not provde functionality to pass any subsystem specific arguments. So vm_type needs to be preserved even though, userspace is aware about it.

So Why do we preserve only vm_type: To keep things simple for this series, As target is guest_memfd.

Currently I dont have discreet plan on what else will
,in future, be needed to be preserved. Which I agree not a absoulute right way to approach. I will layout a rough plan on KVM side preservation.

Having this need for backward compatiblity: I responded here:
https://lore.kernel.org/all/[email protected]/

Quoted 90 lines
Quoted 16 lines
Quoted 16 lines

Review

Quoted 10 lines
(quoted diff) +9 -0
+KVM LIVE UPDATE
+M:	Pasha Tatashin <[email protected]>
+M:	Mike Rapoport <[email protected]>
+M:	Pratyush Yadav <[email protected]>
+R:	Tarun Sahu <[email protected]>
+L:	[email protected]
+L:	[email protected]
+S:	Maintained
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git

NAK on taking changes through a different tree. This is KVM code, period.

In general, I'm skeptical of the dedicated MAINTAINERS entry. It's extremely difficult to tell since this series is little more than a skeleton (either that or liveupdate is way simpler that I was expecting), but I suspect that maintaining

...

Quoted 7 lines
Quoted 3 lines

E.g. the LUO APIs seem pretty straightforward; I assume the bulk of the complexity is going to be in knowing what to save/restore, and how, which is much more about KVM than it is about liveupdate.

I think it is fine if you want to take these changes through the KVM tree, but I would like live update maintainers to be listed as reviewers at least.

Why not simply add a file pattern match to the LIVE UPDATE entry?

MAINTAINERS +1 -1
index 8014b9f8253e..2eb57b22c37f 100644
@@ -15052,8 +15052,8 @@ F:      include/linux/liveupdate.h
 F:     include/uapi/linux/liveupdate.h
 F:     kernel/liveupdate/
 F:     lib/tests/liveupdate.c
-F:     mm/memfd_luo.c
 F:     tools/testing/selftests/liveupdate/
+N:     [^a-z]luo
 
 LLC (802.2)
 L:     [email protected]
Quoted 8 lines

At the same time, I also keep being (pleasantly)
surprised at preservation being relatively simple. For example, the code to preserve a shmem file (via memfd) is roughly 600 lines, a big chunk of which is comments. The code of course has some limitations, but it is good enough for use in production.

For one, we care about ABI breakages and versioning.

Which is amusing to me because that implies KVM does not, and I would hazard to guess that KVM has the biggest ABI surface of any subsystem in the kernel by a country mile (though I'm probably wildly underestimating the effective ABI surface of filesystems).

Quoted 2 lines

The serialized state is a part of live update ABI and changes to it should be ACKed by us.

Meh, "Don't break userspace" is a universal rule in the kernel, I genuinely don't see why liveupdate needs special treatment.

Quoted 3 lines

For another, how the file handlers interact with their dependencies can affect the behaviour that VMMs observe. Those changes should also pass by some live update eyes.

Perhaps in the short term, but IMO, that's not a winning strategy in the long term. From my perspective, that like saying the PAGE CACHE maintainers should review every usage of the filemap APIs, because how the APIs are used impacts the page cache and affects userspace-visible behavior. There are myriad analogies like that throughout the kernel.

Yes, liveupdate is new and shiny, but IMO for it to be successful and maintainable, it needs to be treated like any other core infrastructure in the kernel, not a special snowflake whose details are known only by a handful of people. Because I think it's likely liveupdate goes one of two ways: either liveupdate becomes a very niche thing that is used sparingly throughout the kernel, or it becomes a broadly used feature that is supported by many filesystems and subsystems.

If liveupdate is relegated to niche status, then it probably isn't going to see a significant amount of ongoing development, at which point the folks working on liveupdate will naturally migrate to other projects, and maintenance will largely be left to subsystem maintainers.

If liveupdate is broadly used, then having a single group of people maintain every subsystem's usage won't scale, and maintenance will again largely fall on the shoulder of subsystem maintainers. Which is totally fine and working as intended, because that's exactly what subystem maintainers are signing up for by merging support for liveupdate.

author
#41 lore raw

PATCH v4 6/11 KVM: guest_memfd: Move internal definitions to internal header

In reply to Tarun Sahu

Extract 'struct gmem_file', 'struct gmem_inode', and GMEM_I() from virt/kvm/guest_memfd.c into a new internal header virt/kvm/guest_memfd.h. Also split __kvm_gmem_create() to expose a non-static __kvm_gmem_create_file() helper that returns a 'struct file *' instead of an fd.

These internal definitions and helpers allow upcoming guest_memfd Live Update Orchestrator (LUO) preservation code to access guest_memfd internals and reconstruct guest_memfd file instances from preserved state without installing them into a file descriptor table up front.

---
 virt/kvm/guest_memfd.c | 68 +++++++++++++++++-------------------------
 virt/kvm/guest_memfd.h | 39 ++++++++++++++++++++++++
 2 files changed, 67 insertions(+), 40 deletions(-)
 create mode 100644 virt/kvm/guest_memfd.h
virt/kvm/guest_memfd.c +28 -40
index db57c5766ab6..dd84bba8974b 100644
@@ -7,38 +7,12 @@
 #include <linux/mempolicy.h>
 #include <linux/pseudo_fs.h>
 #include <linux/pagemap.h>
+#include "guest_memfd.h"
 
 #include "kvm_mm.h"
 
 static struct vfsmount *kvm_gmem_mnt;
 
-/*
- * A guest_memfd instance can be associated multiple VMs, each with its own
- * "view" of the underlying physical memory.
- *
- * The gmem's inode is effectively the raw underlying physical storage, and is
- * used to track properties of the physical memory, while each gmem file is
- * effectively a single VM's view of that storage, and is used to track assets
- * specific to its associated VM, e.g. memslots=>gmem bindings.
- */
-struct gmem_file {
-	struct kvm *kvm;
-	struct xarray bindings;
-	struct list_head entry;
-};
-
-struct gmem_inode {
-	struct shared_policy policy;
-	struct inode vfs_inode;
-	struct list_head gmem_file_list;
-
-	u64 flags;
-};
-
-static __always_inline struct gmem_inode *GMEM_I(struct inode *inode)
-{
-	return container_of(inode, struct gmem_inode, vfs_inode);
-}
 
 #define kvm_gmem_for_each_file(f, inode) \
 	list_for_each_entry(f, &GMEM_I(inode)->gmem_file_list, entry)
@@ -557,23 +531,17 @@ bool __weak kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 	return true;
 }
 
-static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
+struct file *__kvm_gmem_create_file(struct kvm *kvm, loff_t size, u64 flags)
 {
 	static const char *name = "[kvm-gmem]";
 	struct gmem_file *f;
 	struct inode *inode;
 	struct file *file;
-	int fd, err;
-
-	fd = get_unused_fd_flags(0);
-	if (fd < 0)
-		return fd;
+	int err;
 
 	f = kzalloc_obj(*f);
-	if (!f) {
-		err = -ENOMEM;
-		goto err_fd;
-	}
+	if (!f)
+		return ERR_PTR(-ENOMEM);
 
 	/* __fput() will take care of fops_put(). */
 	if (!fops_get(&kvm_gmem_fops)) {
@@ -612,8 +580,7 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
 	xa_init(&f->bindings);
 	list_add(&f->entry, &GMEM_I(inode)->gmem_file_list);
 
-	fd_install(fd, file);
-	return fd;
+	return file;
 
 err_inode:
 	iput(inode);
@@ -621,7 +588,28 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
 	fops_put(&kvm_gmem_fops);
 err_gmem:
 	kfree(f);
-err_fd:
+	return ERR_PTR(err);
+}
+
+static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
+{
+	struct file *file;
+	int fd, err;
+
+	fd = get_unused_fd_flags(0);
+	if (fd < 0)
+		return fd;
+
+	file = __kvm_gmem_create_file(kvm, size, flags);
+	if (IS_ERR(file)) {
+		err = PTR_ERR(file);
+		goto err_put_fd;
+	}
+
+	fd_install(fd, file);
+	return fd;
+
+err_put_fd:
 	put_unused_fd(fd);
 	return err;
 }
virt/kvm/guest_memfd.h new file +39 -0
new file mode 100644
index 000000000000..c528b046dd69
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __KVM_GUEST_MEMFD_H__
+#define __KVM_GUEST_MEMFD_H__ 1
+
+#include <linux/kvm_host.h>
+#include <linux/fs.h>
+#include <linux/mempolicy.h>
+
+/*
+ * A guest_memfd instance can be associated multiple VMs, each with its own
+ * "view" of the underlying physical memory.
+ *
+ * The gmem's inode is effectively the raw underlying physical storage, and is
+ * used to track properties of the physical memory, while each gmem file is
+ * effectively a single VM's view of that storage, and is used to track assets
+ * specific to its associated VM, e.g. memslots=>gmem bindings.
+ */
+struct gmem_file {
+	struct kvm *kvm;
+	struct xarray bindings;
+	struct list_head entry;
+};
+
+struct gmem_inode {
+	struct shared_policy policy;
+	struct inode vfs_inode;
+	struct list_head gmem_file_list;
+
+	u64 flags;
+};
+
+static inline struct gmem_inode *GMEM_I(struct inode *inode)
+{
+	return container_of(inode, struct gmem_inode, vfs_inode);
+}
+
+struct file *__kvm_gmem_create_file(struct kvm *kvm, loff_t size, u64 flags);
+
+#endif /* __KVM_GUEST_MEMFD_H__ */
Signature · 1 line
2.55.0.229.g6434b31f56-goog
#42 lore raw

Re: [PATCH v4 06/11] KVM: guest_memfd: Move internal definitions to internal header

Quoted 6 lines · Tarun Sahu

Extract 'struct gmem_file', 'struct gmem_inode', and GMEM_I() from virt/kvm/guest_memfd.c into a new internal header virt/kvm/guest_memfd.h. Also split __kvm_gmem_create() to expose a non-static __kvm_gmem_create_file() helper that returns a 'struct file *' instead of an fd.

This is done basically to support a new virt/kvm/guest_memfd_luo.c file.

Would like to know what Sean thinks of this! I'd like to know for future guest_memfd work too.

Quoted 171 lines

These internal definitions and helpers allow upcoming guest_memfd Live Update Orchestrator (LUO) preservation code to access guest_memfd internals and reconstruct guest_memfd file instances from preserved state without installing them into a file descriptor table up front.

---
 virt/kvm/guest_memfd.c | 68 +++++++++++++++++-------------------------
 virt/kvm/guest_memfd.h | 39 ++++++++++++++++++++++++
 2 files changed, 67 insertions(+), 40 deletions(-)
 create mode 100644 virt/kvm/guest_memfd.h
virt/kvm/guest_memfd.c +28 -40
index db57c5766ab6..dd84bba8974b 100644
@@ -7,38 +7,12 @@
 #include <linux/mempolicy.h>
 #include <linux/pseudo_fs.h>
 #include <linux/pagemap.h>
+#include "guest_memfd.h"
 
 #include "kvm_mm.h"
 
 static struct vfsmount *kvm_gmem_mnt;
 
-/*
- * A guest_memfd instance can be associated multiple VMs, each with its own
- * "view" of the underlying physical memory.
- *
- * The gmem's inode is effectively the raw underlying physical storage, and is
- * used to track properties of the physical memory, while each gmem file is
- * effectively a single VM's view of that storage, and is used to track assets
- * specific to its associated VM, e.g. memslots=>gmem bindings.
- */
-struct gmem_file {
-	struct kvm *kvm;
-	struct xarray bindings;
-	struct list_head entry;
-};
-
-struct gmem_inode {
-	struct shared_policy policy;
-	struct inode vfs_inode;
-	struct list_head gmem_file_list;
-
-	u64 flags;
-};
-
-static __always_inline struct gmem_inode *GMEM_I(struct inode *inode)
-{
-	return container_of(inode, struct gmem_inode, vfs_inode);
-}
 
 #define kvm_gmem_for_each_file(f, inode) \
 	list_for_each_entry(f, &GMEM_I(inode)->gmem_file_list, entry)
@@ -557,23 +531,17 @@ bool __weak kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 	return true;
 }
 
-static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
+struct file *__kvm_gmem_create_file(struct kvm *kvm, loff_t size, u64 flags)
 {
 	static const char *name = "[kvm-gmem]";
 	struct gmem_file *f;
 	struct inode *inode;
 	struct file *file;
-	int fd, err;
-
-	fd = get_unused_fd_flags(0);
-	if (fd < 0)
-		return fd;
+	int err;
 
 	f = kzalloc_obj(*f);
-	if (!f) {
-		err = -ENOMEM;
-		goto err_fd;
-	}
+	if (!f)
+		return ERR_PTR(-ENOMEM);
 
 	/* __fput() will take care of fops_put(). */
 	if (!fops_get(&kvm_gmem_fops)) {
@@ -612,8 +580,7 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
 	xa_init(&f->bindings);
 	list_add(&f->entry, &GMEM_I(inode)->gmem_file_list);
 
-	fd_install(fd, file);
-	return fd;
+	return file;
 
 err_inode:
 	iput(inode);
@@ -621,7 +588,28 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
 	fops_put(&kvm_gmem_fops);
 err_gmem:
 	kfree(f);
-err_fd:
+	return ERR_PTR(err);
+}
+
+static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
+{
+	struct file *file;
+	int fd, err;
+
+	fd = get_unused_fd_flags(0);
+	if (fd < 0)
+		return fd;
+
+	file = __kvm_gmem_create_file(kvm, size, flags);
+	if (IS_ERR(file)) {
+		err = PTR_ERR(file);
+		goto err_put_fd;
+	}
+
+	fd_install(fd, file);
+	return fd;
+
+err_put_fd:
 	put_unused_fd(fd);
 	return err;
 }
virt/kvm/guest_memfd.h new file +39 -1
new file mode 100644
index 000000000000..c528b046dd69
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __KVM_GUEST_MEMFD_H__
+#define __KVM_GUEST_MEMFD_H__ 1
+
+#include <linux/kvm_host.h>
+#include <linux/fs.h>
+#include <linux/mempolicy.h>
+
+/*
+ * A guest_memfd instance can be associated multiple VMs, each with its own
+ * "view" of the underlying physical memory.
+ *
+ * The gmem's inode is effectively the raw underlying physical storage, and is
+ * used to track properties of the physical memory, while each gmem file is
+ * effectively a single VM's view of that storage, and is used to track assets
+ * specific to its associated VM, e.g. memslots=>gmem bindings.
+ */
+struct gmem_file {
+	struct kvm *kvm;
+	struct xarray bindings;
+	struct list_head entry;
+};
+
+struct gmem_inode {
+	struct shared_policy policy;
+	struct inode vfs_inode;
+	struct list_head gmem_file_list;
+
+	u64 flags;
+};
+
+static inline struct gmem_inode *GMEM_I(struct inode *inode)
+{
+	return container_of(inode, struct gmem_inode, vfs_inode);
+}
+
+struct file *__kvm_gmem_create_file(struct kvm *kvm, loff_t size, u64 flags);
+
+#endif /* __KVM_GUEST_MEMFD_H__ */
--

2.55.0.229.g6434b31f56-goog

maintainer
#43 lore raw

Re: [PATCH v4 06/11] KVM: guest_memfd: Move internal definitions to internal header

Quoted 11 lines · Ackerley Tng
Quoted 6 lines · Tarun Sahu

Extract 'struct gmem_file', 'struct gmem_inode', and GMEM_I() from virt/kvm/guest_memfd.c into a new internal header virt/kvm/guest_memfd.h. Also split __kvm_gmem_create() to expose a non-static __kvm_gmem_create_file() helper that returns a 'struct file *' instead of an fd.

This is done basically to support a new virt/kvm/guest_memfd_luo.c file.

My $0.02 on this: I think it is useful to split out the live update code into a separate file. It makes for easier reviews, and puts a clear boundary between core and live update. We have done it for memfd/shmem for example (see mm/memfd_luo.c) and for other live update series in flight like PCI or IOMMU.

Quoted 10 lines

Would like to know what Sean thinks of this! I'd like to know for future guest_memfd work too.

Quoted 6 lines

These internal definitions and helpers allow upcoming guest_memfd Live Update Orchestrator (LUO) preservation code to access guest_memfd internals and reconstruct guest_memfd file instances from preserved state without installing them into a file descriptor table up front.

[...]

Signature · 2 lines
Regards,
Pratyush Yadav
author
#44 lore raw

PATCH v4 7/11 KVM: guest_memfd: Add support for freezing mappings

In reply to Tarun Sahu

Introduce kvm_gmem_freeze() to freeze a guest_memfd inode's mapping, which prevents fallocate() operations and new page fault allocations during preservation.

Use a global SRCU (kvm_gmem_freeze_srcu) to synchronize freeze state checkers without incurring per-fault locking overhead or risking per-CPU memory exhaustion (as per-CPU structure not counted in cgroup) from per-inode SRCU structures by faulty/compromised VMM.

---
 virt/kvm/guest_memfd.c | 117 +++++++++++++++++++++++++++++++++++++----
 virt/kvm/guest_memfd.h |   5 ++
 2 files changed, 111 insertions(+), 11 deletions(-)
virt/kvm/guest_memfd.c +106 -11
index dd84bba8974b..d522cc34f476 100644
@@ -7,11 +7,13 @@
 #include <linux/mempolicy.h>
 #include <linux/pseudo_fs.h>
 #include <linux/pagemap.h>
+#include <linux/srcu.h>
 #include "guest_memfd.h"
 
 #include "kvm_mm.h"
 
 static struct vfsmount *kvm_gmem_mnt;
+static struct srcu_struct kvm_gmem_freeze_srcu;
 
 
 #define kvm_gmem_for_each_file(f, inode) \
@@ -96,6 +98,7 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
 	/* TODO: Support huge pages. */
 	struct mempolicy *policy;
 	struct folio *folio;
+	int idx;
 
 	/*
 	 * Fast-path: See if folio is already present in mapping to avoid
@@ -105,12 +108,20 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
 	if (!IS_ERR(folio))
 		return folio;
 
+	idx = srcu_read_lock(&kvm_gmem_freeze_srcu);
+	if (kvm_gmem_is_frozen(inode)) {
+		srcu_read_unlock(&kvm_gmem_freeze_srcu, idx);
+		return ERR_PTR(-EPERM);
+	}
+
 	policy = mpol_shared_policy_lookup(&GMEM_I(inode)->policy, index);
 	folio = __filemap_get_folio_mpol(inode->i_mapping, index,
 					 FGP_LOCK | FGP_CREAT,
 					 mapping_gfp_mask(inode->i_mapping), policy);
 	mpol_cond_put(policy);
 
+	srcu_read_unlock(&kvm_gmem_freeze_srcu, idx);
+
 	/*
 	 * External interfaces like kvm_gmem_get_pfn() support dealing
 	 * with hugepages to a degree, but internally, guest_memfd currently
@@ -273,16 +284,30 @@ static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
 static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,
 			       loff_t len)
 {
+	struct inode *inode = file_inode(file);
 	int ret;
+	int idx;
 
-	if (!(mode & FALLOC_FL_KEEP_SIZE))
-		return -EOPNOTSUPP;
+	idx = srcu_read_lock(&kvm_gmem_freeze_srcu);
+	if (kvm_gmem_is_frozen(inode)) {
+		srcu_read_unlock(&kvm_gmem_freeze_srcu, idx);
+		return -EPERM;
+	}
 
-	if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE))
-		return -EOPNOTSUPP;
+	if (!(mode & FALLOC_FL_KEEP_SIZE)) {
+		ret = -EOPNOTSUPP;
+		goto out;
+	}
 
-	if (!PAGE_ALIGNED(offset) || !PAGE_ALIGNED(len))
-		return -EINVAL;
+	if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE)) {
+		ret = -EOPNOTSUPP;
+		goto out;
+	}
+
+	if (!PAGE_ALIGNED(offset) || !PAGE_ALIGNED(len)) {
+		ret = -EINVAL;
+		goto out;
+	}
 
 	if (mode & FALLOC_FL_PUNCH_HOLE)
 		ret = kvm_gmem_punch_hole(file_inode(file), offset, len);
@@ -291,6 +316,9 @@ static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,
 
 	if (!ret)
 		file_modified(file);
+
+out:
+	srcu_read_unlock(&kvm_gmem_freeze_srcu, idx);
 	return ret;
 }
 
@@ -948,7 +976,9 @@ static void kvm_gmem_destroy_inode(struct inode *inode)
 
 static void kvm_gmem_free_inode(struct inode *inode)
 {
-	kmem_cache_free(kvm_gmem_inode_cachep, GMEM_I(inode));
+	struct gmem_inode *gi = GMEM_I(inode);
+
+	kmem_cache_free(kvm_gmem_inode_cachep, gi);
 }
 
 static const struct super_operations kvm_gmem_super_operations = {
@@ -1003,12 +1033,21 @@ int kvm_gmem_init(struct module *module)
 	if (!kvm_gmem_inode_cachep)
 		return -ENOMEM;
 
+	ret = init_srcu_struct(&kvm_gmem_freeze_srcu);
+	if (ret)
+		goto err_cache;
+
 	ret = kvm_gmem_init_mount();
-	if (ret) {
-		kmem_cache_destroy(kvm_gmem_inode_cachep);
-		return ret;
-	}
+	if (ret)
+		goto err_srcu;
+
 	return 0;
+
+err_srcu:
+	cleanup_srcu_struct(&kvm_gmem_freeze_srcu);
+err_cache:
+	kmem_cache_destroy(kvm_gmem_inode_cachep);
+	return ret;
 }
 
 void kvm_gmem_exit(void)
@@ -1016,5 +1055,61 @@ void kvm_gmem_exit(void)
 	kern_unmount(kvm_gmem_mnt);
 	kvm_gmem_mnt = NULL;
 	rcu_barrier();
+	cleanup_srcu_struct(&kvm_gmem_freeze_srcu);
 	kmem_cache_destroy(kvm_gmem_inode_cachep);
 }
+
+/**
+ * kvm_gmem_freeze - Freeze or unfreeze a guest_memfd inode mapping.
+ * @inode: The guest_memfd inode.
+ * @freeze: True to freeze, false to unfreeze.
+ *
+ * This API is used strictly during the live update / preservation transition
+ * window to prevent host userspace and guest-side faults from making any
+ * mapping modifications (such as fallocate or page fault allocation)
+ * to the guest_memfd page cache.
+ *
+ * Synchronization Strategy (Sleepable RCU):
+ * To avoid high-contention VFS locks (like inode_lock or
+ * filemap_invalidate_lock) on the vCPU page fault hot paths, this subsystem
+ * implements a lightweight, system-wide Sleepable RCU (SRCU) mechanism
+ * (`kvm_gmem_freeze_srcu`):
+ *
+ * Global vs. Per-Inode SRCU
+ * ======================
+ * A single system-wide global static `srcu_struct` is used instead of a
+ * per-inode SRCU structure to completely prevent unprivileged users from
+ * exhausting the host's per-CPU memory allocator. Because
+ * `init_srcu_struct()` allocates per-CPU memory via `alloc_percpu()`, which
+ * is not accounted by memory cgroups (memcg),
+ * a per-inode SRCU structure would allow a tenant to bypass cgroup limits and
+ * trigger a system-wide Out-of-Memory (OOM) crash simply by spawning a large
+ * number of guest_memfd file descriptors (bounded only by RLIMIT_NOFILE).
+ *
+ * Flag Modification Note:
+ * Since `GUEST_MEMFD_F_MAPPING_FROZEN` is the ONLY flag in
+ * `GMEM_I(inode)->flags` that is mutated dynamically at runtime (all other
+ * flags are creation-time flags which remain strictly read-only), there is
+ * no possibility of concurrent bit-modification races. Therefore, a standard
+ * `WRITE_ONCE` is fully safe and does not require complex `cmpxchg`
+ * synchronization loops.
+ */
+void kvm_gmem_freeze(struct inode *inode, bool freeze)
+{
+	u64 flags = READ_ONCE(GMEM_I(inode)->flags);
+
+	if (freeze)
+		flags |= GUEST_MEMFD_F_MAPPING_FROZEN;
+	else
+		flags &= ~GUEST_MEMFD_F_MAPPING_FROZEN;
+
+	WRITE_ONCE(GMEM_I(inode)->flags, flags);
+
+	if (freeze)
+		synchronize_srcu(&kvm_gmem_freeze_srcu);
+}
+
+bool kvm_gmem_is_frozen(struct inode *inode)
+{
+	return READ_ONCE(GMEM_I(inode)->flags) & GUEST_MEMFD_F_MAPPING_FROZEN;
+}
virt/kvm/guest_memfd.h +5 -0
index c528b046dd69..028c348a1023 100644
@@ -29,11 +29,16 @@ struct gmem_inode {
 	u64 flags;
 };
 
+/* Internal kernel-only flags (must not overlap with UAPI flags) */
+#define GUEST_MEMFD_F_MAPPING_FROZEN	(1ULL << 63)
+
 static inline struct gmem_inode *GMEM_I(struct inode *inode)
 {
 	return container_of(inode, struct gmem_inode, vfs_inode);
 }
 
 struct file *__kvm_gmem_create_file(struct kvm *kvm, loff_t size, u64 flags);
+void kvm_gmem_freeze(struct inode *inode, bool freeze);
+bool kvm_gmem_is_frozen(struct inode *inode);
 
 #endif /* __KVM_GUEST_MEMFD_H__ */
Signature · 1 line
2.55.0.229.g6434b31f56-goog
#45 lore raw

Re: [PATCH v4 07/11] KVM: guest_memfd: Add support for freezing mappings

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Holding the global `kvm_gmem_freeze_srcu` read lock during unbounded `fallocate` operations allows…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Holding the global kvm_gmem_freeze_srcu read lock during unbounded fallocate operations allows an unprivileged tenant to block kvm_gmem_freeze() indefinitely for other VMs. --

--- Patch [7]: [PATCH v4 07/11] KVM: guest_memfd: Add support for freezing mappings --- commit de8f141a41c06197ca510e03554977beeeb4fcdf
Author: Tarun Sahu <[email protected]>

KVM: guest_memfd: Add support for freezing mappings

This commit introduces kvm_gmem_freeze() to freeze a guest_memfd inode's mapping to prevent fallocate and page fault allocations during preservation. It uses a global SRCU to synchronize freeze state checkers without incurring per-fault locking overhead or per-CPU memory exhaustion.

Quoted 4 lines
virt/kvm/guest_memfd.c +0 -0
index fe1adc9ba48d1..a4d9d348e4d16 100644

[ ... ]

Quoted 15 lines
virt/kvm/guest_memfd.c +7 -2
@@ -273,16 +284,30 @@ static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
 static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,
 			       loff_t len)
 {
+	struct inode *inode = file_inode(file);
 	int ret;
+	int idx;
 
-	if (!(mode & FALLOC_FL_KEEP_SIZE))
-		return -EOPNOTSUPP;
+	idx = srcu_read_lock(&kvm_gmem_freeze_srcu);
+	if (kvm_gmem_is_frozen(inode)) {
+		srcu_read_unlock(&kvm_gmem_freeze_srcu, idx);
+		return -EPERM;
+	}

[ ... ]

virt/kvm/guest_memfd.c static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset, · line 323

Quoted 12 lines
virt/kvm/guest_memfd.c +3 -0
 	if (mode & FALLOC_FL_PUNCH_HOLE)
 		ret = kvm_gmem_punch_hole(file_inode(file), offset, len);
 	else
 		ret = kvm_gmem_allocate(file_inode(file), offset, len);
 
 	if (!ret)
 		file_modified(file);
+
+out:
+	srcu_read_unlock(&kvm_gmem_freeze_srcu, idx);
 	return ret;
 }

[Severity: High]
Does this code allow an unprivileged tenant to stall the system by holding the global kvm_gmem_freeze_srcu read lock across unbounded fallocate operations?

If a tenant requests a massive hole punch or allocation, kvm_gmem_fallocate() holds the read lock for the entire duration of kvm_gmem_punch_hole() or kvm_gmem_allocate().

If another VM triggers kvm_gmem_freeze(..., true) during this time, it will block in synchronize_srcu(&kvm_gmem_freeze_srcu) until the massive operation completes.

Could this lead to a cross-tenant denial of service where one VM blocks live updates and preservation operations for all other VMs on the same hypervisor?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
#46 lore raw

Re: [PATCH v4 07/11] KVM: guest_memfd: Add support for freezing mappings

In reply to Tarun Sahu
Quoted 9 lines · Tarun Sahu

Introduce kvm_gmem_freeze() to freeze a guest_memfd inode's mapping, which prevents fallocate() operations and new page fault allocations during preservation.

Use a global SRCU (kvm_gmem_freeze_srcu) to synchronize freeze state checkers without incurring per-fault locking overhead or risking per-CPU memory exhaustion (as per-CPU structure not counted in cgroup) from per-inode SRCU structures by faulty/compromised VMM.

I wonder if everything to do with freezing should be gated behind CONFIG_LIVEUPDATE_GUEST_MEMFD. What do people think?

Quoted 4 lines

[...snip...]

#47 lore raw

Re: [PATCH v4 07/11] KVM: guest_memfd: Add support for freezing mappings

Quoted 6 lines · Ackerley Tng
Quoted 4 lines · Tarun Sahu

Introduce kvm_gmem_freeze() to freeze a guest_memfd inode's mapping, which prevents fallocate() operations and new page fault allocations during preservation.

Quoted 8 lines
Quoted 5 lines

Use a global SRCU (kvm_gmem_freeze_srcu) to synchronize freeze state checkers without incurring per-fault locking overhead or risking per-CPU memory exhaustion (as per-CPU structure not counted in cgroup) from per-inode SRCU structures by faulty/compromised VMM.

I wonder if everything to do with freezing should be gated behind CONFIG_LIVEUPDATE_GUEST_MEMFD. What do people think?

Right now, Only liveupdate is the usecase. There was a discussion around the future usecase when IOMMU FD can use few pages from guest_memfd and will have to prevent punch_hole operation. But that will also avoid new faults. So freezing is not the solution here.

A SEAL type solution like in memfd can be explored later. But this is very far in future so Plenty of time to discuss more ideas.

On gating the freeze across CONFIG_LIVEUPDATE_GUEST_MEMFD sounds good
Idea. As in future, there is a plan to remove the freeze at all from
the preservation path. Before LIVEUPDATE, there was no usecase and in future, that usecase will also be gone. So gating it against the config makes sense.

Would love to know what do people think?

~Tarun

Quoted 5 lines
Quoted 5 lines

[...snip...]

#48 lore raw

Re: [PATCH v4 07/11] KVM: guest_memfd: Add support for freezing mappings

In reply to Tarun Sahu
Quoted 28 lines · Tarun Sahu

[...snip...]

virt/kvm/guest_memfd.c +11 -6
@@ -273,16 +284,30 @@ static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
 static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,
 			       loff_t len)
 {
+	struct inode *inode = file_inode(file);
 	int ret;
+	int idx;
 
-	if (!(mode & FALLOC_FL_KEEP_SIZE))
-		return -EOPNOTSUPP;
+	idx = srcu_read_lock(&kvm_gmem_freeze_srcu);
+	if (kvm_gmem_is_frozen(inode)) {
+		srcu_read_unlock(&kvm_gmem_freeze_srcu, idx);
+		return -EPERM;
+	}
 
-	if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE))
-		return -EOPNOTSUPP;
+	if (!(mode & FALLOC_FL_KEEP_SIZE)) {
+		ret = -EOPNOTSUPP;
+		goto out;
+	}
 
-	if (!PAGE_ALIGNED(offset) || !PAGE_ALIGNED(len))
-		return -EINVAL;

virt/kvm/guest_memfd.c static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len) · line 311

Quoted 10 lines
virt/kvm/guest_memfd.c +9 -0
+	if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE)) {
+		ret = -EOPNOTSUPP;
+		goto out;
+	}
+
+	if (!PAGE_ALIGNED(offset) || !PAGE_ALIGNED(len)) {
+		ret = -EINVAL;
+		goto out;
+	}
 

Continuing from [1], this guard is actually meant for PUNCH_HOLE... I wonder if there could be a deeper refactoring to check when actually punching a hole, basically the equivalent place as kvm_gmem_get_folio() in the flow of fallocating.

Will srcu_read_lock() be taken in a nested way for the fallocate case? It can be taken in a nested way with some intricacies, but maybe if the freeze check can be done just as the folio is about to be removed (as described in the above paragraph), it would avoid this nested locking here.

[1] https://lore.kernel.org/all/[email protected]/

Quoted 15 lines
virt/kvm/guest_memfd.c +3 -0
 	if (mode & FALLOC_FL_PUNCH_HOLE)
 		ret = kvm_gmem_punch_hole(file_inode(file), offset, len);
@@ -291,6 +316,9 @@ static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,
 
 	if (!ret)
 		file_modified(file);
+
+out:
+	srcu_read_unlock(&kvm_gmem_freeze_srcu, idx);
 	return ret;
 }
 
 
[...snip...]
#49 lore raw

Re: [PATCH v4 07/11] KVM: guest_memfd: Add support for freezing mappings

Quoted 46 lines · Ackerley Tng
Quoted 28 lines · Tarun Sahu

[...snip...]

virt/kvm/guest_memfd.c +11 -6
@@ -273,16 +284,30 @@ static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
 static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,
 			       loff_t len)
 {
+	struct inode *inode = file_inode(file);
 	int ret;
+	int idx;
 
-	if (!(mode & FALLOC_FL_KEEP_SIZE))
-		return -EOPNOTSUPP;
+	idx = srcu_read_lock(&kvm_gmem_freeze_srcu);
+	if (kvm_gmem_is_frozen(inode)) {
+		srcu_read_unlock(&kvm_gmem_freeze_srcu, idx);
+		return -EPERM;
+	}
 
-	if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE))
-		return -EOPNOTSUPP;
+	if (!(mode & FALLOC_FL_KEEP_SIZE)) {
+		ret = -EOPNOTSUPP;
+		goto out;
+	}
 
-	if (!PAGE_ALIGNED(offset) || !PAGE_ALIGNED(len))
-		return -EINVAL;

virt/kvm/guest_memfd.c static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len) · line 311

Quoted 10 lines
virt/kvm/guest_memfd.c +9 -0
+	if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE)) {
+		ret = -EOPNOTSUPP;
+		goto out;
+	}
+
+	if (!PAGE_ALIGNED(offset) || !PAGE_ALIGNED(len)) {
+		ret = -EINVAL;
+		goto out;
+	}
 

Continuing from [1], this guard is actually meant for PUNCH_HOLE... I wonder if there could be a deeper refactoring to check when actually punching a hole, basically the equivalent place as kvm_gmem_get_folio() in the flow of fallocating.

Quoted 7 lines

Will srcu_read_lock() be taken in a nested way for the fallocate case? It can be taken in a nested way with some intricacies, but maybe if the freeze check can be done just as the folio is about to be removed (as described in the above paragraph), it would avoid this nested locking here.

[1] https://lore.kernel.org/all/[email protected]/

Instead of avoiding on remove folio, We can just add the freeze check across the PUNCH_HOLE, So there will be no nested lock anymore. Nested lock is just with fallocate populate -> kvm_gmem_get_pfn. So this nested ness will be removed with this update.

~Tarun

Quoted 16 lines
Quoted 16 lines
virt/kvm/guest_memfd.c +3 -0
 	if (mode & FALLOC_FL_PUNCH_HOLE)
 		ret = kvm_gmem_punch_hole(file_inode(file), offset, len);
@@ -291,6 +316,9 @@ static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,
 
 	if (!ret)
 		file_modified(file);
+
+out:
+	srcu_read_unlock(&kvm_gmem_freeze_srcu, idx);
 	return ret;
 }
 
 
[...snip...]
#50 lore raw

Re: [PATCH v4 07/11] KVM: guest_memfd: Add support for freezing mappings

In reply to Tarun Sahu
Quoted 13 lines · Tarun Sahu

+/**
+ * kvm_gmem_freeze - Freeze or unfreeze a guest_memfd inode mapping.
+ * @inode: The guest_memfd inode.
+ * @freeze: True to freeze, false to unfreeze.
+ *
+ * This API is used strictly during the live update / preservation transition
+ * window to prevent host userspace and guest-side faults from making any
+ * mapping modifications (such as fallocate or page fault allocation)
+ * to the guest_memfd page cache.
+ *
+ * Synchronization Strategy (Sleepable RCU):
+ * To avoid high-contention VFS locks (like inode_lock or
+ * filemap_invalidate_lock) on the vCPU page fault hot paths, this subsystem

Quoted 8 lines

+ * implements a lightweight, system-wide Sleepable RCU (SRCU) mechanism
+ * (kvm_gmem_freeze_srcu):
+ *
+ * Global vs. Per-Inode SRCU
+ * ======================
+ * A single system-wide global static srcu_struct is used instead of a
+ * per-inode SRCU structure to completely prevent unprivileged users from
+ * exhausting the host's per-CPU memory allocator.

This argument doesn't hold up given that each kvm structure has two SRCU structs.

#51 lore raw

Re: [PATCH v4 07/11] KVM: guest_memfd: Add support for freezing mappings

Quoted 23 lines · Sean Christopherson
Quoted 13 lines · Tarun Sahu

+/**
+ * kvm_gmem_freeze - Freeze or unfreeze a guest_memfd inode mapping.
+ * @inode: The guest_memfd inode.
+ * @freeze: True to freeze, false to unfreeze.
+ *
+ * This API is used strictly during the live update / preservation transition
+ * window to prevent host userspace and guest-side faults from making any
+ * mapping modifications (such as fallocate or page fault allocation)
+ * to the guest_memfd page cache.
+ *
+ * Synchronization Strategy (Sleepable RCU):
+ * To avoid high-contention VFS locks (like inode_lock or
+ * filemap_invalidate_lock) on the vCPU page fault hot paths, this subsystem

Quoted 8 lines

+ * implements a lightweight, system-wide Sleepable RCU (SRCU) mechanism
+ * (kvm_gmem_freeze_srcu):
+ *
+ * Global vs. Per-Inode SRCU
+ * ======================
+ * A single system-wide global static srcu_struct is used instead of a
+ * per-inode SRCU structure to completely prevent unprivileged users from
+ * exhausting the host's per-CPU memory allocator.

Quoted 1 line

This argument doesn't hold up given that each kvm structure has two SRCU structs.

Ohh okay. I was very skeptic that per-cpu structure are not counted in cgroups and liveupdate is once in a while usecase So avoided making it per-cpu. But otherwise I am fine with going with per-cpu. Will take care of it in next revision.

Thanks
~Tarun

author
#52 lore raw

PATCH v4 8/11 KVM: guest_memfd: Add support for preservation via LUO

In reply to Tarun Sahu

Register a Live Update Orchestrator (LUO) file handler for guest_memfd files to preserve guest memory across kexec reboots.

Freeze the guest_memfd inode via kvm_gmem_freeze() during preservation to prevent concurrent fallocate() calls or page fault allocations. Walk the inode's page cache folios to serialize their metadata and physical PFNs for kexec handover.

On retrieval, reconstruct the guest_memfd file with __kvm_gmem_create_file(), re-insert the preserved folios into the page cache via filemap_add_folio(), and associate the file back to its preserved KVM VM via the VM token.

Currently, this supports fully shared guest_memfd instances backed by PAGE_SIZE pages. Also update MAINTAINERS for guest_memfd LUO preservation.

---
 MAINTAINERS                 |   1 +
 include/linux/kho/abi/kvm.h |  79 +++++-
 virt/kvm/Makefile.kvm       |   2 +-
 virt/kvm/guest_memfd_luo.c  | 515 ++++++++++++++++++++++++++++++++++++
 virt/kvm/kvm_main.c         |   7 +
 virt/kvm/kvm_mm.h           |   4 +
 6 files changed, 601 insertions(+), 7 deletions(-)
 create mode 100644 virt/kvm/guest_memfd_luo.c
MAINTAINERS +1 -0
index 0283f0fd6ef4..8db8bb448b23 100644
@@ -14548,6 +14548,7 @@ L:	[email protected]
 L:	[email protected]
 S:	Maintained
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git
+F:	virt/kvm/guest_memfd_luo.c
 F:	virt/kvm/kvm_luo.c
 
 KVM PARAVIRT (KVM/paravirt)
include/linux/kho/abi/kvm.h +73 -6

97 lines not on this page · raw message

virt/kvm/Makefile.kvm +1 -1
index c1a962159264..d30fca094c42 100644
@@ -13,4 +13,4 @@ kvm-$(CONFIG_HAVE_KVM_IRQ_ROUTING) += $(KVM)/irqchip.o
 kvm-$(CONFIG_HAVE_KVM_DIRTY_RING) += $(KVM)/dirty_ring.o
 kvm-$(CONFIG_HAVE_KVM_PFNCACHE) += $(KVM)/pfncache.o
 kvm-$(CONFIG_KVM_GUEST_MEMFD) += $(KVM)/guest_memfd.o
-kvm-$(CONFIG_LIVEUPDATE_GUEST_MEMFD) += $(KVM)/kvm_luo.o
+kvm-$(CONFIG_LIVEUPDATE_GUEST_MEMFD) += $(KVM)/guest_memfd_luo.o $(KVM)/kvm_luo.o
virt/kvm/guest_memfd_luo.c new file +515 -0

515 lines not on this page · raw message

virt/kvm/kvm_main.c +7 -0
index d9c3dd1afb05..e8e2f10cd71e 100644
@@ -6581,6 +6581,10 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module)
 	if (r)
 		goto err_luo;
 
+	r = kvm_gmem_luo_init();
+	if (r)
+		goto err_gmem_luo;
+
 	/*
 	 * Registration _must_ be the very last thing done, as this exposes
 	 * /dev/kvm to userspace, i.e. all infrastructure must be setup!
@@ -6594,6 +6598,8 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module)
 	return 0;
 
 err_register:
+	kvm_gmem_luo_exit();
+err_gmem_luo:
 	kvm_luo_exit();
 err_luo:
 	kvm_uninit_virtualization();
@@ -6625,6 +6631,7 @@ void kvm_exit(void)
 	 */
 	misc_deregister(&kvm_dev);
 
+	kvm_gmem_luo_exit();
 	kvm_luo_exit();
 
 	kvm_uninit_virtualization();
virt/kvm/kvm_mm.h +4 -0
index 87198715fb01..1295ff8bd504 100644
@@ -103,9 +103,13 @@ static inline void kvm_gmem_unbind(struct kvm_memory_slot *slot)
 #ifdef CONFIG_LIVEUPDATE_GUEST_MEMFD
 int kvm_luo_init(void);
 void kvm_luo_exit(void);
+int kvm_gmem_luo_init(void);
+void kvm_gmem_luo_exit(void);
 #else
 static inline int kvm_luo_init(void) { return 0; }
 static inline void kvm_luo_exit(void) {}
+static inline int kvm_gmem_luo_init(void) { return 0; }
+static inline void kvm_gmem_luo_exit(void) {}
 #endif /* CONFIG_LIVEUPDATE_GUEST_MEMFD */
 
 #endif /* __KVM_MM_H__ */
Signature · 1 line
2.55.0.229.g6434b31f56-goog
#53 lore raw

Re: [PATCH v4 08/11] KVM: guest_memfd: Add support for preservation via LUO

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] `kvm_gmem_luo_finish` incorrectly performs cleanup on successful retrieval, resulting in…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] kvm_gmem_luo_finish incorrectly performs cleanup on successful retrieval, resulting in use-after-free and double-frees of the serialization state and premature freeing of page cache folios.
- [High] Missing kho_is_enabled() check before invoking KHO serialization APIs, risking a NULL pointer dereference.
--

--- Patch [8]: [PATCH v4 08/11] KVM: guest_memfd: Add support for preservation via LUO --- commit 8d81cd3b20bca211fdf1860f352a2bbcf1096ec9
Author: Tarun Sahu <[email protected]>

KVM: guest_memfd: Add support for preservation via LUO

This commit registers a Live Update Orchestrator file handler for guest_memfd files to preserve guest memory across kexec reboots. It freezes the guest_memfd inode during preservation, walks the page cache to serialize metadata, and reconstructs the file and its folios on retrieval.

Quoted 6 lines
virt/kvm/guest_memfd_luo.c new file +0 -0
new file mode 100644
index 0000000000000..c0aa9354e0a78
@@ -0,0 +1,515 @@

[ ... ]

Quoted 5 lines
virt/kvm/guest_memfd_luo.c new file +5 -0
+static int kvm_gmem_luo_preserve(struct liveupdate_file_op_args *args)
+{
+	struct guest_memfd_luo_folio_ser *folios_ser = NULL;
+	u64 count = 0, gmem_flags, abi_flags = 0;
+	struct guest_memfd_luo_ser *ser;

[ ... ]

virt/kvm/guest_memfd_luo.c line 185

Quoted 3 lines
virt/kvm/guest_memfd_luo.c new file +3 -0
+	end_index = size >> PAGE_SHIFT;
+
+	ser = kho_alloc_preserve(sizeof(*ser));

[Severity: High]
Does this path need to verify if the KHO subsystem is actually enabled?

If the kernel is booted with KHO disabled, calling kho_alloc_preserve() and other KHO APIs unconditionally can cause a NULL pointer dereference.

Could we check kho_is_enabled() either here or in kvm_gmem_luo_init() before registering the handler?

[ ... ]

Quoted 5 lines
virt/kvm/guest_memfd_luo.c new file +5 -0
+static void kvm_gmem_luo_finish(struct liveupdate_file_op_args *args)
+{
+	struct guest_memfd_luo_ser *ser;
+	struct guest_memfd_luo_folio_ser *folios_ser;
+

virt/kvm/guest_memfd_luo.c line 471

Quoted 10 lines
virt/kvm/guest_memfd_luo.c new file +10 -0
+	/* Nothing to be done here, if retrieve_status was successful or errored,
+	 * Cleanup is taken care of in retrieval call.
+	 */
+	if (args->retrieve_status)
+		return;
+
+	if (!args->serialized_data)
+		return;
+
+	ser = phys_to_virt(args->serialized_data);

[Severity: High]
Is the check on args->retrieve_status inverted here?

When kvm_gmem_luo_retrieve() succeeds, it returns 0 and frees the ser pointer:

kvm_gmem_luo_retrieve()

    args->file = file;
    kho_restore_free(ser);
    return 0;

Because retrieve_status is 0 on success, this finish function will fall through the if statement instead of returning early. It will then access the freed ser pointer and mistakenly discard all the successfully restored folios.

Would checking if (!args->retrieve_status) avoid this use-after-free regression?

Quoted 11 lines
virt/kvm/guest_memfd_luo.c new file +11 -0
+
+	if (ser->nr_folios) {
+		folios_ser = kho_restore_vmalloc(&ser->folios);
+		if (folios_ser) {
+			kvm_gmem_luo_discard_folios(folios_ser, ser->nr_folios, 0);
+			vfree(folios_ser);
+		}
+	}
+
+	kho_restore_free(ser);
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
#54 lore raw

Re: [PATCH v4 08/11] KVM: guest_memfd: Add support for preservation via LUO

In reply to Tarun Sahu
Quoted 12 lines · Tarun Sahu

[...snip...]

virt/kvm/guest_memfd_luo.c new file +9 -0
+static bool kvm_gmem_luo_can_preserve(struct liveupdate_file_handler *handler, struct file *file)
+{
+	struct inode *inode = file_inode(file);
+	struct gmem_file *gmem_file;
+	struct kvm *kvm;
+
+	if (inode->i_sb->s_magic != GUEST_MEMFD_MAGIC)
+		return false;
+

virt/kvm/guest_memfd_luo.c line 124

Quoted 10 lines
virt/kvm/guest_memfd_luo.c new file +10 -0
+	gmem_file = file->private_data;
+	if (!gmem_file)
+		return false;
+
+	/*
+	 * Only Fully-shared guest_memfd preservation is supported
+	 */
+	if (!(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED))
+		return false;
+

The above check can probably be removed. I think the core thing about this series is "we can preserve any memory that doesn't have any encryption state that goes along with it", and so the check for kvm_arch_has_private_mem() below would be sufficient to gate that.

virt/kvm/guest_memfd_luo.c line 133

Quoted 9 lines
virt/kvm/guest_memfd_luo.c new file +9 -0
+	/*
+	 * It makes sure that no memory can be converted to private
+	 * even if it was initially fully shared (in-place conversions are
+	 * prevented).
+	 */
+	kvm = gmem_file->kvm;
+	if (kvm_arch_has_private_mem(kvm))
+		return false;
+

For now, has_private_mem == has some encryption state that we can't preserve yet.

This also gates off KVM_X86_SW_PROTECTED_VM, but I guess that is intended, since when work is done for preserving encryption state, we can then test using KVM_X86_SW_PROTECTED_VM, and only then, remove this check?

Quoted 9 lines
virt/kvm/guest_memfd_luo.c new file +6 -0
+	if (mapping_large_folio_support(inode->i_mapping))
+		return false;
+
+	return true;
+}
+
 
[...snip...]
#55 lore raw

Re: [PATCH v4 08/11] KVM: guest_memfd: Add support for preservation via LUO

Quoted 21 lines · Ackerley Tng
Quoted 19 lines · Tarun Sahu

[...snip...]

virt/kvm/guest_memfd_luo.c new file +16 -0
+static bool kvm_gmem_luo_can_preserve(struct liveupdate_file_handler *handler, struct file *file)
+{
+	struct inode *inode = file_inode(file);
+	struct gmem_file *gmem_file;
+	struct kvm *kvm;
+
+	if (inode->i_sb->s_magic != GUEST_MEMFD_MAGIC)
+		return false;
+
+	gmem_file = file->private_data;
+	if (!gmem_file)
+		return false;
+
+	/*
+	 * Only Fully-shared guest_memfd preservation is supported
+	 */
Quoted 8 lines

virt/kvm/guest_memfd_luo.c line 124

Quoted 3 lines
virt/kvm/guest_memfd_luo.c new file +3 -0
+	if (!(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED))
+		return false;
+

The above check can probably be removed. I think the core thing about this series is "we can preserve any memory that doesn't have any encryption state that goes along with it", and so the check for kvm_arch_has_private_mem() below would be sufficient to gate that.

Sounds Good. Will remove it in next version. I will rephrase the
Line: "Support only fully shared guest_memfd" => "Support only guest_memfd that does not support private pages and also can not be converted to private"

Quoted 14 lines

virt/kvm/guest_memfd_luo.c line 133

Quoted 10 lines
virt/kvm/guest_memfd_luo.c new file +9 -0
+	/*
+	 * It makes sure that no memory can be converted to private
+	 * even if it was initially fully shared (in-place conversions are
+	 * prevented).
+	 */
+	kvm = gmem_file->kvm;
+	if (kvm_arch_has_private_mem(kvm))
+		return false;
+

For now, has_private_mem == has some encryption state that we can't preserve yet.

Quoted 4 lines

This also gates off KVM_X86_SW_PROTECTED_VM, but I guess that is intended, since when work is done for preserving encryption state, we can then test using KVM_X86_SW_PROTECTED_VM, and only then, remove this check?

Yes, It will be enabled when private memory preservation will be supported.

Quoted 10 lines
Quoted 10 lines
virt/kvm/guest_memfd_luo.c new file +6 -0
+	if (mapping_large_folio_support(inode->i_mapping))
+		return false;
+
+	return true;
+}
+
 
[...snip...]
author
#56 lore raw

PATCH v4 9/11 docs: liveupdate: Add documentation for VM and guest_memfd preservation

In reply to Tarun Sahu

Add Documentation/liveupdate/vmm.rst detailing VMM requirements, scope, ordering constraints, and kernel-doc inclusions for KVM VM and guest_memfd preservation via Live Update Orchestrator (LUO).

Also update Documentation/core-api/liveupdate.rst and MAINTAINERS.

---
 Documentation/core-api/liveupdate.rst |   1 +
 Documentation/liveupdate/vmm.rst      | 107 ++++++++++++++++++++++++++
 MAINTAINERS                           |   1 +
 3 files changed, 109 insertions(+)
 create mode 100644 Documentation/liveupdate/vmm.rst
Documentation/core-api/liveupdate.rst +1 -0
index 5a292d0f3706..bac58a363151 100644
@@ -34,6 +34,7 @@ The following types of file descriptors can be preserved
    :maxdepth: 1
 
    ../mm/memfd_preservation
+   ../liveupdate/vmm
 
 Public API
 ==========
Documentation/liveupdate/vmm.rst new file +107 -0

107 lines not on this page · raw message

MAINTAINERS +1 -0
index 8db8bb448b23..c032f9a0451b 100644
@@ -14548,6 +14548,7 @@ L:	[email protected]
 L:	[email protected]
 S:	Maintained
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git
+F:	Documentation/liveupdate/vmm.rst
 F:	virt/kvm/guest_memfd_luo.c
 F:	virt/kvm/kvm_luo.c
 
Signature · 1 line
2.55.0.229.g6434b31f56-goog
#57 lore raw

Re: [PATCH v4 09/11] docs: liveupdate: Add documentation for VM and guest_memfd preservation

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Mismatched kernel-doc section titles cause Sphinx build warnings and missing documentation. --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Mismatched kernel-doc section titles cause Sphinx build warnings and missing documentation. --

--- Patch [9]: [PATCH v4 09/11] docs: liveupdate: Add documentation for VM and guest_memfd preservation --- commit a930d2f58a9edc06bdcfd10fb383e97d7fe3996c
Author: Tarun Sahu <[email protected]>

docs: liveupdate: Add documentation for VM and guest_memfd preservation

Add Documentation/liveupdate/vmm.rst detailing VMM requirements, scope, ordering constraints, and kernel-doc inclusions for KVM VM and guest_memfd preservation via Live Update Orchestrator (LUO).

Quoted 7 lines
Documentation/liveupdate/vmm.rst new file +1 -0
new file mode 100644
index 0000000000000..b8e8fb2b63c21
@@ -0,0 +1,107 @@
+.. SPDX-License-Identifier: GPL-2.0-or-later

Documentation/liveupdate/vmm.rst line 11

Quoted 10 lines
Documentation/liveupdate/vmm.rst new file +10 -0
+
+=============================
+VM & Guest_Memfd Preservation
+=============================
+
+.. kernel-doc:: virt/kvm/kvm_luo.c
+   :doc: KVM VM Preservation via LUO
+
+.. kernel-doc:: virt/kvm/guest_memfd_luo.c
+   :doc: Guest_Memfd Preservation via LUO

[Severity: Low]
Does this section title exactly match the DOC: title in virt/kvm/guest_memfd_luo.c?

If the source file uses "Guestmemfd Preservation via LUO" without the underscore, this directive will fail to find the documentation and generate a Sphinx build warning.

[ ... ]

Documentation/liveupdate/vmm.rst line 98

Quoted 5 lines
Documentation/liveupdate/vmm.rst new file +5 -0
+VM & Guest_Memfd Preservation ABI
+=================================
+
+.. kernel-doc:: include/linux/kho/abi/kvm.h
+   :doc: DOC: guest_memfd Live Update ABI

[Severity: Low]
Does this section title match the actual title in include/linux/kho/abi/kvm.h?

If the source file uses "KVM and guest_memfd Live Update ABI", this mismatch will cause empty sections in the generated documentation.

Also, should the literal "DOC: " prefix be omitted here? The directive typically only expects the text that follows the prefix.

Quoted 5 lines
Documentation/liveupdate/vmm.rst new file +5 -0
+
+.. kernel-doc:: include/linux/kho/abi/kvm.h
+   :internal:
+
+See Also
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
#58 lore raw

Re: [PATCH v4 09/11] docs: liveupdate: Add documentation for VM and guest_memfd preservation

Quoted 36 lines · [email protected] writes:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Mismatched kernel-doc section titles cause Sphinx build warnings and missing documentation. --

--- Patch [9]: [PATCH v4 09/11] docs: liveupdate: Add documentation for VM and guest_memfd preservation --- commit a930d2f58a9edc06bdcfd10fb383e97d7fe3996c
Author: Tarun Sahu <[email protected]>

docs: liveupdate: Add documentation for VM and guest_memfd preservation

Add Documentation/liveupdate/vmm.rst detailing VMM requirements, scope, ordering constraints, and kernel-doc inclusions for KVM VM and guest_memfd preservation via Live Update Orchestrator (LUO).

Quoted 7 lines
Documentation/liveupdate/vmm.rst new file +1 -0
new file mode 100644
index 0000000000000..b8e8fb2b63c21
@@ -0,0 +1,107 @@
+.. SPDX-License-Identifier: GPL-2.0-or-later

Documentation/liveupdate/vmm.rst line 11

Quoted 10 lines
Documentation/liveupdate/vmm.rst new file +10 -0
+
+=============================
+VM & Guest_Memfd Preservation
+=============================
+
+.. kernel-doc:: virt/kvm/kvm_luo.c
+   :doc: KVM VM Preservation via LUO
+
+.. kernel-doc:: virt/kvm/guest_memfd_luo.c
+   :doc: Guest_Memfd Preservation via LUO

[Severity: Low]
Does this section title exactly match the DOC: title in virt/kvm/guest_memfd_luo.c?

Quoted 3 lines

If the source file uses "Guestmemfd Preservation via LUO" without the underscore, this directive will fail to find the documentation and generate a Sphinx build warning.

I will correct it. Good find!

Quoted 26 lines

[ ... ]

Documentation/liveupdate/vmm.rst line 98

Quoted 5 lines
Documentation/liveupdate/vmm.rst new file +5 -0
+VM & Guest_Memfd Preservation ABI
+=================================
+
+.. kernel-doc:: include/linux/kho/abi/kvm.h
+   :doc: DOC: guest_memfd Live Update ABI

[Severity: Low]
Does this section title match the actual title in include/linux/kho/abi/kvm.h?

If the source file uses "KVM and guest_memfd Live Update ABI", this mismatch will cause empty sections in the generated documentation.

Also, should the literal "DOC: " prefix be omitted here? The directive typically only expects the text that follows the prefix.

Quoted 5 lines
Documentation/liveupdate/vmm.rst new file +5 -0
+
+.. kernel-doc:: include/linux/kho/abi/kvm.h
+   :internal:
+
+See Also
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
author
#59 lore raw

PATCH v4 10/11 KVM: selftests: Split ____vm_create() and add vm_create_from_fd()

In reply to Tarun Sahu

Refactor ____vm_create() in the KVM selftest library to extract its initialization steps into separate helpers vm_init_fields() and vm_init_memory_properties(). Also introduce vm_create_from_fd() to instantiate a struct kvm_vm from an existing VM file descriptor.

This allows test setups (such as guest_memfd preservation tests) to reconstruct and initialize a struct kvm_vm when restoring preserved VMs across live updates. No functional change is introduced for existing tests.

---
 .../testing/selftests/kvm/include/kvm_util.h  |  3 ++
 tools/testing/selftests/kvm/lib/kvm_util.c    | 49 ++++++++++++++++---
 2 files changed, 44 insertions(+), 8 deletions(-)
tools/testing/selftests/kvm/include/kvm_util.h +3 -0
index 04a910164a29..0626efa5a73b 100644
@@ -471,6 +471,9 @@ const char *vm_guest_mode_string(u32 i);
 
 void kvm_vm_free(struct kvm_vm *vmp);
 void kvm_vm_restart(struct kvm_vm *vmp);
+void vm_init_fields(struct kvm_vm *vm, struct vm_shape shape);
+void vm_init_memory_properties(struct kvm_vm *vm);
+struct kvm_vm *vm_create_from_fd(int vm_fd, struct vm_shape shape);
 void kvm_vm_release(struct kvm_vm *vmp);
 void kvm_vm_elf_load(struct kvm_vm *vm, const char *filename);
 int kvm_memfd_alloc(size_t size, bool hugepages);
tools/testing/selftests/kvm/lib/kvm_util.c +41 -8

73 lines not on this page · raw message

Signature · 1 line
2.55.0.229.g6434b31f56-goog
author
#60 lore raw

PATCH v4 11/11 KVM: selftests: Add guest_memfd_preservation_test

In reply to Tarun Sahu

Add guest_memfd_preservation_test to verify that KVM VM state and guest memory backed by guest_memfd are preserved across a kexec reboot via Live Update Orchestrator (LUO).

This test includes multiple tests
- (A) Test to preserve non-guest_memfd, non-memfd which must fail (/dev/null)
- (B) Test to preserve guest_memfd with no INIT_SHARED flag which must fail
- (C) Test to allocate the guest_memfd memory after preservation because preservation makes the guest_memfd frozen. Which must fail as well.
- (D) Test to verify guest_memfd with INIT_SHARED flag preservation across
kexec: Below is the details of stages in this test.

In Stage 1 (pre-kexec), create a VM with a guest_memfd memory slot, run guest code to write a deterministic pattern into guest_memfd, and preserve both the VM and guest_memfd file descriptors via LUO. This stage also tests A, B and C. A, B and C do not require stage2 so they will finish in stage 1 only.

In Stage 2 (post-kexec), retrieve the preserved VM and guest_memfd file descriptors, reconstruct the VM using vm_create_from_fd(), re-map the retrieved guest_memfd, and execute guest code to verify data integrity against the original pattern.

Example usage:

  $ ./guest_memfd_preservation_test --s 1
  $ <kexec>
  $ ./guest_memfd_preservation_test --s 2
---
 MAINTAINERS                                   |   1 +
 tools/testing/selftests/kvm/Makefile.kvm      |   6 +-
 .../kvm/guest_memfd_preservation_test.c       | 357 ++++++++++++++++++
 3 files changed, 363 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/kvm/guest_memfd_preservation_test.c
MAINTAINERS +1 -0
index c032f9a0451b..df764e430d25 100644
@@ -14549,6 +14549,7 @@ L:	[email protected]
 S:	Maintained
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git
 F:	Documentation/liveupdate/vmm.rst
+F:	tools/testing/selftests/kvm/guest_memfd_preservation_test.c
 F:	virt/kvm/guest_memfd_luo.c
 F:	virt/kvm/kvm_luo.c
 
tools/testing/selftests/kvm/Makefile.kvm +5 -1
index 4ace12606e93..02cfd7fc718e 100644
@@ -165,6 +165,8 @@ TEST_GEN_PROGS_x86 += pre_fault_memory_test
 
 # Compiled outputs used by test targets
 TEST_GEN_PROGS_EXTENDED_x86 += x86/nx_huge_pages_test
+# Manual test that forks a persistent background daemon; skip auto CI run
+TEST_GEN_PROGS_EXTENDED_x86 += guest_memfd_preservation_test
 
 TEST_GEN_PROGS_arm64 = $(TEST_GEN_PROGS_COMMON)
 TEST_GEN_PROGS_arm64 += arm64/aarch32_id_regs
@@ -259,6 +261,7 @@ OVERRIDE_TARGETS = 1
 # which causes the environment variable to override the makefile).
 include ../lib.mk
 include ../cgroup/lib/libcgroup.mk
+include ../liveupdate/lib/libliveupdate.mk
 
 INSTALL_HDR_PATH = $(top_srcdir)/usr
 LINUX_HDR_PATH = $(INSTALL_HDR_PATH)/include/
@@ -313,7 +316,8 @@ LIBKVM_S := $(filter %.S,$(LIBKVM))
 LIBKVM_C_OBJ := $(patsubst %.c, $(OUTPUT)/%.o, $(LIBKVM_C))
 LIBKVM_S_OBJ := $(patsubst %.S, $(OUTPUT)/%.o, $(LIBKVM_S))
 LIBKVM_STRING_OBJ := $(patsubst %.c, $(OUTPUT)/%.o, $(LIBKVM_STRING))
-LIBKVM_OBJS = $(LIBKVM_C_OBJ) $(LIBKVM_S_OBJ) $(LIBKVM_STRING_OBJ) $(LIBCGROUP_O)
+LIBKVM_OBJS = $(LIBKVM_C_OBJ) $(LIBKVM_S_OBJ) $(LIBKVM_STRING_OBJ) \
+						$(LIBCGROUP_O) $(LIBLIVEUPDATE_O)
 SPLIT_TEST_GEN_PROGS := $(patsubst %, $(OUTPUT)/%, $(SPLIT_TESTS))
 SPLIT_TEST_GEN_OBJ := $(patsubst %, $(OUTPUT)/$(ARCH)/%.o, $(SPLIT_TESTS))
 
tools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +357 -0

357 lines not on this page · raw message

Signature · 1 line
2.55.0.229.g6434b31f56-goog
#61 lore raw

Re: [PATCH v4 11/11] KVM: selftests: Add guest_memfd_preservation_test

Quoted 4 lines · Tarun Sahu

Add guest_memfd_preservation_test to verify that KVM VM state and guest memory backed by guest_memfd are preserved across a kexec reboot via Live Update Orchestrator (LUO).

Quoted 8 lines

This test includes multiple tests
- (A) Test to preserve non-guest_memfd, non-memfd which must fail (/dev/null)
- (B) Test to preserve guest_memfd with no INIT_SHARED flag which must fail
- (C) Test to allocate the guest_memfd memory after preservation because preservation makes the guest_memfd frozen. Which must fail as well.
- (D) Test to verify guest_memfd with INIT_SHARED flag preservation across
kexec: Below is the details of stages in this test.

I think we should split this into multiple patches, where each patch introduces 1 test.

Quoted 9 lines

[...snip...]

+++ b/tools/testing/selftests/kvm/guest_memfd_preservation_test.c

tools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +4 -0
@@ -0,0 +1,357 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2026, Google LLC.
+ *

tools/testing/selftests/kvm/guest_memfd_preservation_test.c line 14

Quoted 10 lines
tools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +10 -0
+ * Author: Tarun Sahu <[email protected]>
+ *
+ * Test for VM and guest_memfd preservation across kexec (Live Update) via LUO.
+ *
+ * NOTE: This is a MANUAL test and is excluded from automated CI/testing
+ * frameworks because Stage 1 daemonizes into the background to pin resources
+ * and requires a human operator to manually trigger kexec before Stage 2
+ * is executed. Running Stage 1 automatically would leak the background daemon
+ * and cause CI runners to falsely interpret it as a passed test.
+ *

Usually there aren't many printfs in tests, but since this is a MANUAL test, maybe that's ok?

Would like to know what Sean thinks of this, and printfs in general.

Quoted 3 lines

[...snip...]

#62 lore raw

Re: [PATCH v4 11/11] KVM: selftests: Add guest_memfd_preservation_test

Quoted 15 lines · Ackerley Tng
Quoted 4 lines · Tarun Sahu

Add guest_memfd_preservation_test to verify that KVM VM state and guest memory backed by guest_memfd are preserved across a kexec reboot via Live Update Orchestrator (LUO).

Quoted 8 lines

This test includes multiple tests
- (A) Test to preserve non-guest_memfd, non-memfd which must fail (/dev/null)
- (B) Test to preserve guest_memfd with no INIT_SHARED flag which must fail
- (C) Test to allocate the guest_memfd memory after preservation because preservation makes the guest_memfd frozen. Which must fail as well.
- (D) Test to verify guest_memfd with INIT_SHARED flag preservation across
kexec: Below is the details of stages in this test.

Quoted 2 lines

I think we should split this into multiple patches, where each patch introduces 1 test.

Got it. I will split them. I thought that this test already spawn a VM and other test can just reuse the same VM.

Quoted 29 lines
Quoted 10 lines

[...snip...]

+++ b/tools/testing/selftests/kvm/guest_memfd_preservation_test.c

tools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +4 -0
@@ -0,0 +1,357 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2026, Google LLC.
+ *

tools/testing/selftests/kvm/guest_memfd_preservation_test.c line 14

Quoted 10 lines
tools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +10 -0
+ * Author: Tarun Sahu <[email protected]>
+ *
+ * Test for VM and guest_memfd preservation across kexec (Live Update) via LUO.
+ *
+ * NOTE: This is a MANUAL test and is excluded from automated CI/testing
+ * frameworks because Stage 1 daemonizes into the background to pin resources
+ * and requires a human operator to manually trigger kexec before Stage 2
+ * is executed. Running Stage 1 automatically would leak the background daemon
+ * and cause CI runners to falsely interpret it as a passed test.
+ *

Usually there aren't many printfs in tests, but since this is a MANUAL test, maybe that's ok?

Would like to know what Sean thinks of this, and printfs in general.

Quoted 3 lines

[...snip...]

#63 lore raw

Re: [PATCH v4 11/11] KVM: selftests: Add guest_memfd_preservation_test

Quoted 17 lines · Tarun Sahu
Quoted 15 lines · Ackerley Tng
Quoted 12 lines · Tarun Sahu

Add guest_memfd_preservation_test to verify that KVM VM state and guest memory backed by guest_memfd are preserved across a kexec reboot via Live Update Orchestrator (LUO).

This test includes multiple tests
- (A) Test to preserve non-guest_memfd, non-memfd which must fail (/dev/null)
- (B) Test to preserve guest_memfd with no INIT_SHARED flag which must fail
- (C) Test to allocate the guest_memfd memory after preservation because preservation makes the guest_memfd frozen. Which must fail as well.
- (D) Test to verify guest_memfd with INIT_SHARED flag preservation across
kexec: Below is the details of stages in this test.

Quoted 5 lines
Quoted 2 lines

I think we should split this into multiple patches, where each patch introduces 1 test.

Got it. I will split them. I thought that this test already spawn a VM and other test can just reuse the same VM.

From upstream hypervisor liveupdate meeting, I realized you meant to be same file but saperate commits. Sound good to me. I will take care of that.

Quoted 30 lines
Quoted 30 lines
Quoted 11 lines

[...snip...]

+++ b/tools/testing/selftests/kvm/guest_memfd_preservation_test.c

tools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +4 -0
@@ -0,0 +1,357 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2026, Google LLC.
+ *

tools/testing/selftests/kvm/guest_memfd_preservation_test.c line 14

Quoted 10 lines
tools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +10 -0
+ * Author: Tarun Sahu <[email protected]>
+ *
+ * Test for VM and guest_memfd preservation across kexec (Live Update) via LUO.
+ *
+ * NOTE: This is a MANUAL test and is excluded from automated CI/testing
+ * frameworks because Stage 1 daemonizes into the background to pin resources
+ * and requires a human operator to manually trigger kexec before Stage 2
+ * is executed. Running Stage 1 automatically would leak the background daemon
+ * and cause CI runners to falsely interpret it as a passed test.
+ *

Usually there aren't many printfs in tests, but since this is a MANUAL test, maybe that's ok?

Would like to know what Sean thinks of this, and printfs in general.

Quoted 3 lines

[...snip...]

maintainer
#64 lore raw

Re: [PATCH v4 11/11] KVM: selftests: Add guest_memfd_preservation_test

In reply to Ackerley Tng
Quoted 13 lines · Ackerley Tng
Quoted 11 lines · Tarun Sahu

Add guest_memfd_preservation_test to verify that KVM VM state and guest memory backed by guest_memfd are preserved across a kexec reboot via Live Update Orchestrator (LUO).

This test includes multiple tests
- (A) Test to preserve non-guest_memfd, non-memfd which must fail (/dev/null)
- (B) Test to preserve guest_memfd with no INIT_SHARED flag which must fail
- (C) Test to allocate the guest_memfd memory after preservation because preservation makes the guest_memfd frozen. Which must fail as well.
- (D) Test to verify guest_memfd with INIT_SHARED flag preservation across
kexec: Below is the details of stages in this test.

[...]

Quoted 8 lines
Quoted 8 lines

+++ b/tools/testing/selftests/kvm/guest_memfd_preservation_test.c

tools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +6 -0
@@ -0,0 +1,357 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2026, Google LLC.
+ *
+ * Author: Tarun Sahu <[email protected]>
+ *
Quoted 8 lines

tools/testing/selftests/kvm/guest_memfd_preservation_test.c line 14

Quoted 8 lines
tools/testing/selftests/kvm/guest_memfd_preservation_test.c new file +8 -0
+ * Test for VM and guest_memfd preservation across kexec (Live Update) via LUO.
+ *
+ * NOTE: This is a MANUAL test and is excluded from automated CI/testing
+ * frameworks because Stage 1 daemonizes into the background to pin resources
+ * and requires a human operator to manually trigger kexec before Stage 2
+ * is executed. Running Stage 1 automatically would leak the background daemon
+ * and cause CI runners to falsely interpret it as a passed test.
+ *

I think this is fine as a first step but we really should automate these.

We have some infrastructure to automate other live update tests across both stages already by launching a VM under qemu and making the test process as the init task. See tools/testing/selftests/kho/vmtest.sh. For KVM tests you'd need the runner to be able to run nested VMs.

The script right now is pretty bare bones, but I really think we should spend time improving it to the point where we can automate all our tests using it.

Quoted 5 lines

Usually there aren't many printfs in tests, but since this is a MANUAL test, maybe that's ok?

Would like to know what Sean thinks of this, and printfs in general.

Signature · 2 lines
Regards,
Pratyush Yadav
author
#65 lore raw
In reply to Tarun Sahu

Hello,

As per the discussion on the series, I will try to layout the future plan of guest_memfd preservation and VM preservation via LUO. Though guest_memfd itself is in development phase, Deducing the exact plan and ABI beforehand is quite impossible. So I will propose tentative plan as per my current understanding.

Introduction:
Guest_memfd was developed only for CocoVM use-case, But now its target is to unify guest memory for cocoVM and non-cocoVM. Though not all user will be onboarded on using guest_memfd with non-CocoVM, I see current development in guest_memfd for e.g multiple backends will provide its usefulness for non-CocoVM (Maybe get rid of hugetlbfs - traditional PITA).

This preservation series and future development targets the preservation for both cocoVM and non-cocoVM.

Tentative Plan (Overview):
I plan on supporting the guest_memfd preservation on incremental steps. This series builds the foundation for it and only support fully shared guest_memfd backed by PAGE_SIZE pages. Currently, For this base series, Only potential user would be non-CocoVM. But due to performance reasons, at this stage it can not be used (as it is backed by PAGE_SIZE).

Incremental Steps:
A. This base foundation series (Fully shared backed by PAGE_SIZE pages) also require VM preservation [1]. but currently preserving single field only (VM_TYPE).
This sets the foundation for VM preservation via LUO as well. Present KVM uAPIs can support preservation for non-CocoVM backed by non-guest_memfd memory. But they are not sufficient to preserve the cocoVM [2].

B. Next Step, Private guest_memfd preservation on top of in-place conversion series (merged in KVM x86 cocoVM tree) again backed by PAGE_SIZE pages. This will also include VM preservation, for example the data structures that are TDX protected can not be preserved with current KVM uAPIs [2].

C. Parallel to B, guest_memfd preservation for non-CocoVM backed by hugetlb. This work will be on top of guest_memfd support for hugetlb backend [3][4]. Currently, How this support will be developed is in upstream discussion. But The work on preservation will pick it as soon as hugetlb support is integrated.

D. On Top of C, Extends guest_memfd preservation for hugetlb backend for cocoVM (Integrating private guest_memfd support)

E. Preservation for any other backend that will be supported in future.

VM Preservation:
Guest_memfd preservation can not be supported alone, it needs VM preservation as well [1][2]. Even though there are no current user, But there are users who want guest_memfd preservation for non-cocoVM and cocoVM both. And that opens the door for VM preservation as well because present KVM uAPIs are not sufficient (for cocoVM case [2]). This will unify preservation mechanism for VM (and its memory) in linux.

Presently, for non-CocoVM, KVM provides uAPIs to preserve the state of the VM for e.g. vCPU + device state through the (KVM_GET/SET_REGS, SREGS2, XSAVE, MSRS, LAPIC, VCPU_EVENTS, MP_STATE, NESTED_STATE, CLOCK, ...))

Now, for cocoVM, we need to preserve the TD state of the VM and global TDX module state. For which preservation of the following will be needed (Not everything included, it is as much I could think at this time, Also I am not expert on TDX, I will try)

**Global State (Will use LUO FLB objects)**:
1. TDMRs + PAMT locations
2. In-use HKID bitmap

**Per-TD/VM (part of per-VM preservation, It can open door to unify the place of preservation for other fields as well. But again open for discussion)**
3. VM_TYPE (already preserved)
4. TDR Page
5. TDCS Page
6. HKID
7. S-EPT non-leaf pages
8. S-EPT leaf maps
9. per VM memory attributes (unrelated to TDX)

**per-vCPUs (preserved via vCPU fd)**
10. vCPU ID
11. TDVPR page
12. TDCX page
etc.
Note: Also, why not extends
the support for full preservation of vCPUs ignoring existing KVM uAPIs to unify everything at one place: Open to discussion and can be discussed thoroughly once RFC for TDX preservation will be sent.

Guest_memfd Preservation:
1. per gmem memory attributes
2. folios preservation (size/number of pages per folio, index etc)
3. State of hugetlb preservation (again LUO FLB object will be needed to preserve system shared pool information)
4. SPLIT information for hugetlb for shared pages
5. backing type of guest_memfd
6. gmem related flags
7. mem policy
etc.

The above plan is currently based on my current understanding which can be changed as guest_memfd evolve and new findings come to the light while during the work/review of RFC of individual features.

[1]: https://lore.kernel.org/all/[email protected]/#:~:text=Guest_memfd%20can%27t%20be,KVM%20side%20preservation.
[2]
KVM uAPI fetch data to userspace and restore it back in new kernel. for
TDX case, that is not possible becuase of 2 major reasons:
A. all control data structures are protected and encrypted,
Host cannot access them if they do not belong to host.
While LUO just have to handover the PFN to new kernel,
So LUO does not access the underlying data.
B. TDX module can not be freshly initialized, it will lose
the state of the protected VMs and their relation data.
So preserving using KVM uAPIs and reinitializing it and
populating it via userspace is not an option. It has to be
done from kernel. and kexec handover/LUO can handle it very
well by updating TDX initializing sequence in kernel (This
is one of the Idea currently in my mind, topic is open for
exploration).

[3]: https://lore.kernel.org/all/[email protected]/
[4]: https://lore.kernel.org/all/[email protected]/

I wanted to send this before LPC so that we can pick up this for discussion on Tues in either KVM MC (13:00 KVM and Live Update) Slot [5] or Live Update (16:10 Guest_Memfd Preservation For Live Update) Slot [6] or for general discussion on liveupdate ABI (15:50 Live Update Compatibility) Slot [7].

[5]: https://lpc.events/event/20/contributions/2632/
[6]: https://lpc.events/event/20/contributions/2610/
[7]: https://lpc.events/event/20/contributions/2613/

[5] is before [6] & [7], So, We can also have any other slots as per [8] on wed.

[8]: https://lore.kernel.org/all/[email protected]/#r

~Tarun

Quoted 202 lines · Tarun Sahu

Hello,

This is v4 of guest_memfd preservation during liveupdate. As per the discussion from bi-weekly guest-memfd meeting, This series will be go to linux via kvm tree. And Planning to get it merged in upcoming merge window. (7.2-rc1 for release in 7.3)

The changes are rebased on:
kvm/next + liveupdate/next (merge) + [4] + [5] Where,
[4]: luo: APIs to retrieve file internally from session [5]: selftests: liveupdate sefltests library
Here is the github repo:
https://github.com/tar-unix/linux/tree/gmem-pre

Patches [4] and [5] are prequisite of this series and they are rebased/merged on liveupdate/next. kvm/next is behind few patches from liveupdate/next, hence direct cherry-pick of [4] and [5] is not possible. On Linux 7.2 they will be aligned. If we succeed in merging it before I can request liveupdate maintainer to provide the tag.

Steps to test:
1. Compile Kernel with CONFIG_LIVEUPDATE_GUEST_MEMFD=y
2. boot kernel with command line: kho=on liveupdate=on
3. run the following kselftest

        $ .selftests/kvm/guest_memfd_preservation_test --s 1
        $ <kexec> --reuse-cmdline
        $ .selftests/kvm/guest_memfd_preservation_test --s 2

NOTE: Assert the following:

        $ ls /dev/liveupdate
        $ ls /dev/kvm
        $ dmesg | grep liveupdate # (should have kvm_vm_luo &&
                # guest_memfd_luo handler registered)

V4 <- V3
1. Split the refactoring patch into multiple patches [2, 3, 4]
2. Updated commits msg as per the kvm documentation and suggestions
3. Removed KHOSER_PTR and using raw pointer, as series [3] not yet decided to go in the next cycle
4. Implemented more tests in guest_memfd_preservation_test allocation during frozen gmem_inode, non-guest_memfd preservation
5. Moved the vm_create_from_fd to kvm_util.c from the guest_memfd_preservation_test.c
5. Minor updates as per the suggestions

V3 <- RFC V2 [2]
1. Finalize the design
2. resolve sashiko reported bugs
3. Use of KHOSER_PTR instead of raw serialized_data as per [3]

RFC V2 [2] <- RFC V1 [1]
1. Removed mem_attr_array as it is not needed for fully-shared
2. Removed pre-faulted condition
3. Added vm_type preservation for ARM64.
4. Removed liveupdate_get_file_incoming api patch as it is sent separately [4] by Samiullah.

**GUEST MEMFD PRESERVATION**
Below is the cover letter of the series which explains the design.

SCOPE:
1. Fully Shared Guest_memfd
2. Guest_memfd backed by PAGE_SIZE pages

Any VM whose memory is backed by such guest_memfd and satisfy the above conditions can be preserved across liveupdate.

To Add support for liveupdate of any subsystem, we need to implement the following calls: (e.g: subsystem: guest_memfd)

guest_memfd_luo_can_preserve: used by luo subsystem to filter the
guest_memfd files
guest_memfd_luo_preserve: triggered from userspace by PRESERVE ioctl
If can_preserve says the file is preservable by guest_memfd_luo_preserve
guest_memfd_luo_retrieve: triggered from userspace by RETRIEVE ioctl,
recognised by the passed token from userspace. Used in new kernel after kexec.
guest_memfd_luo_unpreserve: triggered from userspace when a session is
closed, which inherently calls unpreserve of all preserved files.
guest_memfd_luo_freeze: triggered just before kexec automatically in
kernel. no userspace involvement.
guest_memfd_luo_finish: triggered by userspace when all files are
retrieved and want to finish the session retrieval.

*PART A: KVM VM preservation*
Introdcued the kvm_luo.c to preserve the vm_file during liveupdate. Currently it only preserve vm_type. This can be expanded to preserve more details in future as need arises for example data related to cocoVM (SEPT etc.).

        int kvm_fd = open("/dev/kvm", O_RDWR);
        int vm_fd = ioctl(kvm_fd, KVM_CREATE_VM, 0);
        preserve_arg = {
        .size = sizeof(preserve_arg),
            .fd = vm_fd,
            .token = VM_TOKEN,
        };
        ioctl(session_fd, LIVEUPDATE_SESSION_PRESERVE_FD, &preserve_arg);

This VM_TOKEN is used by guest_memfd preservation. guest_memfd preservation save this token (fetched internally by luo apis [4]) to associates the guest_memfd with its vm_file. On retrieval, guest_memfd retrieves the vm_file using this token and use its struct kvm to retrieve itself.

PART B: Guest_memfd preservation
Introduce guest_memfd_luo.c to preserve the guest_memfd file during liveupdate. Currently it only preserve guest_memfd files that have:

1. INIT_SHARED flag ensures intially all the memory is shared
2. !kvm_arch_has_private_mem() which ensures no shared pages can be converted to private after [6] lands.

Preservation first freezes the gmem_inode introduced in PATCH [07/11] and preserve each allocated folios from guest_memfd page_cache using KHO.

luo_freeze (not the gmem_inode freeze PATCH[07/11]) call update the vm_token which is preserved across kexec.

Retrieval retrieves the vm_file using vm_token and create the guest_memfd with vm_file->kvm and populate the preserved folios to guest_memfd page_cache.

Added the Documentation for VMM on guest_memfd preservation guidelines To know more about, PATCH[10/11] can be referred which has the more detailed documentation.

Future Plan:
1. Make preservation immune to new allocation and fallocate calls so that we can remove freeze call
2. In future, When IOMMU will use guest_memfd and can allocate page tables for DMA in that area, Later it will be a problem if other user of guest_memfd do punch hole before preservation, that will make the pages to be lost after preservation. This will be taken care of when IOMMU will add support for guest_memfd.
3. guest_memfd preservation for private pages + in-place conversion.
4. In future, when guest_memfd will have hugetlb page support, add support its preservation.

[1]: https://lore.kernel.org/all/[email protected]/
[2]: https://lore.kernel.org/all/[email protected]/

     https://lore.kernel.org/all/c054ba0fb2639932bbe354420d3f4f84cce84905.1780676742.git.tarunsahu@google.com/
[3]: https://lore.kernel.org/all/[email protected]/
[4]: https://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git/commit/?h=next&id=f4d2e4f0d12d88155cd1128ed3294f7827bf4c5d
[5]: https://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git/commit/?h=next&id=2f3f53a3735a7e67bbe9e580cb49372bf9261967
     https://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git/commit/?h=next&id=df1a0d068d481b7d175c5af3970043206409a062

[6]: https://lore.kernel.org/all/[email protected]/

Tarun Sahu (11):
liveupdate: Add LIVEUPDATE_GUEST_MEMFD config option
KVM: Introduce kvm_create_vm_file() helper
KVM: Export kvm_uevent_notify_vm_create()
KVM: Track weak reference to vm_file in struct kvm
KVM: LUO: Support VM preservation across live updates
KVM: guest_memfd: Move internal definitions to internal header
KVM: guest_memfd: Add support for freezing mappings
KVM: guest_memfd: Add support for preservation via LUO
docs: liveupdate: Add documentation for VM and guest_memfd
preservation
KVM: selftests: Split ____vm_create() and add vm_create_from_fd()
KVM: selftests: Add guest_memfd_preservation_test

 Documentation/core-api/liveupdate.rst         |   1 +
 Documentation/liveupdate/vmm.rst              | 107 ++++
 MAINTAINERS                                   |  14 +
 include/linux/kho/abi/kvm.h                   | 106 ++++
 include/linux/kvm_host.h                      |  14 +
 kernel/liveupdate/Kconfig                     |  15 +
 tools/testing/selftests/kvm/Makefile.kvm      |   6 +-
 .../kvm/guest_memfd_preservation_test.c       | 357 ++++++++++++
 .../testing/selftests/kvm/include/kvm_util.h  |   3 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |  49 +-
 virt/kvm/Makefile.kvm                         |   1 +
 virt/kvm/guest_memfd.c                        | 185 +++++--
 virt/kvm/guest_memfd.h                        |  44 ++
 virt/kvm/guest_memfd_luo.c                    | 515 ++++++++++++++++++
 virt/kvm/kvm_luo.c                            | 195 +++++++
 virt/kvm/kvm_main.c                           |  94 +++-
 virt/kvm/kvm_mm.h                             |  15 +
 17 files changed, 1640 insertions(+), 81 deletions(-)
 create mode 100644 Documentation/liveupdate/vmm.rst
 create mode 100644 include/linux/kho/abi/kvm.h
 create mode 100644 tools/testing/selftests/kvm/guest_memfd_preservation_test.c
 create mode 100644 virt/kvm/guest_memfd.h
 create mode 100644 virt/kvm/guest_memfd_luo.c
 create mode 100644 virt/kvm/kvm_luo.c

base-commit: a204badd8432f93b7e862e7dac6db0fe3d65f370 --
2.55.0.229.g6434b31f56-goog