Skip to content

RFC mm: Refactor KVM guest_memfd to introduce guestmem library

Kalyazin, Nikita kvm linux-fsdevel linux-mm lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v6 RFC Latest2 patches 2 repliesIn review

v3 from the cover letter

- https://lore.kernel.org/kvm/[email protected]
- minor changes to address review comments (James)

Patches in v6 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/2mm: guestmem: introduce guestmem libraryMAINTAINERS include/linux/guestmem.h mm/Kconfig +4 more+465 -2711 reply
2/2userfaulfd: add minor mode for guestmemDocumentation/admin-guide/mm/userfaultfd.rst fs/userfaultfd.c include/linux/userfaultfd_k.h +2 more+28 -90 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v6 yet.

Read the v6 thread

Cover letter [RFC PATCH v6 0/2] mm: Refactor KVM guest_memfd to introduce guestmem library

This is a revival of the guestmem library patch series originated from Elliot [1]. The reason I am bringing it up now is it would help implement UserfaultFD support minor mode in guest_memfd.

Background

We are building a Firecracker version that uses guest_memfd to back guest memory [2]. The main objective is to use guest_memfd to remove guest memory from host kernel's direct map to reduce the surface for Spectre-style transient execution issues [3]. Currently, Firecracker supports restoring VMs from snapshots using UserfaultFD [4], which is similar to the postcopy phase of live migration. During restoration, while we rely on a separate mechanism to handle stage-2 faults in guest_memfd [5], UserfaultFD support in guest_memfd is still required to handle faults caused either by the VMM itself or by MMIO access handling on x86.

The major problem in implementing UserfaultFD for guest_memfd is that the MM code (UserfaultFD) needs to call KVM-specific interfaces. Particularly for the minor mode, these are 1) determining the type of the VMA (eg is_vma_guest_memfd()) and 2) obtaining a folio (ie kvm_gmem_get_folio()). Those may not be always available as KVM can be compiled as a module. Peter attempted to approach it via exposing an ops structure where modules (such as KVM) could provide their own callbacks, but it was not deemed to be sufficiently safe as it opens up an unrestricted interface for all modules and may leave MM in an inconsistent state [6].

An alternative way to make these interfaces available to the UserfaultFD code is extracting generic-MM guest_memfd parts into a library (guestmem) under MM where they can be safely consumed by the UserfaultFD code. As far as I know, the original guestmem library series was motivated by adding guest_memfd support in Gunyah hypervisor [7].

This RFC

I took Elliot's v5 (the latest) and rebased it on top of the guest_memfd preview branch [8] because I also wanted to see how it would work with direct map removal [3] and write syscall [9], which are building blocks for the guest_memfd-based Firecracker version. On top of it I added a patch that implements UserfaultFD support for guest_memfd using interfaces provided by the guestmem library to illustrate the complete idea.

I made the following modifications along the way:
- Followed by a comment from Sean, converted invalidate_begin() callback back to void as it cannot fail in KVM, and the related Gunyah requirement is unknown to me
- Extended the guestmem_ops structure with the supports_mmap() callback to provide conditional mmap support in guestmem
- Extended the guestmem library interface with guestmem_allocate(), guestmem_test_no_direct_map(), guestmem_mark_prepared(), guestmem_mmap(), and guestmem_vma_is_guestmem()
- Made (kvm_gmem)/(guestmem)_test_no_direct_map() use mapping_no_direct_map() instead of KVM-specific flag GUEST_MEMFD_FLAG_NO_DIRECT_MAP to make it KVM-independent

Feedback that I would like to receive:
- Is this the right solution to the "UserfaultFD in guest_memfd" problem?
- What requirements from other hypervisors than KVM do we need to consider at this point?
- Does the line between generic-MM and KVM-specific guest_memfd parts look sensible?

Previous iterations of UserfaultFD support in guest_memfd patches:
v3:
- https://lore.kernel.org/kvm/[email protected]
- minor changes to address review comments (James)
v2:
- https://lore.kernel.org/kvm/[email protected]
- implement a full minor trap instead of hybrid missing/minor trap (James/Peter)
- make UFFDIO_CONTINUE implementation generic calling vm_ops->fault()
v1:
- https://lore.kernel.org/kvm/[email protected]

Nikita

[1]: https://lore.kernel.org/kvm/[email protected]
[2]: https://github.com/firecracker-microvm/firecracker/tree/feature/secret-hiding
[3]: https://lore.kernel.org/kvm/[email protected]
[4]: https://github.com/firecracker-microvm/firecracker/blob/main/docs/snapshotting/handling-page-faults-on-snapshot-resume.md
[5]: https://lore.kernel.org/kvm/[email protected]
[6]: https://lore.kernel.org/linux-mm/[email protected]
[7]: https://lore.kernel.org/lkml/[email protected]
[8]: https://git.kernel.org/pub/scm/linux/kernel/git/david/linux.git/log/?h=guestmemfd-preview
[9]: https://lore.kernel.org/kvm/[email protected]

Nikita Kalyazin (2):
mm: guestmem: introduce guestmem library
userfaulfd: add minor mode for guestmem

 Documentation/admin-guide/mm/userfaultfd.rst |   4 +-
 MAINTAINERS                                  |   2 +
 fs/userfaultfd.c                             |   3 +-
 include/linux/guestmem.h                     |  46 +++
 include/linux/userfaultfd_k.h                |   8 +-
 include/uapi/linux/userfaultfd.h             |   8 +-
 mm/Kconfig                                   |   3 +
 mm/Makefile                                  |   1 +
 mm/guestmem.c                                | 380 +++++++++++++++++++
 mm/userfaultfd.c                             |  14 +-
 virt/kvm/Kconfig                             |   1 +
 virt/kvm/guest_memfd.c                       | 303 ++-------------
 12 files changed, 493 insertions(+), 280 deletions(-)
 create mode 100644 include/linux/guestmem.h
 create mode 100644 mm/guestmem.c

base-commit: 911634bac3107b237dcd8fdcb6ac91a22741cbe7

Signature · 1 line
2.50.1