Skip to content

RFC Support iommu(fd) persistence for live update

James Gowans kvm linux-iommu lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest13 patches 19 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/13iommufd: Support marking and tracking persistent iommufdsdrivers/iommu/iommufd/iommufd_private.h drivers/iommu/iommufd/main.c include/uapi/linux/iommufd.h+53 -00 replies
2/13iommufd: Add plumbing for KHO (de)serialisedrivers/iommu/iommufd/Makefile drivers/iommu/iommufd/iommufd_private.h drivers/iommu/iommufd/main.c +1 more+72 -10 replies
3/13iommu/intel: zap context table entries on kexecdrivers/iommu/intel/dmar.c drivers/iommu/intel/iommu.c drivers/iommu/intel/iommu.h+33 -41 reply
4/13iommu: Support marking domains as persistent on allocdrivers/iommu/amd/iommu.c drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c drivers/iommu/intel/iommu.c +3 more+21 -50 replies
5/13iommufd: Serialise persisted iommufds and ioasdrivers/iommu/iommufd/iommufd_private.h drivers/iommu/iommufd/main.c drivers/iommu/iommufd/serialise.c+81 -416 replies
6/13iommufd: Expose persistent iommufd IDs in sysfsdrivers/iommu/iommufd/serialise.c+47 -10 replies
7/13iommufd: Re-hydrate a usable iommufd ctx from sysfsdrivers/iommu/iommufd/io_pagetable.c drivers/iommu/iommufd/iommufd_private.h drivers/iommu/iommufd/main.c +1 more+60 -40 replies
8/13intel-iommu: Add serialise and deserialise boilerplatedrivers/iommu/intel/Makefile drivers/iommu/intel/iommu.c drivers/iommu/intel/iommu.h +1 more+77 -00 replies
9/13intel-iommu: Serialise dmar_domain on KHO activaetdrivers/iommu/intel/iommu.c drivers/iommu/intel/iommu.h drivers/iommu/intel/serialise.c+101 -100 replies
10/13intel-iommu: Re-hydrate persistent domains after kexecdrivers/iommu/intel/iommu.c drivers/iommu/intel/iommu.h drivers/iommu/intel/serialise.c+53 -10 replies
11/13iommu: Add callback to restore persisted iommu_domaindrivers/iommu/intel/iommu.c drivers/iommu/iommufd/serialise.c include/linux/iommu.h+25 -11 reply
12/13iommufd, guestmemfs: Ensure persistent file used for persistent DMAdrivers/iommu/iommufd/ioas.c fs/guestmemfs/file.c include/linux/guestmemfs.h+34 -01 reply
13/13iommufd, guestmemfs: Pin files when mapped for persistent DMAdrivers/iommu/iommufd/ioas.c drivers/iommu/iommufd/iommufd_private.h drivers/iommu/iommufd/serialise.c +4 more+50 -10 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 00/13] Support iommu(fd) persistence for live update

Live update is a mechanism to support updating a hypervisor in a way that has limited impact to running virtual machines. This is done by pausing/serialising running VMs, kexec-ing into a new kernel, starting new VMM processes and then deserialising/resuming the VMs so that they continue running from where they were. When the VMs have DMA devices assigned to them, the IOMMU state and page tables needs to be persisted so that DMA transactions can continue across kexec.

Currently there is no mechanism in Linux to be able to continue running DMA across kexec, and pick up handles to the original DMA mappings after kexec. We are looking for a path to be able to support this capability which is necessary for live update. In this RFC patch series a potential solution is sketched out, and we are looking for feedback on the approach and userspace interfaces.

This RFC is intended to serve as the discussion ground for a Linux Plumbers Conf 2024 session on iommu persistence:
https://lpc.events/event/18/contributions/1686/
... and a BoF session on memory persistence in general:
https://lpc.events/event/18/contributions/1970/
Please join those to further this discussion.

The concept is as follows:
IOMMUFDs are marked as persistent via a new option. When a struct iommu_domain is allocated by a iommufd, that iommu_domain also gets marked as persistend. Before kexec iommufd serialises the metadata for all persistent domains to the KHO device tree blob. Similarly the IOMMU platform driver marks all of the page table pages as persistent. After kexec the persistent IOMMUFDs are expose to userspace in sysfs. Fresh IOMMUFD objects are built up from the data which was passed across in KHO. Userspace can open these sysfs files to get a handle on the IOMMUFD again. Iommufd ensures that only persistent memory can be mapped into persistent address spaces.

This depends on KHO as the foundational framework for passing data across kexec and for marking pages as persistent:
https://lore.kernel.org/all/[email protected]/

It also depends on guestmemfs as the provider of persistent guest RAM to be mapped into the IOMMU:
https://lore.kernel.org/all/[email protected]/

The code is not a complete solution, it is more of a sketch to show the moving parts and proposed interfaces to gather feedback and have the discussion. Only a small portion of the IOMMUFD object and dmar_domains are serialised; it is necessary to figure out all the data which needs to be serialised to have a fully working deserialised object.

Sign-offs are omitted to make it clear that this is not for merging yet.

Adding maintainers from IOMMU drivers, iommufd, kexec and KVM (seeing as this is designed for live update of a hypervisor specifically), and others who have engaged with the topic of memory persistence previously.

James Gowans (13):
iommufd: Support marking and tracking persistent iommufds
iommufd: Add plumbing for KHO (de)serialise
iommu/intel: zap context table entries on kexec
iommu: Support marking domains as persistent on alloc
iommufd: Serialise persisted iommufds and ioas
iommufd: Expose persistent iommufd IDs in sysfs
iommufd: Re-hydrate a usable iommufd ctx from sysfs
intel-iommu: Add serialise and deserialise boilerplate
intel-iommu: Serialise dmar_domain on KHO activaet
intel-iommu: Re-hydrate persistent domains after kexec
iommu: Add callback to restore persisted iommu_domain iommufd, guestmemfs: Ensure persistent file used for persistent DMA iommufd, guestmemfs: Pin files when mapped for persistent DMA

 drivers/iommu/amd/iommu.c                   |   4 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c |   3 +-
 drivers/iommu/intel/Makefile                |   1 +
 drivers/iommu/intel/dmar.c                  |   1 +
 drivers/iommu/intel/iommu.c                 |  84 ++++++--
 drivers/iommu/intel/iommu.h                 |  31 +++
 drivers/iommu/intel/serialise.c             | 174 ++++++++++++++++
 drivers/iommu/iommufd/Makefile              |   1 +
 drivers/iommu/iommufd/hw_pagetable.c        |   5 +-
 drivers/iommu/iommufd/io_pagetable.c        |   2 +-
 drivers/iommu/iommufd/ioas.c                |  26 +++
 drivers/iommu/iommufd/iommufd_private.h     |  34 ++++
 drivers/iommu/iommufd/main.c                |  75 ++++++-
 drivers/iommu/iommufd/selftest.c            |   1 +
 drivers/iommu/iommufd/serialise.c           | 213 ++++++++++++++++++++
 fs/guestmemfs/file.c                        |  25 +++
 fs/guestmemfs/guestmemfs.h                  |   1 +
 fs/guestmemfs/inode.c                       |   4 +
 include/linux/guestmemfs.h                  |  15 ++
 include/linux/iommu.h                       |  16 +-
 include/uapi/linux/iommufd.h                |   5 +
 21 files changed, 698 insertions(+), 23 deletions(-)
 create mode 100644 drivers/iommu/intel/serialise.c
 create mode 100644 drivers/iommu/iommufd/serialise.c
Signature · 1 line
2.34.1