Skip to contentSkip to conversation

iommu: Add live update state preservation

PATCH Superseded Samiullah Khawaja 7 participants 82 messages Apr 27 – Jun 1

Subsystems (8)
KVM Kselftest IOMMU Live update +4 more subsystems

v2 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v2 cover iommu: Add live update state preservation

Hi,

This patch series introduces a mechanism for IOMMU state preservation across live update, including the Intel VT-d driver support implementation.

Please take a look at the following LWN article to learn about KHO and Live Update Orchestrator:

https://lwn.net/Articles/1033364/

This work depends on,

- VFIO CDEV preservation series (v3):
https://lore.kernel.org/kvm/[email protected]/
- PCI device preservation series (v4):
https://lore.kernel.org/all/[email protected]/
- LUO series to add FLB refcounting:
https://lore.kernel.org/lkml/[email protected]/

The kernel tree with all dependencies is uploaded to the following Github location:

https://github.com/samikhawaja/linux/tree/iommu/phase1-v2

Overall Goals:

The goal of this effort is to preserve the IOMMU domains, managed by iommufd, attached to devices preserved through VFIO cdev. This allows DMA mappings and IOMMU context of a device assigned to a VM to be maintained across a kexec live update.

This is achieved by preserving IOMMU page tables using Generic Page Table support, IOMMU root table and the relevant context entries across live update.

The functionality in the previously sent RFC is split into two phases and this series implements the Phase 1. Phase 1 implements the following functionality:

- Foundational work in IOMMU core and VT-d driver to preserve and restore IOMMU translation units, IOMMU domains and devices across liveupdate kexec.
- The preservation is triggered by preserving vfio cdev FD and bound iommufd FD into a live update session.
- An HWPT (and backing IOMMU domain) is only preserved if it contains only file type DMA mappings. Also the memfd being used for such mapping should be SEAL SEAL'd during mapping.
- During live update boot, the state of preserved Intel VT-d, IOMMU domain and devices is restored.
- The restored IOMMU domains are reattached to the preserved devices during early boot.
- The DMA ownership of restored devices is also claimed during live update boot. This means that any attempt to bind a non-vfio drivers with them or binding a new iommufd with them will fail.

Architectural Overview:

The target architecture for IOMMU state preservation across a live update involves coordination between the Live Update Orchestrator, iommufd, and the IOMMU drivers.

The core design uses the Live Update Orchestrator's file descriptor preservation mechanism to preserve iommufd file descriptors. The user marks the iommufd HWPTs for preservation using a new ioctl added in this series. Once done, the preservation of iommufd inside an LUO session is triggered using LUO ioctls. During preservation, the LUO preserve callback for an iommufd walks through the HWPTs it manages to identify the ones that need to be preserved. Once identified, a new IOMMU core API is used to preserve the iommu domain. The IOMMU core uses Generic Page Table to preserve the page tables of these domains. The domains are then marked as preserved.

When the user triggers the preservation of a VFIO cdev that is attached to an iommufd that is preserved, the device attachment state of that VFIO cdev is also preserved using an API exported by iommufd. IOMMUFD fetches all the information that needs to be preserved and calls the IOMMU core API to preserve the device state. The IOMMU core also preserves state of IOMMU that is associated with this device.

The IOMMU core has LUO FLB registered with the iommufd LUO file handler so the preserved iommu domain and iommu hardware unit state is available during boot for early restore in the next kernel.

During boot the driver fetches the preserved state from the IOMMU core and restores the state of preserved IOMMUs. Later when IOMMU core goes through the devices and probes them, the iommu domains of preserved devices are restored and the preserved devices are attached to them. During attachment, the DMA ownership of these devices is also claimed.

Tested:

The new iommufd_liveupdate_kexec_test selftest was used to verify the preservation logic. It was tested using QEMU with virtual IOMMU (VT-d) support with virtio pcie device bound to the vfio-pci driver.

Also Tested on an Intel machine with DSA device bound to vfio-pci driver.

Following steps were followed for verification,

- Bind the test device with vfio-pci driver
- Run test on the machine by running

./iommufd_liveupdate_kexec_test <vfio-cdev-path>

- Trigger Kexec.
- After reboot, try binding the device to a non-vfio pci driver,

echo <device bdf> > /sys/class/bus/drivers/pci-pf-stub/bind

- This should fail with "Device or resource busy".
- Bind the device with vfio-pci driver and run the test again.

  ./iommufd_liveupdate_kexec_test <vfio-cdev-path> --stage 2

- Test verifies that the device cannot be bound with a new iommufd and the session cannot be finished.

Future Work:

- Phase 2 with IOMMUFD restore to reclaim the preserved vfio cdev and restore the preserved HWPTs.
- Full support for PASID preservation.
- Nested IOMMU preservation.
- Extend support to other IOMMU architectures (e.g., AMD-Vi, Arm SMMUv3).
- DMA Alloc preservation support buddy allocator only.
https://github.com/samikhawaja/linux/tree/dma-alloc-preserve

Roadmap:

The doc below gives a break down of the overall work into Patch series needed to complete Liveupdate feature from IOMMU perspective:
https://docs.google.com/document/d/1enDn-uPE9U77U-xHEnzn6HHGKiePSAtMIP8EDU3NO0M

High-Level Sequence Flow:

The following diagrams illustrate the high-level interactions during the preservation phase. Note that function names in the diagram are kept abbreviated to save horizontal space.

Preserve:

Before live update the PREPARE event of Liveupdate Orchestrator invokes callbacks of the registered file and subsystem handlers.

 Userspace (VMM) | LUO Core |    iommufd    |  IOMMU Core   | IOMMU Driver
-----------------|----------|---------------|---------------|-------------
                 |          |               |               |
MARK_HWPT        |          |               |               |
--------------------------->                |               |
                 |          | Mark HWPT for |               |
                 |          | preservation  |               |
                 |          |               |               |
PRESERVE         |          |               |               |
 iommufd_fd      |          |               |               |
----------------->          |               |               |
                 | preserve |               |               |
                 |---------->               |               |
                 |          | For each HWPT |               |
                 |          |-------------->                |
                 |          |               | domain_presrv |
                 |          |               |-------------->
                 |          |               |               | gpt(preserve)
                 |          |               |<--------------|
                 |          |<--------------|               |
                 |<---------|               |               |
                 |          |               |               |
...              |          |               |               |
                 |          |               |               |
PRESERVE,        |          |               |               |
 vfio_cdev_fd    |          |               |               |
----------------->          |               |               |
                 | preserve |               |               |
                 |---------->               |               |
                 |          |               |               |
                 |          | iommu_preserv |               |
                 |          | _device()     |               |
                 |          |-------------->                |
                 |          |               | preserve      |
                 |          |               | (iommu_hw)    |
                 |          |               |-------------->
                 |          |               |               | preserve(root)
                 |          |               |               | preserve(pasid)
                 |          |               |<--------------|
                 |          |               |               |
                 |          |               | preserve      |
                 |          |               | _device(dev)  |
                 |          |               |-------------->
                 |          |               |               |
                 |          |               |<--------------|
                 |          |<--------------|               |
                 |<---------|               |               |

Restore:

After a live update, the preserved state is restored during boot.

 Userspace (VMM) | LUO Core |    iommufd    |  IOMMU Core   | IOMMU Driver
-----------------|----------|---------------|---------------|-------------
                 |          |               |               |
                 |          |               |               | Restore
                 |          |               |               | Root, DIDs
                 |          |               |               |
                 |          |               |               | Register
                 |          |               | probe devices |
                 |          |               |               |
                 |          |               | restore       |
                 |          |               | domain        |
                 |          |               |-------------->
                 |          |               |               | restore
                 |          |               | reattach      |
                 |          |               | domain        |
                 |          |               |-------------->
                 |          |               |               |

Looking forward to your feedback on this.

Changelog:

v2:
- Move IOMMU_LIVEUPDATE under IOMMU_SUPPORT dependencies.
- Update copyright year to 2026.
- Add Kernel-doc for FLB struct.
- Add an ASCII diagram for FLB memory layout.
- Change compatibility to iommu-liveupdate-v1.
- Rename structs to more reasonable names.
- Add comment explaining the rationale for BUG_ON during restoration.
- Rename 'did' to 'attachment_id'.
- Use phys_to_virt() and virt_to_phys() consistently.
- Create separate functions for FLB unpreserve and folio_put.
- Return the virtual address from the restore_array function.
- Free serialized state on finish without checking for NULL.
- Rename 'iommu-lu.h' to 'iommu-liveupdate.h'.
- Suffix max with _per_page (iommu_max_objs_per_page).
- Move unused helpers to the patches that actually need them.
- Rename reserve_obj_ser to alloc_object_ser.
- Only allow vfio drivers to use preserve_device APIs.
- Move ops declarations under CONFIG_IOMMU_LIVEUPDATE guards.
- Add lockdep_assert_held() validations in locked functions.
- Rename device_ser_match to match_device_ser.
- Create iommu_folio_update_stats(folio, nr_pages) helper.
- Explicitly set incoherent flag to false for restored pages.
- Do not use unpreserve_pages() function for error handling.
- Preserve vasz and sign_extend of a domain.
- Make the restored domain free-only.
- Ignore error of pt_descend() during restore as it should never fail (dead code).
- On domain restore, clear all features except the preserved.
- Add a KUnit test to verify that a restored domain can be freed with zeroed features.
- Bypass paging_domain_compatible() checks for restored domains.
- Use an explicit if-check for preserved state to clear the context entries.
- Rename clean_context to clear_unpreserved_context_entries.
- Use "DMAR:" in pr_fmt instead of "iommu:".
- Pass ROOT_ENTRY_NR directly into unpreserve functions.
- Remove IOMMU lock during preserve as not needed.
- Iterate all devices when clearing context entries instead of just PCIe devices.
- Do not populate empty unpreserve_device callbacks if not needed.
- Rename unpreserve_iommu_context to unpreserve_iommu_context_table.
- Check the associated IOMMU in _restore_used_domain_id.
- Add a comment that MMIO base is used as token for identifying IOMMU.
- Add a comment in _restore_used_domain_ids that IDA allocation can safely fail.
- Mark IOMMU state restored after restoring to prevent double restoration and UAF.
- Fix domain leak in iommu_restore_domain error path.
- Check DID start range before allocation.
- Add lockdep_assert_held in group domain restoration path.
- Add WARN_ON to catch unexpected group ownership during device probe.
- Add comment explaining group ownership setup and reclaim during device probe.
- Cleanup the full PASID directory instead of only 4K.
- Rename ioctl to IOMMU_HWPT_LIVEUPDATE_MARK_PRESERVE.
- Use XArray to save the HWPT liveupdate marks.
- Add the ioctl struct to ucmd_buffer.
- Make the ioctl 64-bit aligned (struct padding).
- Add UAPI documentation stating that HWPTs cannot be unmarked.
- Add UAPI documentation stating that only file-based mappings are allowed.
- Rename structs to use "liveupdate" instead of "lu".
- Rework HWPT preservation to prevent TOCTOU and mutex lock inside XA spin lock.
- Check preserved pages at the IOPT level, not the HWPT level.
- Return -EOPNOTSUPP if there are PASID attachments.
- Do not allow detach (or attach) once the device is preserved.
- Add MODULE_IMPORT_NS("IOMMUFD") in vfio_pci_liveupdate.c.
- Make token u64.
- Modify selftest to use live update kexec_test pattern.
- Use helper macros in test instead of repeating ksft_assert
- Rename setup_cdev to open_cdev.
- Define constants (tokens) as u64 to avoid ABI warnings.

v1: https://lore.kernel.org/all/[email protected]/

rfcv2: https://lore.kernel.org/all/[email protected]/

rfcv1: https://lore.kernel.org/all/[email protected]/

Pasha Tatashin (1):
liveupdate: luo_file: Add internal APIs for file preservation

Samiullah Khawaja (13):
iommu: Implement IOMMU Live update FLB callbacks
iommu: Implement IOMMU domain preservation
iommu: Implement device and IOMMU HW preservation iommu/pages: Add APIs to preserve/unpreserve/restore iommu pages
iommupt: Implement preserve/unpreserve/restore callbacks iommu/vt-d: Implement device and iommu preserve/unpreserve ops
iommu: Add APIs to get iommu and device preserved state iommu/vt-d: Restore IOMMU state and reclaimed domain ids
iommu: Restore and reattach preserved domains to devices iommu/vt-d: preserve PASID table of preserved device
iommufd: Add APIs to preserve/unpreserve a vfio cdev vfio/pci: Preserve the iommufd state of the vfio cdev iommufd/selftest: Add test to verify iommufd preservation

YiFei Zhu (2):
iommufd: Implement ioctl to mark HWPT for preservation
iommufd: Persist iommu hardware pagetables for live update

 MAINTAINERS                                   |  12 +
 drivers/iommu/Kconfig                         |  12 +
 drivers/iommu/Makefile                        |   1 +
 drivers/iommu/generic_pt/iommu_pt.h           | 131 ++++
 drivers/iommu/generic_pt/kunit_iommu_pt.h     |  28 +
 drivers/iommu/intel/Makefile                  |   1 +
 drivers/iommu/intel/iommu.c                   | 159 ++++-
 drivers/iommu/intel/iommu.h                   |  50 +-
 drivers/iommu/intel/liveupdate.c              | 337 ++++++++++
 drivers/iommu/intel/nested.c                  |   2 +-
 drivers/iommu/intel/pasid.c                   |   7 +-
 drivers/iommu/intel/pasid.h                   |   9 +
 drivers/iommu/iommu-pages.c                   | 108 +++-
 drivers/iommu/iommu-pages.h                   |  30 +
 drivers/iommu/iommu.c                         |  79 ++-
 drivers/iommu/iommufd/Makefile                |   1 +
 drivers/iommu/iommufd/device.c                | 102 +++
 drivers/iommu/iommufd/io_pagetable.c          |  11 +
 drivers/iommu/iommufd/io_pagetable.h          |   1 +
 drivers/iommu/iommufd/iommufd_private.h       |  46 ++
 drivers/iommu/iommufd/liveupdate.c            | 339 ++++++++++
 drivers/iommu/iommufd/main.c                  |  19 +-
 drivers/iommu/iommufd/pages.c                 |   7 +
 drivers/iommu/liveupdate.c                    | 592 ++++++++++++++++++
 drivers/vfio/device_cdev.c                    |  10 +
 drivers/vfio/pci/vfio_pci_liveupdate.c        |  33 +-
 include/linux/generic_pt/iommu.h              |  19 +-
 include/linux/iommu-liveupdate.h              | 156 +++++
 include/linux/iommu.h                         |  47 ++
 include/linux/iommufd.h                       |  29 +
 include/linux/kho/abi/iommu.h                 | 249 ++++++++
 include/linux/kho/abi/iommufd.h               |  51 ++
 include/linux/liveupdate.h                    |  21 +
 include/uapi/linux/iommufd.h                  |  26 +
 kernel/liveupdate/luo_file.c                  |  69 ++
 kernel/liveupdate/luo_internal.h              |  17 +
 tools/testing/selftests/iommu/Makefile        |  12 +
 .../iommu/iommufd_liveupdate_kexec_test.c     | 239 +++++++
 38 files changed, 3016 insertions(+), 46 deletions(-)
 create mode 100644 drivers/iommu/intel/liveupdate.c
 create mode 100644 drivers/iommu/iommufd/liveupdate.c
 create mode 100644 drivers/iommu/liveupdate.c
 create mode 100644 include/linux/iommu-liveupdate.h
 create mode 100644 include/linux/kho/abi/iommu.h
 create mode 100644 include/linux/kho/abi/iommufd.h
 create mode 100644 tools/testing/selftests/iommu/iommufd_liveupdate_kexec_test.c

base-commit: 2a4c0c11c0193889446cdb6f1540cc2b9aff97dd

Signature · 1 line
2.54.0.545.g6539524ca2-goog
#1 lore raw

PATCH v2 1/16 liveupdate: luo_file: Add internal APIs for file preservation

From: Pasha Tatashin <[email protected]>

The core liveupdate mechanism allows userspace to preserve file descriptors. However, kernel subsystems often manage struct file objects directly and need to participate in the preservation process programmatically without relying solely on userspace interaction.

---
 include/linux/liveupdate.h       | 21 ++++++++++
 kernel/liveupdate/luo_file.c     | 69 ++++++++++++++++++++++++++++++++
 kernel/liveupdate/luo_internal.h | 17 ++++++++
 3 files changed, 107 insertions(+)
include/linux/liveupdate.h +21 -0
index 88722e5caf02..261f61998fce 100644
@@ -25,6 +25,7 @@ struct file;
 /**
  * struct liveupdate_file_op_args - Arguments for file operation callbacks.
  * @handler:          The file handler being called.
+ * @session:          The session this file belongs to.
  * @retrieve_status:  The retrieve status for the 'can_finish / finish'
  *                    operation. A value of 0 means the retrieve has not been
  *                    attempted, a positive value means the retrieve was
@@ -45,6 +46,7 @@ struct file;
  */
 struct liveupdate_file_op_args {
 	struct liveupdate_file_handler *handler;
+	struct liveupdate_session *session;
 	int retrieve_status;
 	struct file *file;
 	u64 serialized_data;
@@ -243,6 +245,13 @@ int liveupdate_flb_get_incoming(struct liveupdate_flb *flb, void **objp);
 void liveupdate_flb_put_incoming(struct liveupdate_flb *flb);
 
 int liveupdate_flb_get_outgoing(struct liveupdate_flb *flb, void **objp);
+/* kernel can internally retrieve files */
+int liveupdate_get_file_incoming(struct liveupdate_session *s, u64 token,
+				 struct file **filep);
+
+/* Get a token for an outgoing file, or -ENOENT if file is not preserved */
+int liveupdate_get_token_outgoing(struct liveupdate_session *s,
+				  struct file *file, u64 *tokenp);
 
 #else /* CONFIG_LIVEUPDATE */
 
@@ -292,5 +301,17 @@ static inline int liveupdate_flb_get_outgoing(struct liveupdate_flb *flb,
 	return -EOPNOTSUPP;
 }
 
+static inline int liveupdate_get_file_incoming(struct liveupdate_session *s,
+					       u64 token, struct file **filep)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline int liveupdate_get_token_outgoing(struct liveupdate_session *s,
+						struct file *file, u64 *tokenp)
+{
+	return -EOPNOTSUPP;
+}
+
 #endif /* CONFIG_LIVEUPDATE */
 #endif /* _LINUX_LIVEUPDATE_H */
kernel/liveupdate/luo_file.c +69 -0

114 lines not on this page · raw message

kernel/liveupdate/luo_internal.h +17 -0
index 875844d7a41d..08b198802e7f 100644
@@ -79,6 +79,23 @@ struct luo_session {
 
 extern struct rw_semaphore luo_register_rwlock;
 
+static inline struct liveupdate_session *luo_session_from_file_set(struct luo_file_set *file_set)
+{
+	struct luo_session *session;
+
+	session = container_of(file_set, struct luo_session, file_set);
+
+	return (struct liveupdate_session *)session;
+}
+
+static inline struct luo_file_set *luo_file_set_from_session_locked(struct liveupdate_session *s)
+{
+	struct luo_session *session = (struct luo_session *)s;
+
+	lockdep_assert_held(&session->mutex);
+	return &session->file_set;
+}
+
 int luo_session_create(const char *name, struct file **filep);
 int luo_session_retrieve(const char *name, struct file **filep);
 int __init luo_session_setup_outgoing(void *fdt);
Signature · 1 line
2.54.0.545.g6539524ca2-goog
#2 lore raw

Re: [PATCH v2 01/16] liveupdate: luo_file: Add internal APIs for file preservation

Quoted 9 lines · Samiullah Khawaja

From: Pasha Tatashin <[email protected]>

The core liveupdate mechanism allows userspace to preserve file descriptors. However, kernel subsystems often manage struct file objects directly and need to participate in the preservation process programmatically without relying solely on userspace interaction.

[..]

Quoted 45 lines
kernel/liveupdate/luo_file.c +41 -0
@@ -924,3 +931,65 @@ void liveupdate_unregister_file_handler(struct liveupdate_file_handler *fh)
 	luo_flb_unregister_all(fh);
 	list_del(&ACCESS_PRIVATE(fh, list));
 }
+EXPORT_SYMBOL_GPL(liveupdate_unregister_file_handler);
+
+/**
+ * liveupdate_get_token_outgoing - Get the token for a preserved file.
+ * @s:      The outgoing liveupdate session.
+ * @file:   The file object to search for.
+ * @tokenp: Output parameter for the found token.
+ *
+ * Searches the list of preserved files in an outgoing session for a matching
+ * file object. If found, the corresponding user-provided token is returned.
+ *
+ * This function is intended for in-kernel callers that need to correlate a
+ * file with its liveupdate token.
+ *
+ * Context: It must be called with session mutex acquired.
+ * Return: 0 on success, -ENOENT if the file is not preserved in this session.
+ */
+int liveupdate_get_token_outgoing(struct liveupdate_session *s,
+				  struct file *file, u64 *tokenp)
+{
+	struct luo_file_set *file_set = luo_file_set_from_session_locked(s);
+	struct luo_file *luo_file;
+	int err = -ENOENT;
+
+	list_for_each_entry(luo_file, &file_set->files_list, list) {
+		if (luo_file->file == file) {
+			if (tokenp)
+				*tokenp = luo_file->token;
+			err = 0;
+			break;
+		}
+	}
+
+	return err;
+}
+
+/**
+ * liveupdate_get_file_incoming - Retrieves a preserved file for in-kernel use.
+ * @s:      The incoming liveupdate session (restored from the previous kernel).
+ * @token:  The unique token identifying the file to retrieve.
+ * @filep:  On success, this will be populated with a pointer to the retrieved

kernel/liveupdate/luo_file.c void liveupdate_unregister_file_handler(struct liveupdate_file_handler *fh) · line 984

Quoted 10 lines
kernel/liveupdate/luo_file.c +10 -0
+ *          'struct file'.
+ *
+ * Provides a kernel-internal API for other subsystems to retrieve their
+ * preserved files after a live update. This function is a simple wrapper
+ * around luo_retrieve_file(), allowing callers to find a file by its token.
+ *
+ * The caller receives a new reference to the file and must call fput() when it
+ * is no longer needed. The file's lifetime is managed by LUO and any userspace
+ * file descriptors. If the caller needs to hold a reference to the file beyond
+ * the immediate scope, it must call get_file() itself.

Thanks for re-wording this and I'm sorry for being a stickler here, I'm a bit concerned that the last part here might lead to reference leaks in downstream drivers.

Looking at the underlying luo_retrieve_file() implementation [1], it explicitly calls get_file() before returning the pointer (both on the initial retrieve and on cached ones). This means the caller inherently receives a reference that they own & the caller is responsible for exactly one fput().

However, that last part of the comment can be misunderstood as the caller doesn't hold a lasting reference unless they call get_file() themselves. This makes the reader assume that LUO is going to automatically reap that initial reference from them.

If a driver author assumes LUO is going to reap it, they will follow that last sentence and call get_file() to stash the pointer safely. They might end up holding two references (thinking one of them will be reaped), and could ultimately leak the struct file when they only call fput() once during teardown.

Should we just drop that last sentence to make the lifecycle contract unambiguous? (i.e., The caller gets a newly bumped reference, and they are responsible for exactly one fput() per call).

kernel/liveupdate/luo_file.c void liveupdate_unregister_file_handler(struct liveupdate_file_handler *fh) · line 995

Quoted 11 lines
kernel/liveupdate/luo_file.c +11 -0
+ *
+ * Context: It must be called with session mutex acquired of a restored session.
+ * Return: 0 on success. Returns -ENOENT if no file with the matching token is
+ *         found, or any other negative errno on failure.
+ */
+int liveupdate_get_file_incoming(struct liveupdate_session *s, u64 token,
+				 struct file **filep)
+{
+	return luo_retrieve_file(luo_file_set_from_session_locked(s),
+				 token, filep);
+}

Nit: Shouldn't we export both of these functions via EXPORT_SYMBOL_GPL? Since, these new APIs are intended for kernel subsystems to participate programmatically, there could be IOMMU drivers (or others) that can be compiled as loadable modules. Thus we should export these APIs via EXPORT_SYMBOL_GPL(). If they aren't exported, any loadable module attempting to use them will compile successfully (due to the header), but will fail to load at runtime with an Unknown symbol error.

IIUC, if a function isn't exported with EXPORT_SYMBOL, it remains hidden inside vmlinux, (i.e. it isn't in the kernel's global symbol table used during modprobe).

Thanks,
Praan

[1] https://elixir.bootlin.com/linux/v7.0-rc3/source/kernel/liveupdate/luo_file.c#L560

#3 lore raw

Re: [PATCH v2 01/16] liveupdate: luo_file: Add internal APIs for file preservation

Quoted 19 lines · Pranjal Shrivastava
Quoted 9 lines · Samiullah Khawaja

From: Pasha Tatashin <[email protected]>

The core liveupdate mechanism allows userspace to preserve file descriptors. However, kernel subsystems often manage struct file objects directly and need to participate in the preservation process programmatically without relying solely on userspace interaction.

[..]

Quoted 6 lines
kernel/liveupdate/luo_file.c +2 -0
@@ -924,3 +931,65 @@ void liveupdate_unregister_file_handler(struct liveupdate_file_handler *fh)
 	luo_flb_unregister_all(fh);
 	list_del(&ACCESS_PRIVATE(fh, list));
 }
+EXPORT_SYMBOL_GPL(liveupdate_unregister_file_handler);
+

[snip]

Quoted 38 lines
Quoted 6 lines
kernel/liveupdate/luo_file.c +6 -0
+
+/**
+ * liveupdate_get_file_incoming - Retrieves a preserved file for in-kernel use.
+ * @s:      The incoming liveupdate session (restored from the previous kernel).
+ * @token:  The unique token identifying the file to retrieve.
+ * @filep:  On success, this will be populated with a pointer to the retrieved

kernel/liveupdate/luo_file.c void liveupdate_unregister_file_handler(struct liveupdate_file_handler *fh) · line 984

Quoted 10 lines
kernel/liveupdate/luo_file.c +10 -0
+ *          'struct file'.
+ *
+ * Provides a kernel-internal API for other subsystems to retrieve their
+ * preserved files after a live update. This function is a simple wrapper
+ * around luo_retrieve_file(), allowing callers to find a file by its token.
+ *
+ * The caller receives a new reference to the file and must call fput() when it
+ * is no longer needed. The file's lifetime is managed by LUO and any userspace
+ * file descriptors. If the caller needs to hold a reference to the file beyond
+ * the immediate scope, it must call get_file() itself.

Thanks for re-wording this and I'm sorry for being a stickler here, I'm a bit concerned that the last part here might lead to reference leaks in downstream drivers.

Looking at the underlying luo_retrieve_file() implementation [1], it explicitly calls get_file() before returning the pointer (both on the initial retrieve and on cached ones). This means the caller inherently receives a reference that they own & the caller is responsible for exactly one fput().

However, that last part of the comment can be misunderstood as the caller doesn't hold a lasting reference unless they call get_file() themselves. This makes the reader assume that LUO is going to automatically reap that initial reference from them.

If a driver author assumes LUO is going to reap it, they will follow that last sentence and call get_file() to stash the pointer safely. They might end up holding two references (thinking one of them will be reaped), and could ultimately leak the struct file when they only call fput() once during teardown.

Quoted 3 lines

Should we just drop that last sentence to make the lifecycle contract unambiguous? (i.e., The caller gets a newly bumped reference, and they are responsible for exactly one fput() per call).

I think these are all valid points. I will remove the last sentences for clarity.

Quoted 21 lines

kernel/liveupdate/luo_file.c void liveupdate_unregister_file_handler(struct liveupdate_file_handler *fh) · line 995

Quoted 12 lines

+ *
+ * Context: It must be called with session mutex acquired of a restored session.
+ * Return: 0 on success. Returns -ENOENT if no file with the matching token is
+ * found, or any other negative errno on failure.
+ */

kernel/liveupdate/luo_file.c +6 -0
+int liveupdate_get_file_incoming(struct liveupdate_session *s, u64 token,
+				 struct file **filep)
+{
+	return luo_retrieve_file(luo_file_set_from_session_locked(s),
+				 token, filep);
+}

Nit: Shouldn't we export both of these functions via EXPORT_SYMBOL_GPL? Since, these new APIs are intended for kernel subsystems to participate programmatically, there could be IOMMU drivers (or others) that can be compiled as loadable modules. Thus we should export these APIs via EXPORT_SYMBOL_GPL(). If they aren't exported, any loadable module attempting to use them will compile successfully (due to the header), but will fail to load at runtime with an Unknown symbol error.

Quoted 3 lines

IIUC, if a function isn't exported with EXPORT_SYMBOL, it remains hidden inside vmlinux, (i.e. it isn't in the kernel's global symbol table used during modprobe).

Agreed. Will Update this.

I think there are other LUO APIs that are not exported and can be used by modules. Let me check those also.

Quoted 6 lines

Thanks,
Praan

[1] https://elixir.bootlin.com/linux/v7.0-rc3/source/kernel/liveupdate/luo_file.c#L560

#4 lore raw

Re: [PATCH v2 01/16] liveupdate: luo_file: Add internal APIs for file preservation

Quoted 12 lines · Samiullah Khawaja
Quoted 11 lines · Pranjal Shrivastava
Quoted 9 lines · Samiullah Khawaja

From: Pasha Tatashin <[email protected]>

The core liveupdate mechanism allows userspace to preserve file descriptors. However, kernel subsystems often manage struct file objects directly and need to participate in the preservation process programmatically without relying solely on userspace interaction.

[..]

Quoted 20 lines
Quoted 20 lines

kernel/liveupdate/luo_file.c void liveupdate_unregister_file_handler(struct liveupdate_file_handler *fh) · line 995

Quoted 11 lines

+ *
+ * Context: It must be called with session mutex acquired of a restored session.
+ * Return: 0 on success. Returns -ENOENT if no file with the matching token is
+ * found, or any other negative errno on failure.
+ */

kernel/liveupdate/luo_file.c +6 -0
+int liveupdate_get_file_incoming(struct liveupdate_session *s, u64 token,
+				 struct file **filep)
+{
+	return luo_retrieve_file(luo_file_set_from_session_locked(s),
+				 token, filep);
+}

Nit: Shouldn't we export both of these functions via EXPORT_SYMBOL_GPL? Since, these new APIs are intended for kernel subsystems to participate programmatically, there could be IOMMU drivers (or others) that can be compiled as loadable modules. Thus we should export these APIs via EXPORT_SYMBOL_GPL(). If they aren't exported, any loadable module attempting to use them will compile successfully (due to the header), but will fail to load at runtime with an Unknown symbol error.

Quoted 8 lines
Quoted 3 lines

IIUC, if a function isn't exported with EXPORT_SYMBOL, it remains hidden inside vmlinux, (i.e. it isn't in the kernel's global symbol table used during modprobe).

Agreed. Will Update this.

I think there are other LUO APIs that are not exported and can be used by modules. Let me check those also.

Hi,
+1 to Pranjal's EXPORT_SYMBOL_GPL nit, and I'd like to extend it. With CONFIG_IOMMUFD=m && CONFIG_IOMMU_LIVEUPDATE=y
the series fails at modpost, not at modprobe — kernel won't even build:

ERROR: modpost: "memfd_get_seals" [iommufd.ko] undefined!
ERROR: modpost: "iommu_preserve_device" [iommufd.ko] undefined!
ERROR: modpost: "iommu_unpreserve_device" [iommufd.ko] undefined!
ERROR: modpost: "liveupdate_register_file_handler" [iommufd.ko] undefined!
ERROR: modpost: "liveupdate_get_token_outgoing" [iommufd.ko] undefined!
ERROR: modpost: "iommufd_liveupdate_register" [iommufd.ko] undefined!
ERROR: modpost: "iommufd_liveupdate_unregister" [iommufd.ko] undefined!
ERROR: modpost: "iommufd_hwpt_liveupdate_mark_preserve" [iommufd.ko] undefined!

Two issues:
1) drivers/iommu/iommufd/Makefile (patch 12) builds liveupdate.o into vmlinux instead of iommufd.ko — needs to be a composite member, not a top-level obj:

     -obj-$(CONFIG_IOMMU_LIVEUPDATE) += liveupdate.o
     +iommufd-$(CONFIG_IOMMU_LIVEUPDATE) += liveupdate.o

This fixes the iommufd_liveupdate_{register,unregister} and iommufd_hwpt_liveupdate_mark_preserve errors above.

2) Beyond the two LUO functions Pranjal mentioned, four more vmlinux symbols need EXPORT_SYMBOL_GPL.

Thanks,
Ankit

Quoted 6 lines
Quoted 6 lines

Thanks,
Praan

[1] https://elixir.bootlin.com/linux/v7.0-rc3/source/kernel/liveupdate/luo_file.c#L560

#5 lore raw

Re: [PATCH v2 01/16] liveupdate: luo_file: Add internal APIs for file preservation

Quoted 59 lines · Ankit Soni
Quoted 12 lines · Samiullah Khawaja
Quoted 11 lines · Pranjal Shrivastava
Quoted 9 lines · Samiullah Khawaja

From: Pasha Tatashin <[email protected]>

The core liveupdate mechanism allows userspace to preserve file descriptors. However, kernel subsystems often manage struct file objects directly and need to participate in the preservation process programmatically without relying solely on userspace interaction.

[..]

Quoted 20 lines
Quoted 20 lines

kernel/liveupdate/luo_file.c void liveupdate_unregister_file_handler(struct liveupdate_file_handler *fh) · line 995

Quoted 11 lines

+ *
+ * Context: It must be called with session mutex acquired of a restored session.
+ * Return: 0 on success. Returns -ENOENT if no file with the matching token is
+ * found, or any other negative errno on failure.
+ */

kernel/liveupdate/luo_file.c +6 -0
+int liveupdate_get_file_incoming(struct liveupdate_session *s, u64 token,
+				 struct file **filep)
+{
+	return luo_retrieve_file(luo_file_set_from_session_locked(s),
+				 token, filep);
+}

Nit: Shouldn't we export both of these functions via EXPORT_SYMBOL_GPL? Since, these new APIs are intended for kernel subsystems to participate programmatically, there could be IOMMU drivers (or others) that can be compiled as loadable modules. Thus we should export these APIs via EXPORT_SYMBOL_GPL(). If they aren't exported, any loadable module attempting to use them will compile successfully (due to the header), but will fail to load at runtime with an Unknown symbol error.

Quoted 8 lines
Quoted 3 lines

IIUC, if a function isn't exported with EXPORT_SYMBOL, it remains hidden inside vmlinux, (i.e. it isn't in the kernel's global symbol table used during modprobe).

Agreed. Will Update this.

I think there are other LUO APIs that are not exported and can be used by modules. Let me check those also.

Hi,
+1 to Pranjal's EXPORT_SYMBOL_GPL nit, and I'd like to extend it. With CONFIG_IOMMUFD=m && CONFIG_IOMMU_LIVEUPDATE=y
the series fails at modpost, not at modprobe — kernel won't even build:

ERROR: modpost: "memfd_get_seals" [iommufd.ko] undefined!
ERROR: modpost: "iommu_preserve_device" [iommufd.ko] undefined!
ERROR: modpost: "iommu_unpreserve_device" [iommufd.ko] undefined!
ERROR: modpost: "liveupdate_register_file_handler" [iommufd.ko] undefined!
ERROR: modpost: "liveupdate_get_token_outgoing" [iommufd.ko] undefined!
ERROR: modpost: "iommufd_liveupdate_register" [iommufd.ko] undefined!
ERROR: modpost: "iommufd_liveupdate_unregister" [iommufd.ko] undefined!
ERROR: modpost: "iommufd_hwpt_liveupdate_mark_preserve" [iommufd.ko] undefined!

Quoted 5 lines

Two issues:
1) drivers/iommu/iommufd/Makefile (patch 12) builds liveupdate.o into vmlinux instead of iommufd.ko — needs to be a composite member, not a top-level obj:

    -obj-$(CONFIG_IOMMU_LIVEUPDATE) += liveupdate.o
    +iommufd-$(CONFIG_IOMMU_LIVEUPDATE) += liveupdate.o

Thanks for this, I will update and test this.

Quoted 7 lines

This fixes the iommufd_liveupdate_{register,unregister} and iommufd_hwpt_liveupdate_mark_preserve errors above.

2) Beyond the two LUO functions Pranjal mentioned, four more vmlinux symbols need EXPORT_SYMBOL_GPL.

Yes, we will have a commit for those in the VFIO series as it also uses those LUO symbols, and this series has a dependency on VFIO series. Please see the following commit from David:

https://lore.kernel.org/kvm/[email protected]/

Quoted 10 lines

Thanks,
Ankit

Quoted 6 lines
Quoted 6 lines

Thanks,
Praan

[1] https://elixir.bootlin.com/linux/v7.0-rc3/source/kernel/liveupdate/luo_file.c#L560

#6 lore raw

PATCH v2 2/16 iommu: Implement IOMMU Live update FLB callbacks

In reply to Samiullah Khawaja

Add liveupdate FLB for IOMMU state preservation. Use KHO preserve memory alloc/free helper functions to allocate memory for the IOMMU Live update FLB object and the serialization structs for device, domain and iommu.

During retrieve, walk through the preserved obj array headers and restore each folio. Also recreate the FLB obj.

---
 MAINTAINERS                      |   9 ++
 drivers/iommu/Kconfig            |  12 ++
 drivers/iommu/Makefile           |   1 +
 drivers/iommu/liveupdate.c       | 198 ++++++++++++++++++++++++++++
 include/linux/iommu-liveupdate.h |  18 +++
 include/linux/kho/abi/iommu.h    | 218 +++++++++++++++++++++++++++++++
 6 files changed, 456 insertions(+)
 create mode 100644 drivers/iommu/liveupdate.c
 create mode 100644 include/linux/iommu-liveupdate.h
 create mode 100644 include/linux/kho/abi/iommu.h
MAINTAINERS +9 -0
index 737c5ed1ce38..980041955abc 100644
@@ -13490,6 +13490,15 @@ F:	include/linux/iova.h
 F:	include/linux/of_iommu.h
 F:	rust/kernel/iommu/
 
+IOMMU LIVEUPDATE
+M:	Samiullah Khawaja <[email protected]>
+R:	Pranjal Shrivastava <[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	drivers/iommu/liveupdate.c
+F:	include/linux/iommu-liveupdate.h
+F:	include/linux/kho/abi/iommu.h
+
 IOMMUFD
 M:	Jason Gunthorpe <[email protected]>
 M:	Kevin Tian <[email protected]>
drivers/iommu/Kconfig +12 -0
index f86262b11416..a2f416df3214 100644
@@ -403,6 +403,18 @@ config IOMMU_DEBUG_PAGEALLOC
 	  line to activate the runtime checks.
 
 	  If unsure, say N.
+
+config IOMMU_LIVEUPDATE
+	bool "IOMMU live update state preservation support"
+	depends on LIVEUPDATE && IOMMUFD
+	help
+	  Enable support for preserving IOMMU state across a kexec live update.
+
+	  This allows devices managed by iommufd to maintain their DMA mappings
+	  during kexec base kernel update.
+
+	  If unsure, say N.
+
 endif # IOMMU_SUPPORT
 
 source "drivers/iommu/generic_pt/Kconfig"
drivers/iommu/Makefile +1 -0
index 0275821f4ef9..c333f4a3ada3 100644
@@ -16,6 +16,7 @@ obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) += io-pgtable-arm.o
 obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE_KUNIT_TEST) += io-pgtable-arm-selftests.o
 obj-$(CONFIG_IOMMU_IO_PGTABLE_DART) += io-pgtable-dart.o
 obj-$(CONFIG_IOMMU_IOVA) += iova.o
+obj-$(CONFIG_IOMMU_LIVEUPDATE) += liveupdate.o
 obj-$(CONFIG_OF_IOMMU)	+= of_iommu.o
 obj-$(CONFIG_MSM_IOMMU) += msm_iommu.o
 obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o
drivers/iommu/liveupdate.c new file +198 -0

198 lines not on this page · raw message

include/linux/iommu-liveupdate.h new file +18 -0
new file mode 100644
index 000000000000..3d1c65ed76fa
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Copyright (C) 2026, Google LLC
+ * Author: Samiullah Khawaja <[email protected]>
+ */
+
+#ifndef _LINUX_IOMMU_LIVEUPDATE_H
+#define _LINUX_IOMMU_LIVEUPDATE_H
+
+#include <linux/iommu.h>
+#include <linux/liveupdate.h>
+#include <linux/kho/abi/iommu.h>
+
+int iommu_liveupdate_register_flb(struct liveupdate_file_handler *handler);
+void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler);
+
+#endif /* _LINUX_IOMMU_LIVEUPDATE_H */
include/linux/kho/abi/iommu.h new file +218 -0

218 lines not on this page · raw message

Signature · 1 line
2.54.0.545.g6539524ca2-goog
#7 lore raw

Re: [PATCH v2 02/16] iommu: Implement IOMMU Live update FLB callbacks

Quoted 33 lines · Samiullah Khawaja

Add liveupdate FLB for IOMMU state preservation. Use KHO preserve memory alloc/free helper functions to allocate memory for the IOMMU Live update FLB object and the serialization structs for device, domain and iommu.

During retrieve, walk through the preserved obj array headers and restore each folio. Also recreate the FLB obj.

drivers/iommu/liveupdate.c new file +24 -0
+static void *iommu_liveupdate_restore_array(u64 array_phys)
+{
+	struct iommu_array_hdr_ser *array_hdr;
+	void *vaddr = array_phys ? phys_to_virt(array_phys) : NULL;
+
+	while (array_phys) {
+		/*
+		 * Failure to restore preserved IOMMU state is considered fatal.
+		 *
+		 * This is because the IOMMU translations for preserved IOMMUs
+		 * were kept enabled in the previous kernel and the preserved
+		 * devices have their IOMMU domains still present. Not being
+		 * able to restore means that the memory mapped into preserved
+		 * domains might be already corrupted by the preserved devices.
+		 *
+		 * There is no way to confirm the integrity of the memory that
+		 * was mapped. BUG_ON is the safest option at this point.
+		 */
+		BUG_ON(!kho_restore_folio(array_phys));
+		array_hdr = phys_to_virt(array_phys);
+		array_phys = array_hdr->next_array_phys;
+	}
+
+	return vaddr;

drivers/iommu/liveupdate.c line 88

Quoted 10 lines
drivers/iommu/liveupdate.c new file +9 -0
+}
 
+static int iommu_liveupdate_flb_retrieve(struct liveupdate_flb_op_args *argp)
+{
+	struct iommu_flb_obj *obj;
+	struct iommu_flb_ser *ser;
+
+	obj = kzalloc_obj(*obj, GFP_KERNEL);
+	if (!obj)
+		return -ENOMEM;

Should this be considered fatal for the same reason iommu_liveupdate_restore_array() is considered fatal? If anything in iommu_liveupdate_flb_retrieve() fails then the risk of corruption as described in iommu_liveupdate_restore_array() is possible.

Quoted 10 lines
drivers/iommu/liveupdate.c new file +10 -0
+
+	/* Data must be present and valid from the previous kernel */
+	BUG_ON(!kho_restore_folio(argp->data));
+
+	mutex_init(&obj->lock);
+	ser = phys_to_virt(argp->data);
+	obj->ser = ser;
+
+	obj->curr_domain_array = iommu_liveupdate_restore_array(ser->iommu_domain_array_phys);
+	obj->curr_device_array = iommu_liveupdate_restore_array(ser->device_array_phys);

drivers/iommu/liveupdate.c line 180

Quoted 10 lines
drivers/iommu/liveupdate.c new file +10 -0
+	obj->curr_iommu_array = iommu_liveupdate_restore_array(ser->iommu_array_phys);
+	argp->obj = obj;
+	return 0;
+}
+
+static struct liveupdate_flb_ops iommu_flb_ops = {
+	.preserve = iommu_liveupdate_flb_preserve,
+	.unpreserve = iommu_liveupdate_flb_unpreserve,
+	.finish = iommu_liveupdate_flb_finish,
+	.retrieve = iommu_liveupdate_flb_retrieve,

nit: I think it's helpful to put these in the order they are expected to be called.

.preserve = iommu_liveupdate_flb_preserve,
.unpreserve = iommu_liveupdate_flb_unpreserve,
.retrieve = iommu_liveupdate_flb_retrieve,
.finish = iommu_liveupdate_flb_finish,

include/linux/kho/abi/iommu.h line 76

Quoted 7 lines
include/linux/kho/abi/iommu.h new file +3 -0
new file mode 100644
index 000000000000..37b967820f14
 
+enum iommu_type_ser {
+	IOMMU_INVALID,
+};

Please document this enum.

include/linux/kho/abi/iommu.h line 87

Quoted 11 lines
include/linux/kho/abi/iommu.h new file +11 -0
+
+/**
+ * struct iommu_hdr_ser - Common header for all serialized IOMMU objects
+ * @ref_count: Reference count for the object
+ * @deleted: Flag indicating if the object is deleted
+ * @incoming: Flag indicating if the object was preserved in previous kernel
+ */
+struct iommu_hdr_ser {
+	u32 ref_count;
+	u32 deleted:1;
+	u32 incoming:1;

Are C bitfields safe to use in Live Update ABI?

Quoted 10 lines
include/linux/kho/abi/iommu.h new file +9 -0
+} __packed;
 
+/**
+ * struct iommu_flb_obj - FLB object allocated in current kernel pointing to
+ *			  preserved state in FLB
+ * @lock: Mutex protecting the object
+ * @ser: Pointer to the serialized state in FLB
+ * @curr_iommu_array: Pointer to the current array of IOMMU instances
+ * @curr_domain_array: Pointer to the current array of domains
+ * @curr_device_array: Pointer to the current array of devices

include/linux/kho/abi/iommu.h line 216

Quoted 10 lines
include/linux/kho/abi/iommu.h new file +10 -0
+ */
+struct iommu_flb_obj {
+	/* @lock: Protects the serialized objects during concurrent preservation */
+	struct mutex lock;
+	struct iommu_flb_ser *ser;
+
+	struct iommu_hw_array_ser *curr_iommu_array;
+	struct iommu_domain_array_ser *curr_domain_array;
+	struct iommu_device_array_ser *curr_device_array;
+} __packed;

This struct is not ABI so it should not be __packed nor defined in this file. I haven't read the whole series yet but this definition can probably go in drivers/iommu/liveupdate.c.

#8 lore raw

Re: [PATCH v2 02/16] iommu: Implement IOMMU Live update FLB callbacks

On Fri, May 01, 2026 at 09:45:19PM +0000, David Matlack wrote:

[...]

Quoted 5 lines
Quoted 5 lines
include/linux/kho/abi/iommu.h new file +5 -0
+
+/**
+ * struct iommu_hdr_ser - Common header for all serialized IOMMU objects
+ * @ref_count: Reference count for the object
+ * @deleted: Flag indicating if the object is deleted
Quoted 9 lines

include/linux/kho/abi/iommu.h line 87

Quoted 6 lines

+ * @incoming: Flag indicating if the object was preserved in previous kernel
+ */

include/linux/kho/abi/iommu.h new file +4 -0
+struct iommu_hdr_ser {
+	u32 ref_count;
+	u32 deleted:1;
+	u32 incoming:1;

Are C bitfields safe to use in Live Update ABI?

AFAIU, they aren't. The C standard does not dictate how bitfields are packed into their underlying types (e.g., whether they are packed MSB to LSB or vice-versa), making them highly dependent on the compiler and arch endianness.

I agree that we cannot rely on bitfields. Let's convert this to something like u32 flags; field and define explicit bitmasks.

Quoted 16 lines
Quoted 16 lines
include/linux/kho/abi/iommu.h new file +15 -0
+} __packed;
 
+/**
+ * struct iommu_flb_obj - FLB object allocated in current kernel pointing to
+ *			  preserved state in FLB
+ * @lock: Mutex protecting the object
+ * @ser: Pointer to the serialized state in FLB
+ * @curr_iommu_array: Pointer to the current array of IOMMU instances
+ * @curr_domain_array: Pointer to the current array of domains
+ * @curr_device_array: Pointer to the current array of devices
+ */
+struct iommu_flb_obj {
+	/* @lock: Protects the serialized objects during concurrent preservation */
+	struct mutex lock;
+	struct iommu_flb_ser *ser;
+
Quoted 8 lines

include/linux/kho/abi/iommu.h line 216

Quoted 4 lines
include/linux/kho/abi/iommu.h new file +4 -0
+	struct iommu_hw_array_ser *curr_iommu_array;
+	struct iommu_domain_array_ser *curr_domain_array;
+	struct iommu_device_array_ser *curr_device_array;
+} __packed;

This struct is not ABI so it should not be __packed nor defined in this file. I haven't read the whole series yet but this definition can probably go in drivers/iommu/liveupdate.c.

+1.

Thanks,
Praan

maintainer
#9 lore raw

Re: [PATCH v2 02/16] iommu: Implement IOMMU Live update FLB callbacks

Quoted 19 lines · Pranjal Shrivastava

On Fri, May 01, 2026 at 09:45:19PM +0000, David Matlack wrote:

[...]

Quoted 5 lines
Quoted 5 lines
include/linux/kho/abi/iommu.h new file +5 -0
+
+/**
+ * struct iommu_hdr_ser - Common header for all serialized IOMMU objects
+ * @ref_count: Reference count for the object
+ * @deleted: Flag indicating if the object is deleted
Quoted 9 lines

include/linux/kho/abi/iommu.h line 87

Quoted 6 lines

+ * @incoming: Flag indicating if the object was preserved in previous kernel
+ */

include/linux/kho/abi/iommu.h new file +4 -0
+struct iommu_hdr_ser {
+	u32 ref_count;
+	u32 deleted:1;
+	u32 incoming:1;

Are C bitfields safe to use in Live Update ABI?

Quoted 4 lines

AFAIU, they aren't. The C standard does not dictate how bitfields are packed into their underlying types (e.g., whether they are packed MSB to LSB or vice-versa), making them highly dependent on the compiler and arch endianness.

I had the same question and Jason told me otherwise:
https://lore.kernel.org/linux-mm/[email protected]/

I took him at his word and didn't really look much deeper. I use them for memfd preservation as well. See struct memfd_luo_folio_ser in include/linux/kho/abi/memfd.h.

Quoted 5 lines

I agree that we cannot rely on bitfields. Let's convert this to something like u32 flags; field and define explicit bitmasks.

Quoted 1 line
Quoted 1 line

+} __packed;

Signature · 2 lines
Regards,
Pratyush Yadav
#10 lore raw

Re: [PATCH v2 02/16] iommu: Implement IOMMU Live update FLB callbacks

Quoted 26 lines · Pratyush Yadav
Quoted 19 lines · Pranjal Shrivastava

On Fri, May 01, 2026 at 09:45:19PM +0000, David Matlack wrote:

[...]

Quoted 5 lines
Quoted 5 lines
include/linux/kho/abi/iommu.h new file +5 -0
+
+/**
+ * struct iommu_hdr_ser - Common header for all serialized IOMMU objects
+ * @ref_count: Reference count for the object
+ * @deleted: Flag indicating if the object is deleted
Quoted 9 lines

include/linux/kho/abi/iommu.h line 87

Quoted 6 lines

+ * @incoming: Flag indicating if the object was preserved in previous kernel
+ */

include/linux/kho/abi/iommu.h new file +4 -0
+struct iommu_hdr_ser {
+	u32 ref_count;
+	u32 deleted:1;
+	u32 incoming:1;

Are C bitfields safe to use in Live Update ABI?

Quoted 4 lines

AFAIU, they aren't. The C standard does not dictate how bitfields are packed into their underlying types (e.g., whether they are packed MSB to LSB or vice-versa), making them highly dependent on the compiler and arch endianness.

Quoted 6 lines

I had the same question and Jason told me otherwise:
https://lore.kernel.org/linux-mm/[email protected]/

I took him at his word and didn't really look much deeper. I use them for memfd preservation as well. See struct memfd_luo_folio_ser in include/linux/kho/abi/memfd.h.

Ah I see.. I guess I agree if x86 has it.. Arm should too.. I'll try to check this.

Thanks
Praan

#11 lore raw

Re: [PATCH v2 02/16] iommu: Implement IOMMU Live update FLB callbacks

In reply to Pranjal Shrivastava
Quoted 19 lines · Pranjal Shrivastava

On Fri, May 01, 2026 at 09:45:19PM +0000, David Matlack wrote:

[...]

Quoted 5 lines
Quoted 5 lines
include/linux/kho/abi/iommu.h new file +5 -0
+
+/**
+ * struct iommu_hdr_ser - Common header for all serialized IOMMU objects
+ * @ref_count: Reference count for the object
+ * @deleted: Flag indicating if the object is deleted
Quoted 9 lines

include/linux/kho/abi/iommu.h line 87

Quoted 6 lines

+ * @incoming: Flag indicating if the object was preserved in previous kernel
+ */

include/linux/kho/abi/iommu.h new file +4 -0
+struct iommu_hdr_ser {
+	u32 ref_count;
+	u32 deleted:1;
+	u32 incoming:1;

Are C bitfields safe to use in Live Update ABI?

Quoted 4 lines

AFAIU, they aren't. The C standard does not dictate how bitfields are packed into their underlying types (e.g., whether they are packed MSB to LSB or vice-versa), making them highly dependent on the compiler and arch endianness.

Bitfield are defined in the SYSV UNIX ELF ABIs, they are stable for kexec, but variable across arches how they lay out in memory.

Jason

#12 lore raw

Re: [PATCH v2 02/16] iommu: Implement IOMMU Live update FLB callbacks

In reply to David Matlack
Quoted 36 lines · David Matlack
Quoted 34 lines · Samiullah Khawaja

Add liveupdate FLB for IOMMU state preservation. Use KHO preserve memory alloc/free helper functions to allocate memory for the IOMMU Live update FLB object and the serialization structs for device, domain and iommu.

During retrieve, walk through the preserved obj array headers and restore each folio. Also recreate the FLB obj.

drivers/iommu/liveupdate.c new file +25 -0
+static void *iommu_liveupdate_restore_array(u64 array_phys)
+{
+	struct iommu_array_hdr_ser *array_hdr;
+	void *vaddr = array_phys ? phys_to_virt(array_phys) : NULL;
+
+	while (array_phys) {
+		/*
+		 * Failure to restore preserved IOMMU state is considered fatal.
+		 *
+		 * This is because the IOMMU translations for preserved IOMMUs
+		 * were kept enabled in the previous kernel and the preserved
+		 * devices have their IOMMU domains still present. Not being
+		 * able to restore means that the memory mapped into preserved
+		 * domains might be already corrupted by the preserved devices.
+		 *
+		 * There is no way to confirm the integrity of the memory that
+		 * was mapped. BUG_ON is the safest option at this point.
+		 */
+		BUG_ON(!kho_restore_folio(array_phys));
+		array_hdr = phys_to_virt(array_phys);
+		array_phys = array_hdr->next_array_phys;
+	}
+
+	return vaddr;
+}
Quoted 14 lines

drivers/iommu/liveupdate.c line 160

Quoted 8 lines
drivers/iommu/liveupdate.c new file +8 -0
+static int iommu_liveupdate_flb_retrieve(struct liveupdate_flb_op_args *argp)
+{
+	struct iommu_flb_obj *obj;
+	struct iommu_flb_ser *ser;
+
+	obj = kzalloc_obj(*obj, GFP_KERNEL);
+	if (!obj)
+		return -ENOMEM;

Should this be considered fatal for the same reason iommu_liveupdate_restore_array() is considered fatal? If anything in iommu_liveupdate_flb_retrieve() fails then the risk of corruption as described in iommu_liveupdate_restore_array() is possible.

Righ... Nice catch. I suppose we should BUG_ON() this because luo_flb_file_finish_one [1] returns void. Thus, if we return -ENOMEM here all we get is a WARN_ON without panic.

We can't statically allocate obj in liveupdate_flb_op_args because obj is a void ptr. I believe we must add a BUG_ON() here.

Quoted 6 lines
Quoted 6 lines
drivers/iommu/liveupdate.c new file +6 -0
+
+	/* Data must be present and valid from the previous kernel */
+	BUG_ON(!kho_restore_folio(argp->data));
+
+	mutex_init(&obj->lock);
+	ser = phys_to_virt(argp->data);
Quoted 8 lines

drivers/iommu/liveupdate.c line 174

Quoted 8 lines
drivers/iommu/liveupdate.c new file +8 -0
+	obj->ser = ser;
+
+	obj->curr_domain_array = iommu_liveupdate_restore_array(ser->iommu_domain_array_phys);
+	obj->curr_device_array = iommu_liveupdate_restore_array(ser->device_array_phys);
+	obj->curr_iommu_array = iommu_liveupdate_restore_array(ser->iommu_array_phys);
+	argp->obj = obj;
+	return 0;
+}

Thanks,
Praan

[1] https://elixir.bootlin.com/linux/v7.1-rc3/source/kernel/liveupdate/luo_flb.c#L208

#13 lore raw

Re: [PATCH v2 02/16] iommu: Implement IOMMU Live update FLB callbacks

Quoted 52 lines · Pranjal Shrivastava
Quoted 36 lines · David Matlack
Quoted 34 lines · Samiullah Khawaja

Add liveupdate FLB for IOMMU state preservation. Use KHO preserve memory alloc/free helper functions to allocate memory for the IOMMU Live update FLB object and the serialization structs for device, domain and iommu.

During retrieve, walk through the preserved obj array headers and restore each folio. Also recreate the FLB obj.

drivers/iommu/liveupdate.c new file +25 -0
+static void *iommu_liveupdate_restore_array(u64 array_phys)
+{
+	struct iommu_array_hdr_ser *array_hdr;
+	void *vaddr = array_phys ? phys_to_virt(array_phys) : NULL;
+
+	while (array_phys) {
+		/*
+		 * Failure to restore preserved IOMMU state is considered fatal.
+		 *
+		 * This is because the IOMMU translations for preserved IOMMUs
+		 * were kept enabled in the previous kernel and the preserved
+		 * devices have their IOMMU domains still present. Not being
+		 * able to restore means that the memory mapped into preserved
+		 * domains might be already corrupted by the preserved devices.
+		 *
+		 * There is no way to confirm the integrity of the memory that
+		 * was mapped. BUG_ON is the safest option at this point.
+		 */
+		BUG_ON(!kho_restore_folio(array_phys));
+		array_hdr = phys_to_virt(array_phys);
+		array_phys = array_hdr->next_array_phys;
+	}
+
+	return vaddr;
+}
Quoted 14 lines

drivers/iommu/liveupdate.c line 160

Quoted 8 lines
drivers/iommu/liveupdate.c new file +8 -0
+static int iommu_liveupdate_flb_retrieve(struct liveupdate_flb_op_args *argp)
+{
+	struct iommu_flb_obj *obj;
+	struct iommu_flb_ser *ser;
+
+	obj = kzalloc_obj(*obj, GFP_KERNEL);
+	if (!obj)
+		return -ENOMEM;

Should this be considered fatal for the same reason iommu_liveupdate_restore_array() is considered fatal? If anything in iommu_liveupdate_flb_retrieve() fails then the risk of corruption as described in iommu_liveupdate_restore_array() is possible.

Quoted 3 lines

Righ... Nice catch. I suppose we should BUG_ON() this because luo_flb_file_finish_one [1] returns void. Thus, if we return -ENOMEM here all we get is a WARN_ON without panic.

The error is propagated by get_flb_incoming(). The finish path has a WARN. But that is fine because finish won't be called as can_finish() will fail if FLB retrieve fails and restore cannot be done.

I will add a comment about this here.

Quoted 23 lines

We can't statically allocate obj in liveupdate_flb_op_args because obj is a void ptr. I believe we must add a BUG_ON() here.

Quoted 6 lines
Quoted 6 lines
drivers/iommu/liveupdate.c new file +6 -0
+
+	/* Data must be present and valid from the previous kernel */
+	BUG_ON(!kho_restore_folio(argp->data));
+
+	mutex_init(&obj->lock);
+	ser = phys_to_virt(argp->data);
Quoted 8 lines

drivers/iommu/liveupdate.c line 174

Quoted 8 lines
drivers/iommu/liveupdate.c new file +8 -0
+	obj->ser = ser;
+
+	obj->curr_domain_array = iommu_liveupdate_restore_array(ser->iommu_domain_array_phys);
+	obj->curr_device_array = iommu_liveupdate_restore_array(ser->device_array_phys);
+	obj->curr_iommu_array = iommu_liveupdate_restore_array(ser->iommu_array_phys);
+	argp->obj = obj;
+	return 0;
+}

Thanks,
Praan

[1] https://elixir.bootlin.com/linux/v7.1-rc3/source/kernel/liveupdate/luo_flb.c#L208

#14 lore raw

Re: [PATCH v2 02/16] iommu: Implement IOMMU Live update FLB callbacks

Quoted 3 lines · Samiullah Khawaja
Quoted 2 lines · Pranjal Shrivastava
Quoted 1 line · David Matlack

On 2026-04-27 05:56 PM, Samiullah Khawaja wrote:

[snip]

Quoted 20 lines
Quoted 16 lines
Quoted 15 lines

drivers/iommu/liveupdate.c line 160

Quoted 9 lines
drivers/iommu/liveupdate.c new file +8 -0
+static int iommu_liveupdate_flb_retrieve(struct liveupdate_flb_op_args *argp)
+{
+	struct iommu_flb_obj *obj;
+	struct iommu_flb_ser *ser;
+
+	obj = kzalloc_obj(*obj, GFP_KERNEL);
+	if (!obj)
+		return -ENOMEM;

Should this be considered fatal for the same reason iommu_liveupdate_restore_array() is considered fatal? If anything in iommu_liveupdate_flb_retrieve() fails then the risk of corruption as described in iommu_liveupdate_restore_array() is possible.

Quoted 3 lines

Righ... Nice catch. I suppose we should BUG_ON() this because luo_flb_file_finish_one [1] returns void. Thus, if we return -ENOMEM here all we get is a WARN_ON without panic.

Quoted 5 lines

The error is propagated by get_flb_incoming(). The finish path has a WARN. But that is fine because finish won't be called as can_finish() will fail if FLB retrieve fails and restore cannot be done.

I will add a comment about this here.

Ack. I was looking at the finish path as it doesn't propagate the error. but it won't be called ig. I assumed that there should be no good situation at boot that would cause -ENOMEM.

In case we're not panicking for an -ENOMEM and letting the user decide, adding a comment would be great.

Thanks,
Praan

#15 lore raw

Re: [PATCH v2 02/16] iommu: Implement IOMMU Live update FLB callbacks

In reply to David Matlack
Quoted 36 lines · David Matlack
Quoted 34 lines · Samiullah Khawaja

Add liveupdate FLB for IOMMU state preservation. Use KHO preserve memory alloc/free helper functions to allocate memory for the IOMMU Live update FLB object and the serialization structs for device, domain and iommu.

During retrieve, walk through the preserved obj array headers and restore each folio. Also recreate the FLB obj.

drivers/iommu/liveupdate.c new file +25 -0
+static void *iommu_liveupdate_restore_array(u64 array_phys)
+{
+	struct iommu_array_hdr_ser *array_hdr;
+	void *vaddr = array_phys ? phys_to_virt(array_phys) : NULL;
+
+	while (array_phys) {
+		/*
+		 * Failure to restore preserved IOMMU state is considered fatal.
+		 *
+		 * This is because the IOMMU translations for preserved IOMMUs
+		 * were kept enabled in the previous kernel and the preserved
+		 * devices have their IOMMU domains still present. Not being
+		 * able to restore means that the memory mapped into preserved
+		 * domains might be already corrupted by the preserved devices.
+		 *
+		 * There is no way to confirm the integrity of the memory that
+		 * was mapped. BUG_ON is the safest option at this point.
+		 */
+		BUG_ON(!kho_restore_folio(array_phys));
+		array_hdr = phys_to_virt(array_phys);
+		array_phys = array_hdr->next_array_phys;
+	}
+
+	return vaddr;
+}
Quoted 13 lines

drivers/iommu/liveupdate.c line 160

Quoted 8 lines
drivers/iommu/liveupdate.c new file +8 -0
+static int iommu_liveupdate_flb_retrieve(struct liveupdate_flb_op_args *argp)
+{
+	struct iommu_flb_obj *obj;
+	struct iommu_flb_ser *ser;
+
+	obj = kzalloc_obj(*obj, GFP_KERNEL);
+	if (!obj)
+		return -ENOMEM;

Should this be considered fatal for the same reason iommu_liveupdate_restore_array() is considered fatal? If anything in iommu_liveupdate_flb_retrieve() fails then the risk of corruption as described in iommu_liveupdate_restore_array() is possible.

This one is a little tricky, it is not really fatal as it doesn't represent a corruption or unexpected behaviour. I think if this fails the probe of IOMMUs should fail and with a rescan, the user can retry this.

I will handle this properly in the caller to make sure a retry can happen for these.

Quoted 25 lines
Quoted 11 lines
drivers/iommu/liveupdate.c new file +10 -0
+
+	/* Data must be present and valid from the previous kernel */
+	BUG_ON(!kho_restore_folio(argp->data));
+
+	mutex_init(&obj->lock);
+	ser = phys_to_virt(argp->data);
+	obj->ser = ser;
+
+	obj->curr_domain_array = iommu_liveupdate_restore_array(ser->iommu_domain_array_phys);
+	obj->curr_device_array = iommu_liveupdate_restore_array(ser->device_array_phys);

drivers/iommu/liveupdate.c line 180

Quoted 10 lines
drivers/iommu/liveupdate.c new file +10 -0
+	obj->curr_iommu_array = iommu_liveupdate_restore_array(ser->iommu_array_phys);
+	argp->obj = obj;
+	return 0;
+}
+
+static struct liveupdate_flb_ops iommu_flb_ops = {
+	.preserve = iommu_liveupdate_flb_preserve,
+	.unpreserve = iommu_liveupdate_flb_unpreserve,
+	.finish = iommu_liveupdate_flb_finish,
+	.retrieve = iommu_liveupdate_flb_retrieve,

nit: I think it's helpful to put these in the order they are expected to be called.

Quoted 4 lines

.preserve = iommu_liveupdate_flb_preserve,
.unpreserve = iommu_liveupdate_flb_unpreserve,
.retrieve = iommu_liveupdate_flb_retrieve,
.finish = iommu_liveupdate_flb_finish,

Agreed. I will do this.

Quoted 10 lines

include/linux/kho/abi/iommu.h line 76

Quoted 8 lines
include/linux/kho/abi/iommu.h new file +3 -0
new file mode 100644
index 000000000000..37b967820f14
 
+enum iommu_type_ser {
+	IOMMU_INVALID,
+};

Please document this enum.

Agreed. Will do.

Quoted 31 lines

include/linux/kho/abi/iommu.h line 87

Quoted 12 lines
include/linux/kho/abi/iommu.h new file +11 -0
+
+/**
+ * struct iommu_hdr_ser - Common header for all serialized IOMMU objects
+ * @ref_count: Reference count for the object
+ * @deleted: Flag indicating if the object is deleted
+ * @incoming: Flag indicating if the object was preserved in previous kernel
+ */
+struct iommu_hdr_ser {
+	u32 ref_count;
+	u32 deleted:1;
+	u32 incoming:1;

Are C bitfields safe to use in Live Update ABI?

Quoted 16 lines
include/linux/kho/abi/iommu.h new file +15 -0
+} __packed;
 
+/**
+ * struct iommu_flb_obj - FLB object allocated in current kernel pointing to
+ *			  preserved state in FLB
+ * @lock: Mutex protecting the object
+ * @ser: Pointer to the serialized state in FLB
+ * @curr_iommu_array: Pointer to the current array of IOMMU instances
+ * @curr_domain_array: Pointer to the current array of domains
+ * @curr_device_array: Pointer to the current array of devices
+ */
+struct iommu_flb_obj {
+	/* @lock: Protects the serialized objects during concurrent preservation */
+	struct mutex lock;
+	struct iommu_flb_ser *ser;
+
Quoted 8 lines

include/linux/kho/abi/iommu.h line 216

Quoted 4 lines
include/linux/kho/abi/iommu.h new file +4 -0
+	struct iommu_hw_array_ser *curr_iommu_array;
+	struct iommu_domain_array_ser *curr_domain_array;
+	struct iommu_device_array_ser *curr_device_array;
+} __packed;

This struct is not ABI so it should not be __packed nor defined in this file. I haven't read the whole series yet but this definition can probably go in drivers/iommu/liveupdate.c.

Agreed on both points. I think it can be moved to liveupdate.c. Will do.

Thanks,
Sami

#16 lore raw

PATCH v2 3/16 iommu: Implement IOMMU domain preservation

In reply to Samiullah Khawaja

Add IOMMU domain ops that can be implemented by the IOMMU drivers if they support IOMMU domain preservation across liveupdate. The new IOMMU domain preserve, unpreserve and restore APIs call these ops to perform respective live update operations.

---
 drivers/iommu/liveupdate.c       | 97 ++++++++++++++++++++++++++++++++
 include/linux/iommu-liveupdate.h | 14 +++++
 include/linux/iommu.h            | 13 +++++
 3 files changed, 124 insertions(+)
drivers/iommu/liveupdate.c +97 -0
index a26099b145c3..f71f14518248 100644
@@ -13,6 +13,9 @@
 #include <linux/iommu.h>
 #include <linux/errno.h>
 
+#define iommu_max_objs_per_page(_array) \
+	((PAGE_SIZE - sizeof(struct iommu_array_hdr_ser)) / sizeof((_array)->objects[0]))
+
 static void *iommu_liveupdate_restore_array(u64 array_phys)
 {
 	struct iommu_array_hdr_ser *array_hdr;
@@ -196,3 +199,97 @@ void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler)
 	liveupdate_unregister_flb(handler, &iommu_flb);
 }
 EXPORT_SYMBOL(iommu_liveupdate_unregister_flb);
+
+static int alloc_object_ser(struct iommu_array_hdr_ser **curr_array_ptr, u64 max_objs)
+{
+	struct iommu_array_hdr_ser *curr_array = *curr_array_ptr;
+	struct iommu_array_hdr_ser *next_array;
+
+	if (curr_array->nr_objects >= max_objs) {
+		next_array = kho_alloc_preserve(PAGE_SIZE);
+		if (IS_ERR(next_array))
+			return PTR_ERR(next_array);
+
+		curr_array->next_array_phys = virt_to_phys(next_array);
+		*curr_array_ptr = next_array;
+		curr_array = next_array;
+	}
+
+	return curr_array->nr_objects++;
+}
+
+static struct iommu_domain_ser *alloc_iommu_domain_ser(struct iommu_flb_obj *flb)
+{
+	int idx;
+
+	idx = alloc_object_ser((struct iommu_array_hdr_ser **)&flb->curr_domain_array,
+			       iommu_max_objs_per_page(flb->curr_domain_array));
+	if (idx < 0)
+		return ERR_PTR(idx);
+
+	flb->curr_domain_array->objects[idx].hdr.ref_count = 1;
+	return &flb->curr_domain_array->objects[idx];
+}
+
+int iommu_domain_preserve(struct iommu_domain *domain, struct iommu_domain_ser **ser)
+{
+	struct iommu_domain_ser *domain_ser;
+	struct iommu_flb_obj *flb_obj;
+	int ret;
+
+	if (!domain->ops->preserve)
+		return -EOPNOTSUPP;
+
+	ret = liveupdate_flb_get_outgoing(&iommu_flb, (void **)&flb_obj);
+	if (ret)
+		return ret;
+
+	guard(mutex)(&flb_obj->lock);
+	domain_ser = alloc_iommu_domain_ser(flb_obj);
+	if (IS_ERR(domain_ser))
+		return PTR_ERR(domain_ser);
+
+	ret = domain->ops->preserve(domain, domain_ser);
+	if (ret) {
+		domain_ser->hdr.deleted = true;
+		return ret;
+	}
+
+	domain->preserved_state = domain_ser;
+	*ser = domain_ser;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(iommu_domain_preserve);
+
+void iommu_domain_unpreserve(struct iommu_domain *domain)
+{
+	struct iommu_domain_ser *domain_ser;
+	struct iommu_flb_obj *flb_obj;
+	int ret;
+
+	if (!domain->ops->unpreserve)
+		return;
+
+	ret = liveupdate_flb_get_outgoing(&iommu_flb, (void **)&flb_obj);
+	if (WARN_ON(ret))
+		return;
+
+	guard(mutex)(&flb_obj->lock);
+
+	if (!domain->preserved_state)
+		return;
+
+	/*
+	 * There is no check for attached devices here. The correctness relies
+	 * on the Live Update Orchestrator's session lifecycle. All resources
+	 * (iommufd, vfio devices) are preserved within a single session. If the
+	 * session is torn down, the .unpreserve callbacks for all files will be
+	 * invoked, ensuring a consistent cleanup without needing explicit
+	 * refcounting for the serialized objects here.
+	 */
+	domain_ser = domain->preserved_state;
+	domain->ops->unpreserve(domain, domain_ser);
+	domain_ser->hdr.deleted = true;
+	domain->preserved_state = NULL;
+}
+EXPORT_SYMBOL_GPL(iommu_domain_unpreserve);
include/linux/iommu-liveupdate.h +14 -0
index 3d1c65ed76fa..6019cfc27428 100644
@@ -12,6 +12,20 @@
 #include <linux/liveupdate.h>
 #include <linux/kho/abi/iommu.h>
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+int iommu_domain_preserve(struct iommu_domain *domain, struct iommu_domain_ser **ser);
+void iommu_domain_unpreserve(struct iommu_domain *domain);
+#else
+static inline int iommu_domain_preserve(struct iommu_domain *domain, struct iommu_domain_ser **ser)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void iommu_domain_unpreserve(struct iommu_domain *domain)
+{
+}
+#endif
+
 int iommu_liveupdate_register_flb(struct liveupdate_file_handler *handler);
 void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler);
 
include/linux/iommu.h +13 -0
index 6f5d1dec3f89..3853a3946733 100644
@@ -14,6 +14,7 @@
 #include <linux/err.h>
 #include <linux/of.h>
 #include <linux/iova_bitmap.h>
+#include <linux/kho/abi/iommu.h>
 #include <uapi/linux/iommufd.h>
 
 #define IOMMU_READ	(1 << 0)
@@ -249,6 +250,10 @@ struct iommu_domain {
 			struct list_head next;
 		};
 	};
+
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	struct iommu_domain_ser *preserved_state;
+#endif
 };
 
 static inline bool iommu_is_dma_domain(struct iommu_domain *domain)
@@ -752,6 +757,11 @@ struct iommu_ops {
  *                           specific mechanisms.
  * @set_pgtable_quirks: Set io page table quirks (IO_PGTABLE_QUIRK_*)
  * @free: Release the domain after use.
+ * @preserve: Preserve the iommu domain for liveupdate.
+ *            Returns 0 on success, a negative errno on failure.
+ * @unpreserve: Unpreserve the iommu domain that was preserved earlier.
+ * @restore: Restore the iommu domain after liveupdate.
+ *           Returns 0 on success, a negative errno on failure.
  */
 struct iommu_domain_ops {
 	int (*attach_dev)(struct iommu_domain *domain, struct device *dev,
@@ -782,6 +792,9 @@ struct iommu_domain_ops {
 				  unsigned long quirks);
 
 	void (*free)(struct iommu_domain *domain);
+	int (*preserve)(struct iommu_domain *domain, struct iommu_domain_ser *ser);
+	void (*unpreserve)(struct iommu_domain *domain, struct iommu_domain_ser *ser);
+	int (*restore)(struct iommu_domain *domain, struct iommu_domain_ser *ser);
 };
 
 /**
Signature · 1 line
2.54.0.545.g6539524ca2-goog
#17 lore raw

Re: [PATCH v2 03/16] iommu: Implement IOMMU domain preservation

drivers/iommu/liveupdate.c void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler) · line 207

Quoted 12 lines · Samiullah Khawaja

Add IOMMU domain ops that can be implemented by the IOMMU drivers if they support IOMMU domain preservation across liveupdate. The new IOMMU domain preserve, unpreserve and restore APIs call these ops to perform respective live update operations.

drivers/iommu/liveupdate.c +5 -0
+static int alloc_object_ser(struct iommu_array_hdr_ser **curr_array_ptr, u64 max_objs)
+{
+	struct iommu_array_hdr_ser *curr_array = *curr_array_ptr;
+	struct iommu_array_hdr_ser *next_array;
+

There's a trade-off being made in this function to leak deleted array elements instead of trying to reuse them that warrants a comment.

Quoted 12 lines
drivers/iommu/liveupdate.c +12 -0
+	if (curr_array->nr_objects >= max_objs) {
+		next_array = kho_alloc_preserve(PAGE_SIZE);
+		if (IS_ERR(next_array))
+			return PTR_ERR(next_array);
+
+		curr_array->next_array_phys = virt_to_phys(next_array);
+		*curr_array_ptr = next_array;
+		curr_array = next_array;
+	}
+
+	return curr_array->nr_objects++;
+}
#18 lore raw

Re: [PATCH v2 03/16] iommu: Implement IOMMU domain preservation

Quoted 8 lines · David Matlack
Quoted 6 lines · Samiullah Khawaja

Add IOMMU domain ops that can be implemented by the IOMMU drivers if they support IOMMU domain preservation across liveupdate. The new IOMMU domain preserve, unpreserve and restore APIs call these ops to perform respective live update operations.

Quoted 8 lines

drivers/iommu/liveupdate.c void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler) · line 207

Quoted 5 lines
drivers/iommu/liveupdate.c +5 -0
+static int alloc_object_ser(struct iommu_array_hdr_ser **curr_array_ptr, u64 max_objs)
+{
+	struct iommu_array_hdr_ser *curr_array = *curr_array_ptr;
+	struct iommu_array_hdr_ser *next_array;
+

There's a trade-off being made in this function to leak deleted array elements instead of trying to reuse them that warrants a comment.

Are you referring to the "deleted" flag in the HDR of each object? Yes we don't look for deleted objects in linked-list of arrays and reuse them for simplicity. I will add a comment that says this.

Quoted 13 lines
Quoted 13 lines
drivers/iommu/liveupdate.c +12 -0
+	if (curr_array->nr_objects >= max_objs) {
+		next_array = kho_alloc_preserve(PAGE_SIZE);
+		if (IS_ERR(next_array))
+			return PTR_ERR(next_array);
+
+		curr_array->next_array_phys = virt_to_phys(next_array);
+		*curr_array_ptr = next_array;
+		curr_array = next_array;
+	}
+
+	return curr_array->nr_objects++;
+}
Reviewed-by on v2 3/16
#19 lore raw

Re: [PATCH v2 03/16] iommu: Implement IOMMU domain preservation

In reply to Samiullah Khawaja
Quoted 44 lines · Samiullah Khawaja

Add IOMMU domain ops that can be implemented by the IOMMU drivers if they support IOMMU domain preservation across liveupdate. The new IOMMU domain preserve, unpreserve and restore APIs call these ops to perform respective live update operations.

---
 drivers/iommu/liveupdate.c       | 97 ++++++++++++++++++++++++++++++++
 include/linux/iommu-liveupdate.h | 14 +++++
 include/linux/iommu.h            | 13 +++++
 3 files changed, 124 insertions(+)
drivers/iommu/liveupdate.c +17 -0
index a26099b145c3..f71f14518248 100644
@@ -13,6 +13,9 @@
 #include <linux/iommu.h>
 #include <linux/errno.h>
 
+#define iommu_max_objs_per_page(_array) \
+	((PAGE_SIZE - sizeof(struct iommu_array_hdr_ser)) / sizeof((_array)->objects[0]))
+
 static void *iommu_liveupdate_restore_array(u64 array_phys)
 {
 	struct iommu_array_hdr_ser *array_hdr;
@@ -196,3 +199,97 @@ void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler)
 	liveupdate_unregister_flb(handler, &iommu_flb);
 }
 EXPORT_SYMBOL(iommu_liveupdate_unregister_flb);
+
+static int alloc_object_ser(struct iommu_array_hdr_ser **curr_array_ptr, u64 max_objs)
+{
+	struct iommu_array_hdr_ser *curr_array = *curr_array_ptr;
+	struct iommu_array_hdr_ser *next_array;
+
+	if (curr_array->nr_objects >= max_objs) {
+		next_array = kho_alloc_preserve(PAGE_SIZE);
+		if (IS_ERR(next_array))
+			return PTR_ERR(next_array);
+
+		curr_array->next_array_phys = virt_to_phys(next_array);
+		*curr_array_ptr = next_array;
+		curr_array = next_array;

drivers/iommu/liveupdate.c void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler) · line 225

Quoted 10 lines
drivers/iommu/liveupdate.c +10 -0
+	}
+
+	return curr_array->nr_objects++;
+}
+
+static struct iommu_domain_ser *alloc_iommu_domain_ser(struct iommu_flb_obj *flb)
+{
+	int idx;
+
+	idx = alloc_object_ser((struct iommu_array_hdr_ser **)&flb->curr_domain_array,

Nit: Such type-casts could be brittle and risk strict-aliasing warnings based on the compiler.

Since alloc_object_ser only updates the top-level tracking pointer, we can make this much cleaner by changing alloc_object_ser to take a void curr_array_ptr. That way, we can pass (void )&flb->curr_domain_array cleanly without the double-pointer type-punning. Something like:

static int alloc_object_ser(void **curr_array_ptr, u64 max_objs)

{
	struct iommu_array_hdr_ser *curr_array = *curr_array_ptr;
	struct iommu_array_hdr_ser *next_array;

	...
}

and we can pass it as:

idx = alloc_object_ser((void **)&flb->curr_domain_array,

		       iommu_max_objs_per_page(flb->curr_domain_array));
Quoted 19 lines
drivers/iommu/liveupdate.c +19 -0
+			       iommu_max_objs_per_page(flb->curr_domain_array));
+	if (idx < 0)
+		return ERR_PTR(idx);
+
+	flb->curr_domain_array->objects[idx].hdr.ref_count = 1;
+	return &flb->curr_domain_array->objects[idx];
+}
+
+int iommu_domain_preserve(struct iommu_domain *domain, struct iommu_domain_ser **ser)
+{
+	struct iommu_domain_ser *domain_ser;
+	struct iommu_flb_obj *flb_obj;
+	int ret;
+
+	if (!domain->ops->preserve)
+		return -EOPNOTSUPP;
+
+	ret = liveupdate_flb_get_outgoing(&iommu_flb, (void **)&flb_obj);
+	if (ret)

drivers/iommu/liveupdate.c void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler) · line 254

Quoted 10 lines
drivers/iommu/liveupdate.c +10 -0
+		return ret;
+
+	guard(mutex)(&flb_obj->lock);
+	domain_ser = alloc_iommu_domain_ser(flb_obj);
+	if (IS_ERR(domain_ser))
+		return PTR_ERR(domain_ser);
+
+	ret = domain->ops->preserve(domain, domain_ser);
+	if (ret) {
+		domain_ser->hdr.deleted = true;

Nit: This will become domain_ser->hdr.flags |= IOMMU_SER_FLAG_DELETED; once we convert the bitfields in PATCH 2 to flags.

Quoted 9 lines
drivers/iommu/liveupdate.c +9 -0
+		return ret;
+	}
+
+	domain->preserved_state = domain_ser;
+	*ser = domain_ser;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(iommu_domain_preserve);
+

[...]

Quoted 14 lines

static inline bool iommu_is_dma_domain(struct iommu_domain *domain)

drivers/iommu/liveupdate.c +5 -0
@@ -752,6 +757,11 @@ struct iommu_ops {
  *                           specific mechanisms.
  * @set_pgtable_quirks: Set io page table quirks (IO_PGTABLE_QUIRK_*)
  * @free: Release the domain after use.
+ * @preserve: Preserve the iommu domain for liveupdate.
+ *            Returns 0 on success, a negative errno on failure.
+ * @unpreserve: Unpreserve the iommu domain that was preserved earlier.
+ * @restore: Restore the iommu domain after liveupdate.
+ *           Returns 0 on success, a negative errno on failure.
  */
 struct iommu_domain_ops {
 	int (*attach_dev)(struct iommu_domain *domain, struct device *dev,

drivers/iommu/liveupdate.c struct iommu_domain_ops { · line 797

Quoted 6 lines
drivers/iommu/liveupdate.c +3 -0
@@ -782,6 +792,9 @@ struct iommu_domain_ops {
 				  unsigned long quirks);
 
 	void (*free)(struct iommu_domain *domain);
+	int (*preserve)(struct iommu_domain *domain, struct iommu_domain_ser *ser);
+	void (*unpreserve)(struct iommu_domain *domain, struct iommu_domain_ser *ser);
+	int (*restore)(struct iommu_domain *domain, struct iommu_domain_ser *ser);

Nit: Should we add restore in the patch we implement it?

drivers/iommu/liveupdate.c struct iommu_domain_ops { · line 798

Quoted 1 line
drivers/iommu/liveupdate.c +0 -0
 };

With the above nits,

Thanks,
Praan

#20 lore raw

Re: [PATCH v2 03/16] iommu: Implement IOMMU domain preservation

Quoted 72 lines · Pranjal Shrivastava
Quoted 44 lines · Samiullah Khawaja

Add IOMMU domain ops that can be implemented by the IOMMU drivers if they support IOMMU domain preservation across liveupdate. The new IOMMU domain preserve, unpreserve and restore APIs call these ops to perform respective live update operations.

---
 drivers/iommu/liveupdate.c       | 97 ++++++++++++++++++++++++++++++++
 include/linux/iommu-liveupdate.h | 14 +++++
 include/linux/iommu.h            | 13 +++++
 3 files changed, 124 insertions(+)
drivers/iommu/liveupdate.c +17 -0
index a26099b145c3..f71f14518248 100644
@@ -13,6 +13,9 @@
 #include <linux/iommu.h>
 #include <linux/errno.h>
 
+#define iommu_max_objs_per_page(_array) \
+	((PAGE_SIZE - sizeof(struct iommu_array_hdr_ser)) / sizeof((_array)->objects[0]))
+
 static void *iommu_liveupdate_restore_array(u64 array_phys)
 {
 	struct iommu_array_hdr_ser *array_hdr;
@@ -196,3 +199,97 @@ void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler)
 	liveupdate_unregister_flb(handler, &iommu_flb);
 }
 EXPORT_SYMBOL(iommu_liveupdate_unregister_flb);
+
+static int alloc_object_ser(struct iommu_array_hdr_ser **curr_array_ptr, u64 max_objs)
+{
+	struct iommu_array_hdr_ser *curr_array = *curr_array_ptr;
+	struct iommu_array_hdr_ser *next_array;
+
+	if (curr_array->nr_objects >= max_objs) {
+		next_array = kho_alloc_preserve(PAGE_SIZE);
+		if (IS_ERR(next_array))
+			return PTR_ERR(next_array);
+
+		curr_array->next_array_phys = virt_to_phys(next_array);
+		*curr_array_ptr = next_array;
+		curr_array = next_array;

drivers/iommu/liveupdate.c void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler) · line 225

Quoted 10 lines
drivers/iommu/liveupdate.c +10 -0
+	}
+
+	return curr_array->nr_objects++;
+}
+
+static struct iommu_domain_ser *alloc_iommu_domain_ser(struct iommu_flb_obj *flb)
+{
+	int idx;
+
+	idx = alloc_object_ser((struct iommu_array_hdr_ser **)&flb->curr_domain_array,

Nit: Such type-casts could be brittle and risk strict-aliasing warnings based on the compiler.

Since alloc_object_ser only updates the top-level tracking pointer, we can make this much cleaner by changing alloc_object_ser to take a void curr_array_ptr. That way, we can pass (void )&flb->curr_domain_array cleanly without the double-pointer type-punning. Something like:

static int alloc_object_ser(void **curr_array_ptr, u64 max_objs)

{
	struct iommu_array_hdr_ser *curr_array = *curr_array_ptr;
	struct iommu_array_hdr_ser *next_array;

	...
}
Quoted 5 lines

and we can pass it as:

idx = alloc_object_ser((void **)&flb->curr_domain_array,

		       iommu_max_objs_per_page(flb->curr_domain_array));

Agreed. I will update this (including other places).

Quoted 24 lines
Quoted 24 lines
drivers/iommu/liveupdate.c +24 -0
+			       iommu_max_objs_per_page(flb->curr_domain_array));
+	if (idx < 0)
+		return ERR_PTR(idx);
+
+	flb->curr_domain_array->objects[idx].hdr.ref_count = 1;
+	return &flb->curr_domain_array->objects[idx];
+}
+
+int iommu_domain_preserve(struct iommu_domain *domain, struct iommu_domain_ser **ser)
+{
+	struct iommu_domain_ser *domain_ser;
+	struct iommu_flb_obj *flb_obj;
+	int ret;
+
+	if (!domain->ops->preserve)
+		return -EOPNOTSUPP;
+
+	ret = liveupdate_flb_get_outgoing(&iommu_flb, (void **)&flb_obj);
+	if (ret)
+		return ret;
+
+	guard(mutex)(&flb_obj->lock);
+	domain_ser = alloc_iommu_domain_ser(flb_obj);
+	if (IS_ERR(domain_ser))
Quoted 8 lines

drivers/iommu/liveupdate.c void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler) · line 254

Quoted 5 lines
drivers/iommu/liveupdate.c +5 -0
+		return PTR_ERR(domain_ser);
+
+	ret = domain->ops->preserve(domain, domain_ser);
+	if (ret) {
+		domain_ser->hdr.deleted = true;

Nit: This will become domain_ser->hdr.flags |= IOMMU_SER_FLAG_DELETED; once we convert the bitfields in PATCH 2 to flags.

Agreed. I think it is being discussed that bitfields are fine in the other thread, but I will switch it to flags for readibility anyway.

Quoted 29 lines
Quoted 10 lines
drivers/iommu/liveupdate.c +9 -0
+		return ret;
+	}
+
+	domain->preserved_state = domain_ser;
+	*ser = domain_ser;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(iommu_domain_preserve);
+

[...]

Quoted 16 lines

static inline bool iommu_is_dma_domain(struct iommu_domain *domain)

drivers/iommu/liveupdate.c +5 -0
@@ -752,6 +757,11 @@ struct iommu_ops {
  *                           specific mechanisms.
  * @set_pgtable_quirks: Set io page table quirks (IO_PGTABLE_QUIRK_*)
  * @free: Release the domain after use.
+ * @preserve: Preserve the iommu domain for liveupdate.
+ *            Returns 0 on success, a negative errno on failure.
+ * @unpreserve: Unpreserve the iommu domain that was preserved earlier.
+ * @restore: Restore the iommu domain after liveupdate.
+ *           Returns 0 on success, a negative errno on failure.
  */
 struct iommu_domain_ops {
 	int (*attach_dev)(struct iommu_domain *domain, struct device *dev,
@@ -782,6 +792,9 @@ struct iommu_domain_ops {
 				  unsigned long quirks);
 
Quoted 6 lines

drivers/iommu/liveupdate.c struct iommu_domain_ops { · line 797

Quoted 4 lines
drivers/iommu/liveupdate.c +3 -0
 	void (*free)(struct iommu_domain *domain);
+	int (*preserve)(struct iommu_domain *domain, struct iommu_domain_ser *ser);
+	void (*unpreserve)(struct iommu_domain *domain, struct iommu_domain_ser *ser);
+	int (*restore)(struct iommu_domain *domain, struct iommu_domain_ser *ser);

Nit: Should we add restore in the patch we implement it?

The ops are implemented in iommupt (separate scope) and it is done after the iommu pages patch as that is also required for it.

Quoted 6 lines

drivers/iommu/liveupdate.c struct iommu_domain_ops { · line 798

Quoted 2 lines
drivers/iommu/liveupdate.c +0 -0
 };

With the above nits,

Thanks Pranjal.

Quoted 3 lines

Thanks,
Praan

Sami

#21 lore raw

Re: [PATCH v2 03/16] iommu: Implement IOMMU domain preservation

Quoted 3 lines · Samiullah Khawaja
Quoted 2 lines · Pranjal Shrivastava
Quoted 1 line · Samiullah Khawaja

Add IOMMU domain ops that can be implemented by the IOMMU drivers if

[snip]

Quoted 9 lines
Quoted 6 lines

drivers/iommu/liveupdate.c void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler) · line 254

Quoted 3 lines
drivers/iommu/liveupdate.c +3 -0
+	ret = domain->ops->preserve(domain, domain_ser);
+	if (ret) {
+		domain_ser->hdr.deleted = true;

Nit: This will become domain_ser->hdr.flags |= IOMMU_SER_FLAG_DELETED; once we convert the bitfields in PATCH 2 to flags.

Agreed. I think it is being discussed that bitfields are fine in the other thread, but I will switch it to flags for readibility anyway.

Ack. Thanks!

Quoted 36 lines
Quoted 29 lines
Quoted 10 lines
drivers/iommu/liveupdate.c +9 -0
+		return ret;
+	}
+
+	domain->preserved_state = domain_ser;
+	*ser = domain_ser;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(iommu_domain_preserve);
+

[...]

Quoted 16 lines

static inline bool iommu_is_dma_domain(struct iommu_domain *domain)

include/linux/iommu.h +5 -0
@@ -752,6 +757,11 @@ struct iommu_ops {
  *                           specific mechanisms.
  * @set_pgtable_quirks: Set io page table quirks (IO_PGTABLE_QUIRK_*)
  * @free: Release the domain after use.
+ * @preserve: Preserve the iommu domain for liveupdate.
+ *            Returns 0 on success, a negative errno on failure.
+ * @unpreserve: Unpreserve the iommu domain that was preserved earlier.
+ * @restore: Restore the iommu domain after liveupdate.
+ *           Returns 0 on success, a negative errno on failure.
  */
 struct iommu_domain_ops {
 	int (*attach_dev)(struct iommu_domain *domain, struct device *dev,
@@ -782,6 +792,9 @@ struct iommu_domain_ops {
 				  unsigned long quirks);
 
Quoted 6 lines

include/linux/iommu.h struct iommu_domain_ops { · line 797

Quoted 4 lines
include/linux/iommu.h +3 -0
 	void (*free)(struct iommu_domain *domain);
+	int (*preserve)(struct iommu_domain *domain, struct iommu_domain_ser *ser);
+	void (*unpreserve)(struct iommu_domain *domain, struct iommu_domain_ser *ser);
+	int (*restore)(struct iommu_domain *domain, struct iommu_domain_ser *ser);

Nit: Should we add restore in the patch we implement it?

Quoted 2 lines

The ops are implemented in iommupt (separate scope) and it is done after the iommu pages patch as that is also required for it.

I missed the restore implementation in Patch 6, I thought that'll be covered in a diff series. Mb. Please ignore this comment.

Quoted 5 lines
Quoted 5 lines

Review struct iommu_domain_ops { · line 798

Quoted 2 lines
(quoted diff) +0 -0
 };

With the above nits,

Quoted 8 lines
Quoted 1 line

Thanks Pranjal.

Quoted 3 lines

Thanks,
Praan

Sami

Thanks.
Praan

#22 lore raw

PATCH v2 4/16 iommu: Implement device and IOMMU HW preservation

In reply to Samiullah Khawaja

Add IOMMU ops to preserve/unpreserve a device. These can be implemented by the IOMMU drivers that support preservation of devices that have their IOMMU domains preserved. During device preservation the state of the associated IOMMU is also preserved as dependency.

---
 drivers/iommu/liveupdate.c       | 162 +++++++++++++++++++++++++++++++
 include/linux/iommu-liveupdate.h |  33 +++++++
 include/linux/iommu.h            |  20 ++++
 3 files changed, 215 insertions(+)
drivers/iommu/liveupdate.c +162 -0
index f71f14518248..765d042e22e3 100644
@@ -11,6 +11,7 @@
 #include <linux/liveupdate.h>
 #include <linux/iommu-liveupdate.h>
 #include <linux/iommu.h>
+#include <linux/pci.h>
 #include <linux/errno.h>
 
 #define iommu_max_objs_per_page(_array) \
@@ -293,3 +294,164 @@ void iommu_domain_unpreserve(struct iommu_domain *domain)
 	domain->preserved_state = NULL;
 }
 EXPORT_SYMBOL_GPL(iommu_domain_unpreserve);
+
+static struct iommu_hw_ser *alloc_iommu_hw_ser(struct iommu_flb_obj *flb)
+{
+	int idx;
+
+	idx = alloc_object_ser((struct iommu_array_hdr_ser **)&flb->curr_iommu_array,
+			       iommu_max_objs_per_page(flb->curr_iommu_array));
+	if (idx < 0)
+		return ERR_PTR(idx);
+
+	flb->curr_iommu_array->objects[idx].hdr.ref_count = 1;
+	return &flb->curr_iommu_array->objects[idx];
+}
+
+static int iommu_preserve_locked(struct iommu_device *iommu,
+				 struct iommu_flb_obj *flb_obj)
+{
+	struct iommu_hw_ser *iommu_hw_ser;
+	int ret;
+
+	if (!iommu->ops->preserve)
+		return -EOPNOTSUPP;
+
+	lockdep_assert_held(&flb_obj->lock);
+	if (iommu->outgoing_preserved_state) {
+		iommu->outgoing_preserved_state->hdr.ref_count++;
+		return 0;
+	}
+
+	iommu_hw_ser = alloc_iommu_hw_ser(flb_obj);
+	if (IS_ERR(iommu_hw_ser))
+		return PTR_ERR(iommu_hw_ser);
+
+	ret = iommu->ops->preserve(iommu, iommu_hw_ser);
+	if (ret) {
+		iommu_hw_ser->hdr.deleted = true;
+		return ret;
+	}
+
+	iommu->outgoing_preserved_state = iommu_hw_ser;
+	return ret;
+}
+
+static void iommu_unpreserve_locked(struct iommu_device *iommu,
+				    struct iommu_flb_obj *flb_obj)
+{
+	struct iommu_hw_ser *iommu_hw_ser = iommu->outgoing_preserved_state;
+
+	lockdep_assert_held(&flb_obj->lock);
+	iommu_hw_ser->hdr.ref_count--;
+	if (iommu_hw_ser->hdr.ref_count)
+		return;
+
+	iommu->outgoing_preserved_state = NULL;
+	iommu->ops->unpreserve(iommu, iommu_hw_ser);
+	iommu_hw_ser->hdr.deleted = true;
+}
+
+static struct iommu_device_ser *alloc_iommu_device_ser(struct iommu_flb_obj *flb)
+{
+	int idx;
+
+	idx = alloc_object_ser((struct iommu_array_hdr_ser **)&flb->curr_device_array,
+			       iommu_max_objs_per_page(flb->curr_device_array));
+	if (idx < 0)
+		return ERR_PTR(idx);
+
+	flb->curr_device_array->objects[idx].hdr.ref_count = 1;
+	return &flb->curr_device_array->objects[idx];
+}
+
+int iommu_preserve_device(struct iommu_domain *domain,
+			  struct device *dev, u64 *preserved_state)
+{
+	struct iommu_flb_obj *flb_obj;
+	struct iommu_device_ser *device_ser;
+	struct dev_iommu *iommu;
+	struct pci_dev *pdev;
+	int ret;
+
+	if (!dev_is_pci(dev))
+		return -EOPNOTSUPP;
+
+	if (!domain->preserved_state)
+		return -EINVAL;
+
+	if (!iommu_group_dma_owner_claimed(dev->iommu_group))
+		return -EINVAL;
+
+	pdev = to_pci_dev(dev);
+	iommu = dev->iommu;
+	if (!iommu->iommu_dev->ops->preserve_device ||
+	    !iommu->iommu_dev->ops->preserve)
+		return -EOPNOTSUPP;
+
+	ret = liveupdate_flb_get_outgoing(&iommu_flb, (void **)&flb_obj);
+	if (ret)
+		return ret;
+
+	guard(mutex)(&flb_obj->lock);
+	device_ser = alloc_iommu_device_ser(flb_obj);
+	if (IS_ERR(device_ser))
+		return PTR_ERR(device_ser);
+
+	ret = iommu_preserve_locked(iommu->iommu_dev, flb_obj);
+	if (ret) {
+		device_ser->hdr.deleted = true;
+		return ret;
+	}
+
+	device_ser->domain_iommu_ser.domain_phys = __pa(domain->preserved_state);
+	device_ser->domain_iommu_ser.iommu_phys = __pa(iommu->iommu_dev->outgoing_preserved_state);
+	device_ser->devid = pci_dev_id(pdev);
+	device_ser->pci_domain_nr = pci_domain_nr(pdev->bus);
+
+	ret = iommu->iommu_dev->ops->preserve_device(dev, device_ser);
+	if (ret) {
+		device_ser->hdr.deleted = true;
+		iommu_unpreserve_locked(iommu->iommu_dev, flb_obj);
+		return ret;
+	}
+
+	dev->iommu->device_ser = device_ser;
+	*preserved_state = virt_to_phys(device_ser);
+	return 0;
+}
+
+void iommu_unpreserve_device(struct iommu_domain *domain, struct device *dev)
+{
+	struct iommu_flb_obj *flb_obj;
+	struct iommu_device_ser *iommu_device_ser;
+	struct dev_iommu *iommu;
+	struct pci_dev *pdev;
+	int ret;
+
+	if (!dev_is_pci(dev))
+		return;
+
+	if (!iommu_group_dma_owner_claimed(dev->iommu_group))
+		return;
+
+	pdev = to_pci_dev(dev);
+	iommu = dev->iommu;
+	if (!iommu->iommu_dev->ops->unpreserve_device ||
+	    !iommu->iommu_dev->ops->unpreserve)
+		return;
+
+	ret = liveupdate_flb_get_outgoing(&iommu_flb, (void **)&flb_obj);
+	if (WARN_ON(ret))
+		return;
+
+	guard(mutex)(&flb_obj->lock);
+	iommu_device_ser = dev_iommu_preserved_state(dev);
+	if (WARN_ON(!iommu_device_ser))
+		return;
+
+	iommu->iommu_dev->ops->unpreserve_device(dev, iommu_device_ser);
+	dev->iommu->device_ser = NULL;
+
+	iommu_unpreserve_locked(iommu->iommu_dev, flb_obj);
+}
include/linux/iommu-liveupdate.h +33 -0
index 6019cfc27428..279c7ab04f09 100644
@@ -8,14 +8,37 @@
 #ifndef _LINUX_IOMMU_LIVEUPDATE_H
 #define _LINUX_IOMMU_LIVEUPDATE_H
 
+#include <linux/device.h>
 #include <linux/iommu.h>
 #include <linux/liveupdate.h>
 #include <linux/kho/abi/iommu.h>
 
 #ifdef CONFIG_IOMMU_LIVEUPDATE
+static inline void *dev_iommu_preserved_state(struct device *dev)
+{
+	struct iommu_device_ser *ser;
+
+	if (!dev->iommu)
+		return NULL;
+
+	ser = dev->iommu->device_ser;
+	if (ser && !ser->hdr.incoming)
+		return ser;
+
+	return NULL;
+}
+
 int iommu_domain_preserve(struct iommu_domain *domain, struct iommu_domain_ser **ser);
 void iommu_domain_unpreserve(struct iommu_domain *domain);
+int iommu_preserve_device(struct iommu_domain *domain,
+			  struct device *dev, u64 *preserved_state);
+void iommu_unpreserve_device(struct iommu_domain *domain, struct device *dev);
 #else
+static inline void *dev_iommu_preserved_state(struct device *dev)
+{
+	return NULL;
+}
+
 static inline int iommu_domain_preserve(struct iommu_domain *domain, struct iommu_domain_ser **ser)
 {
 	return -EOPNOTSUPP;
@@ -24,6 +47,16 @@ static inline int iommu_domain_preserve(struct iommu_domain *domain, struct iomm
 static inline void iommu_domain_unpreserve(struct iommu_domain *domain)
 {
 }
+
+static inline int iommu_preserve_device(struct iommu_domain *domain,
+					struct device *dev, u64 *preserved_state)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void iommu_unpreserve_device(struct iommu_domain *domain, struct device *dev)
+{
+}
 #endif
 
 int iommu_liveupdate_register_flb(struct liveupdate_file_handler *handler);
include/linux/iommu.h +20 -0
index 3853a3946733..1c424b32c5fc 100644
@@ -655,6 +655,10 @@ __iommu_copy_struct_to_user(const struct iommu_user_data *dst_data,
  *               resources shared/passed to user space IOMMU instance. Associate
  *               it with a nesting @parent_domain. It is required for driver to
  *               set @viommu->ops pointing to its own viommu_ops
+ * @preserve_device: Preserve state of a device for liveupdate.
+ * @unpreserve_device: Unpreserve state that was preserved earlier.
+ * @preserve: Preserve state of iommu translation hardware for liveupdate.
+ * @unpreserve: Unpreserve state of iommu that was preserved earlier.
  * @owner: Driver module providing these ops
  * @identity_domain: An always available, always attachable identity
  *                   translation.
@@ -711,6 +715,13 @@ struct iommu_ops {
 			   struct iommu_domain *parent_domain,
 			   const struct iommu_user_data *user_data);
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	int (*preserve_device)(struct device *dev, struct iommu_device_ser *device_ser);
+	void (*unpreserve_device)(struct device *dev, struct iommu_device_ser *device_ser);
+	int (*preserve)(struct iommu_device *iommu, struct iommu_hw_ser *iommu_ser);
+	void (*unpreserve)(struct iommu_device *iommu, struct iommu_hw_ser *iommu_ser);
+#endif
+
 	const struct iommu_domain_ops *default_domain_ops;
 	struct module *owner;
 	struct iommu_domain *identity_domain;
@@ -806,6 +817,8 @@ struct iommu_domain_ops {
  * @singleton_group: Used internally for drivers that have only one group
  * @max_pasids: number of supported PASIDs
  * @ready: set once iommu_device_register() has completed successfully
+ * @outgoing_preserved_state: preserved iommu state of outgoing kernel for
+ * liveupdate.
  */
 struct iommu_device {
 	struct list_head list;
@@ -815,6 +828,10 @@ struct iommu_device {
 	struct iommu_group *singleton_group;
 	u32 max_pasids;
 	bool ready;
+
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	struct iommu_hw_ser *outgoing_preserved_state;
+#endif
 };
 
 /**
@@ -869,6 +886,9 @@ struct dev_iommu {
 	u32				pci_32bit_workaround:1;
 	u32				require_direct:1;
 	u32				shadow_on_flush:1;
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	struct iommu_device_ser		*device_ser;
+#endif
 };
 
 int iommu_device_register(struct iommu_device *iommu,
Signature · 1 line
2.54.0.545.g6539524ca2-goog
#23 lore raw

Re: [PATCH v2 04/16] iommu: Implement device and IOMMU HW preservation

Quoted 1 line · Samiullah Khawaja

Add IOMMU ops to preserve/unpreserve a device. These can be implemented

Can you make this comment more specific about what is being preserved? Saying it preserves a device is vague and maybe even misleading. It's more about about preserving a device's attachment to a specific domain correct?

drivers/iommu/liveupdate.c

Quoted 12 lines

by the IOMMU drivers that support preservation of devices that have their IOMMU domains preserved. During device preservation the state of the associated IOMMU is also preserved as dependency.

drivers/iommu/liveupdate.c +1 -0
index f71f14518248..765d042e22e3 100644
 
+static struct iommu_device_ser *alloc_iommu_device_ser(struct iommu_flb_obj *flb)

It is unforunate that struct iommu_device_ser has nothing to do with struct iommu_device. The former represents an a PCI device, while the latter represents an IOMMU.

include/linux/iommu-liveupdate.h line 35

Quoted 10 lines
include/linux/iommu-liveupdate.h +3 -0
index 6019cfc27428..279c7ab04f09 100644
 
 int iommu_domain_preserve(struct iommu_domain *domain, struct iommu_domain_ser **ser);
 void iommu_domain_unpreserve(struct iommu_domain *domain);
+int iommu_preserve_device(struct iommu_domain *domain,
+			  struct device *dev, u64 *preserved_state);
+void iommu_unpreserve_device(struct iommu_domain *domain, struct device *dev);

The naming scheme is inconsistent... Maybe it can be:

iommu_preserve_domain()
iommu_unpreserve_domain()
iommu_preserve_device() or iommu_preserve_device_attachment() iommu_unpreserve_device() or iommu_unpreserve_device_attachment()

include/linux/iommu.h struct iommu_ops { · line 723

Quoted 11 lines
include/linux/iommu.h +6 -0
index 3853a3946733..1c424b32c5fc 100644
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	int (*preserve_device)(struct device *dev, struct iommu_device_ser *device_ser);
+	void (*unpreserve_device)(struct device *dev, struct iommu_device_ser *device_ser);
+	int (*preserve)(struct iommu_device *iommu, struct iommu_hw_ser *iommu_ser);
+	void (*unpreserve)(struct iommu_device *iommu, struct iommu_hw_ser *iommu_ser);
+#endif

Maybe we can make these names a little more specific:

preserve_device_attachment()
unpreserve_device_attachment()
preserve_iommu()
unpreserve_iommu()

?

#24 lore raw

Re: [PATCH v2 04/16] iommu: Implement device and IOMMU HW preservation

Quoted 7 lines · David Matlack
Quoted 1 line · Samiullah Khawaja

Add IOMMU ops to preserve/unpreserve a device. These can be implemented

Can you make this comment more specific about what is being preserved? Saying it preserves a device is vague and maybe even misleading. It's more about about preserving a device's attachment to a specific domain correct?

There is attachment ID, but the preservation of device can have IOMMU driver specific things, so in core I mostly mention "preseve device specific state". In later patches in this series, we save PASID table using the same callback. I will add more details in the commit message.

Quoted 12 lines
Quoted 11 lines

by the IOMMU drivers that support preservation of devices that have their IOMMU domains preserved. During device preservation the state of the associated IOMMU is also preserved as dependency.

drivers/iommu/liveupdate.c +0 -0
index f71f14518248..765d042e22e3 100644
Quoted 5 lines
Quoted 1 line

+static struct iommu_device_ser *alloc_iommu_device_ser(struct iommu_flb_obj *flb)

It is unforunate that struct iommu_device_ser has nothing to do with struct iommu_device. The former represents an a PCI device, while the latter represents an IOMMU.

Yes, I went through various iterations of trying to name it in a different way but keeping the "iommu_" prefix and the "device" state naturally falls into this. Not sure if iommu_pci_device_ser is suitable?

Quoted 14 lines

include/linux/iommu-liveupdate.h line 35

Quoted 11 lines
include/linux/iommu-liveupdate.h +3 -0
index 6019cfc27428..279c7ab04f09 100644
 
 int iommu_domain_preserve(struct iommu_domain *domain, struct iommu_domain_ser **ser);
 void iommu_domain_unpreserve(struct iommu_domain *domain);
+int iommu_preserve_device(struct iommu_domain *domain,
+			  struct device *dev, u64 *preserved_state);
+void iommu_unpreserve_device(struct iommu_domain *domain, struct device *dev);

The naming scheme is inconsistent... Maybe it can be:

Quoted 4 lines

iommu_preserve_domain()
iommu_unpreserve_domain()
iommu_preserve_device() or iommu_preserve_device_attachment() iommu_unpreserve_device() or iommu_unpreserve_device_attachment()

Agreed, but I am trying to follow the already existing naming scheme for domains that is used for APIs in iommu.c

iommu_domain_free()
iommu_domain_init()
iommu_domain_preserve()

But I think this is rare, I will update to this as you mentioned above:

iommu_preserve_domain()
iommu_unpreserve_domain()
iommu_preserve_device()
iommu_unpreserve_device()

Quoted 15 lines

include/linux/iommu.h struct iommu_ops { · line 723

Quoted 12 lines
include/linux/iommu.h +6 -0
index 3853a3946733..1c424b32c5fc 100644
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	int (*preserve_device)(struct device *dev, struct iommu_device_ser *device_ser);
+	void (*unpreserve_device)(struct device *dev, struct iommu_device_ser *device_ser);
+	int (*preserve)(struct iommu_device *iommu, struct iommu_hw_ser *iommu_ser);
+	void (*unpreserve)(struct iommu_device *iommu, struct iommu_hw_ser *iommu_ser);
+#endif

Maybe we can make these names a little more specific:

Quoted 2 lines

preserve_device_attachment()
unpreserve_device_attachment()

Attachment is too specific. See my comment above.

Quoted 2 lines

preserve_iommu()
unpreserve_iommu()

These are part of iommu_ops and having preserve_iommu() instead of preserve() is redundant I think. Note ops like capable(), hw_info() in the same struct.

Quoted 2 lines

?

maintainer
#25 lore raw

Re: [PATCH v2 04/16] iommu: Implement device and IOMMU HW preservation

In reply to Samiullah Khawaja
Quoted 164 lines · Samiullah Khawaja

Add IOMMU ops to preserve/unpreserve a device. These can be implemented by the IOMMU drivers that support preservation of devices that have their IOMMU domains preserved. During device preservation the state of the associated IOMMU is also preserved as dependency.

---
  drivers/iommu/liveupdate.c       | 162 +++++++++++++++++++++++++++++++
  include/linux/iommu-liveupdate.h |  33 +++++++
  include/linux/iommu.h            |  20 ++++
  3 files changed, 215 insertions(+)
drivers/iommu/liveupdate.c +137 -0
index f71f14518248..765d042e22e3 100644
@@ -11,6 +11,7 @@
  #include <linux/liveupdate.h>
  #include <linux/iommu-liveupdate.h>
  #include <linux/iommu.h>
+#include <linux/pci.h>
  #include <linux/errno.h>
 
  #define iommu_max_objs_per_page(_array) \
@@ -293,3 +294,164 @@ void iommu_domain_unpreserve(struct iommu_domain *domain)
  	domain->preserved_state = NULL;
  }
  EXPORT_SYMBOL_GPL(iommu_domain_unpreserve);
+
+static struct iommu_hw_ser *alloc_iommu_hw_ser(struct iommu_flb_obj *flb)
+{
+	int idx;
+
+	idx = alloc_object_ser((struct iommu_array_hdr_ser **)&flb->curr_iommu_array,
+			       iommu_max_objs_per_page(flb->curr_iommu_array));
+	if (idx < 0)
+		return ERR_PTR(idx);
+
+	flb->curr_iommu_array->objects[idx].hdr.ref_count = 1;
+	return &flb->curr_iommu_array->objects[idx];
+}
+
+static int iommu_preserve_locked(struct iommu_device *iommu,
+				 struct iommu_flb_obj *flb_obj)
+{
+	struct iommu_hw_ser *iommu_hw_ser;
+	int ret;
+
+	if (!iommu->ops->preserve)
+		return -EOPNOTSUPP;
+
+	lockdep_assert_held(&flb_obj->lock);
+	if (iommu->outgoing_preserved_state) {
+		iommu->outgoing_preserved_state->hdr.ref_count++;
+		return 0;
+	}
+
+	iommu_hw_ser = alloc_iommu_hw_ser(flb_obj);
+	if (IS_ERR(iommu_hw_ser))
+		return PTR_ERR(iommu_hw_ser);
+
+	ret = iommu->ops->preserve(iommu, iommu_hw_ser);
+	if (ret) {
+		iommu_hw_ser->hdr.deleted = true;
+		return ret;
+	}
+
+	iommu->outgoing_preserved_state = iommu_hw_ser;
+	return ret;
+}
+
+static void iommu_unpreserve_locked(struct iommu_device *iommu,
+				    struct iommu_flb_obj *flb_obj)
+{
+	struct iommu_hw_ser *iommu_hw_ser = iommu->outgoing_preserved_state;
+
+	lockdep_assert_held(&flb_obj->lock);
+	iommu_hw_ser->hdr.ref_count--;
+	if (iommu_hw_ser->hdr.ref_count)
+		return;
+
+	iommu->outgoing_preserved_state = NULL;
+	iommu->ops->unpreserve(iommu, iommu_hw_ser);
+	iommu_hw_ser->hdr.deleted = true;
+}
+
+static struct iommu_device_ser *alloc_iommu_device_ser(struct iommu_flb_obj *flb)
+{
+	int idx;
+
+	idx = alloc_object_ser((struct iommu_array_hdr_ser **)&flb->curr_device_array,
+			       iommu_max_objs_per_page(flb->curr_device_array));
+	if (idx < 0)
+		return ERR_PTR(idx);
+
+	flb->curr_device_array->objects[idx].hdr.ref_count = 1;
+	return &flb->curr_device_array->objects[idx];
+}
+
+int iommu_preserve_device(struct iommu_domain *domain,
+			  struct device *dev, u64 *preserved_state)
+{
+	struct iommu_flb_obj *flb_obj;
+	struct iommu_device_ser *device_ser;
+	struct dev_iommu *iommu;
+	struct pci_dev *pdev;
+	int ret;
+
+	if (!dev_is_pci(dev))
+		return -EOPNOTSUPP;
+
+	if (!domain->preserved_state)
+		return -EINVAL;
+
+	if (!iommu_group_dma_owner_claimed(dev->iommu_group))
+		return -EINVAL;
+
+	pdev = to_pci_dev(dev);
+	iommu = dev->iommu;
+	if (!iommu->iommu_dev->ops->preserve_device ||
+	    !iommu->iommu_dev->ops->preserve)
+		return -EOPNOTSUPP;
+
+	ret = liveupdate_flb_get_outgoing(&iommu_flb, (void **)&flb_obj);
+	if (ret)
+		return ret;
+
+	guard(mutex)(&flb_obj->lock);
+	device_ser = alloc_iommu_device_ser(flb_obj);
+	if (IS_ERR(device_ser))
+		return PTR_ERR(device_ser);
+
+	ret = iommu_preserve_locked(iommu->iommu_dev, flb_obj);
+	if (ret) {
+		device_ser->hdr.deleted = true;
+		return ret;
+	}
+
+	device_ser->domain_iommu_ser.domain_phys = __pa(domain->preserved_state);
+	device_ser->domain_iommu_ser.iommu_phys = __pa(iommu->iommu_dev->outgoing_preserved_state);
+	device_ser->devid = pci_dev_id(pdev);
+	device_ser->pci_domain_nr = pci_domain_nr(pdev->bus);
+
+	ret = iommu->iommu_dev->ops->preserve_device(dev, device_ser);
+	if (ret) {
+		device_ser->hdr.deleted = true;
+		iommu_unpreserve_locked(iommu->iommu_dev, flb_obj);
+		return ret;
+	}
+
+	dev->iommu->device_ser = device_ser;
+	*preserved_state = virt_to_phys(device_ser);
+	return 0;
+}
+
+void iommu_unpreserve_device(struct iommu_domain *domain, struct device *dev)
+{
+	struct iommu_flb_obj *flb_obj;
+	struct iommu_device_ser *iommu_device_ser;
+	struct dev_iommu *iommu;
+	struct pci_dev *pdev;
+	int ret;
+
+	if (!dev_is_pci(dev))

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 442

Quoted 10 lines
drivers/iommu/liveupdate.c +10 -0
+		return;
+
+	if (!iommu_group_dma_owner_claimed(dev->iommu_group))
+		return;
+
+	pdev = to_pci_dev(dev);
+	iommu = dev->iommu;
+	if (!iommu->iommu_dev->ops->unpreserve_device ||
+	    !iommu->iommu_dev->ops->unpreserve)
+		return;

Is it considered a driver bug if it implements the preserve hooks but not unpreserve ones? This would at least cause a silent memory leak. How about adding a WARN like this?

    if (WARN_ON_ONCE(!iommu->iommu_dev->ops->unpreserve_device ||
                     !iommu->iommu_dev->ops->unpreserve))
        return;

?

Thanks,
baolu

#26 lore raw

Re: [PATCH v2 04/16] iommu: Implement device and IOMMU HW preservation

Quoted 13 lines · Baolu Lu
Quoted 12 lines · Samiullah Khawaja

Add IOMMU ops to preserve/unpreserve a device. These can be implemented by the IOMMU drivers that support preservation of devices that have their IOMMU domains preserved. During device preservation the state of the associated IOMMU is also preserved as dependency.

---
 drivers/iommu/liveupdate.c       | 162 +++++++++++++++++++++++++++++++
 include/linux/iommu-liveupdate.h |  33 +++++++
 include/linux/iommu.h            |  20 ++++
 3 files changed, 215 insertions(+)

[snip]

Quoted 16 lines
Quoted 16 lines
drivers/iommu/liveupdate.c +16 -0
+
+int iommu_preserve_device(struct iommu_domain *domain,
+			  struct device *dev, u64 *preserved_state)
+{
+	struct iommu_flb_obj *flb_obj;
+	struct iommu_device_ser *device_ser;
+	struct dev_iommu *iommu;
+	struct pci_dev *pdev;
+	int ret;
+
+	if (!dev_is_pci(dev))
+		return -EOPNOTSUPP;
+
+	if (!domain->preserved_state)
+		return -EINVAL;
+
Quoted 8 lines

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 390

Quoted 8 lines
drivers/iommu/liveupdate.c +8 -0
+	if (!iommu_group_dma_owner_claimed(dev->iommu_group))
+		return -EINVAL;
+
+	pdev = to_pci_dev(dev);
+	iommu = dev->iommu;
+	if (!iommu->iommu_dev->ops->preserve_device ||
+	    !iommu->iommu_dev->ops->preserve)
+		return -EOPNOTSUPP;

I will check for unpreserve ops here also.

Quoted 5 lines
Quoted 5 lines
drivers/iommu/liveupdate.c +5 -0
+
+	ret = liveupdate_flb_get_outgoing(&iommu_flb, (void **)&flb_obj);
+	if (ret)
+		return ret;
+

[snip]

Quoted 17 lines
Quoted 17 lines
drivers/iommu/liveupdate.c +17 -0
+}
+
+void iommu_unpreserve_device(struct iommu_domain *domain, struct device *dev)
+{
+	struct iommu_flb_obj *flb_obj;
+	struct iommu_device_ser *iommu_device_ser;
+	struct dev_iommu *iommu;
+	struct pci_dev *pdev;
+	int ret;
+
+	if (!dev_is_pci(dev))
+		return;
+
+	if (!iommu_group_dma_owner_claimed(dev->iommu_group))
+		return;
+
+	pdev = to_pci_dev(dev);
Quoted 8 lines

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 442

Quoted 4 lines
drivers/iommu/liveupdate.c +4 -0
+	iommu = dev->iommu;
+	if (!iommu->iommu_dev->ops->unpreserve_device ||
+	    !iommu->iommu_dev->ops->unpreserve)
+		return;

Is it considered a driver bug if it implements the preserve hooks but not unpreserve ones? This would at least cause a silent memory leak. How about adding a WARN like this?

I think the preservation should not be done if the unpreserve ops are not implemented by the driver. I will add these checks in the _preserve functions and return EOPNOTSUPP if it is not implemented by the driver.

Quoted 4 lines
   if (WARN_ON_ONCE(!iommu->iommu_dev->ops->unpreserve_device ||
                    !iommu->iommu_dev->ops->unpreserve))
       return;

We can add this warning here also.

Quoted 6 lines

?

Thanks,
baolu

Thanks,
Sami

#27 lore raw

Re: [PATCH v2 04/16] iommu: Implement device and IOMMU HW preservation

Quoted 42 lines · Samiullah Khawaja
Quoted 13 lines · Baolu Lu
Quoted 12 lines · Samiullah Khawaja

Add IOMMU ops to preserve/unpreserve a device. These can be implemented by the IOMMU drivers that support preservation of devices that have their IOMMU domains preserved. During device preservation the state of the associated IOMMU is also preserved as dependency.

---
 drivers/iommu/liveupdate.c       | 162 +++++++++++++++++++++++++++++++
 include/linux/iommu-liveupdate.h |  33 +++++++
 include/linux/iommu.h            |  20 ++++
 3 files changed, 215 insertions(+)

[snip]

Quoted 16 lines
Quoted 16 lines
drivers/iommu/liveupdate.c +16 -0
+
+int iommu_preserve_device(struct iommu_domain *domain,
+			  struct device *dev, u64 *preserved_state)
+{
+	struct iommu_flb_obj *flb_obj;
+	struct iommu_device_ser *device_ser;
+	struct dev_iommu *iommu;
+	struct pci_dev *pdev;
+	int ret;
+
+	if (!dev_is_pci(dev))
+		return -EOPNOTSUPP;
+
+	if (!domain->preserved_state)
+		return -EINVAL;
+
Quoted 8 lines

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 390

Quoted 8 lines
drivers/iommu/liveupdate.c +8 -0
+	if (!iommu_group_dma_owner_claimed(dev->iommu_group))
+		return -EINVAL;
+
+	pdev = to_pci_dev(dev);
+	iommu = dev->iommu;
+	if (!iommu->iommu_dev->ops->preserve_device ||
+	    !iommu->iommu_dev->ops->preserve)
+		return -EOPNOTSUPP;

I will check for unpreserve ops here also.

[snip]

Quoted 14 lines
Quoted 9 lines

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 442

Quoted 5 lines
drivers/iommu/liveupdate.c +5 -0
+	pdev = to_pci_dev(dev);
+	iommu = dev->iommu;
+	if (!iommu->iommu_dev->ops->unpreserve_device ||
+	    !iommu->iommu_dev->ops->unpreserve)
+		return;

Is it considered a driver bug if it implements the preserve hooks but not unpreserve ones? This would at least cause a silent memory leak. How about adding a WARN like this?

I think the preservation should not be done if the unpreserve ops are not implemented by the driver. I will add these checks in the _preserve functions and return EOPNOTSUPP if it is not implemented by the driver.

Quoted 1 line
Quoted 5 lines
Quoted 3 lines
   if (WARN_ON_ONCE(!iommu->iommu_dev->ops->unpreserve_device ||
                    !iommu->iommu_dev->ops->unpreserve))
       return;

We can add this warning here also.

I'd rather have the iommu device registration fail instead of solely relying on these WARN_ONs here. We can simply add a snippet inside iommu_device_register():

#ifdef CONFIG_IOMMU_LIVEUPDATE
if ((iommu->ops->preserve && !iommu->ops->unpreserve) ||

    (!iommu->ops->preserve && iommu->ops->unpreserve)) {
        pr_err("IOMMU: %s: Asymmetric live-update operations detected\n",
               dev_name(iommu->dev));
        return -EINVAL;
}

#endif

This prevents a half-baked iommu driver from ever spinning up, completely eliminating the need to check for it inside the live-update session paths. We anyway don't support hot-pluggable liveupdate ops atm.

Thanks,
Praan

#28 lore raw

Re: [PATCH v2 04/16] iommu: Implement device and IOMMU HW preservation

Quoted 77 lines · Pranjal Shrivastava
Quoted 42 lines · Samiullah Khawaja
Quoted 13 lines · Baolu Lu
Quoted 12 lines · Samiullah Khawaja

Add IOMMU ops to preserve/unpreserve a device. These can be implemented by the IOMMU drivers that support preservation of devices that have their IOMMU domains preserved. During device preservation the state of the associated IOMMU is also preserved as dependency.

---
 drivers/iommu/liveupdate.c       | 162 +++++++++++++++++++++++++++++++
 include/linux/iommu-liveupdate.h |  33 +++++++
 include/linux/iommu.h            |  20 ++++
 3 files changed, 215 insertions(+)

[snip]

Quoted 16 lines
Quoted 16 lines
drivers/iommu/liveupdate.c +16 -0
+
+int iommu_preserve_device(struct iommu_domain *domain,
+			  struct device *dev, u64 *preserved_state)
+{
+	struct iommu_flb_obj *flb_obj;
+	struct iommu_device_ser *device_ser;
+	struct dev_iommu *iommu;
+	struct pci_dev *pdev;
+	int ret;
+
+	if (!dev_is_pci(dev))
+		return -EOPNOTSUPP;
+
+	if (!domain->preserved_state)
+		return -EINVAL;
+
Quoted 8 lines

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 390

Quoted 8 lines
drivers/iommu/liveupdate.c +8 -0
+	if (!iommu_group_dma_owner_claimed(dev->iommu_group))
+		return -EINVAL;
+
+	pdev = to_pci_dev(dev);
+	iommu = dev->iommu;
+	if (!iommu->iommu_dev->ops->preserve_device ||
+	    !iommu->iommu_dev->ops->preserve)
+		return -EOPNOTSUPP;

I will check for unpreserve ops here also.

[snip]

Quoted 14 lines
Quoted 9 lines

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 442

Quoted 5 lines
drivers/iommu/liveupdate.c +5 -0
+	pdev = to_pci_dev(dev);
+	iommu = dev->iommu;
+	if (!iommu->iommu_dev->ops->unpreserve_device ||
+	    !iommu->iommu_dev->ops->unpreserve)
+		return;

Is it considered a driver bug if it implements the preserve hooks but not unpreserve ones? This would at least cause a silent memory leak. How about adding a WARN like this?

I think the preservation should not be done if the unpreserve ops are not implemented by the driver. I will add these checks in the _preserve functions and return EOPNOTSUPP if it is not implemented by the driver.

Quoted 1 line
Quoted 5 lines
Quoted 3 lines
   if (WARN_ON_ONCE(!iommu->iommu_dev->ops->unpreserve_device ||
                    !iommu->iommu_dev->ops->unpreserve))
       return;

We can add this warning here also.

I'd rather have the iommu device registration fail instead of solely relying on these WARN_ONs here. We can simply add a snippet inside iommu_device_register():

#ifdef CONFIG_IOMMU_LIVEUPDATE
if ((iommu->ops->preserve && !iommu->ops->unpreserve) ||

   (!iommu->ops->preserve && iommu->ops->unpreserve)) {
       pr_err("IOMMU: %s: Asymmetric live-update operations detected\n",
              dev_name(iommu->dev));
       return -EINVAL;
}
Quoted 5 lines

#endif

This prevents a half-baked iommu driver from ever spinning up, completely eliminating the need to check for it inside the live-update session paths. We anyway don't support hot-pluggable liveupdate ops atm.

Hmm.. I understand your point, But I think we should let the registration go through so the IOMMU can be used. Its just that the liveupdate preservation will not be supported. Failing the registration would be a big hammer that I think is not really needed.

Quoted 3 lines

Thanks,
Praan

Sami

#29 lore raw

Re: [PATCH v2 04/16] iommu: Implement device and IOMMU HW preservation

In reply to Samiullah Khawaja
Quoted 25 lines · Samiullah Khawaja

Add IOMMU ops to preserve/unpreserve a device. These can be implemented by the IOMMU drivers that support preservation of devices that have their IOMMU domains preserved. During device preservation the state of the associated IOMMU is also preserved as dependency.

---
 drivers/iommu/liveupdate.c       | 162 +++++++++++++++++++++++++++++++
 include/linux/iommu-liveupdate.h |  33 +++++++
 include/linux/iommu.h            |  20 ++++
 3 files changed, 215 insertions(+)
drivers/iommu/liveupdate.c +1 -0
index f71f14518248..765d042e22e3 100644
@@ -11,6 +11,7 @@
 #include <linux/liveupdate.h>
 #include <linux/iommu-liveupdate.h>
 #include <linux/iommu.h>
+#include <linux/pci.h>
 #include <linux/errno.h>
 
 #define iommu_max_objs_per_page(_array) \

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 303

Quoted 10 lines
drivers/iommu/liveupdate.c +7 -0
@@ -293,3 +294,164 @@ void iommu_domain_unpreserve(struct iommu_domain *domain)
 	domain->preserved_state = NULL;
 }
 EXPORT_SYMBOL_GPL(iommu_domain_unpreserve);
+
+static struct iommu_hw_ser *alloc_iommu_hw_ser(struct iommu_flb_obj *flb)
+{
+	int idx;
+
+	idx = alloc_object_ser((struct iommu_array_hdr_ser **)&flb->curr_iommu_array,
+			       iommu_max_objs_per_page(flb->curr_iommu_array));

Nit: Same thing about brittle casts here, shall we make them void ** and cast then within alloc_object_set ?

Quoted 34 lines
drivers/iommu/liveupdate.c +34 -0
+	if (idx < 0)
+		return ERR_PTR(idx);
+
+	flb->curr_iommu_array->objects[idx].hdr.ref_count = 1;
+	return &flb->curr_iommu_array->objects[idx];
+}
+
+static int iommu_preserve_locked(struct iommu_device *iommu,
+				 struct iommu_flb_obj *flb_obj)
+{
+	struct iommu_hw_ser *iommu_hw_ser;
+	int ret;
+
+	if (!iommu->ops->preserve)
+		return -EOPNOTSUPP;
+
+	lockdep_assert_held(&flb_obj->lock);
+	if (iommu->outgoing_preserved_state) {
+		iommu->outgoing_preserved_state->hdr.ref_count++;
+		return 0;
+	}
+
+	iommu_hw_ser = alloc_iommu_hw_ser(flb_obj);
+	if (IS_ERR(iommu_hw_ser))
+		return PTR_ERR(iommu_hw_ser);
+
+	ret = iommu->ops->preserve(iommu, iommu_hw_ser);
+	if (ret) {
+		iommu_hw_ser->hdr.deleted = true;
+		return ret;
+	}
+
+	iommu->outgoing_preserved_state = iommu_hw_ser;
+	return ret;

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 347

Quoted 10 lines
drivers/iommu/liveupdate.c +10 -0
+}
+
+static void iommu_unpreserve_locked(struct iommu_device *iommu,
+				    struct iommu_flb_obj *flb_obj)
+{
+	struct iommu_hw_ser *iommu_hw_ser = iommu->outgoing_preserved_state;
+
+	lockdep_assert_held(&flb_obj->lock);
+	iommu_hw_ser->hdr.ref_count--;
+	if (iommu_hw_ser->hdr.ref_count)

Shall we add a defensive if (WARN_ON(!iommu_hw_ser)) ? I'm aware we check this on within iommu_unpreserve_device() but we don't seem to check it before calling iommu_unpreserve_locked() in the error path of iommu_preserve_device.

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 351

Quoted 4 lines
drivers/iommu/liveupdate.c +4 -0
+		return;
+
+	iommu->outgoing_preserved_state = NULL;
+	iommu->ops->unpreserve(iommu, iommu_hw_ser);

We seem to assume we'll always have unpreserve implemented? If so, we should check it during the iommu registration itself and fail it, i.e.

inside iommu_device_register() we could add something like:

#ifdef CONFIG_IOMMU_LIVEUPDATE
if ((iommu->ops->preserve && !iommu->ops->unpreserve) ||

    (!iommu->ops->preserve && iommu->ops->unpreserve)) {
        pr_err("IOMMU: %s: Asymmetric live-update operations detected\n",
               dev_name(iommu->dev));
        return -EINVAL;
}

#endif

This prevents a half-baked iommu driver from ever spinning up, completely eliminating the need to check for it inside the live-update session paths.

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 359

Quoted 8 lines
drivers/iommu/liveupdate.c +8 -0
+	iommu_hw_ser->hdr.deleted = true;
+}
+
+static struct iommu_device_ser *alloc_iommu_device_ser(struct iommu_flb_obj *flb)
+{
+	int idx;
+
+	idx = alloc_object_ser((struct iommu_array_hdr_ser **)&flb->curr_device_array,

Nit: Same thing about brittle casts here, shall we make them void ** and cast then within alloc_object_set ?

Quoted 15 lines
drivers/iommu/liveupdate.c +15 -0
+			       iommu_max_objs_per_page(flb->curr_device_array));
+	if (idx < 0)
+		return ERR_PTR(idx);
+
+	flb->curr_device_array->objects[idx].hdr.ref_count = 1;
+	return &flb->curr_device_array->objects[idx];
+}
+
+int iommu_preserve_device(struct iommu_domain *domain,
+			  struct device *dev, u64 *preserved_state)
+{
+	struct iommu_flb_obj *flb_obj;
+	struct iommu_device_ser *device_ser;
+	struct dev_iommu *iommu;
+	struct pci_dev *pdev;

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 384

Quoted 10 lines
drivers/iommu/liveupdate.c +10 -0
+	int ret;
+
+	if (!dev_is_pci(dev))
+		return -EOPNOTSUPP;
+
+	if (!domain->preserved_state)
+		return -EINVAL;
+
+	if (!iommu_group_dma_owner_claimed(dev->iommu_group))
+		return -EINVAL;

Nice.

Quoted 14 lines
drivers/iommu/liveupdate.c +14 -0
+
+	pdev = to_pci_dev(dev);
+	iommu = dev->iommu;
+	if (!iommu->iommu_dev->ops->preserve_device ||
+	    !iommu->iommu_dev->ops->preserve)
+		return -EOPNOTSUPP;
+
+	ret = liveupdate_flb_get_outgoing(&iommu_flb, (void **)&flb_obj);
+	if (ret)
+		return ret;
+
+	guard(mutex)(&flb_obj->lock);
+	device_ser = alloc_iommu_device_ser(flb_obj);
+	if (IS_ERR(device_ser))

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 408

Quoted 10 lines
drivers/iommu/liveupdate.c +10 -0
+		return PTR_ERR(device_ser);
+
+	ret = iommu_preserve_locked(iommu->iommu_dev, flb_obj);
+	if (ret) {
+		device_ser->hdr.deleted = true;
+		return ret;
+	}
+
+	device_ser->domain_iommu_ser.domain_phys = __pa(domain->preserved_state);
+	device_ser->domain_iommu_ser.iommu_phys = __pa(iommu->iommu_dev->outgoing_preserved_state);

Nit: Should these be updated to use virt_to_phys as well?

Quoted 5 lines
drivers/iommu/liveupdate.c +5 -0
+	device_ser->devid = pci_dev_id(pdev);
+	device_ser->pci_domain_nr = pci_domain_nr(pdev->bus);
+
+	ret = iommu->iommu_dev->ops->preserve_device(dev, device_ser);
+	if (ret) {

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 423

Quoted 10 lines
drivers/iommu/liveupdate.c +10 -0
+		device_ser->hdr.deleted = true;
+		iommu_unpreserve_locked(iommu->iommu_dev, flb_obj);
+		return ret;
+	}
+
+	dev->iommu->device_ser = device_ser;
+	*preserved_state = virt_to_phys(device_ser);
+	return 0;
+}
+

[...]

Thanks,
Praan

#30 lore raw

Re: [PATCH v2 04/16] iommu: Implement device and IOMMU HW preservation

Quoted 24 lines · Pranjal Shrivastava
Quoted 23 lines · Samiullah Khawaja

Add IOMMU ops to preserve/unpreserve a device. These can be implemented by the IOMMU drivers that support preservation of devices that have their IOMMU domains preserved. During device preservation the state of the associated IOMMU is also preserved as dependency.

---
 drivers/iommu/liveupdate.c       | 162 +++++++++++++++++++++++++++++++
 include/linux/iommu-liveupdate.h |  33 +++++++
 include/linux/iommu.h            |  20 ++++
 3 files changed, 215 insertions(+)
drivers/iommu/liveupdate.c +1 -0
index f71f14518248..765d042e22e3 100644
@@ -11,6 +11,7 @@
 #include <linux/liveupdate.h>
 #include <linux/iommu-liveupdate.h>
 #include <linux/iommu.h>
+#include <linux/pci.h>
 #include <linux/errno.h>
 
Quoted 15 lines

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 303

Quoted 12 lines
drivers/iommu/liveupdate.c +7 -0
 #define iommu_max_objs_per_page(_array) \
@@ -293,3 +294,164 @@ void iommu_domain_unpreserve(struct iommu_domain *domain)
 	domain->preserved_state = NULL;
 }
 EXPORT_SYMBOL_GPL(iommu_domain_unpreserve);
+
+static struct iommu_hw_ser *alloc_iommu_hw_ser(struct iommu_flb_obj *flb)
+{
+	int idx;
+
+	idx = alloc_object_ser((struct iommu_array_hdr_ser **)&flb->curr_iommu_array,
+			       iommu_max_objs_per_page(flb->curr_iommu_array));

Nit: Same thing about brittle casts here, shall we make them void ** and cast then within alloc_object_set ?

Agreed. Will change this.

Quoted 43 lines
Quoted 43 lines
drivers/iommu/liveupdate.c +41 -0
+	if (idx < 0)
+		return ERR_PTR(idx);
+
+	flb->curr_iommu_array->objects[idx].hdr.ref_count = 1;
+	return &flb->curr_iommu_array->objects[idx];
+}
+
+static int iommu_preserve_locked(struct iommu_device *iommu,
+				 struct iommu_flb_obj *flb_obj)
+{
+	struct iommu_hw_ser *iommu_hw_ser;
+	int ret;
+
+	if (!iommu->ops->preserve)
+		return -EOPNOTSUPP;
+
+	lockdep_assert_held(&flb_obj->lock);
+	if (iommu->outgoing_preserved_state) {
+		iommu->outgoing_preserved_state->hdr.ref_count++;
+		return 0;
+	}
+
+	iommu_hw_ser = alloc_iommu_hw_ser(flb_obj);
+	if (IS_ERR(iommu_hw_ser))
+		return PTR_ERR(iommu_hw_ser);
+
+	ret = iommu->ops->preserve(iommu, iommu_hw_ser);
+	if (ret) {
+		iommu_hw_ser->hdr.deleted = true;
+		return ret;
+	}
+
+	iommu->outgoing_preserved_state = iommu_hw_ser;
+	return ret;
+}
+
+static void iommu_unpreserve_locked(struct iommu_device *iommu,
+				    struct iommu_flb_obj *flb_obj)
+{
+	struct iommu_hw_ser *iommu_hw_ser = iommu->outgoing_preserved_state;
+
Quoted 8 lines

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 347

Quoted 3 lines
drivers/iommu/liveupdate.c +3 -0
+	lockdep_assert_held(&flb_obj->lock);
+	iommu_hw_ser->hdr.ref_count--;
+	if (iommu_hw_ser->hdr.ref_count)

Shall we add a defensive if (WARN_ON(!iommu_hw_ser)) ? I'm aware we check this on within iommu_unpreserve_device() but we don't seem to check it before calling iommu_unpreserve_locked() in the error path of iommu_preserve_device.

Agreed. I will add this.

Quoted 18 lines

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 351

Quoted 5 lines
drivers/iommu/liveupdate.c +4 -0
+		return;
+
+	iommu->outgoing_preserved_state = NULL;
+	iommu->ops->unpreserve(iommu, iommu_hw_ser);

We seem to assume we'll always have unpreserve implemented? If so, we should check it during the iommu registration itself and fail it, i.e.

inside iommu_device_register() we could add something like:

#ifdef CONFIG_IOMMU_LIVEUPDATE
if ((iommu->ops->preserve && !iommu->ops->unpreserve) ||

   (!iommu->ops->preserve && iommu->ops->unpreserve)) {
       pr_err("IOMMU: %s: Asymmetric live-update operations detected\n",
              dev_name(iommu->dev));
       return -EINVAL;
}
Quoted 4 lines

#endif

This prevents a half-baked iommu driver from ever spinning up, completely eliminating the need to check for it inside the live-update session paths.

Replied to this in the other thread where you suggested this inline with Baolu's comment.

Quoted 12 lines

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 359

Quoted 9 lines
drivers/iommu/liveupdate.c +8 -0
+	iommu_hw_ser->hdr.deleted = true;
+}
+
+static struct iommu_device_ser *alloc_iommu_device_ser(struct iommu_flb_obj *flb)
+{
+	int idx;
+
+	idx = alloc_object_ser((struct iommu_array_hdr_ser **)&flb->curr_device_array,

Nit: Same thing about brittle casts here, shall we make them void ** and cast then within alloc_object_set ?

Will update this in next revision.

Quoted 9 lines
Quoted 9 lines
drivers/iommu/liveupdate.c +8 -0
+			       iommu_max_objs_per_page(flb->curr_device_array));
+	if (idx < 0)
+		return ERR_PTR(idx);
+
+	flb->curr_device_array->objects[idx].hdr.ref_count = 1;
+	return &flb->curr_device_array->objects[idx];
+}
+

[snip]

Quoted 11 lines

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 408

Quoted 9 lines
drivers/iommu/liveupdate.c +9 -0
+
+	ret = iommu_preserve_locked(iommu->iommu_dev, flb_obj);
+	if (ret) {
+		device_ser->hdr.deleted = true;
+		return ret;
+	}
+
+	device_ser->domain_iommu_ser.domain_phys = __pa(domain->preserved_state);
+	device_ser->domain_iommu_ser.iommu_phys = __pa(iommu->iommu_dev->outgoing_preserved_state);

Nit: Should these be updated to use virt_to_phys as well?

Will update these.

Quoted 19 lines
Quoted 16 lines
drivers/iommu/liveupdate.c +15 -0
+	device_ser->devid = pci_dev_id(pdev);
+	device_ser->pci_domain_nr = pci_domain_nr(pdev->bus);
+
+	ret = iommu->iommu_dev->ops->preserve_device(dev, device_ser);
+	if (ret) {
+		device_ser->hdr.deleted = true;
+		iommu_unpreserve_locked(iommu->iommu_dev, flb_obj);
+		return ret;
+	}
+
+	dev->iommu->device_ser = device_ser;
+	*preserved_state = virt_to_phys(device_ser);
+	return 0;
+}
+

[...]

Quoted 3 lines

Thanks,
Praan

Sami

#31 lore raw

Re: [PATCH v2 04/16] iommu: Implement device and IOMMU HW preservation

In reply to Samiullah Khawaja
Quoted 7 lines · Samiullah Khawaja

Add IOMMU ops to preserve/unpreserve a device. These can be implemented by the IOMMU drivers that support preservation of devices that have their IOMMU domains preserved. During device preservation the state of the associated IOMMU is also preserved as dependency.

---

[..]

Quoted 5 lines
drivers/iommu/liveupdate.c +5 -0
+void iommu_unpreserve_device(struct iommu_domain *domain, struct device *dev)
+{
+	struct iommu_flb_obj *flb_obj;
+	struct iommu_device_ser *iommu_device_ser;
+	struct dev_iommu *iommu;

drivers/iommu/liveupdate.c void iommu_domain_unpreserve(struct iommu_domain *domain) · line 438

Quoted 10 lines
drivers/iommu/liveupdate.c +10 -0
+	struct pci_dev *pdev;
+	int ret;
+
+	if (!dev_is_pci(dev))
+		return;
+
+	if (!iommu_group_dma_owner_claimed(dev->iommu_group))
+		return;
+
+	pdev = to_pci_dev(dev);

nit

pdev is declared and assigned, but never used here.

Quoted 140 lines
drivers/iommu/liveupdate.c +19 -0
+	iommu = dev->iommu;
+	if (!iommu->iommu_dev->ops->unpreserve_device ||
+	    !iommu->iommu_dev->ops->unpreserve)
+		return;
+
+	ret = liveupdate_flb_get_outgoing(&iommu_flb, (void **)&flb_obj);
+	if (WARN_ON(ret))
+		return;
+
+	guard(mutex)(&flb_obj->lock);
+	iommu_device_ser = dev_iommu_preserved_state(dev);
+	if (WARN_ON(!iommu_device_ser))
+		return;
+
+	iommu->iommu_dev->ops->unpreserve_device(dev, iommu_device_ser);
+	dev->iommu->device_ser = NULL;
+
+	iommu_unpreserve_locked(iommu->iommu_dev, flb_obj);
+}
include/linux/iommu-liveupdate.h +33 -0
index 6019cfc27428..279c7ab04f09 100644
@@ -8,14 +8,37 @@
 #ifndef _LINUX_IOMMU_LIVEUPDATE_H
 #define _LINUX_IOMMU_LIVEUPDATE_H
 
+#include <linux/device.h>
 #include <linux/iommu.h>
 #include <linux/liveupdate.h>
 #include <linux/kho/abi/iommu.h>
 
 #ifdef CONFIG_IOMMU_LIVEUPDATE
+static inline void *dev_iommu_preserved_state(struct device *dev)
+{
+	struct iommu_device_ser *ser;
+
+	if (!dev->iommu)
+		return NULL;
+
+	ser = dev->iommu->device_ser;
+	if (ser && !ser->hdr.incoming)
+		return ser;
+
+	return NULL;
+}
+
 int iommu_domain_preserve(struct iommu_domain *domain, struct iommu_domain_ser **ser);
 void iommu_domain_unpreserve(struct iommu_domain *domain);
+int iommu_preserve_device(struct iommu_domain *domain,
+			  struct device *dev, u64 *preserved_state);
+void iommu_unpreserve_device(struct iommu_domain *domain, struct device *dev);
 #else
+static inline void *dev_iommu_preserved_state(struct device *dev)
+{
+	return NULL;
+}
+
 static inline int iommu_domain_preserve(struct iommu_domain *domain, struct iommu_domain_ser **ser)
 {
 	return -EOPNOTSUPP;
@@ -24,6 +47,16 @@ static inline int iommu_domain_preserve(struct iommu_domain *domain, struct iomm
 static inline void iommu_domain_unpreserve(struct iommu_domain *domain)
 {
 }
+
+static inline int iommu_preserve_device(struct iommu_domain *domain,
+					struct device *dev, u64 *preserved_state)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void iommu_unpreserve_device(struct iommu_domain *domain, struct device *dev)
+{
+}
 #endif
 
 int iommu_liveupdate_register_flb(struct liveupdate_file_handler *handler);
include/linux/iommu.h +20 -0
index 3853a3946733..1c424b32c5fc 100644
@@ -655,6 +655,10 @@ __iommu_copy_struct_to_user(const struct iommu_user_data *dst_data,
  *               resources shared/passed to user space IOMMU instance. Associate
  *               it with a nesting @parent_domain. It is required for driver to
  *               set @viommu->ops pointing to its own viommu_ops
+ * @preserve_device: Preserve state of a device for liveupdate.
+ * @unpreserve_device: Unpreserve state that was preserved earlier.
+ * @preserve: Preserve state of iommu translation hardware for liveupdate.
+ * @unpreserve: Unpreserve state of iommu that was preserved earlier.
  * @owner: Driver module providing these ops
  * @identity_domain: An always available, always attachable identity
  *                   translation.
@@ -711,6 +715,13 @@ struct iommu_ops {
 			   struct iommu_domain *parent_domain,
 			   const struct iommu_user_data *user_data);
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	int (*preserve_device)(struct device *dev, struct iommu_device_ser *device_ser);
+	void (*unpreserve_device)(struct device *dev, struct iommu_device_ser *device_ser);
+	int (*preserve)(struct iommu_device *iommu, struct iommu_hw_ser *iommu_ser);
+	void (*unpreserve)(struct iommu_device *iommu, struct iommu_hw_ser *iommu_ser);
+#endif
+
 	const struct iommu_domain_ops *default_domain_ops;
 	struct module *owner;
 	struct iommu_domain *identity_domain;
@@ -806,6 +817,8 @@ struct iommu_domain_ops {
  * @singleton_group: Used internally for drivers that have only one group
  * @max_pasids: number of supported PASIDs
  * @ready: set once iommu_device_register() has completed successfully
+ * @outgoing_preserved_state: preserved iommu state of outgoing kernel for
+ * liveupdate.
  */
 struct iommu_device {
 	struct list_head list;
@@ -815,6 +828,10 @@ struct iommu_device {
 	struct iommu_group *singleton_group;
 	u32 max_pasids;
 	bool ready;
+
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	struct iommu_hw_ser *outgoing_preserved_state;
+#endif
 };
 
 /**
@@ -869,6 +886,9 @@ struct dev_iommu {
 	u32				pci_32bit_workaround:1;
 	u32				require_direct:1;
 	u32				shadow_on_flush:1;
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	struct iommu_device_ser		*device_ser;
+#endif
 };
 
 int iommu_device_register(struct iommu_device *iommu,
Signature · 1 line
2.54.0.545.g6539524ca2-goog
#32 lore raw

PATCH v2 5/16 iommu/pages: Add APIs to preserve/unpreserve/restore iommu pages

In reply to Samiullah Khawaja

IOMMU pages are allocated/freed using APIs using struct ioptdesc. For the proper preservation and restoration of ioptdesc add helper functions.

---
 drivers/iommu/iommu-pages.c | 108 ++++++++++++++++++++++++++++++++++--
 drivers/iommu/iommu-pages.h |  30 ++++++++++
 2 files changed, 134 insertions(+), 4 deletions(-)
drivers/iommu/iommu-pages.c +104 -4
index 3bab175d8557..b1ffeb930e6d 100644
@@ -6,6 +6,7 @@
 #include "iommu-pages.h"
 #include <linux/dma-mapping.h>
 #include <linux/gfp.h>
+#include <linux/kexec_handover.h>
 #include <linux/mm.h>
 
 #define IOPTDESC_MATCH(pg_elm, elm)                    \
@@ -28,6 +29,13 @@ static inline size_t ioptdesc_mem_size(struct ioptdesc *desc)
 	return 1UL << (folio_order(ioptdesc_folio(desc)) + PAGE_SHIFT);
 }
 
+static inline void iommu_folio_update_stats(struct folio *folio,
+					    unsigned long nr_pages)
+{
+	mod_node_page_state(folio_pgdat(folio), NR_IOMMU_PAGES, nr_pages);
+	lruvec_stat_mod_folio(folio, NR_SECONDARY_PAGETABLE, nr_pages);
+}
+
 /**
  * iommu_alloc_pages_node_sz - Allocate a zeroed page of a given size from
  *                             specific NUMA node
@@ -80,8 +88,7 @@ void *iommu_alloc_pages_node_sz(int nid, gfp_t gfp, size_t size)
 	 * rather large, i.e. multiple gigabytes in size.
 	 */
 	pgcnt = 1UL << order;
-	mod_node_page_state(folio_pgdat(folio), NR_IOMMU_PAGES, pgcnt);
-	lruvec_stat_mod_folio(folio, NR_SECONDARY_PAGETABLE, pgcnt);
+	iommu_folio_update_stats(folio, pgcnt);
 
 	return folio_address(folio);
 }
@@ -95,8 +102,7 @@ static void __iommu_free_desc(struct ioptdesc *iopt)
 	if (IOMMU_PAGES_USE_DMA_API)
 		WARN_ON_ONCE(iopt->incoherent);
 
-	mod_node_page_state(folio_pgdat(folio), NR_IOMMU_PAGES, -pgcnt);
-	lruvec_stat_mod_folio(folio, NR_SECONDARY_PAGETABLE, -pgcnt);
+	iommu_folio_update_stats(folio, -pgcnt);
 	folio_put(folio);
 }
 
@@ -131,6 +137,100 @@ void iommu_put_pages_list(struct iommu_pages_list *list)
 }
 EXPORT_SYMBOL_GPL(iommu_put_pages_list);
 
+#if IS_ENABLED(CONFIG_IOMMU_LIVEUPDATE)
+/**
+ * iommu_unpreserve_page - Unpreserve a page that was preserved in KHO
+ * @virt: Virtual address of a page
+ */
+void iommu_unpreserve_page(void *virt)
+{
+	kho_unpreserve_folio(ioptdesc_folio(virt_to_ioptdesc(virt)));
+}
+EXPORT_SYMBOL_GPL(iommu_unpreserve_page);
+
+/**
+ * iommu_preserve_page - Preserve a page during kexec handover
+ * @virt: Virtual address of the page to preserve
+ *
+ * Returns 0 on success, negative error on failure
+ */
+int iommu_preserve_page(void *virt)
+{
+	return kho_preserve_folio(ioptdesc_folio(virt_to_ioptdesc(virt)));
+}
+EXPORT_SYMBOL_GPL(iommu_preserve_page);
+
+/**
+ * iommu_unpreserve_pages - Unpreserve pages that were preserved in KHO
+ * @list: List of pages to unpreserve
+ */
+void iommu_unpreserve_pages(struct iommu_pages_list *list)
+{
+	struct ioptdesc *iopt;
+
+	list_for_each_entry(iopt, &list->pages, iopt_freelist_elm)
+		kho_unpreserve_folio(ioptdesc_folio(iopt));
+}
+EXPORT_SYMBOL_GPL(iommu_unpreserve_pages);
+
+/**
+ * iommu_restore_page - Restore a page that was preserved in KHO
+ * @phys: Physical address of a page
+ */
+void iommu_restore_page(u64 phys)
+{
+	struct ioptdesc *iopt;
+	struct folio *folio;
+	unsigned long pgcnt;
+	unsigned int order;
+
+	folio = kho_restore_folio(phys);
+	BUG_ON(!folio);
+
+	iopt = folio_ioptdesc(folio);
+
+	/*
+	 * For the restored pages incoherent is set to false as these are not
+	 * mapped using the DMA_API. The remapping of these pages using DMA_API
+	 * is not needed as these are not going to be written to by the new
+	 * kernel.
+	 */
+	iopt->incoherent = false;
+
+	order = folio_order(folio);
+	pgcnt = 1UL << order;
+	iommu_folio_update_stats(folio, pgcnt);
+}
+EXPORT_SYMBOL_GPL(iommu_restore_page);
+
+/**
+ * iommu_preserve_pages - Preserve pages during kexec handover
+ * @list: List of pages to preserve
+ *
+ * Returns 0 on success, negative error on failure
+ */
+int iommu_preserve_pages(struct iommu_pages_list *list)
+{
+	struct ioptdesc *iopt;
+	int ret;
+
+	list_for_each_entry(iopt, &list->pages, iopt_freelist_elm) {
+		ret = kho_preserve_folio(ioptdesc_folio(iopt));
+		if (ret)
+			goto err;
+	}
+
+	return 0;
+
+err:
+	list_for_each_entry_continue_reverse(iopt, &list->pages, iopt_freelist_elm)
+		kho_unpreserve_folio(ioptdesc_folio(iopt));
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(iommu_preserve_pages);
+#endif
+
 /**
  * iommu_pages_start_incoherent - Setup the page for cache incoherent operation
  * @virt: The page to setup
drivers/iommu/iommu-pages.h +30 -0
index ae9da4f571f6..7b9b6bb504b2 100644
@@ -53,6 +53,36 @@ void *iommu_alloc_pages_node_sz(int nid, gfp_t gfp, size_t size);
 void iommu_free_pages(void *virt);
 void iommu_put_pages_list(struct iommu_pages_list *list);
 
+#if IS_ENABLED(CONFIG_IOMMU_LIVEUPDATE)
+int iommu_preserve_page(void *virt);
+void iommu_unpreserve_page(void *virt);
+int iommu_preserve_pages(struct iommu_pages_list *list);
+void iommu_unpreserve_pages(struct iommu_pages_list *list);
+void iommu_restore_page(u64 phys);
+#else
+static inline int iommu_preserve_page(void *virt)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void iommu_unpreserve_page(void *virt)
+{
+}
+
+static inline int iommu_preserve_pages(struct iommu_pages_list *list)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void iommu_unpreserve_pages(struct iommu_pages_list *list, int count)
+{
+}
+
+static inline void iommu_restore_page(u64 phys)
+{
+}
+#endif
+
 /**
  * iommu_pages_list_add - add the page to a iommu_pages_list
  * @list: List to add the page to
Signature · 1 line
2.54.0.545.g6539524ca2-goog
Reviewed-by on v2 5/16
#33 lore raw

Re: [PATCH v2 05/16] iommu/pages: Add APIs to preserve/unpreserve/restore iommu pages

Quoted 21 lines · Samiullah Khawaja

IOMMU pages are allocated/freed using APIs using struct ioptdesc. For the proper preservation and restoration of ioptdesc add helper functions.

---
 drivers/iommu/iommu-pages.c | 108 ++++++++++++++++++++++++++++++++++--
 drivers/iommu/iommu-pages.h |  30 ++++++++++
 2 files changed, 134 insertions(+), 4 deletions(-)
drivers/iommu/iommu-pages.c +1 -0
index 3bab175d8557..b1ffeb930e6d 100644
@@ -6,6 +6,7 @@
 #include "iommu-pages.h"
 #include <linux/dma-mapping.h>
 #include <linux/gfp.h>
+#include <linux/kexec_handover.h>
 #include <linux/mm.h>
 

[snip]

Quoted 26 lines
drivers/iommu/iommu-pages.c +26 -0
+
+/**
+ * iommu_unpreserve_pages - Unpreserve pages that were preserved in KHO
+ * @list: List of pages to unpreserve
+ */
+void iommu_unpreserve_pages(struct iommu_pages_list *list)
+{
+	struct ioptdesc *iopt;
+
+	list_for_each_entry(iopt, &list->pages, iopt_freelist_elm)
+		kho_unpreserve_folio(ioptdesc_folio(iopt));
+}
+EXPORT_SYMBOL_GPL(iommu_unpreserve_pages);
+
+/**
+ * iommu_restore_page - Restore a page that was preserved in KHO
+ * @phys: Physical address of a page
+ */
+void iommu_restore_page(u64 phys)
+{
+	struct ioptdesc *iopt;
+	struct folio *folio;
+	unsigned long pgcnt;
+	unsigned int order;
+
+	folio = kho_restore_folio(phys);

drivers/iommu/iommu-pages.c void iommu_put_pages_list(struct iommu_pages_list *list) · line 197

Quoted 10 lines
drivers/iommu/iommu-pages.c +10 -0
+	BUG_ON(!folio);
+
+	iopt = folio_ioptdesc(folio);
+
+	/*
+	 * For the restored pages incoherent is set to false as these are not
+	 * mapped using the DMA_API. The remapping of these pages using DMA_API
+	 * is not needed as these are not going to be written to by the new
+	 * kernel.
+	 */

Looks good.

Quoted 21 lines
drivers/iommu/iommu-pages.c +21 -0
+	iopt->incoherent = false;
+
+	order = folio_order(folio);
+	pgcnt = 1UL << order;
+	iommu_folio_update_stats(folio, pgcnt);
+}
+EXPORT_SYMBOL_GPL(iommu_restore_page);
+
+/**
+ * iommu_preserve_pages - Preserve pages during kexec handover
+ * @list: List of pages to preserve
+ *
+ * Returns 0 on success, negative error on failure
+ */
+int iommu_preserve_pages(struct iommu_pages_list *list)
+{
+	struct ioptdesc *iopt;
+	int ret;
+
+	list_for_each_entry(iopt, &list->pages, iopt_freelist_elm) {
+		ret = kho_preserve_folio(ioptdesc_folio(iopt));

drivers/iommu/iommu-pages.c void iommu_put_pages_list(struct iommu_pages_list *list) · line 228

Quoted 10 lines
drivers/iommu/iommu-pages.c +10 -0
+		if (ret)
+			goto err;
+	}
+
+	return 0;
+
+err:
+	list_for_each_entry_continue_reverse(iopt, &list->pages, iopt_freelist_elm)
+		kho_unpreserve_folio(ioptdesc_folio(iopt));
+

Looks good.

Quoted 5 lines
drivers/iommu/iommu-pages.c +5 -0
+	return ret;
+}
+EXPORT_SYMBOL_GPL(iommu_preserve_pages);
+#endif
+

[snip]

drivers/iommu/iommu-pages.c

Quoted 6 lines
drivers/iommu/iommu-pages.c +6 -0
+static inline int iommu_preserve_pages(struct iommu_pages_list *list)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void iommu_unpreserve_pages(struct iommu_pages_list *list, int count)

Nit: This is inconsistent from the signature in it's counter part in IS_ENABLED(CONFIG_IOMMU_LIVEUPDATE)

Let's remove int count from the arguments to avoid broken builds.

drivers/iommu/iommu-pages.c

Quoted 8 lines
drivers/iommu/iommu-pages.c +8 -0
+{
+}
+
+static inline void iommu_restore_page(u64 phys)
+{
+}
+#endif
+

With that stub argument fixed,

Thanks,
Praan

#34 lore raw

Re: [PATCH v2 05/16] iommu/pages: Add APIs to preserve/unpreserve/restore iommu pages

Quoted 106 lines · Pranjal Shrivastava
Quoted 21 lines · Samiullah Khawaja

IOMMU pages are allocated/freed using APIs using struct ioptdesc. For the proper preservation and restoration of ioptdesc add helper functions.

---
 drivers/iommu/iommu-pages.c | 108 ++++++++++++++++++++++++++++++++++--
 drivers/iommu/iommu-pages.h |  30 ++++++++++
 2 files changed, 134 insertions(+), 4 deletions(-)
drivers/iommu/iommu-pages.c +1 -0
index 3bab175d8557..b1ffeb930e6d 100644
@@ -6,6 +6,7 @@
 #include "iommu-pages.h"
 #include <linux/dma-mapping.h>
 #include <linux/gfp.h>
+#include <linux/kexec_handover.h>
 #include <linux/mm.h>
 

[snip]

Quoted 26 lines
drivers/iommu/iommu-pages.c +26 -0
+
+/**
+ * iommu_unpreserve_pages - Unpreserve pages that were preserved in KHO
+ * @list: List of pages to unpreserve
+ */
+void iommu_unpreserve_pages(struct iommu_pages_list *list)
+{
+	struct ioptdesc *iopt;
+
+	list_for_each_entry(iopt, &list->pages, iopt_freelist_elm)
+		kho_unpreserve_folio(ioptdesc_folio(iopt));
+}
+EXPORT_SYMBOL_GPL(iommu_unpreserve_pages);
+
+/**
+ * iommu_restore_page - Restore a page that was preserved in KHO
+ * @phys: Physical address of a page
+ */
+void iommu_restore_page(u64 phys)
+{
+	struct ioptdesc *iopt;
+	struct folio *folio;
+	unsigned long pgcnt;
+	unsigned int order;
+
+	folio = kho_restore_folio(phys);

drivers/iommu/iommu-pages.c void iommu_put_pages_list(struct iommu_pages_list *list) · line 197

Quoted 10 lines
drivers/iommu/iommu-pages.c +10 -0
+	BUG_ON(!folio);
+
+	iopt = folio_ioptdesc(folio);
+
+	/*
+	 * For the restored pages incoherent is set to false as these are not
+	 * mapped using the DMA_API. The remapping of these pages using DMA_API
+	 * is not needed as these are not going to be written to by the new
+	 * kernel.
+	 */

Looks good.

Quoted 21 lines
drivers/iommu/iommu-pages.c +21 -0
+	iopt->incoherent = false;
+
+	order = folio_order(folio);
+	pgcnt = 1UL << order;
+	iommu_folio_update_stats(folio, pgcnt);
+}
+EXPORT_SYMBOL_GPL(iommu_restore_page);
+
+/**
+ * iommu_preserve_pages - Preserve pages during kexec handover
+ * @list: List of pages to preserve
+ *
+ * Returns 0 on success, negative error on failure
+ */
+int iommu_preserve_pages(struct iommu_pages_list *list)
+{
+	struct ioptdesc *iopt;
+	int ret;
+
+	list_for_each_entry(iopt, &list->pages, iopt_freelist_elm) {
+		ret = kho_preserve_folio(ioptdesc_folio(iopt));

drivers/iommu/iommu-pages.c void iommu_put_pages_list(struct iommu_pages_list *list) · line 228

Quoted 10 lines
drivers/iommu/iommu-pages.c +10 -0
+		if (ret)
+			goto err;
+	}
+
+	return 0;
+
+err:
+	list_for_each_entry_continue_reverse(iopt, &list->pages, iopt_freelist_elm)
+		kho_unpreserve_folio(ioptdesc_folio(iopt));
+

Looks good.

Quoted 5 lines
drivers/iommu/iommu-pages.c +5 -0
+	return ret;
+}
+EXPORT_SYMBOL_GPL(iommu_preserve_pages);
+#endif
+

[snip]

Quoted 9 lines

drivers/iommu/iommu-pages.c

Quoted 6 lines
drivers/iommu/iommu-pages.c +6 -0
+static inline int iommu_preserve_pages(struct iommu_pages_list *list)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void iommu_unpreserve_pages(struct iommu_pages_list *list, int count)

Nit: This is inconsistent from the signature in it's counter part in IS_ENABLED(CONFIG_IOMMU_LIVEUPDATE)

Oh. I will update this in next revision

Quoted 13 lines

Let's remove int count from the arguments to avoid broken builds.

drivers/iommu/iommu-pages.c

Quoted 8 lines
drivers/iommu/iommu-pages.c +8 -0
+{
+}
+
+static inline void iommu_restore_page(u64 phys)
+{
+}
+#endif
+

With that stub argument fixed,

Quoted 1 line

Thanks Pranjal.

Quoted 3 lines

Thanks,
Praan

#35 lore raw

PATCH v2 6/16 iommupt: Implement preserve/unpreserve/restore callbacks

In reply to Samiullah Khawaja

Implement the iommu domain ops for presevation, unpresevation and restoration of iommu domains for liveupdate. Use the existing page walker to preserve the ioptdesc of the top_table and the lower tables.

Preserve top_level, VASZ and FEAT Sign Extended to restore the domain in the next kernel. On restore the domain has only the preserved features enabled and all the other features are zeroed. This is ok since the restored domain is made immutable and can only be freed. A kunit test is added to verify that the IOMMU domain free can be done with trimmed features.

---
 drivers/iommu/generic_pt/iommu_pt.h       | 131 ++++++++++++++++++++++
 drivers/iommu/generic_pt/kunit_iommu_pt.h |  28 +++++
 include/linux/generic_pt/iommu.h          |  19 +++-
 3 files changed, 177 insertions(+), 1 deletion(-)
drivers/iommu/generic_pt/iommu_pt.h +131 -0
index 19b6daf88f2a..7bca827e3a55 100644
@@ -961,6 +961,133 @@ static int NS(map_range)(struct pt_iommu *iommu_table, dma_addr_t iova,
 	return ret;
 }
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+/**
+ * unpreserve() - Unpreserve page tables and other state of a domain.
+ * @domain: Domain to unpreserve
+ */
+void DOMAIN_NS(unpreserve)(struct iommu_domain *domain, struct iommu_domain_ser *ser)
+{
+	struct pt_iommu *iommu_table =
+		container_of(domain, struct pt_iommu, domain);
+	struct pt_common *common = common_from_iommu(iommu_table);
+	struct pt_range range = pt_all_range(common);
+	struct pt_iommu_collect_args collect = {
+		.free_list = IOMMU_PAGES_LIST_INIT(collect.free_list),
+	};
+
+	iommu_pages_list_add(&collect.free_list, range.top_table);
+	pt_walk_range(&range, __collect_tables, &collect);
+
+	iommu_unpreserve_pages(&collect.free_list);
+}
+EXPORT_SYMBOL_NS_GPL(DOMAIN_NS(unpreserve), "GENERIC_PT_IOMMU");
+
+/**
+ * preserve() - Preserve page tables and other state of a domain.
+ * @domain: Domain to preserve
+ *
+ * Returns: -ERRNO on failure, 0 on success.
+ */
+int DOMAIN_NS(preserve)(struct iommu_domain *domain, struct iommu_domain_ser *ser)
+{
+	struct pt_iommu *iommu_table =
+		container_of(domain, struct pt_iommu, domain);
+	struct pt_common *common = common_from_iommu(iommu_table);
+	struct pt_range range = pt_all_range(common);
+	struct pt_iommu_collect_args collect = {
+		.free_list = IOMMU_PAGES_LIST_INIT(collect.free_list),
+	};
+	int ret;
+
+	iommu_pages_list_add(&collect.free_list, range.top_table);
+	pt_walk_range(&range, __collect_tables, &collect);
+
+	ret = iommu_preserve_pages(&collect.free_list);
+	if (ret)
+		return ret;
+
+	ser->top_table_phys = virt_to_phys(range.top_table);
+	ser->top_level = range.top_level;
+
+	/*
+	 * VASZ and SIGN_EXTEND will be needed in next kernel for collector page
+	 * table walk to restore and free pages.
+	 */
+	ser->vasz = common->max_vasz_lg2;
+	ser->sign_extend = pt_feature(common, PT_FEAT_SIGN_EXTEND);
+
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(DOMAIN_NS(preserve), "GENERIC_PT_IOMMU");
+
+static int __restore_tables(struct pt_range *range, void *arg,
+			    unsigned int level, struct pt_table_p *table)
+{
+	struct pt_state pts = pt_init(range, level, table);
+	int ret;
+
+	for_each_pt_level_entry(&pts) {
+		if (pts.type == PT_ENTRY_TABLE) {
+			iommu_restore_page(virt_to_phys(pts.table_lower));
+
+			/*
+			 * pt_descend can only fail if pts.table_lower is not
+			 * init. So the if statement below is dead code.
+			 */
+			ret = pt_descend(&pts, arg, __restore_tables);
+			if (ret)
+				return ret;
+		}
+	}
+
+	return 0;
+}
+
+static const struct pt_iommu_ops NS(ops_immutable);
+
+/**
+ * restore() - Restore page tables and other state of a domain.
+ * @domain: Domain to preserve
+ *
+ * Returns: -ERRNO on failure, 0 on success.
+ */
+int DOMAIN_NS(restore)(struct iommu_domain *domain, struct iommu_domain_ser *ser)
+{
+	struct pt_iommu *iommu_table =
+		container_of(domain, struct pt_iommu, domain);
+	struct pt_common *common = common_from_iommu(iommu_table);
+	struct pt_range range;
+
+	common->max_vasz_lg2 = ser->vasz;
+
+	/* Make this domain immutable.*/
+	iommu_table->ops = &NS(ops_immutable);
+
+	/*
+	 * It is safe to override this here since this domain is immutable and
+	 * can only be freed.
+	 */
+	common->features = 0;
+	if (ser->sign_extend)
+		common->features |= BIT(PT_FEAT_SIGN_EXTEND);
+
+	range = pt_all_range(common);
+	iommu_restore_page(ser->top_table_phys);
+
+	/* Free new table */
+	iommu_free_pages(range.top_table);
+
+	/* Set the restored top table */
+	pt_top_set(common, phys_to_virt(ser->top_table_phys), ser->top_level);
+
+	/* Restore all pages*/
+	range = pt_all_range(common);
+	return pt_walk_range(&range, __restore_tables, NULL);
+}
+EXPORT_SYMBOL_NS_GPL(DOMAIN_NS(restore), "GENERIC_PT_IOMMU");
+#endif
+
 struct pt_unmap_args {
 	struct iommu_pages_list free_list;
 	pt_vaddr_t unmapped;
@@ -1138,6 +1265,10 @@ static const struct pt_iommu_ops NS(ops) = {
 	.deinit = NS(deinit),
 };
 
+static const struct pt_iommu_ops NS(ops_immutable) = {
+	.deinit = NS(deinit),
+};
+
 static int pt_init_common(struct pt_common *common)
 {
 	struct pt_range top_range = pt_top_range(common);
drivers/iommu/generic_pt/kunit_iommu_pt.h +28 -0
index e8a63c8ea850..af1918d693ed 100644
@@ -426,6 +426,33 @@ static void test_mixed(struct kunit *test)
 	check_iova(test, start, oa, len);
 }
 
+static void test_restore_free(struct kunit *test)
+{
+	struct kunit_iommu_priv *priv = test->priv;
+	struct pt_range top_range = pt_top_range(priv->common);
+	u64 start = 0x3fe400ULL << 12;
+	u64 end = 0x4c0600ULL << 12;
+	pt_vaddr_t len = end - start;
+
+	if (top_range.last_va <= start || sizeof(unsigned long) == 4)
+		kunit_skip(test, "range is too small");
+	if ((priv->safe_pgsize_bitmap & GENMASK(30, 21)) != (BIT(30) | BIT(21)))
+		kunit_skip(test, "incompatible psize");
+
+	/* Map a large mixed range to populate multiple levels of page tables */
+	do_map(test, start, start, len);
+
+	/*
+	 * Simulate a restored state by clearing all features except
+	 * SIGN_EXTEND. This verifies that the generic page table free walker
+	 * can correctly tear down a populated domain when other features are
+	 * zeroed.
+	 */
+	priv->common->features &= BIT(PT_FEAT_SIGN_EXTEND);
+
+	/* The domain will be freed when the test exits. */
+}
+
 static struct kunit_case iommu_test_cases[] = {
 	KUNIT_CASE_FMT(test_increase_level),
 	KUNIT_CASE_FMT(test_map_simple),
@@ -434,6 +461,7 @@ static struct kunit_case iommu_test_cases[] = {
 	KUNIT_CASE_FMT(test_random_map),
 	KUNIT_CASE_FMT(test_pgsize_boundary),
 	KUNIT_CASE_FMT(test_mixed),
+	KUNIT_CASE_FMT(test_restore_free),
 	{},
 };
 
include/linux/generic_pt/iommu.h +18 -1
index dd0edd02a48a..649b3b9eb1a0 100644
@@ -13,6 +13,7 @@ struct iommu_iotlb_gather;
 struct pt_iommu_ops;
 struct pt_iommu_driver_ops;
 struct iommu_dirty_bitmap;
+struct iommu_domain_ser;
 
 /**
  * DOC: IOMMU Radix Page Table
@@ -251,6 +252,12 @@ struct pt_iommu_cfg {
 #define IOMMU_PROTOTYPES(fmt)                                                  \
 	phys_addr_t pt_iommu_##fmt##_iova_to_phys(struct iommu_domain *domain, \
 						  dma_addr_t iova);            \
+	int pt_iommu_##fmt##_preserve(struct iommu_domain *domain,             \
+				      struct iommu_domain_ser *ser);           \
+	void pt_iommu_##fmt##_unpreserve(struct iommu_domain *domain,          \
+					 struct iommu_domain_ser *ser);        \
+	int pt_iommu_##fmt##_restore(struct iommu_domain *domain,              \
+				     struct iommu_domain_ser *ser);            \
 	int pt_iommu_##fmt##_read_and_clear_dirty(                             \
 		struct iommu_domain *domain, unsigned long iova, size_t size,  \
 		unsigned long flags, struct iommu_dirty_bitmap *dirty);        \
@@ -266,12 +273,22 @@ struct pt_iommu_cfg {
 	};                              \
 	IOMMU_PROTOTYPES(fmt)
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+#define IOMMU_PT_LIVEUPDATE_OPS(fmt)			\
+	, .preserve = &pt_iommu_##fmt##_preserve,	\
+	.unpreserve = &pt_iommu_##fmt##_unpreserve,	\
+	.restore = &pt_iommu_##fmt##_restore
+#else
+#define IOMMU_PT_LIVEUPDATE_OPS(fmt)
+#endif
+
 /*
  * A driver uses IOMMU_PT_DOMAIN_OPS to populate the iommu_domain_ops for the
  * iommu_pt
  */
 #define IOMMU_PT_DOMAIN_OPS(fmt)                        \
-	.iova_to_phys = &pt_iommu_##fmt##_iova_to_phys
+	.iova_to_phys = &pt_iommu_##fmt##_iova_to_phys	\
+	IOMMU_PT_LIVEUPDATE_OPS(fmt)
 #define IOMMU_PT_DIRTY_OPS(fmt) \
 	.read_and_clear_dirty = &pt_iommu_##fmt##_read_and_clear_dirty
 
Signature · 1 line
2.54.0.545.g6539524ca2-goog
maintainer
#36 lore raw

Re: [PATCH v2 06/16] iommupt: Implement preserve/unpreserve/restore callbacks

Quoted 239 lines · Samiullah Khawaja

Implement the iommu domain ops for presevation, unpresevation and restoration of iommu domains for liveupdate. Use the existing page walker to preserve the ioptdesc of the top_table and the lower tables.

Preserve top_level, VASZ and FEAT Sign Extended to restore the domain in the next kernel. On restore the domain has only the preserved features enabled and all the other features are zeroed. This is ok since the restored domain is made immutable and can only be freed. A kunit test is added to verify that the IOMMU domain free can be done with trimmed features.

---
  drivers/iommu/generic_pt/iommu_pt.h       | 131 ++++++++++++++++++++++
  drivers/iommu/generic_pt/kunit_iommu_pt.h |  28 +++++
  include/linux/generic_pt/iommu.h          |  19 +++-
  3 files changed, 177 insertions(+), 1 deletion(-)
drivers/iommu/generic_pt/iommu_pt.h +131 -0
index 19b6daf88f2a..7bca827e3a55 100644
@@ -961,6 +961,133 @@ static int NS(map_range)(struct pt_iommu *iommu_table, dma_addr_t iova,
  	return ret;
  }
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+/**
+ * unpreserve() - Unpreserve page tables and other state of a domain.
+ * @domain: Domain to unpreserve
+ */
+void DOMAIN_NS(unpreserve)(struct iommu_domain *domain, struct iommu_domain_ser *ser)
+{
+	struct pt_iommu *iommu_table =
+		container_of(domain, struct pt_iommu, domain);
+	struct pt_common *common = common_from_iommu(iommu_table);
+	struct pt_range range = pt_all_range(common);
+	struct pt_iommu_collect_args collect = {
+		.free_list = IOMMU_PAGES_LIST_INIT(collect.free_list),
+	};
+
+	iommu_pages_list_add(&collect.free_list, range.top_table);
+	pt_walk_range(&range, __collect_tables, &collect);
+
+	iommu_unpreserve_pages(&collect.free_list);
+}
+EXPORT_SYMBOL_NS_GPL(DOMAIN_NS(unpreserve), "GENERIC_PT_IOMMU");
+
+/**
+ * preserve() - Preserve page tables and other state of a domain.
+ * @domain: Domain to preserve
+ *
+ * Returns: -ERRNO on failure, 0 on success.
+ */
+int DOMAIN_NS(preserve)(struct iommu_domain *domain, struct iommu_domain_ser *ser)
+{
+	struct pt_iommu *iommu_table =
+		container_of(domain, struct pt_iommu, domain);
+	struct pt_common *common = common_from_iommu(iommu_table);
+	struct pt_range range = pt_all_range(common);
+	struct pt_iommu_collect_args collect = {
+		.free_list = IOMMU_PAGES_LIST_INIT(collect.free_list),
+	};
+	int ret;
+
+	iommu_pages_list_add(&collect.free_list, range.top_table);
+	pt_walk_range(&range, __collect_tables, &collect);
+
+	ret = iommu_preserve_pages(&collect.free_list);
+	if (ret)
+		return ret;
+
+	ser->top_table_phys = virt_to_phys(range.top_table);
+	ser->top_level = range.top_level;
+
+	/*
+	 * VASZ and SIGN_EXTEND will be needed in next kernel for collector page
+	 * table walk to restore and free pages.
+	 */
+	ser->vasz = common->max_vasz_lg2;
+	ser->sign_extend = pt_feature(common, PT_FEAT_SIGN_EXTEND);
+
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(DOMAIN_NS(preserve), "GENERIC_PT_IOMMU");
+
+static int __restore_tables(struct pt_range *range, void *arg,
+			    unsigned int level, struct pt_table_p *table)
+{
+	struct pt_state pts = pt_init(range, level, table);
+	int ret;
+
+	for_each_pt_level_entry(&pts) {
+		if (pts.type == PT_ENTRY_TABLE) {
+			iommu_restore_page(virt_to_phys(pts.table_lower));
+
+			/*
+			 * pt_descend can only fail if pts.table_lower is not
+			 * init. So the if statement below is dead code.
+			 */
+			ret = pt_descend(&pts, arg, __restore_tables);
+			if (ret)
+				return ret;
+		}
+	}
+
+	return 0;
+}
+
+static const struct pt_iommu_ops NS(ops_immutable);
+
+/**
+ * restore() - Restore page tables and other state of a domain.
+ * @domain: Domain to preserve
+ *
+ * Returns: -ERRNO on failure, 0 on success.
+ */
+int DOMAIN_NS(restore)(struct iommu_domain *domain, struct iommu_domain_ser *ser)
+{
+	struct pt_iommu *iommu_table =
+		container_of(domain, struct pt_iommu, domain);
+	struct pt_common *common = common_from_iommu(iommu_table);
+	struct pt_range range;
+
+	common->max_vasz_lg2 = ser->vasz;
+
+	/* Make this domain immutable.*/
+	iommu_table->ops = &NS(ops_immutable);
+
+	/*
+	 * It is safe to override this here since this domain is immutable and
+	 * can only be freed.
+	 */
+	common->features = 0;
+	if (ser->sign_extend)
+		common->features |= BIT(PT_FEAT_SIGN_EXTEND);
+
+	range = pt_all_range(common);
+	iommu_restore_page(ser->top_table_phys);
+
+	/* Free new table */
+	iommu_free_pages(range.top_table);
+
+	/* Set the restored top table */
+	pt_top_set(common, phys_to_virt(ser->top_table_phys), ser->top_level);
+
+	/* Restore all pages*/
+	range = pt_all_range(common);
+	return pt_walk_range(&range, __restore_tables, NULL);
+}
+EXPORT_SYMBOL_NS_GPL(DOMAIN_NS(restore), "GENERIC_PT_IOMMU");
+#endif
+
  struct pt_unmap_args {
  	struct iommu_pages_list free_list;
  	pt_vaddr_t unmapped;
@@ -1138,6 +1265,10 @@ static const struct pt_iommu_ops NS(ops) = {
  	.deinit = NS(deinit),
  };
 
+static const struct pt_iommu_ops NS(ops_immutable) = {
+	.deinit = NS(deinit),
+};
+
  static int pt_init_common(struct pt_common *common)
  {
  	struct pt_range top_range = pt_top_range(common);
drivers/iommu/generic_pt/kunit_iommu_pt.h +28 -0
index e8a63c8ea850..af1918d693ed 100644
@@ -426,6 +426,33 @@ static void test_mixed(struct kunit *test)
  	check_iova(test, start, oa, len);
  }
 
+static void test_restore_free(struct kunit *test)
+{
+	struct kunit_iommu_priv *priv = test->priv;
+	struct pt_range top_range = pt_top_range(priv->common);
+	u64 start = 0x3fe400ULL << 12;
+	u64 end = 0x4c0600ULL << 12;
+	pt_vaddr_t len = end - start;
+
+	if (top_range.last_va <= start || sizeof(unsigned long) == 4)
+		kunit_skip(test, "range is too small");
+	if ((priv->safe_pgsize_bitmap & GENMASK(30, 21)) != (BIT(30) | BIT(21)))
+		kunit_skip(test, "incompatible psize");
+
+	/* Map a large mixed range to populate multiple levels of page tables */
+	do_map(test, start, start, len);
+
+	/*
+	 * Simulate a restored state by clearing all features except
+	 * SIGN_EXTEND. This verifies that the generic page table free walker
+	 * can correctly tear down a populated domain when other features are
+	 * zeroed.
+	 */
+	priv->common->features &= BIT(PT_FEAT_SIGN_EXTEND);
+
+	/* The domain will be freed when the test exits. */
+}
+
  static struct kunit_case iommu_test_cases[] = {
  	KUNIT_CASE_FMT(test_increase_level),
  	KUNIT_CASE_FMT(test_map_simple),
@@ -434,6 +461,7 @@ static struct kunit_case iommu_test_cases[] = {
  	KUNIT_CASE_FMT(test_random_map),
  	KUNIT_CASE_FMT(test_pgsize_boundary),
  	KUNIT_CASE_FMT(test_mixed),
+	KUNIT_CASE_FMT(test_restore_free),
  	{},
  };
 
include/linux/generic_pt/iommu.h +7 -0
index dd0edd02a48a..649b3b9eb1a0 100644
@@ -13,6 +13,7 @@ struct iommu_iotlb_gather;
  struct pt_iommu_ops;
  struct pt_iommu_driver_ops;
  struct iommu_dirty_bitmap;
+struct iommu_domain_ser;
 
  /**
   * DOC: IOMMU Radix Page Table
@@ -251,6 +252,12 @@ struct pt_iommu_cfg {
  #define IOMMU_PROTOTYPES(fmt)                                                  \
  	phys_addr_t pt_iommu_##fmt##_iova_to_phys(struct iommu_domain *domain, \
  						  dma_addr_t iova);            \
+	int pt_iommu_##fmt##_preserve(struct iommu_domain *domain,             \
+				      struct iommu_domain_ser *ser);           \
+	void pt_iommu_##fmt##_unpreserve(struct iommu_domain *domain,          \
+					 struct iommu_domain_ser *ser);        \
+	int pt_iommu_##fmt##_restore(struct iommu_domain *domain,              \
+				     struct iommu_domain_ser *ser);            \
  	int pt_iommu_##fmt##_read_and_clear_dirty(                             \
  		struct iommu_domain *domain, unsigned long iova, size_t size,  \
  		unsigned long flags, struct iommu_dirty_bitmap *dirty);        \

include/linux/generic_pt/iommu.h struct pt_iommu_cfg { · line 280

Quoted 8 lines
include/linux/generic_pt/iommu.h +5 -0
@@ -266,12 +273,22 @@ struct pt_iommu_cfg {
  	};                              \
  	IOMMU_PROTOTYPES(fmt)
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+#define IOMMU_PT_LIVEUPDATE_OPS(fmt)			\
+	, .preserve = &pt_iommu_##fmt##_preserve,	\
+	.unpreserve = &pt_iommu_##fmt##_unpreserve,	\
+	.restore = &pt_iommu_##fmt##_restore

Nit: would it look better if we put it like this?

#define IOMMU_PT_LIVEUPDATE_OPS(fmt)			\
	, .preserve = &pt_iommu_##fmt##_preserve	\
	, .unpreserve = &pt_iommu_##fmt##_unpreserve	\
	, .restore = &pt_iommu_##fmt##_restore

include/linux/generic_pt/iommu.h struct pt_iommu_cfg { · line 283

Quoted 3 lines
include/linux/generic_pt/iommu.h +3 -0
+#else
+#define IOMMU_PT_LIVEUPDATE_OPS(fmt)
+#endif

Thanks,
baolu

#37 lore raw

Re: [PATCH v2 06/16] iommupt: Implement preserve/unpreserve/restore callbacks

Quoted 38 lines · Baolu Lu
Quoted 27 lines · Samiullah Khawaja

Implement the iommu domain ops for presevation, unpresevation and restoration of iommu domains for liveupdate. Use the existing page walker to preserve the ioptdesc of the top_table and the lower tables.

Preserve top_level, VASZ and FEAT Sign Extended to restore the domain in the next kernel. On restore the domain has only the preserved features enabled and all the other features are zeroed. This is ok since the restored domain is made immutable and can only be freed. A kunit test is added to verify that the IOMMU domain free can be done with trimmed features.

---

include/linux/generic_pt/iommu.h +6 -0
@@ -251,6 +252,12 @@ struct pt_iommu_cfg {
 #define IOMMU_PROTOTYPES(fmt)                                                  \
 	phys_addr_t pt_iommu_##fmt##_iova_to_phys(struct iommu_domain *domain, \
 						  dma_addr_t iova);            \
+	int pt_iommu_##fmt##_preserve(struct iommu_domain *domain,             \
+				      struct iommu_domain_ser *ser);           \
+	void pt_iommu_##fmt##_unpreserve(struct iommu_domain *domain,          \
+					 struct iommu_domain_ser *ser);        \
+	int pt_iommu_##fmt##_restore(struct iommu_domain *domain,              \
+				     struct iommu_domain_ser *ser);            \
 	int pt_iommu_##fmt##_read_and_clear_dirty(                             \
 		struct iommu_domain *domain, unsigned long iova, size_t size,  \
 		unsigned long flags, struct iommu_dirty_bitmap *dirty);        \

include/linux/generic_pt/iommu.h struct pt_iommu_cfg { · line 279

Quoted 7 lines
include/linux/generic_pt/iommu.h +5 -0
@@ -266,12 +273,22 @@ struct pt_iommu_cfg {
 	};                              \
 	IOMMU_PROTOTYPES(fmt)
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+#define IOMMU_PT_LIVEUPDATE_OPS(fmt)			\
+	, .preserve = &pt_iommu_##fmt##_preserve,	\
+	.unpreserve = &pt_iommu_##fmt##_unpreserve,	\
+	.restore = &pt_iommu_##fmt##_restore

Nit: would it look better if we put it like this?

Quoted 4 lines
#define IOMMU_PT_LIVEUPDATE_OPS(fmt)			\
	, .preserve = &pt_iommu_##fmt##_preserve	\
	, .unpreserve = &pt_iommu_##fmt##_unpreserve	\
	, .restore = &pt_iommu_##fmt##_restore

Agreed. I will change this.

Quoted 8 lines

include/linux/generic_pt/iommu.h struct pt_iommu_cfg { · line 283

Quoted 4 lines
include/linux/generic_pt/iommu.h +3 -0
+#else
+#define IOMMU_PT_LIVEUPDATE_OPS(fmt)
+#endif

Thanks,
baolu

#38 lore raw

Re: [PATCH v2 06/16] iommupt: Implement preserve/unpreserve/restore callbacks

In reply to Samiullah Khawaja
Quoted 18 lines · Samiullah Khawaja

Implement the iommu domain ops for presevation, unpresevation and restoration of iommu domains for liveupdate. Use the existing page walker to preserve the ioptdesc of the top_table and the lower tables.

Preserve top_level, VASZ and FEAT Sign Extended to restore the domain in the next kernel. On restore the domain has only the preserved features enabled and all the other features are zeroed. This is ok since the restored domain is made immutable and can only be freed. A kunit test is added to verify that the IOMMU domain free can be done with trimmed features.

---
 drivers/iommu/generic_pt/iommu_pt.h       | 131 ++++++++++++++++++++++
 drivers/iommu/generic_pt/kunit_iommu_pt.h |  28 +++++
 include/linux/generic_pt/iommu.h          |  19 +++-
 3 files changed, 177 insertions(+), 1 deletion(-)

[...]

Quoted 10 lines
drivers/iommu/generic_pt/iommu_pt.h +10 -0
+static const struct pt_iommu_ops NS(ops_immutable);
+
+/**
+ * restore() - Restore page tables and other state of a domain.
+ * @domain: Domain to preserve
+ *
+ * Returns: -ERRNO on failure, 0 on success.
+ */
+int DOMAIN_NS(restore)(struct iommu_domain *domain, struct iommu_domain_ser *ser)
+{

drivers/iommu/generic_pt/iommu_pt.h static int NS(map_range)(struct pt_iommu *iommu_table, dma_addr_t iova, · line 1066

Quoted 10 lines
drivers/iommu/generic_pt/iommu_pt.h +10 -0
+	struct pt_iommu *iommu_table =
+		container_of(domain, struct pt_iommu, domain);
+	struct pt_common *common = common_from_iommu(iommu_table);
+	struct pt_range range;
+
+	common->max_vasz_lg2 = ser->vasz;
+
+	/* Make this domain immutable.*/
+	iommu_table->ops = &NS(ops_immutable);
+

Nit: Let's consider adding a comment since we just have .deinit in ops:

/*
* Restored page tables are strictly transient and only permitted to be
* destroyed via .deinit. Because we only preserve user-assigned devices
* utilizing pass-through frameworks (VFIO / IOMMUFD), any concurrent or
* subsequent map/unmap operations on a restored domain are explicitly
* blocked at the subsystem boundary (e.g., via IOAS immutability seals). */

This makes it explicitly clear that any nullptr dererfs occuring due to immutable_ops->map_range etc. (in the future) are considered as bugs.

Quoted 28 lines
drivers/iommu/generic_pt/iommu_pt.h +24 -0
+	/*
+	 * It is safe to override this here since this domain is immutable and
+	 * can only be freed.
+	 */
+	common->features = 0;
+	if (ser->sign_extend)
+		common->features |= BIT(PT_FEAT_SIGN_EXTEND);
+
+	range = pt_all_range(common);
+	iommu_restore_page(ser->top_table_phys);
+
+	/* Free new table */
+	iommu_free_pages(range.top_table);
+
+	/* Set the restored top table */
+	pt_top_set(common, phys_to_virt(ser->top_table_phys), ser->top_level);
+
+	/* Restore all pages*/
+	range = pt_all_range(common);
+	return pt_walk_range(&range, __restore_tables, NULL);
+}
+EXPORT_SYMBOL_NS_GPL(DOMAIN_NS(restore), "GENERIC_PT_IOMMU");
+#endif
+
 struct pt_unmap_args {
 	struct iommu_pages_list free_list;
 	pt_vaddr_t unmapped;

drivers/iommu/generic_pt/iommu_pt.h static const struct pt_iommu_ops NS(ops) = { · line 1271

Quoted 7 lines
drivers/iommu/generic_pt/iommu_pt.h +4 -0
@@ -1138,6 +1265,10 @@ static const struct pt_iommu_ops NS(ops) = {
 	.deinit = NS(deinit),
 };
 
+static const struct pt_iommu_ops NS(ops_immutable) = {
+	.deinit = NS(deinit),
+};
+

[...]

Thanks,
Praan

#39 lore raw

Re: [PATCH v2 06/16] iommupt: Implement preserve/unpreserve/restore callbacks

Quoted 53 lines · Pranjal Shrivastava
Quoted 18 lines · Samiullah Khawaja

Implement the iommu domain ops for presevation, unpresevation and restoration of iommu domains for liveupdate. Use the existing page walker to preserve the ioptdesc of the top_table and the lower tables.

Preserve top_level, VASZ and FEAT Sign Extended to restore the domain in the next kernel. On restore the domain has only the preserved features enabled and all the other features are zeroed. This is ok since the restored domain is made immutable and can only be freed. A kunit test is added to verify that the IOMMU domain free can be done with trimmed features.

---
 drivers/iommu/generic_pt/iommu_pt.h       | 131 ++++++++++++++++++++++
 drivers/iommu/generic_pt/kunit_iommu_pt.h |  28 +++++
 include/linux/generic_pt/iommu.h          |  19 +++-
 3 files changed, 177 insertions(+), 1 deletion(-)

[...]

Quoted 10 lines
drivers/iommu/generic_pt/iommu_pt.h +10 -0
+static const struct pt_iommu_ops NS(ops_immutable);
+
+/**
+ * restore() - Restore page tables and other state of a domain.
+ * @domain: Domain to preserve
+ *
+ * Returns: -ERRNO on failure, 0 on success.
+ */
+int DOMAIN_NS(restore)(struct iommu_domain *domain, struct iommu_domain_ser *ser)
+{

drivers/iommu/generic_pt/iommu_pt.h static int NS(map_range)(struct pt_iommu *iommu_table, dma_addr_t iova, · line 1066

Quoted 10 lines
drivers/iommu/generic_pt/iommu_pt.h +10 -0
+	struct pt_iommu *iommu_table =
+		container_of(domain, struct pt_iommu, domain);
+	struct pt_common *common = common_from_iommu(iommu_table);
+	struct pt_range range;
+
+	common->max_vasz_lg2 = ser->vasz;
+
+	/* Make this domain immutable.*/
+	iommu_table->ops = &NS(ops_immutable);
+

Nit: Let's consider adding a comment since we just have .deinit in ops:

/*
* Restored page tables are strictly transient and only permitted to be
* destroyed via .deinit. Because we only preserve user-assigned devices
* utilizing pass-through frameworks (VFIO / IOMMUFD), any concurrent or
* subsequent map/unmap operations on a restored domain are explicitly
* blocked at the subsystem boundary (e.g., via IOAS immutability seals). */

Quoted 2 lines

This makes it explicitly clear that any nullptr dererfs occuring due to immutable_ops->map_range etc. (in the future) are considered as bugs.

Sounds great, will add a comment here.

Quoted 39 lines
Quoted 36 lines
drivers/iommu/generic_pt/iommu_pt.h +28 -0
+	/*
+	 * It is safe to override this here since this domain is immutable and
+	 * can only be freed.
+	 */
+	common->features = 0;
+	if (ser->sign_extend)
+		common->features |= BIT(PT_FEAT_SIGN_EXTEND);
+
+	range = pt_all_range(common);
+	iommu_restore_page(ser->top_table_phys);
+
+	/* Free new table */
+	iommu_free_pages(range.top_table);
+
+	/* Set the restored top table */
+	pt_top_set(common, phys_to_virt(ser->top_table_phys), ser->top_level);
+
+	/* Restore all pages*/
+	range = pt_all_range(common);
+	return pt_walk_range(&range, __restore_tables, NULL);
+}
+EXPORT_SYMBOL_NS_GPL(DOMAIN_NS(restore), "GENERIC_PT_IOMMU");
+#endif
+
 struct pt_unmap_args {
 	struct iommu_pages_list free_list;
 	pt_vaddr_t unmapped;
@@ -1138,6 +1265,10 @@ static const struct pt_iommu_ops NS(ops) = {
 	.deinit = NS(deinit),
 };
 
+static const struct pt_iommu_ops NS(ops_immutable) = {
+	.deinit = NS(deinit),
+};
+

[...]

Quoted 2 lines

Thanks,
Praan

Sami

#40 lore raw

Re: [PATCH v2 06/16] iommupt: Implement preserve/unpreserve/restore callbacks

In reply to Samiullah Khawaja

include/linux/generic_pt/iommu.h struct pt_iommu_cfg { · line 283

Quoted 12 lines · Samiullah Khawaja
include/linux/generic_pt/iommu.h +8 -0
@@ -266,12 +273,22 @@ struct pt_iommu_cfg {
 	};                              \
 	IOMMU_PROTOTYPES(fmt)
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+#define IOMMU_PT_LIVEUPDATE_OPS(fmt)			\
+	, .preserve = &pt_iommu_##fmt##_preserve,	\
+	.unpreserve = &pt_iommu_##fmt##_unpreserve,	\
+	.restore = &pt_iommu_##fmt##_restore
+#else
+#define IOMMU_PT_LIVEUPDATE_OPS(fmt)
+#endif

Don't do it like this at all, new iommupt unique ops should be routed through struct pt_iommu_ops

See how map_range/unmap_range now work

Jason

#41 lore raw

Re: [PATCH v2 06/16] iommupt: Implement preserve/unpreserve/restore callbacks

Quoted 17 lines · Jason Gunthorpe

include/linux/generic_pt/iommu.h struct pt_iommu_cfg { · line 283

Quoted 12 lines · Samiullah Khawaja
include/linux/generic_pt/iommu.h +8 -0
@@ -266,12 +273,22 @@ struct pt_iommu_cfg {
 	};                              \
 	IOMMU_PROTOTYPES(fmt)
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+#define IOMMU_PT_LIVEUPDATE_OPS(fmt)			\
+	, .preserve = &pt_iommu_##fmt##_preserve,	\
+	.unpreserve = &pt_iommu_##fmt##_unpreserve,	\
+	.restore = &pt_iommu_##fmt##_restore
+#else
+#define IOMMU_PT_LIVEUPDATE_OPS(fmt)
+#endif

Don't do it like this at all, new iommupt unique ops should be routed through struct pt_iommu_ops

Quoted 1 line

See how map_range/unmap_range now work

Ah yes. That also looks much cleaner. I will update this.

Quoted 2 lines

Jason

Sami

#42 lore raw

PATCH v2 7/16 iommu/vt-d: Implement device and iommu preserve/unpreserve ops

In reply to Samiullah Khawaja

Add implementation of the device and iommu presevation in a separate file. Also set the device and iommu preserve/unpreserve ops in the struct iommu_ops.

During normal shutdown the iommu translation is disabled. Since the root table is preserved during live update, it needs to be cleaned up and the context entries of the unpreserved devices need to be cleared.

---
 MAINTAINERS                      |   1 +
 drivers/iommu/intel/Makefile     |   1 +
 drivers/iommu/intel/iommu.c      |  52 +++++++++++-
 drivers/iommu/intel/iommu.h      |  28 +++++++
 drivers/iommu/intel/liveupdate.c | 139 +++++++++++++++++++++++++++++++
 drivers/iommu/iommu.c            |  18 ++++
 include/linux/iommu-liveupdate.h |  10 +++
 include/linux/iommu.h            |  14 ++++
 include/linux/kho/abi/iommu.h    |  18 ++++
 9 files changed, 277 insertions(+), 4 deletions(-)
 create mode 100644 drivers/iommu/intel/liveupdate.c
MAINTAINERS +1 -0
index 980041955abc..9f5c02c6c8c1 100644
@@ -13495,6 +13495,7 @@ M:	Samiullah Khawaja <[email protected]>
 R:	Pranjal Shrivastava <[email protected]>
 L:	[email protected]
 S:	Maintained
+F:	drivers/iommu/intel/liveupdate.c
 F:	drivers/iommu/liveupdate.c
 F:	include/linux/iommu-liveupdate.h
 F:	include/linux/kho/abi/iommu.h
drivers/iommu/intel/Makefile +1 -0
index ada651c4a01b..d38fc101bc35 100644
@@ -6,3 +6,4 @@ obj-$(CONFIG_INTEL_IOMMU_DEBUGFS) += debugfs.o
 obj-$(CONFIG_INTEL_IOMMU_SVM) += svm.o
 obj-$(CONFIG_IRQ_REMAP) += irq_remapping.o
 obj-$(CONFIG_INTEL_IOMMU_PERF_EVENTS) += perfmon.o
+obj-$(CONFIG_IOMMU_LIVEUPDATE) += liveupdate.o
drivers/iommu/intel/iommu.c +48 -4

88 lines not on this page · raw message

drivers/iommu/intel/iommu.h +28 -0
index ef145560aa98..5e0bc17e76bf 100644
@@ -552,6 +552,8 @@ struct root_entry {
 	u64     hi;
 };
 
+#define ROOT_ENTRY_NR (VTD_PAGE_SIZE / sizeof(struct root_entry))
+
 /*
  * low 64 bits:
  * 0: present
@@ -1284,6 +1286,32 @@ static inline int iopf_for_domain_replace(struct iommu_domain *new,
 	return 0;
 }
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+int intel_iommu_preserve_device(struct device *dev,
+				struct iommu_device_ser *device_ser);
+int intel_iommu_preserve(struct iommu_device *iommu,
+			 struct iommu_hw_ser *iommu_ser);
+void intel_iommu_unpreserve(struct iommu_device *iommu,
+			    struct iommu_hw_ser *iommu_ser);
+#else
+static inline int intel_iommu_preserve_device(struct device *dev,
+					      struct iommu_device_ser *device_ser)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline int intel_iommu_preserve(struct iommu_device *iommu,
+				       struct iommu_hw_ser *iommu_ser)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void intel_iommu_unpreserve(struct iommu_device *iommu,
+					  struct iommu_hw_ser *iommu_ser)
+{
+}
+#endif
+
 #ifdef CONFIG_INTEL_IOMMU_SVM
 void intel_svm_check(struct intel_iommu *iommu);
 struct iommu_domain *intel_svm_domain_alloc(struct device *dev,
drivers/iommu/intel/liveupdate.c new file +139 -0

139 lines not on this page · raw message

drivers/iommu/iommu.c +18 -0
index 782e73a9d45f..0561990f46e3 100644
@@ -307,6 +307,24 @@ void iommu_device_unregister(struct iommu_device *iommu)
 }
 EXPORT_SYMBOL_GPL(iommu_device_unregister);
 
+static int _iommu_for_each_dev_cb(struct device *dev, void *data)
+{
+	struct iommu_dev_iter *iter = data;
+
+	if (dev->iommu && dev->iommu->iommu_dev == iter->iommu)
+		return iter->fn(dev, iter->iommu, iter->arg);
+
+	return 0;
+}
+
+void iommu_for_each_dev(struct iommu_dev_iter *iter)
+{
+	for (int i = 0; i < ARRAY_SIZE(iommu_buses); i++)
+		bus_for_each_dev(iommu_buses[i], NULL, iter,
+				 _iommu_for_each_dev_cb);
+}
+EXPORT_SYMBOL_GPL(iommu_for_each_dev);
+
 #if IS_ENABLED(CONFIG_IOMMUFD_TEST)
 void iommu_device_unregister_bus(struct iommu_device *iommu,
 				 const struct bus_type *bus,
include/linux/iommu-liveupdate.h +10 -0
index 279c7ab04f09..c9d75c6b3be9 100644
@@ -33,6 +33,11 @@ void iommu_domain_unpreserve(struct iommu_domain *domain);
 int iommu_preserve_device(struct iommu_domain *domain,
 			  struct device *dev, u64 *preserved_state);
 void iommu_unpreserve_device(struct iommu_domain *domain, struct device *dev);
+
+static inline void *iommu_preserved_state(struct iommu_device *iommu)
+{
+	return iommu->outgoing_preserved_state;
+}
 #else
 static inline void *dev_iommu_preserved_state(struct device *dev)
 {
@@ -57,6 +62,11 @@ static inline int iommu_preserve_device(struct iommu_domain *domain,
 static inline void iommu_unpreserve_device(struct iommu_domain *domain, struct device *dev)
 {
 }
+
+static inline void *iommu_preserved_state(struct iommu_device *iommu)
+{
+	return NULL;
+}
 #endif
 
 int iommu_liveupdate_register_flb(struct liveupdate_file_handler *handler);
include/linux/iommu.h +14 -0
index 1c424b32c5fc..999be5127c65 100644
@@ -1207,6 +1207,20 @@ static inline void *dev_iommu_priv_get(struct device *dev)
 
 void dev_iommu_priv_set(struct device *dev, void *priv);
 
+typedef int (*iommu_dev_iter_fn)(struct device *dev,
+				 struct iommu_device *iommu, void *arg);
+
+/**
+ * struct iommu_dev_iter - Iterator for devices attached to an IOMMU
+ */
+struct iommu_dev_iter {
+	struct iommu_device *iommu;
+	iommu_dev_iter_fn fn;
+	void *arg;
+};
+
+void iommu_for_each_dev(struct iommu_dev_iter *iter);
+
 extern struct mutex iommu_probe_device_lock;
 int iommu_probe_device(struct device *dev);
 
include/linux/kho/abi/iommu.h +18 -0
index 37b967820f14..5ffedf0dbd5a 100644
@@ -73,6 +73,7 @@
 
 enum iommu_type_ser {
 	IOMMU_INVALID,
+	IOMMU_INTEL,
 };
 
 /**
@@ -132,16 +133,33 @@ struct iommu_device_ser {
 	struct iommu_dev_map_ser domain_iommu_ser;
 } __packed;
 
+/**
+ * struct iommu_intel_ser - Serialized state of an Intel IOMMU instance
+ * @restored: Whether IOMMU state is restored
+ * @phys_addr: Physical address of the IOMMU register base
+ * @root_table: Physical address of the root entry table
+ */
+struct iommu_intel_ser {
+	u8 restored;
+	u8 padding[7];
+	u64 phys_addr;
+	u64 root_table;
+};
+
 /**
  * struct iommu_hw_ser - Serialized state of an IOMMU instance
  * @hdr: Common object header
  * @token: Unique token for the IOMMU
  * @type: IOMMU type serialized state belongs to
+ * @intel: Intel specific serialization data
  */
 struct iommu_hw_ser {
 	struct iommu_hdr_ser hdr;
 	u64 token;
 	u64 type;
+	union {
+		struct iommu_intel_ser intel;
+	};
 } __packed;
 
 /**
Signature · 1 line
2.54.0.545.g6539524ca2-goog
maintainer
#43 lore raw

Re: [PATCH v2 07/16] iommu/vt-d: Implement device and iommu preserve/unpreserve ops

Quoted 7 lines · Samiullah Khawaja

Add implementation of the device and iommu presevation in a separate file. Also set the device and iommu preserve/unpreserve ops in the struct iommu_ops.

During normal shutdown the iommu translation is disabled. Since the root table is preserved during live update, it needs to be cleaned up and the context entries of the unpreserved devices need to be cleared.

This is not related to preserve/unpreserve ops and could be made in a separated patch?

Quoted 20 lines
---
  MAINTAINERS                      |   1 +
  drivers/iommu/intel/Makefile     |   1 +
  drivers/iommu/intel/iommu.c      |  52 +++++++++++-
  drivers/iommu/intel/iommu.h      |  28 +++++++
  drivers/iommu/intel/liveupdate.c | 139 +++++++++++++++++++++++++++++++
  drivers/iommu/iommu.c            |  18 ++++
  include/linux/iommu-liveupdate.h |  10 +++
  include/linux/iommu.h            |  14 ++++
  include/linux/kho/abi/iommu.h    |  18 ++++
  9 files changed, 277 insertions(+), 4 deletions(-)

create mode 100644 drivers/iommu/intel/liveupdate.c

MAINTAINERS +0 -0
index 980041955abc..9f5c02c6c8c1 100644

MAINTAINERS M: Samiullah Khawaja <[email protected]> · line 13498

Quoted 4 lines
MAINTAINERS +1 -0
@@ -13495,6 +13495,7 @@ M:	Samiullah Khawaja <[email protected]>
  R:	Pranjal Shrivastava <[email protected]>
  L:	[email protected]
  S:	Maintained
+F:	drivers/iommu/intel/liveupdate.c

This file is deeply integrated into the Intel IOMMU driver. Maintaining it separately from the rest of the driver would add unnecessary complexity. I suggest merging this entry into the existing Intel IOMMU block so they can be managed together.

Quoted 190 lines
MAINTAINERS +0 -0
  F:	drivers/iommu/liveupdate.c
  F:	include/linux/iommu-liveupdate.h
  F:	include/linux/kho/abi/iommu.h
drivers/iommu/intel/Makefile +1 -0
index ada651c4a01b..d38fc101bc35 100644
@@ -6,3 +6,4 @@ obj-$(CONFIG_INTEL_IOMMU_DEBUGFS) += debugfs.o
  obj-$(CONFIG_INTEL_IOMMU_SVM) += svm.o
  obj-$(CONFIG_IRQ_REMAP) += irq_remapping.o
  obj-$(CONFIG_INTEL_IOMMU_PERF_EVENTS) += perfmon.o
+obj-$(CONFIG_IOMMU_LIVEUPDATE) += liveupdate.o
drivers/iommu/intel/iommu.c +48 -4
index c3d18cd77d2f..68fecd4e57fa 100644
@@ -16,6 +16,7 @@
  #include <linux/crash_dump.h>
  #include <linux/dma-direct.h>
  #include <linux/dmi.h>
+#include <linux/iommu-liveupdate.h>
  #include <linux/memory.h>
  #include <linux/pci.h>
  #include <linux/pci-ats.h>
@@ -52,6 +53,8 @@ static int rwbf_quirk;
 
  #define rwbf_required(iommu)	(rwbf_quirk || cap_rwbf((iommu)->cap))
 
+static void clear_unpreserved_context_entries(struct intel_iommu *iommu);
+
  /*
   * set to 1 to panic kernel if can't successfully enable VT-d
   * (used when kernel is launched w/ TXT)
@@ -60,8 +63,6 @@ static int force_on = 0;
  static int intel_iommu_tboot_noforce;
  static int no_platform_optin;
 
-#define ROOT_ENTRY_NR (VTD_PAGE_SIZE/sizeof(struct root_entry))
-
  /*
   * Take a root_entry and return the Lower Context Table Pointer (LCTP)
   * if marked present.
@@ -2375,8 +2376,11 @@ void intel_iommu_shutdown(void)
  		/* Disable PMRs explicitly here. */
  		iommu_disable_protect_mem_regions(iommu);
 
-		/* Make sure the IOMMUs are switched off */
-		iommu_disable_translation(iommu);
+		/* Make sure the IOMMUs are switched off if not preserved. */
+		if (iommu_preserved_state(&iommu->iommu))
+			clear_unpreserved_context_entries(iommu);
+		else
+			iommu_disable_translation(iommu);
  	}
  }
 
@@ -2899,6 +2903,41 @@ static const struct iommu_dirty_ops intel_second_stage_dirty_ops = {
  	.set_dirty_tracking = intel_iommu_set_dirty_tracking,
  };
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+static int clear_unpreserve_context_entry_fn(struct device *dev,
+					     struct iommu_device *iommu,
+					     void *arg)
+{
+	struct device_domain_info *info;
+
+	info = dev_iommu_priv_get(dev);
+	if (!info)
+		return 0;
+
+	if (dev_is_pci(dev) && dev_iommu_preserved_state(dev))
+		return 0;
+
+	domain_context_clear(info);
+	return 0;
+}
+
+static void clear_unpreserved_context_entries(struct intel_iommu *iommu)
+{
+	struct iommu_dev_iter iter = {
+		.fn = clear_unpreserve_context_entry_fn,
+		.iommu = &iommu->iommu,
+		.arg = NULL,
+
+	};
+
+	iommu_for_each_dev(&iter);
+}
+#else
+static void clear_unpreserved_context_entries(struct intel_iommu *iommu)
+{
+}
+#endif
+
  static struct iommu_domain *
  intel_iommu_domain_alloc_second_stage(struct device *dev,
  				      struct intel_iommu *iommu, u32 flags)
@@ -3926,6 +3965,11 @@ const struct iommu_ops intel_iommu_ops = {
  	.is_attach_deferred	= intel_iommu_is_attach_deferred,
  	.def_domain_type	= device_def_domain_type,
  	.page_response		= intel_iommu_page_response,
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	.preserve_device	= intel_iommu_preserve_device,
+	.preserve		= intel_iommu_preserve,
+	.unpreserve		= intel_iommu_unpreserve,
+#endif
  };
 
  static void quirk_iommu_igfx(struct pci_dev *dev)
drivers/iommu/intel/iommu.h +28 -0
index ef145560aa98..5e0bc17e76bf 100644
@@ -552,6 +552,8 @@ struct root_entry {
  	u64     hi;
  };
 
+#define ROOT_ENTRY_NR (VTD_PAGE_SIZE / sizeof(struct root_entry))
+
  /*
   * low 64 bits:
   * 0: present
@@ -1284,6 +1286,32 @@ static inline int iopf_for_domain_replace(struct iommu_domain *new,
  	return 0;
  }
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+int intel_iommu_preserve_device(struct device *dev,
+				struct iommu_device_ser *device_ser);
+int intel_iommu_preserve(struct iommu_device *iommu,
+			 struct iommu_hw_ser *iommu_ser);
+void intel_iommu_unpreserve(struct iommu_device *iommu,
+			    struct iommu_hw_ser *iommu_ser);
+#else
+static inline int intel_iommu_preserve_device(struct device *dev,
+					      struct iommu_device_ser *device_ser)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline int intel_iommu_preserve(struct iommu_device *iommu,
+				       struct iommu_hw_ser *iommu_ser)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void intel_iommu_unpreserve(struct iommu_device *iommu,
+					  struct iommu_hw_ser *iommu_ser)
+{
+}
+#endif
+
  #ifdef CONFIG_INTEL_IOMMU_SVM
  void intel_svm_check(struct intel_iommu *iommu);
  struct iommu_domain *intel_svm_domain_alloc(struct device *dev,
drivers/iommu/intel/liveupdate.c new file +28 -0
new file mode 100644
index 000000000000..75fa68b701bf
@@ -0,0 +1,139 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+/*
+ * Copyright (C) 2026, Google LLC
+ * Author: Samiullah Khawaja <[email protected]>
+ */
+
+#define pr_fmt(fmt)    "DMAR: liveupdate: " fmt
+
+#include <linux/kexec_handover.h>
+#include <linux/liveupdate.h>
+#include <linux/iommu-liveupdate.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+
+#include "iommu.h"
+#include "../iommu-pages.h"
+
+static void unpreserve_iommu_context_table(struct intel_iommu *iommu, int end)
+{
+	struct context_entry *context;
+	int i;
+
+	for (i = 0; i < end; i++) {
+		context = iommu_context_addr(iommu, i, 0, 0);
+		if (context)
+			iommu_unpreserve_page(context);
+

drivers/iommu/intel/liveupdate.c line 38

Quoted 10 lines
drivers/iommu/intel/liveupdate.c new file +10 -0
+		if (!sm_supported(iommu))
+			continue;
+
+		context = iommu_context_addr(iommu, i, 0x80, 0);
+		if (context)
+			iommu_unpreserve_page(context);
+	}
+}
+
+static int preserve_iommu_context_table(struct intel_iommu *iommu)

Since this function preserves all context tables, should it be renamed to preserve_iommu_context_tables()?

drivers/iommu/intel/liveupdate.c line 49

Quoted 11 lines
drivers/iommu/intel/liveupdate.c new file +11 -0
+{
+	struct context_entry *context;
+	int ret;
+	int i;
+
+	for (i = 0; i < ROOT_ENTRY_NR; i++) {
+		/*
+		 * Alloc the context tables now to make sure the iommu unit is
+		 * properly preserved. These might stay unused and wastes around
+		 * 32MB max in scalable mode.
+		 */

Instead of allocating and preserving context tables for all root entries (as noted, can waste up to 32MB), could we restrict this only to the entries possibly in use by active PCI devices?

Quoted 158 lines
drivers/iommu/intel/liveupdate.c new file +90 -0
+		spin_lock(&iommu->lock);
+		context = iommu_context_addr(iommu, i, 0, 1);
+		spin_unlock(&iommu->lock);
+		if (!context) {
+			ret = -ENOMEM;
+			goto error;
+		}
+		ret = iommu_preserve_page(context);
+		if (ret)
+			goto error;
+
+		if (!sm_supported(iommu))
+			continue;
+
+		spin_lock(&iommu->lock);
+		context = iommu_context_addr(iommu, i, 0x80, 1);
+		spin_unlock(&iommu->lock);
+		if (!context) {
+			ret = -ENOMEM;
+			goto error_sm;
+		}
+		ret = iommu_preserve_page(context);
+		if (ret)
+			goto error_sm;
+	}
+
+	return 0;
+
+error_sm:
+	context = iommu_context_addr(iommu, i, 0, 0);
+	iommu_unpreserve_page(context);
+error:
+	unpreserve_iommu_context_table(iommu, i);
+	return ret;
+}
+
+int intel_iommu_preserve_device(struct device *dev,
+				struct iommu_device_ser *device_ser)
+{
+	struct device_domain_info *info = dev_iommu_priv_get(dev);
+
+	if (!dev_is_pci(dev)) {
+		dev_err(dev, "Cannot preserve non-PCI device\n");
+		return -EOPNOTSUPP;
+	}
+
+	if (!info)
+		return -EINVAL;
+
+	device_ser->domain_iommu_ser.attachment_id = domain_id_iommu(info->domain,
+								     info->iommu);
+	return 0;
+}
+
+int intel_iommu_preserve(struct iommu_device *iommu_dev,
+			 struct iommu_hw_ser *ser)
+{
+	struct intel_iommu *iommu;
+	int ret;
+
+	iommu = container_of(iommu_dev, struct intel_iommu, iommu);
+
+	ret = preserve_iommu_context_table(iommu);
+	if (ret)
+		return ret;
+
+	ret = iommu_preserve_page(iommu->root_entry);
+	if (ret) {
+		unpreserve_iommu_context_table(iommu, ROOT_ENTRY_NR);
+		return ret;
+	}
+
+	ser->intel.phys_addr = iommu->reg_phys;
+	ser->intel.root_table = __pa(iommu->root_entry);
+	ser->type = IOMMU_INTEL;
+	ser->token = ser->intel.phys_addr;
+
+	return 0;
+}
+
+void intel_iommu_unpreserve(struct iommu_device *iommu_dev,
+			    struct iommu_hw_ser *iommu_ser)
+{
+	struct intel_iommu *iommu;
+
+	iommu = container_of(iommu_dev, struct intel_iommu, iommu);
+
+	unpreserve_iommu_context_table(iommu, ROOT_ENTRY_NR);
+	iommu_unpreserve_page(iommu->root_entry);
+}
drivers/iommu/iommu.c +18 -0
index 782e73a9d45f..0561990f46e3 100644
@@ -307,6 +307,24 @@ void iommu_device_unregister(struct iommu_device *iommu)
  }
  EXPORT_SYMBOL_GPL(iommu_device_unregister);
 
+static int _iommu_for_each_dev_cb(struct device *dev, void *data)
+{
+	struct iommu_dev_iter *iter = data;
+
+	if (dev->iommu && dev->iommu->iommu_dev == iter->iommu)
+		return iter->fn(dev, iter->iommu, iter->arg);
+
+	return 0;
+}
+
+void iommu_for_each_dev(struct iommu_dev_iter *iter)
+{
+	for (int i = 0; i < ARRAY_SIZE(iommu_buses); i++)
+		bus_for_each_dev(iommu_buses[i], NULL, iter,
+				 _iommu_for_each_dev_cb);
+}
+EXPORT_SYMBOL_GPL(iommu_for_each_dev);
+
  #if IS_ENABLED(CONFIG_IOMMUFD_TEST)
  void iommu_device_unregister_bus(struct iommu_device *iommu,
  				 const struct bus_type *bus,
include/linux/iommu-liveupdate.h +10 -0
index 279c7ab04f09..c9d75c6b3be9 100644
@@ -33,6 +33,11 @@ void iommu_domain_unpreserve(struct iommu_domain *domain);
  int iommu_preserve_device(struct iommu_domain *domain,
  			  struct device *dev, u64 *preserved_state);
  void iommu_unpreserve_device(struct iommu_domain *domain, struct device *dev);
+
+static inline void *iommu_preserved_state(struct iommu_device *iommu)
+{
+	return iommu->outgoing_preserved_state;
+}
  #else
  static inline void *dev_iommu_preserved_state(struct device *dev)
  {
@@ -57,6 +62,11 @@ static inline int iommu_preserve_device(struct iommu_domain *domain,
  static inline void iommu_unpreserve_device(struct iommu_domain *domain, struct device *dev)
  {
  }
+
+static inline void *iommu_preserved_state(struct iommu_device *iommu)
+{
+	return NULL;
+}
  #endif
 
  int iommu_liveupdate_register_flb(struct liveupdate_file_handler *handler);
include/linux/iommu.h +3 -0
index 1c424b32c5fc..999be5127c65 100644
@@ -1207,6 +1207,20 @@ static inline void *dev_iommu_priv_get(struct device *dev)
 
  void dev_iommu_priv_set(struct device *dev, void *priv);
 
+typedef int (*iommu_dev_iter_fn)(struct device *dev,
+				 struct iommu_device *iommu, void *arg);
+

include/linux/iommu.h static inline void *dev_iommu_priv_get(struct device *dev) · line 1222

Quoted 10 lines
include/linux/iommu.h +10 -0
+/**
+ * struct iommu_dev_iter - Iterator for devices attached to an IOMMU
+ */
+struct iommu_dev_iter {
+	struct iommu_device *iommu;
+	iommu_dev_iter_fn fn;
+	void *arg;
+};
+
+void iommu_for_each_dev(struct iommu_dev_iter *iter);

Is a generic helper necessary here? I am concerned about potential races with concurrent operations especially probe_device or release_device. And also the hot-plug/unplug scenarios?

Since the current preservation logic is limited to PCI devices, it might be safer and simpler to use the existing for_each_pci_dev() for this specific case instead of introducing a new generic helper?

Quoted 40 lines
include/linux/iommu.h +1 -0
+
  extern struct mutex iommu_probe_device_lock;
  int iommu_probe_device(struct device *dev);
 
include/linux/kho/abi/iommu.h +15 -0
index 37b967820f14..5ffedf0dbd5a 100644
@@ -73,6 +73,7 @@
 
  enum iommu_type_ser {
  	IOMMU_INVALID,
+	IOMMU_INTEL,
  };
 
  /**
@@ -132,16 +133,33 @@ struct iommu_device_ser {
  	struct iommu_dev_map_ser domain_iommu_ser;
  } __packed;
 
+/**
+ * struct iommu_intel_ser - Serialized state of an Intel IOMMU instance
+ * @restored: Whether IOMMU state is restored
+ * @phys_addr: Physical address of the IOMMU register base
+ * @root_table: Physical address of the root entry table
+ */
+struct iommu_intel_ser {
+	u8 restored;
+	u8 padding[7];
+	u64 phys_addr;
+	u64 root_table;
+};
+
  /**
   * struct iommu_hw_ser - Serialized state of an IOMMU instance
   * @hdr: Common object header
   * @token: Unique token for the IOMMU
   * @type: IOMMU type serialized state belongs to
+ * @intel: Intel specific serialization data
   */

include/linux/kho/abi/iommu.h struct iommu_device_ser { · line 165

Quoted 10 lines
include/linux/kho/abi/iommu.h +3 -0
  struct iommu_hw_ser {
  	struct iommu_hdr_ser hdr;
  	u64 token;
  	u64 type;
+	union {
+		struct iommu_intel_ser intel;
+	};
  } __packed;
 
  /**

Thanks,
baolu

#44 lore raw

Re: [PATCH v2 07/16] iommu/vt-d: Implement device and iommu preserve/unpreserve ops

Quoted 11 lines · Baolu Lu
Quoted 7 lines · Samiullah Khawaja

Add implementation of the device and iommu presevation in a separate file. Also set the device and iommu preserve/unpreserve ops in the struct iommu_ops.

During normal shutdown the iommu translation is disabled. Since the root table is preserved during live update, it needs to be cleaned up and the context entries of the unpreserved devices need to be cleared.

This is not related to preserve/unpreserve ops and could be made in a separated patch?

Agreed. I will move this stuff to a separate patch.

Quoted 16 lines
Quoted 16 lines
---
 MAINTAINERS                      |   1 +
 drivers/iommu/intel/Makefile     |   1 +
 drivers/iommu/intel/iommu.c      |  52 +++++++++++-
 drivers/iommu/intel/iommu.h      |  28 +++++++
 drivers/iommu/intel/liveupdate.c | 139 +++++++++++++++++++++++++++++++
 drivers/iommu/iommu.c            |  18 ++++
 include/linux/iommu-liveupdate.h |  10 +++
 include/linux/iommu.h            |  14 ++++
 include/linux/kho/abi/iommu.h    |  18 ++++
 9 files changed, 277 insertions(+), 4 deletions(-)
 create mode 100644 drivers/iommu/intel/liveupdate.c
Quoted 14 lines

MAINTAINERS M: Samiullah Khawaja <[email protected]> · line 13498

Quoted 9 lines
MAINTAINERS +1 -0
index 980041955abc..9f5c02c6c8c1 100644
@@ -13495,6 +13495,7 @@ M:	Samiullah Khawaja <[email protected]>
 R:	Pranjal Shrivastava <[email protected]>
 L:	[email protected]
 S:	Maintained
+F:	drivers/iommu/intel/liveupdate.c

This file is deeply integrated into the Intel IOMMU driver. Maintaining it separately from the rest of the driver would add unnecessary complexity. I suggest merging this entry into the existing Intel IOMMU block so they can be managed together.

I understand your concern. This file is driver specific and should definitely be routed through your tree.

My goal with listing it here wasn't to change the merge path, but rather to ensure I am responsible for the liveupdate-specific logic in this file. Because this file acts as the backend to the generic Live Update framework, keeping it tied to the Liveupdate IOMMU entry helps guarantee that future changes stay architecturally aligned and tested across the whole liveupdate feature.

However, I think I will add a dedicated 'IOMMU VT-d LIVEUPDATE' block where you are also listed as maintainer, and the patches for this file should absolutely continue to flow through the Intel IOMMU tree.

Quoted 7 lines
Quoted 7 lines
MAINTAINERS +0 -0
 F:	drivers/iommu/liveupdate.c
 F:	include/linux/iommu-liveupdate.h
 F:	include/linux/kho/abi/iommu.h
drivers/iommu/intel/Makefile +1 -0
index ada651c4a01b..d38fc101bc35 100644
+

[snip]

Quoted 15 lines
Quoted 15 lines
drivers/iommu/intel/Makefile +15 -0
+static void unpreserve_iommu_context_table(struct intel_iommu *iommu, int end)
+{
+	struct context_entry *context;
+	int i;
+
+	for (i = 0; i < end; i++) {
+		context = iommu_context_addr(iommu, i, 0, 0);
+		if (context)
+			iommu_unpreserve_page(context);
+
+		if (!sm_supported(iommu))
+			continue;
+
+		context = iommu_context_addr(iommu, i, 0x80, 0);
+		if (context)
Quoted 8 lines

drivers/iommu/intel/Makefile

Quoted 5 lines
drivers/iommu/intel/Makefile +5 -0
+			iommu_unpreserve_page(context);
+	}
+}
+
+static int preserve_iommu_context_table(struct intel_iommu *iommu)

Since this function preserves all context tables, should it be renamed to preserve_iommu_context_tables()?

Agreed. Will change this.

Quoted 16 lines

drivers/iommu/intel/Makefile

Quoted 12 lines
drivers/iommu/intel/Makefile +11 -0
+{
+	struct context_entry *context;
+	int ret;
+	int i;
+
+	for (i = 0; i < ROOT_ENTRY_NR; i++) {
+		/*
+		 * Alloc the context tables now to make sure the iommu unit is
+		 * properly preserved. These might stay unused and wastes around
+		 * 32MB max in scalable mode.
+		 */

Instead of allocating and preserving context tables for all root entries (as noted, can waste up to 32MB), could we restrict this only to the entries possibly in use by active PCI devices?

I think the hotplug devices or VFs created through SR-IOV will be missed that way. Lets say device A is preserved and the associated iommu is also preserved. And then a new device B is hotplugged and preserved, then the context table for that will be missed.

Since we don't track the context_tables that are preserved, there is no way to incremently preserve the new-ones. Let me look into the behaviour of KHO, maybe we can make the preserve call idempotent and do these incrementally.

Quoted 8 lines
Quoted 8 lines
drivers/iommu/intel/Makefile +7 -0
+		spin_lock(&iommu->lock);
+		context = iommu_context_addr(iommu, i, 0, 1);
+		spin_unlock(&iommu->lock);
+		if (!context) {
+			ret = -ENOMEM;
+			goto error;
+		}

[snip]

Quoted 24 lines
Quoted 9 lines
include/linux/iommu.h +3 -0
index 1c424b32c5fc..999be5127c65 100644
@@ -1207,6 +1207,20 @@ static inline void *dev_iommu_priv_get(struct device *dev)
 void dev_iommu_priv_set(struct device *dev, void *priv);
+typedef int (*iommu_dev_iter_fn)(struct device *dev,
+				 struct iommu_device *iommu, void *arg);
+

include/linux/iommu.h static inline void *dev_iommu_priv_get(struct device *dev) · line 1220

Quoted 10 lines
include/linux/iommu.h +10 -0
+/**
+ * struct iommu_dev_iter - Iterator for devices attached to an IOMMU
+ */
+struct iommu_dev_iter {
+	struct iommu_device *iommu;
+	iommu_dev_iter_fn fn;
+	void *arg;
+};
+
+void iommu_for_each_dev(struct iommu_dev_iter *iter);

Is a generic helper necessary here? I am concerned about potential races with concurrent operations especially probe_device or release_device. And also the hot-plug/unplug scenarios?

Quoted 3 lines

Since the current preservation logic is limited to PCI devices, it might be safer and simpler to use the existing for_each_pci_dev() for this specific case instead of introducing a new generic helper?

The helper is used late during shutdown to clean up context entries for devices that are not preserved. During this phase, the system is quiescent and hotplug is disabled, so there are no races with concurrent probe or release operations.

I was using for_each_pci_dev() in previous version, but the iterator is needed to clear context entries of both PCI and non-PCI devices that are unpreserved.

Please also see related comments on previous version:
https://lore.kernel.org/all/[email protected]/

Quoted 4 lines
Quoted 4 lines
include/linux/iommu.h +1 -0
+
 extern struct mutex iommu_probe_device_lock;
 int iommu_probe_device(struct device *dev);

[snip]

Quoted 7 lines

include/linux/iommu.h

Quoted 3 lines
include/linux/iommu.h +1 -0
+	};
 } __packed;
 /**

Thanks,
baolu

Thanks,
Sami

#45 lore raw

Re: [PATCH v2 07/16] iommu/vt-d: Implement device and iommu preserve/unpreserve ops

Quoted 30 lines · Samiullah Khawaja
Quoted 11 lines · Baolu Lu
Quoted 7 lines · Samiullah Khawaja

Add implementation of the device and iommu presevation in a separate file. Also set the device and iommu preserve/unpreserve ops in the struct iommu_ops.

During normal shutdown the iommu translation is disabled. Since the root table is preserved during live update, it needs to be cleaned up and the context entries of the unpreserved devices need to be cleared.

This is not related to preserve/unpreserve ops and could be made in a separated patch?

Agreed. I will move this stuff to a separate patch.

Quoted 16 lines
Quoted 16 lines

---

MAINTAINERS                      |   1 +
drivers/iommu/intel/Makefile     |   1 +
drivers/iommu/intel/iommu.c      |  52 +++++++++++-
drivers/iommu/intel/iommu.h      |  28 +++++++
drivers/iommu/intel/liveupdate.c | 139 +++++++++++++++++++++++++++++++
drivers/iommu/iommu.c            |  18 ++++
include/linux/iommu-liveupdate.h |  10 +++
include/linux/iommu.h            |  14 ++++
include/linux/kho/abi/iommu.h    |  18 ++++
9 files changed, 277 insertions(+), 4 deletions(-)

create mode 100644 drivers/iommu/intel/liveupdate.c

[snip]

Quoted 17 lines
Quoted 16 lines

drivers/iommu/intel/liveupdate.c line 49

Quoted 12 lines
drivers/iommu/intel/liveupdate.c new file +11 -0
+{
+	struct context_entry *context;
+	int ret;
+	int i;
+
+	for (i = 0; i < ROOT_ENTRY_NR; i++) {
+		/*
+		 * Alloc the context tables now to make sure the iommu unit is
+		 * properly preserved. These might stay unused and wastes around
+		 * 32MB max in scalable mode.
+		 */

Instead of allocating and preserving context tables for all root entries (as noted, can waste up to 32MB), could we restrict this only to the entries possibly in use by active PCI devices?

Quoted 4 lines

I think the hotplug devices or VFs created through SR-IOV will be missed that way. Lets say device A is preserved and the associated iommu is also preserved. And then a new device B is hotplugged and preserved, then the context table for that will be missed.

Ok I thought about it a little more and basically we have following things to consider when we preserve context tables,

- The devices can be hotplugged and preserved, so the context tables of those need to be preserved if we don't allocate all of them first time we preserve iommu, as done here.
- New context tables can be added (after hotplug) for unpreserved devices. And if we don't get another iommu preserve call after these are added, those remain unpreserved, so during shutdown those entries need to be removed from root table or preserved for simplicity.

To solve this we can,

1. Either preserve the new context table when it is added for a preserved iommu. This can be done in iommu_context_addr(). This is simpler and no tracking needed.

2. Or track the preserved context tables using a bitmap and then preserve them incremently whenever a device is preserved. On shutdown during cleanup, we can clear the entries for unpreserved context tables from root table.

I am inclined towards second option. WDYT?

I think we will have to do similar stuff for PASID also down the road to preserve pasid_tables in PASID directory.

Quoted 13 lines

Since we don't track the context_tables that are preserved, there is no way to incremently preserve the new-ones. Let me look into the behaviour of KHO, maybe we can make the preserve call idempotent and do these incrementally.

Quoted 8 lines
Quoted 8 lines
drivers/iommu/intel/liveupdate.c new file +7 -0
+		spin_lock(&iommu->lock);
+		context = iommu_context_addr(iommu, i, 0, 1);
+		spin_unlock(&iommu->lock);
+		if (!context) {
+			ret = -ENOMEM;
+			goto error;
+		}

[snip]

Quoted 7 lines
Quoted 4 lines

Thanks,
baolu

Thanks,
Sami

Sami

#46 lore raw

Re: [PATCH v2 07/16] iommu/vt-d: Implement device and iommu preserve/unpreserve ops

Quoted 72 lines · Samiullah Khawaja
Quoted 30 lines · Samiullah Khawaja
Quoted 11 lines · Baolu Lu
Quoted 7 lines · Samiullah Khawaja

Add implementation of the device and iommu presevation in a separate file. Also set the device and iommu preserve/unpreserve ops in the struct iommu_ops.

During normal shutdown the iommu translation is disabled. Since the root table is preserved during live update, it needs to be cleaned up and the context entries of the unpreserved devices need to be cleared.

This is not related to preserve/unpreserve ops and could be made in a separated patch?

Agreed. I will move this stuff to a separate patch.

Quoted 16 lines
Quoted 16 lines

---

MAINTAINERS                      |   1 +
drivers/iommu/intel/Makefile     |   1 +
drivers/iommu/intel/iommu.c      |  52 +++++++++++-
drivers/iommu/intel/iommu.h      |  28 +++++++
drivers/iommu/intel/liveupdate.c | 139 +++++++++++++++++++++++++++++++
drivers/iommu/iommu.c            |  18 ++++
include/linux/iommu-liveupdate.h |  10 +++
include/linux/iommu.h            |  14 ++++
include/linux/kho/abi/iommu.h    |  18 ++++
9 files changed, 277 insertions(+), 4 deletions(-)

create mode 100644 drivers/iommu/intel/liveupdate.c

[snip]

Quoted 17 lines
Quoted 16 lines

drivers/iommu/intel/liveupdate.c line 49

Quoted 12 lines
drivers/iommu/intel/liveupdate.c new file +11 -0
+{
+	struct context_entry *context;
+	int ret;
+	int i;
+
+	for (i = 0; i < ROOT_ENTRY_NR; i++) {
+		/*
+		 * Alloc the context tables now to make sure the iommu unit is
+		 * properly preserved. These might stay unused and wastes around
+		 * 32MB max in scalable mode.
+		 */

Instead of allocating and preserving context tables for all root entries (as noted, can waste up to 32MB), could we restrict this only to the entries possibly in use by active PCI devices?

Quoted 4 lines

I think the hotplug devices or VFs created through SR-IOV will be missed that way. Lets say device A is preserved and the associated iommu is also preserved. And then a new device B is hotplugged and preserved, then the context table for that will be missed.

Ok I thought about it a little more and basically we have following things to consider when we preserve context tables,

- The devices can be hotplugged and preserved, so the context tables of those need to be preserved if we don't allocate all of them first time we preserve iommu, as done here.
- New context tables can be added (after hotplug) for unpreserved devices. And if we don't get another iommu preserve call after these are added, those remain unpreserved, so during shutdown those entries need to be removed from root table or preserved for simplicity.

To solve this we can,

1. Either preserve the new context table when it is added for a preserved iommu. This can be done in iommu_context_addr(). This is simpler and no tracking needed.

Quoted 6 lines

2. Or track the preserved context tables using a bitmap and then preserve them incremently whenever a device is preserved. On shutdown during cleanup, we can clear the entries for unpreserved context tables from root table.

I am inclined towards second option. WDYT?

Thinking out loud here, I agree that shifting away from the 32MB pre-allocation is the right direction. I'm wondering if we can avoid the overhead of introducing a new tracking bitmap (Option 2) altogether?

Since the IOMMU serialization is a strict dependency for device tracking, could we move the context table preservation directly into the device level op: intel_iommu_preserve_device()?

Whenever a specific device is preserved on-demand:

1. It queries the parent IOMMU to fetch the allocated context table backing its info->bus.

2. It calls iommu_preserve_page(context) for that table. Because KHO's tracking handles duplicates, this should be fine if multiple devices reside on the same bus...

Regarding Scalable Mode, we could just need a simple check in that path:

/* intel_iommu_preserve_device */
/* Preserve the primary/lower context table backing this bus */ context = iommu_context_addr(info->iommu, info->bus, 0, 0); if (context)

	iommu_preserve_page(context);

/* If scalable mode is active, preserve the upper context table as well */

if (sm_supported(info->iommu)) {
	context = iommu_context_addr(info->iommu, info->bus, 0x80, 0);
	if (context)
		iommu_preserve_page(context);
}

WDYT?

Quoted 9 lines

I think we will have to do similar stuff for PASID also down the road to preserve pasid_tables in PASID directory.

Quoted 6 lines

Since we don't track the context_tables that are preserved, there is no way to incremently preserve the new-ones. Let me look into the behaviour of KHO, maybe we can make the preserve call idempotent and do these incrementally.

Quoted 1 line
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines

drivers/iommu/intel/liveupdate.c line 56

Quoted 7 lines
drivers/iommu/intel/liveupdate.c new file +7 -0
+		spin_lock(&iommu->lock);
+		context = iommu_context_addr(iommu, i, 0, 1);
+		spin_unlock(&iommu->lock);
+		if (!context) {
+			ret = -ENOMEM;
+			goto error;
+		}

[snip]

Thanks,
Praan

#47 lore raw

Re: [PATCH v2 07/16] iommu/vt-d: Implement device and iommu preserve/unpreserve ops

Quoted 73 lines · Pranjal Shrivastava
Quoted 72 lines · Samiullah Khawaja
Quoted 30 lines · Samiullah Khawaja
Quoted 11 lines · Baolu Lu
Quoted 7 lines · Samiullah Khawaja

Add implementation of the device and iommu presevation in a separate file. Also set the device and iommu preserve/unpreserve ops in the struct iommu_ops.

During normal shutdown the iommu translation is disabled. Since the root table is preserved during live update, it needs to be cleaned up and the context entries of the unpreserved devices need to be cleared.

This is not related to preserve/unpreserve ops and could be made in a separated patch?

Agreed. I will move this stuff to a separate patch.

Quoted 16 lines
Quoted 16 lines

---

MAINTAINERS                      |   1 +
drivers/iommu/intel/Makefile     |   1 +
drivers/iommu/intel/iommu.c      |  52 +++++++++++-
drivers/iommu/intel/iommu.h      |  28 +++++++
drivers/iommu/intel/liveupdate.c | 139 +++++++++++++++++++++++++++++++
drivers/iommu/iommu.c            |  18 ++++
include/linux/iommu-liveupdate.h |  10 +++
include/linux/iommu.h            |  14 ++++
include/linux/kho/abi/iommu.h    |  18 ++++
9 files changed, 277 insertions(+), 4 deletions(-)

create mode 100644 drivers/iommu/intel/liveupdate.c

[snip]

Quoted 17 lines
Quoted 16 lines

drivers/iommu/intel/liveupdate.c line 49

Quoted 12 lines
drivers/iommu/intel/liveupdate.c new file +11 -0
+{
+	struct context_entry *context;
+	int ret;
+	int i;
+
+	for (i = 0; i < ROOT_ENTRY_NR; i++) {
+		/*
+		 * Alloc the context tables now to make sure the iommu unit is
+		 * properly preserved. These might stay unused and wastes around
+		 * 32MB max in scalable mode.
+		 */

Instead of allocating and preserving context tables for all root entries (as noted, can waste up to 32MB), could we restrict this only to the entries possibly in use by active PCI devices?

Quoted 4 lines

I think the hotplug devices or VFs created through SR-IOV will be missed that way. Lets say device A is preserved and the associated iommu is also preserved. And then a new device B is hotplugged and preserved, then the context table for that will be missed.

Ok I thought about it a little more and basically we have following things to consider when we preserve context tables,

- The devices can be hotplugged and preserved, so the context tables of those need to be preserved if we don't allocate all of them first time we preserve iommu, as done here.
- New context tables can be added (after hotplug) for unpreserved devices. And if we don't get another iommu preserve call after these are added, those remain unpreserved, so during shutdown those entries need to be removed from root table or preserved for simplicity.

To solve this we can,

1. Either preserve the new context table when it is added for a preserved iommu. This can be done in iommu_context_addr(). This is simpler and no tracking needed.

Quoted 8 lines
Quoted 6 lines

2. Or track the preserved context tables using a bitmap and then preserve them incremently whenever a device is preserved. On shutdown during cleanup, we can clear the entries for unpreserved context tables from root table.

I am inclined towards second option. WDYT?

Thinking out loud here, I agree that shifting away from the 32MB

Wait it should be 2MB (256 buses * 8K(low + Hi)), I miswrote it earlier.

Quoted 2 lines

pre-allocation is the right direction. I'm wondering if we can avoid the overhead of introducing a new tracking bitmap (Option 2) altogether?

ROOT_ENTRY_NR is 256 and has 2 context tables for scalable mode, so the max bitmap will be eight u64. It is per IOMMU, but still reasonable.

Quoted 23 lines

Since the IOMMU serialization is a strict dependency for device tracking, could we move the context table preservation directly into the device level op: intel_iommu_preserve_device()?

Whenever a specific device is preserved on-demand:

1. It queries the parent IOMMU to fetch the allocated context table backing its info->bus.

2. It calls iommu_preserve_page(context) for that table. Because KHO's tracking handles duplicates, this should be fine if multiple devices reside on the same bus...

Regarding Scalable Mode, we could just need a simple check in that path:

/* intel_iommu_preserve_device */
/* Preserve the primary/lower context table backing this bus */ context = iommu_context_addr(info->iommu, info->bus, 0, 0); if (context)

	iommu_preserve_page(context);
Quoted 8 lines

/* If scalable mode is active, preserve the upper context table as well */

if (sm_supported(info->iommu)) {
	context = iommu_context_addr(info->iommu, info->bus, 0x80, 0);
	if (context)
		iommu_preserve_page(context);
}

WDYT?

I was thinking on the same lines and was hinting to it in my earlier reply to Baolu. But then I didn't go for it as with this approach we introduce a bunch of complexity or lost state depending on the approach we take.

Basically like I mentioned, the unpreserved context tables need to be removed from root table during shutdown. We can find out which ones are unpreserved by walking the preserved device list yes. But it complicates the overall preservation lifecycle of these context tables by introducing multiple preserved device list walks. Basically on every device unpreserve we will have to decide that whether we unpreserve the context table or some preserved device is still using it. Or if we choose to keep it preserved even if the device is unpreserved, then during cleanup we have no way of differentiation whether a context table was unpreserved and needs clearing in root table.

The bitmap approach is much simpler and only preserves the context tables for the active devices as suggested by Baolu earlier. Note with this approach we preserve the context tables of the active devices and not only the preserved ones, but since these are recreated after kexec anyway, so not really a waste as compared to preserving the maximum possible.

Quoted 18 lines
Quoted 10 lines

I think we will have to do similar stuff for PASID also down the road to preserve pasid_tables in PASID directory.

Quoted 6 lines

Since we don't track the context_tables that are preserved, there is no way to incremently preserve the new-ones. Let me look into the behaviour of KHO, maybe we can make the preserve call idempotent and do these incrementally.

Quoted 1 line
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines

drivers/iommu/intel/liveupdate.c line 56

Quoted 7 lines
drivers/iommu/intel/liveupdate.c new file +7 -0
+		spin_lock(&iommu->lock);
+		context = iommu_context_addr(iommu, i, 0, 1);
+		spin_unlock(&iommu->lock);
+		if (!context) {
+			ret = -ENOMEM;
+			goto error;
+		}
Quoted 4 lines

[snip]

Thanks,
Praan

Sami

#48 lore raw

PATCH v2 8/16 iommu: Add APIs to get iommu and device preserved state

In reply to Samiullah Khawaja

The preserved state of the device and IOMMU needs to be fetched during shutdown and boot in the next kernel. Add APIs that can be used to fetch the preserved state of a device and IOMMU. The APIs will only be used during shutdown and after liveupdate so no locking needed.

---
 drivers/iommu/liveupdate.c       | 57 ++++++++++++++++++++++++++++++++
 include/linux/iommu-liveupdate.h | 31 +++++++++++++++++
 2 files changed, 88 insertions(+)
drivers/iommu/liveupdate.c +57 -0
index 765d042e22e3..60ee29b0c6bd 100644
@@ -17,6 +17,14 @@
 #define iommu_max_objs_per_page(_array) \
 	((PAGE_SIZE - sizeof(struct iommu_array_hdr_ser)) / sizeof((_array)->objects[0]))
 
+#define iommu_liveupdate_for_each_obj(_arr, _obj, _idx)				\
+	for (; (_arr);								\
+	     (_arr) = (_arr)->hdr.next_array_phys ?				\
+	     phys_to_virt((_arr)->hdr.next_array_phys) : NULL)			\
+		for ((_idx) = 0, (_obj) = (_arr)->objects;			\
+		     (_idx) < (_arr)->hdr.nr_objects; (_idx)++, (_obj)++)	\
+			if (!(_obj)->hdr.deleted)
+
 static void *iommu_liveupdate_restore_array(u64 array_phys)
 {
 	struct iommu_array_hdr_ser *array_hdr;
@@ -201,6 +209,55 @@ void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler)
 }
 EXPORT_SYMBOL(iommu_liveupdate_unregister_flb);
 
+int iommu_for_each_preserved_device(iommu_preserved_device_iter_fn fn,
+				    void *arg)
+{
+	struct iommu_flb_obj *flb_obj;
+	struct iommu_device_array_ser *array;
+	struct iommu_device_ser *device_ser;
+	int ret, idx;
+
+	ret = liveupdate_flb_get_incoming(&iommu_flb, (void **)&flb_obj);
+	if (ret)
+		return -ENOENT;
+
+	array = phys_to_virt(flb_obj->ser->device_array_phys);
+	iommu_liveupdate_for_each_obj(array, device_ser, idx) {
+		ret = fn(device_ser, arg);
+		if (ret)
+			goto out;
+	}
+
+out:
+	liveupdate_flb_put_incoming(&iommu_flb);
+	return ret;
+}
+EXPORT_SYMBOL(iommu_for_each_preserved_device);
+
+struct iommu_hw_ser *iommu_get_preserved_data(u64 token, enum iommu_type_ser type)
+{
+	struct iommu_hw_ser *iommu_ser = NULL;
+	struct iommu_hw_array_ser *array;
+	struct iommu_flb_obj *flb_obj;
+	int ret, idx;
+
+	ret = liveupdate_flb_get_incoming(&iommu_flb, (void **)&flb_obj);
+	if (ret)
+		return NULL;
+
+	array = phys_to_virt(flb_obj->ser->iommu_array_phys);
+	iommu_liveupdate_for_each_obj(array, iommu_ser, idx) {
+		if (iommu_ser->token == token && iommu_ser->type == type)
+			goto out;
+	}
+
+	iommu_ser = NULL;
+out:
+	liveupdate_flb_put_incoming(&iommu_flb);
+	return iommu_ser;
+}
+EXPORT_SYMBOL(iommu_get_preserved_data);
+
 static int alloc_object_ser(struct iommu_array_hdr_ser **curr_array_ptr, u64 max_objs)
 {
 	struct iommu_array_hdr_ser *curr_array = *curr_array_ptr;
include/linux/iommu-liveupdate.h +31 -0
index c9d75c6b3be9..0baf6bc2d93f 100644
@@ -13,6 +13,8 @@
 #include <linux/liveupdate.h>
 #include <linux/kho/abi/iommu.h>
 
+typedef int (*iommu_preserved_device_iter_fn)(struct iommu_device_ser *ser,
+					      void *arg);
 #ifdef CONFIG_IOMMU_LIVEUPDATE
 static inline void *dev_iommu_preserved_state(struct device *dev)
 {
@@ -28,6 +30,20 @@ static inline void *dev_iommu_preserved_state(struct device *dev)
 	return NULL;
 }
 
+static inline void *iommu_domain_restored_state(struct iommu_domain *domain)
+{
+	struct iommu_domain_ser *ser;
+
+	ser = domain->preserved_state;
+	if (ser && ser->hdr.incoming)
+		return ser;
+
+	return NULL;
+}
+
+int iommu_for_each_preserved_device(iommu_preserved_device_iter_fn fn,
+				    void *arg);
+struct iommu_hw_ser *iommu_get_preserved_data(u64 token, enum iommu_type_ser type);
 int iommu_domain_preserve(struct iommu_domain *domain, struct iommu_domain_ser **ser);
 void iommu_domain_unpreserve(struct iommu_domain *domain);
 int iommu_preserve_device(struct iommu_domain *domain,
@@ -44,6 +60,21 @@ static inline void *dev_iommu_preserved_state(struct device *dev)
 	return NULL;
 }
 
+static inline void *iommu_domain_restored_state(struct iommu_domain *domain)
+{
+	return NULL;
+}
+
+static inline int iommu_for_each_preserved_device(iommu_preserved_device_iter_fn fn, void *arg)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline struct iommu_hw_ser *iommu_get_preserved_data(u64 token, enum iommu_type_ser type)
+{
+	return NULL;
+}
+
 static inline int iommu_domain_preserve(struct iommu_domain *domain, struct iommu_domain_ser **ser)
 {
 	return -EOPNOTSUPP;
Signature · 1 line
2.54.0.545.g6539524ca2-goog
Reviewed-by on v2 8/16
#49 lore raw

Re: [PATCH v2 08/16] iommu: Add APIs to get iommu and device preserved state

Quoted 17 lines · Samiullah Khawaja

The preserved state of the device and IOMMU needs to be fetched during shutdown and boot in the next kernel. Add APIs that can be used to fetch the preserved state of a device and IOMMU. The APIs will only be used during shutdown and after liveupdate so no locking needed.

---
 drivers/iommu/liveupdate.c       | 57 ++++++++++++++++++++++++++++++++
 include/linux/iommu-liveupdate.h | 31 +++++++++++++++++
 2 files changed, 88 insertions(+)
drivers/iommu/liveupdate.c +0 -0
index 765d042e22e3..60ee29b0c6bd 100644
@@ -17,6 +17,14 @@
 #define iommu_max_objs_per_page(_array) \

drivers/iommu/liveupdate.c line 27

Quoted 10 lines
drivers/iommu/liveupdate.c +8 -0
 	((PAGE_SIZE - sizeof(struct iommu_array_hdr_ser)) / sizeof((_array)->objects[0]))
 
+#define iommu_liveupdate_for_each_obj(_arr, _obj, _idx)				\
+	for (; (_arr);								\
+	     (_arr) = (_arr)->hdr.next_array_phys ?				\
+	     phys_to_virt((_arr)->hdr.next_array_phys) : NULL)			\
+		for ((_idx) = 0, (_obj) = (_arr)->objects;			\
+		     (_idx) < (_arr)->hdr.nr_objects; (_idx)++, (_obj)++)	\
+			if (!(_obj)->hdr.deleted)
+

Nit: While the usage of this macro is safe right now because we use goto out; to short-circuit the scan. Such naked nested loops inside an iterator macro can be a bit of a trap for the future.

If anyone tries to reuse this macro down the road and uses a break; statement, it will only break the innermost object loop and continue scanning subsequent array blocks.

We could nest two macros or it as a helper fn to prevent such bugs.

Quoted 10 lines
drivers/iommu/liveupdate.c +3 -0
 static void *iommu_liveupdate_restore_array(u64 array_phys)
 {
 	struct iommu_array_hdr_ser *array_hdr;
@@ -201,6 +209,55 @@ void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler)
 }
 EXPORT_SYMBOL(iommu_liveupdate_unregister_flb);
 
+int iommu_for_each_preserved_device(iommu_preserved_device_iter_fn fn,
+				    void *arg)
+{

drivers/iommu/liveupdate.c void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler) · line 224

Quoted 10 lines
drivers/iommu/liveupdate.c +10 -0
+	struct iommu_flb_obj *flb_obj;
+	struct iommu_device_array_ser *array;
+	struct iommu_device_ser *device_ser;
+	int ret, idx;
+
+	ret = liveupdate_flb_get_incoming(&iommu_flb, (void **)&flb_obj);
+	if (ret)
+		return -ENOENT;
+
+	array = phys_to_virt(flb_obj->ser->device_array_phys);

Nit: phys_to_virt doesn't return an error for NULL. While we shouldn't be in a case where serialized ptrs are NULL. I'm wondering how painful catching such bugs would be.. should we check for NULL phys_addr here? (and at other places)

drivers/iommu/liveupdate.c void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler) · line 236

Quoted 12 lines
drivers/iommu/liveupdate.c +12 -0
+	iommu_liveupdate_for_each_obj(array, device_ser, idx) {
+		ret = fn(device_ser, arg);
+		if (ret)
+			goto out;
+	}
+
+out:
+	liveupdate_flb_put_incoming(&iommu_flb);
+	return ret;
+}
+EXPORT_SYMBOL(iommu_for_each_preserved_device);
+

With those two nits,

#50 lore raw

Re: [PATCH v2 08/16] iommu: Add APIs to get iommu and device preserved state

Quoted 33 lines · Pranjal Shrivastava
Quoted 17 lines · Samiullah Khawaja

The preserved state of the device and IOMMU needs to be fetched during shutdown and boot in the next kernel. Add APIs that can be used to fetch the preserved state of a device and IOMMU. The APIs will only be used during shutdown and after liveupdate so no locking needed.

---
 drivers/iommu/liveupdate.c       | 57 ++++++++++++++++++++++++++++++++
 include/linux/iommu-liveupdate.h | 31 +++++++++++++++++
 2 files changed, 88 insertions(+)
drivers/iommu/liveupdate.c +0 -0
index 765d042e22e3..60ee29b0c6bd 100644
@@ -17,6 +17,14 @@
 #define iommu_max_objs_per_page(_array) \

drivers/iommu/liveupdate.c line 27

Quoted 10 lines
drivers/iommu/liveupdate.c +8 -0
 	((PAGE_SIZE - sizeof(struct iommu_array_hdr_ser)) / sizeof((_array)->objects[0]))
 
+#define iommu_liveupdate_for_each_obj(_arr, _obj, _idx)				\
+	for (; (_arr);								\
+	     (_arr) = (_arr)->hdr.next_array_phys ?				\
+	     phys_to_virt((_arr)->hdr.next_array_phys) : NULL)			\
+		for ((_idx) = 0, (_obj) = (_arr)->objects;			\
+		     (_idx) < (_arr)->hdr.nr_objects; (_idx)++, (_obj)++)	\
+			if (!(_obj)->hdr.deleted)
+

Nit: While the usage of this macro is safe right now because we use goto out; to short-circuit the scan. Such naked nested loops inside an iterator macro can be a bit of a trap for the future.

Quoted 5 lines

If anyone tries to reuse this macro down the road and uses a break; statement, it will only break the innermost object loop and continue scanning subsequent array blocks.

We could nest two macros or it as a helper fn to prevent such bugs.

Agreed on all points. I will change this to two nested macros.

Quoted 19 lines
Quoted 19 lines
drivers/iommu/liveupdate.c +10 -0
 static void *iommu_liveupdate_restore_array(u64 array_phys)
 {
 	struct iommu_array_hdr_ser *array_hdr;
@@ -201,6 +209,55 @@ void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler)
 }
 EXPORT_SYMBOL(iommu_liveupdate_unregister_flb);
 
+int iommu_for_each_preserved_device(iommu_preserved_device_iter_fn fn,
+				    void *arg)
+{
+	struct iommu_flb_obj *flb_obj;
+	struct iommu_device_array_ser *array;
+	struct iommu_device_ser *device_ser;
+	int ret, idx;
+
+	ret = liveupdate_flb_get_incoming(&iommu_flb, (void **)&flb_obj);
+	if (ret)
Quoted 8 lines

drivers/iommu/liveupdate.c void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler) · line 224

Quoted 3 lines
drivers/iommu/liveupdate.c +3 -0
+		return -ENOENT;
+
+	array = phys_to_virt(flb_obj->ser->device_array_phys);

Nit: phys_to_virt doesn't return an error for NULL. While we shouldn't be in a case where serialized ptrs are NULL. I'm wondering how painful catching such bugs would be.. should we check for NULL phys_addr here? (and at other places)

Agreed. Will update.

Quoted 16 lines

drivers/iommu/liveupdate.c void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler) · line 236

Quoted 13 lines
drivers/iommu/liveupdate.c +12 -0
+	iommu_liveupdate_for_each_obj(array, device_ser, idx) {
+		ret = fn(device_ser, arg);
+		if (ret)
+			goto out;
+	}
+
+out:
+	liveupdate_flb_put_incoming(&iommu_flb);
+	return ret;
+}
+EXPORT_SYMBOL(iommu_for_each_preserved_device);
+

With those two nits,

Quoted 1 line

Thanks,
Sami

#51 lore raw

PATCH v2 9/16 iommu/vt-d: Restore IOMMU state and reclaimed domain ids

In reply to Samiullah Khawaja

During boot fetch the preserved state of IOMMU unit and if found then restore the state.

- Reuse the root_table that was preserved in the previous kernel.
- Reclaim the domain ids of the preserved domains for each preserved devices so these are not acquired by another domain.

---
 drivers/iommu/intel/iommu.c      | 55 ++++++++++++++++++++++--------
 drivers/iommu/intel/iommu.h      |  7 ++++
 drivers/iommu/intel/liveupdate.c | 57 ++++++++++++++++++++++++++++++++
 3 files changed, 105 insertions(+), 14 deletions(-)
drivers/iommu/intel/iommu.c +41 -14
index 68fecd4e57fa..4118a0861f38 100644
@@ -670,10 +670,17 @@ void dmar_fault_dump_ptes(struct intel_iommu *iommu, u16 source_id,
 #endif
 
 /* iommu handling */
-static int iommu_alloc_root_entry(struct intel_iommu *iommu)
+static int iommu_alloc_root_entry(struct intel_iommu *iommu,
+				  struct iommu_hw_ser *iommu_ser)
 {
 	struct root_entry *root;
 
+	if (iommu_ser) {
+		intel_iommu_liveupdate_restore_root_table(iommu, iommu_ser);
+		__iommu_flush_cache(iommu, iommu->root_entry, ROOT_SIZE);
+		return 0;
+	}
+
 	root = iommu_alloc_pages_node_sz(iommu->node, GFP_ATOMIC, SZ_4K);
 	if (!root) {
 		pr_err("Allocating root entry for %s failed\n",
@@ -992,15 +999,16 @@ static void disable_dmar_iommu(struct intel_iommu *iommu)
 		iommu_disable_translation(iommu);
 }
 
-static void free_dmar_iommu(struct intel_iommu *iommu)
+static void free_dmar_iommu(struct intel_iommu *iommu, struct iommu_hw_ser *iommu_ser)
 {
 	if (iommu->copied_tables) {
 		bitmap_free(iommu->copied_tables);
 		iommu->copied_tables = NULL;
 	}
 
-	/* free context mapping */
-	free_context_table(iommu);
+	/* free context mapping if there is no serialized state. */
+	if (!iommu_ser)
+		free_context_table(iommu);
 
 	if (ecap_prs(iommu->ecap))
 		intel_iommu_finish_prq(iommu);
@@ -1611,6 +1619,7 @@ static int copy_translation_tables(struct intel_iommu *iommu)
 
 static int __init init_dmars(void)
 {
+	struct iommu_hw_ser *iommu_ser = NULL;
 	struct dmar_drhd_unit *drhd;
 	struct intel_iommu *iommu;
 	int ret;
@@ -1633,8 +1642,12 @@ static int __init init_dmars(void)
 						   intel_pasid_max_id);
 		}
 
+		iommu_ser = iommu_get_preserved_data(iommu->reg_phys, IOMMU_INTEL);
+
 		intel_iommu_init_qi(iommu);
-		init_translation_status(iommu);
+
+		if (!iommu_ser)
+			init_translation_status(iommu);
 
 		if (translation_pre_enabled(iommu) && !is_kdump_kernel()) {
 			iommu_disable_translation(iommu);
@@ -1648,7 +1661,7 @@ static int __init init_dmars(void)
 		 * we could share the same root & context tables
 		 * among all IOMMU's. Need to Split it later.
 		 */
-		ret = iommu_alloc_root_entry(iommu);
+		ret = iommu_alloc_root_entry(iommu, iommu_ser);
 		if (ret)
 			goto free_iommu;
 
@@ -1732,8 +1745,12 @@ static int __init init_dmars(void)
 
 free_iommu:
 	for_each_active_iommu(iommu, drhd) {
-		disable_dmar_iommu(iommu);
-		free_dmar_iommu(iommu);
+		iommu_ser = iommu_get_preserved_data(iommu->reg_phys, IOMMU_INTEL);
+
+		if (!iommu_ser)
+			disable_dmar_iommu(iommu);
+
+		free_dmar_iommu(iommu, iommu_ser);
 	}
 
 	return ret;
@@ -2107,15 +2124,19 @@ int dmar_parse_one_satc(struct acpi_dmar_header *hdr, void *arg)
 static int intel_iommu_add(struct dmar_drhd_unit *dmaru)
 {
 	struct intel_iommu *iommu = dmaru->iommu;
+	struct iommu_hw_ser *iommu_ser = NULL;
 	int ret;
 
+	/* Use IOMMU HW unit MMIO base to identify the preserved state. */
+	iommu_ser = iommu_get_preserved_data(iommu->reg_phys, IOMMU_INTEL);
+
 	/*
 	 * Disable translation if already enabled prior to OS handover.
 	 */
-	if (iommu->gcmd & DMA_GCMD_TE)
+	if (!iommu_ser && iommu->gcmd & DMA_GCMD_TE)
 		iommu_disable_translation(iommu);
 
-	ret = iommu_alloc_root_entry(iommu);
+	ret = iommu_alloc_root_entry(iommu, iommu_ser);
 	if (ret)
 		goto out;
 
@@ -2150,9 +2171,10 @@ static int intel_iommu_add(struct dmar_drhd_unit *dmaru)
 	return 0;
 
 disable_iommu:
-	disable_dmar_iommu(iommu);
+	if (!iommu_ser)
+		disable_dmar_iommu(iommu);
 out:
-	free_dmar_iommu(iommu);
+	free_dmar_iommu(iommu, iommu_ser);
 	return ret;
 }
 
@@ -2160,6 +2182,7 @@ int dmar_iommu_hotplug(struct dmar_drhd_unit *dmaru, bool insert)
 {
 	int ret = 0;
 	struct intel_iommu *iommu = dmaru->iommu;
+	struct iommu_hw_ser *iommu_ser;
 
 	if (!intel_iommu_enabled)
 		return 0;
@@ -2169,8 +2192,12 @@ int dmar_iommu_hotplug(struct dmar_drhd_unit *dmaru, bool insert)
 	if (insert) {
 		ret = intel_iommu_add(dmaru);
 	} else {
-		disable_dmar_iommu(iommu);
-		free_dmar_iommu(iommu);
+		iommu_ser = iommu_get_preserved_data(iommu->reg_phys, IOMMU_INTEL);
+
+		if (!iommu_ser)
+			disable_dmar_iommu(iommu);
+
+		free_dmar_iommu(iommu, iommu_ser);
 	}
 
 	return ret;
drivers/iommu/intel/iommu.h +7 -0
index 5e0bc17e76bf..b0ec0b471a43 100644
@@ -1293,6 +1293,8 @@ int intel_iommu_preserve(struct iommu_device *iommu,
 			 struct iommu_hw_ser *iommu_ser);
 void intel_iommu_unpreserve(struct iommu_device *iommu,
 			    struct iommu_hw_ser *iommu_ser);
+void intel_iommu_liveupdate_restore_root_table(struct intel_iommu *iommu,
+					       struct iommu_hw_ser *iommu_ser);
 #else
 static inline int intel_iommu_preserve_device(struct device *dev,
 					      struct iommu_device_ser *device_ser)
@@ -1310,6 +1312,11 @@ static inline void intel_iommu_unpreserve(struct iommu_device *iommu,
 					  struct iommu_hw_ser *iommu_ser)
 {
 }
+
+static inline void intel_iommu_liveupdate_restore_root_table(struct intel_iommu *iommu,
+							     struct iommu_hw_ser *iommu_ser)
+{
+}
 #endif
 
 #ifdef CONFIG_INTEL_IOMMU_SVM
drivers/iommu/intel/liveupdate.c +57 -0

63 lines not on this page · raw message

Signature · 1 line
2.54.0.545.g6539524ca2-goog
maintainer
#52 lore raw

Re: [PATCH v2 09/16] iommu/vt-d: Restore IOMMU state and reclaimed domain ids

Quoted 21 lines · Samiullah Khawaja

During boot fetch the preserved state of IOMMU unit and if found then restore the state.

- Reuse the root_table that was preserved in the previous kernel.
- Reclaim the domain ids of the preserved domains for each preserved devices so these are not acquired by another domain.

---
  drivers/iommu/intel/iommu.c      | 55 ++++++++++++++++++++++--------
  drivers/iommu/intel/iommu.h      |  7 ++++
  drivers/iommu/intel/liveupdate.c | 57 ++++++++++++++++++++++++++++++++
  3 files changed, 105 insertions(+), 14 deletions(-)
drivers/iommu/intel/iommu.c +0 -0
index 68fecd4e57fa..4118a0861f38 100644
@@ -670,10 +670,17 @@ void dmar_fault_dump_ptes(struct intel_iommu *iommu, u16 source_id,
  #endif
 

drivers/iommu/intel/iommu.c void dmar_fault_dump_ptes(struct intel_iommu *iommu, u16 source_id, · line 680

Quoted 10 lines
drivers/iommu/intel/iommu.c +5 -1
  /* iommu handling */
-static int iommu_alloc_root_entry(struct intel_iommu *iommu)
+static int iommu_alloc_root_entry(struct intel_iommu *iommu,
+				  struct iommu_hw_ser *iommu_ser)
  {
  	struct root_entry *root;
 
+	if (iommu_ser) {
+		intel_iommu_liveupdate_restore_root_table(iommu, iommu_ser);
+		__iommu_flush_cache(iommu, iommu->root_entry, ROOT_SIZE);

The root table is already active and being used by the hardware at this point, correct? Since the CPU should not be modifying an active table, the __iommu_flush_cache() call seems unnecessary and potentially risky.

Furthermore, if the iommu state has been restored, is it necessary to call iommu_set_root_entry()? Calling it would essentially replace an active, in-use root table with an identical one, which appears redundant.

drivers/iommu/intel/iommu.c void dmar_fault_dump_ptes(struct intel_iommu *iommu, u16 source_id, · line 686

Quoted 6 lines
drivers/iommu/intel/iommu.c +3 -0
+		return 0;
+	}
+
  	root = iommu_alloc_pages_node_sz(iommu->node, GFP_ATOMIC, SZ_4K);
  	if (!root) {
  		pr_err("Allocating root entry for %s failed\n",

Thanks,
baolu

#53 lore raw

Re: [PATCH v2 09/16] iommu/vt-d: Restore IOMMU state and reclaimed domain ids

Quoted 35 lines · Baolu Lu
Quoted 19 lines · Samiullah Khawaja

During boot fetch the preserved state of IOMMU unit and if found then restore the state.

- Reuse the root_table that was preserved in the previous kernel.
- Reclaim the domain ids of the preserved domains for each preserved devices so these are not acquired by another domain.

---
 drivers/iommu/intel/iommu.c      | 55 ++++++++++++++++++++++--------
 drivers/iommu/intel/iommu.h      |  7 ++++
 drivers/iommu/intel/liveupdate.c | 57 ++++++++++++++++++++++++++++++++
 3 files changed, 105 insertions(+), 14 deletions(-)
drivers/iommu/intel/iommu.c +0 -0
index 68fecd4e57fa..4118a0861f38 100644

drivers/iommu/intel/iommu.c void dmar_fault_dump_ptes(struct intel_iommu *iommu, u16 source_id, · line 678

Quoted 10 lines
drivers/iommu/intel/iommu.c +5 -1
@@ -670,10 +670,17 @@ void dmar_fault_dump_ptes(struct intel_iommu *iommu, u16 source_id,
 #endif
 /* iommu handling */
-static int iommu_alloc_root_entry(struct intel_iommu *iommu)
+static int iommu_alloc_root_entry(struct intel_iommu *iommu,
+				  struct iommu_hw_ser *iommu_ser)
 {
 	struct root_entry *root;
+	if (iommu_ser) {
+		intel_iommu_liveupdate_restore_root_table(iommu, iommu_ser);
+		__iommu_flush_cache(iommu, iommu->root_entry, ROOT_SIZE);

The root table is already active and being used by the hardware at this point, correct? Since the CPU should not be modifying an active table, the __iommu_flush_cache() call seems unnecessary and potentially risky.

Quoted 4 lines

Furthermore, if the iommu state has been restored, is it necessary to call iommu_set_root_entry()? Calling it would essentially replace an active, in-use root table with an identical one, which appears redundant.

Agreed to both points. I will remove the redundant flush here.

The set_root_entry() is done outside this function in a separate loop after setup of all iommus, I will check whether the iommu was preserved there and skip set_root_entry().

Quoted 10 lines

drivers/iommu/intel/iommu.c void dmar_fault_dump_ptes(struct intel_iommu *iommu, u16 source_id, · line 686

Quoted 7 lines
drivers/iommu/intel/iommu.c +3 -0
+		return 0;
+	}
+
 	root = iommu_alloc_pages_node_sz(iommu->node, GFP_ATOMIC, SZ_4K);
 	if (!root) {
 		pr_err("Allocating root entry for %s failed\n",

Thanks,
baolu

#54 lore raw

Re: [PATCH v2 09/16] iommu/vt-d: Restore IOMMU state and reclaimed domain ids

In reply to Samiullah Khawaja
Quoted 24 lines · Samiullah Khawaja

During boot fetch the preserved state of IOMMU unit and if found then restore the state.

- Reuse the root_table that was preserved in the previous kernel.
- Reclaim the domain ids of the preserved domains for each preserved devices so these are not acquired by another domain.

---
 drivers/iommu/intel/iommu.c      | 55 ++++++++++++++++++++++--------
 drivers/iommu/intel/iommu.h      |  7 ++++
 drivers/iommu/intel/liveupdate.c | 57 ++++++++++++++++++++++++++++++++
 3 files changed, 105 insertions(+), 14 deletions(-)
drivers/iommu/intel/iommu.c +1 -1
index 68fecd4e57fa..4118a0861f38 100644
@@ -670,10 +670,17 @@ void dmar_fault_dump_ptes(struct intel_iommu *iommu, u16 source_id,
 #endif
 
 /* iommu handling */
-static int iommu_alloc_root_entry(struct intel_iommu *iommu)
+static int iommu_alloc_root_entry(struct intel_iommu *iommu,

drivers/iommu/intel/iommu.c void dmar_fault_dump_ptes(struct intel_iommu *iommu, u16 source_id, · line 683

Quoted 10 lines
drivers/iommu/intel/iommu.c +7 -0
+				  struct iommu_hw_ser *iommu_ser)
 {
 	struct root_entry *root;
 
+	if (iommu_ser) {
+		intel_iommu_liveupdate_restore_root_table(iommu, iommu_ser);
+		__iommu_flush_cache(iommu, iommu->root_entry, ROOT_SIZE);
+		return 0;
+	}
+

Minor nit: I still believe this condition block can be moved into the caller? Since the called fetches iommu_ser, it can call this as a stand alone helper and bypass calling iommu_alloc_root_entry if (iommu_ser).

Quoted 58 lines
drivers/iommu/intel/iommu.c +11 -5
 	root = iommu_alloc_pages_node_sz(iommu->node, GFP_ATOMIC, SZ_4K);
 	if (!root) {
 		pr_err("Allocating root entry for %s failed\n",
@@ -992,15 +999,16 @@ static void disable_dmar_iommu(struct intel_iommu *iommu)
 		iommu_disable_translation(iommu);
 }
 
-static void free_dmar_iommu(struct intel_iommu *iommu)
+static void free_dmar_iommu(struct intel_iommu *iommu, struct iommu_hw_ser *iommu_ser)
 {
 	if (iommu->copied_tables) {
 		bitmap_free(iommu->copied_tables);
 		iommu->copied_tables = NULL;
 	}
 
-	/* free context mapping */
-	free_context_table(iommu);
+	/* free context mapping if there is no serialized state. */
+	if (!iommu_ser)
+		free_context_table(iommu);
 
 	if (ecap_prs(iommu->ecap))
 		intel_iommu_finish_prq(iommu);
@@ -1611,6 +1619,7 @@ static int copy_translation_tables(struct intel_iommu *iommu)
 
 static int __init init_dmars(void)
 {
+	struct iommu_hw_ser *iommu_ser = NULL;
 	struct dmar_drhd_unit *drhd;
 	struct intel_iommu *iommu;
 	int ret;
@@ -1633,8 +1642,12 @@ static int __init init_dmars(void)
 						   intel_pasid_max_id);
 		}
 
+		iommu_ser = iommu_get_preserved_data(iommu->reg_phys, IOMMU_INTEL);
+
 		intel_iommu_init_qi(iommu);
-		init_translation_status(iommu);
+
+		if (!iommu_ser)
+			init_translation_status(iommu);
 
 		if (translation_pre_enabled(iommu) && !is_kdump_kernel()) {
 			iommu_disable_translation(iommu);
@@ -1648,7 +1661,7 @@ static int __init init_dmars(void)
 		 * we could share the same root & context tables
 		 * among all IOMMU's. Need to Split it later.
 		 */
-		ret = iommu_alloc_root_entry(iommu);
+		ret = iommu_alloc_root_entry(iommu, iommu_ser);
 		if (ret)
 			goto free_iommu;
 
@@ -1732,8 +1745,12 @@ static int __init init_dmars(void)
 
 free_iommu:
 	for_each_active_iommu(iommu, drhd) {

drivers/iommu/intel/iommu.c static int __init init_dmars(void) · line 1755

Quoted 10 lines
drivers/iommu/intel/iommu.c +6 -2
-		disable_dmar_iommu(iommu);
-		free_dmar_iommu(iommu);
+		iommu_ser = iommu_get_preserved_data(iommu->reg_phys, IOMMU_INTEL);
+
+		if (!iommu_ser)
+			disable_dmar_iommu(iommu);
+
+		free_dmar_iommu(iommu, iommu_ser);
 	}
 

I'm wondering what happens if init_dmars fails for a preserved iommu? Since we have non NULL iommu_ser, we'll skip disable_dmar_iommu and skip free_context_table() inside free_dmar_iommu() as well. I'm concerned we might leak some resources in this case. If the failure happens after we set restored = 1, even a rmmod -> modprobe cycle won't help fix the leak

drivers/iommu/intel/iommu.c int dmar_parse_one_satc(struct acpi_dmar_header *hdr, void *arg) · line 2131

Quoted 10 lines
drivers/iommu/intel/iommu.c +3 -0
 	return ret;
@@ -2107,15 +2124,19 @@ int dmar_parse_one_satc(struct acpi_dmar_header *hdr, void *arg)
 static int intel_iommu_add(struct dmar_drhd_unit *dmaru)
 {
 	struct intel_iommu *iommu = dmaru->iommu;
+	struct iommu_hw_ser *iommu_ser = NULL;
 	int ret;
 
+	/* Use IOMMU HW unit MMIO base to identify the preserved state. */
+	iommu_ser = iommu_get_preserved_data(iommu->reg_phys, IOMMU_INTEL);

Thanks for adding that comment! Really clever btw :)

Quoted 13 lines
drivers/iommu/intel/iommu.c +3 -2
+
 	/*
 	 * Disable translation if already enabled prior to OS handover.
 	 */
-	if (iommu->gcmd & DMA_GCMD_TE)
+	if (!iommu_ser && iommu->gcmd & DMA_GCMD_TE)
 		iommu_disable_translation(iommu);
 
-	ret = iommu_alloc_root_entry(iommu);
+	ret = iommu_alloc_root_entry(iommu, iommu_ser);
 	if (ret)
 		goto out;
 

[snip]

drivers/iommu/intel/iommu.c

Quoted 8 lines
drivers/iommu/intel/iommu.c +8 -0
+
+static int _restore_used_domain_ids(struct iommu_device_ser *ser, void *arg)
+{
+	int id = ser->domain_iommu_ser.attachment_id;
+	struct iommu_hw_ser *iommu_hw_ser;
+	struct intel_iommu *iommu = arg;
+
+	iommu_hw_ser = phys_to_virt(ser->domain_iommu_ser.iommu_phys);

We should check for iommu_phys being NULL here.. I know corruptions can be funnier but a WARN_ON(!iommu_phys) could help catch NULL corruptions

Quoted 6 lines
drivers/iommu/intel/iommu.c +6 -0
+	if (iommu_hw_ser->type != IOMMU_INTEL)
+		return 0;
+
+	/* Only allocate domain ID from associated IOMMU HW unit */
+	if (iommu_hw_ser->intel.phys_addr != iommu->reg_phys)
+		return 0;

drivers/iommu/intel/iommu.c

Quoted 10 lines
drivers/iommu/intel/iommu.c +10 -0
+
+	/*
+	 * This can fail as multiple preserved devices can share the same domain
+	 * ID. Since this is done during DMAR init so these failures can be
+	 * ignored.
+	 */
+	ida_alloc_range(&iommu->domain_ida, id, id, GFP_ATOMIC);
+	return 0;
+}
+

Thanks,
Praan

#55 lore raw

Re: [PATCH v2 09/16] iommu/vt-d: Restore IOMMU state and reclaimed domain ids

Quoted 38 lines · Pranjal Shrivastava
Quoted 37 lines · Samiullah Khawaja

During boot fetch the preserved state of IOMMU unit and if found then restore the state.

- Reuse the root_table that was preserved in the previous kernel.
- Reclaim the domain ids of the preserved domains for each preserved devices so these are not acquired by another domain.

---
 drivers/iommu/intel/iommu.c      | 55 ++++++++++++++++++++++--------
 drivers/iommu/intel/iommu.h      |  7 ++++
 drivers/iommu/intel/liveupdate.c | 57 ++++++++++++++++++++++++++++++++
 3 files changed, 105 insertions(+), 14 deletions(-)
drivers/iommu/intel/iommu.c +7 -3
 		if (translation_pre_enabled(iommu) && !is_kdump_kernel()) {
 			iommu_disable_translation(iommu);
@@ -1648,7 +1661,7 @@ static int __init init_dmars(void)
 		 * we could share the same root & context tables
 		 * among all IOMMU's. Need to Split it later.
 		 */
-		ret = iommu_alloc_root_entry(iommu);
+		ret = iommu_alloc_root_entry(iommu, iommu_ser);
 		if (ret)
 			goto free_iommu;
 
@@ -1732,8 +1745,12 @@ static int __init init_dmars(void)
 
 free_iommu:
 	for_each_active_iommu(iommu, drhd) {
-		disable_dmar_iommu(iommu);
-		free_dmar_iommu(iommu);
+		iommu_ser = iommu_get_preserved_data(iommu->reg_phys, IOMMU_INTEL);
+
+		if (!iommu_ser)
+			disable_dmar_iommu(iommu);
+
+		free_dmar_iommu(iommu, iommu_ser);
Quoted 9 lines

Review static int __init init_dmars(void) · line 1755

Quoted 2 lines
(quoted diff) +0 -0
 	}
 

I'm wondering what happens if init_dmars fails for a preserved iommu? Since we have non NULL iommu_ser, we'll skip disable_dmar_iommu and skip free_context_table() inside free_dmar_iommu() as well. I'm concerned we might leak some resources in this case. If the failure happens after we set restored = 1, even a rmmod -> modprobe cycle won't help fix the leak

Re-visiting this, I think we don't need to worry as skipping restore is the intended behaviour.. we don't wanna attempt restore on things that have already been restored. Please ignore this

Thanks,
Praan

#56 lore raw

Re: [PATCH v2 09/16] iommu/vt-d: Restore IOMMU state and reclaimed domain ids

In reply to Pranjal Shrivastava
Quoted 29 lines · Pranjal Shrivastava
Quoted 28 lines · Samiullah Khawaja

During boot fetch the preserved state of IOMMU unit and if found then restore the state.

- Reuse the root_table that was preserved in the previous kernel.
- Reclaim the domain ids of the preserved domains for each preserved devices so these are not acquired by another domain.

---
 drivers/iommu/intel/iommu.c      | 55 ++++++++++++++++++++++--------
 drivers/iommu/intel/iommu.h      |  7 ++++
 drivers/iommu/intel/liveupdate.c | 57 ++++++++++++++++++++++++++++++++
 3 files changed, 105 insertions(+), 14 deletions(-)
drivers/iommu/intel/iommu.c +2 -1
index 68fecd4e57fa..4118a0861f38 100644
@@ -670,10 +670,17 @@ void dmar_fault_dump_ptes(struct intel_iommu *iommu, u16 source_id,
 #endif
 
 /* iommu handling */
-static int iommu_alloc_root_entry(struct intel_iommu *iommu)
+static int iommu_alloc_root_entry(struct intel_iommu *iommu,
+				  struct iommu_hw_ser *iommu_ser)
 {
 	struct root_entry *root;
 
Quoted 10 lines

drivers/iommu/intel/iommu.c void dmar_fault_dump_ptes(struct intel_iommu *iommu, u16 source_id, · line 683

Quoted 6 lines
drivers/iommu/intel/iommu.c +6 -0
+	if (iommu_ser) {
+		intel_iommu_liveupdate_restore_root_table(iommu, iommu_ser);
+		__iommu_flush_cache(iommu, iommu->root_entry, ROOT_SIZE);
+		return 0;
+	}
+

Minor nit: I still believe this condition block can be moved into the caller? Since the called fetches iommu_ser, it can call this as a stand alone helper and bypass calling iommu_alloc_root_entry if (iommu_ser).

Agreed. Also as Baolu suggested the cache flush is not needed, so we can move it out as it is self-contained. Will update this.

Quoted 8 lines
Quoted 8 lines
drivers/iommu/intel/iommu.c +0 -0
 	root = iommu_alloc_pages_node_sz(iommu->node, GFP_ATOMIC, SZ_4K);
 	if (!root) {
 		pr_err("Allocating root entry for %s failed\n",
@@ -992,15 +999,16 @@ static void disable_dmar_iommu(struct intel_iommu *iommu)
 		iommu_disable_translation(iommu);
 }
 

[snip]

Quoted 12 lines

drivers/iommu/intel/iommu.c

Quoted 9 lines
drivers/iommu/intel/iommu.c +8 -0
+
+static int _restore_used_domain_ids(struct iommu_device_ser *ser, void *arg)
+{
+	int id = ser->domain_iommu_ser.attachment_id;
+	struct iommu_hw_ser *iommu_hw_ser;
+	struct intel_iommu *iommu = arg;
+
+	iommu_hw_ser = phys_to_virt(ser->domain_iommu_ser.iommu_phys);

We should check for iommu_phys being NULL here.. I know corruptions can be funnier but a WARN_ON(!iommu_phys) could help catch NULL corruptions

Agreed. I will add a WARN_ON here.

Quoted 12 lines
Quoted 12 lines
drivers/iommu/intel/iommu.c +11 -0
+	if (iommu_hw_ser->type != IOMMU_INTEL)
+		return 0;
+
+	/* Only allocate domain ID from associated IOMMU HW unit */
+	if (iommu_hw_ser->intel.phys_addr != iommu->reg_phys)
+		return 0;
+
+	/*
+	 * This can fail as multiple preserved devices can share the same domain
+	 * ID. Since this is done during DMAR init so these failures can be
+	 * ignored.
Quoted 8 lines

drivers/iommu/intel/iommu.c

Quoted 5 lines
drivers/iommu/intel/iommu.c +5 -0
+	 */
+	ida_alloc_range(&iommu->domain_ida, id, id, GFP_ATOMIC);
+	return 0;
+}
+

Thanks,
Praan

Sami

#57 lore raw

PATCH v2 10/16 iommu: Restore and reattach preserved domains to devices

In reply to Samiullah Khawaja

Restore the preserved domains by restoring the page tables using restore IOMMU domain op. Reattach the preserved domain to the device during default domain setup. While attaching, reuse the domain ID that was used in the previous kernel. The context entry setup is not needed as that is preserved during liveupdate.

---
 drivers/iommu/intel/iommu.c      | 49 ++++++++++++++------
 drivers/iommu/intel/iommu.h      |  3 +-
 drivers/iommu/intel/nested.c     |  2 +-
 drivers/iommu/iommu.c            | 61 ++++++++++++++++++++++++-
 drivers/iommu/liveupdate.c       | 78 ++++++++++++++++++++++++++++++++
 include/linux/iommu-liveupdate.h | 50 ++++++++++++++++++++
 6 files changed, 224 insertions(+), 19 deletions(-)
drivers/iommu/intel/iommu.c +34 -15
index 4118a0861f38..b90757164cd8 100644
@@ -1031,7 +1031,8 @@ static bool first_level_by_default(struct intel_iommu *iommu)
 	return true;
 }
 
-int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu)
+int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu,
+			int restore_did)
 {
 	struct iommu_domain_info *info, *curr;
 	int num, ret = -ENOSPC;
@@ -1051,8 +1052,11 @@ int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu)
 		return 0;
 	}
 
-	num = ida_alloc_range(&iommu->domain_ida, IDA_START_DID,
-			      cap_ndoms(iommu->cap) - 1, GFP_KERNEL);
+	if (restore_did >= IDA_START_DID)
+		num = restore_did;
+	else
+		num = ida_alloc_range(&iommu->domain_ida, IDA_START_DID,
+				      cap_ndoms(iommu->cap) - 1, GFP_KERNEL);
 	if (num < 0) {
 		pr_err("%s: No free domain ids\n", iommu->name);
 		goto err_unlock;
@@ -1320,10 +1324,14 @@ static int dmar_domain_attach_device(struct dmar_domain *domain,
 {
 	struct device_domain_info *info = dev_iommu_priv_get(dev);
 	struct intel_iommu *iommu = info->iommu;
+	struct device_ser *device_ser = NULL;
 	unsigned long flags;
 	int ret;
 
-	ret = domain_attach_iommu(domain, iommu);
+	device_ser = dev_iommu_restored_state(dev);
+
+	ret = domain_attach_iommu(domain, iommu,
+				  dev_iommu_restore_did(dev, &domain->domain));
 	if (ret)
 		return ret;
 
@@ -1336,16 +1344,18 @@ static int dmar_domain_attach_device(struct dmar_domain *domain,
 	if (dev_is_real_dma_subdevice(dev))
 		return 0;
 
-	if (!sm_supported(iommu))
-		ret = domain_context_mapping(domain, dev);
-	else if (intel_domain_is_fs_paging(domain))
-		ret = domain_setup_first_level(iommu, domain, dev,
-					       IOMMU_NO_PASID, NULL);
-	else if (intel_domain_is_ss_paging(domain))
-		ret = domain_setup_second_level(iommu, domain, dev,
-						IOMMU_NO_PASID, NULL);
-	else if (WARN_ON(true))
-		ret = -EINVAL;
+	if (!device_ser) {
+		if (!sm_supported(iommu))
+			ret = domain_context_mapping(domain, dev);
+		else if (intel_domain_is_fs_paging(domain))
+			ret = domain_setup_first_level(iommu, domain, dev,
+						       IOMMU_NO_PASID, NULL);
+		else if (intel_domain_is_ss_paging(domain))
+			ret = domain_setup_second_level(iommu, domain, dev,
+							IOMMU_NO_PASID, NULL);
+		else if (WARN_ON(true))
+			ret = -EINVAL;
+	}
 
 	if (ret)
 		goto out_block_translation;
@@ -3170,6 +3180,15 @@ int paging_domain_compatible(struct iommu_domain *domain, struct device *dev)
 	struct intel_iommu *iommu = info->iommu;
 	int ret = -EINVAL;
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	/*
+	 * Restored IOMMU domains are already attached to the device and can
+	 * only be freed. So no need to check the compatibility.
+	 */
+	if (iommu_domain_restored_state(domain))
+		return 0;
+#endif
+
 	if (intel_domain_is_fs_paging(dmar_domain))
 		ret = paging_domain_compatible_first_stage(dmar_domain, iommu);
 	else if (intel_domain_is_ss_paging(dmar_domain))
@@ -3647,7 +3666,7 @@ domain_add_dev_pasid(struct iommu_domain *domain,
 	if (!dev_pasid)
 		return ERR_PTR(-ENOMEM);
 
-	ret = domain_attach_iommu(dmar_domain, iommu);
+	ret = domain_attach_iommu(dmar_domain, iommu, -1);
 	if (ret)
 		goto out_free;
 
drivers/iommu/intel/iommu.h +2 -1
index b0ec0b471a43..8e37acf7de12 100644
@@ -1182,7 +1182,8 @@ void __iommu_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,
  */
 #define QI_OPT_WAIT_DRAIN		BIT(0)
 
-int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu);
+int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu,
+			int restore_did);
 void domain_detach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu);
 void device_block_translation(struct device *dev);
 int paging_domain_compatible(struct iommu_domain *domain, struct device *dev);
drivers/iommu/intel/nested.c +1 -1
index 2b979bec56ce..6e13f697b463 100644
@@ -40,7 +40,7 @@ static int intel_nested_attach_dev(struct iommu_domain *domain,
 		return ret;
 	}
 
-	ret = domain_attach_iommu(dmar_domain, iommu);
+	ret = domain_attach_iommu(dmar_domain, iommu, -1);
 	if (ret) {
 		dev_err_ratelimited(dev, "Failed to attach domain to iommu\n");
 		return ret;
drivers/iommu/iommu.c +59 -2

104 lines not on this page · raw message

drivers/iommu/liveupdate.c +78 -0

87 lines not on this page · raw message

include/linux/iommu-liveupdate.h +50 -0

80 lines not on this page · raw message

Signature · 1 line
2.54.0.545.g6539524ca2-goog
maintainer
#58 lore raw

Re: [PATCH v2 10/16] iommu: Restore and reattach preserved domains to devices

Quoted 7 lines · Samiullah Khawaja

Restore the preserved domains by restoring the page tables using restore IOMMU domain op. Reattach the preserved domain to the device during default domain setup. While attaching, reuse the domain ID that was used in the previous kernel. The context entry setup is not needed as that is preserved during liveupdate.

Quoted 8 lines
---
  drivers/iommu/intel/iommu.c      | 49 ++++++++++++++------
  drivers/iommu/intel/iommu.h      |  3 +-
  drivers/iommu/intel/nested.c     |  2 +-
  drivers/iommu/iommu.c            | 61 ++++++++++++++++++++++++-
  drivers/iommu/liveupdate.c       | 78 ++++++++++++++++++++++++++++++++
  include/linux/iommu-liveupdate.h | 50 ++++++++++++++++++++
  6 files changed, 224 insertions(+), 19 deletions(-)

Please split the changes in the Intel iommu driver and the iommu core into different patches.

drivers/iommu/intel/iommu.c static bool first_level_by_default(struct intel_iommu *iommu) · line 1035

Quoted 12 lines
drivers/iommu/intel/iommu.c +2 -1
index 4118a0861f38..b90757164cd8 100644
@@ -1031,7 +1031,8 @@ static bool first_level_by_default(struct intel_iommu *iommu)
  	return true;
  }
 
-int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu)
+int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu,
+			int restore_did)

How about using a new helper for restored domain? For example,

int domain_reattach_iommu(...)

or

int domain_restore_iommu(...)

drivers/iommu/intel/iommu.c int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu) · line 1056

Quoted 11 lines
drivers/iommu/intel/iommu.c +2 -2
  {
  	struct iommu_domain_info *info, *curr;
  	int num, ret = -ENOSPC;
@@ -1051,8 +1052,11 @@ int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu)
  		return 0;
  	}
 
-	num = ida_alloc_range(&iommu->domain_ida, IDA_START_DID,
-			      cap_ndoms(iommu->cap) - 1, GFP_KERNEL);
+	if (restore_did >= IDA_START_DID)
+		num = restore_did;

For a preserved domain ID, do we need to check whether it has been reserved from the ida pool?

Quoted 9 lines
drivers/iommu/intel/iommu.c +3 -0
+	else
+		num = ida_alloc_range(&iommu->domain_ida, IDA_START_DID,
+				      cap_ndoms(iommu->cap) - 1, GFP_KERNEL);
  	if (num < 0) {
  		pr_err("%s: No free domain ids\n", iommu->name);
  		goto err_unlock;
@@ -1320,10 +1324,14 @@ static int dmar_domain_attach_device(struct dmar_domain *domain,
  {
  	struct device_domain_info *info = dev_iommu_priv_get(dev);

drivers/iommu/intel/iommu.c static int dmar_domain_attach_device(struct dmar_domain *domain, · line 1334

Quoted 10 lines
drivers/iommu/intel/iommu.c +5 -1
  	struct intel_iommu *iommu = info->iommu;
+	struct device_ser *device_ser = NULL;
  	unsigned long flags;
  	int ret;
 
-	ret = domain_attach_iommu(domain, iommu);
+	device_ser = dev_iommu_restored_state(dev);
+
+	ret = domain_attach_iommu(domain, iommu,
+				  dev_iommu_restore_did(dev, &domain->domain));

What is the expected behavior if there is a mismatch between the restored state and the availability of a domain ID? Specifically, if device_ser is found but dev_iommu_restore_did() fails (returns -1), how should the driver proceed in that case?

Quoted 76 lines
drivers/iommu/intel/iommu.c +22 -11
  	if (ret)
  		return ret;
 
@@ -1336,16 +1344,18 @@ static int dmar_domain_attach_device(struct dmar_domain *domain,
  	if (dev_is_real_dma_subdevice(dev))
  		return 0;
 
-	if (!sm_supported(iommu))
-		ret = domain_context_mapping(domain, dev);
-	else if (intel_domain_is_fs_paging(domain))
-		ret = domain_setup_first_level(iommu, domain, dev,
-					       IOMMU_NO_PASID, NULL);
-	else if (intel_domain_is_ss_paging(domain))
-		ret = domain_setup_second_level(iommu, domain, dev,
-						IOMMU_NO_PASID, NULL);
-	else if (WARN_ON(true))
-		ret = -EINVAL;
+	if (!device_ser) {
+		if (!sm_supported(iommu))
+			ret = domain_context_mapping(domain, dev);
+		else if (intel_domain_is_fs_paging(domain))
+			ret = domain_setup_first_level(iommu, domain, dev,
+						       IOMMU_NO_PASID, NULL);
+		else if (intel_domain_is_ss_paging(domain))
+			ret = domain_setup_second_level(iommu, domain, dev,
+							IOMMU_NO_PASID, NULL);
+		else if (WARN_ON(true))
+			ret = -EINVAL;
+	}
 
  	if (ret)
  		goto out_block_translation;
@@ -3170,6 +3180,15 @@ int paging_domain_compatible(struct iommu_domain *domain, struct device *dev)
  	struct intel_iommu *iommu = info->iommu;
  	int ret = -EINVAL;
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	/*
+	 * Restored IOMMU domains are already attached to the device and can
+	 * only be freed. So no need to check the compatibility.
+	 */
+	if (iommu_domain_restored_state(domain))
+		return 0;
+#endif
+
  	if (intel_domain_is_fs_paging(dmar_domain))
  		ret = paging_domain_compatible_first_stage(dmar_domain, iommu);
  	else if (intel_domain_is_ss_paging(dmar_domain))
@@ -3647,7 +3666,7 @@ domain_add_dev_pasid(struct iommu_domain *domain,
  	if (!dev_pasid)
  		return ERR_PTR(-ENOMEM);
 
-	ret = domain_attach_iommu(dmar_domain, iommu);
+	ret = domain_attach_iommu(dmar_domain, iommu, -1);
  	if (ret)
  		goto out_free;
 
drivers/iommu/intel/iommu.h +2 -1
index b0ec0b471a43..8e37acf7de12 100644
@@ -1182,7 +1182,8 @@ void __iommu_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,
   */
  #define QI_OPT_WAIT_DRAIN		BIT(0)
 
-int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu);
+int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu,
+			int restore_did);
  void domain_detach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu);
  void device_block_translation(struct device *dev);
  int paging_domain_compatible(struct iommu_domain *domain, struct device *dev);
drivers/iommu/intel/nested.c +0 -0
index 2b979bec56ce..6e13f697b463 100644

drivers/iommu/intel/nested.c static int intel_nested_attach_dev(struct iommu_domain *domain, · line 46

Quoted 8 lines
drivers/iommu/intel/nested.c +1 -1
@@ -40,7 +40,7 @@ static int intel_nested_attach_dev(struct iommu_domain *domain,
  		return ret;
  	}
 
-	ret = domain_attach_iommu(dmar_domain, iommu);
+	ret = domain_attach_iommu(dmar_domain, iommu, -1);
  	if (ret) {
  		dev_err_ratelimited(dev, "Failed to attach domain to iommu\n");
  		return ret;

Thanks,
baolu

#59 lore raw

Re: [PATCH v2 10/16] iommu: Restore and reattach preserved domains to devices

Quoted 17 lines · Baolu Lu
Quoted 7 lines · Samiullah Khawaja

Restore the preserved domains by restoring the page tables using restore IOMMU domain op. Reattach the preserved domain to the device during default domain setup. While attaching, reuse the domain ID that was used in the previous kernel. The context entry setup is not needed as that is preserved during liveupdate.

Quoted 8 lines
---
 drivers/iommu/intel/iommu.c      | 49 ++++++++++++++------
 drivers/iommu/intel/iommu.h      |  3 +-
 drivers/iommu/intel/nested.c     |  2 +-
 drivers/iommu/iommu.c            | 61 ++++++++++++++++++++++++-
 drivers/iommu/liveupdate.c       | 78 ++++++++++++++++++++++++++++++++
 include/linux/iommu-liveupdate.h | 50 ++++++++++++++++++++
 6 files changed, 224 insertions(+), 19 deletions(-)
Quoted 2 lines

Please split the changes in the Intel iommu driver and the iommu core into different patches.

Agreed. I will split these.

Quoted 15 lines

drivers/iommu/intel/iommu.c static bool first_level_by_default(struct intel_iommu *iommu) · line 1034

Quoted 12 lines
drivers/iommu/intel/iommu.c +2 -1
index 4118a0861f38..b90757164cd8 100644
@@ -1031,7 +1031,8 @@ static bool first_level_by_default(struct intel_iommu *iommu)
 	return true;
 }
-int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu)
+int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu,
+			int restore_did)

How about using a new helper for restored domain? For example,

Quoted 5 lines

int domain_reattach_iommu(...)

or

int domain_restore_iommu(...)

Agreed. I didn't do this to avoid code duplication, but I think this might be much clean. Also in the restore path, the did test can also be done as you pointed out below.

Quoted 14 lines

drivers/iommu/intel/iommu.c int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu) · line 1055

Quoted 11 lines
drivers/iommu/intel/iommu.c +2 -2
 {
 	struct iommu_domain_info *info, *curr;
 	int num, ret = -ENOSPC;
@@ -1051,8 +1052,11 @@ int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu)
 		return 0;
 	}
-	num = ida_alloc_range(&iommu->domain_ida, IDA_START_DID,
-			      cap_ndoms(iommu->cap) - 1, GFP_KERNEL);
+	if (restore_did >= IDA_START_DID)
+		num = restore_did;

For a preserved domain ID, do we need to check whether it has been reserved from the ida pool?

Yes, I will add a sanity check once I move this logic into separate function.

Quoted 16 lines
Quoted 16 lines
drivers/iommu/intel/iommu.c +5 -1
+	else
+		num = ida_alloc_range(&iommu->domain_ida, IDA_START_DID,
+				      cap_ndoms(iommu->cap) - 1, GFP_KERNEL);
 	if (num < 0) {
 		pr_err("%s: No free domain ids\n", iommu->name);
 		goto err_unlock;
@@ -1320,10 +1324,14 @@ static int dmar_domain_attach_device(struct dmar_domain *domain,
 {
 	struct device_domain_info *info = dev_iommu_priv_get(dev);
 	struct intel_iommu *iommu = info->iommu;
+	struct device_ser *device_ser = NULL;
 	unsigned long flags;
 	int ret;
-	ret = domain_attach_iommu(domain, iommu);
+	device_ser = dev_iommu_restored_state(dev);
Quoted 8 lines

drivers/iommu/intel/iommu.c static int dmar_domain_attach_device(struct dmar_domain *domain, · line 1334

Quoted 3 lines
drivers/iommu/intel/iommu.c +3 -0
+
+	ret = domain_attach_iommu(domain, iommu,
+				  dev_iommu_restore_did(dev, &domain->domain));

What is the expected behavior if there is a mismatch between the restored state and the availability of a domain ID? Specifically, if device_ser is found but dev_iommu_restore_did() fails (returns -1), how should the driver proceed in that case?

The restore of the domain should fail in this case, as this is unexpected behaviour, as the DID should be preserved in the previous kernel and in this (new) kernel the did should have been reclaimed during iommu HW restore.

But I think we can sanity check these in the separate restore function. I will rework this.

Quoted 72 lines
Quoted 72 lines
drivers/iommu/intel/iommu.c +22 -11
 	if (ret)
 		return ret;
@@ -1336,16 +1344,18 @@ static int dmar_domain_attach_device(struct dmar_domain *domain,
 	if (dev_is_real_dma_subdevice(dev))
 		return 0;
-	if (!sm_supported(iommu))
-		ret = domain_context_mapping(domain, dev);
-	else if (intel_domain_is_fs_paging(domain))
-		ret = domain_setup_first_level(iommu, domain, dev,
-					       IOMMU_NO_PASID, NULL);
-	else if (intel_domain_is_ss_paging(domain))
-		ret = domain_setup_second_level(iommu, domain, dev,
-						IOMMU_NO_PASID, NULL);
-	else if (WARN_ON(true))
-		ret = -EINVAL;
+	if (!device_ser) {
+		if (!sm_supported(iommu))
+			ret = domain_context_mapping(domain, dev);
+		else if (intel_domain_is_fs_paging(domain))
+			ret = domain_setup_first_level(iommu, domain, dev,
+						       IOMMU_NO_PASID, NULL);
+		else if (intel_domain_is_ss_paging(domain))
+			ret = domain_setup_second_level(iommu, domain, dev,
+							IOMMU_NO_PASID, NULL);
+		else if (WARN_ON(true))
+			ret = -EINVAL;
+	}
 	if (ret)
 		goto out_block_translation;
@@ -3170,6 +3180,15 @@ int paging_domain_compatible(struct iommu_domain *domain, struct device *dev)
 	struct intel_iommu *iommu = info->iommu;
 	int ret = -EINVAL;
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	/*
+	 * Restored IOMMU domains are already attached to the device and can
+	 * only be freed. So no need to check the compatibility.
+	 */
+	if (iommu_domain_restored_state(domain))
+		return 0;
+#endif
+
 	if (intel_domain_is_fs_paging(dmar_domain))
 		ret = paging_domain_compatible_first_stage(dmar_domain, iommu);
 	else if (intel_domain_is_ss_paging(dmar_domain))
@@ -3647,7 +3666,7 @@ domain_add_dev_pasid(struct iommu_domain *domain,
 	if (!dev_pasid)
 		return ERR_PTR(-ENOMEM);
-	ret = domain_attach_iommu(dmar_domain, iommu);
+	ret = domain_attach_iommu(dmar_domain, iommu, -1);
 	if (ret)
 		goto out_free;
drivers/iommu/intel/iommu.h +2 -1
index b0ec0b471a43..8e37acf7de12 100644
@@ -1182,7 +1182,8 @@ void __iommu_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,
  */
 #define QI_OPT_WAIT_DRAIN		BIT(0)
-int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu);
+int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu,
+			int restore_did);
 void domain_detach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu);
 void device_block_translation(struct device *dev);
 int paging_domain_compatible(struct iommu_domain *domain, struct device *dev);
drivers/iommu/intel/nested.c +0 -0
index 2b979bec56ce..6e13f697b463 100644
@@ -40,7 +40,7 @@ static int intel_nested_attach_dev(struct iommu_domain *domain,
 		return ret;
 	}
Quoted 8 lines

drivers/iommu/intel/nested.c static int intel_nested_attach_dev(struct iommu_domain *domain, · line 46

Quoted 5 lines
drivers/iommu/intel/nested.c +1 -1
-	ret = domain_attach_iommu(dmar_domain, iommu);
+	ret = domain_attach_iommu(dmar_domain, iommu, -1);
 	if (ret) {
 		dev_err_ratelimited(dev, "Failed to attach domain to iommu\n");
 		return ret;

Thanks,
baolu

Thanks,
Sami

#60 lore raw

Re: [PATCH v2 10/16] iommu: Restore and reattach preserved domains to devices

In reply to Samiullah Khawaja
Quoted 45 lines · Samiullah Khawaja

Restore the preserved domains by restoring the page tables using restore IOMMU domain op. Reattach the preserved domain to the device during default domain setup. While attaching, reuse the domain ID that was used in the previous kernel. The context entry setup is not needed as that is preserved during liveupdate.

---
 drivers/iommu/intel/iommu.c      | 49 ++++++++++++++------
 drivers/iommu/intel/iommu.h      |  3 +-
 drivers/iommu/intel/nested.c     |  2 +-
 drivers/iommu/iommu.c            | 61 ++++++++++++++++++++++++-
 drivers/iommu/liveupdate.c       | 78 ++++++++++++++++++++++++++++++++
 include/linux/iommu-liveupdate.h | 50 ++++++++++++++++++++
 6 files changed, 224 insertions(+), 19 deletions(-)
drivers/iommu/intel/iommu.c +7 -3
index 4118a0861f38..b90757164cd8 100644
@@ -1031,7 +1031,8 @@ static bool first_level_by_default(struct intel_iommu *iommu)
 	return true;
 }
 
-int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu)
+int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu,
+			int restore_did)
 {
 	struct iommu_domain_info *info, *curr;
 	int num, ret = -ENOSPC;
@@ -1051,8 +1052,11 @@ int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu)
 		return 0;
 	}
 
-	num = ida_alloc_range(&iommu->domain_ida, IDA_START_DID,
-			      cap_ndoms(iommu->cap) - 1, GFP_KERNEL);
+	if (restore_did >= IDA_START_DID)
+		num = restore_did;
+	else
+		num = ida_alloc_range(&iommu->domain_ida, IDA_START_DID,
+				      cap_ndoms(iommu->cap) - 1, GFP_KERNEL);
 	if (num < 0) {
 		pr_err("%s: No free domain ids\n", iommu->name);
 		goto err_unlock;

drivers/iommu/intel/iommu.c static int dmar_domain_attach_device(struct dmar_domain *domain, · line 1327

Quoted 4 lines
drivers/iommu/intel/iommu.c +1 -0
@@ -1320,10 +1324,14 @@ static int dmar_domain_attach_device(struct dmar_domain *domain,
 {
 	struct device_domain_info *info = dev_iommu_priv_get(dev);
 	struct intel_iommu *iommu = info->iommu;
+	struct device_ser *device_ser = NULL;

Typo here? iommu_device_ser.
compiles because dev_iommu_restored_state() returns void * and the variable is only null-checked, never dereferenced.

Ankit

Quoted 390 lines
drivers/iommu/intel/iommu.c +26 -12
 	unsigned long flags;
 	int ret;
 
-	ret = domain_attach_iommu(domain, iommu);
+	device_ser = dev_iommu_restored_state(dev);
+
+	ret = domain_attach_iommu(domain, iommu,
+				  dev_iommu_restore_did(dev, &domain->domain));
 	if (ret)
 		return ret;
 
@@ -1336,16 +1344,18 @@ static int dmar_domain_attach_device(struct dmar_domain *domain,
 	if (dev_is_real_dma_subdevice(dev))
 		return 0;
 
-	if (!sm_supported(iommu))
-		ret = domain_context_mapping(domain, dev);
-	else if (intel_domain_is_fs_paging(domain))
-		ret = domain_setup_first_level(iommu, domain, dev,
-					       IOMMU_NO_PASID, NULL);
-	else if (intel_domain_is_ss_paging(domain))
-		ret = domain_setup_second_level(iommu, domain, dev,
-						IOMMU_NO_PASID, NULL);
-	else if (WARN_ON(true))
-		ret = -EINVAL;
+	if (!device_ser) {
+		if (!sm_supported(iommu))
+			ret = domain_context_mapping(domain, dev);
+		else if (intel_domain_is_fs_paging(domain))
+			ret = domain_setup_first_level(iommu, domain, dev,
+						       IOMMU_NO_PASID, NULL);
+		else if (intel_domain_is_ss_paging(domain))
+			ret = domain_setup_second_level(iommu, domain, dev,
+							IOMMU_NO_PASID, NULL);
+		else if (WARN_ON(true))
+			ret = -EINVAL;
+	}
 
 	if (ret)
 		goto out_block_translation;
@@ -3170,6 +3180,15 @@ int paging_domain_compatible(struct iommu_domain *domain, struct device *dev)
 	struct intel_iommu *iommu = info->iommu;
 	int ret = -EINVAL;
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	/*
+	 * Restored IOMMU domains are already attached to the device and can
+	 * only be freed. So no need to check the compatibility.
+	 */
+	if (iommu_domain_restored_state(domain))
+		return 0;
+#endif
+
 	if (intel_domain_is_fs_paging(dmar_domain))
 		ret = paging_domain_compatible_first_stage(dmar_domain, iommu);
 	else if (intel_domain_is_ss_paging(dmar_domain))
@@ -3647,7 +3666,7 @@ domain_add_dev_pasid(struct iommu_domain *domain,
 	if (!dev_pasid)
 		return ERR_PTR(-ENOMEM);
 
-	ret = domain_attach_iommu(dmar_domain, iommu);
+	ret = domain_attach_iommu(dmar_domain, iommu, -1);
 	if (ret)
 		goto out_free;
 
drivers/iommu/intel/iommu.h +2 -1
index b0ec0b471a43..8e37acf7de12 100644
@@ -1182,7 +1182,8 @@ void __iommu_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,
  */
 #define QI_OPT_WAIT_DRAIN		BIT(0)
 
-int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu);
+int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu,
+			int restore_did);
 void domain_detach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu);
 void device_block_translation(struct device *dev);
 int paging_domain_compatible(struct iommu_domain *domain, struct device *dev);
drivers/iommu/intel/nested.c +1 -1
index 2b979bec56ce..6e13f697b463 100644
@@ -40,7 +40,7 @@ static int intel_nested_attach_dev(struct iommu_domain *domain,
 		return ret;
 	}
 
-	ret = domain_attach_iommu(dmar_domain, iommu);
+	ret = domain_attach_iommu(dmar_domain, iommu, -1);
 	if (ret) {
 		dev_err_ratelimited(dev, "Failed to attach domain to iommu\n");
 		return ret;
drivers/iommu/iommu.c +59 -2
index 0561990f46e3..e888700da53d 100644
@@ -18,6 +18,7 @@
 #include <linux/errno.h>
 #include <linux/host1x_context_bus.h>
 #include <linux/iommu.h>
+#include <linux/iommu-liveupdate.h>
 #include <linux/iommufd.h>
 #include <linux/idr.h>
 #include <linux/err.h>
@@ -505,6 +506,10 @@ static int iommu_init_device(struct device *dev)
 		goto err_free;
 	}
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	dev->iommu->device_ser = iommu_get_device_preserved_data(dev);
+#endif
+
 	iommu_dev = ops->probe_device(dev);
 	if (IS_ERR(iommu_dev)) {
 		ret = PTR_ERR(iommu_dev);
@@ -2204,6 +2209,13 @@ static int __iommu_attach_device(struct iommu_domain *domain,
 	ret = domain->ops->attach_dev(domain, dev, old);
 	if (ret)
 		return ret;
+
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	/* The associated state can be unset once restored. */
+	if (dev_iommu_restored_state(dev))
+		WRITE_ONCE(dev->iommu->device_ser, NULL);
+#endif
+
 	dev->iommu->attach_deferred = 0;
 	trace_attach_device_to_domain(dev);
 	return 0;
@@ -3159,6 +3171,47 @@ int iommu_fwspec_add_ids(struct device *dev, const u32 *ids, int num_ids)
 }
 EXPORT_SYMBOL_GPL(iommu_fwspec_add_ids);
 
+static inline void *__iommu_group_restored_state(struct iommu_group *group)
+{
+	struct device *dev;
+
+	dev = iommu_group_first_dev(group);
+	if (!dev_is_pci(dev))
+		return NULL;
+
+	return dev_iommu_restored_state(dev);
+}
+
+static struct iommu_domain *__iommu_group_restore_domain(struct iommu_group *group)
+{
+	struct iommu_device_ser *device_ser;
+	struct iommu_domain *domain;
+	struct device *dev;
+	void *owner;
+
+	lockdep_assert_held(&group->mutex);
+	dev = iommu_group_first_dev(group);
+	if (!dev_is_pci(dev))
+		return NULL;
+
+	device_ser = dev_iommu_restored_state(dev);
+	if (!device_ser)
+		return NULL;
+
+	domain = iommu_restore_domain(dev, device_ser, &owner);
+	if (WARN_ON(IS_ERR(domain)))
+		return NULL;
+
+	/*
+	 * Ownership of groups with preserved devices is set during boot. These
+	 * will be reclaimed later by the entity (iommufd) that preserved them.
+	 */
+	WARN_ON(group->owner);
+	group->owner = owner;
+	group->owner_cnt = 1;
+	return domain;
+}
+
 /**
  * iommu_setup_default_domain - Set the default_domain for the group
  * @group: Group to change
@@ -3173,8 +3226,8 @@ static int iommu_setup_default_domain(struct iommu_group *group,
 				      int target_type)
 {
 	struct iommu_domain *old_dom = group->default_domain;
+	struct iommu_domain *dom, *restored_domain;
 	struct group_device *gdev;
-	struct iommu_domain *dom;
 	bool direct_failed;
 	int req_type;
 	int ret;
@@ -3218,6 +3271,10 @@ static int iommu_setup_default_domain(struct iommu_group *group,
 	/* We must set default_domain early for __iommu_device_set_domain */
 	group->default_domain = dom;
 	if (!group->domain) {
+		if (__iommu_group_restored_state(group))
+			restored_domain = __iommu_group_restore_domain(group);
+		else
+			restored_domain = dom;
 		/*
 		 * Drivers are not allowed to fail the first domain attach.
 		 * The only way to recover from this is to fail attaching the
@@ -3225,7 +3282,7 @@ static int iommu_setup_default_domain(struct iommu_group *group,
 		 * in group->default_domain so it is freed after.
 		 */
 		ret = __iommu_group_set_domain_internal(
-			group, dom, IOMMU_SET_DOMAIN_MUST_SUCCEED);
+			group, restored_domain, IOMMU_SET_DOMAIN_MUST_SUCCEED);
 		if (WARN_ON(ret))
 			goto out_free_old;
 	} else {
drivers/iommu/liveupdate.c +78 -0
index 60ee29b0c6bd..0888871784ea 100644
@@ -234,6 +234,41 @@ int iommu_for_each_preserved_device(iommu_preserved_device_iter_fn fn,
 }
 EXPORT_SYMBOL(iommu_for_each_preserved_device);
 
+static inline bool match_device_ser(struct iommu_device_ser *match,
+				    struct pci_dev *pdev)
+{
+	return match->devid == pci_dev_id(pdev) && match->pci_domain_nr == pci_domain_nr(pdev->bus);
+}
+
+struct iommu_device_ser *iommu_get_device_preserved_data(struct device *dev)
+{
+	struct iommu_device_ser *device_ser = NULL;
+	struct iommu_device_array_ser *array;
+	struct iommu_flb_obj *flb_obj;
+	int ret, idx;
+
+	if (!dev_is_pci(dev))
+		return NULL;
+
+	ret = liveupdate_flb_get_incoming(&iommu_flb, (void **)&flb_obj);
+	if (ret)
+		return NULL;
+
+	array = phys_to_virt(flb_obj->ser->device_array_phys);
+	iommu_liveupdate_for_each_obj(array, device_ser, idx) {
+		if (match_device_ser(device_ser, to_pci_dev(dev))) {
+			device_ser->hdr.incoming = true;
+			goto out;
+		}
+	}
+
+	device_ser = NULL;
+out:
+	liveupdate_flb_put_incoming(&iommu_flb);
+	return device_ser;
+}
+EXPORT_SYMBOL(iommu_get_device_preserved_data);
+
 struct iommu_hw_ser *iommu_get_preserved_data(u64 token, enum iommu_type_ser type)
 {
 	struct iommu_hw_ser *iommu_ser = NULL;
@@ -512,3 +547,46 @@ void iommu_unpreserve_device(struct iommu_domain *domain, struct device *dev)
 
 	iommu_unpreserve_locked(iommu->iommu_dev, flb_obj);
 }
+
+struct iommu_domain *iommu_restore_domain(struct device *dev,
+					  struct iommu_device_ser *ser,
+					  void **owner)
+{
+	struct iommu_domain_ser *domain_ser;
+	struct iommu_flb_obj *flb_obj;
+	struct iommu_domain *domain;
+	int ret;
+
+	domain_ser = phys_to_virt(ser->domain_iommu_ser.domain_phys);
+
+	ret = liveupdate_flb_get_incoming(&iommu_flb, (void **)&flb_obj);
+	if (ret)
+		return ERR_PTR(ret);
+
+	guard(mutex)(&flb_obj->lock);
+	if (domain_ser->restored_domain) {
+		domain = domain_ser->restored_domain;
+		goto out;
+	}
+
+	domain_ser->hdr.incoming =  true;
+	domain = iommu_paging_domain_alloc(dev);
+	if (IS_ERR(domain))
+		goto out;
+
+	ret = domain->ops->restore(domain, domain_ser);
+	if (ret) {
+		iommu_domain_free(domain);
+		domain = ERR_PTR(ret);
+		goto out;
+	}
+
+	/* The device is owned by the preserved state. */
+	*owner = ser;
+	domain->preserved_state = domain_ser;
+	domain_ser->restored_domain = domain;
+
+out:
+	liveupdate_flb_put_incoming(&iommu_flb);
+	return domain;
+}
include/linux/iommu-liveupdate.h +50 -0
index 0baf6bc2d93f..75d27256c883 100644
@@ -30,6 +30,20 @@ static inline void *dev_iommu_preserved_state(struct device *dev)
 	return NULL;
 }
 
+static inline void *dev_iommu_restored_state(struct device *dev)
+{
+	struct iommu_device_ser *ser;
+
+	if (!dev->iommu)
+		return NULL;
+
+	ser = dev->iommu->device_ser;
+	if (ser && ser->hdr.incoming)
+		return ser;
+
+	return NULL;
+}
+
 static inline void *iommu_domain_restored_state(struct iommu_domain *domain)
 {
 	struct iommu_domain_ser *ser;
@@ -41,8 +55,22 @@ static inline void *iommu_domain_restored_state(struct iommu_domain *domain)
 	return NULL;
 }
 
+static inline int dev_iommu_restore_did(struct device *dev, struct iommu_domain *domain)
+{
+	struct iommu_device_ser *ser = dev_iommu_restored_state(dev);
+
+	if (ser && iommu_domain_restored_state(domain))
+		return ser->domain_iommu_ser.attachment_id;
+
+	return -1;
+}
+
+struct iommu_domain *iommu_restore_domain(struct device *dev,
+					  struct iommu_device_ser *ser,
+					  void **owner);
 int iommu_for_each_preserved_device(iommu_preserved_device_iter_fn fn,
 				    void *arg);
+struct iommu_device_ser *iommu_get_device_preserved_data(struct device *dev);
 struct iommu_hw_ser *iommu_get_preserved_data(u64 token, enum iommu_type_ser type);
 int iommu_domain_preserve(struct iommu_domain *domain, struct iommu_domain_ser **ser);
 void iommu_domain_unpreserve(struct iommu_domain *domain);
@@ -60,16 +88,38 @@ static inline void *dev_iommu_preserved_state(struct device *dev)
 	return NULL;
 }
 
+static inline void *dev_iommu_restored_state(struct device *dev)
+{
+	return NULL;
+}
+
+static inline int dev_iommu_restore_did(struct device *dev, struct iommu_domain *domain)
+{
+	return -1;
+}
+
 static inline void *iommu_domain_restored_state(struct iommu_domain *domain)
 {
 	return NULL;
 }
 
+static inline struct iommu_domain *iommu_restore_domain(struct device *dev,
+							struct iommu_device_ser *ser,
+							void **owner)
+{
+	return NULL;
+}
+
 static inline int iommu_for_each_preserved_device(iommu_preserved_device_iter_fn fn, void *arg)
 {
 	return -EOPNOTSUPP;
 }
 
+static inline struct iommu_device_ser *iommu_get_device_preserved_data(struct device *dev)
+{
+	return NULL;
+}
+
 static inline struct iommu_hw_ser *iommu_get_preserved_data(u64 token, enum iommu_type_ser type)
 {
 	return NULL;
Signature · 1 line
2.54.0.545.g6539524ca2-goog
#61 lore raw

Re: [PATCH v2 10/16] iommu: Restore and reattach preserved domains to devices

Quoted 46 lines · Ankit Soni
Quoted 45 lines · Samiullah Khawaja

Restore the preserved domains by restoring the page tables using restore IOMMU domain op. Reattach the preserved domain to the device during default domain setup. While attaching, reuse the domain ID that was used in the previous kernel. The context entry setup is not needed as that is preserved during liveupdate.

---
 drivers/iommu/intel/iommu.c      | 49 ++++++++++++++------
 drivers/iommu/intel/iommu.h      |  3 +-
 drivers/iommu/intel/nested.c     |  2 +-
 drivers/iommu/iommu.c            | 61 ++++++++++++++++++++++++-
 drivers/iommu/liveupdate.c       | 78 ++++++++++++++++++++++++++++++++
 include/linux/iommu-liveupdate.h | 50 ++++++++++++++++++++
 6 files changed, 224 insertions(+), 19 deletions(-)
drivers/iommu/intel/iommu.c +7 -3
index 4118a0861f38..b90757164cd8 100644
@@ -1031,7 +1031,8 @@ static bool first_level_by_default(struct intel_iommu *iommu)
 	return true;
 }
 
-int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu)
+int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu,
+			int restore_did)
 {
 	struct iommu_domain_info *info, *curr;
 	int num, ret = -ENOSPC;
@@ -1051,8 +1052,11 @@ int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu)
 		return 0;
 	}
 
-	num = ida_alloc_range(&iommu->domain_ida, IDA_START_DID,
-			      cap_ndoms(iommu->cap) - 1, GFP_KERNEL);
+	if (restore_did >= IDA_START_DID)
+		num = restore_did;
+	else
+		num = ida_alloc_range(&iommu->domain_ida, IDA_START_DID,
+				      cap_ndoms(iommu->cap) - 1, GFP_KERNEL);
 	if (num < 0) {
 		pr_err("%s: No free domain ids\n", iommu->name);
 		goto err_unlock;
@@ -1320,10 +1324,14 @@ static int dmar_domain_attach_device(struct dmar_domain *domain,
Quoted 8 lines

drivers/iommu/intel/iommu.c static int dmar_domain_attach_device(struct dmar_domain *domain, · line 1327

Quoted 4 lines
drivers/iommu/intel/iommu.c +1 -0
 {
 	struct device_domain_info *info = dev_iommu_priv_get(dev);
 	struct intel_iommu *iommu = info->iommu;
+	struct device_ser *device_ser = NULL;

Typo here? iommu_device_ser.
compiles because dev_iommu_restored_state() returns void * and the variable is only null-checked, never dereferenced.

Agreed. Will update this.

Quoted 14 lines

Ankit

Quoted 11 lines
drivers/iommu/intel/iommu.c +4 -1
 	unsigned long flags;
 	int ret;
 
-	ret = domain_attach_iommu(domain, iommu);
+	device_ser = dev_iommu_restored_state(dev);
+
+	ret = domain_attach_iommu(domain, iommu,
+				  dev_iommu_restore_did(dev, &domain->domain));
 	if (ret)
 		return ret;
 

[snip]

Quoted 6 lines
Quoted 6 lines

static inline struct iommu_hw_ser *iommu_get_preserved_data(u64 token, enum iommu_type_ser type) {
return NULL;
--
2.54.0.545.g6539524ca2-goog

Thanks,
Sami

#62 lore raw

PATCH v2 11/16 iommu/vt-d: preserve PASID table of preserved device

In reply to Samiullah Khawaja

In scalable mode the PASID table is used to fetch the io page tables. Preserve and restore the PASID table of the preserved devices.

---
 drivers/iommu/intel/iommu.c      |   5 +-
 drivers/iommu/intel/iommu.h      |  12 +++
 drivers/iommu/intel/liveupdate.c | 141 +++++++++++++++++++++++++++++++
 drivers/iommu/intel/pasid.c      |   7 +-
 drivers/iommu/intel/pasid.h      |   9 ++
 include/linux/kho/abi/iommu.h    |  13 +++
 6 files changed, 184 insertions(+), 3 deletions(-)
drivers/iommu/intel/iommu.c +4 -1
index b90757164cd8..6d42051dcf7c 100644
@@ -2951,8 +2951,10 @@ static int clear_unpreserve_context_entry_fn(struct device *dev,
 	if (!info)
 		return 0;
 
-	if (dev_is_pci(dev) && dev_iommu_preserved_state(dev))
+	if (dev_is_pci(dev) && dev_iommu_preserved_state(dev)) {
+		pasid_cleanup_preserved_table(dev);
 		return 0;
+	}
 
 	domain_context_clear(info);
 	return 0;
@@ -4013,6 +4015,7 @@ const struct iommu_ops intel_iommu_ops = {
 	.page_response		= intel_iommu_page_response,
 #ifdef CONFIG_IOMMU_LIVEUPDATE
 	.preserve_device	= intel_iommu_preserve_device,
+	.unpreserve_device	= intel_iommu_unpreserve_device,
 	.preserve		= intel_iommu_preserve,
 	.unpreserve		= intel_iommu_unpreserve,
 #endif
drivers/iommu/intel/iommu.h +12 -0
index 8e37acf7de12..62076a1a0b4d 100644
@@ -1290,12 +1290,15 @@ static inline int iopf_for_domain_replace(struct iommu_domain *new,
 #ifdef CONFIG_IOMMU_LIVEUPDATE
 int intel_iommu_preserve_device(struct device *dev,
 				struct iommu_device_ser *device_ser);
+void intel_iommu_unpreserve_device(struct device *dev,
+				   struct iommu_device_ser *device_ser);
 int intel_iommu_preserve(struct iommu_device *iommu,
 			 struct iommu_hw_ser *iommu_ser);
 void intel_iommu_unpreserve(struct iommu_device *iommu,
 			    struct iommu_hw_ser *iommu_ser);
 void intel_iommu_liveupdate_restore_root_table(struct intel_iommu *iommu,
 					       struct iommu_hw_ser *iommu_ser);
+void pasid_cleanup_preserved_table(struct device *dev);
 #else
 static inline int intel_iommu_preserve_device(struct device *dev,
 					      struct iommu_device_ser *device_ser)
@@ -1303,6 +1306,11 @@ static inline int intel_iommu_preserve_device(struct device *dev,
 	return -EOPNOTSUPP;
 }
 
+static inline void intel_iommu_unpreserve_device(struct device *dev,
+						 struct iommu_device_ser *device_ser)
+{
+}
+
 static inline int intel_iommu_preserve(struct iommu_device *iommu,
 				       struct iommu_hw_ser *iommu_ser)
 {
@@ -1318,6 +1326,10 @@ static inline void intel_iommu_liveupdate_restore_root_table(struct intel_iommu
 							     struct iommu_hw_ser *iommu_ser)
 {
 }
+
+static inline void pasid_cleanup_preserved_table(struct device *dev)
+{
+}
 #endif
 
 #ifdef CONFIG_INTEL_IOMMU_SVM
drivers/iommu/intel/liveupdate.c +141 -0

169 lines not on this page · raw message

drivers/iommu/intel/pasid.c +5 -2
index 89541b74ab8c..5cac8e95f73b 100644
@@ -60,8 +60,11 @@ int intel_pasid_alloc_table(struct device *dev)
 
 	size = max_pasid >> (PASID_PDE_SHIFT - 3);
 	order = size ? get_order(size) : 0;
-	dir = iommu_alloc_pages_node_sz(info->iommu->node, GFP_KERNEL,
-					1 << (order + PAGE_SHIFT));
+
+	dir = intel_pasid_try_restore_table(dev, 1 << (order + PAGE_SHIFT + 3));
+	if (!dir)
+		dir = iommu_alloc_pages_node_sz(info->iommu->node, GFP_KERNEL,
+						1 << (order + PAGE_SHIFT));
 	if (!dir) {
 		kfree(pasid_table);
 		return -ENOMEM;
drivers/iommu/intel/pasid.h +9 -0
index 48d3bb6b68de..44e673a4ad8f 100644
@@ -301,6 +301,15 @@ static inline void pasid_set_eafe(struct pasid_entry *pe)
 
 extern unsigned int intel_pasid_max_id;
 int intel_pasid_alloc_table(struct device *dev);
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+void *intel_pasid_try_restore_table(struct device *dev, u64 max_pasid);
+#else
+static inline void *intel_pasid_try_restore_table(struct device *dev,
+						  u64 max_pasid)
+{
+	return NULL;
+}
+#endif
 void intel_pasid_free_table(struct device *dev);
 struct pasid_table *intel_pasid_get_table(struct device *dev);
 int intel_pasid_setup_first_level(struct intel_iommu *iommu, struct device *dev,
include/linux/kho/abi/iommu.h +13 -0
index 5ffedf0dbd5a..5eeb1e0c9bce 100644
@@ -119,6 +119,16 @@ struct iommu_dev_map_ser {
 	u64 iommu_phys;
 } __packed;
 
+/**
+ * struct iommu_device_intel_ser - Intel specific state of serialized device
+ * @pasid_table: Physical address of pasid table
+ * @max_pasid: Maximum supported pasid
+ */
+struct iommu_device_intel_ser {
+	u64 pasid_table;
+	u64 max_pasid;
+} __packed;
+
 /**
  * struct iommu_device_ser - Serialized state of a device
  * @hdr: Common object header
@@ -131,6 +141,9 @@ struct iommu_device_ser {
 	u32 devid;
 	u32 pci_domain_nr;
 	struct iommu_dev_map_ser domain_iommu_ser;
+	union {
+		struct iommu_device_intel_ser intel;
+	};
 } __packed;
 
 /**
Signature · 1 line
2.54.0.545.g6539524ca2-goog
maintainer
#63 lore raw

Re: [PATCH v2 11/16] iommu/vt-d: preserve PASID table of preserved device

Quoted 169 lines · Samiullah Khawaja

In scalable mode the PASID table is used to fetch the io page tables. Preserve and restore the PASID table of the preserved devices.

---
  drivers/iommu/intel/iommu.c      |   5 +-
  drivers/iommu/intel/iommu.h      |  12 +++
  drivers/iommu/intel/liveupdate.c | 141 +++++++++++++++++++++++++++++++
  drivers/iommu/intel/pasid.c      |   7 +-
  drivers/iommu/intel/pasid.h      |   9 ++
  include/linux/kho/abi/iommu.h    |  13 +++
  6 files changed, 184 insertions(+), 3 deletions(-)
drivers/iommu/intel/iommu.c +4 -1
index b90757164cd8..6d42051dcf7c 100644
@@ -2951,8 +2951,10 @@ static int clear_unpreserve_context_entry_fn(struct device *dev,
  	if (!info)
  		return 0;
 
-	if (dev_is_pci(dev) && dev_iommu_preserved_state(dev))
+	if (dev_is_pci(dev) && dev_iommu_preserved_state(dev)) {
+		pasid_cleanup_preserved_table(dev);
  		return 0;
+	}
 
  	domain_context_clear(info);
  	return 0;
@@ -4013,6 +4015,7 @@ const struct iommu_ops intel_iommu_ops = {
  	.page_response		= intel_iommu_page_response,
  #ifdef CONFIG_IOMMU_LIVEUPDATE
  	.preserve_device	= intel_iommu_preserve_device,
+	.unpreserve_device	= intel_iommu_unpreserve_device,
  	.preserve		= intel_iommu_preserve,
  	.unpreserve		= intel_iommu_unpreserve,
  #endif
drivers/iommu/intel/iommu.h +12 -0
index 8e37acf7de12..62076a1a0b4d 100644
@@ -1290,12 +1290,15 @@ static inline int iopf_for_domain_replace(struct iommu_domain *new,
  #ifdef CONFIG_IOMMU_LIVEUPDATE
  int intel_iommu_preserve_device(struct device *dev,
  				struct iommu_device_ser *device_ser);
+void intel_iommu_unpreserve_device(struct device *dev,
+				   struct iommu_device_ser *device_ser);
  int intel_iommu_preserve(struct iommu_device *iommu,
  			 struct iommu_hw_ser *iommu_ser);
  void intel_iommu_unpreserve(struct iommu_device *iommu,
  			    struct iommu_hw_ser *iommu_ser);
  void intel_iommu_liveupdate_restore_root_table(struct intel_iommu *iommu,
  					       struct iommu_hw_ser *iommu_ser);
+void pasid_cleanup_preserved_table(struct device *dev);
  #else
  static inline int intel_iommu_preserve_device(struct device *dev,
  					      struct iommu_device_ser *device_ser)
@@ -1303,6 +1306,11 @@ static inline int intel_iommu_preserve_device(struct device *dev,
  	return -EOPNOTSUPP;
  }
 
+static inline void intel_iommu_unpreserve_device(struct device *dev,
+						 struct iommu_device_ser *device_ser)
+{
+}
+
  static inline int intel_iommu_preserve(struct iommu_device *iommu,
  				       struct iommu_hw_ser *iommu_ser)
  {
@@ -1318,6 +1326,10 @@ static inline void intel_iommu_liveupdate_restore_root_table(struct intel_iommu
  							     struct iommu_hw_ser *iommu_ser)
  {
  }
+
+static inline void pasid_cleanup_preserved_table(struct device *dev)
+{
+}
  #endif
 
  #ifdef CONFIG_INTEL_IOMMU_SVM
drivers/iommu/intel/liveupdate.c +74 -0
index 50a63812533f..404b485e97b9 100644
@@ -14,6 +14,7 @@
  #include <linux/pci.h>
 
  #include "iommu.h"
+#include "pasid.h"
  #include "../iommu-pages.h"
 
  static void unpreserve_iommu_context_table(struct intel_iommu *iommu, int end)
@@ -140,10 +141,96 @@ void intel_iommu_liveupdate_restore_root_table(struct intel_iommu *iommu,
  	iommu_for_each_preserved_device(_restore_used_domain_ids, iommu);
  }
 
+enum pasid_lu_op {
+	PASID_LU_OP_PRESERVE = 1,
+	PASID_LU_OP_UNPRESERVE,
+	PASID_LU_OP_RESTORE,
+	PASID_LU_OP_FREE,
+};
+
+static int pasid_lu_do_op(void *table, enum pasid_lu_op op)
+{
+	int ret = 0;
+
+	switch (op) {
+	case PASID_LU_OP_PRESERVE:
+		ret = iommu_preserve_page(table);
+		break;
+	case PASID_LU_OP_UNPRESERVE:
+		iommu_unpreserve_page(table);
+		break;
+	case PASID_LU_OP_RESTORE:
+		iommu_restore_page(virt_to_phys(table));
+		break;
+	case PASID_LU_OP_FREE:
+		iommu_free_pages(table);
+		break;
+	}
+
+	return ret;
+}
+
+static int pasid_lu_handle_pd(struct pasid_dir_entry *dir, enum pasid_lu_op op)
+{
+	struct pasid_entry *table;
+	int ret;
+
+	/* Only preserve first table for NO_PASID. */
+	table = get_pasid_table_from_pde(&dir[0]);
+	if (!table)
+		return -EINVAL;
+
+	ret = pasid_lu_do_op(table, op);
+	if (ret)
+		return ret;
+
+	ret = pasid_lu_do_op(dir, op);
+	if (ret)
+		goto err;
+
+	return 0;
+err:
+	if (op == PASID_LU_OP_PRESERVE)
+		pasid_lu_do_op(table, PASID_LU_OP_UNPRESERVE);
+
+	return ret;
+}
+
+void pasid_cleanup_preserved_table(struct device *dev)
+{
+	struct pasid_table *pasid_table;
+	struct pasid_dir_entry *dir;
+	struct pasid_entry *table;
+	size_t dir_size;
+
+	pasid_table = intel_pasid_get_table(dev);
+	if (!pasid_table)
+		return;
+
+	dir = pasid_table->table;
+	table = get_pasid_table_from_pde(&dir[0]);
+	if (!table)
+		return;
+
+	/* Clear everything except the first entry in table. */
+	memset(&table[1], 0, SZ_4K - sizeof(*table));

drivers/iommu/intel/liveupdate.c void intel_iommu_liveupdate_restore_root_table(struct intel_iommu *iommu, · line 226

Quoted 10 lines
drivers/iommu/intel/liveupdate.c +10 -0
+
+	/* Use the folio order to calculate the size of Pasid Directory */
+	dir_size = (1 << (folio_order(virt_to_folio(dir)) + PAGE_SHIFT));
+
+	/* Clear everything except the first entry in directory */
+	memset(&dir[1], 0, dir_size - sizeof(struct pasid_dir_entry));
+
+	clflush_cache_range(&table[0], SZ_4K);
+	clflush_cache_range(&dir[0], dir_size);
+}

The PASID table is currently active and in use by the hardware. Clearing the entries without the necessary hardware cache invalidation is buggy.

It seems this manual clearing is a workaround because PASID domain preservation isn't supported yet. If so, rather than clearing the table blindly, the code should verify if any PASIDs (other than IOMMU_NO_PASID) are actually in use. If there are, the preserve callback should return an error.

Or anything I overlooked here?

Thanks,
baolu

#64 lore raw

Re: [PATCH v2 11/16] iommu/vt-d: preserve PASID table of preserved device

Quoted 14 lines · Baolu Lu
Quoted 13 lines · Samiullah Khawaja

In scalable mode the PASID table is used to fetch the io page tables. Preserve and restore the PASID table of the preserved devices.

---
 drivers/iommu/intel/iommu.c      |   5 +-
 drivers/iommu/intel/iommu.h      |  12 +++
 drivers/iommu/intel/liveupdate.c | 141 +++++++++++++++++++++++++++++++
 drivers/iommu/intel/pasid.c      |   7 +-
 drivers/iommu/intel/pasid.h      |   9 ++
 include/linux/kho/abi/iommu.h    |  13 +++
 6 files changed, 184 insertions(+), 3 deletions(-)

[snip]

Quoted 33 lines
Quoted 19 lines
drivers/iommu/intel/liveupdate.c +19 -0
+
+void pasid_cleanup_preserved_table(struct device *dev)
+{
+	struct pasid_table *pasid_table;
+	struct pasid_dir_entry *dir;
+	struct pasid_entry *table;
+	size_t dir_size;
+
+	pasid_table = intel_pasid_get_table(dev);
+	if (!pasid_table)
+		return;
+
+	dir = pasid_table->table;
+	table = get_pasid_table_from_pde(&dir[0]);
+	if (!table)
+		return;
+
+	/* Clear everything except the first entry in table. */
+	memset(&table[1], 0, SZ_4K - sizeof(*table));

drivers/iommu/intel/liveupdate.c void intel_iommu_liveupdate_restore_root_table(struct intel_iommu *iommu, · line 226

Quoted 10 lines
drivers/iommu/intel/liveupdate.c +10 -0
+
+	/* Use the folio order to calculate the size of Pasid Directory */
+	dir_size = (1 << (folio_order(virt_to_folio(dir)) + PAGE_SHIFT));
+
+	/* Clear everything except the first entry in directory */
+	memset(&dir[1], 0, dir_size - sizeof(struct pasid_dir_entry));
+
+	clflush_cache_range(&table[0], SZ_4K);
+	clflush_cache_range(&dir[0], dir_size);
+}

The PASID table is currently active and in use by the hardware. Clearing the entries without the necessary hardware cache invalidation is buggy.

Quoted 5 lines

It seems this manual clearing is a workaround because PASID domain preservation isn't supported yet. If so, rather than clearing the table blindly, the code should verify if any PASIDs (other than IOMMU_NO_PASID) are actually in use. If there are, the preserve callback should return an error.

Thanks for looking into this, I agree and will remove the clearing logic here.

Yes, we do this check in iommufd, as it is the dma owner of the device, and only DMA owned devices are allowed to be preserved.

During preservation iommufd returns an error if the device has PASID (non NO_PASID) attachments. And once the device is preserved, any PASID attachments are not allowed until the device is unpreserved.

I think I will make this check robust by moving it into core and use pasid_array. It will require some plumbing as pasid_array exists in iommu.c file.

Quoted 5 lines

Or anything I overlooked here?

Thanks,
baolu

Thanks,
Sami

maintainer
#65 lore raw

Re: [PATCH v2 11/16] iommu/vt-d: preserve PASID table of preserved device

Quoted 62 lines · Samiullah Khawaja
Quoted 14 lines · Baolu Lu
Quoted 13 lines · Samiullah Khawaja

In scalable mode the PASID table is used to fetch the io page tables. Preserve and restore the PASID table of the preserved devices.

---
 drivers/iommu/intel/iommu.c      |   5 +- drivers/iommu/intel/iommu.h      |  12 +++ drivers/iommu/intel/liveupdate.c | 141 +++++++++++++++++++++++++++++++ drivers/iommu/intel/pasid.c      |   7 +- drivers/iommu/intel/pasid.h      |   9 ++ include/linux/kho/abi/iommu.h    |  13 +++

 6 files changed, 184 insertions(+), 3 deletions(-)

[snip]

Quoted 33 lines
Quoted 19 lines
drivers/iommu/intel/liveupdate.c +19 -0
+
+void pasid_cleanup_preserved_table(struct device *dev)
+{
+    struct pasid_table *pasid_table;
+    struct pasid_dir_entry *dir;
+    struct pasid_entry *table;
+    size_t dir_size;
+
+    pasid_table = intel_pasid_get_table(dev);
+    if (!pasid_table)
+        return;
+
+    dir = pasid_table->table;
+    table = get_pasid_table_from_pde(&dir[0]);
+    if (!table)
+        return;
+
+    /* Clear everything except the first entry in table. */
+    memset(&table[1], 0, SZ_4K - sizeof(*table));

drivers/iommu/intel/liveupdate.c void intel_iommu_liveupdate_restore_root_table(struct intel_iommu *iommu, · line 226

Quoted 10 lines
drivers/iommu/intel/liveupdate.c +10 -0
+
+    /* Use the folio order to calculate the size of Pasid Directory */
+    dir_size = (1 << (folio_order(virt_to_folio(dir)) + PAGE_SHIFT));
+
+    /* Clear everything except the first entry in directory */
+    memset(&dir[1], 0, dir_size - sizeof(struct pasid_dir_entry));
+
+    clflush_cache_range(&table[0], SZ_4K);
+    clflush_cache_range(&dir[0], dir_size);
+}

The PASID table is currently active and in use by the hardware. Clearing the entries without the necessary hardware cache invalidation is buggy.

Quoted 5 lines

It seems this manual clearing is a workaround because PASID domain preservation isn't supported yet. If so, rather than clearing the table blindly, the code should verify if any PASIDs (other than IOMMU_NO_PASID) are actually in use. If there are, the preserve callback should return an error.

Thanks for looking into this, I agree and will remove the clearing logic here.

Yes, we do this check in iommufd, as it is the dma owner of the device, and only DMA owned devices are allowed to be preserved.

Quoted 7 lines

During preservation iommufd returns an error if the device has PASID (non NO_PASID) attachments. And once the device is preserved, any PASID attachments are not allowed until the device is unpreserved.

I think I will make this check robust by moving it into core and use pasid_array. It will require some plumbing as pasid_array exists in iommu.c file.

Yeah, that would be better.

Thanks,
baolu

#66 lore raw

Re: [PATCH v2 11/16] iommu/vt-d: preserve PASID table of preserved device

In reply to Samiullah Khawaja
Quoted 100 lines · Samiullah Khawaja

In scalable mode the PASID table is used to fetch the io page tables. Preserve and restore the PASID table of the preserved devices.

---
 drivers/iommu/intel/iommu.c      |   5 +-
 drivers/iommu/intel/iommu.h      |  12 +++
 drivers/iommu/intel/liveupdate.c | 141 +++++++++++++++++++++++++++++++
 drivers/iommu/intel/pasid.c      |   7 +-
 drivers/iommu/intel/pasid.h      |   9 ++
 include/linux/kho/abi/iommu.h    |  13 +++
 6 files changed, 184 insertions(+), 3 deletions(-)
drivers/iommu/intel/iommu.c +4 -1
index b90757164cd8..6d42051dcf7c 100644
@@ -2951,8 +2951,10 @@ static int clear_unpreserve_context_entry_fn(struct device *dev,
 	if (!info)
 		return 0;
 
-	if (dev_is_pci(dev) && dev_iommu_preserved_state(dev))
+	if (dev_is_pci(dev) && dev_iommu_preserved_state(dev)) {
+		pasid_cleanup_preserved_table(dev);
 		return 0;
+	}
 
 	domain_context_clear(info);
 	return 0;
@@ -4013,6 +4015,7 @@ const struct iommu_ops intel_iommu_ops = {
 	.page_response		= intel_iommu_page_response,
 #ifdef CONFIG_IOMMU_LIVEUPDATE
 	.preserve_device	= intel_iommu_preserve_device,
+	.unpreserve_device	= intel_iommu_unpreserve_device,
 	.preserve		= intel_iommu_preserve,
 	.unpreserve		= intel_iommu_unpreserve,
 #endif
drivers/iommu/intel/iommu.h +12 -0
index 8e37acf7de12..62076a1a0b4d 100644
@@ -1290,12 +1290,15 @@ static inline int iopf_for_domain_replace(struct iommu_domain *new,
 #ifdef CONFIG_IOMMU_LIVEUPDATE
 int intel_iommu_preserve_device(struct device *dev,
 				struct iommu_device_ser *device_ser);
+void intel_iommu_unpreserve_device(struct device *dev,
+				   struct iommu_device_ser *device_ser);
 int intel_iommu_preserve(struct iommu_device *iommu,
 			 struct iommu_hw_ser *iommu_ser);
 void intel_iommu_unpreserve(struct iommu_device *iommu,
 			    struct iommu_hw_ser *iommu_ser);
 void intel_iommu_liveupdate_restore_root_table(struct intel_iommu *iommu,
 					       struct iommu_hw_ser *iommu_ser);
+void pasid_cleanup_preserved_table(struct device *dev);
 #else
 static inline int intel_iommu_preserve_device(struct device *dev,
 					      struct iommu_device_ser *device_ser)
@@ -1303,6 +1306,11 @@ static inline int intel_iommu_preserve_device(struct device *dev,
 	return -EOPNOTSUPP;
 }
 
+static inline void intel_iommu_unpreserve_device(struct device *dev,
+						 struct iommu_device_ser *device_ser)
+{
+}
+
 static inline int intel_iommu_preserve(struct iommu_device *iommu,
 				       struct iommu_hw_ser *iommu_ser)
 {
@@ -1318,6 +1326,10 @@ static inline void intel_iommu_liveupdate_restore_root_table(struct intel_iommu
 							     struct iommu_hw_ser *iommu_ser)
 {
 }
+
+static inline void pasid_cleanup_preserved_table(struct device *dev)
+{
+}
 #endif
 
 #ifdef CONFIG_INTEL_IOMMU_SVM
drivers/iommu/intel/liveupdate.c +5 -0
index 50a63812533f..404b485e97b9 100644
@@ -14,6 +14,7 @@
 #include <linux/pci.h>
 
 #include "iommu.h"
+#include "pasid.h"
 #include "../iommu-pages.h"
 
 static void unpreserve_iommu_context_table(struct intel_iommu *iommu, int end)
@@ -140,10 +141,96 @@ void intel_iommu_liveupdate_restore_root_table(struct intel_iommu *iommu,
 	iommu_for_each_preserved_device(_restore_used_domain_ids, iommu);
 }
 
+enum pasid_lu_op {
+	PASID_LU_OP_PRESERVE = 1,
+	PASID_LU_OP_UNPRESERVE,
+	PASID_LU_OP_RESTORE,

drivers/iommu/intel/liveupdate.c void intel_iommu_liveupdate_restore_root_table(struct intel_iommu *iommu, · line 157

Quoted 10 lines
drivers/iommu/intel/liveupdate.c +10 -0
+	PASID_LU_OP_FREE,
+};
+
+static int pasid_lu_do_op(void *table, enum pasid_lu_op op)
+{
+	int ret = 0;
+
+	switch (op) {
+	case PASID_LU_OP_PRESERVE:
+		ret = iommu_preserve_page(table);

Nit: This is making me consider renaming the helper as iommu_preserve_folio. I almost thought why are we preserving a single page.

Quoted 15 lines
drivers/iommu/intel/liveupdate.c +15 -0
+		break;
+	case PASID_LU_OP_UNPRESERVE:
+		iommu_unpreserve_page(table);
+		break;
+	case PASID_LU_OP_RESTORE:
+		iommu_restore_page(virt_to_phys(table));
+		break;
+	case PASID_LU_OP_FREE:
+		iommu_free_pages(table);
+		break;
+	}
+
+	return ret;
+}
+

[snip]

Quoted 9 lines
drivers/iommu/intel/liveupdate.c +9 -0
+
+void pasid_cleanup_preserved_table(struct device *dev)
+{
+	struct pasid_table *pasid_table;
+	struct pasid_dir_entry *dir;
+	struct pasid_entry *table;
+	size_t dir_size;
+
+	pasid_table = intel_pasid_get_table(dev);

drivers/iommu/intel/liveupdate.c void intel_iommu_liveupdate_restore_root_table(struct intel_iommu *iommu, · line 216

Quoted 10 lines
drivers/iommu/intel/liveupdate.c +10 -0
+	if (!pasid_table)
+		return;
+
+	dir = pasid_table->table;
+	table = get_pasid_table_from_pde(&dir[0]);
+	if (!table)
+		return;
+
+	/* Clear everything except the first entry in table. */
+	memset(&table[1], 0, SZ_4K - sizeof(*table));

Nit: Is the first entry always 4K or could it change based on PAGE_SIZE?

Quoted 11 lines
drivers/iommu/intel/liveupdate.c +11 -0
+
+	/* Use the folio order to calculate the size of Pasid Directory */
+	dir_size = (1 << (folio_order(virt_to_folio(dir)) + PAGE_SHIFT));
+
+	/* Clear everything except the first entry in directory */
+	memset(&dir[1], 0, dir_size - sizeof(struct pasid_dir_entry));
+
+	clflush_cache_range(&table[0], SZ_4K);
+	clflush_cache_range(&dir[0], dir_size);
+}
+

[...]

drivers/iommu/intel/liveupdate.c void intel_iommu_unpreserve(struct iommu_device *iommu_dev, · line 330

Quoted 10 lines
drivers/iommu/intel/liveupdate.c +10 -0
+void *intel_pasid_try_restore_table(struct device *dev, u64 max_pasid)
+{
+	struct iommu_device_ser *ser = dev_iommu_restored_state(dev);
+
+	if (!ser)
+		return NULL;
+
+	BUG_ON(pasid_lu_handle_pd(phys_to_virt(ser->intel.pasid_table),
+				  PASID_LU_OP_RESTORE));
+	if (WARN_ON_ONCE(ser->intel.max_pasid != max_pasid)) {

I'm wondering if this could be slightly relaxed to:
if (ser->intel.max_pasid < max_pasid) to ensure it's a minimum requirement rather than an exact match?

Quoted 15 lines
drivers/iommu/intel/liveupdate.c +7 -0
+		pasid_lu_handle_pd(phys_to_virt(ser->intel.pasid_table),
+				   PASID_LU_OP_FREE);
+		return NULL;
+	}
+
+	return phys_to_virt(ser->intel.pasid_table);
+}
drivers/iommu/intel/pasid.c +0 -0
index 89541b74ab8c..5cac8e95f73b 100644
@@ -60,8 +60,11 @@ int intel_pasid_alloc_table(struct device *dev)
 
 	size = max_pasid >> (PASID_PDE_SHIFT - 3);
 	order = size ? get_order(size) : 0;

drivers/iommu/intel/pasid.c int intel_pasid_alloc_table(struct device *dev) · line 70

Quoted 10 lines
drivers/iommu/intel/pasid.c +5 -2
-	dir = iommu_alloc_pages_node_sz(info->iommu->node, GFP_KERNEL,
-					1 << (order + PAGE_SHIFT));
+
+	dir = intel_pasid_try_restore_table(dev, 1 << (order + PAGE_SHIFT + 3));
+	if (!dir)
+		dir = iommu_alloc_pages_node_sz(info->iommu->node, GFP_KERNEL,
+						1 << (order + PAGE_SHIFT));
 	if (!dir) {
 		kfree(pasid_table);
 		return -ENOMEM;

Thanks,
Praan

#67 lore raw

Re: [PATCH v2 11/16] iommu/vt-d: preserve PASID table of preserved device

Quoted 14 lines · Pranjal Shrivastava
Quoted 13 lines · Samiullah Khawaja

In scalable mode the PASID table is used to fetch the io page tables. Preserve and restore the PASID table of the preserved devices.

---
 drivers/iommu/intel/iommu.c      |   5 +-
 drivers/iommu/intel/iommu.h      |  12 +++
 drivers/iommu/intel/liveupdate.c | 141 +++++++++++++++++++++++++++++++
 drivers/iommu/intel/pasid.c      |   7 +-
 drivers/iommu/intel/pasid.h      |   9 ++
 include/linux/kho/abi/iommu.h    |  13 +++
 6 files changed, 184 insertions(+), 3 deletions(-)

[snip]

Quoted 4 lines
Quoted 4 lines
drivers/iommu/intel/liveupdate.c +4 -0
+
+static int pasid_lu_do_op(void *table, enum pasid_lu_op op)
+{
+	int ret = 0;
Quoted 8 lines

drivers/iommu/intel/liveupdate.c void intel_iommu_liveupdate_restore_root_table(struct intel_iommu *iommu, · line 157

Quoted 4 lines
drivers/iommu/intel/liveupdate.c +4 -0
+
+	switch (op) {
+	case PASID_LU_OP_PRESERVE:
+		ret = iommu_preserve_page(table);

Nit: This is making me consider renaming the helper as iommu_preserve_folio. I almost thought why are we preserving a single page.

Interestingly the iommu pages API uses plural of page in API name as each iopt_desc can be backed by multiple pages:

iommu_free_pages()
iommu_alloc_pages_*()

So I will rename these to:

iommu_preserve_pages()
iommu_preserve_pages_list(list)
iommu_unpreserve_pages()
iommu_unpreserve_pages_list(list)
iommu_restore_pages()

Quoted 32 lines
Quoted 16 lines
drivers/iommu/intel/liveupdate.c +15 -0
+		break;
+	case PASID_LU_OP_UNPRESERVE:
+		iommu_unpreserve_page(table);
+		break;
+	case PASID_LU_OP_RESTORE:
+		iommu_restore_page(virt_to_phys(table));
+		break;
+	case PASID_LU_OP_FREE:
+		iommu_free_pages(table);
+		break;
+	}
+
+	return ret;
+}
+

[snip]

Quoted 13 lines
drivers/iommu/intel/liveupdate.c +13 -0
+
+void pasid_cleanup_preserved_table(struct device *dev)
+{
+	struct pasid_table *pasid_table;
+	struct pasid_dir_entry *dir;
+	struct pasid_entry *table;
+	size_t dir_size;
+
+	pasid_table = intel_pasid_get_table(dev);
+	if (!pasid_table)
+		return;
+
+	dir = pasid_table->table;
Quoted 8 lines

drivers/iommu/intel/liveupdate.c void intel_iommu_liveupdate_restore_root_table(struct intel_iommu *iommu, · line 216

Quoted 6 lines
drivers/iommu/intel/liveupdate.c +6 -0
+	table = get_pasid_table_from_pde(&dir[0]);
+	if (!table)
+		return;
+
+	/* Clear everything except the first entry in table. */
+	memset(&table[1], 0, SZ_4K - sizeof(*table));

Nit: Is the first entry always 4K or could it change based on PAGE_SIZE?

VT-d uses 4k always, but for clarity I will change this to VTD_PAGE_SIZE.

Quoted 15 lines
Quoted 12 lines
drivers/iommu/intel/liveupdate.c +11 -0
+
+	/* Use the folio order to calculate the size of Pasid Directory */
+	dir_size = (1 << (folio_order(virt_to_folio(dir)) + PAGE_SHIFT));
+
+	/* Clear everything except the first entry in directory */
+	memset(&dir[1], 0, dir_size - sizeof(struct pasid_dir_entry));
+
+	clflush_cache_range(&table[0], SZ_4K);
+	clflush_cache_range(&dir[0], dir_size);
+}
+

[...]

Quoted 14 lines

drivers/iommu/intel/liveupdate.c void intel_iommu_unpreserve(struct iommu_device *iommu_dev, · line 330

Quoted 10 lines
drivers/iommu/intel/liveupdate.c +10 -0
+void *intel_pasid_try_restore_table(struct device *dev, u64 max_pasid)
+{
+	struct iommu_device_ser *ser = dev_iommu_restored_state(dev);
+
+	if (!ser)
+		return NULL;
+
+	BUG_ON(pasid_lu_handle_pd(phys_to_virt(ser->intel.pasid_table),
+				  PASID_LU_OP_RESTORE));
+	if (WARN_ON_ONCE(ser->intel.max_pasid != max_pasid)) {

I'm wondering if this could be slightly relaxed to:
if (ser->intel.max_pasid < max_pasid) to ensure it's a minimum requirement rather than an exact match?

Makes sense. I will update this.

Quoted 14 lines
Quoted 14 lines
drivers/iommu/intel/liveupdate.c +7 -0
+		pasid_lu_handle_pd(phys_to_virt(ser->intel.pasid_table),
+				   PASID_LU_OP_FREE);
+		return NULL;
+	}
+
+	return phys_to_virt(ser->intel.pasid_table);
+}
drivers/iommu/intel/pasid.c +0 -0
index 89541b74ab8c..5cac8e95f73b 100644
@@ -60,8 +60,11 @@ int intel_pasid_alloc_table(struct device *dev)
 
Quoted 15 lines

drivers/iommu/intel/pasid.c int intel_pasid_alloc_table(struct device *dev) · line 70

Quoted 12 lines
drivers/iommu/intel/pasid.c +5 -2
 	size = max_pasid >> (PASID_PDE_SHIFT - 3);
 	order = size ? get_order(size) : 0;
-	dir = iommu_alloc_pages_node_sz(info->iommu->node, GFP_KERNEL,
-					1 << (order + PAGE_SHIFT));
+
+	dir = intel_pasid_try_restore_table(dev, 1 << (order + PAGE_SHIFT + 3));
+	if (!dir)
+		dir = iommu_alloc_pages_node_sz(info->iommu->node, GFP_KERNEL,
+						1 << (order + PAGE_SHIFT));
 	if (!dir) {
 		kfree(pasid_table);
 		return -ENOMEM;

Thanks,
Praan

Sami

#68 lore raw

PATCH v2 12/16 iommufd: Implement ioctl to mark HWPT for preservation

In reply to Samiullah Khawaja

From: YiFei Zhu <[email protected]>

Userspace provides a token to mark the HWPT for preservation. Note that this token is not the LUO token that is used to preserve the iommufd. Once all the required HWPT are marked for preservation, the user can preserve the iommufd into LUO. The iommufd will preserve the HWPTs that are marked for preservation.

The marked HWPTs are tracked using a new XArray mark protected by a new liveupdate mutex. This mutex will also be used during iommufd preservation to protect against any race with the mark preserve ioctl.

The HWPT token will be used during restore to identify this HWPT. The restoration logic is not implemented and will be added later.

---
 MAINTAINERS                             |  1 +
 drivers/iommu/iommufd/Makefile          |  1 +
 drivers/iommu/iommufd/iommufd_private.h | 18 +++++++++
 drivers/iommu/iommufd/liveupdate.c      | 52 +++++++++++++++++++++++++
 drivers/iommu/iommufd/main.c            |  9 +++++
 include/uapi/linux/iommufd.h            | 26 +++++++++++++
 6 files changed, 107 insertions(+)
 create mode 100644 drivers/iommu/iommufd/liveupdate.c
MAINTAINERS +1 -0
index 9f5c02c6c8c1..bf6a2ad61989 100644
@@ -13496,6 +13496,7 @@ R:	Pranjal Shrivastava <[email protected]>
 L:	[email protected]
 S:	Maintained
 F:	drivers/iommu/intel/liveupdate.c
+F:	drivers/iommu/iommufd/liveupdate.c
 F:	drivers/iommu/liveupdate.c
 F:	include/linux/iommu-liveupdate.h
 F:	include/linux/kho/abi/iommu.h
drivers/iommu/iommufd/Makefile +1 -0
index 71d692c9a8f4..c3bf0b6452d3 100644
@@ -17,3 +17,4 @@ obj-$(CONFIG_IOMMUFD_DRIVER) += iova_bitmap.o
 
 iommufd_driver-y := driver.o
 obj-$(CONFIG_IOMMUFD_DRIVER_CORE) += iommufd_driver.o
+obj-$(CONFIG_IOMMU_LIVEUPDATE) += liveupdate.o
drivers/iommu/iommufd/iommufd_private.h +18 -0
index 6ac1965199e9..111f4d42e210 100644
@@ -44,6 +44,11 @@ struct iommufd_ctx {
 	struct file *file;
 	struct xarray objects;
 	struct xarray groups;
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+#define IOMMUFD_OBJ_LIVEUPDATE_MARK XA_MARK_1
+	/* @liveupdate_mutex: Protects the preservation of HWPTs. */
+	struct mutex liveupdate_mutex;
+#endif
 	wait_queue_head_t destroy_wait;
 	struct rw_semaphore ioas_creation_lock;
 	struct maple_tree mt_mmap;
@@ -373,6 +378,10 @@ struct iommufd_hwpt_paging {
 	bool auto_domain : 1;
 	bool enforce_cache_coherency : 1;
 	bool nest_parent : 1;
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	bool liveupdate_preserve : 1;
+	u64 liveupdate_token;
+#endif
 	/* Head at iommufd_ioas::hwpt_list */
 	struct list_head hwpt_item;
 	struct iommufd_sw_msi_maps present_sw_msi;
@@ -706,6 +715,15 @@ iommufd_get_vdevice(struct iommufd_ctx *ictx, u32 id)
 			    struct iommufd_vdevice, obj);
 }
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+int iommufd_hwpt_liveupdate_mark_preserve(struct iommufd_ucmd *ucmd);
+#else
+static inline int iommufd_hwpt_liveupdate_mark_preserve(struct iommufd_ucmd *ucmd)
+{
+	return -ENOTTY;
+}
+#endif
+
 #ifdef CONFIG_IOMMUFD_TEST
 int iommufd_test(struct iommufd_ucmd *ucmd);
 void iommufd_selftest_destroy(struct iommufd_object *obj);
drivers/iommu/iommufd/liveupdate.c new file +52 -0

52 lines not on this page · raw message

drivers/iommu/iommufd/main.c +9 -0
index 8c6d43601afb..0114c1520db4 100644
@@ -313,6 +313,9 @@ static int iommufd_fops_open(struct inode *inode, struct file *filp)
 	init_rwsem(&ictx->ioas_creation_lock);
 	xa_init_flags(&ictx->objects, XA_FLAGS_ALLOC1 | XA_FLAGS_ACCOUNT);
 	xa_init(&ictx->groups);
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	mutex_init(&ictx->liveupdate_mutex);
+#endif
 	ictx->file = filp;
 	mt_init_flags(&ictx->mt_mmap, MT_FLAGS_ALLOC_RANGE);
 	init_waitqueue_head(&ictx->destroy_wait);
@@ -375,6 +378,9 @@ static int iommufd_fops_release(struct inode *inode, struct file *filp)
 	 * iommufd_object_tombstone_user()
 	 */
 	xa_destroy(&ictx->objects);
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	mutex_destroy(&ictx->liveupdate_mutex);
+#endif
 
 	WARN_ON(!xa_empty(&ictx->groups));
 
@@ -420,6 +426,7 @@ union ucmd_buffer {
 	struct iommu_hwpt_alloc hwpt;
 	struct iommu_hwpt_get_dirty_bitmap get_dirty_bitmap;
 	struct iommu_hwpt_invalidate cache;
+	struct iommu_hwpt_liveupdate_mark_preserve mark_preserve;
 	struct iommu_hwpt_set_dirty_tracking set_dirty_tracking;
 	struct iommu_ioas_alloc alloc;
 	struct iommu_ioas_allow_iovas allow_iovas;
@@ -493,6 +500,8 @@ static const struct iommufd_ioctl_op iommufd_ioctl_ops[] = {
 		 __reserved),
 	IOCTL_OP(IOMMU_VIOMMU_ALLOC, iommufd_viommu_alloc_ioctl,
 		 struct iommu_viommu_alloc, out_viommu_id),
+	IOCTL_OP(IOMMU_HWPT_LIVEUPDATE_MARK_PRESERVE, iommufd_hwpt_liveupdate_mark_preserve,
+		 struct iommu_hwpt_liveupdate_mark_preserve, hwpt_token),
 #ifdef CONFIG_IOMMUFD_TEST
 	IOCTL_OP(IOMMU_TEST_CMD, iommufd_test, struct iommu_test_cmd, last),
 #endif
include/uapi/linux/iommufd.h +26 -0
index e998dfbd6960..d96a74b43c9d 100644
@@ -57,6 +57,7 @@ enum {
 	IOMMUFD_CMD_IOAS_CHANGE_PROCESS = 0x92,
 	IOMMUFD_CMD_VEVENTQ_ALLOC = 0x93,
 	IOMMUFD_CMD_HW_QUEUE_ALLOC = 0x94,
+	IOMMUFD_CMD_HWPT_LU_MARK_PRESERVE = 0x95,
 };
 
 /**
@@ -1347,4 +1348,29 @@ struct iommu_hw_queue_alloc {
 	__aligned_u64 length;
 };
 #define IOMMU_HW_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HW_QUEUE_ALLOC)
+
+/**
+ * struct iommu_hwpt_liveupdate_mark_preserve - ioctl(IOMMU_HWPT_LIVEUPDATE_MARK_PRESERVE)
+ * @size: sizeof(struct iommu_hwpt_liveupdate_mark_preserve)
+ * @hwpt_id: Iommufd object ID of the target HWPT
+ * @hwpt_token: Token to identify this hwpt upon restore
+ *
+ * The target HWPT will be preserved during iommufd preservation.
+ * Only file-based memory mappings (e.g. memfd) are supported for HWPTs marked
+ * for preservation. Mapping anonymous memory into a preserved HWPT will result
+ * in a failure during the preservation phase.
+ *
+ * The hwpt_token is provided by userspace. If userspace enters a token
+ * already in use within this iommufd, -EADDRINUSE is returned from this ioctl.
+ *
+ * Note: There is no 'unmark' operation, so any HWPTs pooled in userspace that
+ * are marked for preservation must be destroyed after use.
+ */
+struct iommu_hwpt_liveupdate_mark_preserve {
+	__u32 size;
+	__u32 hwpt_id;
+	__u64 hwpt_token;
+};
+#define IOMMU_HWPT_LIVEUPDATE_MARK_PRESERVE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_LU_MARK_PRESERVE)
+
 #endif
Signature · 1 line
2.54.0.545.g6539524ca2-goog
Reviewed-by on v2 12/16
#69 lore raw

Re: [PATCH v2 12/16] iommufd: Implement ioctl to mark HWPT for preservation

Quoted 17 lines · Samiullah Khawaja

From: YiFei Zhu <[email protected]>

Userspace provides a token to mark the HWPT for preservation. Note that this token is not the LUO token that is used to preserve the iommufd. Once all the required HWPT are marked for preservation, the user can preserve the iommufd into LUO. The iommufd will preserve the HWPTs that are marked for preservation.

The marked HWPTs are tracked using a new XArray mark protected by a new liveupdate mutex. This mutex will also be used during iommufd preservation to protect against any race with the mark preserve ioctl.

The HWPT token will be used during restore to identify this HWPT. The restoration logic is not implemented and will be added later.

[snip]

Quoted 17 lines
drivers/iommu/iommufd/iommufd_private.h +5 -0
index 6ac1965199e9..111f4d42e210 100644
@@ -44,6 +44,11 @@ struct iommufd_ctx {
 	struct file *file;
 	struct xarray objects;
 	struct xarray groups;
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+#define IOMMUFD_OBJ_LIVEUPDATE_MARK XA_MARK_1
+	/* @liveupdate_mutex: Protects the preservation of HWPTs. */
+	struct mutex liveupdate_mutex;
+#endif
 	wait_queue_head_t destroy_wait;
 	struct rw_semaphore ioas_creation_lock;
 	struct maple_tree mt_mmap;

drivers/iommu/iommufd/iommufd_private.h struct iommufd_hwpt_paging { · line 382

Quoted 5 lines
drivers/iommu/iommufd/iommufd_private.h +2 -0
@@ -373,6 +378,10 @@ struct iommufd_hwpt_paging {
 	bool auto_domain : 1;
 	bool enforce_cache_coherency : 1;
 	bool nest_parent : 1;
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	bool liveupdate_preserve : 1;

Nit: Is this bool a remnant from v1 that made into this refactor? I don't see us using it anywhere?

Quoted 9 lines
drivers/iommu/iommufd/iommufd_private.h +2 -0
+	u64 liveupdate_token;
+#endif
 	/* Head at iommufd_ioas::hwpt_list */
 	struct list_head hwpt_item;
 	struct iommufd_sw_msi_maps present_sw_msi;
@@ -706,6 +715,15 @@ iommufd_get_vdevice(struct iommufd_ctx *ictx, u32 id)
 			    struct iommufd_vdevice, obj);
 }
 

[...]

Quoted 18 lines
drivers/iommu/iommufd/iommufd_private.h +18 -0
+int iommufd_hwpt_liveupdate_mark_preserve(struct iommufd_ucmd *ucmd)
+{
+	struct iommu_hwpt_liveupdate_mark_preserve *cmd = ucmd->cmd;
+	struct iommufd_hwpt_paging *hwpt_target;
+	struct iommufd_hwpt_paging *hwpt_paging;
+	struct iommufd_ctx *ictx = ucmd->ictx;
+	struct iommufd_object *obj;
+	unsigned long index;
+	int rc = 0;
+
+	hwpt_target = iommufd_get_hwpt_paging(ucmd, cmd->hwpt_id);
+	if (IS_ERR(hwpt_target))
+		return PTR_ERR(hwpt_target);
+
+	mutex_lock(&ictx->liveupdate_mutex);
+
+	xa_lock(&ictx->objects);
+	xa_for_each_marked(&ictx->objects, index, obj, IOMMUFD_OBJ_LIVEUPDATE_MARK) {

drivers/iommu/iommufd/iommufd_private.h

Quoted 10 lines
drivers/iommu/iommufd/iommufd_private.h +10 -0
+		if (WARN_ON_ONCE(obj->type != IOMMUFD_OBJ_HWPT_PAGING))
+			continue;
+
+		hwpt_paging = to_hwpt_paging(container_of(obj, struct iommufd_hw_pagetable, obj));
+		if (hwpt_paging->liveupdate_token == cmd->hwpt_token) {
+			rc = -EADDRINUSE;
+			goto out_unlock;
+		}
+	}
+

Nit: What happens if the user calls this IOCTL on an HWPT that is already marked but passes a different token?

The xa_for_each_marked loop will not match the new token (so it bypasses -EADDRINUSE case) and the code will silently overwrite the HWPT's existing token?

Since there is no unmark / update UAPI, It can be a fair policy as well, but maybe we should call it out explicitly, maybe a log like:
dev_warn("Overwriting token to %d ...") ?

If NOT, then we should avoid overwriting the token and scream with retval like -EINVAL & dev_err in dmesg.

drivers/iommu/iommufd/iommufd_private.h

Quoted 9 lines
drivers/iommu/iommufd/iommufd_private.h +9 -0
+	__xa_set_mark(&ictx->objects, hwpt_target->common.obj.id, IOMMUFD_OBJ_LIVEUPDATE_MARK);
+	hwpt_target->liveupdate_token = cmd->hwpt_token;
+
+out_unlock:
+	xa_unlock(&ictx->objects);
+	mutex_unlock(&ictx->liveupdate_mutex);
+	iommufd_put_object(ictx, &hwpt_target->common.obj);
+	return rc;
+}

With those two nits addressed,

Thanks,
Praan

#70 lore raw

Re: [PATCH v2 12/16] iommufd: Implement ioctl to mark HWPT for preservation

Quoted 19 lines · Pranjal Shrivastava
Quoted 17 lines · Samiullah Khawaja

From: YiFei Zhu <[email protected]>

Userspace provides a token to mark the HWPT for preservation. Note that this token is not the LUO token that is used to preserve the iommufd. Once all the required HWPT are marked for preservation, the user can preserve the iommufd into LUO. The iommufd will preserve the HWPTs that are marked for preservation.

The marked HWPTs are tracked using a new XArray mark protected by a new liveupdate mutex. This mutex will also be used during iommufd preservation to protect against any race with the mark preserve ioctl.

The HWPT token will be used during restore to identify this HWPT. The restoration logic is not implemented and will be added later.

[snip]

Quoted 33 lines
Quoted 19 lines
drivers/iommu/iommufd/liveupdate.c new file +18 -0
+int iommufd_hwpt_liveupdate_mark_preserve(struct iommufd_ucmd *ucmd)
+{
+	struct iommu_hwpt_liveupdate_mark_preserve *cmd = ucmd->cmd;
+	struct iommufd_hwpt_paging *hwpt_target;
+	struct iommufd_hwpt_paging *hwpt_paging;
+	struct iommufd_ctx *ictx = ucmd->ictx;
+	struct iommufd_object *obj;
+	unsigned long index;
+	int rc = 0;
+
+	hwpt_target = iommufd_get_hwpt_paging(ucmd, cmd->hwpt_id);
+	if (IS_ERR(hwpt_target))
+		return PTR_ERR(hwpt_target);
+
+	mutex_lock(&ictx->liveupdate_mutex);
+
+	xa_lock(&ictx->objects);
+	xa_for_each_marked(&ictx->objects, index, obj, IOMMUFD_OBJ_LIVEUPDATE_MARK) {

drivers/iommu/iommufd/liveupdate.c line 43

Quoted 10 lines
drivers/iommu/iommufd/liveupdate.c new file +10 -0
+		if (WARN_ON_ONCE(obj->type != IOMMUFD_OBJ_HWPT_PAGING))
+			continue;
+
+		hwpt_paging = to_hwpt_paging(container_of(obj, struct iommufd_hw_pagetable, obj));
+		if (hwpt_paging->liveupdate_token == cmd->hwpt_token) {
+			rc = -EADDRINUSE;
+			goto out_unlock;
+		}
+	}
+

Nit: What happens if the user calls this IOCTL on an HWPT that is already marked but passes a different token?

Quoted 7 lines

The xa_for_each_marked loop will not match the new token (so it bypasses -EADDRINUSE case) and the code will silently overwrite the HWPT's existing token?

Since there is no unmark / update UAPI, It can be a fair policy as well, but maybe we should call it out explicitly, maybe a log like:
dev_warn("Overwriting token to %d ...") ?

Agreed. I will do this.

Quoted 16 lines

If NOT, then we should avoid overwriting the token and scream with retval like -EINVAL & dev_err in dmesg.

drivers/iommu/iommufd/liveupdate.c line 52

Quoted 9 lines
drivers/iommu/iommufd/liveupdate.c new file +9 -0
+	__xa_set_mark(&ictx->objects, hwpt_target->common.obj.id, IOMMUFD_OBJ_LIVEUPDATE_MARK);
+	hwpt_target->liveupdate_token = cmd->hwpt_token;
+
+out_unlock:
+	xa_unlock(&ictx->objects);
+	mutex_unlock(&ictx->liveupdate_mutex);
+	iommufd_put_object(ictx, &hwpt_target->common.obj);
+	return rc;
+}

With those two nits addressed,

Quoted 4 lines

Thanks,
Praan

Thanks,
Sami

#71 lore raw

PATCH v2 13/16 iommufd: Persist iommu hardware pagetables for live update

In reply to Samiullah Khawaja

From: YiFei Zhu <[email protected]>

Register iommufd with the LUO framework and implement the preserve and unpreserve ops to save marked HWPTs.

To make sure mappings do not change during preserved state, add a liveupdate_immutable flag to IOAS. When an HWPT is preserved, its IOAS is marked immutable and any map/unmap attempts will fail with -EBUSY. This is synchronized using the domains_rwsem to prevent races with concurrent mapping operations.

The preserve callback iterates over the marked HWPTs, verifies that the backing memory pages are preserved, and calls iommu_domain_preserve() to preserve the associated IOMMU domain.

---
 MAINTAINERS                             |   1 +
 drivers/iommu/iommufd/io_pagetable.c    |  11 +
 drivers/iommu/iommufd/io_pagetable.h    |   1 +
 drivers/iommu/iommufd/iommufd_private.h |  27 ++-
 drivers/iommu/iommufd/liveupdate.c      | 287 ++++++++++++++++++++++++
 drivers/iommu/iommufd/main.c            |  10 +-
 drivers/iommu/iommufd/pages.c           |   7 +
 include/linux/kho/abi/iommufd.h         |  51 +++++
 8 files changed, 393 insertions(+), 2 deletions(-)
 create mode 100644 include/linux/kho/abi/iommufd.h
MAINTAINERS +1 -0
index bf6a2ad61989..6005b737d1c5 100644
@@ -13500,6 +13500,7 @@ F:	drivers/iommu/iommufd/liveupdate.c
 F:	drivers/iommu/liveupdate.c
 F:	include/linux/iommu-liveupdate.h
 F:	include/linux/kho/abi/iommu.h
+F:	include/linux/kho/abi/iommufd.h
 
 IOMMUFD
 M:	Jason Gunthorpe <[email protected]>
drivers/iommu/iommufd/io_pagetable.c +11 -0
index 24d4917105d9..b18dba9dd147 100644
@@ -384,6 +384,11 @@ int iopt_map_pages(struct io_pagetable *iopt, struct list_head *pages_list,
 		return rc;
 
 	down_read(&iopt->domains_rwsem);
+	if (iopt_liveupdate_immutable(iopt)) {
+		rc = -EBUSY;
+		goto out_unlock_domains;
+	}
+
 	rc = iopt_fill_domains_pages(pages_list);
 	if (rc)
 		goto out_unlock_domains;
@@ -755,6 +760,12 @@ static int iopt_unmap_iova_range(struct io_pagetable *iopt, unsigned long start,
 again:
 	down_read(&iopt->domains_rwsem);
 	down_write(&iopt->iova_rwsem);
+
+	if (iopt_liveupdate_immutable(iopt)) {
+		rc = -EBUSY;
+		goto out_unlock_iova;
+	}
+
 	while ((area = iopt_area_iter_first(iopt, start, last))) {
 		unsigned long area_last = iopt_area_last_iova(area);
 		unsigned long area_first = iopt_area_iova(area);
drivers/iommu/iommufd/io_pagetable.h +1 -0
index 27e3e311d395..207ff368d412 100644
@@ -234,6 +234,7 @@ struct iopt_pages {
 		struct {			/* IOPT_ADDRESS_FILE */
 			struct file *file;
 			unsigned long start;
+			u32 seals;
 		};
 		/* IOPT_ADDRESS_DMABUF */
 		struct iopt_pages_dmabuf dmabuf;
drivers/iommu/iommufd/iommufd_private.h +26 -1
index 111f4d42e210..3c88aa115d08 100644
@@ -98,6 +98,9 @@ struct io_pagetable {
 	/* IOVA that cannot be allocated, struct iopt_reserved */
 	struct rb_root_cached reserved_itree;
 	u8 disable_large_pages;
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	bool liveupdate_immutable;
+#endif
 	unsigned long iova_alignment;
 };
 
@@ -379,7 +382,7 @@ struct iommufd_hwpt_paging {
 	bool enforce_cache_coherency : 1;
 	bool nest_parent : 1;
 #ifdef CONFIG_IOMMU_LIVEUPDATE
-	bool liveupdate_preserve : 1;
+	bool liveupdate_preserved : 1;
 	u64 liveupdate_token;
 #endif
 	/* Head at iommufd_ioas::hwpt_list */
@@ -716,12 +719,34 @@ iommufd_get_vdevice(struct iommufd_ctx *ictx, u32 id)
 }
 
 #ifdef CONFIG_IOMMU_LIVEUPDATE
+int iommufd_liveupdate_register(void);
+void iommufd_liveupdate_unregister(void);
+
 int iommufd_hwpt_liveupdate_mark_preserve(struct iommufd_ucmd *ucmd);
+
+static inline bool iopt_liveupdate_immutable(const struct io_pagetable *iopt)
+{
+	return iopt->liveupdate_immutable;
+}
 #else
+static inline int iommufd_liveupdate_register(void)
+{
+	return 0;
+}
+
+static inline void iommufd_liveupdate_unregister(void)
+{
+}
+
 static inline int iommufd_hwpt_liveupdate_mark_preserve(struct iommufd_ucmd *ucmd)
 {
 	return -ENOTTY;
 }
+
+static inline bool iopt_liveupdate_immutable(const struct io_pagetable *iopt)
+{
+	return false;
+}
 #endif
 
 #ifdef CONFIG_IOMMUFD_TEST
drivers/iommu/iommufd/liveupdate.c +287 -0

299 lines not on this page · raw message

drivers/iommu/iommufd/main.c +9 -1
index 0114c1520db4..0a7e7bb586d7 100644
@@ -782,11 +782,18 @@ static int __init iommufd_init(void)
 		if (ret)
 			goto err_misc;
 	}
-	ret = iommufd_test_init();
+
+	ret = iommufd_liveupdate_register();
 	if (ret)
 		goto err_vfio_misc;
+
+	ret = iommufd_test_init();
+	if (ret)
+		goto err_liveupdate;
 	return 0;
 
+err_liveupdate:
+	iommufd_liveupdate_unregister();
 err_vfio_misc:
 	if (IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER))
 		misc_deregister(&vfio_misc_dev);
@@ -798,6 +805,7 @@ static int __init iommufd_init(void)
 static void __exit iommufd_exit(void)
 {
 	iommufd_test_exit();
+	iommufd_liveupdate_unregister();
 	if (IS_ENABLED(CONFIG_IOMMUFD_VFIO_CONTAINER))
 		misc_deregister(&vfio_misc_dev);
 	misc_deregister(&iommu_misc_dev);
drivers/iommu/iommufd/pages.c +7 -0
index 9bdb2945afe1..3b0c0acb8856 100644
@@ -55,6 +55,7 @@
 #include <linux/overflow.h>
 #include <linux/slab.h>
 #include <linux/sched/mm.h>
+#include <linux/memfd.h>
 #include <linux/vfio_pci_core.h>
 
 #include "double_span.h"
@@ -1421,6 +1422,7 @@ struct iopt_pages *iopt_alloc_file_pages(struct file *file,
 
 {
 	struct iopt_pages *pages;
+	int seals;
 
 	pages = iopt_alloc_pages(start_byte, length, writable);
 	if (IS_ERR(pages))
@@ -1428,6 +1430,11 @@ struct iopt_pages *iopt_alloc_file_pages(struct file *file,
 	pages->file = get_file(file);
 	pages->start = start - start_byte;
 	pages->type = IOPT_ADDRESS_FILE;
+
+	seals = memfd_get_seals(file);
+	if (seals > 0)
+		pages->seals = seals;
+
 	return pages;
 }
 
include/linux/kho/abi/iommufd.h new file +51 -0

51 lines not on this page · raw message

Signature · 1 line
2.54.0.545.g6539524ca2-goog
#72 lore raw

Re: [PATCH v2 13/16] iommufd: Persist iommu hardware pagetables for live update

Quoted 17 lines · Samiullah Khawaja

From: YiFei Zhu <[email protected]>

Register iommufd with the LUO framework and implement the preserve and unpreserve ops to save marked HWPTs.

To make sure mappings do not change during preserved state, add a liveupdate_immutable flag to IOAS. When an HWPT is preserved, its IOAS is marked immutable and any map/unmap attempts will fail with -EBUSY. This is synchronized using the domains_rwsem to prevent races with concurrent mapping operations.

The preserve callback iterates over the marked HWPTs, verifies that the backing memory pages are preserved, and calls iommu_domain_preserve() to preserve the associated IOMMU domain.

[...]

Quoted 15 lines
drivers/iommu/iommufd/iommufd_private.h +3 -0
index 111f4d42e210..3c88aa115d08 100644
@@ -98,6 +98,9 @@ struct io_pagetable {
 	/* IOVA that cannot be allocated, struct iopt_reserved */
 	struct rb_root_cached reserved_itree;
 	u8 disable_large_pages;
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	bool liveupdate_immutable;
+#endif
 	unsigned long iova_alignment;
 };
 

drivers/iommu/iommufd/iommufd_private.h struct iommufd_hwpt_paging { · line 385

Quoted 5 lines
drivers/iommu/iommufd/iommufd_private.h +1 -1
@@ -379,7 +382,7 @@ struct iommufd_hwpt_paging {
 	bool enforce_cache_coherency : 1;
 	bool nest_parent : 1;
 #ifdef CONFIG_IOMMU_LIVEUPDATE
-	bool liveupdate_preserve : 1;
+	bool liveupdate_preserved : 1;

Ahh okay, that's why I didn't find any reference earlier. (I searched for liveupdate_preserve), Please ignore the remnant comment in Patch 12.

Quoted 2 lines
drivers/iommu/iommufd/iommufd_private.h +0 -0
 	u64 liveupdate_token;
 #endif

[...]

Quoted 13 lines
drivers/iommu/iommufd/iommufd_private.h +13 -0
+
+static int iommufd_preserve_hwpt(struct iommufd_hwpt_paging *hwpt,
+				 struct iommufd_hwpt_ser *hwpt_ser,
+				 struct liveupdate_session *session)
+{
+	struct iommu_domain_ser *domain_ser;
+	bool ioas_made_immutable = false;
+	int rc;
+
+	if (!hwpt->ioas->iopt.liveupdate_immutable) {
+		/*
+		 * Make IOAS immutable so the DMA mappings do not change while
+		 * the HWPT is preserved. Since one IOAS can have multiple

drivers/iommu/iommufd/iommufd_private.h

Quoted 10 lines
drivers/iommu/iommufd/iommufd_private.h +10 -0
+		 * HWPTs, if an error occurs this call needs to make the IOAS
+		 * mutable again if it was the one that made it immutable.
+		 */
+		ioas_made_immutable = true;
+		ioas_set_immutable(hwpt->ioas, true);
+
+		rc = check_iopt_pages_preserved(session, hwpt);
+		if (rc)
+			goto err;
+	}

Nit:
I'm thinking what happens for a shared IOAS situation? Say, 2 devices, in the same container, behind 2 different IOMMUs, sharing the IOAS. Each device will have it's own HWPT (N:1 mapping for HWPT v/s IOAS)

So, what happens if Device 1 succeeds with preservation but device 2 doesn't? For example:

1. Preserve Device 1:
- It sees liveupdate_immutable is false.
- Sets ioas_made_immutable = true on its local stack.
- Flips IOAS to immutable.
- Preservation succeeds.

2. Preserve Device 2:
- It sees liveupdate_immutable is already true (because Device 1 set it).
- Sets ioas_made_immutable = false on its local stack.
- The Failure: iommu_domain_preserve fails for Device 2.
- The Jump: Hits goto err;

Now, inside the err: label for device 2, it checks if (ioas_made_immutable), since it is FALSE for device 2, it does nothing.

I agree we return an error code to the caller which finally cleans it up well, but I'm considering if we should make liveupdate_immutable refcountable? Since the error handling in iommufd_preserve_hwpt() is logically incomplete for shared IOAS as it only attempts to restore mutability if the current HWPT set immutable = true;

Quoted 17 lines
drivers/iommu/iommufd/iommufd_private.h +17 -0
+
+	hwpt_ser->token = hwpt->liveupdate_token;
+	hwpt_ser->reclaimed = false;
+
+	rc = iommu_domain_preserve(hwpt->common.domain, &domain_ser);
+	if (rc < 0)
+		goto err;
+
+	hwpt_ser->domain_data = virt_to_phys(domain_ser);
+	return 0;
+
+err:
+	if (ioas_made_immutable)
+		ioas_set_immutable(hwpt->ioas, false);
+
+	return rc;
+}

[...]

Quoted 34 lines
drivers/iommu/iommufd/iommufd_private.h +34 -0
+
+static int iommufd_liveupdate_preserve(struct liveupdate_file_op_args *args)
+{
+	struct iommufd_ctx *ictx = iommufd_ctx_from_file(args->file);
+	struct iommufd_hwpt_paging *hwpt;
+	struct iommufd_ser *iommufd_ser;
+	struct iommufd_object *obj;
+	unsigned int nr_hwpts;
+	unsigned long index;
+	unsigned int i;
+	void *mem;
+	int rc;
+
+	if (IS_ERR(ictx))
+		return PTR_ERR(ictx);
+
+	mutex_lock(&ictx->liveupdate_mutex);
+
+	/* Count the number of HWPTs to preserve */
+	nr_hwpts = 0;
+	xa_lock(&ictx->objects);
+	xa_for_each_marked(&ictx->objects, index, obj, IOMMUFD_OBJ_LIVEUPDATE_MARK) {
+		if (obj->type != IOMMUFD_OBJ_HWPT_PAGING)
+			continue;
+
+		hwpt = to_hwpt_paging(container_of(obj, struct iommufd_hw_pagetable, obj));
+		if (!hwpt->common.domain) {
+			rc = -EINVAL;
+			xa_unlock(&ictx->objects);
+			goto out_unlock;
+		}
+		nr_hwpts++;
+	}
+	xa_unlock(&ictx->objects);

drivers/iommu/iommufd/iommufd_private.h

Quoted 10 lines
drivers/iommu/iommufd/iommufd_private.h +10 -0
+
+	mem = kho_alloc_preserve(struct_size(iommufd_ser,
+					     hwpt_array, nr_hwpts));
+	if (!mem) {
+		rc = -ENOMEM;
+		goto out_unlock;
+	}
+
+	iommufd_ser = mem;
+	iommufd_ser->nr_hwpts = nr_hwpts;

Nit: Can there be a TOCTOU here? We first count nr_hwpts in the first loop, but actually preserve them in the loop below. Is it possible for a Guest to race with these loops and destroy a HWPT?

That could cause a bug in the new kernel as it may try to restore nr_hwpts which is one more than the preserved HWPTs.

Quoted 31 lines
drivers/iommu/iommufd/iommufd_private.h +31 -0
+
+	/* Preserve HWPTs */
+	i = 0;
+	xa_lock(&ictx->objects);
+	xa_for_each_marked(&ictx->objects, index, obj, IOMMUFD_OBJ_LIVEUPDATE_MARK) {
+		if (obj->type != IOMMUFD_OBJ_HWPT_PAGING)
+			continue;
+
+		if (!iommufd_lock_obj(obj)) {
+			rc = -ENOENT;
+			xa_unlock(&ictx->objects);
+			goto out_unpreserve;
+		}
+
+		/*
+		 * HWPT is locked so it will not be destroyed. The xarray lock
+		 * can be released here before preserving the HWPT.
+		 */
+		xa_unlock(&ictx->objects);
+		hwpt = to_hwpt_paging(container_of(obj, struct iommufd_hw_pagetable, obj));
+		rc = iommufd_preserve_hwpt(hwpt, &iommufd_ser->hwpt_array[i++], args->session);
+		if (rc) {
+			iommufd_put_object(ictx, obj);
+			goto out_unpreserve;
+		}
+
+		/* Mark as preserved */
+		hwpt->liveupdate_preserved = true;
+		xa_lock(&ictx->objects);
+	}
+	xa_unlock(&ictx->objects);

[...]

Quoted 21 lines
drivers/iommu/iommufd/pages.c +2 -0
index 9bdb2945afe1..3b0c0acb8856 100644
@@ -55,6 +55,7 @@
 #include <linux/overflow.h>
 #include <linux/slab.h>
 #include <linux/sched/mm.h>
+#include <linux/memfd.h>
 #include <linux/vfio_pci_core.h>
 
 #include "double_span.h"
@@ -1421,6 +1422,7 @@ struct iopt_pages *iopt_alloc_file_pages(struct file *file,
 
 {
 	struct iopt_pages *pages;
+	int seals;
 
 	pages = iopt_alloc_pages(start_byte, length, writable);
 	if (IS_ERR(pages))

drivers/iommu/iommufd/pages.c struct iopt_pages *iopt_alloc_file_pages(struct file *file, · line 1437

Quoted 8 lines
drivers/iommu/iommufd/pages.c +5 -0
@@ -1428,6 +1430,11 @@ struct iopt_pages *iopt_alloc_file_pages(struct file *file,
 	pages->file = get_file(file);
 	pages->start = start - start_byte;
 	pages->type = IOPT_ADDRESS_FILE;
+
+	seals = memfd_get_seals(file);
+	if (seals > 0)
+		pages->seals = seals;
+

Can caching memfd seals create a TOCTOU issue?
IIUC, iopt_alloc_file_pages happens at map time, However, the userspace is allowed to map a memfd and then apply the F_ADD_SEALS via fcntl() later in its setup sequence? For example a sequence like:

1. VMM creates a memfd. It has 0 seals.
2. VMM calls IOMMU_IOAS_MAP_FILE. IOMMUFD caches pages->seals = 0.
3. VMM finishes its setup and calls:
fcntl(fd, F_ADD_SEALS, F_SEAL_GROW | F_SEAL_SHRINK | F_SEAL_SEAL).

4.VMM initiates Live Update.
5.check_iopt_pages_preserved looks at the cached pages->seals
(which is still 0), sees the seals are missing, & kills the LiveUpdate with -EINVAL, even though the file is properly sealed..

Thus, I guess we should dynamically check seals via memfd_get_seals()

drivers/iommu/iommufd/pages.c struct iopt_pages *iopt_alloc_file_pages(struct file *file, · line 1440

Quoted 3 lines
drivers/iommu/iommufd/pages.c +0 -0
 	return pages;
 }
 

Thanks,
Praan

#73 lore raw

Re: [PATCH v2 13/16] iommufd: Persist iommu hardware pagetables for live update

Quoted 35 lines · Pranjal Shrivastava
Quoted 17 lines · Samiullah Khawaja

From: YiFei Zhu <[email protected]>

Register iommufd with the LUO framework and implement the preserve and unpreserve ops to save marked HWPTs.

To make sure mappings do not change during preserved state, add a liveupdate_immutable flag to IOAS. When an HWPT is preserved, its IOAS is marked immutable and any map/unmap attempts will fail with -EBUSY. This is synchronized using the domains_rwsem to prevent races with concurrent mapping operations.

The preserve callback iterates over the marked HWPTs, verifies that the backing memory pages are preserved, and calls iommu_domain_preserve() to preserve the associated IOMMU domain.

[...]

Quoted 14 lines
drivers/iommu/iommufd/iommufd_private.h +3 -0
index 111f4d42e210..3c88aa115d08 100644
@@ -98,6 +98,9 @@ struct io_pagetable {
 	/* IOVA that cannot be allocated, struct iopt_reserved */
 	struct rb_root_cached reserved_itree;
 	u8 disable_large_pages;
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	bool liveupdate_immutable;
+#endif
 	unsigned long iova_alignment;
 };
 

[snip]

Quoted 48 lines
Quoted 13 lines
drivers/iommu/iommufd/iommufd_private.h +13 -0
+
+static int iommufd_preserve_hwpt(struct iommufd_hwpt_paging *hwpt,
+				 struct iommufd_hwpt_ser *hwpt_ser,
+				 struct liveupdate_session *session)
+{
+	struct iommu_domain_ser *domain_ser;
+	bool ioas_made_immutable = false;
+	int rc;
+
+	if (!hwpt->ioas->iopt.liveupdate_immutable) {
+		/*
+		 * Make IOAS immutable so the DMA mappings do not change while
+		 * the HWPT is preserved. Since one IOAS can have multiple

drivers/iommu/iommufd/iommufd_private.h

Quoted 10 lines
drivers/iommu/iommufd/iommufd_private.h +10 -0
+		 * HWPTs, if an error occurs this call needs to make the IOAS
+		 * mutable again if it was the one that made it immutable.
+		 */
+		ioas_made_immutable = true;
+		ioas_set_immutable(hwpt->ioas, true);
+
+		rc = check_iopt_pages_preserved(session, hwpt);
+		if (rc)
+			goto err;
+	}

Nit:
I'm thinking what happens for a shared IOAS situation? Say, 2 devices, in the same container, behind 2 different IOMMUs, sharing the IOAS. Each device will have it's own HWPT (N:1 mapping for HWPT v/s IOAS)

So, what happens if Device 1 succeeds with preservation but device 2 doesn't? For example:

1. Preserve Device 1:
- It sees liveupdate_immutable is false.
- Sets ioas_made_immutable = true on its local stack.
- Flips IOAS to immutable.
- Preservation succeeds.

2. Preserve Device 2:
- It sees liveupdate_immutable is already true (because Device 1 set it).
- Sets ioas_made_immutable = false on its local stack.
- The Failure: iommu_domain_preserve fails for Device 2.
- The Jump: Hits goto err;

Now, inside the err: label for device 2, it checks if (ioas_made_immutable), since it is FALSE for device 2, it does nothing.

Quoted 5 lines

I agree we return an error code to the caller which finally cleans it up well, but I'm considering if we should make liveupdate_immutable refcountable? Since the error handling in iommufd_preserve_hwpt() is logically incomplete for shared IOAS as it only attempts to restore mutability if the current HWPT set immutable = true;

This is a valid scenario and as you already pointed that it is handled by the error handling in the caller.

But I agree refcount makes it cleaner. I will update this.

Quoted 70 lines
Quoted 18 lines
drivers/iommu/iommufd/iommufd_private.h +17 -0
+
+	hwpt_ser->token = hwpt->liveupdate_token;
+	hwpt_ser->reclaimed = false;
+
+	rc = iommu_domain_preserve(hwpt->common.domain, &domain_ser);
+	if (rc < 0)
+		goto err;
+
+	hwpt_ser->domain_data = virt_to_phys(domain_ser);
+	return 0;
+
+err:
+	if (ioas_made_immutable)
+		ioas_set_immutable(hwpt->ioas, false);
+
+	return rc;
+}

[...]

Quoted 34 lines
drivers/iommu/iommufd/iommufd_private.h +34 -0
+
+static int iommufd_liveupdate_preserve(struct liveupdate_file_op_args *args)
+{
+	struct iommufd_ctx *ictx = iommufd_ctx_from_file(args->file);
+	struct iommufd_hwpt_paging *hwpt;
+	struct iommufd_ser *iommufd_ser;
+	struct iommufd_object *obj;
+	unsigned int nr_hwpts;
+	unsigned long index;
+	unsigned int i;
+	void *mem;
+	int rc;
+
+	if (IS_ERR(ictx))
+		return PTR_ERR(ictx);
+
+	mutex_lock(&ictx->liveupdate_mutex);
+
+	/* Count the number of HWPTs to preserve */
+	nr_hwpts = 0;
+	xa_lock(&ictx->objects);
+	xa_for_each_marked(&ictx->objects, index, obj, IOMMUFD_OBJ_LIVEUPDATE_MARK) {
+		if (obj->type != IOMMUFD_OBJ_HWPT_PAGING)
+			continue;
+
+		hwpt = to_hwpt_paging(container_of(obj, struct iommufd_hw_pagetable, obj));
+		if (!hwpt->common.domain) {
+			rc = -EINVAL;
+			xa_unlock(&ictx->objects);
+			goto out_unlock;
+		}
+		nr_hwpts++;
+	}
+	xa_unlock(&ictx->objects);

drivers/iommu/iommufd/iommufd_private.h

Quoted 10 lines
drivers/iommu/iommufd/iommufd_private.h +10 -0
+
+	mem = kho_alloc_preserve(struct_size(iommufd_ser,
+					     hwpt_array, nr_hwpts));
+	if (!mem) {
+		rc = -ENOMEM;
+		goto out_unlock;
+	}
+
+	iommufd_ser = mem;
+	iommufd_ser->nr_hwpts = nr_hwpts;

Nit: Can there be a TOCTOU here? We first count nr_hwpts in the first loop, but actually preserve them in the loop below. Is it possible for a Guest to race with these loops and destroy a HWPT?

Quoted 2 lines

That could cause a bug in the new kernel as it may try to restore nr_hwpts which is one more than the preserved HWPTs.

Agreed. The HWPT is locked in the preserve loop below, I will update the nr_hwpts to handle this after that loop to handle this.

Quoted 75 lines
Quoted 32 lines
drivers/iommu/iommufd/iommufd_private.h +31 -0
+
+	/* Preserve HWPTs */
+	i = 0;
+	xa_lock(&ictx->objects);
+	xa_for_each_marked(&ictx->objects, index, obj, IOMMUFD_OBJ_LIVEUPDATE_MARK) {
+		if (obj->type != IOMMUFD_OBJ_HWPT_PAGING)
+			continue;
+
+		if (!iommufd_lock_obj(obj)) {
+			rc = -ENOENT;
+			xa_unlock(&ictx->objects);
+			goto out_unpreserve;
+		}
+
+		/*
+		 * HWPT is locked so it will not be destroyed. The xarray lock
+		 * can be released here before preserving the HWPT.
+		 */
+		xa_unlock(&ictx->objects);
+		hwpt = to_hwpt_paging(container_of(obj, struct iommufd_hw_pagetable, obj));
+		rc = iommufd_preserve_hwpt(hwpt, &iommufd_ser->hwpt_array[i++], args->session);
+		if (rc) {
+			iommufd_put_object(ictx, obj);
+			goto out_unpreserve;
+		}
+
+		/* Mark as preserved */
+		hwpt->liveupdate_preserved = true;
+		xa_lock(&ictx->objects);
+	}
+	xa_unlock(&ictx->objects);

[...]

Quoted 21 lines
drivers/iommu/iommufd/pages.c +2 -0
index 9bdb2945afe1..3b0c0acb8856 100644
@@ -55,6 +55,7 @@
 #include <linux/overflow.h>
 #include <linux/slab.h>
 #include <linux/sched/mm.h>
+#include <linux/memfd.h>
 #include <linux/vfio_pci_core.h>
 
 #include "double_span.h"
@@ -1421,6 +1422,7 @@ struct iopt_pages *iopt_alloc_file_pages(struct file *file,
 
 {
 	struct iopt_pages *pages;
+	int seals;
 
 	pages = iopt_alloc_pages(start_byte, length, writable);
 	if (IS_ERR(pages))

drivers/iommu/iommufd/pages.c struct iopt_pages *iopt_alloc_file_pages(struct file *file, · line 1437

Quoted 8 lines
drivers/iommu/iommufd/pages.c +5 -0
@@ -1428,6 +1430,11 @@ struct iopt_pages *iopt_alloc_file_pages(struct file *file,
 	pages->file = get_file(file);
 	pages->start = start - start_byte;
 	pages->type = IOPT_ADDRESS_FILE;
+
+	seals = memfd_get_seals(file);
+	if (seals > 0)
+		pages->seals = seals;
+

Can caching memfd seals create a TOCTOU issue?
IIUC, iopt_alloc_file_pages happens at map time, However, the userspace is allowed to map a memfd and then apply the F_ADD_SEALS via fcntl() later in its setup sequence? For example a sequence like:

1. VMM creates a memfd. It has 0 seals.
2. VMM calls IOMMU_IOAS_MAP_FILE. IOMMUFD caches pages->seals = 0.
3. VMM finishes its setup and calls:
fcntl(fd, F_ADD_SEALS, F_SEAL_GROW | F_SEAL_SHRINK | F_SEAL_SEAL).

Quoted 4 lines

4.VMM initiates Live Update.
5.check_iopt_pages_preserved looks at the cached pages->seals
(which is still 0), sees the seals are missing, & kills the LiveUpdate with -EINVAL, even though the file is properly sealed..

This is true and it is intentionally this way to make sure that the seal is applied during mapping otherwise user can apply the seal after resizing the memfd and preserve IOMMU mappings that are pointing to unpreserved pages. Consider following:

1. VMM creates a memfd and seals is zero.
2. VMM maps memfd into ioas/hwpt.
3. VMM resizes the memfd.
4. VMM seals memfd
5. VMM preserves the memfd (it only preseves the current size).
6. VMM preserves iommufd and it succeeds as memfd is sealed.

But the pages being referred by the iommu mappings are refcounted in current kernel, but not preserved.

Check the comment in check_iopt_pages_preserved() also. I will add a comment here also.

Quoted 9 lines

Thus, I guess we should dynamically check seals via memfd_get_seals()

drivers/iommu/iommufd/pages.c struct iopt_pages *iopt_alloc_file_pages(struct file *file, · line 1440

Quoted 3 lines
drivers/iommu/iommufd/pages.c +0 -0
 	return pages;
 }
 

Thanks,
Praan

Sami

#74 lore raw

Re: [PATCH v2 13/16] iommufd: Persist iommu hardware pagetables for live update

Quoted 2 lines · Samiullah Khawaja
Quoted 1 line · Pranjal Shrivastava

On Mon, Apr 27, 2026 at 05:56:30PM +0000, Samiullah Khawaja wrote:

[...]

Quoted 46 lines
Quoted 29 lines
Quoted 10 lines

#include "double_span.h"

drivers/iommu/iommufd/pages.c +1 -0
@@ -1421,6 +1422,7 @@ struct iopt_pages *iopt_alloc_file_pages(struct file *file,
 
 {
 	struct iopt_pages *pages;
+	int seals;
 
 	pages = iopt_alloc_pages(start_byte, length, writable);
 	if (IS_ERR(pages))

drivers/iommu/iommufd/pages.c struct iopt_pages *iopt_alloc_file_pages(struct file *file, · line 1437

Quoted 8 lines
drivers/iommu/iommufd/pages.c +5 -0
@@ -1428,6 +1430,11 @@ struct iopt_pages *iopt_alloc_file_pages(struct file *file,
 	pages->file = get_file(file);
 	pages->start = start - start_byte;
 	pages->type = IOPT_ADDRESS_FILE;
+
+	seals = memfd_get_seals(file);
+	if (seals > 0)
+		pages->seals = seals;
+

Can caching memfd seals create a TOCTOU issue?
IIUC, iopt_alloc_file_pages happens at map time, However, the userspace is allowed to map a memfd and then apply the F_ADD_SEALS via fcntl() later in its setup sequence? For example a sequence like:

1. VMM creates a memfd. It has 0 seals.
2. VMM calls IOMMU_IOAS_MAP_FILE. IOMMUFD caches pages->seals = 0.
3. VMM finishes its setup and calls:
fcntl(fd, F_ADD_SEALS, F_SEAL_GROW | F_SEAL_SHRINK | F_SEAL_SEAL).

Quoted 4 lines

4.VMM initiates Live Update.
5.check_iopt_pages_preserved looks at the cached pages->seals
(which is still 0), sees the seals are missing, & kills the LiveUpdate with -EINVAL, even though the file is properly sealed..

This is true and it is intentionally this way to make sure that the seal is applied during mapping otherwise user can apply the seal after resizing the memfd and preserve IOMMU mappings that are pointing to unpreserved pages. Consider following:

1. VMM creates a memfd and seals is zero.
2. VMM maps memfd into ioas/hwpt.
3. VMM resizes the memfd.
4. VMM seals memfd
5. VMM preserves the memfd (it only preseves the current size).
6. VMM preserves iommufd and it succeeds as memfd is sealed.

Quoted 6 lines

But the pages being referred by the iommu mappings are refcounted in current kernel, but not preserved.

Check the comment in check_iopt_pages_preserved() also. I will add a comment here also.

Quoted 1 line

I understand the intent to enforce a policy to Seal-at-Map to ensure consistency. I am wondering if this policy is a little too restrictive. Should we consider performing a dynamic i_size check during preservation instead? I can't think of a good use-case as of now.. (maybe let it be?)

However, if we decide to keep the current policy, we should probably check for the required seals during the IOMMU_IOAS_MAP_FILE ioctl itself. If the seals aren't present, we could pr_warn("Don't expect liveupdate preservation for this memory, bad seals") to let the user know that this specific mapping will be ineligible for Live Update preservation later.

Thanks,
Praan

#75 lore raw

Re: [PATCH v2 13/16] iommufd: Persist iommu hardware pagetables for live update

Quoted 54 lines · Pranjal Shrivastava
Quoted 2 lines · Samiullah Khawaja
Quoted 1 line · Pranjal Shrivastava

On Mon, Apr 27, 2026 at 05:56:30PM +0000, Samiullah Khawaja wrote:

[...]

Quoted 49 lines
Quoted 29 lines
Quoted 10 lines

#include "double_span.h"

drivers/iommu/iommufd/pages.c +1 -0
@@ -1421,6 +1422,7 @@ struct iopt_pages *iopt_alloc_file_pages(struct file *file,
 
 {
 	struct iopt_pages *pages;
+	int seals;
 
 	pages = iopt_alloc_pages(start_byte, length, writable);
 	if (IS_ERR(pages))

drivers/iommu/iommufd/pages.c struct iopt_pages *iopt_alloc_file_pages(struct file *file, · line 1437

Quoted 8 lines
drivers/iommu/iommufd/pages.c +5 -0
@@ -1428,6 +1430,11 @@ struct iopt_pages *iopt_alloc_file_pages(struct file *file,
 	pages->file = get_file(file);
 	pages->start = start - start_byte;
 	pages->type = IOPT_ADDRESS_FILE;
+
+	seals = memfd_get_seals(file);
+	if (seals > 0)
+		pages->seals = seals;
+

Can caching memfd seals create a TOCTOU issue?
IIUC, iopt_alloc_file_pages happens at map time, However, the userspace is allowed to map a memfd and then apply the F_ADD_SEALS via fcntl() later in its setup sequence? For example a sequence like:

1. VMM creates a memfd. It has 0 seals.
2. VMM calls IOMMU_IOAS_MAP_FILE. IOMMUFD caches pages->seals = 0.
3. VMM finishes its setup and calls:
fcntl(fd, F_ADD_SEALS, F_SEAL_GROW | F_SEAL_SHRINK | F_SEAL_SEAL).

Quoted 4 lines

4.VMM initiates Live Update.
5.check_iopt_pages_preserved looks at the cached pages->seals
(which is still 0), sees the seals are missing, & kills the LiveUpdate with -EINVAL, even though the file is properly sealed..

This is true and it is intentionally this way to make sure that the seal is applied during mapping otherwise user can apply the seal after resizing the memfd and preserve IOMMU mappings that are pointing to unpreserved pages. Consider following:

1. VMM creates a memfd and seals is zero.
2. VMM maps memfd into ioas/hwpt.
3. VMM resizes the memfd.
4. VMM seals memfd
5. VMM preserves the memfd (it only preseves the current size).
6. VMM preserves iommufd and it succeeds as memfd is sealed.

But the pages being referred by the iommu mappings are refcounted in current kernel, but not preserved.

Quoted 8 lines
Quoted 3 lines

Check the comment in check_iopt_pages_preserved() also. I will add a comment here also.

Quoted 1 line

I understand the intent to enforce a policy to Seal-at-Map to ensure consistency. I am wondering if this policy is a little too restrictive. Should we consider performing a dynamic i_size check during preservation instead? I can't think of a good use-case as of now.. (maybe let it be?)

On another thought, I guess we should go ahead with the current policy, to ensure we preserve the right pages. If an unsealed file is shrunk and then grown back to its original size before preservation we might have a problem. We can ignore this comment.

Thanks,
Praan

#76 lore raw

PATCH v2 14/16 iommufd: Add APIs to preserve/unpreserve a vfio cdev

In reply to Samiullah Khawaja

Add APIs that can be used to preserve and unpreserve a vfio cdev. Use the APIs exported by the IOMMU core to preserve/unpreserve device.

The LUO token of the preserved iommufd is fetched and returned back to the caller as that can be used during restore to get the restored iommufd. Handle to the preserved state of the device is also returned to reassociate with the restored state after live update kexec.

---
 drivers/iommu/iommufd/device.c          | 102 ++++++++++++++++++++++++
 drivers/iommu/iommufd/iommufd_private.h |   3 +
 include/linux/iommufd.h                 |  29 +++++++
 3 files changed, 134 insertions(+)
drivers/iommu/iommufd/device.c +102 -0
index 170a7005f0bc..d19fece00da3 100644
@@ -2,6 +2,7 @@
 /* Copyright (c) 2021-2022, NVIDIA CORPORATION & AFFILIATES
  */
 #include <linux/iommu.h>
+#include <linux/iommu-liveupdate.h>
 #include <linux/iommufd.h>
 #include <linux/pci-ats.h>
 #include <linux/slab.h>
@@ -610,6 +611,10 @@ int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt,
 	int rc;
 
 	mutex_lock(&igroup->lock);
+	if (iommufd_device_is_preserved(idev)) {
+		rc = -EBUSY;
+		goto err_unlock;
+	}
 
 	attach = xa_cmpxchg(&igroup->pasid_attach, pasid, NULL,
 			    XA_ZERO_ENTRY, GFP_KERNEL);
@@ -1665,3 +1670,100 @@ int iommufd_get_hw_info(struct iommufd_ucmd *ucmd)
 	iommufd_put_object(ucmd->ictx, &idev->obj);
 	return rc;
 }
+
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+static bool _iommufd_device_has_pasid_attachments(struct iommufd_device *idev)
+{
+	struct iommufd_group *igroup = idev->igroup;
+	unsigned long start = IOMMU_NO_PASID;
+
+	if (xa_find_after(&igroup->pasid_attach,
+			  &start, UINT_MAX, XA_PRESENT))
+		return true;
+
+	return false;
+}
+
+int iommufd_device_preserve(struct liveupdate_session *s,
+			    struct iommufd_device *idev,
+			    u64 *iommufd_tokenp,
+			    u64 *preserved_state)
+{
+	struct iommufd_group *igroup = idev->igroup;
+	struct iommufd_hwpt_paging *hwpt_paging;
+	struct iommufd_hw_pagetable *hwpt;
+	struct iommufd_attach *attach;
+	int ret;
+
+	mutex_lock(&igroup->lock);
+	if (_iommufd_device_has_pasid_attachments(idev)) {
+		ret = -EOPNOTSUPP;
+		goto out;
+	}
+
+	attach = xa_load(&igroup->pasid_attach, IOMMU_NO_PASID);
+	if (!attach) {
+		ret = -ENOENT;
+		goto out;
+	}
+
+	hwpt = attach->hwpt;
+	hwpt_paging = find_hwpt_paging(hwpt);
+	if (!hwpt_paging || !hwpt_paging->liveupdate_preserved) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ret = liveupdate_get_token_outgoing(s, idev->ictx->file, iommufd_tokenp);
+	if (ret)
+		goto out;
+
+	ret = iommu_preserve_device(hwpt_paging->common.domain,
+				    idev->dev,
+				    preserved_state);
+
+	if (!ret)
+		igroup->liveupdate_preserved = true;
+out:
+	mutex_unlock(&igroup->lock);
+	return ret;
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_device_preserve, "IOMMUFD");
+
+void iommufd_device_unpreserve(struct liveupdate_session *s,
+			       struct iommufd_device *idev)
+{
+	struct iommufd_group *igroup = idev->igroup;
+	struct iommufd_hwpt_paging *hwpt_paging;
+	struct iommufd_hw_pagetable *hwpt;
+	struct iommufd_attach *attach;
+
+	mutex_lock(&igroup->lock);
+	attach = xa_load(&igroup->pasid_attach, IOMMU_NO_PASID);
+	if (!attach) {
+		WARN(1, "IOMMU_NO_PASID attachment not found");
+		igroup->liveupdate_preserved = false;
+		goto out;
+	}
+
+	hwpt = attach->hwpt;
+	hwpt_paging = find_hwpt_paging(hwpt);
+	if (!hwpt_paging || !hwpt_paging->liveupdate_preserved) {
+		WARN(1, "Attached domain is not preserved");
+		igroup->liveupdate_preserved = false;
+		goto out;
+	}
+
+	iommu_unpreserve_device(hwpt_paging->common.domain, idev->dev);
+	igroup->liveupdate_preserved = false;
+out:
+	mutex_unlock(&igroup->lock);
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_device_unpreserve, "IOMMUFD");
+
+bool iommufd_device_is_preserved(struct iommufd_device *idev)
+{
+	return idev && idev->igroup && idev->igroup->liveupdate_preserved;
+}
+EXPORT_SYMBOL_NS_GPL(iommufd_device_is_preserved, "IOMMUFD");
+#endif
drivers/iommu/iommufd/iommufd_private.h +3 -0
index 3c88aa115d08..9b47eaf92d42 100644
@@ -486,6 +486,9 @@ struct iommufd_group {
 	struct xarray pasid_attach;
 	struct iommufd_sw_msi_maps required_sw_msi;
 	phys_addr_t sw_msi_start;
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	bool liveupdate_preserved;
+#endif
 };
 
 /*
include/linux/iommufd.h +29 -0
index 6e7efe83bc5d..d1fd5d71e0fd 100644
@@ -9,6 +9,7 @@
 #include <linux/err.h>
 #include <linux/errno.h>
 #include <linux/iommu.h>
+#include <linux/liveupdate.h>
 #include <linux/refcount.h>
 #include <linux/types.h>
 #include <linux/xarray.h>
@@ -71,6 +72,34 @@ void iommufd_device_detach(struct iommufd_device *idev, ioasid_t pasid);
 struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev);
 u32 iommufd_device_to_id(struct iommufd_device *idev);
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+int iommufd_device_preserve(struct liveupdate_session *s,
+			    struct iommufd_device *idev,
+			    u64 *iommufd_tokenp,
+			    u64 *preserved_state);
+void iommufd_device_unpreserve(struct liveupdate_session *s,
+			       struct iommufd_device *idev);
+bool iommufd_device_is_preserved(struct iommufd_device *idev);
+#else
+static inline int iommufd_device_preserve(struct liveupdate_session *s,
+					  struct iommufd_device *idev,
+					  u64 *iommufd_tokenp,
+					  u64 *preserved_state)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void iommufd_device_unpreserve(struct liveupdate_session *s,
+					     struct iommufd_device *idev)
+{
+}
+
+static inline bool iommufd_device_is_preserved(struct iommufd_device *idev)
+{
+	return false;
+}
+#endif
+
 struct iommufd_access_ops {
 	u8 needs_pin_pages : 1;
 	void (*unmap)(void *data, unsigned long iova, unsigned long length);
Signature · 1 line
2.54.0.545.g6539524ca2-goog
Reviewed-by on v2 14/16
#77 lore raw

Re: [PATCH v2 14/16] iommufd: Add APIs to preserve/unpreserve a vfio cdev

Quoted 8 lines · Samiullah Khawaja

Add APIs that can be used to preserve and unpreserve a vfio cdev. Use the APIs exported by the IOMMU core to preserve/unpreserve device.

The LUO token of the preserved iommufd is fetched and returned back to the caller as that can be used during restore to get the restored iommufd. Handle to the preserved state of the device is also returned to reassociate with the restored state after live update kexec.

Quoted 7 lines
---
 drivers/iommu/iommufd/device.c          | 102 ++++++++++++++++++++++++
 drivers/iommu/iommufd/iommufd_private.h |   3 +
 include/linux/iommufd.h                 |  29 +++++++
 3 files changed, 134 insertions(+)

The iommufd_device_replace() is ONLY reachable by VFIO which is blocked by PATCH 15. LGTM.

Thanks,
Praan

#78 lore raw

PATCH v2 15/16 vfio/pci: Preserve the iommufd state of the vfio cdev

In reply to Samiullah Khawaja

If the vfio cdev is attached to an iommufd, preserve the state of the attached iommufd also. Basically preserve the iommu specific state of the device and also the attach iommu HW unit.

Once the device and its iommufd attachment is preserved, it cannot be detached or attached to another IOAS until it is unpreserved.

---
 drivers/vfio/device_cdev.c             | 10 ++++++++
 drivers/vfio/pci/vfio_pci_liveupdate.c | 33 +++++++++++++++++++++++++-
 2 files changed, 42 insertions(+), 1 deletion(-)
drivers/vfio/device_cdev.c +10 -0
index 6844684a3d8e..23f083c0891a 100644
@@ -271,6 +271,11 @@ int vfio_df_ioctl_attach_pt(struct vfio_device_file *df,
 	}
 
 	mutex_lock(&device->dev_set->lock);
+	if (iommufd_device_is_preserved(device->iommufd_device)) {
+		ret = -EBUSY;
+		goto out_unlock;
+	}
+
 	if (attach.flags & VFIO_DEVICE_ATTACH_PASID)
 		ret = device->ops->pasid_attach_ioas(device,
 						     attach.pasid,
@@ -329,6 +334,11 @@ int vfio_df_ioctl_detach_pt(struct vfio_device_file *df,
 	}
 
 	mutex_lock(&device->dev_set->lock);
+	if (iommufd_device_is_preserved(device->iommufd_device)) {
+		mutex_unlock(&device->dev_set->lock);
+		return -EBUSY;
+	}
+
 	if (detach.flags & VFIO_DEVICE_DETACH_PASID)
 		device->ops->pasid_detach_ioas(device, detach.pasid);
 	else
drivers/vfio/pci/vfio_pci_liveupdate.c +32 -1
index 976ef17e6103..b56d80379ffc 100644
@@ -108,10 +108,13 @@
 #include <linux/kho/abi/vfio_pci.h>
 #include <linux/liveupdate.h>
 #include <linux/errno.h>
+#include <linux/iommufd.h>
 #include <linux/vfio.h>
 
 #include "vfio_pci_priv.h"
 
+MODULE_IMPORT_NS("IOMMUFD");
+
 static bool vfio_pci_liveupdate_can_preserve(struct liveupdate_file_handler *handler,
 					     struct file *file)
 {
@@ -153,9 +156,26 @@ static int vfio_pci_liveupdate_preserve(struct liveupdate_file_op_args *args)
 	vdev = container_of(device, struct vfio_pci_core_device, vdev);
 	pdev = vdev->pdev;
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	/* If iommufd is attached, preserve the underlying domain */
+	mutex_lock(&device->dev_set->lock);
+	if (device->iommufd_attached) {
+		u64 token, preserved_state;
+
+		ret = iommufd_device_preserve(args->session,
+					      device->iommufd_device,
+					      &token, &preserved_state);
+		if (ret) {
+			mutex_unlock(&device->dev_set->lock);
+			return ret;
+		}
+	}
+	mutex_unlock(&device->dev_set->lock);
+#endif
+
 	ret = pci_liveupdate_preserve(pdev);
 	if (ret)
-		return ret;
+		goto err_iommufd_unpreserve;
 
 	ser = kho_alloc_preserve(sizeof(*ser));
 	if (IS_ERR(ser)) {
@@ -170,6 +190,9 @@ static int vfio_pci_liveupdate_preserve(struct liveupdate_file_op_args *args)
 	args->serialized_data = virt_to_phys(ser);
 	return 0;
 
+err_iommufd_unpreserve:
+	iommufd_device_unpreserve(args->session, device->iommufd_device);
+
 err_unpreserve:
 	pci_liveupdate_unpreserve(pdev);
 	return ret;
@@ -178,6 +201,14 @@ static int vfio_pci_liveupdate_preserve(struct liveupdate_file_op_args *args)
 static void vfio_pci_liveupdate_unpreserve(struct liveupdate_file_op_args *args)
 {
 	struct vfio_device *device = vfio_device_from_file(args->file);
+	struct vfio_pci_core_device_ser *ser;
+
+	ser = phys_to_virt(args->serialized_data);
+	mutex_lock(&device->dev_set->lock);
+	if (device->iommufd_attached)
+		iommufd_device_unpreserve(args->session,
+					  device->iommufd_device);
+	mutex_unlock(&device->dev_set->lock);
 
 	pci_liveupdate_unpreserve(to_pci_dev(device->dev));
 	kho_unpreserve_free(phys_to_virt(args->serialized_data));
Signature · 1 line
2.54.0.545.g6539524ca2-goog
Reviewed-by on v2 15/16
#79 lore raw

Re: [PATCH v2 15/16] vfio/pci: Preserve the iommufd state of the vfio cdev

Quoted 13 lines · Samiullah Khawaja

If the vfio cdev is attached to an iommufd, preserve the state of the attached iommufd also. Basically preserve the iommu specific state of the device and also the attach iommu HW unit.

Once the device and its iommufd attachment is preserved, it cannot be detached or attached to another IOAS until it is unpreserved.

---
 drivers/vfio/device_cdev.c             | 10 ++++++++
 drivers/vfio/pci/vfio_pci_liveupdate.c | 33 +++++++++++++++++++++++++-
 2 files changed, 42 insertions(+), 1 deletion(-)

[...]

Quoted 33 lines
drivers/vfio/pci/vfio_pci_liveupdate.c +14 -0
index 976ef17e6103..b56d80379ffc 100644
@@ -108,10 +108,13 @@
 #include <linux/kho/abi/vfio_pci.h>
 #include <linux/liveupdate.h>
 #include <linux/errno.h>
+#include <linux/iommufd.h>
 #include <linux/vfio.h>
 
 #include "vfio_pci_priv.h"
 
+MODULE_IMPORT_NS("IOMMUFD");
+
 static bool vfio_pci_liveupdate_can_preserve(struct liveupdate_file_handler *handler,
 					     struct file *file)
 {
@@ -153,9 +156,26 @@ static int vfio_pci_liveupdate_preserve(struct liveupdate_file_op_args *args)
 	vdev = container_of(device, struct vfio_pci_core_device, vdev);
 	pdev = vdev->pdev;
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	/* If iommufd is attached, preserve the underlying domain */
+	mutex_lock(&device->dev_set->lock);
+	if (device->iommufd_attached) {
+		u64 token, preserved_state;
+
+		ret = iommufd_device_preserve(args->session,
+					      device->iommufd_device,
+					      &token, &preserved_state);
+		if (ret) {
+			mutex_unlock(&device->dev_set->lock);

drivers/vfio/pci/vfio_pci_liveupdate.c static int vfio_pci_liveupdate_preserve(struct liveupdate_file_op_args *args) · line 178

Quoted 10 lines
drivers/vfio/pci/vfio_pci_liveupdate.c +7 -1
+			return ret;
+		}
+	}
+	mutex_unlock(&device->dev_set->lock);
+#endif
+
 	ret = pci_liveupdate_preserve(pdev);
 	if (ret)
-		return ret;
+		goto err_iommufd_unpreserve;

Nit: Would we hit the WARN in patch 14 if (device->iommufd_attached) was false here? Should we check if (device->iommufd_attached) below before calling unpreserve?

drivers/vfio/pci/vfio_pci_liveupdate.c static int vfio_pci_liveupdate_preserve(struct liveupdate_file_op_args *args) · line 198

Quoted 13 lines
drivers/vfio/pci/vfio_pci_liveupdate.c +3 -0
 	ser = kho_alloc_preserve(sizeof(*ser));
 	if (IS_ERR(ser)) {
@@ -170,6 +190,9 @@ static int vfio_pci_liveupdate_preserve(struct liveupdate_file_op_args *args)
 	args->serialized_data = virt_to_phys(ser);
 	return 0;
 
+err_iommufd_unpreserve:
+	iommufd_device_unpreserve(args->session, device->iommufd_device);
+
 err_unpreserve:
 	pci_liveupdate_unpreserve(pdev);
 	return ret;

With that nit:

Thanks,
Praan

#80 lore raw

Re: [PATCH v2 15/16] vfio/pci: Preserve the iommufd state of the vfio cdev

Quoted 56 lines · Pranjal Shrivastava
Quoted 13 lines · Samiullah Khawaja

If the vfio cdev is attached to an iommufd, preserve the state of the attached iommufd also. Basically preserve the iommu specific state of the device and also the attach iommu HW unit.

Once the device and its iommufd attachment is preserved, it cannot be detached or attached to another IOAS until it is unpreserved.

---
 drivers/vfio/device_cdev.c             | 10 ++++++++
 drivers/vfio/pci/vfio_pci_liveupdate.c | 33 +++++++++++++++++++++++++-
 2 files changed, 42 insertions(+), 1 deletion(-)

[...]

Quoted 39 lines
drivers/vfio/pci/vfio_pci_liveupdate.c +20 -0
index 976ef17e6103..b56d80379ffc 100644
@@ -108,10 +108,13 @@
 #include <linux/kho/abi/vfio_pci.h>
 #include <linux/liveupdate.h>
 #include <linux/errno.h>
+#include <linux/iommufd.h>
 #include <linux/vfio.h>
 
 #include "vfio_pci_priv.h"
 
+MODULE_IMPORT_NS("IOMMUFD");
+
 static bool vfio_pci_liveupdate_can_preserve(struct liveupdate_file_handler *handler,
 					     struct file *file)
 {
@@ -153,9 +156,26 @@ static int vfio_pci_liveupdate_preserve(struct liveupdate_file_op_args *args)
 	vdev = container_of(device, struct vfio_pci_core_device, vdev);
 	pdev = vdev->pdev;
 
+#ifdef CONFIG_IOMMU_LIVEUPDATE
+	/* If iommufd is attached, preserve the underlying domain */
+	mutex_lock(&device->dev_set->lock);
+	if (device->iommufd_attached) {
+		u64 token, preserved_state;
+
+		ret = iommufd_device_preserve(args->session,
+					      device->iommufd_device,
+					      &token, &preserved_state);
+		if (ret) {
+			mutex_unlock(&device->dev_set->lock);
+			return ret;
+		}
+	}
+	mutex_unlock(&device->dev_set->lock);
+#endif
+
Quoted 8 lines

drivers/vfio/pci/vfio_pci_liveupdate.c static int vfio_pci_liveupdate_preserve(struct liveupdate_file_op_args *args) · line 178

Quoted 4 lines
drivers/vfio/pci/vfio_pci_liveupdate.c +1 -1
 	ret = pci_liveupdate_preserve(pdev);
 	if (ret)
-		return ret;
+		goto err_iommufd_unpreserve;

Nit: Would we hit the WARN in patch 14 if (device->iommufd_attached) was false here? Should we check if (device->iommufd_attached) below before calling unpreserve?

Agreed. I will update this.

Quoted 17 lines
Quoted 5 lines
drivers/vfio/pci/vfio_pci_liveupdate.c +0 -0
 	ser = kho_alloc_preserve(sizeof(*ser));
 	if (IS_ERR(ser)) {

drivers/vfio/pci/vfio_pci_liveupdate.c static int vfio_pci_liveupdate_preserve(struct liveupdate_file_op_args *args) · line 198

Quoted 9 lines
drivers/vfio/pci/vfio_pci_liveupdate.c +3 -0
@@ -170,6 +190,9 @@ static int vfio_pci_liveupdate_preserve(struct liveupdate_file_op_args *args)
 	args->serialized_data = virt_to_phys(ser);
 	return 0;
 
+err_iommufd_unpreserve:
+	iommufd_device_unpreserve(args->session, device->iommufd_device);
+
 err_unpreserve:
 	pci_liveupdate_unpreserve(pdev);
 	return ret;

With that nit:

Quoted 4 lines

Thanks,
Praan

Thanks,
Sami

#81 lore raw

PATCH v2 16/16 iommufd/selftest: Add test to verify iommufd preservation

In reply to Samiullah Khawaja

Test iommufd preservation by setting up an iommufd and vfio cdev and preserve it across live update. Test takes VFIO cdev path of a device bound to vfio-pci driver and binds it to an iommufd being preserved. It also preserves the vfio cdev so the iommufd state associated with it is also preserved.

The restore path is tested by restoring the preserved vfio cdev only. On restore, test verifies that the bind with a new iommufd fails as the device is attached to the restored IOMMU domain. Also the LUO session finish fails as the preserved iommufd is not restored.

---
 tools/testing/selftests/iommu/Makefile        |  12 +
 .../iommu/iommufd_liveupdate_kexec_test.c     | 239 ++++++++++++++++++
 2 files changed, 251 insertions(+)
 create mode 100644 tools/testing/selftests/iommu/iommufd_liveupdate_kexec_test.c
tools/testing/selftests/iommu/Makefile +12 -0
index 84abeb2f0949..ab35e8b21580 100644
@@ -7,4 +7,16 @@ TEST_GEN_PROGS :=
 TEST_GEN_PROGS += iommufd
 TEST_GEN_PROGS += iommufd_fail_nth
 
+TEST_GEN_PROGS_EXTENDED += iommufd_liveupdate_kexec_test
+
 include ../lib.mk
+include ../liveupdate/lib/libliveupdate.mk
+
+CFLAGS += -I$(top_srcdir)/tools/include
+CFLAGS += -MD
+CFLAGS += $(EXTRA_CFLAGS)
+
+$(TEST_GEN_PROGS_EXTENDED): %: %.o $(LIBLIVEUPDATE_O)
+	$(CC) $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH) $< $(LIBLIVEUPDATE_O) $(LDLIBS) -o $@
+
+EXTRA_CLEAN += $(LIBLIVEUPDATE_O)
tools/testing/selftests/iommu/iommufd_liveupdate_kexec_test.c new file +239 -0

239 lines not on this page · raw message

Signature · 1 line
2.54.0.545.g6539524ca2-goog