Skip to contentSkip to conversation

iommu/vt-d: Fixes for issues reported by Sashiko

PATCH In review Lu Baolu 2 participants 9 messages Jul 31 – Aug 4

Subsystems (3)
lore
Share & tools
lore

Lu Baolu

author maintainer
#0 lore raw

cover iommu/vt-d: Fixes for issues reported by Sashiko

Hi,

This collects a number of pre-existing issues that were reported by Sashiko while reviewing recent VT-d patches. None of them are regressions from the patches under review; they are latent problems in code those patches happened to touch. They are independent of each other and can be applied in any order.

Thanks,
baolu

Lu Baolu (5):
iommu/vt-d: Fix shift overflow in qi_desc_dev_iotlb_pasid() iommu/vt-d: Clear Present bit before tearing down copied context entry iommu/vt-d: Fix iopf_refcount leak on RID domain replacement iommu/vt-d: Tear down scalable-mode context on probe failure iommu/vt-d: Flush context cache with correct SID when tearing down
aliases

 drivers/iommu/intel/iommu.h | 18 +++++++++++++-----
 drivers/iommu/intel/iommu.c | 22 +++++++++++++++-------
 drivers/iommu/intel/pasid.c |  9 ++++++---
 3 files changed, 34 insertions(+), 15 deletions(-)
Signature · 1 line
2.43.0
author maintainer
#1 lore raw

PATCH 1/5 iommu/vt-d: Fix shift overflow in qi_desc_dev_iotlb_pasid()

Callers request a full Device-TLB flush by passing MAX_AGAW_PFN_WIDTH (64 - VTD_PAGE_SHIFT == 52) as @size_order. Two shifts in qi_desc_dev_iotlb_pasid() are not prepared for a value that large:

  unsigned long mask = 1UL << (VTD_PAGE_SHIFT + size_order - 1);
  ...
  if (!IS_ALIGNED(addr, VTD_PAGE_SIZE << size_order))

The first evaluates to 1UL << 63. On 32-bit builds this is undefined behaviour; in practice x86 masks the shift count to 5 bits, so the expression yields 1UL << 31 and ~mask becomes 0x7fffffff. That value is zero-extended when it is applied to the 64-bit descriptor, so

  desc->qw1 &= ~mask;

clears qw1[63:32] as well as bit 31. The ADDR field, which had just been filled with ones to request the widest possible range, collapses to 0x7ffff000. As the S bit remains set, hardware decodes the least significant zero bit of ADDR and invalidates only 2GiB instead of the entire address space. Device-TLB entries above that boundary survive the unmap, leaving an ATS-capable device able to keep accessing memory that has already been freed.

The second shift, VTD_PAGE_SIZE << size_order, is 1UL << 64 and is therefore undefined on 64-bit builds too. On x86_64 the shift count masks to zero, IS_ALIGNED(addr, 1) is trivially true and the sanity check silently degrades into a no-op.

Compute both quantities in 64-bit and clamp @size_order to the largest range the ADDR field can encode. Capping at 63 - VTD_PAGE_SHIFT keeps the intended "flush everything" behaviour: qw1[62:12] is set, bit 62 is cleared as the size indicator and the S bit is set. The non-PASID variant qi_desc_dev_iotlb() already uses 1ULL and is unaffected.

Fixes f701c9f36bcb7 ("iommu/vt-d: Factor out invalidation descriptor composition")
Assisted-by Claude:claude-opus-5
---
 drivers/iommu/intel/iommu.h | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)
drivers/iommu/intel/iommu.h +12 -4
index c00f44db0020..8a59c7c9d0a6 100644
@@ -1105,12 +1105,20 @@ static inline void qi_desc_dev_iotlb_pasid(u16 sid, u16 pfsid, u32 pasid,
 					   unsigned int size_order,
 					   struct qi_desc *desc)
 {
-	unsigned long mask = 1UL << (VTD_PAGE_SHIFT + size_order - 1);
-
 	desc->qw0 = QI_DEV_EIOTLB_PASID(pasid) | QI_DEV_EIOTLB_SID(sid) |
 		QI_DEV_EIOTLB_QDEP(qdep) | QI_DEIOTLB_TYPE |
 		QI_DEV_IOTLB_PFSID(pfsid);
 
+	/*
+	 * The invalidation range is encoded in the ADDR field, which only
+	 * covers bits 63:12.  Clamp @size_order so that callers asking for a
+	 * full flush (e.g. with MAX_AGAW_PFN_WIDTH) do not overflow the
+	 * shifts below.  The clamped value still spans the whole range that
+	 * the descriptor is able to express.
+	 */
+	if (size_order > 63 - VTD_PAGE_SHIFT)
+		size_order = 63 - VTD_PAGE_SHIFT;
+
 	/*
 	 * If S bit is 0, we only flush a single page. If S bit is set,
 	 * The least significant zero bit indicates the invalidation address
@@ -1120,7 +1128,7 @@ static inline void qi_desc_dev_iotlb_pasid(u16 sid, u16 pfsid, u32 pasid,
 	 * Max Invs Pending (MIP) is set to 0 for now until we have DIT in
 	 * ECAP.
 	 */
-	if (!IS_ALIGNED(addr, VTD_PAGE_SIZE << size_order))
+	if (!IS_ALIGNED(addr, BIT_ULL(VTD_PAGE_SHIFT + size_order)))
 		pr_warn_ratelimited("Invalidate non-aligned address %llx, order %d\n",
 				    addr, size_order);
 
@@ -1136,7 +1144,7 @@ static inline void qi_desc_dev_iotlb_pasid(u16 sid, u16 pfsid, u32 pasid,
 		desc->qw1 |= GENMASK_ULL(size_order + VTD_PAGE_SHIFT - 1,
 					VTD_PAGE_SHIFT);
 		/* Clear size_order bit to indicate size */
-		desc->qw1 &= ~mask;
+		desc->qw1 &= ~BIT_ULL(VTD_PAGE_SHIFT + size_order - 1);
 		/* Set the S bit to indicate flushing more than 1 page */
 		desc->qw1 |= QI_DEV_EIOTLB_SIZE;
 	}
Signature · 1 line
2.43.0
Reviewed-by on v1 1/5
#2 lore raw

Re: [PATCH 1/5] iommu/vt-d: Fix shift overflow in qi_desc_dev_iotlb_pasid()

Quoted 93 lines · Lu Baolu

Callers request a full Device-TLB flush by passing MAX_AGAW_PFN_WIDTH (64 - VTD_PAGE_SHIFT == 52) as @size_order. Two shifts in qi_desc_dev_iotlb_pasid() are not prepared for a value that large:

unsigned long mask = 1UL << (VTD_PAGE_SHIFT + size_order - 1);
...
if (!IS_ALIGNED(addr, VTD_PAGE_SIZE << size_order))

The first evaluates to 1UL << 63. On 32-bit builds this is undefined behaviour; in practice x86 masks the shift count to 5 bits, so the expression yields 1UL << 31 and ~mask becomes 0x7fffffff. That value is zero-extended when it is applied to the 64-bit descriptor, so

desc->qw1 &= ~mask;

clears qw1[63:32] as well as bit 31. The ADDR field, which had just been filled with ones to request the widest possible range, collapses to 0x7ffff000. As the S bit remains set, hardware decodes the least significant zero bit of ADDR and invalidates only 2GiB instead of the entire address space. Device-TLB entries above that boundary survive the unmap, leaving an ATS-capable device able to keep accessing memory that has already been freed.

The second shift, VTD_PAGE_SIZE << size_order, is 1UL << 64 and is therefore undefined on 64-bit builds too. On x86_64 the shift count masks to zero, IS_ALIGNED(addr, 1) is trivially true and the sanity check silently degrades into a no-op.

Compute both quantities in 64-bit and clamp @size_order to the largest range the ADDR field can encode. Capping at 63 - VTD_PAGE_SHIFT keeps the intended "flush everything" behaviour: qw1[62:12] is set, bit 62 is cleared as the size indicator and the S bit is set. The non-PASID variant qi_desc_dev_iotlb() already uses 1ULL and is unaffected.

Fixes f701c9f36bcb7 ("iommu/vt-d: Factor out invalidation descriptor composition")
Assisted-by Claude:claude-opus-5

---
drivers/iommu/intel/iommu.h | 16 ++++++++++++----

1 file changed, 12 insertions(+), 4 deletions(-)
drivers/iommu/intel/iommu.h +12 -4
index c00f44db0020..8a59c7c9d0a6 100644
@@ -1105,12 +1105,20 @@ static inline void qi_desc_dev_iotlb_pasid(u16 sid, u16 pfsid, u32 pasid,
 					   unsigned int size_order,
 					   struct qi_desc *desc)
 {
-	unsigned long mask = 1UL << (VTD_PAGE_SHIFT + size_order - 1);
-
 	desc->qw0 = QI_DEV_EIOTLB_PASID(pasid) | QI_DEV_EIOTLB_SID(sid) |
 		QI_DEV_EIOTLB_QDEP(qdep) | QI_DEIOTLB_TYPE |
 		QI_DEV_IOTLB_PFSID(pfsid);
 
+	/*
+	 * The invalidation range is encoded in the ADDR field, which only
+	 * covers bits 63:12.  Clamp @size_order so that callers asking for a
+	 * full flush (e.g. with MAX_AGAW_PFN_WIDTH) do not overflow the
+	 * shifts below.  The clamped value still spans the whole range that
+	 * the descriptor is able to express.
+	 */
+	if (size_order > 63 - VTD_PAGE_SHIFT)
+		size_order = 63 - VTD_PAGE_SHIFT;
+
 	/*
 	 * If S bit is 0, we only flush a single page. If S bit is set,
 	 * The least significant zero bit indicates the invalidation address
@@ -1120,7 +1128,7 @@ static inline void qi_desc_dev_iotlb_pasid(u16 sid, u16 pfsid, u32 pasid,
 	 * Max Invs Pending (MIP) is set to 0 for now until we have DIT in
 	 * ECAP.
 	 */
-	if (!IS_ALIGNED(addr, VTD_PAGE_SIZE << size_order))
+	if (!IS_ALIGNED(addr, BIT_ULL(VTD_PAGE_SHIFT + size_order)))
 		pr_warn_ratelimited("Invalidate non-aligned address %llx, order %d\n",
 				    addr, size_order);
 
@@ -1136,7 +1144,7 @@ static inline void qi_desc_dev_iotlb_pasid(u16 sid, u16 pfsid, u32 pasid,
 		desc->qw1 |= GENMASK_ULL(size_order + VTD_PAGE_SHIFT - 1,
 					VTD_PAGE_SHIFT);
 		/* Clear size_order bit to indicate size */
-		desc->qw1 &= ~mask;
+		desc->qw1 &= ~BIT_ULL(VTD_PAGE_SHIFT + size_order - 1);
 		/* Set the S bit to indicate flushing more than 1 page */
 		desc->qw1 |= QI_DEV_EIOTLB_SIZE;
 	}
Signature · 1 line
2.43.0
author maintainer
#3 lore raw

PATCH 2/5 iommu/vt-d: Clear Present bit before tearing down copied context entry

In reply to Lu Baolu

copied_context_tear_down() zeroes the 128-bit context entry with context_clear_entry() while the Present bit is still set, and only then issues the context-cache and IOTLB invalidations. This leaves a window in which hardware can fetch a torn entry, with some fields already zeroed while Present is still set, leading to unpredictable behaviour or spurious faults. While x86 provides strong write ordering, the compiler may reorder the writes to the two 64-bit halves of the entry, and the hardware fetch is not guaranteed to be atomic with respect to multiple CPU writes.

There is no cacheline flush before the invalidation either, so on an IOMMU without coherent access to the context table the zeroed entry may not be visible to hardware at the point the invalidation is submitted.

Apply the same ownership handshake described in the VT-d spec, Section 6.5.3.3 ("Guidance to Software for Invalidations"): clear only the Present bit, flush it out to the IOMMU, perform the invalidations, and only then zero the remainder of the entry.

Fixes c7191984e5aad ("iommu/vt-d: Factor out helpers from domain_context_mapping_one()")
Assisted-by Claude:claude-opus-5
---
 drivers/iommu/intel/iommu.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/iommu/intel/iommu.c +5 -1
index 058967b669d9..528b59e5f4ce 100644
@@ -1115,7 +1115,8 @@ static void copied_context_tear_down(struct intel_iommu *iommu,
 	assert_spin_locked(&iommu->lock);
 
 	did_old = context_domain_id(context);
-	context_clear_entry(context);
+	context_clear_present(context);
+	__iommu_flush_cache(iommu, context, sizeof(*context));
 
 	if (did_old < iommu->max_domain_id) {
 		iommu->flush.flush_context(iommu, did_old,
@@ -1126,6 +1127,9 @@ static void copied_context_tear_down(struct intel_iommu *iommu,
 					 DMA_TLB_DSI_FLUSH);
 	}
 
+	context_clear_entry(context);
+	__iommu_flush_cache(iommu, context, sizeof(*context));
+
 	clear_context_copied(iommu, bus, devfn);
 }
 
Signature · 1 line
2.43.0
author maintainer
#4 lore raw

PATCH 3/5 iommu/vt-d: Fix iopf_refcount leak on RID domain replacement

In reply to Lu Baolu

intel_iommu_attach_device() enables IOPF for the new domain but never disables it for the old one. device_block_translation(), called at the start of the function, tears down translation but does not touch any IOPF state; blocking_domain_attach_dev() has to call iopf_for_domain_remove() explicitly before invoking it for exactly this reason.

identity_domain_attach_dev() has the same problem. Its comment claims that no PRI handling is needed because the device has been put in the blocking state, but the blocking state and the IOPF reference count are independent of each other.

As a result, replacing a domain that has an iopf_handler with another domain at RID level leaks a reference in info->iopf_refcount. The count never drops back to zero, so iopf_queue_remove_device() is never called and iommu_disable_pci_pri() triggers its WARN_ON(info->iopf_refcount) when the device is released.

The PASID paths already handle this correctly by way of iopf_for_domain_replace(); convert the two RID paths to do the same. Using the replace helper rather than a bare remove keeps the enable before the disable, so the reference count does not transiently reach zero and evict the device from the IOPF queue.

Fixes 17fce9d2336d ("iommu/vt-d: Put iopf enablement in domain attach path")
Assisted-by Claude:claude-opus-5
---
 drivers/iommu/intel/iommu.c | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)
drivers/iommu/intel/iommu.c +8 -5
index 528b59e5f4ce..6baf1c075dc9 100644
@@ -3158,13 +3158,13 @@ static int intel_iommu_attach_device(struct iommu_domain *domain,
 	if (ret)
 		return ret;
 
-	ret = iopf_for_domain_set(domain, dev);
+	ret = iopf_for_domain_replace(domain, old, dev);
 	if (ret)
 		return ret;
 
 	ret = dmar_domain_attach_device(to_dmar_domain(domain), dev);
 	if (ret)
-		iopf_for_domain_remove(domain, dev);
+		iopf_for_domain_replace(old, domain, dev);
 
 	return ret;
 }
@@ -3870,10 +3870,13 @@ static int identity_domain_attach_dev(struct iommu_domain *domain,
 		return 0;
 
 	/*
-	 * No PRI support with the global identity domain. No need to enable or
-	 * disable PRI in this path as the iommu has been put in the blocking
-	 * state.
+	 * The identity domain has no iopf_handler, so no IOPF reference is
+	 * taken for it.  The reference held by the old domain must still be
+	 * released here; putting the device in the blocking state above does
+	 * not affect the IOPF reference count.
 	 */
+	iopf_for_domain_remove(old, dev);
+
 	if (sm_supported(iommu))
 		ret = intel_pasid_setup_pass_through(iommu, dev, IOMMU_NO_PASID);
 	else
Signature · 1 line
2.43.0
author maintainer
#5 lore raw

PATCH 4/5 iommu/vt-d: Tear down scalable-mode context on probe failure

In reply to Lu Baolu

intel_pasid_setup_sm_context() walks a PCI device’s DMA aliases via pci_for_each_dma_alias() and programs a scalable-mode context entry for each RID. For a device with a dma_alias_mask, the callback is invoked once for the device’s own RID and once for each alias bit, all with the same pci_dev, so device_pasid_table_setup() runs for multiple RIDs.

pci_for_each_dma_alias() stops at the first callback error. Therefore, a failure partway through the walk can leave context entries for already processed RIDs present and still pointing to the device’s PASID table.

On this error path, intel_iommu_probe_device() currently jumps directly to intel_pasid_free_table(), which frees the PASID table without first tearing down those context entries. The IOMMU may then walk a present context entry whose PASID table pointer references freed memory.

intel_iommu_release_device() already performs teardown before freeing the table. Apply the same ordering on the probe failure path.

device_pasid_table_teardown() safely handles RIDs that were never
programmed: iommu_context_addr() returns NULL when no context table has been allocated, and clearing the Present bit of an already non-present entry is a no-op. So unwind is safe for both the alias that failed and any aliases not yet reached.

Fixes 301f1a80487fd ("iommu/vt-d: Setup scalable mode context entry in probe path")
Assisted-by Claude:claude-opus-5
---
 drivers/iommu/intel/iommu.c | 1 +
 1 file changed, 1 insertion(+)
drivers/iommu/intel/iommu.c +1 -0
index 6baf1c075dc9..489e9ab5f698 100644
@@ -3342,6 +3342,7 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev)
 
 	return &iommu->iommu;
 free_table:
+	intel_pasid_teardown_sm_context(dev);
 	intel_pasid_free_table(dev);
 clear_rbtree:
 	device_rbtree_remove(info);
Signature · 1 line
2.43.0
author maintainer
#6 lore raw

PATCH 5/5 iommu/vt-d: Flush context cache with correct SID when tearing down aliases

In reply to Lu Baolu

domain_context_clear_one() and device_pasid_table_teardown() are both invoked once per DMA alias of a device. Each function locates the context entry using the bus/devfn pair provided by the pci_for_each_dma_alias() callback, then calls intel_context_flush_no_pasid(), which constructs a device-selective context-cache invalidation from info->bus and info->devfn (that is, always the requester ID of the device itself).

As a result, for every alias other than the device’s own RID, the context entry that was just cleared in memory is never invalidated in the context cache. Hardware may continue using that stale cached entry. In the scalable-mode teardown path, intel_pasid_free_table() can then free the PASID directory still referenced by that stale entry, allowing the IOMMU to walk freed memory.

Fix this by passing the source ID of the entry being torn down to intel_context_flush_no_pasid(), instead of deriving it from @info.

Fixes f90584f4beb84 ("iommu/vt-d: Add helper to flush caches for context change")
Assisted-by Claude:claude-opus-5
---
 drivers/iommu/intel/iommu.h | 2 +-
 drivers/iommu/intel/iommu.c | 2 +-
 drivers/iommu/intel/pasid.c | 9 ++++++---
 3 files changed, 8 insertions(+), 5 deletions(-)
drivers/iommu/intel/iommu.h +1 -1
index 8a59c7c9d0a6..7f01620bf3a1 100644
@@ -1249,7 +1249,7 @@ void cache_tag_flush_range_np(struct dmar_domain *domain, unsigned long start,
 			      unsigned long end);
 
 void intel_context_flush_no_pasid(struct device_domain_info *info,
-				  struct context_entry *context, u16 did);
+				  struct context_entry *context, u16 did, u16 sid);
 
 int intel_iommu_enable_prq(struct intel_iommu *iommu);
 int intel_iommu_finish_prq(struct intel_iommu *iommu);
drivers/iommu/intel/iommu.c +1 -1
index 489e9ab5f698..2e3b3ab216f8 100644
@@ -1257,7 +1257,7 @@ static void domain_context_clear_one(struct device_domain_info *info, u8 bus, u8
 	context_clear_present(context);
 	__iommu_flush_cache(iommu, context, sizeof(*context));
 	spin_unlock(&iommu->lock);
-	intel_context_flush_no_pasid(info, context, did);
+	intel_context_flush_no_pasid(info, context, did, PCI_DEVID(bus, devfn));
 	context_clear_entry(context);
 	__iommu_flush_cache(iommu, context, sizeof(*context));
 }
drivers/iommu/intel/pasid.c +6 -3
index 81353fd46b37..e4f24d3f19a6 100644
@@ -751,7 +751,7 @@ static void device_pasid_table_teardown(struct device *dev, u8 bus, u8 devfn)
 	context_clear_present(context);
 	__iommu_flush_cache(iommu, context, sizeof(*context));
 	spin_unlock(&iommu->lock);
-	intel_context_flush_no_pasid(info, context, did);
+	intel_context_flush_no_pasid(info, context, did, PCI_DEVID(bus, devfn));
 	context_clear_entry(context);
 	__iommu_flush_cache(iommu, context, sizeof(*context));
 }
@@ -955,9 +955,12 @@ static void __context_flush_dev_iotlb(struct device_domain_info *info)
  * This helper can only be used when IOMMU is working in the legacy mode or
  * IOMMU is in scalable mode but all PASID table entries of the device are
  * non-present.
+ *
+ * @sid identifies the context entry that was modified, which may be a DMA
+ * alias of @info->dev rather than its own requester ID.
  */
 void intel_context_flush_no_pasid(struct device_domain_info *info,
-				  struct context_entry *context, u16 did)
+				  struct context_entry *context, u16 did, u16 sid)
 {
 	struct intel_iommu *iommu = info->iommu;
 
@@ -967,7 +970,7 @@ void intel_context_flush_no_pasid(struct device_domain_info *info,
 	 * when operating in scalable mode. Therefore the @did value doesn't
 	 * matter in scalable mode.
 	 */
-	iommu->flush.flush_context(iommu, did, PCI_DEVID(info->bus, info->devfn),
+	iommu->flush.flush_context(iommu, did, sid,
 				   DMA_CCMD_MASK_NOBIT, DMA_CCMD_DEVICE_INVL);
 
 	/*
Signature · 1 line
2.43.0
Reviewed-by on v1 5/5
#7 lore raw

Re: [PATCH 5/5] iommu/vt-d: Flush context cache with correct SID when tearing down aliases

Quoted 22 lines · Lu Baolu

domain_context_clear_one() and device_pasid_table_teardown() are both invoked once per DMA alias of a device. Each function locates the context entry using the bus/devfn pair provided by the pci_for_each_dma_alias() callback, then calls intel_context_flush_no_pasid(), which constructs a device-selective context-cache invalidation from info->bus and info->devfn (that is, always the requester ID of the device itself).

As a result, for every alias other than the device’s own RID, the context entry that was just cleared in memory is never invalidated in the context cache. Hardware may continue using that stale cached entry. In the scalable-mode teardown path, intel_pasid_free_table() can then free the PASID directory still referenced by that stale entry, allowing the IOMMU to walk freed memory.

Fix this by passing the source ID of the entry being torn down to intel_context_flush_no_pasid(), instead of deriving it from @info.

Fixes f90584f4beb84 ("iommu/vt-d: Add helper to flush caches for context change")
Assisted-by Claude:claude-opus-5
Quoted 6 lines

---

drivers/iommu/intel/iommu.h | 2 +-
drivers/iommu/intel/iommu.c | 2 +-
drivers/iommu/intel/pasid.c | 9 ++++++---
3 files changed, 8 insertions(+), 5 deletions(-)

Sami

author maintainer
#8 lore raw
In reply to Lu Baolu
Quoted 19 lines · Lu Baolu

Hi,

This collects a number of pre-existing issues that were reported by Sashiko while reviewing recent VT-d patches. None of them are regressions from the patches under review; they are latent problems in code those patches happened to touch. They are independent of each other and can be applied in any order.

Thanks,
baolu

Lu Baolu (5):
iommu/vt-d: Fix shift overflow in qi_desc_dev_iotlb_pasid() iommu/vt-d: Clear Present bit before tearing down copied context entry iommu/vt-d: Fix iopf_refcount leak on RID domain replacement iommu/vt-d: Tear down scalable-mode context on probe failure iommu/vt-d: Flush context cache with correct SID when tearing down
aliases

Quoted 4 lines
  drivers/iommu/intel/iommu.h | 18 +++++++++++++-----
  drivers/iommu/intel/iommu.c | 22 +++++++++++++++-------
  drivers/iommu/intel/pasid.c |  9 ++++++---
  3 files changed, 34 insertions(+), 15 deletions(-)

Queued for iommu/vt-d.