iommu/vt-d: Fixes for issues reported by Sashiko
Subsystems (3)
Lu Baolu
maintainercover iommu/vt-d: Fixes for issues reported by Sashiko
Hi,
This collects a number of pre-existing issues that were reported by Sashiko while reviewing recent VT-d patches. None of them are regressions from the patches under review; they are latent problems in code those patches happened to touch. They are independent of each other and can be applied in any order.
Thanks,
baolu
Lu Baolu (5):
iommu/vt-d: Fix shift overflow in qi_desc_dev_iotlb_pasid() iommu/vt-d: Clear Present bit before tearing down copied context entry iommu/vt-d: Fix iopf_refcount leak on RID domain replacement iommu/vt-d: Tear down scalable-mode context on probe failure iommu/vt-d: Flush context cache with correct SID when tearing down
aliases
drivers/iommu/intel/iommu.h | 18 +++++++++++++----- drivers/iommu/intel/iommu.c | 22 +++++++++++++++------- drivers/iommu/intel/pasid.c | 9 ++++++--- 3 files changed, 34 insertions(+), 15 deletions(-)
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH 1/5 iommu/vt-d: Fix shift overflow in qi_desc_dev_iotlb_pasid()
Callers request a full Device-TLB flush by passing MAX_AGAW_PFN_WIDTH (64 - VTD_PAGE_SHIFT == 52) as @size_order. Two shifts in qi_desc_dev_iotlb_pasid() are not prepared for a value that large:
unsigned long mask = 1UL << (VTD_PAGE_SHIFT + size_order - 1); ... if (!IS_ALIGNED(addr, VTD_PAGE_SIZE << size_order))
The first evaluates to 1UL << 63. On 32-bit builds this is undefined behaviour; in practice x86 masks the shift count to 5 bits, so the expression yields 1UL << 31 and ~mask becomes 0x7fffffff. That value is zero-extended when it is applied to the 64-bit descriptor, so
desc->qw1 &= ~mask;
clears qw1[63:32] as well as bit 31. The ADDR field, which had just been filled with ones to request the widest possible range, collapses to 0x7ffff000. As the S bit remains set, hardware decodes the least significant zero bit of ADDR and invalidates only 2GiB instead of the entire address space. Device-TLB entries above that boundary survive the unmap, leaving an ATS-capable device able to keep accessing memory that has already been freed.
The second shift, VTD_PAGE_SIZE << size_order, is 1UL << 64 and is therefore undefined on 64-bit builds too. On x86_64 the shift count masks to zero, IS_ALIGNED(addr, 1) is trivially true and the sanity check silently degrades into a no-op.
Compute both quantities in 64-bit and clamp @size_order to the largest range the ADDR field can encode. Capping at 63 - VTD_PAGE_SHIFT keeps the intended "flush everything" behaviour: qw1[62:12] is set, bit 62 is cleared as the size indicator and the S bit is set. The non-PASID variant qi_desc_dev_iotlb() already uses 1ULL and is unaffected.
--- drivers/iommu/intel/iommu.h | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)
drivers/iommu/intel/iommu.h +12 -4
@@ -1105,12 +1105,20 @@ static inline void qi_desc_dev_iotlb_pasid(u16 sid, u16 pfsid, u32 pasid,unsigned int size_order, struct qi_desc *desc) { - unsigned long mask = 1UL << (VTD_PAGE_SHIFT + size_order - 1); - desc->qw0 = QI_DEV_EIOTLB_PASID(pasid) | QI_DEV_EIOTLB_SID(sid) |QI_DEV_EIOTLB_QDEP (qdep) |QI_DEIOTLB_TYPE |QI_DEV_IOTLB_PFSID (pfsid); +/* +* The invalidation range is encoded in the ADDR field, which only +* covers bits 63:12. Clamp @size_order so that callers asking for a +* full flush (e.g. with MAX_AGAW_PFN_WIDTH) do not overflow the +* shifts below. The clamped value still spans the whole range that +* the descriptor is able to express. +*/ +if (size_order >63 -VTD_PAGE_SHIFT ) + size_order =63 -VTD_PAGE_SHIFT ; +/* * If S bit is 0, we only flush a single page. If S bit is set, * The least significant zero bit indicates the invalidation address
@@ -1120,7 +1128,7 @@ static inline void qi_desc_dev_iotlb_pasid(u16 sid, u16 pfsid, u32 pasid,* Max Invs Pending (MIP) is set to 0 for now until we have DIT in * ECAP. */ -if (!IS_ALIGNED (addr,VTD_PAGE_SIZE << size_order)) +if (!IS_ALIGNED (addr,BIT_ULL (VTD_PAGE_SHIFT + size_order)))pr_warn_ratelimited ("Invalidate non-aligned address %llx, order %d\n" , addr, size_order);
@@ -1136,7 +1144,7 @@ static inline void qi_desc_dev_iotlb_pasid(u16 sid, u16 pfsid, u32 pasid, desc->qw1 |=GENMASK_ULL (size_order +VTD_PAGE_SHIFT -1 ,VTD_PAGE_SHIFT );/* Clear size_order bit to indicate size */ - desc->qw1 &= ~mask; + desc->qw1 &= ~BIT_ULL (VTD_PAGE_SHIFT + size_order -1 );/* Set the S bit to indicate flushing more than 1 page */ desc->qw1 |=QI_DEV_EIOTLB_SIZE ; }
Signature · 1 line
2.43.0
Re: [PATCH 1/5] iommu/vt-d: Fix shift overflow in qi_desc_dev_iotlb_pasid()
Quoted 93 lines · Lu Baolu
Callers request a full Device-TLB flush by passing
MAX_AGAW_PFN_WIDTH(64 -VTD_PAGE_SHIFT== 52) as @size_order. Two shifts inqi_desc_dev_iotlb_pasid()are not prepared for a value that large:unsigned long mask = 1UL << (
VTD_PAGE_SHIFT+size_order- 1);
...
if (!IS_ALIGNED(addr,VTD_PAGE_SIZE<<size_order))The first evaluates to 1UL << 63. On 32-bit builds this is undefined behaviour; in practice x86 masks the shift count to 5 bits, so the expression yields 1UL << 31 and ~mask becomes
0x7fffffff. That value is zero-extended when it is applied to the 64-bit descriptor, so
desc->qw1&= ~mask;clears qw1[63:32] as well as bit 31. The ADDR field, which had just been filled with ones to request the widest possible range, collapses to
0x7ffff000. As the S bit remains set, hardware decodes the least significant zero bit of ADDR and invalidates only 2GiB instead of the entire address space. Device-TLB entries above that boundary survive the unmap, leaving an ATS-capable device able to keep accessing memory that has already been freed.The second shift,
VTD_PAGE_SIZE<<size_order, is 1UL << 64 and is therefore undefined on 64-bit builds too. Onx86_64the shift count masks to zero,IS_ALIGNED(addr, 1)is trivially true and the sanity check silently degrades into a no-op.Compute both quantities in 64-bit and clamp @size_order to the largest range the ADDR field can encode. Capping at 63 -
VTD_PAGE_SHIFTkeeps the intended "flush everything" behaviour: qw1[62:12] is set, bit 62 is cleared as the size indicator and the S bit is set. The non-PASID variantqi_desc_dev_iotlb()already uses 1ULL and is unaffected.Fixes f701c9f36bcb7 ("iommu/vt-d: Factor out invalidation descriptor composition")Reported-by Sashiko <[email protected]>Assisted-by Claude:claude-opus-5Signed-off-by Lu Baolu <[email protected]>---
drivers/iommu/intel/iommu.h| 16 ++++++++++++----1 file changed, 12 insertions(+), 4 deletions(-)drivers/iommu/intel/iommu.h +12
-4@@ -1105,12 +1105,20 @@ static inline void qi_desc_dev_iotlb_pasid(u16 sid, u16 pfsid, u32 pasid,unsigned int size_order, struct qi_desc *desc) { - unsigned long mask = 1UL << (VTD_PAGE_SHIFT + size_order - 1); - desc->qw0 = QI_DEV_EIOTLB_PASID(pasid) | QI_DEV_EIOTLB_SID(sid) |QI_DEV_EIOTLB_QDEP (qdep) |QI_DEIOTLB_TYPE |QI_DEV_IOTLB_PFSID (pfsid); +/* +* The invalidation range is encoded in the ADDR field, which only +* covers bits 63:12. Clamp @size_order so that callers asking for a +* full flush (e.g. with MAX_AGAW_PFN_WIDTH) do not overflow the +* shifts below. The clamped value still spans the whole range that +* the descriptor is able to express. +*/ +if (size_order >63 -VTD_PAGE_SHIFT ) + size_order =63 -VTD_PAGE_SHIFT ; +/* * If S bit is 0, we only flush a single page. If S bit is set, * The least significant zero bit indicates the invalidation address @@ -1120,7 +1128,7 @@ static inline void qi_desc_dev_iotlb_pasid(u16 sid, u16 pfsid, u32 pasid,* Max Invs Pending (MIP) is set to 0 for now until we have DIT in * ECAP. */ -if (!IS_ALIGNED (addr,VTD_PAGE_SIZE << size_order)) +if (!IS_ALIGNED (addr,BIT_ULL (VTD_PAGE_SHIFT + size_order)))pr_warn_ratelimited ("Invalidate non-aligned address %llx, order %d\n" , addr, size_order);@@ -1136,7 +1144,7 @@ static inline void qi_desc_dev_iotlb_pasid(u16 sid, u16 pfsid, u32 pasid, desc->qw1 |=GENMASK_ULL (size_order +VTD_PAGE_SHIFT -1 ,VTD_PAGE_SHIFT );/* Clear size_order bit to indicate size */ - desc->qw1 &= ~mask; + desc->qw1 &= ~BIT_ULL (VTD_PAGE_SHIFT + size_order -1 );/* Set the S bit to indicate flushing more than 1 page */ desc->qw1 |=QI_DEV_EIOTLB_SIZE ; }Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH 2/5 iommu/vt-d: Clear Present bit before tearing down copied context entry
In reply to Lu Baolucopied_context_tear_down() zeroes the 128-bit context entry with context_clear_entry() while the Present bit is still set, and only then issues the context-cache and IOTLB invalidations. This leaves a window in which hardware can fetch a torn entry, with some fields already zeroed while Present is still set, leading to unpredictable behaviour or spurious faults. While x86 provides strong write ordering, the compiler may reorder the writes to the two 64-bit halves of the entry, and the hardware fetch is not guaranteed to be atomic with respect to multiple CPU writes.
There is no cacheline flush before the invalidation either, so on an IOMMU without coherent access to the context table the zeroed entry may not be visible to hardware at the point the invalidation is submitted.
Apply the same ownership handshake described in the VT-d spec, Section 6.5.3.3 ("Guidance to Software for Invalidations"): clear only the Present bit, flush it out to the IOMMU, perform the invalidations, and only then zero the remainder of the entry.
--- drivers/iommu/intel/iommu.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/iommu/intel/iommu.c +5 -1
@@ -1115,7 +1115,8 @@ static void copied_context_tear_down(struct intel_iommu *iommu,assert_spin_locked (&iommu->lock); did_old =context_domain_id (context); -context_clear_entry (context); +context_clear_present (context); +__iommu_flush_cache (iommu, context,sizeof (*context));if (did_old < iommu->max_domain_id) { iommu->flush.flush_context(iommu, did_old,
@@ -1126,6 +1127,9 @@ static void copied_context_tear_down(struct intel_iommu *iommu, DMA_TLB_DSI_FLUSH); } +context_clear_entry (context); +__iommu_flush_cache (iommu, context,sizeof (*context)); +clear_context_copied (iommu, bus, devfn); }
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH 3/5 iommu/vt-d: Fix iopf_refcount leak on RID domain replacement
In reply to Lu Baoluintel_iommu_attach_device() enables IOPF for the new domain but never disables it for the old one. device_block_translation(), called at the start of the function, tears down translation but does not touch any IOPF state; blocking_domain_attach_dev() has to call iopf_for_domain_remove() explicitly before invoking it for exactly this reason.
identity_domain_attach_dev() has the same problem. Its comment claims that no PRI handling is needed because the device has been put in the blocking state, but the blocking state and the IOPF reference count are independent of each other.
As a result, replacing a domain that has an iopf_handler with another domain at RID level leaks a reference in info->iopf_refcount. The count never drops back to zero, so iopf_queue_remove_device() is never called and iommu_disable_pci_pri() triggers its WARN_ON(info->iopf_refcount) when the device is released.
The PASID paths already handle this correctly by way of iopf_for_domain_replace(); convert the two RID paths to do the same. Using the replace helper rather than a bare remove keeps the enable before the disable, so the reference count does not transiently reach zero and evict the device from the IOPF queue.
--- drivers/iommu/intel/iommu.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
drivers/iommu/intel/iommu.c +8 -5
@@ -3158,13 +3158,13 @@ static int intel_iommu_attach_device(struct iommu_domain *domain,if (ret)return ret; - ret =iopf_for_domain_set (domain, dev); + ret =iopf_for_domain_replace (domain, old, dev);if (ret)return ret; ret =dmar_domain_attach_device (to_dmar_domain (domain), dev);if (ret) -iopf_for_domain_remove (domain, dev); +iopf_for_domain_replace (old, domain, dev);return ret; }
@@ -3870,10 +3870,13 @@ static int identity_domain_attach_dev(struct iommu_domain *domain,return 0 ;/* -* No PRI support with the global identity domain. No need to enable or -* disable PRI in this path as the iommu has been put in the blocking -* state. +* The identity domain has no iopf_handler, so no IOPF reference is +* taken for it. The reference held by the old domain must still be +* released here; putting the device in the blocking state above does +* not affect the IOPF reference count. */ +iopf_for_domain_remove (old, dev); +if (sm_supported (iommu)) ret =intel_pasid_setup_pass_through (iommu, dev,IOMMU_NO_PASID );else
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH 4/5 iommu/vt-d: Tear down scalable-mode context on probe failure
In reply to Lu Baoluintel_pasid_setup_sm_context() walks a PCI device’s DMA aliases via pci_for_each_dma_alias() and programs a scalable-mode context entry for each RID. For a device with a dma_alias_mask, the callback is invoked once for the device’s own RID and once for each alias bit, all with the same pci_dev, so device_pasid_table_setup() runs for multiple RIDs.
pci_for_each_dma_alias() stops at the first callback error. Therefore, a failure partway through the walk can leave context entries for already processed RIDs present and still pointing to the device’s PASID table.
On this error path, intel_iommu_probe_device() currently jumps directly to intel_pasid_free_table(), which frees the PASID table without first tearing down those context entries. The IOMMU may then walk a present context entry whose PASID table pointer references freed memory.
intel_iommu_release_device() already performs teardown before freeing the table. Apply the same ordering on the probe failure path.
device_pasid_table_teardown() safely handles RIDs that were never
programmed: iommu_context_addr() returns NULL when no context table has been allocated, and clearing the Present bit of an already non-present entry is a no-op. So unwind is safe for both the alias that failed and any aliases not yet reached.
--- drivers/iommu/intel/iommu.c | 1 + 1 file changed, 1 insertion(+)
drivers/iommu/intel/iommu.c +1 -0
@@ -3342,6 +3342,7 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev)return &iommu->iommu; free_table: +intel_pasid_teardown_sm_context (dev);intel_pasid_free_table (dev); clear_rbtree:device_rbtree_remove (info);
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH 5/5 iommu/vt-d: Flush context cache with correct SID when tearing down aliases
In reply to Lu Baoludomain_context_clear_one() and device_pasid_table_teardown() are both invoked once per DMA alias of a device. Each function locates the context entry using the bus/devfn pair provided by the pci_for_each_dma_alias() callback, then calls intel_context_flush_no_pasid(), which constructs a device-selective context-cache invalidation from info->bus and info->devfn (that is, always the requester ID of the device itself).
As a result, for every alias other than the device’s own RID, the context entry that was just cleared in memory is never invalidated in the context cache. Hardware may continue using that stale cached entry. In the scalable-mode teardown path, intel_pasid_free_table() can then free the PASID directory still referenced by that stale entry, allowing the IOMMU to walk freed memory.
Fix this by passing the source ID of the entry being torn down to intel_context_flush_no_pasid(), instead of deriving it from @info.
--- drivers/iommu/intel/iommu.h | 2 +- drivers/iommu/intel/iommu.c | 2 +- drivers/iommu/intel/pasid.c | 9 ++++++--- 3 files changed, 8 insertions(+), 5 deletions(-)
drivers/iommu/intel/iommu.h +1 -1
@@ -1249,7 +1249,7 @@ void cache_tag_flush_range_np(struct dmar_domain *domain, unsigned long start, unsigned long end);void intel_context_flush_no_pasid (struct device_domain_info *info, -struct context_entry *context,u16 did); +struct context_entry *context,u16 did,u16 sid);int intel_iommu_enable_prq (struct intel_iommu *iommu);int intel_iommu_finish_prq (struct intel_iommu *iommu);
drivers/iommu/intel/iommu.c +1 -1
@@ -1257,7 +1257,7 @@ static void domain_context_clear_one(struct device_domain_info *info, u8 bus, u8context_clear_present (context);__iommu_flush_cache (iommu, context,sizeof (*context));spin_unlock (&iommu->lock); -intel_context_flush_no_pasid (info, context, did); +intel_context_flush_no_pasid (info, context, did,PCI_DEVID (bus, devfn));context_clear_entry (context);__iommu_flush_cache (iommu, context,sizeof (*context)); }
drivers/iommu/intel/pasid.c +6 -3
@@ -751,7 +751,7 @@ static void device_pasid_table_teardown(struct device *dev, u8 bus, u8 devfn)context_clear_present (context);__iommu_flush_cache (iommu, context,sizeof (*context));spin_unlock (&iommu->lock); -intel_context_flush_no_pasid (info, context, did); +intel_context_flush_no_pasid (info, context, did,PCI_DEVID (bus, devfn));context_clear_entry (context);__iommu_flush_cache (iommu, context,sizeof (*context)); }
@@ -955,9 +955,12 @@ static void __context_flush_dev_iotlb(struct device_domain_info *info)* This helper can only be used when IOMMU is working in the legacy mode or * IOMMU is in scalable mode but all PASID table entries of the device are * non-present. +* +* @sid identifies the context entry that was modified, which may be a DMA +* alias of @info->dev rather than its own requester ID. */ void intel_context_flush_no_pasid (struct device_domain_info *info, -struct context_entry *context,u16 did) +struct context_entry *context,u16 did,u16 sid) {struct intel_iommu *iommu = info->iommu;
@@ -967,7 +970,7 @@ void intel_context_flush_no_pasid(struct device_domain_info *info,* when operating in scalable mode. Therefore the @did value doesn't * matter in scalable mode. */ - iommu->flush.flush_context (iommu, did,PCI_DEVID (info->bus, info->devfn), + iommu->flush.flush_context (iommu, did, sid,DMA_CCMD_MASK_NOBIT ,DMA_CCMD_DEVICE_INVL );/*
Signature · 1 line
2.43.0
Re: [PATCH 5/5] iommu/vt-d: Flush context cache with correct SID when tearing down aliases
Quoted 22 lines · Lu Baolu
domain_context_clear_one()anddevice_pasid_table_teardown()are both invoked once per DMA alias of a device. Each function locates the context entry using the bus/devfn pair provided by thepci_for_each_dma_alias()callback, then callsintel_context_flush_no_pasid(), which constructs a device-selective context-cache invalidation frominfo->busandinfo->devfn(that is, always the requester ID of the device itself).As a result, for every alias other than the device’s own RID, the context entry that was just cleared in memory is never invalidated in the context cache. Hardware may continue using that stale cached entry. In the scalable-mode teardown path,
intel_pasid_free_table()can then free the PASID directory still referenced by that stale entry, allowing the IOMMU to walk freed memory.Fix this by passing the source ID of the entry being torn down to
intel_context_flush_no_pasid(), instead of deriving it from @info.Fixes f90584f4beb84 ("iommu/vt-d: Add helper to flush caches for context change")Reported-by Sashiko <[email protected]>Assisted-by Claude:claude-opus-5Signed-off-by Lu Baolu <[email protected]>
Quoted 6 lines
---
drivers/iommu/intel/iommu.h | 2 +- drivers/iommu/intel/iommu.c | 2 +- drivers/iommu/intel/pasid.c | 9 ++++++--- 3 files changed, 8 insertions(+), 5 deletions(-)
Sami
Baolu Lu
maintainerQuoted 19 lines · Lu Baolu
Hi,
This collects a number of pre-existing issues that were reported by Sashiko while reviewing recent VT-d patches. None of them are regressions from the patches under review; they are latent problems in code those patches happened to touch. They are independent of each other and can be applied in any order.
Thanks,
baoluLu Baolu (5):
iommu/vt-d: Fix shift overflow inqi_desc_dev_iotlb_pasid()iommu/vt-d: Clear Present bit before tearing down copied context entry iommu/vt-d: Fixiopf_refcountleak on RID domain replacement iommu/vt-d: Tear down scalable-mode context on probe failure iommu/vt-d: Flush context cache with correct SID when tearing down
aliases
Quoted 4 lines
drivers/iommu/intel/iommu.h | 18 +++++++++++++----- drivers/iommu/intel/iommu.c | 22 +++++++++++++++------- drivers/iommu/intel/pasid.c | 9 ++++++--- 3 files changed, 34 insertions(+), 15 deletions(-)
Queued for iommu/vt-d.