iommu/vt-d: Fixes for v6.1-rc8
Subsystems (2)
Lu Baolu
maintainerHi Joerg,
Below fixes are queued for v6.1. They aim to:
- Add a fix to handle a QAT device quirk.
- Fix 3 PCI device refcount leaks.
This series is also available at github.
https://github.com/LuBaolu/intel-iommu/commits/vtd-fix-for-v6.1-rc8
Please consider it for the iommu/fix branch.
Best regards,
Lu Baolu
Jacob Pan (1):
iommu/vt-d: Add a fix for devices need extra dtlb flush
Xiongfeng Wang (2):
iommu/vt-d: Fix PCI device refcount leak in has_external_pci() iommu/vt-d: Fix PCI device refcount leak in dmar_dev_scope_init()
Yang Yingliang (1):
iommu/vt-d: Fix PCI device refcount leak in prq_event_thread()
drivers/iommu/intel/iommu.h | 4 ++ drivers/iommu/intel/dmar.c | 1 + drivers/iommu/intel/iommu.c | 73 +++++++++++++++++++++++++++++++++++-- drivers/iommu/intel/svm.c | 19 +++++++--- 4 files changed, 88 insertions(+), 9 deletions(-)
Signature · 1 line
2.34.1
Lu Baolu
maintainerPATCH 1/4 iommu/vt-d: Add a fix for devices need extra dtlb flush
From: Jacob Pan <[email protected]>
QAT devices on Intel Sapphire Rapids and Emerald Rapids have a defect in address translation service (ATS). These devices may inadvertently issue ATS invalidation completion before posted writes initiated with translated address that utilized translations matching the invalidation address range, violating the invalidation completion ordering.
This patch adds an extra device TLB invalidation for the affected devices, it is needed to ensure no more posted writes with translated address following the invalidation completion. Therefore, the ordering is preserved and data-corruption is prevented.
Device TLBs are invalidated under the following six conditions:
1. Device driver does DMA API unmap IOVA
2. Device driver unbind a PASID from a process, sva_unbind_device()
3. PASID is torn down, after PASID cache is flushed. e.g. process exit_mmap() due to crash
4. Under SVA usage, called by mmu_notifier.invalidate_range() where VM has to free pages that were unmapped
5. userspace driver unmaps a DMA buffer
6. Cache invalidation in vSVA usage (upcoming)
For #1 and #2, device drivers are responsible for stopping DMA traffic before unmap/unbind. For #3, iommu driver gets mmu_notifier to invalidate TLB the same way as normal user unmap which will do an extra invalidation. The dTLB invalidation after PASID cache flush does not need an extra invalidation.
Therefore, we only need to deal with #4 and #5 in this patch. #1 is also covered by this patch due to common code path with #5.
--- drivers/iommu/intel/iommu.h | 4 +++ drivers/iommu/intel/iommu.c | 69 +++++++++++++++++++++++++++++++++++-- drivers/iommu/intel/svm.c | 5 ++- 3 files changed, 75 insertions(+), 3 deletions(-)
drivers/iommu/intel/iommu.h +4 -0
@@ -623,6 +623,7 @@ struct device_domain_info { u8 pri_enabled:1; u8 ats_supported:1; u8 ats_enabled:1; + u8 dtlb_extra_inval:1; /* Quirk for devices need extra flush */u8 ats_qdep;struct device *dev;/* it's NULL for PCIe-to-PCI bridge */ struct intel_iommu *iommu;/* IOMMU used by this device */
@@ -728,6 +729,9 @@ void qi_flush_piotlb(struct intel_iommu *iommu, u16 did, u32 pasid, u64 addr,void qi_flush_dev_iotlb_pasid (struct intel_iommu *iommu,u16 sid,u16 pfsid,u32 pasid,u16 qdep,u64 addr,unsigned int size_order); +void quirk_extra_dev_tlb_flush (struct device_domain_info *info, +unsigned long address,unsigned long pages, +u32 pasid,u16 qdep);void qi_flush_pasid_cache (struct intel_iommu *iommu,u16 did,u64 granu,u32 pasid);
drivers/iommu/intel/iommu.c +67 -2
91 lines not on this page · raw message
drivers/iommu/intel/svm.c +4 -1
@@ -184,10 +184,13 @@ static void __flush_svm_range_dev(struct intel_svm *svm,return ;qi_flush_piotlb (sdev->iommu, sdev->did, svm->pasid, address, pages, ih); -if (info->ats_enabled) +if (info->ats_enabled) {qi_flush_dev_iotlb_pasid (sdev->iommu, sdev->sid, info->pfsid, svm->pasid, sdev->qdep, address,order_base_2 (pages)); +quirk_extra_dev_tlb_flush (info, address,order_base_2 (pages), + svm->pasid, sdev->qdep); + } } static void intel_flush_svm_range_dev(struct intel_svm *svm,
Signature · 1 line
2.34.1
Baolu Lu
maintainerRe: [PATCH 1/4] iommu/vt-d: Add a fix for devices need extra dtlb flush
Quoted 29 lines · Lu Baolu
From: Jacob Pan<[email protected]>
QAT devices on Intel Sapphire Rapids and Emerald Rapids have a defect in address translation service (ATS). These devices may inadvertently issue ATS invalidation completion before posted writes initiated with translated address that utilized translations matching the invalidation address range, violating the invalidation completion ordering.
This patch adds an extra device TLB invalidation for the affected devices, it is needed to ensure no more posted writes with translated address following the invalidation completion. Therefore, the ordering is preserved and data-corruption is prevented.
Device TLBs are invalidated under the following six conditions:
1. Device driver does DMA API unmap IOVA
2. Device driver unbind a PASID from a process,sva_unbind_device()
3. PASID is torn down, after PASID cache is flushed. e.g. processexit_mmap()due to crash
4. Under SVA usage, called bymmu_notifier.invalidate_range()where VM has to free pages that were unmapped
5. userspace driver unmaps a DMA buffer
6. Cache invalidation in vSVA usage (upcoming)For #1 and #2, device drivers are responsible for stopping DMA traffic before unmap/unbind. For #3, iommu driver gets
mmu_notifierto invalidate TLB the same way as normal user unmap which will do an extra invalidation. The dTLB invalidation after PASID cache flush does not need an extra invalidation.
Quoted 6 lines
Therefore, we only need to deal with #4 and #5 in this patch. #1 is also covered by this patch due to common code path with #5.
Tested-by Yuzhang Luo<[email protected]>Reviewed-by Ashok Raj<[email protected]>Reviewed-by Kevin Tian<[email protected]>
This missed cc stable tag.
Sorry for the inconvenience.
Quoted 8 lines
Signed-off-by Jacob Pan<[email protected]>Signed-off-by Lu Baolu<[email protected]>--- drivers/iommu/intel/iommu.h | 4 +++ drivers/iommu/intel/iommu.c | 69 +++++++++++++++++++++++++++++++++++-- drivers/iommu/intel/svm.c | 5 ++- 3 files changed, 75 insertions(+), 3 deletions(-)
Best regards,
baolu
Lu Baolu
maintainerPATCH 2/4 iommu/vt-d: Fix PCI device refcount leak in prq_event_thread()
In reply to Lu BaoluFrom: Yang Yingliang <[email protected]>
As comment of pci_get_domain_bus_and_slot() says, it returns a pci device with refcount increment, when finish using it, the caller must decrease the reference count by calling pci_dev_put(). So call pci_dev_put() after using the 'pdev' to avoid refcount leak.
Besides, if the 'pdev' is null or intel_svm_prq_report() returns error, there is no need to trace this fault.
--- drivers/iommu/intel/svm.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
drivers/iommu/intel/svm.c +9 -5
@@ -748,12 +748,16 @@ static irqreturn_t prq_event_thread(int irq, void *d)* If prq is to be handled outside iommu driver via receiver of * the fault notifiers, we skip the page response here. */ -if (!pdev ||intel_svm_prq_report (iommu, &pdev->dev, req)) -handle_bad_prq_event (iommu, req,QI_RESP_INVALID ); +if (!pdev) + goto bad_req; -trace_prq_report (iommu, &pdev->dev, req->qw_0, req->qw_1, - req->priv_data[0 ], req->priv_data[1 ], - iommu->prq_seq_number++); +if (intel_svm_prq_report (iommu, &pdev->dev, req)) +handle_bad_prq_event (iommu, req,QI_RESP_INVALID ); +else +trace_prq_report (iommu, &pdev->dev, req->qw_0, req->qw_1, + req->priv_data[0 ], req->priv_data[1 ], + iommu->prq_seq_number++); +pci_dev_put (pdev); prq_advance: head = (head +sizeof (*req)) &PRQ_RING_MASK ; }
Signature · 1 line
2.34.1
Lu Baolu
maintainerPATCH 3/4 iommu/vt-d: Fix PCI device refcount leak in has_external_pci()
In reply to Lu BaoluFrom: Xiongfeng Wang <[email protected]>
for_each_pci_dev() is implemented by pci_get_device(). The comment of pci_get_device() says that it will increase the reference count for the returned pci_dev and also decrease the reference count for the input pci_dev @from if it is not NULL.
If we break for_each_pci_dev() loop with pdev not NULL, we need to call pci_dev_put() to decrease the reference count. Add the missing pci_dev_put() before 'return true' to avoid reference count leak.
--- drivers/iommu/intel/iommu.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/iommu/intel/iommu.c +3 -1
@@ -3873,8 +3873,10 @@ static inline bool has_external_pci(void)struct pci_dev *pdev =NULL ;for_each_pci_dev (pdev) -if (pdev->external_facing) +if (pdev->external_facing) { +pci_dev_put (pdev);return true; + }return false; }
Signature · 1 line
2.34.1
Lu Baolu
maintainerPATCH 4/4 iommu/vt-d: Fix PCI device refcount leak in dmar_dev_scope_init()
In reply to Lu BaoluFrom: Xiongfeng Wang <[email protected]>
for_each_pci_dev() is implemented by pci_get_device(). The comment of pci_get_device() says that it will increase the reference count for the returned pci_dev and also decrease the reference count for the input pci_dev @from if it is not NULL.
If we break for_each_pci_dev() loop with pdev not NULL, we need to call pci_dev_put() to decrease the reference count. Add the missing pci_dev_put() for the error path to avoid reference count leak.
--- drivers/iommu/intel/dmar.c | 1 + 1 file changed, 1 insertion(+)
drivers/iommu/intel/dmar.c +1 -0
@@ -820,6 +820,7 @@ int __init dmar_dev_scope_init(void) info =dmar_alloc_pci_notify_info (dev,BUS_NOTIFY_ADD_DEVICE );if (!info) { +pci_dev_put (dev);return dmar_dev_scope_status; }else {dmar_pci_bus_add_dev (info);
Signature · 1 line
2.34.1
Quoted 7 lines · Lu Baolu
Jacob Pan (1):
iommu/vt-d: Add a fix for devices need extra dtlb flushXiongfeng Wang (2):
iommu/vt-d: Fix PCI device refcount leak inhas_external_pci()iommu/vt-d: Fix PCI device refcount leak indmar_dev_scope_init()
Quoted 8 lines
Yang Yingliang (1):
iommu/vt-d: Fix PCI device refcount leak inprq_event_thread()drivers/iommu/intel/iommu.h | 4 ++ drivers/iommu/intel/dmar.c | 1 + drivers/iommu/intel/iommu.c | 73 +++++++++++++++++++++++++++++++++++-- drivers/iommu/intel/svm.c | 19 +++++++--- 4 files changed, 88 insertions(+), 9 deletions(-)
Applied to iommu/fixes, thanks Baolu.
Hi Joerg,
Quoted 3 lines · Joerg Roedel
Quoted 2 lines · Lu Baolu
Jacob Pan (1):
iommu/vt-d: Add a fix for devices need extra dtlb flush
There is a bug in this patch, I will send a fix patch. Or can you squash the fix below?
From: Jacob Pan <[email protected]>
Date: Fri, 2 Dec 2022 16:22:42 -0800
Subject: [PATCH] iommu/vt-d: Fix buggy QAT device mask
Impacted QAT device IDs that need extra dtlb flush quirk is ranging from 0x4940 to 0x4943. After bitwise AND device ID with 0xfffc the result should be 0x4940 instead of 0x494c to identify these devices.
--- drivers/iommu/intel/iommu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/iommu/intel/iommu.c +1 -1
@@ -1401,7 +1401,7 @@ static void domain_update_iotlb(struct dmar_domain *domain)* This quirk is exempted from risky_device() check because it applies only * to the built-in QAT devices and it doesn't grant additional privileges. */ -#define BUGGY_QAT_DEVID_MASK 0x494c +#define BUGGY_QAT_DEVID_MASK 0x4940static bool dev_needs_extra_dtlb_flush (struct pci_dev *pdev) {if (pdev->vendor !=PCI_VENDOR_ID_INTEL )
sorry about that,
Thanks,
Jacob