fix vt-d hard lockup when hotplug ATS capable device
Subsystems (3)
v11 cover fix vt-d hard lockup when hotplug ATS capable device
This patchset is used to fix vt-d hard lockup reported when surprise unplug ATS capable endpoint device connects to system via PCIe switch as following topology.
+-[0000:15]-+-00.0 Intel Corporation Ice Lake Memory Map/VT-d
| +-00.1 Intel Corporation Ice Lake Mesh 2 PCIe
| +-00.2 Intel Corporation Ice Lake RAS
| +-00.4 Intel Corporation Device 0b23
| \-01.0-[16-1b]----00.0-[17-1b]--+-00.0-[18]----00.0 NVIDIA Corporation Device 2324
| +-01.0-[19]----00.0
Mellanox Technologies MT2910 Family [ConnectX-7]
User brought endpoint device 19:00.0's link down by flapping it's hotplug capable slot 17:01.0 link control register, as sequence DLLSC response, pciehp_ist() will unload device driver and power it off, durning device driver is unloading an iommu device-TLB invalidation (Intel VT-d spec, or 'ATS Invalidation' in PCIe spec) request issued to that link down device, thus a long time completion/timeout waiting in interrupt context causes continuous hard lockup warnning and system hang.
Other detail, see every patch commit log.
patch [1&2] were tested by [email protected] on stable v6.7-rc4. patch [1-5] passed compiling on stable v6.8rc1.
change log:
v11:
- update per latest comment and suggestion from Baolu and YiLiu.
- split refactoring patch into two patches, [3/5] for simplify parameters and [4/5] for pdev parameter passing.
- re-order patches.
- fold target device presence check into qi_check_fault().
- combine patch[2][5] in v10 into one patch[5].
- some commit description correctness.
- add fixes tag to patch[2/5].
- rebased on 6.8rc1
v10:
- refactor qi_submit_sync() and its callers to get pci_dev instance, as Kevin pointed out add target_flush_dev to iommu is not right.
v9:
- unify all spelling of ATS Invalidation adhere to PCIe spec per Bjorn's suggestion.
v8:
- add a patch to break the loop for timeout device-TLB invalidation, as Bjorn said there is possibility device just no response but not gone.
v7:
- reorder patches and revise commit log per Bjorn's guide.
- other code and commit log revise per Lukas' suggestion.
- rebased to stable v6.7-rc6.
v6:
- add two patches to break out device-TLB invalidation if device is gone. v5:
- add a patch try to fix the rare case (surprise remove a device in safe removal process). not work because surprise removal handling can't re-enter when another safe removal is in process.
v4:
- move the PCI device state checking after ATS per Baolu's suggestion. v3:
- fix commit description typo.
v2:
- revise commit[1] description part according to Lukas' suggestion.
- revise commit[2] description to clarify the issue's impact. v1:
- https://lore.kernel.org/lkml/20231213034637.2603013-1-haifeng.zhao@
linux.intel.com/T/
Thanks,
Ethan
Ethan Zhao (5):
PCI: make pci_dev_is_disconnected() helper public for other drivers iommu/vt-d: don't issue ATS Invalidation request when device is
disconnected
iommu/vt-d: simplify parameters of qi_submit_sync() ATS invalidation
callers
iommu/vt-d: pass pdev parameter for qi_check_fault() and refactor
callers
iommu/vt-d: improve ITE fault handling if target device isn't present
drivers/iommu/intel/dmar.c | 71 +++++++++++++++++++++++------ drivers/iommu/intel/iommu.c | 26 +++-------- drivers/iommu/intel/iommu.h | 20 ++++---- drivers/iommu/intel/irq_remapping.c | 2 +- drivers/iommu/intel/nested.c | 9 +--- drivers/iommu/intel/pasid.c | 12 ++--- drivers/iommu/intel/svm.c | 13 +++--- drivers/pci/pci.h | 5 -- include/linux/pci.h | 5 ++ 9 files changed, 93 insertions(+), 70 deletions(-)
Signature · 1 line
2.31.1
PATCH v11 1/5 PCI: make pci_dev_is_disconnected() helper public for other drivers
Make pci_dev_is_disconnected() public so that it can be called from Intel VT-d driver to quickly fix/workaround the surprise removal unplug hang issue for those ATS capable devices on PCIe switch downstream hotplug capable ports.
Beside pci_device_is_present() function, this one has no config space space access, so is light enough to optimize the normal pure surprise removal and safe removal flow.
--- drivers/pci/pci.h | 5 ----- include/linux/pci.h | 5 +++++ 2 files changed, 5 insertions(+), 5 deletions(-)
drivers/pci/pci.h +0 -5
@@ -368,11 +368,6 @@ static inline int pci_dev_set_disconnected(struct pci_dev *dev, void *unused)return 0 ; } -static inline bool pci_dev_is_disconnected (const struct pci_dev *dev) -{ -return dev->error_state == pci_channel_io_perm_failure; -} -/* pci_dev priv_flags */ #define PCI_DEV_ADDED 0#define PCI_DPC_RECOVERED 1
include/linux/pci.h +5 -0
@@ -2512,6 +2512,11 @@ static inline struct pci_dev *pcie_find_root_port(struct pci_dev *dev)return NULL ; } +static inline bool pci_dev_is_disconnected (const struct pci_dev *dev) +{ +return dev->error_state == pci_channel_io_perm_failure; +} +void pci_request_acs (void );bool pci_acs_enabled (struct pci_dev *pdev,u16 acs_flags); bool pci_acs_path_enabled(struct pci_dev *start,
Signature · 1 line
2.31.1
PATCH v11 2/5 iommu/vt-d: don't issue ATS Invalidation request when device is disconnected
In reply to Ethan ZhaoFor those endpoint devices connect to system via hotplug capable ports, users could request a hot reset to the device by flapping device's link through setting the slot's link control register, as pciehp_ist() DLLSC interrupt sequence response, pciehp will unload the device driver and then power it off. thus cause an IOMMU device-TLB invalidation (Intel VT-d spec, or ATS Invalidation in PCIe spec r6.1) request for non-existence target device to be sent and deadly loop to retry that request after ITE fault triggered in interrupt context.
That would cause following continuous hard lockup warning and system hang
[ 4211.433662] pcieport 0000:17:01.0: pciehp: Slot(108): Link Down
[ 4211.433664] pcieport 0000:17:01.0: pciehp: Slot(108): Card not present
[ 4223.822591] NMI watchdog: Watchdog detected hard LOCKUP on cpu 144
[ 4223.822622] CPU: 144 PID: 1422 Comm: irq/57-pciehp Kdump: loaded Tainted: G S
OE kernel version xxxx
[ 4223.822623] Hardware name: vendorname xxxx 666-106,
BIOS 01.01.02.03.01 05/15/2023
[ 4223.822623] RIP: 0010:qi_submit_sync+0x2c0/0x490
[ 4223.822624] Code: 48 be 00 00 00 00 00 08 00 00 49 85 74 24 20 0f 95 c1 48 8b
57 10 83 c1 04 83 3c 1a 03 0f 84 a2 01 00 00 49 8b 04 24 8b 70 34 <40> f6 c6 1
0 74 17 49 8b 04 24 8b 80 80 00 00 00 89 c2 d3 fa 41 39
[ 4223.822624] RSP: 0018:ffffc4f074f0bbb8 EFLAGS: 00000093
[ 4223.822625] RAX: ffffc4f040059000 RBX: 0000000000000014 RCX: 0000000000000005
[ 4223.822625] RDX: ffff9f3841315800 RSI: 0000000000000000 RDI: ffff9f38401a8340
[ 4223.822625] RBP: ffff9f38401a8340 R08: ffffc4f074f0bc00 R09: 0000000000000000
[ 4223.822626] R10: 0000000000000010 R11: 0000000000000018 R12: ffff9f384005e200
[ 4223.822626] R13: 0000000000000004 R14: 0000000000000046 R15: 0000000000000004
[ 4223.822626] FS: 0000000000000000(0000) GS:ffffa237ae400000(0000)
knlGS:0000000000000000
[ 4223.822627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4223.822627] CR2: 00007ffe86515d80 CR3: 000002fd3000a001 CR4: 0000000000770ee0
[ 4223.822627] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4223.822628] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[ 4223.822628] PKRU: 55555554
[ 4223.822628] Call Trace:
[ 4223.822628] qi_flush_dev_iotlb+0xb1/0xd0
[ 4223.822628] __dmar_remove_one_dev_info+0x224/0x250
[ 4223.822629] dmar_remove_one_dev_info+0x3e/0x50
[ 4223.822629] intel_iommu_release_device+0x1f/0x30
[ 4223.822629] iommu_release_device+0x33/0x60
[ 4223.822629] iommu_bus_notifier+0x7f/0x90
[ 4223.822630] blocking_notifier_call_chain+0x60/0x90
[ 4223.822630] device_del+0x2e5/0x420
[ 4223.822630] pci_remove_bus_device+0x70/0x110
[ 4223.822630] pciehp_unconfigure_device+0x7c/0x130
[ 4223.822631] pciehp_disable_slot+0x6b/0x100
[ 4223.822631] pciehp_handle_presence_or_link_change+0xd8/0x320
[ 4223.822631] pciehp_ist+0x176/0x180
[ 4223.822631] ? irq_finalize_oneshot.part.50+0x110/0x110
[ 4223.822632] irq_thread_fn+0x19/0x50
[ 4223.822632] irq_thread+0x104/0x190
[ 4223.822632] ? irq_forced_thread_fn+0x90/0x90
[ 4223.822632] ? irq_thread_check_affinity+0xe0/0xe0
[ 4223.822633] kthread+0x114/0x130
[ 4223.822633] ? __kthread_cancel_work+0x40/0x40
[ 4223.822633] ret_from_fork+0x1f/0x30
[ 4223.822633] Kernel panic - not syncing: Hard LOCKUP
[ 4223.822634] CPU: 144 PID: 1422 Comm: irq/57-pciehp Kdump: loaded Tainted: G S
OE kernel version xxxx
[ 4223.822634] Hardware name: vendorname xxxx 666-106,
BIOS 01.01.02.03.01 05/15/2023
[ 4223.822634] Call Trace:
[ 4223.822634] <NMI>
[ 4223.822635] dump_stack+0x6d/0x88
[ 4223.822635] panic+0x101/0x2d0
[ 4223.822635] ? ret_from_fork+0x11/0x30
[ 4223.822635] nmi_panic.cold.14+0xc/0xc
[ 4223.822636] watchdog_overflow_callback.cold.8+0x6d/0x81
[ 4223.822636] __perf_event_overflow+0x4f/0xf0
[ 4223.822636] handle_pmi_common+0x1ef/0x290
[ 4223.822636] ? __set_pte_vaddr+0x28/0x40
[ 4223.822637] ? flush_tlb_one_kernel+0xa/0x20
[ 4223.822637] ? __native_set_fixmap+0x24/0x30
[ 4223.822637] ? ghes_copy_tofrom_phys+0x70/0x100
[ 4223.822637] ? __ghes_peek_estatus.isra.16+0x49/0xa0
[ 4223.822637] intel_pmu_handle_irq+0xba/0x2b0
[ 4223.822638] perf_event_nmi_handler+0x24/0x40
[ 4223.822638] nmi_handle+0x4d/0xf0
[ 4223.822638] default_do_nmi+0x49/0x100
[ 4223.822638] exc_nmi+0x134/0x180
[ 4223.822639] end_repeat_nmi+0x16/0x67
[ 4223.822639] RIP: 0010:qi_submit_sync+0x2c0/0x490
[ 4223.822639] Code: 48 be 00 00 00 00 00 08 00 00 49 85 74 24 20 0f 95 c1 48 8b
57 10 83 c1 04 83 3c 1a 03 0f 84 a2 01 00 00 49 8b 04 24 8b 70 34 <40> f6 c6 10
74 17 49 8b 04 24 8b 80 80 00 00 00 89 c2 d3 fa 41 39
[ 4223.822640] RSP: 0018:ffffc4f074f0bbb8 EFLAGS: 00000093
[ 4223.822640] RAX: ffffc4f040059000 RBX: 0000000000000014 RCX: 0000000000000005
[ 4223.822640] RDX: ffff9f3841315800 RSI: 0000000000000000 RDI: ffff9f38401a8340
[ 4223.822641] RBP: ffff9f38401a8340 R08: ffffc4f074f0bc00 R09: 0000000000000000
[ 4223.822641] R10: 0000000000000010 R11: 0000000000000018 R12: ffff9f384005e200
[ 4223.822641] R13: 0000000000000004 R14: 0000000000000046 R15: 0000000000000004
[ 4223.822641] ? qi_submit_sync+0x2c0/0x490
[ 4223.822642] ? qi_submit_sync+0x2c0/0x490
[ 4223.822642] </NMI>
[ 4223.822642] qi_flush_dev_iotlb+0xb1/0xd0
[ 4223.822642] __dmar_remove_one_dev_info+0x224/0x250
[ 4223.822643] dmar_remove_one_dev_info+0x3e/0x50
[ 4223.822643] intel_iommu_release_device+0x1f/0x30
[ 4223.822643] iommu_release_device+0x33/0x60
[ 4223.822643] iommu_bus_notifier+0x7f/0x90
[ 4223.822644] blocking_notifier_call_chain+0x60/0x90
[ 4223.822644] device_del+0x2e5/0x420
[ 4223.822644] pci_remove_bus_device+0x70/0x110
[ 4223.822644] pciehp_unconfigure_device+0x7c/0x130
[ 4223.822644] pciehp_disable_slot+0x6b/0x100
[ 4223.822645] pciehp_handle_presence_or_link_change+0xd8/0x320
[ 4223.822645] pciehp_ist+0x176/0x180
[ 4223.822645] ? irq_finalize_oneshot.part.50+0x110/0x110
[ 4223.822645] irq_thread_fn+0x19/0x50
[ 4223.822646] irq_thread+0x104/0x190
[ 4223.822646] ? irq_forced_thread_fn+0x90/0x90
[ 4223.822646] ? irq_thread_check_affinity+0xe0/0xe0
[ 4223.822646] kthread+0x114/0x130
[ 4223.822647] ? __kthread_cancel_work+0x40/0x40
[ 4223.822647] ret_from_fork+0x1f/0x30
[ 4223.822647] Kernel Offset: 0x6400000 from 0xffffffff81000000 (relocationrange: 0xffffffff80000000-0xffffffffbfffffff)
Such issue could be triggered by all kinds of regular surprise removal hotplug operation. like:
1. pull EP(endpoint device) out directly.
2. turn off EP's power.
3. bring the link down.
etc.
this patch aims to work for regular safe removal and surprise removal unplug. these hot unplug handling process could be optimized to fix the ATS Invalidation hang issue by calling pci_dev_is_disconnected() in function devtlb_invalidation_with_pasid() to check target device state to avoid sending meaningless ATS Invalidation request to iommu when device is gone. (see IMPLEMENTATION NOTE in PCIe spec r6.1 section 10.3.1)
For safe removal, device wouldn't be removed until the whole software handling process is done, it wouldn't trigger the hard lock up issue caused by too long ATS Invalidation timeout wait. In safe removal path, device state isn't set to pci_channel_io_perm_failure in pciehp_unconfigure_device() by checking 'presence' parameter, calling pci_dev_is_disconnected() in devtlb_invalidation_with_pasid() will return false there, wouldn't break the function.
For surprise removal, device state is set to pci_channel_io_perm_failure in pciehp_unconfigure_device(), means device is already gone (disconnected) call pci_dev_is_disconnected() in devtlb_invalidation_with_pasid() will return true to break the function not to send ATS Invalidation request to the disconnected device blindly, thus avoid to trigger further ITE fault, and ITE fault will block all invalidation request to be handled. furthermore retry the timeout request could trigger hard lockup.
safe removal (present) & surprise removal (not present)
pciehp_ist()
pciehp_handle_presence_or_link_change()
pciehp_disable_slot()
remove_board()
pciehp_unconfigure_device(presence) {
if (!presence)
pci_walk_bus(parent, pci_dev_set_disconnected, NULL);
}this patch works for regular safe removal and surprise removal of ATS capable endpoint on PCIe switch downstream ports.
--- drivers/iommu/intel/pasid.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/iommu/intel/pasid.c +3 -0
@@ -214,6 +214,9 @@ devtlb_invalidation_with_pasid(struct intel_iommu *iommu,if (!info || !info->ats_enabled)return ; +if (pci_dev_is_disconnected (to_pci_dev (dev))) +return ; + sid = info->bus <<8 | info->devfn; qdep = info->ats_qdep; pfsid = info->pfsid;
Signature · 1 line
2.31.1
PATCH v11 3/5 iommu/vt-d: simplify parameters of qi_submit_sync() ATS invalidation callers
In reply to Ethan Zhaofold parameters back to struct device_domain_info *info instead of extract and pass them, thus decrease the number of the parameter passed for following functions
qi_flush_dev_iotlb()qi_flush_dev_iotlb_pasid()quirk_extra_dev_tlb_flush()
no function change.
--- drivers/iommu/intel/dmar.c | 26 ++++++++++++++++++++++---- drivers/iommu/intel/iommu.c | 26 +++++++------------------- drivers/iommu/intel/iommu.h | 17 ++++++++--------- drivers/iommu/intel/nested.c | 9 ++------- drivers/iommu/intel/pasid.c | 9 ++------- drivers/iommu/intel/svm.c | 7 +++---- 6 files changed, 44 insertions(+), 50 deletions(-)
drivers/iommu/intel/dmar.c +22 -4
@@ -1517,11 +1517,20 @@ void qi_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,qi_submit_sync (iommu, &desc,1 ,0 ); } -void qi_flush_dev_iotlb (struct intel_iommu *iommu,u16 sid,u16 pfsid, -u16 qdep,u64 addr,unsigned mask) +void qi_flush_dev_iotlb (struct intel_iommu *iommu, +struct device_domain_info *info,u64 addr, +unsigned int mask) { +u16 sid, qdep, pfsid;struct qi_desc desc; +if (!info || !info->ats_enabled) +return ; + + sid = info->bus <<8 | info->devfn; + qdep = info->ats_qdep; + pfsid = info->pfsid; +/* * VT-d spec, section 4.3: *
@@ -1590,11 +1599,20 @@ void qi_flush_piotlb(struct intel_iommu *iommu, u16 did, u32 pasid, u64 addr, }/* PASID-based device IOTLB Invalidate */ -void qi_flush_dev_iotlb_pasid (struct intel_iommu *iommu,u16 sid,u16 pfsid, -u32 pasid,u16 qdep,u64 addr,unsigned int size_order) +void qi_flush_dev_iotlb_pasid (struct intel_iommu *iommu, +struct device_domain_info *info,u64 addr,u32 pasid, +unsigned int size_order) {unsigned long mask =1UL << (VTD_PAGE_SHIFT + size_order -1 );struct qi_desc desc = {.qw1 =0 , .qw2 =0 , .qw3 =0 }; +u16 sid, qdep, pfsid; + +if (!info || !dev_is_pci (info->dev)) +return ; + + sid = info->bus <<8 | info->devfn; + qdep = info->ats_qdep; + pfsid = info->pfsid;/* * VT-d spec, section 4.3:
drivers/iommu/intel/iommu.c +7 -19
@@ -1310,16 +1310,11 @@ static void iommu_disable_pci_caps(struct device_domain_info *info)static void __iommu_flush_dev_iotlb (struct device_domain_info *info,u64 addr,unsigned int mask) { -u16 sid, qdep; -if (!info || !info->ats_enabled)return ; - sid = info->bus <<8 | info->devfn; - qdep = info->ats_qdep; -qi_flush_dev_iotlb (info->iommu, sid, info->pfsid, - qdep, addr, mask); -quirk_extra_dev_tlb_flush (info, addr, mask,IOMMU_NO_PASID , qdep); +qi_flush_dev_iotlb (info->iommu, info, addr, mask); +quirk_extra_dev_tlb_flush (info, addr,IOMMU_NO_PASID , mask); } static void iommu_flush_dev_iotlb(struct dmar_domain *domain,
@@ -1342,11 +1337,7 @@ static void iommu_flush_dev_iotlb(struct dmar_domain *domain,if (!info->ats_enabled)continue ; -qi_flush_dev_iotlb_pasid (info->iommu, -PCI_DEVID (info->bus, info->devfn), - info->pfsid, dev_pasid->pasid, - info->ats_qdep, addr, - mask); +qi_flush_dev_iotlb_pasid (info->iommu, info, addr, dev_pasid->pasid, mask); }spin_unlock_irqrestore (&domain->lock, flags); }
@@ -4990,9 +4981,8 @@ static void __init check_tylersburg_isoch(void)* * As a reminder, #6 will *NEED* this quirk as we enable nested translation. */ -void quirk_extra_dev_tlb_flush (struct device_domain_info *info, -unsigned long address,unsigned long mask, -u32 pasid,u16 qdep) +void quirk_extra_dev_tlb_flush (struct device_domain_info *info,u32 pasid, +unsigned long address,unsigned long mask) {u16 sid;
@@ -5001,11 +4991,9 @@ void quirk_extra_dev_tlb_flush(struct device_domain_info *info, sid =PCI_DEVID (info->bus, info->devfn);if (pasid ==IOMMU_NO_PASID ) { -qi_flush_dev_iotlb (info->iommu, sid, info->pfsid, - qdep, address, mask); +qi_flush_dev_iotlb (info->iommu, info, address, mask); }else { -qi_flush_dev_iotlb_pasid (info->iommu, sid, info->pfsid, - pasid, qdep, address, mask); +qi_flush_dev_iotlb_pasid (info->iommu, info, address, pasid, mask); } }
drivers/iommu/intel/iommu.h +8 -9
@@ -1037,18 +1037,17 @@ void qi_flush_context(struct intel_iommu *iommu, u16 did, u16 sid, u8 fm, u64 type);void qi_flush_iotlb (struct intel_iommu *iommu,u16 did,u64 addr,unsigned int size_order,u64 type); -void qi_flush_dev_iotlb (struct intel_iommu *iommu,u16 sid,u16 pfsid, -u16 qdep,u64 addr,unsigned mask); - +void qi_flush_dev_iotlb (struct intel_iommu *iommu, +struct device_domain_info *info,u64 addr, +unsigned int mask);void qi_flush_piotlb (struct intel_iommu *iommu,u16 did,u32 pasid,u64 addr,unsigned long npages,bool ih); -void qi_flush_dev_iotlb_pasid (struct intel_iommu *iommu,u16 sid,u16 pfsid, -u32 pasid,u16 qdep,u64 addr, -unsigned int size_order); -void quirk_extra_dev_tlb_flush (struct device_domain_info *info, -unsigned long address,unsigned long pages, -u32 pasid,u16 qdep); +void qi_flush_dev_iotlb_pasid (struct intel_iommu *iommu, +struct device_domain_info *info,u64 addr, +u32 pasid,unsigned int size_order); +void quirk_extra_dev_tlb_flush (struct device_domain_info *info,u32 pasid, +unsigned long address,unsigned long mask);void qi_flush_pasid_cache (struct intel_iommu *iommu,u16 did,u64 granu,u32 pasid);
drivers/iommu/intel/nested.c +2 -7
@@ -78,18 +78,13 @@ static void nested_flush_dev_iotlb(struct dmar_domain *domain, u64 addr, {struct device_domain_info *info;unsigned long flags; -u16 sid, qdep;spin_lock_irqsave (&domain->lock, flags);list_for_each_entry (info, &domain->devices, link) {if (!info->ats_enabled)continue ; - sid = info->bus <<8 | info->devfn; - qdep = info->ats_qdep; -qi_flush_dev_iotlb (info->iommu, sid, info->pfsid, - qdep, addr, mask); -quirk_extra_dev_tlb_flush (info, addr, mask, -IOMMU_NO_PASID , qdep); +qi_flush_dev_iotlb (info->iommu, info, addr, mask); +quirk_extra_dev_tlb_flush (info,IOMMU_NO_PASID , addr, mask); }spin_unlock_irqrestore (&domain->lock, flags); }
drivers/iommu/intel/pasid.c +2 -7
@@ -208,7 +208,6 @@ devtlb_invalidation_with_pasid(struct intel_iommu *iommu, struct device *dev, u32 pasid) {struct device_domain_info *info; -u16 sid, qdep, pfsid; info =dev_iommu_priv_get (dev);if (!info || !info->ats_enabled)
@@ -217,10 +216,6 @@ devtlb_invalidation_with_pasid(struct intel_iommu *iommu,if (pci_dev_is_disconnected (to_pci_dev (dev)))return ; - sid = info->bus <<8 | info->devfn; - qdep = info->ats_qdep; - pfsid = info->pfsid; -/* * When PASID 0 is used, it indicates RID2PASID(DMA request w/o PASID), * devTLB flush w/o PASID should be used. For non-zero PASID under
@@ -228,9 +223,9 @@ devtlb_invalidation_with_pasid(struct intel_iommu *iommu,* efficient to flush devTLB specific to the PASID. */ if (pasid ==IOMMU_NO_PASID ) -qi_flush_dev_iotlb (iommu, sid, pfsid, qdep,0 ,64 -VTD_PAGE_SHIFT ); +qi_flush_dev_iotlb (iommu, info,0 ,64 -VTD_PAGE_SHIFT );else -qi_flush_dev_iotlb_pasid (iommu, sid, pfsid, pasid, qdep,0 ,64 -VTD_PAGE_SHIFT ); +qi_flush_dev_iotlb_pasid (iommu, info,0 , pasid,64 -VTD_PAGE_SHIFT ); } void intel_pasid_tear_down_entry(struct intel_iommu *iommu, struct device *dev,
drivers/iommu/intel/svm.c +3 -4
@@ -181,11 +181,10 @@ static void __flush_svm_range_dev(struct intel_svm *svm,qi_flush_piotlb (sdev->iommu, sdev->did, svm->pasid, address, pages, ih);if (info->ats_enabled) { -qi_flush_dev_iotlb_pasid (sdev->iommu, sdev->sid, info->pfsid, - svm->pasid, sdev->qdep, address, +qi_flush_dev_iotlb_pasid (sdev->iommu, info, address, svm->pasid,order_base_2 (pages)); -quirk_extra_dev_tlb_flush (info, address,order_base_2 (pages), - svm->pasid, sdev->qdep); +quirk_extra_dev_tlb_flush (info, svm->pasid, address, +order_base_2 (pages)); } }
Signature · 1 line
2.31.1
kernel test robot
build botRe: [PATCH v11 3/5] iommu/vt-d: simplify parameters of qi_submit_sync() ATS invalidation callers
kernel test robot noticed the following build warnings:
Hi Ethan,
kernel test robot noticed the following build warnings:
[auto build test WARNING on pci/next]
[also build test WARNING on pci/for-linus linus/master v6.8-rc1 next-20240125] [cannot apply to joro-iommu/next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Ethan-Zhao/PCI-make-pci_dev_is_disconnected-helper-public-for-other-drivers/20240126-094305 base: https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git next patch link: https://lore.kernel.org/r/20240126014002.481294-4-haifeng.zhao%40linux.intel.com patch subject: [PATCH v11 3/5] iommu/vt-d: simplify parameters of qi_submit_sync() ATS invalidation callers config: i386-buildonly-randconfig-001-20240127 (https://download.01.org/0day-ci/archive/20240127/[email protected]/config) compiler: gcc-7 (Ubuntu 7.5.0-6ubuntu2) 7.5.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240127/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
drivers/iommu/intel/iommu.c: In function 'quirk_extra_dev_tlb_flush':
>> drivers/iommu/intel/iommu.c:4987:6: warning: variable 'sid' set but not used [-Wunused-but-set-variable]
u16 sid;
^~~vim +/sid +4987 drivers/iommu/intel/iommu.c
e65a6897be5e49 Jacob Pan 2022-12-01 4957
e65a6897be5e49 Jacob Pan 2022-12-01 4958 /*
e65a6897be5e49 Jacob Pan 2022-12-01 4959 * Here we deal with a device TLB defect where device may inadvertently issue ATS
e65a6897be5e49 Jacob Pan 2022-12-01 4960 * invalidation completion before posted writes initiated with translated address
e65a6897be5e49 Jacob Pan 2022-12-01 4961 * that utilized translations matching the invalidation address range, violating
e65a6897be5e49 Jacob Pan 2022-12-01 4962 * the invalidation completion ordering.
e65a6897be5e49 Jacob Pan 2022-12-01 4963 * Therefore, any use cases that cannot guarantee DMA is stopped before unmap is
e65a6897be5e49 Jacob Pan 2022-12-01 4964 * vulnerable to this defect. In other words, any dTLB invalidation initiated not
e65a6897be5e49 Jacob Pan 2022-12-01 4965 * under the control of the trusted/privileged host device driver must use this
e65a6897be5e49 Jacob Pan 2022-12-01 4966 * quirk.
e65a6897be5e49 Jacob Pan 2022-12-01 4967 * Device TLBs are invalidated under the following six conditions:
e65a6897be5e49 Jacob Pan 2022-12-01 4968 * 1. Device driver does DMA API unmap IOVA
e65a6897be5e49 Jacob Pan 2022-12-01 4969 * 2. Device driver unbind a PASID from a process, sva_unbind_device()
e65a6897be5e49 Jacob Pan 2022-12-01 4970 * 3. PASID is torn down, after PASID cache is flushed. e.g. process
e65a6897be5e49 Jacob Pan 2022-12-01 4971 * exit_mmap() due to crash
e65a6897be5e49 Jacob Pan 2022-12-01 4972 * 4. Under SVA usage, called by mmu_notifier.invalidate_range() where
e65a6897be5e49 Jacob Pan 2022-12-01 4973 * VM has to free pages that were unmapped
e65a6897be5e49 Jacob Pan 2022-12-01 4974 * 5. Userspace driver unmaps a DMA buffer
e65a6897be5e49 Jacob Pan 2022-12-01 4975 * 6. Cache invalidation in vSVA usage (upcoming)
e65a6897be5e49 Jacob Pan 2022-12-01 4976 *
e65a6897be5e49 Jacob Pan 2022-12-01 4977 * For #1 and #2, device drivers are responsible for stopping DMA traffic
e65a6897be5e49 Jacob Pan 2022-12-01 4978 * before unmap/unbind. For #3, iommu driver gets mmu_notifier to
e65a6897be5e49 Jacob Pan 2022-12-01 4979 * invalidate TLB the same way as normal user unmap which will use this quirk.
e65a6897be5e49 Jacob Pan 2022-12-01 4980 * The dTLB invalidation after PASID cache flush does not need this quirk.
e65a6897be5e49 Jacob Pan 2022-12-01 4981 *
e65a6897be5e49 Jacob Pan 2022-12-01 4982 * As a reminder, #6 will *NEED* this quirk as we enable nested translation.
e65a6897be5e49 Jacob Pan 2022-12-01 4983 */
e2fcf16ac26679 Ethan Zhao 2024-01-25 4984 void quirk_extra_dev_tlb_flush(struct device_domain_info *info, u32 pasid,
e2fcf16ac26679 Ethan Zhao 2024-01-25 4985 unsigned long address, unsigned long mask)
e65a6897be5e49 Jacob Pan 2022-12-01 4986 {
e65a6897be5e49 Jacob Pan 2022-12-01 @4987 u16 sid;
e65a6897be5e49 Jacob Pan 2022-12-01 4988
e65a6897be5e49 Jacob Pan 2022-12-01 4989 if (likely(!info->dtlb_extra_inval))
e65a6897be5e49 Jacob Pan 2022-12-01 4990 return;
e65a6897be5e49 Jacob Pan 2022-12-01 4991
e65a6897be5e49 Jacob Pan 2022-12-01 4992 sid = PCI_DEVID(info->bus, info->devfn);
4298780126c298 Jacob Pan 2023-08-09 4993 if (pasid == IOMMU_NO_PASID) {
e2fcf16ac26679 Ethan Zhao 2024-01-25 4994 qi_flush_dev_iotlb(info->iommu, info, address, mask);
e65a6897be5e49 Jacob Pan 2022-12-01 4995 } else {
e2fcf16ac26679 Ethan Zhao 2024-01-25 4996 qi_flush_dev_iotlb_pasid(info->iommu, info, address, pasid, mask);
e65a6897be5e49 Jacob Pan 2022-12-01 4997 }
e65a6897be5e49 Jacob Pan 2022-12-01 4998 }
dc57875866ab9f Kan Liang 2023-01-31 4999 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH v11 3/5] iommu/vt-d: simplify parameters of qi_submit_sync() ATS invalidation callers
In reply to Ethan Zhaokernel test robot noticed the following build warnings:
Hi Ethan,
kernel test robot noticed the following build warnings:
[auto build test WARNING on pci/next]
[also build test WARNING on pci/for-linus linus/master v6.8-rc1 next-20240125] [cannot apply to joro-iommu/next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Ethan-Zhao/PCI-make-pci_dev_is_disconnected-helper-public-for-other-drivers/20240126-094305 base: https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git next patch link: https://lore.kernel.org/r/20240126014002.481294-4-haifeng.zhao%40linux.intel.com patch subject: [PATCH v11 3/5] iommu/vt-d: simplify parameters of qi_submit_sync() ATS invalidation callers config: x86_64-randconfig-005-20240128 (https://download.01.org/0day-ci/archive/20240128/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240128/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
drivers/iommu/intel/svm.c: In function 'intel_flush_svm_all':
>> drivers/iommu/intel/svm.c:229:67: warning: passing argument 2 of 'qi_flush_dev_iotlb_pasid' makes pointer from integer without a cast [-Wint-conversion]
229 | qi_flush_dev_iotlb_pasid(sdev->iommu, sdev->sid, info->pfsid,
| ~~~~^~~~~
| |
| u16 {aka short unsigned int}
In file included from drivers/iommu/intel/svm.c:22:
drivers/iommu/intel/iommu.h:1047:58: note: expected 'struct device_domain_info *' but argument is of type 'u16' {aka 'short unsigned int'}
1047 | struct device_domain_info *info, u64 addr,
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
drivers/iommu/intel/svm.c:229:25: error: too many arguments to function 'qi_flush_dev_iotlb_pasid'
229 | qi_flush_dev_iotlb_pasid(sdev->iommu, sdev->sid, info->pfsid,
| ^~~~~~~~~~~~~~~~~~~~~~~~
drivers/iommu/intel/iommu.h:1046:6: note: declared here
1046 | void qi_flush_dev_iotlb_pasid(struct intel_iommu *iommu,
| ^~~~~~~~~~~~~~~~~~~~~~~~
drivers/iommu/intel/svm.c:232:25: error: too many arguments to function 'quirk_extra_dev_tlb_flush'
232 | quirk_extra_dev_tlb_flush(info, 0, 64 - VTD_PAGE_SHIFT,
| ^~~~~~~~~~~~~~~~~~~~~~~~~ drivers/iommu/intel/iommu.h:1049:6: note: declared here
1049 | void quirk_extra_dev_tlb_flush(struct device_domain_info *info, u32 pasid,
| ^~~~~~~~~~~~~~~~~~~~~~~~~
vim +/qi_flush_dev_iotlb_pasid +229 drivers/iommu/intel/svm.c
2f26e0a9c9860d drivers/iommu/intel-svm.c David Woodhouse 2015-09-09 217
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 218 static void intel_flush_svm_all(struct intel_svm *svm)
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 219 {
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 220 struct device_domain_info *info;
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 221 struct intel_svm_dev *sdev;
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 222
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 223 rcu_read_lock();
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 224 list_for_each_entry_rcu(sdev, &svm->devs, list) {
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 225 info = dev_iommu_priv_get(sdev->dev);
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 226
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 227 qi_flush_piotlb(sdev->iommu, sdev->did, svm->pasid, 0, -1UL, 0);
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 228 if (info->ats_enabled) {
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 @229 qi_flush_dev_iotlb_pasid(sdev->iommu, sdev->sid, info->pfsid,
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 230 svm->pasid, sdev->qdep,
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 231 0, 64 - VTD_PAGE_SHIFT);
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 232 quirk_extra_dev_tlb_flush(info, 0, 64 - VTD_PAGE_SHIFT,
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 233 svm->pasid, sdev->qdep);
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 234 }
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 235 }
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 236 rcu_read_unlock();
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 237 }
e7ad6c2a4b1aa7 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 238 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH v11 3/5] iommu/vt-d: simplify parameters of qi_submit_sync() ATS invalidation callers
In reply to Ethan Zhaokernel test robot noticed the following build errors:
Hi Ethan,
kernel test robot noticed the following build errors:
[auto build test ERROR on pci/next]
[also build test ERROR on pci/for-linus linus/master v6.8-rc1 next-20240125] [cannot apply to joro-iommu/next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Ethan-Zhao/PCI-make-pci_dev_is_disconnected-helper-public-for-other-drivers/20240126-094305 base: https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git next patch link: https://lore.kernel.org/r/20240126014002.481294-4-haifeng.zhao%40linux.intel.com patch subject: [PATCH v11 3/5] iommu/vt-d: simplify parameters of qi_submit_sync() ATS invalidation callers config: x86_64-randconfig-005-20240128 (https://download.01.org/0day-ci/archive/20240128/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240128/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
drivers/iommu/intel/svm.c: In function 'intel_flush_svm_all':
drivers/iommu/intel/svm.c:229:67: warning: passing argument 2 of 'qi_flush_dev_iotlb_pasid' makes pointer from integer without a cast [-Wint-conversion]
229 | qi_flush_dev_iotlb_pasid(sdev->iommu, sdev->sid, info->pfsid,
| ~~~~^~~~~
| |
| u16 {aka short unsigned int}
In file included from drivers/iommu/intel/svm.c:22:
drivers/iommu/intel/iommu.h:1047:58: note: expected 'struct device_domain_info *' but argument is of type 'u16' {aka 'short unsigned int'}
1047 | struct device_domain_info *info, u64 addr,
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
>> drivers/iommu/intel/svm.c:229:25: error: too many arguments to function 'qi_flush_dev_iotlb_pasid'
229 | qi_flush_dev_iotlb_pasid(sdev->iommu, sdev->sid, info->pfsid,
| ^~~~~~~~~~~~~~~~~~~~~~~~
drivers/iommu/intel/iommu.h:1046:6: note: declared here
1046 | void qi_flush_dev_iotlb_pasid(struct intel_iommu *iommu,
| ^~~~~~~~~~~~~~~~~~~~~~~~
>> drivers/iommu/intel/svm.c:232:25: error: too many arguments to function 'quirk_extra_dev_tlb_flush'
232 | quirk_extra_dev_tlb_flush(info, 0, 64 - VTD_PAGE_SHIFT,
| ^~~~~~~~~~~~~~~~~~~~~~~~~ drivers/iommu/intel/iommu.h:1049:6: note: declared here
1049 | void quirk_extra_dev_tlb_flush(struct device_domain_info *info, u32 pasid,
| ^~~~~~~~~~~~~~~~~~~~~~~~~
vim +/qi_flush_dev_iotlb_pasid +229 drivers/iommu/intel/svm.c
2f26e0a9c9860db drivers/iommu/intel-svm.c David Woodhouse 2015-09-09 217
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 218 static void intel_flush_svm_all(struct intel_svm *svm)
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 219 {
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 220 struct device_domain_info *info;
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 221 struct intel_svm_dev *sdev;
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 222
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 223 rcu_read_lock();
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 224 list_for_each_entry_rcu(sdev, &svm->devs, list) {
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 225 info = dev_iommu_priv_get(sdev->dev);
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 226
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 227 qi_flush_piotlb(sdev->iommu, sdev->did, svm->pasid, 0, -1UL, 0);
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 228 if (info->ats_enabled) {
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 @229 qi_flush_dev_iotlb_pasid(sdev->iommu, sdev->sid, info->pfsid,
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 230 svm->pasid, sdev->qdep,
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 231 0, 64 - VTD_PAGE_SHIFT);
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 @232 quirk_extra_dev_tlb_flush(info, 0, 64 - VTD_PAGE_SHIFT,
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 233 svm->pasid, sdev->qdep);
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 234 }
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 235 }
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 236 rcu_read_unlock();
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 237 }
e7ad6c2a4b1aa71 drivers/iommu/intel/svm.c Lu Baolu 2023-11-22 238 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH v11 4/5 iommu/vt-d: pass pdev parameter for qi_check_fault() and refactor callers
In reply to Ethan Zhaoto check state of ATS capable pci device in qi_check_fault() for surprise removal case, we need to pass the target pci device of ATS invalidation request to qi_check_fault(). if pdev is valid, means current request is for ATS invalidation, vice vesa.
no function change.
--- drivers/iommu/intel/dmar.c | 31 ++++++++++++++++++----------- drivers/iommu/intel/iommu.h | 3 ++- drivers/iommu/intel/irq_remapping.c | 2 +- drivers/iommu/intel/pasid.c | 2 +- drivers/iommu/intel/svm.c | 6 +++--- 5 files changed, 26 insertions(+), 18 deletions(-)
drivers/iommu/intel/dmar.c +19 -12
@@ -1267,7 +1267,8 @@ static void qi_dump_fault(struct intel_iommu *iommu, u32 fault) (unsigned long long)desc->qw1); } -static int qi_check_fault (struct intel_iommu *iommu,int index,int wait_index) +static int qi_check_fault (struct intel_iommu *iommu,int index,int wait_index, +struct pci_dev *pdev) {u32 fault;int head, tail;
@@ -1344,7 +1345,8 @@ static int qi_check_fault(struct intel_iommu *iommu, int index, int wait_index)* can be part of the submission but it will not be polled for completion. */ int qi_submit_sync (struct intel_iommu *iommu,struct qi_desc *desc, -unsigned int count,unsigned long options) +unsigned int count,unsigned long options, +struct pci_dev *pdev) {struct q_inval *qi = iommu->qi;s64 devtlb_start_ktime =0 ;
@@ -1430,7 +1432,7 @@ int qi_submit_sync(struct intel_iommu *iommu, struct qi_desc *desc,* a deadlock where the interrupt context can wait indefinitely * for free slots in the queue. */ - rc =qi_check_fault (iommu, index, wait_index); + rc =qi_check_fault (iommu, index, wait_index, pdev);if (rc)break ;
@@ -1476,7 +1478,7 @@ void qi_global_iec(struct intel_iommu *iommu) desc.qw3 =0 ;/* should never fail */ -qi_submit_sync (iommu, &desc,1 ,0 ); +qi_submit_sync (iommu, &desc,1 ,0 ,NULL ); } void qi_flush_context(struct intel_iommu *iommu, u16 did, u16 sid, u8 fm,
@@ -1490,7 +1492,7 @@ void qi_flush_context(struct intel_iommu *iommu, u16 did, u16 sid, u8 fm, desc.qw2 =0 ; desc.qw3 =0 ; -qi_submit_sync (iommu, &desc,1 ,0 ); +qi_submit_sync (iommu, &desc,1 ,0 ,NULL ); } void qi_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,
@@ -1514,23 +1516,27 @@ void qi_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr, desc.qw2 =0 ; desc.qw3 =0 ; -qi_submit_sync (iommu, &desc,1 ,0 ); +qi_submit_sync (iommu, &desc,1 ,0 ,NULL ); }void qi_flush_dev_iotlb (struct intel_iommu *iommu,struct device_domain_info *info,u64 addr,unsigned int mask) { +struct pci_dev *pdev =NULL ;u16 sid, qdep, pfsid;struct qi_desc desc;if (!info || !info->ats_enabled)return ; +if (info->dev || !dev_is_pci (info->dev)) +return ; + + pdev =to_pci_dev (info->dev); sid = info->bus <<8 | info->devfn; qdep = info->ats_qdep; pfsid = info->pfsid; -/* * VT-d spec, section 4.3: *
@@ -1554,7 +1560,7 @@ void qi_flush_dev_iotlb(struct intel_iommu *iommu, desc.qw2 =0 ; desc.qw3 =0 ; -qi_submit_sync (iommu, &desc,1 ,0 ); +qi_submit_sync (iommu, &desc,1 ,0 , pdev); }/* PASID-based IOTLB invalidation */
@@ -1595,7 +1601,7 @@ void qi_flush_piotlb(struct intel_iommu *iommu, u16 did, u32 pasid, u64 addr,QI_EIOTLB_AM (mask); } -qi_submit_sync (iommu, &desc,1 ,0 ); +qi_submit_sync (iommu, &desc,1 ,0 ,NULL ); }/* PASID-based device IOTLB Invalidate */
@@ -1605,15 +1611,16 @@ void qi_flush_dev_iotlb_pasid(struct intel_iommu *iommu, {unsigned long mask =1UL << (VTD_PAGE_SHIFT + size_order -1 );struct qi_desc desc = {.qw1 =0 , .qw2 =0 , .qw3 =0 }; +struct pci_dev *pdev =NULL ;u16 sid, qdep, pfsid;if (!info || !dev_is_pci (info->dev))return ; + pdev =to_pci_dev (info->dev); sid = info->bus <<8 | info->devfn; qdep = info->ats_qdep; pfsid = info->pfsid; -/* * VT-d spec, section 4.3: *
@@ -1657,7 +1664,7 @@ void qi_flush_dev_iotlb_pasid(struct intel_iommu *iommu, desc.qw1 |=QI_DEV_EIOTLB_SIZE ; } -qi_submit_sync (iommu, &desc,1 ,0 ); +qi_submit_sync (iommu, &desc,1 ,0 , pdev); } void qi_flush_pasid_cache(struct intel_iommu *iommu, u16 did,
@@ -1667,7 +1674,7 @@ void qi_flush_pasid_cache(struct intel_iommu *iommu, u16 did, desc.qw0 =QI_PC_PASID (pasid) |QI_PC_DID (did) |QI_PC_GRAN (granu) |QI_PC_TYPE ; -qi_submit_sync (iommu, &desc,1 ,0 ); +qi_submit_sync (iommu, &desc,1 ,0 ,NULL ); }/*
drivers/iommu/intel/iommu.h +2 -1
@@ -1052,7 +1052,8 @@ void qi_flush_pasid_cache(struct intel_iommu *iommu, u16 did, u64 granu, u32 pasid);int qi_submit_sync (struct intel_iommu *iommu,struct qi_desc *desc, -unsigned int count,unsigned long options); +unsigned int count,unsigned long options, +struct pci_dev *pdev);/* * Options used in qi_submit_sync: * QI_OPT_WAIT_DRAIN - Wait for PRQ drain completion, spec 6.5.2.8.
drivers/iommu/intel/irq_remapping.c +1 -1
@@ -153,7 +153,7 @@ static int qi_flush_iec(struct intel_iommu *iommu, int index, int mask) desc.qw2 =0 ; desc.qw3 =0 ; -return qi_submit_sync (iommu, &desc,1 ,0 ); +return qi_submit_sync (iommu, &desc,1 ,0 ,NULL ); } static int modify_irte(struct irq_2_iommu *irq_iommu,
drivers/iommu/intel/pasid.c +1 -1
@@ -200,7 +200,7 @@ pasid_cache_invalidation_with_pasid(struct intel_iommu *iommu, desc.qw2 =0 ; desc.qw3 =0 ; -qi_submit_sync (iommu, &desc,1 ,0 ); +qi_submit_sync (iommu, &desc,1 ,0 ,NULL ); } static void
drivers/iommu/intel/svm.c +3 -3
@@ -538,7 +538,7 @@ void intel_drain_pasid_prq(struct device *dev, u32 pasid)QI_DEV_IOTLB_PFSID (info->pfsid); qi_retry:reinit_completion (&iommu->prq_complete); -qi_submit_sync (iommu, desc,3 ,QI_OPT_WAIT_DRAIN ); +qi_submit_sync (iommu, desc,3 ,QI_OPT_WAIT_DRAIN ,NULL );if (readl (iommu->reg +DMAR_PRS_REG ) &DMA_PRS_PRO ) {wait_for_completion (&iommu->prq_complete); goto qi_retry;
@@ -641,7 +641,7 @@ static void handle_bad_prq_event(struct intel_iommu *iommu, desc.qw3 =0 ; } -qi_submit_sync (iommu, &desc,1 ,0 ); +qi_submit_sync (iommu, &desc,1 ,0 ,NULL ); }static irqreturn_t prq_event_thread (int irq,void *d)
@@ -797,7 +797,7 @@ int intel_svm_page_response(struct device *dev, ktime_to_ns(ktime_get()) - prm->private_data[0]); } -qi_submit_sync (iommu, &desc,1 ,0 ); +qi_submit_sync (iommu, &desc,1 ,0 ,NULL ); } out:return ret;
Signature · 1 line
2.31.1
PATCH v11 5/5 iommu/vt-d: improve ITE fault handling if target device isn't present
In reply to Ethan ZhaoBecause surprise removal could happen anytime, e.g. user could request safe removal to EP(endpoint device) via sysfs and brings its link down to do surprise removal cocurrently. such aggressive cases would cause ATS invalidation request issued to non-existence target device, then deadly loop to retry that request after ITE fault triggered in interrupt context. this patch aims to optimize the ITE handling by checking the target device presence state to avoid retrying the timeout request blindly, thus avoid hard lockup or system hang.
--- drivers/iommu/intel/dmar.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
drivers/iommu/intel/dmar.c +18 -0
@@ -1272,6 +1272,7 @@ static int qi_check_fault(struct intel_iommu *iommu, int index, int wait_index, {u32 fault;int head, tail; +u64 iqe_err, ite_sid;struct q_inval *qi = iommu->qi;int shift =qi_shift (iommu);
@@ -1316,6 +1317,13 @@ static int qi_check_fault(struct intel_iommu *iommu, int index, int wait_index, tail =readl (iommu->reg +DMAR_IQT_REG ); tail = ((tail >> shift) -1 +QI_LENGTH ) %QI_LENGTH ; +/* +* SID field is valid only when the ITE field is Set in FSTS_REG +* see Intel VT-d spec r4.1, section 11.4.9.9 +*/ + iqe_err =dmar_readq (iommu->reg +DMAR_IQER_REG ); + ite_sid =DMAR_IQER_REG_ITESID (iqe_err); +writel (DMA_FSTS_ITE , iommu->reg +DMAR_FSTS_REG );pr_info ("Invalidation Time-out Error (ITE) cleared\n" );
@@ -1325,6 +1333,16 @@ static int qi_check_fault(struct intel_iommu *iommu, int index, int wait_index, head = (head -2 +QI_LENGTH ) %QI_LENGTH ; }while (head != tail); +/* +* If got ITE, we need to check if the sid of ITE is the same as +* current ATS invalidation target device, if yes, don't try this +* request anymore if the target device isn't present. +* 0 value of ite_sid means old VT-d device, no ite_sid value. +*/ +if (pdev && ite_sid && !pci_device_is_present (pdev) && + ite_sid ==pci_dev_id (pci_physfn (pdev))) +return -ETIMEDOUT ; +if (qi->desc_status[wait_index] ==QI_ABORT )return -EAGAIN ; }
Signature · 1 line
2.31.1