iommu/dma: s390 DMA API conversion and optimized IOTLB flushing
Subsystems (18)
Niklas Schnelle
maintainerv13 cover iommu/dma: s390 DMA API conversion and optimized IOTLB flushing
Hi All,
This patch series converts s390's PCI support from its platform specific DMA API implementation in arch/s390/pci/pci_dma.c to the common DMA IOMMU layer. The conversion itself is done in patches 3-4 with patch 2 providing the final necessary IOMMU driver improvement to handle s390's special IOTLB flush out-of-resource indication in virtualized environments. The conversion itself only touches the s390 IOMMU driver and s390 arch code moving over remaining functions from the s390 DMA API implementation. No changes to common code are necessary.
After patch 4 the basic conversion is done and on our partitioning machine hypervisor LPAR performance matches the previous implementation. When running under z/VM or KVM however, performance plummets to about half of the existing code due to a much higher rate of IOTLB flushes for unmapped pages. Due to the hypervisors use of IOTLB flushes to synchronize their shadow tables these are very expensive and minimizing them is key for regaining the performance loss.
To this end patches 5-6 add a new, single queue, IOTLB flushing scheme as an alternative to the existing per-CPU flush queues. Introducing an alternative scheme was suggested by Robin Murphy[1]. The single queue mode is introduced in patch 4 together with a new .shadow_on_flush flag bit in struct dev_iommu. This allows IOMMU drivers to indicate that their IOTLB flushes do the extra work of shadowing. This then lets the dma-iommu code use a single queue.
Then patch 6 enables variable queue sizes using power of 2 values and shift/mask to keep performance as close to the fixed size queue code as possible. A larger queue size and timeout is used by dma-iommu when shadow_on_flush is set. This same scheme may also be used by other IOMMU drivers with similar requirements. Particularly virtio-iommu may be a candidate.
I tested this code on s390x with LPAR, z/VM and KVM hypervisors on an AMD Ryzen x86 system with native IOMMU and a guest with a modified virtio-iommu[4] that set .shadow_on_flush = true.
This code is also available in the b4/dma_iommu topic branch of my git.kernel.org repository[3] with tags matching the version sent.
NOTE: Due to the large drop in performance I think we should not merge the DMA API conversion (patch 4) until we have a more suited IOVA flushing scheme with similar improvements as the proposed changes.
NOTE: With the DMA API converted mlx5_core is broken on s390x since commit 06cd555f73ca ("net/mlx5: split mlx5_cmd_init() to probe and reload routines"). A fix for the breakage has been proposed[5].
Best regards,
Niklas
[0] https://lore.kernel.org/linux-iommu/[email protected]/
[1] https://lore.kernel.org/linux-iommu/[email protected]/
[2] https://lore.kernel.org/linux-iommu/[email protected]/
[3] https://git.kernel.org/pub/scm/linux/kernel/git/niks/linux.git/
[4] https://lore.kernel.org/lkml/[email protected]/
[5] https://lore.kernel.org/netdev/[email protected]/
---
Changes in v13:
- Rebase on iommu/core which requires removing the platform domain
handling with.
- Moved setting iommu_dma_forcedac = true to s390_iommu_init() so it
happens before the call to iommu_dma_set_pci_32bit_workaround()
- Link to v12: https://lore.kernel.org/r/[email protected]
Changes in v12:
- Rebased on v6.5-rc7
- Changed queue type flag to an enum
- Incorporated feedback from Robin Murphy
- Set options centrally and only once in iommu_dma_init_domain() with
new helper iommu_dma_init_options()
- Do not reset options of failing to init FQ
- Fixed rebase mishap that partially rolled back patch 2
- Simplified patch 4 by simply no claiming the deferred flush
capability for ISM
- Inlined and removed fq_flush_percpu()
- Changed vzalloc() to vmalloc() for queue
- Added Acked-by's from Robin Murphy
- Link to v11: https://lore.kernel.org/r/[email protected]
Changes in v11:
- Rebased on v6.5-rc2
- Added patch to force IOMMU_DOMAIN_DMA on s390 specific ISM devices
- Dropped the patch to properly set DMA mask on ISM devices which went upstream separately.
- s390 IOMMU driver now uses IOMMU_CAP_DEFERRED_FLUSH to enable DMA-FQ leaving no uses of IOMMU_DOMAIN_DMA_FQ in the driver.
- Link to v10: https://lore.kernel.org/r/[email protected]
Changes in v10:
- Rebased on v6.4-rc3
- Removed the .tune_dma_iommu() op in favor of a .shadow_on_flush flag
in struct dev_iommu which then let's the dma-iommu choose a single
queue and larger timeouts and IOVA counts. This leaves the dma-iommu
with full responsibility for the settings.
- The above change affects patches 5 and 6 and lead to a new subject for patch 6 since the flush queue size and timeout is no longer driver controlled
- Link to v9: https://lore.kernel.org/r/[email protected]
Changes in v9:
- Rebased on v6.4-rc2
- Re-ordered iommu_group_store_type() to allow passing the device to iommu_dma_init_fq()
- Link to v8: https://lore.kernel.org/r/[email protected]
---
Niklas Schnelle (6):
iommu: Allow .iotlb_sync_map to fail and handle s390's -ENOMEM return s390/pci: prepare is_passed_through() for dma-iommu s390/pci: Use dma-iommu layer
iommu/s390: Disable deferred flush for ISM devices iommu/dma: Allow a single FQ in addition to per-CPU FQs iommu/dma: Use a large flush queue and timeout for shadow_on_flush
Documentation/admin-guide/kernel-parameters.txt | 9 +- arch/s390/include/asm/pci.h | 7 - arch/s390/include/asm/pci_clp.h | 3 + arch/s390/include/asm/pci_dma.h | 119 +--- arch/s390/pci/Makefile | 2 +- arch/s390/pci/pci.c | 22 +- arch/s390/pci/pci_bus.c | 5 - arch/s390/pci/pci_debug.c | 12 +- arch/s390/pci/pci_dma.c | 735 ------------------------ arch/s390/pci/pci_event.c | 17 +- arch/s390/pci/pci_sysfs.c | 19 +- drivers/iommu/Kconfig | 4 +- drivers/iommu/amd/iommu.c | 5 +- drivers/iommu/apple-dart.c | 5 +- drivers/iommu/dma-iommu.c | 200 +++++-- drivers/iommu/intel/iommu.c | 5 +- drivers/iommu/iommu.c | 20 +- drivers/iommu/msm_iommu.c | 5 +- drivers/iommu/mtk_iommu.c | 5 +- drivers/iommu/s390-iommu.c | 434 ++++++++++++-- drivers/iommu/sprd-iommu.c | 5 +- drivers/iommu/sun50i-iommu.c | 6 +- include/linux/iommu.h | 6 +- 23 files changed, 634 insertions(+), 1016 deletions(-)
---
base-commit: ccb76c5751634541bfd2222374f46e1dea83d525
change-id: 20230310-dma_iommu-5e048c538647
Best regards,
Signature · 8 lines
Niklas Schnelle Linux on Z Development IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Gregor Pillen Geschäftsführung: David Faller Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM Data Privacy Statement - https://www.ibm.com/privacy
Niklas Schnelle
maintainerPATCH v13 1/6 iommu: Allow .iotlb_sync_map to fail and handle s390's -ENOMEM return
On s390 when using a paging hypervisor, .iotlb_sync_map is used to sync mappings by letting the hypervisor inspect the synced IOVA range and updating a shadow table. This however means that .iotlb_sync_map can fail as the hypervisor may run out of resources while doing the sync. This can be due to the hypervisor being unable to pin guest pages, due to a limit on mapped addresses such as vfio_iommu_type1.dma_entry_limit or lack of other resources. Either way such a failure to sync a mapping should result in a DMA_MAPPING_ERROR.
Now especially when running with batched IOTLB flushes for unmap it may be that some IOVAs have already been invalidated but not yet synced via .iotlb_sync_map. Thus if the hypervisor indicates running out of resources, first do a global flush allowing the hypervisor to free resources associated with these mappings as well a retry creating the new mappings and only if that also fails report this error to callers.
--- drivers/iommu/amd/iommu.c | 5 +++-- drivers/iommu/apple-dart.c | 5 +++-- drivers/iommu/intel/iommu.c | 5 +++-- drivers/iommu/iommu.c | 20 ++++++++++++++++---- drivers/iommu/msm_iommu.c | 5 +++-- drivers/iommu/mtk_iommu.c | 5 +++-- drivers/iommu/s390-iommu.c | 29 +++++++++++++++++++++++------ drivers/iommu/sprd-iommu.c | 5 +++-- drivers/iommu/sun50i-iommu.c | 6 ++++-- include/linux/iommu.h | 4 ++-- 10 files changed, 63 insertions(+), 26 deletions(-)
drivers/iommu/amd/iommu.c +3 -2
@@ -2233,14 +2233,15 @@ static int amd_iommu_attach_device(struct iommu_domain *dom,return ret; } -static void amd_iommu_iotlb_sync_map (struct iommu_domain *dom, -unsigned long iova,size_t size) +static int amd_iommu_iotlb_sync_map (struct iommu_domain *dom, +unsigned long iova,size_t size) {struct protection_domain *domain =to_pdomain (dom);struct io_pgtable_ops *ops = &domain->iop.iop.ops;if (ops->map_pages)domain_flush_np_cache (domain, iova, size); +return 0 ; } static int amd_iommu_map_pages(struct iommu_domain *dom, unsigned long iova,
drivers/iommu/apple-dart.c +3 -2
@@ -506,10 +506,11 @@ static void apple_dart_iotlb_sync(struct iommu_domain *domain,apple_dart_domain_flush_tlb (to_dart_domain (domain)); } -static void apple_dart_iotlb_sync_map (struct iommu_domain *domain, -unsigned long iova,size_t size) +static int apple_dart_iotlb_sync_map (struct iommu_domain *domain, +unsigned long iova,size_t size) {apple_dart_domain_flush_tlb (to_dart_domain (domain)); +return 0 ; } static phys_addr_t apple_dart_iova_to_phys(struct iommu_domain *domain,
drivers/iommu/intel/iommu.c +3 -2
@@ -4678,8 +4678,8 @@ static bool risky_device(struct pci_dev *pdev)return false; } -static void intel_iommu_iotlb_sync_map (struct iommu_domain *domain, -unsigned long iova,size_t size) +static int intel_iommu_iotlb_sync_map (struct iommu_domain *domain, +unsigned long iova,size_t size) {struct dmar_domain *dmar_domain =to_dmar_domain (domain);unsigned long pages =aligned_nrpages (iova, size);
@@ -4689,6 +4689,7 @@ static void intel_iommu_iotlb_sync_map(struct iommu_domain *domain,xa_for_each (&dmar_domain->iommu_array, i, info)__mapping_notify_one (info->iommu, dmar_domain, pfn, pages); +return 0 ; }static void intel_iommu_remove_dev_pasid (struct device *dev,ioasid_t pasid)
drivers/iommu/iommu.c +16 -4
@@ -2585,8 +2585,17 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova,return -EINVAL ; ret =__iommu_map (domain, iova, paddr, size, prot, gfp); -if (ret ==0 && ops->iotlb_sync_map) - ops->iotlb_sync_map (domain, iova, size); +if (ret ==0 && ops->iotlb_sync_map) { + ret = ops->iotlb_sync_map (domain, iova, size); +if (ret) + goto out_err; + } + +return ret; + +out_err: +/* undo mappings already done */ +iommu_unmap (domain, iova, size);return ret; }
@@ -2714,8 +2723,11 @@ ssize_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, sg =sg_next (sg); } -if (ops->iotlb_sync_map) - ops->iotlb_sync_map (domain, iova, mapped); +if (ops->iotlb_sync_map) { + ret = ops->iotlb_sync_map (domain, iova, mapped); +if (ret) + goto out_err; + }return mapped; out_err:
drivers/iommu/msm_iommu.c +3 -2
@@ -498,12 +498,13 @@ static int msm_iommu_map(struct iommu_domain *domain, unsigned long iova,return ret; } -static void msm_iommu_sync_map (struct iommu_domain *domain,unsigned long iova, -size_t size) +static int msm_iommu_sync_map (struct iommu_domain *domain,unsigned long iova, +size_t size) {struct msm_priv *priv =to_msm_priv (domain);__flush_iotlb_range (iova, size,SZ_4K , false, priv); +return 0 ; } static size_t msm_iommu_unmap(struct iommu_domain *domain, unsigned long iova,
drivers/iommu/mtk_iommu.c +3 -2
@@ -837,12 +837,13 @@ static void mtk_iommu_iotlb_sync(struct iommu_domain *domain,mtk_iommu_tlb_flush_range_sync (gather->start, length, dom->bank); } -static void mtk_iommu_sync_map (struct iommu_domain *domain,unsigned long iova, -size_t size) +static int mtk_iommu_sync_map (struct iommu_domain *domain,unsigned long iova, +size_t size) {struct mtk_iommu_domain *dom =to_mtk_domain (domain);mtk_iommu_tlb_flush_range_sync (iova, size, dom->bank); +return 0 ; } static phys_addr_t mtk_iommu_iova_to_phys(struct iommu_domain *domain,
drivers/iommu/s390-iommu.c +23 -6
@@ -219,6 +219,12 @@ static void s390_iommu_release_device(struct device *dev)__s390_iommu_detach_device (zdev); } +static int zpci_refresh_all (struct zpci_dev *zdev) +{ +return zpci_refresh_trans ((u64 )zdev->fh <<32 , zdev->start_dma, + zdev->end_dma - zdev->start_dma +1 ); +} +static void s390_iommu_flush_iotlb_all (struct iommu_domain *domain) {struct s390_domain *s390_domain =to_s390_domain (domain);
@@ -226,8 +232,7 @@ static void s390_iommu_flush_iotlb_all(struct iommu_domain *domain)rcu_read_lock ();list_for_each_entry_rcu (zdev, &s390_domain->devices, iommu_list) { -zpci_refresh_trans ((u64 )zdev->fh <<32 , zdev->start_dma, - zdev->end_dma - zdev->start_dma +1 ); +zpci_refresh_all (zdev); }rcu_read_unlock (); }
@@ -251,20 +256,32 @@ static void s390_iommu_iotlb_sync(struct iommu_domain *domain,rcu_read_unlock (); } -static void s390_iommu_iotlb_sync_map (struct iommu_domain *domain, -unsigned long iova,size_t size) +static int s390_iommu_iotlb_sync_map (struct iommu_domain *domain, +unsigned long iova,size_t size) {struct s390_domain *s390_domain =to_s390_domain (domain);struct zpci_dev *zdev; +int ret =0 ;rcu_read_lock ();list_for_each_entry_rcu (zdev, &s390_domain->devices, iommu_list) {if (!zdev->tlb_refresh)continue ; -zpci_refresh_trans ((u64 )zdev->fh <<32 , - iova, size); + ret =zpci_refresh_trans ((u64 )zdev->fh <<32 , + iova, size); +/* +* let the hypervisor discover invalidated entries +* allowing it to free IOVAs and unpin pages +*/ +if (ret == -ENOMEM ) { + ret =zpci_refresh_all (zdev); +if (ret) +break ; + } }rcu_read_unlock (); + +return ret; } static int s390_iommu_validate_trans(struct s390_domain *s390_domain,
drivers/iommu/sprd-iommu.c +3 -2
@@ -341,8 +341,8 @@ static size_t sprd_iommu_unmap(struct iommu_domain *domain, unsigned long iova,return size; } -static void sprd_iommu_sync_map (struct iommu_domain *domain, -unsigned long iova,size_t size) +static int sprd_iommu_sync_map (struct iommu_domain *domain, +unsigned long iova,size_t size) {struct sprd_iommu_domain *dom =to_sprd_domain (domain);unsigned int reg;
@@ -354,6 +354,7 @@ static void sprd_iommu_sync_map(struct iommu_domain *domain,/* clear IOMMU TLB buffer after page table updated */ sprd_iommu_write (dom->sdev, reg,0xffffffff ); +return 0 ; } static void sprd_iommu_sync(struct iommu_domain *domain,
drivers/iommu/sun50i-iommu.c +4 -2
@@ -401,8 +401,8 @@ static void sun50i_iommu_flush_iotlb_all(struct iommu_domain *domain)spin_unlock_irqrestore (&iommu->iommu_lock, flags); } -static void sun50i_iommu_iotlb_sync_map (struct iommu_domain *domain, -unsigned long iova,size_t size) +static int sun50i_iommu_iotlb_sync_map (struct iommu_domain *domain, +unsigned long iova,size_t size) {struct sun50i_iommu_domain *sun50i_domain =to_sun50i_domain (domain);struct sun50i_iommu *iommu = sun50i_domain->iommu;
@@ -411,6 +411,8 @@ static void sun50i_iommu_iotlb_sync_map(struct iommu_domain *domain,spin_lock_irqsave (&iommu->iommu_lock, flags);sun50i_iommu_zap_range (iommu, iova, size);spin_unlock_irqrestore (&iommu->iommu_lock, flags); + +return 0 ; } static void sun50i_iommu_iotlb_sync(struct iommu_domain *domain,
include/linux/iommu.h +2 -2
@@ -350,8 +350,8 @@ struct iommu_domain_ops { struct iommu_iotlb_gather *iotlb_gather);void (*flush_iotlb_all)(struct iommu_domain *domain); -void (*iotlb_sync_map)(struct iommu_domain *domain,unsigned long iova, -size_t size); +int (*iotlb_sync_map)(struct iommu_domain *domain,unsigned long iova, +size_t size);void (*iotlb_sync)(struct iommu_domain *domain,struct iommu_iotlb_gather *iotlb_gather);
Signature · 1 line
2.39.2
Niklas Schnelle
maintainerPATCH v13 2/6 s390/pci: prepare is_passed_through() for dma-iommu
In reply to Niklas SchnelleWith the IOMMU always controlled through the IOMMU driver testing for zdev->s390_domain is not a valid indication of the device being passed-through. Instead test if zdev->kzdev is set.
--- arch/s390/pci/pci_event.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
arch/s390/pci/pci_event.c +11 -4
@@ -59,9 +59,16 @@ static inline bool ers_result_indicates_abort(pci_ers_result_t ers_res) } } -static bool is_passed_through (struct zpci_dev *zdev) +static bool is_passed_through (struct pci_dev *pdev) { -return zdev->s390_domain; +struct zpci_dev *zdev =to_zpci (pdev); +bool ret; + +mutex_lock (&zdev->kzdev_lock); + ret = !!zdev->kzdev; +mutex_unlock (&zdev->kzdev_lock); + +return ret; }static bool is_driver_supported (struct pci_driver *driver)
@@ -176,7 +183,7 @@ static pci_ers_result_t zpci_event_attempt_error_recovery(struct pci_dev *pdev) } pdev->error_state = pci_channel_io_frozen; -if (is_passed_through (to_zpci (pdev))) { +if (is_passed_through (pdev)) {pr_info ("%s: Cannot be recovered in the host because it is a pass-through device\n" ,pci_name (pdev)); goto out_unlock;
@@ -239,7 +246,7 @@ static void zpci_event_io_failure(struct pci_dev *pdev, pci_channel_state_t es)* we will inject the error event and let the guest recover the device * itself. */ -if (is_passed_through (to_zpci (pdev))) +if (is_passed_through (pdev)) goto out; driver =to_pci_driver (pdev->dev.driver);if (driver && driver->err_handler && driver->err_handler->error_detected)
Signature · 1 line
2.39.2
Niklas Schnelle
maintainerPATCH v13 3/6 s390/pci: Use dma-iommu layer
In reply to Niklas SchnelleWhile s390 already has a standard IOMMU driver and previous changes have added I/O TLB flushing operations this driver is currently only used for user-space PCI access such as vfio-pci. For the DMA API s390 instead utilizes its own implementation in arch/s390/pci/pci_dma.c which drives the same hardware and shares some code but requires a complex and fragile hand over between DMA API and IOMMU API use of a device and despite code sharing still leads to significant duplication and maintenance effort. Let's utilize the common code DMAP API implementation from drivers/iommu/dma-iommu.c instead allowing us to get rid of arch/s390/pci/pci_dma.c.
--- Documentation/admin-guide/kernel-parameters.txt | 9 +- arch/s390/include/asm/pci.h | 7 - arch/s390/include/asm/pci_clp.h | 3 + arch/s390/include/asm/pci_dma.h | 119 +--- arch/s390/pci/Makefile | 2 +- arch/s390/pci/pci.c | 22 +- arch/s390/pci/pci_bus.c | 5 - arch/s390/pci/pci_debug.c | 12 +- arch/s390/pci/pci_dma.c | 735 ------------------------ arch/s390/pci/pci_event.c | 2 - arch/s390/pci/pci_sysfs.c | 19 +- drivers/iommu/Kconfig | 4 +- drivers/iommu/s390-iommu.c | 400 +++++++++++-- 13 files changed, 401 insertions(+), 938 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +5 -4
@@ -2220,7 +2220,7 @@ forcing Dual Address Cycle for PCI cards supporting greater than 32-bit addressing. - iommu.strict= [ARM64, X86] Configure TLB invalidation behaviour + iommu.strict= [ARM64, X86, S390] Configure TLB invalidation behaviour Format: { "0" | "1" } 0 - Lazy mode. Request that DMA unmap operations use deferred
@@ -5611,9 +5611,10 @@ s390_iommu= [HW,S390] Set s390 IOTLB flushing mode strict - With strict flushing every unmap operation will result in - an IOTLB flush. Default is lazy flushing before reuse, - which is faster. + With strict flushing every unmap operation will result + in an IOTLB flush. Default is lazy flushing before + reuse, which is faster. Deprecated, equivalent to + iommu.strict=1. s390_iommu_aperture= [KNL,S390] Specifies the size of the per device DMA address space
arch/s390/include/asm/pci.h +0 -7
@@ -159,13 +159,6 @@ struct zpci_dev {unsigned long *dma_table;int tlb_refresh; -spinlock_t iommu_bitmap_lock; -unsigned long *iommu_bitmap; -unsigned long *lazy_bitmap; -unsigned long iommu_size; -unsigned long iommu_pages; -unsigned int next_bit; -struct iommu_device iommu_dev;/* IOMMU core handle */ char res_name[16 ];
arch/s390/include/asm/pci_clp.h +3 -0
@@ -50,6 +50,9 @@ struct clp_fh_list_entry {#define CLP_UTIL_STR_LEN 64#define CLP_PFIP_NR_SEGMENTS 4 +/* PCI function type numbers */ +#define PCI_FUNC_TYPE_ISM 0x5/* ISM device */ +extern bool zpci_unique_uid;struct clp_rsp_slpc_pci {
arch/s390/include/asm/pci_dma.h +9 -110
126 lines not on this page · raw message
arch/s390/pci/Makefile +1 -1
@@ -3,7 +3,7 @@ # Makefile for the s390 PCI subsystem. # -obj-$(CONFIG_PCI) += pci.o pci_irq.o pci_dma.o pci_clp.o pci_sysfs.o \ +obj-$(CONFIG_PCI) += pci.o pci_irq.o pci_clp.o pci_sysfs.o \ pci_event.o pci_debug.o pci_insn.o pci_mmio.o \ pci_bus.o pci_kvm_hook.o obj-$(CONFIG_PCI_IOV) += pci_iov.o
arch/s390/pci/pci.c +5 -17
@@ -124,7 +124,11 @@ int zpci_register_ioat(struct zpci_dev *zdev, u8 dmaas,WARN_ON_ONCE (iota &0x3fff ); fib.pba = base; - fib.pal = limit; +/* Work around off by one in ISM virt device */ +if (zdev->pft ==PCI_FUNC_TYPE_ISM && limit > base) + fib.pal = limit + (1 <<12 ); +else + fib.pal = limit; fib.iota = iota |ZPCI_IOTA_RTTO_FLAG ; fib.gd = zdev->gisa; cc =zpci_mod_fc (req, &fib, status);
@@ -582,7 +586,6 @@ int pcibios_device_add(struct pci_dev *pdev) pdev->no_vf_scan =1 ; pdev->dev.groups = zpci_attr_groups; - pdev->dev.dma_ops = &s390_pci_dma_ops;zpci_map_resources (pdev);for (i =0 ; i <PCI_STD_NUM_BARS ; i++) {
@@ -756,8 +759,6 @@ int zpci_hot_reset_device(struct zpci_dev *zdev)if (zdev->dma_table) rc =zpci_register_ioat (zdev,0 , zdev->start_dma, zdev->end_dma,virt_to_phys (zdev->dma_table), &status); -else - rc =zpci_dma_init_device (zdev);if (rc) {zpci_disable_device (zdev);return rc;
@@ -865,11 +866,6 @@ int zpci_deconfigure_device(struct zpci_dev *zdev)if (zdev->zbus->bus)zpci_bus_remove_device (zdev, false); -if (zdev->dma_table) { - rc =zpci_dma_exit_device (zdev); -if (rc) -return rc; - }if (zdev_enabled (zdev)) { rc =zpci_disable_device (zdev);if (rc)
@@ -918,8 +914,6 @@ void zpci_release_device(struct kref *kref)if (zdev->zbus->bus)zpci_bus_remove_device (zdev, false); -if (zdev->dma_table) -zpci_dma_exit_device (zdev);if (zdev_enabled (zdev))zpci_disable_device (zdev);
@@ -1109,10 +1103,6 @@ static int __init pci_base_init(void)if (rc) goto out_irq; - rc =zpci_dma_init (); -if (rc) - goto out_dma; - rc =clp_scan_pci_devices ();if (rc) goto out_find;
@@ -1122,8 +1112,6 @@ static int __init pci_base_init(void)return 0 ; out_find: -zpci_dma_exit (); -out_dma:zpci_irq_exit (); out_irq:zpci_mem_exit ();
arch/s390/pci/pci_bus.c +0 -5
@@ -47,11 +47,6 @@ static int zpci_bus_prepare_device(struct zpci_dev *zdev) rc =zpci_enable_device (zdev);if (rc)return rc; - rc =zpci_dma_init_device (zdev); -if (rc) { -zpci_disable_device (zdev); -return rc; - } }if (!zdev->has_resources) {
arch/s390/pci/pci_debug.c +9 -3
@@ -53,9 +53,11 @@ static char *pci_fmt3_names[] = { };static char *pci_sw_names[] = { -"Allocated pages" ,"Mapped pages" ,"Unmapped pages" , +"Global RPCITs" , +"Sync Map RPCITs" , +"Sync RPCITs" , }; static void pci_fmb_show(struct seq_file *m, char *name[], int length,
@@ -69,10 +71,14 @@ static void pci_fmb_show(struct seq_file *m, char *name[], int length,static void pci_sw_counter_show (struct seq_file *m) { -struct zpci_dev *zdev = m->private; -atomic64_t *counter = &zdev->allocated_pages; +struct zpci_iommu_ctrs *ctrs =zpci_get_iommu_ctrs (m->private); +atomic64_t *counter;int i; +if (!ctrs) +return ; + + counter = &ctrs->mapped_pages;for (i =0 ; i <ARRAY_SIZE (pci_sw_names); i++, counter++)seq_printf (m,"%26s:\t%llu\n" , pci_sw_names[i],atomic64_read (counter));
arch/s390/pci/pci_dma.c deleted +0 -735
735 lines not on this page · raw message
arch/s390/pci/pci_event.c +0 -2
@@ -313,8 +313,6 @@ static void zpci_event_hard_deconfigured(struct zpci_dev *zdev, u32 fh)/* Even though the device is already gone we still * need to free zPCI resources as part of the disable. */ -if (zdev->dma_table) -zpci_dma_exit_device (zdev);if (zdev_enabled (zdev))zpci_disable_device (zdev); zdev->state =ZPCI_FN_STATE_STANDBY ;
arch/s390/pci/pci_sysfs.c +8 -11
@@ -56,6 +56,7 @@ static ssize_t recover_store(struct device *dev, struct device_attribute *attr,struct pci_dev *pdev =to_pci_dev (dev);struct zpci_dev *zdev =to_zpci (pdev);int ret =0 ; +u8 status;/* Can't use device_remove_self() here as that would lead us to lock * the pci_rescan_remove_lock while holding the device' kernfs lock.
@@ -82,12 +83,6 @@ static ssize_t recover_store(struct device *dev, struct device_attribute *attr,pci_lock_rescan_remove ();if (pci_dev_is_added (pdev)) {pci_stop_and_remove_bus_device (pdev); -if (zdev->dma_table) { - ret =zpci_dma_exit_device (zdev); -if (ret) - goto out; - } -if (zdev_enabled (zdev)) { ret =zpci_disable_device (zdev);/*
@@ -105,14 +100,16 @@ static ssize_t recover_store(struct device *dev, struct device_attribute *attr, ret =zpci_enable_device (zdev);if (ret) goto out; - ret =zpci_dma_init_device (zdev); -if (ret) { -zpci_disable_device (zdev); - goto out; + +if (zdev->dma_table) { + ret =zpci_register_ioat (zdev,0 , zdev->start_dma, zdev->end_dma, +virt_to_phys (zdev->dma_table), &status); +if (ret) +zpci_disable_device (zdev); } -pci_rescan_bus (zdev->zbus->bus); } out: +pci_rescan_bus (zdev->zbus->bus);pci_unlock_rescan_remove ();if (kn)sysfs_unbreak_active_protection (kn);
drivers/iommu/Kconfig +2 -2
@@ -91,7 +91,7 @@ config IOMMU_DEBUGFS choice prompt "IOMMU default domain type" depends on IOMMU_API - default IOMMU_DEFAULT_DMA_LAZY if X86 || IA64 + default IOMMU_DEFAULT_DMA_LAZY if X86 || IA64 || S390 default IOMMU_DEFAULT_DMA_STRICT help Choose the type of IOMMU domain used to manage DMA API usage by
@@ -146,7 +146,7 @@ config OF_IOMMU # IOMMU-agnostic DMA-mapping layer config IOMMU_DMA - def_bool ARM64 || IA64 || X86 + def_bool ARM64 || IA64 || X86 || S390 select DMA_OPS select IOMMU_API select IOMMU_IOVA
drivers/iommu/s390-iommu.c +359 -41
512 lines not on this page · raw message
Signature · 1 line
2.39.2
Niklas Schnelle
maintainerPATCH v13 4/6 iommu/s390: Disable deferred flush for ISM devices
In reply to Niklas SchnelleISM devices are virtual PCI devices used for cross-LPAR communication. Unlike real PCI devices ISM devices do not use the hardware IOMMU but inspects IOMMU translation tables directly on IOTLB flush (s390 RPCIT instruction).
ISM devices keep their DMA allocations static and only very rarely DMA unmap at all. For each IOTLB flush that occurs after unmap the ISM devices will however inspect the area of the IOVA space indicated by the flush. This means that for the global IOTLB flushes used by the flush queue mechanism the entire IOVA space would be inspected. In principle this would be fine, albeit potentially unnecessarily slow, it turns out however that ISM devices are sensitive to seeing IOVA addresses that are currently in use in the IOVA range being flushed. Seeing such in-use IOVA addresses will cause the ISM device to enter an error state and become unusable.
Fix this by claiming IOMMU_CAP_DEFERRED_FLUSH only for non-ISM devices. This makes sure IOTLB flushes only cover IOVAs that have been unmapped and also restricts the range of the IOTLB flush potentially reducing latency spikes.
--- drivers/iommu/s390-iommu.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/iommu/s390-iommu.c +3 -1
@@ -315,11 +315,13 @@ static struct s390_domain *to_s390_domain(struct iommu_domain *dom)static bool s390_iommu_capable (struct device *dev,enum iommu_cap cap) { +struct zpci_dev *zdev =to_zpci_dev (dev); +switch (cap) {case IOMMU_CAP_CACHE_COHERENCY :return true;case IOMMU_CAP_DEFERRED_FLUSH : -return true; +return zdev->pft !=PCI_FUNC_TYPE_ISM ;default :return false; }
Signature · 1 line
2.39.2
Re: [PATCH v13 4/6] iommu/s390: Disable deferred flush for ISM devices
Quoted 16 lines · Niklas Schnelle
ISM devices are virtual PCI devices used for cross-LPAR communication. Unlike real PCI devices ISM devices do not use the hardware IOMMU but inspects IOMMU translation tables directly on IOTLB flush (s390 RPCIT instruction).
ISM devices keep their DMA allocations static and only very rarely DMA unmap at all. For each IOTLB flush that occurs after unmap the ISM devices will however inspect the area of the IOVA space indicated by the flush. This means that for the global IOTLB flushes used by the flush queue mechanism the entire IOVA space would be inspected. In principle this would be fine, albeit potentially unnecessarily slow, it turns out however that ISM devices are sensitive to seeing IOVA addresses that are currently in use in the IOVA range being flushed. Seeing such in-use IOVA addresses will cause the ISM device to enter an error state and become unusable.
Quoted 6 lines
Fix this by claiming
IOMMU_CAP_DEFERRED_FLUSHonly for non-ISM devices. This makes sure IOTLB flushes only cover IOVAs that have been unmapped and also restricts the range of the IOTLB flush potentially reducing latency spikes.Signed-off-by Niklas Schnelle <[email protected]>
Looks like my review tag from v12 got dropped on accident -- no changes since prior version and still looks good to me so
Niklas Schnelle
maintainerPATCH v13 5/6 iommu/dma: Allow a single FQ in addition to per-CPU FQs
In reply to Niklas SchnelleIn some virtualized environments, including s390 paged memory guests, IOTLB flushes are used to update IOMMU shadow tables. Due to this, they are much more expensive than in typical bare metal environments or non-paged s390 guests. In addition they may parallelize poorly in virtualized environments. This changes the trade off for flushing IOVAs such that minimizing the number of IOTLB flushes trumps any benefit of cheaper queuing operations or increased paralellism.
In this scenario per-CPU flush queues pose several problems. Firstly per-CPU memory is often quite limited prohibiting larger queues. Secondly collecting IOVAs per-CPU but flushing via a global timeout reduces the number of IOVAs flushed for each timeout especially on s390 where PCI interrupts may not be bound to a specific CPU.
Let's introduce a single flush queue mode that reuses the same queue logic but only allocates a single global queue. This mode is selected by dma-iommu if a newly introduced .shadow_on_flush flag is set in struct dev_iommu. As a first user the s390 IOMMU driver sets this flag during probe_device. With the unchanged small FQ size and timeouts this setting is worse than per-CPU queues but a follow up patch will make the FQ size and timeout variable. Together this allows the common IOVA flushing code to more closely resemble the global flush behavior used on s390's previous internal DMA API implementation.
--- drivers/iommu/dma-iommu.c | 168 ++++++++++++++++++++++++++++++++++----------- drivers/iommu/s390-iommu.c | 3 + include/linux/iommu.h | 2 + 3 files changed, 134 insertions(+), 39 deletions(-)
drivers/iommu/dma-iommu.c +129 -39
@@ -43,14 +43,26 @@ enum iommu_dma_cookie_type { IOMMU_DMA_MSI_COOKIE, }; +enum iommu_dma_queue_type { +IOMMU_DMA_OPTS_PER_CPU_QUEUE , +IOMMU_DMA_OPTS_SINGLE_QUEUE , +}; + +struct iommu_dma_options { +enum iommu_dma_queue_type qt; +}; + struct iommu_dma_cookie { enum iommu_dma_cookie_type type; union { /* Full allocator for IOMMU_DMA_IOVA_COOKIE */ struct { struct iova_domain iovad; - - struct iova_fq __percpu *fq; /* Flush queue */ + /* Flush queue */ + union { + struct iova_fq *single_fq; + struct iova_fq __percpu *percpu_fq; + }; /* Number of TLB flushes that have been started */ atomic64_t fq_flush_start_cnt; /* Number of TLB flushes that have been finished */
@@ -67,6 +79,8 @@ struct iommu_dma_cookie {/* Domain for flush queue callback; NULL if flush queue not in use */ struct iommu_domain *fq_domain; +/* Options for dma-iommu use */ +struct iommu_dma_options options;struct mutex mutex; };
@@ -124,7 +138,7 @@ static inline unsigned int fq_ring_add(struct iova_fq *fq)return idx; } -static void fq_ring_free (struct iommu_dma_cookie *cookie,struct iova_fq *fq) +static void fq_ring_free_locked (struct iommu_dma_cookie *cookie,struct iova_fq *fq) {u64 counter =atomic64_read (&cookie->fq_flush_finish_cnt);unsigned int idx;
@@ -145,6 +159,15 @@ static void fq_ring_free(struct iommu_dma_cookie *cookie, struct iova_fq *fq) } } +static void fq_ring_free (struct iommu_dma_cookie *cookie,struct iova_fq *fq) +{ +unsigned long flags; + +spin_lock_irqsave (&fq->lock, flags); +fq_ring_free_locked (cookie, fq); +spin_unlock_irqrestore (&fq->lock, flags); +} +static void fq_flush_iotlb (struct iommu_dma_cookie *cookie) {atomic64_inc (&cookie->fq_flush_start_cnt);
@@ -160,14 +183,11 @@ static void fq_flush_timeout(struct timer_list *t)atomic_set (&cookie->fq_timer_on,0 );fq_flush_iotlb (cookie); -for_each_possible_cpu (cpu) { -unsigned long flags; -struct iova_fq *fq; - - fq =per_cpu_ptr (cookie->fq, cpu); -spin_lock_irqsave (&fq->lock, flags); -fq_ring_free (cookie, fq); -spin_unlock_irqrestore (&fq->lock, flags); +if (cookie->options.qt ==IOMMU_DMA_OPTS_SINGLE_QUEUE ) { +fq_ring_free (cookie, cookie->single_fq); + }else { +for_each_possible_cpu (cpu) +fq_ring_free (cookie,per_cpu_ptr (cookie->percpu_fq, cpu)); } }
@@ -188,7 +208,11 @@ static void queue_iova(struct iommu_dma_cookie *cookie,*/ smp_mb (); - fq =raw_cpu_ptr (cookie->fq); +if (cookie->options.qt ==IOMMU_DMA_OPTS_SINGLE_QUEUE ) + fq = cookie->single_fq; +else + fq =raw_cpu_ptr (cookie->percpu_fq); +spin_lock_irqsave (&fq->lock, flags);/*
@@ -196,11 +220,11 @@ static void queue_iova(struct iommu_dma_cookie *cookie,* flushed out on another CPU. This makes the fq_full() check below less * likely to be true. */ -fq_ring_free (cookie, fq); +fq_ring_free_locked (cookie, fq);if (fq_full (fq)) {fq_flush_iotlb (cookie); -fq_ring_free (cookie, fq); +fq_ring_free_locked (cookie, fq); } idx =fq_ring_add (fq);
@@ -219,31 +243,88 @@ static void queue_iova(struct iommu_dma_cookie *cookie, jiffies + msecs_to_jiffies(IOVA_FQ_TIMEOUT)); } -static void iommu_dma_free_fq (struct iommu_dma_cookie *cookie) +static void iommu_dma_free_fq_single (struct iova_fq *fq) +{ +int idx; + +fq_ring_for_each (idx, fq) +put_pages_list (&fq->entries[idx].freelist); +vfree (fq); +} + +static void iommu_dma_free_fq_percpu (struct iova_fq __percpu *percpu_fq) {int cpu, idx; -if (!cookie->fq) -return ; - -del_timer_sync (&cookie->fq_timer);/* The IOVAs will be torn down separately, so just free our queued pages */ for_each_possible_cpu (cpu) { -struct iova_fq *fq =per_cpu_ptr (cookie->fq, cpu); +struct iova_fq *fq =per_cpu_ptr (percpu_fq, cpu);fq_ring_for_each (idx, fq)put_pages_list (&fq->entries[idx].freelist); } -free_percpu (cookie->fq); +free_percpu (percpu_fq); +} + +static void iommu_dma_free_fq (struct iommu_dma_cookie *cookie) +{ +if (!cookie->fq_domain) +return ; + +del_timer_sync (&cookie->fq_timer); +if (cookie->options.qt ==IOMMU_DMA_OPTS_SINGLE_QUEUE ) +iommu_dma_free_fq_single (cookie->single_fq); +else +iommu_dma_free_fq_percpu (cookie->percpu_fq); +} + +static void iommu_dma_init_one_fq (struct iova_fq *fq) +{ +int i; + + fq->head =0 ; + fq->tail =0 ; + +spin_lock_init (&fq->lock); + +for (i =0 ; i <IOVA_FQ_SIZE ; i++) +INIT_LIST_HEAD (&fq->entries[i].freelist); +} + +static int iommu_dma_init_fq_single (struct iommu_dma_cookie *cookie) +{ +struct iova_fq *queue; + + queue =vmalloc (sizeof (*queue)); +if (!queue) +return -ENOMEM ; +iommu_dma_init_one_fq (queue); + cookie->single_fq = queue; + +return 0 ; +} + +static int iommu_dma_init_fq_percpu (struct iommu_dma_cookie *cookie) +{ +struct iova_fq __percpu *queue; +int cpu; + + queue = alloc_percpu(struct iova_fq); +if (!queue) +return -ENOMEM ; + +for_each_possible_cpu (cpu) +iommu_dma_init_one_fq (per_cpu_ptr (queue, cpu)); + cookie->percpu_fq = queue; +return 0 ; }/* sysfs updates are serialised by the mutex of the group owning @domain */ int iommu_dma_init_fq (struct iommu_domain *domain) {struct iommu_dma_cookie *cookie = domain->iova_cookie; -struct iova_fq __percpu *queue; -int i, cpu; +int rc;if (cookie->fq_domain)return 0 ;
@@ -251,26 +332,16 @@ int iommu_dma_init_fq(struct iommu_domain *domain)atomic64_set (&cookie->fq_flush_start_cnt,0 );atomic64_set (&cookie->fq_flush_finish_cnt,0 ); - queue = alloc_percpu(struct iova_fq); -if (!queue) { +if (cookie->options.qt ==IOMMU_DMA_OPTS_SINGLE_QUEUE ) + rc =iommu_dma_init_fq_single (cookie); +else + rc =iommu_dma_init_fq_percpu (cookie); + +if (rc) {pr_warn ("iova flush queue initialization failed\n" );return -ENOMEM ; } -for_each_possible_cpu (cpu) { -struct iova_fq *fq =per_cpu_ptr (queue, cpu); - - fq->head =0 ; - fq->tail =0 ; - -spin_lock_init (&fq->lock); - -for (i =0 ; i <IOVA_FQ_SIZE ; i++) -INIT_LIST_HEAD (&fq->entries[i].freelist); - } - - cookie->fq = queue; -timer_setup (&cookie->fq_timer, fq_flush_timeout,0 );atomic_set (&cookie->fq_timer_on,0 );/*
@@ -554,6 +625,23 @@ static bool dev_use_sg_swiotlb(struct device *dev, struct scatterlist *sg,return false; } +/** +* iommu_dma_init_options - Initialize dma-iommu options +* @options: The options to be initialized +* @dev: Device the options are set for +* +* This allows tuning dma-iommu specific to device properties +*/ +static void iommu_dma_init_options (struct iommu_dma_options *options, +struct device *dev) +{ +/* Shadowing IOTLB flushes do better with a single queue */ +if (dev->iommu->shadow_on_flush) + options->qt =IOMMU_DMA_OPTS_SINGLE_QUEUE ; +else + options->qt =IOMMU_DMA_OPTS_PER_CPU_QUEUE ; +} +/** * iommu_dma_init_domain - Initialise a DMA mapping domain * @domain: IOMMU domain previously prepared by iommu_get_dma_cookie()
@@ -614,6 +702,8 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base,if (ret) goto done_unlock; +iommu_dma_init_options (&cookie->options, dev); +/* If the FQ fails we can simply fall back to strict mode */ if (domain->type ==IOMMU_DOMAIN_DMA_FQ && (!device_iommu_capable (dev,IOMMU_CAP_DEFERRED_FLUSH ) ||iommu_dma_init_fq (domain)))
drivers/iommu/s390-iommu.c +3 -0
@@ -463,6 +463,9 @@ static struct iommu_device *s390_iommu_probe_device(struct device *dev)if (zdev->end_dma >ZPCI_TABLE_SIZE_RT -1 ) zdev->end_dma =ZPCI_TABLE_SIZE_RT -1 ; +if (zdev->tlb_refresh) + dev->iommu->shadow_on_flush =1 ; +return &zdev->iommu_dev; }
include/linux/iommu.h +2 -0
@@ -424,6 +424,7 @@ struct iommu_fault_param {* @attach_deferred: the dma domain attachment is deferred * @pci_32bit_workaround: Limit DMA allocations to 32-bit IOVAs * @require_direct: device requires IOMMU_RESV_DIRECT regions +* @shadow_on_flush: IOTLB flushes are used to sync shadow tables * * TODO: migrate other per device data pointers under iommu_dev_data, e.g. * struct iommu_group *iommu_group;
@@ -439,6 +440,7 @@ struct dev_iommu { u32 attach_deferred:1; u32 pci_32bit_workaround:1; u32 require_direct:1; + u32 shadow_on_flush:1; }; int iommu_device_register(struct iommu_device *iommu,
Signature · 1 line
2.39.2
Niklas Schnelle
maintainerPATCH v13 6/6 iommu/dma: Use a large flush queue and timeout for shadow_on_flush
In reply to Niklas SchnelleFlush queues currently use a fixed compile time size of 256 entries. This being a power of 2 allows the compiler to use shift and mask instead of more expensive modulo operations. With per-CPU flush queues larger queue sizes would hit per-CPU allocation limits, with a single flush queue these limits do not apply however. Also with single queues being particularly suitable for virtualized environments with expensive IOTLB flushes these benefit especially from larger queues and thus fewer flushes.
To this end re-order struct iova_fq so we can use a dynamic array and introduce the flush queue size and timeouts as new options in the iommu_dma_options struct. So as not to lose the shift and mask optimization, use a power of 2 for the length and use explicit shift and mask instead of letting the compiler optimize this.
A large queue size and 1 second timeout is then set for the shadow on flush case set by s390 paged memory guests. This then brings performance on par with the previous s390 specific DMA API implementation.
--- drivers/iommu/dma-iommu.c | 50 ++++++++++++++++++++++++++++++----------------- 1 file changed, 32 insertions(+), 18 deletions(-)
drivers/iommu/dma-iommu.c +32 -18
@@ -50,6 +50,8 @@ enum iommu_dma_queue_type {struct iommu_dma_options {enum iommu_dma_queue_type qt; +size_t fq_size; +unsigned int fq_timeout; };struct iommu_dma_cookie {
@@ -98,10 +100,12 @@ static int __init iommu_dma_forcedac_setup(char *str)early_param ("iommu.forcedac" , iommu_dma_forcedac_setup);/* Number of entries per flush queue */ -#define IOVA_FQ_SIZE 256 +#define IOVA_DEFAULT_FQ_SIZE 256 +#define IOVA_SINGLE_FQ_SIZE 32768/* Timeout (in ms) after which entries are flushed from the queue */ -#define IOVA_FQ_TIMEOUT 10 +#define IOVA_DEFAULT_FQ_TIMEOUT 10 +#define IOVA_SINGLE_FQ_TIMEOUT 1000/* Flush queue entry for deferred flushing */ struct iova_fq_entry {
@@ -113,18 +117,19 @@ struct iova_fq_entry {/* Per-CPU flush queue structure */ struct iova_fq { -struct iova_fq_entry entries[IOVA_FQ_SIZE ]; -unsigned int head, tail;spinlock_t lock; +unsigned int head, tail; +unsigned int mod_mask; +struct iova_fq_entry entries[]; };#define fq_ring_for_each (i, fq) \ - for ((i) = (fq)->head; (i) != (fq)->tail; (i) = ((i) + 1) % IOVA_FQ_SIZE) + for ((i) = (fq)->head; (i) != (fq)->tail; (i) = ((i) + 1) & (fq)->mod_mask)static inline bool fq_full (struct iova_fq *fq) {assert_spin_locked (&fq->lock); -return (((fq->tail +1 ) %IOVA_FQ_SIZE ) == fq->head); +return (((fq->tail +1 ) & fq->mod_mask) == fq->head); }static inline unsigned int fq_ring_add (struct iova_fq *fq)
@@ -133,7 +138,7 @@ static inline unsigned int fq_ring_add(struct iova_fq *fq)assert_spin_locked (&fq->lock); - fq->tail = (idx +1 ) %IOVA_FQ_SIZE ; + fq->tail = (idx +1 ) & fq->mod_mask;return idx; }
@@ -155,7 +160,7 @@ static void fq_ring_free_locked(struct iommu_dma_cookie *cookie, struct iova_fq fq->entries[idx].iova_pfn, fq->entries[idx].pages); - fq->head = (fq->head +1 ) %IOVA_FQ_SIZE ; + fq->head = (fq->head +1 ) & fq->mod_mask; } }
@@ -240,7 +245,7 @@ static void queue_iova(struct iommu_dma_cookie *cookie,if (!atomic_read (&cookie->fq_timer_on) && !atomic_xchg (&cookie->fq_timer_on,1 ))mod_timer (&cookie->fq_timer, - jiffies +msecs_to_jiffies (IOVA_FQ_TIMEOUT )); + jiffies +msecs_to_jiffies (cookie->options.fq_timeout)); }static void iommu_dma_free_fq_single (struct iova_fq *fq)
@@ -279,27 +284,29 @@ static void iommu_dma_free_fq(struct iommu_dma_cookie *cookie)iommu_dma_free_fq_percpu (cookie->percpu_fq); } -static void iommu_dma_init_one_fq (struct iova_fq *fq) +static void iommu_dma_init_one_fq (struct iova_fq *fq,size_t fq_size) {int i; fq->head =0 ; fq->tail =0 ; + fq->mod_mask = fq_size -1 ;spin_lock_init (&fq->lock); -for (i =0 ; i <IOVA_FQ_SIZE ; i++) +for (i =0 ; i < fq_size; i++)INIT_LIST_HEAD (&fq->entries[i].freelist); }static int iommu_dma_init_fq_single (struct iommu_dma_cookie *cookie) { +size_t fq_size = cookie->options.fq_size;struct iova_fq *queue; - queue =vmalloc (sizeof (*queue)); + queue =vmalloc (struct_size (queue, entries, fq_size));if (!queue)return -ENOMEM ; -iommu_dma_init_one_fq (queue); +iommu_dma_init_one_fq (queue, fq_size); cookie->single_fq = queue;return 0 ;
@@ -307,15 +314,17 @@ static int iommu_dma_init_fq_single(struct iommu_dma_cookie *cookie)static int iommu_dma_init_fq_percpu (struct iommu_dma_cookie *cookie) { +size_t fq_size = cookie->options.fq_size;struct iova_fq __percpu *queue;int cpu; - queue = alloc_percpu(struct iova_fq); + queue =__alloc_percpu (struct_size (queue, entries, fq_size), + __alignof__(*queue));if (!queue)return -ENOMEM ;for_each_possible_cpu (cpu) -iommu_dma_init_one_fq (per_cpu_ptr (queue, cpu)); +iommu_dma_init_one_fq (per_cpu_ptr (queue, cpu), fq_size); cookie->percpu_fq = queue;return 0 ; }
@@ -635,11 +644,16 @@ static bool dev_use_sg_swiotlb(struct device *dev, struct scatterlist *sg,static void iommu_dma_init_options (struct iommu_dma_options *options,struct device *dev) { -/* Shadowing IOTLB flushes do better with a single queue */ -if (dev->iommu->shadow_on_flush) +/* Shadowing IOTLB flushes do better with a single large queue */ +if (dev->iommu->shadow_on_flush) { options->qt =IOMMU_DMA_OPTS_SINGLE_QUEUE ; -else + options->fq_timeout =IOVA_SINGLE_FQ_TIMEOUT ; + options->fq_size =IOVA_SINGLE_FQ_SIZE ; + }else { options->qt =IOMMU_DMA_OPTS_PER_CPU_QUEUE ; + options->fq_size =IOVA_DEFAULT_FQ_SIZE ; + options->fq_timeout =IOVA_DEFAULT_FQ_TIMEOUT ; + } }/**
Signature · 1 line
2.39.2
Quoted 7 lines · Niklas Schnelle
Niklas Schnelle (6):
iommu: Allow .iotlb_sync_map to fail and handle s390's-ENOMEMreturn s390/pci: prepareis_passed_through()for dma-iommu s390/pci: Use dma-iommu layer
iommu/s390: Disable deferred flush for ISM devices iommu/dma: Allow a single FQ in addition to per-CPU FQs iommu/dma: Use a large flush queue and timeout forshadow_on_flush
Applied, thanks Niklas.
Niklas Schnelle
maintainerQuoted 11 lines · Joerg Roedel
Quoted 7 lines · Niklas Schnelle
Niklas Schnelle (6):
iommu: Allow .iotlb_sync_map to fail and handle s390's-ENOMEMreturn s390/pci: prepareis_passed_through()for dma-iommu s390/pci: Use dma-iommu layer
iommu/s390: Disable deferred flush for ISM devices iommu/dma: Allow a single FQ in addition to per-CPU FQs iommu/dma: Use a large flush queue and timeout forshadow_on_flushApplied, thanks Niklas.
Thank you Joerg. During some more testing I've found a minor issue where resetting the debug statistics still used the now obsolete counters that should have been removed. So I'll send a follow up patch, sorry for not noticing this sooner.
Thanks,
Niklas