Skip to contentSkip to conversation

iommu/dma: s390 DMA API conversion and optimized IOTLB flushing

PATCH Superseded Niklas Schnelle 6 participants 19 messages Jul 17, 2023 – Aug 29, 2023

Subsystems (21)

v11 of 13 Superseded Latest is v13 →

lore
Share & tools
lore

Niklas Schnelle

author maintainer
#0 lore raw

v11 cover iommu/dma: s390 DMA API conversion and optimized IOTLB flushing

Hi All,

This patch series converts s390's PCI support from its platform specific DMA API implementation in arch/s390/pci/pci_dma.c to the common DMA IOMMU layer. The conversion itself is done in patches 3-4 with patch 2 providing the final necessary IOMMU driver improvement to handle s390's special IOTLB flush out-of-resource indication in virtualized environments. Patches 1-2 may be applied independently. The conversion itself only touches the s390 IOMMU driver and s390 arch code moving over remaining functions from the s390 DMA API implementation. No changes to common code are necessary.

After patch 3 the basic conversion is done and on our partitioning machine hypervisor LPAR performance matches or exceeds the existing code. When running under z/VM or KVM however, performance plummets to about half of the existing code due to a much higher rate of IOTLB flushes for unmapped pages. Due to the hypervisors use of IOTLB flushes to synchronize their shadow tables these are very expensive and minimizing them is key for regaining the performance loss.

To this end patches 4-5 add a new, single queue, IOTLB flushing scheme as an alternative to the existing per-CPU flush queues. Introducing an alternative scheme was suggested by Robin Murphy[1]. In the previous RFC of this conversion Robin suggested reusing more of the existing queuing logic which I incorporated since v2. The single queue mode is introduced in patch 4 together with a new .shadow_on_flush flag bit in struct dev_iommu. This allows IOMMU drivers to indicate that their IOTLB flushes do the extra work of shadowing. This then lets the dma-iommu code use a single queue.

Then patch 5 enables variable queue sizes using power of 2 values and shift/mask to keep performance as close to the existing code as possible. A larger queue size and timeout is then used by dma-iommu when shadow_on_flush is set. This same scheme may also be used by other IOMMU drivers with similar requirements. Particularly virtio-iommu may be a candidate.

In a previous version I verified that the new scheme does work on my x86_64 Ryzen workstation by locally modifying iommu_subsys_init() to default to the single queue mode and verifying its use via "/sys/.../iommu_group/type". I did not find problems with an AMD GPU, Intel NIC (with SR-IOV and KVM pass-through), NVMes or any on board peripherals.

This code is also available in the b4/dma_iommu topic branch of my git.kernel.org repository[3] with tags the version sent.

NOTE: Due to the large drop in performance I think we should not merge the DMA API conversion (patch 4) until we have a more suited IOVA flushing scheme with similar improvements as the proposed changes.

Best regards,
Niklas

[0] https://lore.kernel.org/linux-iommu/[email protected]/
[1] https://lore.kernel.org/linux-iommu/[email protected]/
[2] https://lore.kernel.org/linux-iommu/[email protected]/
[3] https://git.kernel.org/pub/scm/linux/kernel/git/niks/linux.git/

---
Changes in v11:
- Rebased on v6.5-rc2
- Added patch to force IOMMU_DOMAIN_DMA on s390 specific ISM devices
- Dropped the patch to properly set DMA mask on ISM devices which went upstream separately.
- s390 IOMMU driver now uses IOMMU_CAP_DEFERRED_FLUSH to enable DMA-FQ leaving no uses of IOMMU_DOMAIN_DMA_FQ in the driver.
- Link to v10: https://lore.kernel.org/r/[email protected]

Changes in v10:
- Rebased on v6.4-rc3
- Removed the .tune_dma_iommu() op in favor of a .shadow_on_flush flag
in struct dev_iommu which then let's the dma-iommu choose a single
queue and larger timeouts and IOVA counts. This leaves the dma-iommu
with full responsibility for the settings.
- The above change affects patches 5 and 6 and lead to a new subject for patch 6 since the flush queue size and timeout is no longer driver controlled
- Link to v9: https://lore.kernel.org/r/[email protected]

Changes in v9:
- Rebased on v6.4-rc2
- Re-ordered iommu_group_store_type() to allow passing the device to iommu_dma_init_fq()
- Link to v8: https://lore.kernel.org/r/[email protected]

---
Niklas Schnelle (6):
iommu: Allow .iotlb_sync_map to fail and handle s390's -ENOMEM return s390/pci: prepare is_passed_through() for dma-iommu s390/pci: Use dma-iommu layer
iommu/s390: Force ISM devices to use IOMMU_DOMAIN_DMA iommu/dma: Allow a single FQ in addition to per-CPU FQs iommu/dma: Use a large flush queue and timeout for shadow_on_flush

 Documentation/admin-guide/kernel-parameters.txt |   9 +-
 arch/s390/include/asm/pci.h                     |   7 -
 arch/s390/include/asm/pci_clp.h                 |   3 +
 arch/s390/include/asm/pci_dma.h                 | 119 +---
 arch/s390/pci/Makefile                          |   2 +-
 arch/s390/pci/pci.c                             |  22 +-
 arch/s390/pci/pci_bus.c                         |   5 -
 arch/s390/pci/pci_debug.c                       |  12 +-
 arch/s390/pci/pci_dma.c                         | 735 ------------------------
 arch/s390/pci/pci_event.c                       |  17 +-
 arch/s390/pci/pci_sysfs.c                       |  19 +-
 drivers/iommu/Kconfig                           |   4 +-
 drivers/iommu/amd/iommu.c                       |   5 +-
 drivers/iommu/apple-dart.c                      |   5 +-
 drivers/iommu/dma-iommu.c                       | 212 +++++--
 drivers/iommu/intel/iommu.c                     |   5 +-
 drivers/iommu/msm_iommu.c                       |   5 +-
 drivers/iommu/mtk_iommu.c                       |   5 +-
 drivers/iommu/s390-iommu.c                      | 433 ++++++++++++--
 drivers/iommu/sprd-iommu.c                      |   5 +-
 drivers/iommu/sun50i-iommu.c                    |   6 +-
 drivers/iommu/tegra-gart.c                      |   5 +-
 include/linux/iommu.h                           |   6 +-
 23 files changed, 641 insertions(+), 1005 deletions(-)

---
base-commit: fdf0eaf11452d72945af31804e2a1048ee1b574c
change-id: 20230310-dma_iommu-5e048c538647

Best regards,

Signature · 8 lines
Niklas Schnelle
Linux on Z Development

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: David Faller
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM Data Privacy Statement - https://www.ibm.com/privacy 
author maintainer
#1 lore raw

PATCH v11 1/6 iommu: Allow .iotlb_sync_map to fail and handle s390's -ENOMEM return

On s390 when using a paging hypervisor, .iotlb_sync_map is used to sync mappings by letting the hypervisor inspect the synced IOVA range and updating a shadow table. This however means that .iotlb_sync_map can fail as the hypervisor may run out of resources while doing the sync. This can be due to the hypervisor being unable to pin guest pages, due to a limit on mapped addresses such as vfio_iommu_type1.dma_entry_limit or lack of other resources. Either way such a failure to sync a mapping should result in a DMA_MAPPING_ERROR.

Now especially when running with batched IOTLB flushes for unmap it may be that some IOVAs have already been invalidated but not yet synced via .iotlb_sync_map. Thus if the hypervisor indicates running out of resources, first do a global flush allowing the hypervisor to free resources associated with these mappings as well a retry creating the new mappings and only if that also fails report this error to callers.

---
 drivers/iommu/amd/iommu.c    |  5 +++--
 drivers/iommu/apple-dart.c   |  5 +++--
 drivers/iommu/intel/iommu.c  |  5 +++--
 drivers/iommu/iommu.c        | 20 ++++++++++++++++----
 drivers/iommu/msm_iommu.c    |  5 +++--
 drivers/iommu/mtk_iommu.c    |  5 +++--
 drivers/iommu/s390-iommu.c   | 29 +++++++++++++++++++++++------
 drivers/iommu/sprd-iommu.c   |  5 +++--
 drivers/iommu/sun50i-iommu.c |  6 ++++--
 drivers/iommu/tegra-gart.c   |  5 +++--
 include/linux/iommu.h        |  4 ++--
 11 files changed, 66 insertions(+), 28 deletions(-)
drivers/iommu/amd/iommu.c +3 -2
index c3b58a8389b9..019d700ed0eb 100644
@@ -2219,14 +2219,15 @@ static int amd_iommu_attach_device(struct iommu_domain *dom,
 	return ret;
 }
 
-static void amd_iommu_iotlb_sync_map(struct iommu_domain *dom,
-				     unsigned long iova, size_t size)
+static int amd_iommu_iotlb_sync_map(struct iommu_domain *dom,
+				    unsigned long iova, size_t size)
 {
 	struct protection_domain *domain = to_pdomain(dom);
 	struct io_pgtable_ops *ops = &domain->iop.iop.ops;
 
 	if (ops->map_pages)
 		domain_flush_np_cache(domain, iova, size);
+	return 0;
 }
 
 static int amd_iommu_map_pages(struct iommu_domain *dom, unsigned long iova,
drivers/iommu/apple-dart.c +3 -2
index 8af64b57f048..d061493db634 100644
@@ -506,10 +506,11 @@ static void apple_dart_iotlb_sync(struct iommu_domain *domain,
 	apple_dart_domain_flush_tlb(to_dart_domain(domain));
 }
 
-static void apple_dart_iotlb_sync_map(struct iommu_domain *domain,
-				      unsigned long iova, size_t size)
+static int apple_dart_iotlb_sync_map(struct iommu_domain *domain,
+				     unsigned long iova, size_t size)
 {
 	apple_dart_domain_flush_tlb(to_dart_domain(domain));
+	return 0;
 }
 
 static phys_addr_t apple_dart_iova_to_phys(struct iommu_domain *domain,
drivers/iommu/intel/iommu.c +3 -2
index 5c8c5cdc36cf..7c83493f0a42 100644
@@ -4697,8 +4697,8 @@ static bool risky_device(struct pci_dev *pdev)
 	return false;
 }
 
-static void intel_iommu_iotlb_sync_map(struct iommu_domain *domain,
-				       unsigned long iova, size_t size)
+static int intel_iommu_iotlb_sync_map(struct iommu_domain *domain,
+				      unsigned long iova, size_t size)
 {
 	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
 	unsigned long pages = aligned_nrpages(iova, size);
@@ -4708,6 +4708,7 @@ static void intel_iommu_iotlb_sync_map(struct iommu_domain *domain,
 
 	xa_for_each(&dmar_domain->iommu_array, i, info)
 		__mapping_notify_one(info->iommu, dmar_domain, pfn, pages);
+	return 0;
 }
 
 static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid)
drivers/iommu/iommu.c +16 -4
index caaf563d38ae..fd9f79731d6a 100644
@@ -2413,8 +2413,17 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova,
 		return -EINVAL;
 
 	ret = __iommu_map(domain, iova, paddr, size, prot, gfp);
-	if (ret == 0 && ops->iotlb_sync_map)
-		ops->iotlb_sync_map(domain, iova, size);
+	if (ret == 0 && ops->iotlb_sync_map) {
+		ret = ops->iotlb_sync_map(domain, iova, size);
+		if (ret)
+			goto out_err;
+	}
+
+	return ret;
+
+out_err:
+	/* undo mappings already done */
+	iommu_unmap(domain, iova, size);
 
 	return ret;
 }
@@ -2555,8 +2564,11 @@ ssize_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
 			sg = sg_next(sg);
 	}
 
-	if (ops->iotlb_sync_map)
-		ops->iotlb_sync_map(domain, iova, mapped);
+	if (ops->iotlb_sync_map) {
+		ret = ops->iotlb_sync_map(domain, iova, mapped);
+		if (ret)
+			goto out_err;
+	}
 	return mapped;
 
 out_err:
drivers/iommu/msm_iommu.c +3 -2
index 79d89bad5132..47926d3290e6 100644
@@ -486,12 +486,13 @@ static int msm_iommu_map(struct iommu_domain *domain, unsigned long iova,
 	return ret;
 }
 
-static void msm_iommu_sync_map(struct iommu_domain *domain, unsigned long iova,
-			       size_t size)
+static int msm_iommu_sync_map(struct iommu_domain *domain, unsigned long iova,
+			      size_t size)
 {
 	struct msm_priv *priv = to_msm_priv(domain);
 
 	__flush_iotlb_range(iova, size, SZ_4K, false, priv);
+	return 0;
 }
 
 static size_t msm_iommu_unmap(struct iommu_domain *domain, unsigned long iova,
drivers/iommu/mtk_iommu.c +3 -2
index e93906d6e112..c1bcec1979b0 100644
@@ -794,12 +794,13 @@ static void mtk_iommu_iotlb_sync(struct iommu_domain *domain,
 	mtk_iommu_tlb_flush_range_sync(gather->start, length, dom->bank);
 }
 
-static void mtk_iommu_sync_map(struct iommu_domain *domain, unsigned long iova,
-			       size_t size)
+static int mtk_iommu_sync_map(struct iommu_domain *domain, unsigned long iova,
+			      size_t size)
 {
 	struct mtk_iommu_domain *dom = to_mtk_domain(domain);
 
 	mtk_iommu_tlb_flush_range_sync(iova, size, dom->bank);
+	return 0;
 }
 
 static phys_addr_t mtk_iommu_iova_to_phys(struct iommu_domain *domain,
drivers/iommu/s390-iommu.c +23 -6
index fbf59a8db29b..6723d77489e8 100644
@@ -205,6 +205,12 @@ static void s390_iommu_release_device(struct device *dev)
 		__s390_iommu_detach_device(zdev);
 }
 
+static int zpci_refresh_all(struct zpci_dev *zdev)
+{
+	return zpci_refresh_trans((u64)zdev->fh << 32, zdev->start_dma,
+				  zdev->end_dma - zdev->start_dma + 1);
+}
+
 static void s390_iommu_flush_iotlb_all(struct iommu_domain *domain)
 {
 	struct s390_domain *s390_domain = to_s390_domain(domain);
@@ -212,8 +218,7 @@ static void s390_iommu_flush_iotlb_all(struct iommu_domain *domain)
 
 	rcu_read_lock();
 	list_for_each_entry_rcu(zdev, &s390_domain->devices, iommu_list) {
-		zpci_refresh_trans((u64)zdev->fh << 32, zdev->start_dma,
-				   zdev->end_dma - zdev->start_dma + 1);
+		zpci_refresh_all(zdev);
 	}
 	rcu_read_unlock();
 }
@@ -237,20 +242,32 @@ static void s390_iommu_iotlb_sync(struct iommu_domain *domain,
 	rcu_read_unlock();
 }
 
-static void s390_iommu_iotlb_sync_map(struct iommu_domain *domain,
-				      unsigned long iova, size_t size)
+static int s390_iommu_iotlb_sync_map(struct iommu_domain *domain,
+				     unsigned long iova, size_t size)
 {
 	struct s390_domain *s390_domain = to_s390_domain(domain);
 	struct zpci_dev *zdev;
+	int ret = 0;
 
 	rcu_read_lock();
 	list_for_each_entry_rcu(zdev, &s390_domain->devices, iommu_list) {
 		if (!zdev->tlb_refresh)
 			continue;
-		zpci_refresh_trans((u64)zdev->fh << 32,
-				   iova, size);
+		ret = zpci_refresh_trans((u64)zdev->fh << 32,
+					 iova, size);
+		/*
+		 * let the hypervisor discover invalidated entries
+		 * allowing it to free IOVAs and unpin pages
+		 */
+		if (ret == -ENOMEM) {
+			ret = zpci_refresh_all(zdev);
+			if (ret)
+				break;
+		}
 	}
 	rcu_read_unlock();
+
+	return ret;
 }
 
 static int s390_iommu_validate_trans(struct s390_domain *s390_domain,
drivers/iommu/sprd-iommu.c +3 -2
index 39e34fdeccda..18d61fe29ca0 100644
@@ -343,8 +343,8 @@ static size_t sprd_iommu_unmap(struct iommu_domain *domain, unsigned long iova,
 	return size;
 }
 
-static void sprd_iommu_sync_map(struct iommu_domain *domain,
-				unsigned long iova, size_t size)
+static int sprd_iommu_sync_map(struct iommu_domain *domain,
+			       unsigned long iova, size_t size)
 {
 	struct sprd_iommu_domain *dom = to_sprd_domain(domain);
 	unsigned int reg;
@@ -356,6 +356,7 @@ static void sprd_iommu_sync_map(struct iommu_domain *domain,
 
 	/* clear IOMMU TLB buffer after page table updated */
 	sprd_iommu_write(dom->sdev, reg, 0xffffffff);
+	return 0;
 }
 
 static void sprd_iommu_sync(struct iommu_domain *domain,
drivers/iommu/sun50i-iommu.c +4 -2
index 74c5cb93e900..45c90fa27631 100644
@@ -402,8 +402,8 @@ static void sun50i_iommu_flush_iotlb_all(struct iommu_domain *domain)
 	spin_unlock_irqrestore(&iommu->iommu_lock, flags);
 }
 
-static void sun50i_iommu_iotlb_sync_map(struct iommu_domain *domain,
-					unsigned long iova, size_t size)
+static int sun50i_iommu_iotlb_sync_map(struct iommu_domain *domain,
+				       unsigned long iova, size_t size)
 {
 	struct sun50i_iommu_domain *sun50i_domain = to_sun50i_domain(domain);
 	struct sun50i_iommu *iommu = sun50i_domain->iommu;
@@ -412,6 +412,8 @@ static void sun50i_iommu_iotlb_sync_map(struct iommu_domain *domain,
 	spin_lock_irqsave(&iommu->iommu_lock, flags);
 	sun50i_iommu_zap_range(iommu, iova, size);
 	spin_unlock_irqrestore(&iommu->iommu_lock, flags);
+
+	return 0;
 }
 
 static void sun50i_iommu_iotlb_sync(struct iommu_domain *domain,
drivers/iommu/tegra-gart.c +3 -2
index a482ff838b53..44966d7b07ba 100644
@@ -252,10 +252,11 @@ static int gart_iommu_of_xlate(struct device *dev,
 	return 0;
 }
 
-static void gart_iommu_sync_map(struct iommu_domain *domain, unsigned long iova,
-				size_t size)
+static int gart_iommu_sync_map(struct iommu_domain *domain, unsigned long iova,
+			       size_t size)
 {
 	FLUSH_GART_REGS(gart_handle);
+	return 0;
 }
 
 static void gart_iommu_sync(struct iommu_domain *domain,
include/linux/iommu.h +2 -2
index d31642596675..182cc4c71e62 100644
@@ -339,8 +339,8 @@ struct iommu_domain_ops {
 			      struct iommu_iotlb_gather *iotlb_gather);
 
 	void (*flush_iotlb_all)(struct iommu_domain *domain);
-	void (*iotlb_sync_map)(struct iommu_domain *domain, unsigned long iova,
-			       size_t size);
+	int (*iotlb_sync_map)(struct iommu_domain *domain, unsigned long iova,
+			      size_t size);
 	void (*iotlb_sync)(struct iommu_domain *domain,
 			   struct iommu_iotlb_gather *iotlb_gather);
 
Signature · 1 line
2.39.2
author maintainer
#2 lore raw

PATCH v11 2/6 s390/pci: prepare is_passed_through() for dma-iommu

In reply to Niklas Schnelle

With the IOMMU always controlled through the IOMMU driver testing for zdev->s390_domain is not a valid indication of the device being passed-through. Instead test if zdev->kzdev is set.

---
 arch/s390/pci/pci_event.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
arch/s390/pci/pci_event.c +11 -4
index b9324ca2eb94..4ef5a6a1d618 100644
@@ -59,9 +59,16 @@ static inline bool ers_result_indicates_abort(pci_ers_result_t ers_res)
 	}
 }
 
-static bool is_passed_through(struct zpci_dev *zdev)
+static bool is_passed_through(struct pci_dev *pdev)
 {
-	return zdev->s390_domain;
+	struct zpci_dev *zdev = to_zpci(pdev);
+	bool ret;
+
+	mutex_lock(&zdev->kzdev_lock);
+	ret = !!zdev->kzdev;
+	mutex_unlock(&zdev->kzdev_lock);
+
+	return ret;
 }
 
 static bool is_driver_supported(struct pci_driver *driver)
@@ -176,7 +183,7 @@ static pci_ers_result_t zpci_event_attempt_error_recovery(struct pci_dev *pdev)
 	}
 	pdev->error_state = pci_channel_io_frozen;
 
-	if (is_passed_through(to_zpci(pdev))) {
+	if (is_passed_through(pdev)) {
 		pr_info("%s: Cannot be recovered in the host because it is a pass-through device\n",
 			pci_name(pdev));
 		goto out_unlock;
@@ -239,7 +246,7 @@ static void zpci_event_io_failure(struct pci_dev *pdev, pci_channel_state_t es)
 	 * we will inject the error event and let the guest recover the device
 	 * itself.
 	 */
-	if (is_passed_through(to_zpci(pdev)))
+	if (is_passed_through(pdev))
 		goto out;
 	driver = to_pci_driver(pdev->dev.driver);
 	if (driver && driver->err_handler && driver->err_handler->error_detected)
Signature · 1 line
2.39.2
author maintainer
#3 lore raw

PATCH v11 3/6 s390/pci: Use dma-iommu layer

In reply to Niklas Schnelle

While s390 already has a standard IOMMU driver and previous changes have added I/O TLB flushing operations this driver is currently only used for user-space PCI access such as vfio-pci. For the DMA API s390 instead utilizes its own implementation in arch/s390/pci/pci_dma.c which drives the same hardware and shares some code but requires a complex and fragile hand over between DMA API and IOMMU API use of a device and despite code sharing still leads to significant duplication and maintenance effort. Let's utilize the common code DMAP API implementation from drivers/iommu/dma-iommu.c instead allowing us to get rid of arch/s390/pci/pci_dma.c.

---
 Documentation/admin-guide/kernel-parameters.txt |   9 +-
 arch/s390/include/asm/pci.h                     |   7 -
 arch/s390/include/asm/pci_clp.h                 |   3 +
 arch/s390/include/asm/pci_dma.h                 | 119 +---
 arch/s390/pci/Makefile                          |   2 +-
 arch/s390/pci/pci.c                             |  22 +-
 arch/s390/pci/pci_bus.c                         |   5 -
 arch/s390/pci/pci_debug.c                       |  12 +-
 arch/s390/pci/pci_dma.c                         | 735 ------------------------
 arch/s390/pci/pci_event.c                       |   2 -
 arch/s390/pci/pci_sysfs.c                       |  19 +-
 drivers/iommu/Kconfig                           |   4 +-
 drivers/iommu/s390-iommu.c                      | 391 ++++++++++++-
 13 files changed, 407 insertions(+), 923 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +5 -4
index a1457995fd41..0bf936808386 100644
@@ -2188,7 +2188,7 @@
 			  forcing Dual Address Cycle for PCI cards supporting
 			  greater than 32-bit addressing.
 
-	iommu.strict=	[ARM64, X86] Configure TLB invalidation behaviour
+	iommu.strict=	[ARM64, X86, S390] Configure TLB invalidation behaviour
 			Format: { "0" | "1" }
 			0 - Lazy mode.
 			  Request that DMA unmap operations use deferred
@@ -5513,9 +5513,10 @@
 	s390_iommu=	[HW,S390]
 			Set s390 IOTLB flushing mode
 		strict
-			With strict flushing every unmap operation will result in
-			an IOTLB flush. Default is lazy flushing before reuse,
-			which is faster.
+			With strict flushing every unmap operation will result
+			in an IOTLB flush. Default is lazy flushing before
+			reuse, which is faster. Deprecated, equivalent to
+			iommu.strict=1.
 
 	s390_iommu_aperture=	[KNL,S390]
 			Specifies the size of the per device DMA address space
arch/s390/include/asm/pci.h +0 -7
index b248694e0024..3f74f1cf37df 100644
@@ -159,13 +159,6 @@ struct zpci_dev {
 	unsigned long	*dma_table;
 	int		tlb_refresh;
 
-	spinlock_t	iommu_bitmap_lock;
-	unsigned long	*iommu_bitmap;
-	unsigned long	*lazy_bitmap;
-	unsigned long	iommu_size;
-	unsigned long	iommu_pages;
-	unsigned int	next_bit;
-
 	struct iommu_device iommu_dev;  /* IOMMU core handle */
 
 	char res_name[16];
arch/s390/include/asm/pci_clp.h +3 -0
index d6189ed14f84..f0c677ddd270 100644
@@ -50,6 +50,9 @@ struct clp_fh_list_entry {
 #define CLP_UTIL_STR_LEN	64
 #define CLP_PFIP_NR_SEGMENTS	4
 
+/* PCI function type numbers */
+#define PCI_FUNC_TYPE_ISM	0x5	/* ISM device */
+
 extern bool zpci_unique_uid;
 
 struct clp_rsp_slpc_pci {
arch/s390/include/asm/pci_dma.h +9 -110

126 lines not on this page · raw message

arch/s390/pci/Makefile +1 -1
index 5ae31ca9dd44..0547a10406e7 100644
@@ -3,7 +3,7 @@
 # Makefile for the s390 PCI subsystem.
 #
 
-obj-$(CONFIG_PCI)	+= pci.o pci_irq.o pci_dma.o pci_clp.o pci_sysfs.o \
+obj-$(CONFIG_PCI)	+= pci.o pci_irq.o pci_clp.o pci_sysfs.o \
 			   pci_event.o pci_debug.o pci_insn.o pci_mmio.o \
 			   pci_bus.o pci_kvm_hook.o
 obj-$(CONFIG_PCI_IOV)	+= pci_iov.o
arch/s390/pci/pci.c +5 -17
index afc3f33788da..36aea977bbe0 100644
@@ -124,7 +124,11 @@ int zpci_register_ioat(struct zpci_dev *zdev, u8 dmaas,
 
 	WARN_ON_ONCE(iota & 0x3fff);
 	fib.pba = base;
-	fib.pal = limit;
+	/* Work around off by one in ISM virt device */
+	if (zdev->pft == PCI_FUNC_TYPE_ISM && limit > base)
+		fib.pal = limit + (1 << 12);
+	else
+		fib.pal = limit;
 	fib.iota = iota | ZPCI_IOTA_RTTO_FLAG;
 	fib.gd = zdev->gisa;
 	cc = zpci_mod_fc(req, &fib, status);
@@ -619,7 +623,6 @@ int pcibios_device_add(struct pci_dev *pdev)
 		pdev->no_vf_scan = 1;
 
 	pdev->dev.groups = zpci_attr_groups;
-	pdev->dev.dma_ops = &s390_pci_dma_ops;
 	zpci_map_resources(pdev);
 
 	for (i = 0; i < PCI_STD_NUM_BARS; i++) {
@@ -793,8 +796,6 @@ int zpci_hot_reset_device(struct zpci_dev *zdev)
 	if (zdev->dma_table)
 		rc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
 					virt_to_phys(zdev->dma_table), &status);
-	else
-		rc = zpci_dma_init_device(zdev);
 	if (rc) {
 		zpci_disable_device(zdev);
 		return rc;
@@ -902,11 +903,6 @@ int zpci_deconfigure_device(struct zpci_dev *zdev)
 	if (zdev->zbus->bus)
 		zpci_bus_remove_device(zdev, false);
 
-	if (zdev->dma_table) {
-		rc = zpci_dma_exit_device(zdev);
-		if (rc)
-			return rc;
-	}
 	if (zdev_enabled(zdev)) {
 		rc = zpci_disable_device(zdev);
 		if (rc)
@@ -955,8 +951,6 @@ void zpci_release_device(struct kref *kref)
 	if (zdev->zbus->bus)
 		zpci_bus_remove_device(zdev, false);
 
-	if (zdev->dma_table)
-		zpci_dma_exit_device(zdev);
 	if (zdev_enabled(zdev))
 		zpci_disable_device(zdev);
 
@@ -1146,10 +1140,6 @@ static int __init pci_base_init(void)
 	if (rc)
 		goto out_irq;
 
-	rc = zpci_dma_init();
-	if (rc)
-		goto out_dma;
-
 	rc = clp_scan_pci_devices();
 	if (rc)
 		goto out_find;
@@ -1159,8 +1149,6 @@ static int __init pci_base_init(void)
 	return 0;
 
 out_find:
-	zpci_dma_exit();
-out_dma:
 	zpci_irq_exit();
 out_irq:
 	zpci_mem_exit();
arch/s390/pci/pci_bus.c +0 -5
index 32245b970a0c..daa5d7450c7d 100644
@@ -47,11 +47,6 @@ static int zpci_bus_prepare_device(struct zpci_dev *zdev)
 		rc = zpci_enable_device(zdev);
 		if (rc)
 			return rc;
-		rc = zpci_dma_init_device(zdev);
-		if (rc) {
-			zpci_disable_device(zdev);
-			return rc;
-		}
 	}
 
 	if (!zdev->has_resources) {
arch/s390/pci/pci_debug.c +9 -3
index ca6bd98eec13..6dde2263c79d 100644
@@ -53,9 +53,11 @@ static char *pci_fmt3_names[] = {
 };
 
 static char *pci_sw_names[] = {
-	"Allocated pages",
 	"Mapped pages",
 	"Unmapped pages",
+	"Global RPCITs",
+	"Sync Map RPCITs",
+	"Sync RPCITs",
 };
 
 static void pci_fmb_show(struct seq_file *m, char *name[], int length,
@@ -69,10 +71,14 @@ static void pci_fmb_show(struct seq_file *m, char *name[], int length,
 
 static void pci_sw_counter_show(struct seq_file *m)
 {
-	struct zpci_dev *zdev = m->private;
-	atomic64_t *counter = &zdev->allocated_pages;
+	struct zpci_iommu_ctrs  *ctrs = zpci_get_iommu_ctrs(m->private);
+	atomic64_t *counter;
 	int i;
 
+	if (!ctrs)
+		return;
+
+	counter = &ctrs->mapped_pages;
 	for (i = 0; i < ARRAY_SIZE(pci_sw_names); i++, counter++)
 		seq_printf(m, "%26s:\t%llu\n", pci_sw_names[i],
 			   atomic64_read(counter));
arch/s390/pci/pci_dma.c deleted +0 -735

735 lines not on this page · raw message

arch/s390/pci/pci_event.c +0 -2
index 4ef5a6a1d618..4d9773ef9e0a 100644
@@ -313,8 +313,6 @@ static void zpci_event_hard_deconfigured(struct zpci_dev *zdev, u32 fh)
 	/* Even though the device is already gone we still
 	 * need to free zPCI resources as part of the disable.
 	 */
-	if (zdev->dma_table)
-		zpci_dma_exit_device(zdev);
 	if (zdev_enabled(zdev))
 		zpci_disable_device(zdev);
 	zdev->state = ZPCI_FN_STATE_STANDBY;
arch/s390/pci/pci_sysfs.c +8 -11
index cae280e5c047..8a7abac51816 100644
@@ -56,6 +56,7 @@ static ssize_t recover_store(struct device *dev, struct device_attribute *attr,
 	struct pci_dev *pdev = to_pci_dev(dev);
 	struct zpci_dev *zdev = to_zpci(pdev);
 	int ret = 0;
+	u8 status;
 
 	/* Can't use device_remove_self() here as that would lead us to lock
 	 * the pci_rescan_remove_lock while holding the device' kernfs lock.
@@ -82,12 +83,6 @@ static ssize_t recover_store(struct device *dev, struct device_attribute *attr,
 	pci_lock_rescan_remove();
 	if (pci_dev_is_added(pdev)) {
 		pci_stop_and_remove_bus_device(pdev);
-		if (zdev->dma_table) {
-			ret = zpci_dma_exit_device(zdev);
-			if (ret)
-				goto out;
-		}
-
 		if (zdev_enabled(zdev)) {
 			ret = zpci_disable_device(zdev);
 			/*
@@ -105,14 +100,16 @@ static ssize_t recover_store(struct device *dev, struct device_attribute *attr,
 		ret = zpci_enable_device(zdev);
 		if (ret)
 			goto out;
-		ret = zpci_dma_init_device(zdev);
-		if (ret) {
-			zpci_disable_device(zdev);
-			goto out;
+
+		if (zdev->dma_table) {
+			ret = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
+						 virt_to_phys(zdev->dma_table), &status);
+			if (ret)
+				zpci_disable_device(zdev);
 		}
-		pci_rescan_bus(zdev->zbus->bus);
 	}
 out:
+	pci_rescan_bus(zdev->zbus->bus);
 	pci_unlock_rescan_remove();
 	if (kn)
 		sysfs_unbreak_active_protection(kn);
drivers/iommu/Kconfig +2 -2
index 2b12b583ef4b..5dc4c8e8a08c 100644
@@ -91,7 +91,7 @@ config IOMMU_DEBUGFS
 choice
 	prompt "IOMMU default domain type"
 	depends on IOMMU_API
-	default IOMMU_DEFAULT_DMA_LAZY if X86 || IA64
+	default IOMMU_DEFAULT_DMA_LAZY if X86 || IA64 || S390
 	default IOMMU_DEFAULT_DMA_STRICT
 	help
 	  Choose the type of IOMMU domain used to manage DMA API usage by
@@ -146,7 +146,7 @@ config OF_IOMMU
 
 # IOMMU-agnostic DMA-mapping layer
 config IOMMU_DMA
-	def_bool ARM64 || IA64 || X86
+	def_bool ARM64 || IA64 || X86 || S390
 	select DMA_OPS
 	select IOMMU_API
 	select IOMMU_IOVA
drivers/iommu/s390-iommu.c +365 -26

512 lines not on this page · raw message

Signature · 1 line
2.39.2
author maintainer
#4 lore raw

PATCH v11 4/6 iommu/s390: Force ISM devices to use IOMMU_DOMAIN_DMA

In reply to Niklas Schnelle

ISM devices are virtual PCI devices used for cross-LPAR communication. Unlike real PCI devices ISM devices do not use the hardware IOMMU but inspects IOMMU translation tables directly on IOTLB flush (s390 RPCIT instruction).

While ISM devices keep their DMA allocations static and only very rarely DMA unmap at all, For each IOTLB flush that occurs after unmap the ISM devices will inspect the area of the IOVA space indicated by the flush. This means that for the global IOTLB flushes used by the flush queue mechanism the entire IOVA space would be inspected. In principle this would be fine, albeit potentially unnecessarily slow, it turns out however that ISM devices are sensitive to seeing IOVA addresses that are currently in use in the IOVA range being flushed. Seeing such in-use IOVA addresses will cause the ISM device to enter an error state and become unusable.

Fix this by forcing IOMMU_DOMAIN_DMA to be used for ISM devices. This makes sure IOTLB flushes only cover IOVAs that have been unmapped and also restricts the range of the IOTLB flush potentially reducing latency spikes.

---
 drivers/iommu/s390-iommu.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/iommu/s390-iommu.c +10 -0
index f6d6c60e5634..020cc538e4c4 100644
@@ -710,6 +710,15 @@ struct zpci_iommu_ctrs *zpci_get_iommu_ctrs(struct zpci_dev *zdev)
 	return &zdev->s390_domain->ctrs;
 }
 
+static int s390_iommu_def_domain_type(struct device *dev)
+{
+	struct zpci_dev *zdev = to_zpci_dev(dev);
+
+	if (zdev->pft == PCI_FUNC_TYPE_ISM)
+		return IOMMU_DOMAIN_DMA;
+	return 0;
+}
+
 int zpci_init_iommu(struct zpci_dev *zdev)
 {
 	u64 aperture_size;
@@ -789,6 +798,7 @@ static const struct iommu_ops s390_iommu_ops = {
 	.probe_device = s390_iommu_probe_device,
 	.probe_finalize = s390_iommu_probe_finalize,
 	.release_device = s390_iommu_release_device,
+	.def_domain_type = s390_iommu_def_domain_type,
 	.device_group = generic_device_group,
 	.pgsize_bitmap = SZ_4K,
 	.get_resv_regions = s390_iommu_get_resv_regions,
Signature · 1 line
2.39.2
maintainer Reviewed-by on v11 4/6
#5 lore raw

Re: [PATCH v11 4/6] iommu/s390: Force ISM devices to use IOMMU_DOMAIN_DMA

Quoted 16 lines · Niklas Schnelle

ISM devices are virtual PCI devices used for cross-LPAR communication. Unlike real PCI devices ISM devices do not use the hardware IOMMU but inspects IOMMU translation tables directly on IOTLB flush (s390 RPCIT instruction).

While ISM devices keep their DMA allocations static and only very rarely DMA unmap at all, For each IOTLB flush that occurs after unmap the ISM devices will inspect the area of the IOVA space indicated by the flush. This means that for the global IOTLB flushes used by the flush queue mechanism the entire IOVA space would be inspected. In principle this would be fine, albeit potentially unnecessarily slow, it turns out however that ISM devices are sensitive to seeing IOVA addresses that are currently in use in the IOVA range being flushed. Seeing such in-use IOVA addresses will cause the ISM device to enter an error state and become unusable.

Quoted 6 lines

Fix this by forcing IOMMU_DOMAIN_DMA to be used for ISM devices. This makes sure IOTLB flushes only cover IOVAs that have been unmapped and also restricts the range of the IOTLB flush potentially reducing latency spikes.

This makes sense to me.

maintainer
#6 lore raw

Re: [PATCH v11 4/6] iommu/s390: Force ISM devices to use IOMMU_DOMAIN_DMA

In reply to Niklas Schnelle
Quoted 16 lines · Niklas Schnelle

ISM devices are virtual PCI devices used for cross-LPAR communication. Unlike real PCI devices ISM devices do not use the hardware IOMMU but inspects IOMMU translation tables directly on IOTLB flush (s390 RPCIT instruction).

While ISM devices keep their DMA allocations static and only very rarely DMA unmap at all, For each IOTLB flush that occurs after unmap the ISM devices will inspect the area of the IOVA space indicated by the flush. This means that for the global IOTLB flushes used by the flush queue mechanism the entire IOVA space would be inspected. In principle this would be fine, albeit potentially unnecessarily slow, it turns out however that ISM devices are sensitive to seeing IOVA addresses that are currently in use in the IOVA range being flushed. Seeing such in-use IOVA addresses will cause the ISM device to enter an error state and become unusable.

Quoted 4 lines

Fix this by forcing IOMMU_DOMAIN_DMA to be used for ISM devices. This makes sure IOTLB flushes only cover IOVAs that have been unmapped and also restricts the range of the IOTLB flush potentially reducing latency spikes.

Would it not be simpler to return false for IOMMU_CAP_DEFERRED_FLUSH for these devices?

Cheers,
Robin.

Quoted 34 lines
---
  drivers/iommu/s390-iommu.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
drivers/iommu/s390-iommu.c +10 -0
index f6d6c60e5634..020cc538e4c4 100644
@@ -710,6 +710,15 @@ struct zpci_iommu_ctrs *zpci_get_iommu_ctrs(struct zpci_dev *zdev)
  	return &zdev->s390_domain->ctrs;
  }
 
+static int s390_iommu_def_domain_type(struct device *dev)
+{
+	struct zpci_dev *zdev = to_zpci_dev(dev);
+
+	if (zdev->pft == PCI_FUNC_TYPE_ISM)
+		return IOMMU_DOMAIN_DMA;
+	return 0;
+}
+
  int zpci_init_iommu(struct zpci_dev *zdev)
  {
  	u64 aperture_size;
@@ -789,6 +798,7 @@ static const struct iommu_ops s390_iommu_ops = {
  	.probe_device = s390_iommu_probe_device,
  	.probe_finalize = s390_iommu_probe_finalize,
  	.release_device = s390_iommu_release_device,
+	.def_domain_type = s390_iommu_def_domain_type,
  	.device_group = generic_device_group,
  	.pgsize_bitmap = SZ_4K,
  	.get_resv_regions = s390_iommu_get_resv_regions,
author maintainer
#7 lore raw

Re: [PATCH v11 4/6] iommu/s390: Force ISM devices to use IOMMU_DOMAIN_DMA

Quoted 22 lines · Robin Murphy
Quoted 16 lines · Niklas Schnelle

ISM devices are virtual PCI devices used for cross-LPAR communication. Unlike real PCI devices ISM devices do not use the hardware IOMMU but inspects IOMMU translation tables directly on IOTLB flush (s390 RPCIT instruction).

While ISM devices keep their DMA allocations static and only very rarely DMA unmap at all, For each IOTLB flush that occurs after unmap the ISM devices will inspect the area of the IOVA space indicated by the flush. This means that for the global IOTLB flushes used by the flush queue mechanism the entire IOVA space would be inspected. In principle this would be fine, albeit potentially unnecessarily slow, it turns out however that ISM devices are sensitive to seeing IOVA addresses that are currently in use in the IOVA range being flushed. Seeing such in-use IOVA addresses will cause the ISM device to enter an error state and become unusable.

Quoted 4 lines

Fix this by forcing IOMMU_DOMAIN_DMA to be used for ISM devices. This makes sure IOTLB flushes only cover IOVAs that have been unmapped and also restricts the range of the IOTLB flush potentially reducing latency spikes.

Quoted 5 lines

Would it not be simpler to return false for IOMMU_CAP_DEFERRED_FLUSH for these devices?

Cheers,
Robin.

Nice idea thank you. This is indeed less code, basically just return zdev->pft != PCI_FUNC_TYPE_ISM for the IOMMU_CAP_DEFERRED_FLUSH check. I think it's also semantically more clear in that we don't really care about the domain type but about not getting deferred flushes.

Thanks,
Niklas

Quoted 36 lines
Quoted 35 lines
---
  drivers/iommu/s390-iommu.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
drivers/iommu/s390-iommu.c +10 -0
index f6d6c60e5634..020cc538e4c4 100644
@@ -710,6 +710,15 @@ struct zpci_iommu_ctrs *zpci_get_iommu_ctrs(struct zpci_dev *zdev)
  	return &zdev->s390_domain->ctrs;
  }
 
+static int s390_iommu_def_domain_type(struct device *dev)
+{
+	struct zpci_dev *zdev = to_zpci_dev(dev);
+
+	if (zdev->pft == PCI_FUNC_TYPE_ISM)
+		return IOMMU_DOMAIN_DMA;
+	return 0;
+}
+
  int zpci_init_iommu(struct zpci_dev *zdev)
  {
  	u64 aperture_size;
@@ -789,6 +798,7 @@ static const struct iommu_ops s390_iommu_ops = {
  	.probe_device = s390_iommu_probe_device,
  	.probe_finalize = s390_iommu_probe_finalize,
  	.release_device = s390_iommu_release_device,
+	.def_domain_type = s390_iommu_def_domain_type,
  	.device_group = generic_device_group,
  	.pgsize_bitmap = SZ_4K,
  	.get_resv_regions = s390_iommu_get_resv_regions,
author maintainer
#8 lore raw

PATCH v11 5/6 iommu/dma: Allow a single FQ in addition to per-CPU FQs

In reply to Niklas Schnelle

In some virtualized environments, including s390 paged memory guests, IOTLB flushes are used to update IOMMU shadow tables. Due to this, they are much more expensive than in typical bare metal environments or non-paged s390 guests. In addition they may parallelize poorly in virtualized environments. This changes the trade off for flushing IOVAs such that minimizing the number of IOTLB flushes trumps any benefit of cheaper queuing operations or increased paralellism.

In this scenario per-CPU flush queues pose several problems. Firstly per-CPU memory is often quite limited prohibiting larger queues. Secondly collecting IOVAs per-CPU but flushing via a global timeout reduces the number of IOVAs flushed for each timeout especially on s390 where PCI interrupts may not be bound to a specific CPU.

Let's introduce a single flush queue mode that reuses the same queue logic but only allocates a single global queue. This mode is selected by dma-iommu if a newly introduced .shadow_on_flush flag is set in struct dev_iommu. As a first user the s390 IOMMU driver sets this flag during probe_device. With the unchanged small FQ size and timeouts this setting is worse than per-CPU queues but a follow up patch will make the FQ size and timeout variable. Together this allows the common IOVA flushing code to more closely resemble the global flush behavior used on s390's previous internal DMA API implementation.

---
 drivers/iommu/dma-iommu.c  | 178 +++++++++++++++++++++++++++++++++------------
 drivers/iommu/iommu.c      |  20 +----
 drivers/iommu/s390-iommu.c |   3 +
 include/linux/iommu.h      |   2 +
 4 files changed, 142 insertions(+), 61 deletions(-)
drivers/iommu/dma-iommu.c +133 -45
index e57724163835..4ada8b9749c9 100644
@@ -43,14 +43,23 @@ enum iommu_dma_cookie_type {
 	IOMMU_DMA_MSI_COOKIE,
 };
 
+struct dma_iommu_options {
+#define IOMMU_DMA_OPTS_PER_CPU_QUEUE	0L
+#define IOMMU_DMA_OPTS_SINGLE_QUEUE	BIT(0)
+	u64	flags;
+};
+
 struct iommu_dma_cookie {
 	enum iommu_dma_cookie_type	type;
 	union {
 		/* Full allocator for IOMMU_DMA_IOVA_COOKIE */
 		struct {
 			struct iova_domain	iovad;
-
-			struct iova_fq __percpu *fq;	/* Flush queue */
+			/* Flush queue */
+			union {
+				struct iova_fq	*single_fq;
+				struct iova_fq	__percpu *percpu_fq;
+			};
 			/* Number of TLB flushes that have been started */
 			atomic64_t		fq_flush_start_cnt;
 			/* Number of TLB flushes that have been finished */
@@ -67,6 +76,8 @@ struct iommu_dma_cookie {
 
 	/* Domain for flush queue callback; NULL if flush queue not in use */
 	struct iommu_domain		*fq_domain;
+	/* Options for dma-iommu use */
+	struct dma_iommu_options	options;
 	struct mutex			mutex;
 };
 
@@ -124,7 +135,7 @@ static inline unsigned int fq_ring_add(struct iova_fq *fq)
 	return idx;
 }
 
-static void fq_ring_free(struct iommu_dma_cookie *cookie, struct iova_fq *fq)
+static void fq_ring_free_locked(struct iommu_dma_cookie *cookie, struct iova_fq *fq)
 {
 	u64 counter = atomic64_read(&cookie->fq_flush_finish_cnt);
 	unsigned int idx;
@@ -145,6 +156,15 @@ static void fq_ring_free(struct iommu_dma_cookie *cookie, struct iova_fq *fq)
 	}
 }
 
+static void fq_ring_free(struct iommu_dma_cookie *cookie, struct iova_fq *fq)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&fq->lock, flags);
+	fq_ring_free_locked(cookie, fq);
+	spin_unlock_irqrestore(&fq->lock, flags);
+}
+
 static void fq_flush_iotlb(struct iommu_dma_cookie *cookie)
 {
 	atomic64_inc(&cookie->fq_flush_start_cnt);
@@ -152,23 +172,29 @@ static void fq_flush_iotlb(struct iommu_dma_cookie *cookie)
 	atomic64_inc(&cookie->fq_flush_finish_cnt);
 }
 
+static void fq_flush_percpu(struct iommu_dma_cookie *cookie)
+{
+	int cpu;
+
+	for_each_possible_cpu(cpu) {
+		struct iova_fq *fq;
+
+		fq = per_cpu_ptr(cookie->percpu_fq, cpu);
+		fq_ring_free(cookie, fq);
+	}
+}
+
 static void fq_flush_timeout(struct timer_list *t)
 {
 	struct iommu_dma_cookie *cookie = from_timer(cookie, t, fq_timer);
-	int cpu;
 
 	atomic_set(&cookie->fq_timer_on, 0);
 	fq_flush_iotlb(cookie);
 
-	for_each_possible_cpu(cpu) {
-		unsigned long flags;
-		struct iova_fq *fq;
-
-		fq = per_cpu_ptr(cookie->fq, cpu);
-		spin_lock_irqsave(&fq->lock, flags);
-		fq_ring_free(cookie, fq);
-		spin_unlock_irqrestore(&fq->lock, flags);
-	}
+	if (cookie->options.flags & IOMMU_DMA_OPTS_SINGLE_QUEUE)
+		fq_ring_free(cookie, cookie->single_fq);
+	else
+		fq_flush_percpu(cookie);
 }
 
 static void queue_iova(struct iommu_dma_cookie *cookie,
@@ -188,7 +214,11 @@ static void queue_iova(struct iommu_dma_cookie *cookie,
 	 */
 	smp_mb();
 
-	fq = raw_cpu_ptr(cookie->fq);
+	if (cookie->options.flags & IOMMU_DMA_OPTS_SINGLE_QUEUE)
+		fq = cookie->single_fq;
+	else
+		fq = raw_cpu_ptr(cookie->percpu_fq);
+
 	spin_lock_irqsave(&fq->lock, flags);
 
 	/*
@@ -196,11 +226,11 @@ static void queue_iova(struct iommu_dma_cookie *cookie,
 	 * flushed out on another CPU. This makes the fq_full() check below less
 	 * likely to be true.
 	 */
-	fq_ring_free(cookie, fq);
+	fq_ring_free_locked(cookie, fq);
 
 	if (fq_full(fq)) {
 		fq_flush_iotlb(cookie);
-		fq_ring_free(cookie, fq);
+		fq_ring_free_locked(cookie, fq);
 	}
 
 	idx = fq_ring_add(fq);
@@ -219,31 +249,90 @@ static void queue_iova(struct iommu_dma_cookie *cookie,
 			  jiffies + msecs_to_jiffies(IOVA_FQ_TIMEOUT));
 }
 
-static void iommu_dma_free_fq(struct iommu_dma_cookie *cookie)
+static void iommu_dma_free_fq_single(struct iova_fq *fq)
+{
+	int idx;
+
+	if (!fq)
+		return;
+	fq_ring_for_each(idx, fq)
+		put_pages_list(&fq->entries[idx].freelist);
+	vfree(fq);
+}
+
+static void iommu_dma_free_fq_percpu(struct iova_fq __percpu *percpu_fq)
 {
 	int cpu, idx;
 
-	if (!cookie->fq)
-		return;
-
-	del_timer_sync(&cookie->fq_timer);
 	/* The IOVAs will be torn down separately, so just free our queued pages */
 	for_each_possible_cpu(cpu) {
-		struct iova_fq *fq = per_cpu_ptr(cookie->fq, cpu);
+		struct iova_fq *fq = per_cpu_ptr(percpu_fq, cpu);
 
 		fq_ring_for_each(idx, fq)
 			put_pages_list(&fq->entries[idx].freelist);
 	}
 
-	free_percpu(cookie->fq);
+	free_percpu(percpu_fq);
+}
+
+static void iommu_dma_free_fq(struct iommu_dma_cookie *cookie)
+{
+	if (!cookie->fq_domain)
+		return;
+
+	del_timer_sync(&cookie->fq_timer);
+	if (cookie->options.flags & IOMMU_DMA_OPTS_SINGLE_QUEUE)
+		iommu_dma_free_fq_single(cookie->single_fq);
+	else
+		iommu_dma_free_fq_percpu(cookie->percpu_fq);
+}
+
+static void iommu_dma_init_one_fq(struct iova_fq *fq)
+{
+	int i;
+
+	fq->head = 0;
+	fq->tail = 0;
+
+	spin_lock_init(&fq->lock);
+
+	for (i = 0; i < IOVA_FQ_SIZE; i++)
+		INIT_LIST_HEAD(&fq->entries[i].freelist);
+}
+
+static int iommu_dma_init_fq_single(struct iommu_dma_cookie *cookie)
+{
+	struct iova_fq *queue;
+
+	queue = vzalloc(sizeof(*queue));
+	if (!queue)
+		return -ENOMEM;
+	iommu_dma_init_one_fq(queue);
+	cookie->single_fq = queue;
+
+	return 0;
+}
+
+static int iommu_dma_init_fq_percpu(struct iommu_dma_cookie *cookie)
+{
+	struct iova_fq __percpu *queue;
+	int cpu;
+
+	queue = alloc_percpu(struct iova_fq);
+	if (!queue)
+		return -ENOMEM;
+
+	for_each_possible_cpu(cpu)
+		iommu_dma_init_one_fq(per_cpu_ptr(queue, cpu));
+	cookie->percpu_fq = queue;
+	return 0;
 }
 
 /* sysfs updates are serialised by the mutex of the group owning @domain */
 int iommu_dma_init_fq(struct iommu_domain *domain)
 {
 	struct iommu_dma_cookie *cookie = domain->iova_cookie;
-	struct iova_fq __percpu *queue;
-	int i, cpu;
+	int rc;
 
 	if (cookie->fq_domain)
 		return 0;
@@ -251,26 +340,16 @@ int iommu_dma_init_fq(struct iommu_domain *domain)
 	atomic64_set(&cookie->fq_flush_start_cnt,  0);
 	atomic64_set(&cookie->fq_flush_finish_cnt, 0);
 
-	queue = alloc_percpu(struct iova_fq);
-	if (!queue) {
+	if (cookie->options.flags & IOMMU_DMA_OPTS_SINGLE_QUEUE)
+		rc = iommu_dma_init_fq_single(cookie);
+	else
+		rc = iommu_dma_init_fq_percpu(cookie);
+
+	if (rc) {
 		pr_warn("iova flush queue initialization failed\n");
 		return -ENOMEM;
 	}
 
-	for_each_possible_cpu(cpu) {
-		struct iova_fq *fq = per_cpu_ptr(queue, cpu);
-
-		fq->head = 0;
-		fq->tail = 0;
-
-		spin_lock_init(&fq->lock);
-
-		for (i = 0; i < IOVA_FQ_SIZE; i++)
-			INIT_LIST_HEAD(&fq->entries[i].freelist);
-	}
-
-	cookie->fq = queue;
-
 	timer_setup(&cookie->fq_timer, fq_flush_timeout, 0);
 	atomic_set(&cookie->fq_timer_on, 0);
 	/*
@@ -297,6 +376,7 @@ static struct iommu_dma_cookie *cookie_alloc(enum iommu_dma_cookie_type type)
 	if (cookie) {
 		INIT_LIST_HEAD(&cookie->msi_page_list);
 		cookie->type = type;
+		cookie->options.flags = IOMMU_DMA_OPTS_PER_CPU_QUEUE;
 	}
 	return cookie;
 }
@@ -614,10 +694,18 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base,
 	if (ret)
 		goto done_unlock;
 
-	/* If the FQ fails we can simply fall back to strict mode */
-	if (domain->type == IOMMU_DOMAIN_DMA_FQ &&
-	    (!device_iommu_capable(dev, IOMMU_CAP_DEFERRED_FLUSH) || iommu_dma_init_fq(domain)))
-		domain->type = IOMMU_DOMAIN_DMA;
+	if (domain->type == IOMMU_DOMAIN_DMA_FQ) {
+		/* Expensive shadowing IOTLB flushes require some tuning */
+		if (dev->iommu->shadow_on_flush)
+			cookie->options.flags |= IOMMU_DMA_OPTS_SINGLE_QUEUE;
+
+		/* If the FQ fails we can simply fall back to strict mode */
+		if (!device_iommu_capable(dev, IOMMU_CAP_DEFERRED_FLUSH) ||
+		    iommu_dma_init_fq(domain)) {
+			domain->type = IOMMU_DOMAIN_DMA;
+			cookie->options.flags &= ~IOMMU_DMA_OPTS_SINGLE_QUEUE;
+		}
+	}
 
 	ret = iova_reserve_iommu_regions(dev, domain);
 
drivers/iommu/iommu.c +4 -16
index fd9f79731d6a..caaf563d38ae 100644
@@ -2413,17 +2413,8 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova,
 		return -EINVAL;
 
 	ret = __iommu_map(domain, iova, paddr, size, prot, gfp);
-	if (ret == 0 && ops->iotlb_sync_map) {
-		ret = ops->iotlb_sync_map(domain, iova, size);
-		if (ret)
-			goto out_err;
-	}
-
-	return ret;
-
-out_err:
-	/* undo mappings already done */
-	iommu_unmap(domain, iova, size);
+	if (ret == 0 && ops->iotlb_sync_map)
+		ops->iotlb_sync_map(domain, iova, size);
 
 	return ret;
 }
@@ -2564,11 +2555,8 @@ ssize_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
 			sg = sg_next(sg);
 	}
 
-	if (ops->iotlb_sync_map) {
-		ret = ops->iotlb_sync_map(domain, iova, mapped);
-		if (ret)
-			goto out_err;
-	}
+	if (ops->iotlb_sync_map)
+		ops->iotlb_sync_map(domain, iova, mapped);
 	return mapped;
 
 out_err:
drivers/iommu/s390-iommu.c +3 -0
index 020cc538e4c4..63c56a440c9d 100644
@@ -468,6 +468,9 @@ static struct iommu_device *s390_iommu_probe_device(struct device *dev)
 	if (zdev->end_dma > ZPCI_TABLE_SIZE_RT - 1)
 		zdev->end_dma = ZPCI_TABLE_SIZE_RT - 1;
 
+	if (zdev->tlb_refresh)
+		dev->iommu->shadow_on_flush = 1;
+
 	return &zdev->iommu_dev;
 }
 
include/linux/iommu.h +2 -0
index 182cc4c71e62..c3687e066ed7 100644
@@ -409,6 +409,7 @@ struct iommu_fault_param {
  * @priv:	 IOMMU Driver private data
  * @max_pasids:  number of PASIDs this device can consume
  * @attach_deferred: the dma domain attachment is deferred
+ * @shadow_on_flush: IOTLB flushes are used to sync shadow tables
  *
  * TODO: migrate other per device data pointers under iommu_dev_data, e.g.
  *	struct iommu_group	*iommu_group;
@@ -422,6 +423,7 @@ struct dev_iommu {
 	void				*priv;
 	u32				max_pasids;
 	u32				attach_deferred:1;
+	u32				shadow_on_flush:1;
 };
 
 int iommu_device_register(struct iommu_device *iommu,
Signature · 1 line
2.39.2
maintainer Acked-by on v11 5/6
#9 lore raw

Re: [PATCH v11 5/6] iommu/dma: Allow a single FQ in addition to per-CPU FQs

Quoted 24 lines · Niklas Schnelle

In some virtualized environments, including s390 paged memory guests, IOTLB flushes are used to update IOMMU shadow tables. Due to this, they are much more expensive than in typical bare metal environments or non-paged s390 guests. In addition they may parallelize poorly in virtualized environments. This changes the trade off for flushing IOVAs such that minimizing the number of IOTLB flushes trumps any benefit of cheaper queuing operations or increased paralellism.

In this scenario per-CPU flush queues pose several problems. Firstly per-CPU memory is often quite limited prohibiting larger queues. Secondly collecting IOVAs per-CPU but flushing via a global timeout reduces the number of IOVAs flushed for each timeout especially on s390 where PCI interrupts may not be bound to a specific CPU.

Let's introduce a single flush queue mode that reuses the same queue logic but only allocates a single global queue. This mode is selected by dma-iommu if a newly introduced .shadow_on_flush flag is set in struct dev_iommu. As a first user the s390 IOMMU driver sets this flag during probe_device. With the unchanged small FQ size and timeouts this setting is worse than per-CPU queues but a follow up patch will make the FQ size and timeout variable. Together this allows the common IOVA flushing code to more closely resemble the global flush behavior used on s390's previous internal DMA API implementation.

Quoted 6 lines
---
  drivers/iommu/dma-iommu.c  | 178 +++++++++++++++++++++++++++++++++------------
  drivers/iommu/iommu.c      |  20 +----

The hunks in iommu.c appear to be an inadvertent reversion of patch #1?

A few bonus nits below which you can take or leave, but otherwise, with the rebase-mishap fixed:

Quoted 9 lines
  drivers/iommu/s390-iommu.c |   3 +
  include/linux/iommu.h      |   2 +
  4 files changed, 142 insertions(+), 61 deletions(-)
drivers/iommu/dma-iommu.c +0 -0
index e57724163835..4ada8b9749c9 100644

drivers/iommu/dma-iommu.c enum iommu_dma_cookie_type { · line 47

Quoted 5 lines
drivers/iommu/dma-iommu.c +2 -0
@@ -43,14 +43,23 @@ enum iommu_dma_cookie_type {
  	IOMMU_DMA_MSI_COOKIE,
  };
 
+struct dma_iommu_options {
+#define IOMMU_DMA_OPTS_PER_CPU_QUEUE	0L

Nit: if the intent is to add more flags then that will no longer make sense, and if not then we may as well just have a bool ;)

Quoted 81 lines
drivers/iommu/dma-iommu.c +33 -7
+#define IOMMU_DMA_OPTS_SINGLE_QUEUE	BIT(0)
+	u64	flags;
+};
+
  struct iommu_dma_cookie {
  	enum iommu_dma_cookie_type	type;
  	union {
  		/* Full allocator for IOMMU_DMA_IOVA_COOKIE */
  		struct {
  			struct iova_domain	iovad;
-
-			struct iova_fq __percpu *fq;	/* Flush queue */
+			/* Flush queue */
+			union {
+				struct iova_fq	*single_fq;
+				struct iova_fq	__percpu *percpu_fq;
+			};
  			/* Number of TLB flushes that have been started */
  			atomic64_t		fq_flush_start_cnt;
  			/* Number of TLB flushes that have been finished */
@@ -67,6 +76,8 @@ struct iommu_dma_cookie {
 
  	/* Domain for flush queue callback; NULL if flush queue not in use */
  	struct iommu_domain		*fq_domain;
+	/* Options for dma-iommu use */
+	struct dma_iommu_options	options;
  	struct mutex			mutex;
  };
 
@@ -124,7 +135,7 @@ static inline unsigned int fq_ring_add(struct iova_fq *fq)
  	return idx;
  }
 
-static void fq_ring_free(struct iommu_dma_cookie *cookie, struct iova_fq *fq)
+static void fq_ring_free_locked(struct iommu_dma_cookie *cookie, struct iova_fq *fq)
  {
  	u64 counter = atomic64_read(&cookie->fq_flush_finish_cnt);
  	unsigned int idx;
@@ -145,6 +156,15 @@ static void fq_ring_free(struct iommu_dma_cookie *cookie, struct iova_fq *fq)
  	}
  }
 
+static void fq_ring_free(struct iommu_dma_cookie *cookie, struct iova_fq *fq)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&fq->lock, flags);
+	fq_ring_free_locked(cookie, fq);
+	spin_unlock_irqrestore(&fq->lock, flags);
+}
+
  static void fq_flush_iotlb(struct iommu_dma_cookie *cookie)
  {
  	atomic64_inc(&cookie->fq_flush_start_cnt);
@@ -152,23 +172,29 @@ static void fq_flush_iotlb(struct iommu_dma_cookie *cookie)
  	atomic64_inc(&cookie->fq_flush_finish_cnt);
  }
 
+static void fq_flush_percpu(struct iommu_dma_cookie *cookie)
+{
+	int cpu;
+
+	for_each_possible_cpu(cpu) {
+		struct iova_fq *fq;
+
+		fq = per_cpu_ptr(cookie->percpu_fq, cpu);
+		fq_ring_free(cookie, fq);
+	}
+}
+
  static void fq_flush_timeout(struct timer_list *t)
  {
  	struct iommu_dma_cookie *cookie = from_timer(cookie, t, fq_timer);
-	int cpu;
 
  	atomic_set(&cookie->fq_timer_on, 0);
  	fq_flush_iotlb(cookie);
 
-	for_each_possible_cpu(cpu) {
-		unsigned long flags;
-		struct iova_fq *fq;

drivers/iommu/dma-iommu.c static void fq_flush_iotlb(struct iommu_dma_cookie *cookie) · line 197

Quoted 10 lines
drivers/iommu/dma-iommu.c +4 -6
-
-		fq = per_cpu_ptr(cookie->fq, cpu);
-		spin_lock_irqsave(&fq->lock, flags);
-		fq_ring_free(cookie, fq);
-		spin_unlock_irqrestore(&fq->lock, flags);
-	}
+	if (cookie->options.flags & IOMMU_DMA_OPTS_SINGLE_QUEUE)
+		fq_ring_free(cookie, cookie->single_fq);
+	else
+		fq_flush_percpu(cookie);

Nit: honestly I'd just inline that as:

	else for_each_possible_cpu(cpu)
		fq_ring_free(cookie, per_cpu_ptr(cookie->percpu_fq, cpu));

(possibly with extra braces if you don't feel brave enough to join the elite "else for" club...)

Quoted 31 lines
drivers/iommu/dma-iommu.c +7 -3
  }
 
  static void queue_iova(struct iommu_dma_cookie *cookie,
@@ -188,7 +214,11 @@ static void queue_iova(struct iommu_dma_cookie *cookie,
  	 */
  	smp_mb();
 
-	fq = raw_cpu_ptr(cookie->fq);
+	if (cookie->options.flags & IOMMU_DMA_OPTS_SINGLE_QUEUE)
+		fq = cookie->single_fq;
+	else
+		fq = raw_cpu_ptr(cookie->percpu_fq);
+
  	spin_lock_irqsave(&fq->lock, flags);
 
  	/*
@@ -196,11 +226,11 @@ static void queue_iova(struct iommu_dma_cookie *cookie,
  	 * flushed out on another CPU. This makes the fq_full() check below less
  	 * likely to be true.
  	 */
-	fq_ring_free(cookie, fq);
+	fq_ring_free_locked(cookie, fq);
 
  	if (fq_full(fq)) {
  		fq_flush_iotlb(cookie);
-		fq_ring_free(cookie, fq);
+		fq_ring_free_locked(cookie, fq);
  	}
 
  	idx = fq_ring_add(fq);

drivers/iommu/dma-iommu.c static void queue_iova(struct iommu_dma_cookie *cookie, · line 257

Quoted 10 lines
drivers/iommu/dma-iommu.c +6 -1
@@ -219,31 +249,90 @@ static void queue_iova(struct iommu_dma_cookie *cookie,
  			  jiffies + msecs_to_jiffies(IOVA_FQ_TIMEOUT));
  }
 
-static void iommu_dma_free_fq(struct iommu_dma_cookie *cookie)
+static void iommu_dma_free_fq_single(struct iova_fq *fq)
+{
+	int idx;
+
+	if (!fq)
+		return;

Nit: That should never be true if cookie->fq_domain was set

Quoted 46 lines
drivers/iommu/dma-iommu.c +30 -6
+	fq_ring_for_each(idx, fq)
+		put_pages_list(&fq->entries[idx].freelist);
+	vfree(fq);
+}
+
+static void iommu_dma_free_fq_percpu(struct iova_fq __percpu *percpu_fq)
  {
  	int cpu, idx;
 
-	if (!cookie->fq)
-		return;
-
-	del_timer_sync(&cookie->fq_timer);
  	/* The IOVAs will be torn down separately, so just free our queued pages */
  	for_each_possible_cpu(cpu) {
-		struct iova_fq *fq = per_cpu_ptr(cookie->fq, cpu);
+		struct iova_fq *fq = per_cpu_ptr(percpu_fq, cpu);
 
  		fq_ring_for_each(idx, fq)
  			put_pages_list(&fq->entries[idx].freelist);
  	}
 
-	free_percpu(cookie->fq);
+	free_percpu(percpu_fq);
+}
+
+static void iommu_dma_free_fq(struct iommu_dma_cookie *cookie)
+{
+	if (!cookie->fq_domain)
+		return;
+
+	del_timer_sync(&cookie->fq_timer);
+	if (cookie->options.flags & IOMMU_DMA_OPTS_SINGLE_QUEUE)
+		iommu_dma_free_fq_single(cookie->single_fq);
+	else
+		iommu_dma_free_fq_percpu(cookie->percpu_fq);
+}
+
+static void iommu_dma_init_one_fq(struct iova_fq *fq)
+{
+	int i;
+
+	fq->head = 0;
+	fq->tail = 0;
+
+	spin_lock_init(&fq->lock);

drivers/iommu/dma-iommu.c static void queue_iova(struct iommu_dma_cookie *cookie, · line 307

Quoted 10 lines
drivers/iommu/dma-iommu.c +10 -0
+
+	for (i = 0; i < IOVA_FQ_SIZE; i++)
+		INIT_LIST_HEAD(&fq->entries[i].freelist);
+}
+
+static int iommu_dma_init_fq_single(struct iommu_dma_cookie *cookie)
+{
+	struct iova_fq *queue;
+
+	queue = vzalloc(sizeof(*queue));

Nit: vmalloc() - no need to zero the whole thing when the percpu path doesn't.

Quoted 67 lines
drivers/iommu/dma-iommu.c +28 -18
+	if (!queue)
+		return -ENOMEM;
+	iommu_dma_init_one_fq(queue);
+	cookie->single_fq = queue;
+
+	return 0;
+}
+
+static int iommu_dma_init_fq_percpu(struct iommu_dma_cookie *cookie)
+{
+	struct iova_fq __percpu *queue;
+	int cpu;
+
+	queue = alloc_percpu(struct iova_fq);
+	if (!queue)
+		return -ENOMEM;
+
+	for_each_possible_cpu(cpu)
+		iommu_dma_init_one_fq(per_cpu_ptr(queue, cpu));
+	cookie->percpu_fq = queue;
+	return 0;
  }
 
  /* sysfs updates are serialised by the mutex of the group owning @domain */
  int iommu_dma_init_fq(struct iommu_domain *domain)
  {
  	struct iommu_dma_cookie *cookie = domain->iova_cookie;
-	struct iova_fq __percpu *queue;
-	int i, cpu;
+	int rc;
 
  	if (cookie->fq_domain)
  		return 0;
@@ -251,26 +340,16 @@ int iommu_dma_init_fq(struct iommu_domain *domain)
  	atomic64_set(&cookie->fq_flush_start_cnt,  0);
  	atomic64_set(&cookie->fq_flush_finish_cnt, 0);
 
-	queue = alloc_percpu(struct iova_fq);
-	if (!queue) {
+	if (cookie->options.flags & IOMMU_DMA_OPTS_SINGLE_QUEUE)
+		rc = iommu_dma_init_fq_single(cookie);
+	else
+		rc = iommu_dma_init_fq_percpu(cookie);
+
+	if (rc) {
  		pr_warn("iova flush queue initialization failed\n");
  		return -ENOMEM;
  	}
 
-	for_each_possible_cpu(cpu) {
-		struct iova_fq *fq = per_cpu_ptr(queue, cpu);
-
-		fq->head = 0;
-		fq->tail = 0;
-
-		spin_lock_init(&fq->lock);
-
-		for (i = 0; i < IOVA_FQ_SIZE; i++)
-			INIT_LIST_HEAD(&fq->entries[i].freelist);
-	}
-
-	cookie->fq = queue;
-
  	timer_setup(&cookie->fq_timer, fq_flush_timeout, 0);
  	atomic_set(&cookie->fq_timer_on, 0);
  	/*

drivers/iommu/dma-iommu.c static struct iommu_dma_cookie *cookie_alloc(enum iommu_dma_cookie_type type) · line 379

Quoted 4 lines
drivers/iommu/dma-iommu.c +1 -0
@@ -297,6 +376,7 @@ static struct iommu_dma_cookie *cookie_alloc(enum iommu_dma_cookie_type type)
  	if (cookie) {
  		INIT_LIST_HEAD(&cookie->msi_page_list);
  		cookie->type = type;
+		cookie->options.flags = IOMMU_DMA_OPTS_PER_CPU_QUEUE;

You see, this confused me into thinking it does something and I had to go back and check ;)

Quoted 5 lines
drivers/iommu/dma-iommu.c +0 -0
  	}
  	return cookie;
  }
@@ -614,10 +694,18 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base,
  	if (ret)

drivers/iommu/dma-iommu.c static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, · line 700

Quoted 10 lines
drivers/iommu/dma-iommu.c +4 -4
  		goto done_unlock;
 
-	/* If the FQ fails we can simply fall back to strict mode */
-	if (domain->type == IOMMU_DOMAIN_DMA_FQ &&
-	    (!device_iommu_capable(dev, IOMMU_CAP_DEFERRED_FLUSH) || iommu_dma_init_fq(domain)))
-		domain->type = IOMMU_DOMAIN_DMA;
+	if (domain->type == IOMMU_DOMAIN_DMA_FQ) {
+		/* Expensive shadowing IOTLB flushes require some tuning */
+		if (dev->iommu->shadow_on_flush)
+			cookie->options.flags |= IOMMU_DMA_OPTS_SINGLE_QUEUE;

It would probably be better to set this regardless of the domain type, in case the FQ is only brought up later via sysfs.

drivers/iommu/dma-iommu.c static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, · line 706

Quoted 6 lines
drivers/iommu/dma-iommu.c +6 -0
+
+		/* If the FQ fails we can simply fall back to strict mode */
+		if (!device_iommu_capable(dev, IOMMU_CAP_DEFERRED_FLUSH) ||
+		    iommu_dma_init_fq(domain)) {
+			domain->type = IOMMU_DOMAIN_DMA;
+			cookie->options.flags &= ~IOMMU_DMA_OPTS_SINGLE_QUEUE;

...and either way there should be no need to clear it again - if it was true once it will always be true.

Cheers,
Robin.

Quoted 78 lines
drivers/iommu/dma-iommu.c +2 -0
+		}
+	}
 
  	ret = iova_reserve_iommu_regions(dev, domain);
 
drivers/iommu/iommu.c +4 -16
index fd9f79731d6a..caaf563d38ae 100644
@@ -2413,17 +2413,8 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova,
  		return -EINVAL;
 
  	ret = __iommu_map(domain, iova, paddr, size, prot, gfp);
-	if (ret == 0 && ops->iotlb_sync_map) {
-		ret = ops->iotlb_sync_map(domain, iova, size);
-		if (ret)
-			goto out_err;
-	}
-
-	return ret;
-
-out_err:
-	/* undo mappings already done */
-	iommu_unmap(domain, iova, size);
+	if (ret == 0 && ops->iotlb_sync_map)
+		ops->iotlb_sync_map(domain, iova, size);
 
  	return ret;
  }
@@ -2564,11 +2555,8 @@ ssize_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
  			sg = sg_next(sg);
  	}
 
-	if (ops->iotlb_sync_map) {
-		ret = ops->iotlb_sync_map(domain, iova, mapped);
-		if (ret)
-			goto out_err;
-	}
+	if (ops->iotlb_sync_map)
+		ops->iotlb_sync_map(domain, iova, mapped);
  	return mapped;
 
  out_err:
drivers/iommu/s390-iommu.c +3 -0
index 020cc538e4c4..63c56a440c9d 100644
@@ -468,6 +468,9 @@ static struct iommu_device *s390_iommu_probe_device(struct device *dev)
  	if (zdev->end_dma > ZPCI_TABLE_SIZE_RT - 1)
  		zdev->end_dma = ZPCI_TABLE_SIZE_RT - 1;
 
+	if (zdev->tlb_refresh)
+		dev->iommu->shadow_on_flush = 1;
+
  	return &zdev->iommu_dev;
  }
 
include/linux/iommu.h +2 -0
index 182cc4c71e62..c3687e066ed7 100644
@@ -409,6 +409,7 @@ struct iommu_fault_param {
   * @priv:	 IOMMU Driver private data
   * @max_pasids:  number of PASIDs this device can consume
   * @attach_deferred: the dma domain attachment is deferred
+ * @shadow_on_flush: IOTLB flushes are used to sync shadow tables
   *
   * TODO: migrate other per device data pointers under iommu_dev_data, e.g.
   *	struct iommu_group	*iommu_group;
@@ -422,6 +423,7 @@ struct dev_iommu {
  	void				*priv;
  	u32				max_pasids;
  	u32				attach_deferred:1;
+	u32				shadow_on_flush:1;
  };
 
  int iommu_device_register(struct iommu_device *iommu,
author maintainer
#10 lore raw

Re: [PATCH v11 5/6] iommu/dma: Allow a single FQ in addition to per-CPU FQs

Quoted 51 lines · Robin Murphy
Quoted 24 lines · Niklas Schnelle

In some virtualized environments, including s390 paged memory guests, IOTLB flushes are used to update IOMMU shadow tables. Due to this, they are much more expensive than in typical bare metal environments or non-paged s390 guests. In addition they may parallelize poorly in virtualized environments. This changes the trade off for flushing IOVAs such that minimizing the number of IOTLB flushes trumps any benefit of cheaper queuing operations or increased paralellism.

In this scenario per-CPU flush queues pose several problems. Firstly per-CPU memory is often quite limited prohibiting larger queues. Secondly collecting IOVAs per-CPU but flushing via a global timeout reduces the number of IOVAs flushed for each timeout especially on s390 where PCI interrupts may not be bound to a specific CPU.

Let's introduce a single flush queue mode that reuses the same queue logic but only allocates a single global queue. This mode is selected by dma-iommu if a newly introduced .shadow_on_flush flag is set in struct dev_iommu. As a first user the s390 IOMMU driver sets this flag during probe_device. With the unchanged small FQ size and timeouts this setting is worse than per-CPU queues but a follow up patch will make the FQ size and timeout variable. Together this allows the common IOVA flushing code to more closely resemble the global flush behavior used on s390's previous internal DMA API implementation.

Quoted 6 lines
---
  drivers/iommu/dma-iommu.c  | 178 +++++++++++++++++++++++++++++++++------------
  drivers/iommu/iommu.c      |  20 +----

The hunks in iommu.c appear to be an inadvertent reversion of patch #1?

A few bonus nits below which you can take or leave, but otherwise, with the rebase-mishap fixed:

Quoted 12 lines
  drivers/iommu/s390-iommu.c |   3 +
  include/linux/iommu.h      |   2 +
  4 files changed, 142 insertions(+), 61 deletions(-)
drivers/iommu/dma-iommu.c +0 -0
index e57724163835..4ada8b9749c9 100644
@@ -43,14 +43,23 @@ enum iommu_dma_cookie_type {
  	IOMMU_DMA_MSI_COOKIE,
  };
 
Quoted 7 lines

drivers/iommu/dma-iommu.c enum iommu_dma_cookie_type { · line 47

Quoted 2 lines
drivers/iommu/dma-iommu.c +2 -0
+struct dma_iommu_options {
+#define IOMMU_DMA_OPTS_PER_CPU_QUEUE	0L

Nit: if the intent is to add more flags then that will no longer make sense, and if not then we may as well just have a bool ;)

Quoted 1 line
drivers/iommu/dma-iommu.c +1 -0
+#define IOMMU_DMA_OPTS_SINGLE_QUEUE	BIT(0)

My thinking was that the above two options are mutually exclusive with per-CPU encoded as BIT(0) unset and single queue as set. Then other options could still use the other bits. It's true though that the below use of IOMMU_DMA_OPTS_PER_CPU_QUEUE is a nop so maybe just drop that? Or we could use an enum even if I don't forsee more than these 2 queue types.

Quoted 4 lines

drivers/iommu/dma-iommu.c enum iommu_dma_cookie_type {

Quoted 4 lines
drivers/iommu/dma-iommu.c +3 -0
+	u64	flags;
+};
+
  

---8<---

Quoted 53 lines
Quoted 40 lines
drivers/iommu/dma-iommu.c +21 -4
+static void fq_ring_free(struct iommu_dma_cookie *cookie, struct iova_fq *fq)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&fq->lock, flags);
+	fq_ring_free_locked(cookie, fq);
+	spin_unlock_irqrestore(&fq->lock, flags);
+}
+
  static void fq_flush_iotlb(struct iommu_dma_cookie *cookie)
  {
  	atomic64_inc(&cookie->fq_flush_start_cnt);
@@ -152,23 +172,29 @@ static void fq_flush_iotlb(struct iommu_dma_cookie *cookie)
  	atomic64_inc(&cookie->fq_flush_finish_cnt);
  }
 
+static void fq_flush_percpu(struct iommu_dma_cookie *cookie)
+{
+	int cpu;
+
+	for_each_possible_cpu(cpu) {
+		struct iova_fq *fq;
+
+		fq = per_cpu_ptr(cookie->percpu_fq, cpu);
+		fq_ring_free(cookie, fq);
+	}
+}
+
  static void fq_flush_timeout(struct timer_list *t)
  {
  	struct iommu_dma_cookie *cookie = from_timer(cookie, t, fq_timer);
-	int cpu;
 
  	atomic_set(&cookie->fq_timer_on, 0);
  	fq_flush_iotlb(cookie);
 
-	for_each_possible_cpu(cpu) {
-		unsigned long flags;
-		struct iova_fq *fq;

drivers/iommu/dma-iommu.c static void fq_flush_iotlb(struct iommu_dma_cookie *cookie) · line 197

Quoted 10 lines
drivers/iommu/dma-iommu.c +4 -6
-
-		fq = per_cpu_ptr(cookie->fq, cpu);
-		spin_lock_irqsave(&fq->lock, flags);
-		fq_ring_free(cookie, fq);
-		spin_unlock_irqrestore(&fq->lock, flags);
-	}
+	if (cookie->options.flags & IOMMU_DMA_OPTS_SINGLE_QUEUE)
+		fq_ring_free(cookie, cookie->single_fq);
+	else
+		fq_flush_percpu(cookie);

Nit: honestly I'd just inline that as:

Quoted 5 lines
	else for_each_possible_cpu(cpu)
		fq_ring_free(cookie, per_cpu_ptr(cookie->percpu_fq, cpu));

(possibly with extra braces if you don't feel brave enough to join the elite "else for" club...)

I might but it looks like checkpatch.pl isn't a fan:
...
ERROR: trailing statements should be on next line #119: FILE: drivers/iommu/dma-iommu.c:185:
+ else for_each_possible_cpu(cpu)

So braces it is.

Quoted 35 lines
Quoted 35 lines
drivers/iommu/dma-iommu.c +7 -3
  }
 
  static void queue_iova(struct iommu_dma_cookie *cookie,
@@ -188,7 +214,11 @@ static void queue_iova(struct iommu_dma_cookie *cookie,
  	 */
  	smp_mb();
 
-	fq = raw_cpu_ptr(cookie->fq);
+	if (cookie->options.flags & IOMMU_DMA_OPTS_SINGLE_QUEUE)
+		fq = cookie->single_fq;
+	else
+		fq = raw_cpu_ptr(cookie->percpu_fq);
+
  	spin_lock_irqsave(&fq->lock, flags);
 
  	/*
@@ -196,11 +226,11 @@ static void queue_iova(struct iommu_dma_cookie *cookie,
  	 * flushed out on another CPU. This makes the fq_full() check below less
  	 * likely to be true.
  	 */
-	fq_ring_free(cookie, fq);
+	fq_ring_free_locked(cookie, fq);
 
  	if (fq_full(fq)) {
  		fq_flush_iotlb(cookie);
-		fq_ring_free(cookie, fq);
+		fq_ring_free_locked(cookie, fq);
  	}
 
  	idx = fq_ring_add(fq);
@@ -219,31 +249,90 @@ static void queue_iova(struct iommu_dma_cookie *cookie,
  			  jiffies + msecs_to_jiffies(IOVA_FQ_TIMEOUT));
  }
 
Quoted 9 lines

drivers/iommu/dma-iommu.c static void queue_iova(struct iommu_dma_cookie *cookie, · line 257

Quoted 7 lines
drivers/iommu/dma-iommu.c +6 -1
-static void iommu_dma_free_fq(struct iommu_dma_cookie *cookie)
+static void iommu_dma_free_fq_single(struct iova_fq *fq)
+{
+	int idx;
+
+	if (!fq)
+		return;

Nit: That should never be true if cookie->fq_domain was set

True and the _percpu variant doesn't check it either so will drop.

Quoted 52 lines
Quoted 52 lines
drivers/iommu/dma-iommu.c +35 -6
+	fq_ring_for_each(idx, fq)
+		put_pages_list(&fq->entries[idx].freelist);
+	vfree(fq);
+}
+
+static void iommu_dma_free_fq_percpu(struct iova_fq __percpu *percpu_fq)
  {
  	int cpu, idx;
 
-	if (!cookie->fq)
-		return;
-
-	del_timer_sync(&cookie->fq_timer);
  	/* The IOVAs will be torn down separately, so just free our queued pages */
  	for_each_possible_cpu(cpu) {
-		struct iova_fq *fq = per_cpu_ptr(cookie->fq, cpu);
+		struct iova_fq *fq = per_cpu_ptr(percpu_fq, cpu);
 
  		fq_ring_for_each(idx, fq)
  			put_pages_list(&fq->entries[idx].freelist);
  	}
 
-	free_percpu(cookie->fq);
+	free_percpu(percpu_fq);
+}
+
+static void iommu_dma_free_fq(struct iommu_dma_cookie *cookie)
+{
+	if (!cookie->fq_domain)
+		return;
+
+	del_timer_sync(&cookie->fq_timer);
+	if (cookie->options.flags & IOMMU_DMA_OPTS_SINGLE_QUEUE)
+		iommu_dma_free_fq_single(cookie->single_fq);
+	else
+		iommu_dma_free_fq_percpu(cookie->percpu_fq);
+}
+
+static void iommu_dma_init_one_fq(struct iova_fq *fq)
+{
+	int i;
+
+	fq->head = 0;
+	fq->tail = 0;
+
+	spin_lock_init(&fq->lock);
+
+	for (i = 0; i < IOVA_FQ_SIZE; i++)
+		INIT_LIST_HEAD(&fq->entries[i].freelist);
+}
+
Quoted 8 lines

drivers/iommu/dma-iommu.c static void queue_iova(struct iommu_dma_cookie *cookie, · line 307

Quoted 5 lines
drivers/iommu/dma-iommu.c +5 -0
+static int iommu_dma_init_fq_single(struct iommu_dma_cookie *cookie)
+{
+	struct iova_fq *queue;
+
+	queue = vzalloc(sizeof(*queue));

Nit: vmalloc() - no need to zero the whole thing when the percpu path doesn't.

Done

Quoted 67 lines
Quoted 67 lines
drivers/iommu/dma-iommu.c +28 -18
+	if (!queue)
+		return -ENOMEM;
+	iommu_dma_init_one_fq(queue);
+	cookie->single_fq = queue;
+
+	return 0;
+}
+
+static int iommu_dma_init_fq_percpu(struct iommu_dma_cookie *cookie)
+{
+	struct iova_fq __percpu *queue;
+	int cpu;
+
+	queue = alloc_percpu(struct iova_fq);
+	if (!queue)
+		return -ENOMEM;
+
+	for_each_possible_cpu(cpu)
+		iommu_dma_init_one_fq(per_cpu_ptr(queue, cpu));
+	cookie->percpu_fq = queue;
+	return 0;
  }
 
  /* sysfs updates are serialised by the mutex of the group owning @domain */
  int iommu_dma_init_fq(struct iommu_domain *domain)
  {
  	struct iommu_dma_cookie *cookie = domain->iova_cookie;
-	struct iova_fq __percpu *queue;
-	int i, cpu;
+	int rc;
 
  	if (cookie->fq_domain)
  		return 0;
@@ -251,26 +340,16 @@ int iommu_dma_init_fq(struct iommu_domain *domain)
  	atomic64_set(&cookie->fq_flush_start_cnt,  0);
  	atomic64_set(&cookie->fq_flush_finish_cnt, 0);
 
-	queue = alloc_percpu(struct iova_fq);
-	if (!queue) {
+	if (cookie->options.flags & IOMMU_DMA_OPTS_SINGLE_QUEUE)
+		rc = iommu_dma_init_fq_single(cookie);
+	else
+		rc = iommu_dma_init_fq_percpu(cookie);
+
+	if (rc) {
  		pr_warn("iova flush queue initialization failed\n");
  		return -ENOMEM;
  	}
 
-	for_each_possible_cpu(cpu) {
-		struct iova_fq *fq = per_cpu_ptr(queue, cpu);
-
-		fq->head = 0;
-		fq->tail = 0;
-
-		spin_lock_init(&fq->lock);
-
-		for (i = 0; i < IOVA_FQ_SIZE; i++)
-			INIT_LIST_HEAD(&fq->entries[i].freelist);
-	}
-
-	cookie->fq = queue;
-
  	timer_setup(&cookie->fq_timer, fq_flush_timeout, 0);
  	atomic_set(&cookie->fq_timer_on, 0);
  	/*
Quoted 8 lines

drivers/iommu/dma-iommu.c static struct iommu_dma_cookie *cookie_alloc(enum iommu_dma_cookie_type type) · line 379

Quoted 5 lines
drivers/iommu/dma-iommu.c +1 -0
@@ -297,6 +376,7 @@ static struct iommu_dma_cookie *cookie_alloc(enum iommu_dma_cookie_type type)
  	if (cookie) {
  		INIT_LIST_HEAD(&cookie->msi_page_list);
  		cookie->type = type;
+		cookie->options.flags = IOMMU_DMA_OPTS_PER_CPU_QUEUE;

You see, this confused me into thinking it does something and I had to go back and check ;)

I have no real preference we can drop it, make it an enum, a bool or keep it. I felt like this explicit assignment documented per-CPU as the default so see my variant below but yes it does nothing since the cookie was already kzalloced.

Quoted 30 lines
Quoted 6 lines
drivers/iommu/dma-iommu.c +0 -0
  	}
  	return cookie;
  }
@@ -614,10 +694,18 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base,
  	if (ret)

drivers/iommu/dma-iommu.c static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, · line 700

Quoted 10 lines
drivers/iommu/dma-iommu.c +4 -4
  		goto done_unlock;
 
-	/* If the FQ fails we can simply fall back to strict mode */
-	if (domain->type == IOMMU_DOMAIN_DMA_FQ &&
-	    (!device_iommu_capable(dev, IOMMU_CAP_DEFERRED_FLUSH) || iommu_dma_init_fq(domain)))
-		domain->type = IOMMU_DOMAIN_DMA;
+	if (domain->type == IOMMU_DOMAIN_DMA_FQ) {
+		/* Expensive shadowing IOTLB flushes require some tuning */
+		if (dev->iommu->shadow_on_flush)
+			cookie->options.flags |= IOMMU_DMA_OPTS_SINGLE_QUEUE;

It would probably be better to set this regardless of the domain type, in case the FQ is only brought up later via sysfs.

drivers/iommu/dma-iommu.c static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, · line 706

Quoted 6 lines
drivers/iommu/dma-iommu.c +6 -0
+
+		/* If the FQ fails we can simply fall back to strict mode */
+		if (!device_iommu_capable(dev, IOMMU_CAP_DEFERRED_FLUSH) ||
+		    iommu_dma_init_fq(domain)) {
+			domain->type = IOMMU_DOMAIN_DMA;
+			cookie->options.flags &= ~IOMMU_DMA_OPTS_SINGLE_QUEUE;

...and either way there should be no need to clear it again - if it was true once it will always be true.

Quoted 3 lines

Cheers,
Robin.

I'm now experimenting with an iommu_dma_init_options() helper that gets called from the main path of iommu_dma_init() here is how it looks like at the end:

static void iommu_dma_init_options(struct iommu_dma_options *options, struct device *dev)

{
	/* Expensive shadowing IOTLB flushes do better with a single large queue */
	if (dev->iommu->shadow_on_flush) {
		options->flags = IOMMU_DMA_OPTS_SINGLE_QUEUE;
		options->fq_timeout = IOVA_SINGLE_FQ_TIMEOUT;
		options->fq_size = IOVA_SINGLE_FQ_SIZE;
	} else {
		options->flags = IOMMU_DMA_OPTS_PER_CPU_QUEUE;
		options->fq_size = IOVA_DEFAULT_FQ_SIZE;
		options->fq_timeout = IOVA_DEFAULT_FQ_TIMEOUT;
	}
}

Also I noticed that my options struct used the prefix "dma_iommu_" while everything else is "iommu_dma_" so that got fixed too.

Quoted 33 lines
Quoted 33 lines
drivers/iommu/dma-iommu.c +2 -0
+		}
+	}
 
  	ret = iova_reserve_iommu_regions(dev, domain);
 
drivers/iommu/iommu.c +2 -11
index fd9f79731d6a..caaf563d38ae 100644
@@ -2413,17 +2413,8 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova,
  		return -EINVAL;
 
  	ret = __iommu_map(domain, iova, paddr, size, prot, gfp);
-	if (ret == 0 && ops->iotlb_sync_map) {
-		ret = ops->iotlb_sync_map(domain, iova, size);
-		if (ret)
-			goto out_err;
-	}
-
-	return ret;
-
-out_err:
-	/* undo mappings already done */
-	iommu_unmap(domain, iova, size);
+	if (ret == 0 && ops->iotlb_sync_map)
+		ops->iotlb_sync_map(domain, iova, size);
 
  	return ret;
  }
@@ -2564,11 +2555,8 @@ ssize_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
  			sg = sg_next(sg);
  	}
 
Quoted 8 lines

drivers/iommu/iommu.c ssize_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, · line 2560

Quoted 8 lines
drivers/iommu/iommu.c +2 -5
-	if (ops->iotlb_sync_map) {
-		ret = ops->iotlb_sync_map(domain, iova, mapped);
-		if (ret)
-			goto out_err;
-	}
+	if (ops->iotlb_sync_map)
+		ops->iotlb_sync_map(domain, iova, mapped);
  	return mapped;

Good find that is indeed a rebase mishap ;-(

Quoted 37 lines
Quoted 37 lines
drivers/iommu/iommu.c +0 -0
  out_err:
drivers/iommu/s390-iommu.c +3 -0
index 020cc538e4c4..63c56a440c9d 100644
@@ -468,6 +468,9 @@ static struct iommu_device *s390_iommu_probe_device(struct device *dev)
  	if (zdev->end_dma > ZPCI_TABLE_SIZE_RT - 1)
  		zdev->end_dma = ZPCI_TABLE_SIZE_RT - 1;
 
+	if (zdev->tlb_refresh)
+		dev->iommu->shadow_on_flush = 1;
+
  	return &zdev->iommu_dev;
  }
 
include/linux/iommu.h +2 -0
index 182cc4c71e62..c3687e066ed7 100644
@@ -409,6 +409,7 @@ struct iommu_fault_param {
   * @priv:	 IOMMU Driver private data
   * @max_pasids:  number of PASIDs this device can consume
   * @attach_deferred: the dma domain attachment is deferred
+ * @shadow_on_flush: IOTLB flushes are used to sync shadow tables
   *
   * TODO: migrate other per device data pointers under iommu_dev_data, e.g.
   *	struct iommu_group	*iommu_group;
@@ -422,6 +423,7 @@ struct dev_iommu {
  	void				*priv;
  	u32				max_pasids;
  	u32				attach_deferred:1;
+	u32				shadow_on_flush:1;
  };
 
  int iommu_device_register(struct iommu_device *iommu,
maintainer
#11 lore raw

Re: [PATCH v11 5/6] iommu/dma: Allow a single FQ in addition to per-CPU FQs

On 2023-08-23 15:21, Niklas Schnelle wrote:
[...]

Quoted 6 lines
Quoted 6 lines

drivers/iommu/dma-iommu.c enum iommu_dma_cookie_type { · line 47

Quoted 2 lines
drivers/iommu/dma-iommu.c +2 -0
+struct dma_iommu_options {
+#define IOMMU_DMA_OPTS_PER_CPU_QUEUE	0L

Nit: if the intent is to add more flags then that will no longer make sense, and if not then we may as well just have a bool ;)

Quoted 8 lines
Quoted 1 line
Quoted 1 line

+#define IOMMU_DMA_OPTS_SINGLE_QUEUE BIT(0)

My thinking was that the above two options are mutually exclusive with per-CPU encoded as BIT(0) unset and single queue as set. Then other options could still use the other bits. It's true though that the below use of IOMMU_DMA_OPTS_PER_CPU_QUEUE is a nop so maybe just drop that? Or we could use an enum even if I don't forsee more than these 2 queue types.

My point was that the value 0 can only mean "all flags not set", so while we can very much have the semantic of "single queue flag not set means percpu queue", we cannot infer "0 means percpu queue" unless "all flags" and "single queue flag" are the same thing. As soon as any additional flag is defined, 0 then has a different meaning which may well not even be a combination that's useful to put a specific name to.

I'd like to hope it's sufficiently obvious from the implementation that the opposite of a single queue is multiple queues, since contextually this is already all happening in distinct paths from the case of no queue.

Thanks,
Robin.

author maintainer
#12 lore raw

PATCH v11 6/6 iommu/dma: Use a large flush queue and timeout for shadow_on_flush

In reply to Niklas Schnelle

Flush queues currently use a fixed compile time size of 256 entries. This being a power of 2 allows the compiler to use shift and mask instead of more expensive modulo operations. With per-CPU flush queues larger queue sizes would hit per-CPU allocation limits, with a single flush queue these limits do not apply however. Also with single queues being particularly suitable for virtualized environments with expensive IOTLB flushes these benefit especially from larger queues and thus fewer flushes.

To this end re-order struct iova_fq so we can use a dynamic array and introduce the flush queue size and timeouts as new options in the dma_iommu_options struct. So as not to lose the shift and mask optimization, use a power of 2 for the length and use explicit shift and mask instead of letting the compiler optimize this.

A large queue size and 1 second timeout is then set for the shadow on flush case set by s390 paged memory guests. This then brings performance on par with the previous s390 specific DMA API implementation.

---
 drivers/iommu/dma-iommu.c | 50 +++++++++++++++++++++++++++++++----------------
 1 file changed, 33 insertions(+), 17 deletions(-)
drivers/iommu/dma-iommu.c +33 -17
index 4ada8b9749c9..bebc0804ff53 100644
@@ -46,7 +46,9 @@ enum iommu_dma_cookie_type {
 struct dma_iommu_options {
 #define IOMMU_DMA_OPTS_PER_CPU_QUEUE	0L
 #define IOMMU_DMA_OPTS_SINGLE_QUEUE	BIT(0)
-	u64	flags;
+	u64		flags;
+	size_t		fq_size;
+	unsigned int	fq_timeout;
 };
 
 struct iommu_dma_cookie {
@@ -95,10 +97,12 @@ static int __init iommu_dma_forcedac_setup(char *str)
 early_param("iommu.forcedac", iommu_dma_forcedac_setup);
 
 /* Number of entries per flush queue */
-#define IOVA_FQ_SIZE	256
+#define IOVA_DEFAULT_FQ_SIZE	256
+#define IOVA_SINGLE_FQ_SIZE	32768
 
 /* Timeout (in ms) after which entries are flushed from the queue */
-#define IOVA_FQ_TIMEOUT	10
+#define IOVA_DEFAULT_FQ_TIMEOUT	10
+#define IOVA_SINGLE_FQ_TIMEOUT	1000
 
 /* Flush queue entry for deferred flushing */
 struct iova_fq_entry {
@@ -110,18 +114,19 @@ struct iova_fq_entry {
 
 /* Per-CPU flush queue structure */
 struct iova_fq {
-	struct iova_fq_entry entries[IOVA_FQ_SIZE];
-	unsigned int head, tail;
 	spinlock_t lock;
+	unsigned int head, tail;
+	unsigned int mod_mask;
+	struct iova_fq_entry entries[];
 };
 
 #define fq_ring_for_each(i, fq) \
-	for ((i) = (fq)->head; (i) != (fq)->tail; (i) = ((i) + 1) % IOVA_FQ_SIZE)
+	for ((i) = (fq)->head; (i) != (fq)->tail; (i) = ((i) + 1) & (fq)->mod_mask)
 
 static inline bool fq_full(struct iova_fq *fq)
 {
 	assert_spin_locked(&fq->lock);
-	return (((fq->tail + 1) % IOVA_FQ_SIZE) == fq->head);
+	return (((fq->tail + 1) & fq->mod_mask) == fq->head);
 }
 
 static inline unsigned int fq_ring_add(struct iova_fq *fq)
@@ -130,7 +135,7 @@ static inline unsigned int fq_ring_add(struct iova_fq *fq)
 
 	assert_spin_locked(&fq->lock);
 
-	fq->tail = (idx + 1) % IOVA_FQ_SIZE;
+	fq->tail = (idx + 1) & fq->mod_mask;
 
 	return idx;
 }
@@ -152,7 +157,7 @@ static void fq_ring_free_locked(struct iommu_dma_cookie *cookie, struct iova_fq
 			       fq->entries[idx].iova_pfn,
 			       fq->entries[idx].pages);
 
-		fq->head = (fq->head + 1) % IOVA_FQ_SIZE;
+		fq->head = (fq->head + 1) & fq->mod_mask;
 	}
 }
 
@@ -246,7 +251,7 @@ static void queue_iova(struct iommu_dma_cookie *cookie,
 	if (!atomic_read(&cookie->fq_timer_on) &&
 	    !atomic_xchg(&cookie->fq_timer_on, 1))
 		mod_timer(&cookie->fq_timer,
-			  jiffies + msecs_to_jiffies(IOVA_FQ_TIMEOUT));
+			  jiffies + msecs_to_jiffies(cookie->options.fq_timeout));
 }
 
 static void iommu_dma_free_fq_single(struct iova_fq *fq)
@@ -287,27 +292,29 @@ static void iommu_dma_free_fq(struct iommu_dma_cookie *cookie)
 		iommu_dma_free_fq_percpu(cookie->percpu_fq);
 }
 
-static void iommu_dma_init_one_fq(struct iova_fq *fq)
+static void iommu_dma_init_one_fq(struct iova_fq *fq, size_t fq_size)
 {
 	int i;
 
 	fq->head = 0;
 	fq->tail = 0;
+	fq->mod_mask = fq_size - 1;
 
 	spin_lock_init(&fq->lock);
 
-	for (i = 0; i < IOVA_FQ_SIZE; i++)
+	for (i = 0; i < fq_size; i++)
 		INIT_LIST_HEAD(&fq->entries[i].freelist);
 }
 
 static int iommu_dma_init_fq_single(struct iommu_dma_cookie *cookie)
 {
+	size_t fq_size = cookie->options.fq_size;
 	struct iova_fq *queue;
 
-	queue = vzalloc(sizeof(*queue));
+	queue = vzalloc(struct_size(queue, entries, fq_size));
 	if (!queue)
 		return -ENOMEM;
-	iommu_dma_init_one_fq(queue);
+	iommu_dma_init_one_fq(queue, fq_size);
 	cookie->single_fq = queue;
 
 	return 0;
@@ -315,15 +322,17 @@ static int iommu_dma_init_fq_single(struct iommu_dma_cookie *cookie)
 
 static int iommu_dma_init_fq_percpu(struct iommu_dma_cookie *cookie)
 {
+	size_t fq_size = cookie->options.fq_size;
 	struct iova_fq __percpu *queue;
 	int cpu;
 
-	queue = alloc_percpu(struct iova_fq);
+	queue = __alloc_percpu(struct_size(queue, entries, fq_size),
+			       __alignof__(*queue));
 	if (!queue)
 		return -ENOMEM;
 
 	for_each_possible_cpu(cpu)
-		iommu_dma_init_one_fq(per_cpu_ptr(queue, cpu));
+		iommu_dma_init_one_fq(per_cpu_ptr(queue, cpu), fq_size);
 	cookie->percpu_fq = queue;
 	return 0;
 }
@@ -377,6 +386,8 @@ static struct iommu_dma_cookie *cookie_alloc(enum iommu_dma_cookie_type type)
 		INIT_LIST_HEAD(&cookie->msi_page_list);
 		cookie->type = type;
 		cookie->options.flags = IOMMU_DMA_OPTS_PER_CPU_QUEUE;
+		cookie->options.fq_size = IOVA_DEFAULT_FQ_SIZE;
+		cookie->options.fq_timeout = IOVA_DEFAULT_FQ_TIMEOUT;
 	}
 	return cookie;
 }
@@ -696,14 +707,19 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base,
 
 	if (domain->type == IOMMU_DOMAIN_DMA_FQ) {
 		/* Expensive shadowing IOTLB flushes require some tuning */
-		if (dev->iommu->shadow_on_flush)
+		if (dev->iommu->shadow_on_flush) {
 			cookie->options.flags |= IOMMU_DMA_OPTS_SINGLE_QUEUE;
+			cookie->options.fq_timeout = IOVA_SINGLE_FQ_TIMEOUT;
+			cookie->options.fq_size = IOVA_SINGLE_FQ_SIZE;
+		}
 
 		/* If the FQ fails we can simply fall back to strict mode */
 		if (!device_iommu_capable(dev, IOMMU_CAP_DEFERRED_FLUSH) ||
 		    iommu_dma_init_fq(domain)) {
 			domain->type = IOMMU_DOMAIN_DMA;
 			cookie->options.flags &= ~IOMMU_DMA_OPTS_SINGLE_QUEUE;
+			cookie->options.fq_timeout = IOVA_DEFAULT_FQ_TIMEOUT;
+			cookie->options.fq_size = IOVA_DEFAULT_FQ_SIZE;
 		}
 	}
 
Signature · 1 line
2.39.2
maintainer Acked-by on v11 6/6
#13 lore raw

Re: [PATCH v11 6/6] iommu/dma: Use a large flush queue and timeout for shadow_on_flush

Quoted 161 lines · Niklas Schnelle

Flush queues currently use a fixed compile time size of 256 entries. This being a power of 2 allows the compiler to use shift and mask instead of more expensive modulo operations. With per-CPU flush queues larger queue sizes would hit per-CPU allocation limits, with a single flush queue these limits do not apply however. Also with single queues being particularly suitable for virtualized environments with expensive IOTLB flushes these benefit especially from larger queues and thus fewer flushes.

To this end re-order struct iova_fq so we can use a dynamic array and introduce the flush queue size and timeouts as new options in the dma_iommu_options struct. So as not to lose the shift and mask optimization, use a power of 2 for the length and use explicit shift and mask instead of letting the compiler optimize this.

A large queue size and 1 second timeout is then set for the shadow on flush case set by s390 paged memory guests. This then brings performance on par with the previous s390 specific DMA API implementation.

---
  drivers/iommu/dma-iommu.c | 50 +++++++++++++++++++++++++++++++----------------
  1 file changed, 33 insertions(+), 17 deletions(-)
drivers/iommu/dma-iommu.c +25 -16
index 4ada8b9749c9..bebc0804ff53 100644
@@ -46,7 +46,9 @@ enum iommu_dma_cookie_type {
  struct dma_iommu_options {
  #define IOMMU_DMA_OPTS_PER_CPU_QUEUE	0L
  #define IOMMU_DMA_OPTS_SINGLE_QUEUE	BIT(0)
-	u64	flags;
+	u64		flags;
+	size_t		fq_size;
+	unsigned int	fq_timeout;
  };
 
  struct iommu_dma_cookie {
@@ -95,10 +97,12 @@ static int __init iommu_dma_forcedac_setup(char *str)
  early_param("iommu.forcedac", iommu_dma_forcedac_setup);
 
  /* Number of entries per flush queue */
-#define IOVA_FQ_SIZE	256
+#define IOVA_DEFAULT_FQ_SIZE	256
+#define IOVA_SINGLE_FQ_SIZE	32768
 
  /* Timeout (in ms) after which entries are flushed from the queue */
-#define IOVA_FQ_TIMEOUT	10
+#define IOVA_DEFAULT_FQ_TIMEOUT	10
+#define IOVA_SINGLE_FQ_TIMEOUT	1000
 
  /* Flush queue entry for deferred flushing */
  struct iova_fq_entry {
@@ -110,18 +114,19 @@ struct iova_fq_entry {
 
  /* Per-CPU flush queue structure */
  struct iova_fq {
-	struct iova_fq_entry entries[IOVA_FQ_SIZE];
-	unsigned int head, tail;
  	spinlock_t lock;
+	unsigned int head, tail;
+	unsigned int mod_mask;
+	struct iova_fq_entry entries[];
  };
 
  #define fq_ring_for_each(i, fq) \
-	for ((i) = (fq)->head; (i) != (fq)->tail; (i) = ((i) + 1) % IOVA_FQ_SIZE)
+	for ((i) = (fq)->head; (i) != (fq)->tail; (i) = ((i) + 1) & (fq)->mod_mask)
 
  static inline bool fq_full(struct iova_fq *fq)
  {
  	assert_spin_locked(&fq->lock);
-	return (((fq->tail + 1) % IOVA_FQ_SIZE) == fq->head);
+	return (((fq->tail + 1) & fq->mod_mask) == fq->head);
  }
 
  static inline unsigned int fq_ring_add(struct iova_fq *fq)
@@ -130,7 +135,7 @@ static inline unsigned int fq_ring_add(struct iova_fq *fq)
 
  	assert_spin_locked(&fq->lock);
 
-	fq->tail = (idx + 1) % IOVA_FQ_SIZE;
+	fq->tail = (idx + 1) & fq->mod_mask;
 
  	return idx;
  }
@@ -152,7 +157,7 @@ static void fq_ring_free_locked(struct iommu_dma_cookie *cookie, struct iova_fq
  			       fq->entries[idx].iova_pfn,
  			       fq->entries[idx].pages);
 
-		fq->head = (fq->head + 1) % IOVA_FQ_SIZE;
+		fq->head = (fq->head + 1) & fq->mod_mask;
  	}
  }
 
@@ -246,7 +251,7 @@ static void queue_iova(struct iommu_dma_cookie *cookie,
  	if (!atomic_read(&cookie->fq_timer_on) &&
  	    !atomic_xchg(&cookie->fq_timer_on, 1))
  		mod_timer(&cookie->fq_timer,
-			  jiffies + msecs_to_jiffies(IOVA_FQ_TIMEOUT));
+			  jiffies + msecs_to_jiffies(cookie->options.fq_timeout));
  }
 
  static void iommu_dma_free_fq_single(struct iova_fq *fq)
@@ -287,27 +292,29 @@ static void iommu_dma_free_fq(struct iommu_dma_cookie *cookie)
  		iommu_dma_free_fq_percpu(cookie->percpu_fq);
  }
 
-static void iommu_dma_init_one_fq(struct iova_fq *fq)
+static void iommu_dma_init_one_fq(struct iova_fq *fq, size_t fq_size)
  {
  	int i;
 
  	fq->head = 0;
  	fq->tail = 0;
+	fq->mod_mask = fq_size - 1;
 
  	spin_lock_init(&fq->lock);
 
-	for (i = 0; i < IOVA_FQ_SIZE; i++)
+	for (i = 0; i < fq_size; i++)
  		INIT_LIST_HEAD(&fq->entries[i].freelist);
  }
 
  static int iommu_dma_init_fq_single(struct iommu_dma_cookie *cookie)
  {
+	size_t fq_size = cookie->options.fq_size;
  	struct iova_fq *queue;
 
-	queue = vzalloc(sizeof(*queue));
+	queue = vzalloc(struct_size(queue, entries, fq_size));
  	if (!queue)
  		return -ENOMEM;
-	iommu_dma_init_one_fq(queue);
+	iommu_dma_init_one_fq(queue, fq_size);
  	cookie->single_fq = queue;
 
  	return 0;
@@ -315,15 +322,17 @@ static int iommu_dma_init_fq_single(struct iommu_dma_cookie *cookie)
 
  static int iommu_dma_init_fq_percpu(struct iommu_dma_cookie *cookie)
  {
+	size_t fq_size = cookie->options.fq_size;
  	struct iova_fq __percpu *queue;
  	int cpu;
 
-	queue = alloc_percpu(struct iova_fq);
+	queue = __alloc_percpu(struct_size(queue, entries, fq_size),
+			       __alignof__(*queue));
  	if (!queue)
  		return -ENOMEM;
 
  	for_each_possible_cpu(cpu)
-		iommu_dma_init_one_fq(per_cpu_ptr(queue, cpu));
+		iommu_dma_init_one_fq(per_cpu_ptr(queue, cpu), fq_size);
  	cookie->percpu_fq = queue;
  	return 0;
  }

drivers/iommu/dma-iommu.c static struct iommu_dma_cookie *cookie_alloc(enum iommu_dma_cookie_type type) · line 390

Quoted 5 lines
drivers/iommu/dma-iommu.c +2 -0
@@ -377,6 +386,8 @@ static struct iommu_dma_cookie *cookie_alloc(enum iommu_dma_cookie_type type)
  		INIT_LIST_HEAD(&cookie->msi_page_list);
  		cookie->type = type;
  		cookie->options.flags = IOMMU_DMA_OPTS_PER_CPU_QUEUE;
+		cookie->options.fq_size = IOVA_DEFAULT_FQ_SIZE;
+		cookie->options.fq_timeout = IOVA_DEFAULT_FQ_TIMEOUT;

Similar comment as on the previous patch - it would be clearer and more robust to set fq_size and fq_timeout all in one place in the main path of iommu_dma_init_domain(). Otherwise,

Quoted 25 lines
drivers/iommu/dma-iommu.c +6 -1
  	}
  	return cookie;
  }
@@ -696,14 +707,19 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base,
 
  	if (domain->type == IOMMU_DOMAIN_DMA_FQ) {
  		/* Expensive shadowing IOTLB flushes require some tuning */
-		if (dev->iommu->shadow_on_flush)
+		if (dev->iommu->shadow_on_flush) {
  			cookie->options.flags |= IOMMU_DMA_OPTS_SINGLE_QUEUE;
+			cookie->options.fq_timeout = IOVA_SINGLE_FQ_TIMEOUT;
+			cookie->options.fq_size = IOVA_SINGLE_FQ_SIZE;
+		}
 
  		/* If the FQ fails we can simply fall back to strict mode */
  		if (!device_iommu_capable(dev, IOMMU_CAP_DEFERRED_FLUSH) ||
  		    iommu_dma_init_fq(domain)) {
  			domain->type = IOMMU_DOMAIN_DMA;
  			cookie->options.flags &= ~IOMMU_DMA_OPTS_SINGLE_QUEUE;
+			cookie->options.fq_timeout = IOVA_DEFAULT_FQ_TIMEOUT;
+			cookie->options.fq_size = IOVA_DEFAULT_FQ_SIZE;
  		}
  	}
 
author maintainer
#14 lore raw

RFC 1/1 iommu/virtio: Use single flush queue (EXPERIMENTAL)

In reply to Niklas Schnelle

Just like on paged s390 guests with their virtual IOMMU, syncing mappings via virtio-iommu is quite expensive. It can thus benefit from queueing unmapped IOVAs and flushing them in batches but less so from parallel flushes which is what the shadow_on_flush flag introduced for s390 tunes dma-iommu to do.

For this to work .flush_iotlb_all is implemented. Furthermore .iotlb_sync_map is also implemented and used to pull the sync out of the mapping operation for some additional batching and performance gain.

In a basic test with NVMe pass-through to a KVM guest on a Ryzen 3900X these changes together lead to about 19% more IOPS in a fio test and slightly more bandwidth too.

---
Note:
The idea of using the single flush queue scheme from my series "iommu/dma: s390 DMA API conversion and optimized IOTLB flushing"[0] for virtio-iommu was already mentioned in the cover letter. I now wanted to explore this with this patch which may also serve as a test vehicle for the single flush queue scheme usable on non-s390.

Besides limited testing, this is marked experimental mainly because the use of queuing needs to be a concious decision as it allows continued access to unmapped pages for up to a second with the currently proposed single flush queue mechanism. Also it might make sense to split this patch to do the introduction and use of .iotlb_sync_map separately but as a test vehicle I found it easier to consume as a single patch.

[0]: https://lore.kernel.org/linux-iommu/[email protected]/

 drivers/iommu/virtio-iommu.c | 28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)
drivers/iommu/virtio-iommu.c +27 -1
index 3551ed057774..f29eb4ce2b88 100644
@@ -843,7 +843,7 @@ static int viommu_map_pages(struct iommu_domain *domain, unsigned long iova,
 			.flags		= cpu_to_le32(flags),
 		};
 
-		ret = viommu_send_req_sync(vdomain->viommu, &map, sizeof(map));
+		ret = viommu_add_req(vdomain->viommu, &map, sizeof(map));
 		if (ret) {
 			viommu_del_mappings(vdomain, iova, end);
 			return ret;
@@ -909,6 +909,27 @@ static void viommu_iotlb_sync(struct iommu_domain *domain,
 {
 	struct viommu_domain *vdomain = to_viommu_domain(domain);
 
+	if (!vdomain->nr_endpoints)
+		return;
+	viommu_sync_req(vdomain->viommu);
+}
+
+static int viommu_iotlb_sync_map(struct iommu_domain *domain,
+				 unsigned long iova, size_t size)
+{
+	struct viommu_domain *vdomain = to_viommu_domain(domain);
+
+	if (!vdomain->nr_endpoints)
+		return 0;
+	return viommu_sync_req(vdomain->viommu);
+}
+
+static void viommu_flush_iotlb_all(struct iommu_domain *domain)
+{
+	struct viommu_domain *vdomain = to_viommu_domain(domain);
+
+	if (!vdomain->nr_endpoints)
+		return;
 	viommu_sync_req(vdomain->viommu);
 }
 
@@ -991,6 +1012,7 @@ static struct iommu_device *viommu_probe_device(struct device *dev)
 		if (ret)
 			goto err_free_dev;
 	}
+	dev->iommu->shadow_on_flush = 1;
 
 	return &viommu->iommu;
 
@@ -1037,6 +1059,8 @@ static bool viommu_capable(struct device *dev, enum iommu_cap cap)
 	switch (cap) {
 	case IOMMU_CAP_CACHE_COHERENCY:
 		return true;
+	case IOMMU_CAP_DEFERRED_FLUSH:
+		return true;
 	default:
 		return false;
 	}
@@ -1057,7 +1081,9 @@ static struct iommu_ops viommu_ops = {
 		.map_pages		= viommu_map_pages,
 		.unmap_pages		= viommu_unmap_pages,
 		.iova_to_phys		= viommu_iova_to_phys,
+		.flush_iotlb_all	= viommu_flush_iotlb_all,
 		.iotlb_sync		= viommu_iotlb_sync,
+		.iotlb_sync_map		= viommu_iotlb_sync_map,
 		.free			= viommu_domain_free,
 	}
 };

base-commit: 5514392fe77cd45b0d33bf239f13ba594a6759e5

Signature · 1 line
2.39.2
#15 lore raw

Re: [RFC] iommu/virtio: Use single flush queue (EXPERIMENTAL)

Hi Niklas,

Quoted 6 lines · Niklas Schnelle

Just like on paged s390 guests with their virtual IOMMU, syncing mappings via virtio-iommu is quite expensive. It can thus benefit from queueing unmapped IOVAs and flushing them in batches but less so from parallel flushes which is what the shadow_on_flush flag introduced for s390 tunes dma-iommu to do.

Quoted 7 lines

For this to work .flush_iotlb_all is implemented. Furthermore .iotlb_sync_map is also implemented and used to pull the sync out of the mapping operation for some additional batching and performance gain.

In a basic test with NVMe pass-through to a KVM guest on a Ryzen 3900X these changes together lead to about 19% more IOPS in a fio test and slightly more bandwidth too.

Nice, thank you for testing this. I played with a NVMe on an Intel desktop and can confirm similar results. With "sq" meaning single flush queue and "mq" percpu flush queue, "+map" is with .iotlb_sync_map() enabled.

Multithread block randwrite job [1]:

BW compared to host Confidence
(higher better)
host 100.0% ±0.0%
noviommu 99.9 0.0
viommu lazy sq +map 99.9 0.1
viommu lazy mq +map 99.9 0.1
viommu lazy sq 92.2 0.9
viommu lazy mq 91.5 0.9
viommu strict +map 92.7 0.9
viommu strict 81.3 1.0

Single page randrw:

Latency compared to host Confidence
(lower better)
host x1.00 ±.04
noviommu 1.23 .04
viommu lazy sq +map 7.09 .05
viommu lazy mq +map 7.07 .07
viommu lazy sq 7.15 .04
viommu lazy mq 7.11 .05
viommu strict +map 8.82 .05
viommu strict 8.82 .04

So with lazy+map we get the maximum bandwidth reachable on this disk (2.5GiB/s) even with a heavy iommu_map/unmap usage, which is cool. Random access latency also improves with lazy mode.

The difference between single and percpu flush queue isn't really measurable in my multithread test. There is a difference between Lazy sq and mq but the variation between samples outweighs it.

Quoted 10 lines

---
Note:
The idea of using the single flush queue scheme from my series "iommu/dma: s390 DMA API conversion and optimized IOTLB flushing"[0] for virtio-iommu was already mentioned in the cover letter. I now wanted to explore this with this patch which may also serve as a test vehicle for the single flush queue scheme usable on non-s390.

Quoted 4 lines

Besides limited testing, this is marked experimental mainly because the use of queuing needs to be a concious decision as it allows continued access to unmapped pages for up to a second with the currently proposed single flush queue mechanism.

It fits with the iommu.strict=0 / CONFIG_IOMMU_DEFAULT_DMA_LAZY setting, which selects DMA_FQ domains. That option allows a misbehaving device to access memory that has been freed/reallocated, which is what we're enabling here. I believe the risk is pretty much the same for deferred UNMAP as for deferred TLBI, since mappings that we're removing were likely cached in the TLB. Increasing the timeout does make it easier to exploit, but I don't think that changes the policy from an admin's perspective:
only enable lazy mode if you trust device and driver.

On bare metal, we disable DMA_FQ for devices that can be easily hotplugged into unattended machines (through external-facing ports like thunderbolt). On VMs, the concern isn't really about external devices, since they don't automatically get plugged into a VM without user intervention. Here I guess the devices we don't trust will be virtual devices implemented by other VMs. We don't have any method to identify them yet, so iommu.strict=1 and CONFIG_IOMMU_DEFAULT_DMA_STRICT is the best we can do at the moment.

I'm not so sure about enabling shadow_on_flush by default, since the performance difference was too small in my tests. Maybe a module parameter for dma-iommu could configure the flush queue?

Quoted 3 lines

Also it might make sense to split this patch to do the introduction and use of .iotlb_sync_map separately but as a test vehicle I found it easier to consume as a single patch.

Yes, both changes are useful but should be in two patches

Thanks,
Jean

[1] (Running one fio process for each of the two tests, sequentially, 30

    or more times, with 1 warmup.)

    [global]
    filename=/dev/nvme0n1
    runtime=10
    ioengine=libaio
    direct=1
    time_based

    [randwrite_multi]
    group_reporting
    rw=randwrite
    numjobs=16
    iodepth=16
    bs=64k
    size=1G

    [randrw_page]
    rw=randrw
    rwmixread=50
    rwmixwrite=50
    iodepth=128
    bs=4k
    size=1G
Quoted 75 lines

[0]: https://lore.kernel.org/linux-iommu/[email protected]/

 drivers/iommu/virtio-iommu.c | 28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)
drivers/iommu/virtio-iommu.c +27 -1
index 3551ed057774..f29eb4ce2b88 100644
@@ -843,7 +843,7 @@ static int viommu_map_pages(struct iommu_domain *domain, unsigned long iova,
 			.flags		= cpu_to_le32(flags),
 		};
 
-		ret = viommu_send_req_sync(vdomain->viommu, &map, sizeof(map));
+		ret = viommu_add_req(vdomain->viommu, &map, sizeof(map));
 		if (ret) {
 			viommu_del_mappings(vdomain, iova, end);
 			return ret;
@@ -909,6 +909,27 @@ static void viommu_iotlb_sync(struct iommu_domain *domain,
 {
 	struct viommu_domain *vdomain = to_viommu_domain(domain);
 
+	if (!vdomain->nr_endpoints)
+		return;
+	viommu_sync_req(vdomain->viommu);
+}
+
+static int viommu_iotlb_sync_map(struct iommu_domain *domain,
+				 unsigned long iova, size_t size)
+{
+	struct viommu_domain *vdomain = to_viommu_domain(domain);
+
+	if (!vdomain->nr_endpoints)
+		return 0;
+	return viommu_sync_req(vdomain->viommu);
+}
+
+static void viommu_flush_iotlb_all(struct iommu_domain *domain)
+{
+	struct viommu_domain *vdomain = to_viommu_domain(domain);
+
+	if (!vdomain->nr_endpoints)
+		return;
 	viommu_sync_req(vdomain->viommu);
 }
 
@@ -991,6 +1012,7 @@ static struct iommu_device *viommu_probe_device(struct device *dev)
 		if (ret)
 			goto err_free_dev;
 	}
+	dev->iommu->shadow_on_flush = 1;
 
 	return &viommu->iommu;
 
@@ -1037,6 +1059,8 @@ static bool viommu_capable(struct device *dev, enum iommu_cap cap)
 	switch (cap) {
 	case IOMMU_CAP_CACHE_COHERENCY:
 		return true;
+	case IOMMU_CAP_DEFERRED_FLUSH:
+		return true;
 	default:
 		return false;
 	}
@@ -1057,7 +1081,9 @@ static struct iommu_ops viommu_ops = {
 		.map_pages		= viommu_map_pages,
 		.unmap_pages		= viommu_unmap_pages,
 		.iova_to_phys		= viommu_iova_to_phys,
+		.flush_iotlb_all	= viommu_flush_iotlb_all,
 		.iotlb_sync		= viommu_iotlb_sync,
+		.iotlb_sync_map		= viommu_iotlb_sync_map,
 		.free			= viommu_domain_free,
 	}
 };
Quoted 5 lines

base-commit: 5514392fe77cd45b0d33bf239f13ba594a6759e5

Signature · 1 line
2.39.2

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#16 lore raw

Re: [RFC] iommu/virtio: Use single flush queue (EXPERIMENTAL)

Quoted 5 lines · Jean-Philippe Brucker

automatically get plugged into a VM without user intervention. Here I guess the devices we don't trust will be virtual devices implemented by other VMs. We don't have any method to identify them yet, so iommu.strict=1 and CONFIG_IOMMU_DEFAULT_DMA_STRICT is the best we can do at the moment.

VM's should work the same way as bare metal. The hypervisor should pass in an ACPI/etc indication if specific devices are to be untrusted. Otherwise the VM should assume trusted devices.

The hypervisor can already read all the VM's memory, it doesn't make alot of sense for the VM to try and be defensive here in the general case.

Jason
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author maintainer
#17 lore raw

Re: [RFC] iommu/virtio: Use single flush queue (EXPERIMENTAL)

In reply to Jean-Philippe Brucker
Quoted 48 lines · Jean-Philippe Brucker

Hi Niklas,

Quoted 6 lines · Niklas Schnelle

Just like on paged s390 guests with their virtual IOMMU, syncing mappings via virtio-iommu is quite expensive. It can thus benefit from queueing unmapped IOVAs and flushing them in batches but less so from parallel flushes which is what the shadow_on_flush flag introduced for s390 tunes dma-iommu to do.

Quoted 7 lines

For this to work .flush_iotlb_all is implemented. Furthermore .iotlb_sync_map is also implemented and used to pull the sync out of the mapping operation for some additional batching and performance gain.

In a basic test with NVMe pass-through to a KVM guest on a Ryzen 3900X these changes together lead to about 19% more IOPS in a fio test and slightly more bandwidth too.

Nice, thank you for testing this. I played with a NVMe on an Intel desktop and can confirm similar results. With "sq" meaning single flush queue and "mq" percpu flush queue, "+map" is with .iotlb_sync_map() enabled.

Multithread block randwrite job [1]:

BW compared to host Confidence
(higher better)
host 100.0% ±0.0%
noviommu 99.9 0.0
viommu lazy sq +map 99.9 0.1
viommu lazy mq +map 99.9 0.1
viommu lazy sq 92.2 0.9
viommu lazy mq 91.5 0.9
viommu strict +map 92.7 0.9
viommu strict 81.3 1.0

Single page randrw:

Latency compared to host Confidence
(lower better)
host x1.00 ±.04
noviommu 1.23 .04
viommu lazy sq +map 7.09 .05
viommu lazy mq +map 7.07 .07
viommu lazy sq 7.15 .04
viommu lazy mq 7.11 .05
viommu strict +map 8.82 .05
viommu strict 8.82 .04

Quoted 7 lines

So with lazy+map we get the maximum bandwidth reachable on this disk (2.5GiB/s) even with a heavy iommu_map/unmap usage, which is cool. Random access latency also improves with lazy mode.

The difference between single and percpu flush queue isn't really measurable in my multithread test. There is a difference between Lazy sq and mq but the variation between samples outweighs it.

Interesting that there is no advantage when using a single large queue. Maybe virtio-iommu does handle the per-CPU flushing well because of a stronger CPU affinity than on s390x or maybe it has to do with the sync being just a queue drain making only some syncs slow.

Quoted 16 lines
Quoted 11 lines

---
Note:
The idea of using the single flush queue scheme from my series "iommu/dma: s390 DMA API conversion and optimized IOTLB flushing"[0] for virtio-iommu was already mentioned in the cover letter. I now wanted to explore this with this patch which may also serve as a test vehicle for the single flush queue scheme usable on non-s390.

Quoted 4 lines

Besides limited testing, this is marked experimental mainly because the use of queuing needs to be a concious decision as it allows continued access to unmapped pages for up to a second with the currently proposed single flush queue mechanism.

Quoted 8 lines

It fits with the iommu.strict=0 / CONFIG_IOMMU_DEFAULT_DMA_LAZY setting, which selects DMA_FQ domains. That option allows a misbehaving device to access memory that has been freed/reallocated, which is what we're enabling here. I believe the risk is pretty much the same for deferred UNMAP as for deferred TLBI, since mappings that we're removing were likely cached in the TLB. Increasing the timeout does make it easier to exploit, but I don't think that changes the policy from an admin's perspective:
only enable lazy mode if you trust device and driver.

I'm with Jason here. I think for non-CoCo hypervisor's
the hypervisor is trusted so may just forward the trust level of the relevant device. This may of course be different for CoCo VMs but then the VM knows and could e.g. default to iommu.strict=1 or it needs to use swiotlb anyway.

Quoted 14 lines

On bare metal, we disable DMA_FQ for devices that can be easily hotplugged into unattended machines (through external-facing ports like thunderbolt). On VMs, the concern isn't really about external devices, since they don't automatically get plugged into a VM without user intervention. Here I guess the devices we don't trust will be virtual devices implemented by other VMs. We don't have any method to identify them yet, so iommu.strict=1 and CONFIG_IOMMU_DEFAULT_DMA_STRICT is the best we can do at the moment.

I'm not so sure about enabling shadow_on_flush by default, since the performance difference was too small in my tests. Maybe a module parameter for dma-iommu could configure the flush queue?

Quoted 9 lines
Quoted 3 lines

Also it might make sense to split this patch to do the introduction and use of .iotlb_sync_map separately but as a test vehicle I found it easier to consume as a single patch.

Yes, both changes are useful but should be in two patches

Thanks,
Jean

After sending v12 of my series[0] I'm now looking into sending this separately. Rebasing on v6.5-rc7 however I noticed that without my patch "iommu: Allow .iotlb_sync_map to fail and handle s390's -ENOMEM return" we not only get a conflict with my series but also potentially miss errors on map since just like on s390x virtio-iommu's .iotlb_sync_map can fail. So I think it makes most sense to send this based on my series. I hope that's okay for you.

[0]
https://lore.kernel.org/lkml/[email protected]/

#18 lore raw
In reply to Niklas Schnelle
Quoted 10 lines · Niklas Schnelle

Hi All,

This patch series converts s390's PCI support from its platform specific DMA API implementation in arch/s390/pci/pci_dma.c to the common DMA IOMMU layer. The conversion itself is done in patches 3-4 with patch 2 providing the final necessary IOMMU driver improvement to handle s390's special IOTLB flush out-of-resource indication in virtualized environments. Patches 1-2 may be applied independently. The conversion itself only touches the s390 IOMMU driver and s390 arch code moving over remaining functions from the s390 DMA API implementation. No changes to common code are necessary.

It has been another month with no comments, what is needed to get this merged?

Getting the arches converted to the standard dma ops is a big deal, can we please conclude this?

Thanks,
Jason