Skip to contentSkip to conversation

iommu/dma: s390 DMA API conversion and optimized IOTLB flushing

RFC Superseded Niklas Schnelle 3 participants 11 messages Oct 19, 2022 – Oct 20, 2022

Subsystems (8)

v1 of 13 Superseded Latest is v13 →

lore
Share & tools
lore

Niklas Schnelle

author maintainer
#0 lore raw

cover iommu/dma: s390 DMA API conversion and optimized IOTLB flushing

Hi All,

This patch series converts s390's PCI support from its platform specific DMA API implementation in arch/s390/pci/pci_dma.c to the common DMA IOMMU layer. The conversion itself is done in patch 3 and after applying my previous s390 IOMMU series [0, 1] only touches the s390 IOMMU driver and arch code moving over remaining functions from the s390 DMA code. No changes to common code are necessary.

After patch 3 the conversion is complete in principle and on our partitioning machine hypervisor LPAR performance matches or exceeds the existing code. When running under z/VM or KVM however, performance plummets to about half of the existing code due to a much higher rate of IOTLB flushes for unmapped pages. Due to the hypervisors use of IOTLB flushes to synchronize their shadow tables these are very expensive and minimizing flushes is very important.

To counter this performance drop patches 4-5 propose a new, single queue, IOTLB flushing scheme as an alternative to the existing per-CPU flush queues. Introducing an alternative scheme was also suggested by Robin Murphy[2]. The single queue allows batching a much larger number of lazily freed IOVAs and was also chosen as hypervisors tend to serialize IOTLB flushes removing some of the gains of multiple queues. To give some bound on when IOVAs are flushed a timeout of 1 second is used and postponed when a flush happens. In my tests this scheme brought performance under z/VM and KVM on par with the existing code.

As it is implemented in common code this IOTLB flushing scheme can of course be used by other platforms with expensive IOTLB flushes. Particularly virtio-iommu might be a candidate. With this series however only s390 systems that require IOTLB flushes on map default to it while LPAR uses the per-CPU queues.

I did verify that the new scheme does work on my x86 Ryzen workstation by locally modifying drivers/iommu/iommu.c:iommu_subsys_init() to default to it and verifying its use via "cat /sys/bus/pci/devices/*/iommu_group/type". I did not find problems with an AMD GPU, Intel NIC (with SR-IOV), NVMes or any on board peripherals though I did not perform any meaningful performance tests.

Another complication is that on z/VM and KVM our IOTLB flushes can return an error indicating that the hypervisor has run out of IOVAs and needs us to flush the queue before new mappings can be created. In strict mode this of course doesn't happen but with queuing we need to handle it. This is done in patch 6.

As with previous series this is available via my git.kernel.org tree[3] in the dma_iommu_v1 branch with s390_dma_iommu_v1 tag.

Best regards,
Niklas

[0] https://lore.kernel.org/linux-iommu/[email protected]/
[1] https://lore.kernel.org/linux-iommu/[email protected]/
[2] https://lore.kernel.org/linux-iommu/[email protected]/
[3] https://git.kernel.org/pub/scm/linux/kernel/git/niks/linux.git/

Niklas Schnelle (6):
s390/ism: Set DMA coherent mask
s390/pci: prepare is_passed_through() for dma-iommu s390/pci: Use dma-iommu layer
iommu/dma: Prepare for multiple flush queue implementations iommu/dma: Add simple batching flush queue implementation iommu/s390: flush queued IOVAs on RPCIT out of resource indication

 .../admin-guide/kernel-parameters.txt         |   9 +-
 arch/s390/include/asm/pci.h                   |   7 -
 arch/s390/include/asm/pci_dma.h               | 120 +--
 arch/s390/pci/Makefile                        |   2 +-
 arch/s390/pci/pci.c                           |  22 +-
 arch/s390/pci/pci_bus.c                       |   5 -
 arch/s390/pci/pci_debug.c                     |  13 +-
 arch/s390/pci/pci_dma.c                       | 732 ------------------
 arch/s390/pci/pci_event.c                     |  17 +-
 arch/s390/pci/pci_sysfs.c                     |  19 +-
 drivers/iommu/Kconfig                         |   3 +-
 drivers/iommu/dma-iommu.c                     | 307 ++++++--
 drivers/iommu/dma-iommu.h                     |   1 +
 drivers/iommu/iommu.c                         |  19 +-
 drivers/iommu/s390-iommu.c                    | 415 +++++++++-
 drivers/s390/net/ism_drv.c                    |   4 +
 include/linux/iommu.h                         |   6 +
 17 files changed, 707 insertions(+), 994 deletions(-)
 delete mode 100644 arch/s390/pci/pci_dma.c
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

RFC 1/6 s390/ism: Set DMA coherent mask

A future change will convert the DMA API implementation from the architecture specific arch/s390/pci/pci_dma.c to using the common code drivers/iommu/dma-iommu.c which the utilizes the same IOMMU hardware through the s390-iommu driver. Unlike the s390 specific DMA API this requires devices to correctly call dma_set_coherent_mask() to be allowed to use IOVAs >2^32 in dma_alloc_coherent(). This was however not done for ISM devices which require such addresses since currently the DMA aperture for PCI devices starts a 2^32 and all calls to dma_alloc_coherent() would thus fail.

---
 drivers/s390/net/ism_drv.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/s390/net/ism_drv.c +4 -0
index d34bb6ec1490..5890c32a9e1f 100644
@@ -560,6 +560,10 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (ret)
 		goto err_resource;
 
+	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));
+	if (ret)
+		goto err_resource;
+
 	dma_set_seg_boundary(&pdev->dev, SZ_1M - 1);
 	dma_set_max_seg_size(&pdev->dev, SZ_1M);
 	pci_set_master(pdev);
Signature · 1 line
2.34.1
#2 lore raw

Re: [RFC 1/6] s390/ism: Set DMA coherent mask

drivers/s390/net/ism_drv.c static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 563

Quoted 4 lines · Niklas Schnelle
drivers/s390/net/ism_drv.c +1 -0
 	if (ret)
 		goto err_resource;
 
+	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));

Nit: but you could also just replace the call to dma_set_mask abobe with a call to dma_set_mask_and_coherent.

author maintainer
#3 lore raw

Re: [RFC 1/6] s390/ism: Set DMA coherent mask

Quoted 8 lines · Christoph Hellwig

drivers/s390/net/ism_drv.c static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 563

Quoted 4 lines · Niklas Schnelle
drivers/s390/net/ism_drv.c +1 -0
 	if (ret)
 		goto err_resource;
 
+	ret = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));

Nit: but you could also just replace the call to dma_set_mask abobe with a call to dma_set_mask_and_coherent.

That makes sense, thanks. Will be added for v2. Also obviously this change is only a prerequisite and could be applied separately, the changed coherent mask won't affect use with the existing DMA implementation.

author maintainer
#4 lore raw

RFC 2/6 s390/pci: prepare is_passed_through() for dma-iommu

In reply to Niklas Schnelle

With the IOMMU always controlled through the IOMMU driver testing for zdev->s390_domain is not a valid indication of the device being passed-through. Instead test if zdev->kzdev is set.

---
 arch/s390/pci/pci_event.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
arch/s390/pci/pci_event.c +11 -4
index b9324ca2eb94..4ef5a6a1d618 100644
@@ -59,9 +59,16 @@ static inline bool ers_result_indicates_abort(pci_ers_result_t ers_res)
 	}
 }
 
-static bool is_passed_through(struct zpci_dev *zdev)
+static bool is_passed_through(struct pci_dev *pdev)
 {
-	return zdev->s390_domain;
+	struct zpci_dev *zdev = to_zpci(pdev);
+	bool ret;
+
+	mutex_lock(&zdev->kzdev_lock);
+	ret = !!zdev->kzdev;
+	mutex_unlock(&zdev->kzdev_lock);
+
+	return ret;
 }
 
 static bool is_driver_supported(struct pci_driver *driver)
@@ -176,7 +183,7 @@ static pci_ers_result_t zpci_event_attempt_error_recovery(struct pci_dev *pdev)
 	}
 	pdev->error_state = pci_channel_io_frozen;
 
-	if (is_passed_through(to_zpci(pdev))) {
+	if (is_passed_through(pdev)) {
 		pr_info("%s: Cannot be recovered in the host because it is a pass-through device\n",
 			pci_name(pdev));
 		goto out_unlock;
@@ -239,7 +246,7 @@ static void zpci_event_io_failure(struct pci_dev *pdev, pci_channel_state_t es)
 	 * we will inject the error event and let the guest recover the device
 	 * itself.
 	 */
-	if (is_passed_through(to_zpci(pdev)))
+	if (is_passed_through(pdev))
 		goto out;
 	driver = to_pci_driver(pdev->dev.driver);
 	if (driver && driver->err_handler && driver->err_handler->error_detected)
Signature · 1 line
2.34.1
author maintainer
#5 lore raw

RFC 3/6 s390/pci: Use dma-iommu layer

In reply to Niklas Schnelle

While s390 already has a standard IOMMU driver and previous changes have added I/O TLB flushing operations this driver is currently only used for user-space PCI access such as vfio-pci. For the DMA API s390 instead utilizes its own implementation in arch/s390/pci/pci_dma.c which drives the same hardware and shares some code but requires a complex and fragile hand over between DMA API and IOMMU API use of a device and despite code sharing still leads to significant duplication and maintenance effort. Let's utilize the common code DMAP API implementation from drivers/iommu/dma-iommu.c instead allowing us to get rid of arch/s390/pci/pci_dma.c.

---
 .../admin-guide/kernel-parameters.txt         |   9 +-
 arch/s390/include/asm/pci.h                   |   7 -
 arch/s390/include/asm/pci_dma.h               | 120 +--
 arch/s390/pci/Makefile                        |   2 +-
 arch/s390/pci/pci.c                           |  22 +-
 arch/s390/pci/pci_bus.c                       |   5 -
 arch/s390/pci/pci_debug.c                     |  13 +-
 arch/s390/pci/pci_dma.c                       | 732 ------------------
 arch/s390/pci/pci_event.c                     |   2 -
 arch/s390/pci/pci_sysfs.c                     |  19 +-
 drivers/iommu/Kconfig                         |   3 +-
 drivers/iommu/s390-iommu.c                    | 392 +++++++++-
 12 files changed, 409 insertions(+), 917 deletions(-)
 delete mode 100644 arch/s390/pci/pci_dma.c
Documentation/admin-guide/kernel-parameters.txt +5 -4
index a465d5242774..633312c4f800 100644
@@ -2154,7 +2154,7 @@
 			  forcing Dual Address Cycle for PCI cards supporting
 			  greater than 32-bit addressing.
 
-	iommu.strict=	[ARM64, X86] Configure TLB invalidation behaviour
+	iommu.strict=	[ARM64, X86, S390] Configure TLB invalidation behaviour
 			Format: { "0" | "1" }
 			0 - Lazy mode.
 			  Request that DMA unmap operations use deferred
@@ -5387,9 +5387,10 @@
 	s390_iommu=	[HW,S390]
 			Set s390 IOTLB flushing mode
 		strict
-			With strict flushing every unmap operation will result in
-			an IOTLB flush. Default is lazy flushing before reuse,
-			which is faster.
+			With strict flushing every unmap operation will result
+			in an IOTLB flush. Default is lazy flushing before
+			reuse, which is faster. Deprecated, equivalent to
+			iommu.strict=1.
 
 	s390_iommu_aperture=	[KNL,S390]
 			Specifies the size of the per device DMA address space
arch/s390/include/asm/pci.h +0 -7
index b248694e0024..3f74f1cf37df 100644
@@ -159,13 +159,6 @@ struct zpci_dev {
 	unsigned long	*dma_table;
 	int		tlb_refresh;
 
-	spinlock_t	iommu_bitmap_lock;
-	unsigned long	*iommu_bitmap;
-	unsigned long	*lazy_bitmap;
-	unsigned long	iommu_size;
-	unsigned long	iommu_pages;
-	unsigned int	next_bit;
-
 	struct iommu_device iommu_dev;  /* IOMMU core handle */
 
 	char res_name[16];
arch/s390/include/asm/pci_dma.h +10 -110

126 lines not on this page · raw message

arch/s390/pci/Makefile +1 -1
index 5ae31ca9dd44..0547a10406e7 100644
@@ -3,7 +3,7 @@
 # Makefile for the s390 PCI subsystem.
 #
 
-obj-$(CONFIG_PCI)	+= pci.o pci_irq.o pci_dma.o pci_clp.o pci_sysfs.o \
+obj-$(CONFIG_PCI)	+= pci.o pci_irq.o pci_clp.o pci_sysfs.o \
 			   pci_event.o pci_debug.o pci_insn.o pci_mmio.o \
 			   pci_bus.o pci_kvm_hook.o
 obj-$(CONFIG_PCI_IOV)	+= pci_iov.o
arch/s390/pci/pci.c +5 -17
index ef38b1514c77..6b0fe8761509 100644
@@ -124,7 +124,11 @@ int zpci_register_ioat(struct zpci_dev *zdev, u8 dmaas,
 
 	WARN_ON_ONCE(iota & 0x3fff);
 	fib.pba = base;
-	fib.pal = limit;
+	/* Work around off by one in ISM virt device */
+	if (zdev->pft == 0x5 && limit > base)
+		fib.pal = limit + (1 << 12);
+	else
+		fib.pal = limit;
 	fib.iota = iota | ZPCI_IOTA_RTTO_FLAG;
 	fib.gd = zdev->gisa;
 	cc = zpci_mod_fc(req, &fib, status);
@@ -615,7 +619,6 @@ int pcibios_device_add(struct pci_dev *pdev)
 		pdev->no_vf_scan = 1;
 
 	pdev->dev.groups = zpci_attr_groups;
-	pdev->dev.dma_ops = &s390_pci_dma_ops;
 	zpci_map_resources(pdev);
 
 	for (i = 0; i < PCI_STD_NUM_BARS; i++) {
@@ -789,8 +792,6 @@ int zpci_hot_reset_device(struct zpci_dev *zdev)
 	if (zdev->dma_table)
 		rc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
 					virt_to_phys(zdev->dma_table), &status);
-	else
-		rc = zpci_dma_init_device(zdev);
 	if (rc) {
 		zpci_disable_device(zdev);
 		return rc;
@@ -915,11 +916,6 @@ int zpci_deconfigure_device(struct zpci_dev *zdev)
 	if (zdev->zbus->bus)
 		zpci_bus_remove_device(zdev, false);
 
-	if (zdev->dma_table) {
-		rc = zpci_dma_exit_device(zdev);
-		if (rc)
-			return rc;
-	}
 	if (zdev_enabled(zdev)) {
 		rc = zpci_disable_device(zdev);
 		if (rc)
@@ -968,8 +964,6 @@ void zpci_release_device(struct kref *kref)
 	if (zdev->zbus->bus)
 		zpci_bus_remove_device(zdev, false);
 
-	if (zdev->dma_table)
-		zpci_dma_exit_device(zdev);
 	if (zdev_enabled(zdev))
 		zpci_disable_device(zdev);
 
@@ -1159,10 +1153,6 @@ static int __init pci_base_init(void)
 	if (rc)
 		goto out_irq;
 
-	rc = zpci_dma_init();
-	if (rc)
-		goto out_dma;
-
 	rc = clp_scan_pci_devices();
 	if (rc)
 		goto out_find;
@@ -1172,8 +1162,6 @@ static int __init pci_base_init(void)
 	return 0;
 
 out_find:
-	zpci_dma_exit();
-out_dma:
 	zpci_irq_exit();
 out_irq:
 	zpci_mem_exit();
arch/s390/pci/pci_bus.c +0 -5
index 6a8da1b742ae..b15ad15999f8 100644
@@ -49,11 +49,6 @@ static int zpci_bus_prepare_device(struct zpci_dev *zdev)
 		rc = zpci_enable_device(zdev);
 		if (rc)
 			return rc;
-		rc = zpci_dma_init_device(zdev);
-		if (rc) {
-			zpci_disable_device(zdev);
-			return rc;
-		}
 	}
 
 	if (!zdev->has_resources) {
arch/s390/pci/pci_debug.c +10 -3
index ca6bd98eec13..60cec57a3907 100644
@@ -53,9 +53,12 @@ static char *pci_fmt3_names[] = {
 };
 
 static char *pci_sw_names[] = {
-	"Allocated pages",
+/* TODO "Allocated pages", */
 	"Mapped pages",
 	"Unmapped pages",
+	"Global RPCITs",
+	"Sync Map RPCITs",
+	"Sync RPCITs",
 };
 
 static void pci_fmb_show(struct seq_file *m, char *name[], int length,
@@ -69,10 +72,14 @@ static void pci_fmb_show(struct seq_file *m, char *name[], int length,
 
 static void pci_sw_counter_show(struct seq_file *m)
 {
-	struct zpci_dev *zdev = m->private;
-	atomic64_t *counter = &zdev->allocated_pages;
+	struct zpci_iommu_ctrs  *ctrs = zpci_get_iommu_ctrs(m->private);
+	atomic64_t *counter;
 	int i;
 
+	if (!ctrs)
+		return;
+
+	counter = &ctrs->mapped_pages;
 	for (i = 0; i < ARRAY_SIZE(pci_sw_names); i++, counter++)
 		seq_printf(m, "%26s:\t%llu\n", pci_sw_names[i],
 			   atomic64_read(counter));
arch/s390/pci/pci_dma.c deleted +0 -732

732 lines not on this page · raw message

arch/s390/pci/pci_event.c +0 -2
index 4ef5a6a1d618..4d9773ef9e0a 100644
@@ -313,8 +313,6 @@ static void zpci_event_hard_deconfigured(struct zpci_dev *zdev, u32 fh)
 	/* Even though the device is already gone we still
 	 * need to free zPCI resources as part of the disable.
 	 */
-	if (zdev->dma_table)
-		zpci_dma_exit_device(zdev);
 	if (zdev_enabled(zdev))
 		zpci_disable_device(zdev);
 	zdev->state = ZPCI_FN_STATE_STANDBY;
arch/s390/pci/pci_sysfs.c +8 -11
index cae280e5c047..8a7abac51816 100644
@@ -56,6 +56,7 @@ static ssize_t recover_store(struct device *dev, struct device_attribute *attr,
 	struct pci_dev *pdev = to_pci_dev(dev);
 	struct zpci_dev *zdev = to_zpci(pdev);
 	int ret = 0;
+	u8 status;
 
 	/* Can't use device_remove_self() here as that would lead us to lock
 	 * the pci_rescan_remove_lock while holding the device' kernfs lock.
@@ -82,12 +83,6 @@ static ssize_t recover_store(struct device *dev, struct device_attribute *attr,
 	pci_lock_rescan_remove();
 	if (pci_dev_is_added(pdev)) {
 		pci_stop_and_remove_bus_device(pdev);
-		if (zdev->dma_table) {
-			ret = zpci_dma_exit_device(zdev);
-			if (ret)
-				goto out;
-		}
-
 		if (zdev_enabled(zdev)) {
 			ret = zpci_disable_device(zdev);
 			/*
@@ -105,14 +100,16 @@ static ssize_t recover_store(struct device *dev, struct device_attribute *attr,
 		ret = zpci_enable_device(zdev);
 		if (ret)
 			goto out;
-		ret = zpci_dma_init_device(zdev);
-		if (ret) {
-			zpci_disable_device(zdev);
-			goto out;
+
+		if (zdev->dma_table) {
+			ret = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
+						 virt_to_phys(zdev->dma_table), &status);
+			if (ret)
+				zpci_disable_device(zdev);
 		}
-		pci_rescan_bus(zdev->zbus->bus);
 	}
 out:
+	pci_rescan_bus(zdev->zbus->bus);
 	pci_unlock_rescan_remove();
 	if (kn)
 		sysfs_unbreak_active_protection(kn);
drivers/iommu/Kconfig +2 -1
index dc5f7a156ff5..804fb8f42d61 100644
@@ -93,7 +93,7 @@ config IOMMU_DEBUGFS
 choice
 	prompt "IOMMU default domain type"
 	depends on IOMMU_API
-	default IOMMU_DEFAULT_DMA_LAZY if X86 || IA64
+	default IOMMU_DEFAULT_DMA_LAZY if X86 || IA64 || S390
 	default IOMMU_DEFAULT_DMA_STRICT
 	help
 	  Choose the type of IOMMU domain used to manage DMA API usage by
@@ -412,6 +412,7 @@ config ARM_SMMU_V3_SVA
 config S390_IOMMU
 	def_bool y if S390 && PCI
 	depends on S390 && PCI
+	select IOMMU_DMA
 	select IOMMU_API
 	help
 	  Support for the IOMMU API for s390 PCI devices.
drivers/iommu/s390-iommu.c +368 -24

501 lines not on this page · raw message

Signature · 1 line
2.34.1
author maintainer
#6 lore raw

RFC 4/6 iommu/dma: Prepare for multiple flush queue implementations

In reply to Niklas Schnelle

The dma-iommu currently only supports a per CPU, highly parallelized flush queue that depends on relatively cheap global flushes. Especially in virtualized environments these assumptions may not hold true which in the past lead to the need to resort to strict mode. Instead prepare the dma-iommu code for alternative flushing schemes that optimize for different needs.

---
 drivers/iommu/dma-iommu.c | 164 ++++++++++++++++++++++----------------
 1 file changed, 97 insertions(+), 67 deletions(-)
drivers/iommu/dma-iommu.c +97 -67
index 9297b741f5e8..77d969f5aad7 100644
@@ -44,12 +44,13 @@ enum iommu_dma_cookie_type {
 
 struct iommu_dma_cookie {
 	enum iommu_dma_cookie_type	type;
+
 	union {
 		/* Full allocator for IOMMU_DMA_IOVA_COOKIE */
 		struct {
-			struct iova_domain	iovad;
-
-			struct iova_fq __percpu *fq;	/* Flush queue */
+			struct iova_domain		iovad;
+			/* Flush queue */
+			struct iova_percpu __percpu *percpu_fq;
 			/* Number of TLB flushes that have been started */
 			atomic64_t		fq_flush_start_cnt;
 			/* Number of TLB flushes that have been finished */
@@ -83,13 +84,13 @@ static int __init iommu_dma_forcedac_setup(char *str)
 early_param("iommu.forcedac", iommu_dma_forcedac_setup);
 
 /* Number of entries per flush queue */
-#define IOVA_FQ_SIZE	256
+#define IOVA_PERCPU_SIZE	256
 
 /* Timeout (in ms) after which entries are flushed from the queue */
-#define IOVA_FQ_TIMEOUT	10
+#define IOVA_PERCPU_TIMEOUT	10
 
 /* Flush queue entry for deferred flushing */
-struct iova_fq_entry {
+struct iova_percpu_entry {
 	unsigned long iova_pfn;
 	unsigned long pages;
 	struct list_head freelist;
@@ -97,40 +98,40 @@ struct iova_fq_entry {
 };
 
 /* Per-CPU flush queue structure */
-struct iova_fq {
-	struct iova_fq_entry entries[IOVA_FQ_SIZE];
+struct iova_percpu {
+	struct iova_percpu_entry entries[IOVA_PERCPU_SIZE];
 	unsigned int head, tail;
 	spinlock_t lock;
 };
 
-#define fq_ring_for_each(i, fq) \
-	for ((i) = (fq)->head; (i) != (fq)->tail; (i) = ((i) + 1) % IOVA_FQ_SIZE)
+#define ring_for_each_percpu(i, fq) \
+	for ((i) = (fq)->head; (i) != (fq)->tail; (i) = ((i) + 1) % IOVA_PERCPU_SIZE)
 
-static inline bool fq_full(struct iova_fq *fq)
+static inline bool is_full_percpu(struct iova_percpu *fq)
 {
 	assert_spin_locked(&fq->lock);
-	return (((fq->tail + 1) % IOVA_FQ_SIZE) == fq->head);
+	return (((fq->tail + 1) % IOVA_PERCPU_SIZE) == fq->head);
 }
 
-static inline unsigned int fq_ring_add(struct iova_fq *fq)
+static inline unsigned int ring_add_percpu(struct iova_percpu *fq)
 {
 	unsigned int idx = fq->tail;
 
 	assert_spin_locked(&fq->lock);
 
-	fq->tail = (idx + 1) % IOVA_FQ_SIZE;
+	fq->tail = (idx + 1) % IOVA_PERCPU_SIZE;
 
 	return idx;
 }
 
-static void fq_ring_free(struct iommu_dma_cookie *cookie, struct iova_fq *fq)
+static void ring_free_percpu(struct iommu_dma_cookie *cookie, struct iova_percpu *fq)
 {
 	u64 counter = atomic64_read(&cookie->fq_flush_finish_cnt);
 	unsigned int idx;
 
 	assert_spin_locked(&fq->lock);
 
-	fq_ring_for_each(idx, fq) {
+	ring_for_each_percpu(idx, fq) {
 
 		if (fq->entries[idx].counter >= counter)
 			break;
@@ -140,69 +141,66 @@ static void fq_ring_free(struct iommu_dma_cookie *cookie, struct iova_fq *fq)
 			       fq->entries[idx].iova_pfn,
 			       fq->entries[idx].pages);
 
-		fq->head = (fq->head + 1) % IOVA_FQ_SIZE;
+		fq->head = (fq->head + 1) % IOVA_PERCPU_SIZE;
 	}
 }
 
-static void fq_flush_iotlb(struct iommu_dma_cookie *cookie)
+static void flush_iotlb_percpu(struct iommu_dma_cookie *cookie)
 {
 	atomic64_inc(&cookie->fq_flush_start_cnt);
 	cookie->fq_domain->ops->flush_iotlb_all(cookie->fq_domain);
 	atomic64_inc(&cookie->fq_flush_finish_cnt);
 }
 
-static void fq_flush_timeout(struct timer_list *t)
+static void flush_percpu(struct iommu_dma_cookie *cookie)
 {
-	struct iommu_dma_cookie *cookie = from_timer(cookie, t, fq_timer);
 	int cpu;
 
-	atomic_set(&cookie->fq_timer_on, 0);
-	fq_flush_iotlb(cookie);
+	flush_iotlb_percpu(cookie);
 
 	for_each_possible_cpu(cpu) {
 		unsigned long flags;
-		struct iova_fq *fq;
+		struct iova_percpu *fq;
 
-		fq = per_cpu_ptr(cookie->fq, cpu);
+		fq = per_cpu_ptr(cookie->percpu_fq, cpu);
 		spin_lock_irqsave(&fq->lock, flags);
-		fq_ring_free(cookie, fq);
+		ring_free_percpu(cookie, fq);
 		spin_unlock_irqrestore(&fq->lock, flags);
 	}
 }
 
-static void queue_iova(struct iommu_dma_cookie *cookie,
-		unsigned long pfn, unsigned long pages,
-		struct list_head *freelist)
+static void fq_flush_timeout(struct timer_list *t)
 {
-	struct iova_fq *fq;
+	struct iommu_dma_cookie *cookie = from_timer(cookie, t, fq_timer);
+
+	atomic_set(&cookie->fq_timer_on, 0);
+	flush_percpu(cookie);
+}
+
+static void queue_iova_percpu(struct iommu_dma_cookie *cookie,
+			      unsigned long pfn, unsigned long pages,
+			      struct list_head *freelist)
+{
+	struct iova_percpu *fq;
 	unsigned long flags;
 	unsigned int idx;
 
-	/*
-	 * Order against the IOMMU driver's pagetable update from unmapping
-	 * @pte, to guarantee that fq_flush_iotlb() observes that if called
-	 * from a different CPU before we release the lock below. Full barrier
-	 * so it also pairs with iommu_dma_init_fq() to avoid seeing partially
-	 * written fq state here.
-	 */
-	smp_mb();
-
-	fq = raw_cpu_ptr(cookie->fq);
+	fq = raw_cpu_ptr(cookie->percpu_fq);
 	spin_lock_irqsave(&fq->lock, flags);
 
 	/*
 	 * First remove all entries from the flush queue that have already been
-	 * flushed out on another CPU. This makes the fq_full() check below less
+	 * flushed out on another CPU. This makes the fullness check below less
 	 * likely to be true.
 	 */
-	fq_ring_free(cookie, fq);
+	ring_free_percpu(cookie, fq);
 
-	if (fq_full(fq)) {
-		fq_flush_iotlb(cookie);
-		fq_ring_free(cookie, fq);
+	if (is_full_percpu(fq)) {
+		flush_iotlb_percpu(cookie);
+		ring_free_percpu(cookie, fq);
 	}
 
-	idx = fq_ring_add(fq);
+	idx = ring_add_percpu(fq);
 
 	fq->entries[idx].iova_pfn = pfn;
 	fq->entries[idx].pages    = pages;
@@ -210,65 +208,97 @@ static void queue_iova(struct iommu_dma_cookie *cookie,
 	list_splice(freelist, &fq->entries[idx].freelist);
 
 	spin_unlock_irqrestore(&fq->lock, flags);
+}
+
+static void queue_iova(struct iommu_dma_cookie *cookie,
+		       unsigned long pfn, unsigned long pages,
+		struct list_head *freelist)
+{
+	/*
+	 * Order against the IOMMU driver's pagetable update from unmapping
+	 * @pte, to guarantee that flush_iotlb_percpu() observes that if called
+	 * from a different CPU before we release the lock below. Full barrier
+	 * so it also pairs with iommu_dma_init_fq() to avoid seeing
+	 * partially written queue state here.
+	 */
+	smp_mb();
+
+	queue_iova_percpu(cookie, pfn, pages, freelist);
 
 	/* Avoid false sharing as much as possible. */
 	if (!atomic_read(&cookie->fq_timer_on) &&
 	    !atomic_xchg(&cookie->fq_timer_on, 1))
 		mod_timer(&cookie->fq_timer,
-			  jiffies + msecs_to_jiffies(IOVA_FQ_TIMEOUT));
+			  jiffies + msecs_to_jiffies(IOVA_PERCPU_TIMEOUT));
 }
 
-static void iommu_dma_free_fq(struct iommu_dma_cookie *cookie)
+static void iommu_dma_free_percpu(struct iommu_dma_cookie *cookie)
 {
 	int cpu, idx;
 
-	if (!cookie->fq)
+	if (!cookie->percpu_fq)
 		return;
 
-	del_timer_sync(&cookie->fq_timer);
-	/* The IOVAs will be torn down separately, so just free our queued pages */
 	for_each_possible_cpu(cpu) {
-		struct iova_fq *fq = per_cpu_ptr(cookie->fq, cpu);
+		struct iova_percpu *fq = per_cpu_ptr(cookie->percpu_fq, cpu);
 
-		fq_ring_for_each(idx, fq)
+		ring_for_each_percpu(idx, fq)
 			put_pages_list(&fq->entries[idx].freelist);
 	}
 
-	free_percpu(cookie->fq);
+	free_percpu(cookie->percpu_fq);
 }
 
-/* sysfs updates are serialised by the mutex of the group owning @domain */
-int iommu_dma_init_fq(struct iommu_domain *domain)
+static void iommu_dma_free_fq(struct iommu_dma_cookie *cookie)
 {
-	struct iommu_dma_cookie *cookie = domain->iova_cookie;
-	struct iova_fq __percpu *queue;
-	int i, cpu;
+	del_timer_sync(&cookie->fq_timer);
+	/* The IOVAs will be torn down separately, so just free our queued pages */
+	iommu_dma_free_percpu(cookie);
+}
 
-	if (cookie->fq_domain)
-		return 0;
+static int iommu_dma_init_percpu(struct iommu_dma_cookie *cookie)
+{
+	struct iova_percpu __percpu *queue;
+	int i, cpu;
 
 	atomic64_set(&cookie->fq_flush_start_cnt,  0);
 	atomic64_set(&cookie->fq_flush_finish_cnt, 0);
 
-	queue = alloc_percpu(struct iova_fq);
-	if (!queue) {
-		pr_warn("iova flush queue initialization failed\n");
+	queue = alloc_percpu(struct iova_percpu);
+	if (!queue)
 		return -ENOMEM;
-	}
 
 	for_each_possible_cpu(cpu) {
-		struct iova_fq *fq = per_cpu_ptr(queue, cpu);
+		struct iova_percpu *fq = per_cpu_ptr(queue, cpu);
 
 		fq->head = 0;
 		fq->tail = 0;
 
 		spin_lock_init(&fq->lock);
 
-		for (i = 0; i < IOVA_FQ_SIZE; i++)
+		for (i = 0; i < IOVA_PERCPU_SIZE; i++)
 			INIT_LIST_HEAD(&fq->entries[i].freelist);
 	}
 
-	cookie->fq = queue;
+	cookie->percpu_fq = queue;
+
+	return 0;
+}
+
+/* sysfs updates are serialised by the mutex of the group owning @domain */
+int iommu_dma_init_fq(struct iommu_domain *domain)
+{
+	struct iommu_dma_cookie *cookie = domain->iova_cookie;
+	int rc;
+
+	if (cookie->fq_domain)
+		return 0;
+
+	rc = iommu_dma_init_percpu(cookie);
+	if (rc) {
+		pr_warn("iova flush queue initialization failed\n");
+		return rc;
+	}
 
 	timer_setup(&cookie->fq_timer, fq_flush_timeout, 0);
 	atomic_set(&cookie->fq_timer_on, 0);
Signature · 1 line
2.34.1
author maintainer
#7 lore raw

RFC 5/6 iommu/dma: Add simple batching flush queue implementation

In reply to Niklas Schnelle

Having dma-iommu prepared for alternative flush queue implementations we add a simple per-domain flush queue that optimzes for scenarios where global IOTLB flushes are used but are quite expensive and poorly parallelized. This is for example the case when IOTLB flushes are used to trigger updates to an underlying hypervisor's shadow tables and approximates the global flushing scheme previously in use on s390.

This is achieved by having a per-domain global flush queue that allows queuing a much large number of lazily freed IOVAs than the per-CPU flush queues. While using a single queue reduces parallelism this is not a problem when global IOTLB flushes are synchronized in the hypervisor anyway. While this flush queue allows queuing a large number of IOVAs we do limit the time a freed IOVA remains accessible by hardware to 1 second using a timeout that is reset on flush.

Enable this new flush queue implementation by default on s390 systems which use IOTLB flushes to trigger shadowing namely z/VM and KVM guests.

---
 drivers/iommu/dma-iommu.c  | 157 +++++++++++++++++++++++++++++++++++--
 drivers/iommu/iommu.c      |  19 +++--
 drivers/iommu/s390-iommu.c |  11 +++
 include/linux/iommu.h      |   6 ++
 4 files changed, 180 insertions(+), 13 deletions(-)
drivers/iommu/dma-iommu.c +149 -8
index 77d969f5aad7..427fb84f50c3 100644
@@ -49,8 +49,13 @@ struct iommu_dma_cookie {
 		/* Full allocator for IOMMU_DMA_IOVA_COOKIE */
 		struct {
 			struct iova_domain		iovad;
-			/* Flush queue */
-			struct iova_percpu __percpu *percpu_fq;
+			/* Flush queues */
+			union {
+				struct iova_percpu __percpu *percpu_fq;
+				struct iova_simple *simple_fq;
+			};
+			/* Queue timeout in milliseconds */
+			unsigned int		fq_timeout;
 			/* Number of TLB flushes that have been started */
 			atomic64_t		fq_flush_start_cnt;
 			/* Number of TLB flushes that have been finished */
@@ -104,6 +109,119 @@ struct iova_percpu {
 	spinlock_t lock;
 };
 
+/* Simplified batched flush queue for expensive IOTLB flushes */
+#define IOVA_SIMPLE_SIZE	32768
+/* Maximum time in milliseconds an IOVA can remain lazily freed */
+#define IOVA_SIMPLE_TIMEOUT	1000
+
+struct iova_simple_entry {
+	unsigned long iova_pfn;
+	unsigned long pages;
+};
+
+struct iova_simple {
+	/* Unlike iova_percpu we use a single queue lock */
+	spinlock_t lock;
+	unsigned int tail;
+	unsigned long total_pages;
+	struct list_head freelist;
+	struct iova_simple_entry entries[];
+};
+
+static bool is_full_simple(struct iommu_dma_cookie *cookie)
+{
+	struct iommu_domain *fq_domain = cookie->fq_domain;
+	struct iova_domain *iovad = &cookie->iovad;
+	struct iova_simple *sq = cookie->simple_fq;
+	unsigned long aperture_pages;
+
+	assert_spin_locked(&sq->lock);
+
+	/* If more than 7/8 the aperture is batched let's flush */
+	aperture_pages = ((fq_domain->geometry.aperture_end +  1) -
+		fq_domain->geometry.aperture_start) >> iova_shift(iovad);
+	aperture_pages -= aperture_pages >> 3;
+
+	return (sq->tail >= IOVA_SIMPLE_SIZE ||
+		sq->total_pages >= aperture_pages);
+}
+
+static void flush_simple(struct iommu_dma_cookie *cookie)
+{
+	struct iova_simple *sq = cookie->simple_fq;
+	unsigned int i;
+
+	assert_spin_locked(&sq->lock);
+	/* We're flushing so postpone timeout */
+	mod_timer(&cookie->fq_timer,
+		  jiffies + msecs_to_jiffies(cookie->fq_timeout));
+	cookie->fq_domain->ops->flush_iotlb_all(cookie->fq_domain);
+
+	put_pages_list(&sq->freelist);
+	for (i = 0; i < sq->tail; i++) {
+		free_iova_fast(&cookie->iovad,
+			       sq->entries[i].iova_pfn,
+			       sq->entries[i].pages);
+	}
+	sq->tail = 0;
+	sq->total_pages = 0;
+}
+
+static void flush_simple_lock(struct iommu_dma_cookie *cookie)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&cookie->simple_fq->lock, flags);
+	flush_simple(cookie);
+	spin_unlock_irqrestore(&cookie->simple_fq->lock, flags);
+}
+
+static void queue_iova_simple(struct iommu_dma_cookie *cookie,
+			      unsigned long pfn, unsigned long pages,
+			      struct list_head *freelist)
+{
+	struct iova_simple *sq = cookie->simple_fq;
+	unsigned long flags;
+	unsigned int idx;
+
+	spin_lock_irqsave(&sq->lock, flags);
+	if (is_full_simple(cookie))
+		flush_simple(cookie);
+	idx = sq->tail++;
+
+	sq->entries[idx].iova_pfn = pfn;
+	sq->entries[idx].pages    = pages;
+	list_splice(freelist, &sq->freelist);
+	sq->total_pages += pages;
+	spin_unlock_irqrestore(&sq->lock, flags);
+}
+
+static int iommu_dma_init_simple(struct iommu_dma_cookie *cookie)
+{
+	struct iova_simple *queue;
+
+	queue = vzalloc(sizeof(*queue) +
+			IOVA_SIMPLE_SIZE * sizeof(struct iova_simple_entry));
+	if (!queue)
+		return -ENOMEM;
+
+	INIT_LIST_HEAD(&queue->freelist);
+	cookie->fq_timeout = IOVA_SIMPLE_TIMEOUT;
+	cookie->simple_fq = queue;
+
+	return 0;
+}
+
+static void iommu_dma_free_simple(struct iommu_dma_cookie *cookie)
+{
+	if (!cookie->simple_fq)
+		return;
+
+	put_pages_list(&cookie->simple_fq->freelist);
+	vfree(cookie->simple_fq);
+	cookie->simple_fq = NULL;
+}
+
 #define ring_for_each_percpu(i, fq) \
 	for ((i) = (fq)->head; (i) != (fq)->tail; (i) = ((i) + 1) % IOVA_PERCPU_SIZE)
 
@@ -169,12 +287,23 @@ static void flush_percpu(struct iommu_dma_cookie *cookie)
 	}
 }
 
+static void iommu_dma_flush_fq(struct iommu_dma_cookie *cookie)
+{
+	if (!cookie->fq_domain)
+		return;
+
+	if (cookie->fq_domain->type == IOMMU_DOMAIN_DMA_FQ)
+		flush_percpu(cookie);
+	else
+		flush_simple_lock(cookie);
+}
+
 static void fq_flush_timeout(struct timer_list *t)
 {
 	struct iommu_dma_cookie *cookie = from_timer(cookie, t, fq_timer);
 
 	atomic_set(&cookie->fq_timer_on, 0);
-	flush_percpu(cookie);
+	iommu_dma_flush_fq(cookie);
 }
 
 static void queue_iova_percpu(struct iommu_dma_cookie *cookie,
@@ -223,13 +352,16 @@ static void queue_iova(struct iommu_dma_cookie *cookie,
 	 */
 	smp_mb();
 
-	queue_iova_percpu(cookie, pfn, pages, freelist);
+	if (cookie->fq_domain->type == IOMMU_DOMAIN_DMA_FQ)
+		queue_iova_percpu(cookie, pfn, pages, freelist);
+	else
+		queue_iova_simple(cookie, pfn, pages, freelist);
 
 	/* Avoid false sharing as much as possible. */
 	if (!atomic_read(&cookie->fq_timer_on) &&
 	    !atomic_xchg(&cookie->fq_timer_on, 1))
 		mod_timer(&cookie->fq_timer,
-			  jiffies + msecs_to_jiffies(IOVA_PERCPU_TIMEOUT));
+			  jiffies + msecs_to_jiffies(cookie->fq_timeout));
 }
 
 static void iommu_dma_free_percpu(struct iommu_dma_cookie *cookie)
@@ -253,7 +385,10 @@ static void iommu_dma_free_fq(struct iommu_dma_cookie *cookie)
 {
 	del_timer_sync(&cookie->fq_timer);
 	/* The IOVAs will be torn down separately, so just free our queued pages */
-	iommu_dma_free_percpu(cookie);
+	if (cookie->fq_domain->type == IOMMU_DOMAIN_DMA_FQ)
+		iommu_dma_free_percpu(cookie);
+	else
+		iommu_dma_free_simple(cookie);
 }
 
 static int iommu_dma_init_percpu(struct iommu_dma_cookie *cookie)
@@ -280,6 +415,7 @@ static int iommu_dma_init_percpu(struct iommu_dma_cookie *cookie)
 			INIT_LIST_HEAD(&fq->entries[i].freelist);
 	}
 
+	cookie->fq_timeout = IOVA_PERCPU_TIMEOUT;
 	cookie->percpu_fq = queue;
 
 	return 0;
@@ -294,7 +430,10 @@ int iommu_dma_init_fq(struct iommu_domain *domain)
 	if (cookie->fq_domain)
 		return 0;
 
-	rc = iommu_dma_init_percpu(cookie);
+	if (domain->type == IOMMU_DOMAIN_DMA_FQ)
+		rc = iommu_dma_init_percpu(cookie);
+	else
+		rc = iommu_dma_init_simple(cookie);
 	if (rc) {
 		pr_warn("iova flush queue initialization failed\n");
 		return rc;
@@ -613,7 +752,9 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base,
 		goto done_unlock;
 
 	/* If the FQ fails we can simply fall back to strict mode */
-	if (domain->type == IOMMU_DOMAIN_DMA_FQ && iommu_dma_init_fq(domain))
+	if ((domain->type == IOMMU_DOMAIN_DMA_FQ ||
+	     domain->type == IOMMU_DOMAIN_DMA_SQ) &&
+	    iommu_dma_init_fq(domain))
 		domain->type = IOMMU_DOMAIN_DMA;
 
 	ret = iova_reserve_iommu_regions(dev, domain);
drivers/iommu/iommu.c +14 -5
index 4893c2429ca5..2b3a12799702 100644
@@ -140,6 +140,7 @@ static const char *iommu_domain_type_str(unsigned int t)
 		return "Unmanaged";
 	case IOMMU_DOMAIN_DMA:
 	case IOMMU_DOMAIN_DMA_FQ:
+	case IOMMU_DOMAIN_DMA_SQ:
 		return "Translated";
 	default:
 		return "Unknown";
@@ -437,7 +438,8 @@ early_param("iommu.strict", iommu_dma_setup);
 void iommu_set_dma_strict(void)
 {
 	iommu_dma_strict = true;
-	if (iommu_def_domain_type == IOMMU_DOMAIN_DMA_FQ)
+	if (iommu_def_domain_type == IOMMU_DOMAIN_DMA_FQ ||
+	    iommu_def_domain_type == IOMMU_DOMAIN_DMA_SQ)
 		iommu_def_domain_type = IOMMU_DOMAIN_DMA;
 }
 
@@ -638,6 +640,9 @@ static ssize_t iommu_group_show_type(struct iommu_group *group,
 		case IOMMU_DOMAIN_DMA_FQ:
 			type = "DMA-FQ\n";
 			break;
+		case IOMMU_DOMAIN_DMA_SQ:
+			type = "DMA-SQ\n";
+			break;
 		}
 	}
 	mutex_unlock(&group->mutex);
@@ -2908,10 +2913,11 @@ static int iommu_change_dev_def_domain(struct iommu_group *group,
 	}
 
 	/* We can bring up a flush queue without tearing down the domain */
-	if (type == IOMMU_DOMAIN_DMA_FQ && prev_dom->type == IOMMU_DOMAIN_DMA) {
+	if ((type == IOMMU_DOMAIN_DMA_FQ || type == IOMMU_DOMAIN_DMA_SQ) &&
+	    prev_dom->type == IOMMU_DOMAIN_DMA) {
 		ret = iommu_dma_init_fq(prev_dom);
 		if (!ret)
-			prev_dom->type = IOMMU_DOMAIN_DMA_FQ;
+			prev_dom->type = type;
 		goto out;
 	}
 
@@ -2982,6 +2988,8 @@ static ssize_t iommu_group_store_type(struct iommu_group *group,
 		req_type = IOMMU_DOMAIN_DMA;
 	else if (sysfs_streq(buf, "DMA-FQ"))
 		req_type = IOMMU_DOMAIN_DMA_FQ;
+	else if (sysfs_streq(buf, "DMA-SQ"))
+		req_type = IOMMU_DOMAIN_DMA_SQ;
 	else if (sysfs_streq(buf, "auto"))
 		req_type = 0;
 	else
@@ -3033,8 +3041,9 @@ static ssize_t iommu_group_store_type(struct iommu_group *group,
 
 	/* Check if the device in the group still has a driver bound to it */
 	device_lock(dev);
-	if (device_is_bound(dev) && !(req_type == IOMMU_DOMAIN_DMA_FQ &&
-	    group->default_domain->type == IOMMU_DOMAIN_DMA)) {
+	if (device_is_bound(dev) &&
+	    !((req_type == IOMMU_DOMAIN_DMA_FQ || req_type == IOMMU_DOMAIN_DMA_SQ) &&
+	      group->default_domain->type == IOMMU_DOMAIN_DMA)) {
 		pr_err_ratelimited("Device is still bound to driver\n");
 		ret = -EBUSY;
 		goto out;
drivers/iommu/s390-iommu.c +11 -0
index c2b8a7b96b8e..506f8b92931f 100644
@@ -324,6 +324,15 @@ static bool s390_iommu_capable(struct device *dev, enum iommu_cap cap)
 	}
 }
 
+static int s390_iommu_def_domain_type(struct device *dev)
+{
+	struct zpci_dev *zdev = to_zpci_dev(dev);
+
+	if (zdev->tlb_refresh)
+		return IOMMU_DOMAIN_DMA_SQ;
+	return IOMMU_DOMAIN_DMA_FQ;
+}
+
 static struct iommu_domain *s390_domain_alloc(unsigned domain_type)
 {
 	struct s390_domain *s390_domain;
@@ -331,6 +340,7 @@ static struct iommu_domain *s390_domain_alloc(unsigned domain_type)
 	switch (domain_type) {
 	case IOMMU_DOMAIN_DMA:
 	case IOMMU_DOMAIN_DMA_FQ:
+	case IOMMU_DOMAIN_DMA_SQ:
 	case IOMMU_DOMAIN_UNMANAGED:
 		break;
 	default:
@@ -774,6 +784,7 @@ subsys_initcall(s390_iommu_init);
 
 static const struct iommu_ops s390_iommu_ops = {
 	.capable = s390_iommu_capable,
+	.def_domain_type = s390_iommu_def_domain_type,
 	.domain_alloc = s390_domain_alloc,
 	.probe_device = s390_iommu_probe_device,
 	.probe_finalize = s390_iommu_probe_finalize,
include/linux/iommu.h +6 -0
index a325532aeab5..6c3fe62ec0df 100644
@@ -63,6 +63,7 @@ struct iommu_domain_geometry {
 					      implementation              */
 #define __IOMMU_DOMAIN_PT	(1U << 2)  /* Domain is identity mapped   */
 #define __IOMMU_DOMAIN_DMA_FQ	(1U << 3)  /* DMA-API uses flush queue    */
+#define __IOMMU_DOMAIN_DMA_SQ	(1U << 4)  /* DMA-API uses max fill queue */
 
 /*
  * This are the possible domain-types
@@ -77,6 +78,8 @@ struct iommu_domain_geometry {
  *				  certain optimizations for these domains
  *	IOMMU_DOMAIN_DMA_FQ	- As above, but definitely using batched TLB
  *				  invalidation.
+ *	IOMMU_DOMAIN_DMA_SQ	- As above, but batched invalidations are only
+ *				  flushed when running out of queue space.
  */
 #define IOMMU_DOMAIN_BLOCKED	(0U)
 #define IOMMU_DOMAIN_IDENTITY	(__IOMMU_DOMAIN_PT)
@@ -86,6 +89,9 @@ struct iommu_domain_geometry {
 #define IOMMU_DOMAIN_DMA_FQ	(__IOMMU_DOMAIN_PAGING |	\
 				 __IOMMU_DOMAIN_DMA_API |	\
 				 __IOMMU_DOMAIN_DMA_FQ)
+#define IOMMU_DOMAIN_DMA_SQ	(__IOMMU_DOMAIN_PAGING |	\
+				 __IOMMU_DOMAIN_DMA_API |	\
+				 __IOMMU_DOMAIN_DMA_SQ)
 
 struct iommu_domain {
 	unsigned type;
Signature · 1 line
2.34.1
maintainer
#8 lore raw

Re: [RFC 5/6] iommu/dma: Add simple batching flush queue implementation

Quoted 15 lines · Niklas Schnelle

Having dma-iommu prepared for alternative flush queue implementations we add a simple per-domain flush queue that optimzes for scenarios where global IOTLB flushes are used but are quite expensive and poorly parallelized. This is for example the case when IOTLB flushes are used to trigger updates to an underlying hypervisor's shadow tables and approximates the global flushing scheme previously in use on s390.

This is achieved by having a per-domain global flush queue that allows queuing a much large number of lazily freed IOVAs than the per-CPU flush queues. While using a single queue reduces parallelism this is not a problem when global IOTLB flushes are synchronized in the hypervisor anyway. While this flush queue allows queuing a large number of IOVAs we do limit the time a freed IOVA remains accessible by hardware to 1 second using a timeout that is reset on flush.

Quoted 2 lines

Enable this new flush queue implementation by default on s390 systems which use IOTLB flushes to trigger shadowing namely z/VM and KVM guests.

There seems to be way more duplication here than there probably needs to be. Is there any reason we can't keep using the same iova_fq and iova_fq_entry structures and just refactor how cookie->fq is allocated/freed/dereferenced to choose between percpu and direct? Making the queue size and timout variable - and thus potentially tuneable - isn't necessarily a bad thing for percpu queues either (the size can be constrained to powers of two and stored as a mask to keep the wrapping arithemetic efficient).

Thanks,
Robin.

Quoted 370 lines
---
  drivers/iommu/dma-iommu.c  | 157 +++++++++++++++++++++++++++++++++++--
  drivers/iommu/iommu.c      |  19 +++--
  drivers/iommu/s390-iommu.c |  11 +++
  include/linux/iommu.h      |   6 ++
  4 files changed, 180 insertions(+), 13 deletions(-)
drivers/iommu/dma-iommu.c +149 -8
index 77d969f5aad7..427fb84f50c3 100644
@@ -49,8 +49,13 @@ struct iommu_dma_cookie {
  		/* Full allocator for IOMMU_DMA_IOVA_COOKIE */
  		struct {
  			struct iova_domain		iovad;
-			/* Flush queue */
-			struct iova_percpu __percpu *percpu_fq;
+			/* Flush queues */
+			union {
+				struct iova_percpu __percpu *percpu_fq;
+				struct iova_simple *simple_fq;
+			};
+			/* Queue timeout in milliseconds */
+			unsigned int		fq_timeout;
  			/* Number of TLB flushes that have been started */
  			atomic64_t		fq_flush_start_cnt;
  			/* Number of TLB flushes that have been finished */
@@ -104,6 +109,119 @@ struct iova_percpu {
  	spinlock_t lock;
  };
 
+/* Simplified batched flush queue for expensive IOTLB flushes */
+#define IOVA_SIMPLE_SIZE	32768
+/* Maximum time in milliseconds an IOVA can remain lazily freed */
+#define IOVA_SIMPLE_TIMEOUT	1000
+
+struct iova_simple_entry {
+	unsigned long iova_pfn;
+	unsigned long pages;
+};
+
+struct iova_simple {
+	/* Unlike iova_percpu we use a single queue lock */
+	spinlock_t lock;
+	unsigned int tail;
+	unsigned long total_pages;
+	struct list_head freelist;
+	struct iova_simple_entry entries[];
+};
+
+static bool is_full_simple(struct iommu_dma_cookie *cookie)
+{
+	struct iommu_domain *fq_domain = cookie->fq_domain;
+	struct iova_domain *iovad = &cookie->iovad;
+	struct iova_simple *sq = cookie->simple_fq;
+	unsigned long aperture_pages;
+
+	assert_spin_locked(&sq->lock);
+
+	/* If more than 7/8 the aperture is batched let's flush */
+	aperture_pages = ((fq_domain->geometry.aperture_end +  1) -
+		fq_domain->geometry.aperture_start) >> iova_shift(iovad);
+	aperture_pages -= aperture_pages >> 3;
+
+	return (sq->tail >= IOVA_SIMPLE_SIZE ||
+		sq->total_pages >= aperture_pages);
+}
+
+static void flush_simple(struct iommu_dma_cookie *cookie)
+{
+	struct iova_simple *sq = cookie->simple_fq;
+	unsigned int i;
+
+	assert_spin_locked(&sq->lock);
+	/* We're flushing so postpone timeout */
+	mod_timer(&cookie->fq_timer,
+		  jiffies + msecs_to_jiffies(cookie->fq_timeout));
+	cookie->fq_domain->ops->flush_iotlb_all(cookie->fq_domain);
+
+	put_pages_list(&sq->freelist);
+	for (i = 0; i < sq->tail; i++) {
+		free_iova_fast(&cookie->iovad,
+			       sq->entries[i].iova_pfn,
+			       sq->entries[i].pages);
+	}
+	sq->tail = 0;
+	sq->total_pages = 0;
+}
+
+static void flush_simple_lock(struct iommu_dma_cookie *cookie)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&cookie->simple_fq->lock, flags);
+	flush_simple(cookie);
+	spin_unlock_irqrestore(&cookie->simple_fq->lock, flags);
+}
+
+static void queue_iova_simple(struct iommu_dma_cookie *cookie,
+			      unsigned long pfn, unsigned long pages,
+			      struct list_head *freelist)
+{
+	struct iova_simple *sq = cookie->simple_fq;
+	unsigned long flags;
+	unsigned int idx;
+
+	spin_lock_irqsave(&sq->lock, flags);
+	if (is_full_simple(cookie))
+		flush_simple(cookie);
+	idx = sq->tail++;
+
+	sq->entries[idx].iova_pfn = pfn;
+	sq->entries[idx].pages    = pages;
+	list_splice(freelist, &sq->freelist);
+	sq->total_pages += pages;
+	spin_unlock_irqrestore(&sq->lock, flags);
+}
+
+static int iommu_dma_init_simple(struct iommu_dma_cookie *cookie)
+{
+	struct iova_simple *queue;
+
+	queue = vzalloc(sizeof(*queue) +
+			IOVA_SIMPLE_SIZE * sizeof(struct iova_simple_entry));
+	if (!queue)
+		return -ENOMEM;
+
+	INIT_LIST_HEAD(&queue->freelist);
+	cookie->fq_timeout = IOVA_SIMPLE_TIMEOUT;
+	cookie->simple_fq = queue;
+
+	return 0;
+}
+
+static void iommu_dma_free_simple(struct iommu_dma_cookie *cookie)
+{
+	if (!cookie->simple_fq)
+		return;
+
+	put_pages_list(&cookie->simple_fq->freelist);
+	vfree(cookie->simple_fq);
+	cookie->simple_fq = NULL;
+}
+
  #define ring_for_each_percpu(i, fq) \
  	for ((i) = (fq)->head; (i) != (fq)->tail; (i) = ((i) + 1) % IOVA_PERCPU_SIZE)
 
@@ -169,12 +287,23 @@ static void flush_percpu(struct iommu_dma_cookie *cookie)
  	}
  }
 
+static void iommu_dma_flush_fq(struct iommu_dma_cookie *cookie)
+{
+	if (!cookie->fq_domain)
+		return;
+
+	if (cookie->fq_domain->type == IOMMU_DOMAIN_DMA_FQ)
+		flush_percpu(cookie);
+	else
+		flush_simple_lock(cookie);
+}
+
  static void fq_flush_timeout(struct timer_list *t)
  {
  	struct iommu_dma_cookie *cookie = from_timer(cookie, t, fq_timer);
 
  	atomic_set(&cookie->fq_timer_on, 0);
-	flush_percpu(cookie);
+	iommu_dma_flush_fq(cookie);
  }
 
  static void queue_iova_percpu(struct iommu_dma_cookie *cookie,
@@ -223,13 +352,16 @@ static void queue_iova(struct iommu_dma_cookie *cookie,
  	 */
  	smp_mb();
 
-	queue_iova_percpu(cookie, pfn, pages, freelist);
+	if (cookie->fq_domain->type == IOMMU_DOMAIN_DMA_FQ)
+		queue_iova_percpu(cookie, pfn, pages, freelist);
+	else
+		queue_iova_simple(cookie, pfn, pages, freelist);
 
  	/* Avoid false sharing as much as possible. */
  	if (!atomic_read(&cookie->fq_timer_on) &&
  	    !atomic_xchg(&cookie->fq_timer_on, 1))
  		mod_timer(&cookie->fq_timer,
-			  jiffies + msecs_to_jiffies(IOVA_PERCPU_TIMEOUT));
+			  jiffies + msecs_to_jiffies(cookie->fq_timeout));
  }
 
  static void iommu_dma_free_percpu(struct iommu_dma_cookie *cookie)
@@ -253,7 +385,10 @@ static void iommu_dma_free_fq(struct iommu_dma_cookie *cookie)
  {
  	del_timer_sync(&cookie->fq_timer);
  	/* The IOVAs will be torn down separately, so just free our queued pages */
-	iommu_dma_free_percpu(cookie);
+	if (cookie->fq_domain->type == IOMMU_DOMAIN_DMA_FQ)
+		iommu_dma_free_percpu(cookie);
+	else
+		iommu_dma_free_simple(cookie);
  }
 
  static int iommu_dma_init_percpu(struct iommu_dma_cookie *cookie)
@@ -280,6 +415,7 @@ static int iommu_dma_init_percpu(struct iommu_dma_cookie *cookie)
  			INIT_LIST_HEAD(&fq->entries[i].freelist);
  	}
 
+	cookie->fq_timeout = IOVA_PERCPU_TIMEOUT;
  	cookie->percpu_fq = queue;
 
  	return 0;
@@ -294,7 +430,10 @@ int iommu_dma_init_fq(struct iommu_domain *domain)
  	if (cookie->fq_domain)
  		return 0;
 
-	rc = iommu_dma_init_percpu(cookie);
+	if (domain->type == IOMMU_DOMAIN_DMA_FQ)
+		rc = iommu_dma_init_percpu(cookie);
+	else
+		rc = iommu_dma_init_simple(cookie);
  	if (rc) {
  		pr_warn("iova flush queue initialization failed\n");
  		return rc;
@@ -613,7 +752,9 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base,
  		goto done_unlock;
 
  	/* If the FQ fails we can simply fall back to strict mode */
-	if (domain->type == IOMMU_DOMAIN_DMA_FQ && iommu_dma_init_fq(domain))
+	if ((domain->type == IOMMU_DOMAIN_DMA_FQ ||
+	     domain->type == IOMMU_DOMAIN_DMA_SQ) &&
+	    iommu_dma_init_fq(domain))
  		domain->type = IOMMU_DOMAIN_DMA;
 
  	ret = iova_reserve_iommu_regions(dev, domain);
drivers/iommu/iommu.c +14 -5
index 4893c2429ca5..2b3a12799702 100644
@@ -140,6 +140,7 @@ static const char *iommu_domain_type_str(unsigned int t)
  		return "Unmanaged";
  	case IOMMU_DOMAIN_DMA:
  	case IOMMU_DOMAIN_DMA_FQ:
+	case IOMMU_DOMAIN_DMA_SQ:
  		return "Translated";
  	default:
  		return "Unknown";
@@ -437,7 +438,8 @@ early_param("iommu.strict", iommu_dma_setup);
  void iommu_set_dma_strict(void)
  {
  	iommu_dma_strict = true;
-	if (iommu_def_domain_type == IOMMU_DOMAIN_DMA_FQ)
+	if (iommu_def_domain_type == IOMMU_DOMAIN_DMA_FQ ||
+	    iommu_def_domain_type == IOMMU_DOMAIN_DMA_SQ)
  		iommu_def_domain_type = IOMMU_DOMAIN_DMA;
  }
 
@@ -638,6 +640,9 @@ static ssize_t iommu_group_show_type(struct iommu_group *group,
  		case IOMMU_DOMAIN_DMA_FQ:
  			type = "DMA-FQ\n";
  			break;
+		case IOMMU_DOMAIN_DMA_SQ:
+			type = "DMA-SQ\n";
+			break;
  		}
  	}
  	mutex_unlock(&group->mutex);
@@ -2908,10 +2913,11 @@ static int iommu_change_dev_def_domain(struct iommu_group *group,
  	}
 
  	/* We can bring up a flush queue without tearing down the domain */
-	if (type == IOMMU_DOMAIN_DMA_FQ && prev_dom->type == IOMMU_DOMAIN_DMA) {
+	if ((type == IOMMU_DOMAIN_DMA_FQ || type == IOMMU_DOMAIN_DMA_SQ) &&
+	    prev_dom->type == IOMMU_DOMAIN_DMA) {
  		ret = iommu_dma_init_fq(prev_dom);
  		if (!ret)
-			prev_dom->type = IOMMU_DOMAIN_DMA_FQ;
+			prev_dom->type = type;
  		goto out;
  	}
 
@@ -2982,6 +2988,8 @@ static ssize_t iommu_group_store_type(struct iommu_group *group,
  		req_type = IOMMU_DOMAIN_DMA;
  	else if (sysfs_streq(buf, "DMA-FQ"))
  		req_type = IOMMU_DOMAIN_DMA_FQ;
+	else if (sysfs_streq(buf, "DMA-SQ"))
+		req_type = IOMMU_DOMAIN_DMA_SQ;
  	else if (sysfs_streq(buf, "auto"))
  		req_type = 0;
  	else
@@ -3033,8 +3041,9 @@ static ssize_t iommu_group_store_type(struct iommu_group *group,
 
  	/* Check if the device in the group still has a driver bound to it */
  	device_lock(dev);
-	if (device_is_bound(dev) && !(req_type == IOMMU_DOMAIN_DMA_FQ &&
-	    group->default_domain->type == IOMMU_DOMAIN_DMA)) {
+	if (device_is_bound(dev) &&
+	    !((req_type == IOMMU_DOMAIN_DMA_FQ || req_type == IOMMU_DOMAIN_DMA_SQ) &&
+	      group->default_domain->type == IOMMU_DOMAIN_DMA)) {
  		pr_err_ratelimited("Device is still bound to driver\n");
  		ret = -EBUSY;
  		goto out;
drivers/iommu/s390-iommu.c +11 -0
index c2b8a7b96b8e..506f8b92931f 100644
@@ -324,6 +324,15 @@ static bool s390_iommu_capable(struct device *dev, enum iommu_cap cap)
  	}
  }
 
+static int s390_iommu_def_domain_type(struct device *dev)
+{
+	struct zpci_dev *zdev = to_zpci_dev(dev);
+
+	if (zdev->tlb_refresh)
+		return IOMMU_DOMAIN_DMA_SQ;
+	return IOMMU_DOMAIN_DMA_FQ;
+}
+
  static struct iommu_domain *s390_domain_alloc(unsigned domain_type)
  {
  	struct s390_domain *s390_domain;
@@ -331,6 +340,7 @@ static struct iommu_domain *s390_domain_alloc(unsigned domain_type)
  	switch (domain_type) {
  	case IOMMU_DOMAIN_DMA:
  	case IOMMU_DOMAIN_DMA_FQ:
+	case IOMMU_DOMAIN_DMA_SQ:
  	case IOMMU_DOMAIN_UNMANAGED:
  		break;
  	default:
@@ -774,6 +784,7 @@ subsys_initcall(s390_iommu_init);
 
  static const struct iommu_ops s390_iommu_ops = {
  	.capable = s390_iommu_capable,
+	.def_domain_type = s390_iommu_def_domain_type,
  	.domain_alloc = s390_domain_alloc,
  	.probe_device = s390_iommu_probe_device,
  	.probe_finalize = s390_iommu_probe_finalize,
include/linux/iommu.h +6 -0
index a325532aeab5..6c3fe62ec0df 100644
@@ -63,6 +63,7 @@ struct iommu_domain_geometry {
  					      implementation              */
  #define __IOMMU_DOMAIN_PT	(1U << 2)  /* Domain is identity mapped   */
  #define __IOMMU_DOMAIN_DMA_FQ	(1U << 3)  /* DMA-API uses flush queue    */
+#define __IOMMU_DOMAIN_DMA_SQ	(1U << 4)  /* DMA-API uses max fill queue */
 
  /*
   * This are the possible domain-types
@@ -77,6 +78,8 @@ struct iommu_domain_geometry {
   *				  certain optimizations for these domains
   *	IOMMU_DOMAIN_DMA_FQ	- As above, but definitely using batched TLB
   *				  invalidation.
+ *	IOMMU_DOMAIN_DMA_SQ	- As above, but batched invalidations are only
+ *				  flushed when running out of queue space.
   */
  #define IOMMU_DOMAIN_BLOCKED	(0U)
  #define IOMMU_DOMAIN_IDENTITY	(__IOMMU_DOMAIN_PT)
@@ -86,6 +89,9 @@ struct iommu_domain_geometry {
  #define IOMMU_DOMAIN_DMA_FQ	(__IOMMU_DOMAIN_PAGING |	\
  				 __IOMMU_DOMAIN_DMA_API |	\
  				 __IOMMU_DOMAIN_DMA_FQ)
+#define IOMMU_DOMAIN_DMA_SQ	(__IOMMU_DOMAIN_PAGING |	\
+				 __IOMMU_DOMAIN_DMA_API |	\
+				 __IOMMU_DOMAIN_DMA_SQ)
 
  struct iommu_domain {
  	unsigned type;
author maintainer
#9 lore raw

Re: [RFC 5/6] iommu/dma: Add simple batching flush queue implementation

Quoted 28 lines · Robin Murphy
Quoted 15 lines · Niklas Schnelle

Having dma-iommu prepared for alternative flush queue implementations we add a simple per-domain flush queue that optimzes for scenarios where global IOTLB flushes are used but are quite expensive and poorly parallelized. This is for example the case when IOTLB flushes are used to trigger updates to an underlying hypervisor's shadow tables and approximates the global flushing scheme previously in use on s390.

This is achieved by having a per-domain global flush queue that allows queuing a much large number of lazily freed IOVAs than the per-CPU flush queues. While using a single queue reduces parallelism this is not a problem when global IOTLB flushes are synchronized in the hypervisor anyway. While this flush queue allows queuing a large number of IOVAs we do limit the time a freed IOVA remains accessible by hardware to 1 second using a timeout that is reset on flush.

Quoted 2 lines

Enable this new flush queue implementation by default on s390 systems which use IOTLB flushes to trigger shadowing namely z/VM and KVM guests.

There seems to be way more duplication here than there probably needs to be. Is there any reason we can't keep using the same iova_fq and iova_fq_entry structures and just refactor how cookie->fq is allocated/freed/dereferenced to choose between percpu and direct? Making the queue size and timout variable - and thus potentially tuneable - isn't necessarily a bad thing for percpu queues either (the size can be constrained to powers of two and stored as a mask to keep the wrapping arithemetic efficient).

Quoted 2 lines

Thanks,
Robin.

There are a few reasons I did not re-use the same structs. The first reason is that I wasn't sure where the "__percpu" attribute would go. Maybe one would need a union with and without it in the cookie?

Then my queue had a few different constraints. As I flush all entries at once I only need one freelist not one per IOVA. I also don't need the counter for the same reason. Thus I end up with just 16 bytes per IOVA compared with 40 for the per CPU one. Having only one freelist of course also impacts how the flushing works. Now since struct iova_fq directly embeds the entries I was afraid that making them variable sized adds another indirection slowing down the normal case. Looking at it again I'm wondering though if this could be worked around by using a dynamically sized struct like I did for the struct iova_simple. Possibly even with different entry structs. Are there dynamically sized structs where the dynamic member is a union?

I also do handle the timeout slightly different as for the new queue I postpone it when a flush happens. Though I guess that could be done for percpu too, I'm not sure how expensive modifying the timer is and I wanted to absolutely minimize the overhead added for the existing case.

In short I had some reasons but I'm all for sharing more of the logic and code if possible and welcome ideas.

As a motivating aside some of my testing indicates that with all of this we end up significantly exceeding the performance of our current DMA API implementation in the virtualized case.

Thanks,
Niklas

Quoted 371 lines
Quoted 371 lines
---
  drivers/iommu/dma-iommu.c  | 157 +++++++++++++++++++++++++++++++++++--
  drivers/iommu/iommu.c      |  19 +++--
  drivers/iommu/s390-iommu.c |  11 +++
  include/linux/iommu.h      |   6 ++
  4 files changed, 180 insertions(+), 13 deletions(-)
drivers/iommu/dma-iommu.c +149 -8
index 77d969f5aad7..427fb84f50c3 100644
@@ -49,8 +49,13 @@ struct iommu_dma_cookie {
  		/* Full allocator for IOMMU_DMA_IOVA_COOKIE */
  		struct {
  			struct iova_domain		iovad;
-			/* Flush queue */
-			struct iova_percpu __percpu *percpu_fq;
+			/* Flush queues */
+			union {
+				struct iova_percpu __percpu *percpu_fq;
+				struct iova_simple *simple_fq;
+			};
+			/* Queue timeout in milliseconds */
+			unsigned int		fq_timeout;
  			/* Number of TLB flushes that have been started */
  			atomic64_t		fq_flush_start_cnt;
  			/* Number of TLB flushes that have been finished */
@@ -104,6 +109,119 @@ struct iova_percpu {
  	spinlock_t lock;
  };
 
+/* Simplified batched flush queue for expensive IOTLB flushes */
+#define IOVA_SIMPLE_SIZE	32768
+/* Maximum time in milliseconds an IOVA can remain lazily freed */
+#define IOVA_SIMPLE_TIMEOUT	1000
+
+struct iova_simple_entry {
+	unsigned long iova_pfn;
+	unsigned long pages;
+};
+
+struct iova_simple {
+	/* Unlike iova_percpu we use a single queue lock */
+	spinlock_t lock;
+	unsigned int tail;
+	unsigned long total_pages;
+	struct list_head freelist;
+	struct iova_simple_entry entries[];
+};
+
+static bool is_full_simple(struct iommu_dma_cookie *cookie)
+{
+	struct iommu_domain *fq_domain = cookie->fq_domain;
+	struct iova_domain *iovad = &cookie->iovad;
+	struct iova_simple *sq = cookie->simple_fq;
+	unsigned long aperture_pages;
+
+	assert_spin_locked(&sq->lock);
+
+	/* If more than 7/8 the aperture is batched let's flush */
+	aperture_pages = ((fq_domain->geometry.aperture_end +  1) -
+		fq_domain->geometry.aperture_start) >> iova_shift(iovad);
+	aperture_pages -= aperture_pages >> 3;
+
+	return (sq->tail >= IOVA_SIMPLE_SIZE ||
+		sq->total_pages >= aperture_pages);
+}
+
+static void flush_simple(struct iommu_dma_cookie *cookie)
+{
+	struct iova_simple *sq = cookie->simple_fq;
+	unsigned int i;
+
+	assert_spin_locked(&sq->lock);
+	/* We're flushing so postpone timeout */
+	mod_timer(&cookie->fq_timer,
+		  jiffies + msecs_to_jiffies(cookie->fq_timeout));
+	cookie->fq_domain->ops->flush_iotlb_all(cookie->fq_domain);
+
+	put_pages_list(&sq->freelist);
+	for (i = 0; i < sq->tail; i++) {
+		free_iova_fast(&cookie->iovad,
+			       sq->entries[i].iova_pfn,
+			       sq->entries[i].pages);
+	}
+	sq->tail = 0;
+	sq->total_pages = 0;
+}
+
+static void flush_simple_lock(struct iommu_dma_cookie *cookie)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&cookie->simple_fq->lock, flags);
+	flush_simple(cookie);
+	spin_unlock_irqrestore(&cookie->simple_fq->lock, flags);
+}
+
+static void queue_iova_simple(struct iommu_dma_cookie *cookie,
+			      unsigned long pfn, unsigned long pages,
+			      struct list_head *freelist)
+{
+	struct iova_simple *sq = cookie->simple_fq;
+	unsigned long flags;
+	unsigned int idx;
+
+	spin_lock_irqsave(&sq->lock, flags);
+	if (is_full_simple(cookie))
+		flush_simple(cookie);
+	idx = sq->tail++;
+
+	sq->entries[idx].iova_pfn = pfn;
+	sq->entries[idx].pages    = pages;
+	list_splice(freelist, &sq->freelist);
+	sq->total_pages += pages;
+	spin_unlock_irqrestore(&sq->lock, flags);
+}
+
+static int iommu_dma_init_simple(struct iommu_dma_cookie *cookie)
+{
+	struct iova_simple *queue;
+
+	queue = vzalloc(sizeof(*queue) +
+			IOVA_SIMPLE_SIZE * sizeof(struct iova_simple_entry));
+	if (!queue)
+		return -ENOMEM;
+
+	INIT_LIST_HEAD(&queue->freelist);
+	cookie->fq_timeout = IOVA_SIMPLE_TIMEOUT;
+	cookie->simple_fq = queue;
+
+	return 0;
+}
+
+static void iommu_dma_free_simple(struct iommu_dma_cookie *cookie)
+{
+	if (!cookie->simple_fq)
+		return;
+
+	put_pages_list(&cookie->simple_fq->freelist);
+	vfree(cookie->simple_fq);
+	cookie->simple_fq = NULL;
+}
+
  #define ring_for_each_percpu(i, fq) \
  	for ((i) = (fq)->head; (i) != (fq)->tail; (i) = ((i) + 1) % IOVA_PERCPU_SIZE)
 
@@ -169,12 +287,23 @@ static void flush_percpu(struct iommu_dma_cookie *cookie)
  	}
  }
 
+static void iommu_dma_flush_fq(struct iommu_dma_cookie *cookie)
+{
+	if (!cookie->fq_domain)
+		return;
+
+	if (cookie->fq_domain->type == IOMMU_DOMAIN_DMA_FQ)
+		flush_percpu(cookie);
+	else
+		flush_simple_lock(cookie);
+}
+
  static void fq_flush_timeout(struct timer_list *t)
  {
  	struct iommu_dma_cookie *cookie = from_timer(cookie, t, fq_timer);
 
  	atomic_set(&cookie->fq_timer_on, 0);
-	flush_percpu(cookie);
+	iommu_dma_flush_fq(cookie);
  }
 
  static void queue_iova_percpu(struct iommu_dma_cookie *cookie,
@@ -223,13 +352,16 @@ static void queue_iova(struct iommu_dma_cookie *cookie,
  	 */
  	smp_mb();
 
-	queue_iova_percpu(cookie, pfn, pages, freelist);
+	if (cookie->fq_domain->type == IOMMU_DOMAIN_DMA_FQ)
+		queue_iova_percpu(cookie, pfn, pages, freelist);
+	else
+		queue_iova_simple(cookie, pfn, pages, freelist);
 
  	/* Avoid false sharing as much as possible. */
  	if (!atomic_read(&cookie->fq_timer_on) &&
  	    !atomic_xchg(&cookie->fq_timer_on, 1))
  		mod_timer(&cookie->fq_timer,
-			  jiffies + msecs_to_jiffies(IOVA_PERCPU_TIMEOUT));
+			  jiffies + msecs_to_jiffies(cookie->fq_timeout));
  }
 
  static void iommu_dma_free_percpu(struct iommu_dma_cookie *cookie)
@@ -253,7 +385,10 @@ static void iommu_dma_free_fq(struct iommu_dma_cookie *cookie)
  {
  	del_timer_sync(&cookie->fq_timer);
  	/* The IOVAs will be torn down separately, so just free our queued pages */
-	iommu_dma_free_percpu(cookie);
+	if (cookie->fq_domain->type == IOMMU_DOMAIN_DMA_FQ)
+		iommu_dma_free_percpu(cookie);
+	else
+		iommu_dma_free_simple(cookie);
  }
 
  static int iommu_dma_init_percpu(struct iommu_dma_cookie *cookie)
@@ -280,6 +415,7 @@ static int iommu_dma_init_percpu(struct iommu_dma_cookie *cookie)
  			INIT_LIST_HEAD(&fq->entries[i].freelist);
  	}
 
+	cookie->fq_timeout = IOVA_PERCPU_TIMEOUT;
  	cookie->percpu_fq = queue;
 
  	return 0;
@@ -294,7 +430,10 @@ int iommu_dma_init_fq(struct iommu_domain *domain)
  	if (cookie->fq_domain)
  		return 0;
 
-	rc = iommu_dma_init_percpu(cookie);
+	if (domain->type == IOMMU_DOMAIN_DMA_FQ)
+		rc = iommu_dma_init_percpu(cookie);
+	else
+		rc = iommu_dma_init_simple(cookie);
  	if (rc) {
  		pr_warn("iova flush queue initialization failed\n");
  		return rc;
@@ -613,7 +752,9 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base,
  		goto done_unlock;
 
  	/* If the FQ fails we can simply fall back to strict mode */
-	if (domain->type == IOMMU_DOMAIN_DMA_FQ && iommu_dma_init_fq(domain))
+	if ((domain->type == IOMMU_DOMAIN_DMA_FQ ||
+	     domain->type == IOMMU_DOMAIN_DMA_SQ) &&
+	    iommu_dma_init_fq(domain))
  		domain->type = IOMMU_DOMAIN_DMA;
 
  	ret = iova_reserve_iommu_regions(dev, domain);
drivers/iommu/iommu.c +14 -5
index 4893c2429ca5..2b3a12799702 100644
@@ -140,6 +140,7 @@ static const char *iommu_domain_type_str(unsigned int t)
  		return "Unmanaged";
  	case IOMMU_DOMAIN_DMA:
  	case IOMMU_DOMAIN_DMA_FQ:
+	case IOMMU_DOMAIN_DMA_SQ:
  		return "Translated";
  	default:
  		return "Unknown";
@@ -437,7 +438,8 @@ early_param("iommu.strict", iommu_dma_setup);
  void iommu_set_dma_strict(void)
  {
  	iommu_dma_strict = true;
-	if (iommu_def_domain_type == IOMMU_DOMAIN_DMA_FQ)
+	if (iommu_def_domain_type == IOMMU_DOMAIN_DMA_FQ ||
+	    iommu_def_domain_type == IOMMU_DOMAIN_DMA_SQ)
  		iommu_def_domain_type = IOMMU_DOMAIN_DMA;
  }
 
@@ -638,6 +640,9 @@ static ssize_t iommu_group_show_type(struct iommu_group *group,
  		case IOMMU_DOMAIN_DMA_FQ:
  			type = "DMA-FQ\n";
  			break;
+		case IOMMU_DOMAIN_DMA_SQ:
+			type = "DMA-SQ\n";
+			break;
  		}
  	}
  	mutex_unlock(&group->mutex);
@@ -2908,10 +2913,11 @@ static int iommu_change_dev_def_domain(struct iommu_group *group,
  	}
 
  	/* We can bring up a flush queue without tearing down the domain */
-	if (type == IOMMU_DOMAIN_DMA_FQ && prev_dom->type == IOMMU_DOMAIN_DMA) {
+	if ((type == IOMMU_DOMAIN_DMA_FQ || type == IOMMU_DOMAIN_DMA_SQ) &&
+	    prev_dom->type == IOMMU_DOMAIN_DMA) {
  		ret = iommu_dma_init_fq(prev_dom);
  		if (!ret)
-			prev_dom->type = IOMMU_DOMAIN_DMA_FQ;
+			prev_dom->type = type;
  		goto out;
  	}
 
@@ -2982,6 +2988,8 @@ static ssize_t iommu_group_store_type(struct iommu_group *group,
  		req_type = IOMMU_DOMAIN_DMA;
  	else if (sysfs_streq(buf, "DMA-FQ"))
  		req_type = IOMMU_DOMAIN_DMA_FQ;
+	else if (sysfs_streq(buf, "DMA-SQ"))
+		req_type = IOMMU_DOMAIN_DMA_SQ;
  	else if (sysfs_streq(buf, "auto"))
  		req_type = 0;
  	else
@@ -3033,8 +3041,9 @@ static ssize_t iommu_group_store_type(struct iommu_group *group,
 
  	/* Check if the device in the group still has a driver bound to it */
  	device_lock(dev);
-	if (device_is_bound(dev) && !(req_type == IOMMU_DOMAIN_DMA_FQ &&
-	    group->default_domain->type == IOMMU_DOMAIN_DMA)) {
+	if (device_is_bound(dev) &&
+	    !((req_type == IOMMU_DOMAIN_DMA_FQ || req_type == IOMMU_DOMAIN_DMA_SQ) &&
+	      group->default_domain->type == IOMMU_DOMAIN_DMA)) {
  		pr_err_ratelimited("Device is still bound to driver\n");
  		ret = -EBUSY;
  		goto out;
drivers/iommu/s390-iommu.c +11 -0
index c2b8a7b96b8e..506f8b92931f 100644
@@ -324,6 +324,15 @@ static bool s390_iommu_capable(struct device *dev, enum iommu_cap cap)
  	}
  }
 
+static int s390_iommu_def_domain_type(struct device *dev)
+{
+	struct zpci_dev *zdev = to_zpci_dev(dev);
+
+	if (zdev->tlb_refresh)
+		return IOMMU_DOMAIN_DMA_SQ;
+	return IOMMU_DOMAIN_DMA_FQ;
+}
+
  static struct iommu_domain *s390_domain_alloc(unsigned domain_type)
  {
  	struct s390_domain *s390_domain;
@@ -331,6 +340,7 @@ static struct iommu_domain *s390_domain_alloc(unsigned domain_type)
  	switch (domain_type) {
  	case IOMMU_DOMAIN_DMA:
  	case IOMMU_DOMAIN_DMA_FQ:
+	case IOMMU_DOMAIN_DMA_SQ:
  	case IOMMU_DOMAIN_UNMANAGED:
  		break;
  	default:
@@ -774,6 +784,7 @@ subsys_initcall(s390_iommu_init);
 
  static const struct iommu_ops s390_iommu_ops = {
  	.capable = s390_iommu_capable,
+	.def_domain_type = s390_iommu_def_domain_type,
  	.domain_alloc = s390_domain_alloc,
  	.probe_device = s390_iommu_probe_device,
  	.probe_finalize = s390_iommu_probe_finalize,
include/linux/iommu.h +6 -0
index a325532aeab5..6c3fe62ec0df 100644
@@ -63,6 +63,7 @@ struct iommu_domain_geometry {
  					      implementation              */
  #define __IOMMU_DOMAIN_PT	(1U << 2)  /* Domain is identity mapped   */
  #define __IOMMU_DOMAIN_DMA_FQ	(1U << 3)  /* DMA-API uses flush queue    */
+#define __IOMMU_DOMAIN_DMA_SQ	(1U << 4)  /* DMA-API uses max fill queue */
 
  /*
   * This are the possible domain-types
@@ -77,6 +78,8 @@ struct iommu_domain_geometry {
   *				  certain optimizations for these domains
   *	IOMMU_DOMAIN_DMA_FQ	- As above, but definitely using batched TLB
   *				  invalidation.
+ *	IOMMU_DOMAIN_DMA_SQ	- As above, but batched invalidations are only
+ *				  flushed when running out of queue space.
   */
  #define IOMMU_DOMAIN_BLOCKED	(0U)
  #define IOMMU_DOMAIN_IDENTITY	(__IOMMU_DOMAIN_PT)
@@ -86,6 +89,9 @@ struct iommu_domain_geometry {
  #define IOMMU_DOMAIN_DMA_FQ	(__IOMMU_DOMAIN_PAGING |	\
  				 __IOMMU_DOMAIN_DMA_API |	\
  				 __IOMMU_DOMAIN_DMA_FQ)
+#define IOMMU_DOMAIN_DMA_SQ	(__IOMMU_DOMAIN_PAGING |	\
+				 __IOMMU_DOMAIN_DMA_API |	\
+				 __IOMMU_DOMAIN_DMA_SQ)
 
  struct iommu_domain {
  	unsigned type;
author maintainer
#10 lore raw

RFC 6/6 iommu/s390: flush queued IOVAs on RPCIT out of resource indication

In reply to Niklas Schnelle

When RPCIT indicates that the underlying hypervisor has run out of resources it often means that its IOVA space is exhausted and IOVAs need to be freed before new ones can be created. By triggering a flush of the IOVA queue we can get the queued IOVAs freed and also get the new mapping established during the global flush.

---
 drivers/iommu/dma-iommu.c  |  2 +-
 drivers/iommu/dma-iommu.h  |  1 +
 drivers/iommu/s390-iommu.c | 12 ++++++++++++
 3 files changed, 14 insertions(+), 1 deletion(-)
drivers/iommu/dma-iommu.c +1 -1
index 427fb84f50c3..4853f98f3305 100644
@@ -287,7 +287,7 @@ static void flush_percpu(struct iommu_dma_cookie *cookie)
 	}
 }
 
-static void iommu_dma_flush_fq(struct iommu_dma_cookie *cookie)
+void iommu_dma_flush_fq(struct iommu_dma_cookie *cookie)
 {
 	if (!cookie->fq_domain)
 		return;
drivers/iommu/dma-iommu.h +1 -0
index 942790009292..cac06030aa26 100644
@@ -13,6 +13,7 @@ int iommu_get_dma_cookie(struct iommu_domain *domain);
 void iommu_put_dma_cookie(struct iommu_domain *domain);
 
 int iommu_dma_init_fq(struct iommu_domain *domain);
+void iommu_dma_flush_fq(struct iommu_dma_cookie *cookie);
 
 void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list);
 
drivers/iommu/s390-iommu.c +12 -0
index 506f8b92931f..270662584f96 100644
@@ -502,6 +502,10 @@ static void s390_iommu_flush_iotlb_all(struct iommu_domain *domain)
 		atomic64_inc(&s390_domain->ctrs.global_rpcits);
 		rc = zpci_refresh_trans((u64)zdev->fh << 32, zdev->start_dma,
 					zdev->end_dma - zdev->start_dma + 1);
+		if (rc == -ENOMEM) {
+			iommu_dma_flush_fq(domain->iova_cookie);
+			rc = 0;
+		}
 		if (rc)
 			break;
 	}
@@ -525,6 +529,10 @@ static void s390_iommu_iotlb_sync(struct iommu_domain *domain,
 		atomic64_inc(&s390_domain->ctrs.sync_rpcits);
 		rc = zpci_refresh_trans((u64)zdev->fh << 32, gather->start,
 					size);
+		if (rc == -ENOMEM) {
+			iommu_dma_flush_fq(domain->iova_cookie);
+			rc = 0;
+		}
 		if (rc)
 			break;
 	}
@@ -545,6 +553,10 @@ static void s390_iommu_iotlb_sync_map(struct iommu_domain *domain,
 		atomic64_inc(&s390_domain->ctrs.sync_map_rpcits);
 		rc = zpci_refresh_trans((u64)zdev->fh << 32,
 					iova, size);
+		if (rc == -ENOMEM) {
+			iommu_dma_flush_fq(domain->iova_cookie);
+			rc = 0;
+		}
 		if (rc)
 			break;
 	}
Signature · 1 line
2.34.1