Skip to contentSkip to conversation

Add dynamic iommu backed bounce buffers

PATCH In review David Stevens 4 participants 19 messages Aug 6, 2021 – Jul 1, 2022

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

v2 cover Add dynamic iommu backed bounce buffers

From: David Stevens <[email protected]>

This patch series adds support for per-domain dynamic pools of iommu bounce buffers to the dma-iommu API. This allows iommu mappings to be reused while still maintaining strict iommu protection.

This bounce buffer support is used to add a new config option that, when enabled, causes all non-direct streaming mappings below a configurable size to go through the bounce buffers. This serves as an optimization on systems where manipulating iommu mappings is very expensive. For example, virtio-iommu operations in a guest on a linux host require a vmexit, involvement the VMM, and a VFIO syscall. For relatively small DMA operations, memcpy can be significantly faster.

As a performance comparison, on a device with an i5-10210U, I ran fio with a VFIO passthrough NVMe drive and virtio-iommu with '--direct=1 --rw=read --ioengine=libaio --iodepth=64' and block sizes 4k, 16k, 64k, and 128k. Test throughput increased by 2.8x, 4.7x, 3.6x, and 3.6x. Time spent in iommu_dma_unmap_(page|sg) per GB processed decreased by 97%, 94%, 90%, and 87%. Time spent in iommu_dma_map_(page|sg) decreased by >99%, as bounce buffers don't require syncing here in the read case. Running with multiple jobs doesn't serve as a useful performance comparison because virtio-iommu and vfio_iommu_type1 both have big locks that significantly limit mulithreaded DMA performance.

These pooled bounce buffers are also used for subgranule mappings with untrusted devices, replacing the single use bounce buffers used currently. The biggest difference here is that the new implementation maps a whole sglist using a single bounce buffer. The new implementation does not support using bounce buffers for only some segments of the sglist, so it may require more copying. However, the current implementation requires per-segment iommu map/unmap operations for all untrusted sglist mappings (fully aligned sglists included). On a i5-10210U laptop with the internal NVMe drive made to appear untrusted, fio --direct=1 --rw=read --ioengine=libaio --iodepth=64 --bs=64k showed a statistically significant decrease in CPU load from 2.28% -> 2.17% with the new iommu bounce buffer optimization enabled.

Each domain's buffer pool is split into multiple power-of-2 size classes. Each class allocates a fixed number of buffer slot metadata. A large iova range is allocated, and each slot is assigned an iova from the range. This allows the iova to be easily mapped back to the slot, and allows the critical section of most pool operations to be constant time. The one exception is finding a cached buffer to reuse. These are only separated according to R/W permissions - the use of other permissions such as IOMMU_PRIV may require a linear search through the cache. However, these other permissions are rare and likely exhibit high locality, so the should not be a bottleneck in practice.

Since untrusted devices may require bounce buffers, each domain has a fallback rbtree to manage single use buffers. This may be necessary if a very large number of DMA operations are simultaneously in-flight, or for very large individual DMA operations.

This patch set does not use swiotlb. There are two primary ways in which swiotlb isn't compatible with per-domain buffer pools. First, swiotlb allocates buffers to be compatible with a single device, whereas per-domain buffer pools don't handle that during buffer allocation as a single buffer may end up being used by multiple devices. Second, swiotlb allocation establishes the original to bounce buffer mapping, which again doesn't work if buffers can be reused. Effectively the only code that can be shared between the two use cases is allocating slots from the swiotlb's memory. However, given that we're going to be allocating memory for use with an iommu, allocating memory from a block of memory explicitly set aside to deal with a lack of iommu seems kind of contradictory. At best there might be a small performance improvement if wiotlb allocation is faster than regular page allocation, but buffer allocation isn't on the hot path anyway.

Not using the swiotlb has the benefit that memory doesn't have to be preallocated. Instead, bounce buffers consume memory only for in-flight dma transactions (ignoring temporarily cached buffers), which is the smallest amount possible. This makes it easier to use bounce buffers as an optimization on systems with large numbers of devices or in situations where devices are unknown, since it is not necessary to try to tune how much memory needs to be set aside to achieve good performance without costing too much memory.

Finally, this series adds a new DMA_ATTR_PERSISTENT_STREAMING flag. This is meant to address devices which create long lived streaming mappings but manage CPU cache coherency without using the dma_sync_* APIs. Currently, these devices don't function properly with swiotlb=force. The new flag is used to bypass bounce buffers so such devices will function when the new bounce buffer optimization is enabled. The flag is added to the i915 driver, which creates such mappings. It can also be added to various dma-buf implementations as an optimization, although that is not done here.

v1 -> v2:
- Replace existing untrusted bounce buffers with new bounce buffer pools. This includes significant rework to account for untrusted bounce buffers being required instead of an optimization.
- Add flag for persistent streaming mappings.

David Stevens (9):
Revert "iommu: Allow the dma-iommu api to use bounce buffers"
dma-iommu: expose a few helper functions to module
dma-iommu: bounce buffers for untrusted devices
dma-iommu: remove extra buffer search on unmap
dma-iommu: clear only necessary bytes
dma-iommu: add bounce buffer pools
dma-iommu: support iommu bounce buffer optimization
dma-mapping: add persistent streaming mapping flag drm/i915: use DMA_ATTR_PERSISTENT_STREAMING flag

 drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c |   4 +-
 drivers/gpu/drm/i915/i915_gem_gtt.c        |   3 +-
 drivers/iommu/Kconfig                      |  11 +
 drivers/iommu/Makefile                     |   2 +-
 drivers/iommu/dma-iommu.c                  | 268 ++++-----
 drivers/iommu/io-bounce-buffers.c          | 533 +++++++++++++++++
 drivers/iommu/io-bounce-buffers.h          |  49 ++
 drivers/iommu/io-buffer-manager.c          | 633 +++++++++++++++++++++
 drivers/iommu/io-buffer-manager.h          |  94 +++
 include/linux/dma-iommu.h                  |  12 +
 include/linux/dma-mapping.h                |  11 +
 11 files changed, 1460 insertions(+), 160 deletions(-)
 create mode 100644 drivers/iommu/io-bounce-buffers.c
 create mode 100644 drivers/iommu/io-bounce-buffers.h
 create mode 100644 drivers/iommu/io-buffer-manager.c
 create mode 100644 drivers/iommu/io-buffer-manager.h
Signature · 6 lines
2.32.0.605.g8dce9f2422-goog

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#1 lore raw

PATCH v2 1/9 Revert "iommu: Allow the dma-iommu api to use bounce buffers"

From: David Stevens <[email protected]>

A new pooled bounce buffer implementation will be added to reduce IOMMU interactions on platforms with slow IOMMUs. The new implementation can also support using bounce buffers with untrusted devices, so the current basic bounce buffer support can be reverted.

This reverts commit 82612d66d51d3bacdd789e31d2e875d2494b7514.

---
 drivers/iommu/dma-iommu.c | 152 ++++----------------------------------
 1 file changed, 13 insertions(+), 139 deletions(-)
drivers/iommu/dma-iommu.c +13 -139
index 98ba927aee1a..1491b5450246 100644
@@ -20,11 +20,9 @@
 #include <linux/mm.h>
 #include <linux/mutex.h>
 #include <linux/pci.h>
-#include <linux/swiotlb.h>
 #include <linux/scatterlist.h>
 #include <linux/vmalloc.h>
 #include <linux/crash_dump.h>
-#include <linux/dma-direct.h>
 
 struct iommu_dma_msi_page {
 	struct list_head	list;
@@ -493,23 +491,6 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr,
 	iommu_dma_free_iova(cookie, dma_addr, size, iotlb_gather.freelist);
 }
 
-static void __iommu_dma_unmap_swiotlb(struct device *dev, dma_addr_t dma_addr,
-		size_t size, enum dma_data_direction dir,
-		unsigned long attrs)
-{
-	struct iommu_domain *domain = iommu_get_dma_domain(dev);
-	phys_addr_t phys;
-
-	phys = iommu_iova_to_phys(domain, dma_addr);
-	if (WARN_ON(!phys))
-		return;
-
-	__iommu_dma_unmap(dev, dma_addr, size);
-
-	if (unlikely(is_swiotlb_buffer(phys)))
-		swiotlb_tbl_unmap_single(dev, phys, size, dir, attrs);
-}
-
 static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys,
 		size_t size, int prot, u64 dma_mask)
 {
@@ -536,52 +517,6 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys,
 	return iova + iova_off;
 }
 
-static dma_addr_t __iommu_dma_map_swiotlb(struct device *dev, phys_addr_t phys,
-		size_t org_size, dma_addr_t dma_mask, bool coherent,
-		enum dma_data_direction dir, unsigned long attrs)
-{
-	int prot = dma_info_to_prot(dir, coherent, attrs);
-	struct iommu_domain *domain = iommu_get_dma_domain(dev);
-	struct iommu_dma_cookie *cookie = domain->iova_cookie;
-	struct iova_domain *iovad = &cookie->iovad;
-	size_t aligned_size = org_size;
-	void *padding_start;
-	size_t padding_size;
-	dma_addr_t iova;
-
-	/*
-	 * If both the physical buffer start address and size are
-	 * page aligned, we don't need to use a bounce page.
-	 */
-	if (IS_ENABLED(CONFIG_SWIOTLB) && dev_is_untrusted(dev) &&
-	    iova_offset(iovad, phys | org_size)) {
-		aligned_size = iova_align(iovad, org_size);
-		phys = swiotlb_tbl_map_single(dev, phys, org_size,
-					      aligned_size, dir, attrs);
-
-		if (phys == DMA_MAPPING_ERROR)
-			return DMA_MAPPING_ERROR;
-
-		/* Cleanup the padding area. */
-		padding_start = phys_to_virt(phys);
-		padding_size = aligned_size;
-
-		if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC) &&
-		    (dir == DMA_TO_DEVICE ||
-		     dir == DMA_BIDIRECTIONAL)) {
-			padding_start += org_size;
-			padding_size -= org_size;
-		}
-
-		memset(padding_start, 0, padding_size);
-	}
-
-	iova = __iommu_dma_map(dev, phys, aligned_size, prot, dma_mask);
-	if (iova == DMA_MAPPING_ERROR && is_swiotlb_buffer(phys))
-		swiotlb_tbl_unmap_single(dev, phys, org_size, dir, attrs);
-	return iova;
-}
-
 static void __iommu_dma_free_pages(struct page **pages, int count)
 {
 	while (count--)
@@ -776,15 +711,11 @@ static void iommu_dma_sync_single_for_cpu(struct device *dev,
 {
 	phys_addr_t phys;
 
-	if (dev_is_dma_coherent(dev) && !dev_is_untrusted(dev))
+	if (dev_is_dma_coherent(dev))
 		return;
 
 	phys = iommu_iova_to_phys(iommu_get_dma_domain(dev), dma_handle);
-	if (!dev_is_dma_coherent(dev))
-		arch_sync_dma_for_cpu(phys, size, dir);
-
-	if (is_swiotlb_buffer(phys))
-		swiotlb_sync_single_for_cpu(dev, phys, size, dir);
+	arch_sync_dma_for_cpu(phys, size, dir);
 }
 
 static void iommu_dma_sync_single_for_device(struct device *dev,
@@ -792,15 +723,11 @@ static void iommu_dma_sync_single_for_device(struct device *dev,
 {
 	phys_addr_t phys;
 
-	if (dev_is_dma_coherent(dev) && !dev_is_untrusted(dev))
+	if (dev_is_dma_coherent(dev))
 		return;
 
 	phys = iommu_iova_to_phys(iommu_get_dma_domain(dev), dma_handle);
-	if (is_swiotlb_buffer(phys))
-		swiotlb_sync_single_for_device(dev, phys, size, dir);
-
-	if (!dev_is_dma_coherent(dev))
-		arch_sync_dma_for_device(phys, size, dir);
+	arch_sync_dma_for_device(phys, size, dir);
 }
 
 static void iommu_dma_sync_sg_for_cpu(struct device *dev,
@@ -810,17 +737,11 @@ static void iommu_dma_sync_sg_for_cpu(struct device *dev,
 	struct scatterlist *sg;
 	int i;
 
-	if (dev_is_dma_coherent(dev) && !dev_is_untrusted(dev))
+	if (dev_is_dma_coherent(dev))
 		return;
 
-	for_each_sg(sgl, sg, nelems, i) {
-		if (!dev_is_dma_coherent(dev))
-			arch_sync_dma_for_cpu(sg_phys(sg), sg->length, dir);
-
-		if (is_swiotlb_buffer(sg_phys(sg)))
-			swiotlb_sync_single_for_cpu(dev, sg_phys(sg),
-						    sg->length, dir);
-	}
+	for_each_sg(sgl, sg, nelems, i)
+		arch_sync_dma_for_cpu(sg_phys(sg), sg->length, dir);
 }
 
 static void iommu_dma_sync_sg_for_device(struct device *dev,
@@ -830,17 +751,11 @@ static void iommu_dma_sync_sg_for_device(struct device *dev,
 	struct scatterlist *sg;
 	int i;
 
-	if (dev_is_dma_coherent(dev) && !dev_is_untrusted(dev))
+	if (dev_is_dma_coherent(dev))
 		return;
 
-	for_each_sg(sgl, sg, nelems, i) {
-		if (is_swiotlb_buffer(sg_phys(sg)))
-			swiotlb_sync_single_for_device(dev, sg_phys(sg),
-						       sg->length, dir);
-
-		if (!dev_is_dma_coherent(dev))
-			arch_sync_dma_for_device(sg_phys(sg), sg->length, dir);
-	}
+	for_each_sg(sgl, sg, nelems, i)
+		arch_sync_dma_for_device(sg_phys(sg), sg->length, dir);
 }
 
 static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page,
@@ -849,10 +764,10 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page,
 {
 	phys_addr_t phys = page_to_phys(page) + offset;
 	bool coherent = dev_is_dma_coherent(dev);
+	int prot = dma_info_to_prot(dir, coherent, attrs);
 	dma_addr_t dma_handle;
 
-	dma_handle = __iommu_dma_map_swiotlb(dev, phys, size, dma_get_mask(dev),
-			coherent, dir, attrs);
+	dma_handle = __iommu_dma_map(dev, phys, size, prot, dma_get_mask(dev));
 	if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) &&
 	    dma_handle != DMA_MAPPING_ERROR)
 		arch_sync_dma_for_device(phys, size, dir);
@@ -864,7 +779,7 @@ static void iommu_dma_unmap_page(struct device *dev, dma_addr_t dma_handle,
 {
 	if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC))
 		iommu_dma_sync_single_for_cpu(dev, dma_handle, size, dir);
-	__iommu_dma_unmap_swiotlb(dev, dma_handle, size, dir, attrs);
+	__iommu_dma_unmap(dev, dma_handle, size);
 }
 
 /*
@@ -942,39 +857,6 @@ static void __invalidate_sg(struct scatterlist *sg, int nents)
 	}
 }
 
-static void iommu_dma_unmap_sg_swiotlb(struct device *dev, struct scatterlist *sg,
-		int nents, enum dma_data_direction dir, unsigned long attrs)
-{
-	struct scatterlist *s;
-	int i;
-
-	for_each_sg(sg, s, nents, i)
-		__iommu_dma_unmap_swiotlb(dev, sg_dma_address(s),
-				sg_dma_len(s), dir, attrs);
-}
-
-static int iommu_dma_map_sg_swiotlb(struct device *dev, struct scatterlist *sg,
-		int nents, enum dma_data_direction dir, unsigned long attrs)
-{
-	struct scatterlist *s;
-	int i;
-
-	for_each_sg(sg, s, nents, i) {
-		sg_dma_address(s) = __iommu_dma_map_swiotlb(dev, sg_phys(s),
-				s->length, dma_get_mask(dev),
-				dev_is_dma_coherent(dev), dir, attrs);
-		if (sg_dma_address(s) == DMA_MAPPING_ERROR)
-			goto out_unmap;
-		sg_dma_len(s) = s->length;
-	}
-
-	return nents;
-
-out_unmap:
-	iommu_dma_unmap_sg_swiotlb(dev, sg, i, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC);
-	return 0;
-}
-
 /*
  * The DMA API client is passing in a scatterlist which could describe
  * any old buffer layout, but the IOMMU API requires everything to be
@@ -1002,9 +884,6 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg,
 	if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC))
 		iommu_dma_sync_sg_for_device(dev, sg, nents, dir);
 
-	if (dev_is_untrusted(dev))
-		return iommu_dma_map_sg_swiotlb(dev, sg, nents, dir, attrs);
-
 	/*
 	 * Work out how much IOVA space we need, and align the segments to
 	 * IOVA granules for the IOMMU driver to handle. With some clever
@@ -1074,11 +953,6 @@ static void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg,
 	if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC))
 		iommu_dma_sync_sg_for_cpu(dev, sg, nents, dir);
 
-	if (dev_is_untrusted(dev)) {
-		iommu_dma_unmap_sg_swiotlb(dev, sg, nents, dir, attrs);
-		return;
-	}
-
 	/*
 	 * The scatterlist segments are mapped into a single
 	 * contiguous IOVA allocation, so this is incredibly easy.
Signature · 6 lines
2.32.0.605.g8dce9f2422-goog

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#2 lore raw

PATCH v2 2/9 dma-iommu: expose a few helper functions to module

In reply to David Stevens

From: David Stevens <[email protected]>

Expose a few helper functions from dma-iommu to the rest of the module.

---
 drivers/iommu/dma-iommu.c | 23 ++++++++++++-----------
 include/linux/dma-iommu.h |  8 ++++++++
 2 files changed, 20 insertions(+), 11 deletions(-)
drivers/iommu/dma-iommu.c +12 -11
index 1491b5450246..055ccda5eba1 100644
@@ -412,7 +412,7 @@ static int dma_info_to_prot(enum dma_data_direction dir, bool coherent,
 	}
 }
 
-static dma_addr_t iommu_dma_alloc_iova(struct iommu_domain *domain,
+dma_addr_t __iommu_dma_alloc_iova(struct iommu_domain *domain,
 		size_t size, u64 dma_limit, struct device *dev)
 {
 	struct iommu_dma_cookie *cookie = domain->iova_cookie;
@@ -452,7 +452,7 @@ static dma_addr_t iommu_dma_alloc_iova(struct iommu_domain *domain,
 	return (dma_addr_t)iova << shift;
 }
 
-static void iommu_dma_free_iova(struct iommu_dma_cookie *cookie,
+void __iommu_dma_free_iova(struct iommu_dma_cookie *cookie,
 		dma_addr_t iova, size_t size, struct page *freelist)
 {
 	struct iova_domain *iovad = &cookie->iovad;
@@ -488,7 +488,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr,
 
 	if (!cookie->fq_domain)
 		iommu_iotlb_sync(domain, &iotlb_gather);
-	iommu_dma_free_iova(cookie, dma_addr, size, iotlb_gather.freelist);
+	__iommu_dma_free_iova(cookie, dma_addr, size, iotlb_gather.freelist);
 }
 
 static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys,
@@ -506,12 +506,12 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys,
 
 	size = iova_align(iovad, size + iova_off);
 
-	iova = iommu_dma_alloc_iova(domain, size, dma_mask, dev);
+	iova = __iommu_dma_alloc_iova(domain, size, dma_mask, dev);
 	if (!iova)
 		return DMA_MAPPING_ERROR;
 
 	if (iommu_map_atomic(domain, iova, phys - iova_off, size, prot)) {
-		iommu_dma_free_iova(cookie, iova, size, NULL);
+		__iommu_dma_free_iova(cookie, iova, size, NULL);
 		return DMA_MAPPING_ERROR;
 	}
 	return iova + iova_off;
@@ -617,7 +617,8 @@ static struct page **__iommu_dma_alloc_noncontiguous(struct device *dev,
 		return NULL;
 
 	size = iova_align(iovad, size);
-	iova = iommu_dma_alloc_iova(domain, size, dev->coherent_dma_mask, dev);
+	iova = __iommu_dma_alloc_iova(domain, size,
+				      dev->coherent_dma_mask, dev);
 	if (!iova)
 		goto out_free_pages;
 
@@ -643,7 +644,7 @@ static struct page **__iommu_dma_alloc_noncontiguous(struct device *dev,
 out_free_sg:
 	sg_free_table(sgt);
 out_free_iova:
-	iommu_dma_free_iova(cookie, iova, size, NULL);
+	__iommu_dma_free_iova(cookie, iova, size, NULL);
 out_free_pages:
 	__iommu_dma_free_pages(pages, count);
 	return NULL;
@@ -923,7 +924,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg,
 		prev = s;
 	}
 
-	iova = iommu_dma_alloc_iova(domain, iova_len, dma_get_mask(dev), dev);
+	iova = __iommu_dma_alloc_iova(domain, iova_len, dma_get_mask(dev), dev);
 	if (!iova)
 		goto out_restore_sg;
 
@@ -937,7 +938,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg,
 	return __finalise_sg(dev, sg, nents, iova);
 
 out_free_iova:
-	iommu_dma_free_iova(cookie, iova, iova_len, NULL);
+	__iommu_dma_free_iova(cookie, iova, iova_len, NULL);
 out_restore_sg:
 	__invalidate_sg(sg, nents);
 	return 0;
@@ -1226,7 +1227,7 @@ static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev,
 	if (!msi_page)
 		return NULL;
 
-	iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev);
+	iova = __iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev);
 	if (!iova)
 		goto out_free_page;
 
@@ -1240,7 +1241,7 @@ static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev,
 	return msi_page;
 
 out_free_iova:
-	iommu_dma_free_iova(cookie, iova, size, NULL);
+	__iommu_dma_free_iova(cookie, iova, size, NULL);
 out_free_page:
 	kfree(msi_page);
 	return NULL;
include/linux/dma-iommu.h +8 -0
index 758ca4694257..50f676678318 100644
@@ -42,6 +42,14 @@ void iommu_dma_free_cpu_cached_iovas(unsigned int cpu,
 
 extern bool iommu_dma_forcedac;
 
+struct iommu_dma_cookie;
+
+dma_addr_t __iommu_dma_alloc_iova(struct iommu_domain *domain,
+				  size_t size, dma_addr_t dma_limit,
+				  struct device *dev);
+void __iommu_dma_free_iova(struct iommu_dma_cookie *cookie,
+		dma_addr_t iova, size_t size, struct page *freelist);
+
 #else /* CONFIG_IOMMU_DMA */
 
 struct iommu_domain;
Signature · 6 lines
2.32.0.605.g8dce9f2422-goog

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#3 lore raw

Re: [PATCH v2 2/9] dma-iommu: expose a few helper functions to module

Thank you for the patch! Yet something to improve:

Hi David,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on iommu/next]
[also build test ERROR on drm-intel/for-linux-next hch-configfs/for-next linus/master v5.14-rc4] [cannot apply to next-20210805]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/David-Stevens/Add-dynamic-iommu-backed-bounce-buffers/20210806-183631
base:   https://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git next
config: i386-randconfig-a011-20210804 (attached as .config)
compiler: gcc-9 (Debian 9.3.0-22) 9.3.0

reproduce (this is a W=1 build):
# https://github.com/0day-ci/linux/commit/385dff3d789a97ef650912616e9d696fba96cb20
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review David-Stevens/Add-dynamic-iommu-backed-bounce-buffers/20210806-183631 git checkout 385dff3d789a97ef650912616e9d696fba96cb20
# save the attached .config to linux build tree
mkdir build_dir
make W=1 O=build_dir ARCH=i386 SHELL=/bin/bash

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>):

>> drivers/iommu/dma-iommu.c:415:12: error: conflicting types for '__iommu_dma_alloc_iova'

415 | dma_addr_t __iommu_dma_alloc_iova(struct iommu_domain *domain,
| ^~~~~~~~~~~~~~~~~~~~~~
In file included from drivers/iommu/dma-iommu.c:14:
include/linux/dma-iommu.h:47:12: note: previous declaration of '__iommu_dma_alloc_iova' was here
47 | dma_addr_t __iommu_dma_alloc_iova(struct iommu_domain *domain,
| ^~~~~~~~~~~~~~~~~~~~~~

vim +/__iommu_dma_alloc_iova +415 drivers/iommu/dma-iommu.c

414

 > 415	dma_addr_t __iommu_dma_alloc_iova(struct iommu_domain *domain,

416 size_t size, u64 dma_limit, struct device *dev)
417 {
418 struct iommu_dma_cookie *cookie = domain->iova_cookie;
419 struct iova_domain *iovad = &cookie->iovad;
420 unsigned long shift, iova_len, iova = 0;
421
422 if (cookie->type == IOMMU_DMA_MSI_COOKIE) {
423 cookie->msi_iova += size;
424 return cookie->msi_iova - size;
425 }
426
427 shift = iova_shift(iovad);
428 iova_len = size >> shift;
429 /*
430 * Freeing non-power-of-two-sized allocations back into the IOVA caches
431 * will come back to bite us badly, so we have to waste a bit of space
432 * rounding up anything cacheable to make sure that can't happen. The
433 * order of the unadjusted size will still match upon freeing.
434 */
435 if (iova_len < (1 << (IOVA_RANGE_CACHE_MAX_SIZE - 1)))
436 iova_len = roundup_pow_of_two(iova_len);
437
438 dma_limit = min_not_zero(dma_limit, dev->bus_dma_limit);
439
440 if (domain->geometry.force_aperture)
441 dma_limit = min(dma_limit, (u64)domain->geometry.aperture_end);
442
443 /* Try to get PCI devices a SAC address */
444 if (dma_limit > DMA_BIT_MASK(32) && !iommu_dma_forcedac && dev_is_pci(dev))
445 iova = alloc_iova_fast(iovad, iova_len,
446 DMA_BIT_MASK(32) >> shift, false);
447
448 if (!iova)
449 iova = alloc_iova_fast(iovad, iova_len, dma_limit >> shift,
450 true);
451
452 return (dma_addr_t)iova << shift;
453 }
454

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/[email protected]

#4 lore raw

PATCH v2 3/9 dma-iommu: bounce buffers for untrusted devices

In reply to David Stevens

From: David Stevens <[email protected]>

Add support for dynamic bounce buffers to the dma-api for use with subgranule IOMMU mappings with untrusted devices. Bounce buffer management is split into two parts. First, there is a buffer manager that is responsible for allocating and tracking buffers. Second, there is a layer that uses the managed buffers as bounce buffers. It is responsible for managing the IOMMU mapping and for syncing between the original and bounce buffers.

For now, buffer management is very simple - every mapping allocates a new bounce buffer.

---
 drivers/iommu/Makefile            |   2 +-
 drivers/iommu/dma-iommu.c         |  70 +++++-
 drivers/iommu/io-bounce-buffers.c | 358 ++++++++++++++++++++++++++++++
 drivers/iommu/io-bounce-buffers.h |  46 ++++
 drivers/iommu/io-buffer-manager.c | 212 ++++++++++++++++++
 drivers/iommu/io-buffer-manager.h |  43 ++++
 6 files changed, 728 insertions(+), 3 deletions(-)
 create mode 100644 drivers/iommu/io-bounce-buffers.c
 create mode 100644 drivers/iommu/io-bounce-buffers.h
 create mode 100644 drivers/iommu/io-buffer-manager.c
 create mode 100644 drivers/iommu/io-buffer-manager.h
drivers/iommu/Makefile +1 -1
index c0fb0ba88143..4edaf7adc082 100644
@@ -4,7 +4,7 @@ obj-$(CONFIG_IOMMU_API) += iommu.o
 obj-$(CONFIG_IOMMU_API) += iommu-traces.o
 obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
 obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
-obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
+obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o io-bounce-buffers.o io-buffer-manager.o
 obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
 obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o
 obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) += io-pgtable-arm.o
drivers/iommu/dma-iommu.c +68 -2

178 lines not on this page · raw message

drivers/iommu/io-bounce-buffers.c new file +358 -0

358 lines not on this page · raw message

drivers/iommu/io-bounce-buffers.h new file +46 -0

46 lines not on this page · raw message

drivers/iommu/io-buffer-manager.c new file +212 -0

212 lines not on this page · raw message

drivers/iommu/io-buffer-manager.h new file +43 -0

43 lines not on this page · raw message

Signature · 6 lines
2.32.0.605.g8dce9f2422-goog

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#5 lore raw

Re: [PATCH v2 3/9] dma-iommu: bounce buffers for untrusted devices

Thank you for the patch! Perhaps something to improve:

Hi David,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on iommu/next]
[also build test WARNING on drm-intel/for-linux-next hch-configfs/for-next linus/master v5.14-rc4] [cannot apply to next-20210805]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/David-Stevens/Add-dynamic-iommu-backed-bounce-buffers/20210806-183631
base:   https://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git next
config: ia64-randconfig-r025-20210804 (attached as .config)
compiler: ia64-linux-gcc (GCC) 10.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/c5f1f9fa88a7062c1ded50fa165f6b01ed73f161
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review David-Stevens/Add-dynamic-iommu-backed-bounce-buffers/20210806-183631 git checkout c5f1f9fa88a7062c1ded50fa165f6b01ed73f161
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-10.3.0 make.cross ARCH=ia64

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>):

>> drivers/iommu/io-buffer-manager.c:57:24: warning: no previous prototype for 'find_fallback_node' [-Wmissing-prototypes]
      57 | struct io_buffer_node *find_fallback_node(struct rb_root *root, dma_addr_t iova)
         |                        ^~~~~~~~~~~~~~~~~~
>> drivers/iommu/io-buffer-manager.c:75:6: warning: no previous prototype for 'insert_fallback_node' [-Wmissing-prototypes]

75 | bool insert_fallback_node(struct rb_root *root, struct io_buffer_node *node)
| ^~~~~~~~~~~~~~~~~~~~

vim +/find_fallback_node +57 drivers/iommu/io-buffer-manager.c

    56	
  > 57	struct io_buffer_node *find_fallback_node(struct rb_root *root, dma_addr_t iova)
    58	{
    59		struct rb_node *node = root->rb_node;
    60	
    61		while (node) {
    62			struct io_buffer_node *cur =
    63				container_of(node, struct io_buffer_node, node);
    64	
    65			if (iova < cur->info.iova)
    66				node = node->rb_left;
    67			else if (iova >= cur->info.iova + cur->info.size)
    68				node = node->rb_right;
    69			else
    70				return cur;
    71		}
    72		return NULL;
    73	}
    74	
  > 75	bool insert_fallback_node(struct rb_root *root, struct io_buffer_node *node)
    76	{
    77		struct rb_node **new = &(root->rb_node), *parent = NULL;
    78		dma_addr_t node_end = node->info.iova + node->info.size;
    79	
    80		while (*new) {
    81			struct io_buffer_node *cur =
    82				container_of(*new, struct io_buffer_node, node);
    83			dma_addr_t cur_end = cur->info.iova + cur->info.size;
    84	
    85			parent = *new;
    86			if (node_end <= cur->info.iova)
    87				new = &((*new)->rb_left);
    88			else if (node->info.iova >= cur_end)
    89				new = &((*new)->rb_right);
    90			else {
    91				pr_crit("IOVA collision new=[%llx,%llx) old=[%llx,%llx)\n",
    92					node->info.iova, node_end, cur->info.iova,
    93					cur_end);
    94				return false;
    95			}
    96		}
    97	
    98		rb_link_node(&node->node, parent, new);
    99		rb_insert_color(&node->node, root);

100 return true;
101 }
102

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#6 lore raw

RE: [PATCH v2 3/9] dma-iommu: bounce buffers for untrusted devices

In reply to David Stevens

Hi David,

I like this patch set and this is crucial for reducing the significant vIOMMU performance. It looks you totally rewrite the IOMMU mapping/unmapping part and use the dynamically allocated memory from buddy system as bounce buffer instead of using the legacy SWIOTLB bounce buffer. As I know, some legacy devices' DMA could not access the memory larger than 32-bit memory space and the dynamically allocated memory address could exceed the 32-bit memory space. Is it a problem?

Thx,
Dapeng Mi

-----Original Message-----
From: iommu <[email protected]> On Behalf Of David Stevens
Sent: Friday, August 6, 2021 6:34 PM
To: Robin Murphy <[email protected]>
Cc: [email protected]; Sergey Senozhatsky <[email protected]>; [email protected]; David Stevens <[email protected]>; Will Deacon <[email protected]>; Christoph Hellwig <[email protected]>
Subject: [PATCH v2 3/9] dma-iommu: bounce buffers for untrusted devices

From: David Stevens <[email protected]>

Add support for dynamic bounce buffers to the dma-api for use with subgranule IOMMU mappings with untrusted devices. Bounce buffer management is split into two parts. First, there is a buffer manager that is responsible for allocating and tracking buffers. Second, there is a layer that uses the managed buffers as bounce buffers. It is responsible for managing the IOMMU mapping and for syncing between the original and bounce buffers.

For now, buffer management is very simple - every mapping allocates a new bounce buffer.

---
 drivers/iommu/Makefile            |   2 +-
 drivers/iommu/dma-iommu.c         |  70 +++++-
 drivers/iommu/io-bounce-buffers.c | 358 ++++++++++++++++++++++++++++++  drivers/iommu/io-bounce-buffers.h |  46 ++++  drivers/iommu/io-buffer-manager.c | 212 ++++++++++++++++++  drivers/iommu/io-buffer-manager.h |  43 ++++
 6 files changed, 728 insertions(+), 3 deletions(-)  create mode 100644 drivers/iommu/io-bounce-buffers.c  create mode 100644 drivers/iommu/io-bounce-buffers.h  create mode 100644 drivers/iommu/io-buffer-manager.c  create mode 100644 drivers/iommu/io-buffer-manager.h
drivers/iommu/Makefile +2 -1
@@ -4,7 +4,7 @@ obj-$(CONFIG_IOMMU_API) += iommu.o
 obj-$(CONFIG_IOMMU_API) += iommu-traces.o
 obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
 obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
-obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
+obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o io-bounce-buffers.o 
+io-buffer-manager.o
 obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
 obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o

obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) += io-pgtable-arm.o diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 055ccda5eba1..908eb6fb7dc3 100644

drivers/iommu/dma-iommu.c +58 -1

141 lines not on this page · raw message

unsigned long mask = dma_get_seg_boundary(dev);
- int i;
+ int i, ret;

if (static_branch_unlikely(&iommu_deferred_attach_enabled) &&
iommu_deferred_attach(dev, domain))
return 0;

+ if (cookie->bounce_buffers &&
+ io_bounce_buffers_map_sg(cookie->bounce_buffers, dev, sg, nents,
+ prot, dir, attrs, &ret))
+ return ret;
+
if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC))
iommu_dma_sync_sg_for_device(dev, sg, nents, dir);

@@ -947,10 +1009,14 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, static void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg,
int nents, enum dma_data_direction dir, unsigned long attrs) {
+ struct io_bounce_buffers *bounce = dev_to_io_bounce_buffers(dev);
dma_addr_t start, end;
struct scatterlist *tmp;
int i;

+ if (bounce && io_bounce_buffers_unmap_sg(bounce, sg, nents, dir, attrs))
+ return;
+
if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC))
iommu_dma_sync_sg_for_cpu(dev, sg, nents, dir);

drivers/iommu/io-bounce-buffers.c new file +351 -0

351 lines not on this page · raw message

drivers/iommu/io-bounce-buffers.h new file +46 -0

46 lines not on this page · raw message

drivers/iommu/io-buffer-manager.c new file +211 -0

211 lines not on this page · raw message

drivers/iommu/io-buffer-manager.h new file +43 -0

43 lines not on this page · raw message

--
2.32.0.605.g8dce9f2422-goog

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#7 lore raw

Re: [PATCH v2 3/9] dma-iommu: bounce buffers for untrusted devices

Quoted 4 lines · On Tue, Aug 10, 2021 at 10:19 AM Mi, Dapeng1 <[email protected]> wrote:

Hi David,

I like this patch set and this is crucial for reducing the significant vIOMMU performance. It looks you totally rewrite the IOMMU mapping/unmapping part and use the dynamically allocated memory from buddy system as bounce buffer instead of using the legacy SWIOTLB bounce buffer. As I know, some legacy devices' DMA could not access the memory larger than 32-bit memory space and the dynamically allocated memory address could exceed the 32-bit memory space. Is it a problem?

My understanding is that when devices with that sort of limitation sit behind an IOMMU, the IOVA is what matters, not the physical address. The bounce bounce buffers use the same limits for IOVA allocation as the regular dma-iommu path, so compatible IOVAs will be allocated for the bounce buffers.

-David

Quoted 2 lines

Thx,
Dapeng Mi

#8 lore raw

PATCH v2 4/9 dma-iommu: remove extra buffer search on unmap

In reply to David Stevens

From: David Stevens <[email protected]>

Add callback to buffer manager's removal function so that the buffer can be sync'ed during unmap without an extra find operation.

---
 drivers/iommu/io-bounce-buffers.c | 87 +++++++++++++++++++++++++------
 drivers/iommu/io-buffer-manager.c |  6 ++-
 drivers/iommu/io-buffer-manager.h |  6 ++-
 3 files changed, 81 insertions(+), 18 deletions(-)
drivers/iommu/io-bounce-buffers.c +71 -16
index 78b4440b58c8..c7c52a3f8bf7 100644
@@ -153,6 +153,20 @@ static void io_bounce_buffers_do_sync(struct io_bounce_buffers *buffers,
 	}
 }
 
+static void __io_bounce_buffers_sync_single(struct io_bounce_buffers *buffers,
+					    dma_addr_t dma_handle, size_t size,
+					    struct io_bounce_buffer_info *info,
+					    struct page *orig_buffer, int prot,
+					    enum dma_data_direction dir,
+					    bool sync_for_cpu)
+{
+	size_t offset = dma_handle - info->iova;
+
+	io_bounce_buffers_do_sync(buffers, info->bounce_buffer, offset,
+				  orig_buffer, offset, size, dir, prot,
+				  sync_for_cpu);
+}
+
 bool io_bounce_buffers_sync_single(struct io_bounce_buffers *buffers,
 				   dma_addr_t dma_handle, size_t size,
 				   enum dma_data_direction dir,
@@ -160,17 +174,14 @@ bool io_bounce_buffers_sync_single(struct io_bounce_buffers *buffers,
 {
 	struct io_bounce_buffer_info info;
 	void *orig_buffer;
-	size_t offset;
 	int prot;
 
 	if (!io_buffer_manager_find_buffer(&buffers->manager, dma_handle, &info,
 					   &orig_buffer, &prot))
 		return false;
 
-	offset = dma_handle - info.iova;
-	io_bounce_buffers_do_sync(buffers, info.bounce_buffer, offset,
-				  orig_buffer, offset, size, dir, prot,
-				  sync_for_cpu);
+	__io_bounce_buffers_sync_single(buffers, dma_handle, size, &info,
+					orig_buffer, prot, dir, sync_for_cpu);
 	return true;
 }
 
@@ -219,16 +230,56 @@ bool io_bounce_buffers_sync_sg(struct io_bounce_buffers *buffers,
 	return true;
 }
 
+struct unmap_sync_args {
+	struct io_bounce_buffers *buffers;
+	unsigned long attrs;
+	enum dma_data_direction dir;
+	dma_addr_t handle;
+	size_t size;
+	int nents;
+};
+
+static void
+io_bounce_buffers_unmap_page_sync(struct io_bounce_buffer_info *info, int prot,
+				  void *orig_buffer, void *ctx)
+{
+	struct unmap_sync_args *args = ctx;
+
+	if (args->attrs & DMA_ATTR_SKIP_CPU_SYNC)
+		return;
+
+	__io_bounce_buffers_sync_single(args->buffers, args->handle, args->size,
+					info, orig_buffer, prot, args->dir,
+					true);
+}
+
 bool io_bounce_buffers_unmap_page(struct io_bounce_buffers *buffers,
 				  dma_addr_t handle, size_t size,
 				  enum dma_data_direction dir,
 				  unsigned long attrs)
 {
-	if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC))
-		io_bounce_buffers_sync_single(buffers, handle, size, dir, true);
+	struct unmap_sync_args args = { .buffers = buffers,
+					.attrs = attrs,
+					.dir = dir,
+					.handle = handle,
+					.size = size };
+
+	return io_buffer_manager_release_buffer(
+		&buffers->manager, buffers->domain, handle, true,
+		io_bounce_buffers_unmap_page_sync, &args);
+}
+
+static void io_bounce_buffers_unmap_sg_sync(struct io_bounce_buffer_info *info,
+					    int prot, void *orig_buffer,
+					    void *ctx)
+{
+	struct unmap_sync_args *args = ctx;
+
+	if (args->attrs & DMA_ATTR_SKIP_CPU_SYNC)
+		return;
 
-	return io_buffer_manager_release_buffer(&buffers->manager,
-						buffers->domain, handle, true);
+	__io_bounce_buffers_sync_sg(args->buffers, orig_buffer, args->nents,
+				    info->bounce_buffer, args->dir, prot, true);
 }
 
 bool io_bounce_buffers_unmap_sg(struct io_bounce_buffers *buffers,
@@ -236,11 +287,13 @@ bool io_bounce_buffers_unmap_sg(struct io_bounce_buffers *buffers,
 				enum dma_data_direction dir,
 				unsigned long attrs)
 {
-	if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC))
-		io_bounce_buffers_sync_sg(buffers, sgl, nents, dir, true);
+	struct unmap_sync_args args = {
+		.buffers = buffers, .attrs = attrs, .dir = dir, .nents = nents
+	};
 
 	return io_buffer_manager_release_buffer(
-		&buffers->manager, buffers->domain, sg_dma_address(sgl), true);
+		&buffers->manager, buffers->domain, sg_dma_address(sgl), true,
+		io_bounce_buffers_unmap_sg_sync, &args);
 }
 
 static bool io_bounce_buffers_map_buffer(struct io_bounce_buffers *buffers,
@@ -286,8 +339,9 @@ bool io_bounce_buffers_map_page(struct io_bounce_buffers *buffers,
 					  page, offset, size, dir, prot, false);
 
 	if (!io_bounce_buffers_map_buffer(buffers, &info, prot)) {
-		io_buffer_manager_release_buffer(
-			&buffers->manager, buffers->domain, info.iova, false);
+		io_buffer_manager_release_buffer(&buffers->manager,
+						 buffers->domain, info.iova,
+						 false, NULL, NULL);
 		return true;
 	}
 
@@ -328,8 +382,9 @@ bool io_bounce_buffers_map_sg(struct io_bounce_buffers *buffers,
 					    false);
 
 	if (!io_bounce_buffers_map_buffer(buffers, &info, prot)) {
-		io_buffer_manager_release_buffer(
-			&buffers->manager, buffers->domain, info.iova, false);
+		io_buffer_manager_release_buffer(&buffers->manager,
+						 buffers->domain, info.iova,
+						 false, NULL, NULL);
 		return true;
 	}
 
drivers/iommu/io-buffer-manager.c +5 -1
index 24e95a2faa37..79b9759da928 100644
@@ -169,7 +169,8 @@ bool io_buffer_manager_find_buffer(struct io_buffer_manager *manager,
 
 bool io_buffer_manager_release_buffer(struct io_buffer_manager *manager,
 				      struct iommu_domain *domain,
-				      dma_addr_t handle, bool inited)
+				      dma_addr_t handle, bool inited,
+				      prerelease_cb cb, void *ctx)
 {
 	struct io_buffer_node *node;
 	unsigned long flags;
@@ -184,6 +185,9 @@ bool io_buffer_manager_release_buffer(struct io_buffer_manager *manager,
 	if (!node)
 		return false;
 
+	if (cb)
+		cb(&node->info, node->prot, node->orig_buffer, ctx);
+
 	if (inited)
 		free_buffer = io_bounce_buffers_release_buffer_cb(
 			manager, node->info.iova, node->info.size);
drivers/iommu/io-buffer-manager.h +5 -1
index aae560cc8512..0e75d89926ca 100644
@@ -31,9 +31,13 @@ bool io_buffer_manager_find_buffer(struct io_buffer_manager *manager,
 				   struct io_bounce_buffer_info *info,
 				   void **orig_buffer, int *prot);
 
+typedef void (*prerelease_cb)(struct io_bounce_buffer_info *info, int prot,
+			      void *orig_buffer, void *ctx);
+
 bool io_buffer_manager_release_buffer(struct io_buffer_manager *manager,
 				      struct iommu_domain *domain,
-				      dma_addr_t handle, bool inited);
+				      dma_addr_t handle, bool inited,
+				      prerelease_cb cb, void *ctx);
 
 int io_buffer_manager_init(struct io_buffer_manager *manager);
 
Signature · 6 lines
2.32.0.605.g8dce9f2422-goog

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#9 lore raw

PATCH v2 5/9 dma-iommu: clear only necessary bytes

In reply to David Stevens

From: David Stevens <[email protected]>

Only clear the padding bytes in bounce buffers, since syncing from the original buffer already overwrites the non-padding bytes.

---
 drivers/iommu/io-bounce-buffers.c | 64 +++++++++++++++++++++++++++++--
 drivers/iommu/io-buffer-manager.c |  7 +---
 2 files changed, 63 insertions(+), 8 deletions(-)
drivers/iommu/io-bounce-buffers.c +61 -3
index c7c52a3f8bf7..ed05f593a195 100644
@@ -296,14 +296,70 @@ bool io_bounce_buffers_unmap_sg(struct io_bounce_buffers *buffers,
 		io_bounce_buffers_unmap_sg_sync, &args);
 }
 
+static void io_bounce_buffers_clear_padding(struct io_bounce_buffer_info *info,
+					    size_t pad_hd_end,
+					    size_t pad_tl_start)
+{
+	size_t idx, pad_hd_idx, pad_tl_idx, count;
+
+	count = info->size / PAGE_SIZE;
+	pad_hd_idx = pad_hd_end / PAGE_SIZE;
+	pad_tl_idx = pad_tl_start / PAGE_SIZE;
+
+	if (!IS_ALIGNED(pad_hd_end, PAGE_SIZE)) {
+		struct page *page = info->bounce_buffer[pad_hd_idx];
+		size_t len = offset_in_page(pad_hd_end);
+
+		memset_page(page, 0, 0, len);
+		arch_sync_dma_for_device(page_to_phys(page), 0, len);
+	}
+
+	if (!IS_ALIGNED(pad_tl_start, PAGE_SIZE)) {
+		size_t off = offset_in_page(pad_tl_start);
+		size_t len = PAGE_SIZE - off;
+		struct page *page = info->bounce_buffer[pad_tl_idx];
+
+		memset_page(page, off, 0, len);
+		arch_sync_dma_for_device(page_to_phys(page) + off, 0, len);
+
+		pad_tl_idx++;
+	}
+
+	idx = pad_hd_idx ? 0 : pad_tl_idx;
+	while (idx < count) {
+		struct page *page = info->bounce_buffer[idx++];
+
+		clear_highpage(page);
+		arch_sync_dma_for_device(page_to_phys(page), 0, PAGE_SIZE);
+		if (idx == pad_hd_idx)
+			idx = pad_tl_idx;
+	}
+}
+
 static bool io_bounce_buffers_map_buffer(struct io_bounce_buffers *buffers,
 					 struct io_bounce_buffer_info *info,
-					 int prot)
+					 int prot, bool skiped_sync,
+					 size_t offset, size_t orig_size)
 {
 	unsigned int count = info->size >> PAGE_SHIFT;
 	struct sg_table sgt;
 	size_t mapped;
 
+	if (offset || offset + orig_size < info->size || skiped_sync) {
+		// Ensure that nothing is leaked to untrusted devices when
+		// mapping the buffer by clearing any part of the bounce buffer
+		// that wasn't already cleared by syncing.
+		size_t pad_hd_end, pad_tl_start;
+
+		if (skiped_sync) {
+			pad_hd_end = pad_tl_start = 0;
+		} else {
+			pad_hd_end = offset;
+			pad_tl_start = offset + orig_size;
+		}
+		io_bounce_buffers_clear_padding(info, pad_hd_end, pad_tl_start);
+	}
+
 	if (sg_alloc_table_from_pages(&sgt, info->bounce_buffer, count, 0,
 				      info->size, GFP_ATOMIC))
 		return false;
@@ -338,7 +394,8 @@ bool io_bounce_buffers_map_page(struct io_bounce_buffers *buffers,
 		io_bounce_buffers_do_sync(buffers, info.bounce_buffer, offset,
 					  page, offset, size, dir, prot, false);
 
-	if (!io_bounce_buffers_map_buffer(buffers, &info, prot)) {
+	if (!io_bounce_buffers_map_buffer(buffers, &info, prot, skip_cpu_sync,
+					  offset, size)) {
 		io_buffer_manager_release_buffer(&buffers->manager,
 						 buffers->domain, info.iova,
 						 false, NULL, NULL);
@@ -381,7 +438,8 @@ bool io_bounce_buffers_map_sg(struct io_bounce_buffers *buffers,
 					    info.bounce_buffer, dir, prot,
 					    false);
 
-	if (!io_bounce_buffers_map_buffer(buffers, &info, prot)) {
+	if (!io_bounce_buffers_map_buffer(buffers, &info, prot, skip_cpu_sync,
+					  0, size)) {
 		io_buffer_manager_release_buffer(&buffers->manager,
 						 buffers->domain, info.iova,
 						 false, NULL, NULL);
drivers/iommu/io-buffer-manager.c +2 -5
index 79b9759da928..587584fdf26b 100644
@@ -37,13 +37,10 @@ static struct page **io_buffer_manager_alloc_pages(int count, unsigned int nid)
 	// pages first to make accessing the buffer cheaper.
 	for (i = 0; i < count; i++) {
 		pages[i] = alloc_pages_node(
-			nid,
-			GFP_ATOMIC | __GFP_ZERO | __GFP_NORETRY | __GFP_NOWARN,
-			0);
+			nid, GFP_ATOMIC | __GFP_NORETRY | __GFP_NOWARN, 0);
 		if (!pages[i]) {
 			pages[i] = alloc_pages_node(
-				nid, GFP_ATOMIC | __GFP_ZERO | __GFP_HIGHMEM,
-				0);
+				nid, GFP_ATOMIC | __GFP_HIGHMEM, 0);
 			if (!pages[i]) {
 				io_buffer_manager_free_pages(pages, i);
 				return NULL;
Signature · 6 lines
2.32.0.605.g8dce9f2422-goog

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#10 lore raw

PATCH v2 6/9 dma-iommu: add bounce buffer pools

In reply to David Stevens

From: David Stevens <[email protected]>

Add per-domain pools for IOMMU mapped bounce buffers. Each domain has 8 buffer pools, which hold buffers of size 2^n pages. Buffers are allocated on demand, and unused buffers are periodically released from the cache. Single use buffers are still used for mappings that are too large to use any pool, or if there are too many simultaneously in-use streaming mappings.

---
 drivers/iommu/dma-iommu.c         |  24 +-
 drivers/iommu/io-bounce-buffers.c |  42 ++-
 drivers/iommu/io-bounce-buffers.h |   3 +
 drivers/iommu/io-buffer-manager.c | 417 +++++++++++++++++++++++++++++-
 drivers/iommu/io-buffer-manager.h |  49 +++-
 include/linux/dma-iommu.h         |   2 +
 6 files changed, 520 insertions(+), 17 deletions(-)
drivers/iommu/dma-iommu.c +18 -6
index 908eb6fb7dc3..42f85b7a90f0 100644
@@ -378,6 +378,11 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base,
 			return -EFAULT;
 		}
 
+		if (cookie->bounce_buffers &&
+		    !io_bounce_buffer_reinit_check(cookie->bounce_buffers,
+						   dev, base, limit))
+			return -EFAULT;
+
 		return 0;
 	}
 
@@ -436,12 +441,24 @@ static int dma_info_to_prot(enum dma_data_direction dir, bool coherent,
 	}
 }
 
+u64 __iommu_dma_limit(struct iommu_domain *domain, struct device *dev, u64 mask)
+{
+	u64 dma_limit = mask;
+
+	dma_limit = min_not_zero(dma_limit, dev->bus_dma_limit);
+	if (domain->geometry.force_aperture)
+		dma_limit = min(dma_limit, (u64)domain->geometry.aperture_end);
+
+	return dma_limit;
+}
+
 dma_addr_t __iommu_dma_alloc_iova(struct iommu_domain *domain,
-		size_t size, u64 dma_limit, struct device *dev)
+		size_t size, u64 mask, struct device *dev)
 {
 	struct iommu_dma_cookie *cookie = domain->iova_cookie;
 	struct iova_domain *iovad = &cookie->iovad;
 	unsigned long shift, iova_len, iova = 0;
+	u64 dma_limit = __iommu_dma_limit(domain, dev, mask);
 
 	if (cookie->type == IOMMU_DMA_MSI_COOKIE) {
 		cookie->msi_iova += size;
@@ -459,11 +476,6 @@ dma_addr_t __iommu_dma_alloc_iova(struct iommu_domain *domain,
 	if (iova_len < (1 << (IOVA_RANGE_CACHE_MAX_SIZE - 1)))
 		iova_len = roundup_pow_of_two(iova_len);
 
-	dma_limit = min_not_zero(dma_limit, dev->bus_dma_limit);
-
-	if (domain->geometry.force_aperture)
-		dma_limit = min(dma_limit, (u64)domain->geometry.aperture_end);
-
 	/* Try to get PCI devices a SAC address */
 	if (dma_limit > DMA_BIT_MASK(32) && !iommu_dma_forcedac && dev_is_pci(dev))
 		iova = alloc_iova_fast(iovad, iova_len,
drivers/iommu/io-bounce-buffers.c +35 -7

103 lines not on this page · raw message

drivers/iommu/io-bounce-buffers.h +3 -0
index 6d132a27646c..cf329a9704fa 100644
@@ -14,6 +14,9 @@ struct io_bounce_buffers;
 struct io_bounce_buffers *io_bounce_buffers_init(struct device *dev,
 						 struct iommu_domain *domain,
 						 struct iova_domain *iovad);
+bool io_bounce_buffer_reinit_check(struct io_bounce_buffers *buffers,
+				   struct device *dev, dma_addr_t base,
+				   dma_addr_t limit);
 void io_bounce_buffers_destroy(struct io_bounce_buffers *buffers);
 
 bool io_bounce_buffers_sync_single(struct io_bounce_buffers *buffers,
drivers/iommu/io-buffer-manager.c +415 -2

488 lines not on this page · raw message

drivers/iommu/io-buffer-manager.h +47 -2

71 lines not on this page · raw message

include/linux/dma-iommu.h +2 -0
index 50f676678318..944fd491d94f 100644
@@ -49,6 +49,8 @@ dma_addr_t __iommu_dma_alloc_iova(struct iommu_domain *domain,
 				  struct device *dev);
 void __iommu_dma_free_iova(struct iommu_dma_cookie *cookie,
 		dma_addr_t iova, size_t size, struct page *freelist);
+u64 __iommu_dma_limit(struct iommu_domain *domain,
+		      struct device *dev, u64 mask);
 
 #else /* CONFIG_IOMMU_DMA */
 
Signature · 6 lines
2.32.0.605.g8dce9f2422-goog

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#11 lore raw

PATCH v2 7/9 dma-iommu: support iommu bounce buffer optimization

In reply to David Stevens

From: David Stevens <[email protected]>

Add config that uses IOMMU bounce buffer pools to avoid IOMMU interactions as much as possible for relatively small streaming DMA operations. This can lead to significant performance improvements on systems where IOMMU map/unmap operations are very slow, such as when running virtualized.

---
 drivers/iommu/Kconfig             | 11 +++++
 drivers/iommu/dma-iommu.c         |  5 ++-
 drivers/iommu/io-bounce-buffers.c | 70 +++++++++++++++++++++----------
 drivers/iommu/io-buffer-manager.c | 17 +++++---
 drivers/iommu/io-buffer-manager.h |  8 ++--
 include/linux/dma-iommu.h         |  2 +
 6 files changed, 81 insertions(+), 32 deletions(-)
drivers/iommu/Kconfig +11 -0
index 07b7c25cbed8..e573b5c276dc 100644
@@ -422,4 +422,15 @@ config SPRD_IOMMU
 
 	  Say Y here if you want to use the multimedia devices listed above.
 
+config IOMMU_BOUNCE_BUFFERS
+	bool "Use IOMMU bounce buffers"
+	depends on IOMMU_DMA
+	default n
+	help
+	  Use bounce buffers for small, streaming DMA operations. This may
+	  have performance benefits on systems where establishing IOMMU mappings
+	  is particularly expensive, such as when running as a guest.
+
+	  If unsure, say N here.
+
 endif # IOMMU_SUPPORT
drivers/iommu/dma-iommu.c +3 -2
index 42f85b7a90f0..965bc0a2f140 100644
@@ -324,7 +324,7 @@ static void iommu_dma_flush_iotlb_all(struct iova_domain *iovad)
 	domain->ops->flush_iotlb_all(domain);
 }
 
-static bool dev_is_untrusted(struct device *dev)
+bool dev_is_untrusted(struct device *dev)
 {
 	return dev_is_pci(dev) && to_pci_dev(dev)->untrusted;
 }
@@ -402,7 +402,8 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base,
 
 	ret = iova_reserve_iommu_regions(dev, domain);
 
-	if (ret == 0 && dev_is_untrusted(dev)) {
+	if (ret == 0 && (dev_is_untrusted(dev) ||
+			 IS_ENABLED(CONFIG_IOMMU_BOUNCE_BUFFERS))) {
 		cookie->bounce_buffers =
 			io_bounce_buffers_init(dev, domain, iovad);
 		if (IS_ERR(cookie->bounce_buffers))
drivers/iommu/io-bounce-buffers.c +48 -22

157 lines not on this page · raw message

drivers/iommu/io-buffer-manager.c +12 -5
index 1c69df08603c..0f7f003b53bb 100644
@@ -324,7 +324,8 @@ static bool io_buffer_manager_alloc_slot(struct io_buffer_manager *manager,
 
 bool io_buffer_manager_alloc_buffer(struct io_buffer_manager *manager,
 				    struct device *dev, void *orig_buffer,
-				    size_t size, int prot, unsigned int nid,
+				    size_t size, int prot, bool require_bounce,
+				    unsigned int nid,
 				    struct io_bounce_buffer_info *info,
 				    bool *new_buffer)
 {
@@ -336,6 +337,9 @@ bool io_buffer_manager_alloc_buffer(struct io_buffer_manager *manager,
 					 nid, info, new_buffer))
 		return true;
 
+	if (!require_bounce)
+		return false;
+
 	node = kzalloc(sizeof(*node), GFP_ATOMIC);
 	if (!node)
 		return false;
@@ -401,7 +405,7 @@ static bool __io_buffer_manager_find_slot(struct io_buffer_manager *manager,
 }
 
 bool io_buffer_manager_find_buffer(struct io_buffer_manager *manager,
-				   dma_addr_t handle,
+				   dma_addr_t handle, bool may_use_fallback,
 				   struct io_bounce_buffer_info *info,
 				   void **orig_buffer, int *prot)
 {
@@ -415,7 +419,8 @@ bool io_buffer_manager_find_buffer(struct io_buffer_manager *manager,
 		*orig_buffer = slot->orig_buffer;
 		*prot = slot->prot;
 		return true;
-	}
+	} else if (!may_use_fallback)
+		return false;
 
 	spin_lock_irqsave(&manager->fallback_lock, flags);
 	node = find_fallback_node(&manager->fallback_buffers, handle);
@@ -433,7 +438,8 @@ bool io_buffer_manager_find_buffer(struct io_buffer_manager *manager,
 bool io_buffer_manager_release_buffer(struct io_buffer_manager *manager,
 				      struct iommu_domain *domain,
 				      dma_addr_t handle, bool inited,
-				      prerelease_cb cb, void *ctx)
+				      bool may_use_fallback, prerelease_cb cb,
+				      void *ctx)
 {
 	struct io_buffer_slot *slot, **cache;
 	struct io_buffer_pool *pool;
@@ -472,7 +478,8 @@ bool io_buffer_manager_release_buffer(struct io_buffer_manager *manager,
 
 		spin_unlock_irqrestore(&pool->lock, flags);
 		return true;
-	}
+	} else if (!may_use_fallback)
+		return false;
 
 	spin_lock_irqsave(&manager->fallback_lock, flags);
 	node = find_fallback_node(&manager->fallback_buffers, handle);
drivers/iommu/io-buffer-manager.h +5 -3
index 2aa3b9afcb3d..3d32f9366536 100644
@@ -57,12 +57,13 @@ struct io_bounce_buffer_info {
 
 bool io_buffer_manager_alloc_buffer(struct io_buffer_manager *manager,
 				    struct device *dev, void *orig_buffer,
-				    size_t size, int prot, unsigned int nid,
+				    size_t size, int prot, bool use_fallback,
+				    unsigned int nid,
 				    struct io_bounce_buffer_info *info,
 				    bool *new_buffer);
 
 bool io_buffer_manager_find_buffer(struct io_buffer_manager *manager,
-				   dma_addr_t handle,
+				   dma_addr_t handle, bool may_use_fallback,
 				   struct io_bounce_buffer_info *info,
 				   void **orig_buffer, int *prot);
 
@@ -72,7 +73,8 @@ typedef void (*prerelease_cb)(struct io_bounce_buffer_info *info, int prot,
 bool io_buffer_manager_release_buffer(struct io_buffer_manager *manager,
 				      struct iommu_domain *domain,
 				      dma_addr_t handle, bool inited,
-				      prerelease_cb cb, void *ctx);
+				      bool may_use_fallback, prerelease_cb cb,
+				      void *ctx);
 
 int io_buffer_manager_init(struct io_buffer_manager *manager,
 			   struct device *dev, struct iova_domain *iovad,
include/linux/dma-iommu.h +2 -0
index 944fd491d94f..70bed650d5d1 100644
@@ -52,6 +52,8 @@ void __iommu_dma_free_iova(struct iommu_dma_cookie *cookie,
 u64 __iommu_dma_limit(struct iommu_domain *domain,
 		      struct device *dev, u64 mask);
 
+bool dev_is_untrusted(struct device *dev);
+
 #else /* CONFIG_IOMMU_DMA */
 
 struct iommu_domain;
Signature · 6 lines
2.32.0.605.g8dce9f2422-goog

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#12 lore raw

PATCH v2 8/9 dma-mapping: add persistent streaming mapping flag

In reply to David Stevens

From: David Stevens <[email protected]>

Add a DMA_ATTR_PERSISTENT_STREAMING flag which indicates that the streaming mapping is long lived and that the caller will manage coherency either through the dma_sync_* functions or via some other use-case specific mechanism. This flag indicates to the platform that it should optimize for more efficient syncing at the cost of more expensive mapping and unmapping.

This flag is used to skip optional bounce buffers when CONFIG_IOMMU_BOUNCE_BUFFERS is enabled. With respect to these bounce buffers, in most cases the flag is an optimization. However, callers which do not use the dma_sync_* calls to manage coherency must use this flag to work properly when CONFIG_IOMMU_BOUNCE_BUFFERS is enabled.

---
 drivers/iommu/io-bounce-buffers.c | 14 +++++++++++---
 include/linux/dma-mapping.h       | 11 +++++++++++
 2 files changed, 22 insertions(+), 3 deletions(-)
drivers/iommu/io-bounce-buffers.c +11 -3
index af8c2a51eeed..3a0071d5a9ea 100644
@@ -409,8 +409,16 @@ static bool io_bounce_buffers_map_buffer(struct io_bounce_buffers *buffers,
 	return mapped >= info->size;
 }
 
-static bool use_bounce_buffer(bool force_bounce, size_t size)
+static bool use_bounce_buffer(struct device *dev, unsigned long attrs,
+			      bool force_bounce, size_t size)
 {
+	if (attrs & DMA_ATTR_PERSISTENT_STREAMING) {
+		WARN_ONCE(force_bounce,
+			  "Skipping bounce buffer for untrusted mapping %s\n",
+			  dev_name(dev));
+		return false;
+	}
+
 	if (IS_ENABLED(CONFIG_IOMMU_BOUNCE_BUFFERS) &&
 	    size <= always_bounce_limit)
 		return true;
@@ -429,7 +437,7 @@ bool io_bounce_buffers_map_page(struct io_bounce_buffers *buffers,
 	bool force_bounce = buffers->untrusted &&
 			    iova_offset(buffers->iovad, offset | size);
 
-	if (!use_bounce_buffer(force_bounce, size))
+	if (!use_bounce_buffer(dev, attrs, force_bounce, size))
 		return false;
 
 	*handle = DMA_MAPPING_ERROR;
@@ -476,7 +484,7 @@ bool io_bounce_buffers_map_sg(struct io_bounce_buffers *buffers,
 				buffers->iovad, iter->offset | iter->length);
 	}
 
-	if (!use_bounce_buffer(force_bounce, size))
+	if (!use_bounce_buffer(dev, attrs, force_bounce, size))
 		return false;
 
 	*out_nents = 0;
include/linux/dma-mapping.h +11 -0
index 183e7103a66d..5d318753bb79 100644
@@ -61,6 +61,17 @@
  */
 #define DMA_ATTR_PRIVILEGED		(1UL << 9)
 
+/*
+ * DMA_ATTR_PERSISTENT_STREAMING: Indicates that the streaming mapping is long
+ * lived, so syncing performance should be prioritized over mapping/unmapping
+ * performance. Platform code will establish a mapping which only requires CPU
+ * cache synchronization.
+ *
+ * Callers that create long lived mappings and directly handle CPU cache
+ * management without calling using dma_sync_* functions must set this flag.
+ */
+#define DMA_ATTR_PERSISTENT_STREAMING	(1UL << 10)
+
 /*
  * A dma_addr_t can hold any valid DMA or bus address for the platform.  It can
  * be given to a device to use as a DMA source or target.  It is specific to a
Signature · 6 lines
2.32.0.605.g8dce9f2422-goog

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#13 lore raw

PATCH v2 9/9 drm/i915: use DMA_ATTR_PERSISTENT_STREAMING flag

In reply to David Stevens

From: David Stevens <[email protected]>

Use the new DMA_ATTR_PERSISTENT_STREAMING for long lived dma mappings which directly handle CPU cache coherency instead of using dma_sync_*.

---
 drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 4 +++-
 drivers/gpu/drm/i915/i915_gem_gtt.c        | 3 ++-
 2 files changed, 5 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c +3 -1
index 616c3a2f1baf..df982cfb4f34 100644
@@ -48,7 +48,9 @@ static struct sg_table *i915_gem_map_dma_buf(struct dma_buf_attachment *attachme
 		src = sg_next(src);
 	}
 
-	ret = dma_map_sgtable(attachment->dev, st, dir, DMA_ATTR_SKIP_CPU_SYNC);
+	ret = dma_map_sgtable(attachment->dev, st, dir,
+			      DMA_ATTR_SKIP_CPU_SYNC |
+			      DMA_ATTR_PERSISTENT_STREAMING);
 	if (ret)
 		goto err_free_sg;
 
drivers/gpu/drm/i915/i915_gem_gtt.c +2 -1
index 36489be4896b..f27a849631f7 100644
@@ -33,7 +33,8 @@ int i915_gem_gtt_prepare_pages(struct drm_i915_gem_object *obj,
 				     PCI_DMA_BIDIRECTIONAL,
 				     DMA_ATTR_SKIP_CPU_SYNC |
 				     DMA_ATTR_NO_KERNEL_MAPPING |
-				     DMA_ATTR_NO_WARN))
+				     DMA_ATTR_NO_WARN |
+				     DMA_ATTR_PERSISTENT_STREAMING))
 			return 0;
 
 		/*
Signature · 6 lines
2.32.0.605.g8dce9f2422-goog

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#14 lore raw
In reply to David Stevens
Quoted 88 lines · David Stevens

From: David Stevens <[email protected]>

This patch series adds support for per-domain dynamic pools of iommu bounce buffers to the dma-iommu API. This allows iommu mappings to be reused while still maintaining strict iommu protection.

This bounce buffer support is used to add a new config option that, when enabled, causes all non-direct streaming mappings below a configurable size to go through the bounce buffers. This serves as an optimization on systems where manipulating iommu mappings is very expensive. For example, virtio-iommu operations in a guest on a linux host require a vmexit, involvement the VMM, and a VFIO syscall. For relatively small DMA operations, memcpy can be significantly faster.

As a performance comparison, on a device with an i5-10210U, I ran fio with a VFIO passthrough NVMe drive and virtio-iommu with '--direct=1 --rw=read --ioengine=libaio --iodepth=64' and block sizes 4k, 16k, 64k, and 128k. Test throughput increased by 2.8x, 4.7x, 3.6x, and 3.6x. Time spent in iommu_dma_unmap_(page|sg) per GB processed decreased by 97%, 94%, 90%, and 87%. Time spent in iommu_dma_map_(page|sg) decreased by >99%, as bounce buffers don't require syncing here in the read case. Running with multiple jobs doesn't serve as a useful performance comparison because virtio-iommu and vfio_iommu_type1 both have big locks that significantly limit mulithreaded DMA performance.

These pooled bounce buffers are also used for subgranule mappings with untrusted devices, replacing the single use bounce buffers used currently. The biggest difference here is that the new implementation maps a whole sglist using a single bounce buffer. The new implementation does not support using bounce buffers for only some segments of the sglist, so it may require more copying. However, the current implementation requires per-segment iommu map/unmap operations for all untrusted sglist mappings (fully aligned sglists included). On a i5-10210U laptop with the internal NVMe drive made to appear untrusted, fio --direct=1 --rw=read --ioengine=libaio --iodepth=64 --bs=64k showed a statistically significant decrease in CPU load from 2.28% -> 2.17% with the new iommu bounce buffer optimization enabled.

Each domain's buffer pool is split into multiple power-of-2 size classes. Each class allocates a fixed number of buffer slot metadata. A large iova range is allocated, and each slot is assigned an iova from the range. This allows the iova to be easily mapped back to the slot, and allows the critical section of most pool operations to be constant time. The one exception is finding a cached buffer to reuse. These are only separated according to R/W permissions - the use of other permissions such as IOMMU_PRIV may require a linear search through the cache. However, these other permissions are rare and likely exhibit high locality, so the should not be a bottleneck in practice.

Since untrusted devices may require bounce buffers, each domain has a fallback rbtree to manage single use buffers. This may be necessary if a very large number of DMA operations are simultaneously in-flight, or for very large individual DMA operations.

This patch set does not use swiotlb. There are two primary ways in which swiotlb isn't compatible with per-domain buffer pools. First, swiotlb allocates buffers to be compatible with a single device, whereas per-domain buffer pools don't handle that during buffer allocation as a single buffer may end up being used by multiple devices. Second, swiotlb allocation establishes the original to bounce buffer mapping, which again doesn't work if buffers can be reused. Effectively the only code that can be shared between the two use cases is allocating slots from the swiotlb's memory. However, given that we're going to be allocating memory for use with an iommu, allocating memory from a block of memory explicitly set aside to deal with a lack of iommu seems kind of contradictory. At best there might be a small performance improvement if wiotlb allocation is faster than regular page allocation, but buffer allocation isn't on the hot path anyway.

Not using the swiotlb has the benefit that memory doesn't have to be preallocated. Instead, bounce buffers consume memory only for in-flight dma transactions (ignoring temporarily cached buffers), which is the smallest amount possible. This makes it easier to use bounce buffers as an optimization on systems with large numbers of devices or in situations where devices are unknown, since it is not necessary to try to tune how much memory needs to be set aside to achieve good performance without costing too much memory.

Finally, this series adds a new DMA_ATTR_PERSISTENT_STREAMING flag. This is meant to address devices which create long lived streaming mappings but manage CPU cache coherency without using the dma_sync_* APIs. Currently, these devices don't function properly with swiotlb=force. The new flag is used to bypass bounce buffers so such devices will function when the new bounce buffer optimization is enabled. The flag is added to the i915 driver, which creates such mappings. It can also be added to various dma-buf implementations as an optimization, although that is not done here.

Quoted 7 lines

v1 -> v2:
- Replace existing untrusted bounce buffers with new bounce buffer pools. This includes significant rework to account for untrusted bounce buffers being required instead of an optimization.
- Add flag for persistent streaming mappings.

Hi David,

I'm currently looking into converting s390 from our custom IOMMU based DMA API implementation to using dma-iommu.c. We're always using an IOMMU for PCI devices even when doing pass-through to guests (under both the KVM and z/VM hypervisors). In this case I/O TLB flushes, which we use to do the shadowing of the guest I/O translations, are relatively expensive I'm thus very interested in your work. I've tried rebasing it on v5.18 and got it to compile but didn't get DMA to work though it seems to partially work as I don't get probe failures unlike with a completely broken DMA API. Since I might have very well screwed up the rebase and my DMA API conversion is experimental too I was wondering if you're still working on this and might have a current version I could experiment with?

Thanks,
Niklas

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#15 lore raw
Quoted 100 lines · Niklas Schnelle
Quoted 88 lines · David Stevens

From: David Stevens <[email protected]>

This patch series adds support for per-domain dynamic pools of iommu bounce buffers to the dma-iommu API. This allows iommu mappings to be reused while still maintaining strict iommu protection.

This bounce buffer support is used to add a new config option that, when enabled, causes all non-direct streaming mappings below a configurable size to go through the bounce buffers. This serves as an optimization on systems where manipulating iommu mappings is very expensive. For example, virtio-iommu operations in a guest on a linux host require a vmexit, involvement the VMM, and a VFIO syscall. For relatively small DMA operations, memcpy can be significantly faster.

As a performance comparison, on a device with an i5-10210U, I ran fio with a VFIO passthrough NVMe drive and virtio-iommu with '--direct=1 --rw=read --ioengine=libaio --iodepth=64' and block sizes 4k, 16k, 64k, and 128k. Test throughput increased by 2.8x, 4.7x, 3.6x, and 3.6x. Time spent in iommu_dma_unmap_(page|sg) per GB processed decreased by 97%, 94%, 90%, and 87%. Time spent in iommu_dma_map_(page|sg) decreased by >99%, as bounce buffers don't require syncing here in the read case. Running with multiple jobs doesn't serve as a useful performance comparison because virtio-iommu and vfio_iommu_type1 both have big locks that significantly limit mulithreaded DMA performance.

These pooled bounce buffers are also used for subgranule mappings with untrusted devices, replacing the single use bounce buffers used currently. The biggest difference here is that the new implementation maps a whole sglist using a single bounce buffer. The new implementation does not support using bounce buffers for only some segments of the sglist, so it may require more copying. However, the current implementation requires per-segment iommu map/unmap operations for all untrusted sglist mappings (fully aligned sglists included). On a i5-10210U laptop with the internal NVMe drive made to appear untrusted, fio --direct=1 --rw=read --ioengine=libaio --iodepth=64 --bs=64k showed a statistically significant decrease in CPU load from 2.28% -> 2.17% with the new iommu bounce buffer optimization enabled.

Each domain's buffer pool is split into multiple power-of-2 size classes. Each class allocates a fixed number of buffer slot metadata. A large iova range is allocated, and each slot is assigned an iova from the range. This allows the iova to be easily mapped back to the slot, and allows the critical section of most pool operations to be constant time. The one exception is finding a cached buffer to reuse. These are only separated according to R/W permissions - the use of other permissions such as IOMMU_PRIV may require a linear search through the cache. However, these other permissions are rare and likely exhibit high locality, so the should not be a bottleneck in practice.

Since untrusted devices may require bounce buffers, each domain has a fallback rbtree to manage single use buffers. This may be necessary if a very large number of DMA operations are simultaneously in-flight, or for very large individual DMA operations.

This patch set does not use swiotlb. There are two primary ways in which swiotlb isn't compatible with per-domain buffer pools. First, swiotlb allocates buffers to be compatible with a single device, whereas per-domain buffer pools don't handle that during buffer allocation as a single buffer may end up being used by multiple devices. Second, swiotlb allocation establishes the original to bounce buffer mapping, which again doesn't work if buffers can be reused. Effectively the only code that can be shared between the two use cases is allocating slots from the swiotlb's memory. However, given that we're going to be allocating memory for use with an iommu, allocating memory from a block of memory explicitly set aside to deal with a lack of iommu seems kind of contradictory. At best there might be a small performance improvement if wiotlb allocation is faster than regular page allocation, but buffer allocation isn't on the hot path anyway.

Not using the swiotlb has the benefit that memory doesn't have to be preallocated. Instead, bounce buffers consume memory only for in-flight dma transactions (ignoring temporarily cached buffers), which is the smallest amount possible. This makes it easier to use bounce buffers as an optimization on systems with large numbers of devices or in situations where devices are unknown, since it is not necessary to try to tune how much memory needs to be set aside to achieve good performance without costing too much memory.

Finally, this series adds a new DMA_ATTR_PERSISTENT_STREAMING flag. This is meant to address devices which create long lived streaming mappings but manage CPU cache coherency without using the dma_sync_* APIs. Currently, these devices don't function properly with swiotlb=force. The new flag is used to bypass bounce buffers so such devices will function when the new bounce buffer optimization is enabled. The flag is added to the i915 driver, which creates such mappings. It can also be added to various dma-buf implementations as an optimization, although that is not done here.

Quoted 7 lines

v1 -> v2:
- Replace existing untrusted bounce buffers with new bounce buffer pools. This includes significant rework to account for untrusted bounce buffers being required instead of an optimization.
- Add flag for persistent streaming mappings.

Hi David,

Quoted 12 lines

I'm currently looking into converting s390 from our custom IOMMU based DMA API implementation to using dma-iommu.c. We're always using an IOMMU for PCI devices even when doing pass-through to guests (under both the KVM and z/VM hypervisors). In this case I/O TLB flushes, which we use to do the shadowing of the guest I/O translations, are relatively expensive I'm thus very interested in your work. I've tried rebasing it on v5.18 and got it to compile but didn't get DMA to work though it seems to partially work as I don't get probe failures unlike with a completely broken DMA API. Since I might have very well screwed up the rebase and my DMA API conversion is experimental too I was wondering if you're still working on this and might have a current version I could experiment with?

Unfortunately I don't have anything more recent to share. I've come across some performance issues caused by pathological usage patterns in internal usage, but I haven't seen any correctness issues. I'm hoping that I'll be able to address the performance issues and send a rebased series within the next month or so.

It's definitely possible that this series has some bugs. I've tested it on a range of chromebooks and their various hardware and drivers, but that's still all relatively normal x86_64/arm64. If your hardware is more particular about its DMA, this series might be missing something.

-David
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#16 lore raw
Quoted 119 lines · David Stevens
Quoted 99 lines · Niklas Schnelle
Quoted 88 lines · David Stevens

From: David Stevens <[email protected]>

This patch series adds support for per-domain dynamic pools of iommu bounce buffers to the dma-iommu API. This allows iommu mappings to be reused while still maintaining strict iommu protection.

This bounce buffer support is used to add a new config option that, when enabled, causes all non-direct streaming mappings below a configurable size to go through the bounce buffers. This serves as an optimization on systems where manipulating iommu mappings is very expensive. For example, virtio-iommu operations in a guest on a linux host require a vmexit, involvement the VMM, and a VFIO syscall. For relatively small DMA operations, memcpy can be significantly faster.

As a performance comparison, on a device with an i5-10210U, I ran fio with a VFIO passthrough NVMe drive and virtio-iommu with '--direct=1 --rw=read --ioengine=libaio --iodepth=64' and block sizes 4k, 16k, 64k, and 128k. Test throughput increased by 2.8x, 4.7x, 3.6x, and 3.6x. Time spent in iommu_dma_unmap_(page|sg) per GB processed decreased by 97%, 94%, 90%, and 87%. Time spent in iommu_dma_map_(page|sg) decreased by >99%, as bounce buffers don't require syncing here in the read case. Running with multiple jobs doesn't serve as a useful performance comparison because virtio-iommu and vfio_iommu_type1 both have big locks that significantly limit mulithreaded DMA performance.

These pooled bounce buffers are also used for subgranule mappings with untrusted devices, replacing the single use bounce buffers used currently. The biggest difference here is that the new implementation maps a whole sglist using a single bounce buffer. The new implementation does not support using bounce buffers for only some segments of the sglist, so it may require more copying. However, the current implementation requires per-segment iommu map/unmap operations for all untrusted sglist mappings (fully aligned sglists included). On a i5-10210U laptop with the internal NVMe drive made to appear untrusted, fio --direct=1 --rw=read --ioengine=libaio --iodepth=64 --bs=64k showed a statistically significant decrease in CPU load from 2.28% -> 2.17% with the new iommu bounce buffer optimization enabled.

Each domain's buffer pool is split into multiple power-of-2 size classes. Each class allocates a fixed number of buffer slot metadata. A large iova range is allocated, and each slot is assigned an iova from the range. This allows the iova to be easily mapped back to the slot, and allows the critical section of most pool operations to be constant time. The one exception is finding a cached buffer to reuse. These are only separated according to R/W permissions - the use of other permissions such as IOMMU_PRIV may require a linear search through the cache. However, these other permissions are rare and likely exhibit high locality, so the should not be a bottleneck in practice.

Since untrusted devices may require bounce buffers, each domain has a fallback rbtree to manage single use buffers. This may be necessary if a very large number of DMA operations are simultaneously in-flight, or for very large individual DMA operations.

This patch set does not use swiotlb. There are two primary ways in which swiotlb isn't compatible with per-domain buffer pools. First, swiotlb allocates buffers to be compatible with a single device, whereas per-domain buffer pools don't handle that during buffer allocation as a single buffer may end up being used by multiple devices. Second, swiotlb allocation establishes the original to bounce buffer mapping, which again doesn't work if buffers can be reused. Effectively the only code that can be shared between the two use cases is allocating slots from the swiotlb's memory. However, given that we're going to be allocating memory for use with an iommu, allocating memory from a block of memory explicitly set aside to deal with a lack of iommu seems kind of contradictory. At best there might be a small performance improvement if wiotlb allocation is faster than regular page allocation, but buffer allocation isn't on the hot path anyway.

Not using the swiotlb has the benefit that memory doesn't have to be preallocated. Instead, bounce buffers consume memory only for in-flight dma transactions (ignoring temporarily cached buffers), which is the smallest amount possible. This makes it easier to use bounce buffers as an optimization on systems with large numbers of devices or in situations where devices are unknown, since it is not necessary to try to tune how much memory needs to be set aside to achieve good performance without costing too much memory.

Finally, this series adds a new DMA_ATTR_PERSISTENT_STREAMING flag. This is meant to address devices which create long lived streaming mappings but manage CPU cache coherency without using the dma_sync_* APIs. Currently, these devices don't function properly with swiotlb=force. The new flag is used to bypass bounce buffers so such devices will function when the new bounce buffer optimization is enabled. The flag is added to the i915 driver, which creates such mappings. It can also be added to various dma-buf implementations as an optimization, although that is not done here.

Quoted 7 lines

v1 -> v2:
- Replace existing untrusted bounce buffers with new bounce buffer pools. This includes significant rework to account for untrusted bounce buffers being required instead of an optimization.
- Add flag for persistent streaming mappings.

Hi David,

Quoted 12 lines

I'm currently looking into converting s390 from our custom IOMMU based DMA API implementation to using dma-iommu.c. We're always using an IOMMU for PCI devices even when doing pass-through to guests (under both the KVM and z/VM hypervisors). In this case I/O TLB flushes, which we use to do the shadowing of the guest I/O translations, are relatively expensive I'm thus very interested in your work. I've tried rebasing it on v5.18 and got it to compile but didn't get DMA to work though it seems to partially work as I don't get probe failures unlike with a completely broken DMA API. Since I might have very well screwed up the rebase and my DMA API conversion is experimental too I was wondering if you're still working on this and might have a current version I could experiment with?

Unfortunately I don't have anything more recent to share. I've come across some performance issues caused by pathological usage patterns in internal usage, but I haven't seen any correctness issues. I'm hoping that I'll be able to address the performance issues and send a rebased series within the next month or so.

Quoted 7 lines

It's definitely possible that this series has some bugs. I've tested it on a range of chromebooks and their various hardware and drivers, but that's still all relatively normal x86_64/arm64. If your hardware is more particular about its DMA, this series might be missing something.

-David

Hi David,

Thanks for the answer. The only unusual thing about our DMA is that we only do 64 bit DMA and IOVAs are always >2^32. I don't think I triggered a bug in your code though, rather I think I made some mistake in the rebase onto 5.18 as some of the APIs changed a bit. I'm out next week but may try it again and possibly just test on x86_64 if it doesn't work on s390. If you have anything new I'd be interested to hear of course. Also could you say anything more about the pathological usage patterns?

Thanks,
Niklas

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#17 lore raw
Quoted 132 lines · Niklas Schnelle
Quoted 119 lines · David Stevens
Quoted 99 lines · Niklas Schnelle
Quoted 88 lines · David Stevens

From: David Stevens <[email protected]>

This patch series adds support for per-domain dynamic pools of iommu bounce buffers to the dma-iommu API. This allows iommu mappings to be reused while still maintaining strict iommu protection.

This bounce buffer support is used to add a new config option that, when enabled, causes all non-direct streaming mappings below a configurable size to go through the bounce buffers. This serves as an optimization on systems where manipulating iommu mappings is very expensive. For example, virtio-iommu operations in a guest on a linux host require a vmexit, involvement the VMM, and a VFIO syscall. For relatively small DMA operations, memcpy can be significantly faster.

As a performance comparison, on a device with an i5-10210U, I ran fio with a VFIO passthrough NVMe drive and virtio-iommu with '--direct=1 --rw=read --ioengine=libaio --iodepth=64' and block sizes 4k, 16k, 64k, and 128k. Test throughput increased by 2.8x, 4.7x, 3.6x, and 3.6x. Time spent in iommu_dma_unmap_(page|sg) per GB processed decreased by 97%, 94%, 90%, and 87%. Time spent in iommu_dma_map_(page|sg) decreased by >99%, as bounce buffers don't require syncing here in the read case. Running with multiple jobs doesn't serve as a useful performance comparison because virtio-iommu and vfio_iommu_type1 both have big locks that significantly limit mulithreaded DMA performance.

These pooled bounce buffers are also used for subgranule mappings with untrusted devices, replacing the single use bounce buffers used currently. The biggest difference here is that the new implementation maps a whole sglist using a single bounce buffer. The new implementation does not support using bounce buffers for only some segments of the sglist, so it may require more copying. However, the current implementation requires per-segment iommu map/unmap operations for all untrusted sglist mappings (fully aligned sglists included). On a i5-10210U laptop with the internal NVMe drive made to appear untrusted, fio --direct=1 --rw=read --ioengine=libaio --iodepth=64 --bs=64k showed a statistically significant decrease in CPU load from 2.28% -> 2.17% with the new iommu bounce buffer optimization enabled.

Each domain's buffer pool is split into multiple power-of-2 size classes. Each class allocates a fixed number of buffer slot metadata. A large iova range is allocated, and each slot is assigned an iova from the range. This allows the iova to be easily mapped back to the slot, and allows the critical section of most pool operations to be constant time. The one exception is finding a cached buffer to reuse. These are only separated according to R/W permissions - the use of other permissions such as IOMMU_PRIV may require a linear search through the cache. However, these other permissions are rare and likely exhibit high locality, so the should not be a bottleneck in practice.

Since untrusted devices may require bounce buffers, each domain has a fallback rbtree to manage single use buffers. This may be necessary if a very large number of DMA operations are simultaneously in-flight, or for very large individual DMA operations.

This patch set does not use swiotlb. There are two primary ways in which swiotlb isn't compatible with per-domain buffer pools. First, swiotlb allocates buffers to be compatible with a single device, whereas per-domain buffer pools don't handle that during buffer allocation as a single buffer may end up being used by multiple devices. Second, swiotlb allocation establishes the original to bounce buffer mapping, which again doesn't work if buffers can be reused. Effectively the only code that can be shared between the two use cases is allocating slots from the swiotlb's memory. However, given that we're going to be allocating memory for use with an iommu, allocating memory from a block of memory explicitly set aside to deal with a lack of iommu seems kind of contradictory. At best there might be a small performance improvement if wiotlb allocation is faster than regular page allocation, but buffer allocation isn't on the hot path anyway.

Not using the swiotlb has the benefit that memory doesn't have to be preallocated. Instead, bounce buffers consume memory only for in-flight dma transactions (ignoring temporarily cached buffers), which is the smallest amount possible. This makes it easier to use bounce buffers as an optimization on systems with large numbers of devices or in situations where devices are unknown, since it is not necessary to try to tune how much memory needs to be set aside to achieve good performance without costing too much memory.

Finally, this series adds a new DMA_ATTR_PERSISTENT_STREAMING flag. This is meant to address devices which create long lived streaming mappings but manage CPU cache coherency without using the dma_sync_* APIs. Currently, these devices don't function properly with swiotlb=force. The new flag is used to bypass bounce buffers so such devices will function when the new bounce buffer optimization is enabled. The flag is added to the i915 driver, which creates such mappings. It can also be added to various dma-buf implementations as an optimization, although that is not done here.

Quoted 7 lines

v1 -> v2:
- Replace existing untrusted bounce buffers with new bounce buffer pools. This includes significant rework to account for untrusted bounce buffers being required instead of an optimization.
- Add flag for persistent streaming mappings.

Hi David,

Quoted 12 lines

I'm currently looking into converting s390 from our custom IOMMU based DMA API implementation to using dma-iommu.c. We're always using an IOMMU for PCI devices even when doing pass-through to guests (under both the KVM and z/VM hypervisors). In this case I/O TLB flushes, which we use to do the shadowing of the guest I/O translations, are relatively expensive I'm thus very interested in your work. I've tried rebasing it on v5.18 and got it to compile but didn't get DMA to work though it seems to partially work as I don't get probe failures unlike with a completely broken DMA API. Since I might have very well screwed up the rebase and my DMA API conversion is experimental too I was wondering if you're still working on this and might have a current version I could experiment with?

Unfortunately I don't have anything more recent to share. I've come across some performance issues caused by pathological usage patterns in internal usage, but I haven't seen any correctness issues. I'm hoping that I'll be able to address the performance issues and send a rebased series within the next month or so.

Quoted 7 lines

It's definitely possible that this series has some bugs. I've tested it on a range of chromebooks and their various hardware and drivers, but that's still all relatively normal x86_64/arm64. If your hardware is more particular about its DMA, this series might be missing something.

-David

Hi David,

Quoted 8 lines

Thanks for the answer. The only unusual thing about our DMA is that we only do 64 bit DMA and IOVAs are always >2^32. I don't think I triggered a bug in your code though, rather I think I made some mistake in the rebase onto 5.18 as some of the APIs changed a bit. I'm out next week but may try it again and possibly just test on x86_64 if it doesn't work on s390. If you have anything new I'd be interested to hear of course. Also could you say anything more about the pathological usage patterns?

The problem with this implementation is that if you fall outside the max number/size for the bounce buffer pools, then DMA performance can fall off of a cliff. Although those max parameters are tunable, it's always possible to construct a workload that falls outside of the expected bounds. I think this can be addressed by adding a fallback pooling structure. It won't necessarily be as performant as the primary buffer pools, but I think it should work reasonably well for usage patterns that fall outside the primary buffer pools.

-David

#18 lore raw
In reply to David Stevens
Quoted 119 lines · David Stevens
Quoted 99 lines · Niklas Schnelle
Quoted 88 lines · David Stevens

From: David Stevens <[email protected]>

This patch series adds support for per-domain dynamic pools of iommu bounce buffers to the dma-iommu API. This allows iommu mappings to be reused while still maintaining strict iommu protection.

This bounce buffer support is used to add a new config option that, when enabled, causes all non-direct streaming mappings below a configurable size to go through the bounce buffers. This serves as an optimization on systems where manipulating iommu mappings is very expensive. For example, virtio-iommu operations in a guest on a linux host require a vmexit, involvement the VMM, and a VFIO syscall. For relatively small DMA operations, memcpy can be significantly faster.

As a performance comparison, on a device with an i5-10210U, I ran fio with a VFIO passthrough NVMe drive and virtio-iommu with '--direct=1 --rw=read --ioengine=libaio --iodepth=64' and block sizes 4k, 16k, 64k, and 128k. Test throughput increased by 2.8x, 4.7x, 3.6x, and 3.6x. Time spent in iommu_dma_unmap_(page|sg) per GB processed decreased by 97%, 94%, 90%, and 87%. Time spent in iommu_dma_map_(page|sg) decreased by >99%, as bounce buffers don't require syncing here in the read case. Running with multiple jobs doesn't serve as a useful performance comparison because virtio-iommu and vfio_iommu_type1 both have big locks that significantly limit mulithreaded DMA performance.

These pooled bounce buffers are also used for subgranule mappings with untrusted devices, replacing the single use bounce buffers used currently. The biggest difference here is that the new implementation maps a whole sglist using a single bounce buffer. The new implementation does not support using bounce buffers for only some segments of the sglist, so it may require more copying. However, the current implementation requires per-segment iommu map/unmap operations for all untrusted sglist mappings (fully aligned sglists included). On a i5-10210U laptop with the internal NVMe drive made to appear untrusted, fio --direct=1 --rw=read --ioengine=libaio --iodepth=64 --bs=64k showed a statistically significant decrease in CPU load from 2.28% -> 2.17% with the new iommu bounce buffer optimization enabled.

Each domain's buffer pool is split into multiple power-of-2 size classes. Each class allocates a fixed number of buffer slot metadata. A large iova range is allocated, and each slot is assigned an iova from the range. This allows the iova to be easily mapped back to the slot, and allows the critical section of most pool operations to be constant time. The one exception is finding a cached buffer to reuse. These are only separated according to R/W permissions - the use of other permissions such as IOMMU_PRIV may require a linear search through the cache. However, these other permissions are rare and likely exhibit high locality, so the should not be a bottleneck in practice.

Since untrusted devices may require bounce buffers, each domain has a fallback rbtree to manage single use buffers. This may be necessary if a very large number of DMA operations are simultaneously in-flight, or for very large individual DMA operations.

This patch set does not use swiotlb. There are two primary ways in which swiotlb isn't compatible with per-domain buffer pools. First, swiotlb allocates buffers to be compatible with a single device, whereas per-domain buffer pools don't handle that during buffer allocation as a single buffer may end up being used by multiple devices. Second, swiotlb allocation establishes the original to bounce buffer mapping, which again doesn't work if buffers can be reused. Effectively the only code that can be shared between the two use cases is allocating slots from the swiotlb's memory. However, given that we're going to be allocating memory for use with an iommu, allocating memory from a block of memory explicitly set aside to deal with a lack of iommu seems kind of contradictory. At best there might be a small performance improvement if wiotlb allocation is faster than regular page allocation, but buffer allocation isn't on the hot path anyway.

Not using the swiotlb has the benefit that memory doesn't have to be preallocated. Instead, bounce buffers consume memory only for in-flight dma transactions (ignoring temporarily cached buffers), which is the smallest amount possible. This makes it easier to use bounce buffers as an optimization on systems with large numbers of devices or in situations where devices are unknown, since it is not necessary to try to tune how much memory needs to be set aside to achieve good performance without costing too much memory.

Finally, this series adds a new DMA_ATTR_PERSISTENT_STREAMING flag. This is meant to address devices which create long lived streaming mappings but manage CPU cache coherency without using the dma_sync_* APIs. Currently, these devices don't function properly with swiotlb=force. The new flag is used to bypass bounce buffers so such devices will function when the new bounce buffer optimization is enabled. The flag is added to the i915 driver, which creates such mappings. It can also be added to various dma-buf implementations as an optimization, although that is not done here.

Quoted 7 lines

v1 -> v2:
- Replace existing untrusted bounce buffers with new bounce buffer pools. This includes significant rework to account for untrusted bounce buffers being required instead of an optimization.
- Add flag for persistent streaming mappings.

Hi David,

Quoted 12 lines

I'm currently looking into converting s390 from our custom IOMMU based DMA API implementation to using dma-iommu.c. We're always using an IOMMU for PCI devices even when doing pass-through to guests (under both the KVM and z/VM hypervisors). In this case I/O TLB flushes, which we use to do the shadowing of the guest I/O translations, are relatively expensive I'm thus very interested in your work. I've tried rebasing it on v5.18 and got it to compile but didn't get DMA to work though it seems to partially work as I don't get probe failures unlike with a completely broken DMA API. Since I might have very well screwed up the rebase and my DMA API conversion is experimental too I was wondering if you're still working on this and might have a current version I could experiment with?

Unfortunately I don't have anything more recent to share. I've come across some performance issues caused by pathological usage patterns in internal usage, but I haven't seen any correctness issues. I'm hoping that I'll be able to address the performance issues and send a rebased series within the next month or so.

Quoted 7 lines

It's definitely possible that this series has some bugs. I've tested it on a range of chromebooks and their various hardware and drivers, but that's still all relatively normal x86_64/arm64. If your hardware is more particular about its DMA, this series might be missing something.

-David

Hi David,

I finally came around to trying this again. This time I managed to get it working and figure out what was going wrong. The problem was with the call to iommu_dma_alloc_iova() in io_buffer_manager_init(). As this call happens during the IOMMU initialization dma_get_mask(dev) is used before the driver calls dma_set_mask(_and_coherent)() and is thus still the default mask of DMA_BIT_MASK(32) instead of what the device really supports. This breaks s390 because our IOMMU currently only supports apertures starting at an IOVA >= 2^32. For testing I worked around this by just passing DMA_BIT_MASK(64) instead but of course that's not a proper fix. With that in place your patches work on top of my still experimental conversion to use dma-iommu.c on s390.

I can also already confirm that this gives a similar CPU load (especially steal time) reduction on our z/VM hypervisor which does I/O translation table shadowing much like your virtio-iommu test. It also does help performance of my DMA API rework which sadly still lacks behind our current s390 DMA API implementation. I suspect that is because the lazy unmapping used by dma-iommu.c tries to do the unmapping via a timer in the background while our current approach does them all at once when wrapping around the IOVA space. The latter I suspect works better when I/O table shadowing in the hypervisor is serialized. So to summarize for s390 something like your series would be of significant interest.

Best regards,
Niklas

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu