Skip to contentSkip to conversation

Add dynamic iommu backed bounce buffers

PATCH Superseded David Stevens 4 participants 11 messages Jul 7, 2021 – Jul 9, 2021

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Add dynamic iommu backed bounce buffers

Add support for per-domain dynamic pools of iommu bounce buffers to the dma-iommu API. This allows iommu mappings to be reused while still maintaining strict iommu protection. Allocating buffers dynamically instead of using swiotlb carveouts makes per-domain pools more amenable on systems with large numbers of devices or where devices are unknown.

When enabled, all non-direct streaming mappings below a configurable size will go through bounce buffers. Note that this means drivers which don't properly use the DMA API (e.g. i915) cannot use an iommu when this feature is enabled. However, all drivers which work with swiotlb=force should work.

Bounce buffers serve as an optimization in situations where interactions with the iommu are very costly. For example, virtio-iommu operations in a guest on a linux host require a vmexit, involvement the VMM, and a VFIO syscall. For relatively small DMA operations, memcpy can be significantly faster.

As a performance comparison, on a device with an i5-10210U, I ran fio with a VFIO passthrough NVMe drive with '--direct=1 --rw=read --ioengine=libaio --iodepth=64' and block sizes 4k, 16k, 64k, and 128k. Test throughput increased by 2.8x, 4.7x, 3.6x, and 3.6x. Time spent in iommu_dma_unmap_(page|sg) per GB processed decreased by 97%, 94%, 90%, and 87%. Time spent in iommu_dma_map_(page|sg) decreased by >99%, as bounce buffers don't require syncing here in the read case. Running with multiple jobs doesn't serve as a useful performance comparison because virtio-iommu and vfio_iommu_type1 both have big locks that significantly limit mulithreaded DMA performance.

This patch set is based on v5.13-rc7 plus the patches at [1].

David Stevens (4):
dma-iommu: add kalloc gfp flag to alloc helper
dma-iommu: replace device arguments
dma-iommu: expose a few helper functions to module
dma-iommu: Add iommu bounce buffers to dma-iommu api

 drivers/iommu/Kconfig          |  10 +
 drivers/iommu/Makefile         |   1 +
 drivers/iommu/dma-iommu.c      | 119 ++++--
 drivers/iommu/io-buffer-pool.c | 656 +++++++++++++++++++++++++++++++++
 drivers/iommu/io-buffer-pool.h |  91 +++++
 include/linux/dma-iommu.h      |  12 +
 6 files changed, 861 insertions(+), 28 deletions(-)
 create mode 100644 drivers/iommu/io-buffer-pool.c
 create mode 100644 drivers/iommu/io-buffer-pool.h
Signature · 6 lines
2.32.0.93.g670b81a890-goog

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#1 lore raw

PATCH 1/4 dma-iommu: add kalloc gfp flag to alloc helper

From: David Stevens <[email protected]>

Add gfp flag for kalloc calls within __iommu_dma_alloc_pages, so the function can be called from atomic contexts.

---
 drivers/iommu/dma-iommu.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
drivers/iommu/dma-iommu.c +7 -6
index 614f0dd86b08..00993b56c977 100644
@@ -593,7 +593,8 @@ static void __iommu_dma_free_pages(struct page **pages, int count)
 }
 
 static struct page **__iommu_dma_alloc_pages(struct device *dev,
-		unsigned int count, unsigned long order_mask, gfp_t gfp)
+		unsigned int count, unsigned long order_mask,
+		gfp_t page_gfp, gfp_t kalloc_gfp)
 {
 	struct page **pages;
 	unsigned int i = 0, nid = dev_to_node(dev);
@@ -602,15 +603,15 @@ static struct page **__iommu_dma_alloc_pages(struct device *dev,
 	if (!order_mask)
 		return NULL;
 
-	pages = kvzalloc(count * sizeof(*pages), GFP_KERNEL);
+	pages = kvzalloc(count * sizeof(*pages), kalloc_gfp);
 	if (!pages)
 		return NULL;
 
 	/* IOMMU can map any pages, so himem can also be used here */
-	gfp |= __GFP_NOWARN | __GFP_HIGHMEM;
+	page_gfp |= __GFP_NOWARN | __GFP_HIGHMEM;
 
 	/* It makes no sense to muck about with huge pages */
-	gfp &= ~__GFP_COMP;
+	page_gfp &= ~__GFP_COMP;
 
 	while (count) {
 		struct page *page = NULL;
@@ -624,7 +625,7 @@ static struct page **__iommu_dma_alloc_pages(struct device *dev,
 		for (order_mask &= (2U << __fls(count)) - 1;
 		     order_mask; order_mask &= ~order_size) {
 			unsigned int order = __fls(order_mask);
-			gfp_t alloc_flags = gfp;
+			gfp_t alloc_flags = page_gfp;
 
 			order_size = 1U << order;
 			if (order_mask > order_size)
@@ -680,7 +681,7 @@ static struct page **__iommu_dma_alloc_noncontiguous(struct device *dev,
 
 	count = PAGE_ALIGN(size) >> PAGE_SHIFT;
 	pages = __iommu_dma_alloc_pages(dev, count, alloc_sizes >> PAGE_SHIFT,
-					gfp);
+					gfp, GFP_KERNEL);
 	if (!pages)
 		return NULL;
 
Signature · 1 line
2.32.0.93.g670b81a890-goog
maintainer
#2 lore raw

Re: [PATCH 1/4] dma-iommu: add kalloc gfp flag to alloc helper

Quoted 4 lines · David Stevens

From: David Stevens <[email protected]>

Add gfp flag for kalloc calls within __iommu_dma_alloc_pages, so the function can be called from atomic contexts.

Why bother? If you need GFP_ATOMIC for allocating the pages array, then you don't not need it for allocating the pages themselves. It's hardly rocket science to infer one from the other.

Robin.

Quoted 57 lines
---
  drivers/iommu/dma-iommu.c | 13 +++++++------
  1 file changed, 7 insertions(+), 6 deletions(-)
drivers/iommu/dma-iommu.c +7 -6
index 614f0dd86b08..00993b56c977 100644
@@ -593,7 +593,8 @@ static void __iommu_dma_free_pages(struct page **pages, int count)
  }
 
  static struct page **__iommu_dma_alloc_pages(struct device *dev,
-		unsigned int count, unsigned long order_mask, gfp_t gfp)
+		unsigned int count, unsigned long order_mask,
+		gfp_t page_gfp, gfp_t kalloc_gfp)
  {
  	struct page **pages;
  	unsigned int i = 0, nid = dev_to_node(dev);
@@ -602,15 +603,15 @@ static struct page **__iommu_dma_alloc_pages(struct device *dev,
  	if (!order_mask)
  		return NULL;
 
-	pages = kvzalloc(count * sizeof(*pages), GFP_KERNEL);
+	pages = kvzalloc(count * sizeof(*pages), kalloc_gfp);
  	if (!pages)
  		return NULL;
 
  	/* IOMMU can map any pages, so himem can also be used here */
-	gfp |= __GFP_NOWARN | __GFP_HIGHMEM;
+	page_gfp |= __GFP_NOWARN | __GFP_HIGHMEM;
 
  	/* It makes no sense to muck about with huge pages */
-	gfp &= ~__GFP_COMP;
+	page_gfp &= ~__GFP_COMP;
 
  	while (count) {
  		struct page *page = NULL;
@@ -624,7 +625,7 @@ static struct page **__iommu_dma_alloc_pages(struct device *dev,
  		for (order_mask &= (2U << __fls(count)) - 1;
  		     order_mask; order_mask &= ~order_size) {
  			unsigned int order = __fls(order_mask);
-			gfp_t alloc_flags = gfp;
+			gfp_t alloc_flags = page_gfp;
 
  			order_size = 1U << order;
  			if (order_mask > order_size)
@@ -680,7 +681,7 @@ static struct page **__iommu_dma_alloc_noncontiguous(struct device *dev,
 
  	count = PAGE_ALIGN(size) >> PAGE_SHIFT;
  	pages = __iommu_dma_alloc_pages(dev, count, alloc_sizes >> PAGE_SHIFT,
-					gfp);
+					gfp, GFP_KERNEL);
  	if (!pages)
  		return NULL;
 
#3 lore raw

PATCH 2/4 dma-iommu: replace device arguments

In reply to David Stevens

From: David Stevens <[email protected]>

Replace the struct device argument with the device's nid in __iommu_dma_alloc_pages, since it doesn't need the whole struct. This allows it to be called from places which don't have access to the device.

---
 drivers/iommu/dma-iommu.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/iommu/dma-iommu.c +5 -5
index 00993b56c977..98a5c566a303 100644
@@ -592,12 +592,12 @@ static void __iommu_dma_free_pages(struct page **pages, int count)
 	kvfree(pages);
 }
 
-static struct page **__iommu_dma_alloc_pages(struct device *dev,
+static struct page **__iommu_dma_alloc_pages(
 		unsigned int count, unsigned long order_mask,
-		gfp_t page_gfp, gfp_t kalloc_gfp)
+		unsigned int nid, gfp_t page_gfp, gfp_t kalloc_gfp)
 {
 	struct page **pages;
-	unsigned int i = 0, nid = dev_to_node(dev);
+	unsigned int i = 0;
 
 	order_mask &= (2U << MAX_ORDER) - 1;
 	if (!order_mask)
@@ -680,8 +680,8 @@ static struct page **__iommu_dma_alloc_noncontiguous(struct device *dev,
 		alloc_sizes = min_size;
 
 	count = PAGE_ALIGN(size) >> PAGE_SHIFT;
-	pages = __iommu_dma_alloc_pages(dev, count, alloc_sizes >> PAGE_SHIFT,
-					gfp, GFP_KERNEL);
+	pages = __iommu_dma_alloc_pages(count, alloc_sizes >> PAGE_SHIFT,
+					dev_to_node(dev), gfp, GFP_KERNEL);
 	if (!pages)
 		return NULL;
 
Signature · 6 lines
2.32.0.93.g670b81a890-goog

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#4 lore raw

PATCH 3/4 dma-iommu: expose a few helper functions to module

In reply to David Stevens

From: David Stevens <[email protected]>

Expose a few helper functions from dma-iommu to the rest of the module.

---
 drivers/iommu/dma-iommu.c | 27 ++++++++++++++-------------
 include/linux/dma-iommu.h | 12 ++++++++++++
 2 files changed, 26 insertions(+), 13 deletions(-)
drivers/iommu/dma-iommu.c +14 -13
index 98a5c566a303..48267d9f5152 100644
@@ -413,7 +413,7 @@ static int dma_info_to_prot(enum dma_data_direction dir, bool coherent,
 	}
 }
 
-static dma_addr_t iommu_dma_alloc_iova(struct iommu_domain *domain,
+dma_addr_t __iommu_dma_alloc_iova(struct iommu_domain *domain,
 		size_t size, u64 dma_limit, struct device *dev)
 {
 	struct iommu_dma_cookie *cookie = domain->iova_cookie;
@@ -453,7 +453,7 @@ static dma_addr_t iommu_dma_alloc_iova(struct iommu_domain *domain,
 	return (dma_addr_t)iova << shift;
 }
 
-static void iommu_dma_free_iova(struct iommu_dma_cookie *cookie,
+void __iommu_dma_free_iova(struct iommu_dma_cookie *cookie,
 		dma_addr_t iova, size_t size, struct page *freelist)
 {
 	struct iova_domain *iovad = &cookie->iovad;
@@ -489,7 +489,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr,
 
 	if (!cookie->fq_domain)
 		iommu_iotlb_sync(domain, &iotlb_gather);
-	iommu_dma_free_iova(cookie, dma_addr, size, iotlb_gather.freelist);
+	__iommu_dma_free_iova(cookie, dma_addr, size, iotlb_gather.freelist);
 }
 
 static void __iommu_dma_unmap_swiotlb(struct device *dev, dma_addr_t dma_addr,
@@ -525,12 +525,12 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys,
 
 	size = iova_align(iovad, size + iova_off);
 
-	iova = iommu_dma_alloc_iova(domain, size, dma_mask, dev);
+	iova = __iommu_dma_alloc_iova(domain, size, dma_mask, dev);
 	if (!iova)
 		return DMA_MAPPING_ERROR;
 
 	if (iommu_map_atomic(domain, iova, phys - iova_off, size, prot)) {
-		iommu_dma_free_iova(cookie, iova, size, NULL);
+		__iommu_dma_free_iova(cookie, iova, size, NULL);
 		return DMA_MAPPING_ERROR;
 	}
 	return iova + iova_off;
@@ -585,14 +585,14 @@ static dma_addr_t __iommu_dma_map_swiotlb(struct device *dev, phys_addr_t phys,
 	return iova;
 }
 
-static void __iommu_dma_free_pages(struct page **pages, int count)
+void __iommu_dma_free_pages(struct page **pages, int count)
 {
 	while (count--)
 		__free_page(pages[count]);
 	kvfree(pages);
 }
 
-static struct page **__iommu_dma_alloc_pages(
+struct page **__iommu_dma_alloc_pages(
 		unsigned int count, unsigned long order_mask,
 		unsigned int nid, gfp_t page_gfp, gfp_t kalloc_gfp)
 {
@@ -686,7 +686,8 @@ static struct page **__iommu_dma_alloc_noncontiguous(struct device *dev,
 		return NULL;
 
 	size = iova_align(iovad, size);
-	iova = iommu_dma_alloc_iova(domain, size, dev->coherent_dma_mask, dev);
+	iova = __iommu_dma_alloc_iova(domain, size,
+				      dev->coherent_dma_mask, dev);
 	if (!iova)
 		goto out_free_pages;
 
@@ -712,7 +713,7 @@ static struct page **__iommu_dma_alloc_noncontiguous(struct device *dev,
 out_free_sg:
 	sg_free_table(sgt);
 out_free_iova:
-	iommu_dma_free_iova(cookie, iova, size, NULL);
+	__iommu_dma_free_iova(cookie, iova, size, NULL);
 out_free_pages:
 	__iommu_dma_free_pages(pages, count);
 	return NULL;
@@ -1063,7 +1064,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg,
 		prev = s;
 	}
 
-	iova = iommu_dma_alloc_iova(domain, iova_len, dma_get_mask(dev), dev);
+	iova = __iommu_dma_alloc_iova(domain, iova_len, dma_get_mask(dev), dev);
 	if (!iova)
 		goto out_restore_sg;
 
@@ -1077,7 +1078,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg,
 	return __finalise_sg(dev, sg, nents, iova);
 
 out_free_iova:
-	iommu_dma_free_iova(cookie, iova, iova_len, NULL);
+	__iommu_dma_free_iova(cookie, iova, iova_len, NULL);
 out_restore_sg:
 	__invalidate_sg(sg, nents);
 	return 0;
@@ -1370,7 +1371,7 @@ static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev,
 	if (!msi_page)
 		return NULL;
 
-	iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev);
+	iova = __iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev);
 	if (!iova)
 		goto out_free_page;
 
@@ -1384,7 +1385,7 @@ static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev,
 	return msi_page;
 
 out_free_iova:
-	iommu_dma_free_iova(cookie, iova, size, NULL);
+	__iommu_dma_free_iova(cookie, iova, size, NULL);
 out_free_page:
 	kfree(msi_page);
 	return NULL;
include/linux/dma-iommu.h +12 -0
index 6e75a2d689b4..fc9acc581db0 100644
@@ -42,6 +42,18 @@ void iommu_dma_free_cpu_cached_iovas(unsigned int cpu,
 
 extern bool iommu_dma_forcedac;
 
+struct iommu_dma_cookie;
+
+struct page **__iommu_dma_alloc_pages(
+		unsigned int count, unsigned long order_mask,
+		unsigned int nid, gfp_t page_gfp, gfp_t kalloc_gfp);
+void __iommu_dma_free_pages(struct page **pages, int count);
+dma_addr_t __iommu_dma_alloc_iova(struct iommu_domain *domain,
+				  size_t size, dma_addr_t dma_limit,
+				  struct device *dev);
+void __iommu_dma_free_iova(struct iommu_dma_cookie *cookie,
+		dma_addr_t iova, size_t size, struct page *freelist);
+
 #else /* CONFIG_IOMMU_DMA */
 
 struct iommu_domain;
Signature · 6 lines
2.32.0.93.g670b81a890-goog

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#5 lore raw

PATCH 4/4 dma-iommu: Add iommu bounce buffers to dma-iommu api

In reply to David Stevens

From: David Stevens <[email protected]>

Add support for per-domain dynamic pools of iommu bounce buffers to the dma-iommu api. When enabled, all non-direct streaming mappings below a configurable size will go through bounce buffers.

Each domain has its own buffer pool. Each buffer pool is split into multiple power-of-2 size classes. Each class has a number of preallocated slots that can hold bounce buffers. Bounce buffers are allocated on demand, and unmapped bounce buffers are stored in a cache with periodic eviction of unused cache entries. As the buffer pool is an optimization, any failures simply result in falling back to the normal dma-iommu handling.

---
 drivers/iommu/Kconfig          |  10 +
 drivers/iommu/Makefile         |   1 +
 drivers/iommu/dma-iommu.c      |  75 +++-
 drivers/iommu/io-buffer-pool.c | 656 +++++++++++++++++++++++++++++++++
 drivers/iommu/io-buffer-pool.h |  91 +++++
 5 files changed, 826 insertions(+), 7 deletions(-)
 create mode 100644 drivers/iommu/io-buffer-pool.c
 create mode 100644 drivers/iommu/io-buffer-pool.h
drivers/iommu/Kconfig +10 -0
index 1f111b399bca..6eee57b03ff9 100644
@@ -420,4 +420,14 @@ config SPRD_IOMMU
 
 	  Say Y here if you want to use the multimedia devices listed above.
 
+config IOMMU_IO_BUFFER
+	bool "Use IOMMU bounce buffers"
+	depends on IOMMU_DMA
+	help
+	  Use bounce buffers for small, streaming DMA operations. This may
+	  have performance benefits on systems where establishing IOMMU mappings
+	  is particularly expensive, such as when running as a guest.
+
+	  If unsure, say N here.
+
 endif # IOMMU_SUPPORT
drivers/iommu/Makefile +1 -0
index c0fb0ba88143..2287b2e3d92d 100644
@@ -5,6 +5,7 @@ obj-$(CONFIG_IOMMU_API) += iommu-traces.o
 obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
 obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
 obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
+obj-$(CONFIG_IOMMU_IO_BUFFER) += io-buffer-pool.o
 obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
 obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o
 obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) += io-pgtable-arm.o
drivers/iommu/dma-iommu.c +68 -7

169 lines not on this page · raw message

drivers/iommu/io-buffer-pool.c new file +656 -0

656 lines not on this page · raw message

drivers/iommu/io-buffer-pool.h new file +91 -0

91 lines not on this page · raw message

Signature · 1 line
2.32.0.93.g670b81a890-goog
maintainer
#6 lore raw
In reply to David Stevens

Adding Robin too.

Quoted 49 lines · David Stevens

Add support for per-domain dynamic pools of iommu bounce buffers to the dma-iommu API. This allows iommu mappings to be reused while still maintaining strict iommu protection. Allocating buffers dynamically instead of using swiotlb carveouts makes per-domain pools more amenable on systems with large numbers of devices or where devices are unknown.

When enabled, all non-direct streaming mappings below a configurable size will go through bounce buffers. Note that this means drivers which don't properly use the DMA API (e.g. i915) cannot use an iommu when this feature is enabled. However, all drivers which work with swiotlb=force should work.

Bounce buffers serve as an optimization in situations where interactions with the iommu are very costly. For example, virtio-iommu operations in a guest on a linux host require a vmexit, involvement the VMM, and a VFIO syscall. For relatively small DMA operations, memcpy can be significantly faster.

As a performance comparison, on a device with an i5-10210U, I ran fio with a VFIO passthrough NVMe drive with '--direct=1 --rw=read --ioengine=libaio --iodepth=64' and block sizes 4k, 16k, 64k, and 128k. Test throughput increased by 2.8x, 4.7x, 3.6x, and 3.6x. Time spent in iommu_dma_unmap_(page|sg) per GB processed decreased by 97%, 94%, 90%, and 87%. Time spent in iommu_dma_map_(page|sg) decreased by >99%, as bounce buffers don't require syncing here in the read case. Running with multiple jobs doesn't serve as a useful performance comparison because virtio-iommu and vfio_iommu_type1 both have big locks that significantly limit mulithreaded DMA performance.

This patch set is based on v5.13-rc7 plus the patches at [1].

David Stevens (4):
dma-iommu: add kalloc gfp flag to alloc helper
dma-iommu: replace device arguments
dma-iommu: expose a few helper functions to module
dma-iommu: Add iommu bounce buffers to dma-iommu api

 drivers/iommu/Kconfig          |  10 +
 drivers/iommu/Makefile         |   1 +
 drivers/iommu/dma-iommu.c      | 119 ++++--
 drivers/iommu/io-buffer-pool.c | 656 +++++++++++++++++++++++++++++++++
 drivers/iommu/io-buffer-pool.h |  91 +++++
 include/linux/dma-iommu.h      |  12 +
 6 files changed, 861 insertions(+), 28 deletions(-)
 create mode 100644 drivers/iommu/io-buffer-pool.c
 create mode 100644 drivers/iommu/io-buffer-pool.h
Signature · 1 line
2.32.0.93.g670b81a890-goog
maintainer
#7 lore raw
Quoted 8 lines · Joerg Roedel

Adding Robin too.

Quoted 5 lines · David Stevens

Add support for per-domain dynamic pools of iommu bounce buffers to the dma-iommu API. This allows iommu mappings to be reused while still maintaining strict iommu protection. Allocating buffers dynamically instead of using swiotlb carveouts makes per-domain pools more amenable on systems with large numbers of devices or where devices are unknown.

But isn't that just as true for the currently-supported case? All you need is a large enough Thunderbolt enclosure and you could suddenly plug in a dozen untrusted GPUs all wanting to map hundreds of megabytes of memory. If there's a real concern worth addressing, surely it's worth addressing properly for everyone.

Quoted 11 lines
Quoted 11 lines

When enabled, all non-direct streaming mappings below a configurable size will go through bounce buffers. Note that this means drivers which don't properly use the DMA API (e.g. i915) cannot use an iommu when this feature is enabled. However, all drivers which work with swiotlb=force should work.

Bounce buffers serve as an optimization in situations where interactions with the iommu are very costly. For example, virtio-iommu operations in a guest on a linux host require a vmexit, involvement the VMM, and a VFIO syscall. For relatively small DMA operations, memcpy can be significantly faster.

Yup, back when the bounce-buffering stuff first came up I know networking folks were interested in terms of latency for small packets - virtualised IOMMUs are indeed another interesting case I hadn't thought of. It's definitely been on the radar as another use-case we'd like to accommodate with the bounce-buffering scheme. However, that's the thing: bouncing is bouncing and however you look at it it still overlaps so much with the untrusted case - there's no reason that couldn't use pre-mapped bounce buffers too, for instance - that the only necessary difference is really the policy decision of when to bounce. iommu-dma has already grown complicated enough, and having *three* different ways of doing things internally just seems bonkers and untenable. Pre-map the bounce buffers? Absolutely. Dynamically grow them on demand? Yes please! Do it all as a special thing in its own NIH module and leave the existing mess to rot? Sorry, but no.

Thanks,
Robin.

Quoted 31 lines
Quoted 31 lines

As a performance comparison, on a device with an i5-10210U, I ran fio with a VFIO passthrough NVMe drive with '--direct=1 --rw=read --ioengine=libaio --iodepth=64' and block sizes 4k, 16k, 64k, and 128k. Test throughput increased by 2.8x, 4.7x, 3.6x, and 3.6x. Time spent in iommu_dma_unmap_(page|sg) per GB processed decreased by 97%, 94%, 90%, and 87%. Time spent in iommu_dma_map_(page|sg) decreased by >99%, as bounce buffers don't require syncing here in the read case. Running with multiple jobs doesn't serve as a useful performance comparison because virtio-iommu and vfio_iommu_type1 both have big locks that significantly limit mulithreaded DMA performance.

This patch set is based on v5.13-rc7 plus the patches at [1].

David Stevens (4):
dma-iommu: add kalloc gfp flag to alloc helper
dma-iommu: replace device arguments
dma-iommu: expose a few helper functions to module
dma-iommu: Add iommu bounce buffers to dma-iommu api

  drivers/iommu/Kconfig          |  10 +
  drivers/iommu/Makefile         |   1 +
  drivers/iommu/dma-iommu.c      | 119 ++++--
  drivers/iommu/io-buffer-pool.c | 656 +++++++++++++++++++++++++++++++++
  drivers/iommu/io-buffer-pool.h |  91 +++++
  include/linux/dma-iommu.h      |  12 +
  6 files changed, 861 insertions(+), 28 deletions(-)

create mode 100644 drivers/iommu/io-buffer-pool.c create mode 100644 drivers/iommu/io-buffer-pool.h

Signature · 1 line
2.32.0.93.g670b81a890-goog
#8 lore raw
Quoted 11 lines · Robin Murphy
Quoted 8 lines · Joerg Roedel

Adding Robin too.

Quoted 5 lines · David Stevens

Add support for per-domain dynamic pools of iommu bounce buffers to the dma-iommu API. This allows iommu mappings to be reused while still maintaining strict iommu protection. Allocating buffers dynamically instead of using swiotlb carveouts makes per-domain pools more amenable on systems with large numbers of devices or where devices are unknown.

Quoted 5 lines

But isn't that just as true for the currently-supported case? All you need is a large enough Thunderbolt enclosure and you could suddenly plug in a dozen untrusted GPUs all wanting to map hundreds of megabytes of memory. If there's a real concern worth addressing, surely it's worth addressing properly for everyone.

Bounce buffers consume memory, so there is always going to be some limitation on how many devices are supported. This patch series limits the memory consumption at a given point in time to approximately the amount of active DMA transactions. There's really no way to improve significantly on that. The 'approximately' qualification could be removed by adding a shrinker, but that doesn't change things materially.

This is compared to reusing swiotlb, where the amount of memory consumed would be the largest amount of active DMA transactions you want bounce buffers to handle. I see two concrete shortcomings here. First, most of the time you're not doing heavy IO, especially for consumer workloads. Second, it raises the problem of per-device tuning, since you don't want to waste performance by having too few bounce buffers but you also don't want to waste memory by preallocating too many bounce buffers. This tuning becomes more problematic once you start dealing with external devices.

Also, although this doesn't directly address the raised concern, the bounce buffers are only used for relatively small DMA transactions. So large allocations like framebuffers won't actually consume extra memory via bounce buffers.

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

When enabled, all non-direct streaming mappings below a configurable size will go through bounce buffers. Note that this means drivers which don't properly use the DMA API (e.g. i915) cannot use an iommu when this feature is enabled. However, all drivers which work with swiotlb=force should work.

Bounce buffers serve as an optimization in situations where interactions with the iommu are very costly. For example, virtio-iommu operations in a guest on a linux host require a vmexit, involvement the VMM, and a VFIO syscall. For relatively small DMA operations, memcpy can be significantly faster.

Quoted 14 lines

Yup, back when the bounce-buffering stuff first came up I know networking folks were interested in terms of latency for small packets - virtualised IOMMUs are indeed another interesting case I hadn't thought of. It's definitely been on the radar as another use-case we'd like to accommodate with the bounce-buffering scheme. However, that's the thing:
bouncing is bouncing and however you look at it it still overlaps so much with the untrusted case - there's no reason that couldn't use pre-mapped bounce buffers too, for instance - that the only necessary difference is really the policy decision of when to bounce. iommu-dma has already grown complicated enough, and having *three* different ways of doing things internally just seems bonkers and untenable. Pre-map the bounce buffers? Absolutely. Dynamically grow them on demand? Yes please! Do it all as a special thing in its own NIH module and leave the existing mess to rot? Sorry, but no.

I do agree that iommu-dma is getting fairly complicated. Since a virtualized IOMMU uses bounce buffers much more heavily than sub-granule untrusted DMA, and for the reasons stated earlier in this email, I don't think pre-allocated bounce buffers are viable for the virtualized IOMMU case. I can look at migrating the sub-granule untrusted DMA case to dynamic bounce buffers, if that's an acceptable approach.

-David

Quoted 34 lines

Thanks,
Robin.

Quoted 31 lines
Quoted 31 lines

As a performance comparison, on a device with an i5-10210U, I ran fio with a VFIO passthrough NVMe drive with '--direct=1 --rw=read --ioengine=libaio --iodepth=64' and block sizes 4k, 16k, 64k, and 128k. Test throughput increased by 2.8x, 4.7x, 3.6x, and 3.6x. Time spent in iommu_dma_unmap_(page|sg) per GB processed decreased by 97%, 94%, 90%, and 87%. Time spent in iommu_dma_map_(page|sg) decreased by >99%, as bounce buffers don't require syncing here in the read case. Running with multiple jobs doesn't serve as a useful performance comparison because virtio-iommu and vfio_iommu_type1 both have big locks that significantly limit mulithreaded DMA performance.

This patch set is based on v5.13-rc7 plus the patches at [1].

David Stevens (4):
dma-iommu: add kalloc gfp flag to alloc helper
dma-iommu: replace device arguments
dma-iommu: expose a few helper functions to module
dma-iommu: Add iommu bounce buffers to dma-iommu api

  drivers/iommu/Kconfig          |  10 +
  drivers/iommu/Makefile         |   1 +
  drivers/iommu/dma-iommu.c      | 119 ++++--
  drivers/iommu/io-buffer-pool.c | 656 +++++++++++++++++++++++++++++++++
  drivers/iommu/io-buffer-pool.h |  91 +++++
  include/linux/dma-iommu.h      |  12 +
  6 files changed, 861 insertions(+), 28 deletions(-)

create mode 100644 drivers/iommu/io-buffer-pool.c create mode 100644 drivers/iommu/io-buffer-pool.h

--
2.32.0.93.g670b81a890-goog

#9 lore raw
In reply to David Stevens

Hi David,

I like this idea. Thanks for proposing this.

Quoted 5 lines · David Stevens

Add support for per-domain dynamic pools of iommu bounce buffers to the dma-iommu API. This allows iommu mappings to be reused while still maintaining strict iommu protection. Allocating buffers dynamically instead of using swiotlb carveouts makes per-domain pools more amenable on systems with large numbers of devices or where devices are unknown.

Have you ever considered leveraging the per-device swiotlb memory pool added by below series?

https://lore.kernel.org/linux-iommu/20210625123004.GA3170@willie-the-truck/

Quoted 6 lines

When enabled, all non-direct streaming mappings below a configurable size will go through bounce buffers. Note that this means drivers which don't properly use the DMA API (e.g. i915) cannot use an iommu when this feature is enabled. However, all drivers which work with swiotlb=force should work.

If so, why not making it more scalable by adding a callback into vendor iommu drivers? The vendor iommu drivers have enough information to tell whether the bounce buffer is feasible for a specific domain.

Quoted 3 lines

Bounce buffers serve as an optimization in situations where interactions with the iommu are very costly. For example, virtio-iommu operations in

The simulated IOMMU does the same thing.

It's also an optimization for bare metal in cases where the strict mode of cache invalidation is used. CPU moving data is faster than IOMMU cache invalidation if the buffer is small.

Best regards,
baolu

#10 lore raw
Quoted 12 lines · Lu Baolu

Hi David,

I like this idea. Thanks for proposing this.

Quoted 5 lines · David Stevens

Add support for per-domain dynamic pools of iommu bounce buffers to the dma-iommu API. This allows iommu mappings to be reused while still maintaining strict iommu protection. Allocating buffers dynamically instead of using swiotlb carveouts makes per-domain pools more amenable on systems with large numbers of devices or where devices are unknown.

Quoted 4 lines

Have you ever considered leveraging the per-device swiotlb memory pool added by below series?

https://lore.kernel.org/linux-iommu/20210625123004.GA3170@willie-the-truck/

I'm not sure if that's a good fit. The swiotlb pools are allocated during device initialization, so they require setting aside the worst-case amount of memory. That's okay if you only use it with a small number of devices where you know in advance approximately how much memory they use. However, it doesn't work as well if you want to use it with a large number of devices, or with unknown (i.e. hotplugged) devices.

Quoted 10 lines
Quoted 6 lines

When enabled, all non-direct streaming mappings below a configurable size will go through bounce buffers. Note that this means drivers which don't properly use the DMA API (e.g. i915) cannot use an iommu when this feature is enabled. However, all drivers which work with swiotlb=force should work.

If so, why not making it more scalable by adding a callback into vendor iommu drivers? The vendor iommu drivers have enough information to tell whether the bounce buffer is feasible for a specific domain.

I'm not very familiar with the specifics of VT-d or restrictions with the graphics hardware, but at least on the surface it looks like a limitation of the i915 driver's implementation. The driver uses the DMA_ATTR_SKIP_CPU_SYNC flag, but never calls the dma_sync functions, since things are coherent on x86 hardware. However, bounce buffers violate the driver's assumption that there's no need to sync the CPU and device domain. I doubt there's an inherent limitation of the hardware here, it's just how the driver is implemented. Given that, I don't know if it's something the iommu driver needs to handle.

One potential way this could be addressed would be to add explicit support to the DMA API for long-lived streaming mappings. Drivers can get that behavior today via DMA_ATTR_SKIP_CPU_SYNC and dma_sync. However, the DMA API doesn't really have enough information to treat ephemeral and long-lived mappings differently. With a new DMA_ATTR flag for long-lived streaming mappings, the DMA API could skip bounce buffers. That flag could also be used as a performance optimization in the various dma-buf implementations, since they seem to mostly fall into the long-lived streaming category (the handful I checked do call dma_sync, so there isn't a correctness issue).

-David

Quoted 12 lines
Quoted 3 lines

Bounce buffers serve as an optimization in situations where interactions with the iommu are very costly. For example, virtio-iommu operations in

The simulated IOMMU does the same thing.

It's also an optimization for bare metal in cases where the strict mode of cache invalidation is used. CPU moving data is faster than IOMMU cache invalidation if the buffer is small.

Best regards,
baolu