Skip to contentSkip to conversation

move swiotlb noncoherent dma support from arm64 to generic code

PATCH New Christoph Hellwig 2 participants 12 messages Sep 17, 2018 – Sep 18, 2018

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi all,

this series starts with various swiotlb cleanups, then adds support for non-cache coherent devices to the generic swiotlb support, and finally switches arm64 to use the generic code.

Given that this series depends on patches in the dma-mapping tree, or pending for it I've also published a git tree here:

git://git.infradead.org/users/hch/misc.git swiotlb-noncoherent

Gitweb:

http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/swiotlb-noncoherent

#1 lore raw

PATCH 1/9 swiotlb: remove a pointless comment

This comments describes an aspect of the map_sg interface that isn't even exploited by swiotlb.

---
 kernel/dma/swiotlb.c | 6 ------
 1 file changed, 6 deletions(-)
kernel/dma/swiotlb.c +0 -6
index 4f8a6dbf0b60..9062b14bc7f4 100644
@@ -925,12 +925,6 @@ swiotlb_sync_single_for_device(struct device *hwdev, dma_addr_t dev_addr,
  * appropriate dma address and length.  They are obtained via
  * sg_dma_{address,length}(SG).
  *
- * NOTE: An implementation may be able to use a smaller number of
- *       DMA address/length pairs than there are SG table elements.
- *       (for example via virtual mapping capabilities)
- *       The routine returns the number of addr/length pairs actually
- *       used, at most nents.
- *
  * Device ownership issues as mentioned above for swiotlb_map_page are the
  * same here.
  */
Signature · 1 line
2.18.0
#2 lore raw

PATCH 2/9 swiotlb: mark is_swiotlb_buffer static

In reply to Christoph Hellwig
---
 include/linux/swiotlb.h | 1 -
 kernel/dma/swiotlb.c    | 2 +-
 2 files changed, 1 insertion(+), 2 deletions(-)
include/linux/swiotlb.h +0 -1
index 965be92c33b5..7ef541ce8f34 100644
@@ -121,7 +121,6 @@ static inline unsigned int swiotlb_max_segment(void) { return 0; }
 #endif
 
 extern void swiotlb_print_info(void);
-extern int is_swiotlb_buffer(phys_addr_t paddr);
 extern void swiotlb_set_max_segment(unsigned int);
 
 extern const struct dma_map_ops swiotlb_dma_ops;
kernel/dma/swiotlb.c +1 -1
index 9062b14bc7f4..26d3af52956f 100644
@@ -429,7 +429,7 @@ void __init swiotlb_exit(void)
 	max_segment = 0;
 }
 
-int is_swiotlb_buffer(phys_addr_t paddr)
+static int is_swiotlb_buffer(phys_addr_t paddr)
 {
 	return paddr >= io_tlb_start && paddr < io_tlb_end;
 }
Signature · 1 line
2.18.0
#3 lore raw

PATCH 3/9 swiotlb: do not panic on mapping failures

In reply to Christoph Hellwig

All properly written drivers now have error handling in the dma_map_single / dma_map_page callers. As swiotlb_tbl_map_single already prints a useful warning when running out of swiotlb pool swace we can also remove swiotlb_full entirely as it serves no purpose now.

---
 kernel/dma/swiotlb.c | 33 +--------------------------------
 1 file changed, 1 insertion(+), 32 deletions(-)
kernel/dma/swiotlb.c +1 -32
index 26d3af52956f..69bf305ee5f8 100644
@@ -761,34 +761,6 @@ static bool swiotlb_free_buffer(struct device *dev, size_t size,
 	return true;
 }
 
-static void
-swiotlb_full(struct device *dev, size_t size, enum dma_data_direction dir,
-	     int do_panic)
-{
-	if (swiotlb_force == SWIOTLB_NO_FORCE)
-		return;
-
-	/*
-	 * Ran out of IOMMU space for this operation. This is very bad.
-	 * Unfortunately the drivers cannot handle this operation properly.
-	 * unless they check for dma_mapping_error (most don't)
-	 * When the mapping is small enough return a static buffer to limit
-	 * the damage, or panic when the transfer is too big.
-	 */
-	dev_err_ratelimited(dev, "DMA: Out of SW-IOMMU space for %zu bytes\n",
-			    size);
-
-	if (size <= io_tlb_overflow || !do_panic)
-		return;
-
-	if (dir == DMA_BIDIRECTIONAL)
-		panic("DMA: Random memory could be DMA accessed\n");
-	if (dir == DMA_FROM_DEVICE)
-		panic("DMA: Random memory could be DMA written\n");
-	if (dir == DMA_TO_DEVICE)
-		panic("DMA: Random memory could be DMA read\n");
-}
-
 /*
  * Map a single buffer of the indicated size for DMA in streaming mode.  The
  * physical address to use is returned.
@@ -817,10 +789,8 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
 
 	/* Oh well, have to allocate and map a bounce buffer. */
 	map = map_single(dev, phys, size, dir, attrs);
-	if (map == SWIOTLB_MAP_ERROR) {
-		swiotlb_full(dev, size, dir, 1);
+	if (map == SWIOTLB_MAP_ERROR)
 		return __phys_to_dma(dev, io_tlb_overflow_buffer);
-	}
 
 	dev_addr = __phys_to_dma(dev, map);
 
@@ -948,7 +918,6 @@ swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl, int nelems,
 			if (map == SWIOTLB_MAP_ERROR) {
 				/* Don't panic here, we expect map_sg users
 				   to do proper error handling. */
-				swiotlb_full(hwdev, sg->length, dir, 0);
 				attrs |= DMA_ATTR_SKIP_CPU_SYNC;
 				swiotlb_unmap_sg_attrs(hwdev, sgl, i, dir,
 						       attrs);
Signature · 1 line
2.18.0
#4 lore raw

PATCH 4/9 swiotlb: remove the overflow buffer

In reply to Christoph Hellwig

Like all other dma mapping drivers just return an error code instead of an actual memory buffer. The reason for the overflow buffer was that at the time swiotlb was invented there was no way to check for dma mapping errors, but this has long been fixed.

---
 arch/arm64/mm/dma-mapping.c       |  2 +-
 arch/powerpc/kernel/dma-swiotlb.c |  4 +--
 include/linux/dma-direct.h        |  2 ++
 include/linux/swiotlb.h           |  3 --
 kernel/dma/direct.c               |  2 --
 kernel/dma/swiotlb.c              | 59 ++-----------------------------
 6 files changed, 8 insertions(+), 64 deletions(-)
arch/arm64/mm/dma-mapping.c +1 -1
index cdcb73db9ea2..abcae73eea50 100644
@@ -324,7 +324,7 @@ static int __swiotlb_dma_supported(struct device *hwdev, u64 mask)
 static int __swiotlb_dma_mapping_error(struct device *hwdev, dma_addr_t addr)
 {
 	if (swiotlb)
-		return swiotlb_dma_mapping_error(hwdev, addr);
+		return dma_direct_mapping_error(hwdev, addr);
 	return 0;
 }
 
arch/powerpc/kernel/dma-swiotlb.c +2 -2
index 88f3963ca30f..5fc335f4d9cd 100644
@@ -11,7 +11,7 @@
  *
  */
 
-#include <linux/dma-mapping.h>
+#include <linux/dma-direct.h>
 #include <linux/memblock.h>
 #include <linux/pfn.h>
 #include <linux/of_platform.h>
@@ -59,7 +59,7 @@ const struct dma_map_ops powerpc_swiotlb_dma_ops = {
 	.sync_single_for_device = swiotlb_sync_single_for_device,
 	.sync_sg_for_cpu = swiotlb_sync_sg_for_cpu,
 	.sync_sg_for_device = swiotlb_sync_sg_for_device,
-	.mapping_error = swiotlb_dma_mapping_error,
+	.mapping_error = dma_direct_mapping_error,
 	.get_required_mask = swiotlb_powerpc_get_required,
 };
 
include/linux/dma-direct.h +2 -0
index 86a59ba5a7f3..b7984fc78974 100644
@@ -5,6 +5,8 @@
 #include <linux/dma-mapping.h>
 #include <linux/mem_encrypt.h>
 
+#define DIRECT_MAPPING_ERROR		0
+
 #ifdef CONFIG_ARCH_HAS_PHYS_TO_DMA
 #include <asm/dma-direct.h>
 #else
include/linux/swiotlb.h +0 -3
index 7ef541ce8f34..f847c1b265c4 100644
@@ -106,9 +106,6 @@ extern void
 swiotlb_sync_sg_for_device(struct device *hwdev, struct scatterlist *sg,
 			   int nelems, enum dma_data_direction dir);
 
-extern int
-swiotlb_dma_mapping_error(struct device *hwdev, dma_addr_t dma_addr);
-
 extern int
 swiotlb_dma_supported(struct device *hwdev, u64 mask);
 
kernel/dma/direct.c +0 -2
index c954f0a6dc62..c22fa0f7c20d 100644
@@ -13,8 +13,6 @@
 #include <linux/pfn.h>
 #include <linux/set_memory.h>
 
-#define DIRECT_MAPPING_ERROR		0
-
 /*
  * Most architectures use ZONE_DMA for the first 16 Megabytes, but
  * some use it for entirely different regions:
kernel/dma/swiotlb.c +3 -56

141 lines not on this page · raw message

Signature · 1 line
2.18.0
#5 lore raw

PATCH 5/9 swiotlb: merge swiotlb_unmap_page and unmap_single

In reply to Christoph Hellwig
---
 kernel/dma/swiotlb.c | 15 ++++-----------
 1 file changed, 4 insertions(+), 11 deletions(-)
kernel/dma/swiotlb.c +4 -11
index 11dbcd80b4a6..15335f3a1bf3 100644
@@ -765,9 +765,9 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
  * After this call, reads by the cpu to the buffer are guaranteed to see
  * whatever the device wrote there.
  */
-static void unmap_single(struct device *hwdev, dma_addr_t dev_addr,
-			 size_t size, enum dma_data_direction dir,
-			 unsigned long attrs)
+void swiotlb_unmap_page(struct device *hwdev, dma_addr_t dev_addr,
+			size_t size, enum dma_data_direction dir,
+			unsigned long attrs)
 {
 	phys_addr_t paddr = dma_to_phys(hwdev, dev_addr);
 
@@ -790,13 +790,6 @@ static void unmap_single(struct device *hwdev, dma_addr_t dev_addr,
 	dma_mark_clean(phys_to_virt(paddr), size);
 }
 
-void swiotlb_unmap_page(struct device *hwdev, dma_addr_t dev_addr,
-			size_t size, enum dma_data_direction dir,
-			unsigned long attrs)
-{
-	unmap_single(hwdev, dev_addr, size, dir, attrs);
-}
-
 /*
  * Make physical memory consistent for a single streaming mode DMA translation
  * after a transfer.
@@ -900,7 +893,7 @@ swiotlb_unmap_sg_attrs(struct device *hwdev, struct scatterlist *sgl,
 	BUG_ON(dir == DMA_NONE);
 
 	for_each_sg(sgl, sg, nelems, i)
-		unmap_single(hwdev, sg->dma_address, sg_dma_len(sg), dir,
+		swiotlb_unmap_page(hwdev, sg->dma_address, sg_dma_len(sg), dir,
 			     attrs);
 }
 
Signature · 1 line
2.18.0
#6 lore raw

PATCH 6/9 swiotlb: use swiotlb_map_page in swiotlb_map_sg_attrs

In reply to Christoph Hellwig

No need to duplicate the code - map_sg is equivalent to map_page for each page in the scatterlist.

---
 kernel/dma/swiotlb.c | 34 ++++++++++++----------------------
 1 file changed, 12 insertions(+), 22 deletions(-)
kernel/dma/swiotlb.c +12 -22
index 15335f3a1bf3..15755d7a5242 100644
@@ -845,37 +845,27 @@ swiotlb_sync_single_for_device(struct device *hwdev, dma_addr_t dev_addr,
  * same here.
  */
 int
-swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl, int nelems,
+swiotlb_map_sg_attrs(struct device *dev, struct scatterlist *sgl, int nelems,
 		     enum dma_data_direction dir, unsigned long attrs)
 {
 	struct scatterlist *sg;
 	int i;
 
-	BUG_ON(dir == DMA_NONE);
-
 	for_each_sg(sgl, sg, nelems, i) {
-		phys_addr_t paddr = sg_phys(sg);
-		dma_addr_t dev_addr = phys_to_dma(hwdev, paddr);
-
-		if (swiotlb_force == SWIOTLB_FORCE ||
-		    !dma_capable(hwdev, dev_addr, sg->length)) {
-			phys_addr_t map = map_single(hwdev, sg_phys(sg),
-						     sg->length, dir, attrs);
-			if (map == SWIOTLB_MAP_ERROR) {
-				/* Don't panic here, we expect map_sg users
-				   to do proper error handling. */
-				attrs |= DMA_ATTR_SKIP_CPU_SYNC;
-				swiotlb_unmap_sg_attrs(hwdev, sgl, i, dir,
-						       attrs);
-				sg_dma_len(sgl) = 0;
-				return 0;
-			}
-			sg->dma_address = __phys_to_dma(hwdev, map);
-		} else
-			sg->dma_address = dev_addr;
+		sg->dma_address = swiotlb_map_page(dev, sg_page(sg), sg->offset,
+				sg->length, dir, attrs);
+		if (sg->dma_address == DIRECT_MAPPING_ERROR)
+			goto out_error;
 		sg_dma_len(sg) = sg->length;
 	}
+
 	return nelems;
+
+out_error:
+	swiotlb_unmap_sg_attrs(dev, sgl, i, dir,
+			attrs | DMA_ATTR_SKIP_CPU_SYNC);
+	sg_dma_len(sgl) = 0;
+	return 0;
 }
 
 /*
Signature · 1 line
2.18.0
#7 lore raw

PATCH 7/9 swiotlb: refactor swiotlb_map_page

In reply to Christoph Hellwig

Remove the somewhat useless map_single function, and replace it with a swiotlb_bounce_page handler that handles everything related to actually bouncing a page.

---
 kernel/dma/swiotlb.c | 77 +++++++++++++++++++++-----------------------
 1 file changed, 36 insertions(+), 41 deletions(-)
kernel/dma/swiotlb.c +36 -41
index 15755d7a5242..4d7a4d85d71e 100644
@@ -543,26 +543,6 @@ phys_addr_t swiotlb_tbl_map_single(struct device *hwdev,
 	return tlb_addr;
 }
 
-/*
- * Allocates bounce buffer and returns its physical address.
- */
-static phys_addr_t
-map_single(struct device *hwdev, phys_addr_t phys, size_t size,
-	   enum dma_data_direction dir, unsigned long attrs)
-{
-	dma_addr_t start_dma_addr;
-
-	if (swiotlb_force == SWIOTLB_NO_FORCE) {
-		dev_warn_ratelimited(hwdev, "Cannot do DMA to address %pa\n",
-				     &phys);
-		return SWIOTLB_MAP_ERROR;
-	}
-
-	start_dma_addr = __phys_to_dma(hwdev, io_tlb_start);
-	return swiotlb_tbl_map_single(hwdev, start_dma_addr, phys, size,
-				      dir, attrs);
-}
-
 /*
  * tlb_addr is the physical address of the bounce buffer to unmap.
  */
@@ -714,6 +694,34 @@ static bool swiotlb_free_buffer(struct device *dev, size_t size,
 	return true;
 }
 
+static dma_addr_t swiotlb_bounce_page(struct device *dev, phys_addr_t *phys,
+		size_t size, enum dma_data_direction dir, unsigned long attrs)
+{
+	dma_addr_t dma_addr;
+
+	if (unlikely(swiotlb_force == SWIOTLB_NO_FORCE)) {
+		dev_warn_ratelimited(dev,
+			"Cannot do DMA to address %pa\n", phys);
+		return DIRECT_MAPPING_ERROR;
+	}
+
+	/* Oh well, have to allocate and map a bounce buffer. */
+	*phys = swiotlb_tbl_map_single(dev, __phys_to_dma(dev, io_tlb_start),
+			*phys, size, dir, attrs);
+	if (*phys == SWIOTLB_MAP_ERROR)
+		return DIRECT_MAPPING_ERROR;
+
+	/* Ensure that the address returned is DMA'ble */
+	dma_addr = __phys_to_dma(dev, *phys);
+	if (unlikely(!dma_capable(dev, dma_addr, size))) {
+		swiotlb_tbl_unmap_single(dev, *phys, size, dir,
+			attrs | DMA_ATTR_SKIP_CPU_SYNC);
+		return DIRECT_MAPPING_ERROR;
+	}
+
+	return dma_addr;
+}
+
 /*
  * Map a single buffer of the indicated size for DMA in streaming mode.  The
  * physical address to use is returned.
@@ -726,8 +734,8 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
 			    enum dma_data_direction dir,
 			    unsigned long attrs)
 {
-	phys_addr_t map, phys = page_to_phys(page) + offset;
-	dma_addr_t dev_addr = phys_to_dma(dev, phys);
+	phys_addr_t phys = page_to_phys(page) + offset;
+	dma_addr_t dma_addr = phys_to_dma(dev, phys);
 
 	BUG_ON(dir == DMA_NONE);
 	/*
@@ -735,26 +743,13 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
 	 * we can safely return the device addr and not worry about bounce
 	 * buffering it.
 	 */
-	if (dma_capable(dev, dev_addr, size) && swiotlb_force != SWIOTLB_FORCE)
-		return dev_addr;
-
-	trace_swiotlb_bounced(dev, dev_addr, size, swiotlb_force);
-
-	/* Oh well, have to allocate and map a bounce buffer. */
-	map = map_single(dev, phys, size, dir, attrs);
-	if (map == SWIOTLB_MAP_ERROR)
-		return DIRECT_MAPPING_ERROR;
-
-	dev_addr = __phys_to_dma(dev, map);
-
-	/* Ensure that the address returned is DMA'ble */
-	if (dma_capable(dev, dev_addr, size))
-		return dev_addr;
-
-	attrs |= DMA_ATTR_SKIP_CPU_SYNC;
-	swiotlb_tbl_unmap_single(dev, map, size, dir, attrs);
+	if (!dma_capable(dev, dma_addr, size) ||
+	    swiotlb_force == SWIOTLB_FORCE) {
+		trace_swiotlb_bounced(dev, dma_addr, size, swiotlb_force);
+		dma_addr = swiotlb_bounce_page(dev, &phys, size, dir, attrs);
+	}
 
-	return DIRECT_MAPPING_ERROR;
+	return dma_addr;
 }
 
 /*
Signature · 1 line
2.18.0
#8 lore raw

PATCH 8/9 swiotlb: add support for non-coherent DMA

In reply to Christoph Hellwig

Handle architectures that are not cache coherent directly in the main swiotlb code. This involves two related changes:

- call arch_sync_dma_for_{device,cpu} in all the right places from the various dma_map/unmap/sync methods when the device is non-coherent
- call arch_dma_{alloc,free} for devices that are non-coherent

---
 arch/arm64/mm/dma-mapping.c |  6 ++---
 include/linux/swiotlb.h     |  4 ++--
 kernel/dma/swiotlb.c        | 44 ++++++++++++++++++++++++++++++-------
 3 files changed, 41 insertions(+), 13 deletions(-)
arch/arm64/mm/dma-mapping.c +3 -3
index abcae73eea50..07d9c2633f80 100644
@@ -112,7 +112,7 @@ static void *__dma_alloc(struct device *dev, size_t size,
 		return addr;
 	}
 
-	ptr = swiotlb_alloc(dev, size, dma_handle, flags, attrs);
+	ptr = __swiotlb_alloc(dev, size, dma_handle, flags, attrs);
 	if (!ptr)
 		goto no_mem;
 
@@ -133,7 +133,7 @@ static void *__dma_alloc(struct device *dev, size_t size,
 	return coherent_ptr;
 
 no_map:
-	swiotlb_free(dev, size, ptr, *dma_handle, attrs);
+	__swiotlb_free(dev, size, ptr, *dma_handle, attrs);
 no_mem:
 	return NULL;
 }
@@ -151,7 +151,7 @@ static void __dma_free(struct device *dev, size_t size,
 			return;
 		vunmap(vaddr);
 	}
-	swiotlb_free(dev, size, swiotlb_addr, dma_handle, attrs);
+	__swiotlb_free(dev, size, swiotlb_addr, dma_handle, attrs);
 }
 
 static dma_addr_t __swiotlb_map_page(struct device *dev, struct page *page,
include/linux/swiotlb.h +2 -2
index f847c1b265c4..bc809d826d4f 100644
@@ -67,9 +67,9 @@ extern void swiotlb_tbl_sync_single(struct device *hwdev,
 
 /* Accessory functions. */
 
-void *swiotlb_alloc(struct device *hwdev, size_t size, dma_addr_t *dma_handle,
+void *__swiotlb_alloc(struct device *hwdev, size_t size, dma_addr_t *dma_handle,
 		gfp_t flags, unsigned long attrs);
-void swiotlb_free(struct device *dev, size_t size, void *vaddr,
+void __swiotlb_free(struct device *dev, size_t size, void *vaddr,
 		dma_addr_t dma_addr, unsigned long attrs);
 
 extern dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
kernel/dma/swiotlb.c +36 -8

89 lines not on this page · raw message

Signature · 1 line
2.18.0
#9 lore raw

PATCH 9/9 arm64: use the generic swiotlb_dma_ops

In reply to Christoph Hellwig

Now that the generic swiotlb code supports non-coherent DMA we can switch to it for arm64. For that we need to refactor the existing alloc/free/mmap/pgprot helpers to be used as the architecture hooks, and implement the standard arch_sync_dma_for_{device,cpu} hooks for cache maintaincance in the streaming dma hooks, which also implies using the generic dma_coherent flag in struct device.

Note that we need to keep the old is_device_dma_coherent function around for now, so that the shared arm/arm64 Xen code keeps working.

---
 arch/arm64/Kconfig                   |   4 +
 arch/arm64/include/asm/device.h      |   1 -
 arch/arm64/include/asm/dma-mapping.h |   7 +-
 arch/arm64/mm/dma-mapping.c          | 255 +++++----------------------
 4 files changed, 55 insertions(+), 212 deletions(-)
arch/arm64/Kconfig +4 -0
index 1b1a0e95c751..c4db5131d837 100644
@@ -11,6 +11,8 @@ config ARM64
 	select ARCH_CLOCKSOURCE_DATA
 	select ARCH_HAS_DEBUG_VIRTUAL
 	select ARCH_HAS_DEVMEM_IS_ALLOWED
+	select ARCH_HAS_DMA_COHERENT_TO_PFN
+	select ARCH_HAS_DMA_MMAP_PGPROT
 	select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI
 	select ARCH_HAS_ELF_RANDOMIZE
 	select ARCH_HAS_FAST_MULTIPLIER
@@ -24,6 +26,8 @@ config ARM64
 	select ARCH_HAS_SG_CHAIN
 	select ARCH_HAS_STRICT_KERNEL_RWX
 	select ARCH_HAS_STRICT_MODULE_RWX
+	select ARCH_HAS_SYNC_DMA_FOR_DEVICE
+	select ARCH_HAS_SYNC_DMA_FOR_CPU
 	select ARCH_HAS_SYSCALL_WRAPPER
 	select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST
 	select ARCH_HAVE_NMI_SAFE_CMPXCHG
arch/arm64/include/asm/device.h +0 -1
index 5a5fa47a6b18..3dd3d664c5c5 100644
@@ -23,7 +23,6 @@ struct dev_archdata {
 #ifdef CONFIG_XEN
 	const struct dma_map_ops *dev_dma_ops;
 #endif
-	bool dma_coherent;
 };
 
 struct pdev_archdata {
arch/arm64/include/asm/dma-mapping.h +5 -2
index 0a2d13332545..d2414b540af1 100644
@@ -39,10 +39,13 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
 			const struct iommu_ops *iommu, bool coherent);
 #define arch_setup_dma_ops	arch_setup_dma_ops
 
-/* do not use this function in a driver */
+/*
+ * Do not use this function in a driver, it is only provided for
+ * arch/arm/mm/xen.c, which is used by arm64 as well.
+ */
 static inline bool is_device_dma_coherent(struct device *dev)
 {
-	return dev->archdata.dma_coherent;
+	return dev->dma_coherent;
 }
 
 #endif	/* __KERNEL__ */
arch/arm64/mm/dma-mapping.c +46 -209

408 lines not on this page · raw message

Signature · 1 line
2.18.0
reviewer
#10 lore raw
In reply to Christoph Hellwig

Hi Christoph,

Quoted 5 lines · Christoph Hellwig

Hi all,

this series starts with various swiotlb cleanups, then adds support for non-cache coherent devices to the generic swiotlb support, and finally switches arm64 to use the generic code.

I think there's going to be an issue with the embedded folks' grubby hack in arm64's mem_init() which skips initialising SWIOTLB at all with sufficiently little DRAM. I've been waiting for dma-direct-noncoherent-merge so that I could fix that case to swizzle in dma_direct_ops and avoid swiotlb_dma_ops entirely.

Quoted 4 lines

Given that this series depends on patches in the dma-mapping tree, or pending for it I've also published a git tree here:

git://git.infradead.org/users/hch/misc.git swiotlb-noncoherent

However, upon sitting down to eagerly write that patch I've just boot-tested the above branch as-is for a baseline and discovered a rather more significant problem: arch_dma_alloc() is recursing back into __swiotlb_alloc() and blowing the stack. Not good :(

Robin.

----->8-----

[    4.032760] Insufficient stack space to handle exception!
[    4.032765] ESR: 0x96000047 -- DABT (current EL)
[    4.042666] FAR: 0xffff00000a937fb0
[    4.046113] Task stack:     [0xffff00000a938000..0xffff00000a93c000]
[    4.052399] IRQ stack:      [0xffff000008008000..0xffff00000800c000]
[    4.058684] Overflow stack: [0xffff80097ff4b290..0xffff80097ff4c290]
[    4.064972] CPU: 1 PID: 130 Comm: kworker/1:1 Not tainted 4.19.0-rc2+ 
#681
[    4.071775] Hardware name: ARM LTD ARM Juno Development Platform/ARM 
Juno Development Platform, BIOS EDK II Jul 10 2018
[    4.082456] Workqueue: events deferred_probe_work_func
[    4.087542] pstate: 00000005 (nzcv daif -PAN -UAO)
[    4.092283] pc : arch_dma_alloc+0x0/0x198
[    4.096250] lr : dma_direct_alloc+0x20/0x28
[    4.100385] sp : ffff00000a938010
[    4.103660] x29: ffff00000a938010 x28: ffff800974e15238
[    4.108918] x27: ffff000008bf30d8 x26: ffff80097543d400
[    4.114176] x25: 0000000000000300 x24: ffff80097543d400
[    4.119434] x23: ffff800974e15238 x22: 0000000000001000
[    4.124691] x21: ffff80097543d400 x20: 0000000000001000
[    4.129948] x19: 0000000000000300 x18: ffffffffffffffff
[    4.135206] x17: 0000000000000000 x16: ffff000008bf1b58
[    4.140463] x15: ffff0000091eb688 x14: ffff00008a93ba1d
[    4.145720] x13: ffffff0000000000 x12: 0000000000000000
[    4.150977] x11: 0000000000000001 x10: ffffff7f7fff7fff
[    4.156235] x9 : 0000000000000000 x8 : 0000000000000000
[    4.161492] x7 : ffff800974df9810 x6 : 0000000000000000
[    4.166749] x5 : 0000000000000000 x4 : 0000000000000300
[    4.172006] x3 : 00000000006002c0 x2 : ffff800974e15238
[    4.177263] x1 : 0000000000001000 x0 : ffff80097543d400
[    4.182521] Kernel panic - not syncing: kernel stack overflow
[    4.188207] CPU: 1 PID: 130 Comm: kworker/1:1 Not tainted 4.19.0-rc2+ 
#681
[    4.195008] Hardware name: ARM LTD ARM Juno Development Platform/ARM 
Juno Development Platform, BIOS EDK II Jul 10 2018
[    4.205681] Workqueue: events deferred_probe_work_func
[    4.210765] Call trace:
[    4.213183]  dump_backtrace+0x0/0x1f0
[    4.216805]  show_stack+0x14/0x20
[    4.220084]  dump_stack+0x9c/0xbc
[    4.223362]  panic+0x138/0x294
[    4.226381]  __stack_chk_fail+0x0/0x18
[    4.230088]  handle_bad_stack+0x11c/0x130
[    4.234052]  __bad_stack+0x88/0x8c
[    4.237415]  arch_dma_alloc+0x0/0x198
[    4.241036]  __swiotlb_alloc+0x3c/0x178
[    4.244828]  arch_dma_alloc+0xd0/0x198
[    4.248534]  dma_direct_alloc+0x20/0x28
[    4.252327]  __swiotlb_alloc+0x3c/0x178
[    4.256119]  arch_dma_alloc+0xd0/0x198
[    4.259825]  dma_direct_alloc+0x20/0x28
[    4.263617]  __swiotlb_alloc+0x3c/0x178
[    4.267409]  arch_dma_alloc+0xd0/0x198
[    4.271115]  dma_direct_alloc+0x20/0x28
[    4.274907]  __swiotlb_alloc+0x3c/0x178
[    4.278700]  arch_dma_alloc+0xd0/0x198
[    4.282405]  dma_direct_alloc+0x20/0x28
[    4.286198]  __swiotlb_alloc+0x3c/0x178
[    4.289990]  arch_dma_alloc+0xd0/0x198
[    4.293696]  dma_direct_alloc+0x20/0x28
[    4.297489]  __swiotlb_alloc+0x3c/0x178
[    4.301281]  arch_dma_alloc+0xd0/0x198
[    4.304987]  dma_direct_alloc+0x20/0x28
[    4.308779]  __swiotlb_alloc+0x3c/0x178
[    4.312571]  arch_dma_alloc+0xd0/0x198
[    4.316277]  dma_direct_alloc+0x20/0x28
[    4.320069]  __swiotlb_alloc+0x3c/0x178
[    4.323861]  arch_dma_alloc+0xd0/0x198
[    4.327567]  dma_direct_alloc+0x20/0x28
[    4.331359]  __swiotlb_alloc+0x3c/0x178
[    4.335151]  arch_dma_alloc+0xd0/0x198
[    4.338857]  dma_direct_alloc+0x20/0x28
[    4.342650]  __swiotlb_alloc+0x3c/0x178
[    4.346442]  arch_dma_alloc+0xd0/0x198
[    4.350148]  dma_direct_alloc+0x20/0x28
[    4.353940]  __swiotlb_alloc+0x3c/0x178
[    4.357732]  arch_dma_alloc+0xd0/0x198
[    4.361438]  dma_direct_alloc+0x20/0x28
[    4.365230]  __swiotlb_alloc+0x3c/0x178
[    4.369022]  arch_dma_alloc+0xd0/0x198
[    4.372728]  dma_direct_alloc+0x20/0x28
[    4.376520]  __swiotlb_alloc+0x3c/0x178
[    4.380313]  arch_dma_alloc+0xd0/0x198
[    4.384018]  dma_direct_alloc+0x20/0x28
[    4.387811]  __swiotlb_alloc+0x3c/0x178
[    4.391603]  arch_dma_alloc+0xd0/0x198
[    4.395309]  dma_direct_alloc+0x20/0x28
[    4.399101]  __swiotlb_alloc+0x3c/0x178
[    4.402893]  arch_dma_alloc+0xd0/0x198
[    4.406599]  dma_direct_alloc+0x20/0x28
[    4.410391]  __swiotlb_alloc+0x3c/0x178
[    4.414183]  arch_dma_alloc+0xd0/0x198
[    4.417889]  dma_direct_alloc+0x20/0x28
[    4.421682]  __swiotlb_alloc+0x3c/0x178
[    4.425474]  arch_dma_alloc+0xd0/0x198
[    4.429180]  dma_direct_alloc+0x20/0x28
[    4.432972]  __swiotlb_alloc+0x3c/0x178
[    4.436764]  arch_dma_alloc+0xd0/0x198
[    4.440470]  dma_direct_alloc+0x20/0x28
[    4.444262]  __swiotlb_alloc+0x3c/0x178
[    4.448055]  arch_dma_alloc+0xd0/0x198
[    4.451760]  dma_direct_alloc+0x20/0x28
[    4.455553]  __swiotlb_alloc+0x3c/0x178
[    4.459345]  arch_dma_alloc+0xd0/0x198
[    4.463051]  dma_direct_alloc+0x20/0x28
[    4.466843]  __swiotlb_alloc+0x3c/0x178
[    4.470635]  arch_dma_alloc+0xd0/0x198
[    4.474341]  dma_direct_alloc+0x20/0x28
[    4.478134]  __swiotlb_alloc+0x3c/0x178
[    4.481926]  arch_dma_alloc+0xd0/0x198
[    4.485632]  dma_direct_alloc+0x20/0x28
[    4.489424]  __swiotlb_alloc+0x3c/0x178
[    4.493216]  arch_dma_alloc+0xd0/0x198
[    4.496922]  dma_direct_alloc+0x20/0x28
[    4.500714]  __swiotlb_alloc+0x3c/0x178
[    4.504506]  arch_dma_alloc+0xd0/0x198
[    4.508212]  dma_direct_alloc+0x20/0x28
[    4.512004]  __swiotlb_alloc+0x3c/0x178
[    4.515797]  arch_dma_alloc+0xd0/0x198
[    4.519502]  dma_direct_alloc+0x20/0x28
[    4.523295]  __swiotlb_alloc+0x3c/0x178
[    4.527087]  arch_dma_alloc+0xd0/0x198
[    4.530793]  dma_direct_alloc+0x20/0x28
[    4.534585]  __swiotlb_alloc+0x3c/0x178
[    4.538378]  arch_dma_alloc+0xd0/0x198
[    4.542083]  dma_direct_alloc+0x20/0x28
[    4.545876]  __swiotlb_alloc+0x3c/0x178
[    4.549668]  arch_dma_alloc+0xd0/0x198
[    4.553374]  dma_direct_alloc+0x20/0x28
[    4.557166]  __swiotlb_alloc+0x3c/0x178
[    4.560958]  arch_dma_alloc+0xd0/0x198
[    4.564664]  dma_direct_alloc+0x20/0x28
[    4.568456]  __swiotlb_alloc+0x3c/0x178
[    4.572248]  arch_dma_alloc+0xd0/0x198
[    4.575954]  dma_direct_alloc+0x20/0x28
[    4.579747]  __swiotlb_alloc+0x3c/0x178
[    4.583539]  arch_dma_alloc+0xd0/0x198
[    4.587245]  dma_direct_alloc+0x20/0x28
[    4.591037]  __swiotlb_alloc+0x3c/0x178
[    4.594829]  arch_dma_alloc+0xd0/0x198
[    4.598535]  dma_direct_alloc+0x20/0x28
[    4.602327]  __swiotlb_alloc+0x3c/0x178
[    4.606120]  arch_dma_alloc+0xd0/0x198
[    4.609826]  dma_direct_alloc+0x20/0x28
[    4.613618]  __swiotlb_alloc+0x3c/0x178
[    4.617410]  arch_dma_alloc+0xd0/0x198
[    4.621116]  dma_direct_alloc+0x20/0x28
[    4.624908]  __swiotlb_alloc+0x3c/0x178
[    4.628701]  arch_dma_alloc+0xd0/0x198
[    4.632407]  dma_direct_alloc+0x20/0x28
[    4.636199]  __swiotlb_alloc+0x3c/0x178
[    4.639991]  arch_dma_alloc+0xd0/0x198
[    4.643697]  dma_direct_alloc+0x20/0x28
[    4.647490]  __swiotlb_alloc+0x3c/0x178
[    4.651282]  arch_dma_alloc+0xd0/0x198
[    4.654988]  dma_direct_alloc+0x20/0x28
[    4.658780]  __swiotlb_alloc+0x3c/0x178
[    4.662572]  arch_dma_alloc+0xd0/0x198
[    4.666278]  dma_direct_alloc+0x20/0x28
[    4.670070]  __swiotlb_alloc+0x3c/0x178
[    4.673862]  arch_dma_alloc+0xd0/0x198
[    4.677568]  dma_direct_alloc+0x20/0x28
[    4.681361]  __swiotlb_alloc+0x3c/0x178
[    4.685153]  arch_dma_alloc+0xd0/0x198
[    4.688859]  dma_direct_alloc+0x20/0x28
[    4.692651]  __swiotlb_alloc+0x3c/0x178
[    4.696443]  arch_dma_alloc+0xd0/0x198
[    4.700148]  dma_direct_alloc+0x20/0x28
[    4.703941]  __swiotlb_alloc+0x3c/0x178
[    4.707733]  arch_dma_alloc+0xd0/0x198
[    4.711439]  dma_direct_alloc+0x20/0x28
[    4.715231]  __swiotlb_alloc+0x3c/0x178
[    4.719023]  arch_dma_alloc+0xd0/0x198
[    4.722729]  dma_direct_alloc+0x20/0x28
[    4.726522]  __swiotlb_alloc+0x3c/0x178
[    4.730313]  arch_dma_alloc+0xd0/0x198
[    4.734019]  dma_direct_alloc+0x20/0x28
[    4.737811]  __swiotlb_alloc+0x3c/0x178
[    4.741604]  arch_dma_alloc+0xd0/0x198
[    4.745310]  dma_direct_alloc+0x20/0x28
[    4.749102]  __swiotlb_alloc+0x3c/0x178
[    4.752894]  arch_dma_alloc+0xd0/0x198
[    4.756600]  dma_direct_alloc+0x20/0x28
[    4.760393]  __swiotlb_alloc+0x3c/0x178
[    4.764185]  arch_dma_alloc+0xd0/0x198
[    4.767891]  dma_direct_alloc+0x20/0x28
[    4.771683]  __swiotlb_alloc+0x3c/0x178
[    4.775475]  arch_dma_alloc+0xd0/0x198
[    4.779181]  dma_direct_alloc+0x20/0x28
[    4.782973]  __swiotlb_alloc+0x3c/0x178
[    4.786765]  arch_dma_alloc+0xd0/0x198
[    4.790471]  dma_direct_alloc+0x20/0x28
[    4.794263]  __swiotlb_alloc+0x3c/0x178
[    4.798055]  arch_dma_alloc+0xd0/0x198
[    4.801761]  dma_direct_alloc+0x20/0x28
[    4.805553]  __swiotlb_alloc+0x3c/0x178
[    4.809345]  arch_dma_alloc+0xd0/0x198
[    4.813051]  dma_direct_alloc+0x20/0x28
[    4.816844]  __swiotlb_alloc+0x3c/0x178
[    4.820636]  arch_dma_alloc+0xd0/0x198
[    4.824342]  dma_direct_alloc+0x20/0x28
[    4.828134]  __swiotlb_alloc+0x3c/0x178
[    4.831926]  arch_dma_alloc+0xd0/0x198
[    4.835632]  dma_direct_alloc+0x20/0x28
[    4.839425]  __swiotlb_alloc+0x3c/0x178
[    4.843217]  arch_dma_alloc+0xd0/0x198
[    4.846923]  dma_direct_alloc+0x20/0x28
[    4.850715]  __swiotlb_alloc+0x3c/0x178
[    4.854507]  arch_dma_alloc+0xd0/0x198
[    4.858212]  dma_direct_alloc+0x20/0x28
[    4.862005]  __swiotlb_alloc+0x3c/0x178
[    4.865797]  arch_dma_alloc+0xd0/0x198
[    4.869503]  dma_direct_alloc+0x20/0x28
[    4.873295]  __swiotlb_alloc+0x3c/0x178
[    4.877087]  arch_dma_alloc+0xd0/0x198
[    4.880793]  dma_direct_alloc+0x20/0x28
[    4.884586]  __swiotlb_alloc+0x3c/0x178
[    4.888377]  arch_dma_alloc+0xd0/0x198
[    4.892083]  dma_direct_alloc+0x20/0x28
[    4.895875]  __swiotlb_alloc+0x3c/0x178
[    4.899667]  arch_dma_alloc+0xd0/0x198
[    4.903373]  dma_direct_alloc+0x20/0x28
[    4.907165]  __swiotlb_alloc+0x3c/0x178
[    4.910957]  arch_dma_alloc+0xd0/0x198
[    4.914663]  dma_direct_alloc+0x20/0x28
[    4.918455]  __swiotlb_alloc+0x3c/0x178
[    4.922247]  arch_dma_alloc+0xd0/0x198
[    4.925953]  dma_direct_alloc+0x20/0x28
[    4.929746]  __swiotlb_alloc+0x3c/0x178
[    4.933538]  arch_dma_alloc+0xd0/0x198
[    4.937244]  dma_direct_alloc+0x20/0x28
[    4.941036]  __swiotlb_alloc+0x3c/0x178
[    4.944828]  arch_dma_alloc+0xd0/0x198
[    4.948534]  dma_direct_alloc+0x20/0x28
[    4.952327]  __swiotlb_alloc+0x3c/0x178
[    4.956119]  arch_dma_alloc+0xd0/0x198
[    4.959825]  dma_direct_alloc+0x20/0x28
[    4.963617]  __swiotlb_alloc+0x3c/0x178
[    4.967409]  arch_dma_alloc+0xd0/0x198
[    4.971115]  dma_direct_alloc+0x20/0x28
[    4.974907]  __swiotlb_alloc+0x3c/0x178
[    4.978699]  arch_dma_alloc+0xd0/0x198
[    4.982405]  dma_direct_alloc+0x20/0x28
[    4.986197]  __swiotlb_alloc+0x3c/0x178
[    4.989990]  arch_dma_alloc+0xd0/0x198
[    4.993695]  dma_direct_alloc+0x20/0x28
[    4.997488]  __swiotlb_alloc+0x3c/0x178
[    5.001280]  arch_dma_alloc+0xd0/0x198
[    5.004986]  dma_direct_alloc+0x20/0x28
[    5.008778]  __swiotlb_alloc+0x3c/0x178
[    5.012570]  arch_dma_alloc+0xd0/0x198
[    5.016276]  dma_direct_alloc+0x20/0x28
[    5.020069]  __swiotlb_alloc+0x3c/0x178
[    5.023861]  arch_dma_alloc+0xd0/0x198
[    5.027567]  dma_direct_alloc+0x20/0x28
[    5.031359]  __swiotlb_alloc+0x3c/0x178
[    5.035151]  arch_dma_alloc+0xd0/0x198
[    5.038857]  dma_direct_alloc+0x20/0x28
[    5.042650]  __swiotlb_alloc+0x3c/0x178
[    5.046442]  arch_dma_alloc+0xd0/0x198
[    5.050148]  dma_direct_alloc+0x20/0x28
[    5.053940]  __swiotlb_alloc+0x3c/0x178
[    5.057732]  arch_dma_alloc+0xd0/0x198
[    5.061438]  dma_direct_alloc+0x20/0x28
[    5.065230]  __swiotlb_alloc+0x3c/0x178
[    5.069023]  arch_dma_alloc+0xd0/0x198
[    5.072728]  dma_direct_alloc+0x20/0x28
[    5.076521]  __swiotlb_alloc+0x3c/0x178
[    5.080313]  arch_dma_alloc+0xd0/0x198
[    5.084019]  dma_direct_alloc+0x20/0x28
[    5.087811]  __swiotlb_alloc+0x3c/0x178
[    5.091604]  arch_dma_alloc+0xd0/0x198
[    5.095309]  dma_direct_alloc+0x20/0x28
[    5.099102]  __swiotlb_alloc+0x3c/0x178
[    5.102894]  arch_dma_alloc+0xd0/0x198
[    5.106600]  dma_direct_alloc+0x20/0x28
[    5.110392]  __swiotlb_alloc+0x3c/0x178
[    5.114184]  arch_dma_alloc+0xd0/0x198
[    5.117890]  dma_direct_alloc+0x20/0x28
[    5.121682]  __swiotlb_alloc+0x3c/0x178
[    5.125474]  arch_dma_alloc+0xd0/0x198
[    5.129180]  dma_direct_alloc+0x20/0x28
[    5.132973]  __swiotlb_alloc+0x3c/0x178
[    5.136765]  arch_dma_alloc+0xd0/0x198
[    5.140470]  dma_direct_alloc+0x20/0x28
[    5.144262]  __swiotlb_alloc+0x3c/0x178
[    5.148054]  arch_dma_alloc+0xd0/0x198
[    5.151760]  dma_direct_alloc+0x20/0x28
[    5.155552]  __swiotlb_alloc+0x3c/0x178
[    5.159345]  arch_dma_alloc+0xd0/0x198
[    5.163050]  dma_direct_alloc+0x20/0x28
[    5.166843]  __swiotlb_alloc+0x3c/0x178
[    5.170635]  arch_dma_alloc+0xd0/0x198
[    5.174341]  dma_direct_alloc+0x20/0x28
[    5.178133]  __swiotlb_alloc+0x3c/0x178
[    5.181925]  arch_dma_alloc+0xd0/0x198
[    5.185631]  dma_direct_alloc+0x20/0x28
[    5.189423]  __swiotlb_alloc+0x3c/0x178
[    5.193215]  arch_dma_alloc+0xd0/0x198
[    5.196921]  dma_direct_alloc+0x20/0x28
[    5.200714]  __swiotlb_alloc+0x3c/0x178
[    5.204506]  arch_dma_alloc+0xd0/0x198
[    5.208212]  swiotlb_alloc+0x20/0x28
[    5.211748]  pl330_probe+0x344/0xaf0
[    5.215283]  amba_probe+0xe8/0x1b8
[    5.218646]  really_probe+0xdc/0x3d0
[    5.222181]  driver_probe_device+0x5c/0x148
[    5.226318]  __device_attach_driver+0xa8/0x160
[    5.230712]  bus_for_each_drv+0x64/0xc8
[    5.234505]  __device_attach+0xd8/0x158
[    5.238298]  device_initial_probe+0x10/0x18
[    5.242434]  bus_probe_device+0x90/0x98
[    5.246227]  deferred_probe_work_func+0x88/0xe0
[    5.250708]  process_one_work+0x1e0/0x330
[    5.254673]  worker_thread+0x238/0x460
[    5.258380]  kthread+0x128/0x130
[    5.261571]  ret_from_fork+0x10/0x1c
[    5.265109] SMP: stopping secondary CPUs
[    5.268990] Kernel Offset: disabled
[    5.272438] CPU features: 0x0,25806004
[    5.276143] Memory Limit: none
[    5.279168] ---[ end Kernel panic - not syncing: kernel stack 

overflow ]---

#11 lore raw
Quoted 7 lines · Robin Murphy
Quoted 5 lines · Christoph Hellwig

Hi all,

this series starts with various swiotlb cleanups, then adds support for non-cache coherent devices to the generic swiotlb support, and finally switches arm64 to use the generic code.

Quoted 5 lines

I think there's going to be an issue with the embedded folks' grubby hack in arm64's mem_init() which skips initialising SWIOTLB at all with sufficiently little DRAM. I've been waiting for dma-direct-noncoherent-merge so that I could fix that case to swizzle in dma_direct_ops and avoid swiotlb_dma_ops entirely.

I wait for your review of dma-direct-noncoherent-merge to put it into dma-mapping for-next..

That being said one thing I'm investigating is to eventually further merge dma_direct_ops and swiotlb_ops - the reason for that beeing that I want to remove the indirect calls for the common direct mapping case, and if we don't merge them that will get complicated. Note that swiotlb will generally just work if you don't initialize the buffer as long as we never see a physical address large enough to cause bounce buffering.

Quoted 10 lines
Quoted 5 lines

Given that this series depends on patches in the dma-mapping tree, or pending for it I've also published a git tree here:

git://git.infradead.org/users/hch/misc.git swiotlb-noncoherent

However, upon sitting down to eagerly write that patch I've just boot-tested the above branch as-is for a baseline and discovered a rather more significant problem: arch_dma_alloc() is recursing back into __swiotlb_alloc() and blowing the stack. Not good :(

Oops, I messed up when renaming things. Try this patch on top:

kernel/dma/swiotlb.c +2 -2
index 83e597101c6a..c75c721eb74e 100644
@@ -955,7 +955,7 @@ void *__swiotlb_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle,
 	 */
 	gfp |= __GFP_NOWARN;
 
-	vaddr = dma_direct_alloc(dev, size, dma_handle, gfp, attrs);
+	vaddr = dma_direct_alloc_pages(dev, size, dma_handle, gfp, attrs);
 	if (!vaddr)
 		vaddr = swiotlb_alloc_buffer(dev, size, dma_handle, attrs);
 	return vaddr;
@@ -973,7 +973,7 @@ void __swiotlb_free(struct device *dev, size_t size, void *vaddr,
 		dma_addr_t dma_addr, unsigned long attrs)
 {
 	if (!swiotlb_free_buffer(dev, size, dma_addr))
-		dma_direct_free(dev, size, vaddr, dma_addr, attrs);
+		dma_direct_free_pages(dev, size, vaddr, dma_addr, attrs);
 }
 
 static void swiotlb_free(struct device *dev, size_t size, void *vaddr,