Skip to contentSkip to conversation

Allow dynamic allocation of software IO TLB bounce buffers

PATCH In review Petr Tesarik 5 participants 18 messages Aug 1, 2023 – Sep 8, 2023

Subsystems (9)
lore
Share & tools
lore
#0 lore raw

v7 cover Allow dynamic allocation of software IO TLB bounce buffers

From: Petr Tesarik <[email protected]>

Motivation
==========

The software IO TLB was designed with these assumptions:

1) It would not be used much. Small systems (little RAM) don't need it, and big systems (lots of RAM) would have modern DMA controllers and an IOMMU chip to handle legacy devices.
2) A small fixed memory area (64 MiB by default) is sufficient to handle the few cases which require a bounce buffer.
3) 64 MiB is little enough that it has no impact on the rest of the system.
4) Bounce buffers require large contiguous chunks of low memory. Such memory is precious and can be allocated only early at boot.

It turns out they are not always true:

1) Embedded systems may have more than 4GiB RAM but no IOMMU and legacy 32-bit peripheral busses and/or DMA controllers.
2) CoCo VMs use bounce buffers for all I/O but may need substantially more than 64 MiB.
3) Embedded developers put as many features as possible into the available memory. A few dozen "missing" megabytes may limit what features can be implemented.
4) If CMA is available, it can allocate large continuous chunks even after the system has run for some time.

Goals
=====

The goal of this work is to start with a small software IO TLB at boot and expand it later when/if needed.

Design
======

This version of the patch series retains the current slot allocation algorithm with multiple areas to reduce lock contention, but additional slots can be added when necessary.

These alternatives have been considered:

- Allocate and free buffers as needed using direct DMA API. This works quite well, except in CoCo VMs where each allocation/free requires decrypting/encrypting memory, which is a very expensive operation.

- Allocate a very large software IO TLB at boot, but allow to migrate pages to/from it (like CMA does). For systems with CMA, this would mean two big allocations at boot. Finding the balance between CMA, SWIOTLB and rest of available RAM can be challenging. More importantly, there is no clear benefit compared to allocating SWIOTLB memory pools from the CMA.

Implementation Constraints
==========================

These constraints have been taken into account:

1) Minimize impact on devices which do not benefit from the change.
2) Minimize the number of memory decryption/encryption operations.
3) Avoid contention on a lock or atomic variable to preserve parallel scalability.

Additionally, the software IO TLB code is also used to implement restricted DMA pools. These pools are restricted to a pre-defined physical memory region and must not use any other memory. In other words, dynamic allocation of memory pools must be disabled for restricted DMA pools.

Data Structures
===============

The existing struct io_tlb_mem is the central type for a SWIOTLB allocator, but it now contains multiple memory pools::

io_tlb_mem

  +---------+   io_tlb_pool
  | SWIOTLB | 	+-------+   +-------+   +-------+
  |allocator|-->|default|-->|dynamic|-->|dynamic|-->...
  |    	    |  	|memory |   |memory |   |memory |
  +---------+ 	| pool  |   | pool  |   | pool  |
	      	+-------+   +-------+   +-------+

The allocator structure contains global state (such as flags and counters) and structures needed to schedule new allocations. Each memory pool contains the actual buffer slots and metadata. The first memory pool in the list is the default memory pool allocated statically at early boot.

New memory pools are allocated from a kernel worker thread. That's because bounce buffers are allocated when mapping a DMA buffer, which may happen in interrupt context where large atomic allocations would probably fail. Allocation from process context is much more likely to succeed, especially if it can use CMA.

Nonetheless, the onset of a load spike may fill up the SWIOTLB before the worker has a chance to run. In that case, try to allocate a small transient memory pool to accommodate the request. If memory is encrypted and the device cannot do DMA to encrypted memory, this buffer is allocated from the coherent atomic DMA memory pool. Reducing the size of SWIOTLB may therefore require increasing the size of the coherent pool with the "coherent_pool" command-line parameter.

Performance
===========

All testing compared a vanilla v6.4-rc6 kernel with a fully patched kernel. The kernel was booted with "swiotlb=force" to allow stress-testing the software IO TLB on a high-performance device that would otherwise not need it. CONFIG_DEBUG_FS was set to 'y' to match the configuration of popular distribution kernels; it is understood that parallel workloads suffer from contention on the recently added debugfs atomic counters.

These benchmarks were run:

- small: single-threaded I/O of 4 KiB blocks,
- big: single-threaded I/O of 64 KiB blocks,
- 4way: 4-way parallel I/O of 4 KiB blocks.

In all tested cases, the default 64 MiB SWIOTLB would be sufficient (but wasteful). The "default" pair of columns shows performance impact when booted with 64 MiB SWIOTLB (i.e. current state). The "growing" pair of columns shows the impact when booted with a 1 MiB initial SWIOTLB, which grew to 5 MiB at run time. The "var" column in the tables below is the coefficient of variance over 5 runs of the test, the "diff" column is the difference in read-write I/O bandwidth (MiB/s). The very first column is the coefficient of variance in the results of the base unpatched kernel.

First, on an x86 VM against a QEMU virtio SATA driver backed by a RAM-based block device on the host:

base default growing
var var diff var diff
small 1.96% 0.47% -1.5% 0.52% -2.2%
big 2.03% 1.35% +0.9% 2.22% +2.9%
4way 0.80% 0.45% -0.7% 1.22% <0.1%

Second, on a Raspberry Pi4 with 8G RAM and a class 10 A1 microSD card:

base default growing
var var diff var diff
small 1.09% 1.69% +0.5% 2.14% -0.2%
big 0.03% 0.28% -0.5% 0.03% -0.1%
4way 5.15% 2.39% +0.2% 0.66% <0.1%

Third, on a CoCo VM. This was a bigger system, so I also added a 24-thread parallel I/O test:

base default growing
var var diff var diff
small 2.41% 6.02% +1.1% 10.33% +6.7%
big 9.20% 2.81% -0.6% 16.84% -0.2%
4way 0.86% 2.66% -0.1% 2.22% -4.9%
24way 3.19% 6.19% +4.4% 4.08% -5.9%

Note the increased variance of the CoCo VM, although the host was not otherwise loaded. These are caused by the first run, which includes the overhead of allocating additional bounce buffers and sharing them with the hypervisor. The system was not rebooted between successive runs.

Parallel tests suffer from a reduced number of areas in the dynamically allocated memory pools. This can be improved by allocating a larger pool from CMA (not implemented in this series yet).

I have no good explanation for the increase in performance of the 24-thread I/O test with the default (non-growing) memory pool. Although the difference is within variance, it seems to be real. The average bandwidth is consistently above that of the unpatched kernel.

To sum it up:

- All workloads benefit from reduced memory footprint.
- No performance regressions have been observed with the default size of the software IO TLB.
- Most workloads retain their former performance even if the software IO TLB grows at run time.

Changelog
=========

Changes from v6:
- Rebase on dma-mapping for-next tree. Drop changes to pci-dma.c.

Changes from v5:
- Re-introduce is_swiotlb_allocated(), now again required because of commit b035f5a6d852 ("mm: slab: reduce the kmalloc() minimum alignment if DMA bouncing possible").

Changes from v4:
- Guard the code with a CONFIG_SWIOTLB_DYNAMIC option
- Remove is_swiotlb_allocated(); instead, prevent repeated initialization in swiotlb_init_late()
- Rename default_swiotlb_start() to default_swiotlb_base()
- Embed the default struct io_tlb_pool into struct io_tlb_mem
- Do not re-introduce struct io_tlb_pool.used

Changes from v3:
- Provide swiotlb_is_allocated() instead of extending swiotlb_is_active().
- Do not grow SWIOTLB if its address has been queried (affects Octeon).
- Do not grow SWIOTLB if a remap function is used (affects Xen PV).
- Use dma_mask instead of coherent_dma_mask.
- Replace complex ternary operators with if-else blocks.

Changes from v2:
- Complete rewrite using dynamically allocated memory pools rather than a list of individual buffers
- Depend on other SWIOTLB fixes (already sent)
- Fix Xen and MIPS Octeon builds

Changes from RFC:
- Track dynamic buffers per device instead of per swiotlb
- Use a linked list instead of a maple tree
- Move initialization of swiotlb fields of struct device to a helper function
- Rename __lookup_dyn_slot() to lookup_dyn_slot_locked()
- Introduce per-device flag if dynamic buffers are in use
- Add one more user of DMA_ATTR_MAY_SLEEP
- Add kernel-doc comments for new (and some old) code
- Properly escape '*' in dma-attributes.rst

Petr Tesarik (9):
swiotlb: bail out of swiotlb_init_late() if swiotlb is already
allocated
swiotlb: make io_tlb_default_mem local to swiotlb.c
swiotlb: add documentation and rename swiotlb_do_find_slots()
swiotlb: separate memory pool data from other allocator data
swiotlb: add a flag whether SWIOTLB is allowed to grow
swiotlb: if swiotlb is full, fall back to a transient memory pool
swiotlb: determine potential physical address limit
swiotlb: allocate a new memory pool when existing pools are full
swiotlb: search the software IO TLB only if the device makes use of it

 arch/arm/xen/mm.c           |  10 +-
 arch/mips/pci/pci-octeon.c  |   2 +-
 drivers/base/core.c         |   4 +-
 drivers/xen/swiotlb-xen.c   |   2 +-
 include/linux/device.h      |  10 +-
 include/linux/dma-mapping.h |   2 +
 include/linux/swiotlb.h     | 131 +++++--
 kernel/dma/Kconfig          |  13 +
 kernel/dma/direct.c         |   2 +-
 kernel/dma/swiotlb.c        | 683 ++++++++++++++++++++++++++++++++----
 mm/slab_common.c            |   5 +-
 11 files changed, 764 insertions(+), 100 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH v7 1/9 swiotlb: bail out of swiotlb_init_late() if swiotlb is already allocated

From: Petr Tesarik <[email protected]>

If swiotlb is allocated, immediately return 0, so callers do not have to check io_tlb_default_mem.nslabs explicitly.

---
 arch/arm/xen/mm.c    | 10 ++++------
 kernel/dma/swiotlb.c |  3 +++
 2 files changed, 7 insertions(+), 6 deletions(-)
arch/arm/xen/mm.c +4 -6
index 3d826c0b5fee..882cd70c7a2f 100644
@@ -125,12 +125,10 @@ static int __init xen_mm_init(void)
 		return 0;
 
 	/* we can work with the default swiotlb */
-	if (!io_tlb_default_mem.nslabs) {
-		rc = swiotlb_init_late(swiotlb_size_or_default(),
-				       xen_swiotlb_gfp(), NULL);
-		if (rc < 0)
-			return rc;
-	}
+	rc = swiotlb_init_late(swiotlb_size_or_default(),
+			       xen_swiotlb_gfp(), NULL);
+	if (rc < 0)
+		return rc;
 
 	cflush.op = 0;
 	cflush.a.dev_bus_addr = 0;
kernel/dma/swiotlb.c +3 -0
index 1c0a49e6685a..ee57fd9949dc 100644
@@ -384,6 +384,9 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask,
 	bool retried = false;
 	int rc = 0;
 
+	if (io_tlb_default_mem.nslabs)
+		return 0;
+
 	if (swiotlb_force_disable)
 		return 0;
 
Signature · 1 line
2.25.1
#2 lore raw

PATCH v7 2/9 swiotlb: make io_tlb_default_mem local to swiotlb.c

In reply to Petr Tesarik

From: Petr Tesarik <[email protected]>

SWIOTLB implementation details should not be exposed to the rest of the kernel. This will allow to make changes to the implementation without modifying non-swiotlb code.

To avoid breaking existing users, provide helper functions for the few required fields.

As a bonus, using a helper function to initialize struct device allows to get rid of an #ifdef in driver core.

---
 arch/mips/pci/pci-octeon.c |  2 +-
 drivers/base/core.c        |  4 +---
 drivers/xen/swiotlb-xen.c  |  2 +-
 include/linux/swiotlb.h    | 25 +++++++++++++++++++++++-
 kernel/dma/swiotlb.c       | 39 +++++++++++++++++++++++++++++++++++++-
 mm/slab_common.c           |  5 ++---
 6 files changed, 67 insertions(+), 10 deletions(-)
arch/mips/pci/pci-octeon.c +1 -1
index e457a18cbdc5..d19d9d456309 100644
@@ -664,7 +664,7 @@ static int __init octeon_pci_setup(void)
 
 		/* BAR1 movable regions contiguous to cover the swiotlb */
 		octeon_bar1_pci_phys =
-			io_tlb_default_mem.start & ~((1ull << 22) - 1);
+			default_swiotlb_base() & ~((1ull << 22) - 1);
 
 		for (index = 0; index < 32; index++) {
 			union cvmx_pci_bar1_indexx bar1_index;
drivers/base/core.c +1 -3
index 3dff5037943e..46d1d78c5beb 100644
@@ -3108,9 +3108,7 @@ void device_initialize(struct device *dev)
     defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL)
 	dev->dma_coherent = dma_default_coherent;
 #endif
-#ifdef CONFIG_SWIOTLB
-	dev->dma_io_tlb_mem = &io_tlb_default_mem;
-#endif
+	swiotlb_dev_init(dev);
 }
 EXPORT_SYMBOL_GPL(device_initialize);
 
drivers/xen/swiotlb-xen.c +1 -1
index 67aa74d20162..946bd56f0ac5 100644
@@ -381,7 +381,7 @@ xen_swiotlb_sync_sg_for_device(struct device *dev, struct scatterlist *sgl,
 static int
 xen_swiotlb_dma_supported(struct device *hwdev, u64 mask)
 {
-	return xen_phys_to_dma(hwdev, io_tlb_default_mem.end - 1) <= mask;
+	return xen_phys_to_dma(hwdev, default_swiotlb_limit()) <= mask;
 }
 
 const struct dma_map_ops xen_swiotlb_dma_ops = {
include/linux/swiotlb.h +24 -1
index 4e52cd5e0bdc..2d453b3e7771 100644
@@ -110,7 +110,6 @@ struct io_tlb_mem {
 	atomic_long_t used_hiwater;
 #endif
 };
-extern struct io_tlb_mem io_tlb_default_mem;
 
 static inline bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr)
 {
@@ -128,13 +127,22 @@ static inline bool is_swiotlb_force_bounce(struct device *dev)
 
 void swiotlb_init(bool addressing_limited, unsigned int flags);
 void __init swiotlb_exit(void);
+void swiotlb_dev_init(struct device *dev);
 size_t swiotlb_max_mapping_size(struct device *dev);
+bool is_swiotlb_allocated(void);
 bool is_swiotlb_active(struct device *dev);
 void __init swiotlb_adjust_size(unsigned long size);
+phys_addr_t default_swiotlb_base(void);
+phys_addr_t default_swiotlb_limit(void);
 #else
 static inline void swiotlb_init(bool addressing_limited, unsigned int flags)
 {
 }
+
+static inline void swiotlb_dev_init(struct device *dev)
+{
+}
+
 static inline bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr)
 {
 	return false;
@@ -151,6 +159,11 @@ static inline size_t swiotlb_max_mapping_size(struct device *dev)
 	return SIZE_MAX;
 }
 
+static inline bool is_swiotlb_allocated(void)
+{
+	return false;
+}
+
 static inline bool is_swiotlb_active(struct device *dev)
 {
 	return false;
@@ -159,6 +172,16 @@ static inline bool is_swiotlb_active(struct device *dev)
 static inline void swiotlb_adjust_size(unsigned long size)
 {
 }
+
+static inline phys_addr_t default_swiotlb_base(void)
+{
+	return 0;
+}
+
+static inline phys_addr_t default_swiotlb_limit(void)
+{
+	return 0;
+}
 #endif /* CONFIG_SWIOTLB */
 
 extern void swiotlb_print_info(void);
kernel/dma/swiotlb.c +38 -1
index ee57fd9949dc..0840bc15fb53 100644
@@ -71,7 +71,7 @@ struct io_tlb_slot {
 static bool swiotlb_force_bounce;
 static bool swiotlb_force_disable;
 
-struct io_tlb_mem io_tlb_default_mem;
+static struct io_tlb_mem io_tlb_default_mem;
 
 static unsigned long default_nslabs = IO_TLB_DEFAULT_SIZE >> IO_TLB_SHIFT;
 static unsigned long default_nareas;
@@ -489,6 +489,15 @@ void __init swiotlb_exit(void)
 	memset(mem, 0, sizeof(*mem));
 }
 
+/**
+ * swiotlb_dev_init() - initialize swiotlb fields in &struct device
+ * @dev:	Device to be initialized.
+ */
+void swiotlb_dev_init(struct device *dev)
+{
+	dev->dma_io_tlb_mem = &io_tlb_default_mem;
+}
+
 /*
  * Return the offset into a iotlb slot required to keep the device happy.
  */
@@ -953,6 +962,14 @@ size_t swiotlb_max_mapping_size(struct device *dev)
 	return ((size_t)IO_TLB_SIZE) * IO_TLB_SEGSIZE - min_align;
 }
 
+/**
+ * is_swiotlb_allocated() - check if the default software IO TLB is initialized
+ */
+bool is_swiotlb_allocated(void)
+{
+	return io_tlb_default_mem.nslabs;
+}
+
 bool is_swiotlb_active(struct device *dev)
 {
 	struct io_tlb_mem *mem = dev->dma_io_tlb_mem;
@@ -960,6 +977,26 @@ bool is_swiotlb_active(struct device *dev)
 	return mem && mem->nslabs;
 }
 
+/**
+ * default_swiotlb_base() - get the base address of the default SWIOTLB
+ *
+ * Get the lowest physical address used by the default software IO TLB pool.
+ */
+phys_addr_t default_swiotlb_base(void)
+{
+	return io_tlb_default_mem.start;
+}
+
+/**
+ * default_swiotlb_limit() - get the address limit of the default SWIOTLB
+ *
+ * Get the highest physical address used by the default software IO TLB pool.
+ */
+phys_addr_t default_swiotlb_limit(void)
+{
+	return io_tlb_default_mem.end - 1;
+}
+
 #ifdef CONFIG_DEBUG_FS
 
 static int io_tlb_used_get(void *data, u64 *val)
mm/slab_common.c +2 -3
index d1555ea2981a..c639dba9a3b7 100644
@@ -864,10 +864,9 @@ void __init setup_kmalloc_cache_index_table(void)
 
 static unsigned int __kmalloc_minalign(void)
 {
-#ifdef CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC
-	if (io_tlb_default_mem.nslabs)
+	if (IS_ENABLED(CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC) &&
+	    is_swiotlb_allocated())
 		return ARCH_KMALLOC_MINALIGN;
-#endif
 	return dma_get_cache_alignment();
 }
 
Signature · 1 line
2.25.1
#3 lore raw

PATCH v7 3/9 swiotlb: add documentation and rename swiotlb_do_find_slots()

In reply to Petr Tesarik

From: Petr Tesarik <[email protected]>

Add some kernel-doc comments and move the existing documentation of struct io_tlb_slot to its correct location. The latter was forgotten in commit 942a8186eb445 ("swiotlb: move struct io_tlb_slot to swiotlb.c").

Use the opportunity to give swiotlb_do_find_slots() a more descriptive name and make it clear how it differs from swiotlb_find_slots().

---
 include/linux/swiotlb.h | 15 +++++++---
 kernel/dma/swiotlb.c    | 61 +++++++++++++++++++++++++++++++++++++----
 2 files changed, 66 insertions(+), 10 deletions(-)
include/linux/swiotlb.h +11 -4
index 2d453b3e7771..31625ae507ea 100644
@@ -76,10 +76,6 @@ dma_addr_t swiotlb_map(struct device *dev, phys_addr_t phys,
  * @nslabs:	The number of IO TLB blocks (in groups of 64) between @start and
  *		@end. For default swiotlb, this is command line adjustable via
  *		setup_io_tlb_npages.
- * @list:	The free list describing the number of free entries available
- *		from each index.
- * @orig_addr:	The original address corresponding to a mapped entry.
- * @alloc_size:	Size of the allocated buffer.
  * @debugfs:	The dentry to debugfs.
  * @late_alloc:	%true if allocated using the page allocator
  * @force_bounce: %true if swiotlb bouncing is forced
@@ -111,6 +107,17 @@ struct io_tlb_mem {
 #endif
 };
 
+/**
+ * is_swiotlb_buffer() - check if a physical address belongs to a swiotlb
+ * @dev:        Device which has mapped the buffer.
+ * @paddr:      Physical address within the DMA buffer.
+ *
+ * Check if @paddr points into a bounce buffer.
+ *
+ * Return:
+ * * %true if @paddr points into a bounce buffer
+ * * %false otherwise
+ */
 static inline bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr)
 {
 	struct io_tlb_mem *mem = dev->dma_io_tlb_mem;
kernel/dma/swiotlb.c +55 -6

98 lines not on this page · raw message

Signature · 1 line
2.25.1
#4 lore raw

PATCH v7 4/9 swiotlb: separate memory pool data from other allocator data

In reply to Petr Tesarik

From: Petr Tesarik <[email protected]>

Carve out memory pool specific fields from struct io_tlb_mem. The original struct now contains shared data for the whole allocator, while the new struct io_tlb_pool contains data that is specific to one memory pool of (potentially) many.

---
 include/linux/device.h  |   2 +-
 include/linux/swiotlb.h |  45 +++++++----
 kernel/dma/swiotlb.c    | 175 +++++++++++++++++++++++++---------------
 3 files changed, 140 insertions(+), 82 deletions(-)
include/linux/device.h +1 -1
index bbaeabd04b0d..d9754a68ba95 100644
@@ -625,7 +625,7 @@ struct device_physical_location {
  * @dma_pools:	Dma pools (if dma'ble device).
  * @dma_mem:	Internal for coherent mem override.
  * @cma_area:	Contiguous memory area for dma allocations
- * @dma_io_tlb_mem: Pointer to the swiotlb pool used.  Not for driver use.
+ * @dma_io_tlb_mem: Software IO TLB allocator.  Not for driver use.
  * @archdata:	For arch-specific additions.
  * @of_node:	Associated device tree node.
  * @fwnode:	Associated device node supplied by platform firmware.
include/linux/swiotlb.h +28 -17

76 lines not on this page · raw message

kernel/dma/swiotlb.c +111 -64

423 lines not on this page · raw message

Signature · 1 line
2.25.1
#5 lore raw

PATCH v7 5/9 swiotlb: add a flag whether SWIOTLB is allowed to grow

In reply to Petr Tesarik

From: Petr Tesarik <[email protected]>

Add a config option (CONFIG_SWIOTLB_DYNAMIC) to enable or disable dynamic allocation of additional bounce buffers.

If this option is set, mark the default SWIOTLB as able to grow and restricted DMA pools as unable.

However, if the address of the default memory pool is explicitly queried, make the default SWIOTLB also unable to grow. This is currently used to set up PCI BAR movable regions on some Octeon MIPS boards which may not be able to use a SWIOTLB pool elsewhere in physical memory. See octeon_pci_setup() for more details.

If a remap function is specified, it must be also called on any dynamically allocated pools, but there are some issues:

- The remap function may block, so it should not be called from an atomic context.
- There is no corresponding unremap() function if the memory pool is freed.
- The only in-tree implementation (xen_swiotlb_fixup) requires that the number of slots in the memory pool is a multiple of SWIOTLB_SEGSIZE.

Keep it simple for now and disable growing the SWIOTLB if a remap function was specified.

---
 include/linux/swiotlb.h |  4 ++++
 kernel/dma/Kconfig      | 13 +++++++++++++
 kernel/dma/swiotlb.c    | 13 +++++++++++++
 3 files changed, 30 insertions(+)
include/linux/swiotlb.h +4 -0
index 247f0ab8795a..57be2a0a9fbf 100644
@@ -100,6 +100,7 @@ struct io_tlb_pool {
  * @debugfs:	The dentry to debugfs.
  * @force_bounce: %true if swiotlb bouncing is forced
  * @for_alloc:  %true if the pool is used for memory allocation
+ * @can_grow:	%true if more pools can be allocated dynamically.
  * @total_used:	The total number of slots in the pool that are currently used
  *		across all areas. Used only for calculating used_hiwater in
  *		debugfs.
@@ -112,6 +113,9 @@ struct io_tlb_mem {
 	struct dentry *debugfs;
 	bool force_bounce;
 	bool for_alloc;
+#ifdef CONFIG_SWIOTLB_DYNAMIC
+	bool can_grow;
+#endif
 #ifdef CONFIG_DEBUG_FS
 	atomic_long_t total_used;
 	atomic_long_t used_hiwater;
kernel/dma/Kconfig +13 -0
index 562463fe30ea..4c1e9a3c0ab6 100644
@@ -90,6 +90,19 @@ config SWIOTLB
 	bool
 	select NEED_DMA_MAP_STATE
 
+config SWIOTLB_DYNAMIC
+	bool "Dynamic allocation of DMA bounce buffers"
+	default n
+	depends on SWIOTLB
+	help
+	  This enables dynamic resizing of the software IO TLB. The kernel
+	  starts with one memory pool at boot and it will allocate additional
+	  pools as needed. To reduce run-time kernel memory requirements, you
+	  may have to specify a smaller size of the initial pool using
+	  "swiotlb=" on the kernel command line.
+
+	  If unsure, say N.
+
 config DMA_BOUNCE_UNALIGNED_KMALLOC
 	bool
 	depends on SWIOTLB
kernel/dma/swiotlb.c +13 -0
index 6fc2606e014b..767c8fb36a6b 100644
@@ -330,6 +330,11 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags,
 	io_tlb_default_mem.force_bounce =
 		swiotlb_force_bounce || (flags & SWIOTLB_FORCE);
 
+#ifdef CONFIG_SWIOTLB_DYNAMIC
+	if (!remap)
+		io_tlb_default_mem.can_grow = true;
+#endif
+
 	if (!default_nareas)
 		swiotlb_adjust_nareas(num_possible_cpus());
 
@@ -400,6 +405,11 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask,
 
 	io_tlb_default_mem.force_bounce = swiotlb_force_bounce;
 
+#ifdef CONFIG_SWIOTLB_DYNAMIC
+	if (!remap)
+		io_tlb_default_mem.can_grow = true;
+#endif
+
 	if (!default_nareas)
 		swiotlb_adjust_nareas(num_possible_cpus());
 
@@ -1073,6 +1083,9 @@ bool is_swiotlb_active(struct device *dev)
  */
 phys_addr_t default_swiotlb_base(void)
 {
+#ifdef CONFIG_SWIOTLB_DYNAMIC
+	io_tlb_default_mem.can_grow = false;
+#endif
 	return io_tlb_default_mem.defpool.start;
 }
 
Signature · 1 line
2.25.1
#6 lore raw

PATCH v7 6/9 swiotlb: if swiotlb is full, fall back to a transient memory pool

In reply to Petr Tesarik

From: Petr Tesarik <[email protected]>

Try to allocate a transient memory pool if no suitable slots can be found and the respective SWIOTLB is allowed to grow. The transient pool is just enough big for this one bounce buffer. It is inserted into a per-device list of transient memory pools, and it is freed again when the bounce buffer is unmapped.

Transient memory pools are kept in an RCU list. A memory barrier is required after adding a new entry, because any address within a transient buffer must be immediately recognized as belonging to the SWIOTLB, even if it is passed to another CPU.

Deletion does not require any synchronization beyond RCU ordering guarantees. After a buffer is unmapped, its physical addresses may no longer be passed to the DMA API, so the memory range of the corresponding stale entry in the RCU list never matches. If the memory range gets allocated again, then it happens only after a RCU quiescent state.

Since bounce buffers can now be allocated from different pools, add a parameter to swiotlb_alloc_pool() to let the caller know which memory pool is used. Add swiotlb_find_pool() to find the memory pool corresponding to an address. This function is now also used by is_swiotlb_buffer(), because a simple boundary check is no longer sufficient.

The logic in swiotlb_alloc_tlb() is taken from __dma_direct_alloc_pages(), simplified and enhanced to use coherent memory pools if needed.

Note that this is not the most efficient way to provide a bounce buffer, but when a DMA buffer can't be mapped, something may (and will) actually break. At that point it is better to make an allocation, even if it may be an expensive operation.

---
 include/linux/device.h      |   6 +
 include/linux/dma-mapping.h |   2 +
 include/linux/swiotlb.h     |  29 +++-
 kernel/dma/direct.c         |   2 +-
 kernel/dma/swiotlb.c        | 316 +++++++++++++++++++++++++++++++++++-
 5 files changed, 345 insertions(+), 10 deletions(-)
include/linux/device.h +6 -0
index d9754a68ba95..5fd89c9d005c 100644
@@ -626,6 +626,8 @@ struct device_physical_location {
  * @dma_mem:	Internal for coherent mem override.
  * @cma_area:	Contiguous memory area for dma allocations
  * @dma_io_tlb_mem: Software IO TLB allocator.  Not for driver use.
+ * @dma_io_tlb_pools:	List of transient swiotlb memory pools.
+ * @dma_io_tlb_lock:	Protects changes to the list of active pools.
  * @archdata:	For arch-specific additions.
  * @of_node:	Associated device tree node.
  * @fwnode:	Associated device node supplied by platform firmware.
@@ -731,6 +733,10 @@ struct device {
 #endif
 #ifdef CONFIG_SWIOTLB
 	struct io_tlb_mem *dma_io_tlb_mem;
+#endif
+#ifdef CONFIG_SWIOTLB_DYNAMIC
+	struct list_head dma_io_tlb_pools;
+	spinlock_t dma_io_tlb_lock;
 #endif
 	/* arch specific additions */
 	struct dev_archdata	archdata;
include/linux/dma-mapping.h +2 -0
index e13050eb9777..f0ccca16a0ac 100644
@@ -418,6 +418,8 @@ static inline void dma_sync_sgtable_for_device(struct device *dev,
 #define dma_get_sgtable(d, t, v, h, s) dma_get_sgtable_attrs(d, t, v, h, s, 0)
 #define dma_mmap_coherent(d, v, c, h, s) dma_mmap_attrs(d, v, c, h, s, 0)
 
+bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size);
+
 static inline void *dma_alloc_coherent(struct device *dev, size_t size,
 		dma_addr_t *dma_handle, gfp_t gfp)
 {
include/linux/swiotlb.h +28 -1
index 57be2a0a9fbf..66867d2188ba 100644
@@ -80,6 +80,9 @@ dma_addr_t swiotlb_map(struct device *dev, phys_addr_t phys,
  * @area_nslabs: Number of slots in each area.
  * @areas:	Array of memory area descriptors.
  * @slots:	Array of slot descriptors.
+ * @node:	Member of the IO TLB memory pool list.
+ * @rcu:	RCU head for swiotlb_dyn_free().
+ * @transient:  %true if transient memory pool.
  */
 struct io_tlb_pool {
 	phys_addr_t start;
@@ -91,6 +94,11 @@ struct io_tlb_pool {
 	unsigned int area_nslabs;
 	struct io_tlb_area *areas;
 	struct io_tlb_slot *slots;
+#ifdef CONFIG_SWIOTLB_DYNAMIC
+	struct list_head node;
+	struct rcu_head rcu;
+	bool transient;
+#endif
 };
 
 /**
@@ -122,6 +130,20 @@ struct io_tlb_mem {
 #endif
 };
 
+#ifdef CONFIG_SWIOTLB_DYNAMIC
+
+struct io_tlb_pool *swiotlb_find_pool(struct device *dev, phys_addr_t paddr);
+
+#else
+
+static inline struct io_tlb_pool *swiotlb_find_pool(struct device *dev,
+						    phys_addr_t paddr)
+{
+	return &dev->dma_io_tlb_mem->defpool;
+}
+
+#endif
+
 /**
  * is_swiotlb_buffer() - check if a physical address belongs to a swiotlb
  * @dev:        Device which has mapped the buffer.
@@ -137,7 +159,12 @@ static inline bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr)
 {
 	struct io_tlb_mem *mem = dev->dma_io_tlb_mem;
 
-	return mem && paddr >= mem->defpool.start && paddr < mem->defpool.end;
+	if (!mem)
+		return false;
+
+	if (IS_ENABLED(CONFIG_SWIOTLB_DYNAMIC))
+		return swiotlb_find_pool(dev, paddr);
+	return paddr >= mem->defpool.start && paddr < mem->defpool.end;
 }
 
 static inline bool is_swiotlb_force_bounce(struct device *dev)
kernel/dma/direct.c +1 -1
index d29cade048db..9596ae1aa0da 100644
@@ -66,7 +66,7 @@ static gfp_t dma_direct_optimal_gfp_mask(struct device *dev, u64 *phys_limit)
 	return 0;
 }
 
-static bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size)
+bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size)
 {
 	dma_addr_t dma_addr = phys_to_dma_direct(dev, phys);
 
kernel/dma/swiotlb.c +308 -8

401 lines not on this page · raw message

Signature · 1 line
2.25.1
#7 lore raw

PATCH v7 7/9 swiotlb: determine potential physical address limit

In reply to Petr Tesarik

From: Petr Tesarik <[email protected]>

The value returned by default_swiotlb_limit() should be constant, because it is used to decide whether DMA can be used. To allow allocating memory pools on the fly, use the maximum possible physical address rather than the highest address used by the default pool.

For swiotlb_init_remap(), this is either an arch-specific limit used by memblock_alloc_low(), or the highest directly mapped physical address if the initialization flags include SWIOTLB_ANY. For swiotlb_init_late(), the highest address is determined by the GFP flags.

---
 include/linux/swiotlb.h |  2 ++
 kernel/dma/swiotlb.c    | 14 ++++++++++++++
 2 files changed, 16 insertions(+)
include/linux/swiotlb.h +2 -0
index 66867d2188ba..9825fa14abe4 100644
@@ -109,6 +109,7 @@ struct io_tlb_pool {
  * @force_bounce: %true if swiotlb bouncing is forced
  * @for_alloc:  %true if the pool is used for memory allocation
  * @can_grow:	%true if more pools can be allocated dynamically.
+ * @phys_limit:	Maximum allowed physical address.
  * @total_used:	The total number of slots in the pool that are currently used
  *		across all areas. Used only for calculating used_hiwater in
  *		debugfs.
@@ -123,6 +124,7 @@ struct io_tlb_mem {
 	bool for_alloc;
 #ifdef CONFIG_SWIOTLB_DYNAMIC
 	bool can_grow;
+	u64 phys_limit;
 #endif
 #ifdef CONFIG_DEBUG_FS
 	atomic_long_t total_used;
kernel/dma/swiotlb.c +14 -0
index 30d0fcc3ccb9..0fa081defdbd 100644
@@ -334,6 +334,10 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags,
 #ifdef CONFIG_SWIOTLB_DYNAMIC
 	if (!remap)
 		io_tlb_default_mem.can_grow = true;
+	if (flags & SWIOTLB_ANY)
+		io_tlb_default_mem.phys_limit = virt_to_phys(high_memory - 1);
+	else
+		io_tlb_default_mem.phys_limit = ARCH_LOW_ADDRESS_LIMIT;
 #endif
 
 	if (!default_nareas)
@@ -409,6 +413,12 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask,
 #ifdef CONFIG_SWIOTLB_DYNAMIC
 	if (!remap)
 		io_tlb_default_mem.can_grow = true;
+	if (IS_ENABLED(CONFIG_ZONE_DMA) && (gfp_mask & __GFP_DMA))
+		io_tlb_default_mem.phys_limit = DMA_BIT_MASK(zone_dma_bits);
+	else if (IS_ENABLED(CONFIG_ZONE_DMA32) && (gfp_mask & __GFP_DMA32))
+		io_tlb_default_mem.phys_limit = DMA_BIT_MASK(32);
+	else
+		io_tlb_default_mem.phys_limit = virt_to_phys(high_memory - 1);
 #endif
 
 	if (!default_nareas)
@@ -1397,7 +1407,11 @@ phys_addr_t default_swiotlb_base(void)
  */
 phys_addr_t default_swiotlb_limit(void)
 {
+#ifdef CONFIG_SWIOTLB_DYNAMIC
+	return io_tlb_default_mem.phys_limit;
+#else
 	return io_tlb_default_mem.defpool.end - 1;
+#endif
 }
 
 #ifdef CONFIG_DEBUG_FS
Signature · 1 line
2.25.1
#8 lore raw

PATCH v7 8/9 swiotlb: allocate a new memory pool when existing pools are full

In reply to Petr Tesarik

From: Petr Tesarik <[email protected]>

When swiotlb_find_slots() cannot find suitable slots, schedule the allocation of a new memory pool. It is not possible to allocate the pool immediately, because this code may run in interrupt context, which is not suitable for large memory allocations. This means that the memory pool will be available too late for the currently requested mapping, but the stress on the software IO TLB allocator is likely to continue, and subsequent allocations will benefit from the additional pool eventually.

Keep all memory pools for an allocator in an RCU list to avoid locking on the read side. For modifications, add a new spinlock to struct io_tlb_mem.

The spinlock also protects updates to the total number of slabs (nslabs in struct io_tlb_mem), but not reads of the value. Readers may therefore encounter a stale value, but this is not an issue:

- swiotlb_tbl_map_single() and is_swiotlb_active() only check for non-zero value. This is ensured by the existence of the default memory pool, allocated at boot.

- The exact value is used only for non-critical purposes (debugfs, kernel messages).

---
 include/linux/swiotlb.h |   8 +++
 kernel/dma/swiotlb.c    | 148 +++++++++++++++++++++++++++++++++-------
 2 files changed, 131 insertions(+), 25 deletions(-)
include/linux/swiotlb.h +8 -0
index 9825fa14abe4..8371c92a0271 100644
@@ -8,6 +8,7 @@
 #include <linux/types.h>
 #include <linux/limits.h>
 #include <linux/spinlock.h>
+#include <linux/workqueue.h>
 
 struct device;
 struct page;
@@ -104,12 +105,16 @@ struct io_tlb_pool {
 /**
  * struct io_tlb_mem - Software IO TLB allocator
  * @defpool:	Default (initial) IO TLB memory pool descriptor.
+ * @pool:	IO TLB memory pool descriptor (if not dynamic).
  * @nslabs:	Total number of IO TLB slabs in all pools.
  * @debugfs:	The dentry to debugfs.
  * @force_bounce: %true if swiotlb bouncing is forced
  * @for_alloc:  %true if the pool is used for memory allocation
  * @can_grow:	%true if more pools can be allocated dynamically.
  * @phys_limit:	Maximum allowed physical address.
+ * @lock:	Lock to synchronize changes to the list.
+ * @pools:	List of IO TLB memory pool descriptors (if dynamic).
+ * @dyn_alloc:	Dynamic IO TLB pool allocation work.
  * @total_used:	The total number of slots in the pool that are currently used
  *		across all areas. Used only for calculating used_hiwater in
  *		debugfs.
@@ -125,6 +130,9 @@ struct io_tlb_mem {
 #ifdef CONFIG_SWIOTLB_DYNAMIC
 	bool can_grow;
 	u64 phys_limit;
+	spinlock_t lock;
+	struct list_head pools;
+	struct work_struct dyn_alloc;
 #endif
 #ifdef CONFIG_DEBUG_FS
 	atomic_long_t total_used;
kernel/dma/swiotlb.c +123 -25

250 lines not on this page · raw message

Signature · 1 line
2.25.1
#9 lore raw

PATCH v7 9/9 swiotlb: search the software IO TLB only if the device makes use of it

In reply to Petr Tesarik

From: Petr Tesarik <[email protected]>

Skip searching the software IO TLB if a device has never used it, making sure these devices are not affected by the introduction of multiple IO TLB memory pools.

Additional memory barrier is required to ensure that the new value of the flag is visible to other CPUs after mapping a new bounce buffer. For efficiency, the flag check should be inlined, and then the memory barrier must be moved to is_swiotlb_buffer(). However, it can replace the existing barrier in swiotlb_find_pool(), because all callers use is_swiotlb_buffer() first to verify that the buffer address belongs to the software IO TLB.

---
 include/linux/device.h  |  2 ++
 include/linux/swiotlb.h |  7 ++++++-
 kernel/dma/swiotlb.c    | 14 ++++++--------
 3 files changed, 14 insertions(+), 9 deletions(-)
include/linux/device.h +2 -0
index 5fd89c9d005c..6fc808d22bfd 100644
@@ -628,6 +628,7 @@ struct device_physical_location {
  * @dma_io_tlb_mem: Software IO TLB allocator.  Not for driver use.
  * @dma_io_tlb_pools:	List of transient swiotlb memory pools.
  * @dma_io_tlb_lock:	Protects changes to the list of active pools.
+ * @dma_uses_io_tlb: %true if device has used the software IO TLB.
  * @archdata:	For arch-specific additions.
  * @of_node:	Associated device tree node.
  * @fwnode:	Associated device node supplied by platform firmware.
@@ -737,6 +738,7 @@ struct device {
 #ifdef CONFIG_SWIOTLB_DYNAMIC
 	struct list_head dma_io_tlb_pools;
 	spinlock_t dma_io_tlb_lock;
+	bool dma_uses_io_tlb;
 #endif
 	/* arch specific additions */
 	struct dev_archdata	archdata;
include/linux/swiotlb.h +6 -1
index 8371c92a0271..b4536626f8ff 100644
@@ -172,8 +172,13 @@ static inline bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr)
 	if (!mem)
 		return false;
 
-	if (IS_ENABLED(CONFIG_SWIOTLB_DYNAMIC))
+	if (IS_ENABLED(CONFIG_SWIOTLB_DYNAMIC)) {
+		/* Pairs with smp_wmb() in swiotlb_find_slots() and
+		 * swiotlb_dyn_alloc(), which modify the RCU lists.
+		 */
+		smp_rmb();
 		return swiotlb_find_pool(dev, paddr);
+	}
 	return paddr >= mem->defpool.start && paddr < mem->defpool.end;
 }
 
kernel/dma/swiotlb.c +6 -8
index adf80dec42d7..d7eac84f975b 100644
@@ -730,7 +730,7 @@ static void swiotlb_dyn_alloc(struct work_struct *work)
 
 	add_mem_pool(mem, pool);
 
-	/* Pairs with smp_rmb() in swiotlb_find_pool(). */
+	/* Pairs with smp_rmb() in is_swiotlb_buffer(). */
 	smp_wmb();
 }
 
@@ -764,11 +764,6 @@ struct io_tlb_pool *swiotlb_find_pool(struct device *dev, phys_addr_t paddr)
 	struct io_tlb_mem *mem = dev->dma_io_tlb_mem;
 	struct io_tlb_pool *pool;
 
-	/* Pairs with smp_wmb() in swiotlb_find_slots() and
-	 * swiotlb_dyn_alloc(), which modify the RCU lists.
-	 */
-	smp_rmb();
-
 	rcu_read_lock();
 	list_for_each_entry_rcu(pool, &mem->pools, node) {
 		if (paddr >= pool->start && paddr < pool->end)
@@ -813,6 +808,7 @@ void swiotlb_dev_init(struct device *dev)
 #ifdef CONFIG_SWIOTLB_DYNAMIC
 	INIT_LIST_HEAD(&dev->dma_io_tlb_pools);
 	spin_lock_init(&dev->dma_io_tlb_lock);
+	dev->dma_uses_io_tlb = false;
 #endif
 }
 
@@ -1157,9 +1153,11 @@ static int swiotlb_find_slots(struct device *dev, phys_addr_t orig_addr,
 	list_add_rcu(&pool->node, &dev->dma_io_tlb_pools);
 	spin_unlock_irqrestore(&dev->dma_io_tlb_lock, flags);
 
-	/* Pairs with smp_rmb() in swiotlb_find_pool(). */
-	smp_wmb();
 found:
+	dev->dma_uses_io_tlb = true;
+	/* Pairs with smp_rmb() in is_swiotlb_buffer() */
+	smp_wmb();
+
 	*retpool = pool;
 	return index;
 }
Signature · 1 line
2.25.1
#10 lore raw

Re: [PATCH v7 9/9] swiotlb: search the software IO TLB only if the device makes use of it

Quoted 13 lines · Petr Tesarik

From: Petr Tesarik <[email protected]>

Skip searching the software IO TLB if a device has never used it, making sure these devices are not affected by the introduction of multiple IO TLB memory pools.

Additional memory barrier is required to ensure that the new value of the flag is visible to other CPUs after mapping a new bounce buffer. For efficiency, the flag check should be inlined, and then the memory barrier must be moved to is_swiotlb_buffer(). However, it can replace the existing barrier in swiotlb_find_pool(), because all callers use is_swiotlb_buffer() first to verify that the buffer address belongs to the software IO TLB.

Quoted 2 lines

---

Excuse me if this is a silly question, but I'm not able to figure it out on my own...

Quoted 18 lines
 include/linux/device.h  |  2 ++
 include/linux/swiotlb.h |  7 ++++++-
 kernel/dma/swiotlb.c    | 14 ++++++--------
 3 files changed, 14 insertions(+), 9 deletions(-)
include/linux/device.h +1 -0
index 5fd89c9d005c..6fc808d22bfd 100644
@@ -628,6 +628,7 @@ struct device_physical_location {
  * @dma_io_tlb_mem: Software IO TLB allocator.  Not for driver use.
  * @dma_io_tlb_pools:	List of transient swiotlb memory pools.
  * @dma_io_tlb_lock:	Protects changes to the list of active pools.
+ * @dma_uses_io_tlb: %true if device has used the software IO TLB.
  * @archdata:	For arch-specific additions.
  * @of_node:	Associated device tree node.
  * @fwnode:	Associated device node supplied by platform firmware.

include/linux/device.h struct device { · line 741

Quoted 4 lines
include/linux/device.h +1 -0
@@ -737,6 +738,7 @@ struct device {
 #ifdef CONFIG_SWIOTLB_DYNAMIC
 	struct list_head dma_io_tlb_pools;
 	spinlock_t dma_io_tlb_lock;
+	bool dma_uses_io_tlb;

You add this new member here, fine...

Quoted 48 lines
include/linux/device.h +0 -0
 #endif
 	/* arch specific additions */
 	struct dev_archdata	archdata;
include/linux/swiotlb.h +6 -1
index 8371c92a0271..b4536626f8ff 100644
@@ -172,8 +172,13 @@ static inline bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr)
 	if (!mem)
 		return false;
 
-	if (IS_ENABLED(CONFIG_SWIOTLB_DYNAMIC))
+	if (IS_ENABLED(CONFIG_SWIOTLB_DYNAMIC)) {
+		/* Pairs with smp_wmb() in swiotlb_find_slots() and
+		 * swiotlb_dyn_alloc(), which modify the RCU lists.
+		 */
+		smp_rmb();
 		return swiotlb_find_pool(dev, paddr);
+	}
 	return paddr >= mem->defpool.start && paddr < mem->defpool.end;
 }
 
kernel/dma/swiotlb.c +1 -6
index adf80dec42d7..d7eac84f975b 100644
@@ -730,7 +730,7 @@ static void swiotlb_dyn_alloc(struct work_struct *work)
 
 	add_mem_pool(mem, pool);
 
-	/* Pairs with smp_rmb() in swiotlb_find_pool(). */
+	/* Pairs with smp_rmb() in is_swiotlb_buffer(). */
 	smp_wmb();
 }
 
@@ -764,11 +764,6 @@ struct io_tlb_pool *swiotlb_find_pool(struct device *dev, phys_addr_t paddr)
 	struct io_tlb_mem *mem = dev->dma_io_tlb_mem;
 	struct io_tlb_pool *pool;
 
-	/* Pairs with smp_wmb() in swiotlb_find_slots() and
-	 * swiotlb_dyn_alloc(), which modify the RCU lists.
-	 */
-	smp_rmb();
-
 	rcu_read_lock();
 	list_for_each_entry_rcu(pool, &mem->pools, node) {
 		if (paddr >= pool->start && paddr < pool->end)

kernel/dma/swiotlb.c void swiotlb_dev_init(struct device *dev) · line 811

Quoted 4 lines
kernel/dma/swiotlb.c +1 -0
@@ -813,6 +808,7 @@ void swiotlb_dev_init(struct device *dev)
 #ifdef CONFIG_SWIOTLB_DYNAMIC
 	INIT_LIST_HEAD(&dev->dma_io_tlb_pools);
 	spin_lock_init(&dev->dma_io_tlb_lock);
+	dev->dma_uses_io_tlb = false;

...here you initialize it, fine...

Quoted 4 lines
kernel/dma/swiotlb.c +0 -0
 #endif
 }
 

kernel/dma/swiotlb.c static int swiotlb_find_slots(struct device *dev, phys_addr_t orig_addr, · line 1160

Quoted 10 lines
kernel/dma/swiotlb.c +4 -2
@@ -1157,9 +1153,11 @@ static int swiotlb_find_slots(struct device *dev, phys_addr_t orig_addr,
 	list_add_rcu(&pool->node, &dev->dma_io_tlb_pools);
 	spin_unlock_irqrestore(&dev->dma_io_tlb_lock, flags);
 
-	/* Pairs with smp_rmb() in swiotlb_find_pool(). */
-	smp_wmb();
 found:
+	dev->dma_uses_io_tlb = true;
+	/* Pairs with smp_rmb() in is_swiotlb_buffer() */
+	smp_wmb();
+

...and here you set it if swiotlb is used.

But, as far as I can tell, you don't actually *use* this field anywhere. What am I missing?

Thanks,

jon

#11 lore raw

Re: [PATCH v7 9/9] swiotlb: search the software IO TLB only if the device makes use of it

So it seems this code got merged without this question ever being answered. Sorry if it's a dumb one, but I don't think this functionality works as advertised...

Thanks,

jon

Quoted 124 lines · Jonathan Corbet
Quoted 13 lines · Petr Tesarik

From: Petr Tesarik <[email protected]>

Skip searching the software IO TLB if a device has never used it, making sure these devices are not affected by the introduction of multiple IO TLB memory pools.

Additional memory barrier is required to ensure that the new value of the flag is visible to other CPUs after mapping a new bounce buffer. For efficiency, the flag check should be inlined, and then the memory barrier must be moved to is_swiotlb_buffer(). However, it can replace the existing barrier in swiotlb_find_pool(), because all callers use is_swiotlb_buffer() first to verify that the buffer address belongs to the software IO TLB.

Quoted 2 lines

---

Excuse me if this is a silly question, but I'm not able to figure it out on my own...

Quoted 18 lines
 include/linux/device.h  |  2 ++
 include/linux/swiotlb.h |  7 ++++++-
 kernel/dma/swiotlb.c    | 14 ++++++--------
 3 files changed, 14 insertions(+), 9 deletions(-)
include/linux/device.h +1 -0
index 5fd89c9d005c..6fc808d22bfd 100644
@@ -628,6 +628,7 @@ struct device_physical_location {
  * @dma_io_tlb_mem: Software IO TLB allocator.  Not for driver use.
  * @dma_io_tlb_pools:	List of transient swiotlb memory pools.
  * @dma_io_tlb_lock:	Protects changes to the list of active pools.
+ * @dma_uses_io_tlb: %true if device has used the software IO TLB.
  * @archdata:	For arch-specific additions.
  * @of_node:	Associated device tree node.
  * @fwnode:	Associated device node supplied by platform firmware.

include/linux/device.h struct device { · line 741

Quoted 4 lines
include/linux/device.h +1 -0
@@ -737,6 +738,7 @@ struct device {
 #ifdef CONFIG_SWIOTLB_DYNAMIC
 	struct list_head dma_io_tlb_pools;
 	spinlock_t dma_io_tlb_lock;
+	bool dma_uses_io_tlb;

You add this new member here, fine...

Quoted 48 lines
include/linux/device.h +0 -0
 #endif
 	/* arch specific additions */
 	struct dev_archdata	archdata;
include/linux/swiotlb.h +6 -1
index 8371c92a0271..b4536626f8ff 100644
@@ -172,8 +172,13 @@ static inline bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr)
 	if (!mem)
 		return false;
 
-	if (IS_ENABLED(CONFIG_SWIOTLB_DYNAMIC))
+	if (IS_ENABLED(CONFIG_SWIOTLB_DYNAMIC)) {
+		/* Pairs with smp_wmb() in swiotlb_find_slots() and
+		 * swiotlb_dyn_alloc(), which modify the RCU lists.
+		 */
+		smp_rmb();
 		return swiotlb_find_pool(dev, paddr);
+	}
 	return paddr >= mem->defpool.start && paddr < mem->defpool.end;
 }
 
kernel/dma/swiotlb.c +1 -6
index adf80dec42d7..d7eac84f975b 100644
@@ -730,7 +730,7 @@ static void swiotlb_dyn_alloc(struct work_struct *work)
 
 	add_mem_pool(mem, pool);
 
-	/* Pairs with smp_rmb() in swiotlb_find_pool(). */
+	/* Pairs with smp_rmb() in is_swiotlb_buffer(). */
 	smp_wmb();
 }
 
@@ -764,11 +764,6 @@ struct io_tlb_pool *swiotlb_find_pool(struct device *dev, phys_addr_t paddr)
 	struct io_tlb_mem *mem = dev->dma_io_tlb_mem;
 	struct io_tlb_pool *pool;
 
-	/* Pairs with smp_wmb() in swiotlb_find_slots() and
-	 * swiotlb_dyn_alloc(), which modify the RCU lists.
-	 */
-	smp_rmb();
-
 	rcu_read_lock();
 	list_for_each_entry_rcu(pool, &mem->pools, node) {
 		if (paddr >= pool->start && paddr < pool->end)

kernel/dma/swiotlb.c void swiotlb_dev_init(struct device *dev) · line 811

Quoted 4 lines
kernel/dma/swiotlb.c +1 -0
@@ -813,6 +808,7 @@ void swiotlb_dev_init(struct device *dev)
 #ifdef CONFIG_SWIOTLB_DYNAMIC
 	INIT_LIST_HEAD(&dev->dma_io_tlb_pools);
 	spin_lock_init(&dev->dma_io_tlb_lock);
+	dev->dma_uses_io_tlb = false;

...here you initialize it, fine...

Quoted 4 lines
kernel/dma/swiotlb.c +0 -0
 #endif
 }
 

kernel/dma/swiotlb.c static int swiotlb_find_slots(struct device *dev, phys_addr_t orig_addr, · line 1160

Quoted 10 lines
kernel/dma/swiotlb.c +4 -2
@@ -1157,9 +1153,11 @@ static int swiotlb_find_slots(struct device *dev, phys_addr_t orig_addr,
 	list_add_rcu(&pool->node, &dev->dma_io_tlb_pools);
 	spin_unlock_irqrestore(&dev->dma_io_tlb_lock, flags);
 
-	/* Pairs with smp_rmb() in swiotlb_find_pool(). */
-	smp_wmb();
 found:
+	dev->dma_uses_io_tlb = true;
+	/* Pairs with smp_rmb() in is_swiotlb_buffer() */
+	smp_wmb();
+

...and here you set it if swiotlb is used.

But, as far as I can tell, you don't actually *use* this field anywhere. What am I missing?

Thanks,

jon

#12 lore raw

Re: [PATCH v7 9/9] swiotlb: search the software IO TLB only if the device makes use of it

Hi all,

sorry for my late reply; I've been away from my work setup for a month...

Quoted 3 lines · On Wed, 30 Aug 2023 08:55:51 -0600 Jonathan Corbet <[email protected]> wrote:

So it seems this code got merged without this question ever being answered. Sorry if it's a dumb one, but I don't think this functionality works as advertised...

Yes, I believe the check was originally in is_swiotlb_buffer(), but it got lost during one of the numerous rebases of this patch set. Let me send a follow-up patch after making sure it actually works.

Petr T

Quoted 136 lines

Thanks,

jon

Quoted 130 lines · Jonathan Corbet
Quoted 14 lines · Petr Tesarik

From: Petr Tesarik <[email protected]>

Skip searching the software IO TLB if a device has never used it, making sure these devices are not affected by the introduction of multiple IO TLB memory pools.

Additional memory barrier is required to ensure that the new value of the flag is visible to other CPUs after mapping a new bounce buffer. For efficiency, the flag check should be inlined, and then the memory barrier must be moved to is_swiotlb_buffer(). However, it can replace the existing barrier in swiotlb_find_pool(), because all callers use is_swiotlb_buffer() first to verify that the buffer address belongs to the software IO TLB.

Quoted 2 lines

---

Excuse me if this is a silly question, but I'm not able to figure it out on my own...

Quoted 22 lines
 include/linux/device.h  |  2 ++
 include/linux/swiotlb.h |  7 ++++++-
 kernel/dma/swiotlb.c    | 14 ++++++--------
 3 files changed, 14 insertions(+), 9 deletions(-)
include/linux/device.h +0 -0
index 5fd89c9d005c..6fc808d22bfd 100644
@@ -628,6 +628,7 @@ struct device_physical_location {
  * @dma_io_tlb_mem: Software IO TLB allocator.  Not for driver

use.
* @dma_io_tlb_pools: List of transient swiotlb memory
pools.
* @dma_io_tlb_lock: Protects changes to the list of
active pools.
+ * @dma_uses_io_tlb: %true if device has used the software IO TLB.
* @archdata: For arch-specific additions.
* @of_node: Associated device tree node.
* @fwnode: Associated device node supplied by platform
firmware. @@ -737,6 +738,7 @@ struct device {
#ifdef CONFIG_SWIOTLB_DYNAMIC

include/linux/device.h struct device { · line 741

Quoted 3 lines
include/linux/device.h +1 -0
 	struct list_head dma_io_tlb_pools;
 	spinlock_t dma_io_tlb_lock;
+	bool dma_uses_io_tlb;  

You add this new member here, fine...

Quoted 50 lines

#endif

include/linux/device.h +0 -0
 	/* arch specific additions */
 	struct dev_archdata	archdata;
include/linux/swiotlb.h +2 -1
index 8371c92a0271..b4536626f8ff 100644
@@ -172,8 +172,13 @@ static inline bool is_swiotlb_buffer(struct
 device *dev, phys_addr_t paddr) if (!mem)
 		return false;
 
-	if (IS_ENABLED(CONFIG_SWIOTLB_DYNAMIC))
+	if (IS_ENABLED(CONFIG_SWIOTLB_DYNAMIC)) {
+		/* Pairs with smp_wmb() in swiotlb_find_slots()

and

include/linux/swiotlb.h +1 -0
+		 * swiotlb_dyn_alloc(), which modify the RCU

lists.

include/linux/swiotlb.h +3 -0
+		 */
+		smp_rmb();
 		return swiotlb_find_pool(dev, paddr);
+	}
 	return paddr >= mem->defpool.start && paddr <

mem->defpool.end; }

kernel/dma/swiotlb.c +1 -1
index adf80dec42d7..d7eac84f975b 100644
@@ -730,7 +730,7 @@ static void swiotlb_dyn_alloc(struct
 work_struct *work) 
 	add_mem_pool(mem, pool);
 
-	/* Pairs with smp_rmb() in swiotlb_find_pool(). */
+	/* Pairs with smp_rmb() in is_swiotlb_buffer(). */
 	smp_wmb();
 }
 
@@ -764,11 +764,6 @@ struct io_tlb_pool *swiotlb_find_pool(struct

device *dev, phys_addr_t paddr) struct io_tlb_mem *mem = dev->dma_io_tlb_mem; struct io_tlb_pool *pool;

kernel/dma/swiotlb.c +0 -5
-	/* Pairs with smp_wmb() in swiotlb_find_slots() and
-	 * swiotlb_dyn_alloc(), which modify the RCU lists.
-	 */
-	smp_rmb();
-
 	rcu_read_lock();
 	list_for_each_entry_rcu(pool, &mem->pools, node) {
 		if (paddr >= pool->start && paddr < pool->end)

kernel/dma/swiotlb.c void swiotlb_dev_init(struct device *dev) · line 811

Quoted 4 lines
kernel/dma/swiotlb.c +1 -0
@@ -813,6 +808,7 @@ void swiotlb_dev_init(struct device *dev)
 #ifdef CONFIG_SWIOTLB_DYNAMIC
 	INIT_LIST_HEAD(&dev->dma_io_tlb_pools);
 	spin_lock_init(&dev->dma_io_tlb_lock);
+	dev->dma_uses_io_tlb = false;  

...here you initialize it, fine...

Quoted 4 lines
kernel/dma/swiotlb.c +0 -0
 #endif
 }
 

kernel/dma/swiotlb.c static int swiotlb_find_slots(struct device · line 1160

Quoted 10 lines
kernel/dma/swiotlb.c +4 -2
@@ -1157,9 +1153,11 @@ static int swiotlb_find_slots(struct device
 *dev, phys_addr_t orig_addr, list_add_rcu(&pool->node,
 &dev->dma_io_tlb_pools);
 spin_unlock_irqrestore(&dev->dma_io_tlb_lock, flags); 
-	/* Pairs with smp_rmb() in swiotlb_find_pool(). */
-	smp_wmb();
 found:
+	dev->dma_uses_io_tlb = true;
+	/* Pairs with smp_rmb() in is_swiotlb_buffer() */
+	smp_wmb();
+  

...and here you set it if swiotlb is used.

But, as far as I can tell, you don't actually *use* this field anywhere. What am I missing?

Thanks,

jon

#14 lore raw

Re: [PATCH v7 9/9] swiotlb: search the software IO TLB only if the device makes use of it

In reply to Jonathan Corbet
Quoted 12 lines · Jonathan Corbet

kernel/dma/swiotlb.c static int swiotlb_find_slots(struct device *dev, phys_addr_t orig_addr, · line 1160

Quoted 9 lines
kernel/dma/swiotlb.c +4 -2
 	spin_unlock_irqrestore(&dev->dma_io_tlb_lock, flags);
 
-	/* Pairs with smp_rmb() in swiotlb_find_pool(). */
-	smp_wmb();
 found:
+	dev->dma_uses_io_tlb = true;
+	/* Pairs with smp_rmb() in is_swiotlb_buffer() */
+	smp_wmb();
+

...and here you set it if swiotlb is used.

Quoted 2 lines

But, as far as I can tell, you don't actually *use* this field anywhere. What am I missing?

It's very much unused. Petr, I guess you wanted to use this in is_swiotlb_buffer to avoid the lookup unless required. Can you send a follow up?

maintainer Reviewed-by on the v7 series
#15 lore raw
In reply to Petr Tesarik
Quoted 168 lines · Petr Tesarik

From: Petr Tesarik <[email protected]>

Motivation
==========

The software IO TLB was designed with these assumptions:

1) It would not be used much. Small systems (little RAM) don't need it, and big systems (lots of RAM) would have modern DMA controllers and an IOMMU chip to handle legacy devices.
2) A small fixed memory area (64 MiB by default) is sufficient to handle the few cases which require a bounce buffer.
3) 64 MiB is little enough that it has no impact on the rest of the system.
4) Bounce buffers require large contiguous chunks of low memory. Such memory is precious and can be allocated only early at boot.

It turns out they are not always true:

1) Embedded systems may have more than 4GiB RAM but no IOMMU and legacy 32-bit peripheral busses and/or DMA controllers.
2) CoCo VMs use bounce buffers for all I/O but may need substantially more than 64 MiB.
3) Embedded developers put as many features as possible into the available memory. A few dozen "missing" megabytes may limit what features can be implemented.
4) If CMA is available, it can allocate large continuous chunks even after the system has run for some time.

Goals
=====

The goal of this work is to start with a small software IO TLB at boot and expand it later when/if needed.

Design
======

This version of the patch series retains the current slot allocation algorithm with multiple areas to reduce lock contention, but additional slots can be added when necessary.

These alternatives have been considered:

- Allocate and free buffers as needed using direct DMA API. This works quite well, except in CoCo VMs where each allocation/free requires decrypting/encrypting memory, which is a very expensive operation.

- Allocate a very large software IO TLB at boot, but allow to migrate pages to/from it (like CMA does). For systems with CMA, this would mean two big allocations at boot. Finding the balance between CMA, SWIOTLB and rest of available RAM can be challenging. More importantly, there is no clear benefit compared to allocating SWIOTLB memory pools from the CMA.

Implementation Constraints
==========================

These constraints have been taken into account:

1) Minimize impact on devices which do not benefit from the change.
2) Minimize the number of memory decryption/encryption operations.
3) Avoid contention on a lock or atomic variable to preserve parallel scalability.

Additionally, the software IO TLB code is also used to implement restricted DMA pools. These pools are restricted to a pre-defined physical memory region and must not use any other memory. In other words, dynamic allocation of memory pools must be disabled for restricted DMA pools.

Data Structures
===============

The existing struct io_tlb_mem is the central type for a SWIOTLB allocator, but it now contains multiple memory pools::

io_tlb_mem

  +---------+   io_tlb_pool
  | SWIOTLB | 	+-------+   +-------+   +-------+
  |allocator|-->|default|-->|dynamic|-->|dynamic|-->...
  |    	    |  	|memory |   |memory |   |memory |
  +---------+ 	| pool  |   | pool  |   | pool  |
	      	+-------+   +-------+   +-------+

The allocator structure contains global state (such as flags and counters) and structures needed to schedule new allocations. Each memory pool contains the actual buffer slots and metadata. The first memory pool in the list is the default memory pool allocated statically at early boot.

New memory pools are allocated from a kernel worker thread. That's because bounce buffers are allocated when mapping a DMA buffer, which may happen in interrupt context where large atomic allocations would probably fail. Allocation from process context is much more likely to succeed, especially if it can use CMA.

Nonetheless, the onset of a load spike may fill up the SWIOTLB before the worker has a chance to run. In that case, try to allocate a small transient memory pool to accommodate the request. If memory is encrypted and the device cannot do DMA to encrypted memory, this buffer is allocated from the coherent atomic DMA memory pool. Reducing the size of SWIOTLB may therefore require increasing the size of the coherent pool with the "coherent_pool" command-line parameter.

Performance
===========

All testing compared a vanilla v6.4-rc6 kernel with a fully patched kernel. The kernel was booted with "swiotlb=force" to allow stress-testing the software IO TLB on a high-performance device that would otherwise not need it. CONFIG_DEBUG_FS was set to 'y' to match the configuration of popular distribution kernels; it is understood that parallel workloads suffer from contention on the recently added debugfs atomic counters.

These benchmarks were run:

- small: single-threaded I/O of 4 KiB blocks,
- big: single-threaded I/O of 64 KiB blocks,
- 4way: 4-way parallel I/O of 4 KiB blocks.

In all tested cases, the default 64 MiB SWIOTLB would be sufficient (but wasteful). The "default" pair of columns shows performance impact when booted with 64 MiB SWIOTLB (i.e. current state). The "growing" pair of columns shows the impact when booted with a 1 MiB initial SWIOTLB, which grew to 5 MiB at run time. The "var" column in the tables below is the coefficient of variance over 5 runs of the test, the "diff" column is the difference in read-write I/O bandwidth (MiB/s). The very first column is the coefficient of variance in the results of the base unpatched kernel.

First, on an x86 VM against a QEMU virtio SATA driver backed by a RAM-based block device on the host:

base default growing
var var diff var diff
small 1.96% 0.47% -1.5% 0.52% -2.2%
big 2.03% 1.35% +0.9% 2.22% +2.9%
4way 0.80% 0.45% -0.7% 1.22% <0.1%

Second, on a Raspberry Pi4 with 8G RAM and a class 10 A1 microSD card:

base default growing
var var diff var diff
small 1.09% 1.69% +0.5% 2.14% -0.2%
big 0.03% 0.28% -0.5% 0.03% -0.1%
4way 5.15% 2.39% +0.2% 0.66% <0.1%

Third, on a CoCo VM. This was a bigger system, so I also added a 24-thread parallel I/O test:

base default growing
var var diff var diff
small 2.41% 6.02% +1.1% 10.33% +6.7%
big 9.20% 2.81% -0.6% 16.84% -0.2%
4way 0.86% 2.66% -0.1% 2.22% -4.9%
24way 3.19% 6.19% +4.4% 4.08% -5.9%

Note the increased variance of the CoCo VM, although the host was not otherwise loaded. These are caused by the first run, which includes the overhead of allocating additional bounce buffers and sharing them with the hypervisor. The system was not rebooted between successive runs.

Parallel tests suffer from a reduced number of areas in the dynamically allocated memory pools. This can be improved by allocating a larger pool from CMA (not implemented in this series yet).

I have no good explanation for the increase in performance of the 24-thread I/O test with the default (non-growing) memory pool. Although the difference is within variance, it seems to be real. The average bandwidth is consistently above that of the unpatched kernel.

Quoted 8 lines

To sum it up:

- All workloads benefit from reduced memory footprint.
- No performance regressions have been observed with the default size of the software IO TLB.
- Most workloads retain their former performance even if the software IO TLB grows at run time.

For the driver-core touched portions:

#17 lore raw
Quoted 4 lines · Christoph Hellwig

Thanks,

I've applied this to a new swiotlb-dynamic branch that I'll pull into the dma-mapping for-next tree.

Thank you.

I guess I can prepare some follow-up series now. ;-)

Petr T