Skip to contentSkip to conversation

swiotlb: eliminate per-map atomic contention on used/hiwater tracking

PATCH In review Jun Miao 6 participants 20 messages Jun 22 – Aug 13

Subsystems (2)
lore
Share & tools
lore

Jun Miao

author
#0 lore raw

PATCH 1/1 swiotlb: eliminate per-map atomic contention on used/hiwater tracking

Under heavy concurrent DMA traffic, inc_used_and_hiwater() performs an atomic_long_add_return() plus a CAS loop on the global used_hiwater, and dec_used() performs an atomic_long_sub() on total_used. All CPUs contend on the same cacheline, causing measurable throughput degradation at scale. Both counters are debug-only (CONFIG_DEBUG_FS).

Move hiwater tracking into struct io_tlb_area, where area->used is already maintained under area->lock:

- Add unsigned long used_hiwater to struct io_tlb_area. Update it inside the existing area->lock critical section in swiotlb_search_pool_area() immediately after area->used is incremented, at zero additional synchronisation cost.

- Remove atomic_long_t total_used and used_hiwater from struct io_tlb_mem; make inc_used_and_hiwater()/dec_used() empty stubs.

- Unify the two CONFIG_DEBUG_FS / !CONFIG_DEBUG_FS versions of mem_used() into a single implementation that sums per-area used counts with READ_ONCE().

- Rewrite io_tlb_hiwater_get() to sum area->used_hiwater across all areas (locklessly with READ_ONCE()); rewrite io_tlb_hiwater_set() to reset each area under its own spinlock.

Result: zero global atomic operations on the DMA map/unmap hot path.

Without this patch, the inc_used_and_hiwater() accounts for the most time from perf assembly record as following log:

       | spin_unlock_irqrestore():
       | - call    _raw_spin_unlock_irqrestore
       | swiotlb_area_find_slots():
 10.38 |   mov    $0x60(%rsp),%rax
  0.00 |   mov    $0x2b8(%rax),%rcx
       | arch_atomic64_add_return():
  0.00 |   mov    %r15,%rax
  0.00 |   lock    xadd    %rax,0x58(%rcx)
 66.74 |   lea    (%r15,%rax,1),%rdx
       | arch_atomic64_read():
  0.14 |   mov    $0x60(%rcx),%rax
       | inc_used_and_hiwater():
 19.74 |   cmp    %rdx,%rax
       |   jae    43f
       | arch_atomic64_try_cmpxchg():
       |   lock    cmpxchg %rdx,0x60(%rcx)
       |   jne    432
       | swiotlb_area_find_slots():
  0.07 |   mov    %ebx,%eax
  0.04 |   jmp    289
       | wrap_area_index():
       |   xor    %edx,%edx
       |   jmp    3fb
---
 include/linux/swiotlb.h |   7 ---
 kernel/dma/swiotlb.c    | 109 +++++++++++++++++-----------------------
 2 files changed, 45 insertions(+), 71 deletions(-)
include/linux/swiotlb.h +0 -7
index 3dae0f592063..a92496786cb5 100644
@@ -97,11 +97,6 @@ struct io_tlb_pool {
  * @lock:	Lock to synchronize changes to the list.
  * @pools:	List of IO TLB memory pool descriptors (if dynamic).
  * @dyn_alloc:	Dynamic IO TLB pool allocation work.
- * @total_used:	The total number of slots in the pool that are currently used
- *		across all areas. Used only for calculating used_hiwater in
- *		debugfs.
- * @used_hiwater: The high water mark for total_used.  Used only for reporting
- *		in debugfs.
  * @transient_nslabs: The total number of slots in all transient pools that
  *		are currently used across all areas.
  */
@@ -119,8 +114,6 @@ struct io_tlb_mem {
 	struct work_struct dyn_alloc;
 #endif
 #ifdef CONFIG_DEBUG_FS
-	atomic_long_t total_used;
-	atomic_long_t used_hiwater;
 	atomic_long_t transient_nslabs;
 #endif
 };
kernel/dma/swiotlb.c +45 -64

193 lines not on this page · raw message

Signature · 1 line
2.32.0
#1 lore raw

From: Jun Miao <[email protected]> Sent: Monday, June 22, 2026 5:21 AM

Quoted 1 line

[Adding LKML to the Cc: list]

Quoted 3 lines

Under heavy concurrent DMA traffic, inc_used_and_hiwater() performs an atomic_long_add_return() plus a CAS loop on the global used_hiwater, and dec_used() performs an atomic_long_sub() on total_used.

This statement seems overly broad. Normally the swiotlb is rarely used, so heavy concurrent DMA traffic doesn't generate this problem. The bad case would be in a CoCo VM where all DMA traffic goes through the swiotlb. If your statement is based on the CoCo VM case, please call that out.

Quoted 19 lines

All CPUs contend
on the same cacheline, causing measurable throughput degradation at scale. Both counters are debug-only (CONFIG_DEBUG_FS).

Move hiwater tracking into struct io_tlb_area, where area->used is already maintained under area->lock:

- Add unsigned long used_hiwater to struct io_tlb_area. Update it inside the existing area->lock critical section in swiotlb_search_pool_area() immediately after area->used is incremented, at zero additional synchronisation cost.

- Remove atomic_long_t total_used and used_hiwater from struct io_tlb_mem; make inc_used_and_hiwater()/dec_used() empty stubs.

- Unify the two CONFIG_DEBUG_FS / !CONFIG_DEBUG_FS versions of mem_used() into a single implementation that sums per-area used counts with READ_ONCE().

Quoted 5 lines

- Rewrite io_tlb_hiwater_get() to sum area->used_hiwater across all areas (locklessly with READ_ONCE()); rewrite io_tlb_hiwater_set() to reset each area under its own spinlock.

Result: zero global atomic operations on the DMA map/unmap hot path.

Another result (as I describe below in the code) is that the reported hiwater value can now be egregiously wrong. Back when I added reporting the hiwater value in commit 8b0977ecc8b3, there was considerable discussion of the potential impact of the atomic operations, and the decision was to place reporting under CONFIG_DEBUG_FS as a mitigation option if needed.

At the time, the use case for the hiwater mark was to enable a sysadmin to see how much peak swiotlb space is needed for a particular workload. This info would allow the swiotlb size to be set smaller than the rather large default that is used in CoCo VMs, and free up memory for real workload usage. I thought about the approach you've taken here, but my view at the time was that approach is inaccurate enough so as to be fairly worthless for the intended use case.

Quoted 222 lines

Without this patch, the inc_used_and_hiwater() accounts for the most time from perf assembly record as following log:

       | spin_unlock_irqrestore():
       | - call    _raw_spin_unlock_irqrestore
       | swiotlb_area_find_slots():
 10.38 |   mov    $0x60(%rsp),%rax
  0.00 |   mov    $0x2b8(%rax),%rcx
       | arch_atomic64_add_return():
  0.00 |   mov    %r15,%rax
  0.00 |   lock    xadd    %rax,0x58(%rcx)
 66.74 |   lea    (%r15,%rax,1),%rdx
       | arch_atomic64_read():
  0.14 |   mov    $0x60(%rcx),%rax
       | inc_used_and_hiwater():
 19.74 |   cmp    %rdx,%rax
       |   jae    43f
       | arch_atomic64_try_cmpxchg():
       |   lock    cmpxchg %rdx,0x60(%rcx)
       |   jne    432
       | swiotlb_area_find_slots():
  0.07 |   mov    %ebx,%eax
  0.04 |   jmp    289
       | wrap_area_index():
       |   xor    %edx,%edx
       |   jmp    3fb
---
 include/linux/swiotlb.h |   7 ---
 kernel/dma/swiotlb.c    | 109 +++++++++++++++++-----------------------
 2 files changed, 45 insertions(+), 71 deletions(-)
include/linux/swiotlb.h +0 -7
index 3dae0f592063..a92496786cb5 100644
@@ -97,11 +97,6 @@ struct io_tlb_pool {
  * @lock:	Lock to synchronize changes to the list.
  * @pools:	List of IO TLB memory pool descriptors (if dynamic).
  * @dyn_alloc:	Dynamic IO TLB pool allocation work.
- * @total_used:	The total number of slots in the pool that are currently used
- *		across all areas. Used only for calculating used_hiwater in
- *		debugfs.
- * @used_hiwater: The high water mark for total_used.  Used only for reporting
- *		in debugfs.
  * @transient_nslabs: The total number of slots in all transient pools that
  *		are currently used across all areas.
  */
@@ -119,8 +114,6 @@ struct io_tlb_mem {
 	struct work_struct dyn_alloc;
 #endif
 #ifdef CONFIG_DEBUG_FS
-	atomic_long_t total_used;
-	atomic_long_t used_hiwater;
 	atomic_long_t transient_nslabs;
 #endif
 };
kernel/dma/swiotlb.c +13 -63
index 1abd3e6146f4..9b4130e3f8e8 100644
@@ -107,12 +107,15 @@ static unsigned long default_nareas;
  * This is a single area with a single lock.
  *
  * @used:	The number of used IO TLB block.
+ * @used_hiwater: Per-area high-water mark of used slots. Updated under
+ *		@lock; read locklessly for approximate debugfs reporting.
  * @index:	The slot index to start searching in this area for next round.
  * @lock:	The lock to protect the above data structures in the map and
  *		unmap calls.
  */
 struct io_tlb_area {
 	unsigned long used;
+	unsigned long used_hiwater;
 	unsigned int index;
 	spinlock_t lock;
 };
@@ -959,40 +962,6 @@ static unsigned int wrap_area_index(struct io_tlb_pool *mem, unsigned int index)
 	return index;
 }
 
-/*
- * Track the total used slots with a global atomic value in order to have
- * correct information to determine the high water mark. The mem_used()
- * function gives imprecise results because there's no locking across
- * multiple areas.
- */
-#ifdef CONFIG_DEBUG_FS
-static void inc_used_and_hiwater(struct io_tlb_mem *mem, unsigned int nslots)
-{
-	unsigned long old_hiwater, new_used;
-
-	new_used = atomic_long_add_return(nslots, &mem->total_used);
-	old_hiwater = atomic_long_read(&mem->used_hiwater);
-	do {
-		if (new_used <= old_hiwater)
-			break;
-	} while (!atomic_long_try_cmpxchg(&mem->used_hiwater,
-					  &old_hiwater, new_used));
-}
-
-static void dec_used(struct io_tlb_mem *mem, unsigned int nslots)
-{
-	atomic_long_sub(nslots, &mem->total_used);
-}
-
-#else /* !CONFIG_DEBUG_FS */
-static void inc_used_and_hiwater(struct io_tlb_mem *mem, unsigned int nslots)
-{
-}
-static void dec_used(struct io_tlb_mem *mem, unsigned int nslots)
-{
-}
-#endif /* CONFIG_DEBUG_FS */
-
 #ifdef CONFIG_SWIOTLB_DYNAMIC
 #ifdef CONFIG_DEBUG_FS
 static void inc_transient_used(struct io_tlb_mem *mem, unsigned int nslots)
@@ -1132,9 +1101,10 @@ static int swiotlb_search_pool_area(struct device *dev, struct io_tlb_pool *pool
 	 */
 	area->index = wrap_area_index(pool, index + nslots);
 	area->used += nslots;
+	if (area->used > area->used_hiwater)
+		area->used_hiwater = area->used;
 	spin_unlock_irqrestore(&area->lock, flags);
 
-	inc_used_and_hiwater(dev->dma_io_tlb_mem, nslots);
 	return slot_index;
 }
 
@@ -1295,29 +1265,12 @@ static int swiotlb_find_slots(struct device *dev, phys_addr_t orig_addr,
 
 #endif /* CONFIG_SWIOTLB_DYNAMIC */
 
-#ifdef CONFIG_DEBUG_FS
-
-/**
- * mem_used() - get number of used slots in an allocator
- * @mem:	Software IO TLB allocator.
- *
- * The result is accurate in this version of the function, because an atomic
- * counter is available if CONFIG_DEBUG_FS is set.
- *
- * Return: Number of used slots.
- */
-static unsigned long mem_used(struct io_tlb_mem *mem)
-{
-	return atomic_long_read(&mem->total_used);
-}
-
-#else /* !CONFIG_DEBUG_FS */
-
 /**
  * mem_pool_used() - get number of used slots in a memory pool
  * @pool:	Software IO TLB memory pool.
  *
- * The result is not accurate, see mem_used().
+ * Returns the sum of per-area used slot counts. The result is approximate
+ * since there is no locking across areas.
  *
  * Return: Approximate number of used slots.
  */
@@ -1327,7 +1280,7 @@ static unsigned long mem_pool_used(struct io_tlb_pool *pool)
 	unsigned long used = 0;
 
 	for (i = 0; i < pool->nareas; i++)
-		used += pool->areas[i].used;
+		used += READ_ONCE(pool->areas[i].used);
 	return used;
 }
 
@@ -1335,8 +1288,8 @@ static unsigned long mem_pool_used(struct io_tlb_pool *pool)
  * mem_used() - get number of used slots in an allocator
  * @mem:	Software IO TLB allocator.
  *
- * The result is not accurate, because there is no locking of individual
- * areas.
+ * Returns the sum of per-area used slot counts across all pools. The result
+ * is approximate since there is no locking across areas.
  *
  * Return: Approximate number of used slots.
  */
@@ -1357,8 +1310,6 @@ static unsigned long mem_used(struct io_tlb_mem *mem)
 #endif
 }
 
-#endif /* CONFIG_DEBUG_FS */
-
 /**
  * swiotlb_tbl_map_single() - bounce buffer map a single contiguous physical area
  * @dev:		Device which maps the buffer.
@@ -1508,8 +1459,6 @@ static void swiotlb_release_slots(struct device *dev, phys_addr_t tlb_addr,
 		mem->slots[i].list = ++count;
 	area->used -= nslots;
 	spin_unlock_irqrestore(&area->lock, flags);
-
-	dec_used(dev->dma_io_tlb_mem, nslots);
 }
 
 #ifdef CONFIG_SWIOTLB_DYNAMIC
@@ -1531,7 +1480,6 @@ static bool swiotlb_del_transient(struct device *dev, phys_addr_t tlb_addr,
 	if (!pool->transient)
 		return false;
 
-	dec_used(dev->dma_io_tlb_mem, pool->nslabs);
 	swiotlb_del_pool(dev, pool);
 	dec_transient_used(dev->dma_io_tlb_mem, pool->nslabs);
 	return true;
@@ -1710,20 +1658,53 @@ static int io_tlb_used_get(void *data, u64 *val)
 static int io_tlb_hiwater_get(void *data, u64 *val)
 {
 	struct io_tlb_mem *mem = data;
+	unsigned long hiwater = 0;
+	struct io_tlb_pool *pool;
+	int i;
 
-	*val = atomic_long_read(&mem->used_hiwater);

kernel/dma/swiotlb.c static int io_tlb_used_get(void *data, u64 *val) · line 1674

Quoted 10 lines
kernel/dma/swiotlb.c +10 -0
+#ifdef CONFIG_SWIOTLB_DYNAMIC
+	rcu_read_lock();
+	list_for_each_entry_rcu(pool, &mem->pools, node)
+		for (i = 0; i < pool->nareas; i++)
+			hiwater += READ_ONCE(pool->areas[i].used_hiwater);
+	rcu_read_unlock();
+#else
+	pool = &mem->defpool;
+	for (i = 0; i < pool->nareas; i++)
+		hiwater += READ_ONCE(pool->areas[i].used_hiwater);

Let's ignore the SWIOTLB_DYNAMIC case for simplicity. The approach of calculating a separate hiwater mark for each area, and then summing those per-area hiwater marks, can produce very wrong results.

Consider a 64MiB swiotlb in a system with 8 CPUs. There will be 8 areas, each with 8 MiB of space. Suppose the workload putters along with mostly smallish I/Os, say between 4 KiB and 32 KiB. If each area has 16 I/Os in progress, the area hiwater mark might be 256 KiB (16 I/Os averaging 16 KiB each). Summing across areas produces a hiwater mark of 8 * 256 KiB = 2 MiB. But then suppose a 2 MiB I/O comes in. The hiwater mark for the area that handles that I/O will grow to 2+ MiB. After the first big I/O finishes,
another 2 MiB I/O comes in that is handled by a different area, whose
hiwater mark also goes to 2+ MiB. Pretty soon all 8 areas have a hiwater mark of 2+ MiB, and the total hiwater mark is reported as 16+ MiB. The old algorithm would have reported 4+ MiB, which is accurate. With higher CPU counts and more areas, the discrepancy can get much worse. This is a somewhat contrived example, but the problem is real enough to make the reported hiwater mark be unreliable.

I'm sure the contention for the total hiwater mark in the current code is real, but it's in the context of a lot of other CPU work that is being because of the bounce buffering, including copying lots of data to/from the bounce buffers. Is that atomic increment operation a bottleneck even in the end-to-end context of doing DMA through swiotlb bounce buffers?

Another approach to the contention problem would be to have a separate CONFIG option that is narrower than CONFIG_DEBUG_FS, so that the computation of the hiwater mark can be dropped entirely in production environments. Or the setting could be dynamic at runtime via a static_call, defaulting to not computing the hiwater mark while still allowing a sysadmin to turn it on to see workload usage of the swiotlb.

Michael

Quoted 40 lines
kernel/dma/swiotlb.c +22 -2
+#endif
+	*val = hiwater;
 	return 0;
 }
 
 static int io_tlb_hiwater_set(void *data, u64 val)
 {
 	struct io_tlb_mem *mem = data;
+	struct io_tlb_pool *pool;
+	unsigned long flags;
+	int i;
 
 	/* Only allow setting to zero */
 	if (val != 0)
 		return -EINVAL;
 
-	atomic_long_set(&mem->used_hiwater, val);
+#ifdef CONFIG_SWIOTLB_DYNAMIC
+	rcu_read_lock();
+	list_for_each_entry_rcu(pool, &mem->pools, node)
+		for (i = 0; i < pool->nareas; i++) {
+			spin_lock_irqsave(&pool->areas[i].lock, flags);
+			pool->areas[i].used_hiwater = 0;
+			spin_unlock_irqrestore(&pool->areas[i].lock, flags);
+		}
+	rcu_read_unlock();
+#else
+	pool = &mem->defpool;
+	for (i = 0; i < pool->nareas; i++) {
+		spin_lock_irqsave(&pool->areas[i].lock, flags);
+		pool->areas[i].used_hiwater = 0;
+		spin_unlock_irqrestore(&pool->areas[i].lock, flags);
+	}
+#endif
 	return 0;
 }
 
--

2.32.0

reviewer
#2 lore raw

On 2026-06-23 3:35 am, Michael Kelley wrote:
[...]

Quoted 6 lines

Another approach to the contention problem would be to have a separate CONFIG option that is narrower than CONFIG_DEBUG_FS, so that the computation of the hiwater mark can be dropped entirely in production environments. Or the setting could be dynamic at runtime via a static_call, defaulting to not computing the hiwater mark while still allowing a sysadmin to turn it on to see workload usage of the swiotlb.

Yeah, in terms of usefulness to admins, I'd imagine a sysctl to enable stats collection would probably be better than a config option - I doubt that most folks who just want to tune a standard distro kernel for their VM workload would want to have to rebuild it. A command line option might also be a fair compromise if that's how the eventual SWIOTLB tuning will be controlled anyway.

Thanks,
Robin.

#3 lore raw
In reply to Michael Kelley
Quoted 322 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Tuesday, June 23, 2026 10:35 AM
To: Miao, Jun <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Du, Fan <[email protected]>; LKML <[email protected]>
Subject: RE: [PATCH] swiotlb: eliminate per-map atomic contention on used/hiwater tracking

From: Jun Miao <[email protected]> Sent: Monday, June 22, 2026 5:21 AM

Quoted 1 line

[Adding LKML to the Cc: list]

Quoted 3 lines

Under heavy concurrent DMA traffic, inc_used_and_hiwater() performs an atomic_long_add_return() plus a CAS loop on the global used_hiwater, and dec_used() performs an atomic_long_sub() on total_used.

This statement seems overly broad. Normally the swiotlb is rarely used, so heavy concurrent DMA traffic doesn't generate this problem. The bad case would be in a CoCo VM where all DMA traffic goes through the swiotlb. If your statement is based on the CoCo VM case, please call that out.

Quoted 2 lines

All CPUs contend
on the same cacheline, causing measurable throughput degradation at

scale.

Quoted 17 lines

Both counters are debug-only (CONFIG_DEBUG_FS).

Move hiwater tracking into struct io_tlb_area, where area->used is already maintained under area->lock:

- Add unsigned long used_hiwater to struct io_tlb_area. Update it inside the existing area->lock critical section in swiotlb_search_pool_area() immediately after area->used is incremented, at zero additional synchronisation cost.

- Remove atomic_long_t total_used and used_hiwater from struct io_tlb_mem; make inc_used_and_hiwater()/dec_used() empty stubs.

- Unify the two CONFIG_DEBUG_FS / !CONFIG_DEBUG_FS versions of mem_used() into a single implementation that sums per-area used counts with READ_ONCE().

Quoted 5 lines

- Rewrite io_tlb_hiwater_get() to sum area->used_hiwater across all areas (locklessly with READ_ONCE()); rewrite io_tlb_hiwater_set() to reset each area under its own spinlock.

Result: zero global atomic operations on the DMA map/unmap hot path.

Another result (as I describe below in the code) is that the reported hiwater value can now be egregiously wrong. Back when I added reporting the hiwater value in commit 8b0977ecc8b3, there was considerable discussion of the potential impact of the atomic operations, and the decision was to place reporting under CONFIG_DEBUG_FS as a mitigation option if needed.

At the time, the use case for the hiwater mark was to enable a sysadmin to see how much peak swiotlb space is needed for a particular workload. This info would allow the swiotlb size to be set smaller than the rather large default that is used in CoCo VMs, and free up memory for real workload

usage. I thought about the approach you've taken here, but my view at the time was that approach is inaccurate enough so as to be fairly worthless for the intended use case.

Quoted 42 lines

Without this patch, the inc_used_and_hiwater() accounts for the most time from perf assembly record as following log:

       | spin_unlock_irqrestore():
       | - call    _raw_spin_unlock_irqrestore
       | swiotlb_area_find_slots():
 10.38 |   mov    $0x60(%rsp),%rax
  0.00 |   mov    $0x2b8(%rax),%rcx
       | arch_atomic64_add_return():
  0.00 |   mov    %r15,%rax
  0.00 |   lock    xadd    %rax,0x58(%rcx)
 66.74 |   lea    (%r15,%rax,1),%rdx
       | arch_atomic64_read():
  0.14 |   mov    $0x60(%rcx),%rax
       | inc_used_and_hiwater():
 19.74 |   cmp    %rdx,%rax
       |   jae    43f
       | arch_atomic64_try_cmpxchg():
       |   lock    cmpxchg %rdx,0x60(%rcx)
       |   jne    432
       | swiotlb_area_find_slots():
  0.07 |   mov    %ebx,%eax
  0.04 |   jmp    289
       | wrap_area_index():
       |   xor    %edx,%edx
       |   jmp    3fb
---
 include/linux/swiotlb.h |   7 ---
 kernel/dma/swiotlb.c    | 109 +++++++++++++++++-----------------------
 2 files changed, 45 insertions(+), 71 deletions(-)
include/linux/swiotlb.h +0 -0
index 3dae0f592063..a92496786cb5 100644

include/linux/swiotlb.h struct io_tlb_pool { · line 100

Quoted 4 lines
include/linux/swiotlb.h +0 -1
@@ -97,11 +97,6 @@ struct io_tlb_pool {
  * @lock:	Lock to synchronize changes to the list.
  * @pools:	List of IO TLB memory pool descriptors (if dynamic).
  * @dyn_alloc:	Dynamic IO TLB pool allocation work.
- * @total_used:	The total number of slots in the pool that are

currently used

include/linux/swiotlb.h struct io_tlb_pool {

Quoted 3 lines
include/linux/swiotlb.h +0 -3
- *		across all areas. Used only for calculating used_hiwater in
- *		debugfs.
- * @used_hiwater: The high water mark for total_used.  Used only for

reporting

kernel/dma/swiotlb.c static unsigned int wrap_area_index(struct

Quoted 34 lines
include/linux/swiotlb.h +0 -3
- *		in debugfs.
  * @transient_nslabs: The total number of slots in all transient pools that
  *		are currently used across all areas.
  */
@@ -119,8 +114,6 @@ struct io_tlb_mem {
 	struct work_struct dyn_alloc;
 #endif
 #ifdef CONFIG_DEBUG_FS
-	atomic_long_t total_used;
-	atomic_long_t used_hiwater;
 	atomic_long_t transient_nslabs;
 #endif
 };
kernel/dma/swiotlb.c +3 -0
index 1abd3e6146f4..9b4130e3f8e8 100644
@@ -107,12 +107,15 @@ static unsigned long default_nareas;
  * This is a single area with a single lock.
  *
  * @used:	The number of used IO TLB block.
+ * @used_hiwater: Per-area high-water mark of used slots. Updated under
+ *		@lock; read locklessly for approximate debugfs reporting.
  * @index:	The slot index to start searching in this area for next round.
  * @lock:	The lock to protect the above data structures in the map and
  *		unmap calls.
  */
 struct io_tlb_area {
 	unsigned long used;
+	unsigned long used_hiwater;
 	unsigned int index;
 	spinlock_t lock;
 };
@@ -959,40 +962,6 @@ static unsigned int wrap_area_index(struct

io_tlb_pool *mem, unsigned int index)

kernel/dma/swiotlb.c static unsigned int wrap_area_index(struct io_tlb_pool *mem, unsigned int index)

Quoted 11 lines
kernel/dma/swiotlb.c +0 -8
 	return index;
 }
 
-/*
- * Track the total used slots with a global atomic value in order to have
- * correct information to determine the high water mark. The mem_used()
- * function gives imprecise results because there's no locking across
- * multiple areas.
- */
-#ifdef CONFIG_DEBUG_FS
-static void inc_used_and_hiwater(struct io_tlb_mem *mem, unsigned int

nslots)

Quoted 9 lines
kernel/dma/swiotlb.c +0 -9
-{
-	unsigned long old_hiwater, new_used;
-
-	new_used = atomic_long_add_return(nslots, &mem->total_used);
-	old_hiwater = atomic_long_read(&mem->used_hiwater);
-	do {
-		if (new_used <= old_hiwater)
-			break;
-	} while (!atomic_long_try_cmpxchg(&mem->used_hiwater,

kernel/dma/swiotlb.c static unsigned int wrap_area_index(struct io_tlb_pool *mem, unsigned int index)

Quoted 10 lines
kernel/dma/swiotlb.c +0 -10
-					  &old_hiwater, new_used));
-}
-
-static void dec_used(struct io_tlb_mem *mem, unsigned int nslots)
-{
-	atomic_long_sub(nslots, &mem->total_used);
-}
-
-#else /* !CONFIG_DEBUG_FS */
-static void inc_used_and_hiwater(struct io_tlb_mem *mem, unsigned int

nslots)

kernel/dma/swiotlb.c static unsigned int wrap_area_index(struct io_tlb_pool *mem, unsigned int index)

Quoted 10 lines
kernel/dma/swiotlb.c +0 -7
-{
-}
-static void dec_used(struct io_tlb_mem *mem, unsigned int nslots)
-{
-}
-#endif /* CONFIG_DEBUG_FS */
-
 #ifdef CONFIG_SWIOTLB_DYNAMIC
 #ifdef CONFIG_DEBUG_FS
 static void inc_transient_used(struct io_tlb_mem *mem, unsigned int

nslots)

kernel/dma/swiotlb.c static int swiotlb_search_pool_area(struct device

Quoted 1 line
kernel/dma/swiotlb.c +0 -0
@@ -1132,9 +1101,10 @@ static int swiotlb_search_pool_area(struct device

*dev, struct io_tlb_pool *pool

kernel/dma/swiotlb.c static int swiotlb_find_slots(struct device *dev,

Quoted 12 lines
kernel/dma/swiotlb.c +2 -1
 	 */
 	area->index = wrap_area_index(pool, index + nslots);
 	area->used += nslots;
+	if (area->used > area->used_hiwater)
+		area->used_hiwater = area->used;
 	spin_unlock_irqrestore(&area->lock, flags);
 
-	inc_used_and_hiwater(dev->dma_io_tlb_mem, nslots);
 	return slot_index;
 }
 
@@ -1295,29 +1265,12 @@ static int swiotlb_find_slots(struct device *dev,

phys_addr_t orig_addr,

kernel/dma/swiotlb.c static unsigned long mem_pool_used(struct

Quoted 32 lines
kernel/dma/swiotlb.c +2 -19
 #endif /* CONFIG_SWIOTLB_DYNAMIC */
 
-#ifdef CONFIG_DEBUG_FS
-
-/**
- * mem_used() - get number of used slots in an allocator
- * @mem:	Software IO TLB allocator.
- *
- * The result is accurate in this version of the function, because an atomic
- * counter is available if CONFIG_DEBUG_FS is set.
- *
- * Return: Number of used slots.
- */
-static unsigned long mem_used(struct io_tlb_mem *mem)
-{
-	return atomic_long_read(&mem->total_used);
-}
-
-#else /* !CONFIG_DEBUG_FS */
-
 /**
  * mem_pool_used() - get number of used slots in a memory pool
  * @pool:	Software IO TLB memory pool.
  *
- * The result is not accurate, see mem_used().
+ * Returns the sum of per-area used slot counts. The result is approximate
+ * since there is no locking across areas.
  *
  * Return: Approximate number of used slots.
  */
@@ -1327,7 +1280,7 @@ static unsigned long mem_pool_used(struct

io_tlb_pool *pool)

kernel/dma/swiotlb.c static unsigned long mem_pool_used(struct

Quoted 9 lines
kernel/dma/swiotlb.c +1 -1
 	unsigned long used = 0;
 
 	for (i = 0; i < pool->nareas; i++)
-		used += pool->areas[i].used;
+		used += READ_ONCE(pool->areas[i].used);
 	return used;
 }
 
@@ -1335,8 +1288,8 @@ static unsigned long mem_pool_used(struct

io_tlb_pool *pool)

kernel/dma/swiotlb.c static unsigned long mem_used(struct

Quoted 11 lines
kernel/dma/swiotlb.c +2 -2
  * mem_used() - get number of used slots in an allocator
  * @mem:	Software IO TLB allocator.
  *
- * The result is not accurate, because there is no locking of individual
- * areas.
+ * Returns the sum of per-area used slot counts across all pools. The result
+ * is approximate since there is no locking across areas.
  *
  * Return: Approximate number of used slots.
  */
@@ -1357,8 +1310,6 @@ static unsigned long mem_used(struct

io_tlb_mem *mem)

kernel/dma/swiotlb.c static unsigned long mem_used(struct io_tlb_mem *mem)

Quoted 7 lines
kernel/dma/swiotlb.c +0 -2
 #endif
 }
 
-#endif /* CONFIG_DEBUG_FS */
-
 /**
  * swiotlb_tbl_map_single() - bounce buffer map a single contiguous

physical area

kernel/dma/swiotlb.c static void swiotlb_release_slots(struct device

Quoted 2 lines
kernel/dma/swiotlb.c +0 -0
  * @dev:		Device which maps the buffer.
@@ -1508,8 +1459,6 @@ static void swiotlb_release_slots(struct device

*dev, phys_addr_t tlb_addr,

kernel/dma/swiotlb.c static bool swiotlb_del_transient(struct device

Quoted 9 lines
kernel/dma/swiotlb.c +0 -2
 		mem->slots[i].list = ++count;
 	area->used -= nslots;
 	spin_unlock_irqrestore(&area->lock, flags);
-
-	dec_used(dev->dma_io_tlb_mem, nslots);
 }
 
 #ifdef CONFIG_SWIOTLB_DYNAMIC
@@ -1531,7 +1480,6 @@ static bool swiotlb_del_transient(struct device

*dev, phys_addr_t tlb_addr,

Quoted 16 lines
kernel/dma/swiotlb.c +3 -2
 	if (!pool->transient)
 		return false;
 
-	dec_used(dev->dma_io_tlb_mem, pool->nslabs);
 	swiotlb_del_pool(dev, pool);
 	dec_transient_used(dev->dma_io_tlb_mem, pool->nslabs);
 	return true;
@@ -1710,20 +1658,53 @@ static int io_tlb_used_get(void *data, u64 *val)
 static int io_tlb_hiwater_get(void *data, u64 *val)
 {
 	struct io_tlb_mem *mem = data;
+	unsigned long hiwater = 0;
+	struct io_tlb_pool *pool;
+	int i;
 
-	*val = atomic_long_read(&mem->used_hiwater);

kernel/dma/swiotlb.c static int io_tlb_used_get(void *data, u64 *val) · line 1674

Quoted 10 lines
kernel/dma/swiotlb.c +10 -0
+#ifdef CONFIG_SWIOTLB_DYNAMIC
+	rcu_read_lock();
+	list_for_each_entry_rcu(pool, &mem->pools, node)
+		for (i = 0; i < pool->nareas; i++)
+			hiwater += READ_ONCE(pool->areas[i].used_hiwater);
+	rcu_read_unlock();
+#else
+	pool = &mem->defpool;
+	for (i = 0; i < pool->nareas; i++)
+		hiwater += READ_ONCE(pool->areas[i].used_hiwater);

Let's ignore the SWIOTLB_DYNAMIC case for simplicity. The approach of calculating a separate hiwater mark for each area, and then summing those per-area hiwater marks, can produce very wrong results.

Quoted 14 lines

Consider a 64MiB swiotlb in a system with 8 CPUs. There will be 8 areas, each with 8 MiB of space. Suppose the workload putters along with mostly smallish I/Os, say between 4 KiB and 32 KiB. If each area has 16 I/Os in progress, the area hiwater mark might be 256 KiB (16 I/Os averaging 16 KiB each). Summing across areas produces a hiwater mark of 8 * 256 KiB = 2 MiB. But then suppose a 2 MiB I/O comes in. The hiwater mark for the area that handles that I/O will grow to 2+ MiB. After the first big I/O finishes,
another 2 MiB I/O comes in that is handled by a different area, whose
hiwater mark also goes to 2+ MiB. Pretty soon all 8 areas have a hiwater mark of 2+ MiB, and the total hiwater mark is reported as 16+ MiB. The old algorithm would have reported 4+ MiB, which is accurate. With higher CPU counts and more areas, the discrepancy can get much worse. This is a somewhat contrived example, but the problem is real enough to make the reported hiwater mark be unreliable.

You are correct here, I like your way of thinking.

Quoted 5 lines

I'm sure the contention for the total hiwater mark in the current code is real, but it's in the context of a lot of other CPU work that is being because of the bounce buffering, including copying lots of data to/from the bounce buffers. Is that atomic increment operation a bottleneck even in the end-to-end context of doing DMA through swiotlb bounce buffers?

Practical benchmark show case the highest IO performance for each TVM spec. even if a few iperf (4)workers would cause the contention here. My guts feelings, yes, the real workload probably hit the bottleneck here.

Quoted 6 lines

Another approach to the contention problem would be to have a separate CONFIG option that is narrower than CONFIG_DEBUG_FS, so that the computation of the hiwater mark can be dropped entirely in production environments. Or the setting could be dynamic at runtime via a static_call, defaulting to not computing the hiwater mark while still allowing a sysadmin to turn it on to see workload usage of the swiotlb.

That's counter-intuitive from my perspective.
With global counters, the observation, which itself impacts the performance, wouldn't be able to tell the practical characterization, that's commonly lower than max performance, in turn breaks the semantics of what's it for.

Even without those global counters, if user wants to know the hiwater value, snapshotting used value(sum of each area as current behavior) periodically would produce meaningful value for workload evaluation.

Quoted 42 lines

Michael

Quoted 40 lines
kernel/dma/swiotlb.c +22 -2
+#endif
+	*val = hiwater;
 	return 0;
 }
 
 static int io_tlb_hiwater_set(void *data, u64 val)
 {
 	struct io_tlb_mem *mem = data;
+	struct io_tlb_pool *pool;
+	unsigned long flags;
+	int i;
 
 	/* Only allow setting to zero */
 	if (val != 0)
 		return -EINVAL;
 
-	atomic_long_set(&mem->used_hiwater, val);
+#ifdef CONFIG_SWIOTLB_DYNAMIC
+	rcu_read_lock();
+	list_for_each_entry_rcu(pool, &mem->pools, node)
+		for (i = 0; i < pool->nareas; i++) {
+			spin_lock_irqsave(&pool->areas[i].lock, flags);
+			pool->areas[i].used_hiwater = 0;
+			spin_unlock_irqrestore(&pool->areas[i].lock, flags);
+		}
+	rcu_read_unlock();
+#else
+	pool = &mem->defpool;
+	for (i = 0; i < pool->nareas; i++) {
+		spin_lock_irqsave(&pool->areas[i].lock, flags);
+		pool->areas[i].used_hiwater = 0;
+		spin_unlock_irqrestore(&pool->areas[i].lock, flags);
+	}
+#endif
 	return 0;
 }
 
--

2.32.0

#4 lore raw

From: Du, Fan <[email protected]> Sent: Thursday, June 25, 2026 12:30 AM

Quoted 11 lines
Quoted 11 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Tuesday, June 23, 2026 10:35 AM
To: Miao, Jun <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Du, Fan <[email protected]>; LKML <[email protected]>
Subject: RE: [PATCH] swiotlb: eliminate per-map atomic contention on used/hiwater tracking

[snip]

Quoted 31 lines
Quoted 8 lines

kernel/dma/swiotlb.c static int io_tlb_used_get(void *data, u64 *val) · line 1674

Quoted 3 lines
kernel/dma/swiotlb.c +3 -0
+	pool = &mem->defpool;
+	for (i = 0; i < pool->nareas; i++)
+		hiwater += READ_ONCE(pool->areas[i].used_hiwater);

Let's ignore the SWIOTLB_DYNAMIC case for simplicity. The approach of calculating a separate hiwater mark for each area, and then summing those per-area hiwater marks, can produce very wrong results.

Quoted 14 lines

Consider a 64MiB swiotlb in a system with 8 CPUs. There will be 8 areas, each with 8 MiB of space. Suppose the workload putters along with mostly smallish I/Os, say between 4 KiB and 32 KiB. If each area has 16 I/Os in progress, the area hiwater mark might be 256 KiB (16 I/Os averaging 16 KiB each). Summing across areas produces a hiwater mark of 8 * 256 KiB = 2 MiB. But then suppose a 2 MiB I/O comes in. The hiwater mark for the area that handles that I/O will grow to 2+ MiB. After the first big I/O finishes,
another 2 MiB I/O comes in that is handled by a different area, whose
hiwater mark also goes to 2+ MiB. Pretty soon all 8 areas have a hiwater mark of 2+ MiB, and the total hiwater mark is reported as 16+ MiB. The old algorithm would have reported 4+ MiB, which is accurate. With higher CPU counts and more areas, the discrepancy can get much worse. This is a somewhat contrived example, but the problem is real enough to make the reported hiwater mark be unreliable.

You are correct here, I like your way of thinking.

Quoted 5 lines

I'm sure the contention for the total hiwater mark in the current code is real, but it's in the context of a lot of other CPU work that is being because of the bounce buffering, including copying lots of data to/from the bounce buffers. Is that atomic increment operation a bottleneck even in the end-to-end context of doing DMA through swiotlb bounce buffers?

Quoted 3 lines

Practical benchmark show case the highest IO performance for each TVM spec. even if a few iperf (4)workers would cause the contention here. My guts feelings, yes, the real workload probably hit the bottleneck here.

Just curious -- what is the NIC in the TDX VM? I'm most familiar with the Hyper-V case, where the NIC is the Hyper-V synthetic NIC. That driver uses dedicated send and receive buffers that are allocated and decrypted when the NIC is configured. Most NIC traffic goes through those buffers instead of the swiotlb, so I probably haven't seen cases where the swiotlb is the bottleneck for NIC traffic. I do see the swiotlb as the bottleneck for disk I/O traffic, but the data copying tends to be the gate rather than the allocation and freeing of swiotlb buffers.

Quoted 8 lines
Quoted 7 lines

Another approach to the contention problem would be to have a separate CONFIG option that is narrower than CONFIG_DEBUG_FS, so that the computation of the hiwater mark can be dropped entirely in production environments. Or the setting could be dynamic at runtime via a static_call, defaulting to not computing the hiwater mark while still allowing a sysadmin to turn it on to see workload usage of the swiotlb.

Quoted 4 lines

That's counter-intuitive from my perspective.
With global counters, the observation, which itself impacts the performance, wouldn't be able to tell the practical characterization, that's commonly lower than max performance, in turn breaks the semantics of what's it for.

Agreed. If the global counters affect the performance and throughput significantly, having an accurate hiwater mark loses some of its value.

Quoted 4 lines

Even without those global counters, if user wants to know the hiwater value, snapshotting used value(sum of each area as current behavior) periodically would produce meaningful value for workload evaluation.

I'm a little skeptical of the value of just summing current usage. Doing so tends to miss any spikes, and the spikes are the problem. If swiotlb capacity is exceeded even for a short spike, you don't just get a performance blip. You get I/O failures, which at least on the disk side tends to be fatal to the application doing the I/O. Maybe the networking stack recovers well enough and retries, resulting in just a performance reduction. But I've always thought of swiotlb exhaustion as a fairly serious problem to be avoided at all costs. That's why CoCo VMs allocate so much swiotlb space, even though most of it is never used for typical workloads (at least in my experience).

Michael

#5 lore raw
Quoted 95 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Thursday, June 25, 2026 11:54 PM
To: Du, Fan <[email protected]>; Michael Kelley <[email protected]>; Miao, Jun <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; LKML <linux- [email protected]>
Subject: RE: [PATCH] swiotlb: eliminate per-map atomic contention on used/hiwater tracking

From: Du, Fan <[email protected]> Sent: Thursday, June 25, 2026 12:30 AM

Quoted 11 lines
Quoted 11 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Tuesday, June 23, 2026 10:35 AM
To: Miao, Jun <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Du, Fan <[email protected]>; LKML <[email protected]>
Subject: RE: [PATCH] swiotlb: eliminate per-map atomic contention on used/hiwater tracking

[snip]

Quoted 10 lines
Quoted 10 lines

kernel/dma/swiotlb.c static int io_tlb_used_get(void *data, u64 *val) · line 1674

Quoted 3 lines
kernel/dma/swiotlb.c +3 -0
+	pool = &mem->defpool;
+	for (i = 0; i < pool->nareas; i++)
+		hiwater += READ_ONCE(pool->areas[i].used_hiwater);

Let's ignore the SWIOTLB_DYNAMIC case for simplicity. The approach of calculating a separate hiwater mark for each area, and then summing those per-area hiwater marks, can produce very wrong results.

Consider a 64MiB swiotlb in a system with 8 CPUs. There will be 8 areas, each with 8 MiB of space. Suppose the workload putters along with

mostly

Quoted 3 lines
Quoted 3 lines

smallish I/Os, say between 4 KiB and 32 KiB. If each area has 16 I/Os in progress, the area hiwater mark might be 256 KiB (16 I/Os averaging 16 KiB each). Summing across areas produces a hiwater mark of 8 * 256 KiB = 2

MiB.

Quoted 1 line
Quoted 1 line

But then suppose a 2 MiB I/O comes in. The hiwater mark for the area

that

Quoted 11 lines
Quoted 8 lines

handles that I/O will grow to 2+ MiB. After the first big I/O finishes,
another 2 MiB I/O comes in that is handled by a different area, whose
hiwater mark also goes to 2+ MiB. Pretty soon all 8 areas have a hiwater mark of 2+ MiB, and the total hiwater mark is reported as 16+ MiB. The old algorithm would have reported 4+ MiB, which is accurate. With higher CPU counts and more areas, the discrepancy can get much worse. This is a somewhat contrived example, but the problem is real enough to make the reported hiwater mark be unreliable.

You are correct here, I like your way of thinking.

Quoted 7 lines
Quoted 5 lines

I'm sure the contention for the total hiwater mark in the current code is real, but it's in the context of a lot of other CPU work that is being because of the bounce buffering, including copying lots of data to/from the bounce buffers. Is that atomic increment operation a bottleneck even in the end-to-end context of doing DMA through swiotlb bounce buffers?

Practical benchmark show case the highest IO performance for each TVM

spec.

Quoted 2 lines

even if a few iperf (4)workers would cause the contention here. My guts feelings, yes, the real workload probably hit the bottleneck here.

Just curious -- what is the NIC in the TDX VM? I'm most familiar with the Hyper-V case, where the NIC is the Hyper-V synthetic NIC. That driver uses dedicated send and receive buffers that are allocated and decrypted when the NIC is configured. Most NIC traffic goes through those buffers instead of the swiotlb, so I probably haven't seen cases where the swiotlb is the bottleneck for NIC traffic. I do see the swiotlb as the bottleneck for disk I/O traffic, but the data copying tends to be the gate rather than the allocation and freeing of swiotlb buffers.

Quoted 10 lines
Quoted 7 lines

Another approach to the contention problem would be to have a separate CONFIG option that is narrower than CONFIG_DEBUG_FS, so that the computation of the hiwater mark can be dropped entirely in production environments. Or the setting could be dynamic at runtime via a static_call, defaulting to not computing the hiwater mark while still allowing a sysadmin to turn it on to see workload usage of the swiotlb.

That's counter-intuitive from my perspective.
With global counters, the observation, which itself impacts the

performance,

Quoted 1 line

wouldn't be able to tell the practical characterization, that's commonly

lower than

Quoted 1 line

max performance, in turn breaks the semantics of what's it for.

Agreed. If the global counters affect the performance and throughput significantly, having an accurate hiwater mark loses some of its value.

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

Even without those global counters, if user wants to know the hiwater value, snapshotting used value(sum of each area as current behavior) periodically

would

Quoted 1 line

produce meaningful value for workload evaluation.

I'm a little skeptical of the value of just summing current usage. Doing so tends to miss any spikes, and the spikes are the problem. If swiotlb capacity

That's current design when CONFIG_DEBUG_FS is off, and used as swiotlb shortage indicator for user.

Statistically that sampled value is approximate to the true value as always.

Quoted 7 lines

is exceeded even for a short spike, you don't just get a performance blip. You get I/O failures, which at least on the disk side tends to be fatal to the application doing the I/O. Maybe the networking stack recovers well enough and retries, resulting in just a performance reduction. But I've always thought of swiotlb exhaustion as a fairly serious problem to be avoided at all costs. That's why CoCo VMs allocate so much swiotlb space, even though most of it is never used for typical workloads (at least in my experience).

That's dynamic SWIOTLB is designed for.
Only when the IO is so intensive, transient buffer/DMA pool is exhausted quickly before new shared memory pool is created.

Quoted 1 line

Michael

#6 lore raw

From: Du, Fan <[email protected]> Sent: Thursday, June 25, 2026 8:12 PM

Quoted 85 lines
Quoted 85 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Thursday, June 25, 2026 11:54 PM

From: Du, Fan <[email protected]> Sent: Thursday, June 25, 2026 12:30 AM

Quoted 11 lines
Quoted 11 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Tuesday, June 23, 2026 10:35 AM
To: Miao, Jun <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Du, Fan <[email protected]>; LKML <[email protected]>
Subject: RE: [PATCH] swiotlb: eliminate per-map atomic contention on used/hiwater tracking

[snip]

Quoted 31 lines
Quoted 8 lines

kernel/dma/swiotlb.c static int io_tlb_used_get(void *data, u64 *val) · line 1674

Quoted 3 lines
kernel/dma/swiotlb.c +3 -0
+	pool = &mem->defpool;
+	for (i = 0; i < pool->nareas; i++)
+		hiwater += READ_ONCE(pool->areas[i].used_hiwater);

Let's ignore the SWIOTLB_DYNAMIC case for simplicity. The approach of calculating a separate hiwater mark for each area, and then summing those per-area hiwater marks, can produce very wrong results.

Quoted 14 lines

Consider a 64MiB swiotlb in a system with 8 CPUs. There will be 8 areas, each with 8 MiB of space. Suppose the workload putters along with mostly smallish I/Os, say between 4 KiB and 32 KiB. If each area has 16 I/Os in progress, the area hiwater mark might be 256 KiB (16 I/Os averaging 16 KiB each). Summing across areas produces a hiwater mark of 8 * 256 KiB = 2 MiB. But then suppose a 2 MiB I/O comes in. The hiwater mark for the area that handles that I/O will grow to 2+ MiB. After the first big I/O finishes,
another 2 MiB I/O comes in that is handled by a different area, whose
hiwater mark also goes to 2+ MiB. Pretty soon all 8 areas have a hiwater mark of 2+ MiB, and the total hiwater mark is reported as 16+ MiB. The old algorithm would have reported 4+ MiB, which is accurate. With higher CPU counts and more areas, the discrepancy can get much worse. This is a somewhat contrived example, but the problem is real enough to make the reported hiwater mark be unreliable.

You are correct here, I like your way of thinking.

Quoted 5 lines

I'm sure the contention for the total hiwater mark in the current code is real, but it's in the context of a lot of other CPU work that is being because of the bounce buffering, including copying lots of data to/from the bounce buffers. Is that atomic increment operation a bottleneck even in the end-to-end context of doing DMA through swiotlb bounce buffers?

Quoted 3 lines

Practical benchmark show case the highest IO performance for each TVM spec. even if a few iperf (4)workers would cause the contention here. My guts feelings, yes, the real workload probably hit the bottleneck here.

Just curious -- what is the NIC in the TDX VM? I'm most familiar with the Hyper-V case, where the NIC is the Hyper-V synthetic NIC. That driver uses dedicated send and receive buffers that are allocated and decrypted when the NIC is configured. Most NIC traffic goes through those buffers instead of the swiotlb, so I probably haven't seen cases where the swiotlb is the bottleneck for NIC traffic. I do see the swiotlb as the bottleneck for disk I/O traffic, but the data copying tends to be the gate rather than the allocation and freeing of swiotlb buffers.

Quoted 8 lines
Quoted 7 lines

Another approach to the contention problem would be to have a separate CONFIG option that is narrower than CONFIG_DEBUG_FS, so that the computation of the hiwater mark can be dropped entirely in production environments. Or the setting could be dynamic at runtime via a static_call, defaulting to not computing the hiwater mark while still allowing a sysadmin to turn it on to see workload usage of the swiotlb.

Quoted 4 lines

That's counter-intuitive from my perspective.
With global counters, the observation, which itself impacts the performance, wouldn't be able to tell the practical characterization, that's commonly lower than max performance, in turn breaks the semantics of what's it for.

Agreed. If the global counters affect the performance and throughput significantly, having an accurate hiwater mark loses some of its value.

Quoted 4 lines

Even without those global counters, if user wants to know the hiwater value, snapshotting used value(sum of each area as current behavior) periodically would produce meaningful value for workload evaluation.

Quoted 7 lines
Quoted 2 lines

I'm a little skeptical of the value of just summing current usage. Doing so tends to miss any spikes, and the spikes are the problem. If swiotlb capacity

That's current design when CONFIG_DEBUG_FS is off, and used as swiotlb shortage indicator for user.

Statistically that sampled value is approximate to the true value as always.

OK, yes, statistical sampling could work. The kernel queues a work item that runs periodically to sum current usage across all areas. That sum is compared against the previous sum to calculate a hiwater mark. An experiment to compare the statistical hiwater mark against the current exact calculation would be interesting. I wonder how many samples would be needed, and hence how frequently it would need to run, to get a good result.

Quoted 9 lines
Quoted 8 lines

is exceeded even for a short spike, you don't just get a performance blip. You get I/O failures, which at least on the disk side tends to be fatal to the application doing the I/O. Maybe the networking stack recovers well enough and retries, resulting in just a performance reduction. But I've always thought of swiotlb exhaustion as a fairly serious problem to be avoided at all costs. That's why CoCo VMs allocate so much swiotlb space, even though most of it is never used for typical workloads (at least in my experience).

Quoted 3 lines

That's dynamic SWIOTLB is designed for.
Only when the IO is so intensive, transient buffer/DMA pool is exhausted quickly before new shared memory pool is created.

I don't think dynamic SWIOTLB in its current implementation is very useful for large CoCo VMs. Exhausting the atomic DMA pool is one problem. The dynamic swiotlb also can grow in a max of 4 MiB increments, so if 400 MiB is added dynamically, there's a list of 100 entries that must be searched to find space (and that list is currently searched linearly). Furthermore, the pre-allocated swiotlb gets 1 area per CPU, which mostly avoids contention on the area spin locks. But a dynamically added 4 MiB pool can have a max of 16 areas because each area must be at least 256 KiB. So there's more area spin lock contention with higher CPUs counts. And that all assumes that the memory allocator can provide a 4 MiB contiguous area for the new pool. If the swiotlb needs to grow after there's been memory fragmentation, an added pool might be limited to 2 MiB or 1 MiB or smaller, with a corresponding reduction in the area count and increase in area spin lock contention. Overall, dynamic swiotlb in a big CoCo VM results in complex and messy behavior with new failure modes and bottlenecks.

Michael

#7 lore raw
Quoted 129 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Saturday, June 27, 2026 12:00 AM
To: Du, Fan <[email protected]>; Michael Kelley <[email protected]>; Miao, Jun <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; LKML <linux- [email protected]>
Subject: RE: [PATCH] swiotlb: eliminate per-map atomic contention on used/hiwater tracking

From: Du, Fan <[email protected]> Sent: Thursday, June 25, 2026 8:12 PM

Quoted 6 lines
Quoted 6 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Thursday, June 25, 2026 11:54 PM

From: Du, Fan <[email protected]> Sent: Thursday, June 25, 2026 12:30

AM

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Tuesday, June 23, 2026 10:35 AM
To: Miao, Jun <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Du, Fan <[email protected]>; LKML <[email protected]>
Subject: RE: [PATCH] swiotlb: eliminate per-map atomic contention on used/hiwater tracking

Quoted 8 lines
Quoted 8 lines

[snip]

Quoted 6 lines
Quoted 6 lines

kernel/dma/swiotlb.c static int io_tlb_used_get(void *data, u64 *val) · line 1674

Quoted 3 lines
kernel/dma/swiotlb.c +3 -0
+	pool = &mem->defpool;
+	for (i = 0; i < pool->nareas; i++)
+		hiwater += READ_ONCE(pool->areas[i].used_hiwater);

Let's ignore the SWIOTLB_DYNAMIC case for simplicity. The approach of calculating a separate hiwater mark for each area, and then

summing

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

those per-area hiwater marks, can produce very wrong results.

Consider a 64MiB swiotlb in a system with 8 CPUs. There will be 8

areas,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

each with 8 MiB of space. Suppose the workload putters along with

mostly

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

smallish I/Os, say between 4 KiB and 32 KiB. If each area has 16 I/Os in progress, the area hiwater mark might be 256 KiB (16 I/Os averaging 16

KiB

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

each). Summing across areas produces a hiwater mark of 8 * 256 KiB =

2 MiB.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

But then suppose a 2 MiB I/O comes in. The hiwater mark for the area

that

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

handles that I/O will grow to 2+ MiB. After the first big I/O finishes,
another 2 MiB I/O comes in that is handled by a different area, whose
hiwater mark also goes to 2+ MiB. Pretty soon all 8 areas have a

hiwater

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

mark of 2+ MiB, and the total hiwater mark is reported as 16+ MiB.

The

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

old algorithm would have reported 4+ MiB, which is accurate. With higher CPU counts and more areas, the discrepancy can get much

worse.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

This is a somewhat contrived example, but the problem is real enough to make the reported hiwater mark be unreliable.

You are correct here, I like your way of thinking.

Quoted 1 line

I'm sure the contention for the total hiwater mark in the current code

is

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

real, but it's in the context of a lot of other CPU work that is being

because

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

of the bounce buffering, including copying lots of data to/from the

bounce

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

buffers. Is that atomic increment operation a bottleneck even in the end-to-end context of doing DMA through swiotlb bounce buffers?

Practical benchmark show case the highest IO performance for each

TVM spec.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

even if a few iperf (4)workers would cause the contention here. My guts feelings, yes, the real workload probably hit the bottleneck

here.

Quoted 11 lines
Quoted 11 lines

Just curious -- what is the NIC in the TDX VM? I'm most familiar with the Hyper-V case, where the NIC is the Hyper-V synthetic NIC. That driver uses dedicated send and receive buffers that are allocated and decrypted when the NIC is configured. Most NIC traffic goes through those buffers instead of the swiotlb, so I probably haven't seen cases where the swiotlb is the bottleneck for NIC traffic. I do see the swiotlb as the bottleneck for disk I/O traffic, but the data copying tends to be the gate rather than the allocation and freeing of swiotlb buffers.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Another approach to the contention problem would be to have a

separate

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

CONFIG option that is narrower than CONFIG_DEBUG_FS, so that the computation of the hiwater mark can be dropped entirely in

production

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

environments. Or the setting could be dynamic at runtime via a static_call, defaulting to not computing the hiwater mark while still allowing a sysadmin to turn it on to see workload usage of the swiotlb.

That's counter-intuitive from my perspective.
With global counters, the observation, which itself impacts the

performance,

Quoted 1 line
Quoted 1 line
Quoted 1 line

wouldn't be able to tell the practical characterization, that's commonly

lower than

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

max performance, in turn breaks the semantics of what's it for.

Agreed. If the global counters affect the performance and throughput significantly, having an accurate hiwater mark loses some of its value.

Quoted 2 lines

Even without those global counters, if user wants to know the hiwater

value,

Quoted 1 line
Quoted 1 line
Quoted 1 line

snapshotting used value(sum of each area as current behavior)

periodically would

Quoted 4 lines
Quoted 4 lines
Quoted 1 line

produce meaningful value for workload evaluation.

I'm a little skeptical of the value of just summing current usage. Doing so tends to miss any spikes, and the spikes are the problem. If swiotlb

capacity

Quoted 5 lines

That's current design when CONFIG_DEBUG_FS is off, and used as swiotlb shortage indicator for user.

Statistically that sampled value is approximate to the true value as always.

Quoted 7 lines

OK, yes, statistical sampling could work. The kernel queues a work item that runs periodically to sum current usage across all areas. That sum is compared against the previous sum to calculate a hiwater mark. An experiment to compare the statistical hiwater mark against the current exact calculation would be interesting. I wonder how many samples would be needed, and hence how frequently it would need to run, to get a good result.

Kernel doesn't do that sampling, kernel only report the sum as user space requested. Take a look at the flow when DEBUG_FS is off, and essentially it works as vmstat, user set query interval.

Quoted 17 lines
Quoted 4 lines
Quoted 4 lines

is exceeded even for a short spike, you don't just get a performance blip. You get I/O failures, which at least on the disk side tends to be fatal to the application doing the I/O. Maybe the networking stack recovers well

enough

Quoted 1 line
Quoted 1 line

and retries, resulting in just a performance reduction. But I've always

thought

Quoted 2 lines
Quoted 2 lines

of swiotlb exhaustion as a fairly serious problem to be avoided at all costs. That's why CoCo VMs allocate so much swiotlb space, even though most

of

Quoted 4 lines
Quoted 1 line

it is never used for typical workloads (at least in my experience).

That's dynamic SWIOTLB is designed for.
Only when the IO is so intensive, transient buffer/DMA pool is exhausted

quickly

Quoted 1 line

before new shared memory pool is created.

Quoted 15 lines

I don't think dynamic SWIOTLB in its current implementation is very useful for large CoCo VMs. Exhausting the atomic DMA pool is one problem. The dynamic swiotlb also can grow in a max of 4 MiB increments, so if 400 MiB is added dynamically, there's a list of 100 entries that must be searched to find space (and that list is currently searched linearly). Furthermore, the pre-allocated swiotlb gets 1 area per CPU, which mostly avoids contention on the area spin locks. But a dynamically added 4 MiB pool can have a max of 16 areas because each area must be at least 256 KiB. So there's more area spin lock contention with higher CPUs counts. And that all assumes that the memory allocator can provide a 4 MiB contiguous area for the new pool. If the swiotlb needs to grow after there's been memory fragmentation, an added pool might be limited to 2 MiB or 1 MiB or smaller, with a corresponding reduction in the area count and increase in area spin lock contention. Overall, dynamic swiotlb in a big CoCo VM results in complex and messy behavior with new failure modes and bottlenecks.

Fragmentation indeed undermine the requested allocation size as dynamic swiotlb grows by default 64M . In production practice, for large TVM, we see admin will reserve majority of needed size, and let dynamic swiotlb works to add additional buffer. Current dynamic switlb doesn't support shrinker to release unused memory back to buddy yet.

Quoted 1 line

Michael

#8 lore raw

From: Du, Fan <[email protected]> Sent: Saturday, June 27, 2026 4:21 PM

Quoted 101 lines
Quoted 93 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Saturday, June 27, 2026 12:00 AM

From: Du, Fan <[email protected]> Sent: Thursday, June 25, 2026 8:12 PM

Quoted 79 lines
Quoted 79 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Thursday, June 25, 2026 11:54 PM

From: Du, Fan <[email protected]> Sent: Thursday, June 25, 2026 12:30 AM

Quoted 5 lines
Quoted 5 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Tuesday, June 23, 2026 10:35 AM

[snip]

Quoted 31 lines
Quoted 8 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+	pool = &mem->defpool;
+	for (i = 0; i < pool->nareas; i++)
+		hiwater += READ_ONCE(pool->areas[i].used_hiwater);

Let's ignore the SWIOTLB_DYNAMIC case for simplicity. The approach of calculating a separate hiwater mark for each area, and then summing those per-area hiwater marks, can produce very wrong results.

Quoted 14 lines

Consider a 64MiB swiotlb in a system with 8 CPUs. There will be 8 areas, each with 8 MiB of space. Suppose the workload putters along with mostly smallish I/Os, say between 4 KiB and 32 KiB. If each area has 16 I/Os in progress, the area hiwater mark might be 256 KiB (16 I/Os averaging 16 KiB each). Summing across areas produces a hiwater mark of 8 * 256 KiB = 2 MiB. But then suppose a 2 MiB I/O comes in. The hiwater mark for the area that handles that I/O will grow to 2+ MiB. After the first big I/O finishes,
another 2 MiB I/O comes in that is handled by a different area, whose
hiwater mark also goes to 2+ MiB. Pretty soon all 8 areas have a hiwater mark of 2+ MiB, and the total hiwater mark is reported as 16+ MiB. The old algorithm would have reported 4+ MiB, which is accurate. With higher CPU counts and more areas, the discrepancy can get much worse. This is a somewhat contrived example, but the problem is real enough to make the reported hiwater mark be unreliable.

You are correct here, I like your way of thinking.

Quoted 5 lines

I'm sure the contention for the total hiwater mark in the current code is real, but it's in the context of a lot of other CPU work that is being because of the bounce buffering, including copying lots of data to/from the bounce buffers. Is that atomic increment operation a bottleneck even in the end-to-end context of doing DMA through swiotlb bounce buffers?

Quoted 3 lines

Practical benchmark show case the highest IO performance for each TVM spec. even if a few iperf (4)workers would cause the contention here. My guts feelings, yes, the real workload probably hit the bottleneck here.

Just curious -- what is the NIC in the TDX VM? I'm most familiar with the Hyper-V case, where the NIC is the Hyper-V synthetic NIC. That driver uses dedicated send and receive buffers that are allocated and decrypted when the NIC is configured. Most NIC traffic goes through those buffers instead of the swiotlb, so I probably haven't seen cases where the swiotlb is the bottleneck for NIC traffic. I do see the swiotlb as the bottleneck for disk I/O traffic, but the data copying tends to be the gate rather than the allocation and freeing of swiotlb buffers.

Quoted 8 lines
Quoted 7 lines

Another approach to the contention problem would be to have a separate CONFIG option that is narrower than CONFIG_DEBUG_FS, so that the computation of the hiwater mark can be dropped entirely in production environments. Or the setting could be dynamic at runtime via a static_call, defaulting to not computing the hiwater mark while still allowing a sysadmin to turn it on to see workload usage of the swiotlb.

Quoted 4 lines

That's counter-intuitive from my perspective.
With global counters, the observation, which itself impacts the performance, wouldn't be able to tell the practical characterization, that's commonly lower than max performance, in turn breaks the semantics of what's it for.

Agreed. If the global counters affect the performance and throughput significantly, having an accurate hiwater mark loses some of its value.

Quoted 4 lines

Even without those global counters, if user wants to know the hiwater value, snapshotting used value(sum of each area as current behavior) periodically would produce meaningful value for workload evaluation.

Quoted 7 lines
Quoted 2 lines

I'm a little skeptical of the value of just summing current usage. Doing so tends to miss any spikes, and the spikes are the problem. If swiotlb capacity

That's current design when CONFIG_DEBUG_FS is off, and used as swiotlb shortage indicator for user.

Statistically that sampled value is approximate to the true value as always.

Quoted 7 lines

OK, yes, statistical sampling could work. The kernel queues a work item that runs periodically to sum current usage across all areas. That sum is compared against the previous sum to calculate a hiwater mark. An experiment to compare the statistical hiwater mark against the current exact calculation would be interesting. I wonder how many samples would be needed, and hence how frequently it would need to run, to get a good result.

Quoted 3 lines

Kernel doesn't do that sampling, kernel only report the sum as user space requested. Take a look at the flow when DEBUG_FS is off, and essentially it works as vmstat, user set query interval.

Agree -- the kernel doesn't do that now. I wasn't clear in my comment that I was envisioning that the kernel *could* be enhanced to do the statistical sampling and report the result via sysfs.

Quoted 30 lines
Quoted 14 lines
Quoted 10 lines
Quoted 9 lines

is exceeded even for a short spike, you don't just get a performance blip. You get I/O failures, which at least on the disk side tends to be fatal to the application doing the I/O. Maybe the networking stack recovers well enough and retries, resulting in just a performance reduction. But I've always thought of swiotlb exhaustion as a fairly serious problem to be avoided at all costs. That's why CoCo VMs allocate so much swiotlb space, even though most of it is never used for typical workloads (at least in my experience).

Quoted 3 lines

That's dynamic SWIOTLB is designed for.
Only when the IO is so intensive, transient buffer/DMA pool is exhausted quickly before new shared memory pool is created.

Quoted 15 lines

I don't think dynamic SWIOTLB in its current implementation is very useful for large CoCo VMs. Exhausting the atomic DMA pool is one problem. The dynamic swiotlb also can grow in a max of 4 MiB increments, so if 400 MiB is added dynamically, there's a list of 100 entries that must be searched to find space (and that list is currently searched linearly). Furthermore, the pre-allocated swiotlb gets 1 area per CPU, which mostly avoids contention on the area spin locks. But a dynamically added 4 MiB pool can have a max of 16 areas because each area must be at least 256 KiB. So there's more area spin lock contention with higher CPUs counts. And that all assumes that the memory allocator can provide a 4 MiB contiguous area for the new pool. If the swiotlb needs to grow after there's been memory fragmentation, an added pool might be limited to 2 MiB or 1 MiB or smaller, with a corresponding reduction in the area count and increase in area spin lock contention. Overall, dynamic swiotlb in a big CoCo VM results in complex and messy behavior with new failure modes and bottlenecks.

Quoted 2 lines

Fragmentation indeed undermine the requested allocation size as dynamic swiotlb grows by default 64M .

Actually, I don't think the dynamic swiotlb ever grows by 64 MiB, at least not for systems with a 4 KiB page size. Yes, swiotlb_dyn_alloc() requests a new pool with size "default_nslabs", which might be 64 MiB. But the memory ultimately comes from the buddy allocator, which can provide a maximum of 4 MiB (unless MAX_ORDER has been overridden). swiotlb_alloc_pool() tries 64 MiB, which fails, so it then tries 32 MiB, etc., until it gets down to a size that the buddy allocator can provide. The most it will get is 4 MiB, and perhaps less if memory is fragmented.

Michael

Quoted 4 lines

In production practice, for large TVM, we see admin will reserve majority of needed size, and let dynamic swiotlb works to add additional buffer. Current dynamic switlb doesn't support shrinker to release unused memory back to buddy yet.

#9 lore raw
Quoted 152 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Sunday, June 28, 2026 9:30 AM
To: Du, Fan <[email protected]>; Michael Kelley <[email protected]>; Miao, Jun <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; LKML <linux- [email protected]>
Subject: RE: [PATCH] swiotlb: eliminate per-map atomic contention on used/hiwater tracking

From: Du, Fan <[email protected]> Sent: Saturday, June 27, 2026 4:21 PM

Quoted 5 lines
Quoted 5 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Saturday, June 27, 2026 12:00 AM

Quoted 7 lines
Quoted 7 lines

From: Du, Fan <[email protected]> Sent: Thursday, June 25, 2026 8:12 PM

Quoted 6 lines
Quoted 6 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Thursday, June 25, 2026 11:54 PM

From: Du, Fan <[email protected]> Sent: Thursday, June 25, 2026 12:30

AM

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Tuesday, June 23, 2026 10:35 AM

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

[snip]

Quoted 5 lines
Quoted 5 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+	pool = &mem->defpool;
+	for (i = 0; i < pool->nareas; i++)
+		hiwater += READ_ONCE(pool->areas[i].used_hiwater);

Let's ignore the SWIOTLB_DYNAMIC case for simplicity. The

approach

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

of calculating a separate hiwater mark for each area, and then

summing

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

those per-area hiwater marks, can produce very wrong results.

Consider a 64MiB swiotlb in a system with 8 CPUs. There will be 8

areas,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

each with 8 MiB of space. Suppose the workload putters along

with mostly

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

smallish I/Os, say between 4 KiB and 32 KiB. If each area has 16

I/Os in

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

progress, the area hiwater mark might be 256 KiB (16 I/Os

averaging 16 KiB

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

each). Summing across areas produces a hiwater mark of 8 * 256

KiB = 2 MiB.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

But then suppose a 2 MiB I/O comes in. The hiwater mark for the

area that

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

handles that I/O will grow to 2+ MiB. After the first big I/O finishes,
another 2 MiB I/O comes in that is handled by a different area,

whose

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

hiwater mark also goes to 2+ MiB. Pretty soon all 8 areas have a

hiwater

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

mark of 2+ MiB, and the total hiwater mark is reported as 16+ MiB.

The

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

old algorithm would have reported 4+ MiB, which is accurate. With higher CPU counts and more areas, the discrepancy can get much

worse.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

This is a somewhat contrived example, but the problem is real

enough

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

to make the reported hiwater mark be unreliable.

You are correct here, I like your way of thinking.

Quoted 1 line

I'm sure the contention for the total hiwater mark in the current

code is

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

real, but it's in the context of a lot of other CPU work that is being

because

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

of the bounce buffering, including copying lots of data to/from the

bounce

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

buffers. Is that atomic increment operation a bottleneck even in the end-to-end context of doing DMA through swiotlb bounce buffers?

Practical benchmark show case the highest IO performance for each

TVM spec.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

even if a few iperf (4)workers would cause the contention here. My guts feelings, yes, the real workload probably hit the bottleneck

here.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Just curious -- what is the NIC in the TDX VM? I'm most familiar with

the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Hyper-V case, where the NIC is the Hyper-V synthetic NIC. That driver uses dedicated send and receive buffers that are allocated and

decrypted

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

when the NIC is configured. Most NIC traffic goes through those buffers instead of the swiotlb, so I probably haven't seen cases where the

swiotlb

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

is the bottleneck for NIC traffic. I do see the swiotlb as the bottleneck

for

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

disk I/O traffic, but the data copying tends to be the gate rather than

the

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

allocation and freeing of swiotlb buffers.

Quoted 2 lines
Quoted 2 lines

Another approach to the contention problem would be to have a

separate

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

CONFIG option that is narrower than CONFIG_DEBUG_FS, so that

the

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

computation of the hiwater mark can be dropped entirely in

production

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

environments. Or the setting could be dynamic at runtime via a static_call, defaulting to not computing the hiwater mark while still allowing a sysadmin to turn it on to see workload usage of the

swiotlb.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

That's counter-intuitive from my perspective.
With global counters, the observation, which itself impacts the

performance,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

wouldn't be able to tell the practical characterization, that's

commonly lower than

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

max performance, in turn breaks the semantics of what's it for.

Agreed. If the global counters affect the performance and throughput significantly, having an accurate hiwater mark loses some of its value.

Quoted 2 lines

Even without those global counters, if user wants to know the

hiwater value,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

snapshotting used value(sum of each area as current behavior)

periodically would

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

produce meaningful value for workload evaluation.

I'm a little skeptical of the value of just summing current usage. Doing

so

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

tends to miss any spikes, and the spikes are the problem. If swiotlb

capacity

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

That's current design when CONFIG_DEBUG_FS is off, and used as

swiotlb

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

shortage indicator for user.

Statistically that sampled value is approximate to the true value as

always.

Quoted 10 lines
Quoted 8 lines

OK, yes, statistical sampling could work. The kernel queues a work item that runs periodically to sum current usage across all areas. That sum is compared against the previous sum to calculate a hiwater mark. An experiment to compare the statistical hiwater mark against the current exact calculation would be interesting. I wonder how many samples would be needed, and hence how frequently it would need to run, to get a good result.

Kernel doesn't do that sampling, kernel only report the sum as user space

Quoted 8 lines

requested.

Quoted 1 line

Take a look at the flow when DEBUG_FS is off, and essentially it works as

vmstat,

Quoted 1 line

user set query interval.

Agree -- the kernel doesn't do that now. I wasn't clear in my comment that I was envisioning that the kernel *could* be enhanced to do the statistical sampling and report the result via sysfs.

OK, thanks for the inputs!

Based on previous discussions, to keep backward compatibility, how about keep hiwater as it is, while guarded by dynamic knob(default off), and export used memory from sum of per area as current design does, then user can still have a chance to track the overall usage w/o visible overhead introduced by global atomic counter?

Quoted 62 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

is exceeded even for a short spike, you don't just get a performance

blip.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

You get I/O failures, which at least on the disk side tends to be fatal to

the

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

application doing the I/O. Maybe the networking stack recovers well

enough

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

and retries, resulting in just a performance reduction. But I've always

thought

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

of swiotlb exhaustion as a fairly serious problem to be avoided at all

costs.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

That's why CoCo VMs allocate so much swiotlb space, even though

most of

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

it is never used for typical workloads (at least in my experience).

That's dynamic SWIOTLB is designed for.
Only when the IO is so intensive, transient buffer/DMA pool is

exhausted quickly

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

before new shared memory pool is created.

I don't think dynamic SWIOTLB in its current implementation is very useful for large CoCo VMs. Exhausting the atomic DMA pool is one problem. The dynamic swiotlb also can grow in a max of 4 MiB increments, so if 400 MiB

is

Quoted 3 lines
Quoted 3 lines

added dynamically, there's a list of 100 entries that must be searched to find space (and that list is currently searched linearly). Furthermore, the pre-allocated swiotlb gets 1 area per CPU, which mostly avoids

contention

Quoted 1 line
Quoted 1 line

on the area spin locks. But a dynamically added 4 MiB pool can have a

max of

Quoted 2 lines
Quoted 2 lines

16 areas because each area must be at least 256 KiB. So there's more area spin lock contention with higher CPUs counts. And that all assumes that

the

Quoted 1 line
Quoted 1 line

memory allocator can provide a 4 MiB contiguous area for the new pool.

If

Quoted 8 lines
Quoted 5 lines

the swiotlb needs to grow after there's been memory fragmentation, an added pool might be limited to 2 MiB or 1 MiB or smaller, with a corresponding reduction in the area count and increase in area spin lock contention. Overall, dynamic swiotlb in a big CoCo VM results in complex and messy behavior with new failure modes and bottlenecks.

Fragmentation indeed undermine the requested allocation size as dynamic swiotlb grows by default 64M .

Actually, I don't think the dynamic swiotlb ever grows by 64 MiB, at least not for systems with a 4 KiB page size. Yes, swiotlb_dyn_alloc() requests a new pool with size "default_nslabs", which might be 64 MiB. But the memory ultimately comes from the buddy allocator, which can provide a maximum of 4 MiB (unless MAX_ORDER has been overridden). swiotlb_alloc_pool() tries 64 MiB, which fails, so it then tries 32 MiB, etc., until it gets down to a size that the buddy allocator can provide. The most it will get is 4 MiB, and perhaps less if memory is fragmented.

Michael

Quoted 2 lines

In production practice, for large TVM, we see admin will reserve majority of needed size, and let dynamic swiotlb works to add additional

buffer.

Quoted 1 line

Current dynamic switlb doesn't support shrinker to release unused memory

back

Quoted 1 line

to buddy yet.

#10 lore raw

From: Du, Fan <[email protected]> Sent: Tuesday, June 30, 2026 7:54 PM

Quoted 122 lines
Quoted 114 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Sunday, June 28, 2026 9:30 AM
To: Du, Fan <[email protected]>; Michael Kelley <[email protected]>; Miao, Jun <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; LKML <linux- [email protected]>
Subject: RE: [PATCH] swiotlb: eliminate per-map atomic contention on used/hiwater tracking

From: Du, Fan <[email protected]> Sent: Saturday, June 27, 2026 4:21 PM

Quoted 101 lines
Quoted 100 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Saturday, June 27, 2026 12:00 AM

From: Du, Fan <[email protected]> Sent: Thursday, June 25, 2026 8:12 PM

Quoted 79 lines
Quoted 79 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Thursday, June 25, 2026 11:54 PM

From: Du, Fan <[email protected]> Sent: Thursday, June 25, 2026 12:30 AM

Quoted 5 lines
Quoted 5 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Tuesday, June 23, 2026 10:35 AM

[snip]

Quoted 31 lines
Quoted 8 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+	pool = &mem->defpool;
+	for (i = 0; i < pool->nareas; i++)
+		hiwater += READ_ONCE(pool->areas[i].used_hiwater);

Let's ignore the SWIOTLB_DYNAMIC case for simplicity. The approach of calculating a separate hiwater mark for each area, and then summing those per-area hiwater marks, can produce very wrong results.

Quoted 14 lines

Consider a 64MiB swiotlb in a system with 8 CPUs. There will be 8 areas, each with 8 MiB of space. Suppose the workload putters along with mostly smallish I/Os, say between 4 KiB and 32 KiB. If each area has 16 I/Os in progress, the area hiwater mark might be 256 KiB (16 I/Os averaging 16 KiB each). Summing across areas produces a hiwater mark of 8 * 256 KiB = 2 MiB. But then suppose a 2 MiB I/O comes in. The hiwater mark for the area that handles that I/O will grow to 2+ MiB. After the first big I/O finishes,
another 2 MiB I/O comes in that is handled by a different area, whose
hiwater mark also goes to 2+ MiB. Pretty soon all 8 areas have a hiwater mark of 2+ MiB, and the total hiwater mark is reported as 16+ MiB. The old algorithm would have reported 4+ MiB, which is accurate. With higher CPU counts and more areas, the discrepancy can get much worse. This is a somewhat contrived example, but the problem is real enough to make the reported hiwater mark be unreliable.

You are correct here, I like your way of thinking.

Quoted 5 lines

I'm sure the contention for the total hiwater mark in the current code is real, but it's in the context of a lot of other CPU work that is being because of the bounce buffering, including copying lots of data to/from the bounce buffers. Is that atomic increment operation a bottleneck even in the end-to-end context of doing DMA through swiotlb bounce buffers?

Quoted 3 lines

Practical benchmark show case the highest IO performance for each TVM spec. even if a few iperf (4)workers would cause the contention here. My guts feelings, yes, the real workload probably hit the bottleneck here.

Just curious -- what is the NIC in the TDX VM? I'm most familiar with the Hyper-V case, where the NIC is the Hyper-V synthetic NIC. That driver uses dedicated send and receive buffers that are allocated and decrypted when the NIC is configured. Most NIC traffic goes through those buffers instead of the swiotlb, so I probably haven't seen cases where the swiotlb is the bottleneck for NIC traffic. I do see the swiotlb as the bottleneck for disk I/O traffic, but the data copying tends to be the gate rather than the allocation and freeing of swiotlb buffers.

Quoted 8 lines
Quoted 7 lines

Another approach to the contention problem would be to have a separate CONFIG option that is narrower than CONFIG_DEBUG_FS, so that the computation of the hiwater mark can be dropped entirely in production environments. Or the setting could be dynamic at runtime via a static_call, defaulting to not computing the hiwater mark while still allowing a sysadmin to turn it on to see workload usage of the swiotlb.

Quoted 4 lines

That's counter-intuitive from my perspective.
With global counters, the observation, which itself impacts the performance, wouldn't be able to tell the practical characterization, that's commonly lower than max performance, in turn breaks the semantics of what's it for.

Agreed. If the global counters affect the performance and throughput significantly, having an accurate hiwater mark loses some of its value.

Quoted 4 lines

Even without those global counters, if user wants to know the hiwater value, snapshotting used value(sum of each area as current behavior) periodically would produce meaningful value for workload evaluation.

Quoted 7 lines
Quoted 2 lines

I'm a little skeptical of the value of just summing current usage. Doing so tends to miss any spikes, and the spikes are the problem. If swiotlb capacity

That's current design when CONFIG_DEBUG_FS is off, and used as swiotlb shortage indicator for user.

Statistically that sampled value is approximate to the true value as always.

OK, yes, statistical sampling could work. The kernel queues a work item that runs periodically to sum current usage across all areas. That sum is compared against the previous sum to calculate a hiwater mark. An experiment to compare the statistical hiwater mark against the current exact calculation would be interesting. I wonder how many samples would be needed, and hence how frequently it would need to run, to get a good result.

Quoted 7 lines
Quoted 3 lines

Kernel doesn't do that sampling, kernel only report the sum as user space requested. Take a look at the flow when DEBUG_FS is off, and essentially it works as vmstat, user set query interval.

Agree -- the kernel doesn't do that now. I wasn't clear in my comment that I was envisioning that the kernel *could* be enhanced to do the statistical sampling and report the result via sysfs.

Quoted 8 lines

OK, thanks for the inputs!

Based on previous discussions, to keep backward compatibility, how about keep hiwater as it is, while guarded by dynamic knob(default off), and export used memory from sum of per area as current design does, then user can still have a chance to track the overall usage w/o visible overhead introduced by global atomic counter?

Yes, that works for me. A couple of points:

1) If doing the exact hiwater calculation is dynamic and defaults to "off", that dynamic config would replace being under #ifdef CONFIG_DEBUG_FS, right?

2) The mechanism used for dynamic config needs to be one that is selectable on the kernel boot line so that the exact hiwater mark during boot is easily available.

Michael

#11 lore raw
Quoted 203 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Friday, July 3, 2026 12:01 AM
To: Du, Fan <[email protected]>; Michael Kelley <[email protected]>; Miao, Jun <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; LKML <linux- [email protected]>
Subject: RE: [PATCH] swiotlb: eliminate per-map atomic contention on used/hiwater tracking

From: Du, Fan <[email protected]> Sent: Tuesday, June 30, 2026 7:54 PM

Quoted 5 lines
Quoted 5 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Sunday, June 28, 2026 9:30 AM
To: Du, Fan <[email protected]>; Michael Kelley

<[email protected]>;

Quoted 7 lines
Quoted 7 lines

Miao, Jun <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; LKML <linux- [email protected]>
Subject: RE: [PATCH] swiotlb: eliminate per-map atomic contention on used/hiwater tracking

Quoted 7 lines
Quoted 7 lines

From: Du, Fan <[email protected]> Sent: Saturday, June 27, 2026 4:21 PM

Quoted 6 lines
Quoted 6 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Saturday, June 27, 2026 12:00 AM

From: Du, Fan <[email protected]> Sent: Thursday, June 25, 2026 8:12

PM

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Thursday, June 25, 2026 11:54 PM

From: Du, Fan <[email protected]> Sent: Thursday, June 25, 2026

12:30 AM

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines

-----Original Message-----
From: Michael Kelley <[email protected]>
Sent: Tuesday, June 23, 2026 10:35 AM

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

[snip]

Quoted 3 lines
Quoted 3 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+	pool = &mem->defpool;
+	for (i = 0; i < pool->nareas; i++)
+		hiwater += READ_ONCE(pool-

areas[i].used_hiwater);

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Let's ignore the SWIOTLB_DYNAMIC case for simplicity. The

approach

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

of calculating a separate hiwater mark for each area, and then

summing

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

those per-area hiwater marks, can produce very wrong results.

Consider a 64MiB swiotlb in a system with 8 CPUs. There will be

8 areas,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

each with 8 MiB of space. Suppose the workload putters along

with mostly

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

smallish I/Os, say between 4 KiB and 32 KiB. If each area has 16

I/Os in

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

progress, the area hiwater mark might be 256 KiB (16 I/Os

averaging 16 KiB

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

each). Summing across areas produces a hiwater mark of 8 *

256 KiB = 2 MiB.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

But then suppose a 2 MiB I/O comes in. The hiwater mark for

the area that

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

handles that I/O will grow to 2+ MiB. After the first big I/O

finishes,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

another 2 MiB I/O comes in that is handled by a different area,

whose

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

hiwater mark also goes to 2+ MiB. Pretty soon all 8 areas have a

hiwater

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

mark of 2+ MiB, and the total hiwater mark is reported as 16+

MiB. The

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

old algorithm would have reported 4+ MiB, which is accurate.

With

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

higher CPU counts and more areas, the discrepancy can get

much worse.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

This is a somewhat contrived example, but the problem is real

enough

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

to make the reported hiwater mark be unreliable.

You are correct here, I like your way of thinking.

Quoted 1 line

I'm sure the contention for the total hiwater mark in the current

code is

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

real, but it's in the context of a lot of other CPU work that is

being because

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

of the bounce buffering, including copying lots of data to/from

the bounce

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

buffers. Is that atomic increment operation a bottleneck even in

the

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

end-to-end context of doing DMA through swiotlb bounce

buffers?

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Practical benchmark show case the highest IO performance for

each TVM spec.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

even if a few iperf (4)workers would cause the contention here. My guts feelings, yes, the real workload probably hit the

bottleneck here.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Just curious -- what is the NIC in the TDX VM? I'm most familiar

with the

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Hyper-V case, where the NIC is the Hyper-V synthetic NIC. That

driver

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

uses dedicated send and receive buffers that are allocated and

decrypted

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

when the NIC is configured. Most NIC traffic goes through those

buffers

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

instead of the swiotlb, so I probably haven't seen cases where the

swiotlb

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

is the bottleneck for NIC traffic. I do see the swiotlb as the

bottleneck for

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

disk I/O traffic, but the data copying tends to be the gate rather

than the

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

allocation and freeing of swiotlb buffers.

Quoted 2 lines
Quoted 2 lines

Another approach to the contention problem would be to have

a separate

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

CONFIG option that is narrower than CONFIG_DEBUG_FS, so

that the

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

computation of the hiwater mark can be dropped entirely in

production

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

environments. Or the setting could be dynamic at runtime via a static_call, defaulting to not computing the hiwater mark while

still

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

allowing a sysadmin to turn it on to see workload usage of the

swiotlb.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

That's counter-intuitive from my perspective.
With global counters, the observation, which itself impacts the

performance,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

wouldn't be able to tell the practical characterization, that's

commonly lower than

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

max performance, in turn breaks the semantics of what's it for.

Agreed. If the global counters affect the performance and

throughput

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

significantly, having an accurate hiwater mark loses some of its

value.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Even without those global counters, if user wants to know the

hiwater value,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

snapshotting used value(sum of each area as current behavior)

periodically would

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

produce meaningful value for workload evaluation.

I'm a little skeptical of the value of just summing current usage.

Doing so

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

tends to miss any spikes, and the spikes are the problem. If swiotlb

capacity

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

That's current design when CONFIG_DEBUG_FS is off, and used as

swiotlb

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

shortage indicator for user.

Statistically that sampled value is approximate to the true value as

always.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

OK, yes, statistical sampling could work. The kernel queues a work

item

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

that runs periodically to sum current usage across all areas. That sum

is

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

compared against the previous sum to calculate a hiwater mark. An experiment to compare the statistical hiwater mark against the current exact calculation would be interesting. I wonder how many samples

would

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

be needed, and hence how frequently it would need to run, to get a

good

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

result.

Kernel doesn't do that sampling, kernel only report the sum as user

space requested.

Quoted 1 line
Quoted 1 line
Quoted 1 line

Take a look at the flow when DEBUG_FS is off, and essentially it works as

vmstat,

Quoted 9 lines
Quoted 5 lines
Quoted 1 line

user set query interval.

Agree -- the kernel doesn't do that now. I wasn't clear in my comment that I was envisioning that the kernel *could* be enhanced to do the statistical sampling and report the result via sysfs.

OK, thanks for the inputs!

Based on previous discussions, to keep backward compatibility, how about

keep

Quoted 1 line

hiwater as it is, while guarded by dynamic knob(default off), and export

used

Quoted 1 line

memory from sum of per area as current design does, then user can still

have a

Quoted 1 line

chance to track the overall usage w/o visible overhead introduced by global

atomic

Quoted 2 lines

counter?

Quoted 8 lines

Yes, that works for me. A couple of points:

1) If doing the exact hiwater calculation is dynamic and defaults to "off", that dynamic config would replace being under #ifdef CONFIG_DEBUG_FS, right?

2) The mechanism used for dynamic config needs to be one that is selectable on the kernel boot line so that the exact hiwater mark during boot is easily available.

That's what in my mind too, will post the update version, let's have a review then.

Quoted 1 line

Michael

#12 lore raw

PATCH 1/1 dma/swiotlb: make high watermark tracking boot-time opt-in

In reply to Jun Miao

From: chenhuguanshen <[email protected]>

Under heavy concurrent DMA traffic on CoCo VMs, inc_used_and_hiwater() performs an atomic_long_add_return() plus a CAS loop on the global used_hiwater, and dec_used() performs an atomic_long_sub() on total_used. All CPUs contend on the same cacheline, causing measurable throughput degradation at scale.

Historically these counters were only compiled in under CONFIG_DEBUG_FS, which means production kernels with debugfs paid the atomic overhead unconditionally once the symbols were present. Make the tracking boot-time opt-in instead so that it is disabled by default with near-zero overhead via static_call, and can be enabled via "swiotlb=track_hiwater" parameter on demand for debugging.

Changes:

- The previous inc_used_and_hiwater() maintained a separate atomic total_used counter for accurate hiwater updates. This is replaced with mem_used() which sums across areas. While less precise (no cross-area locking), the result is adequate for high watermark statistics and avoids the overhead of maintaining total_used on every alloc/free.

- Refactor high watermark tracking to use static_call, allowing it to be enabled at boot via the swiotlb=track_hiwater parameter when CONFIG_DEBUG_FS is set. When disabled (the default), the static_call resolves to a no-op function, ensuring near-zero overhead.

- Remove the total_used atomic counter from io_tlb_mem since it is no longer needed. The debugfs "io_tlb_used" file already uses mem_used() to report current usage.

---
v1 -> v2:
- Change the patch title.
- Doing the exact hiwater calculation is dynamic and defaults to "off", dynamic config would replace being under #ifdef CONFIG_DEBUG_FS
- The mechanism used for dynamic config needs to be one that is selectable on the kernel boot line so that the exact hiwater mark during boot is easily available.

---
 .../admin-guide/kernel-parameters.txt         |   3 +-
 include/linux/swiotlb.h                       |  10 +-
 kernel/dma/swiotlb.c                          | 125 +++++++++---------
 3 files changed, 68 insertions(+), 70 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +2 -1
index b2d7d3540ded..bb60c55383a0 100644
@@ -7485,7 +7485,7 @@ Kernel parameters
 			Execution Facility on pSeries.
 
 	swiotlb=	[ARM,PPC,MIPS,X86,S390,EARLY]
-			Format: { <int> [,<int>] | force | noforce }
+			Format: { <int> [,<int>] | force | noforce | track_hiwater }
 			<int> -- Number of I/O TLB slabs
 			<int> -- Second integer after comma. Number of swiotlb
 				 areas with their own lock. Will be rounded up
@@ -7493,6 +7493,7 @@ Kernel parameters
 			force -- force using of bounce buffers even if they
 			         wouldn't be automatically used by the kernel
 			noforce -- Never use bounce buffers (for debugging)
+			track_hiwater -- Track high watermark of swiotlb buffers
 
 	switches=	[HW,M68k,EARLY]
 
include/linux/swiotlb.h +3 -7
index 3dae0f592063..e308792243cc 100644
@@ -97,11 +97,8 @@ struct io_tlb_pool {
  * @lock:	Lock to synchronize changes to the list.
  * @pools:	List of IO TLB memory pool descriptors (if dynamic).
  * @dyn_alloc:	Dynamic IO TLB pool allocation work.
- * @total_used:	The total number of slots in the pool that are currently used
- *		across all areas. Used only for calculating used_hiwater in
- *		debugfs.
- * @used_hiwater: The high water mark for total_used.  Used only for reporting
- *		in debugfs.
+ * @used_hiwater: The high watermark of used slots. Can be enabled at
+ *              boot time via swiotlb=track_hiwater.
  * @transient_nslabs: The total number of slots in all transient pools that
  *		are currently used across all areas.
  */
@@ -119,9 +116,8 @@ struct io_tlb_mem {
 	struct work_struct dyn_alloc;
 #endif
 #ifdef CONFIG_DEBUG_FS
-	atomic_long_t total_used;
-	atomic_long_t used_hiwater;
 	atomic_long_t transient_nslabs;
+	atomic_long_t used_hiwater;
 #endif
 };
 
kernel/dma/swiotlb.c +63 -62

175 lines not on this page · raw message

Signature · 1 line
2.34.1
#13 lore raw

RE: [PATCH] dma/swiotlb: make high watermark tracking boot-time opt-in

From: Frank Chen <[email protected]> Sent: Monday, July 13, 2026 8:25 PM

Quoted 16 lines

From: chenhuguanshen <[email protected]>

Under heavy concurrent DMA traffic on CoCo VMs, inc_used_and_hiwater() performs an atomic_long_add_return() plus a CAS loop on the global used_hiwater, and dec_used() performs an atomic_long_sub() on total_used. All CPUs contend on the same cacheline, causing measurable throughput degradation at scale.

Historically these counters were only compiled in under CONFIG_DEBUG_FS, which means production kernels with debugfs paid the atomic overhead unconditionally once the symbols were present. Make the tracking boot-time opt-in instead so that it is disabled by default with near-zero overhead via static_call, and can be enabled via "swiotlb=track_hiwater" parameter on demand for debugging.

Quoted 7 lines

Changes:

- The previous inc_used_and_hiwater() maintained a separate atomic total_used counter for accurate hiwater updates. This is replaced with mem_used() which sums across areas. While less precise (no cross-area locking), the result is adequate for high watermark statistics and avoids the overhead of maintaining total_used on every alloc/free.

I'm not sold on giving up an accurate hiwater calculation. If it's optional, let's have the option be to maintain total_used on every alloc/free so the hiwater calculation is accurate.

Maintaining hiwater based on mem_used() is already pretty low overhead in that it only does a global atomic operation if the hiwater is raised. After a startup transient that establishes a typical steady-state hiwater, most ongoing swiotlb alloc/free operations won't raise the hiwater and so won't do any atomic operations to maintain it.

To recap, there are three levels of hiwater maintenance:

1) None
2) Based on the less precise mem_used()
3) Based on an atomically update global total_used value

The perf difference between #1 and #2 seems minimal outside of the initial ramp to a steady-state hiwater value. The perf impact of #3 can indeed be more significant, especially at high CPU counts.

Having the kernel boot line offer all three is arguably overkill. I'd suggest offering #1 and #3 (with #1 as the default), or #2 and #3 (with #2 as the default and !CONFIG_DEBUG_FS providing #1).

There's another aspect here as well. Robin Murphy had suggested a sysctl option, so that the accurate hiwater maintenance could be turned on/off at runtime, not just a boot time. My earlier comment about boot time was probably unclear. I was thinking in terms of a runtime option that could *also* be set at boot time, which sysctl does. But I'll admit that enabling accurate hiwater calculation at runtime is tricky because some swiotlb buffers may be allocated at the time of the enablement, and total_used could go negative when those are freed. For runtime sysctl enablement, would probably have to do stop_machine() to freeze everything, and set total_used to the value of mem_used() so that everything is correct. I haven't worked through all the details.

All that said, I don't want to make too big of an issue out of this, and end up over-engineering with complexity that exceeds the value. I could be persuaded to have boot time enablement but not runtime enablement. I'm less happy with losing entirely the accurate calculation (my #3 above) that's in the current upstream code.

Michael

Quoted 277 lines

- Refactor high watermark tracking to use static_call, allowing it to be enabled at boot via the swiotlb=track_hiwater parameter when CONFIG_DEBUG_FS is set. When disabled (the default), the static_call resolves to a no-op function, ensuring near-zero overhead.

- Remove the total_used atomic counter from io_tlb_mem since it is no longer needed. The debugfs "io_tlb_used" file already uses mem_used() to report current usage.

---
v1 -> v2:
- Change the patch title.
- Doing the exact hiwater calculation is dynamic and defaults to "off", dynamic config would replace being under #ifdef CONFIG_DEBUG_FS
- The mechanism used for dynamic config needs to be one that is selectable on the kernel boot line so that the exact hiwater mark during boot is easily available.

---
 .../admin-guide/kernel-parameters.txt         |   3 +-
 include/linux/swiotlb.h                       |  10 +-
 kernel/dma/swiotlb.c                          | 125 +++++++++---------
 3 files changed, 68 insertions(+), 70 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +2 -1
index b2d7d3540ded..bb60c55383a0 100644
@@ -7485,7 +7485,7 @@ Kernel parameters
 			Execution Facility on pSeries.
 
 	swiotlb=	[ARM,PPC,MIPS,X86,S390,EARLY]
-			Format: { <int> [,<int>] | force | noforce }
+			Format: { <int> [,<int>] | force | noforce | track_hiwater }
 			<int> -- Number of I/O TLB slabs
 			<int> -- Second integer after comma. Number of swiotlb
 				 areas with their own lock. Will be rounded up
@@ -7493,6 +7493,7 @@ Kernel parameters
 			force -- force using of bounce buffers even if they
 			         wouldn't be automatically used by the kernel
 			noforce -- Never use bounce buffers (for debugging)
+			track_hiwater -- Track high watermark of swiotlb buffers
 
 	switches=	[HW,M68k,EARLY]
 
include/linux/swiotlb.h +3 -7
index 3dae0f592063..e308792243cc 100644
@@ -97,11 +97,8 @@ struct io_tlb_pool {
  * @lock:	Lock to synchronize changes to the list.
  * @pools:	List of IO TLB memory pool descriptors (if dynamic).
  * @dyn_alloc:	Dynamic IO TLB pool allocation work.
- * @total_used:	The total number of slots in the pool that are currently used
- *		across all areas. Used only for calculating used_hiwater in
- *		debugfs.
- * @used_hiwater: The high water mark for total_used.  Used only for reporting
- *		in debugfs.
+ * @used_hiwater: The high watermark of used slots. Can be enabled at
+ *              boot time via swiotlb=track_hiwater.
  * @transient_nslabs: The total number of slots in all transient pools that
  *		are currently used across all areas.
  */
@@ -119,9 +116,8 @@ struct io_tlb_mem {
 	struct work_struct dyn_alloc;
 #endif
 #ifdef CONFIG_DEBUG_FS
-	atomic_long_t total_used;
-	atomic_long_t used_hiwater;
 	atomic_long_t transient_nslabs;
+	atomic_long_t used_hiwater;
 #endif
 };
 
kernel/dma/swiotlb.c +49 -2
index 1abd3e6146f4..5828163c15f7 100644
@@ -180,8 +180,55 @@ static unsigned int limit_nareas(unsigned int nareas, unsigned
 long nslots)
 	return nareas;
 }
 
-static int __init
-setup_io_tlb_npages(char *str)
+#ifdef CONFIG_DEBUG_FS
+static unsigned long mem_used(struct io_tlb_mem *mem);
+
+/*
+ * Track the total used slots to determine the high watermark.
+ * Unlike the previous total_used atomic counter which provided
+ * exact values, mem_used() iterates areas without cross-area
+ * locking and may return imprecise results under concurrent
+ * alloc/free. This is acceptable for high watermark statistics
+ * and avoids the overhead of maintaining a dedicated atomic
+ * counter on every allocation and free.
+ */
+static void account_hiwater_real(struct io_tlb_mem *mem)
+{
+	unsigned long old_hiwater, new_used;
+
+	new_used = mem_used(mem);
+	old_hiwater = atomic_long_read(&mem->used_hiwater);
+	do {
+		if (new_used <= old_hiwater)
+			break;
+	} while (!atomic_long_try_cmpxchg(&mem->used_hiwater,
+					  &old_hiwater, new_used));
+}
+
+static void account_hiwater_nop(struct io_tlb_mem *mem)
+{
+}
+
+DEFINE_STATIC_CALL(swiotlb_account_hiwater, account_hiwater_nop);
+
+static __always_inline void account_hiwater(struct io_tlb_mem *mem)
+{
+	static_call(swiotlb_account_hiwater)(mem);
+}
+
+#else
+static __always_inline void account_hiwater(struct io_tlb_mem *mem)
+{
+}
+#endif
+
+/*
+ * The tracking of used slots high watermark can be enabled
+ * by appending "track_hiwater" to the swiotlb= boot parameter.
+ * When disabled the tracking functions are no-ops with near-zero
+ * overhead via static_call.
+ */
+static int __init setup_io_tlb_npages(char *str)
 {
 	if (isdigit(*str)) {
kernel/dma/swiotlb.c +13 -36
 		/* avoid tail segment of size < IO_TLB_SEGSIZE */
@@ -194,10 +241,21 @@ setup_io_tlb_npages(char *str)
 		swiotlb_adjust_nareas(simple_strtoul(str, &str, 0));
 	if (*str == ',')
 		++str;
-	if (!strcmp(str, "force"))
+	if (!strncmp(str, "force", 5)) {
 		swiotlb_force_bounce = true;
-	else if (!strcmp(str, "noforce"))
+		str += 5;
+	} else if (!strncmp(str, "noforce", 7)) {
 		swiotlb_force_disable = true;
+		str += 7;
+	}
+#ifdef CONFIG_DEBUG_FS
+	if (*str == ',')
+		++str;
+	if (!strncmp(str, "track_hiwater", 13)) {
+		static_call_update(swiotlb_account_hiwater,
+				   account_hiwater_real);
+	}
+#endif
 
 	return 0;
 }
@@ -959,40 +1017,6 @@ static unsigned int wrap_area_index(struct io_tlb_pool
 *mem, unsigned int index)
 	return index;
 }
 
-/*
- * Track the total used slots with a global atomic value in order to have
- * correct information to determine the high water mark. The mem_used()
- * function gives imprecise results because there's no locking across
- * multiple areas.
- */
-#ifdef CONFIG_DEBUG_FS
-static void inc_used_and_hiwater(struct io_tlb_mem *mem, unsigned int nslots)
-{
-	unsigned long old_hiwater, new_used;
-
-	new_used = atomic_long_add_return(nslots, &mem->total_used);
-	old_hiwater = atomic_long_read(&mem->used_hiwater);
-	do {
-		if (new_used <= old_hiwater)
-			break;
-	} while (!atomic_long_try_cmpxchg(&mem->used_hiwater,
-					  &old_hiwater, new_used));
-}
-
-static void dec_used(struct io_tlb_mem *mem, unsigned int nslots)
-{
-	atomic_long_sub(nslots, &mem->total_used);
-}
-
-#else /* !CONFIG_DEBUG_FS */
-static void inc_used_and_hiwater(struct io_tlb_mem *mem, unsigned int nslots)
-{
-}
-static void dec_used(struct io_tlb_mem *mem, unsigned int nslots)
-{
-}
-#endif /* CONFIG_DEBUG_FS */
-
 #ifdef CONFIG_SWIOTLB_DYNAMIC
 #ifdef CONFIG_DEBUG_FS

static void inc_transient_used(struct io_tlb_mem *mem, unsigned int nslots)

kernel/dma/swiotlb.c +0 -0
@@ -1134,7 +1158,7 @@ static int swiotlb_search_pool_area(struct device *dev,

struct io_tlb_pool *pool

kernel/dma/swiotlb.c +1 -19
 	area->used += nslots;
 	spin_unlock_irqrestore(&area->lock, flags);
 
-	inc_used_and_hiwater(dev->dma_io_tlb_mem, nslots);
+	account_hiwater(dev->dma_io_tlb_mem);
 	return slot_index;
 }
@@ -1295,24 +1319,6 @@ static int swiotlb_find_slots(struct device *dev,
 phys_addr_t orig_addr,
 
 #endif /* CONFIG_SWIOTLB_DYNAMIC */
 
-#ifdef CONFIG_DEBUG_FS
-
-/**
- * mem_used() - get number of used slots in an allocator
- * @mem:	Software IO TLB allocator.
- *
- * The result is accurate in this version of the function, because an atomic
- * counter is available if CONFIG_DEBUG_FS is set.
- *
- * Return: Number of used slots.
- */
-static unsigned long mem_used(struct io_tlb_mem *mem)
-{
-	return atomic_long_read(&mem->total_used);
-}
-
-#else /* !CONFIG_DEBUG_FS */
-
 /**
  * mem_pool_used() - get number of used slots in a memory pool

* @pool: Software IO TLB memory pool.

kernel/dma/swiotlb.c +0 -4
@@ -1357,8 +1363,6 @@ static unsigned long mem_used(struct io_tlb_mem *mem)
 #endif
 }
 
-#endif /* CONFIG_DEBUG_FS */
-
 /**
  * swiotlb_tbl_map_single() - bounce buffer map a single contiguous physical area
  * @dev:		Device which maps the buffer.
@@ -1508,8 +1512,6 @@ static void swiotlb_release_slots(struct device *dev,
 phys_addr_t tlb_addr,
 		mem->slots[i].list = ++count;
 	area->used -= nslots;
 	spin_unlock_irqrestore(&area->lock, flags);
-
-	dec_used(dev->dma_io_tlb_mem, nslots);
 }
 

#ifdef CONFIG_SWIOTLB_DYNAMIC

kernel/dma/swiotlb.c +0 -1
@@ -1531,7 +1533,6 @@ static bool swiotlb_del_transient(struct device *dev,
 phys_addr_t tlb_addr,
 	if (!pool->transient)
 		return false;
 
-	dec_used(dev->dma_io_tlb_mem, pool->nslabs);
 	swiotlb_del_pool(dev, pool);
 	dec_transient_used(dev->dma_io_tlb_mem, pool->nslabs);

return true;
--
2.34.1

maintainer
#14 lore raw

Re: [PATCH] dma/swiotlb: make high watermark tracking boot-time opt-in

Quoted 61 lines · Michael Kelley

From: Frank Chen <[email protected]> Sent: Monday, July 13, 2026 8:25 PM

Quoted 15 lines

From: chenhuguanshen <[email protected]>

Under heavy concurrent DMA traffic on CoCo VMs, inc_used_and_hiwater() performs an atomic_long_add_return() plus a CAS loop on the global used_hiwater, and dec_used() performs an atomic_long_sub() on total_used. All CPUs contend on the same cacheline, causing measurable throughput degradation at scale.

Historically these counters were only compiled in under CONFIG_DEBUG_FS, which means production kernels with debugfs paid the atomic overhead unconditionally once the symbols were present. Make the tracking boot-time opt-in instead so that it is disabled by default with near-zero overhead via static_call, and can be enabled via "swiotlb=track_hiwater" parameter on demand for debugging.

Quoted 7 lines

Changes:

- The previous inc_used_and_hiwater() maintained a separate atomic total_used counter for accurate hiwater updates. This is replaced with mem_used() which sums across areas. While less precise (no cross-area locking), the result is adequate for high watermark statistics and avoids the overhead of maintaining total_used on every alloc/free.

I'm not sold on giving up an accurate hiwater calculation. If it's optional, let's have the option be to maintain total_used on every alloc/free so the hiwater calculation is accurate.

Maintaining hiwater based on mem_used() is already pretty low overhead in that it only does a global atomic operation if the hiwater is raised. After a startup transient that establishes a typical steady-state hiwater, most ongoing swiotlb alloc/free operations won't raise the hiwater and so won't do any atomic operations to maintain it.

To recap, there are three levels of hiwater maintenance:

1) None
2) Based on the less precise mem_used()
3) Based on an atomically update global total_used value

The perf difference between #1 and #2 seems minimal outside of the initial ramp to a steady-state hiwater value. The perf impact of #3 can indeed be more significant, especially at high CPU counts.

Having the kernel boot line offer all three is arguably overkill. I'd suggest offering #1 and #3 (with #1 as the default), or #2 and #3 (with #2 as the default and !CONFIG_DEBUG_FS providing #1).

There's another aspect here as well. Robin Murphy had suggested a sysctl option, so that the accurate hiwater maintenance could be turned on/off at runtime, not just a boot time. My earlier comment about boot time was probably unclear. I was thinking in terms of a runtime option that could *also* be set at boot time, which sysctl does. But I'll admit that enabling accurate hiwater calculation at runtime is tricky because some swiotlb buffers may be allocated at the time of the enablement, and total_used could go negative when those are freed. For runtime sysctl enablement, would probably have to do stop_machine() to freeze everything, and set total_used to the value of mem_used() so that everything is correct. I haven't worked through all the details.

Quoted 6 lines

All that said, I don't want to make too big of an issue out of this, and end up over-engineering with complexity that exceeds the value. I could be persuaded to have boot time enablement but not runtime enablement. I'm less happy with losing entirely the accurate calculation (my #3 above) that's in the current upstream code.

Indeed keeping the accurate measurements is important to track potential

spikes of hiwater level. However imho it would be enough to use boot time

enablement.

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland
#15 lore raw

PATCH v3 1/1 dma/swiotlb: decouple high watermark tracking from CONFIG_DEBUG_FS

In reply to Jun Miao

From: chenhuguanshen <[email protected]>

Under heavy concurrent DMA traffic on CoCo VMs, inc_used_and_hiwater() performs an atomic_long_add_return() plus a CAS loop on the global used_hiwater, and dec_used() performs an atomic_long_sub() on total_used. All CPUs contend on the same cacheline, causing measurable throughput degradation at scale.

Historically these counters were only compiled in under CONFIG_DEBUG_FS, which means production kernels without debugfs paid the atomic overhead unconditionally once the symbols were present. Make the tracking boot-time opt-in instead so that it is disabled by default with near-zero overhead via static_call, and can be enabled via "swiotlb=track_hiwater" parameter on demand for debugging.

Changes:

- Introduction of static_call infrastructure: Under CONFIG_DEBUG_FS, real tracking functions and no-op stubs are defined, with DEFINE_STATIC_CALL defaulting to the stubs, making overhead near-zero when disabled.

- Parsing the track_hiwater boot parameter: setup_io_tlb_npages() is updated to recognize track_hiwater, which triggers static_call_update() to dynamically replace the stubs with the real tracking implementations.

- Removal of old compile-time-gated definitions: The old #ifdef/#else block defining inc_used_and_hiwater()/dec_used() is deleted, as its logic is now replaced by the static_call approach.

- Unification of mem_used() implementation: The two separate debugfs/non-debugfs versions are merged into one function that checks track_hiwater_enabled at runtime to decide whether to return an accurate or approximate value.

- Update of struct field comments: Comments on total_used and used_hiwater are changed from "used only in debugfs" to "enabled via swiotlb=track_hiwater boot parameter and exposed via debugfs".

- Documentation of the new boot parameter: The kernel-parameters documentation adds track_hiwater as a valid option for swiotlb= with its description.

---
v1 -> v2:
- Change the patch title.
- Doing the exact hiwater calculation is dynamic and defaults to "off", dynamic config would replace being under #ifdef CONFIG_DEBUG_FS
- The mechanism used for dynamic config needs to be one that is selectable on the kernel boot line so that the exact hiwater mark during boot is easily available.

v2 -> v3:
- When track_hiwater is enabled, we keep the original precise hiwater calculation backed by the global total_used atomic counter. If disabled, we switch to the approximate approach that sums per-area counters to derive the total used slot count.

---
 .../admin-guide/kernel-parameters.txt         |   3 +-
 include/linux/swiotlb.h                       |   8 +-
 kernel/dma/swiotlb.c                          | 153 +++++++++++-------
 3 files changed, 100 insertions(+), 64 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +2 -1
index b5493a7f8f22..2caa9c0b3d7a 100644
@@ -7477,7 +7477,7 @@ Kernel parameters
 			Execution Facility on pSeries.
 
 	swiotlb=	[ARM,PPC,MIPS,X86,S390,EARLY]
-			Format: { <int> [,<int>] | force | noforce }
+			Format: { <int> [,<int>] | force | noforce | track_hiwater}
 			<int> -- Number of I/O TLB slabs
 			<int> -- Second integer after comma. Number of swiotlb
 				 areas with their own lock. Will be rounded up
@@ -7485,6 +7485,7 @@ Kernel parameters
 			force -- force using of bounce buffers even if they
 			         wouldn't be automatically used by the kernel
 			noforce -- Never use bounce buffers (for debugging)
+			track_hiwater -- Track high watermark of swiotlb buffers
 
 	switches=	[HW,M68k,EARLY]
 
include/linux/swiotlb.h +4 -4
index 3dae0f592063..008f169e4006 100644
@@ -98,10 +98,10 @@ struct io_tlb_pool {
  * @pools:	List of IO TLB memory pool descriptors (if dynamic).
  * @dyn_alloc:	Dynamic IO TLB pool allocation work.
  * @total_used:	The total number of slots in the pool that are currently used
- *		across all areas. Used only for calculating used_hiwater in
- *		debugfs.
- * @used_hiwater: The high water mark for total_used.  Used only for reporting
- *		in debugfs.
+ *		across all areas. Used only for calculating used_hiwater via boot
+ *		parameter swiotlb=track_hiwater and exposed via debugfs.
+ * @used_hiwater: The high water mark for total_used.  Can be enabled at boot
+ *		time via swiotlb=track_hiwater and exposed via debugfs.
  * @transient_nslabs: The total number of slots in all transient pools that
  *		are currently used across all areas.
  */
kernel/dma/swiotlb.c +94 -59

195 lines not on this page · raw message

Signature · 1 line
2.53.0
Reviewed-by Tested-by on v3 1/1
#16 lore raw

RE: [PATCH v3] dma/swiotlb: decouple high watermark tracking from CONFIG_DEBUG_FS

From: Frank Chen <[email protected]> Sent: Tuesday, August 4, 2026 7:30 PM

Quoted 11 lines

From: chenhuguanshen <[email protected]>

Under heavy concurrent DMA traffic on CoCo VMs, inc_used_and_hiwater() performs an atomic_long_add_return() plus a CAS loop on the global used_hiwater, and dec_used() performs an atomic_long_sub() on total_used. All CPUs contend on the same cacheline, causing measurable throughput degradation at scale.

Historically these counters were only compiled in under CONFIG_DEBUG_FS, which means production kernels without debugfs paid the atomic overhead

"with debugfs"? "without debugfs" doesn't make sense to me.

Quoted 1 line

unconditionally once the symbols were present. Make the tracking

I'm not clear on the meaning of "once the symbols were present". Could you clarify?

Quoted 3 lines

boot-time opt-in instead so that it is disabled by default with near-zero overhead via static_call, and can be enabled via "swiotlb=track_hiwater" parameter on demand for debugging.

There's another functional implication here as well: With CONFIG_DEBUG_FS and when high water tracking is disabled, the value reported for "io_tlb_used" is approximate instead of instantaneously exact.

Quoted 19 lines

Changes:

- Introduction of static_call infrastructure: Under CONFIG_DEBUG_FS, real tracking functions and no-op stubs are defined, with DEFINE_STATIC_CALL defaulting to the stubs, making overhead near-zero when disabled.

- Parsing the track_hiwater boot parameter: setup_io_tlb_npages() is updated to recognize track_hiwater, which triggers static_call_update() to dynamically replace the stubs with the real tracking implementations.

- Removal of old compile-time-gated definitions: The old #ifdef/#else block defining inc_used_and_hiwater()/dec_used() is deleted, as its logic is now replaced by the static_call approach.

- Unification of mem_used() implementation: The two separate debugfs/non-debugfs versions are merged into one function that checks track_hiwater_enabled at runtime to decide whether to return an accurate or approximate value.

Quoted 6 lines

- Update of struct field comments: Comments on total_used and used_hiwater are changed from "used only in debugfs" to "enabled via swiotlb=track_hiwater boot parameter and exposed via debugfs".

- Documentation of the new boot parameter: The kernel-parameters documentation adds track_hiwater as a valid option for swiotlb= with its description.

To me, the code changes are straightforward and don't need a detailed recapitulation in the commit message. The commit message should focus on the "why" (which you've done) and perhaps a top-level summary of the approach. You've mentioned using static calls to make the overhead near zero when disabled, and that seems sufficient.

Quoted 43 lines

---
v1 -> v2:
- Change the patch title.
- Doing the exact hiwater calculation is dynamic and defaults to "off", dynamic config would replace being under #ifdef CONFIG_DEBUG_FS
- The mechanism used for dynamic config needs to be one that is selectable on the kernel boot line so that the exact hiwater mark during boot is easily available.

v2 -> v3:
- When track_hiwater is enabled, we keep the original precise hiwater calculation backed by the global total_used atomic counter. If disabled, we switch to the approximate approach that sums per-area counters to derive the total used slot count.

---
 .../admin-guide/kernel-parameters.txt         |   3 +-
 include/linux/swiotlb.h                       |   8 +-
 kernel/dma/swiotlb.c                          | 153 +++++++++++-------
 3 files changed, 100 insertions(+), 64 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +1 -1
index b5493a7f8f22..2caa9c0b3d7a 100644
@@ -7477,7 +7477,7 @@ Kernel parameters
 			Execution Facility on pSeries.
 
 	swiotlb=	[ARM,PPC,MIPS,X86,S390,EARLY]
-			Format: { <int> [,<int>] | force | noforce }
+			Format: { <int> [,<int>] | force | noforce | track_hiwater}
 			<int> -- Number of I/O TLB slabs
 			<int> -- Second integer after comma. Number of swiotlb
 				 areas with their own lock. Will be rounded up

Documentation/admin-guide/kernel-parameters.txt Kernel parameters · line 7488

Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +1 -0
@@ -7485,6 +7485,7 @@ Kernel parameters
 			force -- force using of bounce buffers even if they
 			         wouldn't be automatically used by the kernel
 			noforce -- Never use bounce buffers (for debugging)
+			track_hiwater -- Track high watermark of swiotlb buffers

Add: "Applies only when CONFIG_DEBUG_FS is set" ? I see several other entries in kernel-parameters.txt that document dependencies on CONFIG_* values.

Quoted 196 lines
Documentation/admin-guide/kernel-parameters.txt +0 -0
 	switches=	[HW,M68k,EARLY]
 
include/linux/swiotlb.h +4 -4
index 3dae0f592063..008f169e4006 100644
@@ -98,10 +98,10 @@ struct io_tlb_pool {
  * @pools:	List of IO TLB memory pool descriptors (if dynamic).
  * @dyn_alloc:	Dynamic IO TLB pool allocation work.
  * @total_used:	The total number of slots in the pool that are currently used
- *		across all areas. Used only for calculating used_hiwater in
- *		debugfs.
- * @used_hiwater: The high water mark for total_used.  Used only for reporting
- *		in debugfs.
+ *		across all areas. Used only for calculating used_hiwater via boot
+ *		parameter swiotlb=track_hiwater and exposed via debugfs.
+ * @used_hiwater: The high water mark for total_used.  Can be enabled at boot
+ *		time via swiotlb=track_hiwater and exposed via debugfs.
  * @transient_nslabs: The total number of slots in all transient pools that
  *		are currently used across all areas.
  */
kernel/dma/swiotlb.c +84 -54
index 1abd3e6146f4..799eee3bf9cf 100644
@@ -180,6 +180,74 @@ static unsigned int limit_nareas(unsigned int nareas, unsigned long nslots)
 	return nareas;
 }
 
+#ifdef CONFIG_DEBUG_FS
+/*
+ * Track the total used slots with a global atomic value in order to have
+ * correct information to determine the high water mark.
+ */
+static void inc_used_and_hiwater_real(struct io_tlb_mem *mem,
+				      unsigned int nslots)
+{
+	unsigned long old_hiwater, new_used;
+
+	new_used = atomic_long_add_return(nslots, &mem->total_used);
+	old_hiwater = atomic_long_read(&mem->used_hiwater);
+	do {
+		if (new_used <= old_hiwater)
+			break;
+	} while (!atomic_long_try_cmpxchg(&mem->used_hiwater,
+					  &old_hiwater, new_used));
+}
+
+static void dec_used_real(struct io_tlb_mem *mem, unsigned int nslots)
+{
+	atomic_long_sub(nslots, &mem->total_used);
+}
+
+static void inc_used_and_hiwater_nop(struct io_tlb_mem *mem,
+				     unsigned int nslots)
+{
+}
+static void dec_used_nop(struct io_tlb_mem *mem, unsigned int nslots)
+{
+}
+
+DEFINE_STATIC_CALL(swiotlb_inc_used, inc_used_and_hiwater_nop);
+DEFINE_STATIC_CALL(swiotlb_dec_used, dec_used_nop);
+
+static __always_inline void inc_used_and_hiwater(struct io_tlb_mem *mem,
+						  unsigned int nslots)
+{
+	static_call(swiotlb_inc_used)(mem, nslots);
+}
+
+static __always_inline void dec_used(struct io_tlb_mem *mem,
+				     unsigned int nslots)
+{
+	static_call(swiotlb_dec_used)(mem, nslots);
+}
+
+static bool track_hiwater_enabled __read_mostly;
+
+#else
+
+static __always_inline void inc_used_and_hiwater(struct io_tlb_mem *mem,
+						  unsigned int nslots)
+{
+}
+
+static __always_inline void dec_used(struct io_tlb_mem *mem,
+				     unsigned int nslots)
+{
+}
+#endif
+
+/*
+ * The tracking of used slots high watermark can be enabled
+ * by appending "track_hiwater" to the swiotlb= boot parameter.
+ * When disabled the tracking functions are no-ops with near-zero
+ * overhead via static_call.
+ */
 static int __init
 setup_io_tlb_npages(char *str)
 {
@@ -194,10 +262,24 @@ setup_io_tlb_npages(char *str)
 		swiotlb_adjust_nareas(simple_strtoul(str, &str, 0));
 	if (*str == ',')
 		++str;
-	if (!strcmp(str, "force"))
+	if (!strncmp(str, "force", 5)) {
 		swiotlb_force_bounce = true;
-	else if (!strcmp(str, "noforce"))
+		str += 5;
+	} else if (!strncmp(str, "noforce", 7)) {
 		swiotlb_force_disable = true;
+		str += 7;
+	}
+
+#ifdef CONFIG_DEBUG_FS
+	if (*str == ',')
+		++str;
+	if (!strncmp(str, "track_hiwater", 13)) {
+		track_hiwater_enabled = true;
+		static_call_update(swiotlb_inc_used,
+					inc_used_and_hiwater_real);
+		static_call_update(swiotlb_dec_used, dec_used_real);
+	}
+#endif
 
 	return 0;
 }
@@ -959,40 +1041,6 @@ static unsigned int wrap_area_index(struct io_tlb_pool *mem, unsigned int index)
 	return index;
 }
 
-/*
- * Track the total used slots with a global atomic value in order to have
- * correct information to determine the high water mark. The mem_used()
- * function gives imprecise results because there's no locking across
- * multiple areas.
- */
-#ifdef CONFIG_DEBUG_FS
-static void inc_used_and_hiwater(struct io_tlb_mem *mem, unsigned int nslots)
-{
-	unsigned long old_hiwater, new_used;
-
-	new_used = atomic_long_add_return(nslots, &mem->total_used);
-	old_hiwater = atomic_long_read(&mem->used_hiwater);
-	do {
-		if (new_used <= old_hiwater)
-			break;
-	} while (!atomic_long_try_cmpxchg(&mem->used_hiwater,
-					  &old_hiwater, new_used));
-}
-
-static void dec_used(struct io_tlb_mem *mem, unsigned int nslots)
-{
-	atomic_long_sub(nslots, &mem->total_used);
-}
-
-#else /* !CONFIG_DEBUG_FS */
-static void inc_used_and_hiwater(struct io_tlb_mem *mem, unsigned int nslots)
-{
-}
-static void dec_used(struct io_tlb_mem *mem, unsigned int nslots)
-{
-}
-#endif /* CONFIG_DEBUG_FS */
-
 #ifdef CONFIG_SWIOTLB_DYNAMIC
 #ifdef CONFIG_DEBUG_FS
 static void inc_transient_used(struct io_tlb_mem *mem, unsigned int nslots)
@@ -1295,24 +1343,6 @@ static int swiotlb_find_slots(struct device *dev,
 phys_addr_t orig_addr,
 
 #endif /* CONFIG_SWIOTLB_DYNAMIC */
 
-#ifdef CONFIG_DEBUG_FS
-
-/**
- * mem_used() - get number of used slots in an allocator
- * @mem:	Software IO TLB allocator.
- *
- * The result is accurate in this version of the function, because an atomic
- * counter is available if CONFIG_DEBUG_FS is set.
- *
- * Return: Number of used slots.
- */
-static unsigned long mem_used(struct io_tlb_mem *mem)
-{
-	return atomic_long_read(&mem->total_used);
-}
-
-#else /* !CONFIG_DEBUG_FS */
-
 /**
  * mem_pool_used() - get number of used slots in a memory pool

* @pool: Software IO TLB memory pool.

kernel/dma/swiotlb.c static unsigned long mem_pool_used(struct io_tlb_pool *pool) · line 1370

Quoted 8 lines
kernel/dma/swiotlb.c +3 -2
@@ -1335,13 +1365,20 @@ static unsigned long mem_pool_used(struct io_tlb_pool *pool)
  * mem_used() - get number of used slots in an allocator
  * @mem:	Software IO TLB allocator.
  *
- * The result is not accurate, because there is no locking of individual
- * areas.
+ * When trace_hiwater and CONFIG_DEBUG_FS is enabled, the result is accurate
+ * because the total number of used slots is tracked in mem->total_used.
+ * Otherwise, the result is not accurate, because there is no locking of

Rather than saying "the result is not accurate", I'd suggest saying "the result is an approximation". "Not accurate" sounds like it is of dubious usefulness, while an "approximation" can be useful. :-)

Quoted 25 lines
kernel/dma/swiotlb.c +7 -4
+ * individual areas.
  *
- * Return: Approximate number of used slots.
+ * Return: Number of used slots.
  */
 static unsigned long mem_used(struct io_tlb_mem *mem)
 {
+#ifdef CONFIG_DEBUG_FS
+    if (track_hiwater_enabled)
+	    return atomic_long_read(&mem->total_used);
+#endif
+
 #ifdef CONFIG_SWIOTLB_DYNAMIC
 	struct io_tlb_pool *pool;
 	unsigned long used = 0;
@@ -1357,8 +1394,6 @@ static unsigned long mem_used(struct io_tlb_mem *mem)
 #endif
 }
 
-#endif /* CONFIG_DEBUG_FS */
-
 /**
  * swiotlb_tbl_map_single() - bounce buffer map a single contiguous physical area
  * @dev:		Device which maps the buffer.
--
Quoted 2 lines

2.53.0

Modulo the few nits about the commit message, documentation, and comments, this looks good to me. I've tested on x86/x64 against linux-next20260726 with and without CONFIG_DEBUG_FS, and with and without the swiotlb=track_hiwater. All seems good.

#17 lore raw

PATCH v4 1/1 dma/swiotlb: decouple high watermark tracking from CONFIG_DEBUG_FS

In reply to Jun Miao

From: chenhuguanshen <[email protected]>

Under heavy concurrent DMA traffic on CoCo VMs, inc_used_and_hiwater() performs an atomic_long_add_return() plus a CAS loop on the global used_hiwater, and dec_used() performs an atomic_long_sub() on total_used. All CPUs contend on the same cacheline, causing measurable throughput degradation at scale.

Historically these counters were only compiled in under CONFIG_DEBUG_FS, which means production kernels with debugfs paid the atomic overhead unconditionally. Make the tracking boot-time opt-in instead so that it is disabled by default with near-zero overhead via static_call, and can be enabled via "swiotlb=track_hiwater" parameter on demand for debugging.

Note that when CONFIG_DEBUG_FS is enabled but hiwater tracking is disabled, the "io_tlb_used" metric reports an approximate value rather than an instantaneously exact one.

---
v1 -> v2:
- Change the patch title.
- Doing the exact hiwater calculation is dynamic and defaults to "off", dynamic config would replace being under #ifdef CONFIG_DEBUG_FS
- The mechanism used for dynamic config needs to be one that is selectable on the kernel boot line so that the exact hiwater mark during boot is easily available.

v2 -> v3:
- When track_hiwater is enabled, we keep the original precise hiwater calculation backed by the global total_used atomic counter. If disabled, we switch to the approximate approach that sums per-area counters to derive the total used slot count.

v3 -> v4:
- Change commit message and documentation.

---
 .../admin-guide/kernel-parameters.txt         |   4 +-
 include/linux/swiotlb.h                       |   8 +-
 kernel/dma/swiotlb.c                          | 153 +++++++++++-------
 3 files changed, 101 insertions(+), 64 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +3 -1
index b5493a7f8f22..eb3e8db2c178 100644
@@ -7477,7 +7477,7 @@ Kernel parameters
 			Execution Facility on pSeries.
 
 	swiotlb=	[ARM,PPC,MIPS,X86,S390,EARLY]
-			Format: { <int> [,<int>] | force | noforce }
+			Format: { <int> [,<int>] | force | noforce | track_hiwater}
 			<int> -- Number of I/O TLB slabs
 			<int> -- Second integer after comma. Number of swiotlb
 				 areas with their own lock. Will be rounded up
@@ -7485,6 +7485,8 @@ Kernel parameters
 			force -- force using of bounce buffers even if they
 			         wouldn't be automatically used by the kernel
 			noforce -- Never use bounce buffers (for debugging)
+			track_hiwater -- Track high watermark of swiotlb buffers.
+			                 Only available when CONFIG_DEBUG_FS is set.
 
 	switches=	[HW,M68k,EARLY]
 
include/linux/swiotlb.h +4 -4
index 3dae0f592063..008f169e4006 100644
@@ -98,10 +98,10 @@ struct io_tlb_pool {
  * @pools:	List of IO TLB memory pool descriptors (if dynamic).
  * @dyn_alloc:	Dynamic IO TLB pool allocation work.
  * @total_used:	The total number of slots in the pool that are currently used
- *		across all areas. Used only for calculating used_hiwater in
- *		debugfs.
- * @used_hiwater: The high water mark for total_used.  Used only for reporting
- *		in debugfs.
+ *		across all areas. Used only for calculating used_hiwater via boot
+ *		parameter swiotlb=track_hiwater and exposed via debugfs.
+ * @used_hiwater: The high water mark for total_used.  Can be enabled at boot
+ *		time via swiotlb=track_hiwater and exposed via debugfs.
  * @transient_nslabs: The total number of slots in all transient pools that
  *		are currently used across all areas.
  */
kernel/dma/swiotlb.c +94 -59

195 lines not on this page · raw message

Signature · 1 line
2.53.0
Reviewed-by Tested-by on v4 1/1
#18 lore raw

RE: [PATCH v4] dma/swiotlb: decouple high watermark tracking from CONFIG_DEBUG_FS

From: Frank Chen <[email protected]> Sent: Wednesday, August 12, 2026 12:05 AM

Quoted 35 lines

From: chenhuguanshen <[email protected]>

Under heavy concurrent DMA traffic on CoCo VMs, inc_used_and_hiwater() performs an atomic_long_add_return() plus a CAS loop on the global used_hiwater, and dec_used() performs an atomic_long_sub() on total_used. All CPUs contend on the same cacheline, causing measurable throughput degradation at scale.

Historically these counters were only compiled in under CONFIG_DEBUG_FS, which means production kernels with debugfs paid the atomic overhead unconditionally. Make the tracking boot-time opt-in instead so that it is disabled by default with near-zero overhead via static_call, and can be enabled via "swiotlb=track_hiwater" parameter on demand for debugging.

Note that when CONFIG_DEBUG_FS is enabled but hiwater tracking is disabled, the "io_tlb_used" metric reports an approximate value rather than an instantaneously exact one.

---
v1 -> v2:
- Change the patch title.
- Doing the exact hiwater calculation is dynamic and defaults to "off", dynamic config would replace being under #ifdef CONFIG_DEBUG_FS
- The mechanism used for dynamic config needs to be one that is selectable on the kernel boot line so that the exact hiwater mark during boot is easily available.

Quoted 8 lines

v2 -> v3:
- When track_hiwater is enabled, we keep the original precise hiwater calculation backed by the global total_used atomic counter. If disabled, we switch to the approximate approach that sums per-area counters to derive the total used slot count.

v3 -> v4:
- Change commit message and documentation.

LGTM.

maintainer Applied
#19 lore raw

Re: [PATCH v4] dma/swiotlb: decouple high watermark tracking from CONFIG_DEBUG_FS

Quoted 41 lines · Michael Kelley

From: Frank Chen <[email protected]> Sent: Wednesday, August 12, 2026 12:05 AM

Quoted 40 lines

From: chenhuguanshen <[email protected]>

Under heavy concurrent DMA traffic on CoCo VMs, inc_used_and_hiwater() performs an atomic_long_add_return() plus a CAS loop on the global used_hiwater, and dec_used() performs an atomic_long_sub() on total_used. All CPUs contend on the same cacheline, causing measurable throughput degradation at scale.

Historically these counters were only compiled in under CONFIG_DEBUG_FS, which means production kernels with debugfs paid the atomic overhead unconditionally. Make the tracking boot-time opt-in instead so that it is disabled by default with near-zero overhead via static_call, and can be enabled via "swiotlb=track_hiwater" parameter on demand for debugging.

Note that when CONFIG_DEBUG_FS is enabled but hiwater tracking is disabled, the "io_tlb_used" metric reports an approximate value rather than an instantaneously exact one.

---
v1 -> v2:
- Change the patch title.
- Doing the exact hiwater calculation is dynamic and defaults to "off", dynamic config would replace being under #ifdef CONFIG_DEBUG_FS
- The mechanism used for dynamic config needs to be one that is selectable on the kernel boot line so that the exact hiwater mark during boot is easily available.

v2 -> v3:
- When track_hiwater is enabled, we keep the original precise hiwater calculation backed by the global total_used atomic counter. If disabled, we switch to the approximate approach that sums per-area counters to derive the total used slot count.

Quoted 6 lines
Quoted 2 lines

v3 -> v4:
- Change commit message and documentation.

LGTM.

Applied to dma-mapping-for-next, thanks!

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland