Skip to contentSkip to conversation

swiotlb: Split up single swiotlb lock

PATCH Superseded Tianyu Lan 1 participant 1 message Jun 30, 2022

Subsystems (4)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Tianyu Lan

author
#0 lore raw

PATCH v2 1/1 swiotlb: Split up single swiotlb lock

From: Tianyu Lan <[email protected]>

Traditionally swiotlb was not performance critical because it was only used for slow devices. But in some setups, like TDX/SEV confidential guests, all IO has to go through swiotlb. Currently swiotlb only has a single lock. Under high IO load with multiple CPUs this can lead to significat lock contention on the swiotlb lock.

This patch splits the swiotlb bounce buffer pool into individual areas which have their own lock. Each CPU tries to allocate in its own area first. Only if that fails does it search other areas. On freeing the allocation is freed into the area where the memory was originally allocated from.

Area number can be set via swiotlb_adjust_nareas() and swiotlb kernel parameter.

This idea from Andi Kleen patch(https://github.com/intel/tdx/commit/ 4529b5784c141782c72ec9bd9a92df2b68cb7d45).

---
Change since v1:
* Move struct io_tlb_area to swiotlb.c
* Fix some coding style issue.

---
 .../admin-guide/kernel-parameters.txt         |   4 +-
 include/linux/swiotlb.h                       |   5 +
 kernel/dma/swiotlb.c                          | 218 ++++++++++++++----
 3 files changed, 187 insertions(+), 40 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +3 -1
index 2522b11e593f..4a6ad177d4b8 100644
@@ -5904,8 +5904,10 @@
 			it if 0 is given (See Documentation/admin-guide/cgroup-v1/memory.rst)
 
 	swiotlb=	[ARM,IA-64,PPC,MIPS,X86]
-			Format: { <int> | force | noforce }
+			Format: { <int> [,<int>] | force | noforce }
 			<int> -- Number of I/O TLB slabs
+			<int> -- Second integer after comma. Number of swiotlb
+				 areas with their own lock. Must be power of 2.
 			force -- force using of bounce buffers even if they
 			         wouldn't be automatically used by the kernel
 			noforce -- Never use bounce buffers (for debugging)
include/linux/swiotlb.h +5 -0
index 7ed35dd3de6e..5f898c5e9f19 100644
@@ -89,6 +89,8 @@ extern enum swiotlb_force swiotlb_force;
  * @late_alloc:	%true if allocated using the page allocator
  * @force_bounce: %true if swiotlb bouncing is forced
  * @for_alloc:  %true if the pool is used for memory allocation
+ * @nareas:  The area number in the pool.
+ * @area_nslabs: The slot number in the area.
  */
 struct io_tlb_mem {
 	phys_addr_t start;
@@ -102,6 +104,9 @@ struct io_tlb_mem {
 	bool late_alloc;
 	bool force_bounce;
 	bool for_alloc;
+	unsigned int nareas;
+	unsigned int area_nslabs;
+	struct io_tlb_area *areas;
 	struct io_tlb_slot {
 		phys_addr_t orig_addr;
 		size_t alloc_size;
kernel/dma/swiotlb.c +179 -39

403 lines not on this page · raw message

Signature · 1 line
2.25.1