swiotlb: Split up single swiotlb lock
Subsystems (4)
PATCH v3 1/1 swiotlb: Split up single swiotlb lock
From: Tianyu Lan <[email protected]>
Traditionally swiotlb was not performance critical because it was only used for slow devices. But in some setups, like TDX/SEV confidential guests, all IO has to go through swiotlb. Currently swiotlb only has a single lock. Under high IO load with multiple CPUs this can lead to significat lock contention on the swiotlb lock.
This patch splits the swiotlb bounce buffer pool into individual areas which have their own lock. Each CPU tries to allocate in its own area first. Only if that fails does it search other areas. On freeing the allocation is freed into the area where the memory was originally allocated from.
Area number can be set via swiotlb kernel parameter and is default to be possible cpu number. If possible cpu number is not power of 2, area number will be round up to the next power of 2.
This idea from Andi Kleen patch(https://github.com/intel/tdx/commit/ 4529b5784c141782c72ec9bd9a92df2b68cb7d45).
---
Change since v2:
* Use possible cpu number to adjust iotlb area number
Change since v1:
* Move struct io_tlb_area to swiotlb.c
* Fix some coding style issue.
--- .../admin-guide/kernel-parameters.txt | 4 +- include/linux/swiotlb.h | 5 + kernel/dma/swiotlb.c | 222 +++++++++++++++--- 3 files changed, 191 insertions(+), 40 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +3 -1
@@ -5904,8 +5904,10 @@ it if 0 is given (See Documentation/admin-guide/cgroup-v1/memory.rst) swiotlb= [ARM,IA-64,PPC,MIPS,X86] - Format: { <int> | force | noforce } + Format: { <int> [,<int>] | force | noforce } <int> -- Number of I/O TLB slabs + <int> -- Second integer after comma. Number of swiotlb + areas with their own lock. Must be power of 2. force -- force using of bounce buffers even if they wouldn't be automatically used by the kernel noforce -- Never use bounce buffers (for debugging)
include/linux/swiotlb.h +5 -0
@@ -89,6 +89,8 @@ extern enum swiotlb_force swiotlb_force;* @late_alloc: %true if allocated using the page allocator * @force_bounce: %true if swiotlb bouncing is forced * @for_alloc: %true if the pool is used for memory allocation +* @nareas: The area number in the pool. +* @area_nslabs: The slot number in the area. */ struct io_tlb_mem {phys_addr_t start;
@@ -102,6 +104,9 @@ struct io_tlb_mem {bool late_alloc;bool force_bounce;bool for_alloc; +unsigned int nareas; +unsigned int area_nslabs; +struct io_tlb_area *areas;struct io_tlb_slot {phys_addr_t orig_addr;size_t alloc_size;
kernel/dma/swiotlb.c +183 -39
407 lines not on this page · raw message
Signature · 1 line
2.25.1
Quoted 14 lines · Tianyu Lan
From: Tianyu Lan <[email protected]>
Traditionally swiotlb was not performance critical because it was only used for slow devices. But in some setups, like TDX/SEV confidential guests, all IO has to go through swiotlb. Currently swiotlb only has a single lock. Under high IO load with multiple CPUs this can lead to significat lock contention on the swiotlb lock.
This patch splits the swiotlb bounce buffer pool into individual areas which have their own lock. Each CPU tries to allocate in its own area first. Only if that fails does it search other areas. On freeing the allocation is freed into the area where the memory was originally allocated from.
Quoted 6 lines
Area number can be set via swiotlb kernel parameter and is default to be possible cpu number. If possible cpu number is not power of 2, area number will be round up to the next power of 2.
This idea from Andi Kleen patch(https://github.com/intel/tdx/commit/
4529b5784c141782c72ec9bd9a92df2b68cb7d45).
Thanks, this looks much better. I think there is a small problem with how default_nareas is set - we need to use 0 as the default so that an explicit command line value of 1 works. Als have you checked the interaction with swiotlb_adjust_size in detail?
kernel/dma/swiotlb.c +7 -6
@@ -70,7 +70,7 @@ struct io_tlb_mem io_tlb_default_mem;phys_addr_t swiotlb_unencrypted_base;static unsigned long default_nslabs =IO_TLB_DEFAULT_SIZE >>IO_TLB_SHIFT ; -static unsigned long default_nareas =1 ; +static unsigned long default_nareas;/** * struct io_tlb_area - IO TLB memory area descriptor
@@ -90,7 +90,10 @@ struct io_tlb_area {static void swiotlb_adjust_nareas (unsigned int nareas) { -if (!is_power_of_2 (nareas)) +if (default_nareas) +return ; + +if (nareas >1 && !is_power_of_2 (nareas)) nareas =roundup_pow_of_two (nareas); default_nareas = nareas;
@@ -338,8 +341,7 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags,panic ("%s: Failed to allocate %zu bytes align=0x%lx\n" , __func__, alloc_size,PAGE_SIZE ); -if (default_nareas ==1 ) -swiotlb_adjust_nareas (num_possible_cpus ()); +swiotlb_adjust_nareas (num_possible_cpus ()); mem->areas =memblock_alloc (sizeof (struct io_tlb_area ) * default_nareas,SMP_CACHE_BYTES );
@@ -410,8 +412,7 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, (PAGE_SIZE << order) >> 20); } -if (default_nareas ==1 ) -swiotlb_adjust_nareas (num_possible_cpus ()); +swiotlb_adjust_nareas (num_possible_cpus ()); area_order =get_order (array_size (sizeof (*mem->areas), default_nareas));
Quoted 4 lines · Christoph Hellwig
Thanks, this looks much better. I think there is a small problem with how
default_nareasis set - we need to use 0 as the default so that an explicit command line value of 1 works. Als have you checked the interaction withswiotlb_adjust_sizein detail?
Yes, the patch was tested in the Hyper-V SEV VM which always calls swiotlb_adjust_size() to adjust bounce buffer size according to memory
size. It will round up bounce buffer size to the next power of 2 if the memory size is not power of 2.