swiotlb: Split up single swiotlb lock
Subsystems (4)
RFC PATCH v4 1/1 swiotlb: Split up single swiotlb lock
From: Tianyu Lan <[email protected]>
Traditionally swiotlb was not performance critical because it was only used for slow devices. But in some setups, like TDX/SEV confidential guests, all IO has to go through swiotlb. Currently swiotlb only has a single lock. Under high IO load with multiple CPUs this can lead to significat lock contention on the swiotlb lock.
This patch splits the swiotlb bounce buffer pool into individual areas which have their own lock. Each CPU tries to allocate in its own area first. Only if that fails does it search other areas. On freeing the allocation is freed into the area where the memory was originally allocated from.
Area number can be set via swiotlb_adjust_nareas() and swiotlb kernel parameter.
This idea from Andi Kleen patch(https://github.com/intel/tdx/commit/4529b578 4c141782c72ec9bd9a92df2b68cb7d45).
--- .../admin-guide/kernel-parameters.txt | 4 +- include/linux/swiotlb.h | 27 +++ kernel/dma/swiotlb.c | 202 ++++++++++++++---- 3 files changed, 194 insertions(+), 39 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +3 -1
@@ -5869,8 +5869,10 @@ it if 0 is given (See Documentation/admin-guide/cgroup-v1/memory.rst) swiotlb= [ARM,IA-64,PPC,MIPS,X86] - Format: { <int> | force | noforce } + Format: { <int> [,<int>] | force | noforce } <int> -- Number of I/O TLB slabs + <int> -- Second integer after comma. Number of swiotlb + areas with their own lock. Must be power of 2. force -- force using of bounce buffers even if they wouldn't be automatically used by the kernel noforce -- Never use bounce buffers (for debugging)
include/linux/swiotlb.h +27 -0
@@ -62,6 +62,22 @@ dma_addr_t swiotlb_map(struct device *dev, phys_addr_t phys,#ifdef CONFIG_SWIOTLB extern enum swiotlb_force swiotlb_force; +/** +* struct io_tlb_area - IO TLB memory area descriptor +* +* This is a single area with a single lock. +* +* @used: The number of used IO TLB block. +* @index: The slot index to start searching in this area for next round. +* @lock: The lock to protect the above data structures in the map and +* unmap calls. +*/ +struct io_tlb_area { +unsigned long used; +unsigned int index; +spinlock_t lock; +}; +/** * struct io_tlb_mem - IO TLB Memory Pool Descriptor *
@@ -89,6 +105,8 @@ extern enum swiotlb_force swiotlb_force;* @late_alloc: %true if allocated using the page allocator * @force_bounce: %true if swiotlb bouncing is forced * @for_alloc: %true if the pool is used for memory allocation +* @nareas: The area number in the pool. +* @area_nslabs: The slot number in the area. */ struct io_tlb_mem {phys_addr_t start;
@@ -102,6 +120,9 @@ struct io_tlb_mem {bool late_alloc;bool force_bounce;bool for_alloc; +unsigned int nareas; +unsigned int area_nslabs; +struct io_tlb_area *areas;struct io_tlb_slot {phys_addr_t orig_addr;size_t alloc_size;
@@ -130,6 +151,7 @@ unsigned int swiotlb_max_segment(void);size_t swiotlb_max_mapping_size (struct device *dev);bool is_swiotlb_active (struct device *dev);void __init swiotlb_adjust_size (unsigned long size); +void __init swiotlb_adjust_nareas (unsigned int nareas);#else static inline void swiotlb_init (bool addressing_limited,unsigned int flags) {
@@ -162,6 +184,11 @@ static inline bool is_swiotlb_active(struct device *dev)static inline void swiotlb_adjust_size (unsigned long size) { } + +static inline void swiotlb_adjust_nareas (unsigned int nareas) +{ +} +#endif /* CONFIG_SWIOTLB */ extern void swiotlb_print_info (void );
kernel/dma/swiotlb.c +164 -38
400 lines not on this page · raw message
Signature · 6 lines
2.25.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Thanks,
this looks pretty good to me. A few comments below:
Quoted 5 lines · Tianyu Lan
include/linux/swiotlb.h +5
-0+/** +* struct io_tlb_area - IO TLB memory area descriptor +* +* This is a single area with a single lock. +*
include/linux/swiotlb.h dma_addr_t swiotlb_map(struct device *dev, phys_addr_t phys, · line 79
Quoted 10 lines
include/linux/swiotlb.h +10
-0+* @used: The number of used IO TLB block. +* @index: The slot index to start searching in this area for next round. +* @lock: The lock to protect the above data structures in the map and +* unmap calls. +*/ +struct io_tlb_area { +unsigned long used; +unsigned int index; +spinlock_t lock; +};
This can go into swiotlb.c.
Quoted 1 line
+void
__initswiotlb_adjust_nareas(unsigned int nareas);
And this should be marked static.
Quoted 1 line
+#define
DEFAULT_NUM_AREAS1
I'd drop this define, the magic 1 and a > 1 comparism seems to convey how it is used much better as the checks aren't about default or not, but about larger than one.
I also think that we want some good way to size the default, e.g. by number of CPUs or memory size.
Quoted 4 lines
kernel/dma/swiotlb.c +4
-0+void __init swiotlb_adjust_nareas (unsigned int nareas) +{ +if (!is_power_of_2 (nareas)) { +pr_err ("swiotlb: Invalid areas parameter %d.\n" , nareas);
kernel/dma/swiotlb.c unsigned long swiotlb_size_or_default(void) · line 126
Quoted 10 lines
kernel/dma/swiotlb.c +10
-0+return ; + } + + default_nareas = nareas; + +pr_info ("area num %d.\n" , nareas); +/* Round up number of slabs to the next power of 2. +* The last area is going be smaller than the rest if +* default_nslabs is not power of two. +*/
Please follow the normal kernel comment style with a /* on its own line.
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
I will build the next RFC version of 64-bit swiotlb on top of this patch (or next version of this patch), so that it will render a more finalized view of 32-bt/64-bit plus multiple area.
Thank you very much!
Dongli Zhang
Quoted 53 lines · Christoph Hellwig
Thanks,
this looks pretty good to me. A few comments below:
Quoted 5 lines · Tianyu Lan
include/linux/swiotlb.h +5
-0+/** +* struct io_tlb_area - IO TLB memory area descriptor +* +* This is a single area with a single lock. +* include/linux/swiotlb.h dma_addr_t swiotlb_map(struct device *dev, phys_addr_t phys, · line 79
Quoted 10 lines
include/linux/swiotlb.h +10
-0+* @used: The number of used IO TLB block. +* @index: The slot index to start searching in this area for next round. +* @lock: The lock to protect the above data structures in the map and +* unmap calls. +*/ +struct io_tlb_area { +unsigned long used; +unsigned int index; +spinlock_t lock; +};This can go into
swiotlb.c.Quoted 1 line
+void
__initswiotlb_adjust_nareas(unsigned int nareas);And this should be marked static.
Quoted 1 line
+#define
DEFAULT_NUM_AREAS1I'd drop this define, the magic 1 and a > 1 comparism seems to convey how it is used much better as the checks aren't about default or not, but about larger than one.
I also think that we want some good way to size the default, e.g. by number of CPUs or memory size.
Quoted 4 lines
kernel/dma/swiotlb.c +4
-0+void __init swiotlb_adjust_nareas (unsigned int nareas) +{ +if (!is_power_of_2 (nareas)) { +pr_err ("swiotlb: Invalid areas parameter %d.\n" , nareas);kernel/dma/swiotlb.c unsigned long swiotlb_size_or_default(void) · line 126
Quoted 10 lines
kernel/dma/swiotlb.c +10
-0+return ; + } + + default_nareas = nareas; + +pr_info ("area num %d.\n" , nareas); +/* Round up number of slabs to the next power of 2. +* The last area is going be smaller than the rest if +* default_nslabs is not power of two. +*/ Please follow the normal kernel comment style with a /* on its own line.
Quoted 5 lines
_______________________________________________
iommu mailing list
[email protected]
https://urldefense.com/v3/__https://lists.linuxfoundation.org/mailman/listinfo/iommu__;!!ACWV5N9M2RV99hQ!Jd_DYgd6_uOF8IPr8h1tratEG51zFXtwVpaPa_OW3AEJlWe8gOnmA_fGOdaFUfsVcj1sT5oYw2j4vacY$
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 4 lines · Christoph Hellwig
Thanks,
this looks pretty good to me. A few comments below:
Thanks for your review.
Quoted 10 lines · Tianyu Lan
Quoted 9 lines
+/**
+ * structio_tlb_area- IO TLB memory area descriptor
+ *
+ * This is a single area with a single lock.
+ *
+ * @used: The number of used IO TLB block.
+ * @index: The slot index to start searching in this area for next round.
+ * @lock: The lock to protect the above data structures in the map and
+ * unmap calls.
Quoted 8 lines
include/linux/swiotlb.h dma_addr_t swiotlb_map(struct device *dev, phys_addr_t phys, · line 79
Quoted 6 lines
+ */
include/linux/swiotlb.h +5
-0+struct io_tlb_area { +unsigned long used; +unsigned int index; +spinlock_t lock; +};This can go into
swiotlb.c.
struct io_tlb_area is used in the struct io_tlb_mem.
Quoted 5 lines
Quoted 2 lines
+void
__initswiotlb_adjust_nareas(unsigned int nareas);And this should be marked static.
Quoted 8 lines
Quoted 1 line
+#define
DEFAULT_NUM_AREAS1I'd drop this define, the magic 1 and a > 1 comparism seems to convey how it is used much better as the checks aren't about default or not, but about larger than one.
I also think that we want some good way to size the default, e.g. by number of CPUs or memory size.
swiotlb_adjust_nareas() is exposed to platforms to set area number.
When swiotlb_init() is called, smp_init() isn't called at that point and
so standard API of checking cpu number (e.g, num_online_cpus()) doesn't
work. Platforms may have other ways to get cpu number(e.g x86 may ACPI
MADT table entries to get cpu nubmer) and set area number. I will post following patch to set cpu number via swiotlb_adjust_nareas(),
Quoted 18 lines
Quoted 5 lines
kernel/dma/swiotlb.c +4
-0+void __init swiotlb_adjust_nareas (unsigned int nareas) +{ +if (!is_power_of_2 (nareas)) { +pr_err ("swiotlb: Invalid areas parameter %d.\n" , nareas);kernel/dma/swiotlb.c unsigned long swiotlb_size_or_default(void) · line 126
Quoted 10 lines
kernel/dma/swiotlb.c +10
-0+return ; + } + + default_nareas = nareas; + +pr_info ("area num %d.\n" , nareas); +/* Round up number of slabs to the next power of 2. +* The last area is going be smaller than the rest if +* default_nslabs is not power of two. +*/ Please follow the normal kernel comment style with a /* on its own line.
OK. Will update.