Skip to contentSkip to conversation

swiotlb: Split up single swiotlb lock

RFC PATCH Superseded Tianyu Lan 3 participants 4 messages Jun 17, 2022 – Jun 23, 2022

Subsystems (4)
lore
Share & tools
lore

Tianyu Lan

author
#0 lore raw

RFC PATCH v4 1/1 swiotlb: Split up single swiotlb lock

From: Tianyu Lan <[email protected]>

Traditionally swiotlb was not performance critical because it was only used for slow devices. But in some setups, like TDX/SEV confidential guests, all IO has to go through swiotlb. Currently swiotlb only has a single lock. Under high IO load with multiple CPUs this can lead to significat lock contention on the swiotlb lock.

This patch splits the swiotlb bounce buffer pool into individual areas which have their own lock. Each CPU tries to allocate in its own area first. Only if that fails does it search other areas. On freeing the allocation is freed into the area where the memory was originally allocated from.

Area number can be set via swiotlb_adjust_nareas() and swiotlb kernel parameter.

This idea from Andi Kleen patch(https://github.com/intel/tdx/commit/4529b578 4c141782c72ec9bd9a92df2b68cb7d45).

---
 .../admin-guide/kernel-parameters.txt         |   4 +-
 include/linux/swiotlb.h                       |  27 +++
 kernel/dma/swiotlb.c                          | 202 ++++++++++++++----
 3 files changed, 194 insertions(+), 39 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +3 -1
index 8090130b544b..5d46271982d5 100644
@@ -5869,8 +5869,10 @@
 			it if 0 is given (See Documentation/admin-guide/cgroup-v1/memory.rst)
 
 	swiotlb=	[ARM,IA-64,PPC,MIPS,X86]
-			Format: { <int> | force | noforce }
+			Format: { <int> [,<int>] | force | noforce }
 			<int> -- Number of I/O TLB slabs
+			<int> -- Second integer after comma. Number of swiotlb
+				 areas with their own lock. Must be power of 2.
 			force -- force using of bounce buffers even if they
 			         wouldn't be automatically used by the kernel
 			noforce -- Never use bounce buffers (for debugging)
include/linux/swiotlb.h +27 -0
index 7ed35dd3de6e..7157428cf3ac 100644
@@ -62,6 +62,22 @@ dma_addr_t swiotlb_map(struct device *dev, phys_addr_t phys,
 #ifdef CONFIG_SWIOTLB
 extern enum swiotlb_force swiotlb_force;
 
+/**
+ * struct io_tlb_area - IO TLB memory area descriptor
+ *
+ * This is a single area with a single lock.
+ *
+ * @used:	The number of used IO TLB block.
+ * @index:	The slot index to start searching in this area for next round.
+ * @lock:	The lock to protect the above data structures in the map and
+ *		unmap calls.
+ */
+struct io_tlb_area {
+	unsigned long used;
+	unsigned int index;
+	spinlock_t lock;
+};
+
 /**
  * struct io_tlb_mem - IO TLB Memory Pool Descriptor
  *
@@ -89,6 +105,8 @@ extern enum swiotlb_force swiotlb_force;
  * @late_alloc:	%true if allocated using the page allocator
  * @force_bounce: %true if swiotlb bouncing is forced
  * @for_alloc:  %true if the pool is used for memory allocation
+ * @nareas:  The area number in the pool.
+ * @area_nslabs: The slot number in the area.
  */
 struct io_tlb_mem {
 	phys_addr_t start;
@@ -102,6 +120,9 @@ struct io_tlb_mem {
 	bool late_alloc;
 	bool force_bounce;
 	bool for_alloc;
+	unsigned int nareas;
+	unsigned int area_nslabs;
+	struct io_tlb_area *areas;
 	struct io_tlb_slot {
 		phys_addr_t orig_addr;
 		size_t alloc_size;
@@ -130,6 +151,7 @@ unsigned int swiotlb_max_segment(void);
 size_t swiotlb_max_mapping_size(struct device *dev);
 bool is_swiotlb_active(struct device *dev);
 void __init swiotlb_adjust_size(unsigned long size);
+void __init swiotlb_adjust_nareas(unsigned int nareas);
 #else
 static inline void swiotlb_init(bool addressing_limited, unsigned int flags)
 {
@@ -162,6 +184,11 @@ static inline bool is_swiotlb_active(struct device *dev)
 static inline void swiotlb_adjust_size(unsigned long size)
 {
 }
+
+static inline void swiotlb_adjust_nareas(unsigned int nareas)
+{
+}
+
 #endif /* CONFIG_SWIOTLB */
 
 extern void swiotlb_print_info(void);
kernel/dma/swiotlb.c +164 -38

400 lines not on this page · raw message

Signature · 6 lines
2.25.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#1 lore raw

Thanks,

this looks pretty good to me. A few comments below:

Quoted 5 lines · Tianyu Lan
include/linux/swiotlb.h +5 -0
+/**
+ * struct io_tlb_area - IO TLB memory area descriptor
+ *
+ * This is a single area with a single lock.
+ *

include/linux/swiotlb.h dma_addr_t swiotlb_map(struct device *dev, phys_addr_t phys, · line 79

Quoted 10 lines
include/linux/swiotlb.h +10 -0
+ * @used:	The number of used IO TLB block.
+ * @index:	The slot index to start searching in this area for next round.
+ * @lock:	The lock to protect the above data structures in the map and
+ *		unmap calls.
+ */
+struct io_tlb_area {
+	unsigned long used;
+	unsigned int index;
+	spinlock_t lock;
+};

This can go into swiotlb.c.

Quoted 1 line

+void __init swiotlb_adjust_nareas(unsigned int nareas);

And this should be marked static.

Quoted 1 line

+#define DEFAULT_NUM_AREAS 1

I'd drop this define, the magic 1 and a > 1 comparism seems to convey how it is used much better as the checks aren't about default or not, but about larger than one.

I also think that we want some good way to size the default, e.g. by number of CPUs or memory size.

Quoted 4 lines
kernel/dma/swiotlb.c +4 -0
+void __init swiotlb_adjust_nareas(unsigned int nareas)
+{
+	if (!is_power_of_2(nareas)) {
+		pr_err("swiotlb: Invalid areas parameter %d.\n", nareas);

kernel/dma/swiotlb.c unsigned long swiotlb_size_or_default(void) · line 126

Quoted 10 lines
kernel/dma/swiotlb.c +10 -0
+		return;
+	}
+
+	default_nareas = nareas;
+
+	pr_info("area num %d.\n", nareas);
+	/* Round up number of slabs to the next power of 2.
+	 * The last area is going be smaller than the rest if
+	 * default_nslabs is not power of two.
+	 */

Please follow the normal kernel comment style with a /* on its own line.

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#2 lore raw

I will build the next RFC version of 64-bit swiotlb on top of this patch (or next version of this patch), so that it will render a more finalized view of 32-bt/64-bit plus multiple area.

Thank you very much!

Dongli Zhang

Quoted 53 lines · Christoph Hellwig

Thanks,

this looks pretty good to me. A few comments below:

Quoted 5 lines · Tianyu Lan
include/linux/swiotlb.h +5 -0
+/**
+ * struct io_tlb_area - IO TLB memory area descriptor
+ *
+ * This is a single area with a single lock.
+ *

include/linux/swiotlb.h dma_addr_t swiotlb_map(struct device *dev, phys_addr_t phys, · line 79

Quoted 10 lines
include/linux/swiotlb.h +10 -0
+ * @used:	The number of used IO TLB block.
+ * @index:	The slot index to start searching in this area for next round.
+ * @lock:	The lock to protect the above data structures in the map and
+ *		unmap calls.
+ */
+struct io_tlb_area {
+	unsigned long used;
+	unsigned int index;
+	spinlock_t lock;
+};

This can go into swiotlb.c.

Quoted 1 line

+void __init swiotlb_adjust_nareas(unsigned int nareas);

And this should be marked static.

Quoted 1 line

+#define DEFAULT_NUM_AREAS 1

I'd drop this define, the magic 1 and a > 1 comparism seems to convey how it is used much better as the checks aren't about default or not, but about larger than one.

I also think that we want some good way to size the default, e.g. by number of CPUs or memory size.

Quoted 4 lines
kernel/dma/swiotlb.c +4 -0
+void __init swiotlb_adjust_nareas(unsigned int nareas)
+{
+	if (!is_power_of_2(nareas)) {
+		pr_err("swiotlb: Invalid areas parameter %d.\n", nareas);

kernel/dma/swiotlb.c unsigned long swiotlb_size_or_default(void) · line 126

Quoted 10 lines
kernel/dma/swiotlb.c +10 -0
+		return;
+	}
+
+	default_nareas = nareas;
+
+	pr_info("area num %d.\n", nareas);
+	/* Round up number of slabs to the next power of 2.
+	 * The last area is going be smaller than the rest if
+	 * default_nslabs is not power of two.
+	 */

Please follow the normal kernel comment style with a /* on its own line.

Quoted 5 lines

_______________________________________________
iommu mailing list
[email protected]
https://urldefense.com/v3/__https://lists.linuxfoundation.org/mailman/listinfo/iommu__;!!ACWV5N9M2RV99hQ!Jd_DYgd6_uOF8IPr8h1tratEG51zFXtwVpaPa_OW3AEJlWe8gOnmA_fGOdaFUfsVcj1sT5oYw2j4vacY$

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author
#3 lore raw
In reply to Christoph Hellwig
Quoted 4 lines · Christoph Hellwig

Thanks,

this looks pretty good to me. A few comments below:

Thanks for your review.

Quoted 10 lines · Tianyu Lan
Quoted 9 lines

+/**
+ * struct io_tlb_area - IO TLB memory area descriptor
+ *
+ * This is a single area with a single lock.
+ *
+ * @used: The number of used IO TLB block.
+ * @index: The slot index to start searching in this area for next round.
+ * @lock: The lock to protect the above data structures in the map and
+ * unmap calls.

Quoted 8 lines

include/linux/swiotlb.h dma_addr_t swiotlb_map(struct device *dev, phys_addr_t phys, · line 79

Quoted 6 lines

+ */

include/linux/swiotlb.h +5 -0
+struct io_tlb_area {
+	unsigned long used;
+	unsigned int index;
+	spinlock_t lock;
+};

This can go into swiotlb.c.

struct io_tlb_area is used in the struct io_tlb_mem.

Quoted 5 lines
Quoted 2 lines

+void __init swiotlb_adjust_nareas(unsigned int nareas);

And this should be marked static.

Quoted 8 lines
Quoted 1 line

+#define DEFAULT_NUM_AREAS 1

I'd drop this define, the magic 1 and a > 1 comparism seems to convey how it is used much better as the checks aren't about default or not, but about larger than one.

I also think that we want some good way to size the default, e.g. by number of CPUs or memory size.

swiotlb_adjust_nareas() is exposed to platforms to set area number.
When swiotlb_init() is called, smp_init() isn't called at that point and
so standard API of checking cpu number (e.g, num_online_cpus()) doesn't
work. Platforms may have other ways to get cpu number(e.g x86 may ACPI
MADT table entries to get cpu nubmer) and set area number. I will post following patch to set cpu number via swiotlb_adjust_nareas(),

Quoted 18 lines
Quoted 5 lines
kernel/dma/swiotlb.c +4 -0
+void __init swiotlb_adjust_nareas(unsigned int nareas)
+{
+	if (!is_power_of_2(nareas)) {
+		pr_err("swiotlb: Invalid areas parameter %d.\n", nareas);

kernel/dma/swiotlb.c unsigned long swiotlb_size_or_default(void) · line 126

Quoted 10 lines
kernel/dma/swiotlb.c +10 -0
+		return;
+	}
+
+	default_nareas = nareas;
+
+	pr_info("area num %d.\n", nareas);
+	/* Round up number of slabs to the next power of 2.
+	 * The last area is going be smaller than the rest if
+	 * default_nslabs is not power of two.
+	 */

Please follow the normal kernel comment style with a /* on its own line.

OK. Will update.