Skip to contentSkip to conversation

swiotlb: Add a new cc-swiotlb implementation for Confidential VMs

RFC In review GuoRui.Yu 6 participants 19 messages Jan 28, 2023 – Feb 1, 2023

Subsystems (9)
lore
Share & tools
lore

GuoRui.Yu

author
#0 lore raw

This patch series adds a new swiotlb implementation, cc-swiotlb, for Confidential VMs (such as TDX and SEV-SNP). The new cc-swiotlb allocates the DMA TLB buffer dynamically in runtime instead of allocating at boot with a fixed size. Furthermore, future optimization and security enhancement could be applied on cc-swiotlb without "infecting" the legacy swiotlb.

Background
==========
Under COnfidential COmputing (CoCo) scenarios, the VMM cannot access guest memory directly but requires the guest to explicitly mark the memory as shared (decrypted). To make the streaming DMA mappings work, the current implementation relays on legacy SWIOTLB to bounce the DMA buffer between private (encrypted) and shared (decrypted) memory.

However, the legacy swiotlb is designed for compatibility rather than efficiency and CoCo purpose, which will inevitably introduce some unnecessary restrictions.

1. Fixed immutable swiotlb size cannot accommodate to requirements of multiple devices. And 1GiB (current maximum size) of swiotlb in our testbed cannot afford multiple disks reads/writes simultaneously.

2. Fixed immutable IO_TLB_SIZE (2KiB) cannot satisfy various kinds of devices. At the moment, the minimal size of a swiotlb buffer is 2KiB, which will waste memory on small network packets (under 256 bytes) and decrease efficiency on a large block (up to 256KiB) size reads/writes of disks. And it is hard to have a trade-off on legacy swiotlb to rule them all.

3. The legacy swiotlb cannot efficiently support larger swiotlb buffers. In the worst case, the current implementation requires a full scan of the entire swiotlb buffer, which can cause severe performance hits.

Changes in this patch set
=========================
Instead of keeping "infecting" the legacy swiotlb code with CoCo logic, this patch tries to introduce a new cc-swiotlb for Confidential VMs.

Confidential VMs usually have reasonable modern devices (virtio devices, NVME, etc.), which can access memory above 4GiB, cc-swiotlb could allocate TLB buffers at any position dynamically. Since set_memory_{decrypted,encrypted} is time-consuming and cannot be used in interrupt context, a new kernel thread "kccd" has been added to populate new TLB buffers on-demand, which solved the problem 1.

In addition, the cc-swiotlb manages TLB buffers by different sizes (512B, 2KiB, 4KiB, 16KiB, and 512KiB). The above values come from the following observations (boot with 8core, 32 GiB, 1 nvme disk, and 1 virtio-net):
- Allocations of 512 bytes and below account for 3.5% of the total DMA cache allocations;
- Allocations of 2 KiB and below account for 57.7%;
- Allocations of 4 KiB and below account for 91.3%;
- Allocations of 16 KiB and below account for 96.0%;
- Allocations of 512 KiB and below accounted for 100%;
- At the end of booting, cc-swiotlb uses 288 MiB in total.

For comparison, legacy swiotlb reserves memory at 6%, which requires min(1GiB, 32GiB * 0.06) = 1GiB, and will hang when operating multiple disks simultaneously due to no memory for the swiotlb buffer.

These patches were tested with fio (using different iodepth and block size) on a platform with 96 cores, 384 GiB, and 20 NVME disks, and no IO hang or error was observed.

For simplicity, the current RFC version cannot switch between legacy implementation with cmdline but through compile options. I am open to discussing how to integrate the cc-swiotlb into the legacy one.

Patch Organization
==================
- swiotlb: Split common code from swiotlb.{c,h}
- swiotlb: Add a new cc-swiotlb implementation for Confidential VMs
- swiotlb: Add tracepoint swiotlb_unbounced
- cc-swiotlb: Allow set swiotlb watermark from cmdline

Thanks for your time!

Have a nice day,
Guorui

author
#1 lore raw

PATCH 1/4 swiotlb: Split common code from swiotlb.{c,h}

Split swiotlb_bounce, swiotlb_release_slots, and is_swiotlb_buffer from swiotlb.{c,h} to common-swiotlb.c, and prepare for the new swiotlb implementaion.

---
 include/linux/swiotlb.h     | 10 ++---
 kernel/dma/Makefile         |  2 +-
 kernel/dma/common-swiotlb.c | 74 +++++++++++++++++++++++++++++++++
 kernel/dma/swiotlb.c        | 82 ++++---------------------------------
 4 files changed, 88 insertions(+), 80 deletions(-)
 create mode 100644 kernel/dma/common-swiotlb.c
include/linux/swiotlb.h +4 -6
index 35bc4e281c21..c5e74d3f9cbf 100644
@@ -58,6 +58,9 @@ void swiotlb_sync_single_for_cpu(struct device *dev, phys_addr_t tlb_addr,
 		size_t size, enum dma_data_direction dir);
 dma_addr_t swiotlb_map(struct device *dev, phys_addr_t phys,
 		size_t size, enum dma_data_direction dir, unsigned long attrs);
+void swiotlb_bounce(struct device *dev, phys_addr_t tlb_addr, size_t size,
+			   enum dma_data_direction dir);
+void swiotlb_release_slots(struct device *dev, phys_addr_t tlb_addr);
 
 #ifdef CONFIG_SWIOTLB
 
@@ -105,12 +108,7 @@ struct io_tlb_mem {
 };
 extern struct io_tlb_mem io_tlb_default_mem;
 
-static inline bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr)
-{
-	struct io_tlb_mem *mem = dev->dma_io_tlb_mem;
-
-	return mem && paddr >= mem->start && paddr < mem->end;
-}
+bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr);
 
 static inline bool is_swiotlb_force_bounce(struct device *dev)
 {
kernel/dma/Makefile +1 -1
index 21926e46ef4f..fc0ea13bc089 100644
@@ -6,7 +6,7 @@ obj-$(CONFIG_DMA_OPS)			+= dummy.o
 obj-$(CONFIG_DMA_CMA)			+= contiguous.o
 obj-$(CONFIG_DMA_DECLARE_COHERENT)	+= coherent.o
 obj-$(CONFIG_DMA_API_DEBUG)		+= debug.o
-obj-$(CONFIG_SWIOTLB)			+= swiotlb.o
+obj-$(CONFIG_SWIOTLB)			+= swiotlb.o common-swiotlb.o
 obj-$(CONFIG_DMA_COHERENT_POOL)		+= pool.o
 obj-$(CONFIG_MMU)			+= remap.o
 obj-$(CONFIG_DMA_MAP_BENCHMARK)		+= map_benchmark.o
kernel/dma/common-swiotlb.c new file +74 -0

74 lines not on this page · raw message

kernel/dma/swiotlb.c +9 -73

112 lines not on this page · raw message

Signature · 1 line
2.31.1
author
#2 lore raw

PATCH 2/4 swiotlb: Add a new cc-swiotlb implementation for Confidential VMs

In reply to GuoRui.Yu

Under COnfidential COmputing (CoCo) scenarios, the VMM cannot access guest memory directly but requires the guest to explicitly mark the memory as shared (decrypted). To make the streaming DMA mappings work, the current implementation relays on legacy SWIOTLB to bounce the DMA buffer between private (encrypted) and shared (decrypted) memory.

However, the legacy swiotlb is designed for compatibility rather than efficiency and CoCo purpose, which will inevitably introduce some unnecessary restrictions.
1. Fixed immutable swiotlb size cannot accommodate to requirements of multiple devices. And 1GiB (current maximum size) of swiotlb in our testbed cannot afford multiple disks reads/writes simultaneously.
2. Fixed immutable IO_TLB_SIZE (2KiB) cannot satisfy various kinds of devices. At the moment, the minimal size of a swiotlb buffer is 2KiB, which will waste memory on small network packets (under 512 bytes) and decrease efficiency on a large block (up to 256KiB) size reads/writes of disks. And it is hard to have a trade-off on legacy swiotlb to rule them all.
3. The legacy swiotlb cannot efficiently support larger swiotlb buffers. In the worst case, the current implementation requires a full scan of the entire swiotlb buffer, which can cause severe performance hits.

Instead of keeping "infecting" the legacy swiotlb code with CoCo logic, this patch tries to introduce a new cc-swiotlb for Confidential VMs.

Confidential VMs usually have reasonable modern devices (virtio devices, NVME, etc.), which can access memory above 4GiB, cc-swiotlb could allocate TLB buffers dynamically on-demand, and this design solves problem 1.

In addition, the cc-swiotlb manages TLB buffers by different sizes (512B, 2KiB, 4KiB, 16KiB, and 512KiB), which solves problems 2 and 3.

---
 arch/x86/Kconfig             |   3 +-
 arch/x86/include/asm/iommu.h |   2 +-
 arch/x86/kernel/pci-dma.c    |   6 +-
 drivers/base/core.c          |   2 +-
 drivers/iommu/amd/Kconfig    |   2 +-
 drivers/iommu/dma-iommu.c    |   2 +-
 drivers/iommu/intel/Kconfig  |   2 +-
 include/linux/device.h       |   2 +-
 include/linux/swiotlb.h      |  19 +-
 kernel/dma/Kconfig           |  10 +
 kernel/dma/Makefile          |   1 +
 kernel/dma/cc-swiotlb.c      | 492 +++++++++++++++++++++++++++++++++++
 kernel/dma/common-swiotlb.c  |   1 +
 kernel/dma/direct.c          |   4 +-
 kernel/dma/direct.h          |   4 +-
 15 files changed, 537 insertions(+), 15 deletions(-)
 create mode 100644 kernel/dma/cc-swiotlb.c
arch/x86/Kconfig +2 -1
index 3604074a878b..bce367fc4b55 100644
@@ -31,7 +31,7 @@ config X86_64
 	select HAVE_ARCH_SOFT_DIRTY
 	select MODULES_USE_ELF_RELA
 	select NEED_DMA_MAP_STATE
-	select SWIOTLB
+	select SWIOTLB if !X86_MEM_ENCRYPT
 	select ARCH_HAS_ELFCORE_COMPAT
 	select ZONE_DMA32
 
@@ -1535,6 +1535,7 @@ config X86_CPA_STATISTICS
 config X86_MEM_ENCRYPT
 	select ARCH_HAS_FORCE_DMA_UNENCRYPTED
 	select DYNAMIC_PHYSICAL_MASK
+	select SWIOTLB if !CC_SWIOTLB
 	def_bool n
 
 config AMD_MEM_ENCRYPT
arch/x86/include/asm/iommu.h +1 -1
index 2fd52b65deac..a4fbe92b2321 100644
@@ -11,7 +11,7 @@ extern int iommu_detected;
 extern int iommu_merge;
 extern int panic_on_overflow;
 
-#ifdef CONFIG_SWIOTLB
+#if defined(CONFIG_SWIOTLB) || defined(CONFIG_CC_SWIOTLB)
 extern bool x86_swiotlb_enable;
 #else
 #define x86_swiotlb_enable false
arch/x86/kernel/pci-dma.c +3 -3
index 30bbe4abb5d6..41ad97bfc5be 100644
@@ -37,7 +37,7 @@ int no_iommu __read_mostly;
 /* Set this to 1 if there is a HW IOMMU in the system */
 int iommu_detected __read_mostly = 0;
 
-#ifdef CONFIG_SWIOTLB
+#if defined(CONFIG_SWIOTLB) || defined(CONFIG_CC_SWIOTLB)
 bool x86_swiotlb_enable;
 static unsigned int x86_swiotlb_flags;
 
@@ -169,7 +169,7 @@ static __init int iommu_setup(char *p)
 			disable_dac_quirk = true;
 			return 1;
 		}
-#ifdef CONFIG_SWIOTLB
+#if defined(CONFIG_SWIOTLB) || defined(CONFIG_CC_SWIOTLB)
 		if (!strncmp(p, "soft", 4))
 			x86_swiotlb_enable = true;
 #endif
@@ -192,7 +192,7 @@ static int __init pci_iommu_init(void)
 {
 	x86_init.iommu.iommu_init();
 
-#ifdef CONFIG_SWIOTLB
+#if defined(CONFIG_SWIOTLB) || defined(CONFIG_CC_SWIOTLB)
 	/* An IOMMU turned us off. */
 	if (x86_swiotlb_enable) {
 		pr_info("PCI-DMA: Using software bounce buffering for IO (SWIOTLB)\n");
drivers/base/core.c +1 -1
index a3e14143ec0c..0a0b35f8b5c3 100644
@@ -2955,7 +2955,7 @@ void device_initialize(struct device *dev)
     defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL)
 	dev->dma_coherent = dma_default_coherent;
 #endif
-#ifdef CONFIG_SWIOTLB
+#if defined(CONFIG_SWIOTLB) || defined(CONFIG_CC_SWIOTLB)
 	dev->dma_io_tlb_mem = &io_tlb_default_mem;
 #endif
 }
drivers/iommu/amd/Kconfig +1 -1
index 9b5fc3356bf2..f8dcb67d02cc 100644
@@ -2,7 +2,7 @@
 # AMD IOMMU support
 config AMD_IOMMU
 	bool "AMD IOMMU support"
-	select SWIOTLB
+	select SWIOTLB if !X86_MEM_ENCRYPT
 	select PCI_MSI
 	select PCI_ATS
 	select PCI_PRI
drivers/iommu/dma-iommu.c +1 -1
index f798c44e0903..06f6ad1ef59c 100644
@@ -519,7 +519,7 @@ static bool dev_is_untrusted(struct device *dev)
 
 static bool dev_use_swiotlb(struct device *dev)
 {
-	return IS_ENABLED(CONFIG_SWIOTLB) && dev_is_untrusted(dev);
+	return (IS_ENABLED(CONFIG_SWIOTLB) || IS_ENABLED(CONFIG_CC_SWIOTLB)) && dev_is_untrusted(dev);
 }
 
 /**
drivers/iommu/intel/Kconfig +1 -1
index b7dff5092fd2..1f56f78b1880 100644
@@ -17,7 +17,7 @@ config INTEL_IOMMU
 	select IOMMU_IOVA
 	select NEED_DMA_MAP_STATE
 	select DMAR_TABLE
-	select SWIOTLB
+	select SWIOTLB if !X86_MEM_ENCRYPT
 	select IOASID
 	select PCI_ATS
 	select PCI_PRI
include/linux/device.h +1 -1
index 44e3acae7b36..b5b082d7d1de 100644
@@ -607,7 +607,7 @@ struct device {
 	struct cma *cma_area;		/* contiguous memory area for dma
 					   allocations */
 #endif
-#ifdef CONFIG_SWIOTLB
+#if defined(CONFIG_SWIOTLB) || defined(CONFIG_CC_SWIOTLB)
 	struct io_tlb_mem *dma_io_tlb_mem;
 #endif
 	/* arch specific additions */
include/linux/swiotlb.h +18 -1
index c5e74d3f9cbf..0384784ebce4 100644
@@ -106,6 +106,23 @@ struct io_tlb_mem {
 	struct io_tlb_area *areas;
 	struct io_tlb_slot *slots;
 };
+#elif defined(CONFIG_CC_SWIOTLB)
+
+/**
+ * struct io_tlb_mem - io tlb memory pool descriptor
+ *
+ * @force_bounce: %true if swiotlb bouncing is forced
+ * @mapping: the mapping relations between the END address of backing memory and
+ * 	     original DMA address.
+ */
+struct io_tlb_mem {
+	bool force_bounce;
+	struct xarray* mapping;
+	struct dentry *debugfs;
+};
+#endif
+
+#if defined(CONFIG_SWIOTLB) || defined(CONFIG_CC_SWIOTLB)
 extern struct io_tlb_mem io_tlb_default_mem;
 
 bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr);
@@ -155,7 +172,7 @@ static inline bool is_swiotlb_active(struct device *dev)
 static inline void swiotlb_adjust_size(unsigned long size)
 {
 }
-#endif /* CONFIG_SWIOTLB */
+#endif /* CONFIG_SWIOTLB || CONFIG_CC_SWIOTLB */
 
 extern void swiotlb_print_info(void);
 
kernel/dma/Kconfig +10 -0
index 56866aaa2ae1..7e6b20d2091f 100644
@@ -78,8 +78,18 @@ config ARCH_HAS_FORCE_DMA_UNENCRYPTED
 
 config SWIOTLB
 	bool
+	depends on !CC_SWIOTLB
 	select NEED_DMA_MAP_STATE
 
+config CC_SWIOTLB
+	bool "Enable cc-swiotlb for Confidential VMs"
+	default n
+	select NEED_DMA_MAP_STATE
+	help
+	  This enables a cc-swiotlb implementation for Confidential VMs,
+	  which allows allocating the SWIOTLB buffer allocation on runtime.
+	  If unsure, say "n".
+
 config DMA_RESTRICTED_POOL
 	bool "DMA Restricted Pool"
 	depends on OF && OF_RESERVED_MEM && SWIOTLB
kernel/dma/Makefile +1 -0
index fc0ea13bc089..80c68292da3d 100644
@@ -7,6 +7,7 @@ obj-$(CONFIG_DMA_CMA)			+= contiguous.o
 obj-$(CONFIG_DMA_DECLARE_COHERENT)	+= coherent.o
 obj-$(CONFIG_DMA_API_DEBUG)		+= debug.o
 obj-$(CONFIG_SWIOTLB)			+= swiotlb.o common-swiotlb.o
+obj-$(CONFIG_CC_SWIOTLB)		+= cc-swiotlb.o common-swiotlb.o
 obj-$(CONFIG_DMA_COHERENT_POOL)		+= pool.o
 obj-$(CONFIG_MMU)			+= remap.o
 obj-$(CONFIG_DMA_MAP_BENCHMARK)		+= map_benchmark.o
kernel/dma/cc-swiotlb.c new file +492 -0

492 lines not on this page · raw message

kernel/dma/common-swiotlb.c +1 -0
index d477d5f2a71b..553325d5c9cd 100644
@@ -1,4 +1,5 @@
 /* SPDX-License-Identifier: GPL-2.0 */
+#include <linux/cc_platform.h>
 #include <linux/dma-direct.h>
 #include <linux/swiotlb.h>
 
kernel/dma/direct.c +2 -2
index 63859a101ed8..67a0ff973732 100644
@@ -405,7 +405,7 @@ void dma_direct_free_pages(struct device *dev, size_t size,
 }
 
 #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \
-    defined(CONFIG_SWIOTLB)
+    defined(CONFIG_SWIOTLB) || defined(CONFIG_CC_SWIOTLB)
 void dma_direct_sync_sg_for_device(struct device *dev,
 		struct scatterlist *sgl, int nents, enum dma_data_direction dir)
 {
@@ -428,7 +428,7 @@ void dma_direct_sync_sg_for_device(struct device *dev,
 
 #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \
     defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \
-    defined(CONFIG_SWIOTLB)
+    defined(CONFIG_SWIOTLB) || defined(CONFIG_CC_SWIOTLB)
 void dma_direct_sync_sg_for_cpu(struct device *dev,
 		struct scatterlist *sgl, int nents, enum dma_data_direction dir)
 {
kernel/dma/direct.h +2 -2
index e38ffc5e6bdd..e3050ae59839 100644
@@ -23,7 +23,7 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents,
 size_t dma_direct_max_mapping_size(struct device *dev);
 
 #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \
-    defined(CONFIG_SWIOTLB)
+    defined(CONFIG_SWIOTLB) || defined(CONFIG_CC_SWIOTLB)
 void dma_direct_sync_sg_for_device(struct device *dev, struct scatterlist *sgl,
 		int nents, enum dma_data_direction dir);
 #else
@@ -35,7 +35,7 @@ static inline void dma_direct_sync_sg_for_device(struct device *dev,
 
 #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \
     defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \
-    defined(CONFIG_SWIOTLB)
+    defined(CONFIG_SWIOTLB) || defined(CONFIG_CC_SWIOTLB)
 void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl,
 		int nents, enum dma_data_direction dir, unsigned long attrs);
 void dma_direct_sync_sg_for_cpu(struct device *dev,
Signature · 1 line
2.31.1
author
#12 lore raw

PATCH 3/4 swiotlb: Add tracepoint swiotlb_unbounced

In reply to GuoRui.Yu

Add a new tracepoint swiotlb_unbounced to facilitate statistics of swiotlb buffer usage.

---
 include/trace/events/swiotlb.h | 28 ++++++++++++++++++++++++++++
 kernel/dma/common-swiotlb.c    |  1 +
 2 files changed, 29 insertions(+)
include/trace/events/swiotlb.h +28 -0
index da05c9ebd224..0707adc34df5 100644
@@ -35,6 +35,34 @@ TRACE_EVENT(swiotlb_bounced,
 		__entry->force ? "FORCE" : "NORMAL")
 );
 
+TRACE_EVENT(swiotlb_unbounced,
+	TP_PROTO(struct device *dev, dma_addr_t dev_addr, size_t size),
+	TP_ARGS(dev, dev_addr, size),
+
+	TP_STRUCT__entry(
+		__string(dev_name, dev_name(dev))
+		__field(u64, dma_mask)
+		__field(dma_addr_t, dev_addr)
+		__field(size_t, size)
+		__field(bool, force)
+	),
+
+	TP_fast_assign(
+		__assign_str(dev_name, dev_name(dev));
+		__entry->dma_mask = (dev->dma_mask ? *dev->dma_mask : 0);
+		__entry->dev_addr = dev_addr;
+		__entry->size = size;
+		__entry->force = is_swiotlb_force_bounce(dev);
+	),
+
+	TP_printk("dev_name: %s dma_mask=%llx dev_addr=%llx size=%zu %s",
+		__get_str(dev_name),
+		__entry->dma_mask,
+		(unsigned long long)__entry->dev_addr,
+		__entry->size,
+		__entry->force ? "FORCE" : "NORMAL")
+);
+
 #endif /*  _TRACE_SWIOTLB_H */
 
 /* This part must be outside protection */
kernel/dma/common-swiotlb.c +1 -0
index 553325d5c9cd..e3676a8358f2 100644
@@ -46,6 +46,7 @@ void swiotlb_tbl_unmap_single(struct device *dev, phys_addr_t tlb_addr,
 			      size_t mapping_size, enum dma_data_direction dir,
 			      unsigned long attrs)
 {
+	trace_swiotlb_unbounced(dev, phys_to_dma(dev, tlb_addr), mapping_size);
 	/*
 	 * First, sync the memory before unmapping the entry
 	 */
Signature · 1 line
2.31.1
author
#13 lore raw

PATCH 4/4 cc-swiotlb: Allow set swiotlb watermark from cmdline

In reply to GuoRui.Yu
---
 .../admin-guide/kernel-parameters.txt         | 11 ++++++
 kernel/dma/cc-swiotlb.c                       | 34 ++++++++++++++++++-
 2 files changed, 44 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +11 -0
index 6cfa6e3996cf..ad510e392998 100644
@@ -6090,6 +6090,17 @@
 			         wouldn't be automatically used by the kernel
 			noforce -- Never use bounce buffers (for debugging)
 
+	cc_swiotlb=	[X86]
+			Format: { [<int>, <int>, <int>, <int>]| force ]}
+			Set the swiotlb TLB buffer initial size for Confidential Computing like
+			TDX and SEV-SNP.
+			<int> -- Number of I/O TLB of 256 bytes
+			<int> -- Number of I/O TLB of 4 KiB
+			<int> -- Number of I/O TLB of 64 KiB
+			<int> -- Number of I/O TLB of 256 KiB
+			force -- force using of cc bounce buffers even if they
+			         wouldn't be automatically used by the kernel (for debugging)
+
 	switches=	[HW,M68k]
 
 	sysctl.*=	[KNL]
kernel/dma/cc-swiotlb.c +33 -1
index e85d3ce2272f..ff1cc52189b9 100644
@@ -52,6 +52,32 @@ enum SLOT_SIZE {
 
 const unsigned int SLOT_SIZES[NR_OF_SIZES] = {512, 2048, 4096, 16 * 1024, 512 * 1024};
 unsigned int WATERMARK_SLOT_SIZE[NR_OF_SIZES] = {256, 16384, 8192, 1024, 32};
+static bool swiotlb_force_bounce = false;
+
+static int __init
+setup_io_tlb_watermark(char *str)
+{
+	int i;
+	size_t size;
+
+	for (i = 0; i < NR_OF_SIZES; i++) {
+		if (isdigit(*str)) {
+			size = simple_strtoul(str, &str, 0);
+			WATERMARK_SLOT_SIZE[i] = size;
+
+			if (*str == ',')
+				++str;
+		}
+		else
+			break;
+	}
+
+	if (!strcmp(str, "force"))
+		swiotlb_force_bounce = true;
+
+	return 0;
+}
+early_param("cc_swiotlb", setup_io_tlb_watermark);
 
 struct slot_terrace {
 	struct list_head slots_by_size[NR_OF_SIZES];
@@ -330,6 +356,7 @@ static void cc_populate_pages(void)
 	size_t i, j, k;
 	const size_t max_bytes_per_alloc = (MAX_ORDER_NR_PAGES << PAGE_SHIFT);
 	size_t remain_alloc_size, rounds, round_size, round_slot_nr;
+	size_t size = 0;
 
 	for (i = 0; i < NR_OF_SIZES; i++) {
 		if (atomic_read(&terrace.free_count_by_size[i]) > WATERMARK_SLOT_SIZE[i])
@@ -377,6 +404,11 @@ static void cc_populate_pages(void)
 			spin_unlock_irqrestore(&terrace.lock_by_size[i], flags);
 		}
 	}
+
+	for (i = 0; i < NR_OF_SIZES; i++) {
+		size += (size_t)atomic_read(&terrace.total_count_by_size[i]) * SLOT_SIZES[i];
+	}
+	pr_info("bounce buffer size adjusted to %luMB", size >> 20);
 }
 
 static int kccd(void *p)
@@ -437,7 +469,7 @@ void __init swiotlb_init(bool addressing_limit, unsigned int flags)
 		return;
 	}
 
-	io_tlb_default_mem.force_bounce = cc_platform_has(CC_ATTR_MEM_ENCRYPT);
+	io_tlb_default_mem.force_bounce = cc_platform_has(CC_ATTR_MEM_ENCRYPT) || swiotlb_force_bounce;
 
 	xa_init_flags(io_tlb_default_mem.mapping, 0);
 
Signature · 1 line
2.31.1
author
#15 lore raw
In reply to GuoRui.Yu

+ Marek Szyprowski and Robin Murphy.

Quoted 82 lines · GuoRui.Yu

This patch series adds a new swiotlb implementation, cc-swiotlb, for Confidential VMs (such as TDX and SEV-SNP). The new cc-swiotlb allocates the DMA TLB buffer dynamically in runtime instead of allocating at boot with a fixed size. Furthermore, future optimization and security enhancement could be applied on cc-swiotlb without "infecting" the legacy swiotlb.

Background
==========
Under COnfidential COmputing (CoCo) scenarios, the VMM cannot access guest memory directly but requires the guest to explicitly mark the memory as shared (decrypted). To make the streaming DMA mappings work, the current implementation relays on legacy SWIOTLB to bounce the DMA buffer between private (encrypted) and shared (decrypted) memory.

However, the legacy swiotlb is designed for compatibility rather than efficiency and CoCo purpose, which will inevitably introduce some unnecessary restrictions.

1. Fixed immutable swiotlb size cannot accommodate to requirements of multiple devices. And 1GiB (current maximum size) of swiotlb in our testbed cannot afford multiple disks reads/writes simultaneously.

2. Fixed immutable IO_TLB_SIZE (2KiB) cannot satisfy various kinds of devices. At the moment, the minimal size of a swiotlb buffer is 2KiB, which will waste memory on small network packets (under 256 bytes) and decrease efficiency on a large block (up to 256KiB) size reads/writes of disks. And it is hard to have a trade-off on legacy swiotlb to rule them all.

3. The legacy swiotlb cannot efficiently support larger swiotlb buffers. In the worst case, the current implementation requires a full scan of the entire swiotlb buffer, which can cause severe performance hits.

Changes in this patch set
=========================
Instead of keeping "infecting" the legacy swiotlb code with CoCo logic, this patch tries to introduce a new cc-swiotlb for Confidential VMs.

Confidential VMs usually have reasonable modern devices (virtio devices, NVME, etc.), which can access memory above 4GiB, cc-swiotlb could allocate TLB buffers at any position dynamically. Since set_memory_{decrypted,encrypted} is time-consuming and cannot be used in interrupt context, a new kernel thread "kccd" has been added to populate new TLB buffers on-demand, which solved the problem 1.

In addition, the cc-swiotlb manages TLB buffers by different sizes (512B, 2KiB, 4KiB, 16KiB, and 512KiB). The above values come from the following observations (boot with 8core, 32 GiB, 1 nvme disk, and 1 virtio-net):
- Allocations of 512 bytes and below account for 3.5% of the total DMA cache allocations;
- Allocations of 2 KiB and below account for 57.7%;
- Allocations of 4 KiB and below account for 91.3%;
- Allocations of 16 KiB and below account for 96.0%;
- Allocations of 512 KiB and below accounted for 100%;
- At the end of booting, cc-swiotlb uses 288 MiB in total.

For comparison, legacy swiotlb reserves memory at 6%, which requires min(1GiB, 32GiB * 0.06) = 1GiB, and will hang when operating multiple disks simultaneously due to no memory for the swiotlb buffer.

These patches were tested with fio (using different iodepth and block size) on a platform with 96 cores, 384 GiB, and 20 NVME disks, and no IO hang or error was observed.

For simplicity, the current RFC version cannot switch between legacy implementation with cmdline but through compile options. I am open to discussing how to integrate the cc-swiotlb into the legacy one.

Patch Organization
==================
- swiotlb: Split common code from swiotlb.{c,h}
- swiotlb: Add a new cc-swiotlb implementation for Confidential VMs
- swiotlb: Add tracepoint swiotlb_unbounced
- cc-swiotlb: Allow set swiotlb watermark from cmdline

Thanks for your time!

Have a nice day,
Guorui

#3 lore raw
In reply to GuoRui.Yu
Thank you for the patch! Perhaps something to improve:

Hi GuoRui.Yu",

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on tip/x86/core]
[also build test WARNING on joro-iommu/next driver-core/driver-core-testing driver-core/driver-core-next driver-core/driver-core-linus linus/master v6.2-rc5 next-20230127] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/GuoRui-Yu/swiotlb-Add-a-new-cc-swiotlb-implementation-for-Confidential-VMs/20230128-181154
patch link:    https://lore.kernel.org/r/20230128083254.86012-3-GuoRui.Yu%40linux.alibaba.com
patch subject: [PATCH 2/4] swiotlb: Add a new cc-swiotlb implementation for Confidential VMs
config: alpha-allyesconfig (https://download.01.org/0day-ci/archive/20230128/[email protected]/config)
compiler: alpha-linux-gcc (GCC) 12.1.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/9345d2c75b5994cec46ad0abf12f01ed91742e81
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review GuoRui-Yu/swiotlb-Add-a-new-cc-swiotlb-implementation-for-Confidential-VMs/20230128-181154 git checkout 9345d2c75b5994cec46ad0abf12f01ed91742e81
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=alpha olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=alpha SHELL=/bin/bash kernel/dma/

If you fix the issue, kindly add following tag where applicable | Reported-by: kernel test robot <[email protected]>

All warnings (new ones prefixed by >>):

kernel/dma/cc-swiotlb.c: In function 'swiotlb_bounce':

>> kernel/dma/cc-swiotlb.c:174:32: warning: variable 'tlb_end' set but not used [-Wunused-but-set-variable]
     174 |         phys_addr_t orig_addr, tlb_end, slot_start, slot_end = tlb_addr ;
         |                                ^~~~~~~
   kernel/dma/cc-swiotlb.c: At top level:
>> kernel/dma/cc-swiotlb.c:492:13: warning: no previous prototype for 'swiotlb_hint_cpus' [-Wmissing-prototypes]
     492 | void __init swiotlb_hint_cpus(int cpus) {}
         |             ^~~~~~~~~~~~~~~~~

vim +/tlb_end +174 kernel/dma/cc-swiotlb.c

167
168 /*
169 * Bounce: copy the swiotlb buffer from or back to the original dma location
170 */
171 void swiotlb_bounce(struct device *dev, phys_addr_t tlb_addr, size_t size,
172 enum dma_data_direction dir)
173 {

 > 174		phys_addr_t orig_addr, tlb_end, slot_start, slot_end = tlb_addr ;

175 struct io_tlb_mem *mem = dev->dma_io_tlb_mem;
176 struct io_tlb_slot *slot;
177 unsigned long pfn;
178 void *backing_addr;
179 size_t alloc_size;
180
181 slot = xa_find_after(mem->mapping, (unsigned long*)&slot_end, ULONG_MAX, XA_PRESENT);
182 if (!slot) {
183 dev_WARN_ONCE(dev, 1, "TLB buffer not found.\n");
184 return;
185 }
186
187 orig_addr = slot->orig_addr;
188 alloc_size = slot->alloc_size;
189 slot_start = slot_end - slot->orig_size;
190 tlb_end = tlb_addr + size;
191
192 pfn = PFN_DOWN(orig_addr);
193 if (PageHighMem(pfn_to_page(pfn))) {
194 dev_WARN_ONCE(dev, 1, "Unexpected high memory.\n");
195 return;
196 }
197
198 if (size > alloc_size) {
199 dev_WARN_ONCE(dev, 1,
200 "Buffer overflow detected. Allocation size: %zu. Mapping size: %zu.\n",
201 alloc_size, size);
202 size = alloc_size;
203 }
204
205 backing_addr = slot->va + (tlb_addr - slot_start);
206
207 if (dir == DMA_TO_DEVICE) {
208 memcpy(backing_addr, phys_to_virt(orig_addr), size);
209 } else {
210 memcpy(phys_to_virt(orig_addr), backing_addr, size);
211 }
212 }
213

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
reviewer
#4 lore raw
In reply to GuoRui.Yu

Hi--

Quoted 8 lines · GuoRui.Yu
kernel/dma/Kconfig +0 -0
index 56866aaa2ae1..7e6b20d2091f 100644
@@ -78,8 +78,18 @@ config ARCH_HAS_FORCE_DMA_UNENCRYPTED
 
 config SWIOTLB
 	bool

kernel/dma/Kconfig config ARCH_HAS_FORCE_DMA_UNENCRYPTED · line 90

Quoted 10 lines
kernel/dma/Kconfig +8 -0
+	depends on !CC_SWIOTLB
 	select NEED_DMA_MAP_STATE
 
+config CC_SWIOTLB
+	bool "Enable cc-swiotlb for Confidential VMs"
+	default n
+	select NEED_DMA_MAP_STATE
+	help
+	  This enables a cc-swiotlb implementation for Confidential VMs,
+	  which allows allocating the SWIOTLB buffer allocation on runtime.

Two "allocat..." words seems to be redundant. Probably the second one can be dropped.
Also, instead of "on runtime", how about "at runtime"?

kernel/dma/Kconfig config ARCH_HAS_FORCE_DMA_UNENCRYPTED · line 91

Quoted 1 line
kernel/dma/Kconfig +1 -0
+	  If unsure, say "n".

Thanks.

Signature · 1 line
~Randy
#14 lore raw

Re: [PATCH 4/4] cc-swiotlb: Allow set swiotlb watermark from cmdline

In reply to GuoRui.Yu
Thank you for the patch! Perhaps something to improve:

Hi GuoRui.Yu",

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on tip/x86/core]
[also build test WARNING on joro-iommu/next driver-core/driver-core-testing driver-core/driver-core-next driver-core/driver-core-linus linus/master v6.2-rc5 next-20230127] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/GuoRui-Yu/swiotlb-Add-a-new-cc-swiotlb-implementation-for-Confidential-VMs/20230128-181154
patch link:    https://lore.kernel.org/r/20230128083254.86012-5-GuoRui.Yu%40linux.alibaba.com
patch subject: [PATCH 4/4] cc-swiotlb: Allow set swiotlb watermark from cmdline
config: nios2-buildonly-randconfig-r002-20230129 (https://download.01.org/0day-ci/archive/20230129/[email protected]/config)
compiler: nios2-linux-gcc (GCC) 12.1.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/fcbd538c875f5f1086682829671935d7155e4029
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review GuoRui-Yu/swiotlb-Add-a-new-cc-swiotlb-implementation-for-Confidential-VMs/20230128-181154 git checkout fcbd538c875f5f1086682829671935d7155e4029
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=nios2 olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=nios2 SHELL=/bin/bash kernel/dma/

If you fix the issue, kindly add following tag where applicable | Reported-by: kernel test robot <[email protected]>

All warnings (new ones prefixed by >>):

kernel/dma/cc-swiotlb.c: In function 'swiotlb_bounce':
kernel/dma/cc-swiotlb.c:200:32: warning: variable 'tlb_end' set but not used [-Wunused-but-set-variable]

     200 |         phys_addr_t orig_addr, tlb_end, slot_start, slot_end = tlb_addr ;
         |                                ^~~~~~~
   In file included from kernel/dma/cc-swiotlb.c:13:
   kernel/dma/cc-swiotlb.c: In function 'cc_populate_pages':
>> include/linux/kern_levels.h:5:25: warning: format '%lu' expects argument of type 'long unsigned int', but argument 2 has type 'size_t' {aka 'unsigned int'} [-Wformat=]
       5 | #define KERN_SOH        "\001"          /* ASCII Start Of Header */
         |                         ^~~~~~
   include/linux/printk.h:429:25: note: in definition of macro 'printk_index_wrap'
     429 |                 _p_func(_fmt, ##__VA_ARGS__);                           \
         |                         ^~~~
   include/linux/printk.h:530:9: note: in expansion of macro 'printk'
     530 |         printk(KERN_INFO pr_fmt(fmt), ##__VA_ARGS__)
         |         ^~~~~~
   include/linux/kern_levels.h:14:25: note: in expansion of macro 'KERN_SOH'
      14 | #define KERN_INFO       KERN_SOH "6"    /* informational */
         |                         ^~~~~~~~
   include/linux/printk.h:530:16: note: in expansion of macro 'KERN_INFO'
     530 |         printk(KERN_INFO pr_fmt(fmt), ##__VA_ARGS__)
         |                ^~~~~~~~~
   kernel/dma/cc-swiotlb.c:411:9: note: in expansion of macro 'pr_info'
     411 |         pr_info("bounce buffer size adjusted to %luMB", size >> 20);
         |         ^~~~~~~
   kernel/dma/cc-swiotlb.c: At top level:
   kernel/dma/cc-swiotlb.c:524:13: warning: no previous prototype for 'swiotlb_hint_cpus' [-Wmissing-prototypes]
     524 | void __init swiotlb_hint_cpus(int cpus) {}
         |             ^~~~~~~~~~~~~~~~~

vim +5 include/linux/kern_levels.h

314ba3520e513a Joe Perches 2012-07-30 4
04d2c8c83d0e3a Joe Perches 2012-07-30 @5 #define KERN_SOH "\001" /* ASCII Start Of Header */
04d2c8c83d0e3a Joe Perches 2012-07-30 6 #define KERN_SOH_ASCII '\001'
04d2c8c83d0e3a Joe Perches 2012-07-30 7

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
author
#5 lore raw
In reply to Randy Dunlap

Hi Randy,

Quoted 14 lines · Randy Dunlap

Hi--

Quoted 11 lines · GuoRui.Yu
kernel/dma/Kconfig +1 -0
index 56866aaa2ae1..7e6b20d2091f 100644
@@ -78,8 +78,18 @@ config ARCH_HAS_FORCE_DMA_UNENCRYPTED
 
  config SWIOTLB
  	bool
+	depends on !CC_SWIOTLB
  	select NEED_DMA_MAP_STATE
 
Quoted 11 lines

kernel/dma/Kconfig config ARCH_HAS_FORCE_DMA_UNENCRYPTED · line 90

Quoted 7 lines
kernel/dma/Kconfig +7 -0
+config CC_SWIOTLB
+	bool "Enable cc-swiotlb for Confidential VMs"
+	default n
+	select NEED_DMA_MAP_STATE
+	help
+	  This enables a cc-swiotlb implementation for Confidential VMs,
+	  which allows allocating the SWIOTLB buffer allocation on runtime.

Two "allocat..." words seems to be redundant. Probably the second one can be dropped.
Also, instead of "on runtime", how about "at runtime"?

Thanks for your kindly advice, and I will fix it in the next version.

Quoted 4 lines

kernel/dma/Kconfig config ARCH_HAS_FORCE_DMA_UNENCRYPTED · line 91

Quoted 2 lines
kernel/dma/Kconfig +1 -0
+	  If unsure, say "n".

Thanks.

Sincerely,
Guorui

#6 lore raw
In reply to GuoRui.Yu
Quoted 22 lines · GuoRui.Yu

Under COnfidential COmputing (CoCo) scenarios, the VMM cannot access guest memory directly but requires the guest to explicitly mark the memory as shared (decrypted). To make the streaming DMA mappings work, the current implementation relays on legacy SWIOTLB to bounce the DMA buffer between private (encrypted) and shared (decrypted) memory.

However, the legacy swiotlb is designed for compatibility rather than efficiency and CoCo purpose, which will inevitably introduce some unnecessary restrictions.
1. Fixed immutable swiotlb size cannot accommodate to requirements of multiple devices. And 1GiB (current maximum size) of swiotlb in our testbed cannot afford multiple disks reads/writes simultaneously.
2. Fixed immutable IO_TLB_SIZE (2KiB) cannot satisfy various kinds of devices. At the moment, the minimal size of a swiotlb buffer is 2KiB, which will waste memory on small network packets (under 512 bytes) and decrease efficiency on a large block (up to 256KiB) size reads/writes of disks. And it is hard to have a trade-off on legacy swiotlb to rule them all.
3. The legacy swiotlb cannot efficiently support larger swiotlb buffers. In the worst case, the current implementation requires a full scan of the entire swiotlb buffer, which can cause severe performance hits.

Quoted 7 lines

Instead of keeping "infecting" the legacy swiotlb code with CoCo logic, this patch tries to introduce a new cc-swiotlb for Confidential VMs.

Confidential VMs usually have reasonable modern devices (virtio devices, NVME, etc.), which can access memory above 4GiB, cc-swiotlb could allocate TLB buffers dynamically on-demand, and this design solves problem 1.

When you say solving you mean support for growing the size dynamically without pre-allocation?

The IOMMU is traditionally called in non preemptible regions in drivers, and also allocating memory in IO paths is still not considered fully safe due to potential deadlocks. Both makes it difficult to allocate large memory regions dynamically.

It's not clear how you would solve that?

-Andi

author
#7 lore raw
Quoted 32 lines · Andi Kleen
Quoted 22 lines · GuoRui.Yu

Under COnfidential COmputing (CoCo) scenarios, the VMM cannot access guest memory directly but requires the guest to explicitly mark the memory as shared (decrypted). To make the streaming DMA mappings work, the current implementation relays on legacy SWIOTLB to bounce the DMA buffer between private (encrypted) and shared (decrypted) memory.

However, the legacy swiotlb is designed for compatibility rather than efficiency and CoCo purpose, which will inevitably introduce some unnecessary restrictions.
1. Fixed immutable swiotlb size cannot accommodate to requirements of multiple devices. And 1GiB (current maximum size) of swiotlb in our testbed cannot afford multiple disks reads/writes simultaneously.
2. Fixed immutable IO_TLB_SIZE (2KiB) cannot satisfy various kinds of devices. At the moment, the minimal size of a swiotlb buffer is 2KiB, which will waste memory on small network packets (under 512 bytes) and decrease efficiency on a large block (up to 256KiB) size reads/writes of disks. And it is hard to have a trade-off on legacy swiotlb to rule them all.
3. The legacy swiotlb cannot efficiently support larger swiotlb buffers. In the worst case, the current implementation requires a full scan of the entire swiotlb buffer, which can cause severe performance hits.

Quoted 7 lines

Instead of keeping "infecting" the legacy swiotlb code with CoCo logic, this patch tries to introduce a new cc-swiotlb for Confidential VMs.

Confidential VMs usually have reasonable modern devices (virtio devices, NVME, etc.), which can access memory above 4GiB, cc-swiotlb could allocate TLB buffers dynamically on-demand, and this design solves problem 1.

Quoted 7 lines

When you say solving you mean support for growing the size dynamically without pre-allocation?

The IOMMU is traditionally called in non preemptible regions in drivers, and also allocating memory in IO paths is still not considered fully safe due to potential deadlocks. Both makes it difficult to allocate large memory regions dynamically.

It's not clear how you would solve that?

-Andi

Hi Andi,

Thanks for your question!

I try to solve this problem by creating a new kernel thread, "kccd", to populate the TLB buffer in the backgroud.

Specifically,
1. A new kernel thread is created with the help of "arch_initcall", and this kthread is responsible for memory allocation and setting memory attributes (private or shared);
2. The "swiotlb_tbl_map_single" routine only use the spin_lock protected TLB buffers pre-allocated by the kthread;
a) which actually includes ONE memory allocation brought by xarray
insertion "__xa_insert__".
3. After each allocation, the water level of TLB resources will be checked. If the current TLB resources are found to be lower than the preset value (half of the watermark), the kthread will be awakened to fill them.
4. The TLB buffer allocation in the kthread is batched to "(MAX_ORDER_NR_PAGES << PAGE_SHIFT)" to reduce the holding time of spin_lock and number of calls to set_memory_decrypted().

Thanks,
Guorui

#8 lore raw
Quoted 7 lines

I try to solve this problem by creating a new kernel thread, "kccd", to populate the TLB buffer in the backgroud.

Specifically,
1. A new kernel thread is created with the help of "arch_initcall", and this kthread is responsible for memory allocation and setting memory attributes (private or shared);
2. The "swiotlb_tbl_map_single" routine only use the spin_lock protected TLB buffers pre-allocated by the kthread;
  a) which actually includes ONE memory allocation brought by xarray insertion "__xa_insert__".

That already seems dangerous with all the usual problems of memory allocations in IO paths. Normally code at least uses a mempool to avoid the worst dead lock potential.

Quoted 2 lines

3. After each allocation, the water level of TLB resources will be checked. If the current TLB resources are found to be lower than the preset value (half of the watermark), the kthread will be awakened to fill them.
4. The TLB buffer allocation in the kthread is batched to "(MAX_ORDER_NR_PAGES << PAGE_SHIFT)" to reduce the holding time of spin_lock and number of calls to set_memory_decrypted().

Okay, but does this guarantee that it will never run out of memory?

It seems difficult to make such guarantees. What happens for example if the background thread gets starved by something higher priority?

Or if the allocators have such high bandwidth that they can overwhelm any reasonable background thread.

-Andi

maintainer
#17 lore raw
In reply to GuoRui.Yu
Quoted 15 lines · GuoRui.Yu

This patch series adds a new swiotlb implementation, cc-swiotlb, for Confidential VMs (such as TDX and SEV-SNP). The new cc-swiotlb allocates the DMA TLB buffer dynamically in runtime instead of allocating at boot with a fixed size. Furthermore, future optimization and security enhancement could be applied on cc-swiotlb without "infecting" the legacy swiotlb.

Background
==========
Under COnfidential COmputing (CoCo) scenarios, the VMM cannot access guest memory directly but requires the guest to explicitly mark the memory as shared (decrypted). To make the streaming DMA mappings work, the current implementation relays on legacy SWIOTLB to bounce the DMA buffer between private (encrypted) and shared (decrypted) memory.

Quoted 7 lines

However, the legacy swiotlb is designed for compatibility rather than efficiency and CoCo purpose, which will inevitably introduce some unnecessary restrictions.

1. Fixed immutable swiotlb size cannot accommodate to requirements of multiple devices. And 1GiB (current maximum size) of swiotlb in our testbed cannot afford multiple disks reads/writes simultaneously.

That's not a very logical argument - if a particular use-case needs a particular total amount of SWIOTLB capacity, then that's how much it needs. Whether that capacity is allocated up-front or allocated gradually over time doesn't affect that. The obvious solution to this issue as presented is "make the maximum size bigger", not "add a whole other SWIOTLB implementation".

Quoted 6 lines

2. Fixed immutable IO_TLB_SIZE (2KiB) cannot satisfy various kinds of devices. At the moment, the minimal size of a swiotlb buffer is 2KiB, which will waste memory on small network packets (under 256 bytes) and decrease efficiency on a large block (up to 256KiB) size reads/writes of disks. And it is hard to have a trade-off on legacy swiotlb to rule them all.

That's clearly a general argument, so why should any improvement be arbitrarily restricted to Confidential Compute scenarios?

Quoted 3 lines

3. The legacy swiotlb cannot efficiently support larger swiotlb buffers. In the worst case, the current implementation requires a full scan of the entire swiotlb buffer, which can cause severe performance hits.

Isn't that the main reason we just recently introduced the multiple area stuff?

Robin.

Quoted 49 lines

Changes in this patch set
=========================
Instead of keeping "infecting" the legacy swiotlb code with CoCo logic, this patch tries to introduce a new cc-swiotlb for Confidential VMs.

Confidential VMs usually have reasonable modern devices (virtio devices, NVME, etc.), which can access memory above 4GiB, cc-swiotlb could allocate TLB buffers at any position dynamically. Since set_memory_{decrypted,encrypted} is time-consuming and cannot be used in interrupt context, a new kernel thread "kccd" has been added to populate new TLB buffers on-demand, which solved the problem 1.

In addition, the cc-swiotlb manages TLB buffers by different sizes (512B, 2KiB, 4KiB, 16KiB, and 512KiB). The above values come from the following observations (boot with 8core, 32 GiB, 1 nvme disk, and 1 virtio-net):
- Allocations of 512 bytes and below account for 3.5% of the total DMA cache allocations;
- Allocations of 2 KiB and below account for 57.7%;
- Allocations of 4 KiB and below account for 91.3%;
- Allocations of 16 KiB and below account for 96.0%;
- Allocations of 512 KiB and below accounted for 100%;
- At the end of booting, cc-swiotlb uses 288 MiB in total.

For comparison, legacy swiotlb reserves memory at 6%, which requires min(1GiB, 32GiB * 0.06) = 1GiB, and will hang when operating multiple disks simultaneously due to no memory for the swiotlb buffer.

These patches were tested with fio (using different iodepth and block size) on a platform with 96 cores, 384 GiB, and 20 NVME disks, and no IO hang or error was observed.

For simplicity, the current RFC version cannot switch between legacy implementation with cmdline but through compile options. I am open to discussing how to integrate the cc-swiotlb into the legacy one.

Patch Organization
==================
- swiotlb: Split common code from swiotlb.{c,h}
- swiotlb: Add a new cc-swiotlb implementation for Confidential VMs
- swiotlb: Add tracepoint swiotlb_unbounced
- cc-swiotlb: Allow set swiotlb watermark from cmdline

Thanks for your time!

Have a nice day,
Guorui

author
#9 lore raw
In reply to Andi Kleen

Hi Andi,

Quoted 10 lines · Andi Kleen
Quoted 7 lines

I try to solve this problem by creating a new kernel thread, "kccd", to populate the TLB buffer in the backgroud.

Specifically,
1. A new kernel thread is created with the help of "arch_initcall", and this kthread is responsible for memory allocation and setting memory attributes (private or shared);
2. The "swiotlb_tbl_map_single" routine only use the spin_lock protected TLB buffers pre-allocated by the kthread;
  a) which actually includes ONE memory allocation brought by xarray insertion "__xa_insert__".

That already seems dangerous with all the usual problems of memory allocations in IO paths. Normally code at least uses a mempool to avoid the worst dead lock potential.

The __xa_insert__ is called with GFP_NOWAIT (GFP_ATOMIC & ~__GFP_HIGH), and I will try to dig into this to check if there is any chance to have the deadlock.

I also tried my best to test this piece of code, and no issue have been found in the case of a maximum of 700,000 IOPS.

Thanks for your advices from this point, since I have not notice such possibility.

Quoted 7 lines
Quoted 2 lines

3. After each allocation, the water level of TLB resources will be checked. If the current TLB resources are found to be lower than the preset value (half of the watermark), the kthread will be awakened to fill them.
4. The TLB buffer allocation in the kthread is batched to "(MAX_ORDER_NR_PAGES << PAGE_SHIFT)" to reduce the holding time of spin_lock and number of calls to set_memory_decrypted().

Okay, but does this guarantee that it will never run out of memory?

It seems difficult to make such guarantees. What happens for example if the background thread gets starved by something higher priority?

No, this cannot guarantee we always have sufficient TLB caches, so we can also have a "No memory for cc-swiotlb buffer" warning.

But I want to emphasize that in this case, the current implementation is no worse than the legacy implementation. Moreover, dynamic TLB allocation is more suitable for situations where more disks/network devices will be hotplugged, in which case you cannot pre-set a reasonable value.

Quoted 4 lines

Or if the allocators have such high bandwidth that they can overwhelm any reasonable background thread.

-Andi

Sincerely,
Guorui

author
#18 lore raw
In reply to Robin Murphy

Hi Robin,

Quoted 20 lines · Robin Murphy
Quoted 19 lines · GuoRui.Yu

This patch series adds a new swiotlb implementation, cc-swiotlb, for Confidential VMs (such as TDX and SEV-SNP). The new cc-swiotlb allocates the DMA TLB buffer dynamically in runtime instead of allocating at boot with a fixed size. Furthermore, future optimization and security enhancement could be applied on cc-swiotlb without "infecting" the legacy swiotlb.

Background
==========
Under COnfidential COmputing (CoCo) scenarios, the VMM cannot access guest memory directly but requires the guest to explicitly mark the memory as shared (decrypted). To make the streaming DMA mappings work, the current implementation relays on legacy SWIOTLB to bounce the DMA buffer between private (encrypted) and shared (decrypted) memory.

However, the legacy swiotlb is designed for compatibility rather than efficiency and CoCo purpose, which will inevitably introduce some unnecessary restrictions.

Quoted 6 lines
Quoted 3 lines

1. Fixed immutable swiotlb size cannot accommodate to requirements of multiple devices. And 1GiB (current maximum size) of swiotlb in our testbed cannot afford multiple disks reads/writes simultaneously.

That's not a very logical argument - if a particular use-case needs a particular total amount of SWIOTLB capacity, then that's how much it needs. Whether that capacity is allocated up-front or allocated gradually over time doesn't affect that. The obvious solution to this issue as presented is "make the maximum size bigger", not "add a whole other SWIOTLB implementation".

The reason is all about the hotplugged devices.

In public cloud scenarios, customers often need to hotplug disks. The number of disks can be up to about 50 disks, and the maximum IOPS of each disk may reach 1 million IOPS. It may be difficult for the user to estimate the SWIOTLB size he needs to use when the system starts. And the same story also applies to the NIC devices.

Quoted 9 lines
Quoted 6 lines

2. Fixed immutable IO_TLB_SIZE (2KiB) cannot satisfy various kinds of devices. At the moment, the minimal size of a swiotlb buffer is 2KiB, which will waste memory on small network packets (under 256 bytes) and decrease efficiency on a large block (up to 256KiB) size reads/writes of disks. And it is hard to have a trade-off on legacy swiotlb to rule them all.

That's clearly a general argument, so why should any improvement be arbitrarily restricted to Confidential Compute scenarios?

My idea is that other scenarios can generally support modern high-speed devices without swiotlb.

But almost all DMA allocations in CoCo scenarios need to go through swiotlb for bouncing DMA buffer between the private and shared memory, which makes this problem more prominent in CoCo scenarios.

I guess the same logic may also apply to Xen, but I am not sure so it is CoCo-limited feature for now.

Quoted 6 lines
Quoted 3 lines

3. The legacy swiotlb cannot efficiently support larger swiotlb buffers. In the worst case, the current implementation requires a full scan of the entire swiotlb buffer, which can cause severe performance hits.

Isn't that the main reason we just recently introduced the multiple area stuff?

Yes, this issue is mitigated by multiple area stuff partially, and I am actually facing some issues current scanning logic (for example, it keeps scan the tlb buffer forever and causes soft lookups[1]). And I will keep track this issue (candidate) to give your more details about this.

But my more intuitive thought is why do we need such a complex allocation algorithm? If we only supported CoCo scenarios (or maybe Xen), we might be able to have a more deterministic algorithm instead of waiting for the results of the scan.

[1] some details about the soft lookup:

$ cat ./test.sh
count=0
while true; do
        for i in {1..21};do
                mkfs.xfs -f /dev/nvme${i}n1 &> /dev/null &
        done
        wait
        count=$((count+1))
        echo $count
done
$ ./test.sh # It may take some time...
$ # The system is not fully operational from here...
$ echo l > /proc/sysrq-trigger
...

Every other CPU are waiting this core to exit the swiotlb_do_find_slots loop.

[10199.924385] NMI backtrace for cpu 49
[10199.924386] CPU: 49 PID: 951 Comm: kworker/49:1H Kdump: loaded Tainted: G            EL     6.2.0-rc6-swiotlb-upstream-bug-repreduce+ #70
[10199.924388] Workqueue: kblockd blk_mq_run_work_fn
[10199.924391] RIP: 0010:swiotlb_do_find_slots+0x1fe/0x3e0
[10199.924393] Code: 77 21 48 8b 54 24 28 44 8b 7c 24 38 4c 8b 52 48 48 8d 14 49 48 c1 e2 03 45 39 7c 12 10 0f 83 f7 00 00 00 01 e8 bb 00 00 00 00 <41> 39 c0 0f 46 c3 39 f8 0f 85 4f ff ff f
f 48 8b 74 24 40 48 8b 7c
[10199.924394] RSP: 0018:ffffc90003913bb0 EFLAGS: 00000002
[10199.924395] RAX: 00000000000000e5 RBX: 0000000000000000 RCX: 00000000001880e1
[10199.924396] RDX: 00000000024c1518 RSI: 0000000000188000 RDI: 000000000000030c
[10199.924397] RBP: 0000000000000004 R08: 0000000000008000 R09: 0000000000000002
[10199.924397] R10: ffff88dad2400000 R11: 00000000001fffff R12: 00000000d8400000
[10199.924398] R13: 000000012cc2a800 R14: 0000000000200000 R15: 0000000000000002
[10199.924399] FS:  0000000000000000(0000) GS:ffff88dce5040000(0000) knlGS:0000000000000000
[10199.924400] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[10199.924401] CR2: 00007f8eb9dece80 CR3: 000000000640a004 CR4: 0000000000770ee0
[10199.924401] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[10199.924402] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[10199.924403] PKRU: 55555554
[10199.924403] Call Trace:
[10199.924404]  <TASK>
[10199.924405]  swiotlb_tbl_map_single+0xec/0x1f0
[10199.924407]  swiotlb_map+0x5c/0x260
[10199.924409]  ? nvme_pci_setup_prps+0x1ed/0x340
[10199.924411]  dma_direct_map_page+0x12e/0x1c0
[10199.924413]  nvme_map_data+0x304/0x370
[10199.924415]  nvme_prep_rq.part.0+0x31/0x120
[10199.924417]  nvme_queue_rq+0x77/0x1f0
[10199.924420]  blk_mq_dispatch_rq_list+0x17e/0x670
[10199.924422]  __blk_mq_sched_dispatch_requests+0x129/0x140
[10199.924424]  blk_mq_sched_dispatch_requests+0x34/0x60
[10199.924426]  __blk_mq_run_hw_queue+0x91/0xb0
[10199.924428]  process_one_work+0x1df/0x3b0
[10199.924430]  worker_thread+0x49/0x2e0
[10199.924432]  ? rescuer_thread+0x390/0x390
[10199.924433]  kthread+0xe5/0x110
[10199.924435]  ? kthread_complete_and_exit+0x20/0x20
[10199.924436]  ret_from_fork+0x1f/0x30
[10199.924439]  </TASK>

...

[ 9639.596311] NMI backtrace for cpu 48
[ 9639.596313] CPU: 48 PID: 1215 Comm: kworker/48:1H Kdump: loaded Tainted: G            E      6.2.0-rc6-swiotlb-upstream-bug-repreduce+ #70
[ 9639.596315] Workqueue: kblockd blk_mq_run_work_fn
[ 9639.596319] RIP: 0010:native_queued_spin_lock_slowpath+0x18/0x2c0
[ 9639.596322] Code: 75 02 5a c3 56 0f b6 f0 e8 c5 ff ff ff 5e 5a c3 66 90 0f 1f 44 00 00 41 55 41 54 55 48 89 fd 53 66 90 ba 01 00 00 00 8b 45 00 <85> c0 75 10 f0 0f b1 55 00 85 c0 75 f0 5
b 5d 41 5c 41 5d c3 f3 90
[ 9639.596323] RSP: 0018:ffffc90003e5fb38 EFLAGS: 00000002
[ 9639.596325] RAX: 0000000000000001 RBX: ffff88de5e459adc RCX: 0000000000001000
[ 9639.596326] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88de5e459adc
[ 9639.596327] RBP: ffff88de5e459adc R08: 0000000000000000 R09: ffff888102ab6090
[ 9639.596328] R10: ffff88dad2400000 R11: 00000000001fffff R12: 0000000000000293
[ 9639.596329] R13: 000000000730d000 R14: 0000000000200000 R15: ffffffff83540160
[ 9639.596330] FS:  0000000000000000(0000) GS:ffff88dce5000000(0000) knlGS:0000000000000000
[ 9639.596331] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 9639.596332] CR2: 00007fc091da0000 CR3: 000000000640a002 CR4: 0000000000770ee0
[ 9639.596334] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 9639.596334] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[ 9639.596335] PKRU: 55555554
[ 9639.596336] Call Trace:
[ 9639.596337]  <TASK>
[ 9639.596338]  _raw_spin_lock_irqsave+0x37/0x40
[ 9639.596341]  swiotlb_do_find_slots+0xef/0x3e0
[ 9639.596344]  swiotlb_tbl_map_single+0xec/0x1f0
[ 9639.596347]  swiotlb_map+0x5c/0x260
[ 9639.596349]  dma_direct_map_sg+0x7a/0x280
[ 9639.596352]  __dma_map_sg_attrs+0x30/0x70
[ 9639.596355]  dma_map_sgtable+0x1d/0x30
[ 9639.596356]  nvme_map_data+0xce/0x370
[ 9639.596359]  nvme_prep_rq.part.0+0x31/0x120
[ 9639.596362]  nvme_queue_rq+0x77/0x1f0
[ 9639.596364]  blk_mq_dispatch_rq_list+0x17e/0x670
[ 9639.596367]  __blk_mq_sched_dispatch_requests+0x129/0x140
[ 9639.596370]  blk_mq_sched_dispatch_requests+0x34/0x60
[ 9639.596372]  __blk_mq_run_hw_queue+0x91/0xb0
[ 9639.596374]  process_one_work+0x1df/0x3b0
[ 9639.596377]  worker_thread+0x49/0x2e0
[ 9639.596379]  ? rescuer_thread+0x390/0x390
[ 9639.596380]  kthread+0xe5/0x110
[ 9639.596382]  ? kthread_complete_and_exit+0x20/0x20
[ 9639.596383]  ret_from_fork+0x1f/0x30
[ 9639.596387]  </TASK>

...

[ 9639.595665] NMI backtrace for cpu 50
[ 9639.595667] CPU: 50 PID: 0 Comm: swapper/50 Kdump: loaded Tainted: G 
          E      6.2.0-rc6-swiotlb-upstream-bug-repreduce+ #70
[ 9639.595669] RIP: 0010:native_queued_spin_lock_slowpath+0x2e/0x2c0
[ 9639.595672] Code: 00 41 55 41 54 55 48 89 fd 53 66 90 ba 01 00 00 00 8b 45 00 85 c0 75 10 f0 0f b1 55 00 85 c0 75 f0 5b 5d 41 5c 41 5d c3 f3 90 <eb> e5 81 fe 00 01 00 00 74 4e 40 30 f6 8
5 f6 75 71 f0 0f ba 6d 00
[ 9639.595673] RSP: 0018:ffffc90000c38e28 EFLAGS: 00000002
[ 9639.595674] RAX: 0000000000000001 RBX: ffff88de5e459adc RCX: 0000000000000001
[ 9639.595675] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88de5e459adc
[ 9639.595675] RBP: ffff88de5e459adc R08: ffff88dad2400000 R09: 0000000000001000
[ 9639.595676] R10: 0000005b9c570000 R11: ffffc90000c38ff8 R12: 0000000000000087
[ 9639.595677] R13: ffff88de5e459ad0 R14: ffff88de5e459adc R15: 00000000001882e0
[ 9639.595678] FS:  0000000000000000(0000) GS:ffff88dce5080000(0000) knlGS:0000000000000000
[ 9639.595679] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 9639.595679] CR2: 00007f256aa4d10c CR3: 000000010ade0005 CR4: 0000000000770ee0
[ 9639.595680] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 9639.595681] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[ 9639.595681] PKRU: 55555554
[ 9639.595682] Call Trace:
[ 9639.595682]  <IRQ>
[ 9639.595683]  _raw_spin_lock_irqsave+0x37/0x40
[ 9639.595686]  swiotlb_release_slots.isra.0+0x86/0x180
[ 9639.595688]  dma_direct_unmap_sg+0xcf/0x1a0
[ 9639.595690]  nvme_unmap_data.part.0+0x43/0xc0
[ 9639.595693]  nvme_pci_complete_batch+0x71/0xe0
[ 9639.595695]  nvme_irq+0x7b/0x90
[ 9639.595697]  ? nvme_prep_rq_batch+0xc0/0xc0
[ 9639.595700]  __handle_irq_event_percpu+0x46/0x170
[ 9639.595701]  handle_irq_event+0x3a/0x80
[ 9639.595703]  handle_edge_irq+0xae/0x290
[ 9639.595705]  __common_interrupt+0x62/0x100
[ 9639.595707]  common_interrupt+0xb3/0xd0
[ 9639.595709]  </IRQ>
[ 9639.595709]  <TASK>
[ 9639.595710]  asm_common_interrupt+0x22/0x40
[ 9639.595712] RIP: 0010:__tdx_hypercall+0x34/0x80
$ git diff # which allow me to allocate swiotlb memory bigger than 4GiB.
kernel/dma/swiotlb.c +1 -0
index a34c38bbe28f..a6d7aae3c040 100644
@@ -311,6 +311,7 @@ static void *swiotlb_memblock_alloc(unsigned long nslabs, unsigned int flags,
         * allow to pick a location everywhere for hypervisors with guest
         * memory encryption.
         */
+       flags |= SWIOTLB_ANY;
        if (flags & SWIOTLB_ANY)
                tlb = memblock_alloc(bytes, PAGE_SIZE);
        else

Thank you very much for your time and careful reading. Guorui

Quoted 51 lines

Robin.

Quoted 49 lines

Changes in this patch set
=========================
Instead of keeping "infecting" the legacy swiotlb code with CoCo logic, this patch tries to introduce a new cc-swiotlb for Confidential VMs.

Confidential VMs usually have reasonable modern devices (virtio devices, NVME, etc.), which can access memory above 4GiB, cc-swiotlb could allocate TLB buffers at any position dynamically. Since set_memory_{decrypted,encrypted} is time-consuming and cannot be used in interrupt context, a new kernel thread "kccd" has been added to populate new TLB buffers on-demand, which solved the problem 1.

In addition, the cc-swiotlb manages TLB buffers by different sizes (512B, 2KiB, 4KiB, 16KiB, and 512KiB). The above values come from the following observations (boot with 8core, 32 GiB, 1 nvme disk, and 1 virtio-net):
- Allocations of 512 bytes and below account for 3.5% of the total DMA cache allocations;
- Allocations of 2 KiB and below account for 57.7%;
- Allocations of 4 KiB and below account for 91.3%;
- Allocations of 16 KiB and below account for 96.0%;
- Allocations of 512 KiB and below accounted for 100%;
- At the end of booting, cc-swiotlb uses 288 MiB in total.

For comparison, legacy swiotlb reserves memory at 6%, which requires min(1GiB, 32GiB * 0.06) = 1GiB, and will hang when operating multiple disks simultaneously due to no memory for the swiotlb buffer.

These patches were tested with fio (using different iodepth and block size) on a platform with 96 cores, 384 GiB, and 20 NVME disks, and no IO hang or error was observed.

For simplicity, the current RFC version cannot switch between legacy implementation with cmdline but through compile options. I am open to discussing how to integrate the cc-swiotlb into the legacy one.

Patch Organization
==================
- swiotlb: Split common code from swiotlb.{c,h}
- swiotlb: Add a new cc-swiotlb implementation for Confidential VMs
- swiotlb: Add tracepoint swiotlb_unbounced
- cc-swiotlb: Allow set swiotlb watermark from cmdline

Thanks for your time!

Have a nice day,
Guorui

#10 lore raw
In reply to Guorui Yu
Quoted 1 line

No, this cannot guarantee we always have sufficient TLB caches, so we can also have a "No memory for cc-swiotlb buffer" warning.

It's not just a warning, it will be IO errors, right?

Quoted 2 lines

But I want to emphasize that in this case, the current implementation is no worse than the legacy implementation. Moreover, dynamic TLB allocation is more suitable for situations where more disks/network devices will be hotplugged, in which case you cannot pre-set a reasonable value.

That's a reasonable stand point, but have to emphasize that is "probabilistic" in all the descriptions and comments.

I assume you did some stress testing (E.g. all cores submitting at full bandwidth) to validate that it works for you?

-Andi

author
#11 lore raw
Quoted 5 lines · Andi Kleen
Quoted 1 line

No, this cannot guarantee we always have sufficient TLB caches, so we

can also have a "No memory for cc-swiotlb buffer" warning.

It's not just a warning, it will be IO errors, right?

Yes, they are IO errors, but unsustainable such IO errors are not fatal in my limited testing so far, and the system can survive after through them. Again, legacy swiotlb occasionally suffers from TLB starvation.

However, if dynamic allocation of TLB is not allowed at all, the system will be more likely to be overwhelmed by a large of bursting IOs and unable to respond. Such problems are generally transient, so it is difficult to reproduce and debug in a production environment. Users can only set an unreasonably large fixed size and REBOOT to mitigate this problem as much as possible.

Quoted 5 lines
Quoted 2 lines

But I want to emphasize that in this case, the current implementation is no worse than the legacy implementation. Moreover, dynamic TLB allocation is more suitable for situations where more disks/network devices will be hotplugged, in which case you cannot pre-set a reasonable value.

That's a reasonable stand point, but have to emphasize that is "probabilistic" in all the descriptions and comments.

Agreed, but one point to add is that the user can adjust the water level setting to reduce the possibility of interrupt context allocation TLB failure.

According to the current design, the kthread will be awaken to allocate new TLBs when it is lower than half of the water level, so more flexible room can be left by increasing the water level.

Quoted 4 lines

I assume you did some stress testing (E.g. all cores submitting at full bandwidth) to validate that it works for you?

-Andi

Yes, I tested by fio with different block sizes, iodepths and job numbers on my testbed.

And I have noticed that there are some "IO errors" of No memory for cc-swiotlb buffer in the beginning of the test, but it will be eventually disappeared as long as there are enough free memory.

Thanks for your time,
Guorui