swiotlb: Track and report io_tlb_used high water marks in debugfs
PATCH v4 1/1 swiotlb: Track and report io_tlb_used high water marks in debugfs
swiotlb currently reports the total number of slabs and the instantaneous in-use slabs in debugfs. But with increased usage of swiotlb for all I/O in Confidential Computing (coco) VMs, it has become difficult to know how much memory to allocate for swiotlb bounce buffers, either via the automatic algorithm in the kernel or by specifying a value on the kernel boot line. The current automatic algorithm generously allocates swiotlb bounce buffer memory, and may be wasting significant memory in many use cases.
To support better understanding of swiotlb usage, add tracking of the the high water mark for usage of the default swiotlb bounce buffer memory pool and any reserved memory pools. Report these high water marks in debugfs along with the other swiotlb pool metrics. Allow the high water marks to be reset to zero at runtime by writing to them.
---
This patch is built assuming my previous patch is in place to fix io_tlb_used reporting for reserved memory pools.
Changes in v4:
* Break out high water mark accounting into separate functions with stubs when CONFIG_DEBUG_FS=n. [Christoph Hellwig]
* Do high water mark accounting for each reserved memory pool as well as the default pool. Move the accounting variables into struct io_tlb_mem to support this. [Petr Tesarik]
Changes in v3:
* Do high water mark accounting only when CONFIG_DEBUG_FS=y. As as a result, add back the mem_used() function for the "swiotlb buffer is full" error message. [Christoph -- I didn't hear back whether this approach addresses your concern about one additional atomic operation when slots are allocated and again when freed. I've gone ahead with this new version, and we can obviously have further discussion.]
* Remove unnecessary u64 casts. [Christoph Hellwig]
* Track slot usage and the high water mark only for io_tlb_default_mem. Previous versions incorrectly included per-device pools. [Petr Tesarik]
Changes in v2:
* Only reset the high water mark to zero when the specified new value is zero, to prevent confusion about the ability to reset to some other value [Dexuan Cui]
include/linux/swiotlb.h | 7 ++++++ kernel/dma/swiotlb.c | 66 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+)
include/linux/swiotlb.h +7 -0
@@ -87,6 +87,11 @@ dma_addr_t swiotlb_map(struct device *dev, phys_addr_t phys,* @for_alloc: %true if the pool is used for memory allocation * @nareas: The area number in the pool. * @area_nslabs: The slot number in the area. +* @total_used: The total number of slots in the pool that are currently used +* across all areas. Used only for calculating used_hiwater in +* debugfs. +* @used_hiwater: The high water mark for total_used. Used only for reporting +* in debugfs. */ struct io_tlb_mem {phys_addr_t start;
@@ -102,6 +107,8 @@ struct io_tlb_mem {unsigned int area_nslabs;struct io_tlb_area *areas;struct io_tlb_slot *slots; +atomic_long_t total_used; +atomic_long_t used_hiwater; };extern struct io_tlb_mem io_tlb_default_mem;
kernel/dma/swiotlb.c +66 -0
101 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Christoph Hellwig
AppliedThanks,
applied to the dma-mapping for-next tree.