swiotlb: Allow allocating buffer anywhere in memory
Subsystems (2)
PATCH 1/1 swiotlb: Allow allocating buffer anywhere in memory
POWER secure guests (i.e., guests which use the Protection Execution Facility) need to use SWIOTLB to be able to do I/O with the hypervisor, but they don't need the SWIOTLB memory to be in low addresses since the hypervisor doesn't have any addressing limitation.
This solves a SWIOTLB initialization problem we are seeing in secure guests with 128 GB of RAM: they are configured with 4 GB of crashkernel reserved memory, which leaves no space for SWIOTLB in low addresses.
--- arch/powerpc/mm/mem.c | 7 ++++++- include/linux/swiotlb.h | 8 +++++++- kernel/dma/swiotlb.c | 10 +++++++--- 3 files changed, 20 insertions(+), 5 deletions(-)
Normally I would split changes like this into one patch touching generic code and another for the arch-specific part, but in this case I thought it would be unneeded complexity. I can split though if people prefer it that way.
arch/powerpc/mm/mem.c +6 -1
@@ -50,6 +50,7 @@#include <asm/swiotlb.h> #include <asm/rtas.h> #include <asm/kasan.h> +#include <asm/svm.h> #include <mm/mmu_decl.h>
@@ -290,7 +291,11 @@ void __init mem_init(void)* back to to-down. */ memblock_set_bottom_up (true); -swiotlb_init (0 ); +/* +* SVM guests can use the SWIOTLB wherever it is in memory, +* even if not DMA-able. +*/ +swiotlb_init_anywhere (0 ,is_secure_guest ());#endif high_memory = (void *)__va (max_low_pfn *PAGE_SIZE );
include/linux/swiotlb.h +7 -1
@@ -29,7 +29,13 @@ enum swiotlb_force {*/ #define IO_TLB_SHIFT 11 -extern void swiotlb_init (int verbose); +void __init swiotlb_init_anywhere (int verbose,bool allocate_anywhere); + +static inline void swiotlb_init (int verbose) +{ +swiotlb_init_anywhere (verbose, false); +} +int swiotlb_init_with_tbl (char *tlb,unsigned long nslabs,int verbose);extern unsigned long swiotlb_nr_tbl (void );unsigned long swiotlb_size_or_default (void );
kernel/dma/swiotlb.c +7 -3
@@ -244,7 +244,7 @@ int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose)* structures for the software IO TLB used to implement the DMA API. */ void __init -swiotlb_init (int verbose) +swiotlb_init_anywhere (int verbose,bool allocate_anywhere) {size_t default_size =IO_TLB_DEFAULT_SIZE ;unsigned char *vstart;
@@ -257,8 +257,12 @@ swiotlb_init(int verbose) bytes = io_tlb_nslabs <<IO_TLB_SHIFT ; -/* Get IO TLB memory from the low pages */ - vstart =memblock_alloc_low (PAGE_ALIGN (bytes),PAGE_SIZE ); +if (allocate_anywhere) + vstart =memblock_alloc (PAGE_ALIGN (bytes),PAGE_SIZE ); +else +/* Get IO TLB memory from the low pages */ + vstart =memblock_alloc_low (PAGE_ALIGN (bytes),PAGE_SIZE ); +if (vstart && !swiotlb_init_with_tbl (vstart, io_tlb_nslabs, verbose))return ;
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 8 lines · Thiago Jung Bauermann
POWER secure guests (i.e., guests which use the Protection Execution Facility) need to use SWIOTLB to be able to do I/O with the hypervisor, but they don't need the SWIOTLB memory to be in low addresses since the hypervisor doesn't have any addressing limitation.
This solves a SWIOTLB initialization problem we are seeing in secure guests with 128 GB of RAM: they are configured with 4 GB of crashkernel reserved memory, which leaves no space for SWIOTLB in low addresses.
What about just open coding the allocation and using swiotlb_init_with_tbl?
Quoted 75 lines
Signed-off-by Thiago Jung Bauermann <[email protected]>--- arch/powerpc/mm/mem.c | 7 ++++++- include/linux/swiotlb.h | 8 +++++++- kernel/dma/swiotlb.c | 10 +++++++--- 3 files changed, 20 insertions(+), 5 deletions(-)Normally I would split changes like this into one patch touching generic code and another for the arch-specific part, but in this case I thought it would be unneeded complexity. I can split though if people prefer it that way.
arch/powerpc/mm/mem.c +6
-1@@ -50,6 +50,7 @@#include <asm/swiotlb.h> #include <asm/rtas.h> #include <asm/kasan.h> +#include <asm/svm.h> #include <mm/mmu_decl.h> @@ -290,7 +291,11 @@ void __init mem_init(void)* back to to-down. */ memblock_set_bottom_up (true); -swiotlb_init (0 ); +/* +* SVM guests can use the SWIOTLB wherever it is in memory, +* even if not DMA-able. +*/ +swiotlb_init_anywhere (0 ,is_secure_guest ());#endif high_memory = (void *)__va (max_low_pfn *PAGE_SIZE );include/linux/swiotlb.h +7
-1@@ -29,7 +29,13 @@ enum swiotlb_force {*/ #define IO_TLB_SHIFT 11 -extern void swiotlb_init (int verbose); +void __init swiotlb_init_anywhere (int verbose,bool allocate_anywhere); + +static inline void swiotlb_init (int verbose) +{ +swiotlb_init_anywhere (verbose, false); +} +int swiotlb_init_with_tbl (char *tlb,unsigned long nslabs,int verbose);extern unsigned long swiotlb_nr_tbl (void );unsigned long swiotlb_size_or_default (void );kernel/dma/swiotlb.c +1
-2@@ -244,7 +244,7 @@ int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose)* structures for the software IO TLB used to implement the DMA API. */ void __init -swiotlb_init (int verbose) +swiotlb_init_anywhere (int verbose,bool allocate_anywhere) {size_t default_size =IO_TLB_DEFAULT_SIZE ;unsigned char *vstart;@@ -257,8 +257,12 @@ swiotlb_init(int verbose) bytes = io_tlb_nslabs <<IO_TLB_SHIFT ; -/* Get IO TLB memory from the low pages */
kernel/dma/swiotlb.c swiotlb_init(int verbose) · line 268
Quoted 10 lines
kernel/dma/swiotlb.c +6
-1- vstart =memblock_alloc_low (PAGE_ALIGN (bytes),PAGE_SIZE ); +if (allocate_anywhere) + vstart =memblock_alloc (PAGE_ALIGN (bytes),PAGE_SIZE ); +else +/* Get IO TLB memory from the low pages */ + vstart =memblock_alloc_low (PAGE_ALIGN (bytes),PAGE_SIZE ); +if (vstart && !swiotlb_init_with_tbl (vstart, io_tlb_nslabs, verbose))return ;
---end quoted text---
Hello Christoph,
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Thiago Jung Bauermann
POWER secure guests (i.e., guests which use the Protection Execution Facility) need to use SWIOTLB to be able to do I/O with the hypervisor, but they don't need the SWIOTLB memory to be in low addresses since the hypervisor doesn't have any addressing limitation.
Quoted 6 lines
Quoted 3 lines
This solves a SWIOTLB initialization problem we are seeing in secure guests with 128 GB of RAM: they are configured with 4 GB of crashkernel reserved memory, which leaves no space for SWIOTLB in low addresses.
What about just open coding the allocation and using
swiotlb_init_with_tbl?
Yes, that works too. I just sent a v2 implementing that change. I just had to add a small accessor function so that I could set no_iotlb_memory from outside swiotlb.c.
Thank you for the quick review.
Signature · 6 lines
Thiago Jung Bauermann IBM Linux Technology Center _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu