dma: support DMA zone starting above 4GB
v4 cover dma: support DMA zone starting above 4GB
DMA zones code assumes that DMA lower limit is zero. When there is no RAM below 4GB, arm64 platform code sets DMA/DMA32 zone limits to cover the entire RAM[0].
My target platform has RAM starting at 32GB. Devices with 30-bit DMA mask are mapped to 1GB at the bottom of RAM, between 32GB - 33GB. DMA zone over the entire RAM breaks DMA allocation for these devices.
In response to a previous RFC hack[1] Catalin Marinas suggested to add a separate offset value as base address for the DMA zone, and then refined the suggestion to use start of RAM[3]. This series attempts to implement that suggestion.
With this series applied, the DMA zone covers the right RAM range for my platform.
v4:
* Drop last patch. zone_dma_limit includes RAM base address.
* Adjust DMA zone selection in swiotlb as well.
* Don't change max_zone_phys() behaviour
* Update code to fallback to DMA zone when zone_dma_limit > DMA_BIT_MASK(32)
v3:
* Rebase on v6.11-rc1.
* Drop zone_dma_base. Use memblock_start_of_DRAM() instead.
* Drop DT patches. Low DMA range limit no longer needed.
* Add patch to improve dma_direct_optimal_gfp_mask() heuristics as Catalin suggested.
RFC v2:
* Add patch from Catalin[2] changing zone_dma_bits to zone_dma_limit to simplify subsequent patches
* Test on real hardware
RFC v1: https://lore.kernel.org/all/[email protected]/
[0] See commit 791ab8b2e3db ("arm64: Ignore any DMA offsets in the max_zone_phys() calculation")
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
Baruch Siach (1):
dma: improve DMA zone selection
Catalin Marinas (1):
dma: replace zone_dma_bits by zone_dma_limit
arch/arm64/mm/init.c | 30 +++++++++++++++--------------- arch/powerpc/mm/mem.c | 9 ++++----- arch/s390/mm/init.c | 2 +- include/linux/dma-direct.h | 2 +- kernel/dma/direct.c | 10 +++++----- kernel/dma/pool.c | 5 +++-- kernel/dma/swiotlb.c | 6 +++--- 7 files changed, 32 insertions(+), 32 deletions(-)
base-commit: 8400291e289ee6b2bf9779ff1c83a291501f017b
Signature · 1 line
2.43.0
PATCH v4 1/2 dma: improve DMA zone selection
When device DMA limit does not fit in DMA32 zone it should use DMA zone, even when DMA zone is stricter than needed.
Same goes for devices that can't allocate from the entire normal zone. Limit to DMA32 in that case.
--- kernel/dma/direct.c | 6 +++--- kernel/dma/swiotlb.c | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-)
kernel/dma/direct.c +3 -3
@@ -4,7 +4,7 @@* * DMA operations that map physical memory directly without using an IOMMU. */ -#include <linux/memblock.h> /* for max_pfn */ +#include <linux/memblock.h> #include <linux/export.h> #include <linux/mm.h> #include <linux/dma-map-ops.h>
@@ -59,9 +59,9 @@ static gfp_t dma_direct_optimal_gfp_mask(struct device *dev, u64 *phys_limit)* zones. */ *phys_limit =dma_to_phys (dev, dma_limit); -if (*phys_limit <=DMA_BIT_MASK (zone_dma_bits)) +if (*phys_limit <DMA_BIT_MASK (32 ))return GFP_DMA ; -if (*phys_limit <=DMA_BIT_MASK (32 )) +if (*phys_limit <memblock_end_of_DRAM ())return GFP_DMA32 ;return 0 ; }
kernel/dma/swiotlb.c +2 -2
@@ -629,9 +629,9 @@ static struct page *swiotlb_alloc_tlb(struct device *dev, size_t bytes, } gfp &= ~GFP_ZONEMASK ; -if (phys_limit <=DMA_BIT_MASK (zone_dma_bits)) +if (phys_limit <DMA_BIT_MASK (32 )) gfp |= __GFP_DMA; -else if (phys_limit <=DMA_BIT_MASK (32 )) +else if (phys_limit <memblock_end_of_DRAM ()) gfp |= __GFP_DMA32;while (IS_ERR (page =alloc_dma_pages (gfp, bytes, phys_limit))) {
Signature · 1 line
2.43.0
Re: [PATCH v4 1/2] dma: improve DMA zone selection
Quoted 8 lines · Baruch Siach
When device DMA limit does not fit in DMA32 zone it should use DMA zone, even when DMA zone is stricter than needed.
Same goes for devices that can't allocate from the entire normal zone. Limit to DMA32 in that case.
Reported-by Catalin Marinas <[email protected]>Signed-off-by Baruch Siach <[email protected]>
This looks fine to me now.
PATCH v4 2/2 dma: replace zone_dma_bits by zone_dma_limit
In reply to Baruch SiachFrom: Catalin Marinas <[email protected]>
Hardware DMA limit might not be power of 2. When RAM range starts above 0, say 4GB, DMA limit of 30 bits should end at 5GB. A single high bit can not encode this limit.
Use direct phys_addr_t limit address for DMA zone limit.
Current code using zone_dma_bits assume that all address range below limit is suitable for DMA. For some existing platforms this assumption is not correct. DMA range might have non zero lower limit.
Commit 791ab8b2e3db ("arm64: Ignore any DMA offsets in the max_zone_phys() calculation") made arm64 DMA/DMA32 zones span the entire RAM when RAM starts above 32-bits. This breaks hardware with DMA area that start above 32-bits. But the commit log says that "we haven't noticed any such hardware". It turns out that such hardware does exist.
One such platform has RAM starting at 32GB with an internal bus that has the following DMA limits:
#address-cells = <2>; #size-cells = <2>; dma-ranges = <0x00 0xc0000000 0x08 0x00000000 0x00 0x40000000>;
That is, devices under this bus see 1GB of DMA range between 3GB-4GB in their address space. This range is mapped to CPU memory at 32GB-33GB. With current code DMA allocations for devices under this bus are not limited to DMA area, leading to run-time allocation failure.
With the move to zone_dma_limit DMA allocation for constrained devices becomes possible.
The result is DMA zone that properly reflects the hardware constraints as follows:
[ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000800000000-0x000000083fffffff] [ 0.000000] DMA32 empty [ 0.000000] Normal [mem 0x0000000840000000-0x0000000bffffffff]
--- arch/arm64/mm/init.c | 30 +++++++++++++++--------------- arch/powerpc/mm/mem.c | 9 ++++----- arch/s390/mm/init.c | 2 +- include/linux/dma-direct.h | 2 +- kernel/dma/direct.c | 4 ++-- kernel/dma/pool.c | 5 +++-- kernel/dma/swiotlb.c | 2 +- 7 files changed, 27 insertions(+), 27 deletions(-)
arch/arm64/mm/init.c +15 -15
@@ -115,35 +115,35 @@ static void __init arch_reserve_crashkernel(void) }/* -* Return the maximum physical address for a zone accessible by the given bits -* limit. If DRAM starts above 32-bit, expand the zone to the maximum +* Return the maximum physical address for a zone given its limit. +* If DRAM starts above 32-bit, expand the zone to the maximum * available memory, otherwise cap it at 32-bit. */ -static phys_addr_t __init max_zone_phys (unsigned int zone_bits) +static phys_addr_t __init max_zone_phys (phys_addr_t zone_limit) { -phys_addr_t zone_mask =DMA_BIT_MASK (zone_bits);phys_addr_t phys_start =memblock_start_of_DRAM ();if (phys_start >U32_MAX ) - zone_mask =PHYS_ADDR_MAX ; -else if (phys_start > zone_mask) - zone_mask =U32_MAX ; + zone_limit =PHYS_ADDR_MAX ; +else if (phys_start > zone_limit) + zone_limit =U32_MAX ; -return min (zone_mask,memblock_end_of_DRAM () -1 ) +1 ; +return min (zone_limit,memblock_end_of_DRAM () -1 ) +1 ; }static void __init zone_sizes_init (void ) {unsigned long max_zone_pfns[MAX_NR_ZONES ] = {0 }; -unsigned int __maybe_unused acpi_zone_dma_bits; -unsigned int __maybe_unused dt_zone_dma_bits; -phys_addr_t __maybe_unused dma32_phys_limit =max_zone_phys (32 ); +phys_addr_t __maybe_unused acpi_zone_dma_limit; +phys_addr_t __maybe_unused dt_zone_dma_limit; +phys_addr_t __maybe_unused dma32_phys_limit = +max_zone_phys (DMA_BIT_MASK (32 ));#ifdef CONFIG_ZONE_DMA - acpi_zone_dma_bits =fls64 (acpi_iort_dma_get_max_cpu_address ()); - dt_zone_dma_bits =fls64 (of_dma_get_max_cpu_address (NULL )); - zone_dma_bits =min3 (32U , dt_zone_dma_bits, acpi_zone_dma_bits); - arm64_dma_phys_limit =max_zone_phys (zone_dma_bits); + acpi_zone_dma_limit =acpi_iort_dma_get_max_cpu_address (); + dt_zone_dma_limit =of_dma_get_max_cpu_address (NULL ); + zone_dma_limit =min (dt_zone_dma_limit, acpi_zone_dma_limit); + arm64_dma_phys_limit =max_zone_phys (zone_dma_limit); max_zone_pfns[ZONE_DMA ] =PFN_DOWN (arm64_dma_phys_limit);#endif #ifdef CONFIG_ZONE_DMA32
arch/powerpc/mm/mem.c +4 -5
@@ -216,7 +216,7 @@ static int __init mark_nonram_nosave(void)* everything else. GFP_DMA32 page allocations automatically fall back to * ZONE_DMA. * -* By using 31-bit unconditionally, we can exploit zone_dma_bits to inform the +* By using 31-bit unconditionally, we can exploit zone_dma_limit to inform the * generic DMA mapping code. 32-bit only devices (if not handled by an IOMMU * anyway) will take a first dip into ZONE_NORMAL and get otherwise served by * ZONE_DMA.
@@ -252,13 +252,12 @@ void __init paging_init(void)* powerbooks. */ if (IS_ENABLED (CONFIG_PPC32 )) - zone_dma_bits =30 ; + zone_dma_limit =DMA_BIT_MASK (30 );else - zone_dma_bits =31 ; + zone_dma_limit =DMA_BIT_MASK (31 );#ifdef CONFIG_ZONE_DMA - max_zone_pfns[ZONE_DMA ] =min (max_low_pfn, -1UL << (zone_dma_bits -PAGE_SHIFT )); + max_zone_pfns[ZONE_DMA ] =min (max_low_pfn, zone_dma_limit >>PAGE_SHIFT );#endif max_zone_pfns[ZONE_NORMAL ] = max_low_pfn;#ifdef CONFIG_HIGHMEM
arch/s390/mm/init.c +1 -1
@@ -97,7 +97,7 @@ void __init paging_init(void)vmem_map_init ();sparse_init (); - zone_dma_bits =31 ; + zone_dma_limit =DMA_BIT_MASK (31 );memset (max_zone_pfns,0 ,sizeof (max_zone_pfns)); max_zone_pfns[ZONE_DMA ] =virt_to_pfn (MAX_DMA_ADDRESS ); max_zone_pfns[ZONE_NORMAL ] = max_low_pfn;
include/linux/dma-direct.h +1 -1
@@ -12,7 +12,7 @@#include <linux/mem_encrypt.h> #include <linux/swiotlb.h> -extern unsigned int zone_dma_bits; +extern phys_addr_t zone_dma_limit;/* * Record the mapping of CPU physical to DMA addresses for a given region.
kernel/dma/direct.c +2 -2
@@ -20,7 +20,7 @@* it for entirely different regions. In that case the arch code needs to * override the variable below for dma-direct to work properly. */ -unsigned int zone_dma_bits__ro_after_init =24 ; +phys_addr_t zone_dma_limit__ro_after_init =DMA_BIT_MASK (24 );static inline dma_addr_t phys_to_dma_direct (struct device *dev,phys_addr_t phys)
@@ -580,7 +580,7 @@ int dma_direct_supported(struct device *dev, u64 mask)* part of the check. */ if (IS_ENABLED (CONFIG_ZONE_DMA )) - min_mask =min_t (u64, min_mask,DMA_BIT_MASK (zone_dma_bits)); + min_mask =min_t (u64, min_mask, zone_dma_limit);return mask >=phys_to_dma_unencrypted (dev, min_mask); }
kernel/dma/pool.c +3 -2
@@ -70,9 +70,10 @@ static bool cma_in_zone(gfp_t gfp)/* CMA can't cross zone boundaries, see cma_activate_area() */ end =cma_get_base (cma) + size -1 ;if (IS_ENABLED (CONFIG_ZONE_DMA ) && (gfp &GFP_DMA )) -return end <=DMA_BIT_MASK (zone_dma_bits); +return end <= zone_dma_limit; +/* Account for possible zone_dma_limit > DMA_BIT_MASK(32) */ if (IS_ENABLED (CONFIG_ZONE_DMA32 ) && (gfp &GFP_DMA32 )) -return end <=DMA_BIT_MASK (32 ); +return end <=DMA_BIT_MASK (32 ) || end <= zone_dma_limit;return true; }
kernel/dma/swiotlb.c +1 -1
@@ -450,7 +450,7 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask,if (!remap) io_tlb_default_mem.can_grow = true;if (IS_ENABLED (CONFIG_ZONE_DMA ) && (gfp_mask & __GFP_DMA)) - io_tlb_default_mem.phys_limit =DMA_BIT_MASK (zone_dma_bits); + io_tlb_default_mem.phys_limit = zone_dma_limit;else if (IS_ENABLED (CONFIG_ZONE_DMA32 ) && (gfp_mask & __GFP_DMA32)) io_tlb_default_mem.phys_limit =DMA_BIT_MASK (32 );else
Signature · 1 line
2.43.0
Catalin Marinas
maintainerRe: [PATCH v4 2/2] dma: replace zone_dma_bits by zone_dma_limit
Quoted 21 lines · Baruch Siach
arch/arm64/mm/init.c +3
-4@@ -115,35 +115,35 @@ static void __init arch_reserve_crashkernel(void) }/* -* Return the maximum physical address for a zone accessible by the given bits -* limit. If DRAM starts above 32-bit, expand the zone to the maximum +* Return the maximum physical address for a zone given its limit. +* If DRAM starts above 32-bit, expand the zone to the maximum * available memory, otherwise cap it at 32-bit. */ -static phys_addr_t __init max_zone_phys (unsigned int zone_bits) +static phys_addr_t __init max_zone_phys (phys_addr_t zone_limit) { -phys_addr_t zone_mask =DMA_BIT_MASK (zone_bits);phys_addr_t phys_start =memblock_start_of_DRAM ();if (phys_start >U32_MAX )
arch/arm64/mm/init.c static void __init arch_reserve_crashkernel(void) · line 132
Quoted 10 lines
arch/arm64/mm/init.c +4
-4- zone_mask =PHYS_ADDR_MAX ; - else if (phys_start > zone_mask) - zone_mask =U32_MAX ; + zone_limit =PHYS_ADDR_MAX ; + else if (phys_start > zone_limit) + zone_limit =U32_MAX ; -return min (zone_mask,memblock_end_of_DRAM () -1 ) +1 ; +return min (zone_limit,memblock_end_of_DRAM () -1 ) +1 ; }
OK, so no functional change here which is good. But isn't this series missing some additional patches to limit ZONE_DMA? For you platform, the above function expands ZONE_DMA to the whole RAM which IIUC it's not what you want eventually.
Signature · 1 line
Catalin
Catalin Marinas
maintainerRe: [PATCH v4 2/2] dma: replace zone_dma_bits by zone_dma_limit
In reply to Baruch SiachQuoted 22 lines · Baruch Siach
kernel/dma/pool.c +3
-2@@ -70,9 +70,10 @@ static bool cma_in_zone(gfp_t gfp)/* CMA can't cross zone boundaries, see cma_activate_area() */ end =cma_get_base (cma) + size -1 ;if (IS_ENABLED (CONFIG_ZONE_DMA ) && (gfp &GFP_DMA )) -return end <=DMA_BIT_MASK (zone_dma_bits); +return end <= zone_dma_limit; +/* Account for possible zone_dma_limit > DMA_BIT_MASK(32) */ if (IS_ENABLED (CONFIG_ZONE_DMA32 ) && (gfp &GFP_DMA32 )) -return end <=DMA_BIT_MASK (32 ); +return end <=DMA_BIT_MASK (32 ) || end <= zone_dma_limit;return true; }kernel/dma/swiotlb.c +0
-0
kernel/dma/swiotlb.c int swiotlb_init_late(size_t size, gfp_t gfp_mask, · line 455
Quoted 7 lines
kernel/dma/swiotlb.c +1
-1@@ -450,7 +450,7 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask,if (!remap) io_tlb_default_mem.can_grow = true;if (IS_ENABLED (CONFIG_ZONE_DMA ) && (gfp_mask & __GFP_DMA)) - io_tlb_default_mem.phys_limit =DMA_BIT_MASK (zone_dma_bits); + io_tlb_default_mem.phys_limit = zone_dma_limit;else if (IS_ENABLED (CONFIG_ZONE_DMA32 ) && (gfp_mask & __GFP_DMA32)) io_tlb_default_mem.phys_limit =DMA_BIT_MASK (32 );
I think this needs some adjustment as the cma_in_zone() case. Maybe just use max(DMA_BIT_MASK(32), zone_dma_limit) in both cases for consistency.
Signature · 1 line
Catalin