Skip to contentSkip to conversation

ARM64: Setup DMA32 zone size by bootargs

PATCH In review Phil Chang 5 participants 10 messages Sep 15, 2020 – Sep 25, 2020

Subsystems (3)
lore
Share & tools
lore

Phil Chang

author
#0 lore raw

PATCH 1/1 ARM64: Setup DMA32 zone size by bootargs

Allowing the DMA32 zone be configurable in ARM64 but at most 4Gb.

---

 .../admin-guide/kernel-parameters.txt         |  3 ++
 arch/arm64/include/asm/memory.h               |  2 +
 arch/arm64/mm/init.c                          | 39 +++++++++++++++++--
 3 files changed, 41 insertions(+), 3 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +3 -0
index bdc1f33fd3d1..5be6259e9ba8 100644
@@ -980,6 +980,9 @@
 			The filter can be disabled or changed to another
 			driver later using sysfs.
 
+	dma32_size=nn[MG]  [KNL,BOOT,ARM64]
+			Forces the DMA32 zone size of <nn> in MB.
+
 	driver_async_probe=  [KNL]
 			List of driver names to be probed asynchronously.
 			Format: <driver_name1>,<driver_name2>...
arch/arm64/include/asm/memory.h +2 -0
index afa722504bfd..710de08ae8ae 100644
@@ -175,6 +175,8 @@ extern u64			kimage_vaddr;
 /* the offset between the kernel virtual and physical mappings */
 extern u64			kimage_voffset;
 
+extern phys_addr_t		dma32_zone_size;
+
 static inline unsigned long kaslr_offset(void)
 {
 	return kimage_vaddr - KIMAGE_VADDR;
arch/arm64/mm/init.c +36 -3
index 481d22c32a2e..c8af53680d46 100644
@@ -60,6 +60,9 @@ EXPORT_SYMBOL(physvirt_offset);
 struct page *vmemmap __ro_after_init;
 EXPORT_SYMBOL(vmemmap);
 
+phys_addr_t dma32_zone_size __ro_after_init;
+EXPORT_SYMBOL(dma32_zone_size);
+
 /*
  * We create both ZONE_DMA and ZONE_DMA32. ZONE_DMA covers the first 1G of
  * memory as some devices, namely the Raspberry Pi 4, have peripherals with
@@ -242,6 +245,29 @@ static int __init early_mem(char *p)
 }
 early_param("mem", early_mem);
 
+static int __init setup_dma32_zone(char *p)
+{
+	u64 size;
+
+	if (!p)
+		return -EINVAL;
+
+	if (kstrtoull(p, 0, &size))
+		return -EINVAL;
+
+	/* DMA32 zone size should never grater than 4G */
+	if (size > max_zone_phys(32) / SZ_1M)
+		return -EINVAL;
+
+	pr_notice("Setup dma32 zone size to %llu Mb\n", size);
+
+	dma32_zone_size = size * SZ_1M;
+
+	return 0;
+}
+
+early_param("dma32_size", setup_dma32_zone);
+
 static int __init early_init_dt_scan_usablemem(unsigned long node,
 		const char *uname, int depth, void *data)
 {
@@ -392,10 +418,17 @@ void __init arm64_memblock_init(void)
 		arm64_dma_phys_limit = max_zone_phys(ARM64_ZONE_DMA_BITS);
 	}
 
-	if (IS_ENABLED(CONFIG_ZONE_DMA32))
-		arm64_dma32_phys_limit = max_zone_phys(32);
-	else
+	if (IS_ENABLED(CONFIG_ZONE_DMA32)) {
+		if (dma32_zone_size) {
+			arm64_dma32_phys_limit = min(max_zone_phys(32),
+				dma32_zone_size + memblock_start_of_DRAM());
+		} else {
+			arm64_dma32_phys_limit = max_zone_phys(32);
+			dma32_zone_size = arm64_dma32_phys_limit;
+		}
+	} else {
 		arm64_dma32_phys_limit = PHYS_MASK + 1;
+	}
 
 	reserve_crashkernel();
 
Signature · 1 line
2.18.0
maintainer
#1 lore raw
Quoted 1 line · Phil Chang

Allowing the DMA32 zone be configurable in ARM64 but at most 4Gb.

Why? What is your use-case?

Signature · 6 lines
Catalin

_______________________________________________
Linux-mediatek mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-mediatek
#2 lore raw
In reply to Phil Chang
Quoted 1 line · Phil Chang

Allowing the DMA32 zone be configurable in ARM64 but at most 4Gb.

Please add more details why would you like to limit the DMA32 zone.

Quoted 15 lines

---

 .../admin-guide/kernel-parameters.txt         |  3 ++
 arch/arm64/include/asm/memory.h               |  2 +
 arch/arm64/mm/init.c                          | 39 +++++++++++++++++--
 3 files changed, 41 insertions(+), 3 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +0 -0
index bdc1f33fd3d1..5be6259e9ba8 100644

Documentation/admin-guide/kernel-parameters.txt line 984

Quoted 5 lines
Documentation/admin-guide/kernel-parameters.txt +2 -0
@@ -980,6 +980,9 @@
 			The filter can be disabled or changed to another
 			driver later using sysfs.
 
+	dma32_size=nn[MG]  [KNL,BOOT,ARM64]
+			Forces the DMA32 zone size of <nn> in MB.

Most of the kernel parameters that deal with memory sizes allow either of [KMG] suffixes.

Quoted 34 lines
Documentation/admin-guide/kernel-parameters.txt +1 -0
+
 	driver_async_probe=  [KNL]
 			List of driver names to be probed asynchronously.
 			Format: <driver_name1>,<driver_name2>...
arch/arm64/include/asm/memory.h +2 -0
index afa722504bfd..710de08ae8ae 100644
@@ -175,6 +175,8 @@ extern u64			kimage_vaddr;
 /* the offset between the kernel virtual and physical mappings */
 extern u64			kimage_voffset;
 
+extern phys_addr_t		dma32_zone_size;
+
 static inline unsigned long kaslr_offset(void)
 {
 	return kimage_vaddr - KIMAGE_VADDR;
arch/arm64/mm/init.c +3 -0
index 481d22c32a2e..c8af53680d46 100644
@@ -60,6 +60,9 @@ EXPORT_SYMBOL(physvirt_offset);
 struct page *vmemmap __ro_after_init;
 EXPORT_SYMBOL(vmemmap);
 
+phys_addr_t dma32_zone_size __ro_after_init;
+EXPORT_SYMBOL(dma32_zone_size);
+
 /*
  * We create both ZONE_DMA and ZONE_DMA32. ZONE_DMA covers the first 1G of
  * memory as some devices, namely the Raspberry Pi 4, have peripherals with
@@ -242,6 +245,29 @@ static int __init early_mem(char *p)
 }
 early_param("mem", early_mem);

arch/arm64/mm/init.c static int __init early_mem(char *p) · line 256

Quoted 10 lines
arch/arm64/mm/init.c +9 -0
 
+static int __init setup_dma32_zone(char *p)
+{
+	u64 size;
+
+	if (!p)
+		return -EINVAL;
+
+	if (kstrtoull(p, 0, &size))
+		return -EINVAL;

Better to use memparse() here.

Quoted 22 lines
arch/arm64/mm/init.c +14 -1
+
+	/* DMA32 zone size should never grater than 4G */
+	if (size > max_zone_phys(32) / SZ_1M)
+		return -EINVAL;
+
+	pr_notice("Setup dma32 zone size to %llu Mb\n", size);
+
+	dma32_zone_size = size * SZ_1M;
+
+	return 0;
+}
+
+early_param("dma32_size", setup_dma32_zone);
+
 static int __init early_init_dt_scan_usablemem(unsigned long node,
 		const char *uname, int depth, void *data)
 {
@@ -392,10 +418,17 @@ void __init arm64_memblock_init(void)
 		arm64_dma_phys_limit = max_zone_phys(ARM64_ZONE_DMA_BITS);
 	}
 
-	if (IS_ENABLED(CONFIG_ZONE_DMA32))

arch/arm64/mm/init.c void __init arm64_memblock_init(void) · line 428

Quoted 10 lines
arch/arm64/mm/init.c +8 -2
-		arm64_dma32_phys_limit = max_zone_phys(32);
-	else
+	if (IS_ENABLED(CONFIG_ZONE_DMA32)) {
+		if (dma32_zone_size) {
+			arm64_dma32_phys_limit = min(max_zone_phys(32),
+				dma32_zone_size + memblock_start_of_DRAM());
+		} else {
+			arm64_dma32_phys_limit = max_zone_phys(32);
+			dma32_zone_size = arm64_dma32_phys_limit;
+		}

I think this calculations can be hidden in max_zone_phys(), e.g.

arch/arm64/mm/init.c +6 -1
index 481d22c32a2e..be3fdfb35a56 100644
@@ -189,7 +189,12 @@ static void __init reserve_elfcorehdr(void)
 static phys_addr_t __init max_zone_phys(unsigned int zone_bits)
 {
 	phys_addr_t offset = memblock_start_of_DRAM() & GENMASK_ULL(63, zone_bits);
-	return min(offset + (1ULL << zone_bits), memblock_end_of_DRAM());
+	phys_addr_t zone_size = (1ULL << zone_bits);
+
+	if (IS_ENABLED(CONFIG_ZONE_DMA32) && zone_bits == 32 && dma32_zone_size)
+		zone_bits = min(zone_size,dma32_zone_size);
+
+	return min(offset + zone_size, memblock_end_of_DRAM());
 }
 
 static void __init zone_sizes_init(unsigned long min, unsigned long max)
Quoted 8 lines
arch/arm64/mm/init.c +2 -0
+	} else {
 		arm64_dma32_phys_limit = PHYS_MASK + 1;
+	}
 
 	reserve_crashkernel();
 
Signature · 1 line
2.18.0
Signature · 7 lines
Sincerely yours,
Mike.

_______________________________________________
Linux-mediatek mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-mediatek
#3 lore raw
In reply to Phil Chang
Quoted 1 line · Phil Chang

Allowing the DMA32 zone be configurable in ARM64 but at most 4Gb.

I don't think 4,000,000 bits is a particularly sensible limit.

Perhaps you meant bytes, and perhaps you meant 4*1024*1024? That would be "4 GiB".

Signature · 6 lines
Sent from my Android device with K-9 Mail. Please excuse my brevity.

_______________________________________________
Linux-mediatek mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-mediatek
author
#4 lore raw

PATCH 1/1 ARM64: Setup DMA32 zone size by bootargs

In reply to Phil Chang

this patch allowing the DMA32 zone be configurable in ARM64.

---
For some devices, the main memory split into 2 part due to the memory architecture, the efficient and less inefficient part.
One of the use case is fine-tune the dma32 size to contain all the efficient part of memory block on this kind of architecture

 .../admin-guide/kernel-parameters.txt         |  3 +++
 arch/arm64/include/asm/memory.h               |  2 ++
 arch/arm64/mm/init.c                          | 26 ++++++++++++++++++-
 3 files changed, 30 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +3 -0
index bdc1f33fd3d1..ef3bc3d4931a 100644
@@ -980,6 +980,9 @@
 			The filter can be disabled or changed to another
 			driver later using sysfs.
 
+	dma32_size=nn[KMG]  [KNL,BOOT,ARM64]
+			Forces the DMA32 zone size of <nn> in MB.
+
 	driver_async_probe=  [KNL]
 			List of driver names to be probed asynchronously.
 			Format: <driver_name1>,<driver_name2>...
arch/arm64/include/asm/memory.h +2 -0
index afa722504bfd..710de08ae8ae 100644
@@ -175,6 +175,8 @@ extern u64			kimage_vaddr;
 /* the offset between the kernel virtual and physical mappings */
 extern u64			kimage_voffset;
 
+extern phys_addr_t		dma32_zone_size;
+
 static inline unsigned long kaslr_offset(void)
 {
 	return kimage_vaddr - KIMAGE_VADDR;
arch/arm64/mm/init.c +25 -1
index 481d22c32a2e..fd1b85e131cc 100644
@@ -60,6 +60,9 @@ EXPORT_SYMBOL(physvirt_offset);
 struct page *vmemmap __ro_after_init;
 EXPORT_SYMBOL(vmemmap);
 
+phys_addr_t dma32_zone_size __ro_after_init;
+EXPORT_SYMBOL(dma32_zone_size);
+
 /*
  * We create both ZONE_DMA and ZONE_DMA32. ZONE_DMA covers the first 1G of
  * memory as some devices, namely the Raspberry Pi 4, have peripherals with
@@ -189,7 +192,12 @@ static void __init reserve_elfcorehdr(void)
 static phys_addr_t __init max_zone_phys(unsigned int zone_bits)
 {
 	phys_addr_t offset = memblock_start_of_DRAM() & GENMASK_ULL(63, zone_bits);
-	return min(offset + (1ULL << zone_bits), memblock_end_of_DRAM());
+	phys_addr_t zone_size = (1ULL << zone_bits);
+
+	if (IS_ENABLED(CONFIG_ZONE_DMA32) && zone_bits == 32 && dma32_zone_size)
+		zone_size = min(zone_size, dma32_zone_size);
+
+	return min(offset + zone_size, memblock_end_of_DRAM());
 }
 
 static void __init zone_sizes_init(unsigned long min, unsigned long max)
@@ -242,6 +250,22 @@ static int __init early_mem(char *p)
 }
 early_param("mem", early_mem);
 
+static int __init setup_dma32_zone(char *p)
+{
+	unsigned long long size = memparse(p, NULL);
+
+	/* DMA32 zone size should never grater than 4G */
+	if (size > SZ_4G)
+		return -EINVAL;
+
+	pr_notice("Setup dma32 zone size to %llu Mb\n", size >> 20);
+	dma32_zone_size = size;
+
+	return 0;
+}
+
+early_param("dma32_size", setup_dma32_zone);
+
 static int __init early_init_dt_scan_usablemem(unsigned long node,
 		const char *uname, int depth, void *data)
 {
Signature · 1 line
2.18.0
#5 lore raw
Quoted 1 line · Phil Chang

this patch allowing the DMA32 zone be configurable in ARM64.

Hell no. The point of the DMA32 zone is that it is exactly the first 4GiG, and not some random size someone decided without explaining why.

author
#6 lore raw

PATCH 1/1 ARM64: Setup DMA32 zone size by bootargs

this patch allowing the DMA32 zone be configurable in ARM64. For some devices, the main memory split into 2 part due to the memory architecture, the efficient and less inefficient part.
One of the use case is fine-tune the dma32 size to contain all the efficient part of memory block on this kind of architecture

---
Hi

supplement the reason of this usage.

Thanks.

 .../admin-guide/kernel-parameters.txt         |  3 +++
 arch/arm64/include/asm/memory.h               |  2 ++
 arch/arm64/mm/init.c                          | 26 ++++++++++++++++++-
 3 files changed, 30 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +3 -0
index bdc1f33fd3d1..ef3bc3d4931a 100644
@@ -980,6 +980,9 @@
 			The filter can be disabled or changed to another
 			driver later using sysfs.
 
+	dma32_size=nn[KMG]  [KNL,BOOT,ARM64]
+			Forces the DMA32 zone size of <nn> in MB.
+
 	driver_async_probe=  [KNL]
 			List of driver names to be probed asynchronously.
 			Format: <driver_name1>,<driver_name2>...
arch/arm64/include/asm/memory.h +2 -0
index afa722504bfd..710de08ae8ae 100644
@@ -175,6 +175,8 @@ extern u64			kimage_vaddr;
 /* the offset between the kernel virtual and physical mappings */
 extern u64			kimage_voffset;
 
+extern phys_addr_t		dma32_zone_size;
+
 static inline unsigned long kaslr_offset(void)
 {
 	return kimage_vaddr - KIMAGE_VADDR;
arch/arm64/mm/init.c +25 -1
index 481d22c32a2e..fd1b85e131cc 100644
@@ -60,6 +60,9 @@ EXPORT_SYMBOL(physvirt_offset);
 struct page *vmemmap __ro_after_init;
 EXPORT_SYMBOL(vmemmap);
 
+phys_addr_t dma32_zone_size __ro_after_init;
+EXPORT_SYMBOL(dma32_zone_size);
+
 /*
  * We create both ZONE_DMA and ZONE_DMA32. ZONE_DMA covers the first 1G of
  * memory as some devices, namely the Raspberry Pi 4, have peripherals with
@@ -189,7 +192,12 @@ static void __init reserve_elfcorehdr(void)
 static phys_addr_t __init max_zone_phys(unsigned int zone_bits)
 {
 	phys_addr_t offset = memblock_start_of_DRAM() & GENMASK_ULL(63, zone_bits);
-	return min(offset + (1ULL << zone_bits), memblock_end_of_DRAM());
+	phys_addr_t zone_size = (1ULL << zone_bits);
+
+	if (IS_ENABLED(CONFIG_ZONE_DMA32) && zone_bits == 32 && dma32_zone_size)
+		zone_size = min(zone_size, dma32_zone_size);
+
+	return min(offset + zone_size, memblock_end_of_DRAM());
 }
 
 static void __init zone_sizes_init(unsigned long min, unsigned long max)
@@ -242,6 +250,22 @@ static int __init early_mem(char *p)
 }
 early_param("mem", early_mem);
 
+static int __init setup_dma32_zone(char *p)
+{
+	unsigned long long size = memparse(p, NULL);
+
+	/* DMA32 zone size should never grater than 4G */
+	if (size > SZ_4G)
+		return -EINVAL;
+
+	pr_notice("Setup dma32 zone size to %llu Mb\n", size >> 20);
+	dma32_zone_size = size;
+
+	return 0;
+}
+
+early_param("dma32_size", setup_dma32_zone);
+
 static int __init early_init_dt_scan_usablemem(unsigned long node,
 		const char *uname, int depth, void *data)
 {
Signature · 5 lines
2.18.0
_______________________________________________
Linux-mediatek mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-mediatek
maintainer
#7 lore raw
Quoted 6 lines · Phil Chang

this patch allowing the DMA32 zone be configurable in ARM64. For some devices, the main memory split into 2 part due to the memory architecture, the efficient and less inefficient part.
One of the use case is fine-tune the dma32 size to contain all the efficient part of memory block on this kind of architecture

Quoted 7 lines

---
Hi

supplement the reason of this usage.

That's really not a good enough justification to merge such patch. As Christoph said, DMA32 is defined as addressing the first 4GB of RAM.

Is the memory inefficient (presumably slow) for device or for the CPU? Maybe you can pretend it's a separate NUMA node for the CPU. Alternatively, change your device DMA coherent mask and allocate only from ZONE_DMA (currently first 1GB on arm64).

Signature · 6 lines
Catalin

_______________________________________________
Linux-mediatek mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-mediatek
author
#8 lore raw
In reply to Phil Chang

Actually, In a embedded system with 3GB memory, the memory bus width is not the same among the 3GB. (The first 2GB is 48-bit wide, and the latter 1GB is 16-bit wide.)
For memory throughput reason of hardware IPs, we need allocate memory from the first 2GB for the hardware IPs. And that is why we setup the first 2GB as DMA_ZONE, and use GFP_DMA to allocate memory from the range.

maintainer
#9 lore raw
Quoted 2 lines · Phil Chang

Actually, In a embedded system with 3GB memory, the memory bus width is not the same among the 3GB. (The first 2GB is 48-bit wide, and the latter 1GB is 16-bit wide.)

So I guess that's the data bus width. Devices can still access the whole memory, though at different throughputs.

Does this narrow data bus apply only to devices or the CPUs are affected as well?

Quoted 3 lines

For memory throughput reason of hardware IPs, we need allocate memory from the first 2GB for the hardware IPs. And that is why we setup the first 2GB as DMA_ZONE, and use GFP_DMA to allocate memory from the range.

If it's only a throughput problem, it looks to me more like a NUMA configuration. I think you can add the first 2GB and the last GB in separate nodes and define a "numa-node-id" property per device in DT.

Signature · 1 line
Catalin