memory_block_size reduction in memory hot plug, can we reduced it to 1GB,currently it is 4GB
Subsystems (5)
Hello Kernel Memory Team,
We are trying to enable memory hot plug for our device's based on arm64, is it possible to reduce memory_block_size to 1 GB or lesser as currently it is 4GB.
If you have any info on updated patch or any other way we can do this, please let us know.
After applying attached test patch (for reduction of memory_block_size to 1 GB),we are getting below error. Probe is working fine in case of 4GB.
root@sys:/sys/devices/system/memory# echo 0x980000000 > /sys/devices/system/memory/probe echo: write error
Thanks,
Vikash
arch/arm64/configs/exynosauto9_linux_vm_defconfig +0 -0
+CONFIG_SPARSEMEM=y
+CONFIG_MEMORY_HOTPLUG=y
+CONFIG_ARCH_MEMORY_PROBE=y
+CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
drivers/base/memory.c +5 -5
@@ -469,7 +469,7 @@ static DEVICE_ATTR_RW(auto_online_blocks);* as well as ppc64 will do all of their discovery in userspace * and will require this interface. */ -#ifdef CONFIG_ARCH_MEMORY_PROBE +//#ifdef CONFIG_ARCH_MEMORY_PROBE static ssize_t probe_store (struct device *dev,struct device_attribute *attr,const char *buf,size_t count) {
@@ -490,7 +490,7 @@ static ssize_t probe_store(struct device *dev, struct device_attribute *attr, nid =memory_add_physaddr_to_nid (phys_addr); ret =__add_memory (nid, phys_addr, -MIN_MEMORY_BLOCK_SIZE * sections_per_block); + (MIN_MEMORY_BLOCK_SIZE /4 ) * sections_per_block);if (ret) goto out;
@@ -502,7 +502,7 @@ static ssize_t probe_store(struct device *dev, struct device_attribute *attr, }static DEVICE_ATTR_WO (probe); -#endif +//#endif #ifdef CONFIG_MEMORY_FAILURE /*
@@ -756,9 +756,9 @@ bool is_memblock_offlined(struct memory_block *mem) }static struct attribute *memory_root_attrs[] = { -#ifdef CONFIG_ARCH_MEMORY_PROBE +//#ifdef CONFIG_ARCH_MEMORY_PROBE &dev_attr_probe.attr, -#endif +//#endif
Hi
Quoted 4 lines
We are trying to enable memory hot plug for our device's based on arm64, is it possible to reduce
memory_block_sizeto 1 GB or lesser as currently it is 4GB.
... are you sure it's 4GB? I never heard that magic number on arm64 before. The section size is 512 MiB on arm64 with 64k and 128 MiB with 4k.
Quoted 7 lines
If you have any info on updated patch or any other way we can do this, please let us know.
After applying attached test patch (for reduction of
memory_block_sizeto 1 GB),we are getting below error. Probe is working fine in case of 4GB.
I think I am missing something important.
#define MIN_MEMORY_BLOCK_SIZE (1UL << SECTION_SIZE_BITS)
arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 29 arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 27
-> 512 MiB / 128 MiB
arm64 does not override memory_block_size_bytes(), so it matches exactly these numbers.
Signature · 3 lines
Thanks, David / dhildenb
RE: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB
Hi David,
You were right , in my source code
arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 30 arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 27
I need your input on below.
1) currently due to code ( if (phys_addr & ((pages_per_block << PAGE_SHIFT) - 1)) return -EINVAL;) as of now we could give start addr in the increasing step of 40000000 such as 940000000,980000000,9C0000000. However we cannot give 95000000,990000000,9d0000000. Can we give start address which is not in step of 40000000
2) You were right MIN_MEMORY_BLOCK_SIZE is 1 GB, can we further reduce memory region length to 128 MB or it will be good if we can make it customizable( even smaller then 128 MB). Like in our system I want to hot-plug memory which is starting from 9e0000000 to 9e3000000, As we can see memory region is dependent on MIN_MEMORY_BLOCK_SIZE in below code.
ret = __add_memory(nid, phys_addr,
MIN_MEMORY_BLOCK_SIZE * sections_per_block);
My ask for above is due to non-availability of non -reserved memory in continuous bigger chunk in our system.
Please let me know your input if we can do the same.
Thanks,
Vikash
-----Original Message-----
From: David Hildenbrand <[email protected]>
Sent: Sunday, January 9, 2022 5:56 PM
To: Ranjan, Vikash <[email protected]>; [email protected]
Subject: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB
[EXTERNAL EMAIL]
Hi
Quoted 4 lines
We are trying to enable memory hot plug for our device's based on arm64, is it possible to reduce
memory_block_sizeto 1 GB or lesser as currently it is 4GB.
... are you sure it's 4GB? I never heard that magic number on arm64 before. The section size is 512 MiB on arm64 with 64k and 128 MiB with 4k. Vikash--- yes David you were right.
Quoted 7 lines
If you have any info on updated patch or any other way we can do this, please let us know.
After applying attached test patch (for reduction of
memory_block_sizeto 1 GB),we are getting below error. Probe is working fine in case of 4GB.
I think I am missing something important.
#define MIN_MEMORY_BLOCK_SIZE (1UL << SECTION_SIZE_BITS)
arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 29 arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 27
-> 512 MiB / 128 MiB
arm64 does not override memory_block_size_bytes(), so it matches exactly these numbers.
1)
--
Thanks,
David / dhildenb
Re: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB
Quoted 2 lines · Vikash
Hi David,
Hi,
Quoted 2 lines
You were right , in my source code
arch/arm64/include/asm/sparsemem.h:#defineSECTION_SIZE_BITS30 arch/arm64/include/asm/sparsemem.h:#defineSECTION_SIZE_BITS27
Ever since f0b13ee23241 ("arm64/sparsemem: reduce SECTION_SIZE_BITS"), it should be 512 MiB / 128 MiB. That one should be easy to backport also to older kernels IIRC.
Quoted 3 lines
I need your input on below.
1) currently due to code ( if (phys_addr& ((pages_per_block<<PAGE_SHIFT) - 1)) return-EINVAL;) as of now we could give start addr in the increasing step of 40000000 such as 940000000,980000000,9C0000000. However we cannot give 95000000,990000000,9d0000000. Can we give start address which is not in step of 40000000
Right, in your kernel it should be 1 GiB-aligned chunks. Upstream it should be 512 MiB / 128 MiB-aligned chunks.
Memory blocks always have to be aligned, starting in the middle of one is impossible. So with a section size of 512 MiB, best you can do is probing 512 MiB aligned and sized memory blocks. Similarly, with 128 MiB.
Quoted 2 lines
2) You were right
MIN_MEMORY_BLOCK_SIZEis 1 GB, can we further reduce memory region length to 128 MB or it will be good if we can make it customizable( even smaller then 128 MB). Like in our system I want to hot-plug memory which is starting from9e0000000to9e3000000, As we can see memory region is dependent onMIN_MEMORY_BLOCK_SIZEin below code.
Upstream, with 4k pages, 128 MiB should work. Anything smaller than that is impossible to hotplug/probe.
There are certain restrictions that impose this limit: for example, the pageblock size of 512 MiB limits on 64k limits the section size on 64k to 512 MiB as well.
With 4k, we're using 128 MiB, because it's the smallest possible value that still allows for having huge pages to store the vmemmap easily: the memmap of 128 MiB is exactly 2 MiB, corresponding to one huge page on arm64.
We usually don't want smaller memory blocks, because it makes /sys/devices/system/memory/ explode in size, can degrade performance (e.g., vmemmap), and might result in other issues (sub-section hot-add for ZONE_DEVICE)
Quoted 6 lines
ret = __add_memory(nid,
phys_addr,MIN_MEMORY_BLOCK_SIZE * sections_per_block);My ask for above is due to non-availability of non -reserved memory in continuous bigger chunk in our system.
If you can use 4k base pages, with f0b13ee23241 ("arm64/sparsemem:
reduce SECTION_SIZE_BITS") you'll get 128 MiB sections and consequently 128 MiB MIN_MEMORY_BLOCK_SIZE. With that, you can probe 9e0000000 to 9e3000000 by probing 4 individual 128 MiB-sized memory blocks.
Signature · 3 lines
Thanks, David / dhildenb
RE: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB
Thank You David for your input, let me backport/implement your advice. Will update you on this.
Thanks,
Vikash
-----Original Message-----
From: David Hildenbrand <[email protected]>
Sent: Monday, January 10, 2022 9:52 PM
To: Ranjan, Vikash <[email protected]>; [email protected]
Subject: Re: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB
[EXTERNAL EMAIL]
Quoted 2 lines · Vikash
Hi David,
Hi,
Quoted 3 lines
You were right , in my source code arch/arm64/include/asm/sparsemem.h:#define
SECTION_SIZE_BITS30 arch/arm64/include/asm/sparsemem.h:#defineSECTION_SIZE_BITS27
Ever since f0b13ee23241 ("arm64/sparsemem: reduce SECTION_SIZE_BITS"), it should be 512 MiB / 128 MiB. That one should be easy to backport also to older kernels IIRC.
Quoted 8 lines
I need your input on below.
1) currently due to code ( if (phys_addr& ((pages_per_block<<PAGE_SHIFT) - 1)) return-EINVAL;) as of now we could give start addr in the increasing step of 40000000 such as 940000000,980000000,9C0000000. However we cannot give 95000000,990000000,9d0000000. Can we give start address which is not in step of 40000000
Right, in your kernel it should be 1 GiB-aligned chunks. Upstream it should be 512 MiB / 128 MiB-aligned chunks.
Memory blocks always have to be aligned, starting in the middle of one is impossible. So with a section size of 512 MiB, best you can do is probing 512 MiB aligned and sized memory blocks. Similarly, with 128 MiB.
Quoted 2 lines
2) You were right
MIN_MEMORY_BLOCK_SIZEis 1 GB, can we further reduce memory region length to 128 MB or it will be good if we can make it customizable( even smaller then 128 MB). Like in our system I want to hot-plug memory which is starting from9e0000000to9e3000000, As we can see memory region is dependent onMIN_MEMORY_BLOCK_SIZEin below code.
Upstream, with 4k pages, 128 MiB should work. Anything smaller than that is impossible to hotplug/probe.
There are certain restrictions that impose this limit: for example, the pageblock size of 512 MiB limits on 64k limits the section size on 64k to 512 MiB as well.
With 4k, we're using 128 MiB, because it's the smallest possible value that still allows for having huge pages to store the vmemmap easily: the memmap of 128 MiB is exactly 2 MiB, corresponding to one huge page on arm64.
We usually don't want smaller memory blocks, because it makes /sys/devices/system/memory/ explode in size, can degrade performance (e.g., vmemmap), and might result in other issues (sub-section hot-add for ZONE_DEVICE)
Quoted 7 lines
ret = __add_memory(nid,
phys_addr,
MIN_MEMORY_BLOCK_SIZE*sections_per_block);My ask for above is due to non-availability of non -reserved memory in continuous bigger chunk in our system.
If you can use 4k base pages, with f0b13ee23241 ("arm64/sparsemem:
reduce SECTION_SIZE_BITS") you'll get 128 MiB sections and consequently 128 MiB MIN_MEMORY_BLOCK_SIZE. With that, you can probe 9e0000000 to 9e3000000 by probing 4 individual 128 MiB-sized memory blocks.
--
Thanks,
David / dhildenb