Skip to contentSkip to conversation

memory_block_size reduction in memory hot plug, can we reduced it to 1GB,currently it is 4GB

PATCH Ranjan, Vikash 2 participants 5 messages Jan 6, 2022 – Jan 11, 2022

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

Hello Kernel Memory Team,

We are trying to enable memory hot plug for our device's based on arm64, is it possible to reduce memory_block_size to 1 GB or lesser as currently it is 4GB.

If you have any info on updated patch or any other way we can do this, please let us know.
After applying attached test patch (for reduction of memory_block_size to 1 GB),we are getting below error. Probe is working fine in case of 4GB.

 root@sys:/sys/devices/system/memory# echo 0x980000000  > /sys/devices/system/memory/probe
echo: write error

Thanks,
Vikash

arch/arm64/configs/exynosauto9_linux_vm_defconfig +0 -0
index 2567f5bc24f1..54bf4618c9ca 100644

+CONFIG_SPARSEMEM=y
+CONFIG_MEMORY_HOTPLUG=y
+CONFIG_ARCH_MEMORY_PROBE=y
+CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y

drivers/base/memory.c +5 -5
index 5a8c430fb8ff..3d2988b4bfed 100644
@@ -469,7 +469,7 @@ static DEVICE_ATTR_RW(auto_online_blocks);
  * as well as ppc64 will do all of their discovery in userspace
  * and will require this interface.
  */
-#ifdef CONFIG_ARCH_MEMORY_PROBE
+//#ifdef CONFIG_ARCH_MEMORY_PROBE
 static ssize_t probe_store(struct device *dev, struct device_attribute *attr,
                           const char *buf, size_t count)
 {
@@ -490,7 +490,7 @@ static ssize_t probe_store(struct device *dev, struct device_attribute *attr,
 
        nid = memory_add_physaddr_to_nid(phys_addr);
        ret = __add_memory(nid, phys_addr,
-                          MIN_MEMORY_BLOCK_SIZE * sections_per_block);
+                          (MIN_MEMORY_BLOCK_SIZE/4) * sections_per_block);
 
        if (ret)
                goto out;
@@ -502,7 +502,7 @@ static ssize_t probe_store(struct device *dev, struct device_attribute *attr,
 }
 
 static DEVICE_ATTR_WO(probe);
-#endif
+//#endif
 
 #ifdef CONFIG_MEMORY_FAILURE
 /*
@@ -756,9 +756,9 @@ bool is_memblock_offlined(struct memory_block *mem)
 }
 
 static struct attribute *memory_root_attrs[] = {
-#ifdef CONFIG_ARCH_MEMORY_PROBE
+//#ifdef CONFIG_ARCH_MEMORY_PROBE
        &dev_attr_probe.attr,
-#endif
+//#endif
#1 lore raw

Hi

Quoted 4 lines

We are trying to enable memory hot plug for our device's based on arm64, is it possible to reduce memory_block_size to 1 GB or lesser as currently it is 4GB.

... are you sure it's 4GB? I never heard that magic number on arm64 before. The section size is 512 MiB on arm64 with 64k and 128 MiB with 4k.

Quoted 7 lines

If you have any info on updated patch or any other way we can do this, please let us know.

 After applying attached test patch (for reduction of memory_block_size to 1 GB),we are getting below error. Probe is working fine in case of 4GB.

I think I am missing something important.

#define MIN_MEMORY_BLOCK_SIZE (1UL << SECTION_SIZE_BITS)

arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 29 arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 27

-> 512 MiB / 128 MiB

arm64 does not override memory_block_size_bytes(), so it matches exactly these numbers.

Signature · 3 lines
Thanks,

David / dhildenb
#2 lore raw

RE: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB

Hi David,

You were right , in my source code
arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 30 arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 27

I need your input on below.
1) currently due to code ( if (phys_addr & ((pages_per_block << PAGE_SHIFT) - 1)) return -EINVAL;) as of now we could give start addr in the increasing step of 40000000 such as 940000000,980000000,9C0000000. However we cannot give 95000000,990000000,9d0000000. Can we give start address which is not in step of 40000000

2) You were right MIN_MEMORY_BLOCK_SIZE is 1 GB, can we further reduce memory region length to 128 MB or it will be good if we can make it customizable( even smaller then 128 MB). Like in our system I want to hot-plug memory which is starting from 9e0000000 to 9e3000000, As we can see memory region is dependent on MIN_MEMORY_BLOCK_SIZE in below code.

ret = __add_memory(nid, phys_addr,

                           MIN_MEMORY_BLOCK_SIZE * sections_per_block);

My ask for above is due to non-availability of non -reserved memory in continuous bigger chunk in our system.

Please let me know your input if we can do the same.

Thanks,
Vikash

-----Original Message-----
From: David Hildenbrand <[email protected]>
Sent: Sunday, January 9, 2022 5:56 PM
To: Ranjan, Vikash <[email protected]>; [email protected]
Subject: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB

[EXTERNAL EMAIL]

Hi

Quoted 4 lines

We are trying to enable memory hot plug for our device's based on arm64, is it possible to reduce memory_block_size to 1 GB or lesser as currently it is 4GB.

... are you sure it's 4GB? I never heard that magic number on arm64 before. The section size is 512 MiB on arm64 with 64k and 128 MiB with 4k. Vikash--- yes David you were right.

Quoted 7 lines

If you have any info on updated patch or any other way we can do this, please let us know.

 After applying attached test patch (for reduction of
memory_block_size to 1 GB),we are getting below error. Probe is working fine in case of 4GB.

I think I am missing something important.

#define MIN_MEMORY_BLOCK_SIZE (1UL << SECTION_SIZE_BITS)

arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 29 arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 27

-> 512 MiB / 128 MiB
arm64 does not override memory_block_size_bytes(), so it matches exactly these numbers.

1)
--
Thanks,

David / dhildenb

#3 lore raw

Re: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB

Quoted 2 lines · Vikash

Hi David,

Hi,

Quoted 2 lines

You were right , in my source code
arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 30 arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 27

Ever since f0b13ee23241 ("arm64/sparsemem: reduce SECTION_SIZE_BITS"), it should be 512 MiB / 128 MiB. That one should be easy to backport also to older kernels IIRC.

Quoted 3 lines

I need your input on below.
1) currently due to code ( if (phys_addr & ((pages_per_block << PAGE_SHIFT) - 1)) return -EINVAL;) as of now we could give start addr in the increasing step of 40000000 such as 940000000,980000000,9C0000000. However we cannot give 95000000,990000000,9d0000000. Can we give start address which is not in step of 40000000

Right, in your kernel it should be 1 GiB-aligned chunks. Upstream it should be 512 MiB / 128 MiB-aligned chunks.

Memory blocks always have to be aligned, starting in the middle of one is impossible. So with a section size of 512 MiB, best you can do is probing 512 MiB aligned and sized memory blocks. Similarly, with 128 MiB.

Quoted 2 lines

2) You were right MIN_MEMORY_BLOCK_SIZE is 1 GB, can we further reduce memory region length to 128 MB or it will be good if we can make it customizable( even smaller then 128 MB). Like in our system I want to hot-plug memory which is starting from 9e0000000 to 9e3000000, As we can see memory region is dependent on MIN_MEMORY_BLOCK_SIZE in below code.

Upstream, with 4k pages, 128 MiB should work. Anything smaller than that is impossible to hotplug/probe.

There are certain restrictions that impose this limit: for example, the pageblock size of 512 MiB limits on 64k limits the section size on 64k to 512 MiB as well.

With 4k, we're using 128 MiB, because it's the smallest possible value that still allows for having huge pages to store the vmemmap easily: the memmap of 128 MiB is exactly 2 MiB, corresponding to one huge page on arm64.

We usually don't want smaller memory blocks, because it makes /sys/devices/system/memory/ explode in size, can degrade performance (e.g., vmemmap), and might result in other issues (sub-section hot-add for ZONE_DEVICE)

Quoted 6 lines

ret = __add_memory(nid, phys_addr,

                           MIN_MEMORY_BLOCK_SIZE * sections_per_block);

My ask for above is due to non-availability of non -reserved memory in continuous bigger chunk in our system.

If you can use 4k base pages, with f0b13ee23241 ("arm64/sparsemem:
reduce SECTION_SIZE_BITS") you'll get 128 MiB sections and consequently 128 MiB MIN_MEMORY_BLOCK_SIZE. With that, you can probe 9e0000000 to 9e3000000 by probing 4 individual 128 MiB-sized memory blocks.

Signature · 3 lines
Thanks,

David / dhildenb
#4 lore raw

RE: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB

Thank You David for your input, let me backport/implement your advice. Will update you on this.

Thanks,
Vikash

-----Original Message-----
From: David Hildenbrand <[email protected]>
Sent: Monday, January 10, 2022 9:52 PM
To: Ranjan, Vikash <[email protected]>; [email protected]
Subject: Re: [EXTERNAL] Re: memory_block_size reduction in memory hot plug, can we reduced it to 1GB, currently it is 4GB

[EXTERNAL EMAIL]

Quoted 2 lines · Vikash

Hi David,

Hi,

Quoted 3 lines

You were right , in my source code arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 30 arch/arm64/include/asm/sparsemem.h:#define SECTION_SIZE_BITS 27

Ever since f0b13ee23241 ("arm64/sparsemem: reduce SECTION_SIZE_BITS"), it should be 512 MiB / 128 MiB. That one should be easy to backport also to older kernels IIRC.

Quoted 8 lines

I need your input on below.
1) currently due to code ( if (phys_addr & ((pages_per_block << PAGE_SHIFT) - 1)) return -EINVAL;) as of now we could give start addr in the increasing step of 40000000 such as 940000000,980000000,9C0000000. However we cannot give 95000000,990000000,9d0000000. Can we give start address which is not in step of 40000000

Right, in your kernel it should be 1 GiB-aligned chunks. Upstream it should be 512 MiB / 128 MiB-aligned chunks.

Memory blocks always have to be aligned, starting in the middle of one is impossible. So with a section size of 512 MiB, best you can do is probing 512 MiB aligned and sized memory blocks. Similarly, with 128 MiB.

Quoted 2 lines

2) You were right MIN_MEMORY_BLOCK_SIZE is 1 GB, can we further reduce memory region length to 128 MB or it will be good if we can make it customizable( even smaller then 128 MB). Like in our system I want to hot-plug memory which is starting from 9e0000000 to 9e3000000, As we can see memory region is dependent on MIN_MEMORY_BLOCK_SIZE in below code.

Upstream, with 4k pages, 128 MiB should work. Anything smaller than that is impossible to hotplug/probe.

There are certain restrictions that impose this limit: for example, the pageblock size of 512 MiB limits on 64k limits the section size on 64k to 512 MiB as well.

With 4k, we're using 128 MiB, because it's the smallest possible value that still allows for having huge pages to store the vmemmap easily: the memmap of 128 MiB is exactly 2 MiB, corresponding to one huge page on arm64.

We usually don't want smaller memory blocks, because it makes /sys/devices/system/memory/ explode in size, can degrade performance (e.g., vmemmap), and might result in other issues (sub-section hot-add for ZONE_DEVICE)

Quoted 7 lines

ret = __add_memory(nid, phys_addr,
MIN_MEMORY_BLOCK_SIZE *
sections_per_block);

My ask for above is due to non-availability of non -reserved memory in continuous bigger chunk in our system.

If you can use 4k base pages, with f0b13ee23241 ("arm64/sparsemem:
reduce SECTION_SIZE_BITS") you'll get 128 MiB sections and consequently 128 MiB MIN_MEMORY_BLOCK_SIZE. With that, you can probe 9e0000000 to 9e3000000 by probing 4 individual 128 MiB-sized memory blocks.

--
Thanks,

David / dhildenb