Skip to contentSkip to conversation

arm64/sparsemem: reduce SECTION_SIZE_BITS

PATCH Applied Sudarshan Rajagopalan 8 participants 12 messages Jan 21, 2021 – Jan 29, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover arm64/sparsemem: reduce SECTION_SIZE_BITS

This patch is the follow-up from the discussions in the thread [1]. Reducing the section size has the merit of reducing wastage of reserved memory for vmmemmap mappings for sections with large memory holes. Also with smaller section size gives more grunularity and agility for memory hot(un)plugging.

But there are also constraints in reducing SECTION_SIZE_BIT:

- Should accommodate highest order page for a given config
- Should not break PMD mapping in vmemmap for 4K pages
- Should not consume too many page->flags bits reducing space for other info

This patch uses the suggestions from Anshuman Khandual and David Hildenbrand in thread [1] to set the least possible section size to 128MB for 4K and 16K base page size configs for simplicity, and to 512MB for 64K base page size config.

[1] https://lore.kernel.org/lkml/[email protected]/T/#m8ee60ae69db5e9eb06ca7999c43828d49ccb9626

Sudarshan Rajagopalan (1):
arm64/sparsemem: reduce SECTION_SIZE_BITS

 arch/arm64/include/asm/sparsemem.h | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)
Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project
#1 lore raw

PATCH 1/1 arm64/sparsemem: reduce SECTION_SIZE_BITS

memory_block_size_bytes() determines the memory hotplug granularity i.e the amount of memory which can be hot added or hot removed from the kernel. The generic value here being MIN_MEMORY_BLOCK_SIZE (1UL << SECTION_SIZE_BITS) for memory_block_size_bytes() on platforms like arm64 that does not override.

Current SECTION_SIZE_BITS is 30 i.e 1GB which is large and a reduction here increases memory hotplug granularity, thus improving its agility. A reduced section size also reduces memory wastage in vmemmmap mapping for sections with large memory holes. So we try to set the least section size as possible.

A section size bits selection must follow:
(MAX_ORDER - 1 + PAGE_SHIFT) <= SECTION_SIZE_BITS

CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64 and so just following it would help achieve the smallest section size.

SECTION_SIZE_BITS = (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)

SECTION_SIZE_BITS = 22 (11 - 1 + 12) i.e 4MB   for 4K pages
SECTION_SIZE_BITS = 24 (11 - 1 + 14) i.e 16MB  for 16K pages without THP
SECTION_SIZE_BITS = 25 (12 - 1 + 14) i.e 32MB  for 16K pages with THP
SECTION_SIZE_BITS = 26 (11 - 1 + 16) i.e 64MB  for 64K pages without THP

SECTION_SIZE_BITS = 29 (14 - 1 + 16) i.e 512MB for 64K pages with THP

But there are other problems in reducing SECTION_SIZE_BIT. Reducing it by too much would over populate /sys/devices/system/memory/ and also consume too many page->flags bits in the !vmemmap case. Also section size needs to be multiple of 128MB to have PMD based vmemmap mapping with CONFIG_ARM64_4K_PAGES.

Given these constraints, lets just reduce the section size to 128MB for 4K and 16K base page size configs, and to 512MB for 64K base page size config.

Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Anshuman Khandual <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Logan Gunthorpe <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Steven Price <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
---
 arch/arm64/include/asm/sparsemem.h | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)
arch/arm64/include/asm/sparsemem.h +21 -2
index 1f43fcc79738..eb4a75d720ed 100644
@@ -7,7 +7,26 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS	30
-#endif
+
+/*
+ * Section size must be at least 512MB for 64K base
+ * page size config. Otherwise it will be less than
+ * (MAX_ORDER - 1) and the build process will fail.
+ */
+#ifdef CONFIG_ARM64_64K_PAGES
+#define SECTION_SIZE_BITS 29
+
+#else
+
+/*
+ * Section size must be at least 128MB for 4K base
+ * page size config. Otherwise PMD based huge page
+ * entries could not be created for vmemmap mappings.
+ * 16K follows 4K for simplicity.
+ */
+#define SECTION_SIZE_BITS 27
+#endif /* CONFIG_ARM64_64K_PAGES */
+
+#endif /* CONFIG_SPARSEMEM*/
 
 #endif
Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project
#2 lore raw
Quoted 4 lines · Sudarshan Rajagopalan

But there are other problems in reducing SECTION_SIZE_BIT. Reducing it by too much would over populate /sys/devices/system/memory/ and also consume too many page->flags bits in the !vmemmap case. Also section size needs to be multiple of 128MB to have PMD based vmemmap mapping with CONFIG_ARM64_4K_PAGES.

There is also the issue of requiring more space in the TLB cache with smaller page sizes. Or does ARM resolve these into smaller TLB entries anyways (going on my x86 kwon how here)? Anyways if there are only a few TLB entries then the effect could
be significant.

maintainer
#4 lore raw
In reply to Sudarshan Rajagopalan
Quoted 45 lines · Sudarshan Rajagopalan

memory_block_size_bytes() determines the memory hotplug granularity i.e the amount of memory which can be hot added or hot removed from the kernel. The generic value here being MIN_MEMORY_BLOCK_SIZE (1UL << SECTION_SIZE_BITS) for memory_block_size_bytes() on platforms like arm64 that does not override.

Current SECTION_SIZE_BITS is 30 i.e 1GB which is large and a reduction here increases memory hotplug granularity, thus improving its agility. A reduced section size also reduces memory wastage in vmemmmap mapping for sections with large memory holes. So we try to set the least section size as possible.

A section size bits selection must follow:
(MAX_ORDER - 1 + PAGE_SHIFT) <= SECTION_SIZE_BITS

CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64 and so just following it would help achieve the smallest section size.

SECTION_SIZE_BITS = (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)

SECTION_SIZE_BITS = 22 (11 - 1 + 12) i.e 4MB   for 4K pages
SECTION_SIZE_BITS = 24 (11 - 1 + 14) i.e 16MB  for 16K pages without THP
SECTION_SIZE_BITS = 25 (12 - 1 + 14) i.e 32MB  for 16K pages with THP
SECTION_SIZE_BITS = 26 (11 - 1 + 16) i.e 64MB  for 64K pages without THP

SECTION_SIZE_BITS = 29 (14 - 1 + 16) i.e 512MB for 64K pages with THP

But there are other problems in reducing SECTION_SIZE_BIT. Reducing it by too much would over populate /sys/devices/system/memory/ and also consume too many page->flags bits in the !vmemmap case. Also section size needs to be multiple of 128MB to have PMD based vmemmap mapping with CONFIG_ARM64_4K_PAGES.

Given these constraints, lets just reduce the section size to 128MB for 4K and 16K base page size configs, and to 512MB for 64K base page size config.

Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Anshuman Khandual <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Logan Gunthorpe <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Steven Price <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Quoted 3 lines
---
 arch/arm64/include/asm/sparsemem.h | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

Anshuman -- are you happy with this now?

Will

Reviewed-by on v1 1/1
#6 lore raw
In reply to Sudarshan Rajagopalan
Quoted 73 lines · Sudarshan Rajagopalan

memory_block_size_bytes() determines the memory hotplug granularity i.e the amount of memory which can be hot added or hot removed from the kernel. The generic value here being MIN_MEMORY_BLOCK_SIZE (1UL << SECTION_SIZE_BITS) for memory_block_size_bytes() on platforms like arm64 that does not override.

Current SECTION_SIZE_BITS is 30 i.e 1GB which is large and a reduction here increases memory hotplug granularity, thus improving its agility. A reduced section size also reduces memory wastage in vmemmmap mapping for sections with large memory holes. So we try to set the least section size as possible.

A section size bits selection must follow:
(MAX_ORDER - 1 + PAGE_SHIFT) <= SECTION_SIZE_BITS

CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64 and so just following it would help achieve the smallest section size.

SECTION_SIZE_BITS = (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)

SECTION_SIZE_BITS = 22 (11 - 1 + 12) i.e 4MB   for 4K pages
SECTION_SIZE_BITS = 24 (11 - 1 + 14) i.e 16MB  for 16K pages without THP
SECTION_SIZE_BITS = 25 (12 - 1 + 14) i.e 32MB  for 16K pages with THP
SECTION_SIZE_BITS = 26 (11 - 1 + 16) i.e 64MB  for 64K pages without THP

SECTION_SIZE_BITS = 29 (14 - 1 + 16) i.e 512MB for 64K pages with THP

But there are other problems in reducing SECTION_SIZE_BIT. Reducing it by too much would over populate /sys/devices/system/memory/ and also consume too many page->flags bits in the !vmemmap case. Also section size needs to be multiple of 128MB to have PMD based vmemmap mapping with CONFIG_ARM64_4K_PAGES.

Given these constraints, lets just reduce the section size to 128MB for 4K and 16K base page size configs, and to 512MB for 64K base page size config.

Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Anshuman Khandual <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Logan Gunthorpe <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Steven Price <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
---
 arch/arm64/include/asm/sparsemem.h | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)
arch/arm64/include/asm/sparsemem.h +13 -2
index 1f43fcc79738..eb4a75d720ed 100644
@@ -7,7 +7,26 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS	30
-#endif
+
+/*
+ * Section size must be at least 512MB for 64K base
+ * page size config. Otherwise it will be less than
+ * (MAX_ORDER - 1) and the build process will fail.
+ */
+#ifdef CONFIG_ARM64_64K_PAGES
+#define SECTION_SIZE_BITS 29
+
+#else
+
+/*
+ * Section size must be at least 128MB for 4K base

arch/arm64/include/asm/sparsemem.h line 32

Quoted 10 lines
arch/arm64/include/asm/sparsemem.h +8 -0
+ * page size config. Otherwise PMD based huge page
+ * entries could not be created for vmemmap mappings.
+ * 16K follows 4K for simplicity.
+ */
+#define SECTION_SIZE_BITS 27
+#endif /* CONFIG_ARM64_64K_PAGES */
+
+#endif /* CONFIG_SPARSEMEM*/
 
 #endif

I'm happy to see this change.

Signature · 3 lines
Thanks,

David / dhildenb
Acked-by on v1 1/1
#7 lore raw
In reply to Sudarshan Rajagopalan
Quoted 32 lines · Sudarshan Rajagopalan

memory_block_size_bytes() determines the memory hotplug granularity i.e the amount of memory which can be hot added or hot removed from the kernel. The generic value here being MIN_MEMORY_BLOCK_SIZE (1UL << SECTION_SIZE_BITS) for memory_block_size_bytes() on platforms like arm64 that does not override.

Current SECTION_SIZE_BITS is 30 i.e 1GB which is large and a reduction here increases memory hotplug granularity, thus improving its agility. A reduced section size also reduces memory wastage in vmemmmap mapping for sections with large memory holes. So we try to set the least section size as possible.

A section size bits selection must follow:
(MAX_ORDER - 1 + PAGE_SHIFT) <= SECTION_SIZE_BITS

CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64 and so just following it would help achieve the smallest section size.

SECTION_SIZE_BITS = (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)

SECTION_SIZE_BITS = 22 (11 - 1 + 12) i.e 4MB   for 4K pages
SECTION_SIZE_BITS = 24 (11 - 1 + 14) i.e 16MB  for 16K pages without THP
SECTION_SIZE_BITS = 25 (12 - 1 + 14) i.e 32MB  for 16K pages with THP
SECTION_SIZE_BITS = 26 (11 - 1 + 16) i.e 64MB  for 64K pages without THP

SECTION_SIZE_BITS = 29 (14 - 1 + 16) i.e 512MB for 64K pages with THP

But there are other problems in reducing SECTION_SIZE_BIT. Reducing it by too much would over populate /sys/devices/system/memory/ and also consume too many page->flags bits in the !vmemmap case. Also section size needs to be multiple of 128MB to have PMD based vmemmap mapping with CONFIG_ARM64_4K_PAGES.

Given these constraints, lets just reduce the section size to 128MB for 4K and 16K base page size configs, and to 512MB for 64K base page size config.

Quoted 13 lines
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Anshuman Khandual <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Logan Gunthorpe <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Steven Price <[email protected]>
Cc Suren Baghdasaryan <[email protected]>

BTW, after reduction of the section size maybe arm64 should consider opting out of freeing unused memory map.

This will make David even more happy as this will allow dropping custom pfn_valid() ;-)

Quoted 41 lines
---
 arch/arm64/include/asm/sparsemem.h | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)
arch/arm64/include/asm/sparsemem.h +21 -2
index 1f43fcc79738..eb4a75d720ed 100644
@@ -7,7 +7,26 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS	30
-#endif
+
+/*
+ * Section size must be at least 512MB for 64K base
+ * page size config. Otherwise it will be less than
+ * (MAX_ORDER - 1) and the build process will fail.
+ */
+#ifdef CONFIG_ARM64_64K_PAGES
+#define SECTION_SIZE_BITS 29
+
+#else
+
+/*
+ * Section size must be at least 128MB for 4K base
+ * page size config. Otherwise PMD based huge page
+ * entries could not be created for vmemmap mappings.
+ * 16K follows 4K for simplicity.
+ */
+#define SECTION_SIZE_BITS 27
+#endif /* CONFIG_ARM64_64K_PAGES */
+
+#endif /* CONFIG_SPARSEMEM*/
 
 #endif
Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project
Signature · 2 lines
Sincerely yours,
Mike.
maintainer Reviewed-by on v1 1/1
#9 lore raw
In reply to Sudarshan Rajagopalan
Quoted 32 lines · Sudarshan Rajagopalan

memory_block_size_bytes() determines the memory hotplug granularity i.e the amount of memory which can be hot added or hot removed from the kernel. The generic value here being MIN_MEMORY_BLOCK_SIZE (1UL << SECTION_SIZE_BITS) for memory_block_size_bytes() on platforms like arm64 that does not override.

Current SECTION_SIZE_BITS is 30 i.e 1GB which is large and a reduction here increases memory hotplug granularity, thus improving its agility. A reduced section size also reduces memory wastage in vmemmmap mapping for sections with large memory holes. So we try to set the least section size as possible.

A section size bits selection must follow:
(MAX_ORDER - 1 + PAGE_SHIFT) <= SECTION_SIZE_BITS

CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64 and so just following it would help achieve the smallest section size.

SECTION_SIZE_BITS = (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)

SECTION_SIZE_BITS = 22 (11 - 1 + 12) i.e 4MB   for 4K pages
SECTION_SIZE_BITS = 24 (11 - 1 + 14) i.e 16MB  for 16K pages without THP
SECTION_SIZE_BITS = 25 (12 - 1 + 14) i.e 32MB  for 16K pages with THP
SECTION_SIZE_BITS = 26 (11 - 1 + 16) i.e 64MB  for 64K pages without THP

SECTION_SIZE_BITS = 29 (14 - 1 + 16) i.e 512MB for 64K pages with THP

But there are other problems in reducing SECTION_SIZE_BIT. Reducing it by too much would over populate /sys/devices/system/memory/ and also consume too many page->flags bits in the !vmemmap case. Also section size needs to be multiple of 128MB to have PMD based vmemmap mapping with CONFIG_ARM64_4K_PAGES.

Given these constraints, lets just reduce the section size to 128MB for 4K and 16K base page size configs, and to 512MB for 64K base page size config.

Quoted 13 lines
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Anshuman Khandual <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Logan Gunthorpe <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Steven Price <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
maintainer
#3 lore raw
In reply to Christoph Lameter
Quoted 7 lines · Christoph Lameter
Quoted 4 lines · Sudarshan Rajagopalan

But there are other problems in reducing SECTION_SIZE_BIT. Reducing it by too much would over populate /sys/devices/system/memory/ and also consume too many page->flags bits in the !vmemmap case. Also section size needs to be multiple of 128MB to have PMD based vmemmap mapping with CONFIG_ARM64_4K_PAGES.

Quoted 5 lines

There is also the issue of requiring more space in the TLB cache with smaller page sizes. Or does ARM resolve these into smaller TLB entries anyways (going on my x86 kwon how here)? Anyways if there are only a few TLB entries then the effect could
be significant.

There is indeed more TLB pressure with smaller page sizes but this patch doesn't change this.

Signature · 1 line
Catalin
#8 lore raw
In reply to Mike Rapoport
Quoted 47 lines · Mike Rapoport
Quoted 32 lines · Sudarshan Rajagopalan

memory_block_size_bytes() determines the memory hotplug granularity i.e the amount of memory which can be hot added or hot removed from the kernel. The generic value here being MIN_MEMORY_BLOCK_SIZE (1UL << SECTION_SIZE_BITS) for memory_block_size_bytes() on platforms like arm64 that does not override.

Current SECTION_SIZE_BITS is 30 i.e 1GB which is large and a reduction here increases memory hotplug granularity, thus improving its agility. A reduced section size also reduces memory wastage in vmemmmap mapping for sections with large memory holes. So we try to set the least section size as possible.

A section size bits selection must follow:
(MAX_ORDER - 1 + PAGE_SHIFT) <= SECTION_SIZE_BITS

CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64 and so just following it would help achieve the smallest section size.

SECTION_SIZE_BITS = (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)

SECTION_SIZE_BITS = 22 (11 - 1 + 12) i.e 4MB   for 4K pages
SECTION_SIZE_BITS = 24 (11 - 1 + 14) i.e 16MB  for 16K pages without THP
SECTION_SIZE_BITS = 25 (12 - 1 + 14) i.e 32MB  for 16K pages with THP
SECTION_SIZE_BITS = 26 (11 - 1 + 16) i.e 64MB  for 64K pages without THP

SECTION_SIZE_BITS = 29 (14 - 1 + 16) i.e 512MB for 64K pages with THP

But there are other problems in reducing SECTION_SIZE_BIT. Reducing it by too much would over populate /sys/devices/system/memory/ and also consume too many page->flags bits in the !vmemmap case. Also section size needs to be multiple of 128MB to have PMD based vmemmap mapping with CONFIG_ARM64_4K_PAGES.

Given these constraints, lets just reduce the section size to 128MB for 4K and 16K base page size configs, and to 512MB for 64K base page size config.

Quoted 13 lines
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Anshuman Khandual <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Logan Gunthorpe <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Steven Price <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Quoted 7 lines

BTW, after reduction of the section size maybe arm64 should consider opting out of freeing unused memory map.

This will make David even more happy as this will allow dropping custom pfn_valid() ;-)

Mike knows my wildest dreams ;)

Signature · 3 lines
Thanks,

David / dhildenb
maintainer Applied
#10 lore raw
In reply to Sudarshan Rajagopalan
Quoted 8 lines · Sudarshan Rajagopalan

This patch is the follow-up from the discussions in the thread [1]. Reducing the section size has the merit of reducing wastage of reserved memory for vmmemmap mappings for sections with large memory holes. Also with smaller section size gives more grunularity and agility for memory hot(un)plugging.

But there are also constraints in reducing SECTION_SIZE_BIT:

[...]

Applied to arm64 (for-next/misc), thanks!

[1/1] arm64/sparsemem: reduce SECTION_SIZE_BITS

      https://git.kernel.org/arm64/c/f0b13ee23241

Cheers,

Signature · 5 lines
Will

https://fixes.arm64.dev
https://next.arm64.dev
https://will.arm64.dev
Reviewed-by
#5 lore raw
In reply to Will Deacon
Quoted 50 lines · Will Deacon
Quoted 45 lines · Sudarshan Rajagopalan

memory_block_size_bytes() determines the memory hotplug granularity i.e the amount of memory which can be hot added or hot removed from the kernel. The generic value here being MIN_MEMORY_BLOCK_SIZE (1UL << SECTION_SIZE_BITS) for memory_block_size_bytes() on platforms like arm64 that does not override.

Current SECTION_SIZE_BITS is 30 i.e 1GB which is large and a reduction here increases memory hotplug granularity, thus improving its agility. A reduced section size also reduces memory wastage in vmemmmap mapping for sections with large memory holes. So we try to set the least section size as possible.

A section size bits selection must follow:
(MAX_ORDER - 1 + PAGE_SHIFT) <= SECTION_SIZE_BITS

CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64 and so just following it would help achieve the smallest section size.

SECTION_SIZE_BITS = (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)

SECTION_SIZE_BITS = 22 (11 - 1 + 12) i.e 4MB   for 4K pages
SECTION_SIZE_BITS = 24 (11 - 1 + 14) i.e 16MB  for 16K pages without THP
SECTION_SIZE_BITS = 25 (12 - 1 + 14) i.e 32MB  for 16K pages with THP
SECTION_SIZE_BITS = 26 (11 - 1 + 16) i.e 64MB  for 64K pages without THP

SECTION_SIZE_BITS = 29 (14 - 1 + 16) i.e 512MB for 64K pages with THP

But there are other problems in reducing SECTION_SIZE_BIT. Reducing it by too much would over populate /sys/devices/system/memory/ and also consume too many page->flags bits in the !vmemmap case. Also section size needs to be multiple of 128MB to have PMD based vmemmap mapping with CONFIG_ARM64_4K_PAGES.

Given these constraints, lets just reduce the section size to 128MB for 4K and 16K base page size configs, and to 512MB for 64K base page size config.

Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Anshuman Khandual <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Logan Gunthorpe <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Steven Price <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Quoted 3 lines
---
 arch/arm64/include/asm/sparsemem.h | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)
Quoted 1 line

Anshuman -- are you happy with this now?

Yes.

A small nit. There are couple of extra lines in the patch which can be dropped, probably while merging.

#11 lore raw
In reply to Will Deacon
Quoted 11 lines · Will Deacon
Quoted 8 lines · Sudarshan Rajagopalan

This patch is the follow-up from the discussions in the thread [1]. Reducing the section size has the merit of reducing wastage of reserved memory for vmmemmap mappings for sections with large memory holes. Also with smaller section size gives more grunularity and agility for memory hot(un)plugging.

But there are also constraints in reducing SECTION_SIZE_BIT:

[...]

Applied to arm64 (for-next/misc), thanks!

We have been using 128M for 4K pages for a while, using patch [1] without issues.

One thing that needs to be also modified is makedumpfile [2]- line 76, must update it SECTIONS_SIZE_BITS based on the PAGESIZE.

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://sourceforge.net/p/makedumpfile/code/ci/master/tree/arch/arm64.c

Quoted 6 lines

[1/1] arm64/sparsemem: reduce SECTION_SIZE_BITS

      https://git.kernel.org/arm64/c/f0b13ee23241

Cheers,