Skip to contentSkip to conversation

arm64: mm: decrease the section size to reduce the memory reserved for the page map

PATCH In review Wei Li 7 participants 14 messages Dec 4, 2020 – Dec 7, 2020 arm64

lore
Share & tools
lore

Wei Li

author
#0 lore raw

PATCH 1/1 arm64: mm: decrease the section size to reduce the memory reserved for the page map

For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP do not free the reserved memory for the page map, decrease the section size can reduce the waste of reserved memory.

---
 arch/arm64/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/include/asm/sparsemem.h +1 -1
index 1f43fcc79738..8963bd3def28 100644
@@ -7,7 +7,7 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS	30
+#define SECTION_SIZE_BITS	27
 #endif
 
 #endif

--
2.15.0

Reviewed-by on v1 1/1
#1 lore raw
Quoted 14 lines

-----Original Message-----
From: liwei (CM)
Sent: Friday, December 4, 2020 2:45 PM
To: [email protected]; [email protected]; [email protected]; liwei (CM) <[email protected]>
Cc: fengbaopeng <[email protected]>; [email protected]; [email protected]; Song Bao Hua (Barry Song) <[email protected]>; [email protected]; [email protected]; butao <[email protected]>
Subject: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

Quoted 8 lines

For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP do not free the reserved memory for the page map, decrease the section size can reduce the waste of reserved memory.

---

When page size is 4K, for each 1GB memory, we need 16MB vmemmap; For each 128MB memory, we need 2MB vmemmap.

So while we have memory hole like 928MB(1GB-64MB),if SECTION_SIZE_BITS is 30, we waste 15MB; if SECTION_SIZE_BITS is 27, we waste 1MB only.

Quoted 18 lines
 arch/arm64/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/include/asm/sparsemem.h +1 -2
index 1f43fcc79738..8963bd3def28 100644
@@ -7,7 +7,7 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS	30
+#define SECTION_SIZE_BITS	27
 #endif
 
 #endif
--
Quoted 1 line

2.15.0

Thanks
Barry

maintainer
#2 lore raw
In reply to Wei Li
Quoted 16 lines · Wei Li

For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP do not free the reserved memory for the page map, decrease the section size can reduce the waste of reserved memory.

---
 arch/arm64/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/include/asm/sparsemem.h +0 -0
index 1f43fcc79738..8963bd3def28 100644

arch/arm64/include/asm/sparsemem.h line 10

Quoted 5 lines
arch/arm64/include/asm/sparsemem.h +1 -1
@@ -7,7 +7,7 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS	30
+#define SECTION_SIZE_BITS	27

We chose '30' to avoid running out of bits in the page flags. What changed?

With this patch, I can trigger:

./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds SECTION_SIZE #error Allocator MAX_ORDER exceeds SECTION_SIZE

if I bump up NR_CPUS and NODES_SHIFT.

Will

#3 lore raw
Quoted 18 lines · Will Deacon
Quoted 17 lines · Wei Li

For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP do not free the reserved memory for the page map, decrease the section size can reduce the waste of reserved memory.

---
 arch/arm64/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/include/asm/sparsemem.h +0 -0
index 1f43fcc79738..8963bd3def28 100644
@@ -7,7 +7,7 @@
 
Quoted 6 lines

arch/arm64/include/asm/sparsemem.h line 10

Quoted 4 lines
arch/arm64/include/asm/sparsemem.h +1 -1
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS	30
+#define SECTION_SIZE_BITS	27

We chose '30' to avoid running out of bits in the page flags. What changed?

I think that for 64-bit there are still plenty of free bits. I didn't check now, but when I played with SPARSEMEM on m68k there were 8 bits for section out of 32.

Quoted 6 lines

With this patch, I can trigger:

./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds SECTION_SIZE #error Allocator MAX_ORDER exceeds SECTION_SIZE

if I bump up NR_CPUS and NODES_SHIFT.

I don't think it's related to NR_CPUS and NODES_SHIFT. This seems rather 64K pages that cause this.

Not that is shouldn't be addressed.

Quoted 1 line

Will

Signature · 2 lines
Sincerely yours,
Mike.
#4 lore raw
Quoted 49 lines

-----Original Message-----
From: Mike Rapoport [mailto:[email protected]]
Sent: Saturday, December 5, 2020 12:44 AM
To: Will Deacon <[email protected]>
Cc: liwei (CM) <[email protected]>; [email protected]; fengbaopeng <[email protected]>; [email protected]; [email protected]; Song Bao Hua (Barry Song) <[email protected]>; [email protected]; [email protected]; butao <[email protected]>
Subject: Re: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

Quoted 5 lines · Will Deacon
Quoted 4 lines · Wei Li

For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP do not free the reserved memory for the page map, decrease the section size can reduce the waste of reserved memory.

Quoted 8 lines

Review

Quoted 8 lines
---
 arch/arm64/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/arch/arm64/include/asm/sparsemem.h

Quoted 11 lines

arch/arm64/include/asm/sparsemem.h line 10

Quoted 9 lines

index 1f43fcc79738..8963bd3def28 100644

arch/arm64/include/asm/sparsemem.h +1 -1
@@ -7,7 +7,7 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS	30
+#define SECTION_SIZE_BITS	27

We chose '30' to avoid running out of bits in the page flags. What changed?

I think that for 64-bit there are still plenty of free bits. I didn't check now, but when I played with SPARSEMEM on m68k there were 8 bits for section out of 32.

Quoted 3 lines

With this patch, I can trigger:

./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds

SECTION_SIZE

Quoted 8 lines
Quoted 3 lines

#error Allocator MAX_ORDER exceeds SECTION_SIZE

if I bump up NR_CPUS and NODES_SHIFT.

I don't think it's related to NR_CPUS and NODES_SHIFT. This seems rather 64K pages that cause this.

Not that is shouldn't be addressed.

Right now, only 4K PAGES will define ARM64_SWAPPER_USES_SECTION_MAPS. Other cases will use vmemmap_populate_basepages().
The original patch should be only addressing the issue in 4K pages:
https://lore.kernel.org/lkml/[email protected]/

would we do something like the below?
#ifdef CONFIG_ARM64_4K_PAGE
#define SECTION_SIZE_BITS 27
#else
#define SECTION_SIZE_BITS 30
#endif

Quoted 6 lines
Quoted 2 lines

Will

--
Sincerely yours,
Mike.

Thanks
Barry

#5 lore raw
Quoted 63 lines · Song Bao Hua (Barry Song)
Quoted 49 lines

-----Original Message-----
From: Mike Rapoport [mailto:[email protected]]
Sent: Saturday, December 5, 2020 12:44 AM
To: Will Deacon <[email protected]>
Cc: liwei (CM) <[email protected]>; [email protected]; fengbaopeng <[email protected]>; [email protected]; [email protected]; Song Bao Hua (Barry Song) <[email protected]>; [email protected]; [email protected]; butao <[email protected]>
Subject: Re: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

Quoted 5 lines · Will Deacon
Quoted 4 lines · Wei Li

For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP do not free the reserved memory for the page map, decrease the section size can reduce the waste of reserved memory.

Quoted 8 lines

Review

Quoted 8 lines
---
 arch/arm64/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/arch/arm64/include/asm/sparsemem.h

Quoted 11 lines

arch/arm64/include/asm/sparsemem.h line 10

Quoted 9 lines

index 1f43fcc79738..8963bd3def28 100644

arch/arm64/include/asm/sparsemem.h +1 -1
@@ -7,7 +7,7 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS	30
+#define SECTION_SIZE_BITS	27

We chose '30' to avoid running out of bits in the page flags. What changed?

I think that for 64-bit there are still plenty of free bits. I didn't check now, but when I played with SPARSEMEM on m68k there were 8 bits for section out of 32.

Quoted 3 lines

With this patch, I can trigger:

./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds

SECTION_SIZE

Quoted 8 lines
Quoted 3 lines

#error Allocator MAX_ORDER exceeds SECTION_SIZE

if I bump up NR_CPUS and NODES_SHIFT.

I don't think it's related to NR_CPUS and NODES_SHIFT. This seems rather 64K pages that cause this.

Not that is shouldn't be addressed.

Right now, only 4K PAGES will define ARM64_SWAPPER_USES_SECTION_MAPS. Other cases will use vmemmap_populate_basepages().
The original patch should be only addressing the issue in 4K pages:
https://lore.kernel.org/lkml/[email protected]/

Quoted 6 lines

would we do something like the below?
#ifdef CONFIG_ARM64_4K_PAGE
#define SECTION_SIZE_BITS 27
#else
#define SECTION_SIZE_BITS 30
#endif

This is bit arbitrary. Probably 27 can be further reduced for 4K page size. Instead, we should make SECTION_SIZE_BITS explicitly depend upon MAX_ORDER. IOW section size should be the same as the highest order page in the buddy. CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64. A quick test shows SECTION_SIZE_BITS would be 22 on 4K pages and 29 for 64K pages. As a fall back SECTION_SIZE_BITS can still be 30 in case CONFIG_FORCE_MAX_ZONEORDER is not defined.

arch/arm64/include/asm/sparsemem.h +1 -1
@@ -7,7 +7,7 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS       CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS      30
+#define SECTION_SIZE_BITS      (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)
 #endif
 
 #endif

A similar approach exists on ia64 platform as well.

#6 lore raw
Quoted 89 lines

-----Original Message-----
From: Anshuman Khandual [mailto:[email protected]]
Sent: Monday, December 7, 2020 8:31 PM
To: Song Bao Hua (Barry Song) <[email protected]>; Mike Rapoport <[email protected]>; Will Deacon <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; liwei (CM) <[email protected]>; butao <[email protected]>; [email protected]; fengbaopeng <[email protected]>
Subject: Re: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

Quoted 8 lines · Song Bao Hua (Barry Song)
Quoted 8 lines

-----Original Message-----
From: Mike Rapoport [mailto:[email protected]]
Sent: Saturday, December 5, 2020 12:44 AM
To: Will Deacon <[email protected]>
Cc: liwei (CM) <[email protected]>; [email protected]; fengbaopeng <[email protected]>; [email protected];

[email protected];

Quoted 55 lines
Quoted 41 lines

Song Bao Hua (Barry Song) <[email protected]>; [email protected]; [email protected]; butao <[email protected]>
Subject: Re: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

Quoted 5 lines · Will Deacon
Quoted 4 lines · Wei Li

For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP do not free the reserved memory for the page map, decrease the section size can reduce the waste of reserved memory.

Quoted 8 lines

Review

Quoted 8 lines
---
 arch/arm64/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/arch/arm64/include/asm/sparsemem.h

Quoted 11 lines

arch/arm64/include/asm/sparsemem.h line 10

Quoted 9 lines

index 1f43fcc79738..8963bd3def28 100644

arch/arm64/include/asm/sparsemem.h +1 -1
@@ -7,7 +7,7 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS	30
+#define SECTION_SIZE_BITS	27

We chose '30' to avoid running out of bits in the page flags. What changed?

I think that for 64-bit there are still plenty of free bits. I didn't check now, but when I played with SPARSEMEM on m68k there were 8 bits for section out of 32.

Quoted 3 lines

With this patch, I can trigger:

./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds

SECTION_SIZE

Quoted 8 lines
Quoted 3 lines

#error Allocator MAX_ORDER exceeds SECTION_SIZE

if I bump up NR_CPUS and NODES_SHIFT.

I don't think it's related to NR_CPUS and NODES_SHIFT. This seems rather 64K pages that cause this.

Not that is shouldn't be addressed.

Right now, only 4K PAGES will define ARM64_SWAPPER_USES_SECTION_MAPS. Other cases will use vmemmap_populate_basepages().
The original patch should be only addressing the issue in 4K pages:
https://lore.kernel.org/lkml/[email protected]/

Quoted 6 lines

would we do something like the below?
#ifdef CONFIG_ARM64_4K_PAGE
#define SECTION_SIZE_BITS 27
#else
#define SECTION_SIZE_BITS 30
#endif

Quoted 7 lines

This is bit arbitrary. Probably 27 can be further reduced for 4K page size. Instead, we should make SECTION_SIZE_BITS explicitly depend upon MAX_ORDER. IOW section size should be the same as the highest order page in the buddy. CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64. A quick test shows SECTION_SIZE_BITS would be 22 on 4K pages and 29 for 64K pages. As a fall back SECTION_SIZE_BITS can still be 30 in case CONFIG_FORCE_MAX_ZONEORDER is not defined.

This will break the pmd mapping for vmemmap. As for each 128M(2^27), we can have 2MB pmd mapping.

Quoted 13 lines
arch/arm64/include/asm/sparsemem.h +1 -1
@@ -7,7 +7,7 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS       CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS      30
+#define SECTION_SIZE_BITS      (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)
 #endif
 
 #endif
Quoted 1 line

A similar approach exists on ia64 platform as well.

Thanks
Barry

#7 lore raw
In reply to Will Deacon

(+ Marc)

Quoted 26 lines · Will Deacon
Quoted 16 lines · Wei Li

For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP do not free the reserved memory for the page map, decrease the section size can reduce the waste of reserved memory.

---
 arch/arm64/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/include/asm/sparsemem.h +0 -0
index 1f43fcc79738..8963bd3def28 100644

arch/arm64/include/asm/sparsemem.h line 10

Quoted 5 lines
arch/arm64/include/asm/sparsemem.h +1 -1
@@ -7,7 +7,7 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS     CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS    30
+#define SECTION_SIZE_BITS    27

We chose '30' to avoid running out of bits in the page flags. What changed?

Quoted 7 lines

With this patch, I can trigger:

./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds SECTION_SIZE #error Allocator MAX_ORDER exceeds SECTION_SIZE

if I bump up NR_CPUS and NODES_SHIFT.

Does this mean we will run into problems with the GICv3 ITS LPI tables again if we are forced to reduce MAX_ORDER to fit inside SECTION_SIZE_BITS?

#8 lore raw
Quoted 34 lines · Ard Biesheuvel

(+ Marc)

Quoted 31 lines · Will Deacon
Quoted 16 lines · Wei Li

For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP do not free the reserved memory for the page map, decrease the section size can reduce the waste of reserved memory.

---
 arch/arm64/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/include/asm/sparsemem.h +0 -0
index 1f43fcc79738..8963bd3def28 100644

arch/arm64/include/asm/sparsemem.h line 10

Quoted 5 lines
arch/arm64/include/asm/sparsemem.h +1 -1
@@ -7,7 +7,7 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS     CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS    30
+#define SECTION_SIZE_BITS    27

We chose '30' to avoid running out of bits in the page flags. What changed?

With this patch, I can trigger:

./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds SECTION_SIZE #error Allocator MAX_ORDER exceeds SECTION_SIZE

Quoted 6 lines
Quoted 2 lines

if I bump up NR_CPUS and NODES_SHIFT.

Does this mean we will run into problems with the GICv3 ITS LPI tables again if we are forced to reduce MAX_ORDER to fit inside SECTION_SIZE_BITS?

Most probably. We are already massively constraint on platforms such as TX1, and dividing the max allocatable range by 8 isn't going to make it work any better...

        M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#9 lore raw
Quoted 40 lines · Marc Zyngier
Quoted 39 lines · Ard Biesheuvel

(+ Marc)

Quoted 35 lines · Will Deacon
Quoted 16 lines · Wei Li

For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP do not free the reserved memory for the page map, decrease the section size can reduce the waste of reserved memory.

---
 arch/arm64/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/include/asm/sparsemem.h +0 -0
index 1f43fcc79738..8963bd3def28 100644

arch/arm64/include/asm/sparsemem.h line 10

Quoted 5 lines
arch/arm64/include/asm/sparsemem.h +1 -1
@@ -7,7 +7,7 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS     CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS    30
+#define SECTION_SIZE_BITS    27

We chose '30' to avoid running out of bits in the page flags. What changed?

With this patch, I can trigger:

./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds SECTION_SIZE
#error Allocator MAX_ORDER exceeds SECTION_SIZE

if I bump up NR_CPUS and NODES_SHIFT.

Quoted 7 lines
Quoted 3 lines

Does this mean we will run into problems with the GICv3 ITS LPI tables again if we are forced to reduce MAX_ORDER to fit inside SECTION_SIZE_BITS?

Most probably. We are already massively constraint on platforms such as TX1, and dividing the max allocatable range by 8 isn't going to make it work any better...

I don't think MAX_ORDER should shrink. Even if SECTION_SIZE_BITS is reduced it should accomodate the existing MAX_ORDER.

My two pennies.

Quoted 3 lines
        M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Signature · 2 lines
Sincerely yours,
Mike.
#10 lore raw
Quoted 46 lines · Mike Rapoport
Quoted 44 lines · Marc Zyngier
Quoted 42 lines · Ard Biesheuvel

(+ Marc)

Quoted 27 lines · Will Deacon
Quoted 16 lines · Wei Li

For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP do not free the reserved memory for the page map, decrease the section size can reduce the waste of reserved memory.

---
 arch/arm64/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/include/asm/sparsemem.h +0 -0
index 1f43fcc79738..8963bd3def28 100644

arch/arm64/include/asm/sparsemem.h line 10

Quoted 5 lines
arch/arm64/include/asm/sparsemem.h +1 -1
@@ -7,7 +7,7 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS     CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS    30
+#define SECTION_SIZE_BITS    27

We chose '30' to avoid running out of bits in the page flags. What changed?

Quoted 8 lines

With this patch, I can trigger:

./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds SECTION_SIZE
#error Allocator MAX_ORDER exceeds SECTION_SIZE

if I bump up NR_CPUS and NODES_SHIFT.

Does this mean we will run into problems with the GICv3 ITS LPI tables again if we are forced to reduce MAX_ORDER to fit inside SECTION_SIZE_BITS?

Quoted 9 lines
Quoted 3 lines

Most probably. We are already massively constraint on platforms such as TX1, and dividing the max allocatable range by 8 isn't going to make it work any better...

I don't think MAX_ORDER should shrink. Even if SECTION_SIZE_BITS is reduced it should accomodate the existing MAX_ORDER.

My two pennies.

But include/linux/mmzone.h:1170 has this:

#if (MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS #error Allocator MAX_ORDER exceeds SECTION_SIZE #endif

and Will managed to trigger it after applying this patch.

#11 lore raw
Quoted 57 lines · Ard Biesheuvel
Quoted 46 lines · Mike Rapoport
Quoted 44 lines · Marc Zyngier
Quoted 42 lines · Ard Biesheuvel

(+ Marc)

Quoted 27 lines · Will Deacon
Quoted 16 lines · Wei Li

For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP do not free the reserved memory for the page map, decrease the section size can reduce the waste of reserved memory.

---
 arch/arm64/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/include/asm/sparsemem.h +0 -0
index 1f43fcc79738..8963bd3def28 100644

arch/arm64/include/asm/sparsemem.h line 10

Quoted 5 lines
arch/arm64/include/asm/sparsemem.h +1 -1
@@ -7,7 +7,7 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS     CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS    30
+#define SECTION_SIZE_BITS    27

We chose '30' to avoid running out of bits in the page flags. What changed?

Quoted 8 lines

With this patch, I can trigger:

./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds SECTION_SIZE
#error Allocator MAX_ORDER exceeds SECTION_SIZE

if I bump up NR_CPUS and NODES_SHIFT.

Does this mean we will run into problems with the GICv3 ITS LPI tables again if we are forced to reduce MAX_ORDER to fit inside SECTION_SIZE_BITS?

Quoted 9 lines
Quoted 3 lines

Most probably. We are already massively constraint on platforms such as TX1, and dividing the max allocatable range by 8 isn't going to make it work any better...

I don't think MAX_ORDER should shrink. Even if SECTION_SIZE_BITS is reduced it should accomodate the existing MAX_ORDER.

My two pennies.

Quoted 7 lines

But include/linux/mmzone.h:1170 has this:

#if (MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS #error Allocator MAX_ORDER exceeds SECTION_SIZE #endif

and Will managed to trigger it after applying this patch.

Right, because with 64K pages section size of 27 bits is not enough to accomodate MAX_ORDER (2^13 pages of 64K).

Which means that definition of SECTION_SIZE_BITS should take MAX_ORDER into account either statically with

#ifdef ARM64_4K_PAGES
#define SECTION_SIZE_BITS <a number>
#elif ARM64_16K_PAGES
#define SECTION_SIZE_BITS <a larger number>
#elif ARM64_64K_PAGES
#define SECTION_SIZE_BITS <even larger number>
#else
#error "and what is the page size?"
#endif

or dynamically, like e.g. ia64 does:

#ifdef CONFIG_FORCE_MAX_ZONEORDER
#if ((CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS) #undef SECTION_SIZE_BITS
#define SECTION_SIZE_BITS (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) #endif

Signature · 2 lines
Sincerely yours,
Mike.
#12 lore raw
Quoted 82 lines · Mike Rapoport
Quoted 57 lines · Ard Biesheuvel
Quoted 46 lines · Mike Rapoport
Quoted 44 lines · Marc Zyngier
Quoted 42 lines · Ard Biesheuvel

(+ Marc)

Quoted 27 lines · Will Deacon
Quoted 16 lines · Wei Li

For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP do not free the reserved memory for the page map, decrease the section size can reduce the waste of reserved memory.

---
 arch/arm64/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/include/asm/sparsemem.h +0 -0
index 1f43fcc79738..8963bd3def28 100644

arch/arm64/include/asm/sparsemem.h line 10

Quoted 5 lines
arch/arm64/include/asm/sparsemem.h +1 -1
@@ -7,7 +7,7 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS     CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS    30
+#define SECTION_SIZE_BITS    27

We chose '30' to avoid running out of bits in the page flags. What changed?

Quoted 8 lines

With this patch, I can trigger:

./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds SECTION_SIZE
#error Allocator MAX_ORDER exceeds SECTION_SIZE

if I bump up NR_CPUS and NODES_SHIFT.

Does this mean we will run into problems with the GICv3 ITS LPI tables again if we are forced to reduce MAX_ORDER to fit inside SECTION_SIZE_BITS?

Quoted 9 lines
Quoted 3 lines

Most probably. We are already massively constraint on platforms such as TX1, and dividing the max allocatable range by 8 isn't going to make it work any better...

I don't think MAX_ORDER should shrink. Even if SECTION_SIZE_BITS is reduced it should accomodate the existing MAX_ORDER.

My two pennies.

Quoted 7 lines

But include/linux/mmzone.h:1170 has this:

#if (MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS #error Allocator MAX_ORDER exceeds SECTION_SIZE #endif

and Will managed to trigger it after applying this patch.

Right, because with 64K pages section size of 27 bits is not enough to accomodate MAX_ORDER (2^13 pages of 64K).

Which means that definition of SECTION_SIZE_BITS should take MAX_ORDER into account either statically with

#ifdef ARM64_4K_PAGES
#define SECTION_SIZE_BITS <a number>
#elif ARM64_16K_PAGES
#define SECTION_SIZE_BITS <a larger number>
#elif ARM64_64K_PAGES
#define SECTION_SIZE_BITS <even larger number>
#else
#error "and what is the page size?"
#endif

Quoted 7 lines

or dynamically, like e.g. ia64 does:

#ifdef CONFIG_FORCE_MAX_ZONEORDER
#if ((CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS) #undef SECTION_SIZE_BITS
#define SECTION_SIZE_BITS (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) #endif

I had proposed the same on the other thread here. But with this the SECTION_SIZE_BITS becomes 22 in case of 4K page size reducing to an extent where PMD based vmemmap mapping could not be created. Though have not looked into much details yet.

Using CONFIG_FORCE_MAX_ZONEORDER seems to the right thing to do. But if that does not reasonably work for 4K pages, we might have to hard code it as 27 to have huge page vmemmap mappings.

author
#13 lore raw

(+ saberlily + jiapeng)

Quoted 103 lines · Anshuman Khandual
Quoted 82 lines · Mike Rapoport
Quoted 57 lines · Ard Biesheuvel
Quoted 46 lines · Mike Rapoport
Quoted 44 lines · Marc Zyngier
Quoted 42 lines · Ard Biesheuvel

(+ Marc)

Quoted 27 lines · Will Deacon
Quoted 16 lines · Wei Li

For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP do not free the reserved memory for the page map, decrease the section size can reduce the waste of reserved memory.

---
 arch/arm64/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/include/asm/sparsemem.h +0 -0
index 1f43fcc79738..8963bd3def28 100644

arch/arm64/include/asm/sparsemem.h line 10

Quoted 5 lines
arch/arm64/include/asm/sparsemem.h +1 -1
@@ -7,7 +7,7 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS     CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS    30
+#define SECTION_SIZE_BITS    27

We chose '30' to avoid running out of bits in the page flags. What changed?

Quoted 8 lines

With this patch, I can trigger:

./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds SECTION_SIZE
#error Allocator MAX_ORDER exceeds SECTION_SIZE

if I bump up NR_CPUS and NODES_SHIFT.

Does this mean we will run into problems with the GICv3 ITS LPI tables again if we are forced to reduce MAX_ORDER to fit inside SECTION_SIZE_BITS?

Quoted 9 lines
Quoted 3 lines

Most probably. We are already massively constraint on platforms such as TX1, and dividing the max allocatable range by 8 isn't going to make it work any better...

I don't think MAX_ORDER should shrink. Even if SECTION_SIZE_BITS is reduced it should accomodate the existing MAX_ORDER.

My two pennies.

Quoted 7 lines

But include/linux/mmzone.h:1170 has this:

#if (MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS #error Allocator MAX_ORDER exceeds SECTION_SIZE #endif

and Will managed to trigger it after applying this patch.

Right, because with 64K pages section size of 27 bits is not enough to accomodate MAX_ORDER (2^13 pages of 64K).

Which means that definition of SECTION_SIZE_BITS should take MAX_ORDER into account either statically with

#ifdef ARM64_4K_PAGES
#define SECTION_SIZE_BITS <a number>
#elif ARM64_16K_PAGES
#define SECTION_SIZE_BITS <a larger number>
#elif ARM64_64K_PAGES
#define SECTION_SIZE_BITS <even larger number>
#else
#error "and what is the page size?"
#endif

Quoted 7 lines

or dynamically, like e.g. ia64 does:

#ifdef CONFIG_FORCE_MAX_ZONEORDER
#if ((CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS) #undef SECTION_SIZE_BITS
#define SECTION_SIZE_BITS (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) #endif

I had proposed the same on the other thread here. But with this the SECTION_SIZE_BITS becomes 22 in case of 4K page size reducing to an extent where PMD based vmemmap mapping could not be created. Though have not looked into much details yet.

Using CONFIG_FORCE_MAX_ZONEORDER seems to the right thing to do. But if that does not reasonably work for 4K pages, we might have to hard code it as 27 to have huge page vmemmap mappings.
.