arm64: mm: optimize VA space organization for 52-bit

On arm64, the global variable memstart_addr represents the physical address of PAGE_OFFSET, and so physical to virtual translations or vice versa used to come down to simple additions or subtractions involving the values of PAGE_OFFSET and memstart_addr.

When support for 52-bit virtual addressing was introduced, we had to deal with PAGE_OFFSET potentially being outside of the region that can be covered by the virtual range (as the 52-bit VA capable build needs to be able to run on systems that are only 48-bit VA capable), and for this reason, another translation was introduced, and recorded in the global variable physvirt_offset.

However, if we go back to the original definition of memstart_addr, i.e., the physical address of PAGE_OFFSET, it turns out that there is no need for two separate translations: instead, we can simply subtract the size of the unaddressable VA space from memstart_addr to make the available physical memory appear in the 48-bit addressable VA region.

This simplifies things, but also fixes a bug on KASLR builds, which may update memstart_addr later on in arm64_memblock_init(), but fails to update vmemmap and physvirt_offset accordingly.

Fixes 5383cc6efed13 ("arm64: mm: Introduce vabits_actual")

Signed-off-by Ard Biesheuvel <[email protected]>

On arm64, the global variable memstart_addr represents the physical address of PAGE_OFFSET, and so physical to virtual translations or vice versa used to come down to simple additions or subtractions involving the values of PAGE_OFFSET and memstart_addr.

When support for 52-bit virtual addressing was introduced, we had to deal with PAGE_OFFSET potentially being outside of the region that can be covered by the virtual range (as the 52-bit VA capable build needs to be able to run on systems that are only 48-bit VA capable), and for this reason, another translation was introduced, and recorded in the global variable physvirt_offset.

However, if we go back to the original definition of memstart_addr, i.e., the physical address of PAGE_OFFSET, it turns out that there is no need for two separate translations: instead, we can simply subtract the size of the unaddressable VA space from memstart_addr to make the available physical memory appear in the 48-bit addressable VA region.

This simplifies things, but also fixes a bug on KASLR builds, which may update memstart_addr later on in arm64_memblock_init(), but fails to update vmemmap and physvirt_offset accordingly.

Fixes 5383cc6efed13 ("arm64: mm: Introduce vabits_actual")

Signed-off-by Ard Biesheuvel <[email protected]>

On arm64, the global variable memstart_addr represents the physical address of PAGE_OFFSET, and so physical to virtual translations or vice versa used to come down to simple additions or subtractions involving the values of PAGE_OFFSET and memstart_addr.

When support for 52-bit virtual addressing was introduced, we had to deal with PAGE_OFFSET potentially being outside of the region that can be covered by the virtual range (as the 52-bit VA capable build needs to be able to run on systems that are only 48-bit VA capable), and for this reason, another translation was introduced, and recorded in the global variable physvirt_offset.

However, if we go back to the original definition of memstart_addr, i.e., the physical address of PAGE_OFFSET, it turns out that there is no need for two separate translations: instead, we can simply subtract the size of the unaddressable VA space from memstart_addr to make the available physical memory appear in the 48-bit addressable VA region.

This simplifies things, but also fixes a bug on KASLR builds, which may update memstart_addr later on in arm64_memblock_init(), but fails to update vmemmap and physvirt_offset accordingly.

Fixes 5383cc6efed13 ("arm64: mm: Introduce vabits_actual")

Signed-off-by Ard Biesheuvel <[email protected]>

---
 arch/arm64/include/asm/memory.h  |  5 ++--
 arch/arm64/include/asm/pgtable.h |  4 +--
 arch/arm64/mm/init.c             | 30 +++++++-------------
 3 files changed, 14 insertions(+), 25 deletions(-)

For historical reasons, the arm64 kernel VA space is configured as two equally sized halves, i.e., on a 48-bit VA build, the VA space is split into a 47-bit vmalloc region and a 47-bit linear region.

When support for 52-bit virtual addressing was added, this equal split was kept, resulting in a substantial waste of virtual address space in the linear region:

48-bit VA 52-bit VA
0xffff_ffff_ffff_ffff +-------------+ +-------------+
| vmalloc | | vmalloc |
0xffff_8000_0000_0000 +-------------+ _PAGE_END(48) +-------------+

                         |   linear    |               :             :
   0xffff_0000_0000_0000 +-------------+               :             :
                         :             :               :             :
                         :             :               :             :
                         :             :               :             :
                         :             :               :  currently  :
                         :  unusable   :               :             :
                         :             :               :   unused    :
                         :     by      :               :             :
                         :             :               :             :
                         :  hardware   :               :             :
                         :             :               :             :
   0xfff8_0000_0000_0000 :             : _PAGE_END(52) +-------------+
                         :             :               |             |
                         :             :               |             |
                         :             :               |             |
                         :             :               |             |
                         :             :               |             |
                         :  unusable   :               |             |
                         :             :               |   linear    |
                         :     by      :               |             |
                         :             :               |   region    |
                         :  hardware   :               |             |
                         :             :               |             |
                         :             :               |             |
                         :             :               |             |
                         :             :               |             |
                         :             :               |             |
                         :             :               |             |
   0xfff0_0000_0000_0000 +-------------+  PAGE_OFFSET  +-------------+

As illustrated above, the 52-bit VA kernel uses 47 bits for the vmalloc space (as before), to ensure that a single 64k granule kernel image can support any 64k granule capable system, regardless of whether it supports the 52-bit virtual addressing extension. However, due to the fact that the VA space is still split in equal halves, the linear region is only 2^51 bytes in size, wasting almost half of the 52-bit VA space.

Let's fix this, by abandoning the equal split, and simply assigning all VA space outside of the vmalloc region to the linear region.

The KASAN shadow region is reconfigured so that it ends at the start of the vmalloc region, and grows downwards. That way, the arrangement of the vmalloc space (which contains kernel mappings, modules, BPF region, the vmemmap array etc) is identical between non-KASAN and KASAN builds, which aids debugging.

Signed-off-by Ard Biesheuvel <[email protected]>

---
  Documentation/arm64/kasan-offsets.sh |  3 +--
  Documentation/arm64/memory.rst       | 19 +++++++++----------
  arch/arm64/Kconfig                   | 20 ++++++++++----------
  arch/arm64/include/asm/memory.h      | 12 +++++-------
  arch/arm64/mm/init.c                 |  2 +-
  5 files changed, 26 insertions(+), 30 deletions(-)

Hi Ard,

One comment below...

Quoted 68 lines · Ard Biesheuvel

For historical reasons, the arm64 kernel VA space is configured as two equally sized halves, i.e., on a 48-bit VA build, the VA space is split into a 47-bit vmalloc region and a 47-bit linear region.

When support for 52-bit virtual addressing was added, this equal split was kept, resulting in a substantial waste of virtual address space in the linear region:

48-bit VA 52-bit VA
0xffff_ffff_ffff_ffff +-------------+ +-------------+
| vmalloc | | vmalloc |
0xffff_8000_0000_0000 +-------------+ _PAGE_END(48) +-------------+

                         |   linear    |               :             :
   0xffff_0000_0000_0000 +-------------+               :             :
                         :             :               :             :
                         :             :               :             :
                         :             :               :             :
                         :             :               :  currently  :
                         :  unusable   :               :             :
                         :             :               :   unused    :
                         :     by      :               :             :
                         :             :               :             :
                         :  hardware   :               :             :
                         :             :               :             :
   0xfff8_0000_0000_0000 :             : _PAGE_END(52) +-------------+
                         :             :               |             |
                         :             :               |             |
                         :             :               |             |
                         :             :               |             |
                         :             :               |             |
                         :  unusable   :               |             |
                         :             :               |   linear    |
                         :     by      :               |             |
                         :             :               |   region    |
                         :  hardware   :               |             |
                         :             :               |             |
                         :             :               |             |
                         :             :               |             |
                         :             :               |             |
                         :             :               |             |
                         :             :               |             |
   0xfff0_0000_0000_0000 +-------------+  PAGE_OFFSET  +-------------+

As illustrated above, the 52-bit VA kernel uses 47 bits for the vmalloc space (as before), to ensure that a single 64k granule kernel image can support any 64k granule capable system, regardless of whether it supports the 52-bit virtual addressing extension. However, due to the fact that the VA space is still split in equal halves, the linear region is only 2^51 bytes in size, wasting almost half of the 52-bit VA space.

Let's fix this, by abandoning the equal split, and simply assigning all VA space outside of the vmalloc region to the linear region.

The KASAN shadow region is reconfigured so that it ends at the start of the vmalloc region, and grows downwards. That way, the arrangement of the vmalloc space (which contains kernel mappings, modules, BPF region, the vmemmap array etc) is identical between non-KASAN and KASAN builds, which aids debugging.

Signed-off-by Ard Biesheuvel <[email protected]>

---
  Documentation/arm64/kasan-offsets.sh |  3 +--
  Documentation/arm64/memory.rst       | 19 +++++++++----------
  arch/arm64/Kconfig                   | 20 ++++++++++----------
  arch/arm64/include/asm/memory.h      | 12 +++++-------
  arch/arm64/mm/init.c                 |  2 +-
  5 files changed, 26 insertions(+), 30 deletions(-)

[...]

Documentation/arm64/memory.rst AArch64 Linux memory layout with 4KB pages + 4 levels (48-bit):: · line 35

Quoted 13 lines

Documentation/arm64/memory.rst +1 -4

index cf03b3290800..ee51eb66a578 100644
@@ -32,10 +32,10 @@ AArch64 Linux memory layout with 4KB pages + 4 levels (48-bit)::
    -----------------------------------------------------------------------
    0000000000000000  0000ffffffffffff         256TB          user
    ffff000000000000  ffff7fffffffffff         128TB          kernel logical memory map
-  ffff800000000000   ffff9fffffffffff          32TB          kasan shadow region
-  ffffa00000000000   ffffa00007ffffff         128MB          bpf jit region
-  ffffa00008000000   ffffa0000fffffff         128MB          modules
-  ffffa00010000000   fffffdffbffeffff         ~93TB          vmalloc
+[ ffff600000000000   ffff7fffffffffff ]        32TB          [ kasan shadow region ]

We have the KASAN shadow region intersecting with the kernel logical memory map now. Could this present a problem if both the KASAN and phys_to_virt paths land on the same VA? Or is that not possible? (Also what about smaller VA sizes?)

If it is a problem, could carving out the appropriate memblocks (and warning the user) be a way forward?

Quoted 96 lines · Steve Capper

Hi Ard,

One comment below...

Quoted 68 lines · Ard Biesheuvel

For historical reasons, the arm64 kernel VA space is configured as two equally sized halves, i.e., on a 48-bit VA build, the VA space is split into a 47-bit vmalloc region and a 47-bit linear region.

When support for 52-bit virtual addressing was added, this equal split was kept, resulting in a substantial waste of virtual address space in the linear region:

48-bit VA 52-bit VA
0xffff_ffff_ffff_ffff +-------------+ +-------------+
| vmalloc | | vmalloc |
0xffff_8000_0000_0000 +-------------+ _PAGE_END(48) +-------------+
| linear | : :
0xffff_0000_0000_0000 +-------------+ : :
: : : :
: : : :
: : : :
: : : currently :
: unusable : : :
: : : unused :
: by : : :
: : : :
: hardware : : :
: : : :
0xfff8_0000_0000_0000 : : _PAGE_END(52) +-------------+
: : | |
: : | |
: : | |
: : | |
: : | |
: unusable : | |
: : | linear |
: by : | |
: : | region |
: hardware : | |
: : | |
: : | |
: : | |
: : | |
: : | |
: : | |
0xfff0_0000_0000_0000 +-------------+ PAGE_OFFSET +-------------+

As illustrated above, the 52-bit VA kernel uses 47 bits for the vmalloc space (as before), to ensure that a single 64k granule kernel image can support any 64k granule capable system, regardless of whether it supports the 52-bit virtual addressing extension. However, due to the fact that the VA space is still split in equal halves, the linear region is only 2^51 bytes in size, wasting almost half of the 52-bit VA space.

Let's fix this, by abandoning the equal split, and simply assigning all VA space outside of the vmalloc region to the linear region.

The KASAN shadow region is reconfigured so that it ends at the start of the vmalloc region, and grows downwards. That way, the arrangement of the vmalloc space (which contains kernel mappings, modules, BPF region, the vmemmap array etc) is identical between non-KASAN and KASAN builds, which aids debugging.

Signed-off-by Ard Biesheuvel <[email protected]>

---
   Documentation/arm64/kasan-offsets.sh |  3 +--
   Documentation/arm64/memory.rst       | 19 +++++++++----------
   arch/arm64/Kconfig                   | 20 ++++++++++----------
   arch/arm64/include/asm/memory.h      | 12 +++++-------
   arch/arm64/mm/init.c                 |  2 +-
   5 files changed, 26 insertions(+), 30 deletions(-)

[...]

Documentation/arm64/memory.rst AArch64 Linux memory layout with 4KB pages + 4 levels (48-bit):: · line 35

Quoted 13 lines

Documentation/arm64/memory.rst +1 -4

index cf03b3290800..ee51eb66a578 100644
@@ -32,10 +32,10 @@ AArch64 Linux memory layout with 4KB pages + 4 levels (48-bit)::
     -----------------------------------------------------------------------
     0000000000000000  0000ffffffffffff         256TB          user
     ffff000000000000  ffff7fffffffffff         128TB          kernel logical memory map
-  ffff800000000000   ffff9fffffffffff          32TB          kasan shadow region
-  ffffa00000000000   ffffa00007ffffff         128MB          bpf jit region
-  ffffa00008000000   ffffa0000fffffff         128MB          modules
-  ffffa00010000000   fffffdffbffeffff         ~93TB          vmalloc
+[ ffff600000000000   ffff7fffffffffff ]        32TB          [ kasan shadow region ]

We have the KASAN shadow region intersecting with the kernel logical memory map now. Could this present a problem if both the KASAN and phys_to_virt paths land on the same VA? Or is that not possible? (Also what about smaller VA sizes?)

Quoted 3 lines

If it is a problem, could carving out the appropriate memblocks (and warning the user) be a way forward?

Hi Steve,

This is currently taken into account in arm64_memblock_init():

const s64 linear_region_size = PAGE_END - _PAGE_OFFSET(vabits_actual);

When CONFIG_KASAN=y, PAGE_END is #defined as

(KASAN_SHADOW_END - (1UL << (vabits_actual - KASAN_SHADOW_SCALE_SHIFT)))

and so the linear region ends where the KASAN shadow region starts. So even if the static definitions overlap, the shared region will not be used for mapping memory if it is needed for allocating shadow pages.

For historical reasons, the arm64 kernel VA space is configured as two equally sized halves, i.e., on a 48-bit VA build, the VA space is split into a 47-bit vmalloc region and a 47-bit linear region.

When support for 52-bit virtual addressing was added, this equal split was kept, resulting in a substantial waste of virtual address space in the linear region:

48-bit VA 52-bit VA
0xffff_ffff_ffff_ffff +-------------+ +-------------+
| vmalloc | | vmalloc |
0xffff_8000_0000_0000 +-------------+ _PAGE_END(48) +-------------+

                        |   linear    |               :             :
  0xffff_0000_0000_0000 +-------------+               :             :
                        :             :               :             :
                        :             :               :             :
                        :             :               :             :
                        :             :               :  currently  :
                        :  unusable   :               :             :
                        :             :               :   unused    :
                        :     by      :               :             :
                        :             :               :             :
                        :  hardware   :               :             :
                        :             :               :             :
  0xfff8_0000_0000_0000 :             : _PAGE_END(52) +-------------+
                        :             :               |             |
                        :             :               |             |
                        :             :               |             |
                        :             :               |             |
                        :             :               |             |
                        :  unusable   :               |             |
                        :             :               |   linear    |
                        :     by      :               |             |
                        :             :               |   region    |
                        :  hardware   :               |             |
                        :             :               |             |
                        :             :               |             |
                        :             :               |             |
                        :             :               |             |
                        :             :               |             |
                        :             :               |             |
  0xfff0_0000_0000_0000 +-------------+  PAGE_OFFSET  +-------------+

As illustrated above, the 52-bit VA kernel uses 47 bits for the vmalloc space (as before), to ensure that a single 64k granule kernel image can support any 64k granule capable system, regardless of whether it supports the 52-bit virtual addressing extension. However, due to the fact that the VA space is still split in equal halves, the linear region is only 2^51 bytes in size, wasting almost half of the 52-bit VA space.

Quoted 43 lines · Ard Biesheuvel

For historical reasons, the arm64 kernel VA space is configured as two equally sized halves, i.e., on a 48-bit VA build, the VA space is split into a 47-bit vmalloc region and a 47-bit linear region.

When support for 52-bit virtual addressing was added, this equal split was kept, resulting in a substantial waste of virtual address space in the linear region:

48-bit VA 52-bit VA
0xffff_ffff_ffff_ffff +-------------+ +-------------+
| vmalloc | | vmalloc |
0xffff_8000_0000_0000 +-------------+ _PAGE_END(48) +-------------+

                        |   linear    |               :             :
  0xffff_0000_0000_0000 +-------------+               :             :
                        :             :               :             :
                        :             :               :             :
                        :             :               :             :
                        :             :               :  currently  :
                        :  unusable   :               :             :
                        :             :               :   unused    :
                        :     by      :               :             :
                        :             :               :             :
                        :  hardware   :               :             :
                        :             :               :             :
  0xfff8_0000_0000_0000 :             : _PAGE_END(52) +-------------+
                        :             :               |             |
                        :             :               |             |
                        :             :               |             |
                        :             :               |             |
                        :             :               |             |
                        :  unusable   :               |             |
                        :             :               |   linear    |
                        :     by      :               |             |
                        :             :               |   region    |
                        :  hardware   :               |             |
                        :             :               |             |
                        :             :               |             |
                        :             :               |             |
                        :             :               |             |
                        :             :               |             |
                        :             :               |             |
  0xfff0_0000_0000_0000 +-------------+  PAGE_OFFSET  +-------------+

Quoted 6 lines

As illustrated above, the 52-bit VA kernel uses 47 bits for the vmalloc space (as before), to ensure that a single 64k granule kernel image can support any 64k granule capable system, regardless of whether it supports the 52-bit virtual addressing extension. However, due to the fact that the VA space is still split in equal halves, the linear region is only 2^51 bytes in size, wasting almost half of the 52-bit VA space.

Right, that is a huge wastage. Increasing the linear mapping will definitely help in getting more memory used on the system. But now do we have enough vmemmap range to support this enlarged linear mapping ?

Now that we have reverted the introduction of the vmemmap struct page pointer and the separate physvirt_offset, we can simplify things further, and place the vmemmap region in the VA space in such a way that virtual to page translations and vice versa can be implemented using a single arithmetic shift.

One happy coincidence resulting from this is that the 48-bit/4k and 52-bit/64k configurations (which are assumed to be the two most prevalent) end up with the same placement of the vmemmap region. In a subsequent patch, we will take advantage of this, and unify the memory maps even more.

Signed-off-by Ard Biesheuvel <[email protected]>

Now that we have reverted the introduction of the vmemmap struct page pointer and the separate physvirt_offset, we can simplify things further, and place the vmemmap region in the VA space in such a way that virtual to page translations and vice versa can be implemented using a single arithmetic shift.

One happy coincidence resulting from this is that the 48-bit/4k and 52-bit/64k configurations (which are assumed to be the two most prevalent) end up with the same placement of the vmemmap region. In a subsequent patch, we will take advantage of this, and unify the memory maps even more.

Signed-off-by Ard Biesheuvel <[email protected]>

Hi Ard,

Quoted 6 lines · Ard Biesheuvel

Now that we have reverted the introduction of the vmemmap struct page pointer and the separate physvirt_offset, we can simplify things further, and place the vmemmap region in the VA space in such a way that virtual to page translations and vice versa can be implemented using a single arithmetic shift.

Quoted 7 lines

One happy coincidence resulting from this is that the 48-bit/4k and 52-bit/64k configurations (which are assumed to be the two most prevalent) end up with the same placement of the vmemmap region. In a subsequent patch, we will take advantage of this, and unify the memory maps even more.

Signed-off-by Ard Biesheuvel <[email protected]>

This is now commit 8c96400d6a39be76 ("arm64: mm: make vmemmap region a projection of the linear region") in arm64/for-next/core.

arch/arm64/mm/init.c static void __init free_unused_memmap(void) · line 507

Quoted 7 lines

arch/arm64/mm/init.c +1 -0

@@ -504,6 +504,8 @@ static void __init free_unused_memmap(void)
  */
 void __init mem_init(void)
 {
+       BUILD_BUG_ON(!is_power_of_2(sizeof(struct page)));

This check is triggering for me.

If CONFIG_MEMCG=n, sizeof(struct page) = 56.

If CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y, this is mitigated by the explicit alignment:

    #ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE
    #define _struct_page_alignment  __aligned(2 * sizeof(unsigned long))
    #else
    #define _struct_page_alignment
    #endif

    struct page { ... } _struct_page_alignment;

However, HAVE_ALIGNED_STRUCT_PAGE is selected only if SLUB, while my .config is using SLAB.

Quoted 42 lines · Geert Uytterhoeven

Hi Ard,

Quoted 6 lines · Ard Biesheuvel

Now that we have reverted the introduction of the vmemmap struct page pointer and the separate physvirt_offset, we can simplify things further, and place the vmemmap region in the VA space in such a way that virtual to page translations and vice versa can be implemented using a single arithmetic shift.

Quoted 7 lines

One happy coincidence resulting from this is that the 48-bit/4k and 52-bit/64k configurations (which are assumed to be the two most prevalent) end up with the same placement of the vmemmap region. In a subsequent patch, we will take advantage of this, and unify the memory maps even more.

Signed-off-by Ard Biesheuvel <[email protected]>

This is now commit 8c96400d6a39be76 ("arm64: mm: make vmemmap region a projection of the linear region") in arm64/for-next/core.

arch/arm64/mm/init.c static void __init free_unused_memmap(void) · line 507

Quoted 7 lines

arch/arm64/mm/init.c +1 -0

@@ -504,6 +504,8 @@ static void __init free_unused_memmap(void)
  */
 void __init mem_init(void)
 {
+       BUILD_BUG_ON(!is_power_of_2(sizeof(struct page)));

This check is triggering for me.

If CONFIG_MEMCG=n, sizeof(struct page) = 56.

If CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y, this is mitigated by the explicit alignment:

    #ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE
    #define _struct_page_alignment  __aligned(2 * sizeof(unsigned long))
    #else
    #define _struct_page_alignment
    #endif

Quoted 5 lines

    struct page { ... } _struct_page_alignment;

However, HAVE_ALIGNED_STRUCT_PAGE is selected only if SLUB, while my .config is using SLAB.

Thanks for the report. I will look into this.

Quoted 40 lines · Ard Biesheuvel

Quoted 39 lines · Geert Uytterhoeven

Quoted 6 lines · Ard Biesheuvel

Now that we have reverted the introduction of the vmemmap struct page pointer and the separate physvirt_offset, we can simplify things further, and place the vmemmap region in the VA space in such a way that virtual to page translations and vice versa can be implemented using a single arithmetic shift.

Quoted 7 lines

One happy coincidence resulting from this is that the 48-bit/4k and 52-bit/64k configurations (which are assumed to be the two most prevalent) end up with the same placement of the vmemmap region. In a subsequent patch, we will take advantage of this, and unify the memory maps even more.

Signed-off-by Ard Biesheuvel <[email protected]>

This is now commit 8c96400d6a39be76 ("arm64: mm: make vmemmap region a projection of the linear region") in arm64/for-next/core.

arch/arm64/mm/init.c static void __init free_unused_memmap(void) · line 507

Quoted 7 lines

arch/arm64/mm/init.c +1 -0

@@ -504,6 +504,8 @@ static void __init free_unused_memmap(void)
  */
 void __init mem_init(void)
 {
+       BUILD_BUG_ON(!is_power_of_2(sizeof(struct page)));

This check is triggering for me.

If CONFIG_MEMCG=n, sizeof(struct page) = 56.

If CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y, this is mitigated by the explicit alignment:

    #ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE
    #define _struct_page_alignment  __aligned(2 * sizeof(unsigned long))
    #else
    #define _struct_page_alignment
    #endif

Quoted 7 lines

Quoted 5 lines

    struct page { ... } _struct_page_alignment;

However, HAVE_ALIGNED_STRUCT_PAGE is selected only if SLUB, while my .config is using SLAB.

Thanks for the report. I will look into this.

OK, so we can obviously fix this easily by setting CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y unconditionally instead of only 'if SLUB'. The question is whether that is likely to lead to any performance regressions.

So first of all, having a smaller struct page means we can fit more of them into memory. On a 4k pages config with SPARSEMEM_VMEMMAP enabled (which allocates struct pages in 2M blocks), every 2M block can cover 146 MB of DRAM instead of 128 MB. I'm not sure what kind of DRAM arrangement would be needed to take advantage of this in practice, though.

Another aspect is D-cache utilization: cache lines are typically 64 bytes on arm64, and while we can improve Dcache utilization in theory (by virtue of the smaller size), the random access nature of struct pages may well result in the opposite, given that 3 out of 4 struct pages now straddle two cachelines.

Given the above, and given the purpose of this patch series, which was to tidy up and unify different configurations, in order to reduce the size of the validation matrix, I think it would be reasonable to simply set CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y for arm64 in all cases.

Quoted 46 lines · Ard Biesheuvel

Quoted 44 lines · Geert Uytterhoeven

Quoted 6 lines · Ard Biesheuvel

Now that we have reverted the introduction of the vmemmap struct page pointer and the separate physvirt_offset, we can simplify things further, and place the vmemmap region in the VA space in such a way that virtual to page translations and vice versa can be implemented using a single arithmetic shift.

Quoted 7 lines

One happy coincidence resulting from this is that the 48-bit/4k and 52-bit/64k configurations (which are assumed to be the two most prevalent) end up with the same placement of the vmemmap region. In a subsequent patch, we will take advantage of this, and unify the memory maps even more.

Signed-off-by Ard Biesheuvel <[email protected]>

This is now commit 8c96400d6a39be76 ("arm64: mm: make vmemmap region a projection of the linear region") in arm64/for-next/core.

arch/arm64/mm/init.c static void __init free_unused_memmap(void) · line 507

Quoted 7 lines

arch/arm64/mm/init.c +1 -0

@@ -504,6 +504,8 @@ static void __init free_unused_memmap(void)
  */
 void __init mem_init(void)
 {
+       BUILD_BUG_ON(!is_power_of_2(sizeof(struct page)));

This check is triggering for me.

If CONFIG_MEMCG=n, sizeof(struct page) = 56.

If CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y, this is mitigated by the explicit alignment:

    #ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE
    #define _struct_page_alignment  __aligned(2 * sizeof(unsigned long))
    #else
    #define _struct_page_alignment
    #endif

    struct page { ... } _struct_page_alignment;

However, HAVE_ALIGNED_STRUCT_PAGE is selected only if SLUB, while my .config is using SLAB.

Quoted 1 line

Thanks for the report. I will look into this.

OK, so we can obviously fix this easily by setting CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y unconditionally instead of only 'if SLUB'. The question is whether that is likely to lead to any performance regressions.

Quoted 47 lines · Ard Biesheuvel

Quoted 46 lines · Ard Biesheuvel

Quoted 44 lines · Geert Uytterhoeven

Quoted 6 lines · Ard Biesheuvel

Now that we have reverted the introduction of the vmemmap struct page pointer and the separate physvirt_offset, we can simplify things further, and place the vmemmap region in the VA space in such a way that virtual to page translations and vice versa can be implemented using a single arithmetic shift.

Quoted 7 lines

One happy coincidence resulting from this is that the 48-bit/4k and 52-bit/64k configurations (which are assumed to be the two most prevalent) end up with the same placement of the vmemmap region. In a subsequent patch, we will take advantage of this, and unify the memory maps even more.

Signed-off-by Ard Biesheuvel <[email protected]>

This is now commit 8c96400d6a39be76 ("arm64: mm: make vmemmap region a projection of the linear region") in arm64/for-next/core.

arch/arm64/mm/init.c static void __init free_unused_memmap(void) · line 507

Quoted 7 lines

arch/arm64/mm/init.c +1 -0

@@ -504,6 +504,8 @@ static void __init free_unused_memmap(void)
  */
 void __init mem_init(void)
 {
+       BUILD_BUG_ON(!is_power_of_2(sizeof(struct page)));

This check is triggering for me.

If CONFIG_MEMCG=n, sizeof(struct page) = 56.

If CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y, this is mitigated by the explicit alignment:

    #ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE
    #define _struct_page_alignment  __aligned(2 * sizeof(unsigned long))
    #else
    #define _struct_page_alignment
    #endif

    struct page { ... } _struct_page_alignment;

However, HAVE_ALIGNED_STRUCT_PAGE is selected only if SLUB, while my .config is using SLAB.

Quoted 6 lines

Quoted 1 line

Thanks for the report. I will look into this.

OK, so we can obviously fix this easily by setting CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y unconditionally instead of only 'if SLUB'. The question is whether that is likely to lead to any performance regressions.

I'm not sure I understand. The mem_init() bug triggers if sizeof(struct page) is not a power of 2. HAVE_ALIGNED_STRUCT_PAGE forces the page alignment to 16 bytes but, for example, a 48-byte structure is not a power of 2.

Quoted 38 lines · Catalin Marinas

Quoted 36 lines · Ard Biesheuvel

Quoted 23 lines · Ard Biesheuvel

Quoted 22 lines · Geert Uytterhoeven

arch/arm64/mm/init.c static void __init free_unused_memmap(void) · line 507

Quoted 7 lines · Ard Biesheuvel

arch/arm64/mm/init.c +1 -0

@@ -504,6 +504,8 @@ static void __init free_unused_memmap(void)
  */
 void __init mem_init(void)
 {
+       BUILD_BUG_ON(!is_power_of_2(sizeof(struct page)));

This check is triggering for me.

If CONFIG_MEMCG=n, sizeof(struct page) = 56.

If CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y, this is mitigated by the explicit alignment:

    #ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE
    #define _struct_page_alignment  __aligned(2 * sizeof(unsigned long))
    #else
    #define _struct_page_alignment
    #endif

Quoted 7 lines

Quoted 5 lines

    struct page { ... } _struct_page_alignment;

However, HAVE_ALIGNED_STRUCT_PAGE is selected only if SLUB, while my .config is using SLAB.

Thanks for the report. I will look into this.

OK, so we can obviously fix this easily by setting CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y unconditionally instead of only 'if SLUB'. The question is whether that is likely to lead to any performance regressions.

Quoted 4 lines

I'm not sure I understand. The mem_init() bug triggers if sizeof(struct page) is not a power of 2. HAVE_ALIGNED_STRUCT_PAGE forces the page alignment to 16 bytes but, for example, a 48-byte structure is not a power of 2.

True, but looking at include/linux/mm_types.h, I don't see how that would happen.

Quoted 39 lines · Ard Biesheuvel

Quoted 38 lines · Catalin Marinas

Quoted 36 lines · Ard Biesheuvel

Quoted 23 lines · Ard Biesheuvel

Quoted 22 lines · Geert Uytterhoeven

arch/arm64/mm/init.c static void __init free_unused_memmap(void) · line 507

Quoted 7 lines · Ard Biesheuvel

arch/arm64/mm/init.c +1 -0

@@ -504,6 +504,8 @@ static void __init free_unused_memmap(void)
  */
 void __init mem_init(void)
 {
+       BUILD_BUG_ON(!is_power_of_2(sizeof(struct page)));

This check is triggering for me.

If CONFIG_MEMCG=n, sizeof(struct page) = 56.

If CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y, this is mitigated by the explicit alignment:

    #ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE
    #define _struct_page_alignment  __aligned(2 * sizeof(unsigned long))
    #else
    #define _struct_page_alignment
    #endif

Quoted 7 lines

Quoted 5 lines

    struct page { ... } _struct_page_alignment;

However, HAVE_ALIGNED_STRUCT_PAGE is selected only if SLUB, while my .config is using SLAB.

Thanks for the report. I will look into this.

OK, so we can obviously fix this easily by setting CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y unconditionally instead of only 'if SLUB'. The question is whether that is likely to lead to any performance regressions.

Quoted 7 lines

Quoted 4 lines

I'm not sure I understand. The mem_init() bug triggers if sizeof(struct page) is not a power of 2. HAVE_ALIGNED_STRUCT_PAGE forces the page alignment to 16 bytes but, for example, a 48-byte structure is not a power of 2.

True, but looking at include/linux/mm_types.h, I don't see how that would happen.

Not with 48 and probably won't ever go beyond 64 for "production" builds. But say someone wants to experiment with some debug data in struct page and adds a long. The structure (I think 64 now with MEMCG=y) becomes 72, force-aligned to 80. That triggers the build-bug.

Anyway, I don't mind the forced alignment, only that the build-bug you added has a different requirement than what HAVE_ALIGNED_STRUCT_PAGE provides (power of 2 vs 16-byte aligned).

AFAICT, VMEMMAP_START is a compile-time value, PAGE_OFFSET is also compile-time, so can we not revert to the original virt_to_page and page_to_virt macros? They'd not be as efficient but it may not matter much (and if the struct size is a power of 2, the compiler should change the division/multiplication by shifts).

Tidy up the way the top of the kernel VA space is organized, by mirroring the 256 MB region we have below the vmalloc space, and populating it top down with the PCI I/O space, some guard regions, and the fixmap region. The latter region is itself populated top down, and today only covers about 4 MB, and so 224 MB is ample, and no guard region is therefore required.

The resulting layout is identical between 48-bit/4k and 52-bit/64k configurations.

Signed-off-by Ard Biesheuvel <[email protected]>

This series reorganizes the kernel VA space slightly so that 52-bit VA configurations can use more virtual address space, i.e., the usable linear address space almost doubles, from 2^51 to 2^52-2^47.

Patch #1 merges the physvirt_offset and memstart_addr variables, both of which represent translations between the physical address space and the linear region, and there is no need for having both. This fixes a bug too, as the two values were not properly kept in sync when booting with KASLR enabled.

Patch #2 updates the definitions for the boundaries of the linear space, so that 52-bit VA builds use all available space for the linear region.

Patches #3 and #4 simplify the way the vmemmap region is configured, and unify the 48-bit/4k and 52-bit/64k layouts.

Changes since v1:
- drop incorrect use of untagged_addr() from #2
- add patches #3 and #4

Not boot tested yet on a 52-bit VA capable system.

Cc Steve Capper <[email protected]>

Cc Anshuman Khandual <[email protected]>

Ard Biesheuvel (4):
arm64: mm: use single quantity to represent the PA to VA translation
arm64: mm: extend linear region for 52-bit VA configurations
arm64: mm: make vmemmap region a projection of the linear region
arm64: mm: tidy up top of kernel VA space

 Documentation/arm64/kasan-offsets.sh |  3 +-
 Documentation/arm64/memory.rst       | 45 +++++++++-----------
 arch/arm64/Kconfig                   | 20 ++++-----
 arch/arm64/include/asm/memory.h      | 35 +++++++--------
 arch/arm64/include/asm/pgtable.h     |  6 +--
 arch/arm64/mm/init.c                 | 34 ++++++---------
 6 files changed, 63 insertions(+), 80 deletions(-)

--
2.17.1

This series reorganizes the kernel VA space slightly so that 52-bit VA configurations can use more virtual address space, i.e., the usable linear address space almost doubles, from 2^51 to 2^52-2^47.

Patch #1 merges the physvirt_offset and memstart_addr variables, both of which represent translations between the physical address space and the linear region, and there is no need for having both. This fixes a bug too, as the two values were not properly kept in sync when booting with KASLR enabled.

[...]

This series reorganizes the kernel VA space slightly so that 52-bit VA configurations can use more virtual address space, i.e., the usable linear address space almost doubles, from 2^51 to 2^52-2^47.

Patch #1 merges the physvirt_offset and memstart_addr variables, both of which represent translations between the physical address space and the linear region, and there is no need for having both. This fixes a bug too, as the two values were not properly kept in sync when booting with KASLR enabled.

[...]