Skip to contentSkip to conversation

x86/mm/KASLR: Do not adapt size of the direct mapping section for SGI UV system

PATCH Superseded Baoquan He 3 participants 10 messages May 20, 2017 – Aug 31, 2017 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore

Baoquan He

author
#0 lore raw

v2 cover x86/mm/KASLR: Do not adapt size of the direct mapping section for SGI UV system

This is v2 post.

This patchset is trying to fix a bug that SGI UV system casually hang during boot with KASLR enabled. The root cause is that mm KASLR adapts size of the direct mapping section only based on the system RAM size. Then later when map SGI UV MMIOH region into the direct mapping during rest_init() invocation, it might go beyond of the directing mapping section and step into VMALLOC or VMEMMAP area, then BUG_ON triggered.

The fix is adding a helper function is_early_uv_system to check UV system earlier, then call the helper function in kernel_randomize_memory() to check if it's a SGI UV system, if yes, we keep the size of direct mapping section to be 64TB just as nokslr.

With this fix, SGI UV system can have 64TB direct mapping size always, and the starting address of direct mapping/vmalloc/vmemmap and the padding between them can still be randomized to enhance the system security.

v1->v2:
1. Mike suggested making is_early_uv_system() an inline function and be put in include/asm/uv/uv.h so that they can adjust them easier in the future.

2. Split the v1 code into uv part and mm KASLR part as Mike suggested.

Baoquan He (2):
x86/UV: Introduce a helper function to check UV system at earlier
stage
x86/mm/KASLR: Do not adapt the size of the direct mapping section for
SGI UV system

 arch/x86/include/asm/uv/uv.h | 6 ++++++
 arch/x86/mm/kaslr.c          | 3 ++-
 2 files changed, 8 insertions(+), 1 deletion(-)
Signature · 1 line
2.5.5
author
#1 lore raw

PATCH v2 1/2 x86/UV: Introduce a helper function to check UV system at earlier stage

The SGI BIOS adds UVsystab, and only systems running SGI BIOS (and now HPE Hawks2) will have UVsystab. And UVsystab is detected in efi_init() which is at very early stage. So introduce a new helper function is_early_uv_system() for later usage.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Russ Anderson <[email protected]>
Cc Dimitri Sivanich <[email protected]>
Cc Mike Travis <[email protected]>
Cc Frank Ramsay <[email protected]>
---
 arch/x86/include/asm/uv/uv.h | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/include/asm/uv/uv.h +6 -0
index 6686820..159f698 100644
@@ -19,6 +19,11 @@ extern const struct cpumask *uv_flush_tlb_others(const struct cpumask *cpumask,
 						 unsigned long start,
 						 unsigned long end,
 						 unsigned int cpu);
+#include <linux/efi.h>
+static inline int is_early_uv_system(void)
+{
+	return !((efi.uv_systab == EFI_INVALID_TABLE_ADDR) || !efi.uv_systab);
+}
 
 #else	/* X86_UV */
 
@@ -31,6 +36,7 @@ static inline const struct cpumask *
 uv_flush_tlb_others(const struct cpumask *cpumask, struct mm_struct *mm,
 		    unsigned long start, unsigned long end, unsigned int cpu)
 { return cpumask; }
+static inline int is_early_uv_system(void)	{ return 0; }
 
 #endif	/* X86_UV */
 
Signature · 1 line
2.5.5
Acked-by on v2 1/2
#2 lore raw

Re: [PATCH v2 1/2] x86/UV: Introduce a helper function to check UV system at earlier stage

Quoted 43 lines · Baoquan He

The SGI BIOS adds UVsystab, and only systems running SGI BIOS (and now HPE Hawks2) will have UVsystab. And UVsystab is detected in efi_init() which is at very early stage. So introduce a new helper function is_early_uv_system() for later usage.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Russ Anderson <[email protected]>
Cc Dimitri Sivanich <[email protected]>
Cc Mike Travis <[email protected]>
Cc Frank Ramsay <[email protected]>
---
  arch/x86/include/asm/uv/uv.h | 6 ++++++
  1 file changed, 6 insertions(+)
arch/x86/include/asm/uv/uv.h +6 -0
index 6686820..159f698 100644
@@ -19,6 +19,11 @@ extern const struct cpumask *uv_flush_tlb_others(const struct cpumask *cpumask,
  						 unsigned long start,
  						 unsigned long end,
  						 unsigned int cpu);
+#include <linux/efi.h>
+static inline int is_early_uv_system(void)
+{
+	return !((efi.uv_systab == EFI_INVALID_TABLE_ADDR) || !efi.uv_systab);
+}
 
  #else	/* X86_UV */
 
@@ -31,6 +36,7 @@ static inline const struct cpumask *
  uv_flush_tlb_others(const struct cpumask *cpumask, struct mm_struct *mm,
  		    unsigned long start, unsigned long end, unsigned int cpu)
  { return cpumask; }
+static inline int is_early_uv_system(void)	{ return 0; }
 
  #endif	/* X86_UV */
 
author
#3 lore raw

PATCH v2 2/2 x86/mm/KASLR: Do not adapt size of the direct mapping section for SGI UV system

In reply to Baoquan He

On SGI UV system, kernel casually hang with kaslr enabled.

The back trace is:

kernel BUG at arch/x86/mm/init_64.c:311!
invalid opcode: 0000 [#1] SMP
[...]

RIP: 0010:__init_extra_mapping+0x188/0x196
[...]
Call Trace:
 init_extra_mapping_uc+0x13/0x15
 map_high+0x67/0x75
 map_mmioh_high_uv3+0x20a/0x219
 uv_system_init_hub+0x12d9/0x1496
 uv_system_init+0x27/0x29
 native_smp_prepare_cpus+0x28d/0x2d8
 kernel_init_freeable+0xdd/0x253
 ? rest_init+0x80/0x80
 kernel_init+0xe/0x110
 ret_from_fork+0x2c/0x40

The root cause is that SGI UV system needs map its MMIOH region to direct mapping section and the mapping happens in rest_init(). However mm KASLR is done in kernel_randomize_memory() which is much earlier than MMIOH mapping of SGI UV and doesn't count in the MMIOH regions. When kaslr disabled, there are 64TB space for system RAM to do direct mapping. Both system RAM and SGI UV MMIOH region share this 64TB space. With kaslr enabled, mm KASLR only reserves the actual size of system RAM plus 10TB for direct mapping usage. Then later MMIOH mapping of SGI UV could go beyond the upper bound of direct mapping section to step into VMALLOC or VMEMMAP area. Then the BUG_ON() in __init_extra_mapping() will be triggered.

E.g on the SGI UV3 machine where this bug is reported , there are two MMIOH regions:

[    1.519001] UV: Map MMIOH0_HI 0xffc00000000 - 0x100000000000
[    1.523001] UV: Map MMIOH1_HI 0x100000000000 - 0x200000000000

They are [16TB-16G, 16TB) and [16TB, 32TB). On this machine, 512G ram are spread out to 1TB regions. Then above two SGI MMIOH regions also will be mapped into the direct mapping section.

To fix it, we need check if it's SGI UV system by calling is_early_uv_system() in kernel_randomize_memory(). If yes, do not adapt the size of the direct mapping section. Do it now.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Thomas Garnier <[email protected]>
Cc Kees Cook <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Masahiro Yamada <[email protected]>
---
 arch/x86/mm/kaslr.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/mm/kaslr.c +2 -1
index aed2064..20b0456 100644
@@ -27,6 +27,7 @@
 #include <asm/pgtable.h>
 #include <asm/setup.h>
 #include <asm/kaslr.h>
+#include <asm/uv/uv.h>
 
 #include "mm_internal.h"
 
@@ -123,7 +124,7 @@ void __init kernel_randomize_memory(void)
 		CONFIG_RANDOMIZE_MEMORY_PHYSICAL_PADDING;
 
 	/* Adapt phyiscal memory region size based on available memory */
-	if (memory_tb < kaslr_regions[0].size_tb)
+	if (memory_tb < kaslr_regions[0].size_tb && !is_early_uv_system())
 		kaslr_regions[0].size_tb = memory_tb;
 
 	/* Calculate entropy available between regions */
Signature · 1 line
2.5.5
#4 lore raw
Quoted 73 lines · Baoquan He

On SGI UV system, kernel casually hang with kaslr enabled.

The back trace is:

kernel BUG at arch/x86/mm/init_64.c:311!
invalid opcode: 0000 [#1] SMP
[...]

RIP: 0010:__init_extra_mapping+0x188/0x196
[...]
Call Trace:
 init_extra_mapping_uc+0x13/0x15
 map_high+0x67/0x75
 map_mmioh_high_uv3+0x20a/0x219
 uv_system_init_hub+0x12d9/0x1496
 uv_system_init+0x27/0x29
 native_smp_prepare_cpus+0x28d/0x2d8
 kernel_init_freeable+0xdd/0x253
 ? rest_init+0x80/0x80
 kernel_init+0xe/0x110
 ret_from_fork+0x2c/0x40

The root cause is that SGI UV system needs map its MMIOH region to direct mapping section and the mapping happens in rest_init(). However mm KASLR is done in kernel_randomize_memory() which is much earlier than MMIOH mapping of SGI UV and doesn't count in the MMIOH regions. When kaslr disabled, there are 64TB space for system RAM to do direct mapping. Both system RAM and SGI UV MMIOH region share this 64TB space. With kaslr enabled, mm KASLR only reserves the actual size of system RAM plus 10TB for direct mapping usage. Then later MMIOH mapping of SGI UV could go beyond the upper bound of direct mapping section to step into VMALLOC or VMEMMAP area. Then the BUG_ON() in __init_extra_mapping() will be triggered.

E.g on the SGI UV3 machine where this bug is reported , there are two MMIOH regions:

[    1.519001] UV: Map MMIOH0_HI 0xffc00000000 - 0x100000000000
[    1.523001] UV: Map MMIOH1_HI 0x100000000000 - 0x200000000000

They are [16TB-16G, 16TB) and [16TB, 32TB). On this machine, 512G ram are spread out to 1TB regions. Then above two SGI MMIOH regions also will be mapped into the direct mapping section.

To fix it, we need check if it's SGI UV system by calling is_early_uv_system() in kernel_randomize_memory(). If yes, do not adapt the size of the direct mapping section. Do it now.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Thomas Garnier <[email protected]>
Cc Kees Cook <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Masahiro Yamada <[email protected]>
---
 arch/x86/mm/kaslr.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/mm/kaslr.c +1 -0
index aed2064..20b0456 100644
@@ -27,6 +27,7 @@
 #include <asm/pgtable.h>
 #include <asm/setup.h>
 #include <asm/kaslr.h>
+#include <asm/uv/uv.h>
 
 #include "mm_internal.h"
 

arch/x86/mm/kaslr.c void __init kernel_randomize_memory(void) · line 127

Quoted 5 lines
arch/x86/mm/kaslr.c +1 -1
@@ -123,7 +124,7 @@ void __init kernel_randomize_memory(void)
                CONFIG_RANDOMIZE_MEMORY_PHYSICAL_PADDING;
 
        /* Adapt phyiscal memory region size based on available memory */
-       if (memory_tb < kaslr_regions[0].size_tb)
+       if (memory_tb < kaslr_regions[0].size_tb && !is_early_uv_system())

Given your example, any way we could just restrict memory_tb to be 32TB? Or different configurations will result in different mappings?

Quoted 6 lines
arch/x86/mm/kaslr.c +0 -1
                kaslr_regions[0].size_tb = memory_tb;
 
        /* Calculate entropy available between regions */
--

2.5.5

Signature · 1 line
Thomas
author
#5 lore raw
Quoted 19 lines · Thomas Garnier
Quoted 18 lines · Baoquan He
 arch/x86/mm/kaslr.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/mm/kaslr.c +1 -0
index aed2064..20b0456 100644
@@ -27,6 +27,7 @@
 #include <asm/pgtable.h>
 #include <asm/setup.h>
 #include <asm/kaslr.h>
+#include <asm/uv/uv.h>
 
 #include "mm_internal.h"
 
@@ -123,7 +124,7 @@ void __init kernel_randomize_memory(void)
                CONFIG_RANDOMIZE_MEMORY_PHYSICAL_PADDING;
 
Quoted 6 lines

arch/x86/mm/kaslr.c void __init kernel_randomize_memory(void) · line 127

Quoted 3 lines
arch/x86/mm/kaslr.c +1 -1
        /* Adapt phyiscal memory region size based on available memory */
-       if (memory_tb < kaslr_regions[0].size_tb)
+       if (memory_tb < kaslr_regions[0].size_tb && !is_early_uv_system())

Given your example, any way we could just restrict memory_tb to be 32TB? Or different configurations will result in different mappings?

Thanks for looking into this, Thomas!

For that machine where I used to reproduce the bug and test, 32TB memory need be mapped to the direct mapping region. I am not sure if SGI UV system has larger MMIOH region now or in the future in different machine. If they have machine owning MMIOH region bigger than 64TB, then it's a problem SGI UV need fix because that will break system whether kaslr enabled or not.

Hi Mike, Russ and Frank,

About Thomas's question, could you help answer it? Could other SGI UV system has MMIOH region bigger than 32TB?

Thanks
Baoquan

Quoted 4 lines
Quoted 4 lines
arch/x86/mm/kaslr.c +0 -0
                kaslr_regions[0].size_tb = memory_tb;
 
        /* Calculate entropy available between regions */
author
#6 lore raw

Sorry, forget 'To' Mike, Russ and Frank

Quoted 48 lines · Baoquan He
Quoted 19 lines · Thomas Garnier
Quoted 18 lines · Baoquan He
 arch/x86/mm/kaslr.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/mm/kaslr.c +1 -0
index aed2064..20b0456 100644
@@ -27,6 +27,7 @@
 #include <asm/pgtable.h>
 #include <asm/setup.h>
 #include <asm/kaslr.h>
+#include <asm/uv/uv.h>
 
 #include "mm_internal.h"
 
@@ -123,7 +124,7 @@ void __init kernel_randomize_memory(void)
                CONFIG_RANDOMIZE_MEMORY_PHYSICAL_PADDING;
 
Quoted 6 lines

arch/x86/mm/kaslr.c void __init kernel_randomize_memory(void) · line 127

Quoted 3 lines
arch/x86/mm/kaslr.c +1 -1
        /* Adapt phyiscal memory region size based on available memory */
-       if (memory_tb < kaslr_regions[0].size_tb)
+       if (memory_tb < kaslr_regions[0].size_tb && !is_early_uv_system())

Given your example, any way we could just restrict memory_tb to be 32TB? Or different configurations will result in different mappings?

Thanks for looking into this, Thomas!

For that machine where I used to reproduce the bug and test, 32TB memory need be mapped to the direct mapping region. I am not sure if SGI UV system has larger MMIOH region now or in the future in different machine. If they have machine owning MMIOH region bigger than 64TB, then it's a problem SGI UV need fix because that will break system whether kaslr enabled or not.

Hi Mike, Russ and Frank,

About Thomas's question, could you help answer it? Could other SGI UV system has MMIOH region bigger than 32TB?

Thanks
Baoquan

Quoted 4 lines
Quoted 4 lines
arch/x86/mm/kaslr.c +0 -0
                kaslr_regions[0].size_tb = memory_tb;
 
        /* Calculate entropy available between regions */

A message that is not in the archive (sent off-list or not yet received): [email protected]

Reviewed-by
#8 lore raw
Quoted 52 lines · Mike Travis

On 5/21/2017 4:17 PM, Baoquan He wrote:

Sorry, forget 'To' Mike, Russ and Frank

On 05/22/17 at 07:14am, Baoquan He wrote:

On 05/21/17 at 01:38pm, Thomas Garnier wrote:

On Sat, May 20, 2017 at 5:02 AM, Baoquan He <[email protected]> wrote:

 arch/x86/mm/kaslr.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/mm/kaslr.c +2 -1
index aed2064..20b0456 100644
@@ -27,6 +27,7 @@
 #include <asm/pgtable.h>
 #include <asm/setup.h>
 #include <asm/kaslr.h>
+#include <asm/uv/uv.h>
 
 #include "mm_internal.h"
 
@@ -123,7 +124,7 @@ void __init kernel_randomize_memory(void)
                CONFIG_RANDOMIZE_MEMORY_PHYSICAL_PADDING;
 
        /* Adapt phyiscal memory region size based on available memory */
-       if (memory_tb < kaslr_regions[0].size_tb)
+       if (memory_tb < kaslr_regions[0].size_tb && !is_early_uv_system())
 

Given your example, any way we could just restrict memory_tb to be 32TB? Or different configurations will result in different mappings?

Thanks for looking into this, Thomas!

For that machine where I used to reproduce the bug and test, 32TB memory need be mapped to the direct mapping region. I am not sure if SGI UV system has larger MMIOH region now or in the future in different machine. If they have machine owning MMIOH region bigger than 64TB, then it's a problem SGI UV need fix because that will break system whether kaslr enabled or not.

Hi Mike, Russ and Frank,

About Thomas's question, could you help answer it? Could other SGI UV system has MMIOH region bigger than 32TB?

Quoted 5 lines

While the region is much smaller it can occupy address space > 32TB, up to 64TB - <MMIOH size>.
On a system with 64TB, part of the address space is taken from RAM to accommodate this region.
This has been true since UV1.

I see, it would be better to know the different places to tailor the memory_tb accordingly. I understand that might be difficult to do and I rather have KASLR memory randomization working for now.

Reviewed-by [email protected]
Quoted 9 lines

Thanks
Baoquan

                kaslr_regions[0].size_tb = memory_tb;

        /* Calculate entropy available between regions */
Signature · 1 line
Thomas
author
#9 lore raw
In reply to Baoquan He

Hi all,

Since this is a blocker bug found on SGI UV system and only happen on SGI UV system, and expert from HPE SGI UV dev team, Mike Travis sent private mail to me saying that I can add his Acked-by to this patchset if repost, I will repost with updated patch log. Currently without this fix, SGI UV system will panic during boot with very high possibility.

Quoted 84 lines · Baoquan He

On SGI UV system, kernel casually hang with kaslr enabled.

The back trace is:

kernel BUG at arch/x86/mm/init_64.c:311!
invalid opcode: 0000 [#1] SMP
[...]

RIP: 0010:__init_extra_mapping+0x188/0x196
[...]
Call Trace:
 init_extra_mapping_uc+0x13/0x15
 map_high+0x67/0x75
 map_mmioh_high_uv3+0x20a/0x219
 uv_system_init_hub+0x12d9/0x1496
 uv_system_init+0x27/0x29
 native_smp_prepare_cpus+0x28d/0x2d8
 kernel_init_freeable+0xdd/0x253
 ? rest_init+0x80/0x80
 kernel_init+0xe/0x110
 ret_from_fork+0x2c/0x40

The root cause is that SGI UV system needs map its MMIOH region to direct mapping section and the mapping happens in rest_init(). However mm KASLR is done in kernel_randomize_memory() which is much earlier than MMIOH mapping of SGI UV and doesn't count in the MMIOH regions. When kaslr disabled, there are 64TB space for system RAM to do direct mapping. Both system RAM and SGI UV MMIOH region share this 64TB space. With kaslr enabled, mm KASLR only reserves the actual size of system RAM plus 10TB for direct mapping usage. Then later MMIOH mapping of SGI UV could go beyond the upper bound of direct mapping section to step into VMALLOC or VMEMMAP area. Then the BUG_ON() in __init_extra_mapping() will be triggered.

E.g on the SGI UV3 machine where this bug is reported , there are two MMIOH regions:

[    1.519001] UV: Map MMIOH0_HI 0xffc00000000 - 0x100000000000
[    1.523001] UV: Map MMIOH1_HI 0x100000000000 - 0x200000000000

They are [16TB-16G, 16TB) and [16TB, 32TB). On this machine, 512G ram are spread out to 1TB regions. Then above two SGI MMIOH regions also will be mapped into the direct mapping section.

To fix it, we need check if it's SGI UV system by calling is_early_uv_system() in kernel_randomize_memory(). If yes, do not adapt the size of the direct mapping section. Do it now.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Thomas Garnier <[email protected]>
Cc Kees Cook <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Masahiro Yamada <[email protected]>
---
 arch/x86/mm/kaslr.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/mm/kaslr.c +2 -1
index aed2064..20b0456 100644
@@ -27,6 +27,7 @@
 #include <asm/pgtable.h>
 #include <asm/setup.h>
 #include <asm/kaslr.h>
+#include <asm/uv/uv.h>
 
 #include "mm_internal.h"
 
@@ -123,7 +124,7 @@ void __init kernel_randomize_memory(void)
 		CONFIG_RANDOMIZE_MEMORY_PHYSICAL_PADDING;
 
 	/* Adapt phyiscal memory region size based on available memory */
-	if (memory_tb < kaslr_regions[0].size_tb)
+	if (memory_tb < kaslr_regions[0].size_tb && !is_early_uv_system())
 		kaslr_regions[0].size_tb = memory_tb;
 
 	/* Calculate entropy available between regions */
Signature · 1 line
2.5.5
author
#10 lore raw
In reply to Baoquan He

Hi all,

PING!

Is there any further comment or suggetion about this patchset?

Thanks
Baoquan

Quoted 38 lines · Baoquan He

This is v2 post.

This patchset is trying to fix a bug that SGI UV system casually hang during boot with KASLR enabled. The root cause is that mm KASLR adapts size of the direct mapping section only based on the system RAM size. Then later when map SGI UV MMIOH region into the direct mapping during rest_init() invocation, it might go beyond of the directing mapping section and step into VMALLOC or VMEMMAP area, then BUG_ON triggered.

The fix is adding a helper function is_early_uv_system to check UV system earlier, then call the helper function in kernel_randomize_memory() to check if it's a SGI UV system, if yes, we keep the size of direct mapping section to be 64TB just as nokslr.

With this fix, SGI UV system can have 64TB direct mapping size always, and the starting address of direct mapping/vmalloc/vmemmap and the padding between them can still be randomized to enhance the system security.

v1->v2:
1. Mike suggested making is_early_uv_system() an inline function and be put in include/asm/uv/uv.h so that they can adjust them easier in the future.

2. Split the v1 code into uv part and mm KASLR part as Mike suggested.

Baoquan He (2):
x86/UV: Introduce a helper function to check UV system at earlier
stage
x86/mm/KASLR: Do not adapt the size of the direct mapping section for
SGI UV system

 arch/x86/include/asm/uv/uv.h | 6 ++++++
 arch/x86/mm/kaslr.c          | 3 ++-
 2 files changed, 8 insertions(+), 1 deletion(-)
Signature · 1 line
2.5.5