Skip to contentSkip to conversation

x86_64: increase stack size for KASAN_EXTRA

PATCH Superseded Qian Cai 3 participants 5 messages Dec 28, 2018 – Jan 11, 2019 x86

lore
Share & tools
lore

Qian Cai

author
#0 lore raw

PATCH 1/1 x86_64: increase stack size for KASAN_EXTRA

If the kernel is configured with KASAN_EXTRA, the stack size is increasted significantly due to enable this option will set "-fstack-reuse" to "none" in GCC [1]. As the results, it could trigger stack overrun quite often with 32k stack size compiled using GCC 8. For example, this reproducer

https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/\
syscalls/madvise/madvise06.c

could trigger a "corrupted stack end detected inside scheduler" very reliably with CONFIG_SCHED_STACK_END_CHECK enabled.

There are just too many functions that could have a large stack with KASAN_EXTRA due to large local variables that have been called over and over again without being able to reuse the stacks. Some noticiable ones are,

size
7648 shrink_page_list
3584 xfs_rmap_convert
3312 migrate_page_move_mapping
3312 dev_ethtool
3200 migrate_misplaced_transhuge_page
3168 copy_process

There are other 49 functions are over 2k in size while compiling kernel with "-Wframe-larger-than=" even with a related minimal config on this machine. Hence, it is too much work to change Makefiles for each object to compile without "-fsanitize-address-use-after-scope" individually.

[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81715#c23

Although there is a patch in GCC 9 to help the situation, GCC 9 probably won't be released in a few months and then it probably take another 6-month to 1-year for all major distros to include it as a default. Hence, the stack usage with KASAN_EXTRA can be revisited again in 2020 when GCC 9 is everywhere. Until then, this patch will help users avoid stack overrun.

This has already been fixed for arm64 for the same reason via 6e8830674ea (arm64: kasan: Increase stack size for KASAN_EXTRA).

---
 arch/x86/include/asm/page_64_types.h | 4 ++++
 1 file changed, 4 insertions(+)
arch/x86/include/asm/page_64_types.h +4 -0
index 8f657286d599..0ce558a8150d 100644
@@ -7,7 +7,11 @@
 #endif
 
 #ifdef CONFIG_KASAN
+#ifdef CONFIG_KASAN_EXTRA
+#define KASAN_STACK_ORDER 2
+#else
 #define KASAN_STACK_ORDER 1
+#endif
 #else
 #define KASAN_STACK_ORDER 0
 #endif
Signature · 1 line
2.17.2 (Apple Git-113)
author
#1 lore raw

PATCH RESEND 1/1 x86_64: increase stack size for KASAN_EXTRA

If the kernel is configured with KASAN_EXTRA, the stack size is increasted significantly due to enable this option will set "-fstack-reuse" to "none" in GCC [1]. As the results, it could trigger stack overrun quite often with 32k stack size compiled using GCC 8. For example, this reproducer

https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/\
syscalls/madvise/madvise06.c

could trigger a "corrupted stack end detected inside scheduler" very reliably with CONFIG_SCHED_STACK_END_CHECK enabled.

There are just too many functions that could have a large stack with KASAN_EXTRA due to large local variables that have been called over and over again without being able to reuse the stacks. Some noticiable ones are,

size
7648 shrink_page_list
3584 xfs_rmap_convert
3312 migrate_page_move_mapping
3312 dev_ethtool
3200 migrate_misplaced_transhuge_page
3168 copy_process

There are other 49 functions are over 2k in size while compiling kernel with "-Wframe-larger-than=" even with a related minimal config on this machine. Hence, it is too much work to change Makefiles for each object to compile without "-fsanitize-address-use-after-scope" individually.

[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81715#c23

Although there is a patch in GCC 9 to help the situation, GCC 9 probably won't be released in a few months and then it probably take another 6-month to 1-year for all major distros to include it as a default. Hence, the stack usage with KASAN_EXTRA can be revisited again in 2020 when GCC 9 is everywhere. Until then, this patch will help users avoid stack overrun.

This has already been fixed for arm64 for the same reason via 6e8830674ea (arm64: kasan: Increase stack size for KASAN_EXTRA).

---
 arch/x86/include/asm/page_64_types.h | 4 ++++
 1 file changed, 4 insertions(+)
arch/x86/include/asm/page_64_types.h +4 -0
index 8f657286d599..0ce558a8150d 100644
@@ -7,7 +7,11 @@
 #endif
 
 #ifdef CONFIG_KASAN
+#ifdef CONFIG_KASAN_EXTRA
+#define KASAN_STACK_ORDER 2
+#else
 #define KASAN_STACK_ORDER 1
+#endif
 #else
 #define KASAN_STACK_ORDER 0
 #endif
Signature · 1 line
2.17.2 (Apple Git-113)
#2 lore raw
Quoted 39 lines · On Wed, 9 Jan 2019 16:52:09 -0500 Qian Cai <[email protected]> wrote:

If the kernel is configured with KASAN_EXTRA, the stack size is increasted significantly due to enable this option will set "-fstack-reuse" to "none" in GCC [1]. As the results, it could trigger stack overrun quite often with 32k stack size compiled using GCC 8. For example, this reproducer

https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/\
syscalls/madvise/madvise06.c

could trigger a "corrupted stack end detected inside scheduler" very reliably with CONFIG_SCHED_STACK_END_CHECK enabled.

There are just too many functions that could have a large stack with KASAN_EXTRA due to large local variables that have been called over and over again without being able to reuse the stacks. Some noticiable ones are,

size
7648 shrink_page_list
3584 xfs_rmap_convert
3312 migrate_page_move_mapping
3312 dev_ethtool
3200 migrate_misplaced_transhuge_page
3168 copy_process

There are other 49 functions are over 2k in size while compiling kernel with "-Wframe-larger-than=" even with a related minimal config on this machine. Hence, it is too much work to change Makefiles for each object to compile without "-fsanitize-address-use-after-scope" individually.

[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81715#c23

Although there is a patch in GCC 9 to help the situation, GCC 9 probably won't be released in a few months and then it probably take another 6-month to 1-year for all major distros to include it as a default. Hence, the stack usage with KASAN_EXTRA can be revisited again in 2020 when GCC 9 is everywhere. Until then, this patch will help users avoid stack overrun.

Quoted 2 lines

This has already been fixed for arm64 for the same reason via 6e8830674ea (arm64: kasan: Increase stack size for KASAN_EXTRA).

Sounds good.

Quoted 4 lines
arch/x86/include/asm/page_64_types.h +0 -0
@@ -7,7 +7,11 @@
 #endif

arch/x86/include/asm/page_64_types.h line 17

Quoted 10 lines
arch/x86/include/asm/page_64_types.h +4 -0
 
 #ifdef CONFIG_KASAN
+#ifdef CONFIG_KASAN_EXTRA
+#define KASAN_STACK_ORDER 2
+#else
 #define KASAN_STACK_ORDER 1
+#endif
 #else
 #define KASAN_STACK_ORDER 0
 #endif

I'm suspecting this logic could be performed in Kconfig, for all architectures. Add a new always-defined CONFIG_KASAN_STACK_ORDER?

author
#3 lore raw
Quoted 5 lines · Andrew Morton
Quoted 5 lines
arch/x86/include/asm/page_64_types.h +0 -0
@@ -7,7 +7,11 @@
 #endif
 
Quoted 13 lines

arch/x86/include/asm/page_64_types.h line 17

Quoted 9 lines
arch/x86/include/asm/page_64_types.h +4 -0
 #ifdef CONFIG_KASAN
+#ifdef CONFIG_KASAN_EXTRA
+#define KASAN_STACK_ORDER 2
+#else
 #define KASAN_STACK_ORDER 1
+#endif
 #else
 #define KASAN_STACK_ORDER 0
 #endif

I'm suspecting this logic could be performed in Kconfig, for all architectures. Add a new always-defined CONFIG_KASAN_STACK_ORDER?

Sounds doable, but KASAN only support x86_64, arm64, s390 and xtensa so far. I am not sure I care about the later two nor I have any way to test them if they suffer the same problem.

I'll keep looking if more arches start to implement KASAN, and then might be a good time to reduce all those code duplication.

#4 lore raw
In reply to Qian Cai
Quoted 52 lines · Qian Cai

If the kernel is configured with KASAN_EXTRA, the stack size is increasted significantly due to enable this option will set "-fstack-reuse" to "none" in GCC [1]. As the results, it could trigger stack overrun quite often with 32k stack size compiled using GCC 8. For example, this reproducer

https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/\
syscalls/madvise/madvise06.c

could trigger a "corrupted stack end detected inside scheduler" very reliably with CONFIG_SCHED_STACK_END_CHECK enabled.

There are just too many functions that could have a large stack with KASAN_EXTRA due to large local variables that have been called over and over again without being able to reuse the stacks. Some noticiable ones are,

size
7648 shrink_page_list
3584 xfs_rmap_convert
3312 migrate_page_move_mapping
3312 dev_ethtool
3200 migrate_misplaced_transhuge_page
3168 copy_process

There are other 49 functions are over 2k in size while compiling kernel with "-Wframe-larger-than=" even with a related minimal config on this machine. Hence, it is too much work to change Makefiles for each object to compile without "-fsanitize-address-use-after-scope" individually.

[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81715#c23

Although there is a patch in GCC 9 to help the situation, GCC 9 probably won't be released in a few months and then it probably take another 6-month to 1-year for all major distros to include it as a default. Hence, the stack usage with KASAN_EXTRA can be revisited again in 2020 when GCC 9 is everywhere. Until then, this patch will help users avoid stack overrun.

This has already been fixed for arm64 for the same reason via 6e8830674ea (arm64: kasan: Increase stack size for KASAN_EXTRA).

---
 arch/x86/include/asm/page_64_types.h | 4 ++++
 1 file changed, 4 insertions(+)
arch/x86/include/asm/page_64_types.h +0 -0
index 8f657286d599..0ce558a8150d 100644

arch/x86/include/asm/page_64_types.h line 11

Quoted 5 lines
arch/x86/include/asm/page_64_types.h +2 -0
@@ -7,7 +7,11 @@
 #endif
 
 #ifdef CONFIG_KASAN
+#ifdef CONFIG_KASAN_EXTRA
+#define KASAN_STACK_ORDER 2

So the kernel stack becomes 4-order page. That's above PAGE_ALLOC_COSTLY_ORDER, so people will start seeing fork() failures with -ENOMEM due to high memory fragmentation. I don't think we can afford such change.

Give that use-after-scope has proven to be almost useless for the kernel, I think we should just remove it entirely.

Quoted 7 lines
arch/x86/include/asm/page_64_types.h +2 -0
+#else
 #define KASAN_STACK_ORDER 1
+#endif
 #else
 #define KASAN_STACK_ORDER 0
 #endif