powerpc/kasan/book3s_64: warn when running with hash MMU
Subsystems (3)
PATCH 1/1 powerpc/kasan/book3s_64: warn when running with hash MMU
kasan is known to crash at boot on book3s_64 with non-radix MMU. As noted in commit 41b7a347bf14 ("powerpc: Book3S 64-bit outline-only KASAN support"):
A kernel with CONFIG_KASAN=y will crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call to kasan_arch_is_ready().
Such crashes look like this:
BUG: Unable to handle kernel data access at 0xc00e00000308b100 Faulting instruction address: 0xc0000000006d0fcc Oops: Kernel access of bad area, sig: 11 [#1] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.0.0-rc5-02183-g3ab165dea2a2 #13 [...regs...] NIP [c0000000006d0fcc] kasan_byte_accessible+0xc/0x20 LR [c0000000006cd9cc] __kasan_check_byte+0x2c/0xa0 Call Trace: [c00000001688f930] [c00000001688f970] 0xc00000001688f970 (unreliable) [c00000001688f970] [c0000000005f6a74] ksize+0x34/0xa0 [c00000001688f9a0] [c0000000024c03a8] __alloc_skb+0xd8/0x2d0 [c00000001688fa00] [c0000000003c48c0] audit_log_start+0x260/0x660 [c00000001688fb30] [c0000000003c50ec] audit_log+0x3c/0x70 [c00000001688fb60] [c00000000404590c] audit_init+0x188/0x1ac [c00000001688fbe0] [c0000000000127e0] do_one_initcall+0xe0/0x610 [c00000001688fcd0] [c00000000400a1f0] kernel_init_freeable+0x4c0/0x574 [c00000001688fda0] [c000000000013450] kernel_init+0x30/0x1d0 [c00000001688fe10] [c00000000000cd54] ret_from_kernel_thread+0x5c/0x64
If you look carefully enough at the full kernel output, you might notice this message, much earlier:
KASAN not enabled as it requires radix!
But the eventual oops does not carry any indication that the real problem was detected early on and is a known limitation.
Change init_book3s_64.c::kasan_init() to emit a warning backtrace and taint the kernel when not running on radix. When the kernel likely oopses later, the 'W' taint flag in the report should help minimize developer time spent trying to understand what's gone wrong.
--- arch/powerpc/mm/kasan/init_book3s_64.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
arch/powerpc/mm/kasan/init_book3s_64.c +1 -3
@@ -56,10 +56,8 @@ void __init kasan_init(void)u64 i;pte_t zero_pte =pfn_pte (virt_to_pfn (kasan_early_shadow_page),PAGE_KERNEL ); -if (!early_radix_enabled ()) { -pr_warn ("KASAN not enabled as it requires radix!" ); +if (WARN (!early_radix_enabled (),"KASAN known broken on HPT" ))return ; - }for_each_mem_range (i, &start, &end)kasan_init_phys_region ((void *)start, (void *)end);
Signature · 1 line
2.37.1
Michael Ellerman
reviewerQuoted 7 lines · Nathan Lynch
kasan is known to crash at boot on
book3s_64with non-radix MMU. As noted in commit41b7a347bf14("powerpc: Book3S 64-bit outline-only KASAN support"):A kernel with
CONFIG_KASAN=ywill crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call tokasan_arch_is_ready().
I guess I thought there was some plan to fix that.
But maybe I'm misremembering. Looking now it's not entirely straight forward with the way the headers are structured. So I guess I'm wrong about that.
Quoted 27 lines
Such crashes look like this:
BUG: Unable to handle kernel data access at 0xc00e00000308b100 Faulting instruction address: 0xc0000000006d0fcc Oops: Kernel access of bad area, sig: 11 [#1] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.0.0-rc5-02183-g3ab165dea2a2 #13 [...regs...] NIP [c0000000006d0fcc] kasan_byte_accessible+0xc/0x20 LR [c0000000006cd9cc] __kasan_check_byte+0x2c/0xa0 Call Trace: [c00000001688f930] [c00000001688f970] 0xc00000001688f970 (unreliable) [c00000001688f970] [c0000000005f6a74] ksize+0x34/0xa0 [c00000001688f9a0] [c0000000024c03a8] __alloc_skb+0xd8/0x2d0 [c00000001688fa00] [c0000000003c48c0] audit_log_start+0x260/0x660 [c00000001688fb30] [c0000000003c50ec] audit_log+0x3c/0x70 [c00000001688fb60] [c00000000404590c] audit_init+0x188/0x1ac [c00000001688fbe0] [c0000000000127e0] do_one_initcall+0xe0/0x610 [c00000001688fcd0] [c00000000400a1f0] kernel_init_freeable+0x4c0/0x574 [c00000001688fda0] [c000000000013450] kernel_init+0x30/0x1d0 [c00000001688fe10] [c00000000000cd54] ret_from_kernel_thread+0x5c/0x64If you look carefully enough at the full kernel output, you might notice this message, much earlier:
KASAN not enabled as it requires radix!
Quoted 7 lines
But the eventual oops does not carry any indication that the real problem was detected early on and is a known limitation.
Change
init_book3s_64.c::kasan_init()to emit a warning backtrace and taint the kernel when not running on radix. When the kernel likely oopses later, the 'W' taint flag in the report should help minimize developer time spent trying to understand what's gone wrong.
Should we just panic() directly?
It's too earily in kasan_init(), nothing gets to the console.
But we have kasan_late_init() already, the patch below seems to work and gives us:
Booting Linux via __start() @ 0x0000000000400000 ...
[ 0.000000][ T0] hash-mmu: Page sizes from device-tree: [ 0.000000][ T0] hash-mmu: base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0 [ 0.000000][ T0] hash-mmu: base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1 [ 0.000000][ T0] Using 1TB segments [ 0.000000][ T0] hash-mmu: Initializing hash mmu with SLB [ 0.000000][ T0] Linux version 6.0.0-rc2-00225-g8b68240ff7d7-dirty (michael@alpine1-p1) (gcc (Ubuntu 11.2.0-7ubuntu2) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.37) #403 SMP Thu Oct 6 14:53:58 AEDT 2022 [ 0.000000][ T0] KASAN not enabled as it requires radix! ... [ 0.000000][ T0] mem auto-init: stack:off, heap alloc:off, heap free:off [ 0.000000][ T0] stackdepot hash table entries: 1048576 (order: 7, 8388608 bytes, linear) [ 0.000000][ T0] Kernel panic - not syncing: KASAN enabled kernel can't boot with Hash MMU! [ 0.000000][ T0] CPU: 0 PID: 0 Comm: swapper Not tainted 6.0.0-rc2-00225-g8b68240ff7d7-dirty #403 [ 0.000000][ T0] Hardware name: IBM pSeries (emulated by qemu) POWER8 (raw) 0x4d0200 0xf000004 of:SLOF,HEAD pSeries [ 0.000000][ T0] Call Trace: [ 0.000000][ T0] [c0000000031efd20] [c000000000bf5450] dump_stack_lvl+0x74/0xa8 (unreliable) [ 0.000000][ T0] [c0000000031efd60] [c00000000017d160] panic+0x1b0/0x480 [ 0.000000][ T0] [c0000000031efe00] [c00000000201e028] kasan_late_init+0x30/0x34 [ 0.000000][ T0] [c0000000031efe60] [c000000002018c4c] mem_init+0xc8/0xec [ 0.000000][ T0] [c0000000031efe90] [c000000002004538] mm_init+0x84/0xbc [ 0.000000][ T0] [c0000000031eff00] [c0000000020047a0] start_kernel+0x1cc/0x4bc [ 0.000000][ T0] [c0000000031eff90] [c00000000000d998] start_here_common+0x1c/0x84
cheers
arch/powerpc/mm/kasan/init_book3s_64.c +5 -1
@@ -101,4 +101,8 @@ void __init kasan_init(void)void __init kasan_early_init (void ) { } -void __init kasan_late_init (void ) { } +void __init kasan_late_init (void ) +{ +if (!radix_enabled ()) +panic ("KASAN enabled kernel can't boot with Hash MMU!" ); +}
+ KASAN list
Le 06/10/2022 à 06:10, Michael Ellerman a écrit :
Quoted 10 lines
Quoted 7 lines · Nathan Lynch
kasan is known to crash at boot on
book3s_64with non-radix MMU. As noted in commit41b7a347bf14("powerpc: Book3S 64-bit outline-only KASAN support"):A kernel with
CONFIG_KASAN=ywill crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call tokasan_arch_is_ready().I guess I thought there was some plan to fix that.
I was thinking the same.
Do we have a list of the said entry points to the generic code that are lacking a call to kasan_arch_is_ready() ?
Typically, the BUG dump below shows that kasan_byte_accessible() is lacking the check. It should be straight forward to add kasan_arch_is_ready() check to kasan_byte_accessible(), shouldn't it ?
Quoted 16 lines
But maybe I'm misremembering. Looking now it's not entirely straight forward with the way the headers are structured. So I guess I'm wrong about that.
Quoted 11 lines
Such crashes look like this:
BUG: Unable to handle kernel data access at 0xc00e00000308b100 Faulting instruction address: 0xc0000000006d0fcc Oops: Kernel access of bad area, sig: 11 [#1] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.0.0-rc5-02183-g3ab165dea2a2 #13 [...regs...] NIP [c0000000006d0fcc] kasan_byte_accessible+0xc/0x20 LR [c0000000006cd9cc] __kasan_check_byte+0x2c/0xa0 Call Trace:
...
Quoted 7 lines
Quoted 5 lines
Change
init_book3s_64.c::kasan_init()to emit a warning backtrace and taint the kernel when not running on radix. When the kernel likely oopses later, the 'W' taint flag in the report should help minimize developer time spent trying to understand what's gone wrong.Should we just
panic()directly?
But then you loose any sight that the problem is in kasan_byte_accessible() and have to be fixed there.
Christophe
Michael Ellerman
reviewerQuoted 14 lines · Christophe Leroy
+ KASAN list
Le 06/10/2022 à 06:10, Michael Ellerman a écrit :
Quoted 10 lines
Quoted 7 lines · Nathan Lynch
kasan is known to crash at boot on
book3s_64with non-radix MMU. As noted in commit41b7a347bf14("powerpc: Book3S 64-bit outline-only KASAN support"):A kernel with
CONFIG_KASAN=ywill crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call tokasan_arch_is_ready().I guess I thought there was some plan to fix that.
Quoted 8 lines
I was thinking the same.
Do we have a list of the said entry points to the generic code that are lacking a call to
kasan_arch_is_ready()?Typically, the BUG dump below shows that
kasan_byte_accessible()is lacking the check. It should be straight forward to addkasan_arch_is_ready()check tokasan_byte_accessible(), shouldn't it ?
Yes :)
And one other spot, but the patch below boots OK for me. I'll leave it running for a while just in case there's a path I've missed.
cheers
mm/kasan/common.c +3 -0
@@ -377,6 +377,9 @@ bool __kasan_slab_free(struct kmem_cache *cache, void *object,static inline bool ____kasan_kfree_large (void *ptr,unsigned long ip) { +if (!kasan_arch_is_ready ()) +return false; +if (ptr !=page_address (virt_to_head_page (ptr))) {kasan_report_invalid_free (ptr, ip,KASAN_REPORT_INVALID_FREE );return true;
mm/kasan/generic.c +6 -1
@@ -191,7 +191,12 @@ bool kasan_check_range(unsigned long addr, size_t size, bool write,bool kasan_byte_accessible (const void *addr) { -s8 shadow_byte =READ_ONCE (*(s8 *)kasan_mem_to_shadow (addr)); +s8 shadow_byte; + +if (!kasan_arch_is_ready ()) +return true; + + shadow_byte =READ_ONCE (*(s8 *)kasan_mem_to_shadow (addr));return shadow_byte >=0 && shadow_byte <KASAN_GRANULE_SIZE ; }
Quoted 20 lines · Michael Ellerman
Quoted 19 lines · Christophe Leroy
+ KASAN list
Le 06/10/2022 à 06:10, Michael Ellerman a écrit :
Quoted 10 lines
Quoted 7 lines · Nathan Lynch
kasan is known to crash at boot on
book3s_64with non-radix MMU. As noted in commit41b7a347bf14("powerpc: Book3S 64-bit outline-only KASAN support"):A kernel with
CONFIG_KASAN=ywill crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call tokasan_arch_is_ready().I guess I thought there was some plan to fix that.
I was thinking the same.
Do we have a list of the said entry points to the generic code that are lacking a call to
kasan_arch_is_ready()?
Quoted 8 lines
Quoted 3 lines
Typically, the BUG dump below shows that
kasan_byte_accessible()is lacking the check. It should be straight forward to addkasan_arch_is_ready()check tokasan_byte_accessible(), shouldn't it ?Yes :)
And one other spot, but the patch below boots OK for me. I'll leave it running for a while just in case there's a path I've missed.
It works for me too, thanks (p8 pseries qemu).
This avoids the boot-time oops, but kasan remains unimplemented for hash mmu. Raising the question: with the trivial crashes addressed, is the current message ('KASAN not enabled as it requires radix!') sufficient to notify developers (such as me, a week ago) who mean to use kasan on a book3s platform, unaware that it's radix-only? Would a WARN or something more prominent still be justified?
I guess people will figure it out as soon as they think to search the kernel log for 'KASAN'...
Le 10/10/2022 à 16:10, Nathan Lynch a écrit :
Quoted 39 lines
Quoted 20 lines · Michael Ellerman
Quoted 19 lines · Christophe Leroy
+ KASAN list
Le 06/10/2022 à 06:10, Michael Ellerman a écrit :
Quoted 10 lines
Quoted 7 lines · Nathan Lynch
kasan is known to crash at boot on
book3s_64with non-radix MMU. As noted in commit41b7a347bf14("powerpc: Book3S 64-bit outline-only KASAN support"):A kernel with
CONFIG_KASAN=ywill crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call tokasan_arch_is_ready().I guess I thought there was some plan to fix that.
I was thinking the same.
Do we have a list of the said entry points to the generic code that are lacking a call to
kasan_arch_is_ready()?Quoted 8 lines
Quoted 3 lines
Typically, the BUG dump below shows that
kasan_byte_accessible()is lacking the check. It should be straight forward to addkasan_arch_is_ready()check tokasan_byte_accessible(), shouldn't it ?Yes :)
And one other spot, but the patch below boots OK for me. I'll leave it running for a while just in case there's a path I've missed.
It works for me too, thanks (p8 pseries qemu).
This avoids the boot-time oops, but kasan remains unimplemented for hash mmu. Raising the question: with the trivial crashes addressed, is the current message ('KASAN not enabled as it requires radix!') sufficient to notify developers (such as me, a week ago) who mean to use kasan on a book3s platform, unaware that it's radix-only? Would a WARN or something more prominent still be justified?
Quoted 2 lines
I guess people will figure it out as soon as they think to search the kernel log for 'KASAN'...
I don't think the big hammer WARN would be justified. WARN is supposed to be used only with unexpected conditions.
KASAN not working with hash-MMU is expected. A pr_warn() should be enough.
Someone who has a kernel with KASAN built in but who is not interested by KASAN and who is booting it one a HASH-MMU will be terrified by a WARN.
Michael Ellerman
reviewerQuoted 25 lines · Nathan Lynch
Quoted 24 lines · Michael Ellerman
Quoted 22 lines · Christophe Leroy
+ KASAN list
Le 06/10/2022 à 06:10, Michael Ellerman a écrit :
Quoted 10 lines
Quoted 7 lines · Nathan Lynch
kasan is known to crash at boot on
book3s_64with non-radix MMU. As noted in commit41b7a347bf14("powerpc: Book3S 64-bit outline-only KASAN support"):A kernel with
CONFIG_KASAN=ywill crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call tokasan_arch_is_ready().I guess I thought there was some plan to fix that.
I was thinking the same.
Do we have a list of the said entry points to the generic code that are lacking a call to
kasan_arch_is_ready()?Typically, the BUG dump below shows that
kasan_byte_accessible()is lacking the check. It should be straight forward to addkasan_arch_is_ready()check tokasan_byte_accessible(), shouldn't it ?
Quoted 6 lines
Quoted 4 lines
Yes :)
And one other spot, but the patch below boots OK for me. I'll leave it running for a while just in case there's a path I've missed.
It works for me too, thanks (p8 pseries qemu).
It works but I still see the kasan shadow getting mapped, which we would ideally avoid.
From PTDUMP:
---[ kasan shadow mem start ]---
0xc00f000000000000-0xc00f00000006ffff 0x00000000045e0000 448K r w pte valid present dirty accessed 0xc00f3ffffffe0000-0xc00f3fffffffffff 0x0000000004d80000 128K r w pte valid present dirty accessed
I haven't worked out how those are getting mapped.
Quoted 9 lines
This avoids the boot-time oops, but kasan remains unimplemented for hash mmu. Raising the question: with the trivial crashes addressed, is the current message ('KASAN not enabled as it requires radix!') sufficient to notify developers (such as me, a week ago) who mean to use kasan on a book3s platform, unaware that it's radix-only? Would a WARN or something more prominent still be justified?
I guess people will figure it out as soon as they think to search the kernel log for 'KASAN'...
Yeah, I think a warning is a bit too strong. I think that's more likely to lead to bug reports than anything :)
cheers
Le 11/10/2022 à 12:00, Michael Ellerman a écrit :
Quoted 36 lines
Quoted 25 lines · Nathan Lynch
Quoted 24 lines · Michael Ellerman
Quoted 22 lines · Christophe Leroy
+ KASAN list
Le 06/10/2022 à 06:10, Michael Ellerman a écrit :
Quoted 10 lines
Quoted 7 lines · Nathan Lynch
kasan is known to crash at boot on
book3s_64with non-radix MMU. As noted in commit41b7a347bf14("powerpc: Book3S 64-bit outline-only KASAN support"):A kernel with
CONFIG_KASAN=ywill crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call tokasan_arch_is_ready().I guess I thought there was some plan to fix that.
I was thinking the same.
Do we have a list of the said entry points to the generic code that are lacking a call to
kasan_arch_is_ready()?Typically, the BUG dump below shows that
kasan_byte_accessible()is lacking the check. It should be straight forward to addkasan_arch_is_ready()check tokasan_byte_accessible(), shouldn't it ?Quoted 6 lines
Quoted 4 lines
Yes :)
And one other spot, but the patch below boots OK for me. I'll leave it running for a while just in case there's a path I've missed.
It works for me too, thanks (p8 pseries qemu).
It works but I still see the kasan shadow getting mapped, which we would ideally avoid.
Quoted 7 lines
From PTDUMP:
---[ kasan shadow mem start ]---
0xc00f000000000000-0xc00f00000006ffff 0x00000000045e0000 448K r w pte valid present dirty accessed 0xc00f3ffffffe0000-0xc00f3fffffffffff 0x0000000004d80000 128K r w pte valid present dirty accessedI haven't worked out how those are getting mapped.
kasan_populate_vmalloc() maybe ?
Christophe
Le 11/10/2022 à 12:25, Christophe Leroy a écrit :
Quoted 48 lines
Le 11/10/2022 à 12:00, Michael Ellerman a écrit :
Quoted 43 lines
Quoted 25 lines · Nathan Lynch
Quoted 24 lines · Michael Ellerman
Quoted 22 lines · Christophe Leroy
+ KASAN list
Le 06/10/2022 à 06:10, Michael Ellerman a écrit :
Quoted 10 lines
Quoted 7 lines · Nathan Lynch
kasan is known to crash at boot on
book3s_64with non-radix MMU. As noted in commit41b7a347bf14("powerpc: Book3S 64-bit outline-only KASAN support"):A kernel with
CONFIG_KASAN=ywill crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call tokasan_arch_is_ready().I guess I thought there was some plan to fix that.
I was thinking the same.
Do we have a list of the said entry points to the generic code that are lacking a call to
kasan_arch_is_ready()?Typically, the BUG dump below shows that
kasan_byte_accessible()is lacking the check. It should be straight forward to addkasan_arch_is_ready()check tokasan_byte_accessible(), shouldn't it ?Quoted 6 lines
Quoted 4 lines
Yes :)
And one other spot, but the patch below boots OK for me. I'll leave it running for a while just in case there's a path I've missed.
It works for me too, thanks (p8 pseries qemu).
It works but I still see the kasan shadow getting mapped, which we would ideally avoid.
From PTDUMP:
---[ kasan shadow mem start ]---
0xc00f000000000000-0xc00f00000006ffff 0x00000000045e0000 448K r w pte valid present dirty accessed 0xc00f3ffffffe0000-0xc00f3fffffffffff 0x0000000004d80000 128K r w pte valid present dirty accessedI haven't worked out how those are getting mapped.
Alternative patch proposed at https://patchwork.ozlabs.org/project/linuxppc-dev/patch/150768c55722311699fdcf8f5379e8256749f47d.1674716617.git.christophe.leroy@csgroup.eu/
Christophe