Skip to contentSkip to conversation

powerpc/kasan/book3s_64: warn when running with hash MMU

PATCH In review Nathan Lynch 3 participants 9 messages Oct 4, 2022 – Jan 26, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 powerpc/kasan/book3s_64: warn when running with hash MMU

kasan is known to crash at boot on book3s_64 with non-radix MMU. As noted in commit 41b7a347bf14 ("powerpc: Book3S 64-bit outline-only KASAN support"):

A kernel with CONFIG_KASAN=y will crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call to kasan_arch_is_ready().

Such crashes look like this:

  BUG: Unable to handle kernel data access at 0xc00e00000308b100
  Faulting instruction address: 0xc0000000006d0fcc
  Oops: Kernel access of bad area, sig: 11 [#1]
  LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
  CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.0.0-rc5-02183-g3ab165dea2a2 #13
  [...regs...]
  NIP [c0000000006d0fcc] kasan_byte_accessible+0xc/0x20
  LR [c0000000006cd9cc] __kasan_check_byte+0x2c/0xa0
  Call Trace:
  [c00000001688f930] [c00000001688f970] 0xc00000001688f970 (unreliable)
  [c00000001688f970] [c0000000005f6a74] ksize+0x34/0xa0
  [c00000001688f9a0] [c0000000024c03a8] __alloc_skb+0xd8/0x2d0
  [c00000001688fa00] [c0000000003c48c0] audit_log_start+0x260/0x660
  [c00000001688fb30] [c0000000003c50ec] audit_log+0x3c/0x70
  [c00000001688fb60] [c00000000404590c] audit_init+0x188/0x1ac
  [c00000001688fbe0] [c0000000000127e0] do_one_initcall+0xe0/0x610
  [c00000001688fcd0] [c00000000400a1f0] kernel_init_freeable+0x4c0/0x574
  [c00000001688fda0] [c000000000013450] kernel_init+0x30/0x1d0
  [c00000001688fe10] [c00000000000cd54] ret_from_kernel_thread+0x5c/0x64

If you look carefully enough at the full kernel output, you might notice this message, much earlier:

KASAN not enabled as it requires radix!

But the eventual oops does not carry any indication that the real problem was detected early on and is a known limitation.

Change init_book3s_64.c::kasan_init() to emit a warning backtrace and taint the kernel when not running on radix. When the kernel likely oopses later, the 'W' taint flag in the report should help minimize developer time spent trying to understand what's gone wrong.

---
 arch/powerpc/mm/kasan/init_book3s_64.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
arch/powerpc/mm/kasan/init_book3s_64.c +1 -3
index 9300d641cf9a..5d9894d7fb97 100644
@@ -56,10 +56,8 @@ void __init kasan_init(void)
 	u64 i;
 	pte_t zero_pte = pfn_pte(virt_to_pfn(kasan_early_shadow_page), PAGE_KERNEL);
 
-	if (!early_radix_enabled()) {
-		pr_warn("KASAN not enabled as it requires radix!");
+	if (WARN(!early_radix_enabled(), "KASAN known broken on HPT"))
 		return;
-	}
 
 	for_each_mem_range(i, &start, &end)
 		kasan_init_phys_region((void *)start, (void *)end);
Signature · 1 line
2.37.1
#1 lore raw
Quoted 7 lines · Nathan Lynch

kasan is known to crash at boot on book3s_64 with non-radix MMU. As noted in commit 41b7a347bf14 ("powerpc: Book3S 64-bit outline-only KASAN support"):

A kernel with CONFIG_KASAN=y will crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call to kasan_arch_is_ready().

I guess I thought there was some plan to fix that.

But maybe I'm misremembering. Looking now it's not entirely straight forward with the way the headers are structured. So I guess I'm wrong about that.

Quoted 27 lines

Such crashes look like this:

  BUG: Unable to handle kernel data access at 0xc00e00000308b100
  Faulting instruction address: 0xc0000000006d0fcc
  Oops: Kernel access of bad area, sig: 11 [#1]
  LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
  CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.0.0-rc5-02183-g3ab165dea2a2 #13
  [...regs...]
  NIP [c0000000006d0fcc] kasan_byte_accessible+0xc/0x20
  LR [c0000000006cd9cc] __kasan_check_byte+0x2c/0xa0
  Call Trace:
  [c00000001688f930] [c00000001688f970] 0xc00000001688f970 (unreliable)
  [c00000001688f970] [c0000000005f6a74] ksize+0x34/0xa0
  [c00000001688f9a0] [c0000000024c03a8] __alloc_skb+0xd8/0x2d0
  [c00000001688fa00] [c0000000003c48c0] audit_log_start+0x260/0x660
  [c00000001688fb30] [c0000000003c50ec] audit_log+0x3c/0x70
  [c00000001688fb60] [c00000000404590c] audit_init+0x188/0x1ac
  [c00000001688fbe0] [c0000000000127e0] do_one_initcall+0xe0/0x610
  [c00000001688fcd0] [c00000000400a1f0] kernel_init_freeable+0x4c0/0x574
  [c00000001688fda0] [c000000000013450] kernel_init+0x30/0x1d0
  [c00000001688fe10] [c00000000000cd54] ret_from_kernel_thread+0x5c/0x64

If you look carefully enough at the full kernel output, you might notice this message, much earlier:

KASAN not enabled as it requires radix!

Quoted 7 lines

But the eventual oops does not carry any indication that the real problem was detected early on and is a known limitation.

Change init_book3s_64.c::kasan_init() to emit a warning backtrace and taint the kernel when not running on radix. When the kernel likely oopses later, the 'W' taint flag in the report should help minimize developer time spent trying to understand what's gone wrong.

Should we just panic() directly?

It's too earily in kasan_init(), nothing gets to the console.

But we have kasan_late_init() already, the patch below seems to work and gives us:

Booting Linux via __start() @ 0x0000000000400000 ...

  [    0.000000][    T0] hash-mmu: Page sizes from device-tree:
  [    0.000000][    T0] hash-mmu: base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0
  [    0.000000][    T0] hash-mmu: base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1
  [    0.000000][    T0] Using 1TB segments
  [    0.000000][    T0] hash-mmu: Initializing hash mmu with SLB
  [    0.000000][    T0] Linux version 6.0.0-rc2-00225-g8b68240ff7d7-dirty (michael@alpine1-p1) (gcc (Ubuntu 11.2.0-7ubuntu2) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.37) #403 SMP Thu Oct 6 14:53:58 AEDT 2022
  [    0.000000][    T0] KASAN not enabled as it requires radix!
  ...
  [    0.000000][    T0] mem auto-init: stack:off, heap alloc:off, heap free:off
  [    0.000000][    T0] stackdepot hash table entries: 1048576 (order: 7, 8388608 bytes, linear)
  [    0.000000][    T0] Kernel panic - not syncing: KASAN enabled kernel can't boot with Hash MMU!
  [    0.000000][    T0] CPU: 0 PID: 0 Comm: swapper Not tainted 6.0.0-rc2-00225-g8b68240ff7d7-dirty #403
  [    0.000000][    T0] Hardware name: IBM pSeries (emulated by qemu) POWER8 (raw) 0x4d0200 0xf000004 of:SLOF,HEAD pSeries
  [    0.000000][    T0] Call Trace:
  [    0.000000][    T0] [c0000000031efd20] [c000000000bf5450] dump_stack_lvl+0x74/0xa8 (unreliable)
  [    0.000000][    T0] [c0000000031efd60] [c00000000017d160] panic+0x1b0/0x480
  [    0.000000][    T0] [c0000000031efe00] [c00000000201e028] kasan_late_init+0x30/0x34
  [    0.000000][    T0] [c0000000031efe60] [c000000002018c4c] mem_init+0xc8/0xec
  [    0.000000][    T0] [c0000000031efe90] [c000000002004538] mm_init+0x84/0xbc
  [    0.000000][    T0] [c0000000031eff00] [c0000000020047a0] start_kernel+0x1cc/0x4bc
  [    0.000000][    T0] [c0000000031eff90] [c00000000000d998] start_here_common+0x1c/0x84

cheers

arch/powerpc/mm/kasan/init_book3s_64.c +5 -1
index 9300d641cf9a..cc1b4ee8be53 100644
@@ -101,4 +101,8 @@ void __init kasan_init(void)
 
 void __init kasan_early_init(void) { }
 
-void __init kasan_late_init(void) { }
+void __init kasan_late_init(void)
+{
+	if (!radix_enabled())
+		panic("KASAN enabled kernel can't boot with Hash MMU!");
+}
#2 lore raw

+ KASAN list

Le 06/10/2022 à 06:10, Michael Ellerman a écrit :

Quoted 10 lines
Quoted 7 lines · Nathan Lynch

kasan is known to crash at boot on book3s_64 with non-radix MMU. As noted in commit 41b7a347bf14 ("powerpc: Book3S 64-bit outline-only KASAN support"):

A kernel with CONFIG_KASAN=y will crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call to kasan_arch_is_ready().

I guess I thought there was some plan to fix that.

I was thinking the same.

Do we have a list of the said entry points to the generic code that are lacking a call to kasan_arch_is_ready() ?

Typically, the BUG dump below shows that kasan_byte_accessible() is lacking the check. It should be straight forward to add kasan_arch_is_ready() check to kasan_byte_accessible(), shouldn't it ?

Quoted 16 lines

But maybe I'm misremembering. Looking now it's not entirely straight forward with the way the headers are structured. So I guess I'm wrong about that.

Quoted 11 lines

Such crashes look like this:

   BUG: Unable to handle kernel data access at 0xc00e00000308b100
   Faulting instruction address: 0xc0000000006d0fcc
   Oops: Kernel access of bad area, sig: 11 [#1]
   LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
   CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.0.0-rc5-02183-g3ab165dea2a2 #13
   [...regs...]
   NIP [c0000000006d0fcc] kasan_byte_accessible+0xc/0x20
   LR [c0000000006cd9cc] __kasan_check_byte+0x2c/0xa0
   Call Trace:

...

Quoted 7 lines
Quoted 5 lines

Change init_book3s_64.c::kasan_init() to emit a warning backtrace and taint the kernel when not running on radix. When the kernel likely oopses later, the 'W' taint flag in the report should help minimize developer time spent trying to understand what's gone wrong.

Should we just panic() directly?

But then you loose any sight that the problem is in kasan_byte_accessible() and have to be fixed there.

Christophe

#3 lore raw
Quoted 14 lines · Christophe Leroy

+ KASAN list

Le 06/10/2022 à 06:10, Michael Ellerman a écrit :

Quoted 10 lines
Quoted 7 lines · Nathan Lynch

kasan is known to crash at boot on book3s_64 with non-radix MMU. As noted in commit 41b7a347bf14 ("powerpc: Book3S 64-bit outline-only KASAN support"):

A kernel with CONFIG_KASAN=y will crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call to kasan_arch_is_ready().

I guess I thought there was some plan to fix that.

Quoted 8 lines

I was thinking the same.

Do we have a list of the said entry points to the generic code that are lacking a call to kasan_arch_is_ready() ?

Typically, the BUG dump below shows that kasan_byte_accessible() is lacking the check. It should be straight forward to add kasan_arch_is_ready() check to kasan_byte_accessible(), shouldn't it ?

Yes :)

And one other spot, but the patch below boots OK for me. I'll leave it running for a while just in case there's a path I've missed.

cheers

mm/kasan/common.c +3 -0
index 69f583855c8b..5def0118f2cd 100644
@@ -377,6 +377,9 @@ bool __kasan_slab_free(struct kmem_cache *cache, void *object,
 
 static inline bool ____kasan_kfree_large(void *ptr, unsigned long ip)
 {
+	if (!kasan_arch_is_ready())
+		return false;
+
 	if (ptr != page_address(virt_to_head_page(ptr))) {
 		kasan_report_invalid_free(ptr, ip, KASAN_REPORT_INVALID_FREE);
 		return true;
mm/kasan/generic.c +6 -1
index 437fcc7e77cf..017d3c69e3b3 100644
@@ -191,7 +191,12 @@ bool kasan_check_range(unsigned long addr, size_t size, bool write,
 
 bool kasan_byte_accessible(const void *addr)
 {
-	s8 shadow_byte = READ_ONCE(*(s8 *)kasan_mem_to_shadow(addr));
+	s8 shadow_byte;
+
+	if (!kasan_arch_is_ready())
+		return true;
+
+	shadow_byte = READ_ONCE(*(s8 *)kasan_mem_to_shadow(addr));
 
 	return shadow_byte >= 0 && shadow_byte < KASAN_GRANULE_SIZE;
 }
#4 lore raw
Quoted 20 lines · Michael Ellerman
Quoted 19 lines · Christophe Leroy

+ KASAN list

Le 06/10/2022 à 06:10, Michael Ellerman a écrit :

Quoted 10 lines
Quoted 7 lines · Nathan Lynch

kasan is known to crash at boot on book3s_64 with non-radix MMU. As noted in commit 41b7a347bf14 ("powerpc: Book3S 64-bit outline-only KASAN support"):

A kernel with CONFIG_KASAN=y will crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call to kasan_arch_is_ready().

I guess I thought there was some plan to fix that.

I was thinking the same.

Do we have a list of the said entry points to the generic code that are lacking a call to kasan_arch_is_ready() ?

Quoted 8 lines
Quoted 3 lines

Typically, the BUG dump below shows that kasan_byte_accessible() is lacking the check. It should be straight forward to add kasan_arch_is_ready() check to kasan_byte_accessible(), shouldn't it ?

Yes :)

And one other spot, but the patch below boots OK for me. I'll leave it running for a while just in case there's a path I've missed.

It works for me too, thanks (p8 pseries qemu).

This avoids the boot-time oops, but kasan remains unimplemented for hash mmu. Raising the question: with the trivial crashes addressed, is the current message ('KASAN not enabled as it requires radix!') sufficient to notify developers (such as me, a week ago) who mean to use kasan on a book3s platform, unaware that it's radix-only? Would a WARN or something more prominent still be justified?

I guess people will figure it out as soon as they think to search the kernel log for 'KASAN'...

#5 lore raw

Le 10/10/2022 à 16:10, Nathan Lynch a écrit :

Quoted 39 lines
Quoted 20 lines · Michael Ellerman
Quoted 19 lines · Christophe Leroy

+ KASAN list

Le 06/10/2022 à 06:10, Michael Ellerman a écrit :

Quoted 10 lines
Quoted 7 lines · Nathan Lynch

kasan is known to crash at boot on book3s_64 with non-radix MMU. As noted in commit 41b7a347bf14 ("powerpc: Book3S 64-bit outline-only KASAN support"):

A kernel with CONFIG_KASAN=y will crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call to kasan_arch_is_ready().

I guess I thought there was some plan to fix that.

I was thinking the same.

Do we have a list of the said entry points to the generic code that are lacking a call to kasan_arch_is_ready() ?

Quoted 8 lines
Quoted 3 lines

Typically, the BUG dump below shows that kasan_byte_accessible() is lacking the check. It should be straight forward to add kasan_arch_is_ready() check to kasan_byte_accessible(), shouldn't it ?

Yes :)

And one other spot, but the patch below boots OK for me. I'll leave it running for a while just in case there's a path I've missed.

It works for me too, thanks (p8 pseries qemu).

This avoids the boot-time oops, but kasan remains unimplemented for hash mmu. Raising the question: with the trivial crashes addressed, is the current message ('KASAN not enabled as it requires radix!') sufficient to notify developers (such as me, a week ago) who mean to use kasan on a book3s platform, unaware that it's radix-only? Would a WARN or something more prominent still be justified?

Quoted 2 lines

I guess people will figure it out as soon as they think to search the kernel log for 'KASAN'...

I don't think the big hammer WARN would be justified. WARN is supposed to be used only with unexpected conditions.

KASAN not working with hash-MMU is expected. A pr_warn() should be enough.

Someone who has a kernel with KASAN built in but who is not interested by KASAN and who is booting it one a HASH-MMU will be terrified by a WARN.

#6 lore raw
In reply to Nathan Lynch
Quoted 25 lines · Nathan Lynch
Quoted 24 lines · Michael Ellerman
Quoted 22 lines · Christophe Leroy

+ KASAN list

Le 06/10/2022 à 06:10, Michael Ellerman a écrit :

Quoted 10 lines
Quoted 7 lines · Nathan Lynch

kasan is known to crash at boot on book3s_64 with non-radix MMU. As noted in commit 41b7a347bf14 ("powerpc: Book3S 64-bit outline-only KASAN support"):

A kernel with CONFIG_KASAN=y will crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call to kasan_arch_is_ready().

I guess I thought there was some plan to fix that.

I was thinking the same.

Do we have a list of the said entry points to the generic code that are lacking a call to kasan_arch_is_ready() ?

Typically, the BUG dump below shows that kasan_byte_accessible() is lacking the check. It should be straight forward to add kasan_arch_is_ready() check to kasan_byte_accessible(), shouldn't it ?

Quoted 6 lines
Quoted 4 lines

Yes :)

And one other spot, but the patch below boots OK for me. I'll leave it running for a while just in case there's a path I've missed.

It works for me too, thanks (p8 pseries qemu).

It works but I still see the kasan shadow getting mapped, which we would ideally avoid.

From PTDUMP:

---[ kasan shadow mem start ]---

0xc00f000000000000-0xc00f00000006ffff  0x00000000045e0000       448K         r  w       pte  valid  present        dirty  accessed
0xc00f3ffffffe0000-0xc00f3fffffffffff  0x0000000004d80000       128K         r  w       pte  valid  present        dirty  accessed

I haven't worked out how those are getting mapped.

Quoted 9 lines

This avoids the boot-time oops, but kasan remains unimplemented for hash mmu. Raising the question: with the trivial crashes addressed, is the current message ('KASAN not enabled as it requires radix!') sufficient to notify developers (such as me, a week ago) who mean to use kasan on a book3s platform, unaware that it's radix-only? Would a WARN or something more prominent still be justified?

I guess people will figure it out as soon as they think to search the kernel log for 'KASAN'...

Yeah, I think a warning is a bit too strong. I think that's more likely to lead to bug reports than anything :)

cheers

#7 lore raw

Le 11/10/2022 à 12:00, Michael Ellerman a écrit :

Quoted 36 lines
Quoted 25 lines · Nathan Lynch
Quoted 24 lines · Michael Ellerman
Quoted 22 lines · Christophe Leroy

+ KASAN list

Le 06/10/2022 à 06:10, Michael Ellerman a écrit :

Quoted 10 lines
Quoted 7 lines · Nathan Lynch

kasan is known to crash at boot on book3s_64 with non-radix MMU. As noted in commit 41b7a347bf14 ("powerpc: Book3S 64-bit outline-only KASAN support"):

A kernel with CONFIG_KASAN=y will crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call to kasan_arch_is_ready().

I guess I thought there was some plan to fix that.

I was thinking the same.

Do we have a list of the said entry points to the generic code that are lacking a call to kasan_arch_is_ready() ?

Typically, the BUG dump below shows that kasan_byte_accessible() is lacking the check. It should be straight forward to add kasan_arch_is_ready() check to kasan_byte_accessible(), shouldn't it ?

Quoted 6 lines
Quoted 4 lines

Yes :)

And one other spot, but the patch below boots OK for me. I'll leave it running for a while just in case there's a path I've missed.

It works for me too, thanks (p8 pseries qemu).

It works but I still see the kasan shadow getting mapped, which we would ideally avoid.

Quoted 7 lines

From PTDUMP:

---[ kasan shadow mem start ]---

0xc00f000000000000-0xc00f00000006ffff  0x00000000045e0000       448K         r  w       pte  valid  present        dirty  accessed
0xc00f3ffffffe0000-0xc00f3fffffffffff  0x0000000004d80000       128K         r  w       pte  valid  present        dirty  accessed

I haven't worked out how those are getting mapped.

kasan_populate_vmalloc() maybe ?

Christophe

#8 lore raw

Le 11/10/2022 à 12:25, Christophe Leroy a écrit :

Quoted 48 lines

Le 11/10/2022 à 12:00, Michael Ellerman a écrit :

Quoted 43 lines
Quoted 25 lines · Nathan Lynch
Quoted 24 lines · Michael Ellerman
Quoted 22 lines · Christophe Leroy

+ KASAN list

Le 06/10/2022 à 06:10, Michael Ellerman a écrit :

Quoted 10 lines
Quoted 7 lines · Nathan Lynch

kasan is known to crash at boot on book3s_64 with non-radix MMU. As noted in commit 41b7a347bf14 ("powerpc: Book3S 64-bit outline-only KASAN support"):

A kernel with CONFIG_KASAN=y will crash during boot on a machine using HPT translation because not all the entry points to the generic KASAN code are protected with a call to kasan_arch_is_ready().

I guess I thought there was some plan to fix that.

I was thinking the same.

Do we have a list of the said entry points to the generic code that are lacking a call to kasan_arch_is_ready() ?

Typically, the BUG dump below shows that kasan_byte_accessible() is lacking the check. It should be straight forward to add kasan_arch_is_ready() check to kasan_byte_accessible(), shouldn't it ?

Quoted 6 lines
Quoted 4 lines

Yes :)

And one other spot, but the patch below boots OK for me. I'll leave it running for a while just in case there's a path I've missed.

It works for me too, thanks (p8 pseries qemu).

It works but I still see the kasan shadow getting mapped, which we would ideally avoid.

From PTDUMP:

---[ kasan shadow mem start ]---

0xc00f000000000000-0xc00f00000006ffff  0x00000000045e0000       448K         r  w       pte  valid  present        dirty  accessed
0xc00f3ffffffe0000-0xc00f3fffffffffff  0x0000000004d80000       128K         r  w       pte  valid  present        dirty  accessed

I haven't worked out how those are getting mapped.

Alternative patch proposed at https://patchwork.ozlabs.org/project/linuxppc-dev/patch/150768c55722311699fdcf8f5379e8256749f47d.1674716617.git.christophe.leroy@csgroup.eu/

Christophe