Skip to contentSkip to conversation

KASAN debug kernel fails to boot at early stage when CONFIG_SMP=y is set (kernel 6.5-rc5, PowerMac G4 3,6)

PATCH New Erhard Furtner 3 participants 34 messages Aug 10, 2023 – Feb 28, 2024 PowerPC

lore
Share & tools
lore
#0 lore raw

I wanted to fire up my PowerMac G4 MDD (Dual CPU) with a KASAN debug build of kernel 6.5-rc5 for testing purposes. But the kernel fails to boot at a very early stage. I only get a white screen reading "done
found display: /pci@f0000000/ATY,AlteracParent@10/ATY,Alterac_B@1, opening..."

When I use the same kernel .config but disable SMP the machine just boots up fine with the KASAN enabled kernel.

Perhaps a similar issue to https://bugzilla.kernel.org/show_bug.cgi?id=216190 ?

Kernel .config and full dmesg attached.

Regards,
Erhard F.

#1 lore raw

Le 11/08/2023 à 01:48, Erhard Furtner a écrit :

Quoted 3 lines

I wanted to fire up my PowerMac G4 MDD (Dual CPU) with a KASAN debug build of kernel 6.5-rc5 for testing purposes. But the kernel fails to boot at a very early stage. I only get a white screen reading "done
found display: /pci@f0000000/ATY,AlteracParent@10/ATY,Alterac_B@1, opening..."

Can you try with CONFIG_PPC_EARLY_DEBUG and see if you get more information on the screen ?

Christophe

Quoted 9 lines

When I use the same kernel .config but disable SMP the machine just boots up fine with the KASAN enabled kernel.

Perhaps a similar issue to https://bugzilla.kernel.org/show_bug.cgi?id=216190 ?

Kernel .config and full dmesg attached.

Regards,
Erhard F.

#2 lore raw
Quoted 7 lines · On Fri, 11 Aug 2023 06:45:14 +0000 Christophe Leroy…

Le 11/08/2023 à 01:48, Erhard Furtner a écrit :

Quoted 3 lines

I wanted to fire up my PowerMac G4 MDD (Dual CPU) with a KASAN debug build of kernel 6.5-rc5 for testing purposes. But the kernel fails to boot at a very early stage. I only get a white screen reading "done
found display: /pci@f0000000/ATY,AlteracParent@10/ATY,Alterac_B@1, opening..."

Can you try with CONFIG_PPC_EARLY_DEBUG and see if you get more information on the screen ?

With CONFIG_PPC_EARLY_DEBUG set booting continues and I get two more lines on a white screen:

[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Total memory = 2048MB; using 4096kB for hash table

Afterwards the G4 freezes.

By chane I found out another interesting thing: This only happens on cold boots.

If I boot the G4 up with another kernel first and boot the SMP KASAN kernel afterwards it just boots up fine too! On further reboots the SMP KASAN kernel keeps booting up normally. Until I turn the machine off - and next time I turn it on the same SMP KASAN kernel fails booting again as described. Strange...

This behaviour however is reproducible. Tried that procedure 15-20 times just to be sure. Hope you can make something out of it.

Attached is a dmesg of a successful SMP KASAM kernel boot attempt when a working kernel was booted first.

Regards,
Erhard F.

#3 lore raw

Le 13/08/2023 à 21:38, Erhard Furtner a écrit :

Quoted 11 lines
Quoted 7 lines · On Fri, 11 Aug 2023 06:45:14 +0000 Christophe Leroy…

Le 11/08/2023 à 01:48, Erhard Furtner a écrit :

Quoted 3 lines

I wanted to fire up my PowerMac G4 MDD (Dual CPU) with a KASAN debug build of kernel 6.5-rc5 for testing purposes. But the kernel fails to boot at a very early stage. I only get a white screen reading "done
found display: /pci@f0000000/ATY,AlteracParent@10/ATY,Alterac_B@1, opening..."

Can you try with CONFIG_PPC_EARLY_DEBUG and see if you get more information on the screen ?

Quoted 6 lines

With CONFIG_PPC_EARLY_DEBUG set booting continues and I get two more lines on a white screen:

[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Total memory = 2048MB; using 4096kB for hash table

Afterwards the G4 freezes.

Interesting. That means we get stuck somewhere around MMU_init()

We know that MMU_init_hw() is called and runs at least until:

	pr_info("Total memory = %lldMB; using %ldkB for hash table\n",
		(unsigned long long)(total_memory >> 20), Hash_size >> 10);

But we never reach the print in setup_kuap() which is itself called by set_kup():

	pr_info("Activating Kernel Userspace Access Protection\n");

Could you try to narrow more the issue by spreading pr_info() at places in the code below and/or the called functions ? Either we never come back from MMU_init_hw(), or one of mapin_ram() btext_unmap() kasan_mmu_init() fails.

So the piece of code we are interested in is located in arch/powerpc/mm/init_32.c and is:

	/* Initialize the MMU hardware */
	if (ppc_md.progress)
		ppc_md.progress("MMU:hw init", 0x300);

==> MMU_init_hw();

	/* Map in all of RAM starting at KERNELBASE */
	if (ppc_md.progress)
		ppc_md.progress("MMU:mapin", 0x301);
	mapin_ram();

	/* Initialize early top-down ioremap allocator */
	ioremap_bot = IOREMAP_TOP;

	if (ppc_md.progress)
		ppc_md.progress("MMU:exit", 0x211);

	/* From now on, btext is no longer BAT mapped if it was at all */
#ifdef CONFIG_BOOTX_TEXT
	btext_unmap();

#endif

	kasan_mmu_init();

==> setup_kup();

Christophe

Quoted 11 lines

By chane I found out another interesting thing: This only happens on cold boots.

If I boot the G4 up with another kernel first and boot the SMP KASAN kernel afterwards it just boots up fine too! On further reboots the SMP KASAN kernel keeps booting up normally. Until I turn the machine off - and next time I turn it on the same SMP KASAN kernel fails booting again as described. Strange...

This behaviour however is reproducible. Tried that procedure 15-20 times just to be sure. Hope you can make something out of it.

Attached is a dmesg of a successful SMP KASAM kernel boot attempt when a working kernel was booted first.

Regards,
Erhard F.

#4 lore raw
Quoted 39 lines · On Mon, 14 Aug 2023 09:40:44 +0000 Christophe Leroy…

Interesting. That means we get stuck somewhere around MMU_init()

We know that MMU_init_hw() is called and runs at least until:

	pr_info("Total memory = %lldMB; using %ldkB for hash table\n",
		(unsigned long long)(total_memory >> 20), Hash_size >> 10);

But we never reach the print in setup_kuap() which is itself called by set_kup():

	pr_info("Activating Kernel Userspace Access Protection\n");

Could you try to narrow more the issue by spreading pr_info() at places in the code below and/or the called functions ? Either we never come back from MMU_init_hw(), or one of mapin_ram() btext_unmap() kasan_mmu_init() fails.

So the piece of code we are interested in is located in arch/powerpc/mm/init_32.c and is:

	/* Initialize the MMU hardware */
	if (ppc_md.progress)
		ppc_md.progress("MMU:hw init", 0x300);

==> MMU_init_hw();

	/* Map in all of RAM starting at KERNELBASE */
	if (ppc_md.progress)
		ppc_md.progress("MMU:mapin", 0x301);
	mapin_ram();

	/* Initialize early top-down ioremap allocator */
	ioremap_bot = IOREMAP_TOP;

	if (ppc_md.progress)
		ppc_md.progress("MMU:exit", 0x211);

	/* From now on, btext is no longer BAT mapped if it was at all */
#ifdef CONFIG_BOOTX_TEXT
	btext_unmap();
Quoted 5 lines

#endif

	kasan_mmu_init();

==> setup_kup();

I added a pr_info(); right after MMU_init_hw(); and another one right after setup_kup();.

Output of PPC_EARLY_DEBUG changes so that I get an additional black blank line after

[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Total memory = 2048MB; using 4096kB for hash table

and the freeze afterwards. So it looks like we return from MMU_init_hw() but not from setup_kup().

The dmesg of a warm boot (after first booting with kernel 6.4.10) supports that as it also shows the 1st blank line and the 2nd one just after activating KUEP/KUAP:

[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Total memory = 2048MB; using 4096kB for hash table
[    0.000000] 
[    0.000000] Activating Kernel Userspace Access Protection
[    0.000000] Activating Kernel Userspace Execution Prevention
[    0.000000] 
[    0.000000] Linux version 6.5.0-rc6-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #1 SMP Mon Aug 14 18:05:17 CEST 2023

As the 2nd blank line from pr_info() is just after KUAP, KUEP initialization I thought these might be a problem. But if I deactivate KUAP/KUAP in the kernel .config I still get the freeze sot it must be something else.

Regards,
Erhard

#5 lore raw

PATCH 1/1 Add pr_info() traces for investigation

---
 arch/powerpc/kernel/btext.c     | 2 ++
 arch/powerpc/mm/book3s32/kuap.c | 5 +++++
 arch/powerpc/mm/book3s32/mmu.c  | 3 +++
 arch/powerpc/mm/kasan/init_32.c | 3 +++
 arch/powerpc/mm/pgtable_32.c    | 4 ++++
 5 files changed, 17 insertions(+)
arch/powerpc/kernel/btext.c +2 -0
index 19e46fd623b0..ec989e1011f0 100644
@@ -126,7 +126,9 @@ void __init btext_setup_display(int width, int height, int depth, int pitch,
 
 void __init btext_unmap(void)
 {
+	pr_info("%s:%d\n", __func__, __LINE__);
 	boot_text_mapped = 0;
+	pr_info("%s:%d\n", __func__, __LINE__);
 }
 
 /* Here's a small text engine to use during early boot
arch/powerpc/mm/book3s32/kuap.c +5 -0
index 28676cabb005..856d18c135bf 100644
@@ -20,17 +20,22 @@ EXPORT_SYMBOL(kuap_unlock_all_ool);
 
 void setup_kuap(bool disabled)
 {
+	pr_info("%s:%d\n", __func__, __LINE__);
 	if (!disabled) {
+		pr_info("%s:%d\n", __func__, __LINE__);
 		kuap_lock_all_ool();
 		init_mm.context.sr0 |= SR_KS;
 		current->thread.sr0 |= SR_KS;
 	}
+	pr_info("%s:%d\n", __func__, __LINE__);
 
 	if (smp_processor_id() != boot_cpuid)
 		return;
 
+	pr_info("%s:%d\n", __func__, __LINE__);
 	if (disabled)
 		static_branch_enable(&disable_kuap_key);
 	else
 		pr_info("Activating Kernel Userspace Access Protection\n");
+	pr_info("%s:%d\n", __func__, __LINE__);
 }
arch/powerpc/mm/book3s32/mmu.c +3 -0
index 850783cfa9c7..ab0c19acd3f3 100644
@@ -143,6 +143,7 @@ static unsigned long __init __mmu_mapin_ram(unsigned long base, unsigned long to
 {
 	int idx;
 
+	pr_info("%s:%d %lx %lx\n", __func__, __LINE__, base, top);
 	while ((idx = find_free_bat()) != -1 && base != top) {
 		unsigned int size = bat_block_size(base, top);
 
@@ -151,6 +152,7 @@ static unsigned long __init __mmu_mapin_ram(unsigned long base, unsigned long to
 		setbat(idx, PAGE_OFFSET + base, base, size, PAGE_KERNEL_X);
 		base += size;
 	}
+	pr_info("%s:%d %lx\n", __func__, __LINE__, base);
 
 	return base;
 }
@@ -164,6 +166,7 @@ unsigned long __init mmu_mapin_ram(unsigned long base, unsigned long top)
 	size = roundup_pow_of_two((unsigned long)_einittext - PAGE_OFFSET);
 	setibat(0, PAGE_OFFSET, 0, size, PAGE_KERNEL_X);
 
+	pr_info("%s:%d %lx %lx %lx %lx\n", __func__, __LINE__, base, top, border, size);
 	if (debug_pagealloc_enabled_or_kfence()) {
 		pr_debug_once("Read-Write memory mapped without BATs\n");
 		if (base >= border)
arch/powerpc/mm/kasan/init_32.c +3 -0
index a70828a6d935..a0f82d9a1fa6 100644
@@ -126,12 +126,15 @@ void __init kasan_mmu_init(void)
 {
 	int ret;
 
+	pr_info("%s:%d\n", __func__, __LINE__);
 	if (early_mmu_has_feature(MMU_FTR_HPTE_TABLE)) {
 		ret = kasan_init_shadow_page_tables(KASAN_SHADOW_START, KASAN_SHADOW_END);
+		pr_info("%s:%d %d\n", __func__, __LINE__, ret);
 
 		if (ret)
 			panic("kasan: kasan_init_shadow_page_tables() failed");
 	}
+	pr_info("%s:%d\n", __func__, __LINE__);
 }
 
 void __init kasan_init(void)
arch/powerpc/mm/pgtable_32.c +4 -0
index 5c02fd08d61e..d75084f67d9b 100644
@@ -104,6 +104,7 @@ static void __init __mapin_ram_chunk(unsigned long offset, unsigned long top)
 	phys_addr_t p;
 	bool ktext;
 
+	pr_info("%s:%d %lx %lx\n", __func__, __LINE__, offset, top);
 	s = offset;
 	v = PAGE_OFFSET + s;
 	p = memstart_addr + s;
@@ -113,6 +114,7 @@ static void __init __mapin_ram_chunk(unsigned long offset, unsigned long top)
 		v += PAGE_SIZE;
 		p += PAGE_SIZE;
 	}
+	pr_info("%s:%d\n", __func__, __LINE__);
 }
 
 void __init mapin_ram(void)
@@ -120,6 +122,7 @@ void __init mapin_ram(void)
 	phys_addr_t base, end;
 	u64 i;
 
+	pr_info("%s:%d\n", __func__, __LINE__);
 	for_each_mem_range(i, &base, &end) {
 		phys_addr_t top = min(end, total_lowmem);
 
@@ -128,6 +131,7 @@ void __init mapin_ram(void)
 		base = mmu_mapin_ram(base, top);
 		__mapin_ram_chunk(base, top);
 	}
+	pr_info("%s:%d\n", __func__, __LINE__);
 }
 
 void mark_initmem_nx(void)
Signature · 1 line
2.41.0
#6 lore raw
In reply to Erhard Furtner

Le 14/08/2023 à 19:27, Erhard Furtner a écrit :

Quoted 48 lines
Quoted 39 lines · On Mon, 14 Aug 2023 09:40:44 +0000 Christophe Leroy…

Interesting. That means we get stuck somewhere around MMU_init()

We know that MMU_init_hw() is called and runs at least until:

	pr_info("Total memory = %lldMB; using %ldkB for hash table\n",
		(unsigned long long)(total_memory >> 20), Hash_size >> 10);

But we never reach the print in setup_kuap() which is itself called by set_kup():

	pr_info("Activating Kernel Userspace Access Protection\n");

Could you try to narrow more the issue by spreading pr_info() at places in the code below and/or the called functions ? Either we never come back from MMU_init_hw(), or one of mapin_ram() btext_unmap() kasan_mmu_init() fails.

So the piece of code we are interested in is located in arch/powerpc/mm/init_32.c and is:

	/* Initialize the MMU hardware */
	if (ppc_md.progress)
		ppc_md.progress("MMU:hw init", 0x300);

==> MMU_init_hw();

	/* Map in all of RAM starting at KERNELBASE */
	if (ppc_md.progress)
		ppc_md.progress("MMU:mapin", 0x301);
	mapin_ram();

	/* Initialize early top-down ioremap allocator */
	ioremap_bot = IOREMAP_TOP;

	if (ppc_md.progress)
		ppc_md.progress("MMU:exit", 0x211);

	/* From now on, btext is no longer BAT mapped if it was at all */
#ifdef CONFIG_BOOTX_TEXT
	btext_unmap();
Quoted 5 lines

#endif

	kasan_mmu_init();

==> setup_kup();

Quoted 7 lines

I added a pr_info(); right after MMU_init_hw(); and another one right after setup_kup();.

Output of PPC_EARLY_DEBUG changes so that I get an additional black blank line after

[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Total memory = 2048MB; using 4096kB for hash table

and the freeze afterwards. So it looks like we return from MMU_init_hw() but not from setup_kup().

I just sent a patch with additional pr_info() in order to help locate the issue, please provide output with that patch.

Thanks
Christophe

#7 lore raw
Quoted 5 lines · On Tue, 15 Aug 2023 17:25:13 +0000 Christophe Leroy…

I just sent a patch with additional pr_info() in order to help locate the issue, please provide output with that patch.

Thanks
Christophe

Thanks for your efforts Christophe! With the patch applied I get this output + a blank black line:

[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Total memory = 2048MB; using 4096kB for hash table
[    0.000000] mapin_ram:125
[    0.000000] mmu_mapin_ram:169 0 30000000 1400000 2000000
[    0.000000] __mmu_mapin_ram:146 0 1400000
[    0.000000] __mmu_mapin_ram:155 1400000
[    0.000000] __mmu_mapin_ram:146 1400000 30000000
[    0.000000] __mmu_mapin_ram:155 20000000
[    0.000000] __mapin_ram_chunk:107 20000000 30000000
[    0.000000] __mapin_ram_chunk:117
[    0.000000] mapin_ram:134
[    0.000000] btext_unmap:129

Afterwards the freeze. On a warm boot after booting another kernel the output continues with "[ 0.000000] btext_unmap:131" instead of the black line. Full dmesg of a warm boot attached.

Regards,
Erhard

#8 lore raw

Le 15/08/2023 à 22:01, Erhard Furtner a écrit :

Quoted 10 lines
Quoted 5 lines · On Tue, 15 Aug 2023 17:25:13 +0000 Christophe Leroy…

I just sent a patch with additional pr_info() in order to help locate the issue, please provide output with that patch.

Thanks
Christophe

Thanks for your efforts Christophe! With the patch applied I get this output + a blank black line:

Thanks for your tests and your patience.

Quoted 15 lines
[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Total memory = 2048MB; using 4096kB for hash table
[    0.000000] mapin_ram:125
[    0.000000] mmu_mapin_ram:169 0 30000000 1400000 2000000
[    0.000000] __mmu_mapin_ram:146 0 1400000
[    0.000000] __mmu_mapin_ram:155 1400000
[    0.000000] __mmu_mapin_ram:146 1400000 30000000
[    0.000000] __mmu_mapin_ram:155 20000000
[    0.000000] __mapin_ram_chunk:107 20000000 30000000
[    0.000000] __mapin_ram_chunk:117
[    0.000000] mapin_ram:134
[    0.000000] btext_unmap:129
Quoted 1 line

Afterwards the freeze. On a warm boot after booting another kernel the output continues with "[ 0.000000] btext_unmap:131" instead of the black line. Full dmesg of a warm boot attached.

Not sure this is the reason of the crash. Just that it disables bootx console here and doesn't re-enable any console before the crash.

Could you move the call to btext_unmap() after the call to memblock_set_current_limit(lowmem_end_addr); at the end of function MMU_init()

And redo the test, still with my patch applied for the additional pr_infos.

Thanks
Christophe

Quoted 3 lines

Regards,
Erhard

#9 lore raw
Quoted 8 lines · On Wed, 16 Aug 2023 15:56:26 +0000 Christophe Leroy…

Not sure this is the reason of the crash. Just that it disables bootx console here and doesn't re-enable any console before the crash.

Could you move the call to btext_unmap() after the call to memblock_set_current_limit(lowmem_end_addr); at the end of function MMU_init()

And redo the test, still with my patch applied for the additional pr_infos.

I moved btext_unmap() as you suggested and the order of functions displayed changed accordingly. Now the freeze happens later but still after the 1st call to btext_unmap():

[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Total memory = 2048MB; using 4096kB for hash table
[    0.000000] mapin_ram:125
[    0.000000] mmu_mapin_ram:169 0 30000000 1400000 2000000
[    0.000000] __mmu_mapin_ram:146 0 1400000
[    0.000000] __mmu_mapin_ram:155 1400000
[    0.000000] __mmu_mapin_ram:146 1400000 30000000
[    0.000000] __mmu_mapin_ram:155 20000000
[    0.000000] __mapin_ram_chunk:107 20000000 30000000
[    0.000000] __mapin_ram_chunk:117
[    0.000000] mapin_ram:134
[    0.000000] kasan_mmu_init:129
[    0.000000] kasan_mmu_init:132 0
[    0.000000] kasan_mmu_init:137
[    0.000000] setup_kuap:23
[    0.000000] setup_kuap:25
[    0.000000] setup_kuap:30
[    0.000000] setup_kuap:35
[    0.000000] Activating Kernel Userspace Access Protection
[    0.000000] setup_kuap:40
[    0.000000] Activating Kernel Userspace Execution Prevention
[    0.000000] btext_unmap:129

Regards,
Erhard

#10 lore raw
Quoted 28 lines · Erhard Furtner
Quoted 8 lines · On Wed, 16 Aug 2023 15:56:26 +0000 Christophe Leroy…

Not sure this is the reason of the crash. Just that it disables bootx console here and doesn't re-enable any console before the crash.

Could you move the call to btext_unmap() after the call to memblock_set_current_limit(lowmem_end_addr); at the end of function MMU_init()

And redo the test, still with my patch applied for the additional pr_infos.

I moved btext_unmap() as you suggested and the order of functions displayed changed accordingly. Now the freeze happens later but still after the 1st call to btext_unmap():

[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Total memory = 2048MB; using 4096kB for hash table
[    0.000000] mapin_ram:125
[    0.000000] mmu_mapin_ram:169 0 30000000 1400000 2000000
[    0.000000] __mmu_mapin_ram:146 0 1400000
[    0.000000] __mmu_mapin_ram:155 1400000
[    0.000000] __mmu_mapin_ram:146 1400000 30000000
[    0.000000] __mmu_mapin_ram:155 20000000
[    0.000000] __mapin_ram_chunk:107 20000000 30000000
[    0.000000] __mapin_ram_chunk:117
[    0.000000] mapin_ram:134
[    0.000000] kasan_mmu_init:129
[    0.000000] kasan_mmu_init:132 0
[    0.000000] kasan_mmu_init:137
Quoted 8 lines
[    0.000000] setup_kuap:23
[    0.000000] setup_kuap:25
[    0.000000] setup_kuap:30
[    0.000000] setup_kuap:35
[    0.000000] Activating Kernel Userspace Access Protection
[    0.000000] setup_kuap:40
[    0.000000] Activating Kernel Userspace Execution Prevention
[    0.000000] btext_unmap:129

Try just removing the call to btext_unmap() entirely and see how far it goes?

cheers

#11 lore raw
Quoted 3 lines · On Fri, 18 Aug 2023 09:13:21 +1000 Michael Ellerman <[email protected]> wrote:

Try just removing the call to btext_unmap() entirely and see how far it goes?

cheers

Ok, I commented out btext_unmap() in MMU_init() and the freeze just happens before the "Linux version 6.5.0-rc6-PMacG4-dirty [...]"-line:

[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Total memory = 2048MB; using 4096kB for hash table
[    0.000000] mapin_ram:125
[    0.000000] mmu_mapin_ram:169 0 30000000 1400000 2000000
[    0.000000] __mmu_mapin_ram:146 0 1400000
[    0.000000] __mmu_mapin_ram:155 1400000
[    0.000000] __mmu_mapin_ram:146 1400000 30000000
[    0.000000] __mmu_mapin_ram:155 20000000
[    0.000000] __mapin_ram_chunk:107 20000000 30000000
[    0.000000] __mapin_ram_chunk:117
[    0.000000] mapin_ram:134
[    0.000000] kasan_mmu_init:129
[    0.000000] kasan_mmu_init:132 0
[    0.000000] kasan_mmu_init:137
[    0.000000] setup_kuap:23
[    0.000000] setup_kuap:25
[    0.000000] setup_kuap:30
[    0.000000] setup_kuap:35
[    0.000000] Activating Kernel Userspace Access Protection
[    0.000000] setup_kuap:40
[    0.000000] Activating Kernel Userspace Execution Prevention

I also tried a kernel without KUAP/KUEP getting this:

[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Total memory = 2048MB; using 4096kB for hash table
[    0.000000] mapin_ram:125
[    0.000000] mmu_mapin_ram:169 0 30000000 1400000 2000000
[    0.000000] __mmu_mapin_ram:146 0 1400000
[    0.000000] __mmu_mapin_ram:155 1400000
[    0.000000] __mmu_mapin_ram:146 1400000 30000000
[    0.000000] __mmu_mapin_ram:155 20000000
[    0.000000] __mapin_ram_chunk:107 20000000 30000000
[    0.000000] __mapin_ram_chunk:117
[    0.000000] mapin_ram:134
[    0.000000] kasan_mmu_init:129
[    0.000000] kasan_mmu_init:132 0
[    0.000000] kasan_mmu_init:137

Here too the freeze just happens before the line "Linux version 6.5.0-rc6-PMacG4-dirty [...]" in dmesg.

Attached is (warm boot) dmesg without KUAP/KUEP.

Regards,
Erhard

#12 lore raw

Le 18/08/2023 à 11:16, Erhard Furtner a écrit :

Quoted 48 lines
Quoted 3 lines · On Fri, 18 Aug 2023 09:13:21 +1000 Michael Ellerman <[email protected]> wrote:

Try just removing the call to btext_unmap() entirely and see how far it goes?

cheers

Ok, I commented out btext_unmap() in MMU_init() and the freeze just happens before the "Linux version 6.5.0-rc6-PMacG4-dirty [...]"-line:

[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Total memory = 2048MB; using 4096kB for hash table
[    0.000000] mapin_ram:125
[    0.000000] mmu_mapin_ram:169 0 30000000 1400000 2000000
[    0.000000] __mmu_mapin_ram:146 0 1400000
[    0.000000] __mmu_mapin_ram:155 1400000
[    0.000000] __mmu_mapin_ram:146 1400000 30000000
[    0.000000] __mmu_mapin_ram:155 20000000
[    0.000000] __mapin_ram_chunk:107 20000000 30000000
[    0.000000] __mapin_ram_chunk:117
[    0.000000] mapin_ram:134
[    0.000000] kasan_mmu_init:129
[    0.000000] kasan_mmu_init:132 0
[    0.000000] kasan_mmu_init:137
[    0.000000] setup_kuap:23
[    0.000000] setup_kuap:25
[    0.000000] setup_kuap:30
[    0.000000] setup_kuap:35
[    0.000000] Activating Kernel Userspace Access Protection
[    0.000000] setup_kuap:40
[    0.000000] Activating Kernel Userspace Execution Prevention

I also tried a kernel without KUAP/KUEP getting this:

[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Total memory = 2048MB; using 4096kB for hash table
[    0.000000] mapin_ram:125
[    0.000000] mmu_mapin_ram:169 0 30000000 1400000 2000000
[    0.000000] __mmu_mapin_ram:146 0 1400000
[    0.000000] __mmu_mapin_ram:155 1400000
[    0.000000] __mmu_mapin_ram:146 1400000 30000000
[    0.000000] __mmu_mapin_ram:155 20000000
[    0.000000] __mapin_ram_chunk:107 20000000 30000000
[    0.000000] __mapin_ram_chunk:117
[    0.000000] mapin_ram:134
[    0.000000] kasan_mmu_init:129
[    0.000000] kasan_mmu_init:132 0
[    0.000000] kasan_mmu_init:137
Quoted 3 lines

Here too the freeze just happens before the line "Linux version 6.5.0-rc6-PMacG4-dirty [...]" in dmesg.

Attached is (warm boot) dmesg without KUAP/KUEP.

I'm wondering if the problem is just linked to the kernel being built with CONFIG_SMP or if it is the actual startup of a secondary CPU that cause the freeze.

Please leave the btext_unmap() in place because I think it is important to keep it, and start the kernel with the following parameter:

nr_cpus=1

Thanks
Christophe

#13 lore raw
Quoted 8 lines · On Fri, 18 Aug 2023 15:47:38 +0000 Christophe Leroy…

I'm wondering if the problem is just linked to the kernel being built with CONFIG_SMP or if it is the actual startup of a secondary CPU that cause the freeze.

Please leave the btext_unmap() in place because I think it is important to keep it, and start the kernel with the following parameter:

nr_cpus=1

With btext_unmap() back and place and nr_cpus=1 set the freeze still happens after the 1st btext_unmap:129 on cold boots:

[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Total memory = 2048MB; using 4096kB for hash table
[    0.000000] mapin_ram:125
[    0.000000] mmu_mapin_ram:169 0 30000000 1400000 2000000
[    0.000000] __mmu_mapin_ram:146 0 1400000
[    0.000000] __mmu_mapin_ram:155 1400000
[    0.000000] __mmu_mapin_ram:146 1400000 30000000
[    0.000000] __mmu_mapin_ram:155 20000000
[    0.000000] __mapin_ram_chunk:107 20000000 30000000
[    0.000000] __mapin_ram_chunk:117
[    0.000000] mapin_ram:134
[    0.000000] kasan_mmu_init:129
[    0.000000] kasan_mmu_init:132 0
[    0.000000] kasan_mmu_init:137
[    0.000000] btext_unmap:129

Attached is warm boot dmesg with nr_cpus=1 set.

Regards,
Erhard

#14 lore raw

Le 18/08/2023 à 18:23, Erhard Furtner a écrit :

Quoted 14 lines
Quoted 8 lines · On Fri, 18 Aug 2023 15:47:38 +0000 Christophe Leroy…

I'm wondering if the problem is just linked to the kernel being built with CONFIG_SMP or if it is the actual startup of a secondary CPU that cause the freeze.

Please leave the btext_unmap() in place because I think it is important to keep it, and start the kernel with the following parameter:

nr_cpus=1

With btext_unmap() back and place and nr_cpus=1 set the freeze still happens after the 1st btext_unmap:129 on cold boots:

Quoted 16 lines
[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Total memory = 2048MB; using 4096kB for hash table
[    0.000000] mapin_ram:125
[    0.000000] mmu_mapin_ram:169 0 30000000 1400000 2000000
[    0.000000] __mmu_mapin_ram:146 0 1400000
[    0.000000] __mmu_mapin_ram:155 1400000
[    0.000000] __mmu_mapin_ram:146 1400000 30000000
[    0.000000] __mmu_mapin_ram:155 20000000
[    0.000000] __mapin_ram_chunk:107 20000000 30000000
[    0.000000] __mapin_ram_chunk:117
[    0.000000] mapin_ram:134
[    0.000000] kasan_mmu_init:129
[    0.000000] kasan_mmu_init:132 0
[    0.000000] kasan_mmu_init:137
[    0.000000] btext_unmap:129

Thanks,

Can you replace the call to btext_unmap() by a call to btext_map() at the end of MMU_init() ?

If that gives no interesting result, can you leave the call to btext_unmap() and add a call to btext_map() at the very begining of function start_kernel() in init/main.c (You may have to add a include of asm/btext.h)

With that I hope we can see more stuff.

Christophe

#15 lore raw
Quoted 37 lines · On Tue, 22 Aug 2023 07:31:54 +0000 Christophe Leroy…

Le 18/08/2023 à 18:23, Erhard Furtner a écrit :

Quoted 14 lines
Quoted 8 lines · On Fri, 18 Aug 2023 15:47:38 +0000 Christophe Leroy…

I'm wondering if the problem is just linked to the kernel being built with CONFIG_SMP or if it is the actual startup of a secondary CPU that cause the freeze.

Please leave the btext_unmap() in place because I think it is important to keep it, and start the kernel with the following parameter:

nr_cpus=1

With btext_unmap() back and place and nr_cpus=1 set the freeze still happens after the 1st btext_unmap:129 on cold boots:

Quoted 16 lines
[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Total memory = 2048MB; using 4096kB for hash table
[    0.000000] mapin_ram:125
[    0.000000] mmu_mapin_ram:169 0 30000000 1400000 2000000
[    0.000000] __mmu_mapin_ram:146 0 1400000
[    0.000000] __mmu_mapin_ram:155 1400000
[    0.000000] __mmu_mapin_ram:146 1400000 30000000
[    0.000000] __mmu_mapin_ram:155 20000000
[    0.000000] __mapin_ram_chunk:107 20000000 30000000
[    0.000000] __mapin_ram_chunk:117
[    0.000000] mapin_ram:134
[    0.000000] kasan_mmu_init:129
[    0.000000] kasan_mmu_init:132 0
[    0.000000] kasan_mmu_init:137
[    0.000000] btext_unmap:129

Thanks,

Can you replace the call to btext_unmap() by a call to btext_map() at the end of MMU_init() ?

Quoted 6 lines

If that gives no interesting result, can you leave the call to btext_unmap() and add a call to btext_map() at the very begining of function start_kernel() in init/main.c (You may have to add a include of asm/btext.h)

With that I hope we can see more stuff.

Ok, I tested out both methods.

1.) Replace btext_unmap() with btext_map() at the end of MMU_init().

Warm boot again is unspectacular (attached). On cold boots I sometimes get:

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table
mapin_ram:125
mmu_mapin_ram:169 0 30000000 1400000 2000000
__mmu_mapin_ram:146 0 1400000
__mmu_mapin_ram:155 1400000
__mmu_mapin_ram:146 1400000 30000000
__mmu_mapin_ram:155 20000000
__mapin_ram_chunk:107 20000000 30000000
__mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
ioremap() called early from btext_map+0x64/0xdc. Use early_ioremap() instead
Linux version 6.5.0-rc7-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #4 SMP Wed Aug 23 12:59:11 CEST 2023

which shows one line (Linux version...) more than before. Most of the time I get this more interesting output however:

kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
Linux version 6.5.0-rc7-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #4 SMP Wed Aug 23 12:59:11 CEST 2023 KASAN init done
list_add corruption. prev->next should be next (c17100c0), but was 2c030000. (prev=c036ac7c).

------------[ cut here ]------------
kernel BUG at lib/list_debug.c:30!
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at arch/powerpc/include/asm/machdep.h:227 die+0xd8/0x39c
Modules linked in:
CPU: 0 PID: 0 Comm: swapper Tainted: G               T  ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎irty⊥q
NIP:  c0012c64 LR: c0012c58 CTR: 00000000
REGS: c1717d10 TRAP: 0700   Tainted: G               T   (∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎irty⊥q)
MSR:  00021032 <ME,IR,DR,RI>  CR: 24000484 XER: 00000000

GPR00: 00000000 c1717dc0 c1551c40 00000000 00000000 00000000 00000000 00000000
GPR08: 00000000 00000000 00000000 00000000 00000000 00000000 00dd6f30 021f6e90
GPR16: 021f69b0 02201994 00dd6f3c efff3190 00000000 c1717f10 c1455dd8 c11ec6c0
GPR24: 00001032 c0fab540 c1717e10 00000005 c1740000 c1740000 c1746380 c1555a20 NIP [c0012c64] die+0xd8/0x39c
LR [c0012c58] die+0xcc/0x39c

Call Trace:
[c1717dc0] [c0012c58] die+0xcc/0x39c (unreliable)
[c1717e00] [c00047f0] ProgramCheck_virt+0x100/0x150
--- interrupt: 700 at __list_add_valid+0xe8/0x120
NIP:  c0854ca0 LR: c0854ca0 CTR: 00000000
REGS: c1717e10 TRAP: 0700   Tainted: G                T   (∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎irty⊥q)
MSR:  00021032 <ME,IR,DR,RI>  CR: 24000488  XER: 00000000

GPR00: 00000000 c1717ec0 c1551c40 0000005d 00000000 00000000 00000000 00000000
GPR08: 00000000 00000000 00000000 00000000 00000000 00000000 00dd6f30 021f6e90
GPR16: 021f69b0 02201994 00dd6f3c efff3190 00000000 c1717f10 c1455dd8 c11ec6c0
GPR24: f82e2fde c11ec680 fefefefe c11ec700 effec7a8 c036ac7c c036ac7c c17100c0 NIP [c0854ca0] __list_add_valid+0xe8/@x120
LR [c0854ca0] __list_add_valid+0xe8/@x120
--- interrupt: 700
[c1717ee8] [c0c18764] of_alias_scan+0x330/0x44c
[c1717f70] [c140b0fc] setup_arch+0x78/0x44c
[c1717fc0] [c14045b0] start_kernel+0x78/0x2d8
[c1717ff0] [000035d0] 0x35d0

Code: 3fa0c174 915f0060 39290001 913e0040 480f602d 38600001 488189f1 387db620 4835654d 813db620 2c090000 40820008 <0fe00000> 80de0080 3fa0c0fb 3ee0c172
---[ end trace 0000000000000000 ]---
Oops: Exception in kernel mode, sig: 5 [#1]
BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2
Modules linked in:
CPU: 0 PID: 0 Comm: swapper Tainted: G               T  ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎irty⊥q
NIP:  c0854ca0 LR: c0854ca0 CTR: 00000000
REGS: c1717e10 TRAP: 0700   Tainted: G               T   (∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎irty⊥q)
MSR:  00021032 <ME,IR,DR,RI>  CR: 24000488 XER: 00000000

GPR00: 00000000 c1717ec0 c1551c40 0000005d 00000000 00000000 00000000 00000000
GPR08: 00000000 00000000 00000000 00000000 00000000 00000000 00dd6f30 021f6e90
GPR16: 021f69b0 02201994 00dd6f3c efff3190 00000000 c1717f10 c1455dd8 c11ec6c0
GPR24: f82e2fde c11ec680 fefefefe c11ec700 effec7a8 c036ac7c c036ac7c c17100c0 NIP [c0854ca0] __list_add_valid+0xe8/@x120
LR [c0854ca0] __list_add_valid+0xe8/0x120

Call Trace:
[c1717ec0] [c0854ca0] __list_add_valid+0xe8/0x120 (unreliable)
[c1717ee0] [c0c18764] of_alias_scan+0x330/0x44c
[c1717f70] [c140b0fc] setup_arch+0x78/0x44c
[c1717fc0] [c14045b0] start_kernel+0x78/0x2d8
[c1717ff0] [000035d0] 0x35d0
Code: 7fc5f378 38636060 7f84e378 38630200 4b8b9ec9 0fe00000 3c68c110 7fa6eb78 7fe4fb78 38630180 4b8b9ead <0fe00000> 3c60c110 7fe6fb78 7fa5eb78
---[ end trace 0000000000000000 ]---

2.) Add btext_map() at the very begining of function start_kernel() in init/main.c:

On cold boots I sometimes get:

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table
mapin_ram:125
mmu_mapin_ram:169 0 30000000 1400000 2000000
__mmu_mapin_ram:146 0 1400000
__mmu_mapin_ram:155 1400000
__mmu_mapin_ram:146 1400000 30000000
__mmu_mapin_ram:155 20000000
__mapin_ram_chunk:107 20000000 30000000
__mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
btext_unmap:129
btext_unmap:131
Linux version 6.5.0-rc7-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #5 SMP Wed Aug 23 13:59:00 CEST 2023

which shows one line (Linux version...) more than before. Most of the time I get this more interesting output however:

------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at arch/powerpc/include/asm/machdep.h:227 die+0xd8/0x39c
Modules linked in:
BUG: Kernel NULL pointer dereference on read at 0x00000050
Faulting instruction address: 0xc014e3bc
Thread overran stack, or stack corrupted
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at arch/powerpc/include/asm/machdep.h:227 die+0xd8/0x39c
Modules linked in:
BUG: Kernel NULL pointer dereference on read at 0x00000050

Faulting instruction address: 0xc014e3bc
Thread overran stack, or stack corrupted
[...]

Repeated 10-11 times. In both cases I needed to transcribe the dmesg from a picture I took from the screen + OCR. Hope the numbers are correct.

Regards,
Erhard

#16 lore raw
Quoted 78 lines · Erhard Furtner
Quoted 37 lines · On Tue, 22 Aug 2023 07:31:54 +0000 Christophe Leroy…

Le 18/08/2023 à 18:23, Erhard Furtner a écrit :

Quoted 14 lines
Quoted 8 lines · On Fri, 18 Aug 2023 15:47:38 +0000 Christophe Leroy…

I'm wondering if the problem is just linked to the kernel being built with CONFIG_SMP or if it is the actual startup of a secondary CPU that cause the freeze.

Please leave the btext_unmap() in place because I think it is important to keep it, and start the kernel with the following parameter:

nr_cpus=1

With btext_unmap() back and place and nr_cpus=1 set the freeze still happens after the 1st btext_unmap:129 on cold boots:

Quoted 16 lines
[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] Total memory = 2048MB; using 4096kB for hash table
[    0.000000] mapin_ram:125
[    0.000000] mmu_mapin_ram:169 0 30000000 1400000 2000000
[    0.000000] __mmu_mapin_ram:146 0 1400000
[    0.000000] __mmu_mapin_ram:155 1400000
[    0.000000] __mmu_mapin_ram:146 1400000 30000000
[    0.000000] __mmu_mapin_ram:155 20000000
[    0.000000] __mapin_ram_chunk:107 20000000 30000000
[    0.000000] __mapin_ram_chunk:117
[    0.000000] mapin_ram:134
[    0.000000] kasan_mmu_init:129
[    0.000000] kasan_mmu_init:132 0
[    0.000000] kasan_mmu_init:137
[    0.000000] btext_unmap:129

Thanks,

Can you replace the call to btext_unmap() by a call to btext_map() at the end of MMU_init() ?

Quoted 6 lines

If that gives no interesting result, can you leave the call to btext_unmap() and add a call to btext_map() at the very begining of function start_kernel() in init/main.c (You may have to add a include of asm/btext.h)

With that I hope we can see more stuff.

Ok, I tested out both methods.

1.) Replace btext_unmap() with btext_map() at the end of MMU_init().

Warm boot again is unspectacular (attached). On cold boots I sometimes get:

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table
mapin_ram:125
mmu_mapin_ram:169 0 30000000 1400000 2000000
__mmu_mapin_ram:146 0 1400000
__mmu_mapin_ram:155 1400000
__mmu_mapin_ram:146 1400000 30000000
__mmu_mapin_ram:155 20000000
__mapin_ram_chunk:107 20000000 30000000
__mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
ioremap() called early from btext_map+0x64/0xdc. Use early_ioremap() instead
Linux version 6.5.0-rc7-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #4 SMP Wed Aug 23 12:59:11 CEST 2023

which shows one line (Linux version...) more than before. Most of the time I get this more interesting output however:

kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
Linux version 6.5.0-rc7-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #4 SMP Wed Aug 23 12:59:11 CEST 2023 KASAN init done
list_add corruption. prev->next should be next (c17100c0), but was 2c030000. (prev=c036ac7c).

Quoted 4 lines
------------[ cut here ]------------
kernel BUG at lib/list_debug.c:30!
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at arch/powerpc/include/asm/machdep.h:227 die+0xd8/0x39c

This is a WARN hit while handling the original bug.

Can you apply this patch to avoid that happening, so we can see the original but better.

arch/powerpc/kernel/traps.c +0 -2
index eeff136b83d9..341a0635e131 100644
@@ -198,8 +198,6 @@ static unsigned long oops_begin(struct pt_regs *regs)
 	die_owner = cpu;
 	console_verbose();
 	bust_spinlocks(1);
-	if (machine_is(powermac))
-		pmac_backlight_unblank();
 	return flags;
 }
 NOKPROBE_SYMBOL(oops_begin);

cheers

#17 lore raw
Quoted 50 lines · On Thu, 24 Aug 2023 21:36:26 +1000 Michael Ellerman <[email protected]> wrote:
Quoted 25 lines

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table
mapin_ram:125
mmu_mapin_ram:169 0 30000000 1400000 2000000
__mmu_mapin_ram:146 0 1400000
__mmu_mapin_ram:155 1400000
__mmu_mapin_ram:146 1400000 30000000
__mmu_mapin_ram:155 20000000
__mapin_ram_chunk:107 20000000 30000000
__mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
ioremap() called early from btext_map+0x64/0xdc. Use early_ioremap() instead
Linux version 6.5.0-rc7-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #4 SMP Wed Aug 23 12:59:11 CEST 2023

which shows one line (Linux version...) more than before. Most of the time I get this more interesting output however:

kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
Linux version 6.5.0-rc7-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #4 SMP Wed Aug 23 12:59:11 CEST 2023 KASAN init done
list_add corruption. prev->next should be next (c17100c0), but was 2c030000. (prev=c036ac7c).

Quoted 4 lines
------------[ cut here ]------------
kernel BUG at lib/list_debug.c:30!
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at arch/powerpc/include/asm/machdep.h:227 die+0xd8/0x39c  

This is a WARN hit while handling the original bug.

Can you apply this patch to avoid that happening, so we can see the original but better.

arch/powerpc/kernel/traps.c +0 -2
index eeff136b83d9..341a0635e131 100644
@@ -198,8 +198,6 @@ static unsigned long oops_begin(struct pt_regs *regs)
 	die_owner = cpu;
 	console_verbose();
 	bust_spinlocks(1);
-	if (machine_is(powermac))
-		pmac_backlight_unblank();
 	return flags;
 }
 NOKPROBE_SYMBOL(oops_begin);
Quoted 1 line

cheers

Ok, so I tested now:
Replace btext_unmap() with btext_map() at the end of MMU_init() + Michaels patch.

With the patch I get interesting output less often, but when I do it's:

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table
mapin_ram:125
mmu_mapin_ram:169 0 30000000 1400000 2000000
__mmu_mapin_ram:146 0 1400000
__mmu_mapin_ram:155 1400000
__mmu_mapin_ram:146 1400000 30000000
__mmu_mapin_ram:155 20000000
__mapin_ram_chunk:107 20000000 30000000
__mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
Linux version 6.5.0-rc7-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #4 SMP Wed Aug 23 12:59:11 CEST 2023 KASAN init done

BUG: spinlock bad magic on CPU#0, swapper/0
 lock: 0xc16cbc60, .magic: c036ab84, .owner: <none>/-1, .owner_cpu: -1
CPU: 0 PID: 0 Comm: swapper Tainted: G                T xxxxxxxxxxx
Call Trace:
[c1717c20] [c0f4e288] dump_stack_lvl+0x60/0xa4 (unreliable)
[c1717c40] [c01065e8] do_raw_spin_lock+0x15c/0x1a8
[c1717c70] [c0fa3890] _raw_spin_lock_irqsave+0x20/0x40
[c1717c90] [c0c140ec] of_find_property+0x3c/0x140
[c1717cc0] [c0c14204] of_get_property+0x14/0x4c
[c1717ce0] [c0c22c6c] unlatten_dt_nodes+0x76c/0x894
[c1717f10] [c0c22e88] __unflatten_device_tree+0xf4/0x244
[c1717f50] [c1458050] unflatten_device_tree+0x48/0x84
[c1717f70] [c140b100] setup_arch+0x78/0x44c
[c1717fc0] [c14045b8] start_kernel+0x78/0x2d8

[c1717ff0] [000035d0] 0x35d0

and then the freeze. Or less often I get:

[...]

Modules linked in: _various ASCII chars_ |(EK) _various ASCII chars_ §=(EKTN)
BUG: Unable to handle kernel data access on read at 0x813f0200
Faulting instruction address: 0xc014e444
Thread overran stack, or stack corrupted
Oops: Kernel access of bad area, sig: 11 [#3544]
BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2
Modules linked in: _various ASCII chars_ §=(EKTN)
BUG: Unable to handle kernel data access on read at 0x813f0200
Faulting instruction address: 0xc014e444
Thread overran stack, or stack corrupted
Oops: Kernel access of bad area, sig: 11 [#3545]

BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2

Number after "sig: 11" gets counted up rapidly to #3545 so I can't follow the output on the OF console. Remaining output on screen before the freeze are [#3535] to [#3545] but apart from the numbers the adresses in this output do not change. _various ASCII chars_ in the "Modules linked in:" stay the same but are special characters so hard to transcribe.

Hope that helps.

Regards,
Erhard

#18 lore raw

Le 28/08/2023 à 01:17, Erhard Furtner a écrit :

Quoted 90 lines
Quoted 50 lines · On Thu, 24 Aug 2023 21:36:26 +1000 Michael Ellerman <[email protected]> wrote:
Quoted 25 lines

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table
mapin_ram:125
mmu_mapin_ram:169 0 30000000 1400000 2000000
__mmu_mapin_ram:146 0 1400000
__mmu_mapin_ram:155 1400000
__mmu_mapin_ram:146 1400000 30000000
__mmu_mapin_ram:155 20000000
__mapin_ram_chunk:107 20000000 30000000
__mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
ioremap() called early from btext_map+0x64/0xdc. Use early_ioremap() instead
Linux version 6.5.0-rc7-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #4 SMP Wed Aug 23 12:59:11 CEST 2023

which shows one line (Linux version...) more than before. Most of the time I get this more interesting output however:

kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
Linux version 6.5.0-rc7-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #4 SMP Wed Aug 23 12:59:11 CEST 2023 KASAN init done
list_add corruption. prev->next should be next (c17100c0), but was 2c030000. (prev=c036ac7c).

Quoted 4 lines
------------[ cut here ]------------
kernel BUG at lib/list_debug.c:30!
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at arch/powerpc/include/asm/machdep.h:227 die+0xd8/0x39c

This is a WARN hit while handling the original bug.

Can you apply this patch to avoid that happening, so we can see the original but better.

arch/powerpc/kernel/traps.c +0 -2
index eeff136b83d9..341a0635e131 100644
@@ -198,8 +198,6 @@ static unsigned long oops_begin(struct pt_regs *regs)
  	die_owner = cpu;
  	console_verbose();
  	bust_spinlocks(1);
-	if (machine_is(powermac))
-		pmac_backlight_unblank();
  	return flags;
  }
  NOKPROBE_SYMBOL(oops_begin);
Quoted 1 line

cheers

Ok, so I tested now:
Replace btext_unmap() with btext_map() at the end of MMU_init() + Michaels patch.

With the patch I get interesting output less often, but when I do it's:

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table
mapin_ram:125
mmu_mapin_ram:169 0 30000000 1400000 2000000
__mmu_mapin_ram:146 0 1400000
__mmu_mapin_ram:155 1400000
__mmu_mapin_ram:146 1400000 30000000
__mmu_mapin_ram:155 20000000
__mapin_ram_chunk:107 20000000 30000000
__mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
Linux version 6.5.0-rc7-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #4 SMP Wed Aug 23 12:59:11 CEST 2023 KASAN init done

BUG: spinlock bad magic on CPU#0, swapper/0
  lock: 0xc16cbc60, .magic: c036ab84, .owner: <none>/-1, .owner_cpu: -1
CPU: 0 PID: 0 Comm: swapper Tainted: G                T xxxxxxxxxxx
Call Trace:
[c1717c20] [c0f4e288] dump_stack_lvl+0x60/0xa4 (unreliable)
[c1717c40] [c01065e8] do_raw_spin_lock+0x15c/0x1a8
[c1717c70] [c0fa3890] _raw_spin_lock_irqsave+0x20/0x40
[c1717c90] [c0c140ec] of_find_property+0x3c/0x140
[c1717cc0] [c0c14204] of_get_property+0x14/0x4c
[c1717ce0] [c0c22c6c] unlatten_dt_nodes+0x76c/0x894
[c1717f10] [c0c22e88] __unflatten_device_tree+0xf4/0x244
[c1717f50] [c1458050] unflatten_device_tree+0x48/0x84
[c1717f70] [c140b100] setup_arch+0x78/0x44c
[c1717fc0] [c14045b8] start_kernel+0x78/0x2d8
Quoted 1 line

[c1717ff0] [000035d0] 0x35d0

Ok so there is some corrupted memory somewhere.

Can you try what happens when you remove the call to kasan_init() at the start of setup_arch() in arch/powerpc/kernel/setup-common.c

I'd also be curious to know what happens when CONFIG_DEBUG_SPINLOCK is disabled.

Another question which I'm no sure I asked already: Is it a new problem you have got with recent kernels or is it just that you never tried such a config with older kernels ?

Also, when you say you need to start with another SMP kernel first and then you don't have the problem anymore until the next cold reboot, do you mean you have some old kernel with KASAN that works, or is it a kernel without KASAN that you have to start first ?

Thanks
Christophe

Quoted 25 lines

and then the freeze. Or less often I get:

[...]

Modules linked in: _various ASCII chars_ |(EK) _various ASCII chars_ §=(EKTN)
BUG: Unable to handle kernel data access on read at 0x813f0200
Faulting instruction address: 0xc014e444
Thread overran stack, or stack corrupted
Oops: Kernel access of bad area, sig: 11 [#3544]
BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2
Modules linked in: _various ASCII chars_ §=(EKTN)
BUG: Unable to handle kernel data access on read at 0x813f0200
Faulting instruction address: 0xc014e444
Thread overran stack, or stack corrupted
Oops: Kernel access of bad area, sig: 11 [#3545]

BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2

Number after "sig: 11" gets counted up rapidly to #3545 so I can't follow the output on the OF console. Remaining output on screen before the freeze are [#3535] to [#3545] but apart from the numbers the adresses in this output do not change. _various ASCII chars_ in the "Modules linked in:" stay the same but are special characters so hard to transcribe.

Hope that helps.

Regards,
Erhard

#19 lore raw
Quoted 4 lines · On Thu, 31 Aug 2023 05:32:46 +0000 Christophe Leroy…

Ok so there is some corrupted memory somewhere.

Can you try what happens when you remove the call to kasan_init() at the start of setup_arch() in arch/powerpc/kernel/setup-common.c

Ok, so I left the other patches in place + btext_map() instead of btext_unmap() at the end of MMU_init() + Michaels patch and additionally commented-out kasan_init() as stated above. The outcome is rather interesting! Now I deterministically get this output at boot OF console, regardless wheter it's a cold boot or warm boot:

via-pmu: Server Mode is disabled
PMU driver v2 initialized for Core99, firmware: 0c
ioremap() called early from pmac_nvram_init+0x208/0x7c0. Use early_ioremap() instead
nvram: Checking bank 0...
nvram: gen0=3234, gen1=3235
nvram: Active bank is: 1
nvram: OF partition at 0x410
nvram: XP partition at 0x1020
nvram: NR partition at 0x1120
Top of RAM: 0x80000000, Total RAM: 0x80000000
Memory hole size: 0MB
Zone ranges:

  DMA      [mem 0x0000000000000000-0x000000002fffffff]
  Normal   empty
  HighMem  [mem 0x0000000030000000-0x000000007fffffff]

Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000000000000-0x000000007fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000007fffffff]
percpu: Embedded 14 pages/cpu s24608 r8192 d24544 u57344
pcpu-alloc: s24608 r8192 d24544 u57344 alloc=14*4096
pcpu-alloc: [0] 0
Kernel command line: ro root=/dev/sda5 nr_cpus=1 zswap.max_pool_percent=16 slub_debug=FZP page_poison=1 [email protected]/eth0,[email protected]/70:85:C2:30:EC:01 init=/usr/lib/systemd/systemd Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
Built 1 zonelists, mobility grouping on. Total pages: 522560
mem auto-init: stack:all(pattern), heap alloc:off, heap free:off
stackdepot: allocating hash table via alloc_large_system_hash
stackdepot hash table entries: 1048576 (order: 10, 4194304 bytes, linear)
==================================================================

BUG: KASAN: stack-out-of-bounds in __kernel_poison_pages+0x6c/0xd0
Write of size 4896 at addr c17a7000 by task swapper/0

CPU: 0 PID: 0 Comm: swapper Tainted: G                T 6.5.0-rc7-PMacG4-dirty #7
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[c1717ce0] [c0f4ec40] dump_stack_lvl+0x60/0xa4 (unreliable)
[c1717d00] [c0368380] print_report+0x154/0x548
[c1717d50] [c036813c] kasan_report+0xd0/0x160
[c1717db0] [c0369bb4] kasan_check_range+0x1c8/0x308
[c1717dc0] [c036ae88] memset+0x34/0x90
[c1717de0] [c035b6e0] __kernel_poison_pages+0x6c/0xd0
[c1717e00] [c03355e4] __free_pages_ok+0x418/0x500
[c1717e60] [c14372c8] memblock_free_all+0x268/0x400
[c1717f20] [c14103fc] mem_init+0x8c/0x274
[c1717f60] [c1431cd0] mm_core_init+0x240/0x4e0
[c1717fc0] [c1404694] start_kernel+0x150/0x2d8

[c1717f00] [000035d0] 0x35d0

The buggy address belongs to the physical page:
page:(ptrval) refcount:0 mapcount:0 mapping:00000000 index:0x0 pfn:0x17a7
flags: 0x0(zone=0)
page_type: 0xffffffff()
raw: 00000000 eee15380 eee15380 00000000 00000000 00000000 ffffffff 00000000
raw: 00000000
page dumped because: kasan: bad access detected

Memory state around the buggy address:
 c17a7d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 c17a7d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>c17a7e00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1
                                                     ^
 c17a7e80: f1 f1 04 f2 04 f2 00 f3 f3 f3 00 00 00 00 00 00
 c17a7f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
==================================================================

Disabling lock debugging due to kernel taint

Quoted 2 lines

I'd also be curious to know what happens when CONFIG_DEBUG_SPINLOCK is disabled.

Disabling CONFIG_DEBUG_SPINLOCK does not change the output above. ^^

Quoted 3 lines

Another question which I'm no sure I asked already: Is it a new problem you have got with recent kernels or is it just that you never tried such a config with older kernels ?

I wanted to revisit https://bugzilla.kernel.org/show_bug.cgi?id=216041 and verify whether it was resolved. KASAN worked around 2019-2021 on my G4 as I reported some bugs with it around that time and you fixed some of the bugs. ;) Like kernel bugzilla #205099, #216190, #205885.

But it always seemed flaky on the G4 and had it's problems. So I can't tell whether this specific issue was there back then or if it's new. At least bug #216190 was also about KASAN and SMP issues.

Quoted 4 lines

Also, when you say you need to start with another SMP kernel first and then you don't have the problem anymore until the next cold reboot, do you mean you have some old kernel with KASAN that works, or is it a kernel without KASAN that you have to start first ?

First. I start with a non-KASAN SMP kernel and afterwards reboot into a KASAN kernel. But now with kasan_init() commented-out in start of setup_arch() in arch/powerpc/kernel/setup-common.c this does not work anymore. I get the dmesg above all the time, at cold and warm boots.

Regards,
Erhard

#20 lore raw

Le 01/09/2023 à 00:44, Erhard Furtner a écrit :

Quoted 9 lines
Quoted 4 lines · On Thu, 31 Aug 2023 05:32:46 +0000 Christophe Leroy…

Ok so there is some corrupted memory somewhere.

Can you try what happens when you remove the call to kasan_init() at the start of setup_arch() in arch/powerpc/kernel/setup-common.c

Ok, so I left the other patches in place + btext_map() instead of btext_unmap() at the end of MMU_init() + Michaels patch and additionally commented-out kasan_init() as stated above. The outcome is rather interesting! Now I deterministically get this output at boot OF console, regardless wheter it's a cold boot or warm boot:

Ah, my bad. You also need to remove the call to kasan_late_init() in mem_init() in arch/powerpc/mm/mem.c

Nevertheless, your result is interesting as it shows that the boot goes much further when we don't initialise KASAN.

It probably means that kasan_init() messed up things. I will try to dig a bit more in kasan_init() and see what we can look at.

Christophe

Quoted 90 lines

via-pmu: Server Mode is disabled
PMU driver v2 initialized for Core99, firmware: 0c
ioremap() called early from pmac_nvram_init+0x208/0x7c0. Use early_ioremap() instead
nvram: Checking bank 0...
nvram: gen0=3234, gen1=3235
nvram: Active bank is: 1
nvram: OF partition at 0x410
nvram: XP partition at 0x1020
nvram: NR partition at 0x1120
Top of RAM: 0x80000000, Total RAM: 0x80000000
Memory hole size: 0MB
Zone ranges:

   DMA      [mem 0x0000000000000000-0x000000002fffffff]
   Normal   empty
   HighMem  [mem 0x0000000030000000-0x000000007fffffff]

Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000000000000-0x000000007fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000007fffffff]
percpu: Embedded 14 pages/cpu s24608 r8192 d24544 u57344
pcpu-alloc: s24608 r8192 d24544 u57344 alloc=14*4096
pcpu-alloc: [0] 0
Kernel command line: ro root=/dev/sda5 nr_cpus=1 zswap.max_pool_percent=16 slub_debug=FZP page_poison=1 [email protected]/eth0,[email protected]/70:85:C2:30:EC:01 init=/usr/lib/systemd/systemd Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
Built 1 zonelists, mobility grouping on. Total pages: 522560
mem auto-init: stack:all(pattern), heap alloc:off, heap free:off
stackdepot: allocating hash table via alloc_large_system_hash
stackdepot hash table entries: 1048576 (order: 10, 4194304 bytes, linear)
==================================================================

BUG: KASAN: stack-out-of-bounds in __kernel_poison_pages+0x6c/0xd0
Write of size 4896 at addr c17a7000 by task swapper/0

CPU: 0 PID: 0 Comm: swapper Tainted: G                T 6.5.0-rc7-PMacG4-dirty #7
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[c1717ce0] [c0f4ec40] dump_stack_lvl+0x60/0xa4 (unreliable)
[c1717d00] [c0368380] print_report+0x154/0x548
[c1717d50] [c036813c] kasan_report+0xd0/0x160
[c1717db0] [c0369bb4] kasan_check_range+0x1c8/0x308
[c1717dc0] [c036ae88] memset+0x34/0x90
[c1717de0] [c035b6e0] __kernel_poison_pages+0x6c/0xd0
[c1717e00] [c03355e4] __free_pages_ok+0x418/0x500
[c1717e60] [c14372c8] memblock_free_all+0x268/0x400
[c1717f20] [c14103fc] mem_init+0x8c/0x274
[c1717f60] [c1431cd0] mm_core_init+0x240/0x4e0
[c1717fc0] [c1404694] start_kernel+0x150/0x2d8

[c1717f00] [000035d0] 0x35d0

The buggy address belongs to the physical page:
page:(ptrval) refcount:0 mapcount:0 mapping:00000000 index:0x0 pfn:0x17a7
flags: 0x0(zone=0)
page_type: 0xffffffff()
raw: 00000000 eee15380 eee15380 00000000 00000000 00000000 ffffffff 00000000
raw: 00000000
page dumped because: kasan: bad access detected

Memory state around the buggy address:
  c17a7d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  c17a7d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Quoted 1 line

c17a7e00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1

                                                      ^

c17a7e80: f1 f1 04 f2 04 f2 00 f3 f3 f3 00 00 00 00 00 00
c17a7f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
==================================================================
Disabling lock debugging due to kernel taint

Quoted 2 lines

I'd also be curious to know what happens when CONFIG_DEBUG_SPINLOCK is disabled.

Disabling CONFIG_DEBUG_SPINLOCK does not change the output above. ^^

Quoted 3 lines

Another question which I'm no sure I asked already: Is it a new problem you have got with recent kernels or is it just that you never tried such a config with older kernels ?

I wanted to revisit https://bugzilla.kernel.org/show_bug.cgi?id=216041 and verify whether it was resolved. KASAN worked around 2019-2021 on my G4 as I reported some bugs with it around that time and you fixed some of the bugs. ;) Like kernel bugzilla #205099, #216190, #205885.

But it always seemed flaky on the G4 and had it's problems. So I can't tell whether this specific issue was there back then or if it's new. At least bug #216190 was also about KASAN and SMP issues.

Quoted 4 lines

Also, when you say you need to start with another SMP kernel first and then you don't have the problem anymore until the next cold reboot, do you mean you have some old kernel with KASAN that works, or is it a kernel without KASAN that you have to start first ?

First. I start with a non-KASAN SMP kernel and afterwards reboot into a KASAN kernel. But now with kasan_init() commented-out in start of setup_arch() in arch/powerpc/kernel/setup-common.c this does not work anymore. I get the dmesg above all the time, at cold and warm boots.

Regards,
Erhard

#21 lore raw
Quoted 7 lines · On Fri, 1 Sep 2023 07:43:34 +0000 Christophe Leroy…
Quoted 4 lines
Quoted 2 lines

Can you try what happens when you remove the call to kasan_init() at the start of setup_arch() in arch/powerpc/kernel/setup-common.c

Ok, so I left the other patches in place + btext_map() instead of btext_unmap() at the end of MMU_init() + Michaels patch and additionally commented-out kasan_init() as stated above. The outcome is rather interesting! Now I deterministically get this output at boot OF console, regardless wheter it's a cold boot or warm boot:

Ah, my bad. You also need to remove the call to kasan_late_init() in mem_init() in arch/powerpc/mm/mem.c

Not tragic. Meanwhile I commented-out kasan_late_init() and updated to kernel v6.5.1.

dmesg did not change however, getting the same "BUG: KASAN: stack-out-of-bounds in __kernel_poison_pages+0x6c/0xd0" as last time only on v6.5.1.

Regards,
Erhard

#22 lore raw

Le 03/09/2023 à 23:06, Erhard Furtner a écrit :

Quoted 6 lines
Quoted 3 lines · On Fri, 1 Sep 2023 07:43:34 +0000 Christophe Leroy…
Quoted 3 lines
Quoted 2 lines

Can you try what happens when you remove the call to kasan_init() at the start of setup_arch() in arch/powerpc/kernel/setup-common.c

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

Ok, so I left the other patches in place + btext_map() instead of btext_unmap() at the end of MMU_init() + Michaels patch and additionally commented-out kasan_init() as stated above. The outcome is rather interesting! Now I deterministically get this output at boot OF console, regardless wheter it's a cold boot or warm boot:

Ah, my bad. You also need to remove the call to kasan_late_init() in mem_init() in arch/powerpc/mm/mem.c

Not tragic. Meanwhile I commented-out kasan_late_init() and updated to kernel v6.5.1.

dmesg did not change however, getting the same "BUG: KASAN: stack-out-of-bounds in __kernel_poison_pages+0x6c/0xd0" as last time only on v6.5.1.

Ok, so lets come back to normal situation. Can you add back kasan_init() and kasan_late_init(), while keeping the btext changes and Michael's patch.

Then see what result you get with CONFIG_KASAN but without CONFIG_KASAN_VMALLOC

It would help narrow the problem area because kasan_init() does several things based on CONFIG_KASAN_VMALLOC.

Thanks
Christophe

#23 lore raw

Le 04/09/2023 à 16:48, Christophe Leroy a écrit :

Quoted 28 lines

Le 03/09/2023 à 23:06, Erhard Furtner a écrit :

Quoted 16 lines
Quoted 4 lines · On Fri, 1 Sep 2023 07:43:34 +0000 Christophe Leroy…
Quoted 4 lines
Quoted 3 lines

Can you try what happens when you remove the call to kasan_init() at the
start of setup_arch() in arch/powerpc/kernel/setup-common.c

Quoted 8 lines
Quoted 5 lines

Ok, so I left the other patches in place + btext_map() instead of btext_unmap() at the end of MMU_init() + Michaels patch and additionally commented-out kasan_init() as stated above. The outcome is rather interesting! Now I deterministically get this output at boot OF console, regardless wheter it's a cold boot or warm boot:

Ah, my bad. You also need to remove the call to kasan_late_init() in mem_init() in arch/powerpc/mm/mem.c

Quoted 7 lines

Not tragic. Meanwhile I commented-out kasan_late_init() and updated to kernel v6.5.1.

dmesg did not change however, getting the same "BUG: KASAN:
stack-out-of-bounds in __kernel_poison_pages+0x6c/0xd0" as last time only on v6.5.1.

Quoted 9 lines

Ok, so lets come back to normal situation. Can you add back kasan_init() and kasan_late_init(), while keeping the btext changes and Michael's patch.

Then see what result you get with CONFIG_KASAN but without CONFIG_KASAN_VMALLOC

It would help narrow the problem area because kasan_init() does several things based on CONFIG_KASAN_VMALLOC.

Another thing that could be interesting to test is to remove (or comment out) the following line in arch/powerpc/mm/kasan/Makefile :

   obj-$(CONFIG_PPC_BOOK3S_32)	+= book3s_32.o

That way, the weak version of kasan_init_region() will be used instead of the one in book3s_32.c

#24 lore raw
Quoted 8 lines · On Mon, 4 Sep 2023 14:55:17 +0000 Christophe Leroy…
Quoted 8 lines

Ok, so lets come back to normal situation. Can you add back kasan_init() and kasan_late_init(), while keeping the btext changes and Michael's patch.

Then see what result you get with CONFIG_KASAN but without CONFIG_KASAN_VMALLOC

It would help narrow the problem area because kasan_init() does several things based on CONFIG_KASAN_VMALLOC.

I can't unselect KASAN_VMALLOC, it is forced on. Can only be unselected on PPC_BOOK3S_64 it seems.

Symbol: KASAN_VMALLOC [=y]
Type : bool
Defined at lib/Kconfig.kasan:179
Prompt: Check accesses to vmalloc allocations Depends on: KASAN [=y] && HAVE_ARCH_KASAN_VMALLOC [=y] Location:

     -> Kernel hacking
       -> Memory Debugging
         -> KASAN: dynamic memory safety error detector (KASAN [=y])
           -> Check accesses to vmalloc allocations (KASAN_VMALLOC [=y])

Selected by [y]:
- PPC [=y] && KASAN [=y] && MODULES [=y]
Selected by [n]:
- PPC_BOOK3S_64 [=n] && <choice> && KASAN [=y] && <choice> && KASAN [=y]

Quoted 7 lines

Another thing that could be interesting to test is to remove (or comment out) the following line in arch/powerpc/mm/kasan/Makefile :

   obj-$(CONFIG_PPC_BOOK3S_32)	+= book3s_32.o

That way, the weak version of kasan_init_region() will be used instead of the one in book3s_32.c

When I comment out this line the kernel boots fine! Even with btext_unmap() stuff back to default and using 2 CPUs again.

Regards,
Erhard

#25 lore raw
In reply to Christophe Leroy
Quoted 7 lines · On Mon, 4 Sep 2023 14:55:17 +0000 Christophe Leroy…

Another thing that could be interesting to test is to remove (or comment out) the following line in arch/powerpc/mm/kasan/Makefile :

   obj-$(CONFIG_PPC_BOOK3S_32)	+= book3s_32.o

That way, the weak version of kasan_init_region() will be used instead of the one in book3s_32.c

Tinkered around with older kernels and .config options meanwhile. I found out it's not a new issue, also happens on kernel v6.0. Have not tried older kernels yet.

Also on v6.0 the issue disappears when I comment out "obj-$(CONFIG_PPC_BOOK3S_32) += book3s_32.o" in arch/powerpc/mm/kasan/Makefile.

Even more interesting is the issue disappears too when I deselect INIT_STACK_ALL_PATTERN in my kernel .config. The kernel boots just fine with KASAN when INIT_STACK_NONE=y is set! True for kernel v6.0 and v6.6-rc1.

Current v6.6-rc1 kernel .config and dmesg attached.

Regards,
Erhard

#26 lore raw

Le 12/09/2023 à 02:11, Erhard Furtner a écrit :

Quoted 11 lines
Quoted 7 lines · On Mon, 4 Sep 2023 14:55:17 +0000 Christophe Leroy…

Another thing that could be interesting to test is to remove (or comment out) the following line in arch/powerpc/mm/kasan/Makefile :

    obj-$(CONFIG_PPC_BOOK3S_32)	+= book3s_32.o

That way, the weak version of kasan_init_region() will be used instead of the one in book3s_32.c

Quoted 8 lines

Tinkered around with older kernels and .config options meanwhile. I found out it's not a new issue, also happens on kernel v6.0. Have not tried older kernels yet.

Also on v6.0 the issue disappears when I comment out "obj-$(CONFIG_PPC_BOOK3S_32) += book3s_32.o" in arch/powerpc/mm/kasan/Makefile.

Even more interesting is the issue disappears too when I deselect INIT_STACK_ALL_PATTERN in my kernel .config. The kernel boots just fine with KASAN when INIT_STACK_NONE=y is set! True for kernel v6.0 and v6.6-rc1.

Current v6.6-rc1 kernel .config and dmesg attached.

I suspect something wrong when we set the BATs.

Can you try with the following additional traces:

(quoted diff) +0 -0

b/arch/powerpc/mm/kasan/book3s_32.c
index 450a67ef0bbe..9954b7a3b7ae 100644

arch/powerpc/mm/kasan/book3s_32.c +2 -0
@@ -15,6 +15,7 @@ int __init kasan_init_region(void *start, size_t size)
  	phys_addr_t phys;
  	int ret;
 
+	pr_err("%s: %px %x %lx %lx\n", __func__, start, size, k_start, k_end);
  	while (k_nobat < k_end) {
  		unsigned int k_size = bat_block_size(k_nobat, k_end);
  		int idx = find_free_bat();
@@ -28,6 +29,7 @@ int __init kasan_init_region(void *start, size_t size)
  		if (!phys)
  			break;
 
+		pr_err("%s: setbat %d %lx %x %x\n", __func__, idx, k_nobat, phys, 

k_size);
setbat(idx, k_nobat, phys, k_size, PAGE_KERNEL); k_nobat += k_size;
}
@@ -47,6 +49,7 @@ int __init kasan_init_region(void *start, size_t size)

  	kasan_update_early_region(k_start, k_nobat, __pte(0));

+ pr_err("%s: loop %lx %lx\n", __func__, k_nobat, k_end);
for (k_cur = k_nobat; k_cur < k_end; k_cur += PAGE_SIZE) {
pmd_t *pmd = pmd_off_k(k_cur);
pte_t pte = pfn_pte(PHYS_PFN(phys + k_cur - k_nobat), PAGE_KERNEL);

You'd then get something like:

Total memory = 2048MB; using 4096kB for hash table Activating Kernel Userspace Access Protection
Activating Kernel Userspace Execution Prevention
Linux version 6.6.0-rc1+ ([email protected]) (powerpc64-linux-gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.39) #461 Tue

Sep 12 09:37:11 CEST 2023

kasan_init_region: c0000000 30000000 f8000000 fe000000
kasan_init_region: setbat 3 f8000000 7c000000 4000000
kasan_init_region: loop fc000000 fe000000 KASAN init done

Thanks
Christophe

#27 lore raw
Quoted 43 lines · On Tue, 12 Sep 2023 07:47:50 +0000 Christophe Leroy…

I suspect something wrong when we set the BATs.

Can you try with the following additional traces:

arch/powerpc/mm/kasan/book3s_32.c +2 -0
index 450a67ef0bbe..9954b7a3b7ae 100644
@@ -15,6 +15,7 @@ int __init kasan_init_region(void *start, size_t size)
  	phys_addr_t phys;
  	int ret;
 
+	pr_err("%s: %px %x %lx %lx\n", __func__, start, size, k_start, k_end);
  	while (k_nobat < k_end) {
  		unsigned int k_size = bat_block_size(k_nobat, k_end);
  		int idx = find_free_bat();
@@ -28,6 +29,7 @@ int __init kasan_init_region(void *start, size_t size)
  		if (!phys)
  			break;
 
+		pr_err("%s: setbat %d %lx %x %x\n", __func__, idx, k_nobat, phys, 
 k_size);
  		setbat(idx, k_nobat, phys, k_size, PAGE_KERNEL);
  		k_nobat += k_size;
  	}
arch/powerpc/mm/kasan/book3s_32.c +1 -0
@@ -47,6 +49,7 @@ int __init kasan_init_region(void *start, size_t size)
 
  	kasan_update_early_region(k_start, k_nobat, __pte(0));
 
+	pr_err("%s: loop %lx %lx\n", __func__, k_nobat, k_end);
  	for (k_cur = k_nobat; k_cur < k_end; k_cur += PAGE_SIZE) {
  		pmd_t *pmd = pmd_off_k(k_cur);
  		pte_t pte = pfn_pte(PHYS_PFN(phys + k_cur - k_nobat), PAGE_KERNEL);

You'd then get something like:

Total memory = 2048MB; using 4096kB for hash table Activating Kernel Userspace Access Protection
Activating Kernel Userspace Execution Prevention
Linux version 6.6.0-rc1+ ([email protected]) (powerpc64-linux-gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.39) #461 Tue

Quoted 5 lines
Sep 12 09:37:11 CEST 2023

kasan_init_region: c0000000 30000000 f8000000 fe000000
kasan_init_region: setbat 3 f8000000 7c000000 4000000
kasan_init_region: loop fc000000 fe000000 KASAN init done

Applied this patch on top of 6.6-rc1, your old pr_info() patch and Michaels patch.

This way booting gets stuck at:
printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table mapin_ram:125
mmu_mapin_ram:169 0 30000000 1400000 2000000 __mmu_mapin_ram:146 0 1400000 __mmu_mapin_ram:155 1400000 __mmu_mapin_ram:146 1400000 30000000 __mmu_mapin_ram:155 20000000 __mapin_ram_chunk:107 20000000 30000000 __mapin_ram_chunk:117
mapin_ram:134
btext_unmap:129

Having moved btext_unmap() after memblock_set_current_limit(lowmem_end_addr) in init_32.c I get:

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table mapin_ram:125
mmu_mapin_ram:169 0 30000000 1400000 2000000 __mmu_mapin_ram:146 0 1400000 __mmu_mapin_ram:155 1400000 __mmu_mapin_ram:146 1400000 30000000 __mmu_mapin_ram:155 20000000 __mapin_ram_chunk:107 20000000 30000000 __mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
btext_unmap:129

Changing the already moved btext_unmap() to btext_map() I get:

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table
mapin_ram:125
mmu_mapin_ram:169 0 30000000 1400000 2000000
__mmu_mapin_ram:146 0 1400000
__mmu_mapin_ram:155 1400000
__mmu_mapin_ram:146 1400000 30000000
__mmu_mapin_ram:155 20000000
__mapin_ram_chunk:107 20000000 30000000
__mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
ioremap() called early from btext_map+0x64/0xdc. Use early_ioremap() instead
Linux version 6.6.0-rc1-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #5 SMP Tue Sep 12 16:50:47 CEST 2023
kasan_init_region: c0000000 30000000 f8000000 fe000000
kasan_init_region: loop f8000000 fe000000

So I get no "kasan_init_region: setbat" line and don't reach "KASAN init done".

Regards,
Erhard

#28 lore raw

Le 12/09/2023 à 17:59, Erhard Furtner a écrit :

Quoted 21 lines

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table
mapin_ram:125
mmu_mapin_ram:169 0 30000000 1400000 2000000
__mmu_mapin_ram:146 0 1400000
__mmu_mapin_ram:155 1400000
__mmu_mapin_ram:146 1400000 30000000
__mmu_mapin_ram:155 20000000
__mapin_ram_chunk:107 20000000 30000000
__mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
ioremap() called early from btext_map+0x64/0xdc. Use early_ioremap() instead
Linux version 6.6.0-rc1-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #5 SMP Tue Sep 12 16:50:47 CEST 2023
kasan_init_region: c0000000 30000000 f8000000 fe000000
kasan_init_region: loop f8000000 fe000000

Quoted 1 line

So I get no "kasan_init_region: setbat" line and don't reach "KASAN init done".

Ah ok, maybe your CPU only has 4 BATs and they are all used, following change would tell us.

arch/powerpc/mm/book3s32/mmu.c +1 -0
index 850783cfa9c7..bd26767edce7 100644
@@ -86,6 +86,7 @@ int __init find_free_bat(void)
  		if (!(bat[1].batu & 3))
  			return b;
  	}
+	pr_err("NO FREE BAT (%d)\n", n);
  	return -1;
  }
 

Or you have 8 BATs in which case it's an alignment problem, you need to increase CONFIG_DATA_SHIFT to 23, for that you need CONFIG_ADVANCED and CONFIG_DATA_SHIFT_BOOL

But regardless of that there is a problem we need to find out, because it should work without BATs.

As the BATs allocation fails, it falls back to :

	phys = memblock_phys_alloc_range(k_end - k_start, PAGE_SIZE, 0,
						 MEMBLOCK_ALLOC_ANYWHERE);
		if (!phys)
			return -ENOMEM;
	}

	ret = kasan_init_shadow_page_tables(k_start, k_end);
	if (ret)
		return ret;

	for (k_cur = k_start; k_cur < k_end; k_cur += PAGE_SIZE) {
		pmd_t *pmd = pmd_off_k(k_cur);
		pte_t pte = pfn_pte(PHYS_PFN(phys + k_cur - k_start), PAGE_KERNEL);

		__set_pte_at(&init_mm, k_cur, pte_offset_kernel(pmd, k_cur), pte, 0);
	}
	flush_tlb_kernel_range(k_start, k_end);
	memset(kasan_mem_to_shadow(start), 0, k_end - k_start);

While the __weak function that you confirmed working is:

	ret = kasan_init_shadow_page_tables(k_start, k_end);
	if (ret)
		return ret;

	block = memblock_alloc(k_end - k_start, PAGE_SIZE);
	if (!block)
		return -ENOMEM;

	for (k_cur = k_start & PAGE_MASK; k_cur < k_end; k_cur += PAGE_SIZE) {
		pmd_t *pmd = pmd_off_k(k_cur);
		void *va = block + k_cur - k_start;
		pte_t pte = pfn_pte(PHYS_PFN(__pa(va)), PAGE_KERNEL);

		__set_pte_at(&init_mm, k_cur, pte_offset_kernel(pmd, k_cur), pte, 0);
	}
	flush_tlb_kernel_range(k_start, k_end);

I'm having hard time to understand what's could be wrong at the first place.

Could you try following change:

(quoted diff) +0 -0

b/arch/powerpc/mm/kasan/book3s_32.c
index 9954b7a3b7ae..e04f21908c6a 100644

arch/powerpc/mm/kasan/book3s_32.c +1 -1
@@ -38,7 +38,7 @@ int __init kasan_init_region(void *start, size_t size)
 
  	if (k_nobat < k_end) {
  		phys = memblock_phys_alloc_range(k_end - k_nobat, PAGE_SIZE, 0,
-						 MEMBLOCK_ALLOC_ANYWHERE);
+						 MEMBLOCK_ALLOC_ACCESSIBLE);
  		if (!phys)
  			return -ENOMEM;
  	}

And also that one:

(quoted diff) +0 -0

b/arch/powerpc/mm/kasan/init_32.c
index a70828a6d935..bc1c075489f4 100644

arch/powerpc/mm/kasan/init_32.c +0 -0
@@ -84,6 +84,9 @@ kasan_update_early_region(unsigned long k_start, 

unsigned long k_end, pte_t pte)
{
unsigned long k_cur;

+ if (k_start == k_end)
+ return;
+
for (k_cur = k_start; k_cur != k_end; k_cur += PAGE_SIZE) {
pmd_t *pmd = pmd_off_k(k_cur);
pte_t *ptep = pte_offset_kernel(pmd, k_cur);

Thanks
Christophe

author Applied
#29 lore raw
Quoted 15 lines · On Tue, 12 Sep 2023 17:39:10 +0000 Christophe Leroy…

Ah ok, maybe your CPU only has 4 BATs and they are all used, following change would tell us.

arch/powerpc/mm/book3s32/mmu.c +1 -0
index 850783cfa9c7..bd26767edce7 100644
@@ -86,6 +86,7 @@ int __init find_free_bat(void)
  		if (!(bat[1].batu & 3))
  			return b;
  	}
+	pr_err("NO FREE BAT (%d)\n", n);
  	return -1;
  }
 
Quoted 3 lines

Or you have 8 BATs in which case it's an alignment problem, you need to increase CONFIG_DATA_SHIFT to 23, for that you need CONFIG_ADVANCED and CONFIG_DATA_SHIFT_BOOL

Applied all your patches. According to the output my G4 DP seems to have 8 BATs. Now I get the following output on screen before the freeze:

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table
mapin_ram:125
mmu_mapin_ram:170 0 30000000 1400000 2000000
__mmu_mapin_ram:147 0 1400000
__mmu_mapin_ram:156 1400000
__mmu_mapin_ram:147 1400000 30000000
NO FREE BAT (8)
__mmu_mapin_ram:156 20000000
__mapin_ram_chunk:107 20000000 30000000
__mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:132
kasan_mmu_init:135 0
kasan_mmu_init:140
ioremap() called early from btext_map+0x64/0xdc. Use early_ioremap() instead
Linux version 6.6.0-rc1-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #5 SMP Tue Sep 12 16:50:47 CEST 2023
kasan_init_region: c0000000 30000000 f8000000 fe000000
NO FREE BAT (8)
kasan_init_region: loop f8000000 fe000000

So I set CONFIG_DATA_SHIFT=23 as suggested but then I only get:

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table mapin_ram:125
mmu_mapin_ram:170 0 30000000 1400000 2000000 __mmu_mapin_ram:147 0 1400000 __mmu_mapin_ram:156 1400000 __mmu_mapin_ram:147 1400000 30000000
NO FREE BAT (8)
__mmu_mapin_ram:156 20000000 __mapin_ram_chunk:107 20000000 30000000 __mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:132
kasan_mmu_init:135 0
kasan_mmu_init:140

With btext_unmap() left in place in init_32.c I only get:

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table mapin_ram:125
mmu_mapin_ram:170 0 30000000 1400000 2000000 __mmu_mapin_ram:147 0 1400000 __mmu_mapin_ram:156 1400000 __mmu_mapin_ram:147 1400000 30000000
NO FREE BAT (8)
__mmu_mapin_ram:156 20000000 __mapin_ram_chunk:107 20000000 30000000 __mapin_ram_chunk:117
mapin_ram:134
btext_unmap:129

Hope the output sheds at least some light on what's going wrong!

Regards,
Erhard

#30 lore raw

Le 12/09/2023 à 22:09, Erhard Furtner a écrit :

Quoted 81 lines
Quoted 15 lines · On Tue, 12 Sep 2023 17:39:10 +0000 Christophe Leroy…

Ah ok, maybe your CPU only has 4 BATs and they are all used, following change would tell us.

arch/powerpc/mm/book3s32/mmu.c +1 -0
index 850783cfa9c7..bd26767edce7 100644
@@ -86,6 +86,7 @@ int __init find_free_bat(void)
   		if (!(bat[1].batu & 3))
   			return b;
   	}
+	pr_err("NO FREE BAT (%d)\n", n);
   	return -1;
   }
 
Quoted 3 lines

Or you have 8 BATs in which case it's an alignment problem, you need to increase CONFIG_DATA_SHIFT to 23, for that you need CONFIG_ADVANCED and CONFIG_DATA_SHIFT_BOOL

Applied all your patches. According to the output my G4 DP seems to have 8 BATs. Now I get the following output on screen before the freeze:

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table
mapin_ram:125
mmu_mapin_ram:170 0 30000000 1400000 2000000
__mmu_mapin_ram:147 0 1400000
__mmu_mapin_ram:156 1400000
__mmu_mapin_ram:147 1400000 30000000
NO FREE BAT (8)
__mmu_mapin_ram:156 20000000
__mapin_ram_chunk:107 20000000 30000000
__mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:132
kasan_mmu_init:135 0
kasan_mmu_init:140
ioremap() called early from btext_map+0x64/0xdc. Use early_ioremap() instead
Linux version 6.6.0-rc1-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #5 SMP Tue Sep 12 16:50:47 CEST 2023
kasan_init_region: c0000000 30000000 f8000000 fe000000
NO FREE BAT (8)
kasan_init_region: loop f8000000 fe000000

So I set CONFIG_DATA_SHIFT=23 as suggested but then I only get:

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table mapin_ram:125
mmu_mapin_ram:170 0 30000000 1400000 2000000 __mmu_mapin_ram:147 0 1400000 __mmu_mapin_ram:156 1400000 __mmu_mapin_ram:147 1400000 30000000
NO FREE BAT (8)
__mmu_mapin_ram:156 20000000 __mapin_ram_chunk:107 20000000 30000000 __mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:132
kasan_mmu_init:135 0
kasan_mmu_init:140

With btext_unmap() left in place in init_32.c I only get:

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table mapin_ram:125
mmu_mapin_ram:170 0 30000000 1400000 2000000 __mmu_mapin_ram:147 0 1400000 __mmu_mapin_ram:156 1400000 __mmu_mapin_ram:147 1400000 30000000
NO FREE BAT (8)
__mmu_mapin_ram:156 20000000 __mapin_ram_chunk:107 20000000 30000000 __mapin_ram_chunk:117
mapin_ram:134
btext_unmap:129

Quoted 2 lines

Hope the output sheds at least some light on what's going wrong!

Can you send me (off-list ?) the two vmlinux binaries (The one with data shift 22 and the one with 23) and also your latest .config

About the shift 23, I suspect that ppc32 still doesn't like when .init is past the first 32 Mb.

Thanks
Christophe

#31 lore raw
In reply to Christophe Leroy

Le 12/09/2023 à 19:39, Christophe Leroy a écrit :

Quoted 128 lines

Le 12/09/2023 à 17:59, Erhard Furtner a écrit :

Quoted 21 lines

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table
mapin_ram:125
mmu_mapin_ram:169 0 30000000 1400000 2000000
__mmu_mapin_ram:146 0 1400000
__mmu_mapin_ram:155 1400000
__mmu_mapin_ram:146 1400000 30000000
__mmu_mapin_ram:155 20000000
__mapin_ram_chunk:107 20000000 30000000
__mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
ioremap() called early from btext_map+0x64/0xdc. Use early_ioremap() instead
Linux version 6.6.0-rc1-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #5 SMP Tue Sep 12 16:50:47 CEST 2023
kasan_init_region: c0000000 30000000 f8000000 fe000000
kasan_init_region: loop f8000000 fe000000

Quoted 1 line

So I get no "kasan_init_region: setbat" line and don't reach "KASAN init done".

Ah ok, maybe your CPU only has 4 BATs and they are all used, following change would tell us.

arch/powerpc/mm/book3s32/mmu.c +1 -0
index 850783cfa9c7..bd26767edce7 100644
@@ -86,6 +86,7 @@ int __init find_free_bat(void)
   		if (!(bat[1].batu & 3))
   			return b;
   	}
+	pr_err("NO FREE BAT (%d)\n", n);
   	return -1;
   }
 

Or you have 8 BATs in which case it's an alignment problem, you need to increase CONFIG_DATA_SHIFT to 23, for that you need CONFIG_ADVANCED and CONFIG_DATA_SHIFT_BOOL

But regardless of that there is a problem we need to find out, because it should work without BATs.

As the BATs allocation fails, it falls back to :

	phys = memblock_phys_alloc_range(k_end - k_start, PAGE_SIZE, 0,
						 MEMBLOCK_ALLOC_ANYWHERE);
		if (!phys)
			return -ENOMEM;
	}

	ret = kasan_init_shadow_page_tables(k_start, k_end);
	if (ret)
		return ret;

	for (k_cur = k_start; k_cur < k_end; k_cur += PAGE_SIZE) {
		pmd_t *pmd = pmd_off_k(k_cur);
		pte_t pte = pfn_pte(PHYS_PFN(phys + k_cur - k_start), PAGE_KERNEL);

		__set_pte_at(&init_mm, k_cur, pte_offset_kernel(pmd, k_cur), pte, 0);
	}
	flush_tlb_kernel_range(k_start, k_end);
	memset(kasan_mem_to_shadow(start), 0, k_end - k_start);

While the __weak function that you confirmed working is:

	ret = kasan_init_shadow_page_tables(k_start, k_end);
	if (ret)
		return ret;

	block = memblock_alloc(k_end - k_start, PAGE_SIZE);
	if (!block)
		return -ENOMEM;

	for (k_cur = k_start & PAGE_MASK; k_cur < k_end; k_cur += PAGE_SIZE) {
		pmd_t *pmd = pmd_off_k(k_cur);
		void *va = block + k_cur - k_start;
		pte_t pte = pfn_pte(PHYS_PFN(__pa(va)), PAGE_KERNEL);

		__set_pte_at(&init_mm, k_cur, pte_offset_kernel(pmd, k_cur), pte, 0);
	}
	flush_tlb_kernel_range(k_start, k_end);

I'm having hard time to understand what's could be wrong at the first place.

Could you try following change:

arch/powerpc/mm/kasan/book3s_32.c +1 -1
index 9954b7a3b7ae..e04f21908c6a 100644
@@ -38,7 +38,7 @@ int __init kasan_init_region(void *start, size_t size)
 
   	if (k_nobat < k_end) {
   		phys = memblock_phys_alloc_range(k_end - k_nobat, PAGE_SIZE, 0,
-						 MEMBLOCK_ALLOC_ANYWHERE);
+						 MEMBLOCK_ALLOC_ACCESSIBLE);
   		if (!phys)
   			return -ENOMEM;
   	}

And also that one:

arch/powerpc/mm/kasan/init_32.c +3 -0
index a70828a6d935..bc1c075489f4 100644
@@ -84,6 +84,9 @@ kasan_update_early_region(unsigned long k_start,
 unsigned long k_end, pte_t pte)
   {
   	unsigned long k_cur;
 
+	if (k_start == k_end)
+		return;
+
   	for (k_cur = k_start; k_cur != k_end; k_cur += PAGE_SIZE) {
   		pmd_t *pmd = pmd_off_k(k_cur);
Quoted 4 lines
   		pte_t *ptep = pte_offset_kernel(pmd, k_cur);

I tested the two vmlinux you sent me offlist, they both start without problem on QEMU.

Regarding the use of BATs, in fact a shift of 23 is still not enough to get free BATs for KASAN. But at least it allows you to map all linear mem with BATS whereas a shift of 22 would require 9 BATs :

With shift 22 you have BATs with size : 4+4+8+16+32+64+128+256+256 With shift 23 you have BATs with size : 8+8+16+32+64+128+256+256

So lets forget that for the moment, allthought you may try with CONFIG_STRICT_KERNEL_RWX, in that case you should have enough BATs.

But lets try to refocus on the real problem.

In your last mail you say you tried with all patches. Did it include the two above changes ?

If not can you perform the tests with those two changes in addition, first one by one then both together depending on the result ?

Many thanks for your help and perseverance
Christophe

#32 lore raw
Quoted 124 lines · On Thu, 14 Sep 2023 04:54:17 +0000 Christophe Leroy…

Le 12/09/2023 à 19:39, Christophe Leroy a écrit :

Quoted 123 lines

Le 12/09/2023 à 17:59, Erhard Furtner a écrit :

Quoted 21 lines

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table
mapin_ram:125
mmu_mapin_ram:169 0 30000000 1400000 2000000
__mmu_mapin_ram:146 0 1400000
__mmu_mapin_ram:155 1400000
__mmu_mapin_ram:146 1400000 30000000
__mmu_mapin_ram:155 20000000
__mapin_ram_chunk:107 20000000 30000000
__mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
ioremap() called early from btext_map+0x64/0xdc. Use early_ioremap() instead
Linux version 6.6.0-rc1-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #5 SMP Tue Sep 12 16:50:47 CEST 2023
kasan_init_region: c0000000 30000000 f8000000 fe000000
kasan_init_region: loop f8000000 fe000000

Quoted 1 line

So I get no "kasan_init_region: setbat" line and don't reach "KASAN init done".

Ah ok, maybe your CPU only has 4 BATs and they are all used, following change would tell us.

arch/powerpc/mm/book3s32/mmu.c +1 -0
index 850783cfa9c7..bd26767edce7 100644
@@ -86,6 +86,7 @@ int __init find_free_bat(void)
   		if (!(bat[1].batu & 3))
   			return b;
   	}
+	pr_err("NO FREE BAT (%d)\n", n);
   	return -1;
   }
 

Or you have 8 BATs in which case it's an alignment problem, you need to increase CONFIG_DATA_SHIFT to 23, for that you need CONFIG_ADVANCED and CONFIG_DATA_SHIFT_BOOL

But regardless of that there is a problem we need to find out, because it should work without BATs.

As the BATs allocation fails, it falls back to :

	phys = memblock_phys_alloc_range(k_end - k_start, PAGE_SIZE, 0,
						 MEMBLOCK_ALLOC_ANYWHERE);
		if (!phys)
			return -ENOMEM;
	}

	ret = kasan_init_shadow_page_tables(k_start, k_end);
	if (ret)
		return ret;

	for (k_cur = k_start; k_cur < k_end; k_cur += PAGE_SIZE) {
		pmd_t *pmd = pmd_off_k(k_cur);
		pte_t pte = pfn_pte(PHYS_PFN(phys + k_cur - k_start), PAGE_KERNEL);

		__set_pte_at(&init_mm, k_cur, pte_offset_kernel(pmd, k_cur), pte, 0);
	}
	flush_tlb_kernel_range(k_start, k_end);
	memset(kasan_mem_to_shadow(start), 0, k_end - k_start);

While the __weak function that you confirmed working is:

	ret = kasan_init_shadow_page_tables(k_start, k_end);
	if (ret)
		return ret;

	block = memblock_alloc(k_end - k_start, PAGE_SIZE);
	if (!block)
		return -ENOMEM;

	for (k_cur = k_start & PAGE_MASK; k_cur < k_end; k_cur += PAGE_SIZE) {
		pmd_t *pmd = pmd_off_k(k_cur);
		void *va = block + k_cur - k_start;
		pte_t pte = pfn_pte(PHYS_PFN(__pa(va)), PAGE_KERNEL);

		__set_pte_at(&init_mm, k_cur, pte_offset_kernel(pmd, k_cur), pte, 0);
	}
	flush_tlb_kernel_range(k_start, k_end);

I'm having hard time to understand what's could be wrong at the first place.

Could you try following change:

arch/powerpc/mm/kasan/book3s_32.c +1 -1
index 9954b7a3b7ae..e04f21908c6a 100644
@@ -38,7 +38,7 @@ int __init kasan_init_region(void *start, size_t size)
 
   	if (k_nobat < k_end) {
   		phys = memblock_phys_alloc_range(k_end - k_nobat, PAGE_SIZE, 0,
-						 MEMBLOCK_ALLOC_ANYWHERE);
+						 MEMBLOCK_ALLOC_ACCESSIBLE);
   		if (!phys)
   			return -ENOMEM;
   	}

And also that one:

arch/powerpc/mm/kasan/init_32.c +0 -0
index a70828a6d935..bc1c075489f4 100644
@@ -84,6 +84,9 @@ kasan_update_early_region(unsigned long k_start,
 unsigned long k_end, pte_t pte)
   {
   	unsigned long k_cur;
 
Quoted 12 lines
Quoted 9 lines
arch/powerpc/mm/kasan/init_32.c +3 -0
+	if (k_start == k_end)
+		return;
+
   	for (k_cur = k_start; k_cur != k_end; k_cur += PAGE_SIZE) {
   		pmd_t *pmd = pmd_off_k(k_cur);
   		pte_t *ptep = pte_offset_kernel(pmd, k_cur);

I tested the two vmlinux you sent me offlist, they both start without problem on QEMU.

For me no problems show up on QEMU either. But QEMU does not seem able to mimic my G4 DPs configuration. That would be a dual CPU G4 + SMP config.

Quoted 2 lines

So lets forget that for the moment, allthought you may try with CONFIG_STRICT_KERNEL_RWX, in that case you should have enough BATs.

CONFIG_STRICT_KERNEL_RWX=y was enabled all along on my kernel .config. But for comparison I disabled it. If I disable STRICT_KERNEL_RWX I get no output about BATs whatsoever. Details below.

Quoted 5 lines

In your last mail you say you tried with all patches. Did it include the two above changes ?

If not can you perform the tests with those two changes in addition, first one by one then both together depending on the result ?

I think I did apply both but I re-did the checks just to be sure. For my 'all patches applied' config please check the attached git diff.

dmesg with patch 1 "MEMBLOCK_ALLOC_ACCESSIBLE);" applied:

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table
mapin_ram:125
mmu_mapin_ram:170 0 30000000 1400000 2000000
__mmu_mapin_ram:147 0 1400000
__mmu_mapin_ram:156 1400000
__mmu_mapin_ram:147 1400000 30000000
NO FREE BAT (8)
__mmu_mapin_ram:156 20000000
__mapin_ram_chunk:107 20000000 30000000
__mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
ioremap() called early from btext_map+0x64/0xdc. Use early_ioremap() instead
Linux version 6.6.0-rc1-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #23 SMP Thu Sep 14 13:05:23 CEST 2023
kasan_init_region: c0000000 30000000 f8000000 fe000000
NO FREE BAT (8)
kasan_init_region: loop f8000000 fe000000

dmesg with patch 2 "if (k_start == k_end) return;" applied:

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table
mapin_ram:125
mmu_mapin_ram:170 0 30000000 1400000 2000000
__mmu_mapin_ram:147 0 1400000
__mmu_mapin_ram:156 1400000
__mmu_mapin_ram:147 1400000 30000000
NO FREE BAT (8)
__mmu_mapin_ram:156 20000000
__mapin_ram_chunk:107 20000000 30000000
__mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:132
kasan_mmu_init:135 0
kasan_mmu_init:140
ioremap() called early from btext_map+0x64/0xdc. Use early_ioremap() instead
Linux version 6.6.0-rc1-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #23 SMP Thu Sep 14 13:05:23 CEST 2023
kasan_init_region: c0000000 30000000 f8000000 fe000000
NO FREE BAT (8)
kasan_init_region: loop f8000000 fe000000

dmesg with both KASAN patches applied:

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table
mapin_ram:125
mmu_mapin_ram:170 0 30000000 1400000 2000000
__mmu_mapin_ram:147 0 1400000
__mmu_mapin_ram:156 1400000
__mmu_mapin_ram:147 1400000 30000000
NO FREE BAT (8)
__mmu_mapin_ram:156 20000000
__mapin_ram_chunk:107 20000000 30000000
__mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:132
kasan_mmu_init:135 0
kasan_mmu_init:140
ioremap() called early from btext_map+0x64/0xdc. Use early_ioremap() instead
Linux version 6.6.0-rc1-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #23 SMP Thu Sep 14 13:05:23 CEST 2023
kasan_init_region: c0000000 30000000 f8000000 fe000000
NO FREE BAT (8)
kasan_init_region: loop f8000000 fe000000

dmesg with both KASAN patches and STRICT_KERNEL_RWX=n applied:

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table mapin_ram:125
mmu_mapin_ram:170 0 30000000 1400000 2000000 __mmu_mapin_ram:147 0 1400000 __mmu_mapin_ram:156 1400000 __mmu_mapin_ram:147 1400000 30000000 __mmu_mapin_ram:156 20000000 __mapin_ram_chunk:107 20000000 30000000 __mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:132
kasan_mmu_init:135 0
kasan_mmu_init:140

Quoted 2 lines

Many thanks for your help and perseverance
Christophe

You're welcome! Same to you! :)

Regards,
Erhard

arch/powerpc/kernel/btext.c +2 -0
index 19e46fd623b0..ec989e1011f0 100644
@@ -126,7 +126,9 @@ void __init btext_setup_display(int width, int height, int depth, int pitch,
 
 void __init btext_unmap(void)
 {
+	pr_info("%s:%d\n", __func__, __LINE__);
 	boot_text_mapped = 0;
+	pr_info("%s:%d\n", __func__, __LINE__);
 }
 
 /* Here's a small text engine to use during early boot
arch/powerpc/kernel/traps.c +0 -2
index eeff136b83d9..341a0635e131 100644
@@ -198,8 +198,6 @@ static unsigned long oops_begin(struct pt_regs *regs)
 	die_owner = cpu;
 	console_verbose();
 	bust_spinlocks(1);
-	if (machine_is(powermac))
-		pmac_backlight_unblank();
 	return flags;
 }
 NOKPROBE_SYMBOL(oops_begin);
arch/powerpc/mm/book3s32/kuap.c +5 -0
index 3a8815555a48..79b9fb0adfb4 100644
@@ -5,18 +5,23 @@
 
 void setup_kuap(bool disabled)
 {
+	pr_info("%s:%d\n", __func__, __LINE__);
 	if (!disabled) {
+		pr_info("%s:%d\n", __func__, __LINE__);
 		update_user_segments(mfsr(0) | SR_KS);
 		isync();        /* Context sync required after mtsr() */
 		init_mm.context.sr0 |= SR_KS;
 		current->thread.sr0 |= SR_KS;
 	}
+	pr_info("%s:%d\n", __func__, __LINE__);
 
 	if (smp_processor_id() != boot_cpuid)
 		return;
 
+	pr_info("%s:%d\n", __func__, __LINE__);
 	if (disabled)
 		cur_cpu_spec->mmu_features &= ~MMU_FTR_KUAP;
 	else
 		pr_info("Activating Kernel Userspace Access Protection\n");
+	pr_info("%s:%d\n", __func__, __LINE__);
 }
arch/powerpc/mm/book3s32/mmu.c +4 -0
index 850783cfa9c7..65a3bb4213cd 100644
@@ -86,6 +86,7 @@ int __init find_free_bat(void)
 		if (!(bat[1].batu & 3))
 			return b;
 	}
+	pr_err("NO FREE BAT (%d)\n", n);
 	return -1;
 }
 
@@ -143,6 +144,7 @@ static unsigned long __init __mmu_mapin_ram(unsigned long base, unsigned long to
 {
 	int idx;
 
+	pr_info("%s:%d %lx %lx\n", __func__, __LINE__, base, top);
 	while ((idx = find_free_bat()) != -1 && base != top) {
 		unsigned int size = bat_block_size(base, top);
 
@@ -151,6 +153,7 @@ static unsigned long __init __mmu_mapin_ram(unsigned long base, unsigned long to
 		setbat(idx, PAGE_OFFSET + base, base, size, PAGE_KERNEL_X);
 		base += size;
 	}
+	pr_info("%s:%d %lx\n", __func__, __LINE__, base);
 
 	return base;
 }
@@ -164,6 +167,7 @@ unsigned long __init mmu_mapin_ram(unsigned long base, unsigned long top)
 	size = roundup_pow_of_two((unsigned long)_einittext - PAGE_OFFSET);
 	setibat(0, PAGE_OFFSET, 0, size, PAGE_KERNEL_X);
 
+	pr_info("%s:%d %lx %lx %lx %lx\n", __func__, __LINE__, base, top, border, size);
 	if (debug_pagealloc_enabled_or_kfence()) {
 		pr_debug_once("Read-Write memory mapped without BATs\n");
 		if (base >= border)
arch/powerpc/mm/init_32.c +5 -5
index d8adc452f431..54a45eced552 100644
@@ -117,11 +117,6 @@ void __init MMU_init(void)
 	if (ppc_md.progress)
 		ppc_md.progress("MMU:exit", 0x211);
 
-	/* From now on, btext is no longer BAT mapped if it was at all */
-#ifdef CONFIG_BOOTX_TEXT
-	btext_unmap();
-#endif
-
 	kasan_mmu_init();
 
 	setup_kup();
@@ -130,4 +125,9 @@ void __init MMU_init(void)
 
 	/* Shortly after that, the entire linear mapping will be available */
 	memblock_set_current_limit(lowmem_end_addr);
+
+	/* From now on, btext is no longer BAT mapped if it was at all */
+#ifdef CONFIG_BOOTX_TEXT
+	btext_map();
+#endif
 }
arch/powerpc/mm/kasan/book3s_32.c +4 -1
index 450a67ef0bbe..e04f21908c6a 100644
@@ -15,6 +15,7 @@ int __init kasan_init_region(void *start, size_t size)
 	phys_addr_t phys;
 	int ret;
 
+	pr_err("%s: %px %x %lx %lx\n", __func__, start, size, k_start, k_end);
 	while (k_nobat < k_end) {
 		unsigned int k_size = bat_block_size(k_nobat, k_end);
 		int idx = find_free_bat();
@@ -28,6 +29,7 @@ int __init kasan_init_region(void *start, size_t size)
 		if (!phys)
 			break;
 
+		pr_err("%s: setbat %d %lx %x %x\n", __func__, idx, k_nobat, phys, k_size);
 		setbat(idx, k_nobat, phys, k_size, PAGE_KERNEL);
 		k_nobat += k_size;
 	}
@@ -36,7 +38,7 @@ int __init kasan_init_region(void *start, size_t size)
 
 	if (k_nobat < k_end) {
 		phys = memblock_phys_alloc_range(k_end - k_nobat, PAGE_SIZE, 0,
-						 MEMBLOCK_ALLOC_ANYWHERE);
+						 MEMBLOCK_ALLOC_ACCESSIBLE);
 		if (!phys)
 			return -ENOMEM;
 	}
@@ -47,6 +49,7 @@ int __init kasan_init_region(void *start, size_t size)
 
 	kasan_update_early_region(k_start, k_nobat, __pte(0));
 
+	pr_err("%s: loop %lx %lx\n", __func__, k_nobat, k_end);
 	for (k_cur = k_nobat; k_cur < k_end; k_cur += PAGE_SIZE) {
 		pmd_t *pmd = pmd_off_k(k_cur);
 		pte_t pte = pfn_pte(PHYS_PFN(phys + k_cur - k_nobat), PAGE_KERNEL);
arch/powerpc/mm/kasan/init_32.c +6 -0
index a70828a6d935..d734e0e74942 100644
@@ -84,6 +84,9 @@ kasan_update_early_region(unsigned long k_start, unsigned long k_end, pte_t pte)
 {
 	unsigned long k_cur;
 
+	if (k_start == k_end)
+		return;
+
 	for (k_cur = k_start; k_cur != k_end; k_cur += PAGE_SIZE) {
 		pmd_t *pmd = pmd_off_k(k_cur);
 		pte_t *ptep = pte_offset_kernel(pmd, k_cur);
@@ -126,12 +129,15 @@ void __init kasan_mmu_init(void)
 {
 	int ret;
 
+	pr_info("%s:%d\n", __func__, __LINE__);
 	if (early_mmu_has_feature(MMU_FTR_HPTE_TABLE)) {
 		ret = kasan_init_shadow_page_tables(KASAN_SHADOW_START, KASAN_SHADOW_END);
+		pr_info("%s:%d %d\n", __func__, __LINE__, ret);
 
 		if (ret)
 			panic("kasan: kasan_init_shadow_page_tables() failed");
 	}
+	pr_info("%s:%d\n", __func__, __LINE__);
 }
 
 void __init kasan_init(void)
arch/powerpc/mm/pgtable_32.c +4 -0
index 5c02fd08d61e..d75084f67d9b 100644
@@ -104,6 +104,7 @@ static void __init __mapin_ram_chunk(unsigned long offset, unsigned long top)
 	phys_addr_t p;
 	bool ktext;
 
+	pr_info("%s:%d %lx %lx\n", __func__, __LINE__, offset, top);
 	s = offset;
 	v = PAGE_OFFSET + s;
 	p = memstart_addr + s;
@@ -113,6 +114,7 @@ static void __init __mapin_ram_chunk(unsigned long offset, unsigned long top)
 		v += PAGE_SIZE;
 		p += PAGE_SIZE;
 	}
+	pr_info("%s:%d\n", __func__, __LINE__);
 }
 
 void __init mapin_ram(void)
@@ -120,6 +122,7 @@ void __init mapin_ram(void)
 	phys_addr_t base, end;
 	u64 i;
 
+	pr_info("%s:%d\n", __func__, __LINE__);
 	for_each_mem_range(i, &base, &end) {
 		phys_addr_t top = min(end, total_lowmem);
 
@@ -128,6 +131,7 @@ void __init mapin_ram(void)
 		base = mmu_mapin_ram(base, top);
 		__mapin_ram_chunk(base, top);
 	}
+	pr_info("%s:%d\n", __func__, __LINE__);
 }
 
 void mark_initmem_nx(void)
#33 lore raw
In reply to Christophe Leroy
Quoted 149 lines · On Thu, 14 Sep 2023 04:54:17 +0000 Christophe Leroy…

Le 12/09/2023 à 19:39, Christophe Leroy a écrit :

Quoted 128 lines

Le 12/09/2023 à 17:59, Erhard Furtner a écrit :

Quoted 21 lines

printk: bootconsole [udbg0] enabled
Total memory = 2048MB; using 4096kB for hash table
mapin_ram:125
mmu_mapin_ram:169 0 30000000 1400000 2000000
__mmu_mapin_ram:146 0 1400000
__mmu_mapin_ram:155 1400000
__mmu_mapin_ram:146 1400000 30000000
__mmu_mapin_ram:155 20000000
__mapin_ram_chunk:107 20000000 30000000
__mapin_ram_chunk:117
mapin_ram:134
kasan_mmu_init:129
kasan_mmu_init:132 0
kasan_mmu_init:137
ioremap() called early from btext_map+0x64/0xdc. Use early_ioremap() instead
Linux version 6.6.0-rc1-PMacG4-dirty (root@T1000) (gcc (Gentoo 12.3.1_p20230526 p2) 12.3.1 20230526, GNU ld (Gentoo 2.40 p7) 2.40.0) #5 SMP Tue Sep 12 16:50:47 CEST 2023
kasan_init_region: c0000000 30000000 f8000000 fe000000
kasan_init_region: loop f8000000 fe000000

Quoted 1 line

So I get no "kasan_init_region: setbat" line and don't reach "KASAN init done".

Ah ok, maybe your CPU only has 4 BATs and they are all used, following change would tell us.

arch/powerpc/mm/book3s32/mmu.c +1 -0
index 850783cfa9c7..bd26767edce7 100644
@@ -86,6 +86,7 @@ int __init find_free_bat(void)
   		if (!(bat[1].batu & 3))
   			return b;
   	}
+	pr_err("NO FREE BAT (%d)\n", n);
   	return -1;
   }
 

Or you have 8 BATs in which case it's an alignment problem, you need to increase CONFIG_DATA_SHIFT to 23, for that you need CONFIG_ADVANCED and CONFIG_DATA_SHIFT_BOOL

But regardless of that there is a problem we need to find out, because it should work without BATs.

As the BATs allocation fails, it falls back to :

	phys = memblock_phys_alloc_range(k_end - k_start, PAGE_SIZE, 0,
						 MEMBLOCK_ALLOC_ANYWHERE);
		if (!phys)
			return -ENOMEM;
	}

	ret = kasan_init_shadow_page_tables(k_start, k_end);
	if (ret)
		return ret;

	for (k_cur = k_start; k_cur < k_end; k_cur += PAGE_SIZE) {
		pmd_t *pmd = pmd_off_k(k_cur);
		pte_t pte = pfn_pte(PHYS_PFN(phys + k_cur - k_start), PAGE_KERNEL);

		__set_pte_at(&init_mm, k_cur, pte_offset_kernel(pmd, k_cur), pte, 0);
	}
	flush_tlb_kernel_range(k_start, k_end);
	memset(kasan_mem_to_shadow(start), 0, k_end - k_start);

While the __weak function that you confirmed working is:

	ret = kasan_init_shadow_page_tables(k_start, k_end);
	if (ret)
		return ret;

	block = memblock_alloc(k_end - k_start, PAGE_SIZE);
	if (!block)
		return -ENOMEM;

	for (k_cur = k_start & PAGE_MASK; k_cur < k_end; k_cur += PAGE_SIZE) {
		pmd_t *pmd = pmd_off_k(k_cur);
		void *va = block + k_cur - k_start;
		pte_t pte = pfn_pte(PHYS_PFN(__pa(va)), PAGE_KERNEL);

		__set_pte_at(&init_mm, k_cur, pte_offset_kernel(pmd, k_cur), pte, 0);
	}
	flush_tlb_kernel_range(k_start, k_end);

I'm having hard time to understand what's could be wrong at the first place.

Could you try following change:

arch/powerpc/mm/kasan/book3s_32.c +1 -1
index 9954b7a3b7ae..e04f21908c6a 100644
@@ -38,7 +38,7 @@ int __init kasan_init_region(void *start, size_t size)
 
   	if (k_nobat < k_end) {
   		phys = memblock_phys_alloc_range(k_end - k_nobat, PAGE_SIZE, 0,
-						 MEMBLOCK_ALLOC_ANYWHERE);
+						 MEMBLOCK_ALLOC_ACCESSIBLE);
   		if (!phys)
   			return -ENOMEM;
   	}

And also that one:

arch/powerpc/mm/kasan/init_32.c +3 -0
index a70828a6d935..bc1c075489f4 100644
@@ -84,6 +84,9 @@ kasan_update_early_region(unsigned long k_start,
 unsigned long k_end, pte_t pte)
   {
   	unsigned long k_cur;
 
+	if (k_start == k_end)
+		return;
+
   	for (k_cur = k_start; k_cur != k_end; k_cur += PAGE_SIZE) {
   		pmd_t *pmd = pmd_off_k(k_cur);
Quoted 4 lines
   		pte_t *ptep = pte_offset_kernel(pmd, k_cur);

I tested the two vmlinux you sent me offlist, they both start without problem on QEMU.

Regarding the use of BATs, in fact a shift of 23 is still not enough to get free BATs for KASAN. But at least it allows you to map all linear mem with BATS whereas a shift of 22 would require 9 BATs :

With shift 22 you have BATs with size : 4+4+8+16+32+64+128+256+256 With shift 23 you have BATs with size : 8+8+16+32+64+128+256+256

So lets forget that for the moment, allthought you may try with CONFIG_STRICT_KERNEL_RWX, in that case you should have enough BATs.

But lets try to refocus on the real problem.

Quoted 8 lines

In your last mail you say you tried with all patches. Did it include the two above changes ?

If not can you perform the tests with those two changes in addition, first one by one then both together depending on the result ?

Many thanks for your help and perseverance
Christophe

Revisited this issue with kernel v6.8-rc6 on the same machine.

Now this strange KASAN cold boot issue is gone or at least I can no longer reproduce it. Be it with KASAN_OUTLINE or KASAN_INLINE, SMP boot works just fine on my G4 DP. Which is a good thing. :)

Regards,
Erhard