mm: cma: allocate cma areas bottom-up

With kaslr the kernel image is placed at a random place, so starting the bottom-up allocation with the kernel_end can result in an allocation failure and a warning like this one:

[    0.002920] hugetlb_cma: reserve 2048 MiB, up to 2048 MiB per node
[    0.002921] ------------[ cut here ]------------
[    0.002922] memblock: bottom-up allocation failed, memory hotremove may be affected
[    0.002937] WARNING: CPU: 0 PID: 0 at mm/memblock.c:332 memblock_find_in_range_node+0x178/0x25a
[    0.002956] Call Trace:
[    0.002961]  ? memblock_alloc_range_nid+0x8d/0x11e
[    0.002963]  ? cma_declare_contiguous_nid+0x2c4/0x38c
[    0.002964]  ? hugetlb_cma_reserve+0xdc/0x128
[    0.002968]  ? flush_tlb_one_kernel+0xc/0x20
[    0.002969]  ? native_set_fixmap+0x82/0xd0
[    0.002971]  ? flat_get_apic_id+0x5/0x10
[    0.002973]  ? register_lapic_address+0x8e/0x97
[    0.002975]  ? setup_arch+0x8a5/0xc3f
[    0.002978]  ? start_kernel+0x66/0x547
[    0.002980]  ? load_ucode_bsp+0x4c/0xcd
[    0.002982]  ? secondary_startup_64_no_verify+0xb0/0xbb
[    0.002986] random: get_random_bytes called from __warn+0xab/0x110 with crng_init=0

At the same time, the kernel image is protected with memblock_reserve(), so we can just start searching at PAGE_SIZE. In this case the bottom-up allocation has the same chances to success as a top-down allocation, so there is no reason to fallback in the case of a failure. All together it simplifies the logic.

Hi Roman,

Quoted 23 lines · Roman Gushchin

With kaslr the kernel image is placed at a random place, so starting the bottom-up allocation with the kernel_end can result in an allocation failure and a warning like this one:

[    0.002920] hugetlb_cma: reserve 2048 MiB, up to 2048 MiB per node
[    0.002921] ------------[ cut here ]------------
[    0.002922] memblock: bottom-up allocation failed, memory hotremove may be affected
[    0.002937] WARNING: CPU: 0 PID: 0 at mm/memblock.c:332 memblock_find_in_range_node+0x178/0x25a
[    0.002956] Call Trace:
[    0.002961]  ? memblock_alloc_range_nid+0x8d/0x11e
[    0.002963]  ? cma_declare_contiguous_nid+0x2c4/0x38c
[    0.002964]  ? hugetlb_cma_reserve+0xdc/0x128
[    0.002968]  ? flush_tlb_one_kernel+0xc/0x20
[    0.002969]  ? native_set_fixmap+0x82/0xd0
[    0.002971]  ? flat_get_apic_id+0x5/0x10
[    0.002973]  ? register_lapic_address+0x8e/0x97
[    0.002975]  ? setup_arch+0x8a5/0xc3f
[    0.002978]  ? start_kernel+0x66/0x547
[    0.002980]  ? load_ucode_bsp+0x4c/0xcd
[    0.002982]  ? secondary_startup_64_no_verify+0xb0/0xbb
[    0.002986] random: get_random_bytes called from __warn+0xab/0x110 with crng_init=0

Quoted 5 lines

At the same time, the kernel image is protected with memblock_reserve(), so we can just start searching at PAGE_SIZE. In this case the bottom-up allocation has the same chances to success as a top-down allocation, so there is no reason to fallback in the case of a failure. All together it simplifies the logic.

I figure out that it was introduced by
commit 79442ed189acb ("memblock.c: introduce bottom-up allocation mode")

According to this commit, The purpose of bottom up allocation is to allocate memory from the unhotpluggable node.

With kaslr the kernel image is placed at a random place, so starting the bottom-up allocation with the kernel_end can result in an allocation failure and a warning like this one:

[    0.002920] hugetlb_cma: reserve 2048 MiB, up to 2048 MiB per node
[    0.002921] ------------[ cut here ]------------
[    0.002922] memblock: bottom-up allocation failed, memory hotremove may be affected
[    0.002937] WARNING: CPU: 0 PID: 0 at mm/memblock.c:332 memblock_find_in_range_node+0x178/0x25a
[    0.002937] Modules linked in:
[    0.002939] CPU: 0 PID: 0 Comm: swapper Not tainted 5.10.0+ #1169
[    0.002940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-1.fc33 04/01/2014
[    0.002942] RIP: 0010:memblock_find_in_range_node+0x178/0x25a
[    0.002944] Code: e9 6d ff ff ff 48 85 c0 0f 85 da 00 00 00 80 3d 9b 35 df 00 00 75 15 48 c7 c7 c0 75 59 88 c6 05 8b 35 df 00 01 e8 25 8a fa ff <0f> 0b 48 c7 44 24 20 ff ff ff ff 44 89 e6 44 89 ea 48 c7 c1 70 5c
[    0.002945] RSP: 0000:ffffffff88803d18 EFLAGS: 00010086 ORIG_RAX: 0000000000000000
[    0.002947] RAX: 0000000000000000 RBX: 0000000240000000 RCX: 00000000ffffdfff
[    0.002948] RDX: 00000000ffffdfff RSI: 00000000ffffffea RDI: 0000000000000046
[    0.002948] RBP: 0000000100000000 R08: ffffffff88922788 R09: 0000000000009ffb
[    0.002949] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: 0000000000000000
[    0.002950] R13: 0000000000000000 R14: 0000000080000000 R15: 00000001fb42c000
[    0.002952] FS:  0000000000000000(0000) GS:ffffffff88f71000(0000) knlGS:0000000000000000
[    0.002953] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.002954] CR2: ffffa080fb401000 CR3: 00000001fa80a000 CR4: 00000000000406b0
[    0.002956] Call Trace:
[    0.002961]  ? memblock_alloc_range_nid+0x8d/0x11e
[    0.002963]  ? cma_declare_contiguous_nid+0x2c4/0x38c
[    0.002964]  ? hugetlb_cma_reserve+0xdc/0x128
[    0.002968]  ? flush_tlb_one_kernel+0xc/0x20
[    0.002969]  ? native_set_fixmap+0x82/0xd0
[    0.002971]  ? flat_get_apic_id+0x5/0x10
[    0.002973]  ? register_lapic_address+0x8e/0x97
[    0.002975]  ? setup_arch+0x8a5/0xc3f
[    0.002978]  ? start_kernel+0x66/0x547
[    0.002980]  ? load_ucode_bsp+0x4c/0xcd
[    0.002982]  ? secondary_startup_64_no_verify+0xb0/0xbb
[    0.002986] random: get_random_bytes called from __warn+0xab/0x110 with crng_init=0
[    0.002988] ---[ end trace f151227d0b39be70 ]---

At the same time, the kernel image is protected with memblock_reserve(), so we can just start searching at PAGE_SIZE. In this case the bottom-up allocation has the same chances to success as a top-down allocation, so there is no reason to fallback in the case of a failure. All together it simplifies the logic.

Signed-off-by Roman Gushchin <[email protected]>

---
 mm/memblock.c | 49 ++++++-------------------------------------------
 1 file changed, 6 insertions(+), 43 deletions(-)

mm/memblock.c +6 -43

index b68ee86788af..10bd7d1ef0f4 100644
@@ -275,14 +275,6 @@ __memblock_find_range_top_down(phys_addr_t start, phys_addr_t end,
  *
  * Find @size free area aligned to @align in the specified range and node.
  *
- * When allocation direction is bottom-up, the @start should be greater
- * than the end of the kernel image. Otherwise, it will be trimmed. The
- * reason is that we want the bottom-up allocation just near the kernel
- * image so it is highly likely that the allocated memory and the kernel
- * will reside in the same node.
- *
- * If bottom-up allocation failed, will try to allocate memory top-down.
- *
  * Return:
  * Found address on success, 0 on failure.
  */

@@ -291,8 +283,6 @@ static phys_addr_t __init_memblock memblock_find_in_range_node(phys_addr_t size,
 					phys_addr_t end, int nid,
 					enum memblock_flags flags)
 {
-	phys_addr_t kernel_end, ret;
-
 	/* pump up @end */
 	if (end == MEMBLOCK_ALLOC_ACCESSIBLE ||
 	    end == MEMBLOCK_ALLOC_KASAN)

@@ -301,40 +291,13 @@ static phys_addr_t __init_memblock memblock_find_in_range_node(phys_addr_t size,
 	/* avoid allocating the first page */
 	start = max_t(phys_addr_t, start, PAGE_SIZE);
 	end = max(start, end);
-	kernel_end = __pa_symbol(_end);
-
-	/*
-	 * try bottom-up allocation only when bottom-up mode
-	 * is set and @end is above the kernel image.
-	 */
-	if (memblock_bottom_up() && end > kernel_end) {
-		phys_addr_t bottom_up_start;
-
-		/* make sure we will allocate above the kernel */
-		bottom_up_start = max(start, kernel_end);
 
-		/* ok, try bottom-up allocation first */
-		ret = __memblock_find_range_bottom_up(bottom_up_start, end,
-						      size, align, nid, flags);
-		if (ret)
-			return ret;
-
-		/*
-		 * we always limit bottom-up allocation above the kernel,
-		 * but top-down allocation doesn't have the limit, so
-		 * retrying top-down allocation may succeed when bottom-up
-		 * allocation failed.
-		 *
-		 * bottom-up allocation is expected to be fail very rarely,
-		 * so we use WARN_ONCE() here to see the stack trace if
-		 * fail happens.
-		 */
-		WARN_ONCE(IS_ENABLED(CONFIG_MEMORY_HOTREMOVE),
-			  "memblock: bottom-up allocation failed, memory hotremove may be affected\n");
-	}
-
-	return __memblock_find_range_top_down(start, end, size, align, nid,
-					      flags);
+	if (memblock_bottom_up())
+		return __memblock_find_range_bottom_up(start, end, size, align,
+						       nid, flags);
+	else
+		return __memblock_find_range_top_down(start, end, size, align,
+						      nid, flags);
 }
 
 /**

Signature · 1 line

2.26.2

Quoted 1 line

Signed-off-by Roman Gushchin <[email protected]>

Reviewed-by Mike Rapoport <[email protected]>

Quoted 3 lines · Mike Rapoport

Quoted 1 line

Signed-off-by Roman Gushchin <[email protected]>

Reviewed-by Mike Rapoport <[email protected]>

I've seen a couple of spurious triggers of the WARN_ONCE() removed by this patch. This happens on some ppc64le bare metal (powernv) server machines with CONFIG_SWIOTLB=y and crashkernel=4G, as described in a candidate patch I posted to solve this issue in a different way:

https://lore.kernel.org/linuxppc-dev/[email protected]/

Since this patch solves that problem, is it possible to include it in the next feasible v5.11-rcX, with the following tag?

Fixes 8fabc623238e ("powerpc: Ensure that swiotlb buffer is allocated from low memory")

Quoted 11 lines · On Fri, 22 Jan 2021 01:37:14 -0300 Thiago Jung Bauermann…

Quoted 3 lines · Mike Rapoport

Quoted 1 line

Signed-off-by Roman Gushchin <[email protected]>

Reviewed-by Mike Rapoport <[email protected]>

I've seen a couple of spurious triggers of the WARN_ONCE() removed by this patch. This happens on some ppc64le bare metal (powernv) server machines with CONFIG_SWIOTLB=y and crashkernel=4G, as described in a candidate patch I posted to solve this issue in a different way:

Quoted 4 lines

https://lore.kernel.org/linuxppc-dev/[email protected]/

Since this patch solves that problem, is it possible to include it in the next feasible v5.11-rcX, with the following tag?

We could do this, if we're confident that this patch doesn't depend on [1/2] "mm: cma: allocate cma areas bottom-up"? I think it is...

Quoted 1 line

Fixes 8fabc623238e ("powerpc: Ensure that swiotlb buffer is allocated from low memory")

I added that.

Quoted 13 lines · Andrew Morton

Quoted 11 lines · On Fri, 22 Jan 2021 01:37:14 -0300 Thiago Jung Bauermann…

Quoted 3 lines · Mike Rapoport

Quoted 1 line

Signed-off-by Roman Gushchin <[email protected]>

Reviewed-by Mike Rapoport <[email protected]>

I've seen a couple of spurious triggers of the WARN_ONCE() removed by this patch. This happens on some ppc64le bare metal (powernv) server machines with CONFIG_SWIOTLB=y and crashkernel=4G, as described in a candidate patch I posted to solve this issue in a different way:

Quoted 6 lines

Quoted 4 lines

https://lore.kernel.org/linuxppc-dev/[email protected]/

Since this patch solves that problem, is it possible to include it in the next feasible v5.11-rcX, with the following tag?

We could do this,

Quoted 2 lines

if we're confident that this patch doesn't depend on [1/2] "mm: cma: allocate cma areas bottom-up"? I think it is...

A think it does not depend on cma bottom-up allocation, it's rather the other way around: without this CMA bottom-up allocation could fail with KASLR enabled.

Still, this patch may need updates to the way x86 does early reservations:

https://lore.kernel.org/lkml/[email protected]

Quoted 3 lines

Quoted 1 line

Fixes 8fabc623238e ("powerpc: Ensure that swiotlb buffer is allocated from low memory")

I added that.

Quoted 18 lines · Andrew Morton

Quoted 15 lines · On Fri, 22 Jan 2021 01:37:14 -0300 Thiago Jung Bauermann…

Quoted 3 lines · Mike Rapoport

Quoted 1 line

Signed-off-by Roman Gushchin <[email protected]>

Reviewed-by Mike Rapoport <[email protected]>

I've seen a couple of spurious triggers of the WARN_ONCE() removed by this patch. This happens on some ppc64le bare metal (powernv) server machines with CONFIG_SWIOTLB=y and crashkernel=4G, as described in a candidate patch I posted to solve this issue in a different way:

https://lore.kernel.org/linuxppc-dev/[email protected]/

Since this patch solves that problem, is it possible to include it in the next feasible v5.11-rcX, with the following tag?

Quoted 2 lines

We could do this, if we're confident that this patch doesn't depend on [1/2] "mm: cma: allocate cma areas bottom-up"? I think it is...

A think it does not depend on cma bottom-up allocation, it's rather the other way around: without this CMA bottom-up allocation could fail with KASLR enabled.

Still, this patch may need updates to the way x86 does early reservations:

https://lore.kernel.org/lkml/[email protected]

With kaslr the kernel image is placed at a random place, so starting the bottom-up allocation with the kernel_end can result in an allocation failure and a warning like this one:

[    0.002920] hugetlb_cma: reserve 2048 MiB, up to 2048 MiB per node
[    0.002921] ------------[ cut here ]------------
[    0.002922] memblock: bottom-up allocation failed, memory hotremove may be affected
[    0.002937] WARNING: CPU: 0 PID: 0 at mm/memblock.c:332 memblock_find_in_range_node+0x178/0x25a
[    0.002937] Modules linked in:
[    0.002939] CPU: 0 PID: 0 Comm: swapper Not tainted 5.10.0+ #1169
[    0.002940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-1.fc33 04/01/2014
[    0.002942] RIP: 0010:memblock_find_in_range_node+0x178/0x25a
[    0.002944] Code: e9 6d ff ff ff 48 85 c0 0f 85 da 00 00 00 80 3d 9b 35 df 00 00 75 15 48 c7 c7 c0 75 59 88 c6 05 8b 35 df 00 01 e8 25 8a fa ff <0f> 0b 48 c7 44 24 20 ff ff ff ff 44 89 e6 44 89 ea 48 c7 c1 70 5c
[    0.002945] RSP: 0000:ffffffff88803d18 EFLAGS: 00010086 ORIG_RAX: 0000000000000000
[    0.002947] RAX: 0000000000000000 RBX: 0000000240000000 RCX: 00000000ffffdfff
[    0.002948] RDX: 00000000ffffdfff RSI: 00000000ffffffea RDI: 0000000000000046
[    0.002948] RBP: 0000000100000000 R08: ffffffff88922788 R09: 0000000000009ffb
[    0.002949] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: 0000000000000000
[    0.002950] R13: 0000000000000000 R14: 0000000080000000 R15: 00000001fb42c000
[    0.002952] FS:  0000000000000000(0000) GS:ffffffff88f71000(0000) knlGS:0000000000000000
[    0.002953] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.002954] CR2: ffffa080fb401000 CR3: 00000001fa80a000 CR4: 00000000000406b0
[    0.002956] Call Trace:
[    0.002961]  ? memblock_alloc_range_nid+0x8d/0x11e
[    0.002963]  ? cma_declare_contiguous_nid+0x2c4/0x38c
[    0.002964]  ? hugetlb_cma_reserve+0xdc/0x128
[    0.002968]  ? flush_tlb_one_kernel+0xc/0x20
[    0.002969]  ? native_set_fixmap+0x82/0xd0
[    0.002971]  ? flat_get_apic_id+0x5/0x10
[    0.002973]  ? register_lapic_address+0x8e/0x97
[    0.002975]  ? setup_arch+0x8a5/0xc3f
[    0.002978]  ? start_kernel+0x66/0x547
[    0.002980]  ? load_ucode_bsp+0x4c/0xcd
[    0.002982]  ? secondary_startup_64_no_verify+0xb0/0xbb
[    0.002986] random: get_random_bytes called from __warn+0xab/0x110 with crng_init=0
[    0.002988] ---[ end trace f151227d0b39be70 ]---

At the same time, the kernel image is protected with memblock_reserve(), so we can just start searching at PAGE_SIZE. In this case the bottom-up allocation has the same chances to success as a top-down allocation, so there is no reason to fallback in the case of a failure. All together it simplifies the logic.

Signed-off-by Roman Gushchin <[email protected]>

Quoted 37 lines · Roman Gushchin

With kaslr the kernel image is placed at a random place, so starting the bottom-up allocation with the kernel_end can result in an allocation failure and a warning like this one:

[    0.002920] hugetlb_cma: reserve 2048 MiB, up to 2048 MiB per node
[    0.002921] ------------[ cut here ]------------
[    0.002922] memblock: bottom-up allocation failed, memory hotremove may be affected
[    0.002937] WARNING: CPU: 0 PID: 0 at mm/memblock.c:332 memblock_find_in_range_node+0x178/0x25a
[    0.002937] Modules linked in:
[    0.002939] CPU: 0 PID: 0 Comm: swapper Not tainted 5.10.0+ #1169
[    0.002940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-1.fc33 04/01/2014
[    0.002942] RIP: 0010:memblock_find_in_range_node+0x178/0x25a
[    0.002944] Code: e9 6d ff ff ff 48 85 c0 0f 85 da 00 00 00 80 3d 9b 35 df 00 00 75 15 48 c7 c7 c0 75 59 88 c6 05 8b 35 df 00 01 e8 25 8a fa ff <0f> 0b 48 c7 44 24 20 ff ff ff ff 44 89 e6 44 89 ea 48 c7 c1 70 5c
[    0.002945] RSP: 0000:ffffffff88803d18 EFLAGS: 00010086 ORIG_RAX: 0000000000000000
[    0.002947] RAX: 0000000000000000 RBX: 0000000240000000 RCX: 00000000ffffdfff
[    0.002948] RDX: 00000000ffffdfff RSI: 00000000ffffffea RDI: 0000000000000046
[    0.002948] RBP: 0000000100000000 R08: ffffffff88922788 R09: 0000000000009ffb
[    0.002949] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: 0000000000000000
[    0.002950] R13: 0000000000000000 R14: 0000000080000000 R15: 00000001fb42c000
[    0.002952] FS:  0000000000000000(0000) GS:ffffffff88f71000(0000) knlGS:0000000000000000
[    0.002953] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.002954] CR2: ffffa080fb401000 CR3: 00000001fa80a000 CR4: 00000000000406b0
[    0.002956] Call Trace:
[    0.002961]  ? memblock_alloc_range_nid+0x8d/0x11e
[    0.002963]  ? cma_declare_contiguous_nid+0x2c4/0x38c
[    0.002964]  ? hugetlb_cma_reserve+0xdc/0x128
[    0.002968]  ? flush_tlb_one_kernel+0xc/0x20
[    0.002969]  ? native_set_fixmap+0x82/0xd0
[    0.002971]  ? flat_get_apic_id+0x5/0x10
[    0.002973]  ? register_lapic_address+0x8e/0x97
[    0.002975]  ? setup_arch+0x8a5/0xc3f
[    0.002978]  ? start_kernel+0x66/0x547
[    0.002980]  ? load_ucode_bsp+0x4c/0xcd
[    0.002982]  ? secondary_startup_64_no_verify+0xb0/0xbb
[    0.002986] random: get_random_bytes called from __warn+0xab/0x110 with crng_init=0
[    0.002988] ---[ end trace f151227d0b39be70 ]---

Quoted 7 lines

At the same time, the kernel image is protected with memblock_reserve(), so we can just start searching at PAGE_SIZE. In this case the bottom-up allocation has the same chances to success as a top-down allocation, so there is no reason to fallback in the case of a failure. All together it simplifies the logic.

Signed-off-by Roman Gushchin <[email protected]>

Hi Roman, Thomas and other linux-mips folks,

Kamal and myself have been unable to boot v5.11 on MIPS since this commit, reverting it makes our MIPS platforms boot successfully. We do not see a warning like this one in the commit message, instead what happens appear to be a corrupted Device Tree which prevents the parsing of the "rdb" node and leading to the interrupt controllers not being registered, and the system eventually not booting.

The Device Tree is built-into the kernel image and resides at arch/mips/boot/dts/brcm/bcm97435svmb.dts.

Do you have any idea what could be wrong with MIPS specifically here?

Hi Florian,

Quoted 56 lines · Florian Fainelli

Quoted 37 lines · Roman Gushchin

With kaslr the kernel image is placed at a random place, so starting the bottom-up allocation with the kernel_end can result in an allocation failure and a warning like this one:

[    0.002920] hugetlb_cma: reserve 2048 MiB, up to 2048 MiB per node
[    0.002921] ------------[ cut here ]------------
[    0.002922] memblock: bottom-up allocation failed, memory hotremove may be affected
[    0.002937] WARNING: CPU: 0 PID: 0 at mm/memblock.c:332 memblock_find_in_range_node+0x178/0x25a
[    0.002937] Modules linked in:
[    0.002939] CPU: 0 PID: 0 Comm: swapper Not tainted 5.10.0+ #1169
[    0.002940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-1.fc33 04/01/2014
[    0.002942] RIP: 0010:memblock_find_in_range_node+0x178/0x25a
[    0.002944] Code: e9 6d ff ff ff 48 85 c0 0f 85 da 00 00 00 80 3d 9b 35 df 00 00 75 15 48 c7 c7 c0 75 59 88 c6 05 8b 35 df 00 01 e8 25 8a fa ff <0f> 0b 48 c7 44 24 20 ff ff ff ff 44 89 e6 44 89 ea 48 c7 c1 70 5c
[    0.002945] RSP: 0000:ffffffff88803d18 EFLAGS: 00010086 ORIG_RAX: 0000000000000000
[    0.002947] RAX: 0000000000000000 RBX: 0000000240000000 RCX: 00000000ffffdfff
[    0.002948] RDX: 00000000ffffdfff RSI: 00000000ffffffea RDI: 0000000000000046
[    0.002948] RBP: 0000000100000000 R08: ffffffff88922788 R09: 0000000000009ffb
[    0.002949] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: 0000000000000000
[    0.002950] R13: 0000000000000000 R14: 0000000080000000 R15: 00000001fb42c000
[    0.002952] FS:  0000000000000000(0000) GS:ffffffff88f71000(0000) knlGS:0000000000000000
[    0.002953] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.002954] CR2: ffffa080fb401000 CR3: 00000001fa80a000 CR4: 00000000000406b0
[    0.002956] Call Trace:
[    0.002961]  ? memblock_alloc_range_nid+0x8d/0x11e
[    0.002963]  ? cma_declare_contiguous_nid+0x2c4/0x38c
[    0.002964]  ? hugetlb_cma_reserve+0xdc/0x128
[    0.002968]  ? flush_tlb_one_kernel+0xc/0x20
[    0.002969]  ? native_set_fixmap+0x82/0xd0
[    0.002971]  ? flat_get_apic_id+0x5/0x10
[    0.002973]  ? register_lapic_address+0x8e/0x97
[    0.002975]  ? setup_arch+0x8a5/0xc3f
[    0.002978]  ? start_kernel+0x66/0x547
[    0.002980]  ? load_ucode_bsp+0x4c/0xcd
[    0.002982]  ? secondary_startup_64_no_verify+0xb0/0xbb
[    0.002986] random: get_random_bytes called from __warn+0xab/0x110 with crng_init=0
[    0.002988] ---[ end trace f151227d0b39be70 ]---

Quoted 7 lines

At the same time, the kernel image is protected with memblock_reserve(), so we can just start searching at PAGE_SIZE. In this case the bottom-up allocation has the same chances to success as a top-down allocation, so there is no reason to fallback in the case of a failure. All together it simplifies the logic.

Signed-off-by Roman Gushchin <[email protected]>

Hi Roman, Thomas and other linux-mips folks,

Kamal and myself have been unable to boot v5.11 on MIPS since this commit, reverting it makes our MIPS platforms boot successfully. We do not see a warning like this one in the commit message, instead what happens appear to be a corrupted Device Tree which prevents the parsing of the "rdb" node and leading to the interrupt controllers not being registered, and the system eventually not booting.

Quoted 4 lines

The Device Tree is built-into the kernel image and resides at arch/mips/boot/dts/brcm/bcm97435svmb.dts.

Do you have any idea what could be wrong with MIPS specifically here?

Apparently there is a memblock allocation in one of the functions called from arch_mem_init() between plat_mem_setup() and early_init_fdt_reserve_self().

If you have serial available that early we can try to track it down with forcing memblock_debug in mm/memblock.c to 1:

mm/memblock.c +1 -1

index afaefa8fc6ab..83034245f8d5 100644
@@ -151,7 +151,7 @@ static __refdata struct memblock_type *memblock_memory = &memblock.memory;
                        pr_info(fmt, ##__VA_ARGS__);                    \
        } while (0)
 
-static int memblock_debug __initdata_memblock;
+static int memblock_debug __initdata_memblock = 1;
 static bool system_has_some_mirror __initdata_memblock = false;
 static int memblock_can_resize __initdata_memblock;
 static int memblock_memory_in_slab __initdata_memblock = 0;

Regardless, I think that moving DT self reservation just after plat_mem_setup() is safe and it'll make things more robust.

arch/mips/kernel/setup.c +2 -0

index 279be0153f8b..f476b99a7bcd 100644
@@ -623,6 +623,8 @@ static void __init arch_mem_init(char **cmdline_p)
 {
 	/* call board setup routine */
 	plat_mem_setup();
+	early_init_fdt_reserve_self();
+	early_init_fdt_scan_reserved_mem();
 	memblock_set_bottom_up(true);
 
 	bootcmdline_init();

Hi Mike,

Quoted 21 lines · Mike Rapoport

Hi Florian,

Quoted 18 lines · Florian Fainelli

[...]

Hi Roman, Thomas and other linux-mips folks,

Kamal and myself have been unable to boot v5.11 on MIPS since this commit, reverting it makes our MIPS platforms boot successfully. We do not see a warning like this one in the commit message, instead what happens appear to be a corrupted Device Tree which prevents the parsing of the "rdb" node and leading to the interrupt controllers not being registered, and the system eventually not booting.

The Device Tree is built-into the kernel image and resides at arch/mips/boot/dts/brcm/bcm97435svmb.dts.

Do you have any idea what could be wrong with MIPS specifically here?

Quoted 4 lines

Apparently there is a memblock allocation in one of the functions called from arch_mem_init() between plat_mem_setup() and early_init_fdt_reserve_self().

[...]

[    0.000000] Linux version 5.11.0-g5695e5161974 (florian@localhost)
(mipsel-linux-gcc (GCC) 8.3.0, GNU ld (GNU Binutils) 2.32) #84 SMP Sun
Feb 28 10:01:50 PST 2021
[    0.000000] CPU0 revision is: 00025b00 (Broadcom BMIPS5200)
[    0.000000] FPU revision is: 00130001

[    0.000000] memblock_add: [0x00000000-0x0fffffff]
early_init_dt_scan_memory+0x160/0x1e0
[    0.000000] memblock_add: [0x20000000-0x4fffffff]
early_init_dt_scan_memory+0x160/0x1e0
[    0.000000] memblock_add: [0x90000000-0xcfffffff]

early_init_dt_scan_memory+0x160/0x1e0

[    0.000000] MIPS: machine is Broadcom BCM97435SVMB
[    0.000000] earlycon: ns16550a0 at MMIO32 0x10406b00 (options '')
[    0.000000] printk: bootconsole [ns16550a0] enabled

[    0.000000] memblock_reserve: [0x00aa7600-0x00aaa0a0]

setup_arch+0x128/0x69c

[    0.000000] memblock_reserve: [0x00010000-0x018313cf]

setup_arch+0x1f8/0x69c

[    0.000000] Initrd not found or empty - disabling initrd

[    0.000000] memblock_alloc_try_nid: 10913 bytes align=0x40 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x00001000-0x00003aa0]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32680 bytes align=0x4 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x00003aa4-0x0000ba4b]

memblock_alloc_range_nid+0xf8/0x198

[    0.000000] memblock_alloc_try_nid: 25 bytes align=0x4 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x0000ba4c-0x0000ba64]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_reserve: [0x0096a000-0x00969fff]
setup_arch+0x3fc/0x69c
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 setup_arch+0x4e0/0x69c
[    0.000000] memblock_reserve: [0x0000ba80-0x0000ba9f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 setup_arch+0x4e0/0x69c
[    0.000000] memblock_reserve: [0x0000bb00-0x0000bb1f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 setup_arch+0x4e0/0x69c
[    0.000000] memblock_reserve: [0x0000bb80-0x0000bb9f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Primary instruction cache 32kB, VIPT, 4-way, linesize 64
bytes.
[    0.000000] Primary data cache 32kB, 4-way, VIPT, no aliases,
linesize 32 bytes
[    0.000000] MIPS secondary cache 512kB, 8-way, linesize 128 bytes.
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0xffffffff fixrange_init+0x90/0xf4
[    0.000000] memblock_reserve: [0x0000c000-0x0000cfff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0xffffffff fixrange_init+0x90/0xf4
[    0.000000] memblock_reserve: [0x0000d000-0x0000dfff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0xffffffff fixrange_init+0x90/0xf4
[    0.000000] memblock_reserve: [0x0000e000-0x0000efff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x000000000fffffff]
[    0.000000]   HighMem  [mem 0x0000000010000000-0x00000000cfffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000000fffffff]
[    0.000000]   node   0: [mem 0x0000000020000000-0x000000004fffffff]
[    0.000000]   node   0: [mem 0x0000000090000000-0x00000000cfffffff]
[    0.000000] Initmem setup node 0 [mem
0x0000000000000000-0x00000000cfffffff]
[    0.000000] memblock_alloc_try_nid: 27262976 bytes align=0x80 nid=0
from=0x00000000 max_addr=0x00000000
alloc_node_mem_map.constprop.135+0x6c/0xc8
[    0.000000] memblock_reserve: [0x01831400-0x032313ff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=0
from=0x00000000 max_addr=0x00000000 setup_usemap+0x64/0x98
[    0.000000] memblock_reserve: [0x0000bc00-0x0000bc1f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 384 bytes align=0x80 nid=0
from=0x00000000 max_addr=0x00000000 setup_usemap+0x64/0x98
[    0.000000] memblock_reserve: [0x0000bc80-0x0000bdff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] MEMBLOCK configuration:
[    0.000000]  memory size = 0x80000000 reserved size = 0x0322f032
[    0.000000]  memory.cnt  = 0x3
[    0.000000]  memory[0x0]     [0x00000000-0x0fffffff], 0x10000000
bytes flags: 0x0
[    0.000000]  memory[0x1]     [0x20000000-0x4fffffff], 0x30000000
bytes flags: 0x0
[    0.000000]  memory[0x2]     [0x90000000-0xcfffffff], 0x40000000
bytes flags: 0x0
[    0.000000]  reserved.cnt  = 0xa
[    0.000000]  reserved[0x0]   [0x00001000-0x00003aa0], 0x00002aa1
bytes flags: 0x0
[    0.000000]  reserved[0x1]   [0x00003aa4-0x0000ba64], 0x00007fc1
bytes flags: 0x0
[    0.000000]  reserved[0x2]   [0x0000ba80-0x0000ba9f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x3]   [0x0000bb00-0x0000bb1f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x4]   [0x0000bb80-0x0000bb9f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x5]   [0x0000bc00-0x0000bc1f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x6]   [0x0000bc80-0x0000bdff], 0x00000180
bytes flags: 0x0
[    0.000000]  reserved[0x7]   [0x0000c000-0x0000efff], 0x00003000
bytes flags: 0x0
[    0.000000]  reserved[0x8]   [0x00010000-0x018313cf], 0x018213d0
bytes flags: 0x0
[    0.000000]  reserved[0x9]   [0x01831400-0x032313ff], 0x01a00000
bytes flags: 0x0
[    0.000000] memblock_alloc_try_nid: 30 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 start_kernel+0x12c/0x654
[    0.000000] memblock_reserve: [0x0000be00-0x0000be1d]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 30 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 start_kernel+0x150/0x654
[    0.000000] memblock_reserve: [0x0000be80-0x0000be9d]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_embed_first_chunk+0x3b0/0x884
[    0.000000] memblock_reserve: [0x0000f000-0x0000ffff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_embed_first_chunk+0x5a4/0x884
[    0.000000] memblock_reserve: [0x03231400-0x032323ff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 294912 bytes align=0x1000 nid=-1
from=0x01000000 max_addr=0x00000000 pcpu_dfl_fc_alloc+0x24/0x30
[    0.000000] memblock_reserve: [0x03233000-0x0327afff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_free: [0x03245000-0x03244fff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] memblock_free: [0x03257000-0x03256fff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] memblock_free: [0x03269000-0x03268fff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] memblock_free: [0x0327b000-0x0327afff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] percpu: Embedded 18 pages/cpu s50704 r0 d23024 u73728
[    0.000000] memblock_alloc_try_nid: 4 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x178/0x6ec
[    0.000000] memblock_reserve: [0x0000bf00-0x0000bf03]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x1a8/0x6ec
[    0.000000] memblock_reserve: [0x0000bf80-0x0000bf83]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 16 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x1dc/0x6ec
[    0.000000] memblock_reserve: [0x03232400-0x0323240f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 16 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x20c/0x6ec
[    0.000000] memblock_reserve: [0x03232480-0x0323248f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 128 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x558/0x6ec
[    0.000000] memblock_reserve: [0x03232500-0x0323257f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 92 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0x8c/0x294
[    0.000000] memblock_reserve: [0x03232580-0x032325db]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 768 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0xe0/0x294
[    0.000000] memblock_reserve: [0x03232600-0x032328ff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 772 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0x124/0x294
[    0.000000] memblock_reserve: [0x03232900-0x03232c03]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 192 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0x158/0x294
[    0.000000] memblock_reserve: [0x03232c80-0x03232d3f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_free: [0x0000f000-0x0000ffff]
pcpu_embed_first_chunk+0x838/0x884
[    0.000000] memblock_free: [0x03231400-0x032323ff]
pcpu_embed_first_chunk+0x850/0x884
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 523776
[    0.000000] Kernel command line: console=ttyS0,115200 earlycon
[    0.000000] memblock_alloc_try_nid: 131072 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 alloc_large_system_hash+0x1f8/0x33c
[    0.000000] memblock_reserve: [0x0327b000-0x0329afff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Dentry cache hash table entries: 32768 (order: 5, 131072
bytes, linear)
[    0.000000] memblock_alloc_try_nid: 65536 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 alloc_large_system_hash+0x1f8/0x33c
[    0.000000] memblock_reserve: [0x0329b000-0x032aafff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536

bytes, linear)

[    0.000000] memblock_reserve: [0x00000000-0x000003ff]

trap_init+0x70/0x4e8

[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 2045268K/2097152K available (8226K kernel code,
1070K rwdata, 1336K rodata, 13808K init, 260K bss, 51884K reserved, 0K
cma-reserved, 1835008K highmem)
[    0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay
is 25 jiffies.
[    0.000000] NR_IRQS: 256

[    0.000000] OF: Bad cell count for /rdb
[    0.000000] irq_bcm7038_l1: failed to remap intc L1 registers
[    0.000000] OF: of_irq_init: children remain, but no parents

[    0.000000] random: get_random_bytes called from
start_kernel+0x444/0x654 with crng_init=0
[    0.000000] sched_clock: 32 bits at 250 Hz, resolution 4000000ns,

wraps every 8589934590000000ns

and with your patch applied which unfortunately did not work we have the following:

[...]

[...]

Signature · 1 line

Florian

Hi folks,
What you've got here seems a more complicated problem than it could originally look like. Please, see my comments below.

(Note I've discarded some of the email logs, which of no interest to the discovered problem. Please also note that I haven't got any Broadcom hardware to test out a solution suggested below.)

Quoted 24 lines · Florian Fainelli

Hi Mike,

Quoted 21 lines · Mike Rapoport

Hi Florian,

Quoted 18 lines · Florian Fainelli

[...]

Hi Roman, Thomas and other linux-mips folks,

Kamal and myself have been unable to boot v5.11 on MIPS since this commit, reverting it makes our MIPS platforms boot successfully. We do not see a warning like this one in the commit message, instead what happens appear to be a corrupted Device Tree which prevents the parsing of the "rdb" node and leading to the interrupt controllers not being registered, and the system eventually not booting.

The Device Tree is built-into the kernel image and resides at arch/mips/boot/dts/brcm/bcm97435svmb.dts.

Do you have any idea what could be wrong with MIPS specifically here?

Most likely the problem you've discovered has been there for quite some time. The patch you are referring to just caused it to be triggered by extending the early allocation range. See before that patch was accepted the early memory allocations had been performed in the range:
[kernel_end, RAM_END].
The patch changed that, so the early allocations are done within [RAM_START + PAGE_SIZE, RAM_END].

In normal situations it's safe to do that as long as all the critical memory regions (including the memory residing a space below the kernel) have been reserved. But as soon as a memory with some critical structures haven't been reserved, the kernel may allocate it to be used for instance for early initializations with obviously unpredictable but most of the times unpleasant consequences.

Quoted 4 lines

Quoted 4 lines

Apparently there is a memblock allocation in one of the functions called from arch_mem_init() between plat_mem_setup() and early_init_fdt_reserve_self().

Mike, alas according to the log provided by Florian that's not the reason of the problem. Please, see my considerations below.

Quoted 8 lines

[...]

[    0.000000] Linux version 5.11.0-g5695e5161974 (florian@localhost)
(mipsel-linux-gcc (GCC) 8.3.0, GNU ld (GNU Binutils) 2.32) #84 SMP Sun
Feb 28 10:01:50 PST 2021
[    0.000000] CPU0 revision is: 00025b00 (Broadcom BMIPS5200)
[    0.000000] FPU revision is: 00130001

Quoted 6 lines

[    0.000000] memblock_add: [0x00000000-0x0fffffff]
early_init_dt_scan_memory+0x160/0x1e0
[    0.000000] memblock_add: [0x20000000-0x4fffffff]
early_init_dt_scan_memory+0x160/0x1e0
[    0.000000] memblock_add: [0x90000000-0xcfffffff]

early_init_dt_scan_memory+0x160/0x1e0

Here the memory has been added to the memblock allocator.

Quoted 6 lines

[    0.000000] MIPS: machine is Broadcom BCM97435SVMB
[    0.000000] earlycon: ns16550a0 at MMIO32 0x10406b00 (options '')
[    0.000000] printk: bootconsole [ns16550a0] enabled

[    0.000000] memblock_reserve: [0x00aa7600-0x00aaa0a0]

setup_arch+0x128/0x69c

Here the fdt memory has been reserved. (Note it's built into the kernel.)

Quoted 2 lines

[    0.000000] memblock_reserve: [0x00010000-0x018313cf]

setup_arch+0x1f8/0x69c

Here the kernel itself together with built-in dtb have been reserved. So far so good.

Quoted 11 lines

[    0.000000] Initrd not found or empty - disabling initrd

[    0.000000] memblock_alloc_try_nid: 10913 bytes align=0x40 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x00001000-0x00003aa0]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32680 bytes align=0x4 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x00003aa4-0x0000ba4b]

Quoted 1 line

memblock_alloc_range_nid+0xf8/0x198

The log above most likely belongs to the call-chain:
setup_arch()

+-> arch_mem_init()
    +-> device_tree_init() - BMIPS specific method
        +-> unflatten_and_copy_device_tree()

So to speak here we've copied the fdt from the original space [0x00aa7600-0x00aaa0a0] into [0x00001000-0x00003aa0] and unflattened it to [0x00003aa4-0x0000ba4b].

The problem is that a bit later the next call-chain is performed:
setup_arch()

+-> plat_smp_setup()
    +-> mp_ops->smp_setup(); - registered by prom_init()->register_bmips_smp_ops();
        +-> if (!board_ebase_setup)
                 board_ebase_setup = &bmips_ebase_setup;

So at the moment of the CPU traps initialization the bmips_ebase_setup() method is called. What trap_init() does isn't compatible with the allocation performed by the unflatten_and_copy_device_tree() method. See the next comment.

Quoted 171 lines

[    0.000000] memblock_alloc_try_nid: 25 bytes align=0x4 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x0000ba4c-0x0000ba64]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_reserve: [0x0096a000-0x00969fff]
setup_arch+0x3fc/0x69c
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 setup_arch+0x4e0/0x69c
[    0.000000] memblock_reserve: [0x0000ba80-0x0000ba9f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 setup_arch+0x4e0/0x69c
[    0.000000] memblock_reserve: [0x0000bb00-0x0000bb1f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 setup_arch+0x4e0/0x69c
[    0.000000] memblock_reserve: [0x0000bb80-0x0000bb9f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Primary instruction cache 32kB, VIPT, 4-way, linesize 64
bytes.
[    0.000000] Primary data cache 32kB, 4-way, VIPT, no aliases,
linesize 32 bytes
[    0.000000] MIPS secondary cache 512kB, 8-way, linesize 128 bytes.
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0xffffffff fixrange_init+0x90/0xf4
[    0.000000] memblock_reserve: [0x0000c000-0x0000cfff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0xffffffff fixrange_init+0x90/0xf4
[    0.000000] memblock_reserve: [0x0000d000-0x0000dfff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0xffffffff fixrange_init+0x90/0xf4
[    0.000000] memblock_reserve: [0x0000e000-0x0000efff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x000000000fffffff]
[    0.000000]   HighMem  [mem 0x0000000010000000-0x00000000cfffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000000fffffff]
[    0.000000]   node   0: [mem 0x0000000020000000-0x000000004fffffff]
[    0.000000]   node   0: [mem 0x0000000090000000-0x00000000cfffffff]
[    0.000000] Initmem setup node 0 [mem
0x0000000000000000-0x00000000cfffffff]
[    0.000000] memblock_alloc_try_nid: 27262976 bytes align=0x80 nid=0
from=0x00000000 max_addr=0x00000000
alloc_node_mem_map.constprop.135+0x6c/0xc8
[    0.000000] memblock_reserve: [0x01831400-0x032313ff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=0
from=0x00000000 max_addr=0x00000000 setup_usemap+0x64/0x98
[    0.000000] memblock_reserve: [0x0000bc00-0x0000bc1f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 384 bytes align=0x80 nid=0
from=0x00000000 max_addr=0x00000000 setup_usemap+0x64/0x98
[    0.000000] memblock_reserve: [0x0000bc80-0x0000bdff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] MEMBLOCK configuration:
[    0.000000]  memory size = 0x80000000 reserved size = 0x0322f032
[    0.000000]  memory.cnt  = 0x3
[    0.000000]  memory[0x0]     [0x00000000-0x0fffffff], 0x10000000
bytes flags: 0x0
[    0.000000]  memory[0x1]     [0x20000000-0x4fffffff], 0x30000000
bytes flags: 0x0
[    0.000000]  memory[0x2]     [0x90000000-0xcfffffff], 0x40000000
bytes flags: 0x0
[    0.000000]  reserved.cnt  = 0xa
[    0.000000]  reserved[0x0]   [0x00001000-0x00003aa0], 0x00002aa1
bytes flags: 0x0
[    0.000000]  reserved[0x1]   [0x00003aa4-0x0000ba64], 0x00007fc1
bytes flags: 0x0
[    0.000000]  reserved[0x2]   [0x0000ba80-0x0000ba9f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x3]   [0x0000bb00-0x0000bb1f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x4]   [0x0000bb80-0x0000bb9f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x5]   [0x0000bc00-0x0000bc1f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x6]   [0x0000bc80-0x0000bdff], 0x00000180
bytes flags: 0x0
[    0.000000]  reserved[0x7]   [0x0000c000-0x0000efff], 0x00003000
bytes flags: 0x0
[    0.000000]  reserved[0x8]   [0x00010000-0x018313cf], 0x018213d0
bytes flags: 0x0
[    0.000000]  reserved[0x9]   [0x01831400-0x032313ff], 0x01a00000
bytes flags: 0x0
[    0.000000] memblock_alloc_try_nid: 30 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 start_kernel+0x12c/0x654
[    0.000000] memblock_reserve: [0x0000be00-0x0000be1d]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 30 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 start_kernel+0x150/0x654
[    0.000000] memblock_reserve: [0x0000be80-0x0000be9d]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_embed_first_chunk+0x3b0/0x884
[    0.000000] memblock_reserve: [0x0000f000-0x0000ffff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_embed_first_chunk+0x5a4/0x884
[    0.000000] memblock_reserve: [0x03231400-0x032323ff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 294912 bytes align=0x1000 nid=-1
from=0x01000000 max_addr=0x00000000 pcpu_dfl_fc_alloc+0x24/0x30
[    0.000000] memblock_reserve: [0x03233000-0x0327afff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_free: [0x03245000-0x03244fff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] memblock_free: [0x03257000-0x03256fff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] memblock_free: [0x03269000-0x03268fff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] memblock_free: [0x0327b000-0x0327afff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] percpu: Embedded 18 pages/cpu s50704 r0 d23024 u73728
[    0.000000] memblock_alloc_try_nid: 4 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x178/0x6ec
[    0.000000] memblock_reserve: [0x0000bf00-0x0000bf03]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x1a8/0x6ec
[    0.000000] memblock_reserve: [0x0000bf80-0x0000bf83]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 16 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x1dc/0x6ec
[    0.000000] memblock_reserve: [0x03232400-0x0323240f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 16 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x20c/0x6ec
[    0.000000] memblock_reserve: [0x03232480-0x0323248f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 128 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x558/0x6ec
[    0.000000] memblock_reserve: [0x03232500-0x0323257f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 92 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0x8c/0x294
[    0.000000] memblock_reserve: [0x03232580-0x032325db]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 768 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0xe0/0x294
[    0.000000] memblock_reserve: [0x03232600-0x032328ff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 772 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0x124/0x294
[    0.000000] memblock_reserve: [0x03232900-0x03232c03]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 192 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0x158/0x294
[    0.000000] memblock_reserve: [0x03232c80-0x03232d3f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_free: [0x0000f000-0x0000ffff]
pcpu_embed_first_chunk+0x838/0x884
[    0.000000] memblock_free: [0x03231400-0x032323ff]
pcpu_embed_first_chunk+0x850/0x884
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 523776
[    0.000000] Kernel command line: console=ttyS0,115200 earlycon
[    0.000000] memblock_alloc_try_nid: 131072 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 alloc_large_system_hash+0x1f8/0x33c
[    0.000000] memblock_reserve: [0x0327b000-0x0329afff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Dentry cache hash table entries: 32768 (order: 5, 131072
bytes, linear)
[    0.000000] memblock_alloc_try_nid: 65536 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 alloc_large_system_hash+0x1f8/0x33c
[    0.000000] memblock_reserve: [0x0329b000-0x032aafff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536

Quoted 4 lines

bytes, linear)

[    0.000000] memblock_reserve: [0x00000000-0x000003ff]

trap_init+0x70/0x4e8

Most likely someplace here the corruption has happened. The log above has just reserved a memory for NMI/reset vectors:
arch/mips/kernel/traps.c: trap_init(void): Line 2373.

But then the board_ebase_setup() pointer is dereferenced and called, which has been initialized with bmips_ebase_setup() earlier and which overwrites the ebase variable with: 0x80001000 as this is CPU_BMIPS5000 CPU. So any further calls of the functions like set_handler()/set_except_vector()/set_vi_srs_handler()/etc may cause a corruption of the memory above 0x80001000, which as we have discovered belongs to fdt and unflattened device tree.

Quoted 11 lines

[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 2045268K/2097152K available (8226K kernel code,
1070K rwdata, 1336K rodata, 13808K init, 260K bss, 51884K reserved, 0K
cma-reserved, 1835008K highmem)
[    0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay
is 25 jiffies.
[    0.000000] NR_IRQS: 256

Quoted 3 lines

[    0.000000] OF: Bad cell count for /rdb
[    0.000000] irq_bcm7038_l1: failed to remap intc L1 registers
[    0.000000] OF: of_irq_init: children remain, but no parents

So here is the first time we have got the consequence of the corruption popped up. Luckily it's just the "Bad cells count" error. We could have got much less obvious log here up to getting a crash at some place further...

Quoted 10 lines

[    0.000000] random: get_random_bytes called from
start_kernel+0x444/0x654 with crng_init=0
[    0.000000] sched_clock: 32 bits at 250 Hz, resolution 4000000ns,

wraps every 8589934590000000ns

and with your patch applied which unfortunately did not work we have the following:

[...]

So a patch like this shall workaround the corruption:

Hi Serge,

Quoted 353 lines · Serge Semin

Hi folks,
What you've got here seems a more complicated problem than it could originally look like. Please, see my comments below.

(Note I've discarded some of the email logs, which of no interest to the discovered problem. Please also note that I haven't got any Broadcom hardware to test out a solution suggested below.)

Quoted 24 lines · Florian Fainelli

Hi Mike,

Quoted 21 lines · Mike Rapoport

Hi Florian,

Quoted 18 lines · Florian Fainelli

[...]

Hi Roman, Thomas and other linux-mips folks,

Kamal and myself have been unable to boot v5.11 on MIPS since this commit, reverting it makes our MIPS platforms boot successfully. We do not see a warning like this one in the commit message, instead what happens appear to be a corrupted Device Tree which prevents the parsing of the "rdb" node and leading to the interrupt controllers not being registered, and the system eventually not booting.

The Device Tree is built-into the kernel image and resides at arch/mips/boot/dts/brcm/bcm97435svmb.dts.

Do you have any idea what could be wrong with MIPS specifically here?

Most likely the problem you've discovered has been there for quite some time. The patch you are referring to just caused it to be triggered by extending the early allocation range. See before that patch was accepted the early memory allocations had been performed in the range:
[kernel_end, RAM_END].
The patch changed that, so the early allocations are done within [RAM_START + PAGE_SIZE, RAM_END].

In normal situations it's safe to do that as long as all the critical memory regions (including the memory residing a space below the kernel) have been reserved. But as soon as a memory with some critical structures haven't been reserved, the kernel may allocate it to be used for instance for early initializations with obviously unpredictable but most of the times unpleasant consequences.

Quoted 4 lines

Quoted 4 lines

Apparently there is a memblock allocation in one of the functions called from arch_mem_init() between plat_mem_setup() and early_init_fdt_reserve_self().

Mike, alas according to the log provided by Florian that's not the reason of the problem. Please, see my considerations below.

Quoted 8 lines

[...]

[    0.000000] Linux version 5.11.0-g5695e5161974 (florian@localhost)
(mipsel-linux-gcc (GCC) 8.3.0, GNU ld (GNU Binutils) 2.32) #84 SMP Sun
Feb 28 10:01:50 PST 2021
[    0.000000] CPU0 revision is: 00025b00 (Broadcom BMIPS5200)
[    0.000000] FPU revision is: 00130001

Quoted 6 lines

[    0.000000] memblock_add: [0x00000000-0x0fffffff]
early_init_dt_scan_memory+0x160/0x1e0
[    0.000000] memblock_add: [0x20000000-0x4fffffff]
early_init_dt_scan_memory+0x160/0x1e0
[    0.000000] memblock_add: [0x90000000-0xcfffffff]

early_init_dt_scan_memory+0x160/0x1e0

Here the memory has been added to the memblock allocator.

Quoted 6 lines

[    0.000000] MIPS: machine is Broadcom BCM97435SVMB
[    0.000000] earlycon: ns16550a0 at MMIO32 0x10406b00 (options '')
[    0.000000] printk: bootconsole [ns16550a0] enabled

[    0.000000] memblock_reserve: [0x00aa7600-0x00aaa0a0]

setup_arch+0x128/0x69c

Here the fdt memory has been reserved. (Note it's built into the kernel.)

Quoted 2 lines

[    0.000000] memblock_reserve: [0x00010000-0x018313cf]

setup_arch+0x1f8/0x69c

Here the kernel itself together with built-in dtb have been reserved. So far so good.

Quoted 11 lines

[    0.000000] Initrd not found or empty - disabling initrd

[    0.000000] memblock_alloc_try_nid: 10913 bytes align=0x40 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x00001000-0x00003aa0]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32680 bytes align=0x4 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x00003aa4-0x0000ba4b]

Quoted 1 line

memblock_alloc_range_nid+0xf8/0x198

The log above most likely belongs to the call-chain:
setup_arch()

+-> arch_mem_init()
    +-> device_tree_init() - BMIPS specific method
        +-> unflatten_and_copy_device_tree()

So to speak here we've copied the fdt from the original space [0x00aa7600-0x00aaa0a0] into [0x00001000-0x00003aa0] and unflattened it to [0x00003aa4-0x0000ba4b].

The problem is that a bit later the next call-chain is performed:
setup_arch()

+-> plat_smp_setup()
    +-> mp_ops->smp_setup(); - registered by prom_init()->register_bmips_smp_ops();
        +-> if (!board_ebase_setup)
                 board_ebase_setup = &bmips_ebase_setup;

So at the moment of the CPU traps initialization the bmips_ebase_setup() method is called. What trap_init() does isn't compatible with the allocation performed by the unflatten_and_copy_device_tree() method. See the next comment.

Quoted 171 lines

[    0.000000] memblock_alloc_try_nid: 25 bytes align=0x4 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x0000ba4c-0x0000ba64]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_reserve: [0x0096a000-0x00969fff]
setup_arch+0x3fc/0x69c
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 setup_arch+0x4e0/0x69c
[    0.000000] memblock_reserve: [0x0000ba80-0x0000ba9f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 setup_arch+0x4e0/0x69c
[    0.000000] memblock_reserve: [0x0000bb00-0x0000bb1f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 setup_arch+0x4e0/0x69c
[    0.000000] memblock_reserve: [0x0000bb80-0x0000bb9f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Primary instruction cache 32kB, VIPT, 4-way, linesize 64
bytes.
[    0.000000] Primary data cache 32kB, 4-way, VIPT, no aliases,
linesize 32 bytes
[    0.000000] MIPS secondary cache 512kB, 8-way, linesize 128 bytes.
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0xffffffff fixrange_init+0x90/0xf4
[    0.000000] memblock_reserve: [0x0000c000-0x0000cfff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0xffffffff fixrange_init+0x90/0xf4
[    0.000000] memblock_reserve: [0x0000d000-0x0000dfff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0xffffffff fixrange_init+0x90/0xf4
[    0.000000] memblock_reserve: [0x0000e000-0x0000efff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x000000000fffffff]
[    0.000000]   HighMem  [mem 0x0000000010000000-0x00000000cfffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000000fffffff]
[    0.000000]   node   0: [mem 0x0000000020000000-0x000000004fffffff]
[    0.000000]   node   0: [mem 0x0000000090000000-0x00000000cfffffff]
[    0.000000] Initmem setup node 0 [mem
0x0000000000000000-0x00000000cfffffff]
[    0.000000] memblock_alloc_try_nid: 27262976 bytes align=0x80 nid=0
from=0x00000000 max_addr=0x00000000
alloc_node_mem_map.constprop.135+0x6c/0xc8
[    0.000000] memblock_reserve: [0x01831400-0x032313ff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=0
from=0x00000000 max_addr=0x00000000 setup_usemap+0x64/0x98
[    0.000000] memblock_reserve: [0x0000bc00-0x0000bc1f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 384 bytes align=0x80 nid=0
from=0x00000000 max_addr=0x00000000 setup_usemap+0x64/0x98
[    0.000000] memblock_reserve: [0x0000bc80-0x0000bdff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] MEMBLOCK configuration:
[    0.000000]  memory size = 0x80000000 reserved size = 0x0322f032
[    0.000000]  memory.cnt  = 0x3
[    0.000000]  memory[0x0]     [0x00000000-0x0fffffff], 0x10000000
bytes flags: 0x0
[    0.000000]  memory[0x1]     [0x20000000-0x4fffffff], 0x30000000
bytes flags: 0x0
[    0.000000]  memory[0x2]     [0x90000000-0xcfffffff], 0x40000000
bytes flags: 0x0
[    0.000000]  reserved.cnt  = 0xa
[    0.000000]  reserved[0x0]   [0x00001000-0x00003aa0], 0x00002aa1
bytes flags: 0x0
[    0.000000]  reserved[0x1]   [0x00003aa4-0x0000ba64], 0x00007fc1
bytes flags: 0x0
[    0.000000]  reserved[0x2]   [0x0000ba80-0x0000ba9f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x3]   [0x0000bb00-0x0000bb1f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x4]   [0x0000bb80-0x0000bb9f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x5]   [0x0000bc00-0x0000bc1f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x6]   [0x0000bc80-0x0000bdff], 0x00000180
bytes flags: 0x0
[    0.000000]  reserved[0x7]   [0x0000c000-0x0000efff], 0x00003000
bytes flags: 0x0
[    0.000000]  reserved[0x8]   [0x00010000-0x018313cf], 0x018213d0
bytes flags: 0x0
[    0.000000]  reserved[0x9]   [0x01831400-0x032313ff], 0x01a00000
bytes flags: 0x0
[    0.000000] memblock_alloc_try_nid: 30 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 start_kernel+0x12c/0x654
[    0.000000] memblock_reserve: [0x0000be00-0x0000be1d]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 30 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 start_kernel+0x150/0x654
[    0.000000] memblock_reserve: [0x0000be80-0x0000be9d]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_embed_first_chunk+0x3b0/0x884
[    0.000000] memblock_reserve: [0x0000f000-0x0000ffff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_embed_first_chunk+0x5a4/0x884
[    0.000000] memblock_reserve: [0x03231400-0x032323ff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 294912 bytes align=0x1000 nid=-1
from=0x01000000 max_addr=0x00000000 pcpu_dfl_fc_alloc+0x24/0x30
[    0.000000] memblock_reserve: [0x03233000-0x0327afff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_free: [0x03245000-0x03244fff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] memblock_free: [0x03257000-0x03256fff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] memblock_free: [0x03269000-0x03268fff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] memblock_free: [0x0327b000-0x0327afff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] percpu: Embedded 18 pages/cpu s50704 r0 d23024 u73728
[    0.000000] memblock_alloc_try_nid: 4 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x178/0x6ec
[    0.000000] memblock_reserve: [0x0000bf00-0x0000bf03]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x1a8/0x6ec
[    0.000000] memblock_reserve: [0x0000bf80-0x0000bf83]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 16 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x1dc/0x6ec
[    0.000000] memblock_reserve: [0x03232400-0x0323240f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 16 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x20c/0x6ec
[    0.000000] memblock_reserve: [0x03232480-0x0323248f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 128 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x558/0x6ec
[    0.000000] memblock_reserve: [0x03232500-0x0323257f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 92 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0x8c/0x294
[    0.000000] memblock_reserve: [0x03232580-0x032325db]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 768 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0xe0/0x294
[    0.000000] memblock_reserve: [0x03232600-0x032328ff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 772 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0x124/0x294
[    0.000000] memblock_reserve: [0x03232900-0x03232c03]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 192 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0x158/0x294
[    0.000000] memblock_reserve: [0x03232c80-0x03232d3f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_free: [0x0000f000-0x0000ffff]
pcpu_embed_first_chunk+0x838/0x884
[    0.000000] memblock_free: [0x03231400-0x032323ff]
pcpu_embed_first_chunk+0x850/0x884
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 523776
[    0.000000] Kernel command line: console=ttyS0,115200 earlycon
[    0.000000] memblock_alloc_try_nid: 131072 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 alloc_large_system_hash+0x1f8/0x33c
[    0.000000] memblock_reserve: [0x0327b000-0x0329afff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Dentry cache hash table entries: 32768 (order: 5, 131072
bytes, linear)
[    0.000000] memblock_alloc_try_nid: 65536 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 alloc_large_system_hash+0x1f8/0x33c
[    0.000000] memblock_reserve: [0x0329b000-0x032aafff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536

Quoted 4 lines

bytes, linear)

[    0.000000] memblock_reserve: [0x00000000-0x000003ff]

trap_init+0x70/0x4e8

Most likely someplace here the corruption has happened. The log above has just reserved a memory for NMI/reset vectors:
arch/mips/kernel/traps.c: trap_init(void): Line 2373.

But then the board_ebase_setup() pointer is dereferenced and called, which has been initialized with bmips_ebase_setup() earlier and which overwrites the ebase variable with: 0x80001000 as this is CPU_BMIPS5000 CPU. So any further calls of the functions like set_handler()/set_except_vector()/set_vi_srs_handler()/etc may cause a corruption of the memory above 0x80001000, which as we have discovered belongs to fdt and unflattened device tree.

Quoted 11 lines

[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 2045268K/2097152K available (8226K kernel code,
1070K rwdata, 1336K rodata, 13808K init, 260K bss, 51884K reserved, 0K
cma-reserved, 1835008K highmem)
[    0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay
is 25 jiffies.
[    0.000000] NR_IRQS: 256

Quoted 3 lines

[    0.000000] OF: Bad cell count for /rdb
[    0.000000] irq_bcm7038_l1: failed to remap intc L1 registers
[    0.000000] OF: of_irq_init: children remain, but no parents

So here is the first time we have got the consequence of the corruption popped up. Luckily it's just the "Bad cells count" error. We could have got much less obvious log here up to getting a crash at some place further...

Quoted 10 lines

[    0.000000] random: get_random_bytes called from
start_kernel+0x444/0x654 with crng_init=0
[    0.000000] sched_clock: 32 bits at 250 Hz, resolution 4000000ns,

wraps every 8589934590000000ns

and with your patch applied which unfortunately did not work we have the following:

[...]

So a patch like this shall workaround the corruption:

arch/mips/bmips/setup.c +0 -0

@@ -174,6 +174,8 @@ void __init plat_mem_setup(void)
 
 	__dt_setup_arch(dtb);
 

arch/mips/bmips/setup.c void __init plat_mem_setup(void) · line 181

Quoted 5 lines

arch/mips/bmips/setup.c +2 -0

+	memblock_reserve(0x0, 0x1000 + 0x100*64);
+
 	for (q = bmips_quirk_list; q->quirk_fn; q++) {
 		if (of_flat_dt_is_compatible(of_get_flat_dt_root(),
 					     q->compatible)) {

This patch works, thanks a lot for the troubleshooting and analysis! How about the following which would be more generic and works as well and should be more universal since it does not require each architecture to provide an appropriate call to memblock_reserve():

arch/mips/kernel/traps.c +8 -4

index e0352958e2f7..b0a173b500e8 100644
@@ -2367,10 +2367,7 @@ void __init trap_init(void)
 
        if (!cpu_has_mips_r2_r6) {
                ebase = CAC_BASE;
-               ebase_pa = virt_to_phys((void *)ebase);
                vec_size = 0x400;
-
-               memblock_reserve(ebase_pa, vec_size);
        } else {
                if (cpu_has_veic || cpu_has_vint)
                        vec_size = 0x200 + VECTORSPACING*64;

@@ -2410,6 +2407,14 @@ void __init trap_init(void)
 
        if (board_ebase_setup)
                board_ebase_setup();
+
+       /* board_ebase_setup() can change the exception base address
+        * reserve it now after changes were made.
+        */
+       if (!cpu_has_mips_r2_r6) {
+               ebase_pa = virt_to_phys((void *)ebase);
+               memblock_reserve(ebase_pa, vec_size);
+       }
        per_cpu_trap_init(true);
        memblock_set_bottom_up(false);
-- 

Florian

Quoted 362 lines · Florian Fainelli

Hi Serge,

Quoted 358 lines · Serge Semin

Hi folks,
What you've got here seems a more complicated problem than it could originally look like. Please, see my comments below.

(Note I've discarded some of the email logs, which of no interest to the discovered problem. Please also note that I haven't got any Broadcom hardware to test out a solution suggested below.)

Quoted 24 lines · Florian Fainelli

Hi Mike,

Quoted 21 lines · Mike Rapoport

Hi Florian,

Quoted 18 lines · Florian Fainelli

[...]

Hi Roman, Thomas and other linux-mips folks,

Kamal and myself have been unable to boot v5.11 on MIPS since this commit, reverting it makes our MIPS platforms boot successfully. We do not see a warning like this one in the commit message, instead what happens appear to be a corrupted Device Tree which prevents the parsing of the "rdb" node and leading to the interrupt controllers not being registered, and the system eventually not booting.

The Device Tree is built-into the kernel image and resides at arch/mips/boot/dts/brcm/bcm97435svmb.dts.

Do you have any idea what could be wrong with MIPS specifically here?

Most likely the problem you've discovered has been there for quite some time. The patch you are referring to just caused it to be triggered by extending the early allocation range. See before that patch was accepted the early memory allocations had been performed in the range:
[kernel_end, RAM_END].
The patch changed that, so the early allocations are done within [RAM_START + PAGE_SIZE, RAM_END].

In normal situations it's safe to do that as long as all the critical memory regions (including the memory residing a space below the kernel) have been reserved. But as soon as a memory with some critical structures haven't been reserved, the kernel may allocate it to be used for instance for early initializations with obviously unpredictable but most of the times unpleasant consequences.

Quoted 4 lines

Quoted 4 lines

Apparently there is a memblock allocation in one of the functions called from arch_mem_init() between plat_mem_setup() and early_init_fdt_reserve_self().

Mike, alas according to the log provided by Florian that's not the reason of the problem. Please, see my considerations below.

Quoted 8 lines

[...]

[    0.000000] Linux version 5.11.0-g5695e5161974 (florian@localhost)
(mipsel-linux-gcc (GCC) 8.3.0, GNU ld (GNU Binutils) 2.32) #84 SMP Sun
Feb 28 10:01:50 PST 2021
[    0.000000] CPU0 revision is: 00025b00 (Broadcom BMIPS5200)
[    0.000000] FPU revision is: 00130001

Quoted 6 lines

[    0.000000] memblock_add: [0x00000000-0x0fffffff]
early_init_dt_scan_memory+0x160/0x1e0
[    0.000000] memblock_add: [0x20000000-0x4fffffff]
early_init_dt_scan_memory+0x160/0x1e0
[    0.000000] memblock_add: [0x90000000-0xcfffffff]

early_init_dt_scan_memory+0x160/0x1e0

Here the memory has been added to the memblock allocator.

Quoted 6 lines

[    0.000000] MIPS: machine is Broadcom BCM97435SVMB
[    0.000000] earlycon: ns16550a0 at MMIO32 0x10406b00 (options '')
[    0.000000] printk: bootconsole [ns16550a0] enabled

[    0.000000] memblock_reserve: [0x00aa7600-0x00aaa0a0]

setup_arch+0x128/0x69c

Here the fdt memory has been reserved. (Note it's built into the kernel.)

Quoted 2 lines

[    0.000000] memblock_reserve: [0x00010000-0x018313cf]

setup_arch+0x1f8/0x69c

Here the kernel itself together with built-in dtb have been reserved. So far so good.

Quoted 11 lines

[    0.000000] Initrd not found or empty - disabling initrd

[    0.000000] memblock_alloc_try_nid: 10913 bytes align=0x40 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x00001000-0x00003aa0]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32680 bytes align=0x4 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x00003aa4-0x0000ba4b]

Quoted 1 line

memblock_alloc_range_nid+0xf8/0x198

The log above most likely belongs to the call-chain:
setup_arch()

+-> arch_mem_init()
    +-> device_tree_init() - BMIPS specific method
        +-> unflatten_and_copy_device_tree()

So to speak here we've copied the fdt from the original space [0x00aa7600-0x00aaa0a0] into [0x00001000-0x00003aa0] and unflattened it to [0x00003aa4-0x0000ba4b].

The problem is that a bit later the next call-chain is performed:
setup_arch()

+-> plat_smp_setup()
    +-> mp_ops->smp_setup(); - registered by prom_init()->register_bmips_smp_ops();
        +-> if (!board_ebase_setup)
                 board_ebase_setup = &bmips_ebase_setup;

So at the moment of the CPU traps initialization the bmips_ebase_setup() method is called. What trap_init() does isn't compatible with the allocation performed by the unflatten_and_copy_device_tree() method. See the next comment.

Quoted 171 lines

[    0.000000] memblock_alloc_try_nid: 25 bytes align=0x4 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x0000ba4c-0x0000ba64]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_reserve: [0x0096a000-0x00969fff]
setup_arch+0x3fc/0x69c
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 setup_arch+0x4e0/0x69c
[    0.000000] memblock_reserve: [0x0000ba80-0x0000ba9f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 setup_arch+0x4e0/0x69c
[    0.000000] memblock_reserve: [0x0000bb00-0x0000bb1f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 setup_arch+0x4e0/0x69c
[    0.000000] memblock_reserve: [0x0000bb80-0x0000bb9f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Primary instruction cache 32kB, VIPT, 4-way, linesize 64
bytes.
[    0.000000] Primary data cache 32kB, 4-way, VIPT, no aliases,
linesize 32 bytes
[    0.000000] MIPS secondary cache 512kB, 8-way, linesize 128 bytes.
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0xffffffff fixrange_init+0x90/0xf4
[    0.000000] memblock_reserve: [0x0000c000-0x0000cfff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0xffffffff fixrange_init+0x90/0xf4
[    0.000000] memblock_reserve: [0x0000d000-0x0000dfff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0xffffffff fixrange_init+0x90/0xf4
[    0.000000] memblock_reserve: [0x0000e000-0x0000efff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x000000000fffffff]
[    0.000000]   HighMem  [mem 0x0000000010000000-0x00000000cfffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000000fffffff]
[    0.000000]   node   0: [mem 0x0000000020000000-0x000000004fffffff]
[    0.000000]   node   0: [mem 0x0000000090000000-0x00000000cfffffff]
[    0.000000] Initmem setup node 0 [mem
0x0000000000000000-0x00000000cfffffff]
[    0.000000] memblock_alloc_try_nid: 27262976 bytes align=0x80 nid=0
from=0x00000000 max_addr=0x00000000
alloc_node_mem_map.constprop.135+0x6c/0xc8
[    0.000000] memblock_reserve: [0x01831400-0x032313ff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=0
from=0x00000000 max_addr=0x00000000 setup_usemap+0x64/0x98
[    0.000000] memblock_reserve: [0x0000bc00-0x0000bc1f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 384 bytes align=0x80 nid=0
from=0x00000000 max_addr=0x00000000 setup_usemap+0x64/0x98
[    0.000000] memblock_reserve: [0x0000bc80-0x0000bdff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] MEMBLOCK configuration:
[    0.000000]  memory size = 0x80000000 reserved size = 0x0322f032
[    0.000000]  memory.cnt  = 0x3
[    0.000000]  memory[0x0]     [0x00000000-0x0fffffff], 0x10000000
bytes flags: 0x0
[    0.000000]  memory[0x1]     [0x20000000-0x4fffffff], 0x30000000
bytes flags: 0x0
[    0.000000]  memory[0x2]     [0x90000000-0xcfffffff], 0x40000000
bytes flags: 0x0
[    0.000000]  reserved.cnt  = 0xa
[    0.000000]  reserved[0x0]   [0x00001000-0x00003aa0], 0x00002aa1
bytes flags: 0x0
[    0.000000]  reserved[0x1]   [0x00003aa4-0x0000ba64], 0x00007fc1
bytes flags: 0x0
[    0.000000]  reserved[0x2]   [0x0000ba80-0x0000ba9f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x3]   [0x0000bb00-0x0000bb1f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x4]   [0x0000bb80-0x0000bb9f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x5]   [0x0000bc00-0x0000bc1f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x6]   [0x0000bc80-0x0000bdff], 0x00000180
bytes flags: 0x0
[    0.000000]  reserved[0x7]   [0x0000c000-0x0000efff], 0x00003000
bytes flags: 0x0
[    0.000000]  reserved[0x8]   [0x00010000-0x018313cf], 0x018213d0
bytes flags: 0x0
[    0.000000]  reserved[0x9]   [0x01831400-0x032313ff], 0x01a00000
bytes flags: 0x0
[    0.000000] memblock_alloc_try_nid: 30 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 start_kernel+0x12c/0x654
[    0.000000] memblock_reserve: [0x0000be00-0x0000be1d]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 30 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 start_kernel+0x150/0x654
[    0.000000] memblock_reserve: [0x0000be80-0x0000be9d]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_embed_first_chunk+0x3b0/0x884
[    0.000000] memblock_reserve: [0x0000f000-0x0000ffff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_embed_first_chunk+0x5a4/0x884
[    0.000000] memblock_reserve: [0x03231400-0x032323ff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 294912 bytes align=0x1000 nid=-1
from=0x01000000 max_addr=0x00000000 pcpu_dfl_fc_alloc+0x24/0x30
[    0.000000] memblock_reserve: [0x03233000-0x0327afff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_free: [0x03245000-0x03244fff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] memblock_free: [0x03257000-0x03256fff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] memblock_free: [0x03269000-0x03268fff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] memblock_free: [0x0327b000-0x0327afff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] percpu: Embedded 18 pages/cpu s50704 r0 d23024 u73728
[    0.000000] memblock_alloc_try_nid: 4 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x178/0x6ec
[    0.000000] memblock_reserve: [0x0000bf00-0x0000bf03]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x1a8/0x6ec
[    0.000000] memblock_reserve: [0x0000bf80-0x0000bf83]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 16 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x1dc/0x6ec
[    0.000000] memblock_reserve: [0x03232400-0x0323240f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 16 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x20c/0x6ec
[    0.000000] memblock_reserve: [0x03232480-0x0323248f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 128 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x558/0x6ec
[    0.000000] memblock_reserve: [0x03232500-0x0323257f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 92 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0x8c/0x294
[    0.000000] memblock_reserve: [0x03232580-0x032325db]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 768 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0xe0/0x294
[    0.000000] memblock_reserve: [0x03232600-0x032328ff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 772 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0x124/0x294
[    0.000000] memblock_reserve: [0x03232900-0x03232c03]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 192 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0x158/0x294
[    0.000000] memblock_reserve: [0x03232c80-0x03232d3f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_free: [0x0000f000-0x0000ffff]
pcpu_embed_first_chunk+0x838/0x884
[    0.000000] memblock_free: [0x03231400-0x032323ff]
pcpu_embed_first_chunk+0x850/0x884
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 523776
[    0.000000] Kernel command line: console=ttyS0,115200 earlycon
[    0.000000] memblock_alloc_try_nid: 131072 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 alloc_large_system_hash+0x1f8/0x33c
[    0.000000] memblock_reserve: [0x0327b000-0x0329afff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Dentry cache hash table entries: 32768 (order: 5, 131072
bytes, linear)
[    0.000000] memblock_alloc_try_nid: 65536 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 alloc_large_system_hash+0x1f8/0x33c
[    0.000000] memblock_reserve: [0x0329b000-0x032aafff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536

Quoted 4 lines

bytes, linear)

[    0.000000] memblock_reserve: [0x00000000-0x000003ff]

trap_init+0x70/0x4e8

Most likely someplace here the corruption has happened. The log above has just reserved a memory for NMI/reset vectors:
arch/mips/kernel/traps.c: trap_init(void): Line 2373.

But then the board_ebase_setup() pointer is dereferenced and called, which has been initialized with bmips_ebase_setup() earlier and which overwrites the ebase variable with: 0x80001000 as this is CPU_BMIPS5000 CPU. So any further calls of the functions like set_handler()/set_except_vector()/set_vi_srs_handler()/etc may cause a corruption of the memory above 0x80001000, which as we have discovered belongs to fdt and unflattened device tree.

Quoted 11 lines

[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 2045268K/2097152K available (8226K kernel code,
1070K rwdata, 1336K rodata, 13808K init, 260K bss, 51884K reserved, 0K
cma-reserved, 1835008K highmem)
[    0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay
is 25 jiffies.
[    0.000000] NR_IRQS: 256

Quoted 3 lines

[    0.000000] OF: Bad cell count for /rdb
[    0.000000] irq_bcm7038_l1: failed to remap intc L1 registers
[    0.000000] OF: of_irq_init: children remain, but no parents

So here is the first time we have got the consequence of the corruption popped up. Luckily it's just the "Bad cells count" error. We could have got much less obvious log here up to getting a crash at some place further...

Quoted 10 lines

[    0.000000] random: get_random_bytes called from
start_kernel+0x444/0x654 with crng_init=0
[    0.000000] sched_clock: 32 bits at 250 Hz, resolution 4000000ns,

wraps every 8589934590000000ns

and with your patch applied which unfortunately did not work we have the following:

[...]

So a patch like this shall workaround the corruption:

arch/mips/bmips/setup.c +2 -0

@@ -174,6 +174,8 @@ void __init plat_mem_setup(void)
 
 	__dt_setup_arch(dtb);
 
+	memblock_reserve(0x0, 0x1000 + 0x100*64);
+
 	for (q = bmips_quirk_list; q->quirk_fn; q++) {
 		if (of_flat_dt_is_compatible(of_get_flat_dt_root(),
 					     q->compatible)) {

Quoted 4 lines

This patch works, thanks a lot for the troubleshooting and analysis! How about the following which would be more generic and works as well and should be more universal since it does not require each architecture to provide an appropriate call to memblock_reserve():

Hm, are you sure it's working?

If so, my analysis hasn't been quite
correct. My suggestion was based on the memory initializations, allocations and reservations trace. So here is the sequence of most crucial of them:
1) Memblock initialization:
start_kernel()->setup_arch()->arch_mem_init()->plat_mem_setup()->__dt_setup_arch() (At this point I suggested to place the exceptions memory
reservation.)
2) Base FDT memory reservation:

   start_kernel()->setup_arch()->arch_mem_init()->early_init_fdt_reserve_self()
3) FDT "reserved-memory" nodes parsing and corresponding memory ranges
   reservation:
   start_kernel()->setup_arch()->arch_mem_init()->early_init_fdt_scan_reserved_mem()
4) Reserve kernel itself, some critical sections like initrd and
   crash-kernel:
   start_kernel()->setup_arch()->arch_mem_init()->bootmem_init()...

5) Copy and unflatten the built-into the kernel device tree (BMIPS-platform code):
start_kernel()->setup_arch()->arch_mem_init()->device_tree_init() This is the very first time an allocation from the memblock pool is performed. Since we haven't reserved a memory for the exception vectors yet, the memblock allocator is free to return that memory range for any other use. Needless to say if we try to use that memory later without consulting with memblock, we may and in our case will get into troubles.
6) Many random early memblock allocations for kernel use before buddy and sl*b allocators are up and running...
Note if for some fortunate reason the allocations made in 5) didn't overlap the exceptions memory, here we have much more chances to do that with obviously fatal consequences of the ranges independent usage.
7) Trap/exception vectors initialization and !memory reservation! for them:
start_kernel()->trap_init()
Only at this point we get to reserve the memory for the vectors.
8) Init and run buddy/sl*b allocators:

   start_kernel()->mm_init()->...mem_init()...

There are a lot of allocations done in 5) and 6) before the trap_init() is called in 7). You can see that in your log. That's why I have doubts that your patch worked well. Most likely you've forgotten to revert the workaround suggested by me in the previous message. Could you make sure that you didn't and re-test your patch again? If it still works then I might have confused something and it's strange that my patch worked in the first place...

A food for thoughts for everyone (Thomas, Mark, please join the discussion). What we've got here is a bit bigger problem. AFAICS
if bottom-up allocation is enabled (it's our case) memblock_find_in_range_node() performs the allocation above the very first PAGE_SIZE memory chunk (see that method code for details). So we are currently on a safe side for some older MIPS platforms. But the platform with VEIC/VINT may get into the same troubles here if they didn't reserve exception memory early enough before the kernel starts random allocations from memblock. So we either need to provide a generic workaround for that or make sure each platform gets to reserve vectors itself for instance in the plat_mem_setup() method.

-Sergey

Quoted 32 lines

arch/mips/kernel/traps.c +8 -4

index e0352958e2f7..b0a173b500e8 100644
@@ -2367,10 +2367,7 @@ void __init trap_init(void)
 
        if (!cpu_has_mips_r2_r6) {
                ebase = CAC_BASE;
-               ebase_pa = virt_to_phys((void *)ebase);
                vec_size = 0x400;
-
-               memblock_reserve(ebase_pa, vec_size);
        } else {
                if (cpu_has_veic || cpu_has_vint)
                        vec_size = 0x200 + VECTORSPACING*64;

@@ -2410,6 +2407,14 @@ void __init trap_init(void)
 
        if (board_ebase_setup)
                board_ebase_setup();
+
+       /* board_ebase_setup() can change the exception base address
+        * reserve it now after changes were made.
+        */
+       if (!cpu_has_mips_r2_r6) {
+               ebase_pa = virt_to_phys((void *)ebase);
+               memblock_reserve(ebase_pa, vec_size);
+       }
        per_cpu_trap_init(true);
        memblock_set_bottom_up(false);
-- 

Florian

Quoted 364 lines · Serge Semin

Quoted 362 lines · Florian Fainelli

Hi Serge,

Quoted 358 lines · Serge Semin

Hi folks,
What you've got here seems a more complicated problem than it could originally look like. Please, see my comments below.

(Note I've discarded some of the email logs, which of no interest to the discovered problem. Please also note that I haven't got any Broadcom hardware to test out a solution suggested below.)

Quoted 24 lines · Florian Fainelli

Hi Mike,

Quoted 21 lines · Mike Rapoport

Hi Florian,

Quoted 18 lines · Florian Fainelli

[...]

Hi Roman, Thomas and other linux-mips folks,

Kamal and myself have been unable to boot v5.11 on MIPS since this commit, reverting it makes our MIPS platforms boot successfully. We do not see a warning like this one in the commit message, instead what happens appear to be a corrupted Device Tree which prevents the parsing of the "rdb" node and leading to the interrupt controllers not being registered, and the system eventually not booting.

The Device Tree is built-into the kernel image and resides at arch/mips/boot/dts/brcm/bcm97435svmb.dts.

Do you have any idea what could be wrong with MIPS specifically here?

Most likely the problem you've discovered has been there for quite some time. The patch you are referring to just caused it to be triggered by extending the early allocation range. See before that patch was accepted the early memory allocations had been performed in the range:
[kernel_end, RAM_END].
The patch changed that, so the early allocations are done within [RAM_START + PAGE_SIZE, RAM_END].

In normal situations it's safe to do that as long as all the critical memory regions (including the memory residing a space below the kernel) have been reserved. But as soon as a memory with some critical structures haven't been reserved, the kernel may allocate it to be used for instance for early initializations with obviously unpredictable but most of the times unpleasant consequences.

Quoted 4 lines

Quoted 4 lines

Apparently there is a memblock allocation in one of the functions called from arch_mem_init() between plat_mem_setup() and early_init_fdt_reserve_self().

Mike, alas according to the log provided by Florian that's not the reason of the problem. Please, see my considerations below.

Quoted 8 lines

[...]

[    0.000000] Linux version 5.11.0-g5695e5161974 (florian@localhost)
(mipsel-linux-gcc (GCC) 8.3.0, GNU ld (GNU Binutils) 2.32) #84 SMP Sun
Feb 28 10:01:50 PST 2021
[    0.000000] CPU0 revision is: 00025b00 (Broadcom BMIPS5200)
[    0.000000] FPU revision is: 00130001

Quoted 6 lines

[    0.000000] memblock_add: [0x00000000-0x0fffffff]
early_init_dt_scan_memory+0x160/0x1e0
[    0.000000] memblock_add: [0x20000000-0x4fffffff]
early_init_dt_scan_memory+0x160/0x1e0
[    0.000000] memblock_add: [0x90000000-0xcfffffff]

early_init_dt_scan_memory+0x160/0x1e0

Here the memory has been added to the memblock allocator.

Quoted 6 lines

[    0.000000] MIPS: machine is Broadcom BCM97435SVMB
[    0.000000] earlycon: ns16550a0 at MMIO32 0x10406b00 (options '')
[    0.000000] printk: bootconsole [ns16550a0] enabled

[    0.000000] memblock_reserve: [0x00aa7600-0x00aaa0a0]

setup_arch+0x128/0x69c

Here the fdt memory has been reserved. (Note it's built into the kernel.)

Quoted 2 lines

[    0.000000] memblock_reserve: [0x00010000-0x018313cf]

setup_arch+0x1f8/0x69c

Here the kernel itself together with built-in dtb have been reserved. So far so good.

Quoted 11 lines

[    0.000000] Initrd not found or empty - disabling initrd

[    0.000000] memblock_alloc_try_nid: 10913 bytes align=0x40 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x00001000-0x00003aa0]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32680 bytes align=0x4 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x00003aa4-0x0000ba4b]

Quoted 1 line

memblock_alloc_range_nid+0xf8/0x198

The log above most likely belongs to the call-chain:
setup_arch()

+-> arch_mem_init()
    +-> device_tree_init() - BMIPS specific method
        +-> unflatten_and_copy_device_tree()

So to speak here we've copied the fdt from the original space [0x00aa7600-0x00aaa0a0] into [0x00001000-0x00003aa0] and unflattened it to [0x00003aa4-0x0000ba4b].

The problem is that a bit later the next call-chain is performed:
setup_arch()

+-> plat_smp_setup()
    +-> mp_ops->smp_setup(); - registered by prom_init()->register_bmips_smp_ops();
        +-> if (!board_ebase_setup)
                 board_ebase_setup = &bmips_ebase_setup;

So at the moment of the CPU traps initialization the bmips_ebase_setup() method is called. What trap_init() does isn't compatible with the allocation performed by the unflatten_and_copy_device_tree() method. See the next comment.

Quoted 171 lines

[    0.000000] memblock_alloc_try_nid: 25 bytes align=0x4 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x0000ba4c-0x0000ba64]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_reserve: [0x0096a000-0x00969fff]
setup_arch+0x3fc/0x69c
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 setup_arch+0x4e0/0x69c
[    0.000000] memblock_reserve: [0x0000ba80-0x0000ba9f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 setup_arch+0x4e0/0x69c
[    0.000000] memblock_reserve: [0x0000bb00-0x0000bb1f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 setup_arch+0x4e0/0x69c
[    0.000000] memblock_reserve: [0x0000bb80-0x0000bb9f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Primary instruction cache 32kB, VIPT, 4-way, linesize 64
bytes.
[    0.000000] Primary data cache 32kB, 4-way, VIPT, no aliases,
linesize 32 bytes
[    0.000000] MIPS secondary cache 512kB, 8-way, linesize 128 bytes.
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0xffffffff fixrange_init+0x90/0xf4
[    0.000000] memblock_reserve: [0x0000c000-0x0000cfff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0xffffffff fixrange_init+0x90/0xf4
[    0.000000] memblock_reserve: [0x0000d000-0x0000dfff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0xffffffff fixrange_init+0x90/0xf4
[    0.000000] memblock_reserve: [0x0000e000-0x0000efff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x000000000fffffff]
[    0.000000]   HighMem  [mem 0x0000000010000000-0x00000000cfffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000000fffffff]
[    0.000000]   node   0: [mem 0x0000000020000000-0x000000004fffffff]
[    0.000000]   node   0: [mem 0x0000000090000000-0x00000000cfffffff]
[    0.000000] Initmem setup node 0 [mem
0x0000000000000000-0x00000000cfffffff]
[    0.000000] memblock_alloc_try_nid: 27262976 bytes align=0x80 nid=0
from=0x00000000 max_addr=0x00000000
alloc_node_mem_map.constprop.135+0x6c/0xc8
[    0.000000] memblock_reserve: [0x01831400-0x032313ff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32 bytes align=0x80 nid=0
from=0x00000000 max_addr=0x00000000 setup_usemap+0x64/0x98
[    0.000000] memblock_reserve: [0x0000bc00-0x0000bc1f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 384 bytes align=0x80 nid=0
from=0x00000000 max_addr=0x00000000 setup_usemap+0x64/0x98
[    0.000000] memblock_reserve: [0x0000bc80-0x0000bdff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] MEMBLOCK configuration:
[    0.000000]  memory size = 0x80000000 reserved size = 0x0322f032
[    0.000000]  memory.cnt  = 0x3
[    0.000000]  memory[0x0]     [0x00000000-0x0fffffff], 0x10000000
bytes flags: 0x0
[    0.000000]  memory[0x1]     [0x20000000-0x4fffffff], 0x30000000
bytes flags: 0x0
[    0.000000]  memory[0x2]     [0x90000000-0xcfffffff], 0x40000000
bytes flags: 0x0
[    0.000000]  reserved.cnt  = 0xa
[    0.000000]  reserved[0x0]   [0x00001000-0x00003aa0], 0x00002aa1
bytes flags: 0x0
[    0.000000]  reserved[0x1]   [0x00003aa4-0x0000ba64], 0x00007fc1
bytes flags: 0x0
[    0.000000]  reserved[0x2]   [0x0000ba80-0x0000ba9f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x3]   [0x0000bb00-0x0000bb1f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x4]   [0x0000bb80-0x0000bb9f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x5]   [0x0000bc00-0x0000bc1f], 0x00000020
bytes flags: 0x0
[    0.000000]  reserved[0x6]   [0x0000bc80-0x0000bdff], 0x00000180
bytes flags: 0x0
[    0.000000]  reserved[0x7]   [0x0000c000-0x0000efff], 0x00003000
bytes flags: 0x0
[    0.000000]  reserved[0x8]   [0x00010000-0x018313cf], 0x018213d0
bytes flags: 0x0
[    0.000000]  reserved[0x9]   [0x01831400-0x032313ff], 0x01a00000
bytes flags: 0x0
[    0.000000] memblock_alloc_try_nid: 30 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 start_kernel+0x12c/0x654
[    0.000000] memblock_reserve: [0x0000be00-0x0000be1d]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 30 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 start_kernel+0x150/0x654
[    0.000000] memblock_reserve: [0x0000be80-0x0000be9d]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x1000 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_embed_first_chunk+0x3b0/0x884
[    0.000000] memblock_reserve: [0x0000f000-0x0000ffff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4096 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_embed_first_chunk+0x5a4/0x884
[    0.000000] memblock_reserve: [0x03231400-0x032323ff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 294912 bytes align=0x1000 nid=-1
from=0x01000000 max_addr=0x00000000 pcpu_dfl_fc_alloc+0x24/0x30
[    0.000000] memblock_reserve: [0x03233000-0x0327afff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_free: [0x03245000-0x03244fff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] memblock_free: [0x03257000-0x03256fff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] memblock_free: [0x03269000-0x03268fff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] memblock_free: [0x0327b000-0x0327afff]
pcpu_embed_first_chunk+0x7a0/0x884
[    0.000000] percpu: Embedded 18 pages/cpu s50704 r0 d23024 u73728
[    0.000000] memblock_alloc_try_nid: 4 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x178/0x6ec
[    0.000000] memblock_reserve: [0x0000bf00-0x0000bf03]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 4 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x1a8/0x6ec
[    0.000000] memblock_reserve: [0x0000bf80-0x0000bf83]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 16 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x1dc/0x6ec
[    0.000000] memblock_reserve: [0x03232400-0x0323240f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 16 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x20c/0x6ec
[    0.000000] memblock_reserve: [0x03232480-0x0323248f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 128 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_setup_first_chunk+0x558/0x6ec
[    0.000000] memblock_reserve: [0x03232500-0x0323257f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 92 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0x8c/0x294
[    0.000000] memblock_reserve: [0x03232580-0x032325db]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 768 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0xe0/0x294
[    0.000000] memblock_reserve: [0x03232600-0x032328ff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 772 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0x124/0x294
[    0.000000] memblock_reserve: [0x03232900-0x03232c03]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 192 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 pcpu_alloc_first_chunk+0x158/0x294
[    0.000000] memblock_reserve: [0x03232c80-0x03232d3f]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_free: [0x0000f000-0x0000ffff]
pcpu_embed_first_chunk+0x838/0x884
[    0.000000] memblock_free: [0x03231400-0x032323ff]
pcpu_embed_first_chunk+0x850/0x884
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 523776
[    0.000000] Kernel command line: console=ttyS0,115200 earlycon
[    0.000000] memblock_alloc_try_nid: 131072 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 alloc_large_system_hash+0x1f8/0x33c
[    0.000000] memblock_reserve: [0x0327b000-0x0329afff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Dentry cache hash table entries: 32768 (order: 5, 131072
bytes, linear)
[    0.000000] memblock_alloc_try_nid: 65536 bytes align=0x80 nid=-1
from=0x00000000 max_addr=0x00000000 alloc_large_system_hash+0x1f8/0x33c
[    0.000000] memblock_reserve: [0x0329b000-0x032aafff]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536

Quoted 4 lines

bytes, linear)

[    0.000000] memblock_reserve: [0x00000000-0x000003ff]

trap_init+0x70/0x4e8

Most likely someplace here the corruption has happened. The log above has just reserved a memory for NMI/reset vectors:
arch/mips/kernel/traps.c: trap_init(void): Line 2373.

But then the board_ebase_setup() pointer is dereferenced and called, which has been initialized with bmips_ebase_setup() earlier and which overwrites the ebase variable with: 0x80001000 as this is CPU_BMIPS5000 CPU. So any further calls of the functions like set_handler()/set_except_vector()/set_vi_srs_handler()/etc may cause a corruption of the memory above 0x80001000, which as we have discovered belongs to fdt and unflattened device tree.

Quoted 11 lines

[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 2045268K/2097152K available (8226K kernel code,
1070K rwdata, 1336K rodata, 13808K init, 260K bss, 51884K reserved, 0K
cma-reserved, 1835008K highmem)
[    0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay
is 25 jiffies.
[    0.000000] NR_IRQS: 256

Quoted 3 lines

[    0.000000] OF: Bad cell count for /rdb
[    0.000000] irq_bcm7038_l1: failed to remap intc L1 registers
[    0.000000] OF: of_irq_init: children remain, but no parents

So here is the first time we have got the consequence of the corruption popped up. Luckily it's just the "Bad cells count" error. We could have got much less obvious log here up to getting a crash at some place further...

Quoted 10 lines

[    0.000000] random: get_random_bytes called from
start_kernel+0x444/0x654 with crng_init=0
[    0.000000] sched_clock: 32 bits at 250 Hz, resolution 4000000ns,

wraps every 8589934590000000ns

and with your patch applied which unfortunately did not work we have the following:

[...]

So a patch like this shall workaround the corruption:

arch/mips/bmips/setup.c +2 -0

@@ -174,6 +174,8 @@ void __init plat_mem_setup(void)
 
 	__dt_setup_arch(dtb);
 
+	memblock_reserve(0x0, 0x1000 + 0x100*64);
+
 	for (q = bmips_quirk_list; q->quirk_fn; q++) {
 		if (of_flat_dt_is_compatible(of_get_flat_dt_root(),
 					     q->compatible)) {

Quoted 6 lines

Quoted 4 lines

This patch works, thanks a lot for the troubleshooting and analysis! How about the following which would be more generic and works as well and should be more universal since it does not require each architecture to provide an appropriate call to memblock_reserve():

Hm, are you sure it's working?

I was until I noticed that I was working on top of a revert of Roman's patch sorry about the brain fart here.

Quoted 38 lines

If so, my analysis hasn't been quite
correct. My suggestion was based on the memory initializations, allocations and reservations trace. So here is the sequence of most crucial of them:
1) Memblock initialization:
start_kernel()->setup_arch()->arch_mem_init()->plat_mem_setup()->__dt_setup_arch() (At this point I suggested to place the exceptions memory
reservation.)
2) Base FDT memory reservation:

   start_kernel()->setup_arch()->arch_mem_init()->early_init_fdt_reserve_self()
3) FDT "reserved-memory" nodes parsing and corresponding memory ranges
   reservation:
   start_kernel()->setup_arch()->arch_mem_init()->early_init_fdt_scan_reserved_mem()
4) Reserve kernel itself, some critical sections like initrd and
   crash-kernel:
   start_kernel()->setup_arch()->arch_mem_init()->bootmem_init()...

5) Copy and unflatten the built-into the kernel device tree (BMIPS-platform code):
start_kernel()->setup_arch()->arch_mem_init()->device_tree_init() This is the very first time an allocation from the memblock pool is performed. Since we haven't reserved a memory for the exception vectors yet, the memblock allocator is free to return that memory range for any other use. Needless to say if we try to use that memory later without consulting with memblock, we may and in our case will get into troubles.
6) Many random early memblock allocations for kernel use before buddy and sl*b allocators are up and running...
Note if for some fortunate reason the allocations made in 5) didn't overlap the exceptions memory, here we have much more chances to do that with obviously fatal consequences of the ranges independent usage.
7) Trap/exception vectors initialization and !memory reservation! for them:
start_kernel()->trap_init()
Only at this point we get to reserve the memory for the vectors.
8) Init and run buddy/sl*b allocators:

   start_kernel()->mm_init()->...mem_init()...

Quoted 7 lines

There are a lot of allocations done in 5) and 6) before the trap_init() is called in 7). You can see that in your log. That's why I have doubts that your patch worked well. Most likely you've forgotten to revert the workaround suggested by me in the previous message. Could you make sure that you didn't and re-test your patch again? If it still works then I might have confused something and it's strange that my patch worked in the first place...

I would like to submit a fix for 5.12-rc1 and get it back ported into 5.11 so we have BMIPS machines boot again, that will be essentially your earlier proposed fix.

BMIPS is the only "legacy" MIPS platform that defines an exception base, so while this problem may certainly exist with other platforms, I do wonder how likely it is there, though?

Quoted 48 lines

A food for thoughts for everyone (Thomas, Mark, please join the discussion). What we've got here is a bit bigger problem. AFAICS
if bottom-up allocation is enabled (it's our case) memblock_find_in_range_node() performs the allocation above the very first PAGE_SIZE memory chunk (see that method code for details). So we are currently on a safe side for some older MIPS platforms. But the platform with VEIC/VINT may get into the same troubles here if they didn't reserve exception memory early enough before the kernel starts random allocations from memblock. So we either need to provide a generic workaround for that or make sure each platform gets to reserve vectors itself for instance in the plat_mem_setup() method.

-Sergey

Quoted 32 lines

arch/mips/kernel/traps.c +8 -4

index e0352958e2f7..b0a173b500e8 100644
@@ -2367,10 +2367,7 @@ void __init trap_init(void)
 
        if (!cpu_has_mips_r2_r6) {
                ebase = CAC_BASE;
-               ebase_pa = virt_to_phys((void *)ebase);
                vec_size = 0x400;
-
-               memblock_reserve(ebase_pa, vec_size);
        } else {
                if (cpu_has_veic || cpu_has_vint)
                        vec_size = 0x200 + VECTORSPACING*64;

@@ -2410,6 +2407,14 @@ void __init trap_init(void)
 
        if (board_ebase_setup)
                board_ebase_setup();
+
+       /* board_ebase_setup() can change the exception base address
+        * reserve it now after changes were made.
+        */
+       if (!cpu_has_mips_r2_r6) {
+               ebase_pa = virt_to_phys((void *)ebase);
+               memblock_reserve(ebase_pa, vec_size);
+       }
        per_cpu_trap_init(true);
        memblock_set_bottom_up(false);
-- 

Florian

Signature · 1 line

Florian

BMIPS is one of the few platforms that do change the exception base. After commit 2dcb39645441 ("memblock: do not start bottom-up allocations with kernel_end") we started seeing BMIPS boards fail to boot with the built-in FDT being corrupted.

Before the cited commit, early allocations would be in the [kernel_end, RAM_END] range, but after commit they would be within [RAM_START + PAGE_SIZE, RAM_END].

The custom exception base handler that is installed by bmips_ebase_setup() done for BMIPS5000 CPUs ends-up trampling on the memory region allocated by unflatten_and_copy_device_tree() thus corrupting the FDT used by the kernel.

To fix this, we need to perform an early reservation of the custom exception that is going to be installed and this needs to happen at plat_mem_setup() time to ensure that unflatten_and_copy_device_tree() finds a space that is suitable, away from reserved memory.

Huge thanks to Serget for analysing and proposing a solution to this issue.

Fixes Fixes: 2dcb39645441 ("memblock: do not start bottom-up allocations with kernel_end")

Debugged-by Serge Semin <[email protected]>

Reported-by Kamal Dasu <[email protected]>

Signed-off-by Florian Fainelli <[email protected]>

---
Thomas,

This is intended as a stop-gap solution for 5.12-rc1 and to be picked up by the stable team for 5.11. We should find a safer way to avoid these problems for 5.13 maybe.

 arch/mips/bmips/setup.c       | 22 ++++++++++++++++++++++
 arch/mips/include/asm/traps.h |  2 ++
 2 files changed, 24 insertions(+)

arch/mips/bmips/setup.c +22 -0

index 31bcfa4e08b9..0088bd45b892 100644
@@ -149,6 +149,26 @@ void __init plat_time_init(void)
 	mips_hpt_frequency = freq;
 }
 
+static void __init bmips_ebase_reserve(void)
+{
+	phys_addr_t base, size = VECTORSPACING * 64;
+
+	switch (current_cpu_type()) {
+	default:
+	case CPU_BMIPS4350:
+		return;
+	case CPU_BMIPS3300:
+	case CPU_BMIPS4380:
+		base = 0x0400;
+		break;
+	case CPU_BMIPS5000:
+		base = 0x1000;
+		break;
+	}
+
+	memblock_reserve(base, size);
+}
+
 void __init plat_mem_setup(void)
 {
 	void *dtb;

@@ -169,6 +189,8 @@ void __init plat_mem_setup(void)
 
 	__dt_setup_arch(dtb);
 
+	bmips_ebase_reserve();
+
 	for (q = bmips_quirk_list; q->quirk_fn; q++) {
 		if (of_flat_dt_is_compatible(of_get_flat_dt_root(),
 					     q->compatible)) {

arch/mips/include/asm/traps.h +2 -0

index 6aa8f126a43d..0ba6bb7f9618 100644
@@ -14,6 +14,8 @@
 #define MIPS_BE_FIXUP	1		/* return to the fixup code */
 #define MIPS_BE_FATAL	2		/* treat as an unrecoverable error */
 
+#define VECTORSPACING 0x100	/* for EI/VI mode */
+
 extern void (*board_be_init)(void);
 extern int (*board_be_handler)(struct pt_regs *regs, int is_fixup);
 

Signature · 1 line

2.25.1

BMIPS is one of the few platforms that do change the exception base. After commit 2dcb39645441 ("memblock: do not start bottom-up allocations with kernel_end") we started seeing BMIPS boards fail to boot with the built-in FDT being corrupted.

Before the cited commit, early allocations would be in the [kernel_end, RAM_END] range, but after commit they would be within [RAM_START + PAGE_SIZE, RAM_END].

The custom exception base handler that is installed by bmips_ebase_setup() done for BMIPS5000 CPUs ends-up trampling on the memory region allocated by unflatten_and_copy_device_tree() thus corrupting the FDT used by the kernel.

To fix this, we need to perform an early reservation of the custom exception that is going to be installed and this needs to happen at plat_mem_setup() time to ensure that unflatten_and_copy_device_tree() finds a space that is suitable, away from reserved memory.

Huge thanks to Serget for analysing and proposing a solution to this issue.

Fixes Fixes: 2dcb39645441 ("memblock: do not start bottom-up allocations with kernel_end")

Debugged-by Serge Semin <[email protected]>

Reported-by Kamal Dasu <[email protected]>

Signed-off-by Florian Fainelli <[email protected]>

---

Thomas,

This is intended as a stop-gap solution for 5.12-rc1 and to be picked up by the stable team for 5.11. We should find a safer way to avoid these problems for 5.13 maybe.

 arch/mips/bmips/setup.c       | 22 ++++++++++++++++++++++
 arch/mips/include/asm/traps.h |  2 ++
 2 files changed, 24 insertions(+)

arch/mips/bmips/setup.c +22 -0

index 31bcfa4e08b9..0088bd45b892 100644
@@ -149,6 +149,26 @@ void __init plat_time_init(void)
 	mips_hpt_frequency = freq;
 }
 
+static void __init bmips_ebase_reserve(void)
+{
+	phys_addr_t base, size = VECTORSPACING * 64;
+
+	switch (current_cpu_type()) {
+	default:
+	case CPU_BMIPS4350:
+		return;
+	case CPU_BMIPS3300:
+	case CPU_BMIPS4380:
+		base = 0x0400;
+		break;
+	case CPU_BMIPS5000:
+		base = 0x1000;
+		break;
+	}
+
+	memblock_reserve(base, size);
+}
+
 void __init plat_mem_setup(void)
 {
 	void *dtb;

@@ -169,6 +189,8 @@ void __init plat_mem_setup(void)
 
 	__dt_setup_arch(dtb);
 
+	bmips_ebase_reserve();
+
 	for (q = bmips_quirk_list; q->quirk_fn; q++) {
 		if (of_flat_dt_is_compatible(of_get_flat_dt_root(),
 					     q->compatible)) {

arch/mips/include/asm/traps.h +0 -0

index 6aa8f126a43d..0ba6bb7f9618 100644

arch/mips/include/asm/traps.h +1 -0

+
 extern void (*board_be_init)(void);
 extern int (*board_be_handler)(struct pt_regs *regs, int is_fixup);

Signature · 2 lines

-- 
2.25.1

BMIPS is one of the few platforms that do change the exception base. After commit 2dcb39645441 ("memblock: do not start bottom-up allocations with kernel_end") we started seeing BMIPS boards fail to boot with the built-in FDT being corrupted.

Before the cited commit, early allocations would be in the [kernel_end, RAM_END] range, but after commit they would be within [RAM_START + PAGE_SIZE, RAM_END].

The custom exception base handler that is installed by bmips_ebase_setup() done for BMIPS5000 CPUs ends-up trampling on the memory region allocated by unflatten_and_copy_device_tree() thus corrupting the FDT used by the kernel.

To fix this, we need to perform an early reservation of the custom exception that is going to be installed and this needs to happen at plat_mem_setup() time to ensure that unflatten_and_copy_device_tree() finds a space that is suitable, away from reserved memory.

Huge thanks to Serget for analysing and proposing a solution to this issue.

Fixes Fixes: 2dcb39645441 ("memblock: do not start bottom-up allocations with kernel_end")

Debugged-by Serge Semin <[email protected]>

Reported-by Kamal Dasu <[email protected]>

Signed-off-by Florian Fainelli <[email protected]>

---
Thomas,

This is intended as a stop-gap solution for 5.12-rc1 and to be picked up by the stable team for 5.11. We should find a safer way to avoid these problems for 5.13 maybe.

 arch/mips/bmips/setup.c       | 22 ++++++++++++++++++++++
 arch/mips/include/asm/traps.h |  2 ++
 2 files changed, 24 insertions(+)

arch/mips/bmips/setup.c +22 -0

index 31bcfa4e08b9..0088bd45b892 100644
@@ -149,6 +149,26 @@ void __init plat_time_init(void)
 	mips_hpt_frequency = freq;
 }
 
+static void __init bmips_ebase_reserve(void)
+{
+	phys_addr_t base, size = VECTORSPACING * 64;
+
+	switch (current_cpu_type()) {
+	default:
+	case CPU_BMIPS4350:
+		return;
+	case CPU_BMIPS3300:
+	case CPU_BMIPS4380:
+		base = 0x0400;
+		break;
+	case CPU_BMIPS5000:
+		base = 0x1000;
+		break;
+	}
+
+	memblock_reserve(base, size);
+}
+
 void __init plat_mem_setup(void)
 {
 	void *dtb;

@@ -169,6 +189,8 @@ void __init plat_mem_setup(void)
 
 	__dt_setup_arch(dtb);
 
+	bmips_ebase_reserve();
+
 	for (q = bmips_quirk_list; q->quirk_fn; q++) {
 		if (of_flat_dt_is_compatible(of_get_flat_dt_root(),
 					     q->compatible)) {

arch/mips/include/asm/traps.h +2 -0

index 6aa8f126a43d..0ba6bb7f9618 100644
@@ -14,6 +14,8 @@
 #define MIPS_BE_FIXUP	1		/* return to the fixup code */
 #define MIPS_BE_FATAL	2		/* treat as an unrecoverable error */
 
+#define VECTORSPACING 0x100	/* for EI/VI mode */
+
 extern void (*board_be_init)(void);
 extern int (*board_be_handler)(struct pt_regs *regs, int is_fixup);
 

Signature · 1 line

2.25.1

BMIPS is one of the few platforms that do change the exception base. After commit 2dcb39645441 ("memblock: do not start bottom-up allocations with kernel_end") we started seeing BMIPS boards fail to boot with the built-in FDT being corrupted.

Before the cited commit, early allocations would be in the [kernel_end, RAM_END] range, but after commit they would be within [RAM_START + PAGE_SIZE, RAM_END].

The custom exception base handler that is installed by bmips_ebase_setup() done for BMIPS5000 CPUs ends-up trampling on the memory region allocated by unflatten_and_copy_device_tree() thus corrupting the FDT used by the kernel.

To fix this, we need to perform an early reservation of the custom exception that is going to be installed and this needs to happen at plat_mem_setup() time to ensure that unflatten_and_copy_device_tree() finds a space that is suitable, away from reserved memory.

Huge thanks to Serget for analysing and proposing a solution to this issue.

Fixes Fixes: 2dcb39645441 ("memblock: do not start bottom-up allocations with kernel_end")

Debugged-by Serge Semin <[email protected]>

Reported-by Kamal Dasu <[email protected]>

Signed-off-by Florian Fainelli <[email protected]>

---
Thomas,

This is intended as a stop-gap solution for 5.12-rc1 and to be picked up by the stable team for 5.11. We should find a safer way to avoid these problems for 5.13 maybe.

Quoted 26 lines · Florian Fainelli

BMIPS is one of the few platforms that do change the exception base. After commit 2dcb39645441 ("memblock: do not start bottom-up allocations with kernel_end") we started seeing BMIPS boards fail to boot with the built-in FDT being corrupted.

Before the cited commit, early allocations would be in the [kernel_end, RAM_END] range, but after commit they would be within [RAM_START + PAGE_SIZE, RAM_END].

The custom exception base handler that is installed by bmips_ebase_setup() done for BMIPS5000 CPUs ends-up trampling on the memory region allocated by unflatten_and_copy_device_tree() thus corrupting the FDT used by the kernel.

To fix this, we need to perform an early reservation of the custom exception that is going to be installed and this needs to happen at plat_mem_setup() time to ensure that unflatten_and_copy_device_tree() finds a space that is suitable, away from reserved memory.

Huge thanks to Serget for analysing and proposing a solution to this issue.

Fixes Fixes: 2dcb39645441 ("memblock: do not start bottom-up allocations with kernel_end")

Debugged-by Serge Semin <[email protected]>

Reported-by Kamal Dasu <[email protected]>

Signed-off-by Florian Fainelli <[email protected]>

Quoted 6 lines

---
Thomas,

This is intended as a stop-gap solution for 5.12-rc1 and to be picked up by the stable team for 5.11. We should find a safer way to avoid these problems for 5.13 maybe.

let's try to make it in one ago. Hwo about reserving vector space in cpu_probe, if it's known there and leave the rest to trap_init() ?

Below patch got a quick test on IP22 (real hardware) and malta (qemu). Not sure, if I got all BMIPS parts correct, so please check/test.

BTW. do we really need to EXPORT_SYMBOL ebase ?

Quoted 34 lines · Thomas Bogendoerfer

Quoted 32 lines · Florian Fainelli

BMIPS is one of the few platforms that do change the exception base. After commit 2dcb39645441 ("memblock: do not start bottom-up allocations with kernel_end") we started seeing BMIPS boards fail to boot with the built-in FDT being corrupted.

Before the cited commit, early allocations would be in the [kernel_end, RAM_END] range, but after commit they would be within [RAM_START + PAGE_SIZE, RAM_END].

The custom exception base handler that is installed by bmips_ebase_setup() done for BMIPS5000 CPUs ends-up trampling on the memory region allocated by unflatten_and_copy_device_tree() thus corrupting the FDT used by the kernel.

To fix this, we need to perform an early reservation of the custom exception that is going to be installed and this needs to happen at plat_mem_setup() time to ensure that unflatten_and_copy_device_tree() finds a space that is suitable, away from reserved memory.

Huge thanks to Serget for analysing and proposing a solution to this issue.

Fixes Fixes: 2dcb39645441 ("memblock: do not start bottom-up allocations with kernel_end")

Debugged-by Serge Semin <[email protected]>

Reported-by Kamal Dasu <[email protected]>

Signed-off-by Florian Fainelli <[email protected]>

---
Thomas,

This is intended as a stop-gap solution for 5.12-rc1 and to be picked up by the stable team for 5.11. We should find a safer way to avoid these problems for 5.13 maybe.

Quoted 5 lines

let's try to make it in one ago. Hwo about reserving vector space in cpu_probe, if it's known there and leave the rest to trap_init() ?

Below patch got a quick test on IP22 (real hardware) and malta (qemu). Not sure, if I got all BMIPS parts correct, so please check/test.

Works for me here:

perfect, I only forgot about R3k... I'll submit a formal patch submission later today.

Quoted 2 lines · Thomas Bogendoerfer

perfect, I only forgot about R3k... I'll submit a formal patch submission later today.

What's up with the R3k (the usual trigger for me) here?

Quoted 1 line

What's up with the R3k (the usual trigger for me) here?

I've moved r3k cpu_probe() to it's own file and when moving ebase reservation to cpu_probe(), I need to add it there as well. So just a mechanic step, I've missed.

Hi Serge,

Quoted 129 lines · Serge Semin

Hi folks,
What you've got here seems a more complicated problem than it could originally look like. Please, see my comments below.

(Note I've discarded some of the email logs, which of no interest to the discovered problem. Please also note that I haven't got any Broadcom hardware to test out a solution suggested below.)

Quoted 24 lines · Florian Fainelli

Hi Mike,

Quoted 21 lines · Mike Rapoport

Hi Florian,

Quoted 18 lines · Florian Fainelli

[...]

Hi Roman, Thomas and other linux-mips folks,

Kamal and myself have been unable to boot v5.11 on MIPS since this commit, reverting it makes our MIPS platforms boot successfully. We do not see a warning like this one in the commit message, instead what happens appear to be a corrupted Device Tree which prevents the parsing of the "rdb" node and leading to the interrupt controllers not being registered, and the system eventually not booting.

The Device Tree is built-into the kernel image and resides at arch/mips/boot/dts/brcm/bcm97435svmb.dts.

Do you have any idea what could be wrong with MIPS specifically here?

Most likely the problem you've discovered has been there for quite some time. The patch you are referring to just caused it to be triggered by extending the early allocation range. See before that patch was accepted the early memory allocations had been performed in the range:
[kernel_end, RAM_END].
The patch changed that, so the early allocations are done within [RAM_START + PAGE_SIZE, RAM_END].

In normal situations it's safe to do that as long as all the critical memory regions (including the memory residing a space below the kernel) have been reserved. But as soon as a memory with some critical structures haven't been reserved, the kernel may allocate it to be used for instance for early initializations with obviously unpredictable but most of the times unpleasant consequences.

Quoted 4 lines

Quoted 4 lines

Apparently there is a memblock allocation in one of the functions called from arch_mem_init() between plat_mem_setup() and early_init_fdt_reserve_self().

Mike, alas according to the log provided by Florian that's not the reason of the problem. Please, see my considerations below.

Quoted 8 lines

[...]

[    0.000000] Linux version 5.11.0-g5695e5161974 (florian@localhost)
(mipsel-linux-gcc (GCC) 8.3.0, GNU ld (GNU Binutils) 2.32) #84 SMP Sun
Feb 28 10:01:50 PST 2021
[    0.000000] CPU0 revision is: 00025b00 (Broadcom BMIPS5200)
[    0.000000] FPU revision is: 00130001

Quoted 6 lines

[    0.000000] memblock_add: [0x00000000-0x0fffffff]
early_init_dt_scan_memory+0x160/0x1e0
[    0.000000] memblock_add: [0x20000000-0x4fffffff]
early_init_dt_scan_memory+0x160/0x1e0
[    0.000000] memblock_add: [0x90000000-0xcfffffff]

early_init_dt_scan_memory+0x160/0x1e0

Here the memory has been added to the memblock allocator.

Quoted 6 lines

[    0.000000] MIPS: machine is Broadcom BCM97435SVMB
[    0.000000] earlycon: ns16550a0 at MMIO32 0x10406b00 (options '')
[    0.000000] printk: bootconsole [ns16550a0] enabled

[    0.000000] memblock_reserve: [0x00aa7600-0x00aaa0a0]

setup_arch+0x128/0x69c

Here the fdt memory has been reserved. (Note it's built into the kernel.)

Quoted 2 lines

[    0.000000] memblock_reserve: [0x00010000-0x018313cf]

setup_arch+0x1f8/0x69c

Here the kernel itself together with built-in dtb have been reserved. So far so good.

Quoted 11 lines

[    0.000000] Initrd not found or empty - disabling initrd

[    0.000000] memblock_alloc_try_nid: 10913 bytes align=0x40 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x00001000-0x00003aa0]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32680 bytes align=0x4 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x00003aa4-0x0000ba4b]

Quoted 1 line

memblock_alloc_range_nid+0xf8/0x198

The log above most likely belongs to the call-chain:
setup_arch()

+-> arch_mem_init()
    +-> device_tree_init() - BMIPS specific method
        +-> unflatten_and_copy_device_tree()

So to speak here we've copied the fdt from the original space [0x00aa7600-0x00aaa0a0] into [0x00001000-0x00003aa0] and unflattened it to [0x00003aa4-0x0000ba4b].

The problem is that a bit later the next call-chain is performed:
setup_arch()

+-> plat_smp_setup()
    +-> mp_ops->smp_setup(); - registered by prom_init()->register_bmips_smp_ops();
        +-> if (!board_ebase_setup)
                 board_ebase_setup = &bmips_ebase_setup;

So at the moment of the CPU traps initialization the bmips_ebase_setup() method is called. What trap_init() does isn't compatible with the allocation performed by the unflatten_and_copy_device_tree() method. See the next comment.

Quoted 3 lines

[    0.000000] memblock_alloc_try_nid: 25 bytes align=0x4 nid=-1

from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84

Quoted 54 lines

Quoted 5 lines

[    0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536

bytes, linear)

[    0.000000] memblock_reserve: [0x00000000-0x000003ff]

trap_init+0x70/0x4e8

Most likely someplace here the corruption has happened. The log above has just reserved a memory for NMI/reset vectors:
arch/mips/kernel/traps.c: trap_init(void): Line 2373.

But then the board_ebase_setup() pointer is dereferenced and called, which has been initialized with bmips_ebase_setup() earlier and which overwrites the ebase variable with: 0x80001000 as this is CPU_BMIPS5000 CPU. So any further calls of the functions like set_handler()/set_except_vector()/set_vi_srs_handler()/etc may cause a corruption of the memory above 0x80001000, which as we have discovered belongs to fdt and unflattened device tree.

Quoted 11 lines

[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 2045268K/2097152K available (8226K kernel code,
1070K rwdata, 1336K rodata, 13808K init, 260K bss, 51884K reserved, 0K
cma-reserved, 1835008K highmem)
[    0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay
is 25 jiffies.
[    0.000000] NR_IRQS: 256

Quoted 3 lines

[    0.000000] OF: Bad cell count for /rdb
[    0.000000] irq_bcm7038_l1: failed to remap intc L1 registers
[    0.000000] OF: of_irq_init: children remain, but no parents

So here is the first time we have got the consequence of the corruption popped up. Luckily it's just the "Bad cells count" error. We could have got much less obvious log here up to getting a crash at some place further...

Quoted 10 lines

[    0.000000] random: get_random_bytes called from
start_kernel+0x444/0x654 with crng_init=0
[    0.000000] sched_clock: 32 bits at 250 Hz, resolution 4000000ns,

wraps every 8589934590000000ns

and with your patch applied which unfortunately did not work we have the following:

[...]

So a patch like this shall workaround the corruption:

arch/mips/bmips/setup.c void __init plat_mem_setup(void) · line 181

Quoted 8 lines

arch/mips/bmips/setup.c +2 -0

@@ -174,6 +174,8 @@ void __init plat_mem_setup(void)
 
 	__dt_setup_arch(dtb);
 
+	memblock_reserve(0x0, 0x1000 + 0x100*64);
+
 	for (q = bmips_quirk_list; q->quirk_fn; q++) {
 		if (of_flat_dt_is_compatible(of_get_flat_dt_root(),
 					     q->compatible)) {

This patch works, thanks a lot for the troubleshooting and analysis! How about the following which would be more generic and works as well and should be more universal since it does not require each architecture to provide an appropriate call to memblock_reserve():

arch/mips/kernel/traps.c +0 -3

index e0352958e2f7..b0a173b500e8 100644
@@ -2367,10 +2367,7 @@ void __init trap_init(void)
 
        if (!cpu_has_mips_r2_r6) {
                ebase = CAC_BASE;
-               ebase_pa = virt_to_phys((void *)ebase);
                vec_size = 0x400;
-
-               memblock_reserve(ebase_pa, vec_size);
        } else {
                if (cpu_has_veic || cpu_has_vint)
                        vec_size = 0x200 + VECTORSPACING*64;

@@ -2410,6 +2407,14 @@ void __init trap_init(void)
 

arch/mips/kernel/traps.c +0 -0

        per_cpu_trap_init(true);
        memblock_set_bottom_up(false);

Quoted 132 lines · Florian Fainelli

Hi Serge,

Quoted 129 lines · Serge Semin

Hi folks,
What you've got here seems a more complicated problem than it could originally look like. Please, see my comments below.

(Note I've discarded some of the email logs, which of no interest to the discovered problem. Please also note that I haven't got any Broadcom hardware to test out a solution suggested below.)

Quoted 24 lines · Florian Fainelli

Hi Mike,

Quoted 21 lines · Mike Rapoport

Hi Florian,

Quoted 18 lines · Florian Fainelli

[...]

Hi Roman, Thomas and other linux-mips folks,

Kamal and myself have been unable to boot v5.11 on MIPS since this commit, reverting it makes our MIPS platforms boot successfully. We do not see a warning like this one in the commit message, instead what happens appear to be a corrupted Device Tree which prevents the parsing of the "rdb" node and leading to the interrupt controllers not being registered, and the system eventually not booting.

The Device Tree is built-into the kernel image and resides at arch/mips/boot/dts/brcm/bcm97435svmb.dts.

Do you have any idea what could be wrong with MIPS specifically here?

Most likely the problem you've discovered has been there for quite some time. The patch you are referring to just caused it to be triggered by extending the early allocation range. See before that patch was accepted the early memory allocations had been performed in the range:
[kernel_end, RAM_END].
The patch changed that, so the early allocations are done within [RAM_START + PAGE_SIZE, RAM_END].

In normal situations it's safe to do that as long as all the critical memory regions (including the memory residing a space below the kernel) have been reserved. But as soon as a memory with some critical structures haven't been reserved, the kernel may allocate it to be used for instance for early initializations with obviously unpredictable but most of the times unpleasant consequences.

Quoted 4 lines

Quoted 4 lines

Apparently there is a memblock allocation in one of the functions called from arch_mem_init() between plat_mem_setup() and early_init_fdt_reserve_self().

Mike, alas according to the log provided by Florian that's not the reason of the problem. Please, see my considerations below.

Quoted 8 lines

[...]

[    0.000000] Linux version 5.11.0-g5695e5161974 (florian@localhost)
(mipsel-linux-gcc (GCC) 8.3.0, GNU ld (GNU Binutils) 2.32) #84 SMP Sun
Feb 28 10:01:50 PST 2021
[    0.000000] CPU0 revision is: 00025b00 (Broadcom BMIPS5200)
[    0.000000] FPU revision is: 00130001

Quoted 6 lines

[    0.000000] memblock_add: [0x00000000-0x0fffffff]
early_init_dt_scan_memory+0x160/0x1e0
[    0.000000] memblock_add: [0x20000000-0x4fffffff]
early_init_dt_scan_memory+0x160/0x1e0
[    0.000000] memblock_add: [0x90000000-0xcfffffff]

early_init_dt_scan_memory+0x160/0x1e0

Here the memory has been added to the memblock allocator.

Quoted 6 lines

[    0.000000] MIPS: machine is Broadcom BCM97435SVMB
[    0.000000] earlycon: ns16550a0 at MMIO32 0x10406b00 (options '')
[    0.000000] printk: bootconsole [ns16550a0] enabled

[    0.000000] memblock_reserve: [0x00aa7600-0x00aaa0a0]

setup_arch+0x128/0x69c

Here the fdt memory has been reserved. (Note it's built into the kernel.)

Quoted 2 lines

[    0.000000] memblock_reserve: [0x00010000-0x018313cf]

setup_arch+0x1f8/0x69c

Here the kernel itself together with built-in dtb have been reserved. So far so good.

Quoted 11 lines

[    0.000000] Initrd not found or empty - disabling initrd

[    0.000000] memblock_alloc_try_nid: 10913 bytes align=0x40 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x00001000-0x00003aa0]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32680 bytes align=0x4 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x00003aa4-0x0000ba4b]

Quoted 1 line

memblock_alloc_range_nid+0xf8/0x198

The log above most likely belongs to the call-chain:
setup_arch()

+-> arch_mem_init()
    +-> device_tree_init() - BMIPS specific method
        +-> unflatten_and_copy_device_tree()

So to speak here we've copied the fdt from the original space [0x00aa7600-0x00aaa0a0] into [0x00001000-0x00003aa0] and unflattened it to [0x00003aa4-0x0000ba4b].

The problem is that a bit later the next call-chain is performed:
setup_arch()

+-> plat_smp_setup()
    +-> mp_ops->smp_setup(); - registered by prom_init()->register_bmips_smp_ops();
        +-> if (!board_ebase_setup)
                 board_ebase_setup = &bmips_ebase_setup;

So at the moment of the CPU traps initialization the bmips_ebase_setup() method is called. What trap_init() does isn't compatible with the allocation performed by the unflatten_and_copy_device_tree() method. See the next comment.

Quoted 3 lines

[    0.000000] memblock_alloc_try_nid: 25 bytes align=0x4 nid=-1

from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84

...

Quoted 85 lines

Quoted 54 lines

Quoted 5 lines

[    0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536

bytes, linear)

[    0.000000] memblock_reserve: [0x00000000-0x000003ff]

trap_init+0x70/0x4e8

Most likely someplace here the corruption has happened. The log above has just reserved a memory for NMI/reset vectors:
arch/mips/kernel/traps.c: trap_init(void): Line 2373.

But then the board_ebase_setup() pointer is dereferenced and called, which has been initialized with bmips_ebase_setup() earlier and which overwrites the ebase variable with: 0x80001000 as this is CPU_BMIPS5000 CPU. So any further calls of the functions like set_handler()/set_except_vector()/set_vi_srs_handler()/etc may cause a corruption of the memory above 0x80001000, which as we have discovered belongs to fdt and unflattened device tree.

Quoted 11 lines

[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 2045268K/2097152K available (8226K kernel code,
1070K rwdata, 1336K rodata, 13808K init, 260K bss, 51884K reserved, 0K
cma-reserved, 1835008K highmem)
[    0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay
is 25 jiffies.
[    0.000000] NR_IRQS: 256

Quoted 3 lines

[    0.000000] OF: Bad cell count for /rdb
[    0.000000] irq_bcm7038_l1: failed to remap intc L1 registers
[    0.000000] OF: of_irq_init: children remain, but no parents

So here is the first time we have got the consequence of the corruption popped up. Luckily it's just the "Bad cells count" error. We could have got much less obvious log here up to getting a crash at some place further...

Quoted 10 lines

[    0.000000] random: get_random_bytes called from
start_kernel+0x444/0x654 with crng_init=0
[    0.000000] sched_clock: 32 bits at 250 Hz, resolution 4000000ns,

wraps every 8589934590000000ns

and with your patch applied which unfortunately did not work we have the following:

[...]

So a patch like this shall workaround the corruption:

arch/mips/bmips/setup.c void __init plat_mem_setup(void) · line 181

Quoted 8 lines

arch/mips/bmips/setup.c +2 -0

@@ -174,6 +174,8 @@ void __init plat_mem_setup(void)
 
 	__dt_setup_arch(dtb);
 
+	memblock_reserve(0x0, 0x1000 + 0x100*64);
+
 	for (q = bmips_quirk_list; q->quirk_fn; q++) {
 		if (of_flat_dt_is_compatible(of_get_flat_dt_root(),
 					     q->compatible)) {

This patch works, thanks a lot for the troubleshooting and analysis! How about the following which would be more generic and works as well and should be more universal since it does not require each architecture to provide an appropriate call to memblock_reserve():

arch/mips/kernel/traps.c +0 -3

index e0352958e2f7..b0a173b500e8 100644
@@ -2367,10 +2367,7 @@ void __init trap_init(void)
 
        if (!cpu_has_mips_r2_r6) {
                ebase = CAC_BASE;
-               ebase_pa = virt_to_phys((void *)ebase);
                vec_size = 0x400;
-
-               memblock_reserve(ebase_pa, vec_size);
        } else {
                if (cpu_has_veic || cpu_has_vint)
                        vec_size = 0x200 + VECTORSPACING*64;

@@ -2410,6 +2407,14 @@ void __init trap_init(void)
 

arch/mips/kernel/traps.c void __init trap_init(void) · line 2417

Quoted 10 lines

arch/mips/kernel/traps.c +8 -0

        if (board_ebase_setup)
                board_ebase_setup();
+
+       /* board_ebase_setup() can change the exception base address
+        * reserve it now after changes were made.
+        */
+       if (!cpu_has_mips_r2_r6) {
+               ebase_pa = virt_to_phys((void *)ebase);
+               memblock_reserve(ebase_pa, vec_size);
+       }

With this it's still possible to have memblock allocations around ebase_pa before it is reserved.

I think we have two options here to solve it in more or less generic way:

* split the reservation of ebase from traps_init() and move it earlier to setup_arch(). I didn't check what board_ebase_setup() do, if they need to allocate memory it would not work.

* add an API to memblock to set lower limit for allocations and then set the lower limit, to e.g. kernel load address in arch_mem_init(). This may add complexity for configurations with relocatable kernel and kaslr.

Quoted 221 lines · Mike Rapoport

Quoted 132 lines · Florian Fainelli

Hi Serge,

Quoted 129 lines · Serge Semin

Hi folks,
What you've got here seems a more complicated problem than it could originally look like. Please, see my comments below.

(Note I've discarded some of the email logs, which of no interest to the discovered problem. Please also note that I haven't got any Broadcom hardware to test out a solution suggested below.)

Quoted 24 lines · Florian Fainelli

Hi Mike,

Quoted 21 lines · Mike Rapoport

Hi Florian,

Quoted 18 lines · Florian Fainelli

[...]

Hi Roman, Thomas and other linux-mips folks,

Kamal and myself have been unable to boot v5.11 on MIPS since this commit, reverting it makes our MIPS platforms boot successfully. We do not see a warning like this one in the commit message, instead what happens appear to be a corrupted Device Tree which prevents the parsing of the "rdb" node and leading to the interrupt controllers not being registered, and the system eventually not booting.

The Device Tree is built-into the kernel image and resides at arch/mips/boot/dts/brcm/bcm97435svmb.dts.

Do you have any idea what could be wrong with MIPS specifically here?

Most likely the problem you've discovered has been there for quite some time. The patch you are referring to just caused it to be triggered by extending the early allocation range. See before that patch was accepted the early memory allocations had been performed in the range:
[kernel_end, RAM_END].
The patch changed that, so the early allocations are done within [RAM_START + PAGE_SIZE, RAM_END].

In normal situations it's safe to do that as long as all the critical memory regions (including the memory residing a space below the kernel) have been reserved. But as soon as a memory with some critical structures haven't been reserved, the kernel may allocate it to be used for instance for early initializations with obviously unpredictable but most of the times unpleasant consequences.

Quoted 4 lines

Quoted 4 lines

Apparently there is a memblock allocation in one of the functions called from arch_mem_init() between plat_mem_setup() and early_init_fdt_reserve_self().

Mike, alas according to the log provided by Florian that's not the reason of the problem. Please, see my considerations below.

Quoted 8 lines

[...]

[    0.000000] Linux version 5.11.0-g5695e5161974 (florian@localhost)
(mipsel-linux-gcc (GCC) 8.3.0, GNU ld (GNU Binutils) 2.32) #84 SMP Sun
Feb 28 10:01:50 PST 2021
[    0.000000] CPU0 revision is: 00025b00 (Broadcom BMIPS5200)
[    0.000000] FPU revision is: 00130001

Quoted 6 lines

[    0.000000] memblock_add: [0x00000000-0x0fffffff]
early_init_dt_scan_memory+0x160/0x1e0
[    0.000000] memblock_add: [0x20000000-0x4fffffff]
early_init_dt_scan_memory+0x160/0x1e0
[    0.000000] memblock_add: [0x90000000-0xcfffffff]

early_init_dt_scan_memory+0x160/0x1e0

Here the memory has been added to the memblock allocator.

Quoted 6 lines

[    0.000000] MIPS: machine is Broadcom BCM97435SVMB
[    0.000000] earlycon: ns16550a0 at MMIO32 0x10406b00 (options '')
[    0.000000] printk: bootconsole [ns16550a0] enabled

[    0.000000] memblock_reserve: [0x00aa7600-0x00aaa0a0]

setup_arch+0x128/0x69c

Here the fdt memory has been reserved. (Note it's built into the kernel.)

Quoted 2 lines

[    0.000000] memblock_reserve: [0x00010000-0x018313cf]

setup_arch+0x1f8/0x69c

Here the kernel itself together with built-in dtb have been reserved. So far so good.

Quoted 11 lines

[    0.000000] Initrd not found or empty - disabling initrd

[    0.000000] memblock_alloc_try_nid: 10913 bytes align=0x40 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x00001000-0x00003aa0]
memblock_alloc_range_nid+0xf8/0x198
[    0.000000] memblock_alloc_try_nid: 32680 bytes align=0x4 nid=-1
from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84
[    0.000000] memblock_reserve: [0x00003aa4-0x0000ba4b]

Quoted 1 line

memblock_alloc_range_nid+0xf8/0x198

The log above most likely belongs to the call-chain:
setup_arch()

+-> arch_mem_init()
    +-> device_tree_init() - BMIPS specific method
        +-> unflatten_and_copy_device_tree()

So to speak here we've copied the fdt from the original space [0x00aa7600-0x00aaa0a0] into [0x00001000-0x00003aa0] and unflattened it to [0x00003aa4-0x0000ba4b].

The problem is that a bit later the next call-chain is performed:
setup_arch()

+-> plat_smp_setup()
    +-> mp_ops->smp_setup(); - registered by prom_init()->register_bmips_smp_ops();
        +-> if (!board_ebase_setup)
                 board_ebase_setup = &bmips_ebase_setup;

So at the moment of the CPU traps initialization the bmips_ebase_setup() method is called. What trap_init() does isn't compatible with the allocation performed by the unflatten_and_copy_device_tree() method. See the next comment.

Quoted 3 lines

[    0.000000] memblock_alloc_try_nid: 25 bytes align=0x4 nid=-1

from=0x00000000 max_addr=0x00000000
early_init_dt_alloc_memory_arch+0x40/0x84

...

Quoted 85 lines

Quoted 54 lines

Quoted 5 lines

[    0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536

bytes, linear)

[    0.000000] memblock_reserve: [0x00000000-0x000003ff]

trap_init+0x70/0x4e8

Most likely someplace here the corruption has happened. The log above has just reserved a memory for NMI/reset vectors:
arch/mips/kernel/traps.c: trap_init(void): Line 2373.

But then the board_ebase_setup() pointer is dereferenced and called, which has been initialized with bmips_ebase_setup() earlier and which overwrites the ebase variable with: 0x80001000 as this is CPU_BMIPS5000 CPU. So any further calls of the functions like set_handler()/set_except_vector()/set_vi_srs_handler()/etc may cause a corruption of the memory above 0x80001000, which as we have discovered belongs to fdt and unflattened device tree.

Quoted 11 lines

[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 2045268K/2097152K available (8226K kernel code,
1070K rwdata, 1336K rodata, 13808K init, 260K bss, 51884K reserved, 0K
cma-reserved, 1835008K highmem)
[    0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay
is 25 jiffies.
[    0.000000] NR_IRQS: 256

Quoted 3 lines

[    0.000000] OF: Bad cell count for /rdb
[    0.000000] irq_bcm7038_l1: failed to remap intc L1 registers
[    0.000000] OF: of_irq_init: children remain, but no parents

So here is the first time we have got the consequence of the corruption popped up. Luckily it's just the "Bad cells count" error. We could have got much less obvious log here up to getting a crash at some place further...

Quoted 10 lines

[    0.000000] random: get_random_bytes called from
start_kernel+0x444/0x654 with crng_init=0
[    0.000000] sched_clock: 32 bits at 250 Hz, resolution 4000000ns,

wraps every 8589934590000000ns

and with your patch applied which unfortunately did not work we have the following:

[...]

So a patch like this shall workaround the corruption:

arch/mips/bmips/setup.c void __init plat_mem_setup(void) · line 181

Quoted 8 lines

arch/mips/bmips/setup.c +2 -0

@@ -174,6 +174,8 @@ void __init plat_mem_setup(void)
 
 	__dt_setup_arch(dtb);
 
+	memblock_reserve(0x0, 0x1000 + 0x100*64);
+
 	for (q = bmips_quirk_list; q->quirk_fn; q++) {
 		if (of_flat_dt_is_compatible(of_get_flat_dt_root(),
 					     q->compatible)) {

This patch works, thanks a lot for the troubleshooting and analysis! How about the following which would be more generic and works as well and should be more universal since it does not require each architecture to provide an appropriate call to memblock_reserve():

arch/mips/kernel/traps.c +0 -3

index e0352958e2f7..b0a173b500e8 100644
@@ -2367,10 +2367,7 @@ void __init trap_init(void)
 
        if (!cpu_has_mips_r2_r6) {
                ebase = CAC_BASE;
-               ebase_pa = virt_to_phys((void *)ebase);
                vec_size = 0x400;
-
-               memblock_reserve(ebase_pa, vec_size);
        } else {
                if (cpu_has_veic || cpu_has_vint)
                        vec_size = 0x200 + VECTORSPACING*64;

@@ -2410,6 +2407,14 @@ void __init trap_init(void)
 

Quoted 10 lines

arch/mips/kernel/traps.c void __init trap_init(void) · line 2417

Quoted 10 lines

arch/mips/kernel/traps.c +8 -0

        if (board_ebase_setup)
                board_ebase_setup();
+
+       /* board_ebase_setup() can change the exception base address
+        * reserve it now after changes were made.
+        */
+       if (!cpu_has_mips_r2_r6) {
+               ebase_pa = virt_to_phys((void *)ebase);
+               memblock_reserve(ebase_pa, vec_size);
+       }

Hi folks!

First, I'm really sorry for breaking things and also being silent for last couple of days: I was almost completely offline. Thank you for working on this!

Quoted 9 lines

With this it's still possible to have memblock allocations around ebase_pa before it is reserved.

I think we have two options here to solve it in more or less generic way:

* split the reservation of ebase from traps_init() and move it earlier to setup_arch(). I didn't check what board_ebase_setup() do, if they need to allocate memory it would not work.

It seems that it doesn't allocate any memory, so it sounds like a good option. But doesn't the ebase initialization depend on the memblock allocator?

I see in trap_init():

    if (!cpu_has_mips_r2_r6) {
        ...
    } else {
        ...
	ebase_pa = memblock_phys_alloc(vec_size, 1 << fls(vec_size));
	...
	if (!IS_ENABLED(CONFIG_EVA) && !WARN_ON(ebase_pa >= 0x20000000))
	    ebase = CKSEG0ADDR(ebase_pa);
        else
            ebase = (unsigned long)phys_to_virt(ebase_pa);

Quoted 6 lines

* add an API to memblock to set lower limit for allocations and then set the lower limit, to e.g. kernel load address in arch_mem_init(). This may add complexity for configurations with relocatable kernel and kaslr.

This option looks more like a workaround to me, but maybe it's ok too.

Thanks!

Currently cma areas without a fixed base are allocated close to the end of the node. This placement is sub-optimal because of compaction:
it brings pages into the cma area. In particular, it can bring in hot executable pages, even if there is a plenty of free memory on the machine. This results in cma allocation failures.

Instead let's place cma areas close to the beginning of a node. In this case the compaction will help to free cma areas, resulting in better cma allocation success rates.

If there is enough memory let's try to allocate bottom-up starting with 4GB to exclude any possible interference with DMA32. On smaller machines or in a case of a failure, stick with the old behavior.

16GB vm, 2GB cma area:
With this patch:

[    0.000000] Command line: root=/dev/vda3 rootflags=subvol=/root systemd.unified_cgroup_hierarchy=1 enforcing=0 console=ttyS0,115200 hugetlb_cma=2G
[    0.002928] hugetlb_cma: reserve 2048 MiB, up to 2048 MiB per node
[    0.002930] cma: Reserved 2048 MiB at 0x0000000100000000
[    0.002931] hugetlb_cma: reserved 2048 MiB on node 0

Without this patch:

[    0.000000] Command line: root=/dev/vda3 rootflags=subvol=/root systemd.unified_cgroup_hierarchy=1 enforcing=0 console=ttyS0,115200 hugetlb_cma=2G
[    0.002930] hugetlb_cma: reserve 2048 MiB, up to 2048 MiB per node
[    0.002933] cma: Reserved 2048 MiB at 0x00000003c0000000
[    0.002934] hugetlb_cma: reserved 2048 MiB on node 0

v2:
- switched to memblock_set_bottom_up(true), by Mike
- start with 4GB, by Mike

Signed-off-by Roman Gushchin <[email protected]>

---
 mm/cma.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

mm/cma.c +0 -0

index 7f415d7cda9f..21fd40c092f0 100644
@@ -337,6 +337,22 @@ int __init cma_declare_contiguous_nid(phys_addr_t base,
 			limit = highmem_start;
 		}
 

mm/cma.c +6 -0

+			memblock_set_bottom_up(true);
+			addr = memblock_alloc_range_nid(size, alignment, SZ_4G,
+							limit, nid, true);
+			memblock_set_bottom_up(false);
+		}
+
 		if (!addr) {
 			addr = memblock_alloc_range_nid(size, alignment, base,
 					limit, nid, true);

Signature · 1 line

2.26.2

Quoted 27 lines · Roman Gushchin

Currently cma areas without a fixed base are allocated close to the end of the node. This placement is sub-optimal because of compaction:
it brings pages into the cma area. In particular, it can bring in hot executable pages, even if there is a plenty of free memory on the machine. This results in cma allocation failures.

Instead let's place cma areas close to the beginning of a node. In this case the compaction will help to free cma areas, resulting in better cma allocation success rates.

If there is enough memory let's try to allocate bottom-up starting with 4GB to exclude any possible interference with DMA32. On smaller machines or in a case of a failure, stick with the old behavior.

16GB vm, 2GB cma area:
With this patch:

[    0.000000] Command line: root=/dev/vda3 rootflags=subvol=/root systemd.unified_cgroup_hierarchy=1 enforcing=0 console=ttyS0,115200 hugetlb_cma=2G
[    0.002928] hugetlb_cma: reserve 2048 MiB, up to 2048 MiB per node
[    0.002930] cma: Reserved 2048 MiB at 0x0000000100000000
[    0.002931] hugetlb_cma: reserved 2048 MiB on node 0

Without this patch:

[    0.000000] Command line: root=/dev/vda3 rootflags=subvol=/root systemd.unified_cgroup_hierarchy=1 enforcing=0 console=ttyS0,115200 hugetlb_cma=2G
[    0.002930] hugetlb_cma: reserve 2048 MiB, up to 2048 MiB per node
[    0.002933] cma: Reserved 2048 MiB at 0x00000003c0000000
[    0.002934] hugetlb_cma: reserved 2048 MiB on node 0

Quoted 5 lines

v2:
- switched to memblock_set_bottom_up(true), by Mike
- start with 4GB, by Mike

Signed-off-by Roman Gushchin <[email protected]>

With one nit below

Reviewed-by Mike Rapoport <[email protected]>

Quoted 12 lines

---
 mm/cma.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

mm/cma.c +0 -0

index 7f415d7cda9f..21fd40c092f0 100644
@@ -337,6 +337,22 @@ int __init cma_declare_contiguous_nid(phys_addr_t base,
 			limit = highmem_start;
 		}
 

mm/cma.c int __init cma_declare_contiguous_nid(phys_addr_t base, · line 349

Quoted 10 lines

mm/cma.c +10 -0

+		/*
+		 * If there is enough memory, try a bottom-up allocation first.
+		 * It will place the new cma area close to the start of the node
+		 * and guarantee that the compaction is moving pages out of the
+		 * cma area and not into it.
+		 * Avoid using first 4GB to not interfere with constrained zones
+		 * like DMA/DMA32.
+		 */
+		if (!memblock_bottom_up() &&
+		    memblock_end >= SZ_4G + size) {

This seems short enough to fit a single line

Subject: [PATCH v3 1/2] mm: cma: allocate cma areas bottom-up

Quoted 1 line · On Mon, 21 Dec 2020 09:05:51 -0800 Roman Gushchin <[email protected]> wrote:

Subject: [PATCH v3 1/2] mm: cma: allocate cma areas bottom-up

i386 allmodconfig:

In file included from ./include/vdso/const.h:5,

                 from ./include/linux/const.h:4,
                 from ./include/linux/bits.h:5,
                 from ./include/linux/bitops.h:6,
                 from ./include/linux/kernel.h:11,
                 from ./include/asm-generic/bug.h:20,
                 from ./arch/x86/include/asm/bug.h:93,
                 from ./include/linux/bug.h:5,
                 from ./include/linux/mmdebug.h:5,
                 from ./include/linux/mm.h:9,
                 from ./include/linux/memblock.h:13,
                 from mm/cma.c:24:
mm/cma.c: In function ‘cma_declare_contiguous_nid’:
./include/uapi/linux/const.h:20:19: warning: conversion from ‘long long unsigned int’ to ‘phys_addr_t’ {aka ‘unsigned int’} changes value from ‘4294967296’ to ‘0’ [-Woverflow]
 #define __AC(X,Y) (X##Y)
                   ^~~~~~
./include/uapi/linux/const.h:21:18: note: in expansion of macro ‘__AC’
 #define _AC(X,Y) __AC(X,Y)
                  ^~~~
./include/linux/sizes.h:46:18: note: in expansion of macro ‘_AC’
 #define SZ_4G    _AC(0x100000000, ULL)
                  ^~~

mm/cma.c:349:53: note: in expansion of macro ‘SZ_4G’
addr = memblock_alloc_range_nid(size, alignment, SZ_4G,
^~~~~

Quoted 28 lines · Andrew Morton

Quoted 1 line · On Mon, 21 Dec 2020 09:05:51 -0800 Roman Gushchin <[email protected]> wrote:

Subject: [PATCH v3 1/2] mm: cma: allocate cma areas bottom-up

i386 allmodconfig:

In file included from ./include/vdso/const.h:5,

                 from ./include/linux/const.h:4,
                 from ./include/linux/bits.h:5,
                 from ./include/linux/bitops.h:6,
                 from ./include/linux/kernel.h:11,
                 from ./include/asm-generic/bug.h:20,
                 from ./arch/x86/include/asm/bug.h:93,
                 from ./include/linux/bug.h:5,
                 from ./include/linux/mmdebug.h:5,
                 from ./include/linux/mm.h:9,
                 from ./include/linux/memblock.h:13,
                 from mm/cma.c:24:
mm/cma.c: In function ‘cma_declare_contiguous_nid’:
./include/uapi/linux/const.h:20:19: warning: conversion from ‘long long unsigned int’ to ‘phys_addr_t’ {aka ‘unsigned int’} changes value from ‘4294967296’ to ‘0’ [-Woverflow]
 #define __AC(X,Y) (X##Y)
                   ^~~~~~
./include/uapi/linux/const.h:21:18: note: in expansion of macro ‘__AC’
 #define _AC(X,Y) __AC(X,Y)
                  ^~~~
./include/linux/sizes.h:46:18: note: in expansion of macro ‘_AC’
 #define SZ_4G    _AC(0x100000000, ULL)
                  ^~~

Quoted 4 lines

mm/cma.c:349:53: note: in expansion of macro ‘SZ_4G’
addr = memblock_alloc_range_nid(size, alignment, SZ_4G,
^~~~~

I thought that (!memblock_bottom_up() && memblock_end >= SZ_4G + size) can't be true on a 32-bit platform, so the whole if clause can be compiled out. Maybe it's because memblock_end can be equal to SZ_4G and if the size == 0...

I have no better idea than wrapping everything into #if BITS_PER_LONG > 32
#endif.

Thanks!

--

mm/cma.c +2 -0

index 4fe74c9d83b0..5d69b498603a 100644
@@ -344,12 +344,14 @@ int __init cma_declare_contiguous_nid(phys_addr_t base,
                 * Avoid using first 4GB to not interfere with constrained zones
                 * like DMA/DMA32.
                 */
+#if BITS_PER_LONG > 32
                if (!memblock_bottom_up() && memblock_end >= SZ_4G + size) {
                        memblock_set_bottom_up(true);
                        addr = memblock_alloc_range_nid(size, alignment, SZ_4G,
                                                        limit, nid, true);
                        memblock_set_bottom_up(false);
                }
+#endif
 
                if (!addr) {
                        addr = memblock_alloc_range_nid(size, alignment, base,

Quoted 38 lines · Roman Gushchin

Quoted 28 lines · Andrew Morton

Quoted 1 line · On Mon, 21 Dec 2020 09:05:51 -0800 Roman Gushchin <[email protected]> wrote:

Subject: [PATCH v3 1/2] mm: cma: allocate cma areas bottom-up

i386 allmodconfig:

In file included from ./include/vdso/const.h:5,

                 from ./include/linux/const.h:4,
                 from ./include/linux/bits.h:5,
                 from ./include/linux/bitops.h:6,
                 from ./include/linux/kernel.h:11,
                 from ./include/asm-generic/bug.h:20,
                 from ./arch/x86/include/asm/bug.h:93,
                 from ./include/linux/bug.h:5,
                 from ./include/linux/mmdebug.h:5,
                 from ./include/linux/mm.h:9,
                 from ./include/linux/memblock.h:13,
                 from mm/cma.c:24:
mm/cma.c: In function ‘cma_declare_contiguous_nid’:
./include/uapi/linux/const.h:20:19: warning: conversion from ‘long long unsigned int’ to ‘phys_addr_t’ {aka ‘unsigned int’} changes value from ‘4294967296’ to ‘0’ [-Woverflow]
 #define __AC(X,Y) (X##Y)
                   ^~~~~~
./include/uapi/linux/const.h:21:18: note: in expansion of macro ‘__AC’
 #define _AC(X,Y) __AC(X,Y)
                  ^~~~
./include/linux/sizes.h:46:18: note: in expansion of macro ‘_AC’
 #define SZ_4G    _AC(0x100000000, ULL)
                  ^~~

Quoted 4 lines

mm/cma.c:349:53: note: in expansion of macro ‘SZ_4G’
addr = memblock_alloc_range_nid(size, alignment, SZ_4G,
^~~~~

I thought that (!memblock_bottom_up() && memblock_end >= SZ_4G + size) can't be true on a 32-bit platform, so the whole if clause can be compiled out. Maybe it's because memblock_end can be equal to SZ_4G and if the size == 0...

Quoted 3 lines

I have no better idea than wrapping everything into #if BITS_PER_LONG > 32
#endif.

32-bit systems can have more than 32 bit in the physical address. I think a better option would be to use CONFIG_PHYS_ADDR_T_64BIT