Skip to contentSkip to conversation

U-Boot loaded RAMDisk crashes Linux on MT7623

DISCUSSION Daniel Golle 2 participants 3 messages Jul 19, 2021 – Jul 20, 2021

lore
Share & tools
lore
#0 lore raw

Hi,

I writing in the hope that someone has a good idea about why U-boot is handing over a broken memory address for a loaded ramdisk which results in Linux crashing very early on boot on MediaTek's MT7623N SoC (ARMv7). If anyone has a good idea why this is happening, I'd be very glad, as this currently prevents me from updating that target in OpenWrt. Background:
OpenWrt used to have the initramdisk built-into the kernel itself. Having it separate is nicer as then you won't need to recompile the kernel or even have a compiler installed in order to modify the ramdisk. This already works great on MT7622 and it'd be great to have it the same way on MT7623 (and MT7629 in future).

So when loading a uImage.FIT with RAMDisk subimage to me it looks like U-Boot is not translating the address of the ramdisk correctly, see logs below:

U-Boot> tftpboot 0x88000000 openwrt-mediatek-mt7623-bpi_bananapi-r2-initramfs-recovery.itb Using ethernet@1b100000 device
TFTP from server 192.168.5.2; our IP address is 192.168.5.100
Filename 'openwrt-mediatek-mt7623-bpi_bananapi-r2-initramfs-recovery.itb'. Load address: 0x88000000
Loading: #################################################################

         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         13 MiB/s

done
Bytes transferred = 10492520 (a01a68 hex)
U-Boot> bootm
## Loading kernel from FIT Image at 88000000 ...
Using 'config-1' configuration
Trying 'kernel-1' kernel subimage

     Description:  ARM OpenWrt Linux-5.10.51
     Type:         Kernel Image
     Compression:  gzip compressed
     Data Start:   0x880000e4
     Data Size:    4944975 Bytes = 4.7 MiB
     Architecture: ARM
     OS:           Linux
     Load Address: 0x80008000
     Entry Point:  0x80008000
     Hash algo:    crc32
     Hash value:   9da8225f
     Hash algo:    sha1
     Hash value:   ea4e69501bed0925ecdee0bb6b3a3b489fedc38c

Verifying Hash Integrity ... crc32+ sha1+ OK
## Loading ramdisk from FIT Image at 88000000 ...
Using 'config-1' configuration
Trying 'initrd-1' ramdisk subimage

     Description:  ARM OpenWrt bpi_bananapi-r2 initrd
     Type:         RAMDisk Image
     Compression:  Unknown Compression
     Data Start:   0x884b7668
     Data Size:    5511960 Bytes = 5.3 MiB

Architecture: ARM
OS: Linux
Load Address: unavailable
Entry Point: unavailable
Hash algo: crc32
Hash value: 01e5fbf0
Hash algo: sha1
Hash value: 6ceb78df26920d97dee505ddeb0318e0c1522ba0
Verifying Hash Integrity ... crc32+ sha1+ OK
WARNING: 'compression' nodes for ramdisks are deprecated, please fix your .its file! ## Loading fdt from FIT Image at 88000000 ...
Using 'config-1' configuration
Trying 'fdt-1' fdt subimage

     Description:  ARM OpenWrt bpi_bananapi-r2 device tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x889f9288
     Data Size:    33453 Bytes = 32.7 KiB
     Architecture: ARM
     Hash algo:    crc32
     Hash value:   a2599155
     Hash algo:    sha1
     Hash value:   59c64737be8bda92b33417dfadca87e9c4662be2
   Verifying Hash Integrity ... crc32+ sha1+ OK
   Booting using the fdt blob at 0x889f9288
   Uncompressing Kernel Image
   Loading Ramdisk to ff4b9000, end ff9fab18 ... OK
                                   ^^^^^^^^^^

Using Device Tree in place at 889f9288, end 88a04534

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.10.51 (daniel@box) (arm-openwrt-linux-muslgnueabi-gcc (OpenWrt GCC 8.4.0 r17073+11-8bb4437c01) 8.4.0, GNU ld (GNU Binutils) 2.34) #0 SMP PREEMPT Mon Jul 19 12:26:15 2021
[    0.000000] CPU: ARMv7 Processor [410fc073] revision 3 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: Bananapi BPI-R2
[    0.000000] earlycon: uart8250 at MMIO32 0x11004000 (options '')
[    0.000000] printk: bootconsole [uart8250] enabled
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000080000000-0x00000000afffffff]
[    0.000000]   HighMem  [mem 0x00000000b0000000-0x00000000ffffefff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000ffffefff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000ffffefff]
[    0.000000] 8<--- cut here ---
[    0.000000] Unable to handle kernel paging request at virtual address 3f9fab0c
                                                                        ^^^^^^^^^^

Xf8fabXX looks familiar, see above. Just 3XXXXXXX seems wrong, or at least not where physical memory ended up being mapped in Linux. I also don't understand why U-Boot is reloacing the RAMdisk in first place, shouldn't it be fine to just pass the address where it is loaded inside the FIT structure (0x884b7668)?

[    0.000000] pgd = (ptrval)
[    0.000000] [3f9fab0c] *pgd=00000000
[    0.000000] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[    0.000000] Modules linked in:
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.10.51 #0
[    0.000000] Hardware name: Mediatek Cortex-A7 (Device Tree)
[    0.000000] PC is at memcmp+0x10/0x50
[    0.000000] LR is at start_kernel+0x8c/0x508
[    0.000000] pc : [<c04b8098>]    lr : [<c0c00a08>]    psr: 200000d3
[    0.000000] sp : c0d01fc0  ip : 3f9fab0c  fp : 00000000
[    0.000000] r10: 10c5387d  r9 : 3f9fab08  r8 : c09c007c
[    0.000000] r7 : 3f9fab18  r6 : c0da1038  r5 : c0d04ec0  r4 : 3f9fab0c
[    0.000000] r3 : 00000023  r2 : 0000000c  r1 : c09c007c  r0 : 3f9fab0c
[    0.000000] Flags: nzCv  IRQs off  FIQs off  Mode SVC_32  ISA ARM  Segment none
[    0.000000] Control: 10c5387d  Table: 8000406a  DAC: 00000051
[    0.000000] Process swapper (pid: 0, stack limit = 0x(ptrval))
[    0.000000] Stack: (0xc0d01fc0 to 0xc0d02000)
[    0.000000] 1fc0: 00000000 00000000 00000000 00000000 c0c28a54 00000000 00000000 c0c00330
[    0.000000] 1fe0: 00000051 10c0387d 00000000 889f9288 410fc073 00000000 00000000 00000000
[    0.000000] [<c04b8098>] (memcmp) from [<00000000>] (0x0)
[    0.000000] Code: e3520000 0a00000f e1a0c000 e5d13000 (e5d00000) 
[    0.000000] random: get_random_bytes called from print_oops_end_marker+0x24/0x4c with crng_init=0
[    0.000000] ---[ end trace 0000000000000000 ]---
[    0.000000] Kernel panic - not syncing: Fatal exception
[    0.000000] Rebooting in 1 seconds..
[    0.000000] Reboot failed -- System halted

Anyway, all ideas are welcome :)

Cheers

Daniel

#1 lore raw

Hi again,

Quoted 17 lines · Daniel Golle

...
Loading Ramdisk to ff4b9000, end ff9fab18 ... OK

                                   ^^^^^^^^^^

Using Device Tree in place at 889f9288, end 88a04534

Starting kernel ...
...

[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000080000000-0x00000000afffffff]
[    0.000000]   HighMem  [mem 0x00000000b0000000-0x00000000ffffefff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000ffffefff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000ffffefff]
[    0.000000] 8<--- cut here ---
[    0.000000] Unable to handle kernel paging request at virtual address 3f9fab0c
                                                                        ^^^^^^^^^^
Quoted 1 line

...

I've found the answer myself, thanks to another patch posted to this list, solving the same problem for another board:
[RFC 1/1] board: sifive: unmatched: use zero copy for initrd

So just like there, setting initrd_high (in this case to 0xafffffff) solved the problem.

Sorry for the noise :)

#2 lore raw
In reply to Daniel Golle

Hi Daniel,

Am Montag, den 19.07.2021, 18:34 +0100 schrieb Daniel Golle:

Quoted 80 lines

Hi,

I writing in the hope that someone has a good idea about why U-boot is
handing over a broken memory address for a loaded ramdisk which results
in Linux crashing very early on boot on MediaTek's MT7623N SoC (ARMv7).
If anyone has a good idea why this is happening, I'd be very glad, as this currently prevents me from updating that target in OpenWrt. Background:
OpenWrt used to have the initramdisk built-into the kernel itself. Having it separate is nicer as then you won't need to recompile the kernel or even have a compiler installed in order to modify the ramdisk. This already works great on MT7622 and it'd be great to have it the same way on MT7623 (and MT7629 in future).

So when loading a uImage.FIT with RAMDisk subimage to me it looks like
U-Boot is not translating the address of the ramdisk correctly, see logs below:

U-Boot> tftpboot 0x88000000 openwrt-mediatek-mt7623-bpi_bananapi-r2- initramfs-recovery.itb
Using ethernet@1b100000 device
TFTP from server 192.168.5.2; our IP address is 192.168.5.100 Filename 'openwrt-mediatek-mt7623-bpi_bananapi-r2-initramfs- recovery.itb'.
Load address: 0x88000000
Loading:
#################################################################

         ############################################################
#####
         ############################################################
#####
         ############################################################
#####
         ############################################################
#####
         ############################################################
#####
         ############################################################
#####
         ############################################################
#####
         ############################################################
#####
         ############################################################
#####
         ############################################################
#####
         13 MiB/s

done
Bytes transferred = 10492520 (a01a68 hex)
U-Boot> bootm
## Loading kernel from FIT Image at 88000000 ...
Using 'config-1' configuration
Trying 'kernel-1' kernel subimage

     Description:  ARM OpenWrt Linux-5.10.51
     Type:         Kernel Image
     Compression:  gzip compressed
     Data Start:   0x880000e4
     Data Size:    4944975 Bytes = 4.7 MiB
     Architecture: ARM
     OS:           Linux
     Load Address: 0x80008000
     Entry Point:  0x80008000
     Hash algo:    crc32
     Hash value:   9da8225f
     Hash algo:    sha1
     Hash value:   ea4e69501bed0925ecdee0bb6b3a3b489fedc38c

Verifying Hash Integrity ... crc32+ sha1+ OK
## Loading ramdisk from FIT Image at 88000000 ...
Using 'config-1' configuration
Trying 'initrd-1' ramdisk subimage

     Description:  ARM OpenWrt bpi_bananapi-r2 initrd
     Type:         RAMDisk Image
     Compression:  Unknown Compression
     Data Start:   0x884b7668
     Data Size:    5511960 Bytes = 5.3 MiB
Quoted 4 lines

Architecture: ARM
OS: Linux
Load Address: unavailable
Entry Point: unavailable

you could try to explicitely set load and entry address to 0.

Quoted 54 lines
     Hash algo:    crc32
     Hash value:   01e5fbf0
     Hash algo:    sha1
     Hash value:   6ceb78df26920d97dee505ddeb0318e0c1522ba0

Verifying Hash Integrity ... crc32+ sha1+ OK
WARNING: 'compression' nodes for ramdisks are deprecated, please fix your .its file!
## Loading fdt from FIT Image at 88000000 ...
Using 'config-1' configuration
Trying 'fdt-1' fdt subimage

     Description:  ARM OpenWrt bpi_bananapi-r2 device tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x889f9288
     Data Size:    33453 Bytes = 32.7 KiB
     Architecture: ARM
     Hash algo:    crc32
     Hash value:   a2599155
     Hash algo:    sha1
     Hash value:   59c64737be8bda92b33417dfadca87e9c4662be2
   Verifying Hash Integrity ... crc32+ sha1+ OK
   Booting using the fdt blob at 0x889f9288
   Uncompressing Kernel Image
   Loading Ramdisk to ff4b9000, end ff9fab18 ... OK
                                   ^^^^^^^^^^

Using Device Tree in place at 889f9288, end 88a04534

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.10.51 (daniel@box) (arm-openwrt-linux-
muslgnueabi-gcc (OpenWrt GCC 8.4.0 r17073+11-8bb4437c01) 8.4.0, GNU
ld (GNU Binutils) 2.34) #0 SMP PREEMPT Mon Jul 19 12:26:15 2021
[    0.000000] CPU: ARMv7 Processor [410fc073] revision 3 (ARMv7),
cr=10c5387d
[    0.000000] CPU: div instructions available: patching division
code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing
instruction cache
[    0.000000] OF: fdt: Machine model: Bananapi BPI-R2
[    0.000000] earlycon: uart8250 at MMIO32 0x11004000 (options '')
[    0.000000] printk: bootconsole [uart8250] enabled
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000080000000-0x00000000afffffff]
[    0.000000]   HighMem  [mem 0x00000000b0000000-0x00000000ffffefff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-
0x00000000ffffefff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-
0x00000000ffffefff]
[    0.000000] 8<--- cut here ---
[    0.000000] Unable to handle kernel paging request at virtual
Quoted 8 lines

address 3f9fab0c

   ^^^^^^^^^^

Xf8fabXX looks familiar, see above. Just 3XXXXXXX seems wrong, or at least not where physical memory ended up being mapped in Linux. I also don't understand why U-Boot is reloacing the RAMdisk in first place, shouldn't it be fine to just pass the address where it is loaded inside the FIT structure (0x884b7668)?

The bootm command calls boot_ramdisk_high()
when CONFIG_SYS_BOOT_RAMDISK_HIGH is enabled. This is the case for most archs.

If you look at that function, you can somewhat control the behavior either via env variables "initrd_high", "bootm_low", "bootm_size" or via gd->ram_base, gd->ram_size, gd->ram_top. Maybe you could debug and play with that function. According to the code an in-place initramfs is only possible, if you do a "setenv initrd_high 0xffffffff". But you should also check if ARM Linux has some special alignment requirements for initramfs and if a address of 0x884b7668 is useable (for example on MIPS, there is a 16 kiB boundary requirement and moving the initramfs with boot_ramdisk_high() accomplishes that).

Quoted 42 lines
[    0.000000] pgd = (ptrval)
[    0.000000] [3f9fab0c] *pgd=00000000
[    0.000000] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[    0.000000] Modules linked in:
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.10.51 #0
[    0.000000] Hardware name: Mediatek Cortex-A7 (Device Tree)
[    0.000000] PC is at memcmp+0x10/0x50
[    0.000000] LR is at start_kernel+0x8c/0x508
[    0.000000] pc : [<c04b8098>]    lr : [<c0c00a08>]    psr:
200000d3
[    0.000000] sp : c0d01fc0  ip : 3f9fab0c  fp : 00000000
[    0.000000] r10: 10c5387d  r9 : 3f9fab08  r8 : c09c007c
[    0.000000] r7 : 3f9fab18  r6 : c0da1038  r5 : c0d04ec0  r4 :
3f9fab0c
[    0.000000] r3 : 00000023  r2 : 0000000c  r1 : c09c007c  r0 :
3f9fab0c
[    0.000000] Flags: nzCv  IRQs off  FIQs off  Mode SVC_32  ISA
ARM  Segment none
[    0.000000] Control: 10c5387d  Table: 8000406a  DAC: 00000051
[    0.000000] Process swapper (pid: 0, stack limit = 0x(ptrval))
[    0.000000] Stack: (0xc0d01fc0 to 0xc0d02000)
[    0.000000] 1fc0: 00000000 00000000 00000000 00000000 c0c28a54
00000000 00000000 c0c00330
[    0.000000] 1fe0: 00000051 10c0387d 00000000 889f9288 410fc073
00000000 00000000 00000000
[    0.000000] [<c04b8098>] (memcmp) from [<00000000>] (0x0)
[    0.000000] Code: e3520000 0a00000f e1a0c000 e5d13000 (e5d00000) 
[    0.000000] random: get_random_bytes called from
print_oops_end_marker+0x24/0x4c with crng_init=0
[    0.000000] ---[ end trace 0000000000000000 ]---
[    0.000000] Kernel panic - not syncing: Fatal exception
[    0.000000] Rebooting in 1 seconds..
[    0.000000] Reboot failed -- System halted

Anyway, all ideas are welcome :)

Cheers

Daniel

Signature · 1 line
- Daniel