Skip to contentSkip to conversation

bootefi failures on armv7

U-Boot Ilias Apalodimas 3 participants 7 messages Apr 10, 2019 – Apr 11, 2019

lore
Share & tools
lore
#0 lore raw

Hi Alexander, Heinrich,

(Resending since i was too quick on the trigger forgot to cc the public mailing list. Sorry for the mess)

I can't get an armv7 board (STM32MP157C-DK2) to boot with bootefi on specific kernel/uboot configurations.
u-boot version: U-Boot 2019.04-rc3
kernel version: 5.0.0-rc3

What works:
- bootefi + 'dtb=' on kernel cmdline regardless of kernel config
- bootefi + fdtcontroladdr if CONFIG_ARM_LPAE=y on my kernel config

What doesn't work:
bootefi + fdtcontroladdr if CONFIG_ARM_LPAE is not set.

Once this config is selected the kernel hangs on a BUG_ON() in arm_pte_alloc()

Adding a dump_stack() there throws the relevant information.

[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.0.0-rc3-29427-g769f1f8f9b56-dirty #127
[    0.000000] Hardware name: STM32 (Device Tree Support)
[    0.000000] [<c03123ec>] (unwind_backtrace) from [<c030ce0c>] (show_stack+0x10/0x14)
[    0.000000] [<c030ce0c>] (show_stack) from [<c0e27250>] (dump_stack+0x8c/0xa0)
[    0.000000] [<c0e27250>] (dump_stack) from [<c1507e9c>] (arm_pte_alloc+0x74/0xb8)
[    0.000000] [<c1507e9c>] (arm_pte_alloc) from [<c15081ec>] (__create_mapping+0x30c/0x36c)
[    0.000000] [<c15081ec>] (__create_mapping) from [<c15089f8>] (paging_init+0x234/0x648)
[    0.000000] [<c15089f8>] (paging_init) from [<c1504950>] (setup_arch+0x660/0xcac)
[    0.000000] [<c1504950>] (setup_arch) from [<c1500a4c>] (start_kernel+0x70/0x458)
[    0.000000] [<c1500a4c>] (start_kernel) from [<00000000>] (  (null)) 

Kernel's EFI map:

[    0.000000] efi:   0x0000c0000000-0x0000c1ffffff [Boot Data          |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2000000-0x0000c2860fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2861000-0x0000c7cfffff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7d00000-0x0000c7efffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f00000-0x0000c7f0bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f0c000-0x0000dc705fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc706000-0x0000dc709fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc70a000-0x0000dcf6afff [Loader Code        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6b000-0x0000dcf6bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6c000-0x0000dcf72fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf73000-0x0000dcf73fff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf74000-0x0000dcf75fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf76000-0x0000dff80fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff81000-0x0000dff81fff [Runtime Code       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff82000-0x0000dfffffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]

I've found some hacky workarounds on that and i'd like some input on what's actually wrong

- Workaround 1, add a bogus bootefi cmd

instead of
load mmc 0:5 $kernel_addr_r efi/BOOT/zImage;
bootefi $kernel_addr_r $fdtcontroladdr

doing
bootefi $kernel_addr_r $fdtcontroladdr
load mmc 0:5 $kernel_addr_r efi/BOOT/zImage
bootefi $kernel_addr_r $fdtcontroladdr' works

The first bootefi command will fail since there's nothing loaded on that address yet. What happens is that u-boot adds an extra memory_map, via efi_add_memory_map() and the kernel boots

Kernel's EFI map:

[    0.000000] efi: Processing EFI memory map:
[    0.000000] efi:   0x0000c0000000-0x0000c1ffffff [Boot Data          |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2000000-0x0000c2860fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2861000-0x0000c7cf3fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7cf4000-0x0000c7ef3fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7ef4000-0x0000c7f0bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f0c000-0x0000dc704fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc705000-0x0000dc708fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc709000-0x0000dcf69fff [Loader Code        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6a000-0x0000dcf6cfff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6d000-0x0000dcf6dfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6e000-0x0000dcf74fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf75000-0x0000dcf75fff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf76000-0x0000dff80fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff81000-0x0000dff81fff [Runtime Code       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff82000-0x0000dfffffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]

- Workaround 2, copy fdt one page before the 127mb limit

cmd/bootefi.c +4 -2
index 3619a20e6433..a0b6156845da 100644
@@ -108,11 +108,13 @@ static efi_status_t copy_fdt(void **fdtp)
 	 * Safe fdt location is at 127 MiB.
 	 * On the sandbox convert from the sandbox address space.
 	 */
-	new_fdt_addr = (uintptr_t)map_sysmem(fdt_ram_start + 0x7f00000 +
-					     fdt_size, 0);
+	//new_fdt_addr = (uintptr_t)map_sysmem(fdt_ram_start + 0x7f00000 +
+					     //fdt_size, 0);
+	new_fdt_addr = (uintptr_t)map_sysmem(fdt_ram_start + 0x7f00000, fdt_size);
 	ret = efi_allocate_pages(EFI_ALLOCATE_MAX_ADDRESS,
 				 EFI_RUNTIME_SERVICES_DATA, fdt_pages,
 				 &new_fdt_addr);
+	printf("FDTADDR: %llx pages %lu sz %x\n", new_fdt_addr, fdt_pages, fdt_size);
 	if (ret != EFI_SUCCESS) {
 		/* If we can't put it there, put it somewhere */
 		new_fdt_addr = (ulong)memalign(EFI_PAGE_SIZE, fdt_size);

This loads the fdt on c7ef4000 (which is more than a page). Changing the address from 0x7f00000 to 7EFF000, on the original code, works as well

Kernel's EFI map (with the patch) :

[    0.000000] efi: Processing EFI memory map:
[    0.000000] efi:   0x0000c0000000-0x0000c1ffffff [Boot Data          |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2000000-0x0000c2860fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2861000-0x0000c7cf3fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7cf4000-0x0000c7ef3fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7ef4000-0x0000c7efffff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f00000-0x0000dc705fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc706000-0x0000dc709fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc70a000-0x0000dcf6afff [Loader Code        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6b000-0x0000dcf6bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6c000-0x0000dcf72fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf73000-0x0000dcf73fff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf74000-0x0000dcf75fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf76000-0x0000dff80fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff81000-0x0000dff81fff [Runtime Code       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff82000-0x0000dfffffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]

- Workaround 3, align fdt on 1mb boundries

cmd/bootefi.c +2 -1
index 3619a20e6433..37e849fad773 100644
@@ -101,7 +101,8 @@ static efi_status_t copy_fdt(void **fdtp)
 	 * needs to be expanded later.
 	 */
 	fdt = *fdtp;
-	fdt_pages = efi_size_in_pages(fdt_totalsize(fdt) + 0x3000);
+	//fdt_pages = efi_size_in_pages(fdt_totalsize(fdt) + 0x3000);
+	fdt_pages = efi_size_in_pages(0x100000);
 	fdt_size = fdt_pages << EFI_PAGE_SHIFT;
 
 	/*

Kernel's EFI map:

[    0.000000] efi: Processing EFI memory map:
[    0.000000] efi:   0x0000c0000000-0x0000c1ffffff [Boot Data          |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2000000-0x0000c2860fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2861000-0x0000c7cfffff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7d00000-0x0000c7efffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f00000-0x0000c7ffffff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c8000000-0x0000dc705fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc706000-0x0000dc709fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc70a000-0x0000dcf6afff [Loader Code        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6b000-0x0000dcf6bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6c000-0x0000dcf72fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf73000-0x0000dcf73fff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf74000-0x0000dcf75fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf76000-0x0000dff80fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff81000-0x0000dff81fff [Runtime Code       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff82000-0x0000dfffffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]

Thanks
/Ilias

#1 lore raw
Quoted 92 lines · Ilias Apalodimas

Hi Alexander, Heinrich,

(Resending since i was too quick on the trigger forgot to cc the public mailing list. Sorry for the mess)

I can't get an armv7 board (STM32MP157C-DK2) to boot with bootefi on specific kernel/uboot configurations.
u-boot version: U-Boot 2019.04-rc3
kernel version: 5.0.0-rc3

What works:
- bootefi + 'dtb=' on kernel cmdline regardless of kernel config
- bootefi + fdtcontroladdr if CONFIG_ARM_LPAE=y on my kernel config

What doesn't work:
bootefi + fdtcontroladdr if CONFIG_ARM_LPAE is not set.

Once this config is selected the kernel hangs on a BUG_ON() in arm_pte_alloc()

Adding a dump_stack() there throws the relevant information.

[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.0.0-rc3-29427-g769f1f8f9b56-dirty #127
[    0.000000] Hardware name: STM32 (Device Tree Support)
[    0.000000] [<c03123ec>] (unwind_backtrace) from [<c030ce0c>] (show_stack+0x10/0x14)
[    0.000000] [<c030ce0c>] (show_stack) from [<c0e27250>] (dump_stack+0x8c/0xa0)
[    0.000000] [<c0e27250>] (dump_stack) from [<c1507e9c>] (arm_pte_alloc+0x74/0xb8)
[    0.000000] [<c1507e9c>] (arm_pte_alloc) from [<c15081ec>] (__create_mapping+0x30c/0x36c)
[    0.000000] [<c15081ec>] (__create_mapping) from [<c15089f8>] (paging_init+0x234/0x648)
[    0.000000] [<c15089f8>] (paging_init) from [<c1504950>] (setup_arch+0x660/0xcac)
[    0.000000] [<c1504950>] (setup_arch) from [<c1500a4c>] (start_kernel+0x70/0x458)
[    0.000000] [<c1500a4c>] (start_kernel) from [<00000000>] (  (null))

Kernel's EFI map:

[    0.000000] efi:   0x0000c0000000-0x0000c1ffffff [Boot Data          |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2000000-0x0000c2860fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2861000-0x0000c7cfffff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7d00000-0x0000c7efffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f00000-0x0000c7f0bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f0c000-0x0000dc705fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc706000-0x0000dc709fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc70a000-0x0000dcf6afff [Loader Code        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6b000-0x0000dcf6bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6c000-0x0000dcf72fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf73000-0x0000dcf73fff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf74000-0x0000dcf75fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf76000-0x0000dff80fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff81000-0x0000dff81fff [Runtime Code       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff82000-0x0000dfffffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]

I've found some hacky workarounds on that and i'd like some input on what's actually wrong

- Workaround 1, add a bogus bootefi cmd

instead of
load mmc 0:5 $kernel_addr_r efi/BOOT/zImage;
bootefi $kernel_addr_r $fdtcontroladdr

doing
bootefi $kernel_addr_r $fdtcontroladdr
load mmc 0:5 $kernel_addr_r efi/BOOT/zImage
bootefi $kernel_addr_r $fdtcontroladdr' works

The first bootefi command will fail since there's nothing loaded on that address yet. What happens is that u-boot adds an extra memory_map, via efi_add_memory_map() and the kernel boots

Kernel's EFI map:

[    0.000000] efi: Processing EFI memory map:
[    0.000000] efi:   0x0000c0000000-0x0000c1ffffff [Boot Data          |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2000000-0x0000c2860fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2861000-0x0000c7cf3fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7cf4000-0x0000c7ef3fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7ef4000-0x0000c7f0bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f0c000-0x0000dc704fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc705000-0x0000dc708fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc709000-0x0000dcf69fff [Loader Code        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6a000-0x0000dcf6cfff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6d000-0x0000dcf6dfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6e000-0x0000dcf74fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf75000-0x0000dcf75fff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf76000-0x0000dff80fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff81000-0x0000dff81fff [Runtime Code       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff82000-0x0000dfffffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]

- Workaround 2, copy fdt one page before the 127mb limit

cmd/bootefi.c +0 -0
index 3619a20e6433..a0b6156845da 100644

cmd/bootefi.c static efi_status_t copy_fdt(void **fdtp) · line 108

Quoted 1 line
cmd/bootefi.c +0 -0
@@ -108,11 +108,13 @@ static efi_status_t copy_fdt(void **fdtp)
  	 * Safe fdt location is at 127 MiB.

Hello Ilias,

thanks for reporting the issue.

@Simon, @Alex
Why did we ever introduce this value?
127 MB in baf70c02107 ("efi: Relocate FDT to 127MB instead of 128MB") 128 MB in ad0c1a3d2ce ("efi_loader: Put fdt into convenient location")

baf70c02107 says that the Sandbox has only 128 MiB. But why should we need to hard code this here. When the Sandbox has initialized the memory map efi_allocate_pool() will take care of finding an address in the available memory anyway.

ad0c1a3d2ce mentions 128 MB as recommended value but does not provide any reference. I think we should provide the source of this recommendation.

According to
https://www.st.com/en/microcontrollers-microprocessors/stm32mp157.html
STM32MP157 comes with support for TrustZone and SecureBoot.

https://www.st.com/en/microcontrollers-microprocessors/stm32mp157.html#tools-software:

"The developer package comes with

Boot chain based on TF-A and U-Boot
OP-TEE secure OS running on the Arm® Cortex®-A in secure mode."

The Linux device tree does not reserve a memory region for the trusted firmware (TF-A) and the trusted execution environment (OP-TEE) Shouldn't there be a reserved memory area for both.

If TF-A or OP-TEE reside near 0xC8000000, this would explain the observed behavior.

@Ilias
Do you use TF-A or OP-TEE in your setup?

Best regards

Heinrich

Quoted 76 lines
cmd/bootefi.c +4 -2
  	 * On the sandbox convert from the sandbox address space.
  	 */
-	new_fdt_addr = (uintptr_t)map_sysmem(fdt_ram_start + 0x7f00000 +
-					     fdt_size, 0);
+	//new_fdt_addr = (uintptr_t)map_sysmem(fdt_ram_start + 0x7f00000 +
+					     //fdt_size, 0);
+	new_fdt_addr = (uintptr_t)map_sysmem(fdt_ram_start + 0x7f00000, fdt_size);
  	ret = efi_allocate_pages(EFI_ALLOCATE_MAX_ADDRESS,
  				 EFI_RUNTIME_SERVICES_DATA, fdt_pages,
  				 &new_fdt_addr);
+	printf("FDTADDR: %llx pages %lu sz %x\n", new_fdt_addr, fdt_pages, fdt_size);
  	if (ret != EFI_SUCCESS) {
  		/* If we can't put it there, put it somewhere */
  		new_fdt_addr = (ulong)memalign(EFI_PAGE_SIZE, fdt_size);

This loads the fdt on c7ef4000 (which is more than a page). Changing the address from 0x7f00000 to 7EFF000, on the original code, works as well

Kernel's EFI map (with the patch) :

[    0.000000] efi: Processing EFI memory map:
[    0.000000] efi:   0x0000c0000000-0x0000c1ffffff [Boot Data          |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2000000-0x0000c2860fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2861000-0x0000c7cf3fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7cf4000-0x0000c7ef3fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7ef4000-0x0000c7efffff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f00000-0x0000dc705fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc706000-0x0000dc709fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc70a000-0x0000dcf6afff [Loader Code        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6b000-0x0000dcf6bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6c000-0x0000dcf72fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf73000-0x0000dcf73fff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf74000-0x0000dcf75fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf76000-0x0000dff80fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff81000-0x0000dff81fff [Runtime Code       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff82000-0x0000dfffffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]

- Workaround 3, align fdt on 1mb boundries

cmd/bootefi.c +2 -1
index 3619a20e6433..37e849fad773 100644
@@ -101,7 +101,8 @@ static efi_status_t copy_fdt(void **fdtp)
  	 * needs to be expanded later.
  	 */
  	fdt = *fdtp;
-	fdt_pages = efi_size_in_pages(fdt_totalsize(fdt) + 0x3000);
+	//fdt_pages = efi_size_in_pages(fdt_totalsize(fdt) + 0x3000);
+	fdt_pages = efi_size_in_pages(0x100000);
  	fdt_size = fdt_pages << EFI_PAGE_SHIFT;
 
  	/*

Kernel's EFI map:

[    0.000000] efi: Processing EFI memory map:
[    0.000000] efi:   0x0000c0000000-0x0000c1ffffff [Boot Data          |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2000000-0x0000c2860fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2861000-0x0000c7cfffff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7d00000-0x0000c7efffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f00000-0x0000c7ffffff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c8000000-0x0000dc705fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc706000-0x0000dc709fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc70a000-0x0000dcf6afff [Loader Code        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6b000-0x0000dcf6bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6c000-0x0000dcf72fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf73000-0x0000dcf73fff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf74000-0x0000dcf75fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf76000-0x0000dff80fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff81000-0x0000dff81fff [Runtime Code       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff82000-0x0000dfffffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]

Thanks
/Ilias

#2 lore raw

Hello Heinrich,

Quoted 29 lines

@Simon, @Alex
Why did we ever introduce this value?
127 MB in baf70c02107 ("efi: Relocate FDT to 127MB instead of 128MB") 128 MB in ad0c1a3d2ce ("efi_loader: Put fdt into convenient location")

baf70c02107 says that the Sandbox has only 128 MiB. But why should we need to hard code this here. When the Sandbox has initialized the memory map efi_allocate_pool() will take care of finding an address in the available memory anyway.

ad0c1a3d2ce mentions 128 MB as recommended value but does not provide any reference. I think we should provide the source of this recommendation.

According to
https://www.st.com/en/microcontrollers-microprocessors/stm32mp157.html
STM32MP157 comes with support for TrustZone and SecureBoot.

https://www.st.com/en/microcontrollers-microprocessors/stm32mp157.html#tools-software:

"The developer package comes with

Boot chain based on TF-A and U-Boot
OP-TEE secure OS running on the Arm® Cortex®-A in secure mode."

The Linux device tree does not reserve a memory region for the trusted firmware (TF-A) and the trusted execution environment (OP-TEE) Shouldn't there be a reserved memory area for both.

Quoted 5 lines

If TF-A or OP-TEE reside near 0xC8000000, this would explain the observed behavior.

@Ilias
Do you use TF-A or OP-TEE in your setup?

No none of those are present in my tests.

Note that in both workaround 1/2 fdt ends up at the same address (0xc7ef4000).

Thanks
/Ilias

#3 lore raw
In reply to Ilias Apalodimas
Quoted 5 lines · Ilias Apalodimas

Hi Alexander, Heinrich,

(Resending since i was too quick on the trigger forgot to cc the public mailing list. Sorry for the mess)

Quoted 8 lines

I can't get an armv7 board (STM32MP157C-DK2) to boot with bootefi on specific kernel/uboot configurations.
u-boot version: U-Boot 2019.04-rc3
kernel version: 5.0.0-rc3

What works:
- bootefi + 'dtb=' on kernel cmdline regardless of kernel config
- bootefi + fdtcontroladdr if CONFIG_ARM_LPAE=y on my kernel config

CONFIG_ARM_LPAE=y influences the number of page table levels:

#ifdef CONFIG_ARM_LPAE
#include <asm/pgtable-3level.h>
#else
#include <asm/pgtable-2level.h>
#endif

Hence in arch/arm/mm/mmu.c, arm_pte_alloc() different values of PTE_HWTABLE_OFF and PTE_HWTABLE_SIZE will be used.

Quoted 18 lines

What doesn't work:
bootefi + fdtcontroladdr if CONFIG_ARM_LPAE is not set.

Once this config is selected the kernel hangs on a BUG_ON() in arm_pte_alloc()

Adding a dump_stack() there throws the relevant information.

[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.0.0-rc3-29427-g769f1f8f9b56-dirty #127
[    0.000000] Hardware name: STM32 (Device Tree Support)
[    0.000000] [<c03123ec>] (unwind_backtrace) from [<c030ce0c>] (show_stack+0x10/0x14)
[    0.000000] [<c030ce0c>] (show_stack) from [<c0e27250>] (dump_stack+0x8c/0xa0)
[    0.000000] [<c0e27250>] (dump_stack) from [<c1507e9c>] (arm_pte_alloc+0x74/0xb8)
[    0.000000] [<c1507e9c>] (arm_pte_alloc) from [<c15081ec>] (__create_mapping+0x30c/0x36c)
[    0.000000] [<c15081ec>] (__create_mapping) from [<c15089f8>] (paging_init+0x234/0x648)
[    0.000000] [<c15089f8>] (paging_init) from [<c1504950>] (setup_arch+0x660/0xcac)
[    0.000000] [<c1504950>] (setup_arch) from [<c1500a4c>] (start_kernel+0x70/0x458)
[    0.000000] [<c1500a4c>] (start_kernel) from [<00000000>] (  (null))
Quoted 6 lines

Kernel's EFI map:

[    0.000000] efi:   0x0000c0000000-0x0000c1ffffff [Boot Data          |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2000000-0x0000c2860fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2861000-0x0000c7cfffff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7d00000-0x0000c7efffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f00000-0x0000c7f0bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]

So this seems to be the allocation for the devicetree starting at 127 MiB from RAM start. You seem not to be using the Linux device tree:
Linux 5.0 has 25004 bytes in arch/arm/boot/dts/stm32mp157c-ed1.dtb. So this fits into 7 pages + 3 pages we add to it. But here 12 pages are allocated.

Up to this point I cannot see where U-Boot is doing anything incorrectly. I would not want to change the 127 MiB value to some other value that might fail on other boards.

Maybe you could contact the Linux mm maintainers to get an understanding what implicit assumptions the Linux kernel makes when building page tables or how to debug the problem.

Best regards

Heinrich

Quoted 132 lines
[    0.000000] efi:   0x0000c7f0c000-0x0000dc705fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc706000-0x0000dc709fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc70a000-0x0000dcf6afff [Loader Code        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6b000-0x0000dcf6bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6c000-0x0000dcf72fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf73000-0x0000dcf73fff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf74000-0x0000dcf75fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf76000-0x0000dff80fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff81000-0x0000dff81fff [Runtime Code       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff82000-0x0000dfffffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]

I've found some hacky workarounds on that and i'd like some input on what's actually wrong

- Workaround 1, add a bogus bootefi cmd

instead of
load mmc 0:5 $kernel_addr_r efi/BOOT/zImage;
bootefi $kernel_addr_r $fdtcontroladdr

doing
bootefi $kernel_addr_r $fdtcontroladdr
load mmc 0:5 $kernel_addr_r efi/BOOT/zImage
bootefi $kernel_addr_r $fdtcontroladdr' works

The first bootefi command will fail since there's nothing loaded on that address yet. What happens is that u-boot adds an extra memory_map, via efi_add_memory_map() and the kernel boots

Kernel's EFI map:

[    0.000000] efi: Processing EFI memory map:
[    0.000000] efi:   0x0000c0000000-0x0000c1ffffff [Boot Data          |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2000000-0x0000c2860fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2861000-0x0000c7cf3fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7cf4000-0x0000c7ef3fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7ef4000-0x0000c7f0bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f0c000-0x0000dc704fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc705000-0x0000dc708fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc709000-0x0000dcf69fff [Loader Code        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6a000-0x0000dcf6cfff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6d000-0x0000dcf6dfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6e000-0x0000dcf74fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf75000-0x0000dcf75fff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf76000-0x0000dff80fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff81000-0x0000dff81fff [Runtime Code       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff82000-0x0000dfffffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]

- Workaround 2, copy fdt one page before the 127mb limit

cmd/bootefi.c +4 -2
index 3619a20e6433..a0b6156845da 100644
@@ -108,11 +108,13 @@ static efi_status_t copy_fdt(void **fdtp)
  	 * Safe fdt location is at 127 MiB.
  	 * On the sandbox convert from the sandbox address space.
  	 */
-	new_fdt_addr = (uintptr_t)map_sysmem(fdt_ram_start + 0x7f00000 +
-					     fdt_size, 0);
+	//new_fdt_addr = (uintptr_t)map_sysmem(fdt_ram_start + 0x7f00000 +
+					     //fdt_size, 0);
+	new_fdt_addr = (uintptr_t)map_sysmem(fdt_ram_start + 0x7f00000, fdt_size);
  	ret = efi_allocate_pages(EFI_ALLOCATE_MAX_ADDRESS,
  				 EFI_RUNTIME_SERVICES_DATA, fdt_pages,
  				 &new_fdt_addr);
+	printf("FDTADDR: %llx pages %lu sz %x\n", new_fdt_addr, fdt_pages, fdt_size);
  	if (ret != EFI_SUCCESS) {
  		/* If we can't put it there, put it somewhere */
  		new_fdt_addr = (ulong)memalign(EFI_PAGE_SIZE, fdt_size);

This loads the fdt on c7ef4000 (which is more than a page). Changing the address from 0x7f00000 to 7EFF000, on the original code, works as well

Kernel's EFI map (with the patch) :

[    0.000000] efi: Processing EFI memory map:
[    0.000000] efi:   0x0000c0000000-0x0000c1ffffff [Boot Data          |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2000000-0x0000c2860fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2861000-0x0000c7cf3fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7cf4000-0x0000c7ef3fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7ef4000-0x0000c7efffff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f00000-0x0000dc705fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc706000-0x0000dc709fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc70a000-0x0000dcf6afff [Loader Code        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6b000-0x0000dcf6bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6c000-0x0000dcf72fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf73000-0x0000dcf73fff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf74000-0x0000dcf75fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf76000-0x0000dff80fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff81000-0x0000dff81fff [Runtime Code       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff82000-0x0000dfffffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]

- Workaround 3, align fdt on 1mb boundries

cmd/bootefi.c +2 -1
index 3619a20e6433..37e849fad773 100644
@@ -101,7 +101,8 @@ static efi_status_t copy_fdt(void **fdtp)
  	 * needs to be expanded later.
  	 */
  	fdt = *fdtp;
-	fdt_pages = efi_size_in_pages(fdt_totalsize(fdt) + 0x3000);
+	//fdt_pages = efi_size_in_pages(fdt_totalsize(fdt) + 0x3000);
+	fdt_pages = efi_size_in_pages(0x100000);
  	fdt_size = fdt_pages << EFI_PAGE_SHIFT;
 
  	/*

Kernel's EFI map:

[    0.000000] efi: Processing EFI memory map:
[    0.000000] efi:   0x0000c0000000-0x0000c1ffffff [Boot Data          |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2000000-0x0000c2860fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2861000-0x0000c7cfffff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7d00000-0x0000c7efffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f00000-0x0000c7ffffff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c8000000-0x0000dc705fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc706000-0x0000dc709fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc70a000-0x0000dcf6afff [Loader Code        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6b000-0x0000dcf6bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6c000-0x0000dcf72fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf73000-0x0000dcf73fff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf74000-0x0000dcf75fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf76000-0x0000dff80fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff81000-0x0000dff81fff [Runtime Code       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff82000-0x0000dfffffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]

Thanks
/Ilias

#4 lore raw
Quoted 17 lines · Ilias Apalodimas
Quoted 5 lines

Hi Alexander, Heinrich,

(Resending since i was too quick on the trigger forgot to cc the public mailing list. Sorry for the mess)

Quoted 8 lines

I can't get an armv7 board (STM32MP157C-DK2) to boot with bootefi on specific kernel/uboot configurations.
u-boot version: U-Boot 2019.04-rc3
kernel version: 5.0.0-rc3

What works:
- bootefi + 'dtb=' on kernel cmdline regardless of kernel config
- bootefi + fdtcontroladdr if CONFIG_ARM_LPAE=y on my kernel config

CONFIG_ARM_LPAE=y influences the number of page table levels:

Quoted 8 lines

#ifdef CONFIG_ARM_LPAE
#include <asm/pgtable-3level.h>
#else
#include <asm/pgtable-2level.h>
#endif

Hence in arch/arm/mm/mmu.c, arm_pte_alloc() different values of PTE_HWTABLE_OFF and PTE_HWTABLE_SIZE will be used.

Yes exactly. That was my understanding as well

Quoted 26 lines
Quoted 19 lines

What doesn't work:
bootefi + fdtcontroladdr if CONFIG_ARM_LPAE is not set.

Once this config is selected the kernel hangs on a BUG_ON() in arm_pte_alloc()

Adding a dump_stack() there throws the relevant information.

[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.0.0-rc3-29427-g769f1f8f9b56-dirty #127
[    0.000000] Hardware name: STM32 (Device Tree Support)
[    0.000000] [<c03123ec>] (unwind_backtrace) from [<c030ce0c>] (show_stack+0x10/0x14)
[    0.000000] [<c030ce0c>] (show_stack) from [<c0e27250>] (dump_stack+0x8c/0xa0)
[    0.000000] [<c0e27250>] (dump_stack) from [<c1507e9c>] (arm_pte_alloc+0x74/0xb8)
[    0.000000] [<c1507e9c>] (arm_pte_alloc) from [<c15081ec>] (__create_mapping+0x30c/0x36c)
[    0.000000] [<c15081ec>] (__create_mapping) from [<c15089f8>] (paging_init+0x234/0x648)
[    0.000000] [<c15089f8>] (paging_init) from [<c1504950>] (setup_arch+0x660/0xcac)
[    0.000000] [<c1504950>] (setup_arch) from [<c1500a4c>] (start_kernel+0x70/0x458)
[    0.000000] [<c1500a4c>] (start_kernel) from [<00000000>] (  (null))
Quoted 6 lines

Kernel's EFI map:

[    0.000000] efi:   0x0000c0000000-0x0000c1ffffff [Boot Data          |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2000000-0x0000c2860fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2861000-0x0000c7cfffff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7d00000-0x0000c7efffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f00000-0x0000c7f0bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
Quoted 2 lines

So this seems to be the allocation for the devicetree starting at 127 MiB from RAM start.

From a u-boot point of view everything seems ok.
The start address + size of the DTB are correctly 'fed' into the Linux EFI memory mappings and everything seems properly aligned.

Quoted 4 lines

You seem not to be using the Linux device tree:
Linux 5.0 has 25004 bytes in arch/arm/boot/dts/stm32mp157c-ed1.dtb. So this fits into 7 pages + 3 pages we add to it. But here 12 pages are allocated.

No i am using Patrick's tree from https://github.com/patrickdelaunay/u-boot
on the 2019.x branch. I am using that DTB.

Quoted 4 lines

Up to this point I cannot see where U-Boot is doing anything incorrectly. I would not want to change the 127 MiB value to some other value that might fail on other boards.

Of course. I never suggested we should change that, especially since we don't have any evvidence that says 'it's a uboot problem'.
I just tried to move the FDT around and see if anything fixes my issue, while trying to debug page table creation as well.

Quoted 4 lines

Maybe you could contact the Linux mm maintainers to get an understanding what implicit assumptions the Linux kernel makes when building page tables or how to debug the problem.

Yep that's the next step

Thanks for looking at this

/Ilias

#5 lore raw
In reply to Ilias Apalodimas
Quoted 113 lines · Ilias Apalodimas

Hi Alexander, Heinrich,

(Resending since i was too quick on the trigger forgot to cc the public mailing list. Sorry for the mess)

I can't get an armv7 board (STM32MP157C-DK2) to boot with bootefi on specific kernel/uboot configurations.
u-boot version: U-Boot 2019.04-rc3
kernel version: 5.0.0-rc3

What works:
- bootefi + 'dtb=' on kernel cmdline regardless of kernel config
- bootefi + fdtcontroladdr if CONFIG_ARM_LPAE=y on my kernel config

What doesn't work:
bootefi + fdtcontroladdr if CONFIG_ARM_LPAE is not set.

Once this config is selected the kernel hangs on a BUG_ON() in arm_pte_alloc()

Adding a dump_stack() there throws the relevant information.

[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.0.0-rc3-29427-g769f1f8f9b56-dirty #127
[    0.000000] Hardware name: STM32 (Device Tree Support)
[    0.000000] [<c03123ec>] (unwind_backtrace) from [<c030ce0c>] (show_stack+0x10/0x14)
[    0.000000] [<c030ce0c>] (show_stack) from [<c0e27250>] (dump_stack+0x8c/0xa0)
[    0.000000] [<c0e27250>] (dump_stack) from [<c1507e9c>] (arm_pte_alloc+0x74/0xb8)
[    0.000000] [<c1507e9c>] (arm_pte_alloc) from [<c15081ec>] (__create_mapping+0x30c/0x36c)
[    0.000000] [<c15081ec>] (__create_mapping) from [<c15089f8>] (paging_init+0x234/0x648)
[    0.000000] [<c15089f8>] (paging_init) from [<c1504950>] (setup_arch+0x660/0xcac)
[    0.000000] [<c1504950>] (setup_arch) from [<c1500a4c>] (start_kernel+0x70/0x458)
[    0.000000] [<c1500a4c>] (start_kernel) from [<00000000>] (  (null))

Kernel's EFI map:

[    0.000000] efi:   0x0000c0000000-0x0000c1ffffff [Boot Data          |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2000000-0x0000c2860fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2861000-0x0000c7cfffff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7d00000-0x0000c7efffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f00000-0x0000c7f0bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f0c000-0x0000dc705fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc706000-0x0000dc709fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc70a000-0x0000dcf6afff [Loader Code        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6b000-0x0000dcf6bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6c000-0x0000dcf72fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf73000-0x0000dcf73fff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf74000-0x0000dcf75fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf76000-0x0000dff80fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff81000-0x0000dff81fff [Runtime Code       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff82000-0x0000dfffffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]

I've found some hacky workarounds on that and i'd like some input on what's actually wrong

- Workaround 1, add a bogus bootefi cmd

instead of
load mmc 0:5 $kernel_addr_r efi/BOOT/zImage;
bootefi $kernel_addr_r $fdtcontroladdr

doing
bootefi $kernel_addr_r $fdtcontroladdr
load mmc 0:5 $kernel_addr_r efi/BOOT/zImage
bootefi $kernel_addr_r $fdtcontroladdr' works

The first bootefi command will fail since there's nothing loaded on that address yet. What happens is that u-boot adds an extra memory_map, via efi_add_memory_map() and the kernel boots

Kernel's EFI map:

[    0.000000] efi: Processing EFI memory map:
[    0.000000] efi:   0x0000c0000000-0x0000c1ffffff [Boot Data          |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2000000-0x0000c2860fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2861000-0x0000c7cf3fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7cf4000-0x0000c7ef3fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7ef4000-0x0000c7f0bfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7f0c000-0x0000dc704fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc705000-0x0000dc708fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dc709000-0x0000dcf69fff [Loader Code        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6a000-0x0000dcf6cfff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6d000-0x0000dcf6dfff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf6e000-0x0000dcf74fff [Reserved           |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf75000-0x0000dcf75fff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dcf76000-0x0000dff80fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff81000-0x0000dff81fff [Runtime Code       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000dff82000-0x0000dfffffff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]

- Workaround 2, copy fdt one page before the 127mb limit

cmd/bootefi.c +4 -2
index 3619a20e6433..a0b6156845da 100644
@@ -108,11 +108,13 @@ static efi_status_t copy_fdt(void **fdtp)
         * Safe fdt location is at 127 MiB.
         * On the sandbox convert from the sandbox address space.
         */
-       new_fdt_addr = (uintptr_t)map_sysmem(fdt_ram_start + 0x7f00000 +
-                                            fdt_size, 0);
+       //new_fdt_addr = (uintptr_t)map_sysmem(fdt_ram_start + 0x7f00000 +
+                                            //fdt_size, 0);
+       new_fdt_addr = (uintptr_t)map_sysmem(fdt_ram_start + 0x7f00000, fdt_size);
        ret = efi_allocate_pages(EFI_ALLOCATE_MAX_ADDRESS,
                                 EFI_RUNTIME_SERVICES_DATA, fdt_pages,
                                 &new_fdt_addr);
+       printf("FDTADDR: %llx pages %lu sz %x\n", new_fdt_addr, fdt_pages, fdt_size);
        if (ret != EFI_SUCCESS) {
                /* If we can't put it there, put it somewhere */
                new_fdt_addr = (ulong)memalign(EFI_PAGE_SIZE, fdt_size);

This loads the fdt on c7ef4000 (which is more than a page). Changing the address from 0x7f00000 to 7EFF000, on the original code, works as well

Quoted 7 lines

Kernel's EFI map (with the patch) :

[    0.000000] efi: Processing EFI memory map:
[    0.000000] efi:   0x0000c0000000-0x0000c1ffffff [Boot Data          |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2000000-0x0000c2860fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2861000-0x0000c7cf3fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7cf4000-0x0000c7ef3fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7ef4000-0x0000c7efffff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]

As an aside, putting the FDT in runtime data is not the right thing to do.

Runtime data sections are intended for data that is used by the runtime services implementations themselves, which is why they automatically get the EFI_MEMORY_RUNTIME attribute as well, and get mapped into the EFI runtime address space. They also get flagged as NOMAP regions, which means they get omitted from the linear map, which causes unnecessary page table fragmentation leading to more TLB pressure.

I recommend using boot services data here, or acpi reclaim if you are concerned about the OS not reserving the region correctly.

#6 lore raw

Hi Ard,

Quoted 23 lines
Quoted 5 lines

This loads the fdt on c7ef4000 (which is more than a page). Changing the address from 0x7f00000 to 7EFF000, on the original code, works as well

Quoted 7 lines

Kernel's EFI map (with the patch) :

[    0.000000] efi: Processing EFI memory map:
[    0.000000] efi:   0x0000c0000000-0x0000c1ffffff [Boot Data          |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2000000-0x0000c2860fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c2861000-0x0000c7cf3fff [Conventional Memory|   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7cf4000-0x0000c7ef3fff [Loader Data        |   |  |  |  |  |  |  |   |WB|  |  |  ]
[    0.000000] efi:   0x0000c7ef4000-0x0000c7efffff [Runtime Data       |RUN|  |  |  |  |  |  |   |WB|  |  |  ]

As an aside, putting the FDT in runtime data is not the right thing to do.

Runtime data sections are intended for data that is used by the runtime services implementations themselves, which is why they automatically get the EFI_MEMORY_RUNTIME attribute as well, and get mapped into the EFI runtime address space. They also get flagged as NOMAP regions, which means they get omitted from the linear map, which causes unnecessary page table fragmentation leading to more TLB pressure.

Quoted 2 lines

I recommend using boot services data here, or acpi reclaim if you are concerned about the OS not reserving the region correctly.

This also fixes my boot issue.
I still think the initial analysis is right and this is still a kernel issue. Chaning the mem type to EFI_ACPI_RECLAIM_MEMORY removes the NOMAP flag from the memory and 'avoids' the kernel issue.

Thanks
/Ilias