Skip to contentSkip to conversation

arm64: efi: leave MMU and caches on at boot

PATCH Superseded Ard Biesheuvel 2 participants 9 messages Oct 18, 2022 – Nov 7, 2022

Subsystems (2)
arm64 EFI

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Ard Biesheuvel

author maintainer
#0 lore raw

v4 cover arm64: efi: leave MMU and caches on at boot

The purpose of this series is to remove any explicit cache maintenance for coherency during early boot that becomes unnecessary if we simply retain the cacheable 1:1 mapping of all of system RAM provided by EFI, and use it to populate the ID map page tables. After setting up this preliminary ID map, we disable the MMU, drop to EL1, reprogram the MAIR, TCR and SCTLR registers as before, and proceed as usual, avoiding the need for any manipulations of memory while the MMU and caches are off.

The only properties of the firmware provided 1:1 map we rely on is that it does not require any explicit cache maintenance for coherency, and that it covers the entire memory footprint of the image, including the BSS and padding at the end - all else is under control of the kernel itself, as before.

Changes since v3:
- drop EFI_LOADER_CODE memory type patch that has been queued in the mean time
- rebased onto [partial] series that moves efi-entry.S into the libstub/ source directory [0]
- fixed a correctness issue in patch #2

[0] https://lore.kernel.org/all/[email protected]/

Applying the last patch in the series will require some coordination with the EFI tree, but I can manage that using a secondary 'late' PR during the merge window, if needed.

Cc Will Deacon <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Mark Rutland <[email protected]>

Ard Biesheuvel (6):
arm64: lds: reduce effective minimum image alignment to 64k
arm64: kernel: move identity map out of .text mapping
arm64: head: record the MMU state at primary entry
arm64: head: avoid cache invalidation when entering with the MMU on
arm64: head: clean the ID map page to the PoC
arm64: efi/libstub: enter with the MMU on

 arch/arm64/include/asm/efi.h               |  7 --
 arch/arm64/kernel/head.S                   | 70 +++++++++++++++-----
 arch/arm64/kernel/image-vars.h             |  5 +-
 arch/arm64/kernel/vmlinux.lds.S            | 13 +++-
 arch/arm64/mm/cache.S                      |  5 +-
 arch/arm64/mm/proc.S                       |  2 -
 drivers/firmware/efi/libstub/arm64-entry.S | 57 ----------------
 drivers/firmware/efi/libstub/arm64-stub.c  | 19 +++++-
 include/linux/efi.h                        |  6 +-
 9 files changed, 89 insertions(+), 95 deletions(-)
 delete mode 100644 drivers/firmware/efi/libstub/arm64-entry.S
Signature · 1 line
2.35.1
author maintainer
#1 lore raw

PATCH v4 1/6 arm64: lds: reduce effective minimum image alignment to 64k

Our segment alignment is 64k for all configurations, and coincidentally, this is the largest alignment supported by the PE/COFF executable format used by EFI. This means that generally, there is no need to move the image around in memory after it has been loaded by the firmware, which can be advantageous as it also permits us to rely on the memory attributes set by the firmware (R-X for [_text, __inittext_end] and RW- for [__initdata_begin, _end].

However, the minimum alignment of the image is actually 128k on 64k pages configurations with CONFIG_VMAP_STACK=y, due to the existence of a single 128k aligned object in the image, which is the stack of the init task.

Let's work around this by adding some padding before the init stack allocation, so we can round down the stack pointer to a suitably aligned value if the image is not aligned to 128k in memory.

Note that this does not affect the boot protocol, which still requires 2 MiB alignment for bare metal boot, but is only part of the internal contract between the EFI stub and the kernel proper.

Signed-off-by Ard Biesheuvel <[email protected]>
---
 arch/arm64/include/asm/efi.h              |  7 -------
 arch/arm64/kernel/head.S                  |  3 +++
 arch/arm64/kernel/vmlinux.lds.S           | 11 ++++++++++-
 drivers/firmware/efi/libstub/arm64-stub.c |  2 +-
 include/linux/efi.h                       |  6 +-----
 5 files changed, 15 insertions(+), 14 deletions(-)
arch/arm64/include/asm/efi.h +0 -7
index 439e2bc5d5d8..3177e76de708 100644
@@ -54,13 +54,6 @@ efi_status_t __efi_rt_asm_wrapper(void *, const char *, ...);
 
 /* arch specific definitions used by the stub code */
 
-/*
- * In some configurations (e.g. VMAP_STACK && 64K pages), stacks built into the
- * kernel need greater alignment than we require the segments to be padded to.
- */
-#define EFI_KIMG_ALIGN	\
-	(SEGMENT_ALIGN > THREAD_ALIGN ? SEGMENT_ALIGN : THREAD_ALIGN)
-
 /*
  * On arm64, we have to ensure that the initrd ends up in the linear region,
  * which is a 1 GB aligned region of size '1UL << (VA_BITS_MIN - 1)' that is
arch/arm64/kernel/head.S +3 -0
index 2196aad7b55b..f168e3309704 100644
@@ -398,6 +398,9 @@ SYM_FUNC_END(create_kernel_mapping)
 	msr	sp_el0, \tsk
 
 	ldr	\tmp1, [\tsk, #TSK_STACK]
+#if THREAD_ALIGN > SEGMENT_ALIGN
+	bic	\tmp1, \tmp1, #THREAD_ALIGN - 1
+#endif
 	add	sp, \tmp1, #THREAD_SIZE
 	sub	sp, sp, #PT_REGS_SIZE
arch/arm64/kernel/vmlinux.lds.S +10 -1
index 45131e354e27..0efccdf52be2 100644
@@ -274,7 +274,16 @@ SECTIONS
 
 	_data = .;
 	_sdata = .;
-	RW_DATA(L1_CACHE_BYTES, PAGE_SIZE, THREAD_ALIGN)
+#if THREAD_ALIGN > SEGMENT_ALIGN
+	/*
+	 * Add some padding for the init stack so we can fix up any potential
+	 * misalignment at runtime. In practice, this can only occur on 64k
+	 * pages configurations with CONFIG_VMAP_STACK=y.
+	 */
+	. += THREAD_ALIGN - SEGMENT_ALIGN;
+	ASSERT(. == init_stack, "init_stack not at start of RW_DATA as expected")
+#endif
+	RW_DATA(L1_CACHE_BYTES, PAGE_SIZE, SEGMENT_ALIGN)
 
 	/*
 	 * Data written with the MMU off but read with the MMU on requires
drivers/firmware/efi/libstub/arm64-stub.c +1 -1
index e767a5ac8c3d..6229f42c797f 100644
@@ -97,7 +97,7 @@ efi_status_t handle_kernel_image(unsigned long *image_addr,
 	 * 2M alignment if KASLR was explicitly disabled, even if it was not
 	 * going to be activated to begin with.
 	 */
-	u64 min_kimg_align = efi_nokaslr ? MIN_KIMG_ALIGN : EFI_KIMG_ALIGN;
+	u64 min_kimg_align = efi_nokaslr ? MIN_KIMG_ALIGN : SEGMENT_ALIGN;
 
 	if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) {
 		efi_guid_t li_fixed_proto = LINUX_EFI_LOADED_IMAGE_FIXED_GUID;
include/linux/efi.h +1 -5
index 256e70e42114..1a395c24fdc0 100644
@@ -422,11 +422,7 @@ void efi_native_runtime_setup(void);
 /*
  * This GUID may be installed onto the kernel image's handle as a NULL protocol
  * to signal to the stub that the placement of the image should be respected,
- * and moving the image in physical memory is undesirable. To ensure
- * compatibility with 64k pages kernels with virtually mapped stacks, and to
- * avoid defeating physical randomization, this protocol should only be
- * installed if the image was placed at a randomized 128k aligned address in
- * memory.
+ * and moving the image in physical memory is undesirable.
  */
 #define LINUX_EFI_LOADED_IMAGE_FIXED_GUID	EFI_GUID(0xf5a37b6d, 0x3344, 0x42a5,  0xb6, 0xbb, 0x97, 0x86, 0x48, 0xc1, 0x89, 0x0a)
Signature · 1 line
2.35.1
author maintainer
#2 lore raw

PATCH v4 2/6 arm64: kernel: move identity map out of .text mapping

In reply to Ard Biesheuvel

Reorganize the ID map slightly so that only code that is executed with the MMU off or via the 1:1 mapping remains. This allows us to move the identity map out of the .text segment, as it will no longer need executable permissions via the kernel mapping.

Signed-off-by Ard Biesheuvel <[email protected]>
---
 arch/arm64/kernel/head.S        | 28 +++++++++++---------
 arch/arm64/kernel/vmlinux.lds.S |  2 +-
 arch/arm64/mm/proc.S            |  2 --
 3 files changed, 16 insertions(+), 16 deletions(-)
arch/arm64/kernel/head.S +15 -13
index f168e3309704..25a84ce1700c 100644
@@ -543,19 +543,6 @@ SYM_INNER_LABEL(init_el2, SYM_L_LOCAL)
 	eret
 SYM_FUNC_END(init_kernel_el)
 
-/*
- * Sets the __boot_cpu_mode flag depending on the CPU boot mode passed
- * in w0. See arch/arm64/include/asm/virt.h for more info.
- */
-SYM_FUNC_START_LOCAL(set_cpu_boot_mode_flag)
-	adr_l	x1, __boot_cpu_mode
-	cmp	w0, #BOOT_CPU_MODE_EL2
-	b.ne	1f
-	add	x1, x1, #4
-1:	str	w0, [x1]			// Save CPU boot mode
-	ret
-SYM_FUNC_END(set_cpu_boot_mode_flag)
-
 	/*
 	 * This provides a "holding pen" for platforms to hold all secondary
 	 * cores are held until we're ready for them to initialise.

@@ -600,6 +587,7 @@ SYM_FUNC_START_LOCAL(secondary_startup)
 	br	x8
 SYM_FUNC_END(secondary_startup)
 
+	.text
 SYM_FUNC_START_LOCAL(__secondary_switched)
 	mov	x0, x20
 	bl	set_cpu_boot_mode_flag

@@ -628,6 +616,19 @@ SYM_FUNC_START_LOCAL(__secondary_too_slow)
 	b	__secondary_too_slow
 SYM_FUNC_END(__secondary_too_slow)
 
+/*
+ * Sets the __boot_cpu_mode flag depending on the CPU boot mode passed
+ * in w0. See arch/arm64/include/asm/virt.h for more info.
+ */
+SYM_FUNC_START_LOCAL(set_cpu_boot_mode_flag)
+	adr_l	x1, __boot_cpu_mode
+	cmp	w0, #BOOT_CPU_MODE_EL2
+	b.ne	1f
+	add	x1, x1, #4
+1:	str	w0, [x1]			// Save CPU boot mode
+	ret
+SYM_FUNC_END(set_cpu_boot_mode_flag)
+
 /*
  * The booting CPU updates the failed status @__early_cpu_boot_status,
  * with MMU turned off.

@@ -659,6 +660,7 @@ SYM_FUNC_END(__secondary_too_slow)
  * Checks if the selected granule size is supported by the CPU.
  * If it isn't, park the CPU
  */
+	.section ".idmap.text","awx"
 SYM_FUNC_START(__enable_mmu)
 	mrs	x3, ID_AA64MMFR0_EL1
 	ubfx	x3, x3, #ID_AA64MMFR0_EL1_TGRAN_SHIFT, 4
arch/arm64/kernel/vmlinux.lds.S +1 -1
index 0efccdf52be2..5002d869fa7f 100644
@@ -168,7 +168,6 @@ SECTIONS
 			LOCK_TEXT
 			KPROBES_TEXT
 			HYPERVISOR_TEXT
-			IDMAP_TEXT
 			*(.gnu.warning)
 		. = ALIGN(16);
 		*(.got)			/* Global offset table		*/

@@ -195,6 +194,7 @@ SECTIONS
 		TRAMP_TEXT
 		HIBERNATE_TEXT
 		KEXEC_TEXT
+		IDMAP_TEXT
 		. = ALIGN(PAGE_SIZE);
 	}
arch/arm64/mm/proc.S +0 -2
index b9ecbbae1e1a..d7ca6f23fb0d 100644
@@ -110,7 +110,6 @@ SYM_FUNC_END(cpu_do_suspend)
  *
  * x0: Address of context pointer
  */
-	.pushsection ".idmap.text", "awx"
 SYM_FUNC_START(cpu_do_resume)
 	ldp	x2, x3, [x0]
 	ldp	x4, x5, [x0, #16]

@@ -166,7 +165,6 @@ alternative_else_nop_endif
 	isb
 	ret
 SYM_FUNC_END(cpu_do_resume)
-	.popsection
 #endif
 
 	.pushsection ".idmap.text", "awx"
Signature · 1 line
2.35.1
author maintainer
#3 lore raw

PATCH v4 3/6 arm64: head: record the MMU state at primary entry

In reply to Ard Biesheuvel

Prepare for being able to deal with primary entry with the MMU and caches enabled, by recording whether or not we entered with the MMU on in register x19.

While at it, add disable_mmu_workaround macro invocations to init_kernel_el, as its manipulation of SCTLR_ELx may amount to disabling of the MMU after subsequent patches.

Signed-off-by Ard Biesheuvel <[email protected]>
---
 arch/arm64/kernel/head.S | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
arch/arm64/kernel/head.S +18 -0
index 25a84ce1700c..643797b21a1c 100644
@@ -77,6 +77,7 @@
 	 * primary lowlevel boot path:
 	 *
 	 *  Register   Scope                      Purpose
+	 *  x19        primary_entry() .. start_kernel()        whether we entered with the MMU on
 	 *  x20        primary_entry() .. __primary_switch()    CPU boot mode
 	 *  x21        primary_entry() .. start_kernel()        FDT pointer passed at boot in x0
 	 *  x22        create_idmap() .. start_kernel()         ID map VA of the DT blob

@@ -86,6 +87,7 @@
 	 *  x28        create_idmap()                           callee preserved temp register
 	 */
 SYM_CODE_START(primary_entry)
+	bl	record_mmu_state
 	bl	preserve_boot_args
 	bl	init_kernel_el			// w0=cpu_boot_mode
 	mov	x20, x0

@@ -109,6 +111,19 @@ SYM_CODE_START(primary_entry)
 	b	__primary_switch
 SYM_CODE_END(primary_entry)
 
+SYM_CODE_START_LOCAL(record_mmu_state)
+	mrs	x19, CurrentEL
+	cmp	x19, #CurrentEL_EL2
+	mrs	x19, sctlr_el1
+	b.ne	0f
+	mrs	x19, sctlr_el2
+0:	tst	x19, #SCTLR_ELx_C		// Z := (C == 0)
+	and	x19, x19, #SCTLR_ELx_M		// isolate M bit
+	ccmp	x19, xzr, #4, ne		// Z |= (M == 0)
+	cset	x19, ne				// set x19 if !Z
+	ret
+SYM_CODE_END(record_mmu_state)
+
 /*
  * Preserve the arguments passed by the bootloader in x0 .. x3
  */

@@ -497,6 +512,7 @@ SYM_FUNC_START(init_kernel_el)
 
 SYM_INNER_LABEL(init_el1, SYM_L_LOCAL)
 	mov_q	x0, INIT_SCTLR_EL1_MMU_OFF
+	pre_disable_mmu_workaround
 	msr	sctlr_el1, x0
 	isb
 	mov_q	x0, INIT_PSTATE_EL1

@@ -529,11 +545,13 @@ SYM_INNER_LABEL(init_el2, SYM_L_LOCAL)
 	cbz	x0, 1f
 
 	/* Set a sane SCTLR_EL1, the VHE way */
+	pre_disable_mmu_workaround
 	msr_s	SYS_SCTLR_EL12, x1
 	mov	x2, #BOOT_CPU_FLAG_E2H
 	b	2f
 
 1:
+	pre_disable_mmu_workaround
 	msr	sctlr_el1, x1
 	mov	x2, xzr
 2:
Signature · 1 line
2.35.1
author maintainer
#4 lore raw

PATCH v4 4/6 arm64: head: avoid cache invalidation when entering with the MMU on

In reply to Ard Biesheuvel

If we enter with the MMU on, there is no need for explicit cache invalidation for stores to memory, as they will be coherent with the caches.

Let's take advantage of this, and create the ID map with the MMU still enabled if that is how we entered, and avoid any cache invalidation calls in that case.

Signed-off-by Ard Biesheuvel <[email protected]>
---
 arch/arm64/kernel/head.S | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
arch/arm64/kernel/head.S +5 -2
index 643797b21a1c..5de2ba3539a8 100644
@@ -89,9 +89,9 @@
 SYM_CODE_START(primary_entry)
 	bl	record_mmu_state
 	bl	preserve_boot_args
+	bl	create_idmap
 	bl	init_kernel_el			// w0=cpu_boot_mode
 	mov	x20, x0
-	bl	create_idmap
 
 	/*
 	 * The following calls CPU setup code, see arch/arm64/mm/proc.S for

@@ -134,11 +134,13 @@ SYM_CODE_START_LOCAL(preserve_boot_args)
 	stp	x21, x1, [x0]			// x0 .. x3 at kernel entry
 	stp	x2, x3, [x0, #16]
 
+	cbnz	x19, 0f				// skip cache invalidation if MMU is on
 	dmb	sy				// needed before dc ivac with
 						// MMU off
 
 	add	x1, x0, #0x20			// 4 x 8 bytes
 	b	dcache_inval_poc		// tail call
+0:	ret
 SYM_CODE_END(preserve_boot_args)
 
 SYM_FUNC_START_LOCAL(clear_page_tables)

@@ -375,12 +377,13 @@ SYM_FUNC_START_LOCAL(create_idmap)
 	 * accesses (MMU disabled), invalidate those tables again to
 	 * remove any speculatively loaded cache lines.
 	 */
+	cbnz	x19, 0f				// skip cache invalidation if MMU is on
 	dmb	sy
 
 	adrp	x0, init_idmap_pg_dir
 	adrp	x1, init_idmap_pg_end
 	bl	dcache_inval_poc
-	ret	x28
+0:	ret	x28
 SYM_FUNC_END(create_idmap)
 
 SYM_FUNC_START_LOCAL(create_kernel_mapping)
Signature · 1 line
2.35.1
author maintainer
#5 lore raw

PATCH v4 5/6 arm64: head: clean the ID map page to the PoC

In reply to Ard Biesheuvel

If we enter with the MMU and caches enabled, the caller may not have performed any cache maintenance. So clean the ID mapped page to the PoC, to ensure that instruction and data accesses with the MMU off see the correct data.

Note that this means primary_entry() itself needs to be moved into the ID map as well, as we will return from init_kernel_el() with the MMU and caches off.

Signed-off-by Ard Biesheuvel <[email protected]>
---
 arch/arm64/kernel/head.S | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
arch/arm64/kernel/head.S +13 -1
index 5de2ba3539a8..c8b8ed8477c1 100644
@@ -70,7 +70,7 @@
 
 	__EFI_PE_HEADER
 
-	__INIT
+	.section ".idmap.text","awx"
 
 	/*
 	 * The following callee saved general purpose registers are used on the

@@ -90,6 +90,17 @@ SYM_CODE_START(primary_entry)
 	bl	record_mmu_state
 	bl	preserve_boot_args
 	bl	create_idmap
+
+	/*
+	 * If we entered with the MMU and caches on, clean the ID mapped part
+	 * of the primary boot code to the PoC so we can safely execute it with
+	 * the MMU off.
+	 */
+	cbz	x19, 0f
+	adrp	x0, __idmap_text_start
+	adr_l	x1, __idmap_text_end
+	bl	dcache_clean_poc
+0:
 	bl	init_kernel_el			// w0=cpu_boot_mode
 	mov	x20, x0
 

@@ -111,6 +122,7 @@ SYM_CODE_START(primary_entry)
 	b	__primary_switch
 SYM_CODE_END(primary_entry)
 
+	__INIT
 SYM_CODE_START_LOCAL(record_mmu_state)
 	mrs	x19, CurrentEL
 	cmp	x19, #CurrentEL_EL2
Signature · 1 line
2.35.1
author maintainer
#6 lore raw

PATCH v4 6/6 arm64: efi/libstub: enter with the MMU on

In reply to Ard Biesheuvel

Instead of disabling the MMU and caches before jumping to the kernel's entry point, just call it directly, and keep the MMU and caches enabled.

This removes the need for any unconditional cache invalidation to the PoC in the entry path (although cache maintenance of the code portion of the image is still necessary for I/D coherency if the image was moved around in memory). It also allows us to get rid of the asm routine, as doing the jump is easily done from C code.

Signed-off-by Ard Biesheuvel <[email protected]>
---
 arch/arm64/kernel/image-vars.h             |  5 +-
 arch/arm64/mm/cache.S                      |  5 +-
 drivers/firmware/efi/libstub/arm64-entry.S | 57 --------------------
 drivers/firmware/efi/libstub/arm64-stub.c  | 17 +++++-
 4 files changed, 22 insertions(+), 62 deletions(-)
arch/arm64/kernel/image-vars.h +3 -2
index 74d20835cf91..13e082e946c5 100644
@@ -10,7 +10,7 @@
 #error This file should only be included in vmlinux.lds.S
 #endif
 
-PROVIDE(__efistub_primary_entry_offset	= primary_entry - _text);
+PROVIDE(__efistub_primary_entry		= primary_entry);
 
 /*
  * The EFI stub has its own symbol namespace prefixed by __efistub_, to

@@ -28,10 +28,11 @@ PROVIDE(__efistub_strnlen		= __pi_strnlen);
 PROVIDE(__efistub_strcmp		= __pi_strcmp);
 PROVIDE(__efistub_strncmp		= __pi_strncmp);
 PROVIDE(__efistub_strrchr		= __pi_strrchr);
-PROVIDE(__efistub_dcache_clean_poc	= __pi_dcache_clean_poc);
+PROVIDE(__efistub_caches_clean_inval_pou = __pi_caches_clean_inval_pou);
 
 PROVIDE(__efistub__text			= _text);
 PROVIDE(__efistub__end			= _end);
+PROVIDE(__efistub___inittext_end       	= __inittext_end);
 PROVIDE(__efistub__edata		= _edata);
 PROVIDE(__efistub_screen_info		= screen_info);
 PROVIDE(__efistub__ctype		= _ctype);
arch/arm64/mm/cache.S +3 -2
index 081058d4e436..8c3b3ee9b1d7 100644
@@ -52,10 +52,11 @@ alternative_else_nop_endif
  *	- start   - virtual start address of region
  *	- end     - virtual end address of region
  */
-SYM_FUNC_START(caches_clean_inval_pou)
+SYM_FUNC_START(__pi_caches_clean_inval_pou)
 	caches_clean_inval_pou_macro
 	ret
-SYM_FUNC_END(caches_clean_inval_pou)
+SYM_FUNC_END(__pi_caches_clean_inval_pou)
+SYM_FUNC_ALIAS(caches_clean_inval_pou, __pi_caches_clean_inval_pou)
 
 /*
  *	caches_clean_inval_user_pou(start,end)
drivers/firmware/efi/libstub/arm64-entry.S deleted +0 -57

57 lines not on this page · raw message

drivers/firmware/efi/libstub/arm64-stub.c +16 -1
index 6229f42c797f..9c7e2c1aace2 100644
@@ -86,7 +86,7 @@ efi_status_t handle_kernel_image(unsigned long *image_addr,
 				 efi_handle_t image_handle)
 {
 	efi_status_t status;
-	unsigned long kernel_size, kernel_memsize = 0;
+	unsigned long kernel_size, kernel_codesize, kernel_memsize;
 	u32 phys_seed = 0;
 
 	/*

@@ -130,6 +130,7 @@ efi_status_t handle_kernel_image(unsigned long *image_addr,
 			SEGMENT_ALIGN >> 10);
 
 	kernel_size = _edata - _text;
+	kernel_codesize = __inittext_end - _text;
 	kernel_memsize = kernel_size + (_end - _edata);
 	*reserve_size = kernel_memsize;
 

@@ -173,6 +174,7 @@ efi_status_t handle_kernel_image(unsigned long *image_addr,
 
 	*image_addr = *reserve_addr;
 	memcpy((void *)*image_addr, _text, kernel_size);
+	caches_clean_inval_pou(*image_addr, *image_addr + kernel_codesize);
 
 clean_image_to_poc:
 	/*

@@ -184,3 +186,16 @@ efi_status_t handle_kernel_image(unsigned long *image_addr,
 
 	return EFI_SUCCESS;
 }
+
+asmlinkage void primary_entry(void);
+
+void __noreturn efi_enter_kernel(unsigned long entrypoint,
+				 unsigned long fdt_addr,
+				 unsigned long fdt_size)
+{
+	void (* __noreturn enter_kernel)(u64, u64, u64, u64);
+	u64 offset = (char *)primary_entry - _text;
+
+	enter_kernel = (void *)entrypoint + offset;
+	enter_kernel(fdt_addr, 0, 0, 0);
+}
Signature · 1 line
2.35.1
maintainer
#7 lore raw
In reply to Ard Biesheuvel

Hi Ard,

Quoted 14 lines · Ard Biesheuvel

The purpose of this series is to remove any explicit cache maintenance for coherency during early boot that becomes unnecessary if we simply retain the cacheable 1:1 mapping of all of system RAM provided by EFI, and use it to populate the ID map page tables. After setting up this preliminary ID map, we disable the MMU, drop to EL1, reprogram the MAIR, TCR and SCTLR registers as before, and proceed as usual, avoiding the need for any manipulations of memory while the MMU and caches are off.

The only properties of the firmware provided 1:1 map we rely on is that it does not require any explicit cache maintenance for coherency, and that it covers the entire memory footprint of the image, including the BSS and padding at the end - all else is under control of the kernel itself, as before.

Quoted 6 lines

Changes since v3:
- drop EFI_LOADER_CODE memory type patch that has been queued in the mean time
- rebased onto [partial] series that moves efi-entry.S into the libstub/ source directory [0]
- fixed a correctness issue in patch #2

I really like this series, but I'm also very nervous about supporting booting the kernel with the MMU enabled outside of EFI. The booting documentation prohibits this, but we don't appear to take any steps to prevent this case with your series. Perhaps we shouldn't, but I do think it would be worth trying to warn+taint if we detect it so that we don't spend too much time debugging strange memory issues on platforms that try to use such a configuration.

What do you think?

Cheers,

Will

author maintainer
#8 lore raw
Quoted 33 lines · Will Deacon

Hi Ard,

Quoted 14 lines · Ard Biesheuvel

The purpose of this series is to remove any explicit cache maintenance for coherency during early boot that becomes unnecessary if we simply retain the cacheable 1:1 mapping of all of system RAM provided by EFI, and use it to populate the ID map page tables. After setting up this preliminary ID map, we disable the MMU, drop to EL1, reprogram the MAIR, TCR and SCTLR registers as before, and proceed as usual, avoiding the need for any manipulations of memory while the MMU and caches are off.

The only properties of the firmware provided 1:1 map we rely on is that it does not require any explicit cache maintenance for coherency, and that it covers the entire memory footprint of the image, including the BSS and padding at the end - all else is under control of the kernel itself, as before.

Quoted 6 lines

Changes since v3:
- drop EFI_LOADER_CODE memory type patch that has been queued in the mean time
- rebased onto [partial] series that moves efi-entry.S into the libstub/ source directory [0]
- fixed a correctness issue in patch #2

I really like this series, but I'm also very nervous about supporting booting the kernel with the MMU enabled outside of EFI. The booting documentation prohibits this, but we don't appear to take any steps to prevent this case with your series. Perhaps we shouldn't, but I do think it would be worth trying to warn+taint if we detect it so that we don't spend too much time debugging strange memory issues on platforms that try to use such a configuration.

Quoted 2 lines

What do you think?

I share your concern, and capturing the value of SCTLR at boot and warning about it later should be trivial to do. In fact, we already do something similar for the alignment, where only EFI is permitted to deviate from the 2 MiB alignment requirement of the image's placement in memory. I'll add something in the same spot.

Note that I need to respin this in any case - the EL2 startup code needs to be cleaned to the PoC as well, given that it will also execute with MMU and caches off at EL2 when finalise_el2() is called.

I was about to get back to this so I should have a v5 tomorrow.