Skip to contentSkip to conversation

efi/arm: deal with HYP mode boot

PATCH In review Ard Biesheuvel 4 participants 14 messages Jun 7, 2020 – Jun 11, 2020

Subsystems (2)
lore
Share & tools
lore

Ard Biesheuvel

author maintainer
#0 lore raw

v2 cover efi/arm: deal with HYP mode boot

Some changes to make booting in HYP mode via EFI more robust.

Changes since v2:
- Use the correct register for testing/disabling the HYP MMU
- Don't pretend we're in SVC mode for the HYP+caches case, but set up a HYP exception handler for HVC instructions issued from HYP mode*** BLURB HERE ***

Ard Biesheuvel (2):
efi/arm: decompressor: deal with HYP mode boot gracefully efi/arm: libstub: print boot mode and MMU state at boot

 arch/arm/boot/compressed/head.S           | 61 ++++++++++++++++++++
 drivers/firmware/efi/libstub/arm32-stub.c | 11 ++++
 2 files changed, 72 insertions(+)
Signature · 1 line
2.26.2
author maintainer
#1 lore raw

PATCH v2 1/2 efi/arm: decompressor: deal with HYP mode boot gracefully

EFI on ARM only supports short descriptors, and given that it mandates that the MMU and caches are on, it is implied that booting in HYP mode is not supported.

However, implementations of EFI exist (i.e., U-Boot) that ignore this requirement, which is not entirely unreasonable, given that it makes HYP mode inaccessible to the operating system.

So let's make sure that we can deal with this condition gracefully. We already tolerate booting the EFI stub with the caches off (even though this violates the EFI spec as well), and so we should deal with HYP mode boot with MMU and caches either on or off.

- When the MMU and caches are on, we can ignore the HYP stub altogether, since we can carry on executing at HYP. We do need to ensure that we disable the MMU at HYP before entering the kernel proper.

- When the MMU and caches are off, we have to drop to SVC mode so that we can set up the page tables using short descriptors. In this case, we need to install the HYP stub as usual, so that we can return to HYP mode before handing over to the kernel proper.

---
 arch/arm/boot/compressed/head.S | 61 ++++++++++++++++++++
 1 file changed, 61 insertions(+)
arch/arm/boot/compressed/head.S +61 -0
index c79db44ba128..3476e85c31e7 100644
@@ -1410,7 +1410,11 @@ memdump:	mov	r12, r0
 __hyp_reentry_vectors:
 		W(b)	.			@ reset
 		W(b)	.			@ undef
+#ifdef CONFIG_EFI_STUB
+		W(b)	__enter_kernel_from_hyp	@ hvc from HYP
+#else
 		W(b)	.			@ svc
+#endif
 		W(b)	.			@ pabort
 		W(b)	.			@ dabort
 		W(b)	__enter_kernel		@ hyp
@@ -1429,14 +1433,71 @@ __enter_kernel:
 reloc_code_end:
 
 #ifdef CONFIG_EFI_STUB
+__enter_kernel_from_hyp:
+		mrc	p15, 4, r0, c1, c0, 0	@ read HSCTLR
+		bic	r0, r0, #0x5		@ disable MMU and caches
+		mcr	p15, 4, r0, c1, c0, 0	@ write HSCTLR
+		isb
+		b	__enter_kernel
+
 ENTRY(efi_enter_kernel)
 		mov	r4, r0			@ preserve image base
 		mov	r8, r1			@ preserve DT pointer
 
+ ARM(		adrl	r0, call_cache_fn	)
+ THUMB(		adr	r0, call_cache_fn	)
+		adr	r1, 0f			@ clean the region of code we
+		bl	cache_clean_flush	@ may run with the MMU off
+
+#ifdef CONFIG_ARM_VIRT_EXT
+		@
+		@ The EFI spec does not support booting on ARM in HYP mode,
+		@ since it mandates that the MMU and caches are on, with all
+		@ 32-bit addressable DRAM mapped 1:1 using short descriptors.
+		@
+		@ While the EDK2 reference implementation adheres to this,
+		@ U-Boot might decide to enter the EFI stub in HYP mode
+		@ anyway, with the MMU and caches either on or off.
+		@
+		mrs	r0, cpsr		@ get the current mode
+		msr	spsr_cxsf, r0		@ record boot mode
+		and	r0, r0, #MODE_MASK	@ are we running in HYP mode?
+		cmp	r0, #HYP_MODE
+		bne	.Lefi_svc
+
+		mrc	p15, 4, r1, c1, c0, 0	@ read HSCTLR
+		tst	r1, #0x1		@ MMU enabled at HYP?
+		beq	1f
+
+		@
+		@ When running in HYP mode with the caches on, we're better
+		@ off just carrying on using the cached 1:1 mapping that the
+		@ firmware provided. Set up the HYP vectors so HVC instructions
+		@ issued from HYP mode take us to the correct handler code. We
+		@ will disable the MMU before jumping to the kernel proper.
+		@
+		adr	r0, __hyp_reentry_vectors
+		mcr	p15, 4, r0, c12, c0, 0	@ set HYP vector base (HVBAR)
+		isb
+		b	.Lefi_hyp
+
+		@
+		@ When running in HYP mode with the caches off, we need to drop
+		@ into SVC mode now, and let the decompressor set up its cached
+		@ 1:1 mapping as usual.
+		@
+1:		mov	r9, r4			@ preserve image base
+		bl	__hyp_stub_install	@ install HYP stub vectors
+		safe_svcmode_maskall	r1	@ drop to SVC mode
+		orr	r4, r9, #1		@ restore image base and set LSB
+		b	.Lefi_hyp
+.Lefi_svc:
+#endif
 		mrc	p15, 0, r0, c1, c0, 0	@ read SCTLR
 		tst	r0, #0x1		@ MMU enabled?
 		orreq	r4, r4, #1		@ set LSB if not
 
+.Lefi_hyp:
 		mov	r0, r8			@ DT start
 		add	r1, r8, r2		@ DT end
 		bl	cache_clean_flush
Signature · 1 line
2.26.2
author maintainer
#13 lore raw

PATCH v2 2/2 efi/arm: libstub: print boot mode and MMU state at boot

In reply to Ard Biesheuvel

On 32-bit ARM, we may boot at HYP mode, or with the MMU and caches off (or both), even though the EFI spec actually does not support this. Take note of this in the EFI stub diagnostic output so that we can easily see whether this is the case or not.

---
 drivers/firmware/efi/libstub/arm32-stub.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/firmware/efi/libstub/arm32-stub.c +11 -0
index 40243f524556..659aaa5003a2 100644
@@ -9,8 +9,19 @@
 
 efi_status_t check_platform_features(void)
 {
+	u32 cpsr, sctlr;
 	int block;
 
+	asm("mrs %0, cpsr" : "=r"(cpsr));
+	if ((cpsr & MODE_MASK) == HYP_MODE)
+		asm("mrc p15, 4, %0, c1, c0, 0" : "=r"(sctlr));
+	else
+		asm("mrc p15, 0, %0, c1, c0, 0" : "=r"(sctlr));
+
+	efi_info("Running in %s mode with MMU %sabled\n",
+		 ((cpsr & MODE_MASK) == HYP_MODE) ? "HYP" : "SVC",
+		 (sctlr & 1) ? "en" : "dis");
+
 	/* non-LPAE kernels can run anywhere */
 	if (!IS_ENABLED(CONFIG_ARM_LPAE))
 		return EFI_SUCCESS;
Signature · 1 line
2.26.2
#2 lore raw

Re: [PATCH v2 1/2] efi/arm: decompressor: deal with HYP mode boot gracefully

In reply to Ard Biesheuvel
Quoted 13 lines · Ard Biesheuvel

EFI on ARM only supports short descriptors, and given that it mandates that the MMU and caches are on, it is implied that booting in HYP mode is not supported.

However, implementations of EFI exist (i.e., U-Boot) that ignore this requirement, which is not entirely unreasonable, given that it makes HYP mode inaccessible to the operating system.

So let's make sure that we can deal with this condition gracefully. We already tolerate booting the EFI stub with the caches off (even though this violates the EFI spec as well), and so we should deal with HYP mode boot with MMU and caches either on or off.

Quoted 8 lines

- When the MMU and caches are on, we can ignore the HYP stub altogether, since we can carry on executing at HYP. We do need to ensure that we disable the MMU at HYP before entering the kernel proper.

- When the MMU and caches are off, we have to drop to SVC mode so that we can set up the page tables using short descriptors. In this case, we need to install the HYP stub as usual, so that we can return to HYP mode before handing over to the kernel proper.

To me it is still unclear why you need this patch. Please, describe the problem this patch fixes.

Is there any device that you cannot boot without the patch?

Quoted 72 lines
---
 arch/arm/boot/compressed/head.S | 61 ++++++++++++++++++++
 1 file changed, 61 insertions(+)
arch/arm/boot/compressed/head.S +46 -0
index c79db44ba128..3476e85c31e7 100644
@@ -1410,7 +1410,11 @@ memdump:	mov	r12, r0
 __hyp_reentry_vectors:
 		W(b)	.			@ reset
 		W(b)	.			@ undef
+#ifdef CONFIG_EFI_STUB
+		W(b)	__enter_kernel_from_hyp	@ hvc from HYP
+#else
 		W(b)	.			@ svc
+#endif
 		W(b)	.			@ pabort
 		W(b)	.			@ dabort
 		W(b)	__enter_kernel		@ hyp
@@ -1429,14 +1433,71 @@ __enter_kernel:
 reloc_code_end:
 
 #ifdef CONFIG_EFI_STUB
+__enter_kernel_from_hyp:
+		mrc	p15, 4, r0, c1, c0, 0	@ read HSCTLR
+		bic	r0, r0, #0x5		@ disable MMU and caches
+		mcr	p15, 4, r0, c1, c0, 0	@ write HSCTLR
+		isb
+		b	__enter_kernel
+
 ENTRY(efi_enter_kernel)
 		mov	r4, r0			@ preserve image base
 		mov	r8, r1			@ preserve DT pointer
 
+ ARM(		adrl	r0, call_cache_fn	)
+ THUMB(		adr	r0, call_cache_fn	)
+		adr	r1, 0f			@ clean the region of code we
+		bl	cache_clean_flush	@ may run with the MMU off
+
+#ifdef CONFIG_ARM_VIRT_EXT
+		@
+		@ The EFI spec does not support booting on ARM in HYP mode,
+		@ since it mandates that the MMU and caches are on, with all
+		@ 32-bit addressable DRAM mapped 1:1 using short descriptors.
+		@
+		@ While the EDK2 reference implementation adheres to this,
+		@ U-Boot might decide to enter the EFI stub in HYP mode
+		@ anyway, with the MMU and caches either on or off.
+		@
+		mrs	r0, cpsr		@ get the current mode
+		msr	spsr_cxsf, r0		@ record boot mode
+		and	r0, r0, #MODE_MASK	@ are we running in HYP mode?
+		cmp	r0, #HYP_MODE
+		bne	.Lefi_svc
+
+		mrc	p15, 4, r1, c1, c0, 0	@ read HSCTLR
+		tst	r1, #0x1		@ MMU enabled at HYP?
+		beq	1f
+
+		@
+		@ When running in HYP mode with the caches on, we're better
+		@ off just carrying on using the cached 1:1 mapping that the
+		@ firmware provided. Set up the HYP vectors so HVC instructions
+		@ issued from HYP mode take us to the correct handler code. We
+		@ will disable the MMU before jumping to the kernel proper.
+		@
+		adr	r0, __hyp_reentry_vectors
+		mcr	p15, 4, r0, c12, c0, 0	@ set HYP vector base (HVBAR)
+		isb

arch/arm/boot/compressed/head.S __enter_kernel: · line 1491

Quoted 10 lines
arch/arm/boot/compressed/head.S +10 -0
+		b	.Lefi_hyp
+
+		@
+		@ When running in HYP mode with the caches off, we need to drop
+		@ into SVC mode now, and let the decompressor set up its cached
+		@ 1:1 mapping as usual.
+		@
+1:		mov	r9, r4			@ preserve image base
+		bl	__hyp_stub_install	@ install HYP stub vectors
+		safe_svcmode_maskall	r1	@ drop to SVC mode

Are you returning to HYP mode somewhere?

What is the effect on PSCI?

The Allwinner/Sunxi boards must be booted in HYP mode to have PSCI according to https://linux-sunxi.org/PSCI

Did you test that you still can reboot those boards?

Cc: Chen-Yu Tsai <[email protected]>
(maintainer ARM/Allwinner sunXi SoC support)

Best regards

Heinrich

arch/arm/boot/compressed/head.S __enter_kernel: · line 1503

Quoted 13 lines
arch/arm/boot/compressed/head.S +5 -0
+		orr	r4, r9, #1		@ restore image base and set LSB
+		b	.Lefi_hyp
+.Lefi_svc:
+#endif
 		mrc	p15, 0, r0, c1, c0, 0	@ read SCTLR
 		tst	r0, #0x1		@ MMU enabled?
 		orreq	r4, r4, #1		@ set LSB if not
 
+.Lefi_hyp:
 		mov	r0, r8			@ DT start
 		add	r1, r8, r2		@ DT end
 		bl	cache_clean_flush

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

author maintainer
#3 lore raw

Re: [PATCH v2 1/2] efi/arm: decompressor: deal with HYP mode boot gracefully

Quoted 24 lines · Heinrich Schuchardt
Quoted 13 lines · Ard Biesheuvel

EFI on ARM only supports short descriptors, and given that it mandates that the MMU and caches are on, it is implied that booting in HYP mode is not supported.

However, implementations of EFI exist (i.e., U-Boot) that ignore this requirement, which is not entirely unreasonable, given that it makes HYP mode inaccessible to the operating system.

So let's make sure that we can deal with this condition gracefully. We already tolerate booting the EFI stub with the caches off (even though this violates the EFI spec as well), and so we should deal with HYP mode boot with MMU and caches either on or off.

Quoted 8 lines

- When the MMU and caches are on, we can ignore the HYP stub altogether, since we can carry on executing at HYP. We do need to ensure that we disable the MMU at HYP before entering the kernel proper.

- When the MMU and caches are off, we have to drop to SVC mode so that we can set up the page tables using short descriptors. In this case, we need to install the HYP stub as usual, so that we can return to HYP mode before handing over to the kernel proper.

Quoted 5 lines

To me it is still unclear why you need this patch. Please, describe the problem this patch fixes.

Is there any device that you cannot boot without the patch?

The code as it is today is broken, and if it works, it only does so by accident.

(There were some recent changes but the old code is broken in a similar way)

When we enter via the stub, we used to call cache_off() to disable the caches before calling the decompressor entry point. However, that disables the SVC mode caches, not the HYP mode caches, and so if we enter via EFI at HYP, we will call __hyp_stub_install() with the HYP mod MMU and caches enabled, which is explicitly forbidden (see hyp-stub.S)

With the recent change, the EFI entry code doesn't call cache_off() anymore, but that does not remove the problem, it just moves it to the point where we hand over to the kernel proper.

The problem is really on the u-boot side, and so we either have to follow the letter of the EFI spec and ban the practice of booting in HYP mode or with the caches off, or we work around this like I do in this patch. Doing nothing is not really an option.

If we want EBBR and U-boot as EFI firmware to succeed, we should really fix issues such as these, and not pretend everything is fine when we know it is broken but happens to work on the few boards that we test. This is the reason we have architecture and firmware specs in the first place, and it is really quite unfortunate that we did not catch these u-boot issues before.

As I said, I think booting at HYP can be tolerated, since the OS loses access to it otherwise (and maybe we should even update the EFI spec to allow this). But fiddling with the caches like we do should really be avoided (and the GRUB hack really needs to go as well)

Quoted 76 lines
Quoted 76 lines
---
 arch/arm/boot/compressed/head.S | 61 ++++++++++++++++++++
 1 file changed, 61 insertions(+)
arch/arm/boot/compressed/head.S +50 -0
index c79db44ba128..3476e85c31e7 100644
@@ -1410,7 +1410,11 @@ memdump:       mov     r12, r0
 __hyp_reentry_vectors:
              W(b)    .                       @ reset
              W(b)    .                       @ undef
+#ifdef CONFIG_EFI_STUB
+             W(b)    __enter_kernel_from_hyp @ hvc from HYP
+#else
              W(b)    .                       @ svc
+#endif
              W(b)    .                       @ pabort
              W(b)    .                       @ dabort
              W(b)    __enter_kernel          @ hyp
@@ -1429,14 +1433,71 @@ __enter_kernel:
 reloc_code_end:
 
 #ifdef CONFIG_EFI_STUB
+__enter_kernel_from_hyp:
+             mrc     p15, 4, r0, c1, c0, 0   @ read HSCTLR
+             bic     r0, r0, #0x5            @ disable MMU and caches
+             mcr     p15, 4, r0, c1, c0, 0   @ write HSCTLR
+             isb
+             b       __enter_kernel
+
 ENTRY(efi_enter_kernel)
              mov     r4, r0                  @ preserve image base
              mov     r8, r1                  @ preserve DT pointer
 
+ ARM(                adrl    r0, call_cache_fn       )
+ THUMB(              adr     r0, call_cache_fn       )
+             adr     r1, 0f                  @ clean the region of code we
+             bl      cache_clean_flush       @ may run with the MMU off
+
+#ifdef CONFIG_ARM_VIRT_EXT
+             @
+             @ The EFI spec does not support booting on ARM in HYP mode,
+             @ since it mandates that the MMU and caches are on, with all
+             @ 32-bit addressable DRAM mapped 1:1 using short descriptors.
+             @
+             @ While the EDK2 reference implementation adheres to this,
+             @ U-Boot might decide to enter the EFI stub in HYP mode
+             @ anyway, with the MMU and caches either on or off.
+             @
+             mrs     r0, cpsr                @ get the current mode
+             msr     spsr_cxsf, r0           @ record boot mode
+             and     r0, r0, #MODE_MASK      @ are we running in HYP mode?
+             cmp     r0, #HYP_MODE
+             bne     .Lefi_svc
+
+             mrc     p15, 4, r1, c1, c0, 0   @ read HSCTLR
+             tst     r1, #0x1                @ MMU enabled at HYP?
+             beq     1f
+
+             @
+             @ When running in HYP mode with the caches on, we're better
+             @ off just carrying on using the cached 1:1 mapping that the
+             @ firmware provided. Set up the HYP vectors so HVC instructions
+             @ issued from HYP mode take us to the correct handler code. We
+             @ will disable the MMU before jumping to the kernel proper.
+             @
+             adr     r0, __hyp_reentry_vectors
+             mcr     p15, 4, r0, c12, c0, 0  @ set HYP vector base (HVBAR)
+             isb
+             b       .Lefi_hyp
+
+             @
+             @ When running in HYP mode with the caches off, we need to drop
Quoted 9 lines

arch/arm/boot/compressed/head.S __enter_kernel: · line 1491

Quoted 6 lines
arch/arm/boot/compressed/head.S +6 -0
+             @ into SVC mode now, and let the decompressor set up its cached
+             @ 1:1 mapping as usual.
+             @
+1:           mov     r9, r4                  @ preserve image base
+             bl      __hyp_stub_install      @ install HYP stub vectors
+             safe_svcmode_maskall    r1      @ drop to SVC mode

Are you returning to HYP mode somewhere?

Yes.

Quoted 2 lines

What is the effect on PSCI?

If you boot Linux in HYP then you cannot have PSCI in HYP as well. Linux will take ownership of HYP mode, and remove whatever was there. If you want to run PSCI at HYP, then the OS needs to boot in SVC mode.

Quoted 3 lines

The Allwinner/Sunxi boards must be booted in HYP mode to have PSCI according to https://linux-sunxi.org/PSCI

See above. If PSCI runs in HYP, the OS needs to run at SVC

Quoted 2 lines

Did you test that you still can reboot those boards?

No, I don't have such a board.

Quoted 21 lines

Cc: Chen-Yu Tsai <[email protected]>
(maintainer ARM/Allwinner sunXi SoC support)

Best regards

Heinrich

Quoted 13 lines
arch/arm/boot/compressed/head.S +5 -0
+             orr     r4, r9, #1              @ restore image base and set LSB
+             b       .Lefi_hyp
+.Lefi_svc:
+#endif
              mrc     p15, 0, r0, c1, c0, 0   @ read SCTLR
              tst     r0, #0x1                @ MMU enabled?
              orreq   r4, r4, #1              @ set LSB if not
 
+.Lefi_hyp:
              mov     r0, r8                  @ DT start
              add     r1, r8, r2              @ DT end
              bl      cache_clean_flush
#4 lore raw

Re: [PATCH v2 1/2] efi/arm: decompressor: deal with HYP mode boot gracefully

Quoted 161 lines · Ard Biesheuvel
Quoted 24 lines · Heinrich Schuchardt
Quoted 13 lines · Ard Biesheuvel

EFI on ARM only supports short descriptors, and given that it mandates that the MMU and caches are on, it is implied that booting in HYP mode is not supported.

However, implementations of EFI exist (i.e., U-Boot) that ignore this requirement, which is not entirely unreasonable, given that it makes HYP mode inaccessible to the operating system.

So let's make sure that we can deal with this condition gracefully. We already tolerate booting the EFI stub with the caches off (even though this violates the EFI spec as well), and so we should deal with HYP mode boot with MMU and caches either on or off.

Quoted 8 lines

- When the MMU and caches are on, we can ignore the HYP stub altogether, since we can carry on executing at HYP. We do need to ensure that we disable the MMU at HYP before entering the kernel proper.

- When the MMU and caches are off, we have to drop to SVC mode so that we can set up the page tables using short descriptors. In this case, we need to install the HYP stub as usual, so that we can return to HYP mode before handing over to the kernel proper.

Quoted 5 lines

To me it is still unclear why you need this patch. Please, describe the problem this patch fixes.

Is there any device that you cannot boot without the patch?

The code as it is today is broken, and if it works, it only does so by accident.

(There were some recent changes but the old code is broken in a similar way)

When we enter via the stub, we used to call cache_off() to disable the caches before calling the decompressor entry point. However, that disables the SVC mode caches, not the HYP mode caches, and so if we enter via EFI at HYP, we will call __hyp_stub_install() with the HYP mod MMU and caches enabled, which is explicitly forbidden (see hyp-stub.S)

With the recent change, the EFI entry code doesn't call cache_off() anymore, but that does not remove the problem, it just moves it to the point where we hand over to the kernel proper.

The problem is really on the u-boot side, and so we either have to follow the letter of the EFI spec and ban the practice of booting in HYP mode or with the caches off, or we work around this like I do in this patch. Doing nothing is not really an option.

If we want EBBR and U-boot as EFI firmware to succeed, we should really fix issues such as these, and not pretend everything is fine when we know it is broken but happens to work on the few boards that we test. This is the reason we have architecture and firmware specs in the first place, and it is really quite unfortunate that we did not catch these u-boot issues before.

As I said, I think booting at HYP can be tolerated, since the OS loses access to it otherwise (and maybe we should even update the EFI spec to allow this). But fiddling with the caches like we do should really be avoided (and the GRUB hack really needs to go as well)

Quoted 76 lines
Quoted 76 lines
---
 arch/arm/boot/compressed/head.S | 61 ++++++++++++++++++++
 1 file changed, 61 insertions(+)
arch/arm/boot/compressed/head.S +50 -0
index c79db44ba128..3476e85c31e7 100644
@@ -1410,7 +1410,11 @@ memdump:       mov     r12, r0
 __hyp_reentry_vectors:
              W(b)    .                       @ reset
              W(b)    .                       @ undef
+#ifdef CONFIG_EFI_STUB
+             W(b)    __enter_kernel_from_hyp @ hvc from HYP
+#else
              W(b)    .                       @ svc
+#endif
              W(b)    .                       @ pabort
              W(b)    .                       @ dabort
              W(b)    __enter_kernel          @ hyp
@@ -1429,14 +1433,71 @@ __enter_kernel:
 reloc_code_end:
 
 #ifdef CONFIG_EFI_STUB
+__enter_kernel_from_hyp:
+             mrc     p15, 4, r0, c1, c0, 0   @ read HSCTLR
+             bic     r0, r0, #0x5            @ disable MMU and caches
+             mcr     p15, 4, r0, c1, c0, 0   @ write HSCTLR
+             isb
+             b       __enter_kernel
+
 ENTRY(efi_enter_kernel)
              mov     r4, r0                  @ preserve image base
              mov     r8, r1                  @ preserve DT pointer
 
+ ARM(                adrl    r0, call_cache_fn       )
+ THUMB(              adr     r0, call_cache_fn       )
+             adr     r1, 0f                  @ clean the region of code we
+             bl      cache_clean_flush       @ may run with the MMU off
+
+#ifdef CONFIG_ARM_VIRT_EXT
+             @
+             @ The EFI spec does not support booting on ARM in HYP mode,
+             @ since it mandates that the MMU and caches are on, with all
+             @ 32-bit addressable DRAM mapped 1:1 using short descriptors.
+             @
+             @ While the EDK2 reference implementation adheres to this,
+             @ U-Boot might decide to enter the EFI stub in HYP mode
+             @ anyway, with the MMU and caches either on or off.
+             @
+             mrs     r0, cpsr                @ get the current mode
+             msr     spsr_cxsf, r0           @ record boot mode
+             and     r0, r0, #MODE_MASK      @ are we running in HYP mode?
+             cmp     r0, #HYP_MODE
+             bne     .Lefi_svc
+
+             mrc     p15, 4, r1, c1, c0, 0   @ read HSCTLR
+             tst     r1, #0x1                @ MMU enabled at HYP?
+             beq     1f
+
+             @
+             @ When running in HYP mode with the caches on, we're better
+             @ off just carrying on using the cached 1:1 mapping that the
+             @ firmware provided. Set up the HYP vectors so HVC instructions
+             @ issued from HYP mode take us to the correct handler code. We
+             @ will disable the MMU before jumping to the kernel proper.
+             @
+             adr     r0, __hyp_reentry_vectors
+             mcr     p15, 4, r0, c12, c0, 0  @ set HYP vector base (HVBAR)
+             isb
+             b       .Lefi_hyp
+
+             @
+             @ When running in HYP mode with the caches off, we need to drop
Quoted 9 lines

arch/arm/boot/compressed/head.S __enter_kernel: · line 1491

Quoted 6 lines
arch/arm/boot/compressed/head.S +6 -0
+             @ into SVC mode now, and let the decompressor set up its cached
+             @ 1:1 mapping as usual.
+             @
+1:           mov     r9, r4                  @ preserve image base
+             bl      __hyp_stub_install      @ install HYP stub vectors
+             safe_svcmode_maskall    r1      @ drop to SVC mode

Are you returning to HYP mode somewhere?

Yes.

Quoted 2 lines

What is the effect on PSCI?

If you boot Linux in HYP then you cannot have PSCI in HYP as well. Linux will take ownership of HYP mode, and remove whatever was there. If you want to run PSCI at HYP, then the OS needs to boot in SVC mode.

Quoted 5 lines
Quoted 3 lines

The Allwinner/Sunxi boards must be booted in HYP mode to have PSCI according to https://linux-sunxi.org/PSCI

See above. If PSCI runs in HYP, the OS needs to run at SVC

Heinrich probably misunderstood this. That page is saying that for PSCI to be available to Linux, U-boot must be booting Linux in non-secure mode. That is because the PSCI implementation runs in secure monitor mode under Linux. Linux can boot in HYP or SVC as it so chooses.

Quoted 4 lines
Quoted 2 lines

Did you test that you still can reboot those boards?

No, I don't have such a board.

I'm not sure what reboot has to do with this, since the U-boot PSCI implementation for sunxi does not support reboot or power-off. It only supports CPU power on and off. The whole reason for this is to support SMP in a way that HYP mode can be used.

Hope this clears things up. I'm not familiar with the specifics of EFI nor do I use it in my setup.

Regards
ChenYu

Quoted 21 lines
Quoted 21 lines

Cc: Chen-Yu Tsai <[email protected]>
(maintainer ARM/Allwinner sunXi SoC support)

Best regards

Heinrich

Quoted 13 lines
arch/arm/boot/compressed/head.S +5 -0
+             orr     r4, r9, #1              @ restore image base and set LSB
+             b       .Lefi_hyp
+.Lefi_svc:
+#endif
              mrc     p15, 0, r0, c1, c0, 0   @ read SCTLR
              tst     r0, #0x1                @ MMU enabled?
              orreq   r4, r4, #1              @ set LSB if not
 
+.Lefi_hyp:
              mov     r0, r8                  @ DT start
              add     r1, r8, r2              @ DT end
              bl      cache_clean_flush
#5 lore raw

Re: [PATCH v2 1/2] efi/arm: decompressor: deal with HYP mode boot gracefully

In reply to Ard Biesheuvel
Quoted 164 lines · Ard Biesheuvel
Quoted 24 lines · Heinrich Schuchardt
Quoted 13 lines · Ard Biesheuvel

EFI on ARM only supports short descriptors, and given that it mandates that the MMU and caches are on, it is implied that booting in HYP mode is not supported.

However, implementations of EFI exist (i.e., U-Boot) that ignore this requirement, which is not entirely unreasonable, given that it makes HYP mode inaccessible to the operating system.

So let's make sure that we can deal with this condition gracefully. We already tolerate booting the EFI stub with the caches off (even though this violates the EFI spec as well), and so we should deal with HYP mode boot with MMU and caches either on or off.

Quoted 8 lines

- When the MMU and caches are on, we can ignore the HYP stub altogether, since we can carry on executing at HYP. We do need to ensure that we disable the MMU at HYP before entering the kernel proper.

- When the MMU and caches are off, we have to drop to SVC mode so that we can set up the page tables using short descriptors. In this case, we need to install the HYP stub as usual, so that we can return to HYP mode before handing over to the kernel proper.

Quoted 5 lines

To me it is still unclear why you need this patch. Please, describe the problem this patch fixes.

Is there any device that you cannot boot without the patch?

The code as it is today is broken, and if it works, it only does so by accident.

(There were some recent changes but the old code is broken in a similar way)

When we enter via the stub, we used to call cache_off() to disable the caches before calling the decompressor entry point. However, that disables the SVC mode caches, not the HYP mode caches, and so if we enter via EFI at HYP, we will call __hyp_stub_install() with the HYP mod MMU and caches enabled, which is explicitly forbidden (see hyp-stub.S)

With the recent change, the EFI entry code doesn't call cache_off() anymore, but that does not remove the problem, it just moves it to the point where we hand over to the kernel proper.

The problem is really on the u-boot side, and so we either have to follow the letter of the EFI spec and ban the practice of booting in HYP mode or with the caches off, or we work around this like I do in this patch. Doing nothing is not really an option.

If we want EBBR and U-boot as EFI firmware to succeed, we should really fix issues such as these, and not pretend everything is fine when we know it is broken but happens to work on the few boards that we test. This is the reason we have architecture and firmware specs in the first place, and it is really quite unfortunate that we did not catch these u-boot issues before.

As I said, I think booting at HYP can be tolerated, since the OS loses access to it otherwise (and maybe we should even update the EFI spec to allow this). But fiddling with the caches like we do should really be avoided (and the GRUB hack really needs to go as well)

Quoted 76 lines
Quoted 76 lines
---
 arch/arm/boot/compressed/head.S | 61 ++++++++++++++++++++
 1 file changed, 61 insertions(+)
arch/arm/boot/compressed/head.S +50 -0
index c79db44ba128..3476e85c31e7 100644
@@ -1410,7 +1410,11 @@ memdump:       mov     r12, r0
 __hyp_reentry_vectors:
              W(b)    .                       @ reset
              W(b)    .                       @ undef
+#ifdef CONFIG_EFI_STUB
+             W(b)    __enter_kernel_from_hyp @ hvc from HYP
+#else
              W(b)    .                       @ svc
+#endif
              W(b)    .                       @ pabort
              W(b)    .                       @ dabort
              W(b)    __enter_kernel          @ hyp
@@ -1429,14 +1433,71 @@ __enter_kernel:
 reloc_code_end:
 
 #ifdef CONFIG_EFI_STUB
+__enter_kernel_from_hyp:
+             mrc     p15, 4, r0, c1, c0, 0   @ read HSCTLR
+             bic     r0, r0, #0x5            @ disable MMU and caches
+             mcr     p15, 4, r0, c1, c0, 0   @ write HSCTLR
+             isb
+             b       __enter_kernel
+
 ENTRY(efi_enter_kernel)
              mov     r4, r0                  @ preserve image base
              mov     r8, r1                  @ preserve DT pointer
 
+ ARM(                adrl    r0, call_cache_fn       )
+ THUMB(              adr     r0, call_cache_fn       )
+             adr     r1, 0f                  @ clean the region of code we
+             bl      cache_clean_flush       @ may run with the MMU off
+
+#ifdef CONFIG_ARM_VIRT_EXT
+             @
+             @ The EFI spec does not support booting on ARM in HYP mode,
+             @ since it mandates that the MMU and caches are on, with all
+             @ 32-bit addressable DRAM mapped 1:1 using short descriptors.
+             @
+             @ While the EDK2 reference implementation adheres to this,
+             @ U-Boot might decide to enter the EFI stub in HYP mode
+             @ anyway, with the MMU and caches either on or off.
+             @
+             mrs     r0, cpsr                @ get the current mode
+             msr     spsr_cxsf, r0           @ record boot mode
+             and     r0, r0, #MODE_MASK      @ are we running in HYP mode?
+             cmp     r0, #HYP_MODE
+             bne     .Lefi_svc
+
+             mrc     p15, 4, r1, c1, c0, 0   @ read HSCTLR
+             tst     r1, #0x1                @ MMU enabled at HYP?
+             beq     1f
+
+             @
+             @ When running in HYP mode with the caches on, we're better
+             @ off just carrying on using the cached 1:1 mapping that the
+             @ firmware provided. Set up the HYP vectors so HVC instructions
+             @ issued from HYP mode take us to the correct handler code. We
+             @ will disable the MMU before jumping to the kernel proper.
+             @
+             adr     r0, __hyp_reentry_vectors
+             mcr     p15, 4, r0, c12, c0, 0  @ set HYP vector base (HVBAR)
+             isb
+             b       .Lefi_hyp
+
+             @
+             @ When running in HYP mode with the caches off, we need to drop
Quoted 9 lines

arch/arm/boot/compressed/head.S __enter_kernel: · line 1491

Quoted 6 lines
arch/arm/boot/compressed/head.S +6 -0
+             @ into SVC mode now, and let the decompressor set up its cached
+             @ 1:1 mapping as usual.
+             @
+1:           mov     r9, r4                  @ preserve image base
+             bl      __hyp_stub_install      @ install HYP stub vectors
+             safe_svcmode_maskall    r1      @ drop to SVC mode

Are you returning to HYP mode somewhere?

Yes.

Quoted 2 lines

What is the effect on PSCI?

If you boot Linux in HYP then you cannot have PSCI in HYP as well. Linux will take ownership of HYP mode, and remove whatever was there. If you want to run PSCI at HYP, then the OS needs to boot in SVC mode.

Quoted 3 lines

The Allwinner/Sunxi boards must be booted in HYP mode to have PSCI according to https://linux-sunxi.org/PSCI

Quoted 6 lines

See above. If PSCI runs in HYP, the OS needs to run at SVC

Quoted 2 lines

Did you test that you still can reboot those boards?

No, I don't have such a board.

Hello Ard,

thanks for supplying a branch for testing:
https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=efi-arm-hyp-mode

The OrangePi PC boots fine with this branch. PSCI is enabled. Rebooting the system works fine. See log below.

With the patch 2/2 you add an output line for the exceptions level and the MMU status. Above you state "We already tolerate booting the EFI stub with the caches off." This relates to a workaround in U-Boot accomodating old GRUB versions (CONFIG_EFI_GRUB_ARM32_WORKAROUND=y).

Would a further diagnostic line showing if D-cache and I-cache is enabled make sense?

Loading Linux 5.7.0-armmp-lpae+ ...
Loading initial ramdisk ...
EFI stub: Running in HYP mode with MMU enabled
EFI stub: Booting Linux Kernel...
EFI stub: ERROR: Could not determine UEFI Secure Boot status. EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map... EHCI failed to shut down host controller.

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.7.0-armmp-lpae+ (user@node)
(arm-linux-gnueabihf-gcc (Debian 9.3.0-13) 9.3.0, GNU ld (GNU Binutils
for Debian) 2.34) #10 SMP Mon Jun 8 03:44:37 CEST 2020
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7),
cr=30c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing
instruction cache
[    0.000000] OF: fdt: Machine model: Xunlong Orange Pi PC
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] efi: EFI v2.80 by Das U-Boot
[    0.000000] efi: RTPROP=0x78f30040 SMBIOS=0x78f2a000
MEMRESERVE=0x6a1fa040
[    0.000000] cma: Reserved 16 MiB at 0x000000007f000000
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000040000000-0x000000006fffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x0000000070000000-0x000000007fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000040000000-0x0000000078f07fff]
[    0.000000]   node   0: [mem 0x0000000078f08000-0x0000000078f09fff]
[    0.000000]   node   0: [mem 0x0000000078f0a000-0x0000000078f24fff]
[    0.000000]   node   0: [mem 0x0000000078f25000-0x0000000078f28fff]
[    0.000000]   node   0: [mem 0x0000000078f29000-0x0000000078f29fff]
[    0.000000]   node   0: [mem 0x0000000078f2a000-0x0000000078f2afff]
[    0.000000]   node   0: [mem 0x0000000078f2b000-0x0000000078f2cfff]
[    0.000000]   node   0: [mem 0x0000000078f2d000-0x0000000078f2dfff]
[    0.000000]   node   0: [mem 0x0000000078f2e000-0x0000000078f2ffff]
[    0.000000]   node   0: [mem 0x0000000078f30000-0x0000000078f32fff]
[    0.000000]   node   0: [mem 0x0000000078f33000-0x0000000078f33fff]
[    0.000000]   node   0: [mem 0x0000000078f34000-0x0000000078f34fff]
[    0.000000]   node   0: [mem 0x0000000078f35000-0x0000000078f35fff]
[    0.000000]   node   0: [mem 0x0000000078f36000-0x0000000078f36fff]
[    0.000000]   node   0: [mem 0x0000000078f37000-0x0000000078f38fff]
[    0.000000]   node   0: [mem 0x0000000078f39000-0x0000000078f3efff]
[    0.000000]   node   0: [mem 0x0000000078f3f000-0x000000007df65fff]
[    0.000000]   node   0: [mem 0x000000007df66000-0x000000007df66fff]
[    0.000000]   node   0: [mem 0x000000007df67000-0x000000007dfb9fff]
[    0.000000]   node   0: [mem 0x000000007dfba000-0x000000007dfbcfff]
[    0.000000]   node   0: [mem 0x000000007dfbd000-0x000000007fffffff]
[    0.000000] Initmem setup node 0 [mem
0x0000000040000000-0x000000007fffffff]
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: Using PSCI v0.1 Function IDs from DT

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

author maintainer
#6 lore raw

Re: [PATCH v2 1/2] efi/arm: decompressor: deal with HYP mode boot gracefully

Quoted 173 lines · Heinrich Schuchardt
Quoted 164 lines · Ard Biesheuvel
Quoted 24 lines · Heinrich Schuchardt
Quoted 13 lines · Ard Biesheuvel

EFI on ARM only supports short descriptors, and given that it mandates that the MMU and caches are on, it is implied that booting in HYP mode is not supported.

However, implementations of EFI exist (i.e., U-Boot) that ignore this requirement, which is not entirely unreasonable, given that it makes HYP mode inaccessible to the operating system.

So let's make sure that we can deal with this condition gracefully. We already tolerate booting the EFI stub with the caches off (even though this violates the EFI spec as well), and so we should deal with HYP mode boot with MMU and caches either on or off.

Quoted 8 lines

- When the MMU and caches are on, we can ignore the HYP stub altogether, since we can carry on executing at HYP. We do need to ensure that we disable the MMU at HYP before entering the kernel proper.

- When the MMU and caches are off, we have to drop to SVC mode so that we can set up the page tables using short descriptors. In this case, we need to install the HYP stub as usual, so that we can return to HYP mode before handing over to the kernel proper.

Quoted 5 lines

To me it is still unclear why you need this patch. Please, describe the problem this patch fixes.

Is there any device that you cannot boot without the patch?

The code as it is today is broken, and if it works, it only does so by accident.

(There were some recent changes but the old code is broken in a similar way)

When we enter via the stub, we used to call cache_off() to disable the caches before calling the decompressor entry point. However, that disables the SVC mode caches, not the HYP mode caches, and so if we enter via EFI at HYP, we will call __hyp_stub_install() with the HYP mod MMU and caches enabled, which is explicitly forbidden (see hyp-stub.S)

With the recent change, the EFI entry code doesn't call cache_off() anymore, but that does not remove the problem, it just moves it to the point where we hand over to the kernel proper.

The problem is really on the u-boot side, and so we either have to follow the letter of the EFI spec and ban the practice of booting in HYP mode or with the caches off, or we work around this like I do in this patch. Doing nothing is not really an option.

If we want EBBR and U-boot as EFI firmware to succeed, we should really fix issues such as these, and not pretend everything is fine when we know it is broken but happens to work on the few boards that we test. This is the reason we have architecture and firmware specs in the first place, and it is really quite unfortunate that we did not catch these u-boot issues before.

As I said, I think booting at HYP can be tolerated, since the OS loses access to it otherwise (and maybe we should even update the EFI spec to allow this). But fiddling with the caches like we do should really be avoided (and the GRUB hack really needs to go as well)

Quoted 76 lines
Quoted 76 lines
---
 arch/arm/boot/compressed/head.S | 61 ++++++++++++++++++++
 1 file changed, 61 insertions(+)
arch/arm/boot/compressed/head.S +50 -0
index c79db44ba128..3476e85c31e7 100644
@@ -1410,7 +1410,11 @@ memdump:       mov     r12, r0
 __hyp_reentry_vectors:
              W(b)    .                       @ reset
              W(b)    .                       @ undef
+#ifdef CONFIG_EFI_STUB
+             W(b)    __enter_kernel_from_hyp @ hvc from HYP
+#else
              W(b)    .                       @ svc
+#endif
              W(b)    .                       @ pabort
              W(b)    .                       @ dabort
              W(b)    __enter_kernel          @ hyp
@@ -1429,14 +1433,71 @@ __enter_kernel:
 reloc_code_end:
 
 #ifdef CONFIG_EFI_STUB
+__enter_kernel_from_hyp:
+             mrc     p15, 4, r0, c1, c0, 0   @ read HSCTLR
+             bic     r0, r0, #0x5            @ disable MMU and caches
+             mcr     p15, 4, r0, c1, c0, 0   @ write HSCTLR
+             isb
+             b       __enter_kernel
+
 ENTRY(efi_enter_kernel)
              mov     r4, r0                  @ preserve image base
              mov     r8, r1                  @ preserve DT pointer
 
+ ARM(                adrl    r0, call_cache_fn       )
+ THUMB(              adr     r0, call_cache_fn       )
+             adr     r1, 0f                  @ clean the region of code we
+             bl      cache_clean_flush       @ may run with the MMU off
+
+#ifdef CONFIG_ARM_VIRT_EXT
+             @
+             @ The EFI spec does not support booting on ARM in HYP mode,
+             @ since it mandates that the MMU and caches are on, with all
+             @ 32-bit addressable DRAM mapped 1:1 using short descriptors.
+             @
+             @ While the EDK2 reference implementation adheres to this,
+             @ U-Boot might decide to enter the EFI stub in HYP mode
+             @ anyway, with the MMU and caches either on or off.
+             @
+             mrs     r0, cpsr                @ get the current mode
+             msr     spsr_cxsf, r0           @ record boot mode
+             and     r0, r0, #MODE_MASK      @ are we running in HYP mode?
+             cmp     r0, #HYP_MODE
+             bne     .Lefi_svc
+
+             mrc     p15, 4, r1, c1, c0, 0   @ read HSCTLR
+             tst     r1, #0x1                @ MMU enabled at HYP?
+             beq     1f
+
+             @
+             @ When running in HYP mode with the caches on, we're better
+             @ off just carrying on using the cached 1:1 mapping that the
+             @ firmware provided. Set up the HYP vectors so HVC instructions
+             @ issued from HYP mode take us to the correct handler code. We
+             @ will disable the MMU before jumping to the kernel proper.
+             @
+             adr     r0, __hyp_reentry_vectors
+             mcr     p15, 4, r0, c12, c0, 0  @ set HYP vector base (HVBAR)
+             isb
+             b       .Lefi_hyp
+
+             @
+             @ When running in HYP mode with the caches off, we need to drop
Quoted 9 lines

arch/arm/boot/compressed/head.S __enter_kernel: · line 1491

Quoted 6 lines
arch/arm/boot/compressed/head.S +6 -0
+             @ into SVC mode now, and let the decompressor set up its cached
+             @ 1:1 mapping as usual.
+             @
+1:           mov     r9, r4                  @ preserve image base
+             bl      __hyp_stub_install      @ install HYP stub vectors
+             safe_svcmode_maskall    r1      @ drop to SVC mode

Are you returning to HYP mode somewhere?

Yes.

Quoted 2 lines

What is the effect on PSCI?

If you boot Linux in HYP then you cannot have PSCI in HYP as well. Linux will take ownership of HYP mode, and remove whatever was there. If you want to run PSCI at HYP, then the OS needs to boot in SVC mode.

Quoted 3 lines

The Allwinner/Sunxi boards must be booted in HYP mode to have PSCI according to https://linux-sunxi.org/PSCI

Quoted 6 lines

See above. If PSCI runs in HYP, the OS needs to run at SVC

Quoted 2 lines

Did you test that you still can reboot those boards?

No, I don't have such a board.

Quoted 8 lines

Hello Ard,

thanks for supplying a branch for testing:
https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=efi-arm-hyp-mode

The OrangePi PC boots fine with this branch. PSCI is enabled. Rebooting the system works fine. See log below.

Thanks for testing!

Quoted 3 lines

With the patch 2/2 you add an output line for the exceptions level and the MMU status. Above you state "We already tolerate booting the EFI stub with the caches off."

Indeed.

Quoted 3 lines

This relates to a workaround in U-Boot
accomodating old GRUB versions (CONFIG_EFI_GRUB_ARM32_WORKAROUND=y).

Not entirely. The GRUB hack disables the caches during ExitBootServices() while the rest of the EFI stub runs with the caches on. This is actually worse than simply never enabling the caches in the first place. But we currently deal with both cases, by running the decompressor's cache_on() routine in SVC mode.

Quoted 3 lines

Would a further diagnostic line showing if D-cache and I-cache is enabled make sense?

Enabling the I-cache can be done independently, so it does not really matter (you cannot have dirty cachelines in the I-cache, and instruction fetches are special memory accesses that can be identified as cacheable accesses to normal memory regardless of whether the MMU is enabled and whether a mapping exists)

Enabling the D-cache only has an effect if you enable the MMU, as otherwise, all data accesses will be implicitly qualified as non-cacheable device accesses. Since EFI requires a 1:1 mapping, the only reason for enabling the MMU in the first place is the ability to enable the D-cache.

So in summary, the M bit is the interesting bit.

Quoted 2 lines

Thanks.

Quoted 46 lines

Loading Linux 5.7.0-armmp-lpae+ ...
Loading initial ramdisk ...
EFI stub: Running in HYP mode with MMU enabled
EFI stub: Booting Linux Kernel...
EFI stub: ERROR: Could not determine UEFI Secure Boot status. EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map... EHCI failed to shut down host controller.

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.7.0-armmp-lpae+ (user@node)
(arm-linux-gnueabihf-gcc (Debian 9.3.0-13) 9.3.0, GNU ld (GNU Binutils
for Debian) 2.34) #10 SMP Mon Jun 8 03:44:37 CEST 2020
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7),
cr=30c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing
instruction cache
[    0.000000] OF: fdt: Machine model: Xunlong Orange Pi PC
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] efi: EFI v2.80 by Das U-Boot
[    0.000000] efi: RTPROP=0x78f30040 SMBIOS=0x78f2a000
MEMRESERVE=0x6a1fa040
[    0.000000] cma: Reserved 16 MiB at 0x000000007f000000
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000040000000-0x000000006fffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x0000000070000000-0x000000007fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000040000000-0x0000000078f07fff]
[    0.000000]   node   0: [mem 0x0000000078f08000-0x0000000078f09fff]
[    0.000000]   node   0: [mem 0x0000000078f0a000-0x0000000078f24fff]
[    0.000000]   node   0: [mem 0x0000000078f25000-0x0000000078f28fff]
[    0.000000]   node   0: [mem 0x0000000078f29000-0x0000000078f29fff]
[    0.000000]   node   0: [mem 0x0000000078f2a000-0x0000000078f2afff]
[    0.000000]   node   0: [mem 0x0000000078f2b000-0x0000000078f2cfff]
[    0.000000]   node   0: [mem 0x0000000078f2d000-0x0000000078f2dfff]
[    0.000000]   node   0: [mem 0x0000000078f2e000-0x0000000078f2ffff]
[    0.000000]   node   0: [mem 0x0000000078f30000-0x0000000078f32fff]
[    0.000000]   node   0: [mem 0x0000000078f33000-0x0000000078f33fff]
[    0.000000]   node   0: [mem 0x0000000078f34000-0x0000000078f34fff]
[    0.000000]   node   0: [mem 0x0000000078f35000-0x0000000078f35fff]
[    0.000000]   node   0: [mem 0x0000000078f36000-0x0000000078f36fff]
[    0.000000]   node   0: [mem 0x0000000078f37000-0x0000000078f38fff]
[    0.000000]   node   0: [mem 0x0000000078f39000-0x0000000078f3efff]
[    0.000000]   node   0: [mem 0x0000000078f3f000-0x000000007df65fff]
Quoted 8 lines
[    0.000000]   node   0: [mem 0x000000007df66000-0x000000007df66fff]
[    0.000000]   node   0: [mem 0x000000007df67000-0x000000007dfb9fff]
[    0.000000]   node   0: [mem 0x000000007dfba000-0x000000007dfbcfff]
[    0.000000]   node   0: [mem 0x000000007dfbd000-0x000000007fffffff]
[    0.000000] Initmem setup node 0 [mem
0x0000000040000000-0x000000007fffffff]
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: Using PSCI v0.1 Function IDs from DT

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

author maintainer
#7 lore raw

Re: [PATCH v2 1/2] efi/arm: decompressor: deal with HYP mode boot gracefully

In reply to Heinrich Schuchardt
Quoted 26 lines · Heinrich Schuchardt
Quoted 24 lines · Ard Biesheuvel
Quoted 23 lines · Heinrich Schuchardt
Quoted 21 lines · Ard Biesheuvel

EFI on ARM only supports short descriptors, and given that it mandates that the MMU and caches are on, it is implied that booting in HYP mode is not supported.

However, implementations of EFI exist (i.e., U-Boot) that ignore this requirement, which is not entirely unreasonable, given that it makes HYP mode inaccessible to the operating system.

So let's make sure that we can deal with this condition gracefully. We already tolerate booting the EFI stub with the caches off (even though this violates the EFI spec as well), and so we should deal with HYP mode boot with MMU and caches either on or off.

- When the MMU and caches are on, we can ignore the HYP stub altogether, since we can carry on executing at HYP. We do need to ensure that we disable the MMU at HYP before entering the kernel proper.

- When the MMU and caches are off, we have to drop to SVC mode so that we can set up the page tables using short descriptors. In this case, we need to install the HYP stub as usual, so that we can return to HYP mode before handing over to the kernel proper.

...

Quoted 14 lines

Hello Ard,

thanks for supplying a branch for testing:
https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=efi-arm-hyp-mode

The OrangePi PC boots fine with this branch. PSCI is enabled. Rebooting the system works fine. See log below.

With the patch 2/2 you add an output line for the exceptions level and the MMU status. Above you state "We already tolerate booting the EFI stub with the caches off." This relates to a workaround in U-Boot accomodating old GRUB versions (CONFIG_EFI_GRUB_ARM32_WORKAROUND=y).

Quoted 8 lines

Would a further diagnostic line showing if D-cache and I-cache is enabled make sense?

Loading Linux 5.7.0-armmp-lpae+ ...
Loading initial ramdisk ...
EFI stub: Running in HYP mode with MMU enabled

BTW is this with or without the GRUB hack?

Quoted 51 lines

EFI stub: Booting Linux Kernel...
EFI stub: ERROR: Could not determine UEFI Secure Boot status. EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map... EHCI failed to shut down host controller.

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.7.0-armmp-lpae+ (user@node)
(arm-linux-gnueabihf-gcc (Debian 9.3.0-13) 9.3.0, GNU ld (GNU Binutils
for Debian) 2.34) #10 SMP Mon Jun 8 03:44:37 CEST 2020
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7),
cr=30c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing
instruction cache
[    0.000000] OF: fdt: Machine model: Xunlong Orange Pi PC
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] efi: EFI v2.80 by Das U-Boot
[    0.000000] efi: RTPROP=0x78f30040 SMBIOS=0x78f2a000
MEMRESERVE=0x6a1fa040
[    0.000000] cma: Reserved 16 MiB at 0x000000007f000000
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000040000000-0x000000006fffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x0000000070000000-0x000000007fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000040000000-0x0000000078f07fff]
[    0.000000]   node   0: [mem 0x0000000078f08000-0x0000000078f09fff]
[    0.000000]   node   0: [mem 0x0000000078f0a000-0x0000000078f24fff]
[    0.000000]   node   0: [mem 0x0000000078f25000-0x0000000078f28fff]
[    0.000000]   node   0: [mem 0x0000000078f29000-0x0000000078f29fff]
[    0.000000]   node   0: [mem 0x0000000078f2a000-0x0000000078f2afff]
[    0.000000]   node   0: [mem 0x0000000078f2b000-0x0000000078f2cfff]
[    0.000000]   node   0: [mem 0x0000000078f2d000-0x0000000078f2dfff]
[    0.000000]   node   0: [mem 0x0000000078f2e000-0x0000000078f2ffff]
[    0.000000]   node   0: [mem 0x0000000078f30000-0x0000000078f32fff]
[    0.000000]   node   0: [mem 0x0000000078f33000-0x0000000078f33fff]
[    0.000000]   node   0: [mem 0x0000000078f34000-0x0000000078f34fff]
[    0.000000]   node   0: [mem 0x0000000078f35000-0x0000000078f35fff]
[    0.000000]   node   0: [mem 0x0000000078f36000-0x0000000078f36fff]
[    0.000000]   node   0: [mem 0x0000000078f37000-0x0000000078f38fff]
[    0.000000]   node   0: [mem 0x0000000078f39000-0x0000000078f3efff]
[    0.000000]   node   0: [mem 0x0000000078f3f000-0x000000007df65fff]
[    0.000000]   node   0: [mem 0x000000007df66000-0x000000007df66fff]
[    0.000000]   node   0: [mem 0x000000007df67000-0x000000007dfb9fff]
[    0.000000]   node   0: [mem 0x000000007dfba000-0x000000007dfbcfff]
[    0.000000]   node   0: [mem 0x000000007dfbd000-0x000000007fffffff]
[    0.000000] Initmem setup node 0 [mem
0x0000000040000000-0x000000007fffffff]
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: Using PSCI v0.1 Function IDs from DT
author maintainer
#8 lore raw

Re: [PATCH v2 1/2] efi/arm: decompressor: deal with HYP mode boot gracefully

Quoted 46 lines · Ard Biesheuvel
Quoted 26 lines · Heinrich Schuchardt
Quoted 24 lines · Ard Biesheuvel
Quoted 23 lines · Heinrich Schuchardt
Quoted 21 lines · Ard Biesheuvel

EFI on ARM only supports short descriptors, and given that it mandates that the MMU and caches are on, it is implied that booting in HYP mode is not supported.

However, implementations of EFI exist (i.e., U-Boot) that ignore this requirement, which is not entirely unreasonable, given that it makes HYP mode inaccessible to the operating system.

So let's make sure that we can deal with this condition gracefully. We already tolerate booting the EFI stub with the caches off (even though this violates the EFI spec as well), and so we should deal with HYP mode boot with MMU and caches either on or off.

- When the MMU and caches are on, we can ignore the HYP stub altogether, since we can carry on executing at HYP. We do need to ensure that we disable the MMU at HYP before entering the kernel proper.

- When the MMU and caches are off, we have to drop to SVC mode so that we can set up the page tables using short descriptors. In this case, we need to install the HYP stub as usual, so that we can return to HYP mode before handing over to the kernel proper.

...

Quoted 17 lines

Hello Ard,

thanks for supplying a branch for testing:
https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=efi-arm-hyp-mode

The OrangePi PC boots fine with this branch. PSCI is enabled. Rebooting the system works fine. See log below.

With the patch 2/2 you add an output line for the exceptions level and the MMU status. Above you state "We already tolerate booting the EFI stub with the caches off." This relates to a workaround in U-Boot accomodating old GRUB versions (CONFIG_EFI_GRUB_ARM32_WORKAROUND=y).

Would a further diagnostic line showing if D-cache and I-cache is enabled make sense?

Quoted 8 lines
Quoted 5 lines

Loading Linux 5.7.0-armmp-lpae+ ...
Loading initial ramdisk ...
EFI stub: Running in HYP mode with MMU enabled

BTW is this with or without the GRUB hack?

I've given this a spin myself on a RPi4 running 32-bit U-boot, and everything works as expected, both with and without the GRUB hack enabled.

Russell, given that this only affects code inside #ifdef CONFIG_EFI_STUB, do you have any objections to me taking this as a fix via the EFI tree? I have a set of fixes I need to queue up and send out anyway, and I intend to do so early next week.

#9 lore raw

Re: [PATCH v2 1/2] efi/arm: decompressor: deal with HYP mode boot gracefully

Quoted 8 lines · Ard Biesheuvel

I've given this a spin myself on a RPi4 running 32-bit U-boot, and everything works as expected, both with and without the GRUB hack enabled.

Russell, given that this only affects code inside #ifdef CONFIG_EFI_STUB, do you have any objections to me taking this as a fix via the EFI tree? I have a set of fixes I need to queue up and send out anyway, and I intend to do so early next week.

Please don't, I'll be basing my branches off -rc1 (as normal), and if you then submit this as a fix through the EFI tree for merging after rc1, and then send me further EFI work to go through the ARM tree, we'll end up in exactly the same merge issues as we did prior to this merge window.

Thanks.

Signature · 7 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
author maintainer
#10 lore raw

Re: [PATCH v2 1/2] efi/arm: decompressor: deal with HYP mode boot gracefully

Quoted 11 lines · Russell King - ARM Linux admin
Quoted 8 lines · Ard Biesheuvel

I've given this a spin myself on a RPi4 running 32-bit U-boot, and everything works as expected, both with and without the GRUB hack enabled.

Russell, given that this only affects code inside #ifdef CONFIG_EFI_STUB, do you have any objections to me taking this as a fix via the EFI tree? I have a set of fixes I need to queue up and send out anyway, and I intend to do so early next week.

Quoted 6 lines

Please don't, I'll be basing my branches off -rc1 (as normal), and if you then submit this as a fix through the EFI tree for merging after rc1, and then send me further EFI work to go through the ARM tree, we'll end up in exactly the same merge issues as we did prior to this merge window.

Fair enough. What do you suggest instead? Shall I drop this into the patch system?

#11 lore raw

Re: [PATCH v2 1/2] efi/arm: decompressor: deal with HYP mode boot gracefully

Quoted 20 lines · Ard Biesheuvel
Quoted 11 lines · Russell King - ARM Linux admin
Quoted 8 lines · Ard Biesheuvel

I've given this a spin myself on a RPi4 running 32-bit U-boot, and everything works as expected, both with and without the GRUB hack enabled.

Russell, given that this only affects code inside #ifdef CONFIG_EFI_STUB, do you have any objections to me taking this as a fix via the EFI tree? I have a set of fixes I need to queue up and send out anyway, and I intend to do so early next week.

Quoted 6 lines

Please don't, I'll be basing my branches off -rc1 (as normal), and if you then submit this as a fix through the EFI tree for merging after rc1, and then send me further EFI work to go through the ARM tree, we'll end up in exactly the same merge issues as we did prior to this merge window.

Quoted 2 lines

Fair enough. What do you suggest instead? Shall I drop this into the patch system?

Is it a regression? If so, sending it prior to -rc1 is permissible. If not, please drop it in the patch system.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
author maintainer
#12 lore raw

Re: [PATCH v2 1/2] efi/arm: decompressor: deal with HYP mode boot gracefully

Quoted 22 lines · Russell King - ARM Linux admin
Quoted 20 lines · Ard Biesheuvel
Quoted 17 lines · Russell King - ARM Linux admin
Quoted 8 lines · Ard Biesheuvel

I've given this a spin myself on a RPi4 running 32-bit U-boot, and everything works as expected, both with and without the GRUB hack enabled.

Russell, given that this only affects code inside #ifdef CONFIG_EFI_STUB, do you have any objections to me taking this as a fix via the EFI tree? I have a set of fixes I need to queue up and send out anyway, and I intend to do so early next week.

Please don't, I'll be basing my branches off -rc1 (as normal), and if you then submit this as a fix through the EFI tree for merging after rc1, and then send me further EFI work to go through the ARM tree, we'll end up in exactly the same merge issues as we did prior to this merge window.

Quoted 6 lines
Quoted 2 lines

Fair enough. What do you suggest instead? Shall I drop this into the patch system?

Is it a regression? If so, sending it prior to -rc1 is permissible. If not, please drop it in the patch system.

If you boot via the EFI stub in HYP mode with the caches off (or with U-boot's GRUB hack enabled which fiddles with the caches halfway through), it appears that you cannot boot current mainline. This is an oversight on my part - the EFI spec does not permit doing either of those things, and while EDK2 behaves in this regard, u-boot can be configured in various different non-conforming ways. (Note that v5.7 and before will leave the MMU and caches enabled at HYP upon entering the kernel proper after booting via EFI, so this is not something that was 100% correct before, but at least it booted most of the time)

So this is a regression, but since the EFI tree goes through -tip, I won't be able to get this fix in before -rc1 is released. Therefore, I will be dropping this into the patch system in any case, and leave it up to you to decide when it gets sent onwards.