Allow the kernel to be booted in HYP mode

From: Dave Martin <[email protected]>

This patch does two things:

* Ensure that asynchronous aborts are masked at kernel entry. The bootloader should be masking these anyway, but this reduces the damage window just in case it doesn't.

* Enter svc mode via exception return to ensure that CPU state is properly serialised. This does not matter when switching from an ordinary privileged mode ("PL1" modes in ARMv7-AR rev C parlance), but it potentially does matter when switching from a another privileged mode such as hyp mode.

This should allow the kernel to boot safely either from svc mode or hyp mode, even if no support for use of the ARM Virtualization Extensions is built into the kernel.

Signed-off-by Dave Martin <[email protected]>

Signed-off-by Marc Zyngier <[email protected]>

arch/arm/kernel/head.S +0 -1

@@ -83,8 +83,12 @@ ENTRY(stext)
  THUMB(	.thumb			)	@ switch to Thumb now.
  THUMB(1:			)
 
-	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode

Just bisected this down in linux-next for breaking booting of my omap2420 ARMv6 based n8x0..

Quoted 7 lines

arch/arm/kernel/head.S +0 -1

@@ -83,8 +83,12 @@ ENTRY(stext)
  THUMB(	.thumb			)	@ switch to Thumb now.
  THUMB(1:			)
 
-	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode

arch/arm/kernel/head.S ENTRY(stext) · line 94

Quoted 10 lines

arch/arm/kernel/head.S +6 -1

-						@ and irqs disabled
+#ifdef CONFIG_ARM_VIRT_EXT
+	bl	__hyp_stub_install
+#endif
+	@ ensure svc mode and all interrupts masked
+	safe_svcmode_maskall r9
+
 	mrc	p15, 0, r9, c0, c0		@ get processor id
 	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
 	movs	r10, r5				@ invalid processor (r5=0)?

..and looks like undoing this part fixes it. Any ideas?

I quickly tried disabling ARCH_OMAP3 and ARCH_OMAP4 so it's ARMv6 but that does not help.

Quoted 23 lines · Tony Lindgren

Just bisected this down in linux-next for breaking booting of my omap2420 ARMv6 based n8x0..

Quoted 7 lines

arch/arm/kernel/head.S +0 -1

@@ -83,8 +83,12 @@ ENTRY(stext)
  THUMB(	.thumb			)	@ switch to Thumb now.
  THUMB(1:			)
 
-	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode

arch/arm/kernel/head.S ENTRY(stext) · line 94

Quoted 10 lines

arch/arm/kernel/head.S +6 -1

-						@ and irqs disabled
+#ifdef CONFIG_ARM_VIRT_EXT
+	bl	__hyp_stub_install
+#endif
+	@ ensure svc mode and all interrupts masked
+	safe_svcmode_maskall r9
+
 	mrc	p15, 0, r9, c0, c0		@ get processor id
 	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
 	movs	r10, r5				@ invalid processor (r5=0)?

..and looks like undoing this part fixes it. Any ideas?

Quoted 2 lines

I quickly tried disabling ARCH_OMAP3 and ARCH_OMAP4 so it's ARMv6 but that does not help.

You really should Cc me when you hit something like this. I was thinking about sending my tree (which contains these changes) this evening but if they're breaking stuff, I'd prefer to delay that stuff at least for a while.

Quoted 24 lines · Russell King - ARM Linux

Quoted 23 lines · Tony Lindgren

Just bisected this down in linux-next for breaking booting of my omap2420 ARMv6 based n8x0..

Quoted 7 lines

arch/arm/kernel/head.S +0 -1

@@ -83,8 +83,12 @@ ENTRY(stext)
  THUMB(	.thumb			)	@ switch to Thumb now.
  THUMB(1:			)
 
-	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode

arch/arm/kernel/head.S ENTRY(stext) · line 94

Quoted 10 lines

arch/arm/kernel/head.S +6 -1

-						@ and irqs disabled
+#ifdef CONFIG_ARM_VIRT_EXT
+	bl	__hyp_stub_install
+#endif
+	@ ensure svc mode and all interrupts masked
+	safe_svcmode_maskall r9
+
 	mrc	p15, 0, r9, c0, c0		@ get processor id
 	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
 	movs	r10, r5				@ invalid processor (r5=0)?

..and looks like undoing this part fixes it. Any ideas?

Quoted 2 lines

Quoted 2 lines

I quickly tried disabling ARCH_OMAP3 and ARCH_OMAP4 so it's ARMv6 but that does not help.

Quoted 25 lines · Tony Lindgren

Quoted 24 lines · Russell King - ARM Linux

Quoted 23 lines · Tony Lindgren

Just bisected this down in linux-next for breaking booting of my omap2420 ARMv6 based n8x0..

Quoted 7 lines

arch/arm/kernel/head.S +0 -1

@@ -83,8 +83,12 @@ ENTRY(stext)
  THUMB(	.thumb			)	@ switch to Thumb now.
  THUMB(1:			)
 
-	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode

arch/arm/kernel/head.S ENTRY(stext) · line 94

Quoted 10 lines

arch/arm/kernel/head.S +6 -1

-						@ and irqs disabled
+#ifdef CONFIG_ARM_VIRT_EXT
+	bl	__hyp_stub_install
+#endif
+	@ ensure svc mode and all interrupts masked
+	safe_svcmode_maskall r9
+
 	mrc	p15, 0, r9, c0, c0		@ get processor id
 	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
 	movs	r10, r5				@ invalid processor (r5=0)?

..and looks like undoing this part fixes it. Any ideas?

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

I quickly tried disabling ARCH_OMAP3 and ARCH_OMAP4 so it's ARMv6 but that does not help.

The same kernel boots on 2430sdp, which is the same ARMv6 core as 2430 if I remember correctly. So this hints that it has something to do with the bits set differently by the bootloader?

Quoted 26 lines · Tony Lindgren

Quoted 25 lines · Tony Lindgren

Quoted 24 lines · Russell King - ARM Linux

Quoted 23 lines · Tony Lindgren

Just bisected this down in linux-next for breaking booting of my omap2420 ARMv6 based n8x0..

Quoted 7 lines

arch/arm/kernel/head.S +0 -1

@@ -83,8 +83,12 @@ ENTRY(stext)
  THUMB(	.thumb			)	@ switch to Thumb now.
  THUMB(1:			)
 
-	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode

arch/arm/kernel/head.S ENTRY(stext) · line 94

Quoted 10 lines

arch/arm/kernel/head.S +6 -1

-						@ and irqs disabled
+#ifdef CONFIG_ARM_VIRT_EXT
+	bl	__hyp_stub_install
+#endif
+	@ ensure svc mode and all interrupts masked
+	safe_svcmode_maskall r9
+
 	mrc	p15, 0, r9, c0, c0		@ get processor id
 	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
 	movs	r10, r5				@ invalid processor (r5=0)?

..and looks like undoing this part fixes it. Any ideas?

Quoted 6 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

I quickly tried disabling ARCH_OMAP3 and ARCH_OMAP4 so it's ARMv6 but that does not help.

The same kernel boots on 2430sdp, which is the same ARMv6 core as 2430 if I remember correctly. So this hints that it has something to do with the bits set differently by the bootloader?

Possibly.

What if you apply this on top:

arch/arm/include/asm/assembler.h +0 -0

index 683a1e6b60..b276c26e19 100644

Hi,

Quoted 14 lines · Marc Zyngier

From: Dave Martin <[email protected]>

This patch does two things:

* Ensure that asynchronous aborts are masked at kernel entry. The bootloader should be masking these anyway, but this reduces the damage window just in case it doesn't.

* Enter svc mode via exception return to ensure that CPU state is properly serialised. This does not matter when switching from an ordinary privileged mode ("PL1" modes in ARMv7-AR rev C parlance), but it potentially does matter when switching from a another privileged mode such as hyp mode.

Quoted 6 lines

This should allow the kernel to boot safely either from svc mode or hyp mode, even if no support for use of the ARM Virtualization Extensions is built into the kernel.

Signed-off-by Dave Martin <[email protected]>

Signed-off-by Marc Zyngier <[email protected]>

Just bisected this down in linux-next for breaking booting of my omap2420 ARMv6 based n8x0..

Quoted 7 lines

arch/arm/kernel/head.S +0 -1

@@ -83,8 +83,12 @@ ENTRY(stext)
  THUMB(	.thumb			)	@ switch to Thumb now.
  THUMB(1:			)
 
-	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode

arch/arm/kernel/head.S ENTRY(stext) · line 94

Quoted 10 lines

arch/arm/kernel/head.S +6 -1

-						@ and irqs disabled
+#ifdef CONFIG_ARM_VIRT_EXT
+	bl	__hyp_stub_install
+#endif
+	@ ensure svc mode and all interrupts masked
+	safe_svcmode_maskall r9
+
 	mrc	p15, 0, r9, c0, c0		@ get processor id
 	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
 	movs	r10, r5				@ invalid processor (r5=0)?

..and looks like undoing this part fixes it. Any ideas?

I quickly tried disabling ARCH_OMAP3 and ARCH_OMAP4 so it's ARMv6 but that does not help.

Hi Tony,

Quoted 47 lines · Tony Lindgren

Hi,

Quoted 14 lines · Marc Zyngier

From: Dave Martin <[email protected]>

This patch does two things:

* Ensure that asynchronous aborts are masked at kernel entry. The bootloader should be masking these anyway, but this reduces the damage window just in case it doesn't.

* Enter svc mode via exception return to ensure that CPU state is properly serialised. This does not matter when switching from an ordinary privileged mode ("PL1" modes in ARMv7-AR rev C parlance), but it potentially does matter when switching from a another privileged mode such as hyp mode.

Quoted 6 lines

This should allow the kernel to boot safely either from svc mode or hyp mode, even if no support for use of the ARM Virtualization Extensions is built into the kernel.

Signed-off-by Dave Martin <[email protected]>

Signed-off-by Marc Zyngier <[email protected]>

Just bisected this down in linux-next for breaking booting of my omap2420 ARMv6 based n8x0..

Quoted 7 lines

arch/arm/kernel/head.S +0 -1

@@ -83,8 +83,12 @@ ENTRY(stext)
  THUMB(	.thumb			)	@ switch to Thumb now.
  THUMB(1:			)
 
-	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode

arch/arm/kernel/head.S ENTRY(stext) · line 94

Quoted 10 lines

arch/arm/kernel/head.S +6 -1

-						@ and irqs disabled
+#ifdef CONFIG_ARM_VIRT_EXT
+	bl	__hyp_stub_install
+#endif
+	@ ensure svc mode and all interrupts masked
+	safe_svcmode_maskall r9
+
 	mrc	p15, 0, r9, c0, c0		@ get processor id
 	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
 	movs	r10, r5				@ invalid processor (r5=0)?

..and looks like undoing this part fixes it. Any ideas?

Quoted 2 lines

I quickly tried disabling ARCH_OMAP3 and ARCH_OMAP4 so it's ARMv6 but that does not help.

If you compiled for v6 only, we can safely exclude __hyp_stub_install, and I assume that you get past the decompressor.

If so, that indicates some side effect of the safe_svcmode_maskall macro, and I suspect the "movs pc, lr" bit.

Can you try the attached patch? It basically falls back to the previous behaviour if not entered in HYP mode.

Quoted 3 lines · Marc Zyngier

If so, that indicates some side effect of the safe_svcmode_maskall macro, and I suspect the "movs pc, lr" bit.

That would be surprizing if the "movs pc, lr" was to blame. This should work on all architectures.

However the A bit might be to blame.

Quoted 2 lines

Can you try the attached patch? It basically falls back to the previous behaviour if not entered in HYP mode.

This is likely to work of course. However I think we should try to pinpoint the exact problem i.e. whether it is the A bit or the "movs pc, lr" which makes a difference (it is unlikely to be both).

So I was about to suggest to test this patch as well:

arch/arm/include/asm/assembler.h +1 -2

index 683a1e6b60..118e22ee46 100644
@@ -254,8 +254,7 @@
 	mov	lr , \reg
 	and	lr , lr , #MODE_MASK
 	cmp	lr , #HYP_MODE
-	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
-	bic	\reg , \reg , #MODE_MASK
+	mov	\reg , #PSR_I_BIT | PSR_F_BIT
 	orr	\reg , \reg , #SVC_MODE
 THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
 	msr	spsr_cxsf, \reg

Nicolas

Quoted 51 lines · Marc Zyngier

Hi Tony,

Quoted 47 lines · Tony Lindgren

Hi,

Quoted 14 lines · Marc Zyngier

From: Dave Martin <[email protected]>

This patch does two things:

* Ensure that asynchronous aborts are masked at kernel entry. The bootloader should be masking these anyway, but this reduces the damage window just in case it doesn't.

* Enter svc mode via exception return to ensure that CPU state is properly serialised. This does not matter when switching from an ordinary privileged mode ("PL1" modes in ARMv7-AR rev C parlance), but it potentially does matter when switching from a another privileged mode such as hyp mode.

Quoted 6 lines

This should allow the kernel to boot safely either from svc mode or hyp mode, even if no support for use of the ARM Virtualization Extensions is built into the kernel.

Signed-off-by Dave Martin <[email protected]>

Signed-off-by Marc Zyngier <[email protected]>

Just bisected this down in linux-next for breaking booting of my omap2420 ARMv6 based n8x0..

Quoted 7 lines

arch/arm/kernel/head.S +0 -1

@@ -83,8 +83,12 @@ ENTRY(stext)
  THUMB(	.thumb			)	@ switch to Thumb now.
  THUMB(1:			)
 
-	setmode	PSR_F_BIT | PSR_I_BIT | SVC_MODE, r9 @ ensure svc mode

arch/arm/kernel/head.S ENTRY(stext) · line 94

Quoted 10 lines

arch/arm/kernel/head.S +6 -1

-						@ and irqs disabled
+#ifdef CONFIG_ARM_VIRT_EXT
+	bl	__hyp_stub_install
+#endif
+	@ ensure svc mode and all interrupts masked
+	safe_svcmode_maskall r9
+
 	mrc	p15, 0, r9, c0, c0		@ get processor id
 	bl	__lookup_processor_type		@ r5=procinfo r9=cpuid
 	movs	r10, r5				@ invalid processor (r5=0)?

..and looks like undoing this part fixes it. Any ideas?

Quoted 8 lines

Quoted 2 lines

I quickly tried disabling ARCH_OMAP3 and ARCH_OMAP4 so it's ARMv6 but that does not help.

If you compiled for v6 only, we can safely exclude __hyp_stub_install, and
I assume that you get past the decompressor.

If so, that indicates some side effect of the safe_svcmode_maskall

Quoted 1 line

and I suspect the "movs pc, lr" bit.

That would be surprizing if the "movs pc, lr" was to blame. This should

work on all architectures.

However the A bit might be to blame.

Quoted 2 lines

Can you try the attached patch? It basically falls back to the previous behaviour if not entered in HYP mode.

This is likely to work of course. However I think we should try to pinpoint the exact problem i.e. whether it is the A bit or the "movs pc,

lr" which makes a difference (it is unlikely to be both).

So I was about to suggest to test this patch as well:

arch/arm/include/asm/assembler.h +0 -0

index 683a1e6b60..118e22ee46 100644

If you compiled for v6 only, we can safely exclude __hyp_stub_install, and I assume that you get past the decompressor.

If so, that indicates some side effect of the safe_svcmode_maskall macro, and I suspect the "movs pc, lr" bit.

Can you try the attached patch? It basically falls back to the previous behaviour if not entered in HYP mode.

arch/arm/include/asm/assembler.h +5 -4

index 658a15d..b21b97f 100644
@@ -254,16 +254,17 @@
 	mov	lr , \reg
 	and	lr , lr , #MODE_MASK
 	cmp	lr , #HYP_MODE
-	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
+	orr	\reg , \reg , #PSR_I_BIT | PSR_F_BIT
 	bic	\reg , \reg , #MODE_MASK
 	orr	\reg , \reg , #SVC_MODE
 THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
-	msr	spsr_cxsf, \reg
-	adr	lr, BSYM(2f)
 	bne	1f
+	orr	\reg, \reg, #PSR_A_BIT
+	adr	lr, BSYM(2f)
+	msr	spsr_cxsf, \reg
 	__MSR_ELR_HYP(14)
 	__ERET
-1:	movs	pc, lr
+1:	msr	cpsr_c, \reg
 2:
 .endm
 

Quoted 5 lines · Marc Zyngier

If so, that indicates some side effect of the safe_svcmode_maskall macro, and I suspect the "movs pc, lr" bit.

Can you try the attached patch? It basically falls back to the previous behaviour if not entered in HYP mode.

Yes, with this it boots OK.

Quoted 8 lines · Tony Lindgren

Quoted 5 lines · Marc Zyngier

If so, that indicates some side effect of the safe_svcmode_maskall macro, and I suspect the "movs pc, lr" bit.

Can you try the attached patch? It basically falls back to the previous behaviour if not entered in HYP mode.

Quoted 1 line

Yes, with this it boots OK.

OK. In that case, I suggest this patch be sent to Russell to fix this issue so he could push the ARM stuff to Linus ASAP.

Acked-by Nicolas Pitre <[email protected]>

Quoted 14 lines · Nicolas Pitre

Quoted 11 lines · Tony Lindgren

Quoted 7 lines · Marc Zyngier

If so, that indicates some side effect of the safe_svcmode_maskall macro,
and I suspect the "movs pc, lr" bit.

Can you try the attached patch? It basically falls back to the previous
behaviour if not entered in HYP mode.

Yes, with this it boots OK.

Quoted 4 lines

OK. In that case, I suggest this patch be sent to Russell to fix this issue so he could push the ARM stuff to Linus ASAP.

Acked-by Nicolas Pitre <[email protected]>

Yes we can come back to this:

Tested-by Tony Lindgren <[email protected]>

Quoted 8 lines · Tony Lindgren

Quoted 5 lines · Marc Zyngier

If so, that indicates some side effect of the safe_svcmode_maskall macro, and I suspect the "movs pc, lr" bit.

Can you try the attached patch? It basically falls back to the previous behaviour if not entered in HYP mode.

Quoted 1 line

Yes, with this it boots OK.

OK. In that case, I suggest this patch be sent to Russell to fix this issue so he could push the ARM stuff to Linus ASAP.

Acked-by Nicolas Pitre <[email protected]>

If so, that indicates some side effect of the safe_svcmode_maskall macro, and I suspect the "movs pc, lr" bit.

Can you try the attached patch? It basically falls back to the previous behaviour if not entered in HYP mode.

arch/arm/include/asm/assembler.h +1 -3

index 658a15d..b21b97f 100644
@@ -254,16 +254,17 @@
 	mov	lr , \reg
 	and	lr , lr , #MODE_MASK
 	cmp	lr , #HYP_MODE
-	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
+	orr	\reg , \reg , #PSR_I_BIT | PSR_F_BIT
 	bic	\reg , \reg , #MODE_MASK
 	orr	\reg , \reg , #SVC_MODE
 THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
-	msr	spsr_cxsf, \reg
-	adr	lr, BSYM(2f)
 	bne	1f

Quoted 6 lines · Marc Zyngier

If so, that indicates some side effect of the safe_svcmode_maskall macro, and I suspect the "movs pc, lr" bit.

Can you try the attached patch? It basically falls back to the previous behaviour if not entered in HYP mode.

...

Quoted 16 lines

arch/arm/include/asm/assembler.h +1 -3

index 658a15d..b21b97f 100644
@@ -254,16 +254,17 @@
 	mov	lr , \reg
 	and	lr , lr , #MODE_MASK
 	cmp	lr , #HYP_MODE
-	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
+	orr	\reg , \reg , #PSR_I_BIT | PSR_F_BIT
 	bic	\reg , \reg , #MODE_MASK
 	orr	\reg , \reg , #SVC_MODE
 THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
-	msr	spsr_cxsf, \reg
-	adr	lr, BSYM(2f)
 	bne	1f

arch/arm/include/asm/assembler.h line 270

Quoted 10 lines

arch/arm/include/asm/assembler.h +4 -1

+	orr	\reg, \reg, #PSR_A_BIT
+	adr	lr, BSYM(2f)
+	msr	spsr_cxsf, \reg
 	__MSR_ELR_HYP(14)
 	__ERET
-1:	movs	pc, lr
+1:	msr	cpsr_c, \reg
 2:
 .endm
 

The minimal version of this that still boots on my n800 is just the last change of the above patch:

Quoted 42 lines · Tony Lindgren

Quoted 6 lines · Marc Zyngier

If so, that indicates some side effect of the safe_svcmode_maskall macro, and I suspect the "movs pc, lr" bit.

Can you try the attached patch? It basically falls back to the previous behaviour if not entered in HYP mode.

...

Quoted 16 lines

arch/arm/include/asm/assembler.h +1 -3

index 658a15d..b21b97f 100644
@@ -254,16 +254,17 @@
 	mov	lr , \reg
 	and	lr , lr , #MODE_MASK
 	cmp	lr , #HYP_MODE
-	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
+	orr	\reg , \reg , #PSR_I_BIT | PSR_F_BIT
 	bic	\reg , \reg , #MODE_MASK
 	orr	\reg , \reg , #SVC_MODE
 THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
-	msr	spsr_cxsf, \reg
-	adr	lr, BSYM(2f)
 	bne	1f

arch/arm/include/asm/assembler.h line 270

Quoted 10 lines

arch/arm/include/asm/assembler.h +4 -1

+	orr	\reg, \reg, #PSR_A_BIT
+	adr	lr, BSYM(2f)
+	msr	spsr_cxsf, \reg
 	__MSR_ELR_HYP(14)
 	__ERET
-1:	movs	pc, lr
+1:	msr	cpsr_c, \reg
 2:
 .endm
 

The minimal version of this that still boots on my n800 is just the last change of the above patch:

arch/arm/include/asm/assembler.h THUMB( orr \reg , \reg , #PSR_T_BIT ) · line 269

Quoted 8 lines

arch/arm/include/asm/assembler.h +1 -1

@@ -263,7 +263,7 @@ THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
 	bne	1f
 	__MSR_ELR_HYP(14)
 	__ERET
-1:	movs	pc, lr
+1:	msr	cpsr_c, \reg
 2:
 .endm
 

In an attempt to narrow this down...

Can you follow this (i.e., _after_ a known successful switch to SVC mode)

(a)
mrs \reg, cpsr
msr spsr_cxsf, \reg
adr \reg, 3f
movs pc, lr
3:

and (b)

mrs \reg, cpsr
orr \reg, \reg, #CPSR_A_BIT
msr cpsr_cxsf, \reg

and (c)

mrs \reg, cpsr
orr \reg, \reg, #CPSR_A_BIT
msr spsr_cxsf, \reg
adr \reg, 3f
movs pc, lr
3:

If only (a) works, this would suggest that the attempt to set the A bit is causing the problem.

If only (b) works, this suggests that the A bit is OK but that some invalid hardware state, or something else we don't understand, is causing exception returns to fail in general.

If (a) and (b) work but (c) fails, this suggests that specifically trying to set the A bit via an exception return is problematic.

If all of them work then this suggests some invalid hardware state or something else we don't understand, but which is cleared by the initial msr cpsr_c which clobbers the processor mode.

Quoted 37 lines · Dave Martin

In an attempt to narrow this down...

Can you follow this (i.e., _after_ a known successful switch to SVC mode)

(a)
mrs \reg, cpsr
msr spsr_cxsf, \reg
adr \reg, 3f
movs pc, lr
3:

and (b)

mrs \reg, cpsr
orr \reg, \reg, #CPSR_A_BIT
msr cpsr_cxsf, \reg

and (c)

mrs \reg, cpsr
orr \reg, \reg, #CPSR_A_BIT
msr spsr_cxsf, \reg
adr \reg, 3f
movs pc, lr
3:

If only (a) works, this would suggest that the attempt to set the A bit is causing the problem.

If only (b) works, this suggests that the A bit is OK but that some invalid hardware state, or something else we don't understand, is causing exception returns to fail in general.

Quoted 6 lines

If (a) and (b) work but (c) fails, this suggests that specifically trying to set the A bit via an exception return is problematic.

If all of them work then this suggests some invalid hardware state or something else we don't understand, but which is cleared by the initial msr cpsr_c which clobbers the processor mode.

Playing with an Integrator-CP with a 1136 tile, the only way I could cause the thing to fail was to boot the thing in SYSTEM mode.

Tony, can you possibly also try the attached patch?

arch/arm/include/asm/assembler.h +3 -0

index 683a1e6..7714ec7 100644
@@ -253,6 +253,8 @@
 	mrs	\reg , cpsr
 	mov	lr , \reg
 	and	lr , lr , #MODE_MASK
+	cmp	lr , #SYSTEM_MODE	/* Yet another braindead platform? */
+	beq	omgsys
 	cmp	lr , #HYP_MODE
 	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
 	bic	\reg , \reg , #MODE_MASK

@@ -264,6 +266,7 @@ THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
 	__MSR_ELR_HYP(14)
 	__ERET
 1:	movs	pc, lr
+omgsys: msr	cpsr_c, \reg
 2:
 .endm
 

Quoted 42 lines · Tony Lindgren

Quoted 6 lines · Marc Zyngier

If so, that indicates some side effect of the safe_svcmode_maskall macro, and I suspect the "movs pc, lr" bit.

Can you try the attached patch? It basically falls back to the previous behaviour if not entered in HYP mode.

...

Quoted 16 lines

arch/arm/include/asm/assembler.h +1 -3

index 658a15d..b21b97f 100644
@@ -254,16 +254,17 @@
 	mov	lr , \reg
 	and	lr , lr , #MODE_MASK
 	cmp	lr , #HYP_MODE
-	orr	\reg , \reg , #PSR_A_BIT | PSR_I_BIT | PSR_F_BIT
+	orr	\reg , \reg , #PSR_I_BIT | PSR_F_BIT
 	bic	\reg , \reg , #MODE_MASK
 	orr	\reg , \reg , #SVC_MODE
 THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
-	msr	spsr_cxsf, \reg
-	adr	lr, BSYM(2f)
 	bne	1f

arch/arm/include/asm/assembler.h line 270

Quoted 10 lines

arch/arm/include/asm/assembler.h +4 -1

+	orr	\reg, \reg, #PSR_A_BIT
+	adr	lr, BSYM(2f)
+	msr	spsr_cxsf, \reg
 	__MSR_ELR_HYP(14)
 	__ERET
-1:	movs	pc, lr
+1:	msr	cpsr_c, \reg
 2:
 .endm
 

The minimal version of this that still boots on my n800 is just the last change of the above patch:

arch/arm/include/asm/assembler.h THUMB( orr \reg , \reg , #PSR_T_BIT ) · line 269

Quoted 8 lines

arch/arm/include/asm/assembler.h +1 -1

@@ -263,7 +263,7 @@ THUMB(	orr	\reg , \reg , #PSR_T_BIT	)
 	bne	1f
 	__MSR_ELR_HYP(14)
 	__ERET
-1:	movs	pc, lr
+1:	msr	cpsr_c, \reg
 2:
 .endm
 

In an attempt to narrow this down...

Can you follow this (i.e., _after_ a known successful switch to SVC mode)

(a)
mrs \reg, cpsr
msr spsr_cxsf, \reg
adr \reg, 3f
movs pc, lr
3:

and (b)

mrs \reg, cpsr
orr \reg, \reg, #CPSR_A_BIT
msr cpsr_cxsf, \reg

and (c)

mrs \reg, cpsr
orr \reg, \reg, #CPSR_A_BIT
msr spsr_cxsf, \reg
adr \reg, 3f
movs pc, lr
3:

If only (a) works, this would suggest that the attempt to set the A bit is causing the problem.

If only (b) works, this suggests that the A bit is OK but that some invalid hardware state, or something else we don't understand, is causing exception returns to fail in general.

If (a) and (b) work but (c) fails, this suggests that specifically trying to set the A bit via an exception return is problematic.

If all of them work then this suggests some invalid hardware state or something else we don't understand, but which is cleared by the initial msr cpsr_c which clobbers the processor mode.