Skip to contentSkip to conversation

arm64: Booting clarifications and fine grained traps setup

PATCH In review Mark Brown 3 participants 11 messages Apr 1, 2021 – Apr 8, 2021

Subsystems (2)
lore
Share & tools
lore

Mark Brown

author
#0 lore raw

v2 cover arm64: Booting clarifications and fine grained traps setup

Currently there are a number of areas where we are not explicit about how system registers should be configured when booting the kernel, provide more detailed guidance to help ensure people get things right. I'm not entirely happy with the wording on use of specified reset values so it is a separate patch at the end, I worry that the requirement is too strict.

This was triggered by noticing that it was in spec to leave fine grained traps to EL2 enabled so explicitly disable them at boot, this is sent as part of the same series since doing so means we require that EL3 provide access to fine grained traps in the same document.

v2:
- Pull booting and FGT setup into a single series.
- Require system registers at or below the level the kernel is entered to be initialised, not just those at that level.
- Rather than documenting the desired EL2 setup for FGT add a general statement about how we expect registers to be set up.
- Reorder register initialisation in el2_setup.h.
- Document requirement for SCR_EL3.FGTEn to be 1.

Mark Brown (4):
arm64: Document requirements for fine grained traps at boot
arm64: Disable fine grained traps on boot
arm64: Require that system registers at all visible ELs be initialized
arm64: Document values for system registers on boot

 Documentation/arm64/booting.rst    | 15 ++++++++++++---
 arch/arm64/include/asm/el2_setup.h | 21 +++++++++++++++++++++
 arch/arm64/include/asm/sysreg.h    |  6 ++++++
 3 files changed, 39 insertions(+), 3 deletions(-)

base-commit: 1e28eed17697bcf343c6743f0028cc3b5dd88bf0

Signature · 7 lines
2.20.1


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
author
#1 lore raw

PATCH v2 1/4 arm64: Document requirements for fine grained traps at boot

The arm64 FEAT_FGT extension introduces a set of traps to EL2 for accesses to small sets of registers and instructions from EL1 and EL0, access to which is controlled by EL3. Require access to it so that it is available to us in future and so that we can ensure these traps are disabled during boot.

---
 Documentation/arm64/booting.rst | 6 ++++++
 1 file changed, 6 insertions(+)
Documentation/arm64/booting.rst +6 -0
index 7552dbc1cc54..92ec0bea1af5 100644
@@ -270,6 +270,12 @@ Before jumping into the kernel, the following conditions must be met:
       having 0b1 set for the corresponding bit for each of the auxiliary
       counters present.
 
+  For CPUs with the Fine Grained Traps (FEAT_FGT) extension present:
+
+  - If EL3 is present and the kernel is entered at EL2:
+
+    - SCR_EL3.FGTEn (bit 27) must be initialised to 0b1.
+
 The requirements described above for CPU mode, caches, MMUs, architected
 timers, coherency and system registers apply to all CPUs.  All CPUs must
 enter the kernel in the same exception level.
Signature · 7 lines
2.20.1


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Acked-by on v2 1/4
#2 lore raw

Re: [PATCH v2 1/4] arm64: Document requirements for fine grained traps at boot

Quoted 19 lines · Mark Brown

The arm64 FEAT_FGT extension introduces a set of traps to EL2 for accesses to small sets of registers and instructions from EL1 and EL0, access to which is controlled by EL3. Require access to it so that it is available to us in future and so that we can ensure these traps are disabled during boot.

---
 Documentation/arm64/booting.rst | 6 ++++++
 1 file changed, 6 insertions(+)
Documentation/arm64/booting.rst +0 -0
index 7552dbc1cc54..92ec0bea1af5 100644
@@ -270,6 +270,12 @@ Before jumping into the kernel, the following conditions must be met:
       having 0b1 set for the corresponding bit for each of the auxiliary
       counters present.

Documentation/arm64/booting.rst Before jumping into the kernel, the following conditions must be met: · line 281

Quoted 10 lines
Documentation/arm64/booting.rst +6 -0
 
+  For CPUs with the Fine Grained Traps (FEAT_FGT) extension present:
+
+  - If EL3 is present and the kernel is entered at EL2:
+
+    - SCR_EL3.FGTEn (bit 27) must be initialised to 0b1.
+
 The requirements described above for CPU mode, caches, MMUs, architected
 timers, coherency and system registers apply to all CPUs.  All CPUs must
 enter the kernel in the same exception level.
	M.
Signature · 6 lines
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
author
#3 lore raw

PATCH v2 2/4 arm64: Disable fine grained traps on boot

In reply to Mark Brown

The arm64 FEAT_FGT extension introduces a set of traps to EL2 for accesses to small sets of registers and instructions from EL1 and EL0. Currently Linux makes no use of this feature, ensure that it is not active at boot by disabling the traps during EL2 setup.

---
 arch/arm64/include/asm/el2_setup.h | 21 +++++++++++++++++++++
 arch/arm64/include/asm/sysreg.h    |  6 ++++++
 2 files changed, 27 insertions(+)
arch/arm64/include/asm/el2_setup.h +21 -0
index d77d358f9395..b3f2d3bb0938 100644
@@ -131,6 +131,26 @@
 .Lskip_sve_\@:
 .endm
 
+/* Disable any fine grained traps */
+.macro __init_el2_fgt
+	mrs	x1, id_aa64mmfr0_el1
+	ubfx	x1, x1, #ID_AA64MMFR0_FGT_SHIFT, #4
+	cbz	x1, .Lskip_fgt_\@
+
+	msr_s	SYS_HDFGRTR_EL2, xzr
+	msr_s	SYS_HDFGWTR_EL2, xzr
+	msr_s	SYS_HFGRTR_EL2, xzr
+	msr_s	SYS_HFGWTR_EL2, xzr
+	msr_s	SYS_HFGITR_EL2, xzr
+
+	mrs	x1, id_aa64pfr0_el1		// AMU traps UNDEF without AMU
+	ubfx	x1, x1, #ID_AA64PFR0_AMU_SHIFT, #4
+	cbz	x1, .Lskip_fgt_\@
+
+	msr_s	SYS_HAFGRTR_EL2, xzr
+.Lskip_fgt_\@:
+.endm
+
 .macro __init_el2_nvhe_prepare_eret
 	mov	x0, #INIT_PSTATE_EL1
 	msr	spsr_el2, x0
@@ -155,6 +175,7 @@
 	__init_el2_nvhe_idregs
 	__init_el2_nvhe_cptr
 	__init_el2_nvhe_sve
+	__init_el2_fgt
 	__init_el2_nvhe_prepare_eret
 .endm
 
arch/arm64/include/asm/sysreg.h +6 -0
index d4a5fca984c3..b35468927363 100644
@@ -475,9 +475,15 @@
 #define SYS_PMCCFILTR_EL0		sys_reg(3, 3, 14, 15, 7)
 
 #define SYS_SCTLR_EL2			sys_reg(3, 4, 1, 0, 0)
+#define SYS_HFGRTR_EL2			sys_reg(3, 4, 1, 1, 4)
+#define SYS_HFGWTR_EL2			sys_reg(3, 4, 1, 1, 5)
+#define SYS_HFGITR_EL2			sys_reg(3, 4, 1, 1, 6)
 #define SYS_ZCR_EL2			sys_reg(3, 4, 1, 2, 0)
 #define SYS_TRFCR_EL2			sys_reg(3, 4, 1, 2, 1)
 #define SYS_DACR32_EL2			sys_reg(3, 4, 3, 0, 0)
+#define SYS_HDFGRTR_EL2			sys_reg(3, 4, 3, 1, 4)
+#define SYS_HDFGWTR_EL2			sys_reg(3, 4, 3, 1, 5)
+#define SYS_HAFGRTR_EL2			sys_reg(3, 4, 3, 1, 6)
 #define SYS_SPSR_EL2			sys_reg(3, 4, 4, 0, 0)
 #define SYS_ELR_EL2			sys_reg(3, 4, 4, 0, 1)
 #define SYS_IFSR32_EL2			sys_reg(3, 4, 5, 0, 1)
Signature · 7 lines
2.20.1


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Reviewed-by on v2 2/4
#4 lore raw

Re: [PATCH v2 2/4] arm64: Disable fine grained traps on boot

Quoted 7 lines · Mark Brown

The arm64 FEAT_FGT extension introduces a set of traps to EL2 for accesses to small sets of registers and instructions from EL1 and EL0. Currently Linux makes no use of this feature, ensure that it is not active at boot by disabling the traps during EL2 setup.

	M.
Signature · 6 lines
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
author
#5 lore raw

PATCH v2 3/4 arm64: Require that system registers at all visible ELs be initialized

In reply to Mark Brown

Currently we require that software at a higher exception level initialise all registers at the exception level the kernel will be entered prior to starting the kernel in order to ensure that there is nothing uninitialised which could result in an UNKNOWN state while running the kernel. The expectation is that the software running at the highest exception levels will be tightly coupled to the system and can ensure that all available features are appropriately initialised and that the kernel can initialise anything else.

There is a gap here in the case where new registers are added to lower exception levels that require initialisation but the kernel does not yet understand them. Extend the requirement to also include exception levels below the one where the kernel is entered to cover this.

---
 Documentation/arm64/booting.rst | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
Documentation/arm64/booting.rst +4 -3
index 92ec0bea1af5..4fcc00add117 100644
@@ -202,9 +202,10 @@ Before jumping into the kernel, the following conditions must be met:
 
 - System registers
 
-  All writable architected system registers at the exception level where
-  the kernel image will be entered must be initialised by software at a
-  higher exception level to prevent execution in an UNKNOWN state.
+  All writable architected system registers at or below the exception
+  level where the kernel image will be entered must be initialised by
+  software at a higher exception level to prevent execution in an UNKNOWN
+  state.
 
   - SCR_EL3.FIQ must have the same value across all CPUs the kernel is
     executing on.
Signature · 7 lines
2.20.1


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Acked-by on v2 3/4
#6 lore raw

Re: [PATCH v2 3/4] arm64: Require that system registers at all visible ELs be initialized

Quoted 10 lines · Mark Brown

Currently we require that software at a higher exception level initialise all registers at the exception level the kernel will be entered prior to starting the kernel in order to ensure that there is nothing uninitialised which could result in an UNKNOWN state while running the kernel. The expectation is that the software running at the highest exception levels will be tightly coupled to the system and can ensure that all available features are appropriately initialised and that the kernel can initialise anything else.

Quoted 7 lines

There is a gap here in the case where new registers are added to lower exception levels that require initialisation but the kernel does not yet understand them. Extend the requirement to also include exception levels below the one where the kernel is entered to cover this.

	M.
Signature · 6 lines
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
author
#7 lore raw

PATCH v2 4/4 arm64: Document values for system registers on boot

In reply to Mark Brown

When booting require that the system registers available at or below the exception level the kernel is entered be initialised but do not specify what values should be used in the general case, creating some potential for issues if the kernel does not subsequently configure those registers explicitly (for example if they are not yet used by the kernel) or where their effects may create issues during early configuration.

Specify that where the architecture provides a reset value that value must be used.

---
 Documentation/arm64/booting.rst | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Documentation/arm64/booting.rst +3 -1
index 4fcc00add117..d49f568eb79d 100644
@@ -205,7 +205,9 @@ Before jumping into the kernel, the following conditions must be met:
   All writable architected system registers at or below the exception
   level where the kernel image will be entered must be initialised by
   software at a higher exception level to prevent execution in an UNKNOWN
-  state.
+  state.  Where these architected system registers have reset values
+  specified by the architecture they must be initialised to those values
+  unless specified more specifically.
 
   - SCR_EL3.FIQ must have the same value across all CPUs the kernel is
     executing on.
Signature · 7 lines
2.20.1


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
#8 lore raw

Re: [PATCH v2 4/4] arm64: Document values for system registers on boot

Hi Mark,

Quoted 35 lines · Mark Brown

When booting require that the system registers available at or below the exception level the kernel is entered be initialised but do not specify what values should be used in the general case, creating some potential for issues if the kernel does not subsequently configure those registers explicitly (for example if they are not yet used by the kernel) or where their effects may create issues during early configuration.

Specify that where the architecture provides a reset value that value must be used.

---
 Documentation/arm64/booting.rst | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Documentation/arm64/booting.rst +3 -1
index 4fcc00add117..d49f568eb79d 100644
@@ -205,7 +205,9 @@ Before jumping into the kernel, the following conditions must be met:
   All writable architected system registers at or below the exception
   level where the kernel image will be entered must be initialised by
   software at a higher exception level to prevent execution in an UNKNOWN
-  state.
+  state.  Where these architected system registers have reset values
+  specified by the architecture they must be initialised to those values
+  unless specified more specifically.
 
   - SCR_EL3.FIQ must have the same value across all CPUs the kernel is
     executing on.
Signature · 1 line
2.20.1

Is that always enforceable? Is that even desirable?

Take for example ICC_SRE_EL2.SRE. The reset value for that bit is
0. But it is extremely likely that the FW has set this bit to 1 in order to be able to use interrupts with the sysreg interface. However, as outlined in the GIC spec:

<quote>
If software changes this bit from 1 to 0, the results are UNPREDICTABLE. </quote>

We could go and specify this one, but I fear there is a lot of things we'd have to make explicit...

Thanks,

	M.
Signature · 6 lines
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
author
#9 lore raw

Re: [PATCH v2 4/4] arm64: Document values for system registers on boot

Quoted 5 lines · Marc Zyngier
Quoted 2 lines · Mark Brown

Specify that where the architecture provides a reset value that value must be used.

Quoted 8 lines

Documentation/arm64/booting.rst Before jumping into the kernel, the following conditions must be met: · line 210

Quoted 6 lines

level where the kernel image will be entered must be initialised by software at a higher exception level to prevent execution in an UNKNOWN

Documentation/arm64/booting.rst +3 -1
-  state.
+  state.  Where these architected system registers have reset values
+  specified by the architecture they must be initialised to those values
+  unless specified more specifically.

Is that always enforceable? Is that even desirable?

I'm not convinced on either front frankly, like I said in the cover letter I was worrying that it is too strict a requirement. I couldn't think of a better/more appropriate wording and figured it might start a discussion.

Quoted 8 lines

Take for example ICC_SRE_EL2.SRE. The reset value for that bit is
0. But it is extremely likely that the FW has set this bit to 1 in order to be able to use interrupts with the sysreg interface. However, as outlined in the GIC spec:

<quote>
If software changes this bit from 1 to 0, the results are UNPREDICTABLE. </quote>

That definitely means it's unenforcable without an explict override for that case, having now seen that case I'm sure there's others.

Quoted 2 lines

We could go and specify this one, but I fear there is a lot of things we'd have to make explicit...

Yeah, that's pushing me towards dropping this patch. We could add a general statement recommending that people use the defaults without actually requiring it but that's just vague and unenforcable instead of strict and unimplementable which isn't particularly better. If we're going to have to keep updating the document it's going to be less effort all round to just update for cases we can't handle in code rather than keep on adding exceptions and require firmware to reset lots of stuff we don't *really* have strict requirements for.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

maintainer
#10 lore raw

Re: (subset) [PATCH v2 0/4] arm64: Booting clarifications and fine grained traps setup

In reply to Mark Brown
Quoted 8 lines · Mark Brown

Currently there are a number of areas where we are not explicit about how system registers should be configured when booting the kernel, provide more detailed guidance to help ensure people get things right. I'm not entirely happy with the wording on use of specified reset values so it is a separate patch at the end, I worry that the requirement is too strict.

[...]

Applied to arm64 (for-next/fgt-boot-init), thanks!

[1/4] arm64: Document requirements for fine grained traps at boot

      https://git.kernel.org/arm64/c/3e237387bb76
[2/4] arm64: Disable fine grained traps on boot
      https://git.kernel.org/arm64/c/31c00d2aeaa2
[3/4] arm64: Require that system registers at all visible ELs be initialized
      https://git.kernel.org/arm64/c/230800cd315c
Signature · 7 lines
Catalin


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel