Skip to contentSkip to conversation

arm64: kvm: avoid referencing cpu_hwcaps from hyp

PATCH Applied Mark Rutland 3 participants 11 messages Oct 26, 2020 – Oct 30, 2020

Subsystems (2)
lore
Share & tools
lore

Mark Rutland

author reviewer
#0 lore raw

v2 cover arm64: kvm: avoid referencing cpu_hwcaps from hyp

In a few places we use cpus_have_const_cap() in hyp code, usually because this is hidden within a helper that's also used in regular kernel context. As cpus_have_const_cap() generates code to read the cpu_hwcaps array before capabilities are finalized, this means we generate some potentially-unsound references to regular kernel VAs, but this these are redundant as capabilities are finalized before we initialize the kvm hyp code.

This series gets rid of the redundant code by automatically upgrading cpust_have_const_cap() to cpus_have_final_cap() when used in hyp code. This allows us to avoid creating an NVHE alias for the cpu_hwcaps array, so we can catch if we accidentally introduce an runtime reference to this (e.g. via cpus_have_cap()).

Since v1 [1]:
* Trivial rebase to v5.10-rc1

[1] https://lore.kernel.org/r/[email protected]

Mark Rutland (3):
arm64: kvm: factor out is_{vhe,nvhe}_hyp_code()
arm64: cpufeature: reorder cpus_have_{const,final}_cap()
arm64: cpufeature: upgrade hyp caps to final

 arch/arm64/include/asm/cpufeature.h | 40 ++++++++++++++++++++++++++++---------
 arch/arm64/include/asm/virt.h       |  9 ++++-----
 arch/arm64/kernel/image-vars.h      |  1 -
 3 files changed, 35 insertions(+), 15 deletions(-)
Signature · 6 lines
2.11.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author reviewer
#1 lore raw

PATCH v2 1/3 arm64: kvm: factor out is_{vhe,nvhe}_hyp_code()

Currently has_vhe() detects whether it is being compiled for VHE/NVHE hyp code based on preprocessor definitions, and uses this knowledge to avoid redundant runtime checks.

There are other cases where we'd like to use this knowledge, so let's factor the preprocessor checks out into separate helpers.

There should be no functional change as a result of this patch.

Cc David Brazdil <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm64/include/asm/virt.h | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)
arch/arm64/include/asm/virt.h +16 -5
index 09977acc007d1..300be14ba77b2 100644
@@ -83,16 +83,27 @@ static inline bool is_kernel_in_hyp_mode(void)
 	return read_sysreg(CurrentEL) == CurrentEL_EL2;
 }
 
+static __always_inline bool is_vhe_hyp_code(void)
+{
+	/* Only defined for code run in VHE hyp context */
+	return __is_defined(__KVM_VHE_HYPERVISOR__);
+}
+
+static __always_inline bool is_nvhe_hyp_code(void)
+{
+	/* Only defined for code run in NVHE hyp context */
+	return __is_defined(__KVM_NVHE_HYPERVISOR__);
+}
+
 static __always_inline bool has_vhe(void)
 {
 	/*
-	 * The following macros are defined for code specic to VHE/nVHE.
-	 * If has_vhe() is inlined into those compilation units, it can
-	 * be determined statically. Otherwise fall back to caps.
+	 * Code only run in VHE/NVHE hyp context can assume VHE is present or
+	 * absent. Otherwise fall back to caps.
 	 */
-	if (__is_defined(__KVM_VHE_HYPERVISOR__))
+	if (is_vhe_hyp_code())
 		return true;
-	else if (__is_defined(__KVM_NVHE_HYPERVISOR__))
+	else if (is_nvhe_hyp_code())
 		return false;
 	else
 		return cpus_have_final_cap(ARM64_HAS_VIRT_HOST_EXTN);
Signature · 6 lines
2.11.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
maintainer Acked-by on v2 1/3
#2 lore raw

Re: [PATCHv2 1/3] arm64: kvm: factor out is_{vhe,nvhe}_hyp_code()

Quoted 9 lines · Mark Rutland

Currently has_vhe() detects whether it is being compiled for VHE/NVHE hyp code based on preprocessor definitions, and uses this knowledge to avoid redundant runtime checks.

There are other cases where we'd like to use this knowledge, so let's factor the preprocessor checks out into separate helpers.

There should be no functional change as a result of this patch.

Quoted 7 lines
Cc David Brazdil <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm64/include/asm/virt.h | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

Will
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author reviewer
#3 lore raw

PATCH v2 2/3 arm64: cpufeature: reorder cpus_have_{const, final}_cap()

In reply to Mark Rutland

In a subsequent patch we'll modify cpus_have_const_cap() to call cpus_have_final_cap(), and hence we need to define cpus_have_final_cap() first.

To make subsequent changes easier to follow, this patch reorders the two without making any other changes.

There should be no functional change as a result of this patch.

Cc David Brazdil <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm64/include/asm/cpufeature.h | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
arch/arm64/include/asm/cpufeature.h +8 -8
index f7e7144af174c..5d18c54507e6a 100644
@@ -428,35 +428,35 @@ static __always_inline bool __cpus_have_const_cap(int num)
 }
 
 /*
- * Test for a capability, possibly with a runtime check.
+ * Test for a capability without a runtime check.
  *
- * Before capabilities are finalized, this behaves as cpus_have_cap().
+ * Before capabilities are finalized, this will BUG().
  * After capabilities are finalized, this is patched to avoid a runtime check.
  *
  * @num must be a compile-time constant.
  */
-static __always_inline bool cpus_have_const_cap(int num)
+static __always_inline bool cpus_have_final_cap(int num)
 {
 	if (system_capabilities_finalized())
 		return __cpus_have_const_cap(num);
 	else
-		return cpus_have_cap(num);
+		BUG();
 }
 
 /*
- * Test for a capability without a runtime check.
+ * Test for a capability, possibly with a runtime check.
  *
- * Before capabilities are finalized, this will BUG().
+ * Before capabilities are finalized, this behaves as cpus_have_cap().
  * After capabilities are finalized, this is patched to avoid a runtime check.
  *
  * @num must be a compile-time constant.
  */
-static __always_inline bool cpus_have_final_cap(int num)
+static __always_inline bool cpus_have_const_cap(int num)
 {
 	if (system_capabilities_finalized())
 		return __cpus_have_const_cap(num);
 	else
-		BUG();
+		return cpus_have_cap(num);
 }
 
 static inline void cpus_set_cap(unsigned int num)
Signature · 6 lines
2.11.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
maintainer
#4 lore raw

Re: [PATCHv2 2/3] arm64: cpufeature: reorder cpus_have_{const,final}_cap()

Quoted 8 lines · Mark Rutland

In a subsequent patch we'll modify cpus_have_const_cap() to call cpus_have_final_cap(), and hence we need to define cpus_have_final_cap() first.

To make subsequent changes easier to follow, this patch reorders the two without making any other changes.

There should be no functional change as a result of this patch.

You say this...

Quoted 24 lines
Cc David Brazdil <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm64/include/asm/cpufeature.h | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
arch/arm64/include/asm/cpufeature.h +2 -2
index f7e7144af174c..5d18c54507e6a 100644
@@ -428,35 +428,35 @@ static __always_inline bool __cpus_have_const_cap(int num)
 }
 
 /*
- * Test for a capability, possibly with a runtime check.
+ * Test for a capability without a runtime check.
  *
- * Before capabilities are finalized, this behaves as cpus_have_cap().
+ * Before capabilities are finalized, this will BUG().
  * After capabilities are finalized, this is patched to avoid a runtime check.
  *

arch/arm64/include/asm/cpufeature.h static __always_inline bool __cpus_have_const_cap(int num) · line 443

Quoted 10 lines
arch/arm64/include/asm/cpufeature.h +2 -2
  * @num must be a compile-time constant.
  */
-static __always_inline bool cpus_have_const_cap(int num)
+static __always_inline bool cpus_have_final_cap(int num)
 {
 	if (system_capabilities_finalized())
 		return __cpus_have_const_cap(num);
 	else
-		return cpus_have_cap(num);
+		BUG();

... but isn't the failure case of calling cpus_have_final_cap() early now different? What does BUG() do at EL2 w/ nVHE?

Will
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

maintainer Acked-by
#5 lore raw

Re: [PATCHv2 2/3] arm64: cpufeature: reorder cpus_have_{const,final}_cap()

Quoted 41 lines · Will Deacon
Quoted 8 lines · Mark Rutland

In a subsequent patch we'll modify cpus_have_const_cap() to call cpus_have_final_cap(), and hence we need to define cpus_have_final_cap() first.

To make subsequent changes easier to follow, this patch reorders the two without making any other changes.

There should be no functional change as a result of this patch.

You say this...

Quoted 29 lines
Cc David Brazdil <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm64/include/asm/cpufeature.h | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
arch/arm64/include/asm/cpufeature.h +3 -3
index f7e7144af174c..5d18c54507e6a 100644
@@ -428,35 +428,35 @@ static __always_inline bool __cpus_have_const_cap(int num)
 }
 
 /*
- * Test for a capability, possibly with a runtime check.
+ * Test for a capability without a runtime check.
  *
- * Before capabilities are finalized, this behaves as cpus_have_cap().
+ * Before capabilities are finalized, this will BUG().
  * After capabilities are finalized, this is patched to avoid a runtime check.
  *
  * @num must be a compile-time constant.
  */
-static __always_inline bool cpus_have_const_cap(int num)
+static __always_inline bool cpus_have_final_cap(int num)
 {
Quoted 8 lines

arch/arm64/include/asm/cpufeature.h static __always_inline bool __cpus_have_const_cap(int num) · line 443

Quoted 5 lines
arch/arm64/include/asm/cpufeature.h +1 -1
 	if (system_capabilities_finalized())
 		return __cpus_have_const_cap(num);
 	else
-		return cpus_have_cap(num);
+		BUG();

... but isn't the failure case of calling cpus_have_final_cap() early now different? What does BUG() do at EL2 w/ nVHE?

Ah no, sorry, I see you're just moving things around and the diff makes it look confusing (that and I've been up since 5:30 for KVM Forum).

So on closer inspection:

Will
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author reviewer
#6 lore raw

Re: [PATCHv2 2/3] arm64: cpufeature: reorder cpus_have_{const,final}_cap()

Quoted 12 lines · Will Deacon
Quoted 11 lines · Will Deacon
Quoted 8 lines · Mark Rutland

In a subsequent patch we'll modify cpus_have_const_cap() to call cpus_have_final_cap(), and hence we need to define cpus_have_final_cap() first.

To make subsequent changes easier to follow, this patch reorders the two without making any other changes.

There should be no functional change as a result of this patch.

You say this...

[...]

Quoted 12 lines
Quoted 11 lines

arch/arm64/include/asm/cpufeature.h static __always_inline bool __cpus_have_const_cap(int num) · line 443

Quoted 8 lines
arch/arm64/include/asm/cpufeature.h +2 -2
-static __always_inline bool cpus_have_const_cap(int num)
+static __always_inline bool cpus_have_final_cap(int num)
 {
 	if (system_capabilities_finalized())
 		return __cpus_have_const_cap(num);
 	else
-		return cpus_have_cap(num);
+		BUG();

... but isn't the failure case of calling cpus_have_final_cap() early now different? What does BUG() do at EL2 w/ nVHE?

Quoted 2 lines

Ah no, sorry, I see you're just moving things around and the diff makes it look confusing (that and I've been up since 5:30 for KVM Forum).

Indeed; the diff was even more confusing before I split this from the changes in the next patch!

Quoted 3 lines

So on closer inspection:

Cheers!

Mark.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author reviewer
#7 lore raw

PATCH v2 3/3 arm64: cpufeature: upgrade hyp caps to final

In reply to Mark Rutland

We finalize caps before initializing kvm hyp code, and any use of cpus_have_const_cap() in kvm hyp code generates redundant and potentially unsound code to read the cpu_hwcaps array.

A number of helper functions used in both hyp context and regular kernel context use cpus_have_const_cap(), as some regular kernel code runs before the capabilities are finalized. It's tedious and error-prone to write separate copies of these for hyp and non-hyp code.

So that we can avoid the redundant code, let's automatically upgrade cpus_have_const_cap() to cpus_have_final_cap() when used in hyp context. With this change, there's never a reason to access to cpu_hwcaps array from hyp code, and we don't need to create an NVHE alias for this.

This should have no effect on non-hyp code.

Cc David Brazdil <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm64/include/asm/cpufeature.h | 26 ++++++++++++++++++++++++--
 arch/arm64/include/asm/virt.h       | 12 ------------
 arch/arm64/kernel/image-vars.h      |  1 -
 3 files changed, 24 insertions(+), 15 deletions(-)
arch/arm64/include/asm/cpufeature.h +24 -2
index 5d18c54507e6a..97244d4feca9c 100644
@@ -375,6 +375,23 @@ cpucap_multi_entry_cap_matches(const struct arm64_cpu_capabilities *entry,
 	return false;
 }
 
+static __always_inline bool is_vhe_hyp_code(void)
+{
+	/* Only defined for code run in VHE hyp context */
+	return __is_defined(__KVM_VHE_HYPERVISOR__);
+}
+
+static __always_inline bool is_nvhe_hyp_code(void)
+{
+	/* Only defined for code run in NVHE hyp context */
+	return __is_defined(__KVM_NVHE_HYPERVISOR__);
+}
+
+static __always_inline bool is_hyp_code(void)
+{
+	return is_vhe_hyp_code() || is_nvhe_hyp_code();
+}
+
 extern DECLARE_BITMAP(cpu_hwcaps, ARM64_NCAPS);
 extern struct static_key_false cpu_hwcap_keys[ARM64_NCAPS];
 extern struct static_key_false arm64_const_caps_ready;
@@ -444,8 +461,11 @@ static __always_inline bool cpus_have_final_cap(int num)
 }
 
 /*
- * Test for a capability, possibly with a runtime check.
+ * Test for a capability, possibly with a runtime check for non-hyp code.
  *
+ * For hyp code, this behaves the same as cpus_have_final_cap().
+ *
+ * For non-hyp code:
  * Before capabilities are finalized, this behaves as cpus_have_cap().
  * After capabilities are finalized, this is patched to avoid a runtime check.
  *
@@ -453,7 +473,9 @@ static __always_inline bool cpus_have_final_cap(int num)
  */
 static __always_inline bool cpus_have_const_cap(int num)
 {
-	if (system_capabilities_finalized())
+	if (is_hyp_code())
+		return cpus_have_final_cap(num);
+	else if (system_capabilities_finalized())
 		return __cpus_have_const_cap(num);
 	else
 		return cpus_have_cap(num);
arch/arm64/include/asm/virt.h +0 -12
index 300be14ba77b2..6069be50baf9f 100644
@@ -83,18 +83,6 @@ static inline bool is_kernel_in_hyp_mode(void)
 	return read_sysreg(CurrentEL) == CurrentEL_EL2;
 }
 
-static __always_inline bool is_vhe_hyp_code(void)
-{
-	/* Only defined for code run in VHE hyp context */
-	return __is_defined(__KVM_VHE_HYPERVISOR__);
-}
-
-static __always_inline bool is_nvhe_hyp_code(void)
-{
-	/* Only defined for code run in NVHE hyp context */
-	return __is_defined(__KVM_NVHE_HYPERVISOR__);
-}
-
 static __always_inline bool has_vhe(void)
 {
 	/*
arch/arm64/kernel/image-vars.h +0 -1
index 61684a5009148..c615b285ff5b3 100644
@@ -87,7 +87,6 @@ KVM_NVHE_ALIAS(__icache_flags);
 /* Kernel symbols needed for cpus_have_final/const_caps checks. */
 KVM_NVHE_ALIAS(arm64_const_caps_ready);
 KVM_NVHE_ALIAS(cpu_hwcap_keys);
-KVM_NVHE_ALIAS(cpu_hwcaps);
 
 /* Static keys which are set if a vGIC trap should be handled in hyp. */
 KVM_NVHE_ALIAS(vgic_v2_cpuif_trap);
Signature · 6 lines
2.11.0

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
maintainer Acked-by on v2 3/3
#8 lore raw

Re: [PATCHv2 3/3] arm64: cpufeature: upgrade hyp caps to final

Quoted 25 lines · Mark Rutland

We finalize caps before initializing kvm hyp code, and any use of cpus_have_const_cap() in kvm hyp code generates redundant and potentially unsound code to read the cpu_hwcaps array.

A number of helper functions used in both hyp context and regular kernel context use cpus_have_const_cap(), as some regular kernel code runs before the capabilities are finalized. It's tedious and error-prone to write separate copies of these for hyp and non-hyp code.

So that we can avoid the redundant code, let's automatically upgrade cpus_have_const_cap() to cpus_have_final_cap() when used in hyp context. With this change, there's never a reason to access to cpu_hwcaps array from hyp code, and we don't need to create an NVHE alias for this.

This should have no effect on non-hyp code.

Cc David Brazdil <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm64/include/asm/cpufeature.h | 26 ++++++++++++++++++++++++--
 arch/arm64/include/asm/virt.h       | 12 ------------
 arch/arm64/kernel/image-vars.h      |  1 -
 3 files changed, 24 insertions(+), 15 deletions(-)

[...]

arch/arm64/kernel/image-vars.h KVM_NVHE_ALIAS(__icache_flags); · line 90

Quoted 9 lines
arch/arm64/kernel/image-vars.h +0 -1
index 61684a5009148..c615b285ff5b3 100644
@@ -87,7 +87,6 @@ KVM_NVHE_ALIAS(__icache_flags);
 /* Kernel symbols needed for cpus_have_final/const_caps checks. */
 KVM_NVHE_ALIAS(arm64_const_caps_ready);
 KVM_NVHE_ALIAS(cpu_hwcap_keys);
-KVM_NVHE_ALIAS(cpu_hwcaps);

Nice!

Will
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

maintainer
#9 lore raw
In reply to Mark Rutland

[+ Catalin]

Quoted 25 lines · Mark Rutland

In a few places we use cpus_have_const_cap() in hyp code, usually because this is hidden within a helper that's also used in regular kernel context. As cpus_have_const_cap() generates code to read the cpu_hwcaps array before capabilities are finalized, this means we generate some potentially-unsound references to regular kernel VAs, but this these are redundant as capabilities are finalized before we initialize the kvm hyp code.

This series gets rid of the redundant code by automatically upgrading cpust_have_const_cap() to cpus_have_final_cap() when used in hyp code. This allows us to avoid creating an NVHE alias for the cpu_hwcaps array, so we can catch if we accidentally introduce an runtime reference to this (e.g. via cpus_have_cap()).

Since v1 [1]:
* Trivial rebase to v5.10-rc1

[1] https://lore.kernel.org/r/[email protected]

Mark Rutland (3):
arm64: kvm: factor out is_{vhe,nvhe}_hyp_code()
arm64: cpufeature: reorder cpus_have_{const,final}_cap()
arm64: cpufeature: upgrade hyp caps to final

Quoted 4 lines
 arch/arm64/include/asm/cpufeature.h | 40 ++++++++++++++++++++++++++++---------
 arch/arm64/include/asm/virt.h       |  9 ++++-----
 arch/arm64/kernel/image-vars.h      |  1 -
 3 files changed, 35 insertions(+), 15 deletions(-)

Catalin, Will: can I get an Ack for patches 2 and 3? I'd be quite happy to drop yet another reference from the nVHE object...

Thanks,

	M.
Signature · 5 lines
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
maintainer Applied
#10 lore raw
In reply to Mark Rutland
Quoted 9 lines · Mark Rutland

In a few places we use cpus_have_const_cap() in hyp code, usually because this is hidden within a helper that's also used in regular kernel context. As cpus_have_const_cap() generates code to read the cpu_hwcaps array before capabilities are finalized, this means we generate some potentially-unsound references to regular kernel VAs, but this these are redundant as capabilities are finalized before we initialize the kvm hyp code.

[...]

Applied to next, thanks!

[1/3] KVM: arm64: Factor out is_{vhe,nvhe}_hyp_code()

      commit: e9a33caec90e05673e2f7fb7c80f172031964d25
[2/3] arm64: cpufeature: reorder cpus_have_{const, final}_cap()
      commit: dfc4e3f08903ed8fe0b66cc25b64524a82654166
[3/3] arm64: cpufeature: upgrade hyp caps to final
      commit: d86de40decaa14e6613af1b2783bf4d589d0f38b

Cheers,

	M.
Signature · 7 lines
Without deviation from the norm, progress is not possible.


_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm