Skip to contentSkip to conversation

Enable Linux guests on Hyper-V on ARM64

PATCH Superseded Michael Kelley 3 participants 16 messages Mar 8, 2021 – Apr 5, 2021

Subsystems (6)

v9 of 12 Superseded Latest is v12 →

lore
Share & tools
lore
#0 lore raw

v9 cover Enable Linux guests on Hyper-V on ARM64

This series enables Linux guests running on Hyper-V on ARM64 hardware. New ARM64-specific code in arch/arm64/hyperv initializes Hyper-V, including its interrupts and hypercall mechanism. Existing architecture independent drivers for Hyper-V's VMbus and synthetic devices just work when built for ARM64. Hyper-V code is built and included in the image and modules only if CONFIG_HYPERV is enabled.

The seven patches are organized as follows:

1) Update include/linux/arm-smccc.h to provide an HVC wrapper variant that returns results in other than X0 thru X3

2) Add definitions and functions for making Hyper-V hypercalls and getting/setting virtual processor registers provided by Hyper-V

3) Add architecture specific definitions needed by the architecture independent Hyper-V clocksource driver in drivers/clocksource/hyperv_timer.c. Update the clocksource driver to be initialized on ARM64.

4) Add functions needed by the arch independent VMbus driver for reporting a panic to Hyper-V and as stubs for the kexec and crash handlers.

5) Add Hyper-V initialization code and utility functions that report Hyper-v status.

6) Export screen_info so it may be used by the Hyper-V frame buffer driver built as a module. It is already exported for x86, powerpc, and alpha architectures.

7) Make CONFIG_HYPERV selectable on ARM64 in addition to x86/x64.

Hyper-V on ARM64 runs with a 4 Kbyte page size, but allows guests with 4K/16K/64K page size. Linux guests with this ARM64 enablement code work with all three supported ARM64 page sizes.

The Hyper-V vPCI driver at drivers/pci/host/pci-hyperv.c has x86/x64-specific code and is not being built for ARM64. Fixing this driver to enable vPCI devices on ARM64 will be done later.

In a few cases, terminology from the x86/x64 world has been carried over into the ARM64 code ("MSR", "TSC"). Hyper-V still uses the x86/x64 terminology and has not replaced it with something more generic, so the code uses the Hyper-V terminology. This will be fixed when Hyper-V updates the usage in the TLFS.

This patch set is based on the hyperv-next branch of the code tree
https://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git/

Changes in v9:
* Added Patch 1 to enable making an SMCCC compliant hypercall that returns results in other than registers X0 thru X3, per version 1.2 and later of the SMCCC spec.
* Using the ability to return results in registers X6 and X7, converted hv_get_vpreg_128() to use a "fast" hypercall that passes inputs and outputs in registers, and in doing so eliminated a lot of memory allocation complexity.
* Cleaned up some extra blank lines and use of spaces in aligning local variables. [Sunil Muthuswamy]
* Based on discussion about future directions, reverted the population of hv_vp_index array to use a cpuhp state instead of a hypercall, which is like it was in v7 and earlier.

Changes in v8:
* Removed a lot of code based on refactoring the boundary between arch independent and arch dependent code for Hyper-V, per comments from Arnd Bergmann. The removed code was either duplicated on the x86 side, or has been folded into architecture independent code as not really being architecture dependent.
* Added config dependency on !CONFIG_CPU_BIG_ENDIAN [Arnd Bergmann]
* Reworked the approach to Hyper-V initialization. The functionality is the same, but is now structured like the Xen code with an early init function called in setup_arch() and an early initcall to finish the initialization. [Arnd Bergmann]

Changes in v7:
* Separately upstreamed split of hyperv-tlfs.h into arch dependent and independent versions. In this patch set, update the ARM64 hyperv-tlfs.h to include architecture independent definitions. This approach eliminates a lot of lines of otherwise duplicated code on the ARM64 side.
* Break ARM64 mshyperv.h into smaller pieces. Have an initial baseline, and add code along with patches for a particular functional area. [Marc Zyngier]
* In mshyperv.h, use static inline functions instead of #defines where possible. [Arnd Bergmann]
* Use VMbus INTID obtained from ACPI DSDT instead of hardcoding. The STIMER INTID is still hardcoded because it is needed before Linux has initialized the ACPI subsystem, so it can't be obtained from the DSDT. Wedging it into the GTDT seems dubious, so was not done. [Marc Zyngier]
* Update Hyper-V page size allocation functions to use alloc_page() if PAGE_SIZE == HV_HYP_PAGE_SIZE [Arnd Bergmann]
* Various other minor changes based on feedback and to rebase to latest linux-next [Marc Zyngier and Arnd Bergmann]

Changes in v6:
* Use SMCCC hypercall interface instead of direct invocation of HVC instruction and the Hyper-V hypercall interface [Marc Zyngier]
* Reimplemented functions to alloc/free Hyper-V size pages using kmalloc/kfree since kmalloc now guarantees alignment of power of 2 size allocations [Marc Zyngier]
* Export screen_info in arm64 architecture so it can be used by the Hyper-V buffer driver built as a module
* Renamed source file arch/arm64/hyperv/hv_init.c to hv_core.c to better reflect its content
* Fixed the bit position of certain feature flags presented by Hyper-V to the guest. The bit positions on ARM64 don't match the position on x86 like originally thought.
* Minor fixups to rebase to 5.6-rc5 linux-next

Changes in v5:
* Minor fixups to rebase to 5.4-rc1 linux-next

Changes in v4:
* Moved clock-related code into an architecture independent Hyper-V clocksource driver that is already upstream. Clock related code is removed from this patch set except for the ARM64 specific interrupt handler. [Marc Zyngier]
* Separately upstreamed the split of mshyperv.h into arch independent and arch dependent portions. The arch independent portion has been removed from this patch set.
* Divided patch #2 of the series into multiple smaller patches [Marc Zyngier]
* Changed a dozen or so smaller things based on feedback [Marc Zyngier, Will Deacon]
* Added functions to alloc/free Hyper-V size pages for use by drivers for Hyper-V synthetic devices when updated to not assume guest page size and Hyper-v page size are the same

Changes in v3:
* Added initialization of hv_vp_index array like was recently added on x86 branch [KY Srinivasan]
* Changed Hyper-V ARM64 register symbols to be all uppercase instead of mixed case [KY Srinivasan]
* Separated mshyperv.h into two files, one architecture independent and one architecture dependent. After this code is upstream, will make changes to the x86 code to use the architecture independent file and remove duplication. And once we have a multi-architecture Hyper-V TLFS, will do a separate patch to split hyperv-tlfs.h in the same way. [KY Srinivasan]
* Minor tweaks to rebase to latest linux-next code

Changes in v2:
* Removed patch to implement slow_virt_to_phys() on ARM64. Use of slow_virt_to_phys() in arch independent Hyper-V drivers has been eliminated by commit 6ba34171bcbd ("Drivers: hv: vmbus: Remove use of slow_virt_to_phys()")
* Minor tweaks to rebase to latest linux-next code *** BLURB HERE ***

Michael Kelley (7):
smccc: Add HVC call variant with result registers other than 0 thru 3
arm64: hyperv: Add Hyper-V hypercall and register access utilities
arm64: hyperv: Add Hyper-V clocksource/clockevent support
arm64: hyperv: Add kexec and panic handlers
arm64: hyperv: Initialize hypervisor on boot
arm64: efi: Export screen_info
Drivers: hv: Enable Hyper-V code to be built on ARM64

 MAINTAINERS                          |   3 +
 arch/arm64/Kbuild                    |   1 +
 arch/arm64/hyperv/Makefile           |   2 +
 arch/arm64/hyperv/hv_core.c          | 178 +++++++++++++++++++++++++++++++++++
 arch/arm64/hyperv/mshyperv.c         | 173 ++++++++++++++++++++++++++++++++++
 arch/arm64/include/asm/hyperv-tlfs.h |  69 ++++++++++++++
 arch/arm64/include/asm/mshyperv.h    |  72 ++++++++++++++
 arch/arm64/kernel/efi.c              |   1 +
 arch/arm64/kernel/setup.c            |   4 +
 drivers/clocksource/hyperv_timer.c   |  14 +++
 drivers/hv/Kconfig                   |   3 +-
 include/linux/arm-smccc.h            |  29 ++++--
 12 files changed, 542 insertions(+), 7 deletions(-)
 create mode 100644 arch/arm64/hyperv/Makefile
 create mode 100644 arch/arm64/hyperv/hv_core.c
 create mode 100644 arch/arm64/hyperv/mshyperv.c
 create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
 create mode 100644 arch/arm64/include/asm/mshyperv.h
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH v9 1/7 smccc: Add HVC call variant with result registers other than 0 thru 3

Hypercalls to Hyper-V on ARM64 may return results in registers other than X0 thru X3, as permitted by the SMCCC spec version 1.2 and later. Accommodate this by adding a variant of arm_smccc_1_1_hvc that allows the caller to specify which 3 registers are returned in addition to X0.

---
There are several ways to support returning results from registers other than X0 thru X3, and Hyper-V usage should be compatible with whatever the maintainers prefer. What's implemented in this patch may be the most flexible, but it has the downside of not being a true function interface in that args 0 thru 2 must be fixed strings, and not general "C" expressions.

Other alternatives include:
* Create a variant that hard codes to return X5 thru X7, though in the future there may be Hyper-V hypercalls that need a different hard-coded variant.
* Return all of X0 thru X7 in a larger result structure. That approach may execute more memory stores, but performance is unlikely to be an issue for the Hyper-V hypercalls that would use it. However, it's possible in the future that Hyper-V results might be beyond X7, as allowed by the SMCCC v1.3 spec.
* The macro __arm_smccc_1_1() could be cloned in Hyper-V specific code and modified to meet Hyper-V specific needs, but this seems undesirable since Hyper-V is operating within the v1.2 spec.

In any of these cases, the call might be renamed from "_1_1_" to "_1_2_" to reflect conformance to the later spec version.

 include/linux/arm-smccc.h | 29 +++++++++++++++++++++++------
 1 file changed, 23 insertions(+), 6 deletions(-)
include/linux/arm-smccc.h +23 -6
index f860645..acda958 100644
@@ -300,12 +300,12 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1,
  * entitled to optimise the whole sequence away. "volatile" is what
  * makes it stick.
  */
-#define __arm_smccc_1_1(inst, ...)					\
+#define __arm_smccc_1_1(inst, reg1, reg2, reg3, ...)			\
 	do {								\
 		register unsigned long r0 asm("r0");			\
-		register unsigned long r1 asm("r1");			\
-		register unsigned long r2 asm("r2");			\
-		register unsigned long r3 asm("r3"); 			\
+		register unsigned long r1 asm(reg1);			\
+		register unsigned long r2 asm(reg2);			\
+		register unsigned long r3 asm(reg3);			\
 		__declare_args(__count_args(__VA_ARGS__), __VA_ARGS__);	\
 		asm volatile(inst "\n" :				\
 			     "=r" (r0), "=r" (r1), "=r" (r2), "=r" (r3)	\
@@ -328,7 +328,8 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1,
  * to the SMC instruction. The return values are updated with the content
  * from register 0 to 3 on return from the SMC instruction if not NULL.
  */
-#define arm_smccc_1_1_smc(...)	__arm_smccc_1_1(SMCCC_SMC_INST, __VA_ARGS__)
+#define arm_smccc_1_1_smc(...)\
+	__arm_smccc_1_1(SMCCC_SMC_INST, "r1", "r2", "r3", __VA_ARGS__)
 
 /*
  * arm_smccc_1_1_hvc() - make an SMCCC v1.1 compliant HVC call
@@ -344,7 +345,23 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1,
  * to the HVC instruction. The return values are updated with the content
  * from register 0 to 3 on return from the HVC instruction if not NULL.
  */
-#define arm_smccc_1_1_hvc(...)	__arm_smccc_1_1(SMCCC_HVC_INST, __VA_ARGS__)
+#define arm_smccc_1_1_hvc(...) \
+	__arm_smccc_1_1(SMCCC_HVC_INST, "r1", "r2", "r3", __VA_ARGS__)
+
+/*
+ * arm_smccc_1_1_hvc_reg() - make an SMCCC v1.1 compliant HVC call
+ * specifying output registers
+ *
+ * This is a variant of arm_smccc_1_1_hvc() that allows specifying
+ * three registers from which result values will be returned in
+ * addition to r0.
+ *
+ * @a0-a2: register specifications for 3 return registers (e.g., "r5")
+ * @a3-a10: arguments passed in registers 0 to 7
+ * @res: result values from register 0 and the three registers specified
+ * in a0-a2.
+ */
+#define arm_smccc_1_1_hvc_reg(...) __arm_smccc_1_1(SMCCC_HVC_INST, __VA_ARGS__)
 
 /*
  * Like arm_smccc_1_1* but always returns SMCCC_RET_NOT_SUPPORTED.
Signature · 1 line
1.8.3.1
maintainer
#2 lore raw

Re: [PATCH v9 1/7] smccc: Add HVC call variant with result registers other than 0 thru 3

Hi Michael,

Quoted 5 lines · Michael Kelley

Hypercalls to Hyper-V on ARM64 may return results in registers other than X0 thru X3, as permitted by the SMCCC spec version 1.2 and later. Accommodate this by adding a variant of arm_smccc_1_1_hvc that allows the caller to specify which 3 registers are returned in addition to X0.

Quoted 8 lines

---
There are several ways to support returning results from registers other than X0 thru X3, and Hyper-V usage should be compatible with whatever the maintainers prefer. What's implemented in this patch may be the most flexible, but it has the downside of not being a true function interface in that args 0 thru 2 must be fixed strings, and not general "C" expressions.

For the benefit of others here, SMCCCv1.2 allows:

* SMC64/HVC64 to use all of x1-x17 for both parameters and return values
* SMC32/HVC32 to use all of r1-r7 for both parameters and return values

The rationale for this was to make it possible to pass a large number of arguments in one call without the hypervisor/firmware needing to access the memory of the caller.

My preference would be to add arm_smccc_1_2_{hvc,smc}() assembly functions which read all the permitted argument registers from a struct, and write all the permitted result registers to a struct, leaving it to callers to set those up and decompose them.

That way we only have to write one implementation that all callers can use, which'll be far easier to maintain. I suspect that in general the cost of temporarily bouncing the values through memory will be dominated by whatever the hypervisor/firmware is going to do, and if it's not we can optimize that away in future.

Quoted 9 lines

Other alternatives include:
* Create a variant that hard codes to return X5 thru X7, though in the future there may be Hyper-V hypercalls that need a different hard-coded variant.
* Return all of X0 thru X7 in a larger result structure. That approach may execute more memory stores, but performance is unlikely to be an issue for the Hyper-V hypercalls that would use it. However, it's possible in the future that Hyper-V results might be beyond X7, as allowed by the SMCCC v1.3 spec.

As above, something of this sort would be my preferred approach.

Thanks,
Mark.

Quoted 70 lines

* The macro __arm_smccc_1_1() could be cloned in Hyper-V specific code and modified to meet Hyper-V specific needs, but this seems undesirable since Hyper-V is operating within the v1.2 spec.

In any of these cases, the call might be renamed from "_1_1_" to "_1_2_" to reflect conformance to the later spec version.

 include/linux/arm-smccc.h | 29 +++++++++++++++++++++++------
 1 file changed, 23 insertions(+), 6 deletions(-)
include/linux/arm-smccc.h +23 -6
index f860645..acda958 100644
@@ -300,12 +300,12 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1,
  * entitled to optimise the whole sequence away. "volatile" is what
  * makes it stick.
  */
-#define __arm_smccc_1_1(inst, ...)					\
+#define __arm_smccc_1_1(inst, reg1, reg2, reg3, ...)			\
 	do {								\
 		register unsigned long r0 asm("r0");			\
-		register unsigned long r1 asm("r1");			\
-		register unsigned long r2 asm("r2");			\
-		register unsigned long r3 asm("r3"); 			\
+		register unsigned long r1 asm(reg1);			\
+		register unsigned long r2 asm(reg2);			\
+		register unsigned long r3 asm(reg3);			\
 		__declare_args(__count_args(__VA_ARGS__), __VA_ARGS__);	\
 		asm volatile(inst "\n" :				\
 			     "=r" (r0), "=r" (r1), "=r" (r2), "=r" (r3)	\
@@ -328,7 +328,8 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1,
  * to the SMC instruction. The return values are updated with the content
  * from register 0 to 3 on return from the SMC instruction if not NULL.
  */
-#define arm_smccc_1_1_smc(...)	__arm_smccc_1_1(SMCCC_SMC_INST, __VA_ARGS__)
+#define arm_smccc_1_1_smc(...)\
+	__arm_smccc_1_1(SMCCC_SMC_INST, "r1", "r2", "r3", __VA_ARGS__)
 
 /*
  * arm_smccc_1_1_hvc() - make an SMCCC v1.1 compliant HVC call
@@ -344,7 +345,23 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1,
  * to the HVC instruction. The return values are updated with the content
  * from register 0 to 3 on return from the HVC instruction if not NULL.
  */
-#define arm_smccc_1_1_hvc(...)	__arm_smccc_1_1(SMCCC_HVC_INST, __VA_ARGS__)
+#define arm_smccc_1_1_hvc(...) \
+	__arm_smccc_1_1(SMCCC_HVC_INST, "r1", "r2", "r3", __VA_ARGS__)
+
+/*
+ * arm_smccc_1_1_hvc_reg() - make an SMCCC v1.1 compliant HVC call
+ * specifying output registers
+ *
+ * This is a variant of arm_smccc_1_1_hvc() that allows specifying
+ * three registers from which result values will be returned in
+ * addition to r0.
+ *
+ * @a0-a2: register specifications for 3 return registers (e.g., "r5")
+ * @a3-a10: arguments passed in registers 0 to 7
+ * @res: result values from register 0 and the three registers specified
+ * in a0-a2.
+ */
+#define arm_smccc_1_1_hvc_reg(...) __arm_smccc_1_1(SMCCC_HVC_INST, __VA_ARGS__)
 
 /*
  * Like arm_smccc_1_1* but always returns SMCCC_RET_NOT_SUPPORTED.
Signature · 1 line
1.8.3.1
#3 lore raw

RE: [PATCH v9 1/7] smccc: Add HVC call variant with result registers other than 0 thru 3

From: Mark Rutland <[email protected]> Sent: Wednesday, March 24, 2021 9:55 AM

Quoted 32 lines

Hi Michael,

Quoted 5 lines · Michael Kelley

Hypercalls to Hyper-V on ARM64 may return results in registers other than X0 thru X3, as permitted by the SMCCC spec version 1.2 and later. Accommodate this by adding a variant of arm_smccc_1_1_hvc that allows the caller to specify which 3 registers are returned in addition to X0.

Quoted 8 lines

---
There are several ways to support returning results from registers other than X0 thru X3, and Hyper-V usage should be compatible with whatever the maintainers prefer. What's implemented in this patch may be the most flexible, but it has the downside of not being a true function interface in that args 0 thru 2 must be fixed strings, and not general "C" expressions.

For the benefit of others here, SMCCCv1.2 allows:

* SMC64/HVC64 to use all of x1-x17 for both parameters and return values
* SMC32/HVC32 to use all of r1-r7 for both parameters and return values

The rationale for this was to make it possible to pass a large number of arguments in one call without the hypervisor/firmware needing to access the memory of the caller.

My preference would be to add arm_smccc_1_2_{hvc,smc}() assembly functions which read all the permitted argument registers from a struct, and write all the permitted result registers to a struct, leaving it to callers to set those up and decompose them.

Quoted 6 lines

That way we only have to write one implementation that all callers can use, which'll be far easier to maintain. I suspect that in general the cost of temporarily bouncing the values through memory will be dominated by whatever the hypervisor/firmware is going to do, and if it's not we can optimize that away in future.

Thanks for the feedback, and I'm working on implementing this approach. But I've hit a snag in that gcc limits the "asm" statement to 30 arguments, which gives us 15 registers as parameters and 15 registers as return values, instead of the 18 each allowed by SMCCC v1.2. I will continue with the 15 register limit for now, unless someone knows a way to exceed that. The alternative would be to go to pure assembly language.

I'll post a standalone RFC patch when I have something that works. My C pre-processor wizardry is limited, so others will probably know some tricks that can improve on my first cut.

Michael

maintainer
#4 lore raw

Re: [PATCH v9 1/7] smccc: Add HVC call variant with result registers other than 0 thru 3

Quoted 21 lines · Michael Kelley

From: Mark Rutland <[email protected]> Sent: Wednesday, March 24, 2021 9:55 AM

Quoted 14 lines

For the benefit of others here, SMCCCv1.2 allows:

* SMC64/HVC64 to use all of x1-x17 for both parameters and return values
* SMC32/HVC32 to use all of r1-r7 for both parameters and return values

The rationale for this was to make it possible to pass a large number of arguments in one call without the hypervisor/firmware needing to access the memory of the caller.

My preference would be to add arm_smccc_1_2_{hvc,smc}() assembly functions which read all the permitted argument registers from a struct, and write all the permitted result registers to a struct, leaving it to callers to set those up and decompose them.

Quoted 5 lines

That way we only have to write one implementation that all callers can use, which'll be far easier to maintain. I suspect that in general the cost of temporarily bouncing the values through memory will be dominated by whatever the hypervisor/firmware is going to do, and if it's not we can optimize that away in future.

Quoted 6 lines

Thanks for the feedback, and I'm working on implementing this approach. But I've hit a snag in that gcc limits the "asm" statement to 30 arguments, which gives us 15 registers as parameters and 15 registers as return values, instead of the 18 each allowed by SMCCC v1.2. I will continue with the 15 register limit for now, unless someone knows a way to exceed that. The alternative would be to go to pure assembly language.

I realise in retrospect this is not clear, but when I said "assembly functions" I had meant raw assembly functions rather than inline assembly.

We already have __arm_smccc_smc and __arm_smccc_hvc assembly functions in arch/{arm,arm64}/kernel/smccc-call.S, and I'd expected we'd add the full fat SMCCCv1.2 variants there.

Thanks,
Mark.

#5 lore raw

RE: [PATCH v9 1/7] smccc: Add HVC call variant with result registers other than 0 thru 3

From: Mark Rutland <[email protected]> Sent: Thursday, March 25, 2021 2:56 AM

Quoted 30 lines · Michael Kelley
Quoted 21 lines

From: Mark Rutland <[email protected]> Sent: Wednesday, March 24, 2021 9:55 AM

Quoted 14 lines

For the benefit of others here, SMCCCv1.2 allows:

* SMC64/HVC64 to use all of x1-x17 for both parameters and return values
* SMC32/HVC32 to use all of r1-r7 for both parameters and return values

The rationale for this was to make it possible to pass a large number of arguments in one call without the hypervisor/firmware needing to access the memory of the caller.

My preference would be to add arm_smccc_1_2_{hvc,smc}() assembly functions which read all the permitted argument registers from a struct, and write all the permitted result registers to a struct, leaving it to callers to set those up and decompose them.

Quoted 5 lines

That way we only have to write one implementation that all callers can use, which'll be far easier to maintain. I suspect that in general the cost of temporarily bouncing the values through memory will be dominated by whatever the hypervisor/firmware is going to do, and if it's not we can optimize that away in future.

Quoted 6 lines

Thanks for the feedback, and I'm working on implementing this approach. But I've hit a snag in that gcc limits the "asm" statement to 30 arguments, which gives us 15 registers as parameters and 15 registers as return values, instead of the 18 each allowed by SMCCC v1.2. I will continue with the 15 register limit for now, unless someone knows a way to exceed that. The alternative would be to go to pure assembly language.

Quoted 8 lines

I realise in retrospect this is not clear, but when I said "assembly functions" I had meant raw assembly functions rather than inline assembly.

We already have __arm_smccc_smc and __arm_smccc_hvc assembly functions in arch/{arm,arm64}/kernel/smccc-call.S, and I'd expected we'd add the full fat SMCCCv1.2 variants there.

FWIW, here's an inline assembly version that I have working. On the plus side, gcc does a decent job of optimizing. It doesn't store to memory any result registers that aren't consumed by the caller. On the downside, it's limited to 15 args and 15 results as noted previously. So it doesn't meet your goal of fully implementing the v1.2 spec. Also, all 15 input arguments must be initialized or gcc complains about using uninitialized values.

This version should handle both the 32-bit and 64-bit worlds, though I've only tested in the 64-bit world.

I've made the input and output structures be arrays rather than listing each register as a separate field. Either approach should work, and I'm not sure what the tradeoffs are.

But if building on what Sudeep Holla has already done with raw assembly is preferred, I'm OK with that as well.

Michael

include/linux/arm-smccc.h +107 -0
index acda958..e98cf07 100644
@@ -408,5 +408,112 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1,
 		method;							\
 	})
 
+
+#ifdef CONFIG_ARM64
+#define SMCCC_1_2_REG_COUNT	18
+
+#define ARM64_ADDITIONAL_RESULT_DECLARATIONS				\
+		register unsigned long r8 asm("r8");			\
+		register unsigned long r9 asm("r9");			\
+		register unsigned long r10 asm("r10");			\
+		register unsigned long r11 asm("r11");			\
+		register unsigned long r12 asm("r12");			\
+		register unsigned long r13 asm("r13");			\
+		register unsigned long r14 asm("r14");
+
+#define ARM64_ADDITIONAL_ARG_DECLARATIONS				\
+		register unsigned long arg8 asm("r8") = __a->reg[8];	\
+		register unsigned long arg9 asm("r9") = __a->reg[9];	\
+		register unsigned long arg10 asm("r10") = __a->reg[10];	\
+		register unsigned long arg11 asm("r11") = __a->reg[11];	\
+		register unsigned long arg12 asm("r12") = __a->reg[12];	\
+		register unsigned long arg13 asm("r13") = __a->reg[13];	\
+		register unsigned long arg14 asm("r14") = __a->reg[14];
+
+#define ARM64_ADDITIONAL_OUTPUT_REGS					\
+			, "=r" (r8), "=r" (r9), "=r" (r10), "=r" (r11),	\
+			"=r" (r12), "=r" (r13), "=r" (r14)
+
+#define ARM64_ADDITIONAL_INPUT_REGS					\
+			, "r" (arg8), "r" (arg9), "r" (arg10),		\
+			"r" (arg11), "r" (arg12), "r" (arg13),		\
+			"r" (arg14)
+
+#define ARM64_ADDITIONAL_RESULTS					\
+			__r->reg[8] = r8;				\
+			__r->reg[9] = r9;				\
+			__r->reg[10] = r10;				\
+			__r->reg[11] = r11;				\
+			__r->reg[12] = r12;				\
+			__r->reg[13] = r13;				\
+			__r->reg[14] = r14;
+
+#else /* CONFIG_ARM64 */
+
+#define SMCCC_1_2_REG_COUNT	8
+#define ARM64_ADDITIONAL_RESULT_DECLARATIONS
+#define ARM64_ADDITIONAL_ARG_DECLARATIONS
+#define ARM64_ADDITIONAL_OUTPUT_REGS
+#define ARM64_ADDITIONAL_INPUT_REGS
+#define ARM64_ADDITIONAL_RESULTS
+
+#endif /* CONFIG_ARM64 */
+
+struct arm_smccc_1_2_regs {
+	unsigned long reg[SMCCC_1_2_REG_COUNT];
+};
+
+
+#define __arm_smccc_1_2(inst, args, res)				\
+	do {								\
+		struct arm_smccc_1_2_regs * __a = args;			\
+		struct arm_smccc_1_2_regs * __r = res;			\
+		register unsigned long r0 asm("r0");			\
+		register unsigned long r1 asm("r1");			\
+		register unsigned long r2 asm("r2");			\
+		register unsigned long r3 asm("r3");			\
+		register unsigned long r4 asm("r4");			\
+		register unsigned long r5 asm("r5");			\
+		register unsigned long r6 asm("r6");			\
+		register unsigned long r7 asm("r7");			\
+		ARM64_ADDITIONAL_RESULT_DECLARATIONS			\
+		register unsigned long arg0 asm("r0") = (u32)(__a->reg[0]); \
+		register unsigned long arg1 asm("r1") = __a->reg[1];	\
+		register unsigned long arg2 asm("r2") = __a->reg[2];	\
+		register unsigned long arg3 asm("r3") = __a->reg[3];	\
+		register unsigned long arg4 asm("r4") = __a->reg[4];	\
+		register unsigned long arg5 asm("r5") = __a->reg[5];	\
+		register unsigned long arg6 asm("r6") = __a->reg[6];	\
+		register unsigned long arg7 asm("r7") = __a->reg[7];	\
+		ARM64_ADDITIONAL_ARG_DECLARATIONS			\
+		asm volatile(inst "\n" :				\
+			"=r" (r0), "=r" (r1), "=r" (r2), "=r" (r3),	\
+			"=r" (r4), "=r" (r5), "=r" (r6), "=r" (r7)	\
+			ARM64_ADDITIONAL_OUTPUT_REGS :			\
+			"r" (arg0), "r" (arg1), "r" (arg2),		\
+			"r" (arg3), "r" (arg4), "r" (arg5),		\
+			"r" (arg6), "r" (arg7)				\
+			ARM64_ADDITIONAL_INPUT_REGS :			\
+			"memory");					\
+		if(__r) {						\
+			__r->reg[0] = r0;				\
+			__r->reg[1] = r1;				\
+			__r->reg[2] = r2;				\
+			__r->reg[3] = r3;				\
+			__r->reg[4] = r4;				\
+			__r->reg[5] = r5;				\
+			__r->reg[6] = r6;				\
+			__r->reg[7] = r7;				\
+			ARM64_ADDITIONAL_RESULTS			\
+		}							\
+	} while (0)
+
+#define arm_smccc_1_2_smc(args, res)					\
+	__arm_smccc_1_2(SMCCC_SMC_INST, args, res)
+
+#define arm_smccc_1_2_hvc(args, res)					\
+	__arm_smccc_1_2(SMCCC_HVC_INST, args, res)
+
+
 #endif /*__ASSEMBLY__*/
 #endif /*__LINUX_ARM_SMCCC_H*/
#6 lore raw

PATCH v9 2/7 arm64: hyperv: Add Hyper-V hypercall and register access utilities

In reply to Michael Kelley

hyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level Functional Spec (TLFS), and #includes the architecture-independent part of hyperv-tlfs.h in include/asm-generic. The published TLFS is distinctly oriented to x86/x64, so the ARM64-specific hyperv-tlfs.h includes information for ARM64 that is not yet formally published. The TLFS is available here:

docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs

mshyperv.h defines Linux-specific structures and routines for interacting with Hyper-V on ARM64, and #includes the architecture- independent part of mshyperv.h in include/asm-generic.

Use these definitions to provide utility functions to make Hyper-V hypercalls and to get and set Hyper-V provided registers associated with a virtual processor.

---
 MAINTAINERS                          |   3 +
 arch/arm64/Kbuild                    |   1 +
 arch/arm64/hyperv/Makefile           |   2 +
 arch/arm64/hyperv/hv_core.c          | 126 +++++++++++++++++++++++++++++++++++
 arch/arm64/include/asm/hyperv-tlfs.h |  69 +++++++++++++++++++
 arch/arm64/include/asm/mshyperv.h    |  54 +++++++++++++++
 6 files changed, 255 insertions(+)
 create mode 100644 arch/arm64/hyperv/Makefile
 create mode 100644 arch/arm64/hyperv/hv_core.c
 create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
 create mode 100644 arch/arm64/include/asm/mshyperv.h
MAINTAINERS +3 -0
index bfc1b86..7a47a90 100644
@@ -8232,6 +8232,9 @@ T:	git git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git
 F:	Documentation/ABI/stable/sysfs-bus-vmbus
 F:	Documentation/ABI/testing/debugfs-hyperv
 F:	Documentation/networking/device_drivers/ethernet/microsoft/netvsc.rst
+F:	arch/arm64/hyperv
+F:	arch/arm64/include/asm/hyperv-tlfs.h
+F:	arch/arm64/include/asm/mshyperv.h
 F:	arch/x86/hyperv
 F:	arch/x86/include/asm/hyperv-tlfs.h
 F:	arch/x86/include/asm/mshyperv.h
arch/arm64/Kbuild +1 -0
index d646582..7a37608 100644
@@ -3,4 +3,5 @@ obj-y			+= kernel/ mm/
 obj-$(CONFIG_NET)	+= net/
 obj-$(CONFIG_KVM)	+= kvm/
 obj-$(CONFIG_XEN)	+= xen/
+obj-$(subst m,y,$(CONFIG_HYPERV))	+= hyperv/
 obj-$(CONFIG_CRYPTO)	+= crypto/
arch/arm64/hyperv/Makefile new file +2 -0
new file mode 100644
index 0000000..1697d30
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-y		:= hv_core.o
arch/arm64/hyperv/hv_core.c new file +126 -0

126 lines not on this page · raw message

arch/arm64/include/asm/hyperv-tlfs.h new file +69 -0

69 lines not on this page · raw message

arch/arm64/include/asm/mshyperv.h new file +54 -0

54 lines not on this page · raw message

Signature · 1 line
1.8.3.1
Reviewed-by on v9 2/7
#7 lore raw

RE: [PATCH v9 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities

Quoted 17 lines
arch/arm64/include/asm/mshyperv.h new file +16 -1
+static inline u64 hv_get_register(unsigned int reg)
+{
+	return hv_get_vpreg(reg);
+}
+
+/* SMCCC hypercall parameters */
+#define HV_SMCCC_FUNC_NUMBER	1
+#define HV_FUNC_ID	ARM_SMCCC_CALL_VAL(			\
+				ARM_SMCCC_STD_CALL,		\
+				ARM_SMCCC_SMC_64,		\
+				ARM_SMCCC_OWNER_VENDOR_HYP,	\
+				HV_SMCCC_FUNC_NUMBER)
+
+#include <asm-generic/mshyperv.h>
+
+#endif
--
Quoted 1 line

1.8.3.1

#8 lore raw

PATCH v9 3/7 arm64: hyperv: Add Hyper-V clocksource/clockevent support

In reply to Michael Kelley

Add architecture specific definitions and functions needed by the architecture independent Hyper-V clocksource driver. Update the Hyper-V clocksource driver to be initialized on ARM64.

---
 arch/arm64/include/asm/mshyperv.h  | 12 ++++++++++++
 drivers/clocksource/hyperv_timer.c | 14 ++++++++++++++
 2 files changed, 26 insertions(+)
arch/arm64/include/asm/mshyperv.h +12 -0
index c448704..b17299c 100644
@@ -21,6 +21,7 @@
 #include <linux/types.h>
 #include <linux/arm-smccc.h>
 #include <asm/hyperv-tlfs.h>
+#include <clocksource/arm_arch_timer.h>
 
 /*
  * Declare calls to get and set Hyper-V VP register values on ARM64, which
@@ -41,6 +42,17 @@ static inline u64 hv_get_register(unsigned int reg)
 	return hv_get_vpreg(reg);
 }
 
+/* Define the interrupt ID used by STIMER0 Direct Mode interrupts. This
+ * value can't come from ACPI tables because it is needed before the
+ * Linux ACPI subsystem is initialized.
+ */
+#define HYPERV_STIMER0_VECTOR	31
+
+static inline u64 hv_get_raw_timer(void)
+{
+	return arch_timer_read_counter();
+}
+
 /* SMCCC hypercall parameters */
 #define HV_SMCCC_FUNC_NUMBER	1
 #define HV_FUNC_ID	ARM_SMCCC_CALL_VAL(			\
drivers/clocksource/hyperv_timer.c +14 -0
index ba04cb3..044efb1 100644
@@ -473,3 +473,17 @@ void __init hv_init_clocksource(void)
 	hv_setup_sched_clock(read_hv_sched_clock_msr);
 }
 EXPORT_SYMBOL_GPL(hv_init_clocksource);
+
+/* Initialize everything on ARM64 */
+static int __init hyperv_timer_init(struct acpi_table_header *table)
+{
+	if (!hv_is_hyperv_initialized())
+		return -EINVAL;
+
+	hv_init_clocksource();
+	if (hv_stimer_alloc(true))
+		return -EINVAL;
+
+	return 0;
+}
+TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_timer_init);
Signature · 1 line
1.8.3.1
#9 lore raw

PATCH v9 4/7 arm64: hyperv: Add kexec and panic handlers

In reply to Michael Kelley

Add function to inform Hyper-V about a guest panic.

Also add functions to set up and remove kexec and panic handlers, which are currently unused on ARM64 but are called from architecture independent code in the VMbus driver.

This code is built only when CONFIG_HYPERV is enabled.

---
 arch/arm64/hyperv/Makefile   |  2 +-
 arch/arm64/hyperv/hv_core.c  | 52 ++++++++++++++++++++++++++++++++++++++++++
 arch/arm64/hyperv/mshyperv.c | 54 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 107 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/hyperv/mshyperv.c
arch/arm64/hyperv/Makefile +1 -1
index 1697d30..87c31c0 100644
@@ -1,2 +1,2 @@
 # SPDX-License-Identifier: GPL-2.0
-obj-y		:= hv_core.o
+obj-y		:= hv_core.o mshyperv.o
arch/arm64/hyperv/hv_core.c +52 -0

55 lines not on this page · raw message

arch/arm64/hyperv/mshyperv.c new file +54 -0

54 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#10 lore raw

PATCH v9 5/7 arm64: hyperv: Initialize hypervisor on boot

In reply to Michael Kelley

Add ARM64-specific code to initialize the Hyper-V hypervisor when booting as a guest VM. Provide functions and data structures indicating hypervisor status that are needed by VMbus driver.

This code is built only when CONFIG_HYPERV is enabled.

---
 arch/arm64/hyperv/mshyperv.c      | 119 ++++++++++++++++++++++++++++++++++++++
 arch/arm64/include/asm/mshyperv.h |   6 ++
 arch/arm64/kernel/setup.c         |   4 ++
 3 files changed, 129 insertions(+)
arch/arm64/hyperv/mshyperv.c +119 -0
index d202b4c..125e83e 100644
@@ -14,6 +14,125 @@
 #include <linux/types.h>
 #include <linux/export.h>
 #include <linux/ptrace.h>
+#include <linux/errno.h>
+#include <linux/acpi.h>
+#include <linux/version.h>
+#include <linux/cpuhotplug.h>
+#include <linux/slab.h>
+#include <linux/cpumask.h>
+#include <asm/mshyperv.h>
+
+static bool		hyperv_initialized;
+struct ms_hyperv_info	ms_hyperv __ro_after_init;
+EXPORT_SYMBOL_GPL(ms_hyperv);
+
+u32	*hv_vp_index;
+EXPORT_SYMBOL_GPL(hv_vp_index);
+
+u32	hv_max_vp_index;
+EXPORT_SYMBOL_GPL(hv_max_vp_index);
+
+static int hv_cpu_init(unsigned int cpu)
+{
+	hv_vp_index[cpu] = hv_get_vpreg(HV_REGISTER_VP_INDEX);
+	return 0;
+}
+
+void __init hyperv_early_init(void)
+{
+	struct hv_get_vp_registers_output	result;
+	u32	a, b, c, d;
+	u64	guest_id;
+
+	/*
+	 * If we're in a VM on Hyper-V, the ACPI hypervisor_id field will
+	 * have the string "MsHyperV".
+	 */
+	if (strncmp((char *)&acpi_gbl_FADT.hypervisor_id, "MsHyperV", 8))
+		return;
+
+	/* Setup the guest ID */
+	guest_id = generate_guest_id(0, LINUX_VERSION_CODE, 0);
+	hv_set_vpreg(HV_REGISTER_GUEST_OSID, guest_id);
+
+	/* Get the features and hints from Hyper-V */
+	hv_get_vpreg_128(HV_REGISTER_FEATURES, &result);
+	ms_hyperv.features = result.as32.a;
+	ms_hyperv.misc_features = result.as32.c;
+
+	hv_get_vpreg_128(HV_REGISTER_ENLIGHTENMENTS, &result);
+	ms_hyperv.hints = result.as32.a;
+
+	pr_info("Hyper-V: Features 0x%x, hints 0x%x, misc 0x%x\n",
+		ms_hyperv.features, ms_hyperv.hints, ms_hyperv.misc_features);
+
+	/*
+	 * If Hyper-V has crash notifications, set crash_kexec_post_notifiers
+	 * so that we will report the panic to Hyper-V before running kdump.
+	 */
+	if (ms_hyperv.misc_features & HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE)
+		crash_kexec_post_notifiers = true;
+
+	/* Get information about the Hyper-V host version */
+	hv_get_vpreg_128(HV_REGISTER_HYPERVISOR_VERSION, &result);
+	a = result.as32.a;
+	b = result.as32.b;
+	c = result.as32.c;
+	d = result.as32.d;
+	pr_info("Hyper-V: Host Build %d.%d.%d.%d-%d-%d\n",
+		b >> 16, b & 0xFFFF, a,	d & 0xFFFFFF, c, d >> 24);
+
+	hyperv_initialized = true;
+}
+
+static int __init hyperv_init(void)
+{
+	int	i;
+
+	/* Allocate and initialize percpu VP index array */
+	hv_max_vp_index = num_possible_cpus();
+	hv_vp_index = kmalloc_array(hv_max_vp_index, sizeof(*hv_vp_index),
+				    GFP_KERNEL);
+	if (!hv_vp_index) {
+		hv_max_vp_index = 0;
+		return -ENOMEM;
+	}
+
+	for (i = 0; i < hv_max_vp_index; i++)
+		hv_vp_index[i] = VP_INVAL;
+
+	if (cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "arm64/hyperv_init:online",
+					hv_cpu_init, NULL) < 0) {
+		hv_max_vp_index = 0;
+		kfree(hv_vp_index);
+		hv_vp_index = NULL;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+early_initcall(hyperv_init);
+
+/* This routine is called before kexec/kdump. It does required cleanup. */
+void hyperv_cleanup(void)
+{
+	hv_set_vpreg(HV_REGISTER_GUEST_OSID, 0);
+
+}
+EXPORT_SYMBOL_GPL(hyperv_cleanup);
+
+bool hv_is_hyperv_initialized(void)
+{
+	return hyperv_initialized;
+}
+EXPORT_SYMBOL_GPL(hv_is_hyperv_initialized);
+
+bool hv_is_hibernation_supported(void)
+{
+	return false;
+}
+EXPORT_SYMBOL_GPL(hv_is_hibernation_supported);
 
 /*
  * The VMbus handler functions are no-ops on ARM64 because
arch/arm64/include/asm/mshyperv.h +6 -0
index b17299c..86ca5c5 100644
@@ -23,6 +23,12 @@
 #include <asm/hyperv-tlfs.h>
 #include <clocksource/arm_arch_timer.h>
 
+#if IS_ENABLED(CONFIG_HYPERV)
+void __init hyperv_early_init(void);
+#else
+static inline void hyperv_early_init(void) {};
+#endif
+
 /*
  * Declare calls to get and set Hyper-V VP register values on ARM64, which
  * requires a hypercall.
arch/arm64/kernel/setup.c +4 -0
index c18aacd..1df8285 100644
@@ -49,6 +49,7 @@
 #include <asm/traps.h>
 #include <asm/efi.h>
 #include <asm/xen/hypervisor.h>
+#include <asm/mshyperv.h>
 #include <asm/mmu_context.h>
 
 static int num_standard_resources;
@@ -340,6 +341,9 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
 	if (acpi_disabled)
 		unflatten_device_tree();
 
+	/* Do after acpi_boot_table_init() so local FADT is available */
+	hyperv_early_init();
+
 	bootmem_init();
 
 	kasan_init();
Signature · 1 line
1.8.3.1
Reviewed-by on v9 5/7
#11 lore raw

RE: [PATCH v9 5/7] arm64: hyperv: Initialize hypervisor on boot

Quoted 12 lines

static int num_standard_resources;

arch/arm64/kernel/setup.c +3 -1
@@ -340,6 +341,9 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
 	if (acpi_disabled)
 		unflatten_device_tree();
 
+	/* Do after acpi_boot_table_init() so local FADT is available */
+	hyperv_early_init();
+
 	bootmem_init();
 
 	kasan_init();
--
Quoted 1 line

1.8.3.1

#12 lore raw

PATCH v9 6/7 arm64: efi: Export screen_info

In reply to Michael Kelley

The Hyper-V frame buffer driver may be built as a module, and it needs access to screen_info. So export screen_info.

---
 arch/arm64/kernel/efi.c | 1 +
 1 file changed, 1 insertion(+)
arch/arm64/kernel/efi.c +1 -0
index fa02efb..e1be6c4 100644
@@ -55,6 +55,7 @@ static __init pteval_t create_mapping_protection(efi_memory_desc_t *md)
 
 /* we will fill this structure from the stub, so don't put it in .bss */
 struct screen_info screen_info __section(".data");
+EXPORT_SYMBOL(screen_info);
 
 int __init efi_create_mapping(struct mm_struct *mm, efi_memory_desc_t *md)
 {
Signature · 1 line
1.8.3.1
#13 lore raw

PATCH v9 7/7 Drivers: hv: Enable Hyper-V code to be built on ARM64

In reply to Michael Kelley

Update drivers/hv/Kconfig so CONFIG_HYPERV can be selected on ARM64, causing the Hyper-V specific code to be built.

---
 drivers/hv/Kconfig | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/hv/Kconfig +2 -1
index 79e5356..d492682 100644
@@ -4,7 +4,8 @@ menu "Microsoft Hyper-V guest support"
 
 config HYPERV
 	tristate "Microsoft Hyper-V client drivers"
-	depends on X86 && ACPI && X86_LOCAL_APIC && HYPERVISOR_GUEST
+	depends on ACPI && ((X86 && X86_LOCAL_APIC && HYPERVISOR_GUEST) \
+		|| (ARM64 && !CPU_BIG_ENDIAN))
 	select PARAVIRT
 	select X86_HV_CALLBACK_VECTOR
 	help
Signature · 1 line
1.8.3.1
#14 lore raw
In reply to Michael Kelley

From: Michael Kelley <[email protected]> Sent: Monday, March 8, 2021 11:57 AM

Quoted 8 lines

This series enables Linux guests running on Hyper-V on ARM64 hardware. New ARM64-specific code in arch/arm64/hyperv initializes Hyper-V, including its interrupts and hypercall mechanism. Existing architecture independent drivers for Hyper-V's VMbus and synthetic devices just work when built for ARM64. Hyper-V code is built and included in the image and modules only if CONFIG_HYPERV is enabled.

ARM64 maintainers --

What are the prospects for getting your review and Ack on this patch set? We're wanting to get the Hyper-V support on ARM64 finally accepted upstream. Previous comments should be addressed in this revision, with perhaps a remaining discussion point around the alternate SMCCC hypercall interface in Patch 1 that makes use of changes in v1.2 of the SMCCC spec. There are several viable approaches that I've noted in the patch, depending on your preferences.

Michael

Quoted 180 lines

The seven patches are organized as follows:

1) Update include/linux/arm-smccc.h to provide an HVC wrapper variant that returns results in other than X0 thru X3

2) Add definitions and functions for making Hyper-V hypercalls and getting/setting virtual processor registers provided by Hyper-V

3) Add architecture specific definitions needed by the architecture independent Hyper-V clocksource driver in drivers/clocksource/hyperv_timer.c. Update the clocksource driver to be initialized on ARM64.

4) Add functions needed by the arch independent VMbus driver for reporting a panic to Hyper-V and as stubs for the kexec and crash handlers.

5) Add Hyper-V initialization code and utility functions that report Hyper-v status.

6) Export screen_info so it may be used by the Hyper-V frame buffer driver built as a module. It is already exported for x86, powerpc, and alpha architectures.

7) Make CONFIG_HYPERV selectable on ARM64 in addition to x86/x64.

Hyper-V on ARM64 runs with a 4 Kbyte page size, but allows guests with 4K/16K/64K page size. Linux guests with this ARM64 enablement code work with all three supported ARM64 page sizes.

The Hyper-V vPCI driver at drivers/pci/host/pci-hyperv.c has x86/x64-specific code and is not being built for ARM64. Fixing this driver to enable vPCI devices on ARM64 will be done later.

In a few cases, terminology from the x86/x64 world has been carried over into the ARM64 code ("MSR", "TSC"). Hyper-V still uses the x86/x64 terminology and has not replaced it with something more generic, so the code uses the Hyper-V terminology. This will be fixed when Hyper-V updates the usage in the TLFS.

This patch set is based on the hyperv-next branch of the code tree
https://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git/

Changes in v9:
* Added Patch 1 to enable making an SMCCC compliant hypercall that returns results in other than registers X0 thru X3, per version 1.2 and later of the SMCCC spec.
* Using the ability to return results in registers X6 and X7, converted hv_get_vpreg_128() to use a "fast" hypercall that passes inputs and outputs in registers, and in doing so eliminated a lot of memory allocation complexity.
* Cleaned up some extra blank lines and use of spaces in aligning local variables. [Sunil Muthuswamy]
* Based on discussion about future directions, reverted the population of hv_vp_index array to use a cpuhp state instead of a hypercall, which is like it was in v7 and earlier.

Changes in v8:
* Removed a lot of code based on refactoring the boundary between arch independent and arch dependent code for Hyper-V, per comments from Arnd Bergmann. The removed code was either duplicated on the x86 side, or has been folded into architecture independent code as not really being architecture dependent.
* Added config dependency on !CONFIG_CPU_BIG_ENDIAN [Arnd Bergmann]
* Reworked the approach to Hyper-V initialization. The functionality is the same, but is now structured like the Xen code with an early init function called in setup_arch() and an early initcall to finish the initialization. [Arnd Bergmann]

Changes in v7:
* Separately upstreamed split of hyperv-tlfs.h into arch dependent and independent versions. In this patch set, update the ARM64 hyperv-tlfs.h to include architecture independent definitions. This approach eliminates a lot of lines of otherwise duplicated code on the ARM64 side.
* Break ARM64 mshyperv.h into smaller pieces. Have an initial baseline, and add code along with patches for a particular functional area. [Marc Zyngier]
* In mshyperv.h, use static inline functions instead of #defines where possible. [Arnd Bergmann]
* Use VMbus INTID obtained from ACPI DSDT instead of hardcoding. The STIMER INTID is still hardcoded because it is needed before Linux has initialized the ACPI subsystem, so it can't be obtained from the DSDT. Wedging it into the GTDT seems dubious, so was not done. [Marc Zyngier]
* Update Hyper-V page size allocation functions to use alloc_page() if PAGE_SIZE == HV_HYP_PAGE_SIZE [Arnd Bergmann]
* Various other minor changes based on feedback and to rebase to latest linux-next [Marc Zyngier and Arnd Bergmann]

Changes in v6:
* Use SMCCC hypercall interface instead of direct invocation of HVC instruction and the Hyper-V hypercall interface [Marc Zyngier]
* Reimplemented functions to alloc/free Hyper-V size pages using kmalloc/kfree since kmalloc now guarantees alignment of power of 2 size allocations [Marc Zyngier]
* Export screen_info in arm64 architecture so it can be used by the Hyper-V buffer driver built as a module
* Renamed source file arch/arm64/hyperv/hv_init.c to hv_core.c to better reflect its content
* Fixed the bit position of certain feature flags presented by Hyper-V to the guest. The bit positions on ARM64 don't match the position on x86 like originally thought.
* Minor fixups to rebase to 5.6-rc5 linux-next

Changes in v5:
* Minor fixups to rebase to 5.4-rc1 linux-next

Changes in v4:
* Moved clock-related code into an architecture independent Hyper-V clocksource driver that is already upstream. Clock related code is removed from this patch set except for the ARM64 specific interrupt handler. [Marc Zyngier]
* Separately upstreamed the split of mshyperv.h into arch independent and arch dependent portions. The arch independent portion has been removed from this patch set.
* Divided patch #2 of the series into multiple smaller patches [Marc Zyngier]
* Changed a dozen or so smaller things based on feedback [Marc Zyngier, Will Deacon]
* Added functions to alloc/free Hyper-V size pages for use by drivers for Hyper-V synthetic devices when updated to not assume guest page size and Hyper-v page size are the same

Changes in v3:
* Added initialization of hv_vp_index array like was recently added on x86 branch [KY Srinivasan]
* Changed Hyper-V ARM64 register symbols to be all uppercase instead of mixed case [KY Srinivasan]
* Separated mshyperv.h into two files, one architecture independent and one architecture dependent. After this code is upstream, will make changes to the x86 code to use the architecture independent file and remove duplication. And once we have a multi-architecture Hyper-V TLFS, will do a separate patch to split hyperv-tlfs.h in the same way. [KY Srinivasan]
* Minor tweaks to rebase to latest linux-next code

Changes in v2:
* Removed patch to implement slow_virt_to_phys() on ARM64. Use of slow_virt_to_phys() in arch independent Hyper-V drivers has been eliminated by commit 6ba34171bcbd ("Drivers: hv: vmbus: Remove use of slow_virt_to_phys()")
* Minor tweaks to rebase to latest linux-next code *** BLURB HERE ***

Michael Kelley (7):
smccc: Add HVC call variant with result registers other than 0 thru 3
arm64: hyperv: Add Hyper-V hypercall and register access utilities
arm64: hyperv: Add Hyper-V clocksource/clockevent support
arm64: hyperv: Add kexec and panic handlers
arm64: hyperv: Initialize hypervisor on boot
arm64: efi: Export screen_info
Drivers: hv: Enable Hyper-V code to be built on ARM64

 MAINTAINERS                          |   3 +
 arch/arm64/Kbuild                    |   1 +
 arch/arm64/hyperv/Makefile           |   2 +
 arch/arm64/hyperv/hv_core.c          | 178 +++++++++++++++++++++++++++++++++++
 arch/arm64/hyperv/mshyperv.c         | 173 ++++++++++++++++++++++++++++++++++
 arch/arm64/include/asm/hyperv-tlfs.h |  69 ++++++++++++++
 arch/arm64/include/asm/mshyperv.h    |  72 ++++++++++++++
 arch/arm64/kernel/efi.c              |   1 +
 arch/arm64/kernel/setup.c            |   4 +
 drivers/clocksource/hyperv_timer.c   |  14 +++
 drivers/hv/Kconfig                   |   3 +-
 include/linux/arm-smccc.h            |  29 ++++--
 12 files changed, 542 insertions(+), 7 deletions(-)
 create mode 100644 arch/arm64/hyperv/Makefile
 create mode 100644 arch/arm64/hyperv/hv_core.c
 create mode 100644 arch/arm64/hyperv/mshyperv.c
 create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h
 create mode 100644 arch/arm64/include/asm/mshyperv.h

--
1.8.3.1

#15 lore raw

From: Michael Kelley <[email protected]>

Quoted 28 lines

Sent: Wednesday, March 24, 2021 8:55 AM
To: [email protected]; [email protected]; Mark Rutland <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; KY Srinivasan <[email protected]>
Subject: RE: [PATCH v9 0/7] Enable Linux guests on Hyper-V on ARM64

From: Michael Kelley <[email protected]> Sent: Monday, March 8, 2021 11:57 AM

Quoted 8 lines

This series enables Linux guests running on Hyper-V on ARM64 hardware. New ARM64-specific code in arch/arm64/hyperv initializes Hyper-V, including its interrupts and hypercall mechanism. Existing architecture independent drivers for Hyper-V's VMbus and synthetic devices just work when built for ARM64. Hyper-V code is built and included in the image and modules only if CONFIG_HYPERV is enabled.

ARM64 maintainers --

What are the prospects for getting your review and Ack on this patch set? We're wanting to get the Hyper-V support on ARM64 finally accepted upstream. Previous comments should be addressed in this revision, with perhaps a remaining discussion point around the alternate SMCCC hypercall interface in Patch 1 that makes use of changes in v1.2 of the SMCCC spec. There are several viable approaches that I've noted in the patch, depending on your preferences.

Quoted 1 line

Michael

Thanks, Mark, for jumping in on the SMCCC hypercall interface. But I'm still looking for feedback or ACKs on the other patches in the series. There's only one place in Patch 2 of the series that needs the SMCCC v1.2 interface, and I'd like to be able to respond to any remaining issues with the other patches while the SMCCC details are finished up.

Michael