Enable Linux guests on Hyper-V on ARM64
Subsystems (7)
v10 cover Enable Linux guests on Hyper-V on ARM64
This series enables Linux guests running on Hyper-V on ARM64 hardware. New ARM64-specific code in arch/arm64/hyperv initializes Hyper-V, including its interrupts and hypercall mechanism. Existing architecture independent drivers for Hyper-V's VMbus and synthetic devices just work when built for ARM64. Hyper-V code is built and included in the image and modules only if CONFIG_HYPERV is enabled.
The seven patches are organized as follows:
1) Fix some minor places where x86-dependencies crept into architecture independent Hyper-V code
2) Add definitions and functions for making Hyper-V hypercalls and getting/setting virtual processor registers provided by Hyper-V
3) Add architecture specific definitions needed by the architecture independent Hyper-V clocksource driver in drivers/clocksource/hyperv_timer.c. Update the clocksource driver to be initialized on ARM64.
4) Add functions needed by the arch independent VMbus driver for reporting a panic to Hyper-V and as stubs for the kexec and crash handlers.
5) Add Hyper-V initialization code and utility functions that report Hyper-v status.
6) Export screen_info so it may be used by the Hyper-V frame buffer driver built as a module. It is already exported for x86, powerpc, and alpha architectures.
7) Make CONFIG_HYPERV selectable on ARM64 in addition to x86/x64.
Hyper-V on ARM64 runs with a 4 Kbyte page size, but allows guests with 4K/16K/64K page size. Linux guests with this patch series work with all three supported ARM64 page sizes.
The Hyper-V vPCI driver at drivers/pci/host/pci-hyperv.c has x86/x64-specific code and is not being built for ARM64. Fixing this driver to enable vPCI devices on ARM64 will be done later. The Hyper-V balloon driver also recently picked up some x86 dependencies that will be fixed separately, so it currently doesn't build for ARM64 and CONFIG_HYPERV_BALLOON must be "N".
In a few cases, terminology from the x86/x64 world has been carried over into the ARM64 code ("MSR", "TSC"). Hyper-V still uses the x86/x64 terminology and has not replaced it with something more generic, so the code uses the Hyper-V terminology. This will be fixed when Hyper-V updates the usage in the TLFS.
This patch set is based on the 5.13-rc1 code tree, plus a patch from Sudeep Holla that implements SMCCC v1.2 HVC calls:
https://lore.kernel.org/linux-arm-kernel/[email protected]/
Changes in v10:
* Drop previous Patch 1 for SMCCC v1.2 HVC calls in favor of Sudeep Holla's patch [Mark Rutland]
* Use new helper functions in 5.13 for checking Hyper-V hypercall return status
* Set up per-CPU hypercall argument page that is now used by the Hyper-V balloon driver
* Rebase on 5.13-RC1
Changes in v9:
* Added Patch 1 to enable making an SMCCC compliant hypercall that returns results in other than registers X0 thru X3, per version 1.2 and later of the SMCCC spec.
* Using the ability to return results in registers X6 and X7, converted hv_get_vpreg_128() to use a "fast" hypercall that passes inputs and outputs in registers, and in doing so eliminated a lot of memory allocation complexity.
* Cleaned up some extra blank lines and use of spaces in aligning local variables. [Sunil Muthuswamy]
* Based on discussion about future directions, reverted the population of hv_vp_index array to use a cpuhp state instead of a hypercall, which is like it was in v7 and earlier.
Changes in v8:
* Removed a lot of code based on refactoring the boundary between arch independent and arch dependent code for Hyper-V, per comments from Arnd Bergmann. The removed code was either duplicated on the x86 side, or has been folded into architecture independent code as not really being architecture dependent.
* Added config dependency on !CONFIG_CPU_BIG_ENDIAN [Arnd Bergmann]
* Reworked the approach to Hyper-V initialization. The functionality is the same, but is now structured like the Xen code with an early init function called in setup_arch() and an early initcall to finish the initialization. [Arnd Bergmann]
Changes in v7:
* Separately upstreamed split of hyperv-tlfs.h into arch dependent and independent versions. In this patch set, update the ARM64 hyperv-tlfs.h to include architecture independent definitions. This approach eliminates a lot of lines of otherwise duplicated code on the ARM64 side.
* Break ARM64 mshyperv.h into smaller pieces. Have an initial baseline, and add code along with patches for a particular functional area. [Marc Zyngier]
* In mshyperv.h, use static inline functions instead of #defines where possible. [Arnd Bergmann]
* Use VMbus INTID obtained from ACPI DSDT instead of hardcoding. The STIMER INTID is still hardcoded because it is needed before Linux has initialized the ACPI subsystem, so it can't be obtained from the DSDT. Wedging it into the GTDT seems dubious, so was not done. [Marc Zyngier]
* Update Hyper-V page size allocation functions to use alloc_page() if PAGE_SIZE == HV_HYP_PAGE_SIZE [Arnd Bergmann]
* Various other minor changes based on feedback and to rebase to latest linux-next [Marc Zyngier and Arnd Bergmann]
Changes in v6:
* Use SMCCC hypercall interface instead of direct invocation of HVC instruction and the Hyper-V hypercall interface [Marc Zyngier]
* Reimplemented functions to alloc/free Hyper-V size pages using kmalloc/kfree since kmalloc now guarantees alignment of power of 2 size allocations [Marc Zyngier]
* Export screen_info in arm64 architecture so it can be used by the Hyper-V buffer driver built as a module
* Renamed source file arch/arm64/hyperv/hv_init.c to hv_core.c to better reflect its content
* Fixed the bit position of certain feature flags presented by Hyper-V to the guest. The bit positions on ARM64 don't match the position on x86 like originally thought.
* Minor fixups to rebase to 5.6-rc5 linux-next
Changes in v5:
* Minor fixups to rebase to 5.4-rc1 linux-next
Changes in v4:
* Moved clock-related code into an architecture independent Hyper-V clocksource driver that is already upstream. Clock related code is removed from this patch set except for the ARM64 specific interrupt handler. [Marc Zyngier]
* Separately upstreamed the split of mshyperv.h into arch independent and arch dependent portions. The arch independent portion has been removed from this patch set.
* Divided patch #2 of the series into multiple smaller patches [Marc Zyngier]
* Changed a dozen or so smaller things based on feedback [Marc Zyngier, Will Deacon]
* Added functions to alloc/free Hyper-V size pages for use by drivers for Hyper-V synthetic devices when updated to not assume guest page size and Hyper-v page size are the same
Changes in v3:
* Added initialization of hv_vp_index array like was recently added on x86 branch [KY Srinivasan]
* Changed Hyper-V ARM64 register symbols to be all uppercase instead of mixed case [KY Srinivasan]
* Separated mshyperv.h into two files, one architecture independent and one architecture dependent. After this code is upstream, will make changes to the x86 code to use the architecture independent file and remove duplication. And once we have a multi-architecture Hyper-V TLFS, will do a separate patch to split hyperv-tlfs.h in the same way. [KY Srinivasan]
* Minor tweaks to rebase to latest linux-next code
Changes in v2:
* Removed patch to implement slow_virt_to_phys() on ARM64. Use of slow_virt_to_phys() in arch independent Hyper-V drivers has been eliminated by commit 6ba34171bcbd ("Drivers: hv: vmbus: Remove use of slow_virt_to_phys()")
* Minor tweaks to rebase to latest linux-next code
Michael Kelley (7):
asm-generic: hyperv: Fix incorrect architecture dependencies
arm64: hyperv: Add Hyper-V hypercall and register access utilities
arm64: hyperv: Add Hyper-V clocksource/clockevent support
arm64: hyperv: Add kexec and panic handlers
arm64: hyperv: Initialize hypervisor on boot
arm64: efi: Export screen_info
Drivers: hv: Enable Hyper-V code to be built on ARM64
MAINTAINERS | 3 + arch/arm64/Kbuild | 1 + arch/arm64/hyperv/Makefile | 2 + arch/arm64/hyperv/hv_core.c | 182 +++++++++++++++++++++++++++++++ arch/arm64/hyperv/mshyperv.c | 206 +++++++++++++++++++++++++++++++++++ arch/arm64/include/asm/hyperv-tlfs.h | 69 ++++++++++++ arch/arm64/include/asm/mshyperv.h | 72 ++++++++++++ arch/arm64/kernel/efi.c | 1 + arch/arm64/kernel/setup.c | 4 + arch/x86/include/asm/mshyperv.h | 3 - drivers/clocksource/hyperv_timer.c | 14 +++ drivers/hv/Kconfig | 3 +- include/asm-generic/mshyperv.h | 5 + 13 files changed, 561 insertions(+), 4 deletions(-) create mode 100644 arch/arm64/hyperv/Makefile create mode 100644 arch/arm64/hyperv/hv_core.c create mode 100644 arch/arm64/hyperv/mshyperv.c create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h create mode 100644 arch/arm64/include/asm/mshyperv.h
Signature · 7 lines
1.8.3.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v10 1/7 asm-generic: hyperv: Fix incorrect architecture dependencies
Move the declaration of hv_root_partition and hyperv_pcpu_input_arg from the x86-specific mshyperv.h to the arch independent mshyperv.h since they are used by arch independent code. While here, add a missing #include needed to compile correctly on ARM64.
--- arch/x86/include/asm/mshyperv.h | 3 --- include/asm-generic/mshyperv.h | 5 +++++ 2 files changed, 5 insertions(+), 3 deletions(-)
arch/x86/include/asm/mshyperv.h +0 -3
@@ -36,7 +36,6 @@ static inline u64 hv_get_register(unsigned int reg)extern int hyperv_init_cpuhp;extern void *hv_hypercall_pg; -extern void __percpu **hyperv_pcpu_input_arg;extern void __percpu **hyperv_pcpu_output_arg;extern u64 hv_current_partition_id;
@@ -170,8 +169,6 @@ int hyperv_fill_flush_guest_mapping_list(struct hv_guest_mapping_flush_list *flush, u64 start_gfn, u64 end_gfn); -extern bool hv_root_partition; -#ifdef CONFIG_X86_64 void hv_apic_init (void );void __init hv_init_spinlocks (void );
include/asm-generic/mshyperv.h +5 -0
@@ -22,6 +22,7 @@#include <linux/atomic.h> #include <linux/bitops.h> #include <linux/cpumask.h> +#include <linux/nmi.h> #include <asm/ptrace.h> #include <asm/hyperv-tlfs.h>
@@ -151,6 +152,8 @@ static inline void vmbus_signal_eom(struct hv_message *msg, u32 old_msg_type)extern int vmbus_interrupt;extern int vmbus_irq; +extern bool hv_root_partition; +#if IS_ENABLED (CONFIG_HYPERV )/* * Hypervisor's notion of virtual processor ID is different from
@@ -161,6 +164,8 @@ static inline void vmbus_signal_eom(struct hv_message *msg, u32 old_msg_type)extern u32 *hv_vp_index;extern u32 hv_max_vp_index; +extern void __percpu **hyperv_pcpu_input_arg; +/* Sentinel value for an uninitialized entry in hv_vp_index array */ #define VP_INVAL U32_MAX
Signature · 7 lines
1.8.3.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v10 2/7 arm64: hyperv: Add Hyper-V hypercall and register access utilities
In reply to Michael Kelleyhyperv-tlfs.h defines Hyper-V interfaces from the Hyper-V Top Level Functional Spec (TLFS), and #includes the architecture-independent part of hyperv-tlfs.h in include/asm-generic. The published TLFS is distinctly oriented to x86/x64, so the ARM64-specific hyperv-tlfs.h includes information for ARM64 that is not yet formally published. The TLFS is available here:
docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
mshyperv.h defines Linux-specific structures and routines for interacting with Hyper-V on ARM64, and #includes the architecture- independent part of mshyperv.h in include/asm-generic.
Use these definitions to provide utility functions to make Hyper-V hypercalls and to get and set Hyper-V provided registers associated with a virtual processor.
--- MAINTAINERS | 3 + arch/arm64/Kbuild | 1 + arch/arm64/hyperv/Makefile | 2 + arch/arm64/hyperv/hv_core.c | 130 +++++++++++++++++++++++++++++++++++ arch/arm64/include/asm/hyperv-tlfs.h | 69 +++++++++++++++++++ arch/arm64/include/asm/mshyperv.h | 54 +++++++++++++++ 6 files changed, 259 insertions(+) create mode 100644 arch/arm64/hyperv/Makefile create mode 100644 arch/arm64/hyperv/hv_core.c create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h create mode 100644 arch/arm64/include/asm/mshyperv.h
MAINTAINERS +3 -0
@@ -8437,6 +8437,9 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git F: Documentation/ABI/stable/sysfs-bus-vmbus F: Documentation/ABI/testing/debugfs-hyperv F: Documentation/networking/device_drivers/ethernet/microsoft/netvsc.rst +F: arch/arm64/hyperv +F: arch/arm64/include/asm/hyperv-tlfs.h +F: arch/arm64/include/asm/mshyperv.h F: arch/x86/hyperv F: arch/x86/include/asm/hyperv-tlfs.h F: arch/x86/include/asm/mshyperv.h
arch/arm64/Kbuild +1 -0
@@ -3,4 +3,5 @@ obj-y += kernel/ mm/ obj-$(CONFIG_NET) += net/ obj-$(CONFIG_KVM) += kvm/ obj-$(CONFIG_XEN) += xen/ +obj-$(subst m,y,$(CONFIG_HYPERV)) += hyperv/ obj-$(CONFIG_CRYPTO) += crypto/
arch/arm64/hyperv/Makefile new file +2 -0
@@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0 +obj-y := hv_core.o
arch/arm64/hyperv/hv_core.c new file +130 -0
130 lines not on this page · raw message
arch/arm64/include/asm/hyperv-tlfs.h new file +69 -0
69 lines not on this page · raw message
arch/arm64/include/asm/mshyperv.h new file +54 -0
54 lines not on this page · raw message
Signature · 7 lines
1.8.3.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Mark Rutland
reviewerRe: [PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities
Quoted 33 lines · Michael Kelley
hyperv-tlfs.hdefines Hyper-V interfaces from the Hyper-V Top Level Functional Spec (TLFS), and #includes the architecture-independent part ofhyperv-tlfs.hininclude/asm-generic. The published TLFS is distinctly oriented to x86/x64, so the ARM64-specifichyperv-tlfs.hincludes information for ARM64 that is not yet formally published. The TLFS is available here:docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
mshyperv.hdefines Linux-specific structures and routines for interacting with Hyper-V on ARM64, and #includes the architecture- independent part ofmshyperv.hininclude/asm-generic.Use these definitions to provide utility functions to make Hyper-V hypercalls and to get and set Hyper-V provided registers associated with a virtual processor.
Signed-off-by Michael Kelley <[email protected]>Reviewed-by Sunil Muthuswamy <[email protected]>--- MAINTAINERS | 3 + arch/arm64/Kbuild | 1 + arch/arm64/hyperv/Makefile | 2 + arch/arm64/hyperv/hv_core.c | 130 +++++++++++++++++++++++++++++++++++ arch/arm64/include/asm/hyperv-tlfs.h | 69 +++++++++++++++++++ arch/arm64/include/asm/mshyperv.h | 54 +++++++++++++++ 6 files changed, 259 insertions(+) create mode 100644 arch/arm64/hyperv/Makefile create mode 100644 arch/arm64/hyperv/hv_core.c create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h create mode 100644 arch/arm64/include/asm/mshyperv.harch/arm64/hyperv/hv_core.c new file +1
-0+/*
arch/arm64/hyperv/hv_core.c line 31
Quoted 10 lines
arch/arm64/hyperv/hv_core.c new file +10
-0+* hv_do_hypercall- Invoke the specified hypercall +*/ +u64 hv_do_hypercall (u64 control,void *input,void *output) +{ +struct arm_smccc_res res; +u64 input_address; +u64 output_address; + + input_address = input ?virt_to_phys (input) :0 ; + output_address = output ?virt_to_phys (output) :0 ;
I may have asked this before, but are input and output always linear map pointers, or can they ever be vmalloc pointers?
Otherwise, this looks fine to me.
Mark.
RE: [PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities
From: Mark Rutland <[email protected]> Sent: Friday, May 14, 2021 5:53 AM
Quoted 49 lines · Michael Kelley
Quoted 33 lines
hyperv-tlfs.hdefines Hyper-V interfaces from the Hyper-V Top Level Functional Spec (TLFS), and #includes the architecture-independent part ofhyperv-tlfs.hininclude/asm-generic. The published TLFS is distinctly oriented to x86/x64, so the ARM64-specifichyperv-tlfs.hincludes information for ARM64 that is not yet formally published. The TLFS is available here:docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
mshyperv.hdefines Linux-specific structures and routines for interacting with Hyper-V on ARM64, and #includes the architecture- independent part ofmshyperv.hininclude/asm-generic.Use these definitions to provide utility functions to make Hyper-V hypercalls and to get and set Hyper-V provided registers associated with a virtual processor.
Signed-off-by Michael Kelley <[email protected]>Reviewed-by Sunil Muthuswamy <[email protected]>--- MAINTAINERS | 3 + arch/arm64/Kbuild | 1 + arch/arm64/hyperv/Makefile | 2 + arch/arm64/hyperv/hv_core.c | 130 +++++++++++++++++++++++++++++++++++ arch/arm64/include/asm/hyperv-tlfs.h | 69 +++++++++++++++++++ arch/arm64/include/asm/mshyperv.h | 54 +++++++++++++++ 6 files changed, 259 insertions(+) create mode 100644 arch/arm64/hyperv/Makefile create mode 100644 arch/arm64/hyperv/hv_core.c create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h create mode 100644 arch/arm64/include/asm/mshyperv.harch/arm64/hyperv/hv_core.c new file +1
-0+/* arch/arm64/hyperv/hv_core.c line 31
Quoted 10 lines
arch/arm64/hyperv/hv_core.c new file +10
-0+* hv_do_hypercall- Invoke the specified hypercall +*/ +u64 hv_do_hypercall (u64 control,void *input,void *output) +{ +struct arm_smccc_res res; +u64 input_address; +u64 output_address; + + input_address = input ?virt_to_phys (input) :0 ; + output_address = output ?virt_to_phys (output) :0 ;I may have asked this before, but are
inputandoutputalways linear map pointers, or can they ever be vmalloc pointers?
Quoted 2 lines
Otherwise, this looks fine to me.
The caller must ensure that hypercall arguments are aligned to 4 Kbytes, and no larger than 4 Kbytes, since that's the page size used by Hyper-V regardless of the guest page size. A per-CPU 4 Kbyte memory area (hyperv_pcpu_input_arg) meeting these requirements is pre-allocated that callers can use for this purpose. On the x86 side there's also a similar hyperv_pcpu_output_arg, but there are no use cases for it on ARM64.
Michael
Mark Rutland
reviewerRe: [PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities
Quoted 50 lines · Michael Kelley
From: Mark Rutland <[email protected]> Sent: Friday, May 14, 2021 5:53 AM
Quoted 49 lines · Michael Kelley
Quoted 33 lines
hyperv-tlfs.hdefines Hyper-V interfaces from the Hyper-V Top Level Functional Spec (TLFS), and #includes the architecture-independent part ofhyperv-tlfs.hininclude/asm-generic. The published TLFS is distinctly oriented to x86/x64, so the ARM64-specifichyperv-tlfs.hincludes information for ARM64 that is not yet formally published. The TLFS is available here:docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
mshyperv.hdefines Linux-specific structures and routines for interacting with Hyper-V on ARM64, and #includes the architecture- independent part ofmshyperv.hininclude/asm-generic.Use these definitions to provide utility functions to make Hyper-V hypercalls and to get and set Hyper-V provided registers associated with a virtual processor.
Signed-off-by Michael Kelley <[email protected]>Reviewed-by Sunil Muthuswamy <[email protected]>--- MAINTAINERS | 3 + arch/arm64/Kbuild | 1 + arch/arm64/hyperv/Makefile | 2 + arch/arm64/hyperv/hv_core.c | 130 +++++++++++++++++++++++++++++++++++ arch/arm64/include/asm/hyperv-tlfs.h | 69 +++++++++++++++++++ arch/arm64/include/asm/mshyperv.h | 54 +++++++++++++++ 6 files changed, 259 insertions(+) create mode 100644 arch/arm64/hyperv/Makefile create mode 100644 arch/arm64/hyperv/hv_core.c create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h create mode 100644 arch/arm64/include/asm/mshyperv.harch/arm64/hyperv/hv_core.c new file +1
-0+/* arch/arm64/hyperv/hv_core.c line 31
Quoted 10 lines
arch/arm64/hyperv/hv_core.c new file +10
-0+* hv_do_hypercall- Invoke the specified hypercall +*/ +u64 hv_do_hypercall (u64 control,void *input,void *output) +{ +struct arm_smccc_res res; +u64 input_address; +u64 output_address; + + input_address = input ?virt_to_phys (input) :0 ; + output_address = output ?virt_to_phys (output) :0 ;I may have asked this before, but are
inputandoutputalways linear map pointers, or can they ever be vmalloc pointers?
Quoted 7 lines
Quoted 1 line
Otherwise, this looks fine to me.
The caller must ensure that hypercall arguments are aligned to 4 Kbytes, and no larger than 4 Kbytes, since that's the page size used by Hyper-V regardless of the guest page size. A per-CPU 4 Kbyte memory area (
hyperv_pcpu_input_arg) meeting these requirements is pre-allocated that callers can use for this purpose.
What I was trying to find out was how that was allocated, as vmalloc()'d pointers aren't legitimate to pass to virt_to_phys().
From scanning ahead to patch 5, I see that memory comes from kmalloc(), and so it is legitimate to use virt_to_phys().
I see; and from patch 5 I see that memory come from kmalloc(), and will therefore be part of the linear map, and so virt_to_phys() is legitimate.
What I was asking here was how that memory was allocated. So long as those are the only buffers used, this looks fine to me.
Thanks,
Mark.
RE: [PATCH v10 2/7] arm64: hyperv: Add Hyper-V hypercall and register access utilities
From: Mark Rutland <[email protected]> Sent: Monday, May 17, 2021 4:45 AM
Quoted 68 lines · Michael Kelley
Quoted 27 lines
From: Mark Rutland <[email protected]> Sent: Friday, May 14, 2021 5:53 AM
Quoted 26 lines · Michael Kelley
Quoted 24 lines
hyperv-tlfs.hdefines Hyper-V interfaces from the Hyper-V Top Level Functional Spec (TLFS), and #includes the architecture-independent part ofhyperv-tlfs.hininclude/asm-generic. The published TLFS is distinctly oriented to x86/x64, so the ARM64-specifichyperv-tlfs.hincludes information for ARM64 that is not yet formally published. The TLFS is available here:docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
mshyperv.hdefines Linux-specific structures and routines for interacting with Hyper-V on ARM64, and #includes the architecture- independent part ofmshyperv.hininclude/asm-generic.Use these definitions to provide utility functions to make Hyper-V hypercalls and to get and set Hyper-V provided registers associated with a virtual processor.
Signed-off-by Michael Kelley <[email protected]>Reviewed-by Sunil Muthuswamy <[email protected]>--- MAINTAINERS | 3 + arch/arm64/Kbuild | 1 + arch/arm64/hyperv/Makefile | 2 + arch/arm64/hyperv/hv_core.c | 130+++++++++++++++++++++++++++++++++++
Quoted 23 lines
Quoted 23 lines
Quoted 9 lines
arch/arm64/include/asm/hyperv-tlfs.h | 69 +++++++++++++++++++ arch/arm64/include/asm/mshyperv.h | 54 +++++++++++++++ 6 files changed, 259 insertions(+) create mode 100644 arch/arm64/hyperv/Makefile create mode 100644 arch/arm64/hyperv/hv_core.c create mode 100644 arch/arm64/include/asm/hyperv-tlfs.h create mode 100644 arch/arm64/include/asm/mshyperv.harch/arm64/hyperv/hv_core.c new file +1
-0+/* arch/arm64/hyperv/hv_core.c line 31
Quoted 10 lines
arch/arm64/hyperv/hv_core.c new file +10
-0+* hv_do_hypercall- Invoke the specified hypercall +*/ +u64 hv_do_hypercall (u64 control,void *input,void *output) +{ +struct arm_smccc_res res; +u64 input_address; +u64 output_address; + + input_address = input ?virt_to_phys (input) :0 ; + output_address = output ?virt_to_phys (output) :0 ;I may have asked this before, but are
inputandoutputalways linear map pointers, or can they ever be vmalloc pointers?Quoted 7 lines
Quoted 1 line
Otherwise, this looks fine to me.
The caller must ensure that hypercall arguments are aligned to 4 Kbytes, and no larger than 4 Kbytes, since that's the page size used by Hyper-V regardless of the guest page size. A per-CPU 4 Kbyte memory area (
hyperv_pcpu_input_arg) meeting these requirements is pre-allocated that callers can use for this purpose.What I was trying to find out was how that was allocated, as vmalloc()'d pointers aren't legitimate to pass to
virt_to_phys().From scanning ahead to patch 5, I see that memory comes from
kmalloc(), and so it is legitimate to usevirt_to_phys().
Quoted 7 lines
I see; and from patch 5 I see that memory come from
kmalloc(), and will therefore be part of the linear map, and sovirt_to_phys()is legitimate.What I was asking here was how that memory was allocated. So long as those are the only buffers used, this looks fine to me.
There is no vmalloc() or stack local memory used as arguments to a
hypercall. Call sites know about this requirement along with the
requirement of being aligned to 4 Kbytes and no larger than 4 Kbytes.
Michael
PATCH v10 3/7 arm64: hyperv: Add Hyper-V clocksource/clockevent support
In reply to Michael KelleyAdd architecture specific definitions and functions needed by the architecture independent Hyper-V clocksource driver. Update the Hyper-V clocksource driver to be initialized on ARM64.
--- arch/arm64/include/asm/mshyperv.h | 12 ++++++++++++ drivers/clocksource/hyperv_timer.c | 14 ++++++++++++++ 2 files changed, 26 insertions(+)
arch/arm64/include/asm/mshyperv.h +12 -0
@@ -21,6 +21,7 @@#include <linux/types.h> #include <linux/arm-smccc.h> #include <asm/hyperv-tlfs.h> +#include <clocksource/arm_arch_timer.h> /* * Declare calls to get and set Hyper-V VP register values on ARM64, which
@@ -41,6 +42,17 @@ static inline u64 hv_get_register(unsigned int reg)return hv_get_vpreg (reg); } +/* Define the interrupt ID used by STIMER0 Direct Mode interrupts. This +* value can't come from ACPI tables because it is needed before the +* Linux ACPI subsystem is initialized. +*/ +#define HYPERV_STIMER0_VECTOR 31 + +static inline u64 hv_get_raw_timer (void ) +{ +return arch_timer_read_counter (); +} +/* SMCCC hypercall parameters */ #define HV_SMCCC_FUNC_NUMBER 1#define HV_FUNC_ID ARM_SMCCC_CALL_VAL( \
drivers/clocksource/hyperv_timer.c +14 -0
@@ -569,3 +569,17 @@ void __init hv_init_clocksource(void)hv_setup_sched_clock (read_hv_sched_clock_msr); }EXPORT_SYMBOL_GPL (hv_init_clocksource); + +/* Initialize everything on ARM64 */ +static int __init hyperv_timer_init (struct acpi_table_header *table) +{ +if (!hv_is_hyperv_initialized ()) +return -EINVAL ; + +hv_init_clocksource (); +if (hv_stimer_alloc (true)) +return -EINVAL ; + +return 0 ; +} +TIMER_ACPI_DECLARE (hyperv,ACPI_SIG_GTDT , hyperv_timer_init);
Signature · 7 lines
1.8.3.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Mark Rutland
reviewerRe: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
Hi Michael,
Quoted 4 lines · Michael Kelley
Add architecture specific definitions and functions needed by the architecture independent Hyper-V clocksource driver. Update the Hyper-V clocksource driver to be initialized on ARM64.
Previously we've said that for a clocksource we must use the architected counter, since that's necessary for things like the VDSO to work correctly and efficiently.
Given that, I'm a bit confused that we're registering a per-cpu clocksource that is in part based on the architected counter. Likewise, I don't entirely follow why it's necessary to PV the clock_event_device.
Are the architected counter and timer reliable without this PV infrastructure? Why do we need to PV either of those?
Thanks,
Mark.
Quoted 63 lines
Signed-off-by Michael Kelley <[email protected]>Reviewed-by Sunil Muthuswamy <[email protected]>--- arch/arm64/include/asm/mshyperv.h | 12 ++++++++++++ drivers/clocksource/hyperv_timer.c | 14 ++++++++++++++ 2 files changed, 26 insertions(+)arch/arm64/include/asm/mshyperv.h +12
-0@@ -21,6 +21,7 @@#include <linux/types.h> #include <linux/arm-smccc.h> #include <asm/hyperv-tlfs.h> +#include <clocksource/arm_arch_timer.h> /* * Declare calls to get and set Hyper-V VP register values on ARM64, which @@ -41,6 +42,17 @@ static inline u64 hv_get_register(unsigned int reg)return hv_get_vpreg (reg); } +/* Define the interrupt ID used by STIMER0 Direct Mode interrupts. This +* value can't come from ACPI tables because it is needed before the +* Linux ACPI subsystem is initialized. +*/ +#define HYPERV_STIMER0_VECTOR 31 + +static inline u64 hv_get_raw_timer (void ) +{ +return arch_timer_read_counter (); +} +/* SMCCC hypercall parameters */ #define HV_SMCCC_FUNC_NUMBER 1#define HV_FUNC_ID ARM_SMCCC_CALL_VAL( \drivers/clocksource/hyperv_timer.c +14
-0@@ -569,3 +569,17 @@ void __init hv_init_clocksource(void)hv_setup_sched_clock (read_hv_sched_clock_msr); }EXPORT_SYMBOL_GPL (hv_init_clocksource); + +/* Initialize everything on ARM64 */ +static int __init hyperv_timer_init (struct acpi_table_header *table) +{ +if (!hv_is_hyperv_initialized ()) +return -EINVAL ; + +hv_init_clocksource (); +if (hv_stimer_alloc (true)) +return -EINVAL ; + +return 0 ; +} +TIMER_ACPI_DECLARE (hyperv,ACPI_SIG_GTDT , hyperv_timer_init);Signature · 1 line
1.8.3.1
RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
From: Mark Rutland <[email protected]> Sent: Friday, May 14, 2021 5:37 AM
Quoted 17 lines
Hi Michael,
Quoted 4 lines · Michael Kelley
Add architecture specific definitions and functions needed by the architecture independent Hyper-V clocksource driver. Update the Hyper-V clocksource driver to be initialized on ARM64.
Previously we've said that for a clocksource we must use the architected counter, since that's necessary for things like the VDSO to work correctly and efficiently.
Given that, I'm a bit confused that we're registering a per-cpu clocksource that is in part based on the architected counter. Likewise, I don't entirely follow why it's necessary to PV the
clock_event_device.
Quoted 6 lines
Are the architected counter and timer reliable without this PV infrastructure? Why do we need to PV either of those?
Thanks,
Mark.
For the clocksource, we have a requirement to live migrate VMs between Hyper-V hosts running on hardware that may have different arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz requirement). The Hyper-V virtualization does scaling to handle the frequency difference. And yes, there's a tradeoff with vDSO not working, though we have an out-of-tree vDSO implementation that we can use when necessary.
For clockevents, the only timer interrupt that Hyper-V provides in a guest VM is its virtualized "STIMER" interrupt. There's no virtualization of the ARM arch timer in the guest.
Michael
Quoted 63 lines
Quoted 63 lines
Signed-off-by Michael Kelley <[email protected]>Reviewed-by Sunil Muthuswamy <[email protected]>--- arch/arm64/include/asm/mshyperv.h | 12 ++++++++++++ drivers/clocksource/hyperv_timer.c | 14 ++++++++++++++ 2 files changed, 26 insertions(+)arch/arm64/include/asm/mshyperv.h +12
-0@@ -21,6 +21,7 @@#include <linux/types.h> #include <linux/arm-smccc.h> #include <asm/hyperv-tlfs.h> +#include <clocksource/arm_arch_timer.h> /* * Declare calls to get and set Hyper-V VP register values on ARM64, which @@ -41,6 +42,17 @@ static inline u64 hv_get_register(unsigned int reg)return hv_get_vpreg (reg); } +/* Define the interrupt ID used by STIMER0 Direct Mode interrupts. This +* value can't come from ACPI tables because it is needed before the +* Linux ACPI subsystem is initialized. +*/ +#define HYPERV_STIMER0_VECTOR 31 + +static inline u64 hv_get_raw_timer (void ) +{ +return arch_timer_read_counter (); +} +/* SMCCC hypercall parameters */ #define HV_SMCCC_FUNC_NUMBER 1#define HV_FUNC_ID ARM_SMCCC_CALL_VAL( \drivers/clocksource/hyperv_timer.c +14
-1@@ -569,3 +569,17 @@ void __init hv_init_clocksource(void)hv_setup_sched_clock (read_hv_sched_clock_msr); }EXPORT_SYMBOL_GPL (hv_init_clocksource); + +/* Initialize everything on ARM64 */ +static int __init hyperv_timer_init (struct acpi_table_header *table) +{ +if (!hv_is_hyperv_initialized ()) +return -EINVAL ; + +hv_init_clocksource (); +if (hv_stimer_alloc (true)) +return -EINVAL ; + +return 0 ; +} +TIMER_ACPI_DECLARE (hyperv,ACPI_SIG_GTDT , hyperv_timer_init); --1.8.3.1
Mark Rutland
reviewerRe: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
Quoted 21 lines · Michael Kelley
From: Mark Rutland <[email protected]> Sent: Friday, May 14, 2021 5:37 AM
Quoted 14 lines · Michael Kelley
Quoted 4 lines
Add architecture specific definitions and functions needed by the architecture independent Hyper-V clocksource driver. Update the Hyper-V clocksource driver to be initialized on ARM64.
Previously we've said that for a clocksource we must use the architected counter, since that's necessary for things like the VDSO to work correctly and efficiently.
Given that, I'm a bit confused that we're registering a per-cpu clocksource that is in part based on the architected counter. Likewise, I don't entirely follow why it's necessary to PV the
clock_event_device.Quoted 5 lines
Are the architected counter and timer reliable without this PV infrastructure? Why do we need to PV either of those?
Thanks,
Mark.
Quoted 7 lines
For the clocksource, we have a requirement to live migrate VMs between Hyper-V hosts running on hardware that may have different arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz requirement). The Hyper-V virtualization does scaling to handle the frequency difference. And yes, there's a tradeoff with vDSO not working, though we have an out-of-tree vDSO implementation that we can use when necessary.
Just to be clear, the vDSO is *one example* of something that won't function correctly. More generally, because this undermines core architectural guarantees, it requires more invasive changes (e.g. we'd have to weaken the sanity checks, and not use the counter in things like kexec paths), impacts any architectural features tied to the generic timer/counter (e.g. the event stream, SPE and tracing, future features), and means that other SW (e.g. bootloaders and other EFI applications) are unlikley to function correctly in this environment.
I am very much not keen on trying to PV this.
What does the guest see when it reads CNTFRQ_EL0? Does this match the real HW value (and can this change over time)? Or is this entirely synthetic?
What do the ACPI tables look like in the guest? Is there a GTDT table at all?
How does the counter event stream behave?
Are there other architectural features which Hyper-V does not implement for a guest?
Is there anything else that may change across a migration? e.g. MIDR? MPIDR? Any of the ID registers?
Quoted 3 lines
For clockevents, the only timer interrupt that Hyper-V provides in a guest VM is its virtualized "STIMER" interrupt. There's no virtualization of the ARM arch timer in the guest.
I think that is rather unfortunate, given it's a core architectural feature. Is it just the interrupt that's missing? i.e. does all the PE-local functionality behave as the architecture requires?
Thanks,
Mark.
Quoted 64 lines
Quoted 64 lines
Quoted 64 lines
Signed-off-by Michael Kelley <[email protected]>Reviewed-by Sunil Muthuswamy <[email protected]>--- arch/arm64/include/asm/mshyperv.h | 12 ++++++++++++ drivers/clocksource/hyperv_timer.c | 14 ++++++++++++++ 2 files changed, 26 insertions(+)arch/arm64/include/asm/mshyperv.h +12
-0@@ -21,6 +21,7 @@#include <linux/types.h> #include <linux/arm-smccc.h> #include <asm/hyperv-tlfs.h> +#include <clocksource/arm_arch_timer.h> /* * Declare calls to get and set Hyper-V VP register values on ARM64, which @@ -41,6 +42,17 @@ static inline u64 hv_get_register(unsigned int reg)return hv_get_vpreg (reg); } +/* Define the interrupt ID used by STIMER0 Direct Mode interrupts. This +* value can't come from ACPI tables because it is needed before the +* Linux ACPI subsystem is initialized. +*/ +#define HYPERV_STIMER0_VECTOR 31 + +static inline u64 hv_get_raw_timer (void ) +{ +return arch_timer_read_counter (); +} +/* SMCCC hypercall parameters */ #define HV_SMCCC_FUNC_NUMBER 1#define HV_FUNC_ID ARM_SMCCC_CALL_VAL( \drivers/clocksource/hyperv_timer.c +14
-1@@ -569,3 +569,17 @@ void __init hv_init_clocksource(void)hv_setup_sched_clock (read_hv_sched_clock_msr); }EXPORT_SYMBOL_GPL (hv_init_clocksource); + +/* Initialize everything on ARM64 */ +static int __init hyperv_timer_init (struct acpi_table_header *table) +{ +if (!hv_is_hyperv_initialized ()) +return -EINVAL ; + +hv_init_clocksource (); +if (hv_stimer_alloc (true)) +return -EINVAL ; + +return 0 ; +} +TIMER_ACPI_DECLARE (hyperv,ACPI_SIG_GTDT , hyperv_timer_init); --1.8.3.1
RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
From: Mark Rutland <[email protected]> Sent: Monday, May 17, 2021 6:08 AM
Quoted 49 lines · Michael Kelley
Quoted 21 lines
From: Mark Rutland <[email protected]> Sent: Friday, May 14, 2021 5:37 AM
Quoted 14 lines · Michael Kelley
Quoted 4 lines
Add architecture specific definitions and functions needed by the architecture independent Hyper-V clocksource driver. Update the Hyper-V clocksource driver to be initialized on ARM64.
Previously we've said that for a clocksource we must use the architected counter, since that's necessary for things like the VDSO to work correctly and efficiently.
Given that, I'm a bit confused that we're registering a per-cpu clocksource that is in part based on the architected counter. Likewise, I don't entirely follow why it's necessary to PV the
clock_event_device.Quoted 5 lines
Are the architected counter and timer reliable without this PV infrastructure? Why do we need to PV either of those?
Thanks,
Mark.Quoted 7 lines
For the clocksource, we have a requirement to live migrate VMs between Hyper-V hosts running on hardware that may have different arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz requirement). The Hyper-V virtualization does scaling to handle the frequency difference. And yes, there's a tradeoff with vDSO not working, though we have an out-of-tree vDSO implementation that we can use when necessary.
Just to be clear, the vDSO is *one example* of something that won't function correctly. More generally, because this undermines core architectural guarantees, it requires more invasive changes (e.g. we'd have to weaken the sanity checks, and not use the counter in things like kexec paths), impacts any architectural features tied to the generic timer/counter (e.g. the event stream, SPE and tracing, future features), and means that other SW (e.g. bootloaders and other EFI applications) are unlikley to function correctly in this environment.
I am very much not keen on trying to PV this.
What does the guest see when it reads
CNTFRQ_EL0? Does this match the real HW value (and can this change over time)? Or is this entirely synthetic?What do the ACPI tables look like in the guest? Is there a GTDT table at all?
Quoted 7 lines
How does the counter event stream behave?
Are there other architectural features which Hyper-V does not implement for a guest?
Is there anything else that may change across a migration? e.g. MIDR? MPIDR? Any of the ID registers?
The ARMv8 architectural system counter and associated registers are visible
and functional in a VM on Hyper-V. The "arch_sys_counter" clocksource is
instantiated by the arm_arch_timer.c driver based on the GTDT in the guest, and a Linux guest on Hyper-V runs fine with this clocksource. Low level code like bootloaders and EFI applications work normally.
The Hyper-V virtualization provides another Linux clocksource that is an overlay on the arch counter and that provides time consistency across a live migration. Live migration of ARM64 VMs on Hyper-V is not functional today, but the Hyper-V team believes they can make it functional. I have not explored with them the live migration implications of things beyond time consistency, like event streams, CNTFRQ_EL0, MIDR/MPIDR, etc.
Would a summary of your point be that live migration across hardware with different arch counter frequencies is likely to not be possible with 100% fidelity because of these other dependencies on the arch counter frequency? (hence the fixed 1 GHz frequency in ARM v8.6)
Quoted 8 lines
Quoted 4 lines
For clockevents, the only timer interrupt that Hyper-V provides in a guest VM is its virtualized "STIMER" interrupt. There's no virtualization of the ARM arch timer in the guest.
I think that is rather unfortunate, given it's a core architectural feature. Is it just the interrupt that's missing? i.e. does all the PE-local functionality behave as the architecture requires?
Right off the bat, I don't know about timer-related PE-local functionality as it's not exercised in a Linux VM on Hyper-V that is using STIMER-based clockevents. I'll explore with the Hyper-V team. My impression is that enabling the ARM arch timer in a guest VM is more work for Hyper-V than just wiring up an interrupt.
Michael
Quoted 69 lines
Thanks,
Mark.Quoted 10 lines
Quoted 10 lines
Review
Quoted 10 lines
Signed-off-by Michael Kelley <[email protected]>Reviewed-by Sunil Muthuswamy <[email protected]>--- arch/arm64/include/asm/mshyperv.h | 12 ++++++++++++ drivers/clocksource/hyperv_timer.c | 14 ++++++++++++++ 2 files changed, 26 insertions(+)(quoted diff) +0
-0
b/arch/arm64/include/asm/mshyperv.hQuoted 54 lines
Quoted 54 lines
Quoted 54 lines
index c448704..b17299c 100644
arch/arm64/include/asm/mshyperv.h +12
-0@@ -21,6 +21,7 @@#include <linux/types.h> #include <linux/arm-smccc.h> #include <asm/hyperv-tlfs.h> +#include <clocksource/arm_arch_timer.h> /* * Declare calls to get and set Hyper-V VP register values on ARM64, which @@ -41,6 +42,17 @@ static inline u64 hv_get_register(unsigned int reg)return hv_get_vpreg (reg); } +/* Define the interrupt ID used by STIMER0 Direct Mode interrupts. This +* value can't come from ACPI tables because it is needed before the +* Linux ACPI subsystem is initialized. +*/ +#define HYPERV_STIMER0_VECTOR 31 + +static inline u64 hv_get_raw_timer (void ) +{ +return arch_timer_read_counter (); +} +/* SMCCC hypercall parameters */ #define HV_SMCCC_FUNC_NUMBER 1#define HV_FUNC_ID ARM_SMCCC_CALL_VAL( \drivers/clocksource/hyperv_timer.c +14
-1@@ -569,3 +569,17 @@ void __init hv_init_clocksource(void)hv_setup_sched_clock (read_hv_sched_clock_msr); }EXPORT_SYMBOL_GPL (hv_init_clocksource); + +/* Initialize everything on ARM64 */ +static int __init hyperv_timer_init (struct acpi_table_header *table) +{ +if (!hv_is_hyperv_initialized ()) +return -EINVAL ; + +hv_init_clocksource (); +if (hv_stimer_alloc (true)) +return -EINVAL ; + +return 0 ; +} +TIMER_ACPI_DECLARE (hyperv,ACPI_SIG_GTDT , hyperv_timer_init); --1.8.3.1
Mark Rutland
reviewerRe: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
Quoted 55 lines · Michael Kelley
From: Mark Rutland <[email protected]> Sent: Monday, May 17, 2021 6:08 AM
Quoted 54 lines · Michael Kelley
Quoted 21 lines
From: Mark Rutland <[email protected]> Sent: Friday, May 14, 2021 5:37 AM
Quoted 14 lines · Michael Kelley
Quoted 4 lines
Add architecture specific definitions and functions needed by the architecture independent Hyper-V clocksource driver. Update the Hyper-V clocksource driver to be initialized on ARM64.
Previously we've said that for a clocksource we must use the architected counter, since that's necessary for things like the VDSO to work correctly and efficiently.
Given that, I'm a bit confused that we're registering a per-cpu clocksource that is in part based on the architected counter. Likewise, I don't entirely follow why it's necessary to PV the
clock_event_device.Quoted 5 lines
Are the architected counter and timer reliable without this PV infrastructure? Why do we need to PV either of those?
Thanks,
Mark.Quoted 7 lines
For the clocksource, we have a requirement to live migrate VMs between Hyper-V hosts running on hardware that may have different arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz requirement). The Hyper-V virtualization does scaling to handle the frequency difference. And yes, there's a tradeoff with vDSO not working, though we have an out-of-tree vDSO implementation that we can use when necessary.
Just to be clear, the vDSO is *one example* of something that won't function correctly. More generally, because this undermines core architectural guarantees, it requires more invasive changes (e.g. we'd have to weaken the sanity checks, and not use the counter in things like kexec paths), impacts any architectural features tied to the generic timer/counter (e.g. the event stream, SPE and tracing, future features), and means that other SW (e.g. bootloaders and other EFI applications) are unlikley to function correctly in this environment.
I am very much not keen on trying to PV this.
What does the guest see when it reads
CNTFRQ_EL0? Does this match the real HW value (and can this change over time)? Or is this entirely synthetic?What do the ACPI tables look like in the guest? Is there a GTDT table at all?
How does the counter event stream behave?
Are there other architectural features which Hyper-V does not implement for a guest?
Quoted 8 lines
Quoted 2 lines
Is there anything else that may change across a migration? e.g. MIDR? MPIDR? Any of the ID registers?
The ARMv8 architectural system counter and associated registers are visible
and functional in a VM on Hyper-V. The "arch_sys_counter" clocksource is
instantiated by thearm_arch_timer.cdriver based on the GTDT in the guest, and a Linux guest on Hyper-V runs fine with this clocksource. Low level code like bootloaders and EFI applications work normally.
That's good to hear!
One potential issue worth noting is that as those pieces of software are unlikely to handle counter frequency changes reliably, and so may not behave correctly if live-migrated.
Quoted 11 lines
The Hyper-V virtualization provides another Linux clocksource that is an overlay on the arch counter and that provides time consistency across a live migration. Live migration of ARM64 VMs on Hyper-V is not functional today, but the Hyper-V team believes they can make it functional. I have not explored with them the live migration implications of things beyond time consistency, like event streams,
CNTFRQ_EL0, MIDR/MPIDR, etc.Would a summary of your point be that live migration across hardware with different arch counter frequencies is likely to not be possible with 100% fidelity because of these other dependencies on the arch counter frequency? (hence the fixed 1 GHz frequency in ARM v8.6)
Yes.
In addition, there are a larger set of things necessarily exposed to VMs that mean that live migration isn't all that practical except betweenm identical machines (where the counter frequency should be identical), and the timer frequency might just be the canary in the coalmine. For example, the cache properties enumerated in CTR_EL0 cannot necessarily be emulated on another machine.
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
For clockevents, the only timer interrupt that Hyper-V provides in a guest VM is its virtualized "STIMER" interrupt. There's no virtualization of the ARM arch timer in the guest.
I think that is rather unfortunate, given it's a core architectural feature. Is it just the interrupt that's missing? i.e. does all the PE-local functionality behave as the architecture requires?
Quoted 6 lines
Right off the bat, I don't know about timer-related PE-local functionality as it's not exercised in a Linux VM on Hyper-V that is using STIMER-based clockevents. I'll explore with the Hyper-V team. My impression is that enabling the ARM arch timer in a guest VM is more work for Hyper-V than just wiring up an interrupt.
Thanks for chasing this up!
Mark.
RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
From: Mark Rutland <[email protected]> Sent: Tuesday, May 18, 2021 10:00 AM
Quoted 101 lines · Michael Kelley
Quoted 55 lines
From: Mark Rutland <[email protected]> Sent: Monday, May 17, 2021 6:08 AM
Quoted 54 lines · Michael Kelley
Quoted 21 lines
From: Mark Rutland <[email protected]> Sent: Friday, May 14, 2021 5:37 AM
Quoted 14 lines · Michael Kelley
Quoted 4 lines
Add architecture specific definitions and functions needed by the architecture independent Hyper-V clocksource driver. Update the Hyper-V clocksource driver to be initialized on ARM64.
Previously we've said that for a clocksource we must use the architected counter, since that's necessary for things like the VDSO to work correctly and efficiently.
Given that, I'm a bit confused that we're registering a per-cpu clocksource that is in part based on the architected counter. Likewise, I don't entirely follow why it's necessary to PV the
clock_event_device.Quoted 5 lines
Are the architected counter and timer reliable without this PV infrastructure? Why do we need to PV either of those?
Thanks,
Mark.Quoted 7 lines
For the clocksource, we have a requirement to live migrate VMs between Hyper-V hosts running on hardware that may have different arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz requirement). The Hyper-V virtualization does scaling to handle the frequency difference. And yes, there's a tradeoff with vDSO not working, though we have an out-of-tree vDSO implementation that we can use when necessary.
Just to be clear, the vDSO is *one example* of something that won't function correctly. More generally, because this undermines core architectural guarantees, it requires more invasive changes (e.g. we'd have to weaken the sanity checks, and not use the counter in things like kexec paths), impacts any architectural features tied to the generic timer/counter (e.g. the event stream, SPE and tracing, future features), and means that other SW (e.g. bootloaders and other EFI applications) are unlikley to function correctly in this environment.
I am very much not keen on trying to PV this.
What does the guest see when it reads
CNTFRQ_EL0? Does this match the real HW value (and can this change over time)? Or is this entirely synthetic?What do the ACPI tables look like in the guest? Is there a GTDT table at all?
How does the counter event stream behave?
Are there other architectural features which Hyper-V does not implement for a guest?
Quoted 8 lines
Quoted 2 lines
Is there anything else that may change across a migration? e.g. MIDR? MPIDR? Any of the ID registers?
The ARMv8 architectural system counter and associated registers are visible
and functional in a VM on Hyper-V. The "arch_sys_counter" clocksource is
instantiated by thearm_arch_timer.cdriver based on the GTDT in the guest, and a Linux guest on Hyper-V runs fine with this clocksource. Low level code like bootloaders and EFI applications work normally.That's good to hear!
One potential issue worth noting is that as those pieces of software are unlikely to handle counter frequency changes reliably, and so may not behave correctly if live-migrated.
Quoted 11 lines
The Hyper-V virtualization provides another Linux clocksource that is an overlay on the arch counter and that provides time consistency across a live migration. Live migration of ARM64 VMs on Hyper-V is not functional today, but the Hyper-V team believes they can make it functional. I have not explored with them the live migration implications of things beyond time consistency, like event streams,
CNTFRQ_EL0, MIDR/MPIDR, etc.Would a summary of your point be that live migration across hardware with different arch counter frequencies is likely to not be possible with 100% fidelity because of these other dependencies on the arch counter frequency? (hence the fixed 1 GHz frequency in ARM v8.6)
Yes.
In addition, there are a larger set of things necessarily exposed to VMs that mean that live migration isn't all that practical except betweenm identical machines (where the counter frequency should be identical), and the timer frequency might just be the canary in the coalmine. For example, the cache properties enumerated in
CTR_EL0cannot necessarily be emulated on another machine.Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
For clockevents, the only timer interrupt that Hyper-V provides in a guest VM is its virtualized "STIMER" interrupt. There's no virtualization of the ARM arch timer in the guest.
I think that is rather unfortunate, given it's a core architectural feature. Is it just the interrupt that's missing? i.e. does all the PE-local functionality behave as the architecture requires?
Quoted 9 lines
Quoted 6 lines
Right off the bat, I don't know about timer-related PE-local functionality as it's not exercised in a Linux VM on Hyper-V that is using STIMER-based clockevents. I'll explore with the Hyper-V team. My impression is that enabling the ARM arch timer in a guest VM is more work for Hyper-V than just wiring up an interrupt.
Thanks for chasing this up!
I've had a couple rounds of discussions with the Hyper-V team. For
the clocksource we've agreed to table the live migration discussion, and I'll resubmit the code so that arm_arch_timer.c provides the standard arch_sys_counter clocksource. As noted previously, this just works for a Hyper-V guest. The live migration discussion may come back later after a deeper investigation by Hyper-V.
For clockevents, there's not a near term fix. It's more than just plumbing an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM. From their perspective there's also benefit in having a timer abstraction that's independent of the architecture, and in the Linux guest, the STIMER code is common across x86/x64 and ARM64. It follows the standard Linux clockevents model, as it should. The code is already in use in out-of-tree builds in the Linux VMs included in Windows 10 on ARM64 as part of the so-called "Windows Subsystem for Linux".
So I'm hoping we can get this core support for ARM64 guests on Hyper-V into upstream using the existing STIMER support. At some point, Hyper-V will do the virtualization of the ARM64 arch timer, but we don't want to have to stay out-of-tree until after that happens.
Thoughts?
Michael
Mark Rutland
reviewerRe: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
Hi Michael,
[trimming the bulk of the thrread]
Quoted 6 lines · Michael Kelley
I've had a couple rounds of discussions with the Hyper-V team. For
the clocksource we've agreed to table the live migration discussion, and I'll resubmit the code so thatarm_arch_timer.cprovides the standardarch_sys_counterclocksource. As noted previously, this just works for a Hyper-V guest. The live migration discussion may come back later after a deeper investigation by Hyper-V.
Great; thanks for this!
Quoted 9 lines
For clockevents, there's not a near term fix. It's more than just plumbing an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM. From their perspective there's also benefit in having a timer abstraction that's independent of the architecture, and in the Linux guest, the STIMER code is common across x86/x64 and ARM64. It follows the standard Linux clockevents model, as it should. The code is already in use in out-of-tree builds in the Linux VMs included in Windows 10 on ARM64 as part of the so-called "Windows Subsystem for Linux".
Quoted 4 lines
So I'm hoping we can get this core support for ARM64 guests on Hyper-V into upstream using the existing STIMER support. At some point, Hyper-V will do the virtualization of the ARM64 arch timer, but we don't want to have to stay out-of-tree until after that happens.
My main concern here is making sure that we can rely on architected properties, and don't have to special-case architected bits for hyperv (or any other hypervisor), since that inevitably causes longer-term pain.
While in abstract I'm not as worried about using the timer clock_event_device specifically, that same driver provides the clocksource and the event stream, and I want those to work as usual, without being tied into the hyperv code. IIUC that will require some work, since the driver won't register if the GTDT is missing timer interrupts (or if there is no GTDT).
I think it really depends on what that looks like.
Thanks,
Mark.
RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
From: Mark Rutland <[email protected]> Sent: Thursday, June 10, 2021 9:45 AM
Quoted 34 lines
Hi Michael,
[trimming the bulk of the thrread]
Quoted 6 lines · Michael Kelley
I've had a couple rounds of discussions with the Hyper-V team. For
the clocksource we've agreed to table the live migration discussion, and I'll resubmit the code so thatarm_arch_timer.cprovides the standardarch_sys_counterclocksource. As noted previously, this just works for a Hyper-V guest. The live migration discussion may come back later after a deeper investigation by Hyper-V.Great; thanks for this!
Quoted 9 lines
For clockevents, there's not a near term fix. It's more than just plumbing an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM. From their perspective there's also benefit in having a timer abstraction that's independent of the architecture, and in the Linux guest, the STIMER code is common across x86/x64 and ARM64. It follows the standard Linux clockevents model, as it should. The code is already in use in out-of-tree builds in the Linux VMs included in Windows 10 on ARM64 as part of the so-called "Windows Subsystem for Linux".
Quoted 4 lines
So I'm hoping we can get this core support for ARM64 guests on Hyper-V into upstream using the existing STIMER support. At some point, Hyper-V will do the virtualization of the ARM64 arch timer, but we don't want to have to stay out-of-tree until after that happens.
My main concern here is making sure that we can rely on architected properties, and don't have to special-case architected bits for hyperv (or any other hypervisor), since that inevitably causes longer-term pain.
Quoted 8 lines
While in abstract I'm not as worried about using the timer
clock_event_devicespecifically, that same driver provides the clocksource and the event stream, and I want those to work as usual, without being tied into the hyperv code. IIUC that will require some work, since the driver won't register if the GTDT is missing timer interrupts (or if there is no GTDT).I think it really depends on what that looks like.
Mark,
Here are the details:
The existing initialization and registration code in arm_arch_timer.c works in a Hyper-V guest with no changes. As previously mentioned, the GTDT exists and is correctly populated. Even though it isn't used, there's a PPI INTID specified for the virtual timer, just so the "arm_sys_timer" clockevent can be initialized and registered. The IRQ shows up in the output of "cat /proc/interrupts" with zero counts for all CPUs since no interrupts are ever generated. The EL1 virtual timer registers (CNTV_CVAL_EL0, CNTV_TVAL_EL0, and CNTV_CTL_EL0) are accessible in the VM. The "arm_sys_timer" clockevent is left in a shutdown state with CNTV_CTL_EL0.ENABLE set to zero when the Hyper-V STIMER clockevent is registered with a higher rating.
Event streams are initialized and the __delay() implementation for ARM64 inside the kernel works. However, on the Ampere eMAG hardware I'm using for testing, the WFE instruction returns more quickly than it should even though the event stream fields in CNTKCTL_EL1 are correct. I have a query in to the Hyper-V team to see if they are trapping WFE and just returning, vs. perhaps the eMAG processor takes the easy way out and has WFE just return immediately. I'm not knowledgeable about other uses of timer event streams, so let me know if there are other usage scenarios I should check.
Finally, the "arch_sys_counter" clocksource gets initialized and setup correctly. If the Hyper-V clocksource is also initialized, you can flip between the two clocksources at runtime as expected. If the Hyper-V clocksource is not setup, then Linux in the VM runs fine with the "arch_sys_counter" clocksource.
Michael
RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
From: Michael Kelley <[email protected]> Sent: Sunday, June 13, 2021 7:42 PM
Quoted 61 lines
From: Mark Rutland <[email protected]> Sent: Thursday, June 10, 2021 9:45 AM
Quoted 34 lines
Hi Michael,
[trimming the bulk of the thrread]
Quoted 6 lines · Michael Kelley
I've had a couple rounds of discussions with the Hyper-V team. For
the clocksource we've agreed to table the live migration discussion, and I'll resubmit the code so thatarm_arch_timer.cprovides the standardarch_sys_counterclocksource. As noted previously, this just works for a Hyper-V guest. The live migration discussion may come back later after a deeper investigation by Hyper-V.Great; thanks for this!
Quoted 9 lines
For clockevents, there's not a near term fix. It's more than just plumbing an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM. From their perspective there's also benefit in having a timer abstraction that's independent of the architecture, and in the Linux guest, the STIMER code is common across x86/x64 and ARM64. It follows the standard Linux clockevents model, as it should. The code is already in use in out-of-tree builds in the Linux VMs included in Windows 10 on ARM64 as part of the so-called "Windows Subsystem for Linux".
Quoted 4 lines
So I'm hoping we can get this core support for ARM64 guests on Hyper-V into upstream using the existing STIMER support. At some point, Hyper-V will do the virtualization of the ARM64 arch timer, but we don't want to have to stay out-of-tree until after that happens.
My main concern here is making sure that we can rely on architected properties, and don't have to special-case architected bits for hyperv (or any other hypervisor), since that inevitably causes longer-term pain.
Quoted 8 lines
While in abstract I'm not as worried about using the timer
clock_event_devicespecifically, that same driver provides the clocksource and the event stream, and I want those to work as usual, without being tied into the hyperv code. IIUC that will require some work, since the driver won't register if the GTDT is missing timer interrupts (or if there is no GTDT).I think it really depends on what that looks like.
Mark,
Here are the details:
The existing initialization and registration code in
arm_arch_timer.cworks in a Hyper-V guest with no changes. As previously mentioned, the GTDT exists and is correctly populated. Even though it isn't used, there's a PPI INTID specified for the virtual timer, just so the "arm_sys_timer" clockevent can be initialized and registered. The IRQ shows up in the output of "cat/proc/interrupts" with zero counts for all CPUs since no interrupts are ever generated. The EL1 virtual timer registers (CNTV_CVAL_EL0,CNTV_TVAL_EL0, andCNTV_CTL_EL0) are accessible in the VM. The "arm_sys_timer" clockevent is left in a shutdown state with CNTV_CTL_EL0.ENABLE set to zero when the Hyper-V STIMER clockevent is registered with a higher rating.
Quoted 10 lines
Event streams are initialized and the
__delay()implementation for ARM64 inside the kernel works. However, on the Ampere eMAG hardware I'm using for testing, the WFE instruction returns more quickly than it should even though the event stream fields inCNTKCTL_EL1are correct. I have a query in to the Hyper-V team to see if they are trapping WFE and just returning, vs. perhaps the eMAG processor takes the easy way out and has WFE just return immediately. I'm not knowledgeable about other uses of timer event streams, so let me know if there are other usage scenarios I should check.
I confirmed that Hyper-V is not trapping the WFE instruction. And on a Marvell TX2 and on an Ampere Altra, the counter event stream and WFE provide the expected delay. Evidently WFE on the eMAG doesn't actually delay. Bottom line: event streams work as expected in a Hyper-V VM. No changes needed to arm_arch_timer.[ch].
Michael
Quoted 8 lines
Finally, the "
arch_sys_counter" clocksource gets initialized and setup correctly. If the Hyper-V clocksource is also initialized, you can flip between the two clocksources at runtime as expected. If the Hyper-V clocksource is not setup, then Linux in the VM runs fine with the "arch_sys_counter" clocksource.Michael
Mark Rutland
reviewerRe: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
In reply to Michael KelleyHi Michael,
Thanks for all this; comments inline below. I've added Marc Zyngier, who co-maintains the architected timer code.
Quoted 43 lines · Michael Kelley
From: Mark Rutland <[email protected]> Sent: Thursday, June 10, 2021 9:45 AM
Quoted 29 lines · Michael Kelley
Quoted 6 lines
I've had a couple rounds of discussions with the Hyper-V team. For
the clocksource we've agreed to table the live migration discussion, and I'll resubmit the code so thatarm_arch_timer.cprovides the standardarch_sys_counterclocksource. As noted previously, this just works for a Hyper-V guest. The live migration discussion may come back later after a deeper investigation by Hyper-V.Great; thanks for this!
Quoted 9 lines
For clockevents, there's not a near term fix. It's more than just plumbing an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM. From their perspective there's also benefit in having a timer abstraction that's independent of the architecture, and in the Linux guest, the STIMER code is common across x86/x64 and ARM64. It follows the standard Linux clockevents model, as it should. The code is already in use in out-of-tree builds in the Linux VMs included in Windows 10 on ARM64 as part of the so-called "Windows Subsystem for Linux".
Quoted 4 lines
So I'm hoping we can get this core support for ARM64 guests on Hyper-V into upstream using the existing STIMER support. At some point, Hyper-V will do the virtualization of the ARM64 arch timer, but we don't want to have to stay out-of-tree until after that happens.
My main concern here is making sure that we can rely on architected properties, and don't have to special-case architected bits for hyperv (or any other hypervisor), since that inevitably causes longer-term pain.
Quoted 8 lines
While in abstract I'm not as worried about using the timer
clock_event_devicespecifically, that same driver provides the clocksource and the event stream, and I want those to work as usual, without being tied into the hyperv code. IIUC that will require some work, since the driver won't register if the GTDT is missing timer interrupts (or if there is no GTDT).I think it really depends on what that looks like.
Mark,
Here are the details:
Quoted 11 lines
The existing initialization and registration code in
arm_arch_timer.cworks in a Hyper-V guest with no changes. As previously mentioned, the GTDT exists and is correctly populated. Even though it isn't used, there's a PPI INTID specified for the virtual timer, just so the "arm_sys_timer" clockevent can be initialized and registered. The IRQ shows up in the output of "cat/proc/interrupts" with zero counts for all CPUs since no interrupts are ever generated. The EL1 virtual timer registers (CNTV_CVAL_EL0,CNTV_TVAL_EL0, andCNTV_CTL_EL0) are accessible in the VM. The "arm_sys_timer" clockevent is left in a shutdown state with CNTV_CTL_EL0.ENABLE set to zero when the Hyper-V STIMER clockevent is registered with a higher rating.
This concerns me, since we're lying to the kernel, and assuming that it will never try to use this timer. I appreciate that evidently we don't happen to rely on that today if you register a higher priority timer, but that does open us up to future fragility (e.g. if we added sanity checks when registering timers), and IIRC there are ways for userspace to change the clockevent device today.
Quoted 10 lines
Event streams are initialized and the
__delay()implementation for ARM64 inside the kernel works. However, on the Ampere eMAG hardware I'm using for testing, the WFE instruction returns more quickly than it should even though the event stream fields inCNTKCTL_EL1are correct. I have a query in to the Hyper-V team to see if they are trapping WFE and just returning, vs. perhaps the eMAG processor takes the easy way out and has WFE just return immediately. I'm not knowledgeable about other uses of timer event streams, so let me know if there are other usage scenarios I should check.
I saw your reply confirming that this is gnerally working as expected (and that Hyper-V is not trapping WFE) so this sounds fine to me.
Quoted 5 lines
Finally, the "
arch_sys_counter" clocksource gets initialized and setup correctly. If the Hyper-V clocksource is also initialized, you can flip between the two clocksources at runtime as expected. If the Hyper-V clocksource is not setup, then Linux in the VM runs fine with the "arch_sys_counter" clocksource.
Great!
As above, my remaining concern here is fragility around the clockevent_device; I'm not keen that we're lying (in the GTDT) that interrupts are wired up when they not functional, and while you can get away with that today, that relies on kernel implementation details that could change.
Ideally, Hyper-V would provide the architectural timer (as it's already claiming to in the GTDT), things would "just work", and the Hyper-V timer would be an optimization rather than a functional necessity.
You mentioned above that Hyper-V will virtualize the timer "at some point" -- is that already planned, and when is that likely to be?
Marc, do you have any thoughts on this?
Thanks,
Mark.
Re: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
Quoted 62 lines · Mark Rutland
Hi Michael,
Thanks for all this; comments inline below. I've added Marc Zyngier, who co-maintains the architected timer code.
Quoted 43 lines · Michael Kelley
From: Mark Rutland <[email protected]> Sent: Thursday, June 10, 2021 9:45 AM
Quoted 29 lines · Michael Kelley
Quoted 6 lines
I've had a couple rounds of discussions with the Hyper-V team. For
the clocksource we've agreed to table the live migration discussion, and I'll resubmit the code so thatarm_arch_timer.cprovides the standardarch_sys_counterclocksource. As noted previously, this just works for a Hyper-V guest. The live migration discussion may come back later after a deeper investigation by Hyper-V.Great; thanks for this!
Quoted 9 lines
For clockevents, there's not a near term fix. It's more than just plumbing an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM. From their perspective there's also benefit in having a timer abstraction that's independent of the architecture, and in the Linux guest, the STIMER code is common across x86/x64 and ARM64. It follows the standard Linux clockevents model, as it should. The code is already in use in out-of-tree builds in the Linux VMs included in Windows 10 on ARM64 as part of the so-called "Windows Subsystem for Linux".
Quoted 4 lines
So I'm hoping we can get this core support for ARM64 guests on Hyper-V into upstream using the existing STIMER support. At some point, Hyper-V will do the virtualization of the ARM64 arch timer, but we don't want to have to stay out-of-tree until after that happens.
My main concern here is making sure that we can rely on architected properties, and don't have to special-case architected bits for hyperv (or any other hypervisor), since that inevitably causes longer-term pain.
Quoted 8 lines
While in abstract I'm not as worried about using the timer
clock_event_devicespecifically, that same driver provides the clocksource and the event stream, and I want those to work as usual, without being tied into the hyperv code. IIUC that will require some work, since the driver won't register if the GTDT is missing timer interrupts (or if there is no GTDT).I think it really depends on what that looks like.
Mark,
Here are the details:
Quoted 11 lines
The existing initialization and registration code in
arm_arch_timer.cworks in a Hyper-V guest with no changes. As previously mentioned, the GTDT exists and is correctly populated. Even though it isn't used, there's a PPI INTID specified for the virtual timer, just so the "arm_sys_timer" clockevent can be initialized and registered. The IRQ shows up in the output of "cat/proc/interrupts" with zero counts for all CPUs since no interrupts are ever generated. The EL1 virtual timer registers (CNTV_CVAL_EL0,CNTV_TVAL_EL0, andCNTV_CTL_EL0) are accessible in the VM. The "arm_sys_timer" clockevent is left in a shutdown state with CNTV_CTL_EL0.ENABLE set to zero when the Hyper-V STIMER clockevent is registered with a higher rating.
Quoted 6 lines
This concerns me, since we're lying to the kernel, and assuming that it will never try to use this timer. I appreciate that evidently we don't happen to rely on that today if you register a higher priority timer, but that does open us up to future fragility (e.g. if we added sanity checks when registering timers), and IIRC there are ways for userspace to change the clockevent device today.
Indeed. Userspace can perfectly unbind the clockevent using /sys/devices/system/clockevents/clockevent*/unbind_device, and the kernel will be happy to switch to the next per-cpu timer, which happens to be the arch timer. Oh wait...
Quoted 29 lines
Quoted 11 lines
Event streams are initialized and the
__delay()implementation for ARM64 inside the kernel works. However, on the Ampere eMAG hardware I'm using for testing, the WFE instruction returns more quickly than it should even though the event stream fields inCNTKCTL_EL1are correct. I have a query in to the Hyper-V team to see if they are trapping WFE and just returning, vs. perhaps the eMAG processor takes the easy way out and has WFE just return immediately. I'm not knowledgeable about other uses of timer event streams, so let me know if there are other usage scenarios I should check.I saw your reply confirming that this is gnerally working as expected (and that Hyper-V is not trapping WFE) so this sounds fine to me.
Quoted 5 lines
Finally, the "
arch_sys_counter" clocksource gets initialized and setup correctly. If the Hyper-V clocksource is also initialized, you can flip between the two clocksources at runtime as expected. If the Hyper-V clocksource is not setup, then Linux in the VM runs fine with the "arch_sys_counter" clocksource.Great!
As above, my remaining concern here is fragility around the
clockevent_device; I'm not keen that we're lying (in the GTDT) that interrupts are wired up when they not functional, and while you can get away with that today, that relies on kernel implementation details that could change.
Quoted 8 lines
Ideally, Hyper-V would provide the architectural timer (as it's already claiming to in the GTDT), things would "just work", and the Hyper-V timer would be an optimization rather than a functional necessity.
You mentioned above that Hyper-V will virtualize the timer "at some point" -- is that already planned, and when is that likely to be?
Marc, do you have any thoughts on this?
Overall, lying to the kernel is a bad idea. Only implementing half of the architecture is another bad idea. I doubt the combination of two bad ideas produces a good one.
If Hyper-V guests need to use another timer (for migration purposes?), that's fine. But we rely on both the base architecture to be completely implemented *and* on advertised features to be functional. I think this has been our position since the first Hyper-V patches were posted... 3 years ago?
What is the hold up for reliably virtualising the arch timer, including interrupt delivery?
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent support
From: Marc Zyngier <[email protected]> Sent: Wednesday, June 23, 2021 1:56 AM
Quoted 119 lines
Quoted 62 lines · Mark Rutland
Hi Michael,
Thanks for all this; comments inline below. I've added Marc Zyngier, who co-maintains the architected timer code.
Quoted 43 lines · Michael Kelley
From: Mark Rutland <[email protected]> Sent: Thursday, June 10, 2021 9:45 AM
Quoted 29 lines · Michael Kelley
Quoted 6 lines
I've had a couple rounds of discussions with the Hyper-V team. For
the clocksource we've agreed to table the live migration discussion, and I'll resubmit the code so thatarm_arch_timer.cprovides the standardarch_sys_counterclocksource. As noted previously, this just works for a Hyper-V guest. The live migration discussion may come back later after a deeper investigation by Hyper-V.Great; thanks for this!
Quoted 9 lines
For clockevents, there's not a near term fix. It's more than just plumbing an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM. From their perspective there's also benefit in having a timer abstraction that's independent of the architecture, and in the Linux guest, the STIMER code is common across x86/x64 and ARM64. It follows the standard Linux clockevents model, as it should. The code is already in use in out-of-tree builds in the Linux VMs included in Windows 10 on ARM64 as part of the so-called "Windows Subsystem for Linux".
Quoted 4 lines
So I'm hoping we can get this core support for ARM64 guests on Hyper-V into upstream using the existing STIMER support. At some point, Hyper-V will do the virtualization of the ARM64 arch timer, but we don't want to have to stay out-of-tree until after that happens.
My main concern here is making sure that we can rely on architected properties, and don't have to special-case architected bits for hyperv (or any other hypervisor), since that inevitably causes longer-term pain.
Quoted 8 lines
While in abstract I'm not as worried about using the timer
clock_event_devicespecifically, that same driver provides the clocksource and the event stream, and I want those to work as usual, without being tied into the hyperv code. IIUC that will require some work, since the driver won't register if the GTDT is missing timer interrupts (or if there is no GTDT).I think it really depends on what that looks like.
Mark,
Here are the details:
Quoted 11 lines
The existing initialization and registration code in
arm_arch_timer.cworks in a Hyper-V guest with no changes. As previously mentioned, the GTDT exists and is correctly populated. Even though it isn't used, there's a PPI INTID specified for the virtual timer, just so the "arm_sys_timer" clockevent can be initialized and registered. The IRQ shows up in the output of "cat/proc/interrupts" with zero counts for all CPUs since no interrupts are ever generated. The EL1 virtual timer registers (CNTV_CVAL_EL0,CNTV_TVAL_EL0, andCNTV_CTL_EL0) are accessible in the VM. The "arm_sys_timer" clockevent is left in a shutdown state with CNTV_CTL_EL0.ENABLE set to zero when the Hyper-V STIMER clockevent is registered with a higher rating.Quoted 6 lines
This concerns me, since we're lying to the kernel, and assuming that it will never try to use this timer. I appreciate that evidently we don't happen to rely on that today if you register a higher priority timer, but that does open us up to future fragility (e.g. if we added sanity checks when registering timers), and IIRC there are ways for userspace to change the clockevent device today.
Indeed. Userspace can perfectly unbind the clockevent using
/sys/devices/system/clockevents/clockevent*/unbind_device, and the kernel will be happy to switch to the next per-cpu timer, which happens to be the arch timer. Oh wait...Quoted 29 lines
Quoted 11 lines
Event streams are initialized and the
__delay()implementation for ARM64 inside the kernel works. However, on the Ampere eMAG hardware I'm using for testing, the WFE instruction returns more quickly than it should even though the event stream fields inCNTKCTL_EL1are correct. I have a query in to the Hyper-V team to see if they are trapping WFE and just returning, vs. perhaps the eMAG processor takes the easy way out and has WFE just return immediately. I'm not knowledgeable about other uses of timer event streams, so let me know if there are other usage scenarios I should check.I saw your reply confirming that this is gnerally working as expected (and that Hyper-V is not trapping WFE) so this sounds fine to me.
Quoted 5 lines
Finally, the "
arch_sys_counter" clocksource gets initialized and setup correctly. If the Hyper-V clocksource is also initialized, you can flip between the two clocksources at runtime as expected. If the Hyper-V clocksource is not setup, then Linux in the VM runs fine with the "arch_sys_counter" clocksource.Great!
As above, my remaining concern here is fragility around the
clockevent_device; I'm not keen that we're lying (in the GTDT) that interrupts are wired up when they not functional, and while you can get away with that today, that relies on kernel implementation details that could change.Quoted 8 lines
Ideally, Hyper-V would provide the architectural timer (as it's already claiming to in the GTDT), things would "just work", and the Hyper-V timer would be an optimization rather than a functional necessity.
You mentioned above that Hyper-V will virtualize the timer "at some point" -- is that already planned, and when is that likely to be?
Marc, do you have any thoughts on this?
Overall, lying to the kernel is a bad idea. Only implementing half of the architecture is another bad idea. I doubt the combination of two bad ideas produces a good one.
Quoted 8 lines
If Hyper-V guests need to use another timer (for migration purposes?), that's fine. But we rely on both the base architecture to be completely implemented *and* on advertised features to be functional. I think this has been our position since the first Hyper-V patches were posted... 3 years ago?
What is the hold up for reliably virtualising the arch timer, including interrupt delivery?
Marc (and Mark) --
In our early interactions about the Hyper-V clocks and timers, the code was a bit spread out, and you suggested moving all the clocksource and clockevent stuff to a driver under drivers/clocksource. See
https://lore.kernel.org/lkml/[email protected]/.
That was a good change independent of any ARM64 considerations, but I read (or perhaps overread) your comments to say that it was OK to use these Hyper-V para-virtualized clocks/timers instead of the ARM64 architectural ones in a Hyper-V VM. They work and it's what the Hyper-V guys wanted to do anyway, so having Hyper-V offer the ARM64 arch counter and timer in a VM hasn't been a priority. They had other stuff that didn't work at all on ARM64, so that's where their attention went.
I agree that it would be better to have the ARM64 arch counter/timer fully implemented in a Hyper-V VM. But we're wanting to find a practical way to move forward, and in doing so confine any rough edges to Hyper-V VMs and the Hyper-V specific code in the kernel tree. We're maintaining and shipping the code out-of-tree based on Hyper-V ARM64 current behavior and would like to get this core enablement code upstream. Sure, unbinding the Hyper-V clockevent doesn't work, but that's not a problem in any use cases we see from customers.
All that said, our discussions with the Hyper-V team are continuing. We're still in the process of seeing what's practical to get and when.
Michael
Quoted 7 lines
Thanks,
M.--
Without deviation from the norm, progress is not possible.
PATCH v10 4/7 arm64: hyperv: Add kexec and panic handlers
In reply to Michael KelleyAdd function to inform Hyper-V about a guest panic.
Also add functions to set up and remove kexec and panic handlers, which are currently unused on ARM64 but are called from architecture independent code in the VMbus driver.
This code is built only when CONFIG_HYPERV is enabled.
--- arch/arm64/hyperv/Makefile | 2 +- arch/arm64/hyperv/hv_core.c | 52 ++++++++++++++++++++++++++++++++++++++++++ arch/arm64/hyperv/mshyperv.c | 54 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 107 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/hyperv/mshyperv.c
arch/arm64/hyperv/Makefile +1 -1
@@ -1,2 +1,2 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y := hv_core.o +obj-y := hv_core.o mshyperv.o
arch/arm64/hyperv/hv_core.c +52 -0
55 lines not on this page · raw message
arch/arm64/hyperv/mshyperv.c new file +54 -0
54 lines not on this page · raw message
Signature · 7 lines
1.8.3.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v10 5/7 arm64: hyperv: Initialize hypervisor on boot
In reply to Michael KelleyAdd ARM64-specific code to initialize the Hyper-V hypervisor when booting as a guest VM. Provide functions and data structures indicating hypervisor status that are needed by VMbus driver.
This code is built only when CONFIG_HYPERV is enabled.
--- arch/arm64/hyperv/mshyperv.c | 152 ++++++++++++++++++++++++++++++++++++++ arch/arm64/include/asm/mshyperv.h | 6 ++ arch/arm64/kernel/setup.c | 4 + 3 files changed, 162 insertions(+)
arch/arm64/hyperv/mshyperv.c +152 -0
@@ -14,6 +14,158 @@#include <linux/types.h> #include <linux/export.h> #include <linux/ptrace.h> +#include <linux/errno.h> +#include <linux/acpi.h> +#include <linux/version.h> +#include <linux/cpuhotplug.h> +#include <linux/slab.h> +#include <linux/cpumask.h> +#include <asm/mshyperv.h> + +static bool hyperv_initialized; +struct ms_hyperv_info ms_hyperv__ro_after_init ; +EXPORT_SYMBOL_GPL (ms_hyperv); + +bool hv_root_partition; +EXPORT_SYMBOL_GPL (hv_root_partition); + +u32 *hv_vp_index; +EXPORT_SYMBOL_GPL (hv_vp_index); + +u32 hv_max_vp_index; +EXPORT_SYMBOL_GPL (hv_max_vp_index); + +void __percpu **hyperv_pcpu_input_arg; +EXPORT_SYMBOL_GPL (hyperv_pcpu_input_arg); + +static int hv_cpu_init (unsigned int cpu) +{ +void **input_arg = (void **)this_cpu_ptr (hyperv_pcpu_input_arg); + + hv_vp_index[cpu] =hv_get_vpreg (HV_REGISTER_VP_INDEX ); + + *input_arg =kmalloc (HV_HYP_PAGE_SIZE ,GFP_KERNEL ); +if (!(*input_arg)) +return -ENOMEM ; + +return 0 ; +} + +void __init hyperv_early_init (void ) +{ +struct hv_get_vp_registers_output result; +u32 a, b, c, d; +u64 guest_id; + +/* +* If we're in a VM on Hyper-V, the ACPI hypervisor_id field will +* have the string "MsHyperV". +*/ +if (strncmp ((char *)&acpi_gbl_FADT.hypervisor_id,"MsHyperV" ,8 )) +return ; + +/* Setup the guest ID */ + guest_id =generate_guest_id (0 ,LINUX_VERSION_CODE ,0 ); +hv_set_vpreg (HV_REGISTER_GUEST_OSID , guest_id); + +/* Get the features and hints from Hyper-V */ +hv_get_vpreg_128 (HV_REGISTER_FEATURES , &result); + ms_hyperv.features = result.as32.a; + ms_hyperv.priv_high = result.as32.b; + ms_hyperv.misc_features = result.as32.c; + +hv_get_vpreg_128 (HV_REGISTER_ENLIGHTENMENTS , &result); + ms_hyperv.hints = result.as32.a; + +pr_info ("Hyper-V: privilege flags low 0x%x, high 0x%x, hints 0x%x, misc 0x%x\n" , + ms_hyperv.features, ms_hyperv.priv_high, ms_hyperv.hints, + ms_hyperv.misc_features); + +/* +* If Hyper-V has crash notifications, set crash_kexec_post_notifiers +* so that we will report the panic to Hyper-V before running kdump. +*/ +if (ms_hyperv.misc_features &HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE ) + crash_kexec_post_notifiers = true; + +/* Get information about the Hyper-V host version */ +hv_get_vpreg_128 (HV_REGISTER_HYPERVISOR_VERSION , &result); + a = result.as32.a; + b = result.as32.b; + c = result.as32.c; + d = result.as32.d; +pr_info ("Hyper-V: Host Build %d.%d.%d.%d-%d-%d\n" , + b >>16 , b &0xFFFF , a, d &0xFFFFFF , c, d >>24 ); + + hyperv_initialized = true; +} + +static int __init hyperv_init (void ) +{ +int i, ret; + + hyperv_pcpu_input_arg = alloc_percpu(void *); +if (!hyperv_pcpu_input_arg) +return -ENOMEM ; + +/* Allocate and initialize percpu VP index array */ + hv_max_vp_index =num_possible_cpus (); + hv_vp_index =kmalloc_array (hv_max_vp_index,sizeof (*hv_vp_index), +GFP_KERNEL ); +if (!hv_vp_index) { + ret = -ENOMEM ; + goto free_input_arg; + } + +for (i =0 ; i < hv_max_vp_index; i++) + hv_vp_index[i] =VP_INVAL ; + +if (cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"arm64/hyperv_init:online" , + hv_cpu_init,NULL ) <0 ) { + ret = -EINVAL ; + goto free_vp_index; + } + +return 0 ; + +free_vp_index: +kfree (hv_vp_index); + hv_vp_index =NULL ; + +free_input_arg: + hv_max_vp_index =0 ; +free_percpu (hyperv_pcpu_input_arg); + hyperv_pcpu_input_arg =NULL ; +return ret; +} + +early_initcall (hyperv_init); + +/* This routine is called before kexec/kdump. It does required cleanup. */ +void hyperv_cleanup (void ) +{ +hv_set_vpreg (HV_REGISTER_GUEST_OSID ,0 ); + +} +EXPORT_SYMBOL_GPL (hyperv_cleanup); + +bool hv_is_hyperv_initialized (void ) +{ +return hyperv_initialized; +} +EXPORT_SYMBOL_GPL (hv_is_hyperv_initialized); + +bool hv_is_hibernation_supported (void ) +{ +return false; +} +EXPORT_SYMBOL_GPL (hv_is_hibernation_supported); + +bool hv_is_isolation_supported (void ) +{ +return false; +} +EXPORT_SYMBOL_GPL (hv_is_isolation_supported);/* * The VMbus handler functions are no-ops on ARM64 because
arch/arm64/include/asm/mshyperv.h +6 -0
@@ -23,6 +23,12 @@#include <asm/hyperv-tlfs.h> #include <clocksource/arm_arch_timer.h> +#if IS_ENABLED (CONFIG_HYPERV ) +void __init hyperv_early_init (void ); +#else +static inline void hyperv_early_init (void ) {}; +#endif +/* * Declare calls to get and set Hyper-V VP register values on ARM64, which * requires a hypercall.
arch/arm64/kernel/setup.c +4 -0
@@ -49,6 +49,7 @@#include <asm/traps.h> #include <asm/efi.h> #include <asm/xen/hypervisor.h> +#include <asm/mshyperv.h> #include <asm/mmu_context.h> static int num_standard_resources;
@@ -355,6 +356,9 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)if (acpi_disabled)unflatten_device_tree (); +/* Do after acpi_boot_table_init() so local FADT is available */ +hyperv_early_init (); +bootmem_init ();kasan_init ();
Signature · 7 lines
1.8.3.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Wei Liu
maintainerRe: [PATCH v10 5/7] arm64: hyperv: Initialize hypervisor on boot
Quoted 8 lines · Michael Kelley
Add ARM64-specific code to initialize the Hyper-V hypervisor when booting as a guest VM. Provide functions and data structures indicating hypervisor status that are needed by VMbus driver.
This code is built only when
CONFIG_HYPERVis enabled.Signed-off-by Michael Kelley <[email protected]>
[...]
Quoted 16 lines
/*
* Declare calls to get and set Hyper-V VP register values on ARM64, which
* requires a hypercall.arch/arm64/kernel/setup.c +1
-0@@ -49,6 +49,7 @@#include <asm/traps.h> #include <asm/efi.h> #include <asm/xen/hypervisor.h> +#include <asm/mshyperv.h> #include <asm/mmu_context.h> static int num_standard_resources;
arch/arm64/kernel/setup.c void __init __no_sanitize_address setup_arch(char **cmdline_p) · line 361
Quoted 6 lines
arch/arm64/kernel/setup.c +3
-0@@ -355,6 +356,9 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)if (acpi_disabled)unflatten_device_tree (); +/* Do after acpi_boot_table_init() so local FADT is available */ +hyperv_early_init (); +
Arm maintainers, this requires your attention. Thanks.
The rest is Hyper-V specific, feel free to skip that portion.
Wei.
Quoted 6 lines
arch/arm64/kernel/setup.c +0
-0bootmem_init ();kasan_init ();Signature · 1 line
1.8.3.1
PATCH v10 6/7 arm64: efi: Export screen_info
In reply to Michael KelleyThe Hyper-V frame buffer driver may be built as a module, and it needs access to screen_info. So export screen_info.
--- arch/arm64/kernel/efi.c | 1 + 1 file changed, 1 insertion(+)
arch/arm64/kernel/efi.c +1 -0
@@ -55,6 +55,7 @@ static __init pteval_t create_mapping_protection(efi_memory_desc_t *md)/* we will fill this structure from the stub, so don't put it in .bss */ struct screen_info screen_info __section(".data"); +EXPORT_SYMBOL (screen_info);int __init efi_create_mapping (struct mm_struct *mm,efi_memory_desc_t *md) {
Signature · 7 lines
1.8.3.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v10 7/7 Drivers: hv: Enable Hyper-V code to be built on ARM64
In reply to Michael KelleyUpdate drivers/hv/Kconfig so CONFIG_HYPERV can be selected on ARM64, causing the Hyper-V specific code to be built.
--- drivers/hv/Kconfig | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/hv/Kconfig +2 -1
@@ -4,7 +4,8 @@ menu "Microsoft Hyper-V guest support" config HYPERV tristate "Microsoft Hyper-V client drivers" - depends on X86 && ACPI && X86_LOCAL_APIC && HYPERVISOR_GUEST + depends on ACPI && ((X86 && X86_LOCAL_APIC && HYPERVISOR_GUEST) \ + || (ARM64 && !CPU_BIG_ENDIAN)) select PARAVIRT select X86_HV_CALLBACK_VECTOR help
Signature · 7 lines
1.8.3.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Hi Will,
Quoted 5 lines · Michael Kelley
This patch set is based on the 5.13-rc1 code tree, plus a patch from Sudeep Holla that implements SMCCC v1.2 HVC calls:
https://lore.kernel.org/linux-arm-kernel/[email protected]/
Assuming that you will be handling v5.14, I am asking you. I plan to post the above mention patch that implements SMCCC v1.2 SMC/HVC independent of my FFA series with small change as suggested by Mark R.
Irrespective of readiness of this series or FFA, is it possible to pull the patch and share a branch based on v5.13-rc1(Arm SoC requirement), so that if FFA is ready I can pull the branch along with FFA patches and send it to Arm SoC. You can do the same for this series if it gets ready. I am just trying to avoid last minute confusion. Let me know if you have any alternative suggestions.
--
Regards,
Sudeep