Independent per-CPU data section for nVHE
Subsystems (3)
v4 cover Independent per-CPU data section for nVHE
Introduce '.hyp.data..percpu' as part of ongoing effort to make nVHE hyp code self-contained and independent of the rest of the kernel.
Main benefits:
* independent nVHE per-CPU data section that can be unmapped from host,
* more robust linking of nVHE hyp code,
* no need for hyp-specific macros to access per-CPU variables.
The series is structured as follows:
- patch 1: Improve existing hyp build rules. This could be sent and merged independently of per-CPU but this series builds on it.
- patches 2-3: Minor cleanups.
- patches 4-5: Replace hyp helpers for accessing per-CPU variables with common helpers modified to work correctly in hyp. Per-CPU variables can now be accessed with one API anywhere.
- patches 6-8: Where VHE and nVHE use per-CPU variables defined in kernel proper, move their definitions to hyp/ where they are duplicated and owned by VHE/nVHE, respectively. Non-VHE hyp code now refers only to per-CPU variables defined in its source files. Helpers are added so that kernel proper can continue to access nVHE hyp variables, same way as it does with other nVHE symbols.
- patches 9-10: Introduce '.hyp.data..percpu' ELF section and allocate memory for every CPU core during KVM init. All nVHE per-CPU state is now grouped together in ELF and in memory. Introducing a new per-CPU variable does not require adding new memory mappings any more. nVHE hyp code cannot accidentally refer to kernel-proper per-CPU data as it only has the pointer to its own per-CPU memory.
Patches are rebased on v5.9-rc6 and available in branch 'topic/percpu-v4' at:
https://android-kvm.googlesource.com/linux
For maintainers: In case of interest, there are patches that remove the need for redefining macros under DEBUG_PREEMPT available at the same repo, branch 'topic/percpu-v3-debug-preempt'. Since they are non-trivial, I am not going to post them here so late in the 5.10 window. I plan to post them for 5.11 when they will also be useful for other patches.
Changes v3 -> v4:
* Drop patch that marked pages allocated for hyp reserved
* Copy arm64_ssbd_callback_required at cpu_init_hyp_mode
* Use read_sysreg in __hyp_my_cpu_offset
* Simplify per-CPU region allocation code
* Use same subsection name regex as other kernel linker scripts
* Rename helper per-CPU macros
* Add .gitignore for hyp linker script
Changes v2 -> v3:
* Use PERCPU_INPUT in hyp.ld instead of modifying PERCPU_SECTION
* Only pass linker script once to LD (fix error message)
* Renamed '.hyp.o' to '.nvhe.o'
* Use __KVM_VHE_HYPERVISOR__ to select TPIDR_EL2 instead of alternatives
* Move all prefixing-related macros to hyp_image.h
Changes v1 -> v2:
* partially link hyp code, add linker script
David Brazdil (10):
kvm: arm64: Partially link nVHE hyp code, simplify HYPCOPY
kvm: arm64: Move nVHE hyp namespace macros to hyp_image.h
kvm: arm64: Only define __kvm_ex_table for CONFIG_KVM
kvm: arm64: Remove __hyp_this_cpu_read
kvm: arm64: Remove hyp_adr/ldr_this_cpu
kvm: arm64: Add helpers for accessing nVHE hyp per-cpu vars
kvm: arm64: Duplicate arm64_ssbd_callback_required for nVHE hyp
kvm: arm64: Create separate instances of kvm_host_data for VHE/nVHE
kvm: arm64: Set up hyp percpu data for nVHE
kvm: arm64: Remove unnecessary hyp mappings
arch/arm64/include/asm/assembler.h | 29 +++++--- arch/arm64/include/asm/hyp_image.h | 36 ++++++++++ arch/arm64/include/asm/kvm_asm.h | 82 +++++++++++------------ arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/include/asm/kvm_mmu.h | 22 ++---- arch/arm64/include/asm/percpu.h | 28 +++++++- arch/arm64/kernel/image-vars.h | 4 -- arch/arm64/kernel/vmlinux.lds.S | 13 ++++ arch/arm64/kvm/arm.c | 61 +++++++++++++---- arch/arm64/kvm/hyp/hyp-entry.S | 2 +- arch/arm64/kvm/hyp/include/hyp/debug-sr.h | 4 +- arch/arm64/kvm/hyp/include/hyp/switch.h | 8 +-- arch/arm64/kvm/hyp/nvhe/.gitignore | 2 + arch/arm64/kvm/hyp/nvhe/Makefile | 60 +++++++++-------- arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 19 ++++++ arch/arm64/kvm/hyp/nvhe/switch.c | 8 ++- arch/arm64/kvm/hyp/vhe/switch.c | 5 +- arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 4 +- arch/arm64/kvm/pmu.c | 13 ++-- 19 files changed, 272 insertions(+), 130 deletions(-) create mode 100644 arch/arm64/include/asm/hyp_image.h create mode 100644 arch/arm64/kvm/hyp/nvhe/.gitignore create mode 100644 arch/arm64/kvm/hyp/nvhe/hyp.lds.S
--
2.28.0.681.g6f77f65b4e-goog
PATCH v4 1/10 kvm: arm64: Partially link nVHE hyp code, simplify HYPCOPY
Relying on objcopy to prefix the ELF section names of the nVHE hyp code is brittle and prevents us from using wildcards to match specific section names.
Improve the build rules by partially linking all '.nvhe.o' files and prefixing their ELF section names using a linker script. Continue using objcopy for prefixing ELF symbol names.
One immediate advantage of this approach is that all subsections matching a pattern can be merged into a single prefixed section, eg. .text and .text.* can be linked into a single '.hyp.text'. This removes the need for -fno-reorder-functions on GCC and will be useful in the future too: LTO builds use .text subsections, compilers routinely generate .rodata subsections, etc.
Partially linking all hyp code into a single object file also makes it easier to analyze.
--- arch/arm64/include/asm/hyp_image.h | 24 ++++++++++++ arch/arm64/kvm/hyp/nvhe/.gitignore | 2 + arch/arm64/kvm/hyp/nvhe/Makefile | 60 ++++++++++++++++-------------- arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 13 +++++++ 4 files changed, 72 insertions(+), 27 deletions(-) create mode 100644 arch/arm64/include/asm/hyp_image.h create mode 100644 arch/arm64/kvm/hyp/nvhe/.gitignore create mode 100644 arch/arm64/kvm/hyp/nvhe/hyp.lds.S
arch/arm64/include/asm/hyp_image.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (C) 2020 Google LLC. +* Written by David Brazdil <[email protected]> +*/ + +#ifndef __ARM64_HYP_IMAGE_H__ +#define __ARM64_HYP_IMAGE_H__ + +#ifdef LINKER_SCRIPT + +/* +* KVM nVHE ELF section names are prefixed with .hyp, to separate them +* from the kernel proper. +*/ +#define HYP_SECTION_NAME (NAME ) .hyp##NAME + +/* Defines an ELF hyp section from input section @NAME and its subsections. */ +#define HYP_SECTION (NAME ) \ + HYP_SECTION_NAME(NAME) : { *(NAME NAME##.*) } + +#endif /* LINKER_SCRIPT */ + +#endif /* __ARM64_HYP_IMAGE_H__ */
arch/arm64/kvm/hyp/nvhe/.gitignore new file +2 -0
@@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0-only +hyp.lds
arch/arm64/kvm/hyp/nvhe/Makefile +33 -27
73 lines not on this page · raw message
arch/arm64/kvm/hyp/nvhe/hyp.lds.S new file +13 -0
@@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2020 Google LLC. + * Written by David Brazdil <[email protected]> + * + * Linker script used for partial linking of nVHE EL2 object files. + */ + +#include <asm/hyp_image.h> + +SECTIONS { + HYP_SECTION(.text) +}
Signature · 6 lines
2.28.0.681.g6f77f65b4e-goog _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v4 2/10 kvm: arm64: Move nVHE hyp namespace macros to hyp_image.h
In reply to David BrazdilMinor cleanup to move all macros related to prefixing nVHE hyp section and symbol names into one place: hyp_image.h.
--- arch/arm64/include/asm/hyp_image.h | 12 ++++++++++++ arch/arm64/include/asm/kvm_asm.h | 8 +------- arch/arm64/kernel/image-vars.h | 2 -- arch/arm64/kernel/vmlinux.lds.S | 1 + 4 files changed, 14 insertions(+), 9 deletions(-)
arch/arm64/include/asm/hyp_image.h +12 -0
@@ -7,6 +7,12 @@#ifndef __ARM64_HYP_IMAGE_H__#define __ARM64_HYP_IMAGE_H__ +/* +* KVM nVHE code has its own symbol namespace prefixed with __kvm_nvhe_, +* to separate it from the kernel proper. +*/ +#define kvm_nvhe_sym (sym) __kvm_nvhe_##sym +#ifdef LINKER_SCRIPT /*
@@ -19,6 +25,12 @@#define HYP_SECTION (NAME ) \ HYP_SECTION_NAME(NAME) : { *(NAME NAME##.*) } +/* +* Defines a linker script alias of a kernel-proper symbol referenced by +* KVM nVHE hyp code. +*/ +#define KVM_NVHE_ALIAS (sym) kvm_nvhe_sym(sym) = sym; +#endif /* LINKER_SCRIPT */ #endif /* __ARM64_HYP_IMAGE_H__ */
arch/arm64/include/asm/kvm_asm.h +1 -7
@@ -7,6 +7,7 @@#ifndef __ARM_KVM_ASM_H__#define __ARM_KVM_ASM_H__ +#include <asm/hyp_image.h> #include <asm/virt.h> #define VCPU_WORKAROUND_2_FLAG_SHIFT 0
@@ -42,13 +43,6 @@#include <linux/mm.h> -/* -* Translate name of a symbol defined in nVHE hyp to the name seen -* by kernel proper. All nVHE symbols are prefixed by the build system -* to avoid clashes with the VHE variants. -*/ -#define kvm_nvhe_sym (sym) __kvm_nvhe_##sym -#define DECLARE_KVM_VHE_SYM (sym) extern char sym[]#define DECLARE_KVM_NVHE_SYM (sym) extern char kvm_nvhe_sym(sym)[]
arch/arm64/kernel/image-vars.h +0 -2
@@ -61,8 +61,6 @@ __efistub__ctype = _ctype;* memory mappings. */ -#define KVM_NVHE_ALIAS (sym) __kvm_nvhe_##sym = sym; -/* Alternative callbacks for init-time patching of nVHE hyp code. */ KVM_NVHE_ALIAS (arm64_enable_wa2_handling);KVM_NVHE_ALIAS (kvm_patch_vector_branch);
arch/arm64/kernel/vmlinux.lds.S +1 -0
@@ -9,6 +9,7 @@ #include <asm-generic/vmlinux.lds.h> #include <asm/cache.h> +#include <asm/hyp_image.h> #include <asm/kernel-pgtable.h> #include <asm/memory.h> #include <asm/page.h>
Signature · 6 lines
2.28.0.681.g6f77f65b4e-goog _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v4 3/10 kvm: arm64: Only define __kvm_ex_table for CONFIG_KVM
In reply to David BrazdilMinor cleanup that only creates __kvm_ex_table ELF section and related symbols if CONFIG_KVM is enabled. Also useful as more hyp-specific sections will be added.
--- arch/arm64/kernel/vmlinux.lds.S | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm64/kernel/vmlinux.lds.S +4 -0
@@ -22,11 +22,15 @@ ENTRY(_text) jiffies = jiffies_64; +#ifdef CONFIG_KVM #define HYPERVISOR_EXTABLE \ . = ALIGN(SZ_8); \ __start___kvm_ex_table = .; \ *(__kvm_ex_table) \ __stop___kvm_ex_table = .; +#else /* CONFIG_KVM */ +#define HYPERVISOR_EXTABLE +#endif #define HYPERVISOR_TEXT \ /* \
Signature · 6 lines
2.28.0.681.g6f77f65b4e-goog _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v4 4/10 kvm: arm64: Remove __hyp_this_cpu_read
In reply to David Brazdilthis_cpu_ptr is meant for use in kernel proper because it selects between TPIDR_EL1/2 based on nVHE/VHE. __hyp_this_cpu_ptr was used in hyp to always select TPIDR_EL2. Unify all users behind this_cpu_ptr and friends by selecting _EL2 register under __KVM_NVHE_HYPERVISOR__. VHE continues selecting the register using alternatives.
Under CONFIG_DEBUG_PREEMPT, the kernel helpers perform a preemption check which is omitted by the hyp helpers. Preserve the behavior for nVHE by overriding the corresponding macros under __KVM_NVHE_HYPERVISOR__. Extend the checks into VHE hyp code.
--- arch/arm64/include/asm/kvm_asm.h | 20 ---------------- arch/arm64/include/asm/percpu.h | 28 +++++++++++++++++++++-- arch/arm64/kvm/hyp/include/hyp/debug-sr.h | 4 ++-- arch/arm64/kvm/hyp/include/hyp/switch.h | 8 +++---- arch/arm64/kvm/hyp/nvhe/switch.c | 2 +- arch/arm64/kvm/hyp/vhe/switch.c | 2 +- arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 4 ++-- 7 files changed, 36 insertions(+), 32 deletions(-)
arch/arm64/include/asm/kvm_asm.h +0 -20
@@ -143,26 +143,6 @@ extern char __smccc_workaround_1_smc[__SMCCC_WORKAROUND_1_SMC_SZ]; addr; \ }) -/* -* Home-grown __this_cpu_{ptr,read} variants that always work at HYP, -* provided that sym is really a *symbol* and not a pointer obtained from -* a data structure. As for SHIFT_PERCPU_PTR(), the creative casting keeps -* sparse quiet. -*/ -#define __hyp_this_cpu_ptr (sym) \ - ({ \ - void *__ptr; \ - __verify_pcpu_ptr(&sym); \ - __ptr = hyp_symbol_addr(sym); \ - __ptr += read_sysreg(tpidr_el2); \ - (typeof(sym) __kernel __force *)__ptr; \ - }) - -#define __hyp_this_cpu_read (sym) \ - ({ \ - *__hyp_this_cpu_ptr(sym); \ - }) -#define __KVM_EXTABLE (from, to) \ " .pushsection __kvm_ex_table, \"a\"\n" \ " .align 3\n" \
arch/arm64/include/asm/percpu.h +26 -2
@@ -19,7 +19,16 @@ static inline void set_my_cpu_offset(unsigned long off) :: "r" (off) : "memory"); } -static inline unsigned long __my_cpu_offset (void ) +static inline unsigned long __hyp_my_cpu_offset (void ) +{ +/* +* Non-VHE hyp code runs with preemption disabled. No need to hazard +* the register access against barrier() as in __kern_my_cpu_offset. +*/ +return read_sysreg (tpidr_el2); +} + +static inline unsigned long __kern_my_cpu_offset (void ) {unsigned long off;
@@ -35,7 +44,12 @@ static inline unsigned long __my_cpu_offset(void)return off; } -#define __my_cpu_offset __my_cpu_offset() + +#ifdef __KVM_NVHE_HYPERVISOR__ +#define __my_cpu_offset __hyp_my_cpu_offset() +#else +#define __my_cpu_offset __kern_my_cpu_offset() +#endif #define PERCPU_RW_OPS (sz) \ static inline unsigned long __percpu_read_##sz(void *ptr) \
@@ -227,4 +241,14 @@ PERCPU_RET_OP(add, add, ldadd)#include <asm-generic/percpu.h> +/* Redefine macros for nVHE hyp under DEBUG_PREEMPT to avoid its dependencies. */ +#if defined(__KVM_NVHE_HYPERVISOR__) && defined(CONFIG_DEBUG_PREEMPT ) +#undef this_cpu_ptr +#define this_cpu_ptr raw_cpu_ptr +#undef __this_cpu_read +#define __this_cpu_read raw_cpu_read +#undef __this_cpu_write +#define __this_cpu_write raw_cpu_write +#endif +#endif /* __ASM_PERCPU_H */
arch/arm64/kvm/hyp/include/hyp/debug-sr.h +2 -2
@@ -135,7 +135,7 @@ static inline void __debug_switch_to_guest_common(struct kvm_vcpu *vcpu)if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ; - host_ctxt = &__hyp_this_cpu_ptr (kvm_host_data)->host_ctxt; + host_ctxt = &this_cpu_ptr (&kvm_host_data)->host_ctxt; guest_ctxt = &vcpu->arch.ctxt; host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg =kern_hyp_va (vcpu->arch.debug_ptr);
@@ -154,7 +154,7 @@ static inline void __debug_switch_to_host_common(struct kvm_vcpu *vcpu)if (!(vcpu->arch.flags &KVM_ARM64_DEBUG_DIRTY ))return ; - host_ctxt = &__hyp_this_cpu_ptr (kvm_host_data)->host_ctxt; + host_ctxt = &this_cpu_ptr (&kvm_host_data)->host_ctxt; guest_ctxt = &vcpu->arch.ctxt; host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg =kern_hyp_va (vcpu->arch.debug_ptr);
arch/arm64/kvm/hyp/include/hyp/switch.h +4 -4
@@ -386,7 +386,7 @@ static inline bool __hyp_handle_ptrauth(struct kvm_vcpu *vcpu) !esr_is_ptrauth_trap(kvm_vcpu_get_esr(vcpu)))return false; - ctxt = &__hyp_this_cpu_ptr (kvm_host_data)->host_ctxt; + ctxt = &this_cpu_ptr (&kvm_host_data)->host_ctxt;__ptrauth_save_key (ctxt,APIA );__ptrauth_save_key (ctxt,APIB );__ptrauth_save_key (ctxt,APDA );
@@ -495,7 +495,7 @@ static inline void __set_guest_arch_workaround_state(struct kvm_vcpu *vcpu)* guest wants it disabled, so be it... */ if (__needs_ssbd_off (vcpu) && -__hyp_this_cpu_read (arm64_ssbd_callback_required)) +__this_cpu_read (arm64_ssbd_callback_required))arm_smccc_1_1_smc (ARM_SMCCC_ARCH_WORKAROUND_2 ,0 ,NULL );#endif }
@@ -507,7 +507,7 @@ static inline void __set_host_arch_workaround_state(struct kvm_vcpu *vcpu)* If the guest has disabled the workaround, bring it back on. */ if (__needs_ssbd_off (vcpu) && -__hyp_this_cpu_read (arm64_ssbd_callback_required)) +__this_cpu_read (arm64_ssbd_callback_required))arm_smccc_1_1_smc (ARM_SMCCC_ARCH_WORKAROUND_2 ,1 ,NULL );#endif }
@@ -521,7 +521,7 @@ static inline void __kvm_unexpected_el2_exception(void) entry =hyp_symbol_addr (__start___kvm_ex_table); end =hyp_symbol_addr (__stop___kvm_ex_table); - host_ctxt = &__hyp_this_cpu_ptr (kvm_host_data)->host_ctxt; + host_ctxt = &this_cpu_ptr (&kvm_host_data)->host_ctxt;while (entry < end) { addr = (unsigned long )&entry->insn + entry->insn;
arch/arm64/kvm/hyp/nvhe/switch.c +1 -1
@@ -175,7 +175,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) vcpu =kern_hyp_va (vcpu); - host_ctxt = &__hyp_this_cpu_ptr (kvm_host_data)->host_ctxt; + host_ctxt = &this_cpu_ptr (&kvm_host_data)->host_ctxt; host_ctxt->__hyp_running_vcpu = vcpu; guest_ctxt = &vcpu->arch.ctxt;
arch/arm64/kvm/hyp/vhe/switch.c +1 -1
@@ -108,7 +108,7 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)struct kvm_cpu_context *guest_ctxt;u64 exit_code; - host_ctxt = &__hyp_this_cpu_ptr (kvm_host_data)->host_ctxt; + host_ctxt = &this_cpu_ptr (&kvm_host_data)->host_ctxt; host_ctxt->__hyp_running_vcpu = vcpu; guest_ctxt = &vcpu->arch.ctxt;
arch/arm64/kvm/hyp/vhe/sysreg-sr.c +2 -2
@@ -66,7 +66,7 @@ void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu)struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;struct kvm_cpu_context *host_ctxt; - host_ctxt = &__hyp_this_cpu_ptr (kvm_host_data)->host_ctxt; + host_ctxt = &this_cpu_ptr (&kvm_host_data)->host_ctxt;__sysreg_save_user_state (host_ctxt);/*
@@ -100,7 +100,7 @@ void kvm_vcpu_put_sysregs_vhe(struct kvm_vcpu *vcpu)struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;struct kvm_cpu_context *host_ctxt; - host_ctxt = &__hyp_this_cpu_ptr (kvm_host_data)->host_ctxt; + host_ctxt = &this_cpu_ptr (&kvm_host_data)->host_ctxt;deactivate_traps_vhe_put ();__sysreg_save_el1_state (guest_ctxt);
Signature · 6 lines
2.28.0.681.g6f77f65b4e-goog _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v4 04/10] kvm: arm64: Remove __hyp_this_cpu_read
Quoted 13 lines · David Brazdil
this_cpu_ptris meant for use in kernel proper because it selects between TPIDR_EL1/2 based on nVHE/VHE.__hyp_this_cpu_ptrwas used in hyp to always selectTPIDR_EL2. Unify all users behindthis_cpu_ptrand friends by selecting_EL2register under__KVM_NVHE_HYPERVISOR__. VHE continues selecting the register using alternatives.Under
CONFIG_DEBUG_PREEMPT, the kernel helpers perform a preemption check which is omitted by the hyp helpers. Preserve the behavior for nVHE by overriding the corresponding macros under__KVM_NVHE_HYPERVISOR__. Extend the checks into VHE hyp code.Acked-by Andrew Scull <[email protected]>Signed-off-by David Brazdil <[email protected]>
Quoted 9 lines
--- arch/arm64/include/asm/kvm_asm.h | 20 ---------------- arch/arm64/include/asm/percpu.h | 28 +++++++++++++++++++++-- arch/arm64/kvm/hyp/include/hyp/debug-sr.h | 4 ++-- arch/arm64/kvm/hyp/include/hyp/switch.h | 8 +++---- arch/arm64/kvm/hyp/nvhe/switch.c | 2 +- arch/arm64/kvm/hyp/vhe/switch.c | 2 +- arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 4 ++-- 7 files changed, 36 insertions(+), 32 deletions(-)
Looks good, thanks for the respin!
Will
PATCH v4 5/10 kvm: arm64: Remove hyp_adr/ldr_this_cpu
In reply to David BrazdilThe hyp_adr/ldr_this_cpu helpers were introduced for use in hyp code because they always needed to use TPIDR_EL2 for base, while adr/ldr_this_cpu from kernel proper would select between TPIDR_EL2 and _EL1 based on VHE/nVHE.
Simplify this now that the hyp mode case can be handled using the __KVM_VHE/NVHE_HYPERVISOR__ macros.
--- arch/arm64/include/asm/assembler.h | 29 +++++++++++++++++++---------- arch/arm64/include/asm/kvm_asm.h | 14 +------------- arch/arm64/kvm/hyp/hyp-entry.S | 2 +- 3 files changed, 21 insertions(+), 24 deletions(-)
arch/arm64/include/asm/assembler.h +19 -10
@@ -218,6 +218,23 @@ lr .req x30 // link register str \src, [\tmp, :lo12:\sym] .endm + /* + * @dst: destination register (32 or 64 bit wide) + */ +#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__) + .macro this_cpu_offset, dst + mrs \dst, tpidr_el2 + .endm +#else + .macro this_cpu_offset, dst +alternative_if_not ARM64_HAS_VIRT_HOST_EXTN + mrs \dst, tpidr_el1 +alternative_else + mrs \dst, tpidr_el2 +alternative_endif + .endm +#endif +/* * @dst: Result of per_cpu(sym, smp_processor_id()) (can be SP) * @sym: The name of the per-cpu variable
@@ -226,11 +243,7 @@ lr .req x30 // link register .macro adr_this_cpu, dst, sym, tmp adrp \tmp, \sym add \dst, \tmp, #:lo12:\sym -alternative_if_not ARM64_HAS_VIRT_HOST_EXTN - mrs \tmp, tpidr_el1 -alternative_else - mrs \tmp, tpidr_el2 -alternative_endif + this_cpu_offset \tmp add \dst, \dst, \tmp .endm
@@ -241,11 +254,7 @@ alternative_endif*/ .macro ldr_this_cpu dst, sym, tmp adr_l \dst, \sym -alternative_if_not ARM64_HAS_VIRT_HOST_EXTN - mrs \tmp, tpidr_el1 -alternative_else - mrs \tmp, tpidr_el2 -alternative_endif + this_cpu_offset \tmp ldr \dst, [\dst, \tmp] .endm
arch/arm64/include/asm/kvm_asm.h +1 -13
@@ -173,20 +173,8 @@ extern char __smccc_workaround_1_smc[__SMCCC_WORKAROUND_1_SMC_SZ];#else /* __ASSEMBLY__ */ -.macro hyp_adr_this_cpu reg, sym, tmp - adr_l \reg, \sym - mrs \tmp, tpidr_el2 - add \reg, \reg, \tmp -.endm - -.macro hyp_ldr_this_cpu reg, sym, tmp - adr_l \reg, \sym - mrs \tmp, tpidr_el2 - ldr \reg, [\reg, \tmp] -.endm - .macro get_host_ctxt reg, tmp - hyp_adr_this_cpu \reg, kvm_host_data, \tmp + adr_this_cpu \reg, kvm_host_data, \tmp add \reg, \reg, #HOST_DATA_CONTEXT .endm
arch/arm64/kvm/hyp/hyp-entry.S +1 -1
@@ -132,7 +132,7 @@ alternative_cb_end str x0, [x2, #VCPU_WORKAROUND_FLAGS] /* Check that we actually need to perform the call */ - hyp_ldr_this_cpu x0, arm64_ssbd_callback_required, x2 + ldr_this_cpu x0, arm64_ssbd_callback_required, x2 cbz x0, wa2_end mov w0, #ARM_SMCCC_ARCH_WORKAROUND_2
Signature · 6 lines
2.28.0.681.g6f77f65b4e-goog _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Will Deacon
maintainerRe: [PATCH v4 05/10] kvm: arm64: Remove hyp_adr/ldr_this_cpu
Quoted 22 lines · David Brazdil
The hyp_adr/ldr_this_cpu helpers were introduced for use in hyp code because they always needed to use
TPIDR_EL2for base, while adr/ldr_this_cpu from kernel proper would select betweenTPIDR_EL2and_EL1based on VHE/nVHE.Simplify this now that the hyp mode case can be handled using the __KVM_VHE/NVHE_HYPERVISOR__ macros.
Acked-by Andrew Scull <[email protected]>Acked-by Will Deacon <[email protected]>Signed-off-by David Brazdil <[email protected]>--- arch/arm64/include/asm/assembler.h | 29 +++++++++++++++++++---------- arch/arm64/include/asm/kvm_asm.h | 14 +------------- arch/arm64/kvm/hyp/hyp-entry.S | 2 +- 3 files changed, 21 insertions(+), 24 deletions(-)arch/arm64/include/asm/assembler.h +0
-0
arch/arm64/include/asm/assembler.h lr .req x30 // link register · line 222
Quoted 5 lines
arch/arm64/include/asm/assembler.h +2
-0@@ -218,6 +218,23 @@ lr .req x30 // link register str \src, [\tmp, :lo12:\sym] .endm +/* +* @dst: destination register (32 or 64 bit wide)
nit: this comment is wrong as I don't think mrs can take a W register as the destination argument. I'm assuming Marc can fix that up.
Will
Marc Zyngier
maintainerRe: [PATCH v4 05/10] kvm: arm64: Remove hyp_adr/ldr_this_cpu
Quoted 26 lines · Will Deacon
Quoted 25 lines · David Brazdil
The hyp_adr/ldr_this_cpu helpers were introduced for use in hyp code because they always needed to use
TPIDR_EL2for base, while adr/ldr_this_cpu from kernel proper would select betweenTPIDR_EL2and_EL1based on VHE/nVHE.Simplify this now that the hyp mode case can be handled using the __KVM_VHE/NVHE_HYPERVISOR__ macros.
Acked-by Andrew Scull <[email protected]>Acked-by Will Deacon <[email protected]>Signed-off-by David Brazdil <[email protected]>--- arch/arm64/include/asm/assembler.h | 29 +++++++++++++++++++---------- arch/arm64/include/asm/kvm_asm.h | 14 +------------- arch/arm64/kvm/hyp/hyp-entry.S | 2 +- 3 files changed, 21 insertions(+), 24 deletions(-)arch/arm64/include/asm/assembler.h +0
-0@@ -218,6 +218,23 @@ lr .req x30 // link register str \src, [\tmp, :lo12:\sym] .endm
Quoted 5 lines
arch/arm64/include/asm/assembler.h lr .req x30 // link register · line 222
Quoted 2 lines
arch/arm64/include/asm/assembler.h +2
-0+/* +* @dst: destination register (32 or 64 bit wide) nit: this comment is wrong as I don't think mrs can take a W register as the destination argument. I'm assuming Marc can fix that up.
Indeed. I'll fix it locally.
Another thing is that this patch is going to clash with the Ghostbuster branch (the hyp-entry.S hunk goes), but we can deal with that.
M.
Signature · 5 lines
Jazz is not dead. It just smells funny... _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v4 6/10 kvm: arm64: Add helpers for accessing nVHE hyp per-cpu vars
In reply to David BrazdilDefining a per-CPU variable in hyp/nvhe will result in its name being prefixed with __kvm_nvhe_. Add helpers for declaring these variables in kernel proper and accessing them with this_cpu_ptr and per_cpu_ptr.
--- arch/arm64/include/asm/kvm_asm.h | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_asm.h +23 -2
@@ -54,9 +54,21 @@DECLARE_KVM_VHE_SYM (sym); \ DECLARE_KVM_NVHE_SYM(sym) +#define DECLARE_KVM_VHE_PER_CPU (type, sym) \ + DECLARE_PER_CPU(type, sym) +#define DECLARE_KVM_NVHE_PER_CPU (type, sym) \ + DECLARE_PER_CPU(type, kvm_nvhe_sym(sym)) + +#define DECLARE_KVM_HYP_PER_CPU (type, sym) \ + DECLARE_KVM_VHE_PER_CPU(type, sym); \ + DECLARE_KVM_NVHE_PER_CPU(type, sym) +#define CHOOSE_VHE_SYM (sym) sym#define CHOOSE_NVHE_SYM (sym) kvm_nvhe_sym(sym) +#define this_cpu_ptr_nvhe_sym (sym) this_cpu_ptr(&kvm_nvhe_sym(sym)) +#define per_cpu_ptr_nvhe_sym (sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu) +#ifndef __KVM_NVHE_HYPERVISOR__/* * BIG FAT WARNINGS:
@@ -69,12 +81,21 @@* - Don't let the nVHE hypervisor have access to this, as it will * pick the *wrong* symbol (yes, it runs at EL2...). */ -#define CHOOSE_HYP_SYM (sym) (is_kernel_in_hyp_mode() ? CHOOSE_VHE_SYM(sym) \ +#define CHOOSE_HYP_SYM (sym) (is_kernel_in_hyp_mode() \ + ? CHOOSE_VHE_SYM(sym) \ : CHOOSE_NVHE_SYM(sym)) +#define this_cpu_ptr_hyp_sym (sym) (is_kernel_in_hyp_mode() \ + ? this_cpu_ptr(&sym) \ + : this_cpu_ptr_nvhe_sym(sym)) +#define per_cpu_ptr_hyp_sym (sym, cpu) (is_kernel_in_hyp_mode() \ + ? per_cpu_ptr(&sym, cpu) \ + : per_cpu_ptr_nvhe_sym(sym, cpu))#else /* The nVHE hypervisor shouldn't even try to access anything */ extern void *__nvhe_undefined_symbol; -#define CHOOSE_HYP_SYM (sym) __nvhe_undefined_symbol +#define CHOOSE_HYP_SYM (sym) __nvhe_undefined_symbol +#define this_cpu_ptr_hyp_sym (sym) (&__nvhe_undefined_symbol) +#define per_cpu_ptr_hyp_sym (sym, cpu) (&__nvhe_undefined_symbol)#endif /* Translate a kernel address @ptr into its equivalent linear mapping */
Signature · 6 lines
2.28.0.681.g6f77f65b4e-goog _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v4 7/10 kvm: arm64: Duplicate arm64_ssbd_callback_required for nVHE hyp
In reply to David BrazdilHyp keeps track of which cores require SSBD callback by accessing a kernel-proper global variable. Create an nVHE symbol of the same name and copy the value from kernel proper to nVHE as KVM is being enabled on a core.
Done in preparation for separating percpu memory owned by kernel proper and nVHE.
--- arch/arm64/include/asm/kvm_mmu.h | 14 +++++++++++++- arch/arm64/kernel/image-vars.h | 1 - arch/arm64/kvm/arm.c | 3 +++ arch/arm64/kvm/hyp/nvhe/switch.c | 3 +++ 4 files changed, 19 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_mmu.h +13 -1
@@ -529,6 +529,7 @@ static inline int kvm_map_vectors(void)#ifdef CONFIG_ARM64_SSBD DECLARE_PER_CPU_READ_MOSTLY (u64, arm64_ssbd_callback_required); +DECLARE_KVM_NVHE_PER_CPU (u64, arm64_ssbd_callback_required);static inline int hyp_map_aux_data (void ) {
@@ -537,18 +538,29 @@ static inline int hyp_map_aux_data(void)for_each_possible_cpu (cpu) {u64 *ptr; - ptr =per_cpu_ptr (&arm64_ssbd_callback_required, cpu); + ptr =per_cpu_ptr_nvhe_sym (arm64_ssbd_callback_required, cpu); err =create_hyp_mappings (ptr, ptr +1 ,PAGE_HYP );if (err)return err; } return 0; } + +static inline void hyp_init_aux_data (void ) +{ +u64 *ptr; + +/* Copy arm64_ssbd_callback_required value from kernel to hyp. */ + ptr =this_cpu_ptr_nvhe_sym (arm64_ssbd_callback_required); + *ptr =__this_cpu_read (arm64_ssbd_callback_required); +}#else static inline int hyp_map_aux_data (void ) {return 0 ; } + +static inline void hyp_init_aux_data (void ) {}#endif #define kvm_phys_to_vttbr (addr) phys_to_ttbr(addr)
arch/arm64/kernel/image-vars.h +0 -1
@@ -67,7 +67,6 @@ KVM_NVHE_ALIAS(kvm_patch_vector_branch);KVM_NVHE_ALIAS (kvm_update_va_mask);/* Global kernel state accessed by nVHE hyp code. */ -KVM_NVHE_ALIAS (arm64_ssbd_callback_required);KVM_NVHE_ALIAS (kvm_host_data);KVM_NVHE_ALIAS (kvm_vgic_global_state);
arch/arm64/kvm/arm.c +3 -0
@@ -1298,6 +1298,9 @@ static void cpu_init_hyp_mode(void) arm64_get_ssbd_state() == ARM64_SSBD_FORCE_DISABLE) {kvm_call_hyp_nvhe (__kvm_enable_ssbs); } + +/* Copy information whether SSBD callback is required to hyp. */ +hyp_init_aux_data (); }static void cpu_hyp_reset (void )
arch/arm64/kvm/hyp/nvhe/switch.c +3 -0
@@ -27,6 +27,9 @@#include <asm/processor.h> #include <asm/thread_info.h> +/* Non-VHE copy of the kernel symbol. */ +DEFINE_PER_CPU_READ_MOSTLY (u64, arm64_ssbd_callback_required); +static void __activate_traps (struct kvm_vcpu *vcpu) {u64 val;
Signature · 6 lines
2.28.0.681.g6f77f65b4e-goog _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v4 07/10] kvm: arm64: Duplicate arm64_ssbd_callback_required for nVHE hyp
Quoted 8 lines · David Brazdil
Hyp keeps track of which cores require SSBD callback by accessing a kernel-proper global variable. Create an nVHE symbol of the same name and copy the value from kernel proper to nVHE as KVM is being enabled on a core.
Done in preparation for separating percpu memory owned by kernel proper and nVHE.
Quoted 7 lines
Signed-off-by David Brazdil <[email protected]>--- arch/arm64/include/asm/kvm_mmu.h | 14 +++++++++++++- arch/arm64/kernel/image-vars.h | 1 - arch/arm64/kvm/arm.c | 3 +++ arch/arm64/kvm/hyp/nvhe/switch.c | 3 +++ 4 files changed, 19 insertions(+), 2 deletions(-)
Marc: please take a look at for-next/ghostbusters on the arm64 tree, as that has the patches which remove this stuff entirely. I had to rebase the branch today because I screwed up some of the SoBs but the HEAD is now stable at 780c083a8f84.
Will
PATCH v4 8/10 kvm: arm64: Create separate instances of kvm_host_data for VHE/nVHE
In reply to David BrazdilHost CPU context is stored in a global per-cpu variable kvm_host_data. In preparation for introducing independent per-CPU region for nVHE hyp, create two separate instances of kvm_host_data, one for VHE and one for nVHE.
--- arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/kernel/image-vars.h | 1 - arch/arm64/kvm/arm.c | 5 ++--- arch/arm64/kvm/hyp/nvhe/switch.c | 3 +++ arch/arm64/kvm/hyp/vhe/switch.c | 3 +++ arch/arm64/kvm/pmu.c | 8 ++++---- 6 files changed, 13 insertions(+), 9 deletions(-)
arch/arm64/include/asm/kvm_host.h +1 -1
@@ -565,7 +565,7 @@ void kvm_set_sei_esr(struct kvm_vcpu *vcpu, u64 syndrome);struct kvm_vcpu *kvm_mpidr_to_vcpu (struct kvm *kvm,unsigned long mpidr); -DECLARE_PER_CPU (kvm_host_data_t, kvm_host_data); +DECLARE_KVM_HYP_PER_CPU (kvm_host_data_t, kvm_host_data);static inline void kvm_init_host_cpu_context (struct kvm_cpu_context *cpu_ctxt) {
arch/arm64/kernel/image-vars.h +0 -1
@@ -67,7 +67,6 @@ KVM_NVHE_ALIAS(kvm_patch_vector_branch);KVM_NVHE_ALIAS (kvm_update_va_mask);/* Global kernel state accessed by nVHE hyp code. */ -KVM_NVHE_ALIAS (kvm_host_data);KVM_NVHE_ALIAS (kvm_vgic_global_state);/* Kernel constant needed to compute idmap addresses. */
arch/arm64/kvm/arm.c +2 -3
@@ -46,7 +46,6 @@ __asm__(".arch_extension virt" );#endif -DEFINE_PER_CPU (kvm_host_data_t, kvm_host_data); static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);/* The VMID used in the VTTBR */
@@ -1311,7 +1310,7 @@ static void cpu_hyp_reset(void)static void cpu_hyp_reinit (void ) { -kvm_init_host_cpu_context (&this_cpu_ptr (&kvm_host_data)->host_ctxt); +kvm_init_host_cpu_context (&this_cpu_ptr_hyp_sym (kvm_host_data)->host_ctxt);cpu_hyp_reset ();
@@ -1546,7 +1545,7 @@ static int init_hyp_mode(void)for_each_possible_cpu (cpu) {kvm_host_data_t *cpu_data; - cpu_data =per_cpu_ptr (&kvm_host_data, cpu); + cpu_data =per_cpu_ptr_hyp_sym (kvm_host_data, cpu); err =create_hyp_mappings (cpu_data, cpu_data +1 ,PAGE_HYP );if (err) {
arch/arm64/kvm/hyp/nvhe/switch.c +3 -0
@@ -30,6 +30,9 @@/* Non-VHE copy of the kernel symbol. */ DEFINE_PER_CPU_READ_MOSTLY (u64, arm64_ssbd_callback_required); +/* Non-VHE instance of kvm_host_data. */ +DEFINE_PER_CPU (kvm_host_data_t, kvm_host_data); +static void __activate_traps (struct kvm_vcpu *vcpu) {u64 val;
arch/arm64/kvm/hyp/vhe/switch.c +3 -0
@@ -28,6 +28,9 @@const char __hyp_panic_string[] ="HYP panic:\nPS:%08llx PC:%016llx ESR:%08llx\nFAR:%016llx HPFAR:%016llx PAR:%016llx\nVCPU:%p\n" ; +/* VHE instance of kvm_host_data. */ +DEFINE_PER_CPU (kvm_host_data_t, kvm_host_data); +static void __activate_traps (struct kvm_vcpu *vcpu) {u64 val;
arch/arm64/kvm/pmu.c +4 -4
@@ -31,7 +31,7 @@ static bool kvm_pmu_switch_needed(struct perf_event_attr *attr)*/ void kvm_set_pmu_events (u32 set,struct perf_event_attr *attr) { -struct kvm_host_data *ctx =this_cpu_ptr (&kvm_host_data); +struct kvm_host_data *ctx =this_cpu_ptr_hyp_sym (kvm_host_data);if (!kvm_pmu_switch_needed (attr))return ;
@@ -47,7 +47,7 @@ void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr)*/ void kvm_clr_pmu_events (u32 clr) { -struct kvm_host_data *ctx =this_cpu_ptr (&kvm_host_data); +struct kvm_host_data *ctx =this_cpu_ptr_hyp_sym (kvm_host_data); ctx->pmu_events.events_host &= ~clr; ctx->pmu_events.events_guest &= ~clr;
@@ -173,7 +173,7 @@ void kvm_vcpu_pmu_restore_guest(struct kvm_vcpu *vcpu)return ;preempt_disable (); - host =this_cpu_ptr (&kvm_host_data); + host =this_cpu_ptr_hyp_sym (kvm_host_data); events_guest = host->pmu_events.events_guest; events_host = host->pmu_events.events_host;
@@ -193,7 +193,7 @@ void kvm_vcpu_pmu_restore_host(struct kvm_vcpu *vcpu)if (!has_vhe ())return ; - host =this_cpu_ptr (&kvm_host_data); + host =this_cpu_ptr_hyp_sym (kvm_host_data); events_guest = host->pmu_events.events_guest; events_host = host->pmu_events.events_host;
Signature · 6 lines
2.28.0.681.g6f77f65b4e-goog _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v4 9/10 kvm: arm64: Set up hyp percpu data for nVHE
In reply to David BrazdilAdd hyp percpu section to linker script and rename the corresponding ELF sections of hyp/nvhe object files. This moves all nVHE-specific percpu variables to the new hyp percpu section.
Allocate sufficient amount of memory for all percpu hyp regions at global KVM init time and create corresponding hyp mappings.
The base addresses of hyp percpu regions are kept in a dynamically allocated array in the kernel.
Add NULL checks in PMU event-reset code as it may run before KVM memory is initialized.
--- arch/arm64/include/asm/kvm_asm.h | 19 +++++++++-- arch/arm64/kernel/vmlinux.lds.S | 8 +++++ arch/arm64/kvm/arm.c | 55 +++++++++++++++++++++++++++++-- arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 6 ++++ arch/arm64/kvm/pmu.c | 5 ++- 5 files changed, 87 insertions(+), 6 deletions(-)
arch/arm64/include/asm/kvm_asm.h +17 -2
@@ -66,8 +66,19 @@#define CHOOSE_VHE_SYM (sym) sym#define CHOOSE_NVHE_SYM (sym) kvm_nvhe_sym(sym) -#define this_cpu_ptr_nvhe_sym (sym) this_cpu_ptr(&kvm_nvhe_sym(sym)) -#define per_cpu_ptr_nvhe_sym (sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu) +/* +* Compute pointer to a symbol defined in nVHE percpu region. +* Returns NULL if percpu memory has not been allocated yet. +*/ +#define this_cpu_ptr_nvhe_sym (sym) per_cpu_ptr_nvhe_sym(sym, smp_processor_id()) +#define per_cpu_ptr_nvhe_sym (sym, cpu) \ + ({ \ + unsigned long base, off; \ + base = kvm_arm_hyp_percpu_base[cpu]; \ + off = (unsigned long)&CHOOSE_NVHE_SYM(sym) - \ + (unsigned long)&CHOOSE_NVHE_SYM(__per_cpu_start); \ + base ? (typeof(CHOOSE_NVHE_SYM(sym))*)(base + off) : NULL; \ + })#ifndef __KVM_NVHE_HYPERVISOR__/*
@@ -117,6 +128,10 @@ DECLARE_KVM_HYP_SYM(__kvm_hyp_vector);#define __kvm_hyp_init CHOOSE_NVHE_SYM(__kvm_hyp_init)#define __kvm_hyp_vector CHOOSE_HYP_SYM(__kvm_hyp_vector) +extern unsigned long kvm_arm_hyp_percpu_base[NR_CPUS ]; +DECLARE_KVM_NVHE_SYM (__per_cpu_start); +DECLARE_KVM_NVHE_SYM (__per_cpu_end); +#ifdef CONFIG_KVM_INDIRECT_VECTORS extern atomic_t arm64_el2_vector_last_slot;DECLARE_KVM_HYP_SYM (__bp_harden_hyp_vecs);
arch/arm64/kernel/vmlinux.lds.S +8 -0
@@ -28,8 +28,15 @@ jiffies = jiffies_64; __start___kvm_ex_table = .; \ *(__kvm_ex_table) \ __stop___kvm_ex_table = .; + +#define HYPERVISOR_PERCPU_SECTION \ + . = ALIGN(PAGE_SIZE); \ + HYP_SECTION_NAME(.data..percpu) : { \ + *(HYP_SECTION_NAME(.data..percpu)) \ + } #else /* CONFIG_KVM */ #define HYPERVISOR_EXTABLE +#define HYPERVISOR_PERCPU_SECTION #endif #define HYPERVISOR_TEXT \
@@ -195,6 +202,7 @@ SECTIONS } PERCPU_SECTION(L1_CACHE_BYTES) + HYPERVISOR_PERCPU_SECTION .rela.dyn : ALIGN(8) { *(.rela .rela*)
arch/arm64/kvm/arm.c +52 -3
92 lines not on this page · raw message
arch/arm64/kvm/hyp/nvhe/hyp.lds.S +6 -0
@@ -7,7 +7,13 @@ */ #include <asm/hyp_image.h> +#include <asm-generic/vmlinux.lds.h> +#include <asm/cache.h> +#include <asm/memory.h> SECTIONS { HYP_SECTION(.text) + HYP_SECTION_NAME(.data..percpu) : { + PERCPU_INPUT(L1_CACHE_BYTES) + } }
arch/arm64/kvm/pmu.c +4 -1
@@ -33,7 +33,7 @@ void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr) {struct kvm_host_data *ctx =this_cpu_ptr_hyp_sym (kvm_host_data); -if (!kvm_pmu_switch_needed (attr)) +if (!ctx || !kvm_pmu_switch_needed (attr))return ;if (!attr->exclude_host)
@@ -49,6 +49,9 @@ void kvm_clr_pmu_events(u32 clr) {struct kvm_host_data *ctx =this_cpu_ptr_hyp_sym (kvm_host_data); +if (!ctx) +return ; + ctx->pmu_events.events_host &= ~clr; ctx->pmu_events.events_guest &= ~clr; }
Signature · 6 lines
2.28.0.681.g6f77f65b4e-goog _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v4 09/10] kvm: arm64: Set up hyp percpu data for nVHE
Quoted 13 lines · David Brazdil
Add hyp percpu section to linker script and rename the corresponding ELF sections of hyp/nvhe object files. This moves all nVHE-specific percpu variables to the new hyp percpu section.
Allocate sufficient amount of memory for all percpu hyp regions at global KVM init time and create corresponding hyp mappings.
The base addresses of hyp percpu regions are kept in a dynamically allocated array in the kernel.
Add NULL checks in PMU event-reset code as it may run before KVM memory is initialized.
Quoted 8 lines
Signed-off-by David Brazdil <[email protected]>--- arch/arm64/include/asm/kvm_asm.h | 19 +++++++++-- arch/arm64/kernel/vmlinux.lds.S | 8 +++++ arch/arm64/kvm/arm.c | 55 +++++++++++++++++++++++++++++-- arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 6 ++++ arch/arm64/kvm/pmu.c | 5 ++- 5 files changed, 87 insertions(+), 6 deletions(-)
But one comment for Marc below...
Quoted 27 lines
arch/arm64/include/asm/kvm_asm.h +13
-2@@ -66,8 +66,19 @@#define CHOOSE_VHE_SYM (sym) sym#define CHOOSE_NVHE_SYM (sym) kvm_nvhe_sym(sym) -#define this_cpu_ptr_nvhe_sym (sym) this_cpu_ptr(&kvm_nvhe_sym(sym)) -#define per_cpu_ptr_nvhe_sym (sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu) +/* +* Compute pointer to a symbol defined in nVHE percpu region. +* Returns NULL if percpu memory has not been allocated yet. +*/ +#define this_cpu_ptr_nvhe_sym (sym) per_cpu_ptr_nvhe_sym(sym, smp_processor_id()) +#define per_cpu_ptr_nvhe_sym (sym, cpu) \ + ({ \ + unsigned long base, off; \ + base = kvm_arm_hyp_percpu_base[cpu]; \ + off = (unsigned long)&CHOOSE_NVHE_SYM(sym) - \ + (unsigned long)&CHOOSE_NVHE_SYM(__per_cpu_start); \ + base ? (typeof(CHOOSE_NVHE_SYM(sym))*)(base + off) : NULL; \ + })#ifndef __KVM_NVHE_HYPERVISOR__/*
arch/arm64/include/asm/kvm_asm.h DECLARE_KVM_HYP_SYM(__kvm_hyp_vector); · line 137
Quoted 10 lines
arch/arm64/include/asm/kvm_asm.h +4
-0@@ -117,6 +128,10 @@ DECLARE_KVM_HYP_SYM(__kvm_hyp_vector);#define __kvm_hyp_init CHOOSE_NVHE_SYM(__kvm_hyp_init)#define __kvm_hyp_vector CHOOSE_HYP_SYM(__kvm_hyp_vector) +extern unsigned long kvm_arm_hyp_percpu_base[NR_CPUS ]; +DECLARE_KVM_NVHE_SYM (__per_cpu_start); +DECLARE_KVM_NVHE_SYM (__per_cpu_end); +#ifdef CONFIG_KVM_INDIRECT_VECTORS extern atomic_t arm64_el2_vector_last_slot;DECLARE_KVM_HYP_SYM (__bp_harden_hyp_vecs);
The changes in this file will collide quite badly with Andrew's "handler" branch which you've already queued, so you'll probably want to chat with Andrew and David when you get to resolve that. In particular, I think it would be good if the this_cpu_ptr_nvhe_sym() macro only ends up in the EL1 block, since I don't think that its use of smp_processor_id() is safe at EL2. That's not a problem as it stands, as its only used by the host.
Will
PATCH v4 10/10 kvm: arm64: Remove unnecessary hyp mappings
In reply to David BrazdilWith all nVHE per-CPU variables being part of the hyp per-CPU region, mapping them individual is not necessary any longer. They are mapped to hyp as part of the overall per-CPU region.
--- arch/arm64/include/asm/kvm_mmu.h | 20 -------------------- arch/arm64/kvm/arm.c | 16 ---------------- 2 files changed, 36 deletions(-)
arch/arm64/include/asm/kvm_mmu.h +0 -20
@@ -531,21 +531,6 @@ static inline int kvm_map_vectors(void)DECLARE_PER_CPU_READ_MOSTLY (u64, arm64_ssbd_callback_required);DECLARE_KVM_NVHE_PER_CPU (u64, arm64_ssbd_callback_required); -static inline int hyp_map_aux_data (void ) -{ -int cpu, err; - -for_each_possible_cpu (cpu) { -u64 *ptr; - - ptr =per_cpu_ptr_nvhe_sym (arm64_ssbd_callback_required, cpu); - err =create_hyp_mappings (ptr, ptr +1 ,PAGE_HYP ); -if (err) -return err; - } -return 0 ; -} -static inline void hyp_init_aux_data (void ) {u64 *ptr;
@@ -555,11 +540,6 @@ static inline void hyp_init_aux_data(void) *ptr = __this_cpu_read(arm64_ssbd_callback_required); }#else -static inline int hyp_map_aux_data (void ) -{ -return 0 ; -} -static inline void hyp_init_aux_data (void ) {}#endif
arch/arm64/kvm/arm.c +0 -16
@@ -1591,22 +1591,6 @@ static int init_hyp_mode(void) } } -for_each_possible_cpu (cpu) { -kvm_host_data_t *cpu_data; - - cpu_data =per_cpu_ptr_hyp_sym (kvm_host_data, cpu); - err =create_hyp_mappings (cpu_data, cpu_data +1 ,PAGE_HYP ); - -if (err) { -kvm_err ("Cannot map host CPU state: %d\n" , err); - goto out_err; - } - } - - err =hyp_map_aux_data (); -if (err) -kvm_err ("Cannot map host auxiliary data: %d\n" , err); -return 0 ; out_err:
Signature · 6 lines
2.28.0.681.g6f77f65b4e-goog _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v4 10/10] kvm: arm64: Remove unnecessary hyp mappings
Quoted 5 lines · David Brazdil
With all nVHE per-CPU variables being part of the hyp per-CPU region, mapping them individual is not necessary any longer. They are mapped to hyp as part of the overall per-CPU region.
Acked-by Andrew Scull<[email protected]>
^^^ Missing space between "Scull" and "<".
Quoted 5 lines
Signed-off-by David Brazdil <[email protected]>--- arch/arm64/include/asm/kvm_mmu.h | 20 -------------------- arch/arm64/kvm/arm.c | 16 ---------------- 2 files changed, 36 deletions(-)
Will
Quoted 2 lines · David Brazdil
Introduce '.hyp.data..percpu' as part of ongoing effort to make nVHE hyp code self-contained and independent of the rest of the kernel.
The percpu subsystems point is to enable the use of special hardware instructions that can perform address calculation and a memory operation in one interruptible instruction. This is in particular useful to avoid higher overhead for memory management related counters because preempt disable/enable etc can be avoided.
ARM cannot do that and thus has a LC/SC loop.
This is a patchset for ARM64 so its not clear to me what kind of advantage there would be against a simple implementation that does a regular fetch from a base address with an offset.
Quoted 4 lines
Main benefits:
* independent nVHE per-CPU data section that can be unmapped from host,
* more robust linking of nVHE hyp code,
* no need for hyp-specific macros to access per-CPU variables.
Maybe simply don't use percpu variables for your arm code? Those pointers to data will be much more indepedent of the rest of the kernel and allow a much higher degree of being self-contained.
Quoted 9 lines · David Brazdil
Introduce '.hyp.data..percpu' as part of ongoing effort to make nVHE hyp code self-contained and independent of the rest of the kernel.
Main benefits:
* independent nVHE per-CPU data section that can be unmapped from host,
* more robust linking of nVHE hyp code,
* no need for hyp-specific macros to access per-CPU variables.[...]
Applied to next, thanks!
[01/10] kvm: arm64: Partially link nVHE hyp code, simplify HYPCOPY
commit: ab25464bdabd45f283cc1194e332040f89071106
[02/10] kvm: arm64: Move nVHE hyp namespace macros to hyp_image.h
commit: ce492a16ffb8814d9651c3fdafc363bfa1b01189
[03/10] kvm: arm64: Only define __kvm_ex_table for CONFIG_KVM
commit: 3471ee06e33e413d7fa73c1aa3092e6e794b9e05
[04/10] kvm: arm64: Remove __hyp_this_cpu_read
commit: 717cf94adb54095d14a6674baea73123188f2901
[05/10] kvm: arm64: Remove hyp_adr/ldr_this_cpu
commit: ea391027d35546d9155f1350123b5af8bddec706
[06/10] kvm: arm64: Add helpers for accessing nVHE hyp per-cpu vars
commit: 572494995bc3d282336bfd8162741929402910b9
[07/10] kvm: arm64: Duplicate arm64_ssbd_callback_required for nVHE hyp
commit: df4c8214a18d202fa0ec221a001f640e020f7e44
[08/10] kvm: arm64: Create separate instances of kvm_host_data for VHE/nVHE
commit: 2a1198c9b436402582f7beed57028044b819329c
[09/10] kvm: arm64: Set up hyp percpu data for nVHE
commit: 30c953911c4370bfb622ee1c2fcc7e78c84df800
[10/10] kvm: arm64: Remove unnecessary hyp mappings
commit: a3bb9c3a00551726590137e3974495ce6cf6b758Cheers,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.